STABILIZING OFF-POLICY REINFORCEMENT LEARNING WITH CONSERVATIVE POLICY GRADIENTS
Year: 2019

Abstract:
In recent years, advances in deep learning have enabled the application of reinforcement learning algorithms in complex domains. However, they lack the theoretical guarantees which are present in the tabular setting and suffer from many stability and reproducibility problems [10]. In this work, we suggest a simple approach for improving stability in off-policy actor-critic deep reinforcement learning regimes. Experiments on continuous action spaces, in the MuJoCo control suite, show that our proposed method reduces the variance of the process and improves the overall performance across most domains.

heading: 1 INTRODUCTION
text: Reinforcement Learning (RL) is a dynamical learning paradigm, in which the algorithm (also known as the ‘agent’) learns through sequential interaction with the environment. On each round, the agent performs an action, which transitions it into a new state, and is provided with a state-dependent reward. The goal of the agent is to maximize the cumulative reward it observes throughout these interactions with the environment. When considering tabular RL, in which there is a finite number of states and actions, there exist efficient algorithms with theoretical convergence and stability guarantees [29; 13; 14; 34; 5; 4]. However, this is not the case when considering deep RL approaches – when a neural network is used to cope with large state spaces, such as images [19; 11], and/or large action spaces, for example in continuous control [17; 27].
The issue with deep RL is twofold: (i) the optimization process is finite, and as such does not follow many of the requirements for the process to converge, such as decaying learning rates [2], and (ii) the non-linearity of the function approximators (e.g., neural networks) results in a highly non-convex optimization landscape. Due to this, as can be seen in all empirical works [22; 26; 17; 6; 9], the learning process exhibits high variance; during a short time of training, a near-optimal policy may become arbitrarily bad. As a result, in recent years, the stability of RL algorithms has become a major concern of the research community [10].
Instead of tackling the stability problems in the optimization process, many previous works focused on different aspects of the loss function [6; 9; 30]. Then, they usually apply a standard variant of gradient descent algorithm on the modified loss function. While these approaches are capable of finding relatively good policies, they are strictly inferior when compared to their tabular counterparts – the tabular approaches ensure convergence and stability, whereas the practical variants exhibit instability and high variance.
In this work, we suggest a novel approach that improves the stability of actor-critic methods. Specifically, we draw the connection between deep RL approaches and their tabular counterparts and highlight the issues inherent in deep RL algorithms. Through this connection, we propose a solution, in the form of a hierarchical training procedure. As opposed to the tabular case, our approach is not ensured to converge, but rather has stability guarantees, i.e., the stationary distribution of the policies is shown to be stable, meaning that with high probability the performance of the policy improves and does not suffer high degradation. We show, empirically across a range of continuous control tasks in MuJoCo, that our approach indeed results in improved behavior both in terms of stability and overall performance.

heading: 2 BACKGROUND AND NOTATION
text: We consider an infinite-horizon discounted Markov Decision Process [23, MDP]. An MDP is defined as the 5-tuple (S,A, P, r, γ), where S is the state space, A the action space,
Algorithm 1 Policy Iteration 1: Initialize π0 2: k = 0 3: while not converged do 4: while not converged do . Policy Evaluation 5: for s ∈ S do 6: vπk(s) = r(s, πk(s)) + γ ∑ s′∈S P (s, πk(s), s ′)vπk(s′)
7: for s ∈ S do . Policy Improvement 8: πk+1(s) ∈ arg maxa∈A r(s,a) + γ ∑ s′∈S P (s,a, s ′)vπk(s′)
9: k = k + 1 10: return policy πk
Algorithm 2 DDPG, TD3 and our Conservative update 1: Input: algorithm, learning rate α, averaging parameter τ , steps T and exploration N . 2: Initialize policy θ, critics φ1, φ2 and replay buffer R 3: Set θ− = θ, φi− = φ
i, t = 0 4: while t < T do 5: Collect data and append to replay buffer R 6: t = t+ 1 7: {si,ai, ri, si+1} = R.sample() . Training phase 8: if DDPG then 9: φ = φ− α∇φ||ri + γQφ−(si+1, πθ−(si+1))−Qφ(si,ai)||22 10: else . TD3 11: φ1 = φ1 − α∇φ1 ||ri + γminiQφi−(si+1, πθ−(si+1))−Qφ1(si,ai)|| 2 2 12: φ2 = φ2 − α∇φ2 ||ri + γminiQφi−(si+1, πθ−(si+1))−Qφ2(si,ai)|| 2 2

heading: 13: θ = θ + α∇θQφ1−(si, πθ(si))
text: 14: φi− = (1− τ)φi− + τφi . Update target networks 15: if Conservative Update then 16: if t%Efreq == 0 then 17: evaluate πθ and πθ− for Enum episodes each 18: if P (Jπθ > J π̂θ− ) ≥ 1− p then 19: θ− = θ 20: else 21: θ− = (1− τ)θ− + τθ
Colored lines represent similarities between the classic and modern approach.
P : S × S ×A 7→ [0, 1] is a transition kernel, r : S ×A→ [rmin, rmax] is a reward function and γ ∈ (0, 1) is the discount factor. The goal of an RL agent is to learn a policy π(s,a) that maps states into a distribution over actions. The quality of a policy is measured by its value, vπ(s) = Eπ[ ∑∞ t=0 γ
tr(st,at) | s0 = s], which is the average cumulative reward when starting from state s and acting according to π. Another important quantity is the Q-function, Qπ(s,a) = Eπ[ ∑∞ t=0 γ
tr(st,at) | s0 = s,a0 = a], which is strongly connected to the value through the relation vπ(s) = ∫ a Qπ(s,a)π(s,a)da. We denote an optimal policy that maximizes the value simultaneously across all of the states by π∗ ∈ arg maxπ vπ and the optimal value function by v∗ = vπ ∗ . When the Q-function and the policy π are represented by a parametric function, we denote them by Qφ and πθ, where φ and θ are the parameters of the function (e.g. the parameters of a deep neural network).
While our goal is to maximize vπ , in practice we measure an empirical estimation of Es∼ρvπ(s), where ρ is the initial state distribution. To accommodate for this fact, we define the cumulative reward of a sampled trajectory by Jπ . This is a random variable, as randomness may occur due to (i) the initial state distribution, (ii) the transition kernel and reward function and (iii) the policy.

heading: 2.1 CLASSIC REINFORCEMENT LEARNING ALGORITHMS
text: Policy Iteration [12, PI] is an algorithm for solving tabular MDPs with convergence guarantees. PI (Algorithm 1) iterates between two steps: (i) policy evaluation in which the current policy πk is evaluated, producing the value vπk and (ii) policy improvement in which the policy πk+1 is updated greedily w.r.t. vπk . This iterative scheme is proven to converge to the optimal policy.
PI schemes have also been extended to the approximate case. Scherrer [25] provides an overview of such approaches, which are called Approximate Policy Iteration (API). In API we consider the scenario in which ||v̂π̂k − vπk ||∞ ≤ . This approximation error can be caused due to two factors, the first being the inability to estimate the value and the second being the inability to find the greedy policy, i.e., the arg max at each state. For instance, such errors may occur due to the functional class being used, e.g., neural networks. Most importantly, the sub-optimality of these approaches is bounded by ||vπ̂∞ − v∗||∞ ≤ (1−γ)2 [25], i.e., the sub-optimality of the final policy is proportional to the approximation error .

heading: 2.2 OFF-POLICY DEEP REINFORCEMENT LEARNING FOR CONTINUOUS CONTROL
text: DDPG [17] and its latest improvement TD3 [6] use neural networks as function approximators in order to find the optimal policy. Both approaches utilize a target network, which is crucial for the success of the methods. While in their work, this was merely considered as a design parameter, we argue that the importance of these target networks lies in the connection to Policy Iteration.
Deep Deterministic Policy Gradients [17, DDPG] (Algorithm 2) aims to directly learn a deterministic policy in a continuous action space. DDPG has both policy evaluation and policy improvement steps. The Q-function, also known as the critic, is trained to estimate the utility of the target policy πθ− , whereas the online policy (the actor) is trained to find the 1-step greedy update w.r.t. this estimation. We argue that DDPG and PI are connected through the target network update procedure. In PI, the previous policy πk is evaluated by vπk , followed by an improvement step in which πk+1 becomes the one-step greedy policy w.r.t. this estimate. DDPG follows a similar scheme, in which the critic Qφ estimates the utility of the policy πθ− , followed by the online policy πθ which is updated w.r.t. this estimate. In this work, we compare to TD3 [6], an improved version of DDPG.
Twin Delayed DDPG [6, TD3] improves the DDPG algorithm by tackling the problem of overestimation. Similarly to how Double-DQN [32] overcame the overestimation problem by introducing an additional Q function, in TD3 they introduce an additional critic. They show, both theoretically and empirically, that by bootstrapping the minimal value between both estimators miniQi(s, π(s)) they are able to overcome the problem of overestimation.

heading: 3 INSTABILITY IN POLICY-BASED DEEP REINFORCEMENT LEARNING
text: In the previous section, we provided an overview of PI, which has convergence guarantees, and two off-policy DRL variants. However, since in practical applications the policy is represented using a non-linear neural network, a small change in the parameters θ may result in a very large change in the output. We illustrate this issue in Figure 1, where a state of the art off-policy approach learns to control the Hopper and InvertedPendulum robots.
In these figures, we show the individual learning curves over several seeds, without performing temporal smoothing1, whereas the bold line represents the mean performance across runs. At each evaluation period, we evaluate the performance of the agent across 100 episodes. These results paint a clear picture – while the agent is capable of learning a policy which attains a score of approximately 3500 in Hopper and the optimal score of 1000 in InvertedPendulum, it is highly unstable, i.e., a policy which was near-optimal at time t might become arbitrarily bad at time t+ 1.
1Temporal smoothing is a common practice in empirical studies, in which each data-point in the graph represents a moving average over the past N evaluations. While this makes the graphs easier to read, it hides a lot of important information such as the variance of each seed during training.

heading: 4 CONSERVATIVE POLICY GRADIENTS
text: While there exists a connection between the tabular approach and that which is used with non-linear function approximators, due to the non-convex optimization landscape, a small gradient step can lead to an arbitrarily bad solution. As such, current policy-based methods with non-linear function approximation (neural networks) behave poorly when compared to their tabular counterparts.
We propose a simple, conservative, yet theoretically grounded approach for tackling these issues. Taking inspiration from the classic approaches, we observe that the gradient-based algorithms lack the improvement guarantees. To overcome this issue, we propose a hierarchical learning procedure: while the online network is trained as before (as seen in Algorithm 2), the target network is updated periodically if, with high probability, the online policy is better.
In simple terms, our proposal is as follows: every T time-steps, evaluate both the target network2 and the online network. If with confidence of 1− δ the online network is better than the target, perform the switch θ− = θ. In addition, between updates, the target network is kept fixed, as opposed to the standard approach [17; 6] which performs Polyak averaging [21]. We provide additional discussion in Appendix D.
The following proposition motivates this approach. We discretize the range [vmin, vmax] into bins of size ∆ and formulate the learning process as a random-walk, in which with probability 1− δ, the policy improves (1 step to the right) and decreases otherwise. The random-walk is bounded at both ends by vmin and vmax, such that the value is projected back into this range.
Proposition 1. Let vπt be a random walk admitting values in the set V = {vmin, vmin + ∆, . . . , vmax −∆, vmax}. Also, assume that w.p. δ: vπt+1 = min{vπt + ∆, vmax} and vπt+1 = max{vπt −∆, vmin} otherwise. Then, the stationary distribution v of vπt equals P(v = x) = rx 1−r
1−r(vmax−vmin)/∆+1 for any x ∈ V , where r = δ 1−δ .
Proof Sketch. We can write the recurrence relations by:
P(v = vmin) = (1− δ)P(v = vmin) + δP(v = vmin + ∆) P(v = x) = (1− δ)P(v = x−∆) + δP(v = x+ ∆) P(v = vmax) = (1− δ)P(v = vmax −∆) + δP(v = vmax)
the stationary distribution follows.
An illustration of this process is provided in Appendix A. When the probability of improvement is 50%, i.e., we are unable to determine which policy is better, the stationary distribution over the value may be arbitrarily bad. However, as the confidence in the improvement of the policy increases, the distribution shifts towards a delta function at the optimal value function.
Remark 1. In practice, the optimal attainable policy is a function of the initial starting parameters. Due to the non-convexity of the optimization landscape, not all initializations are ensured to be capable of attaining an optimal policy.
2The target network needs to be re-evaluated in order to ensure an unbiased estimate of its performance.

heading: 5 EXPERIMENTS
text: Implementing our approach in an off-policy actor-critic scheme is straightforward. While current approaches [17; 6] hold two policies, an online and a target policy, such that the target policy is updated each step using a Polyak averaging procedure [21] – our approach, illustrated in Fig. 2, requires the target policy to be updated only when the online policy is superior, with high probability.
Since in our framework, the online policy is aimed at finding the 1-step greedy policy, and due to the continuous nature of the action space, we add a ‘trust region’ element to the loss λ||πθ(s)−πθ−(s)||22 such that the online policy is kept close to the target. In addition, we found that collecting data interchangeably from the online and the target network leads to a further improvement in stability.
For all the experiments, we plot 5 different random seeds for each configuration, discarding failure seeds3. At each evaluation step, each seed is evaluated for 100 episodes, and the average performance is presented. We plot both the individual seeds (light curves) and the average results across seeds (dark curves). As opposed to previous works, we do not smooth the graphs using a moving average procedure, which enables a better analysis of the experiments’ variance and stability.

heading: 5.1 EVALUATING THE POLICIES
text: Our approach requires determining whether or not an online policy is better, with probability 1− δ. Assuming Jπ , the performance of a sampled trajectory for any deterministic policy π, is normally distributed, we may consider the t-test for evaluating improvement. We formulate the null hypothesis as µπθ = EρJπθ ≤ EρJ
πθ− = µπθ− . In other words, we hypothesize that the online policy has not improved over the target policy, thus we use the Welch’s one-tailed t-test [33]. The t-statistic
is then defined as t = ( µ̂πθ − µ̂πθ− ) / (√ σ̂2µθ/nπθ + σ̂ 2 µθ− /nπθ− ) , where µ̂π is the empirical
estimate of EρJπ , σ̂π is its standard deviation and nπ the number of sampled trajectories. The tstatistic enables us to find the p-value, which represents the confidence in the null hypothesis. Under Gaussian assumption, the p-value can be directly calculated by p = 1 − Φ(t), where Φ(x) is the cumulative distribution function of a standard normal random variable. Hence, when p < 1− δ we have a high enough confidence that the null hypothesis is incorrect - thus the new policy is better, in which case we update the target policy.

heading: 5.2 PARAMETER SELECTION
text: For this approach, there are three parameters which require tuning. (i) how often we evaluate the online policy, (ii) how many episodes are used for evaluation and (iii) the minimal confidence required for policy improvement. Notice that there is a connection between the confidence and the number of evaluation episodes; as the number of evaluations grows, the confidence itself increases. Hence, when there is a relatively high variance in the evaluations, a low number of samples will result in a low confidence, even when the empirical mean is higher. On the other hand, the evaluation frequency controls how fast the process converges. We evaluate the effect of these parameters and provide comprehensive results in Appendix B.
3We observed that some seeds in deep RL result in utter failure, for instance, a score of 40 in Hopper. This issue is relatively rare, and also happens in the baselines, but hinders the proper evaluation of the policy.
Confidence Value δ: The confidence level is 1 − δ, i.e., P(EρJπθ > EρJπθ− ) ≥ 1 − δ. While a natural approach would be to attempt a high level of confidence, our empirical results show otherwise. Demanding a confidence which is too high, makes it much harder to find an improving policy, resulting in a stable yet strictly sub-optimal policy. On the other hand, when very low, the approach becomes unstable. We found that δ ∈ [0.1, 0.2] work well in practice.
Evaluation Episodes K: The number of evaluations is correlated to the confidence. A higher number of evaluations increases the confidence in the empirical mean, enabling the algorithm to determine that a policy is better even when the difference is small. However, this comes at a cost – evaluating the policy takes time and a multitude of evaluations may drastically lengthen training.
Evaluation Frequency N : When comparing evaluation frequency, we see that when the evaluation is performed every 10, 000 steps, it requires a high probability of improvement to successfully converge. However, when the evaluation is performed more often, the process behaves well even for a lower improvement probability (fewer evaluation episodes).
Based on these insights and the results in Appendix B, we opted to run with a minimal confidence for swapping of 90%, evaluation every 1, 000 steps over 10 episodes (sampled trajectories).

heading: 5.3 RESULTS
text: We compared our approach to the baseline – TD3 without our policy selection scheme. The results are presented in Figure 3. These results show a high resemblance to the theoretical model, showing that, across all domains, the conservative approach (Conservative-TD3) reduces the variance of individual seeds, also seen in Table 1, and results in increased stability across most domains.
InvertedPendulum and InvertedDoublePendulum serve as perfect examples. While TD3 is capable of finding an optimal policy - e.g., the maximal score - subsequent policies may be arbitrarily bad. By adding the hierarchical selection scheme, our approach is capable of reducing this behavior. That is, once an optimal policy is found, it is not likely to be replaced with an inferior one.
An interesting result can be seen in the Walker2d domain. While some seeds attain near-optimal performance, others converge to a sub-optimal policy. As our approach searches for a high-confidence improvement, it may sometimes get stuck. However, the inner-process variance, i.e., the variance of individual seeds during training, is much lower across all domains.
Finally, we observe sub-optimal behavior in the Ant domain, both in terms of individual seeds and average performance across seeds. The parameters (confidence, number of evaluation episodes and
evaluation frequency) were selected based on the Hopper domain, however it is important to note that these parameters are not optimal for each domain.
Figure 4 presents the standard deviation (STD) of both the online (Figure 4a) and target (Figure 4b) networks, and the difference between the empirical means |µ̂π − µ̂π− | (Figure 4c). This clearly shows that in Ant, there is a very large variance, compared the difference between the means, which results in a low confidence. To overcome this issue, we provide an additional experiment, for the Ant domain based on these insights (Figure 4d), which considered a minimal confidence of 20% (90% in Figure 3). These results show that while the parameters are transferable across domains, the optimal parameters found on one domain are not necessarily optimal for another.

heading: 5.3.1 STABILITY ANALYSIS
text: To analyze the stability of each algorithm (TD3 vs C-TD3), we consider the variance of the policy during the last 100 episodes of training. In Table 1 we present the maximum of the standard deviation in different seeds, where per-seed STD is calculated over the 100 last episodes of training.
Our empirical results match the theoretical behavior. Indeed, the inner-seed variance is dramatically reduced when using the conservative update rule, as opposed to the non-conservative TD3.

heading: 5.4 POTENTIAL EXTENSIONS
text: In this work, we presented an approach for providing probabilistic improvement guarantees. The performance of a trajectory is a random variable, as the initial state, the transition kernel and sometimes the policy – may all be stochastic. To determine which policy π− or π (the target or online) is better, we performed N evaluations, assumed that the samples are normally distributed and performed a t-test. Below we present several intuitive extensions to our work:
Unknown distribution: While the Gaussian assumption worked well in practice, it may not always be correct. In such a scenario, we see two possible routes: (i) known range, in which the minimal and maximal possible values are known, where one may use the Empirical Bernstein [18], or (ii) when the range is unknown, one may consider a bootstrap-based approach for estimating the confidence intervals around the mean [1].
Upper/Lower Confidence Bounds: Another approach is to consider improvement by a minimal margin. For instance, we may set the update rule to P(vπ − ≥ vπ−) ≥ 1− δ, so that the statistical test ensures that with high confidence, the online policy has improved by at least over the target.
Adaptive Sampling: In our approach, we considered a rather small number of evaluations, and thus resorted to an equal number of evaluations of each policy – up to 100 evaluations for each policy. However, when provided with a larger budget, it is of great interest to combine adaptive sampling procedures. Gabillon et al. [7] propose such an approach, which takes into account the empirical mean and variance of each policy evaluation in order to determine which policy to evaluate. At the end of this process, the policy with the highest empirical mean is returned.
In our initial experiments, we tested the UCB/LCB approach. However, we found that is domainspecific. While in some domains the attained rewards are relatively high, and the policies improve greatly between iterations, in others the improvement is rather small. This led to an early convergence to sub-optimal policies. Nevertheless, this approach may prove beneficial in certain settings and should be considered. Additionally, we tested policy swap based on the empirical estimate of the mean, without ensuring a proper confidence. While this approach worked relatively well in practice, as it lacks proper confidence bounds, we decided to focus on the t-test based approach.

heading: 6 RELATED WORK
text: Trust Region Optimization: Recent works in on-policy RL have also taken inspiration from Conservative Policy Iteration [15, CPI]. Schulman et al. [26] formulated learning through a trust region optimization scheme, which was later optimized in Schulman et al. [27]. These approaches bound the KL-divergence between the policy before and after the update. While TRPO greatly improved the stability compared to previous policy gradient approaches, such as A2C [20], it lacks the theoretical guarantees.
Safe Policy Improvement: Given a baseline policy, safe policy improvement (SPI) algorithms aim to find a new policy that is better than the baseline with high probability [31; 8; 16]. Although this may seem similar to our update stage, there are two major differences: (i) most SPI algorithms directly calculate the improving policy, which is much less efficient than continuously updating the policy and performing the switch when improvement is achieved, and (ii) due to safety constraints, SPI algorithms usually perform off-policy evaluation for the suggested policy, which is still an open problem in high dimensional problems. In contrast, we evaluate it directly and thus enjoy a better sample complexity.
Evolution Strategies: ES [24; 28; 3] is a gradient-free approach for solving reinforcement learning tasks. At each iteration, given the previous policy πθi , a set of augmented policies is created {πθi+ξi}i and evaluated, where ξi is some random noise. The ES update rule can be seen as a form of numerical gradient estimation (i.e., finite differences) [35]. There are some similarities between our approach and ES. While ES is a gradient-free method, it uses empirical evaluations of the policies to determine the parameter update direction. Our approach uses these evaluations in order to determine whether or not the policy has improved, and in contrast to ES, will only change the policy if there is a high enough confidence for improvement.

heading: 7 CONCLUSIONS
text: In this work, we tackled the stability problems of policy-based reinforcement learning approaches. Even though the empirical gradient (SGD) is a noisy estimate of the ideal descent direction, and due to the structure of the optimization landscape and the large step sizes used in practice, the policy’s performance varies greatly during training.
Drawing from the connection between actor-critic approaches and approximate policy iteration schemes, we introduced a hierarchical training scheme, which we analyzed by comparing this approach to a bounded random-walk. While this scheme is not an exact representation of the optimization process, it provides an intuition on the behavior of such a probabilistic improvement scheme, i.e., as the probability of improvement increases, the stationary distribution over the performance of the policy shifts towards the optimal solution.
Finally, we proposed the Conservative-TD3, which performs a periodic evaluation of both the online and target policies such that it updates the target network only when there is a higher enough confidence that the online network has surpassed its performance. We validate this on several continuous control tasks in MuJoCo; empirical evidence shows that indeed this approach behaves similarly to the theoretical analysis. When compared to the baseline, we observe a dramatic reduction in variance, increased stability and an overall improvement in the performance itself.

heading: A RANDOM WALK
text: We expect our approach to approximately behave as a random walk on a bounded region. As the probability of taking a step towards the right increases, the stationary distribution shifts towards the optimal value. On the other hand, as the uncertainty grows, i.e., the probability of improvement becomes closer to 12 , the stationary distribution is uniform.

heading: B PARAMETER ABLATION TESTS
text: In this section, we present the simulation results for all possible parameter combinations, in the Hopper domain. Specifically, we tested minimal required improvement probability of [0.5, 0.8, 0.9, 0.95], number of evaluations per evaluation steps of [5, 10, 100] and evaluation frequency of [1000, 10000].
The graphs above show the effect of tuning the evaluation frequency, the number of evaluations and the minimal required confidence. As the evaluation becomes more frequent, the agent learns faster. As our approach requires the critic to evaluate the target policy π−, the online policy π aims to find the 1-step greedy policy. This may happen rather quickly, thus a more frequent evaluation scheme will improve faster. On the other hand, the number of evaluation episodes is correlated to the confidence in improvement, i.e., the confidence we have that π is indeed ’better’ than π−. We see that as the number of evaluations rises, the agent is capable of finding good solutions even when a higher confidence is required. However, a lower confidence results in a noisier training regime and a small number of evaluations leads to a low confidence – hence high confidence requirements with a low number of evaluations often leads to the inability to converge to a good solution.

heading: C ONLINE VS. TARGET NETWORKS
text: The common approach is to present the performance of the online network. In these graphs, we show that, as opposed to the common belief, the target network suffers less from instability and offers improved performance. These results are intuitive once the connection to policy iteration schemes is made.

heading: D EMPIRICAL DETAILS AND DISCUSSION
text: We follow the same hyper-parameter scheme as is used for the TD3 algorithm [6]. The first 10, 000 steps are played using randomly sampled actions, during which the target policy is updated in a Polyak averaging technique. After which we initialize the conservative periodic policy updates.
In addition, we noticed that while the target policy has a high probability stability assurance, this does not hold for the online policy. As such, we found that often the online policy converges to a strictly sub-optimal solution. We overcome this issue using two heuristics, (i) using a trust-region update rule in which we penalize (an additional loss term) the distance between the action predicted by the online and the target policies. As the critic estimates the performance of the target policy, the estimation error grows as the online policy diverges from the target. Finally, (ii) if the policy has not been updated for over 20, 000 steps, we assume that it has converged to a sub-optimal region of the loss landscape. At such an event, we set the new online parameters as θ = θ + u where u ∼ N (0, 0.1) – this is akin to an ES search procedure in which noise is added to the parameters in order to overcome local extrema. This approach helps, but does not ensure, preventing the approach from ”flatlining” in which the online policy is no longer updated.
While the online policy benefits from the periodic updating procedure, we observed that when the critic is also updated periodically, the performance becomes strictly sub-optimal. We believe that this is since this introduces an additional time-scale, where three timescale algorithms are known to be tricky to tune and properly operate.
Finally, we observed that in certain initializations both TD3 and C-TD3 were incapable of learning. This resulted in a flat-line throughout the entire training regime, e.g., results such as a score of 40 in Hopper. Although rare, this is indeed an important problem in DRL, which needs to be addressed properly, and is not apparent in supervised learning regimes. However, as this is not the focus of this paper, we discard such seeds to enable the focus on the proper training regimes.

Review summaries:
This paper proposes an approach called conservative policy gradients to stabilize the training of deep policy gradient methods. At fixed intervals, the current policy and a separate target policy are evaluated with a number of rollouts. The target policy is then updated to match the current policy only if the current policy is better than the target policy. Experiments show that the proposed method, when applied to TD3, reduces the variance in performance through the training.

This paper proposes a simple method for stabilizing the off-policy deep reinforcement learning algorithm, which updates the target network only when the online network performs better than the target network in order to ensure the stability guarantees. More specifically, at every T time steps, they execute both the online network and the target network so as to evaluate the performance of each policy. Then, they update the target network only when the online network outperforms the target network with high probability. The experimental results show that the proposed Conservative-TD3 (C-TD3) is less prone to performance degradation during training.