Is High Variance Unavoidable in RL? A Case Study in Continuous Control
Authors: Johan Bjorck
Carla P Gomes
Kilian Q Weinberger
Venue: ICLR 2022 Poster (2022)
Keywords: reinforcement learning
continuous control

Abstract: Reinforcement learning (RL) experiments have notoriously high variance, and minor details can have disproportionately large effects on measured outcomes. This is problematic for creating reproducible research and also serves as an obstacle when applying RL to sensitive real-world applications. In this paper, we investigate causes for this perceived instability. To allow for an in-depth analysis, we focus on a specifically popular setup with high variance -- continuous control from pixels with an actor-critic agent. In this setting, we demonstrate that poor outlier runs which completely fail to learn are an important source of variance, but that weight initialization and initial exploration are not at fault. We show that one cause for these outliers is unstable network parametrization which leads to saturating nonlinearities. We investigate several fixes to this issue and find that simply normalizing penultimate features is surprisingly effective. For sparse tasks, we also find that partially disabling clipped double Q-learning decreases variance. By combining fixes we significantly decrease variances, lowering the average standard deviation across 21 tasks by a factor >3 for a state-of-the-art agent. This demonstrates that the perceived variance is not necessarily inherent to RL. Instead, it may be addressed via simple modifications and we argue that developing low-variance agents is an important goal for the RL community.

Published as a conference paper at ICLR 2022
Is HIGH VARIANCE UNAVOIDABLE IN RL? A CASE STUDY IN CONTINUOUS CONTROL
Johan Bjorek, Carla P. Gomes, Kilian Q. Weinberger Cornell University
ABSTRACT
Reinforcement learning (RL) experiments have notoriously high variance, and mi- nor details can have disproportionately large effects on measured outcomes. This is problematic for creating reproducible research and also serves when applying RI to sensitive real-world applications. In this paper, we inves- tigate causes for this perceived instability. To allow for in-depth analysis, we focus on a specifically popular setup with high variance - continuous control from pixels with actor-critic agent. In this setting, demonstrate that poor outlier runs which completely fail to learn are an important source of variance, but that weight initialization and initial exploration cause for these outliers is unstable network parametrization which leads to satu- rating nonlinearities. We investigate several fixes to this issue and find that simply normalizing penultimate features is surprisingly effective. For sparse tasks, we also find that partially disabling clipped double Q-learning decreases variance. By combining fixes we significantly decrease variances, lowering the average stan- dard deviation across 21 tasks by factor > for state-of-the-ar agent. This demonstrates that the perceived variance is not necessarily inherent to RL. In- stead, it may be addressed via simple modifications and we argue that developing low-variance agents is an important goal for the RL community.
obstacle
not fault. We show that
INTRODUCTION
With the advent of deep learning, reinforcement learning (RL) with function approximators has be- come a promising strategy for learning complex behavior (Silver et al., 2017 Mnih et al. 2015). While RI. can work remarkably well, deep RI. experiment have been observed to exhibit high vari- ance (Chan et al., 2020 Clary et al, 2018 Lynnerup et al. 2020 Specifically, the rewards obtained the random seeds used for training, even if no other modifications are made. Furthermore, implementation level details (Engstrom et al. 2019) and even hardware scheduling (Henderson et al. 2018) often have outsized effect on performance.
by RL agents commonly vary significantly depending
High variance is problematic for creating reproducible RL research, and achieving statistically sig. nificant results over large RI. benchmarks is often prohibitively time-consuming (Colas al. 2018 Agarwal et al. 2021). Furthermore, high variance provides an obstacle for real-world applica- tions of RL, where safety and predictability can be critical (Garcıa Fernández) [0015 Srinivasan et al., 2020 Especially relevant RL applications include medical testing (Bastani et al. 202
autonomous vehicles (Bellemare al. 2020) and communications infrastructure (Liu et al. 2020
In this paper, we investigate causes for high variance in RL. To limit the paper's scope and compu- tational footprint we focus on continuous control from pixels with state-of-the-arr (SOTA) actor- critic algorithm (Lillicrap et al.] 2021 Yarats et al.] 2021b). This is a popular setup that exhibits high variance, and where actor-critic algorithms often achieve SOTA results (Laskin al., 2020b a Kostrikov et al.) 2020). In this setting, we first show that an important cause of variance is outlier runs that fail early and never recover. is natural to hypothesize that the typical sources of ran- domness, network initialization, and initial exploration, are to blame. However, we demonstrate that these do in fact have little impact on the final performance. We similarly provide experiments that suggest that neither poor feature representations fault. We instead demonstrate that unstable network parametrization can drive the variance and lead to excessively large activations that saturate action distributions. We propose several methods to
excessive sensitivity of the environment is

---

Published as a conference paper at ICLR 2022
avoid such issues and find that they decrease variance. The most effective method is simply setting penultimate features to unit norm a strategy we call penultimate normalization which avoids exploding activations by design. We also find that partially disabling double clipped Q-learning (Fujimooo et al. 2018), a strategy we call asymmetrically double clipped Q-learning, reduces vari- ance for many sparse tasks. By combining multiple approaches we increase average rewards while decreasing variance as measured of 21 tasks. The average standard deviation decreases by a factor 3 and for some tasks by more than an order of magnitude. Our experiments suggest that high vari- ance is not necessarily inherent to RL. Instead, one can directly optimize for lower variance without hurting the average reward. We argue that developing such vor-variance agents is an important goal for the RL community - both for the sake of reproducible research and real-world deployment.
2 BACKGROUND
We consider continuous control modeled as a Markov decision process (MDP) with a continuous action space AA and state space S. In practice, the action space A is often bounded to account for physical limitations, c.g., how much force robotic joints can exert. A popular choice is simply 1, 1] (dsm(A) Within this setting, Deep Deterministic Policy Gradients (DDPG) (Lillicrap) 2021) is performant algorithm that forms the backbone of many state-of-the-art agents Laskin et al. 2020b al Kostrikov et al., 2020 Yarats et al. 2021b). DDPG uses a neural network (the critic) to predict the Q-value Qo (ay, 8,) for each action pair. The parameters @ of the critic are trained by minimizing the soft Bellman residual:
A
E ((Qo(ar,s)) st) [ra + (EQ(
Here, rt is the reward and E[Q(a++1 S, Sc+11 1]] is the Q-value estimated by the target critic- - a network whose weights can e.g. be the exponentially averaged weights of the critic. One can use multi-step returns for the critic target (Sutton 1988). Another common trick is double clipped q-learning, i.e. using two Q-networks Q1, Q2 and taking Q = min(Q1, Q2) decrease Q-value overestimation (Fujimoto et al., 2018). The policy is obtained via an actor-network with parameters 0, which maps each state s, distribution no (8,) over actions. The action is obtained by calculating (s)" the vector output of the actor network. Thereafter a tanh non-linearity is applied elementwise to ensure E [-1, 1]"d for n (A.. The policy is then obtained as:
ag (s) = tanh(a?"" (s)) +
is some additive noise which encourages exploration, e.g. Gaussian variable with time- dependen variance o². Various noise types have been employed (Achiam 2018 Lillicrap et al. 2001 Yarats
et al. 2021b). Since the Q-values correspond to the expected discounted rewards, the actions ag should maximize Qp (a),,))) Thus, to obtain a gradient update for the actor one uses the gradient VoQo (ag, s). effectively calculating gradients through the critic network. In control from pixels, the state s is typically an image (possibly data augmented) processed by a convolutional network.
3 BENCHMARKING VARIANCE
Experimental Setup. To limit the paper's scope and computational footprint, we focus on contin- uous control from pixels with actor-critic algorithm. This setup is relatively close to real-worke robotics, where reliable performance is important. We consider the standard continuous control benchmark deepmind control (dm-control) (Tassa al., (2020). Within dm-control, long line of state-of-the--rt models relies on the actor-critic setup (Laskin et al. 2020b a Kostrikov et al. 2020) We consider the recent data autta-augmenttiionn based DDPG-type agent DRQv2 (Yarats et al. (021b)
which provides state-of-tt of-the--rtttttt performance and runs fast. We use the default hyperparameters that Yarats et al. (2021b) uses on the medium benchmark (listed in Appendix (A) throughout the paper. We will refer to this agent as the baseline agent. We are interested in using large number of random seeds to obtain accurate statistics and thus need to limit the number environments we consider. To select environments we simply rank the dm-control tasks based upon their relative variance as measured by Yarats et al. (202bb). We select the five tasks with the highest relative variance- finger turn hard, walker walk, hopper hop, acrobot swingup, and reacher hard (see Table 4 in Appendix A for details). In Section[.22 we consider a larger set of tasks. We will refer to these as envirouments or tasks interchangeably. For each run, train the agent for one million frames, equivalently 1 1,000
2

---

Published as a conference paper at ICLR 2022
episodes, and evaluate over ten episodes. We use 10 40 seeds for each method consider, these are not shared between experiments, but we reuse the runs for the baseline agent across figures.
3.1 VERIFYING VARIANCE
We first aim to verify that the environments indeed exhibit high variance. To this end, we run the baseline agent with 40 random seeds on each task. shows the result of this experiment for three tasks, see Appendix for the two other tasks. We show both individual runs as well as mean and standard deviations. For some tasks, the variance is dominated by outlier runs that fail to learn early in the training and never recover. E.g. for the walker task, we see several runs which have not improved during the first 500 episodes, and most of these do not improve in the following 500 episodes either. Except for these outliers, the variance of the walker task is relatively small. We also note that these outliers separate themselves relatively early on, suggesting that there are some issues in early training that cause variance. If it possible to eliminate poor outlier runs, would expect the variance to go down. Motivated by these observations, and the walker task in particular, we will investigate various potential causes of outlier runs and high variance.
3.2 DOES INITIALIZATION MATTER?
There are multiple sources of randomness specific to the early training iterations. The agent net- works are randomly initialized and the agent is given some initial experience, collected by random actions, start training on. Could these two initial factors be the cause of the high variance? To study this hypothesis we measure the correlation between two runs that share the same network initialization and initial experience, but otherwise have independent randomness (from e.g. the en- vironment, action sampling, and mini-batch sampling). Across all tasks, we repeat this process for ten combinations of initializations and initial experience, from each such combination we start two runs A, B which do not share other sources of randomness. For each time-step we measure the Pear- son correlation between score(A) and score(B). This correlation, averaged across all timesteps, is shown in Figure 2 (left). We that the correlation is very small, suggesting that the initialization and initial experience does not have large effect on the outcome. Specifically, the hypothesis that there good and bad initializations that leads to good or bad performance is inconsistent with the results of Figure 2 (left). Correlation across time is noisy but centers around zero, see Appendix A
walker
8crobat
hopper
800
140
200
reward
400
70
500
1000
500
1000
500
1000
300
300
800
reward
200
400
100
500
1000
0
500
1000
500
1000
episodes
episodes
episodes
Figure 1: Learning curves over 40 seeds for the baseline agent. The upper row shows the mean reward, with one standard deviation indicated by shading. The bottom row shows individual runs. For some tasks, the variance is dominated by few outlier runs that completely fail to learn This is especially clear the walker task. Removing such outlier runs would decrease the variance.
3

---

Published as a conference paper at ICLR 2022
0.06
correlation
800
reward
0.03
400
0.00
-0.03
finger
hopper
walker acrobot reacher
0.0
0.5
1.0
random action probability
Figure 2: (Left) Correlation between runs that share network initialization and starting experience, but otherwise have independent other sources of randomness (e.g. from mini-batch sampling). Across all tasks, the correlation is small. This suggests that initialization and starting experience have limited effect on the outcomes. (Right) The average reward for the walker task when interpo- lating between a learned policy (random action probability 0) and random policy (random action probability 1). The rewards obtained decrease gracefully as we adopt more random policy. This suggests that the environment is relatively forgiving with respect to actions.
3.3 ARE THE ENVIRONMENTS TOO SENSITIVE?
Another cause of variance is the environments. Perhaps the environments require very specific policies to yield rewards, and unless the agent stumbles upon a good policy early on, learning fails. To study the sensitivity of the environments we consider gradually exchanging a good learned policy with random policy. Specifically, at every timestep 4, with probability p. we exchange the policy's 's action with a random action. We use the final policy of a performant run with the baseline agent the learned policy. The effect of this, for ten final policies learned for the walker environment, is shown in Figure - (right). Performance degrades gracefully as we gradually move to a random policy. This suggests that the environment is relatively stable with regard actions. Results for other environments are similar and are shown in Appendix [A] Furthermore, in Appendix we verify that the variance arising from evaluating the agent finite number of episodes is small compared to the inter-run variance. Thus, environment sensitivity and evaluation methodology do not appear to explain the excessive variance.
3.4 IS FEATURE LEARNING TO BLAME?
In the early stages of training, the image encoder will extract poor features. Without good represen- tations, it is hard to leam good policy, and without policy generating interesting behavior it is hard learn good representations. the encoder to extract good features should help to decrease the variance. To test this hypothesis, we consider initializing the weights of the convolutional encoder to some good pretrained values with above-average final performance and
such a vicious cycle was causing runs to fail, jump-starting
before training normal. For each task, pick use its convolutional weights at episode 500. We refer to this strategy as pretrain. We also consider the same strategy but decrease the learning rate of the pre-trained weights by factor of 10x. We refer to this strategy as pretrain + small Ir. In addition to this fix, we also consider improving the features with self-supervised learning during early training (Ting & Tian 2020). We use the loss function proposed in Grill et al. (2020) which is useful for smaller batch sizes (we use 256) and has already successfully been applied RL. (Schwarzer et al. (02T). We use standard methods in self-supervised learning: target network with exponentially moving weights and head network. See Appendix A for details. We consider adding the self-supervised loss to the first 10,000 and 20,000 steps and refer to these modifications self-supervised 10k and self-supervised 20k. The results for this experiment are shown Table [ For these fixes, the large variance persists even though the performance improves marginally with self-supervised learning. We thus hypothesize that representation learning is not the main bottleneck. Indeed, experiments in later sections of the paper show that one significantly decrease variance without improving representation learning.
4

---

Published as a conference paper at ICLR 2022
Table I: Performance when using self-supervised learning pretrained weights for the concolu-
tional encoder. Improvements are small and variance remain high. This suggests that poor features
might not cause the high variance. Indeed, Table shows that performance can improved signif- icantly without better representation learning. Metrics are calculated over 10 seeds.
metric
method
acrobot
finger turn
hopper hop
reacher
walker
avg.
baseline
122.0
278.6
176.2
678.0
769.1
404.8
μ
self-supervised 10k
141.4
456.2
176.5
671.6
677.1
424.6
self-supervised 20k
159.4
325.3
209.5
659.6
773.8
425.5
μ
pretrain
160.0
303.7
153.3
620.8
581.4
363.8
pretrain small Ir
157.3
358.8
124.8
679.0
674.2
398.8
baseline
76.0
221.2
90.3
167.3
350.1
181.0
self-supervised 10k
68.4
276.1
104.6
222.4
425.9
219.5
self-supervised 20k
67.7
191.9
71.6
164.8
374.4
174.1
pretrain
39.8
146.7
86.8
168.7
429.1
174.2
pretrain small Ir
71.8
204.6
102.7
165.4
393.5
187.6
3.5 STABILITY NETWORK PARAMETRIZATION
We have considered the effects of initialization, feature learning, and environment sensitivity and found that none of these factors satisfactorily explain the observed variance. Another factor to consider is the stability of the network parametrization. Unstable parametrization could make the learning process chaotic and thus cause high variance. To study this hypothesis we investigate the failure modes of individual runs. In Figure 3 we consider some runs that fail as well as some successful runs for the walker task. Plots for runs are available Appendix [A] We plot three quantities during training: rewards, actor gradient norm V[Ve], and average absolute value of actions |al- All agents initially have poor rewards, but once rewards start to improve, they improve steadily. During this initial period of no little learning, the gradients of the actor-network (row two Figure are small. Recall that the actor optimized via gradients: VgQo as (s))
run
run 3
4
1000
reward
500
0
1.
|a] 0.5
1000
episode
episode
episode
episode
Figure 3: Each row shows a different quantity during training for the baseline agent. Row shows rewards, row shows actor gradient norm Véll, and row shows average absolute action |a]. Each column corresponds to an independent run of the walker task: run is successful, runs and 3 eventually learn, while run 4 fails learn. Before learning starts, all runs suffer from small gradients. During this period, the average absolute value of actions is close to 1. This causes small gradients since the gradient of tanh(x) small when ttan(x)) 1.
5

---

Published as a conference paper at ICLR 2022
E: aQ. da. If this gradient is close zero, might suspect that da, = 0. This could easily 0a; a0
a0
happen if the tanh non-line non-lincarity in equation 2 is saturated. In row three of Figure 3 we show the
average absolute value of the actions, which are all bound to lie in [-1, 1]. We indeed see that for the runs that fail the average absolute action stays around throughout training, which implies
that the gradients for the actor are small since tanh(z)] = implies år tanh(z) x 0. The cause
of saturating tanh must be excessively large activations, ie.. ag"(s in equation 2 becoming too large, which saturates the tanh non-linearity. In fact, all runs seem initially suffer from saturating tanh, but most runs escape from this poor policy. Somewhat similarly, Wang et al. (2020) has observed that saturating nonlinearities can impede exploration when removing the entropy bonus from SAC agents. However, for the DDPG agent saturation does not harm exploration since the exploration noise is added outside the non-linearity as per eq. 2 Instead, the issue here appears to be vanishing gradients which impedes learning. The Q-values could of course also suffer from exploding activations, but since there are no saturating nonlinearities for the Q-values, such issues are harder to pinpoint. These experiments suggest that avoiding excessively large activations could reduce the number of poor outlier runs, and thus decrease variance.
4 IMPROVING STABILITY
Motivated by our qualitative observations now consider a few ideas for improving the stability of learning, hoping that these will decrease the variance. We consider several methods:
Normalizing activations. As per eq. (2), the actor selects actions using a ag" (s), where (s) is deep neural network. Figure B shows that learning can fail due to excessively large activations. To avoid this propose to parametrize (s) as a linear function of features with fixed magnitude. Specifically, let As be the penultimate features of the actor-network. We propose to simply nor- T.A (s) for a linear function I.. Note that no mean subtraction is performed. We refer to this straightforward modification as penultimate normalization or pnorm for short. When applied to the actor, we simply call the strategy actor pnorm. Our motivation has been to stabilize the actor output, but the critic network could also benefit from the stability offered by normalizing penultimate features. We refer to the strategy of normalizing both the actor and the critic as both pnorm. We also consider layer norm (Ba et al. 2016) applied similarly both the actor and critic. We also consider the normal- ization scheme of Wang et al. 2220)), which directly constrains the output of the actor. We refer to this method as outpul norm. At last, following (Gogianu et al. (202T), we consider adding spectral normalization (Miyato al. 2018) to the penultimate layer.
malize AB (s) = Ao (s) (s)) and let arre be a linear function of Ag. i.e. (s)
baseline
Penalizing saturating actions. Another strategy for avoiding actions with large magnitude is to add term to the loss function which directly penal- izes large actions. One can imagine many such loss functions, we here con- sider the simple and natural choice of squared loss {(8. >>|aa,"* (s)]l? We can simply add the square loss times some hyperparameter À to the loss function. By tuning X, one should be able to remove the saturating actions illustrated in Figure 3 We heuristically find that X = 0.000001 works well in practice (see Appendix A) and use this value. We refer this modification as penalty, Since the Adam optimizer is invariant under re-scaling the loss, the specific value of has a small effect as long as VoQo (s)) x 0, which happen as per Figure 3
actor pnomm
Learning rate warmup. One common strategy to improve stability is to start the learning rate at zero and increase it linearly to its top value during early training (Goyal et al., 2017). This can improve numerical stability (Ma & Yarats 2021). We consider using this strategy and linearly increasing the learning rate from zero over the first 10,000 steps. We refer to this strategy Ir warmup.
both pnorm
Small weights final layer. Following Andrychowicz et al. 2020). we con- sider using smaller weights for the final layers of the actor and the critic. Specifically, we downscale these weights by a factor of 100 at initialization time. We refer to this strategy as scale down.
100 episode
200
Figure 4: Average Q- value during training.

---

Published as conference paper at ICLR 2022
Table 2: Performance for various stability improvements. These methods often decrease variance,
especially for the walker task where variance decreases by orders of magnitude. Larger stability often leads to higher reward, likely as poor outlier runs are eliminated. Normalizing features gives consistent improvements. Metrics are calculated over 10 seeds.
metric
method
acrobot
finger turn
hopper hop
reacher
walker
avg.
baseline
122.0
278.6
176.2
678.0
769.1
404.8
penalty
128.0
431.6
241.7
584.6
956.3
468.4
actor pnorm
117.8
372.7
240.5
701.7
956.7
477.9
44
both pnorm
192.2
605.8
188.7
892.7
961.5
568.2
layer norm
43.5
661.2
218.6
659.7
952.4
507.1
μ
Ir warmup 150.3
265.4
173.7
660.9
670.4
384.1
grad clip
138.2
214.0
206.0
520.0
675.6
350.7
grad clip 10
79.3
166.2
157.7
548.4
678.7
326.0
spectral
104.0
101.0
201.0
213.6
775.6
279.1
scale down
127.2
258.4
111.7
508.7
386.4
278.5
output norm
13.3
306.3
216.9
699.4
952.1
437.6
baseline
76.0
221.2
90.3
167.3
350.1
181.0
penalty
42.1
160.6
46.5
236.8
15.4
100.3
actor pnorm
39.9
227.9
42.7
101.9
8.0
84.1
0
both pnorm
60.1
193.4
78.4
122.5
7.0
92.3
0
layer norm
57.2
228.4
112.5
297.1
13.5
141.7
0
Ir warmup 53.8
133.6
98.2
133.4
423.4
168.5
0
grad clip
60.6
127.1
92.4
231.9
425.3
187.5
0
grad clip 10
56.1
143.9
113.6
243.2
426.3
196.6
spectral
37.0
61.4
75.0
224.8
128.0
105.2
0
scale down
27.8
186.9
96.6
241.3
444.0
199.3
ourput norm
16.6
190.4
21.4
180.4
17.2
85.2
Gradient clipping. Another stabilizing strategy is gradient clipping (Zhang et al. 2020). To avoid occasional exploding gradients, simply clips the norm of gradients that above some thresh- old. This strategy is popular in NLP (Gehring et al.] [017]] [eters et al.] 2018) and sometimes used in RL (Raffin al., 2019).. We consider clipping the gradient when the norm, calculated inde- pendently for the actor, critic, and convolutional encoder, is larger than or 10. We refer to these strategies as grad clip and grad clip 10.
Results. In Table 2 we evaluate the effects of these ideas. Penultimate normalization has the best performance and lowest variability average, whereas layer norm can make performance worse. Since normalizing the critic improves performance, the critic might also suffer from instabilities. This instability can observed by simply plotting the average Q-values during training for various agents, see Figure 4 We study this question in detail in Appendix C The penalty method improves performance while many other methods decrease average reward, and lowering variance by using poor policy is not hard to achieve and not particularly desirable. The output normalization method of Waag et al. (2020) performs worse than actor normalization, and importantly fails completely on the acrobot task. We hypothesize that directly constraining the output space is harmful on the acrobot task. These experiments show that it is possible to significantly decrease the variance of the baseline agent by applying stabilization methods. Additionally, decreasing the variance often increases the average reward - likely as bad outlier runs are eliminated.
4.1 COMBINING FIXES
We have found that many of the proposed methods can decrease variance and improve average reward. For optimal performance, we now consider combining three orthogonal methods which improve performance individually actor and critic penultimate normalization, pre-tanh penalty, and self-supervised learning (we apply it to the first 10,000 steps). Table B we compare the performance between the baseline and an agent using these three modifications (which we refer to as combined) over 40 seeds. see that the variances go down while the average score goes up. In fact, for the walker task, the variance decreases by orders of magnitude. Eliminating poor

---

Published as conference paper at ICLR 2022
Table 3: Performance when combining three improvements: penultimate normalization, action penalty and early self-supervised learning. The variance decreases and the performance improves across all tasks. This demonstrates that one can substantially decrease variance in RL without hurt- ing average reward. Metries caleulated over 40 seeds.
metric
method
acrobot
finger turn
hopper hop
reacher
walker
avg.
/4
baseline 122.0
278.6
176.2
678.0
769.1
404.8
/4
combined
227.7
653.5
231.1
893.4
964.2
594.0
baseline 76.0
221.2
90.3
167.3
350.1
181.0
combined
59.0
183.1
40.8
120.0
6.2
81.8
outlier runs seems to benefit the average performance in addition to decreasing variance. While our methods have mostly been motivated by outlier runs that fail catastrophically, they likely help more normal runs too. Learning curves for the combined agent are given in Appendix [A] Appendix verify that our fixes enable the agent to learn successfully with a larger learning rate.
4.2 ADDITIONAL TASKS
We now apply the combined agent of Table tasks identified as easy or medium by Yarats et al. (2021b). Note that the combined agent has been constructed using only of these tasks, so this experiment measures generalization. For computa- tional reasons, the additional 16 tasks only use 10 seeds each. We use the baseline hyperparameters across all tasks without tuning (Yarats et al. (2021b) tunes parameters on per-task basis). Results are shown in Figure 5] Our methods consistently improve the average reward while the average standard deviation decreases by factor > Furthermore, a few additional tasks (finger spin, hopper stand, walker run, walker stand) have their variance decreased by order of magnitude or more. The tasks chectah run and quadruped walk perform worse than the baseline, although this could potentially be alleviated by tuning hyperparameters. Just increasing the learning rate these tasks improve performance, see Figure 13 in Appendix A These results demonstrate that our mod- ifications significantly decrease variance and that directly optimizing variance be feasible and fruitful. Performance profiles (Agarwal et al. 2021) are available in Figure o6 Appendix A
additional tasks. Specifically, consider all 21
combined
900
avg. reward
combined+
600
300
combined
std. reward
300
combined+
200
100
Figure 5: Performance for tasks. (Top) Average reward per task. The combined agents improve the average reward, and most tasks benefit from our modifications. (Bottom) Standard deviation of reward, broken up by task. The combined agents have less variability, and for few tasks the variance decreases by orders of magnitude. Except for measurements in Table 3 we use 10 seeds.
8

---

Published as a conference paper at ICLR 2022
run 2
rum
run 4
a 0.5
500
1000
500
1000
500
1000
500
1000
episode
episode
episode
episode
Figure 6: The average absolute value of the actions for the combined agent. We show four repre- sentative runs of walker task. Compared to the baseline algorithm in Figure 3 the actions do not saturate early in training despite increasing rapidly, indicating improved stability.
4.3 SPARSE TASKS
Consider the five tasks in Figure 5 where the combined agent has the highest standard deviation. Four out of these have sparse rewards finger turn hard, reacher hard, pendulum swingup, and cartpole swingup sparse. Once stability has been addressed, sparsity is major driver of residual variance. Motivated by this, we consider three fixes that could help for sparse tasks. Firstly, not train- ing until least one non-zero reward has been observed. Secondly, extending the self-supervised training throughout the whole training process. Thirdly, exchanging the clipped double Q-learning Q min(Qu 1, Q2) (Fujimoto et al., 2018) with Q = avg(Q Q2). decrease pessimistic bias. Through ablations (see Appendix B.. we find that doing this for the critic loss (eq. (T) but retaining the clipping Q min(Q1. Q2) for the actor loss performs the best. We dub this strategy asym- metrically clipped double Q-learning. In Appendix B we present an ablation study over these three modifications, finding that using more self-supervised learning and asymmetrically clipped double Q-learning improves performance. We add these two modifications to our combined agent, which we now call combined+++. In Figure we show the performance of this agent (seeds are independent from experiments in Appendix B). Performance improves, especially for sparse tasks, and compared to the baseline agent the average standard deviation decreases by a factor > 3.
5 EPILOGUE
Related Work. There are multiple papers documenting the high variance of RL (Henderson et al,
2018), focusing on implementation details (Engstrom et al. 22119), continuous control (Mania al.
2018 Atari games (Clary et al.] 2018) and real-world robots (Lynnerup et al.) 2020). Given this
high variance, there is ample work on how accurately measure performance in RL (Colas al.,
2019 Chan et al. 2020 Jordan et al.] 2020) Agarwal et al. 2021 Islam et al. 2017). Our strategy of simply reducing the variance of RL algorithms complementary to developing robust metrics.
Within Q-learning, there is a large amount of work aiming to improve and stabilize Q-estimates
(Fujimooo et al.] 2018, Nikishin et al. 2018 Anschel et al. [017 Jia et al. 2020 Chen al. 2018).
These apply to the discrete DQN setting, distinct from continuous control. There is also work on
stabilizing RL for specific architectures (Parisotto et al.) 2020) and task types (Kumar et al., 2019
Mao et al. 2019). Normalization in RL have been investigated by (Salimans & Kingma (2016);
Bjorek et al. (2021b); Gogianu et al. (202T), and Wang et al. (2020). Wang et al. (2020) directly normalizes the output of the actor, which e.g. for oinddimeasil task shrinks the action space to (tanh(11),tanl)) tanh(1)] (modulo noise). As per Table 2 this method underperforms our methods.
Limitations. We have only considered a single SOTA agent (DRQv2 of (Yarats et al. 202 lb) on one popular suite of tasks. Furthermore, we focus on an agent with tanh nonlinearities for the policy. While this is an important agent class, other important agent types (Hafner et al. 2021 and environments (Bellemare et al., 2013) remain. We show that it is possible to decrease variance with simple modifications, although which methods are effective likely depends upon the base algorithm. While many of our fixes can potentially be added to other agents, remains to be experimentally tested if doing so improves performance. We leave this question for future work. While our fixes typically are only few lines of code, they nonetheless increase the agent complexity.
Conclusion. We have investigated variance in popular continuous control setup and proposed several methods to decrease it. We demonstrate across 21 tasks that these dramatically decrease variance while improving average rewards for a state-of-the-ee agent. Our experiments show that directly optimizing for lower variance can be both feasible and fruitful.

---

Published as a conference paper at ICLR 2022
Ethics Statement. There many positive RL applications examples include safe autonomous cars, accelerated scientific discovery, and medical testing. Studying causes of high variance, and proposing fixes to this issue, benefit real-world adoption of RL techniques. Additionally, ducing the variance of RI. experiments would aid the research community as reproducibility can be problematic in the field. However, there also many potentially harmful military applications. Our study is academic in nature and only considers simulated toy environments, but insights from our work could nonetheless be useful in harmful applications. We do not believe that our work differs from other RL work significantly concerning ethics.
Reproducability Statement. Appendix A we detail the infrastructure and software used. The perhaps most important component of generating reproducible findings in RL is to use many seeds. To this end, we have focused on five environments but evaluated these thoroughly. The main exper- iments of the paper (Table use 40 seeds whereas all others use 10 seeds.
Acknowledgement. This research is supported in part by the grants from the National Science Foundation (II--618134, III-1526012, IIS 149882, IIS-1724282, and TRIPODS- 1740822), the Office of Naval Research DOD (N00014- 17-1-2175), Bill and Melinda Gates Foundation. thankful for generous support by Zillow and SAP America Inc. This material is based upon work supported by the National Science Foundation under Grant Number CCF- 1522054. We are also grateful for support through grant AFOSR-MURE (FA9550-18-1) 18-1-0136). Any opinions, findings, conclusions, or recommendations expressed here are those the authors and do not necessarily reflect the views of the sponsors. We thank Rich Bernstein, Wenting Zhao, and Ziwei Liu for help with the manuscript. At last, we thank our anonymous reviewers for fruitful interactions.
are
REFERENCES
Joshua Achiam. Spinning Up in Deep Reinforcement Learning. 2018
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G Bellemare. Deep reinforcement learning the edge of the statistical precipice. In Thirty- Conference on Neural Information Processing Systems, 2021.
Marcin Andrychowiez, Anton Raichuk, Piotr Stańczyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, et al. What matters for on-policy deep actor-critic methods? a large-scale study. In International Conference Learning Representations, 2020.
Oron Anschel, Nir Baram, and Nahum Shimkin. Averaged-dqu: Variance reduction and stabilization for deep reinforcement learning. In International conference machine learning, pp. 176-185. PMLR, 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. 1607.06600 2016.
arXiv preprint
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 2019.
Hamsa Bastani, Kimon Drakopoulos, Vishal Gupta, Jon Vlachogiannis, Christos Hadjicristodoulou, Pagona Lagiou, Gkikas Magiorkinis, Dimitrios Paraskevis, and Sotirios Tsiodras. Efficient and targeted covid-19 border testing via reinforcement learning, 2021.
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environ- ment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47: 253-279, 2013.
Mare G Bellemare, Salvatore Candido, Pablo Samuel Castro, Jun Gong, Marlos C Machado, Sub- hodeep Moitra, Sameera S Ponda, and Ziyu Wang. Autonomous navigation of stratospheric bal- loons using reinforeement learning. Nature, 588(7836):77-82, 2020.
Johan Bjorck, Carla Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normal- ization. arXiv preprint arXiv: 02375, 2018.
10

---

Published as a conference paper at ICLR 2022
Johan Bjorck, Xiangyu Chen, Christopher De Sa, Carla P Gomes, and Kilian Weinberger. Low- precision reinforcement learning. arXiv preprint arXiv:2102.13565, 2021a.
Nils Bjorck, Carla Gomes, and Kilian Q Weinberger. Towards deeper deep reinforcement learning with spectral normalization. Advances in Neural Information Processing Systems, 34, 2021b.
Stephanie CY Chan, Samuel Fishman, John Canny, Anoop Korattikara, and Sergio Guadarrama. Mcasuring the reliability of reinforcement learning algorithms. 2020.
Shi-Yong Yong Chen, Yang Yu, Qing Da, Jun Tan, Hai-Kuan Huang, and Hai-Hong Tang. Stabilizing reinforcement learning in dynamic environment with application to online recommendation. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1187-1196, 2018.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Kaleigh Clary, Emma Tosch, John Foley, and David Jensen. Let's play again: Variability of deep reinforcement learning agents in atari environments. 2018.
Cédric Colas, Olivier Sigaud, and Pierre- Yves Oudeyer. How many random seeds? statistical power analysis in deep reinforcement learning experiments. arXiv preprint /806.v:8006.8800 2018.
Cédric Colas, Olivier Sigaud, and Pierre- Oudeyer. A hitchhiker's s guide to statistical compar- isons reinforcement learning algorithms. ICLR 2019 Workshop RML, 2019.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Firdaus Janoos, Larry Rudolph, and Aleksander Madry. Implementation matters in deep rl: A case study ppo and trpo. In International conference on learning representations, 2019.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International Conference Machine Learning, pp. 1587-1596. PMLR, 2018.
Javier Garcia and Fernando Feráández. A comprehensive survey safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-1480, 2015.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. Convolutional sequence to sequence learning. In International Conference Machine Learning, pp. 1243- 1252. PMLR, 2017.
Florin Gogianu, Tudor Berariu, Mihaela Rosca, Claudia Clopath, Lucian Busoniu, and Razvan Pas- canu. Spectral normalisation for deep reinforcement learning: optimisation perspective. 2021.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: Dtdiv:/7060 706.02667. 2017.
Evan Greensmith, Peter L Bartlett, and Jonathan Baxter. Variance reduction techniques for gradient estimates in reinforcement learning. Journal of Machine Learning Research, 5(9), 2004.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. Advances in neural information processing systems, 2020.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. ICLR, 2021.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xic, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Peter Henderson, Riashat Islam, Philip Bachman, Joelle Pineau, Doina Precup, and David Meger. Deep reinforcement learning that matters. In AAAI, 2018.

---

Published as a conference paper at ICLR 2022
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 1448-566 PMLR. 2015.
Riashat Islam, Peter Henderson, Maziar Gomrokchi, and Doina Precup. Reproducibility of bench- marked deep reinforcement learning tasks for continuous control. ICML 2017 RML Workshop, 2017.
Haonan Jia, Xiao Zhang, Jun Xu, Wei Zeng, Hao Jiang, Xiaohui Yan, and Ji-Rong Wen. Variance re- duction for deep q-learning using stochastic recursive gradient. arXiv preprint arXiv:2007. 2020.
Longlong Jing and Yingli Tian. Self-supervised visual feature learning with deep neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 2020.
Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. Advances in neural information processing systems, 26:315-323, 2013.
Scott Jordan, Yash Chandak, Daniel Cohen, Mengxue Zhang, and Philip Thomas. Evaluating the performance of reinforeement learning algorithms. In International Conference on Machine Learning, pp. 4962-4973. PMLR, 2020.
Jens Kober, J Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(11))1288-1274 2013.
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. 2020.
Aviral Kumar, Justin Fu, G. Tucker, and Sergey Levine. Stabilizing off-policy q-learning via boot- strapping error reduction. In NeurIPS, 2019.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Re- inforcement learning with augmented data. 2020a.
Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. In Proceedings of the 37th Annual International Conference on Machine Learning (ICML), 2020b.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning in training neural networks. 2019.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. ICLR, 2021.
Yan Liu, Zhiyuan Jiang, Shunqing Zhang, and Shugong Xu. Deep reinforcement learning-basse based beam tracking for low-latency services in vehicular networks. In ICC 2020-2020 IEEE Interna- tional Conference on Communications (ICC), pp. 1-7. IEEE, 2020.
Ziwei Liu, Ban Xuan Dong, Mayank Misra, Yangyang Sun, Joseph Strzalka, Shrayesh N Patel, Fernando A Escobedo, Paul F Nealey, and Christopher K Ober. Self-assembly behavior of an oligothiophene-basee conjugated liquid crystal and its implication for ionic conductivity charac- teristics, Advanced Functional Materials, 29(2):1805220 2019.
Nicolai A Lynnerup, Laura Nolling, Rasmus Hasle, and John Hallam. A survey on reproducibility by evaluating deep reinforcement learning algorithms on real-world robots. In Conference on Robot Learning. pp. 466-489. PMLR, 2020.
Jerry Ma and Denis Yarats. On the adequacy of untuned warmup for adaptive optimization. AAAI, 2021.
Horia Mania, Aurelia Guy, and Benjamin Recht. Simple random search of static linear policies is competitive for reinforcement learning. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pp. 1805-1814, 2018
12

---

Published as a conference paper at ICLR 2022
Hongzi Mao, Shaileshh Bojja Venkatakrishnan, Malte Schwarzkopf, and Mohammad Alizadeh. Variance reduction for reinforcement learning in input-driven environments. ICLR, 2019.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. ICLR, 2018.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529 -33999533333303 2015.
Evgenii Nikishin, Pavel Izmailov, Ben Athiwaratkun, Dmitrii Podoprikhin, Timur Garipov, Pavel Shvechikov, Dmitry Vetrov, and Andrew Gordon Wilson. Improving stability in deep reinforce- ment learning with weight averaging. In Uncertainty in artificial intelligence workshop on uncer- tainty in Deep learning, 2018.
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International Conference on Machine Learning, pp. 7487-7498.
PMLR, 2020.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In NAACL, 2018.
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dor- Betp://e....ooooo//aaaaaaaaeeeee i 2019.
mann. Stable baselines3.
Tim Salimans and Durk Kingma, Weight normalization: simple reparameterization accelerate training of deep neural networks. Advances in neural information processing systems, 29:901- 909, 2016.
Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with momentum predictive representations. ICLR, 2021.
David Silver. Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 355-339,.599 2017.
Krishnan Srinivasan, Benjamin Eysenbach, Sehoon Ha, Jie Tan, and Chelsea Finn. Learning to be safe: Deep with a safety critic. arXiv preprint arXiv:2010. 14603, 2020.
Richard S Sutton. Learning to predict by the methods of temporal differences. Machine learning, (1):9 44, 1988.
Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. dm control: Software and tasks for continuous control, 2020.
Che Wang, Yanqiu Wu, Quan Vuong, and Keith Ross. Striving for simplicity and performance in off-policy drl: Output normalization and non-uniform sampling. In International Conference Machine Learning, pp. 10070-10080. PMLR, 2020.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous con-
trol: Improved data-augmented reinforcement learning. arXiv preprint arX6v:2107.00665, 2021a.
Denis Yarats, Rob Fergus, Alessandro Lazarie, and Lerrel Pinto. Mastering visual continuous con-
trol: Improved data- aata-auumented reinforcement learning, arXiv preprint arXiv:2107.09644 202
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. ICLR, 2020.
Zhenpeng Zhou, Steven Kearnes, Li Li. Richard N Zare, and Patrick Riley. Optimization of molecules via deep reinforcement learning. Scientific reports, 9(1):1-10, 2019.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
Tuning X. For tuning the hyperparameters
we search over the set {le-1, 1e-2, 1e-3, 3, 1e-4, 1e-5, 1e-6). For each value, we run the hopper task for short amount of time with five different seeds. We found that too large À penalized actions too much, and that even very small values like 1e-6 sufficed to give improvement.
Details on self-supervised Learning. We extract features from the convolutional network and the first feedforward layer of the critic. which projects the feature map to 50-dimensions. Feature vectors are normalized to unit norm before being fed into the self-supervised loss (Bachman et al., 2019). We use exponentially moving weights for the target network (He al. 2020). using the same decay parameters as for the target critic. We use a head for the online network which is simply two-layer MLP with hidden dimension 512 and final dimension 50 (Chen et al. 2020p. The two augmentations consider are simply two adjacent frames with augmentations that the DDPG agent of Yarats et al. (2021b) considers naturally. We use the square loss as proposed in Grill et al. (2020), which has been shown to be useful in smaller batch sizes. The self superviied loss is then simply added to the normal loss function for the critic.
experiments Nvidia Tesla V100 GPUs and Intel Xeon CPUs. The GPUs use CUDA 11.1 and CUDNN 8.0.0.5. We use Py PoTch 1.9.0 and python 3.8. 10. Our experi- ments are based upon the open-source DRQv2 implementation Yarats al. (2021b).
Infrastructure. We
Table 4: Variance across tasks reported by Yarats et al. (2021b). Mean reward and task variance often increase in tandem - to avoid bias towards easy tasks with high mean and thus high variance, rank tasks based upon relative variance a o/u. We variance, removing one duplicate: finger turn easy/hard. Used tasks are highlighted. Scores are measured just before million frames (i.e. frame 980000) since all runs have scores the million mark. All numbers are rounded to two decimal digits.
the five tasks with the highest relative
task
o/P
finger turn hard
0.74
268.90
198.36
walker walk
0.48
769.66
371.48
hopper hop
0.36
221.13
80.63
finger turn easy
0.31
558.52
174.19
acrobot swingup
0.31
177.65
54.77
reacher hard
0.22
628.29
135.57
walker run
0.20
538.59
110.02
cup catch
0.13
909.95
118.06
finger spin
0.12
860.30
104.18
quadruped run
0.12
446.49
53,51
quadruped walk
0.12
732.52
87.19
cartpole balance sparse
0.12
962.30
113.10
reach duplo
0.08
202.51
16.99
cartpole swingup sparse
0.08
760.11
60.11
cheetah run
0,07
710.24
52.43
reacher easy
0.05
931.54
50.86
pendulum swingup
0.04
838.49
36.38
hopper stand
0.02
917.85
20.45
cartpole swingup
0.02
864.68
15.47
walker stand
0.01
980.72
5.21
cartpole balance
0.01
993.45
5.00
14

---

Published as a conference paper at ICLR 2022
finger
reacher
900
400
600
reward
M
200
300
500
1000
500
1000
900
800
600
reward
400
300
500 episodes
1000
500 episodes
1000
Figure 7: Learning curves for the two tasks not shown in Figure [ The upper shows the mean reward with one standard deviation indicated by shading. The bottom row show learning curves for individual runs. Although the signal is noisy, several outlier runs fail to learn
Table 5: Hyperparameters used throughout the paper. These follow Yarats et al. (2021b) for the medium tasks.
parameter
value
learning rate
lc-4
soft-update
le-2
update frequency
stddev. schedule
linear(1.0, 0.1, 500000)
stddev. clip
0.3
action repeat
seed frames
4e3
batch size
256
n-step returns
discount 7
0.99
15

---

Published as a conference paper at ICLR 2022
acrobet
finger spin
hopper hop
120
200
reward
400
200
0
0-
0.5
0.0
1.0
0.0
1.0
prob
prob
prob
reacher
walker walk
900
800
reward
300
prob
prob
Figure 8: Rewards obtained when gradually replacing learned policy with random policy. At every time-step I, the action suggested by the learned policy replaced with random action with probability p. We start from final policy learned by the baseline algorithm and average over ten such policies. Across all tasks, the rewards decrease gracefully with a more random policy, suggesting that the environments relatively robust to perturbed policies.
finger
6.7
correlation
0.0-
W
0.0-
-0.6
-0.
-0.8
episode
walker
happer
1.0
1.0
correlation
-0.5
1000
500
1000
episode
episode
Figure 9: Correlation between the evaluation scores of two runs that share network initialization and seed experience, but otherwise have different randomness. Correlation is calculated across ten runs and plotted against time. The correlation noisy for many environments but oscillates around zero, suggesting that there is little actual correlation and that network initialization and seed experience have a small effect on performance.
16

---

Published as a conference paper at ICLR 2022
run
run 2
run 3
run 4
1000
reward
500
10
100
wm
50
[a
0.5
0.0
500
1000
500
1000
500
1000
500
1000
episade
episade
episode
episode
Figure 10: Four quantities shown during training for the combined agent. We show reward, norm of actor gradient V[]l norm of actor weight I| and average absolute action (a). The actions never saturate and learning starts quickly. Compare with Figure 3 which shows the same quantities for the baseline algorithm.
17

---

Published as a conference paper at ICLR 2022
hopper
finger
walker
900-
600
300
300
0
0
500
1000
500
10000
500
1000
300
800
800
400-
100
500
1000
episodes
episodes
episodes
300
800
4.00
400
500
1000
400
800
200
400
100
0
400
800
500
1000
episodes
episodes
Figure 11: Learning curves when using the combined agent. We show mean reward with one stan- dard deviation indicated by shading and learning curves for individual runs.
18

---

Published as a conference paper at ICLR 2022
1000
train reward
600
400
200
500
1000
500
1000
500
1000
500
1000
episode
episode
episode
episode
Figure 12: Rewards obtained during training for the runs illustrated in Figure rewards nonzero from the start of training. This implies that the agent observes rewards and can train on these. Thus, lack of rewards is not the cause of poor performance.
3
Note that the
cheetah run
quadruped walk
900
baseline
combined
600
combined+Ir
reward
600
300
300
baseline
combined
combined+IIr
0
500
1000
0
500 episodes
1000
episodes
Figure 13: The performance of the combined agents on two tasks when increasing the learning rate by a factor of 10. Performance improves markedly on these tasks. This highlights how hyperparam- eter tuning could further improve the results of Figure
rum
run
rum 3
run
run
[a]
0.5
500
1000
500
1000
500
1000
500
1000
500
1000
rum
6
rUL 7
run 8
rum
9
rum 10
[a]
0.25
500
1000
500
1000
1000
1000
1000
episode
episode
episade
episode
episode
Figure 14: We repeat Figure for all 10 seeds of the both pnorm agent of Table 2 Note that the action distribution does not saturate.
19

---

Published as a conference paper at ICLR 2022
r1l11
rmn
rum
rum 4
run
a
0.8
500
1000
500
1000
500
1000
500
1000
500
1000
rum
rum
rmm 8
rmn 9
rum 10
1.0
|a]
0.8
500 episode
1000
500
1000
0
500
1000
500 episode
1000
500 episode
1000
episode
episode
Figure 15: We repeat Figure 6 for all 10 seeds of the penalty agent of Table 2 Note that the action distribution does not saturate.
5 original tasks
16 additional tasks
1.00
1.00
combree
0.75
0.75
Fraction with
0.50
0.25
0.25
0.00
Score (T)
Score (r)
Figure 16: Performance profiles as computer by the rliable library of Agarwal et al. (021)). (Left) profiles over the original 5 tasks of Table with 10 seeds for combined++ and 40 seeds for the two other methods. (Right) profiles over the additional 16 tasks of Figure 5] with 10 seeds per method. Our combined agents improve performance, especially for the lowest quantiles. The effect is most dramatic for the original tasks, this is natural since we initially selected these as they suffered from high variance. Also note that the bascline the recent SOTA DRQv2 agent Yarats al. (2021b), which is very competitive so any improvements to it can be very significant.
20

---

Published as a conference paper at ICLR 2022
[ll]
66
[n]
050/004
optsce
opoxds
02:030
121006
[|]]
[4]
[F7]]
190
180
202
509
300
105
[4]
197)
197
194
[d]
[o]
[80]
146
140
1.0
100
600
54
105000
sptritt
1030tt
<pbods
<2305:
Figure 17: We repeat Figure for all 40 seeds. The plots show five quantities for the baseline agent of the walker task: reward, absolute average action (a], actor weight norm || actor gradient norm || and the logarithm of the actor gradient norm log / We add 1e-10 10 to the logarithm for numerical stability. Results are similar to Figure 3
21

---

Published as a conference paper at ICLR 2022
B IMPROVEMENTS FOR SPARSE TASKS
In Figure consider the five tasks where the combined agent suffers from the highest standard deviation. Four of these tasks have sparse rewards. This suggests that sparsity major driver of variance. To further decrease the variance, we now investigate fixes for four sparse tasks cartpole swingup sparse, pendulum swingup, reacher hard, and finger turn hard. We will consider three additional modifications which aim to specifically combat reward sparsity:
Avoiding training without rewards. For the sparse tasks, an agent might not see any rewards during e.g. the first 100 episodes. Perhaps training in the absence of any rewards harms the network weights. The fix we consider simple, do not use any gradient updates before at least one non-zero reward has been observed.
Increasing self-supervised learning. Since the agent might potentially not observe any rewards for 100 episodes, we consider extending the self-supervised learning ensure some supervisory signal. We simply extend the self-supervised time from 10k steps to the entire training process.
Removing clipped double q-learning. The agent of Yarats et al. (2021a) employs clipped dou- ble q-learning (Fujimoto et al., 2018) i... the agent uses two critic networks Q1, Q2 and takes Q min( Q1, Q2) to avoid overestimation. We hypothesise that the min-operator causes excessive pessimism in sparse environments where reward often are zero. For example, if Q1 = and Q2 > have min (Q1, Q2) 0. Thus, if one Q-network outputs only zeros and the rewards almost only zeros, the critic target of eq. ( will be mostly zeros and the other Q-network will be encouraged output mostly zeros too. Thus, the Q-network could potentially get stuck with mostly outputting zeros, an issue shown in Figure 18 We thus propose to use Qa = avg Q1, Q2). Through ablation experiments Table we find that using Qa avg(Q1. Q2) only for the critic loss (i.e.
eq. I but retaining Q = min Q2) for the actor update, works the best. We call this strategy asymmetrically clipped double Q-learning.
Results for these experiments are shown in Table 7 We see that decreasing pessimism and increasing self-supervised learning help the most. Increasing supervised learning helps the most for cart- pole swingup sparse and pendulum swingup. Note that these two tasks were not used in the ablation experiments of Table 2 Additionally, whereas feature learning was not the bottleneck for the base- line agent, it might be the bottleneck for the combined agent. Avoiding training without rewards does not seem to improve performance significantly. We conclude that increased self-supervised learning and asymmetrically clipped double Q-learning improve performance for tasks with sparse rewards. We add these two modifications to the combined agent and call the resulting agent com- bined++. Its performance across all 21 tasks is shown in Figure 5 It achieves even better results, and its average standard deviation is smaller than the baseline by factor > 3.
Table 6: Performance for various types of clipped double q-learning (Fujimoto et al. 2018). We compare exchanging Q min(Qi, Q2) with avg(Q1, Q2)) at locations when computing the critic loss (as in eq. 1 and when computing the actor loss. The baseline always uses min, avg both uses avg both locations. and avg critic/actor uses the avg only when defining the critic/actor loss. Using Q = avg(Q1. Q2) only when defining the critic loss performs the best. We call the resulting strategy asymmetrically clipped double Q-learning.
metric method cartpole sparse
finger turn hard
pendulum
reacher hard avg
J baseline 641.8
596.5
648.0
872.6
689.7
μ avg actor 80.9
422.2
776.0
672.6
487.9
μ
avg both 393.0
432.4
647.6
548.4
505.4
μ
avg critic
761.5
811.3
639.0
939.2
787.8
baseline 321.5
292.4
283.5
95.5
248.2
avg actor 242.6
258.8
138.9
164.8
201.3
avg both 393.3
171.9
323.1
312.0
300.1
0
avg critic
127.5
89.4
318.6
37.3
143.2
22

---

Published as a conference paper at ICLR 2022
Table 7: Performance for three fixes for sparse tasks: asymmetrically clipped double Q-lcarning (ac), learning until at least one non-zero reward has been observed (nz) and more self-supervised learning (ssl). We show mean reward and standard deviation of rewards, using ten seeds. Us- ing asymmetrically clipped double Q-learning and more self-supervised learning are the best fixes. Increasing the self-supervised learning helps the most for cartpole swingup sparse and pendulum swingup, tasks which were part of the five tasks consider first.
metric
method
cartpole sparse
finger turn
pendulum reacher hard
μ
baseline
641.8
596.5
648.0
872.6
ssltac
802.5
788.0
828.0
930.2
ssl+nz
607.6
659.4
813.5
869.3
ac+nz
561.4
770.6
624.6
860.9
618.2
663.4
657.2
882.9
ac
761.5
811.3
639.0
939.2
ssl
235.0
574.6
686.7
882.7
ssltnz +ac
742.7
729.8
758.0
922.7
baseline
321.5
292.4
283.5
95.5
0
ssltac
29.4
135.6
24.2
47.6
ssltnz
286.1
173.4
29.8
78.4
0
ac+nz
306.2
150.4
330.0
80.3
267.5
192.1
252.5
117.7
ac
127.5
89.4
318.6
37.3
ssl
359.7
236.7
272.5
80.1
0
ssl+nz +ac
151.2
130.9
220.8
45.3
23

---

Published as a conference paper at ICLR 2022
rum
run 2
run 3
run 4
run 5
reward
500
fna((t)
0.0
fnz(reward)
0.2
0.0
rum 6
rum 7
rum 9
run 10
reward
500
[nz(Q1)
0.0
(ne(araard)
0.4
0.0
100
200
100
200
0
100
200
0
100
200
0
100
200
episode
episode
episode
episode
episode
Figure 18: We illustrate three quantities for 10 runs of the pendulum swingup task using the com-
bined agent: the reward, [Q,)Qt) the fraction of non-zero Q-target values obtained over the training batches, and fnz (reward) the fraction of non-zero rewards obtained over the training batches. As shown in the two middle rows, the Q-values frequently "get stuck" at outputting almost only zeros.
24

---

Published as a conference paper at ICLR 2022
C NORMALIZATION FOR THE CRITIC
We now study why penultimate normalization helpful for the critic network. We consider three agents from Table ! baseline, actor pnorm, and both pnorm. The both pnorm agent uses penul- timate normalization for the critic, whereas the others do not. We plot two additional quantities during early training for the reacher hard task, using 10 new seeds. In Figure [9 we show the aver- age Q-values, measured over training batches, achieved by actions generated by the actor network. We see that the curves are much smoother when normalizing the critic, whereas they oscillate ag- gressively otherwise. Quickly oscillating Q-values makes it hard for the actor to learn good policy. We hypothesize that stabilizing the Q-values over time the primary mechanism by which critic normalization helps. In Figure 20 we show the average absolute difference between Q-values before and after a gradient update, as measured over training batches. We see that normalizing the critic often leads to Q-value changes which stabilizes learning as per Figure 19 Whereas the Q-value changes can trivially be made smaller by decreasing the learning rate, changes in Q-values depend non-linearly the learning rate. Further- more, too low learning rates often hurt generalization (Li et al. 2019). For increased legibility, the curves smoothed by averaging with window of width ten.
smaller and more stable over time. This in turn likely
25

---

Published as a conference paper at ICLR 2022
baseline
20
ا
بحمم
40
Mamatr
مممم
actor pnorm
10
Q
both pnorm
50
Q
20
100
200
100
200
100
200
0
100
200
0
100
200
episode
episode
episode
episode
episode
Figure 19: Average Q-values during early training for three agents on the reacher hard task. The both pnorm agent, which uses penultimate normalization for the critic, has Q-values that grow steadily during training. The other agents have Q-values which often fluctuate wildly during training. This suggests that penultimate normalization for the critic stabilizes Q-values, which provides less noisy signal to the actor.
26

---

Published as a conference paper at ICLR 2022
baseline
0.1
AQ
ببممممم
سسمم
0.1
AQ
0.0
actor pnorm
ب
AQ
0.1
0.0
AQ
0.1
both pnorm
الملربسس
AQ
0.025
0.000
0.05
AQ
w ابسل
0.00
100
200
100
200
100
200
100
100
200
episode
episode
episode
episode
episode
Figure 20: Average absolute difference AQ between Q-values before and after a gradient step. The Q-value changes are generally smaller and more stable over time for the both pnorm agent (which uses penultimate normalization for the critic). This likely stabilizes the Q-values as per Figure 19
27

---

Published as a conference paper at ICLR 2022
D LEARNING RATE STABILITY
To verify that our fixes increase the stability we now consider changing the learning rate. high learning rate can cause divergence (Li al. 2019). and learning rate stability can thus serve proxy for learning stability. We consider learning rates in the set 21e-4,3e-44 1e-3, 3e-3J, where 1c-4 is the default learning rate used by Yarats et al. (202bb) Results for the walker task are shown in Table We see that the baseline algorithm fails for large learning rates, the average rewards drop quickly while variance decreases when effective policy learned. We show the same experiment for additional tasks, but only one learning rate, in Table 2 Again, we see that the agent can tolerate much higher learning rates when using our modifications.
Table Performance when changing learning rates. The baseline algorithm fails with larger learn- ing rates whereas the combined agent succeeds. This suggests that our fixes improve the stability. Metrics are calculated over 10 seeds on the walker task.
metric
method
le-4
3e-4
le-3
3e-3
baseline combined
769.1
330.2
26.8
25.5
964.2
962.2
954.9
960.1
baseline
350.1
415.3
3.9
6.4
combined
6.2
4.7
5.6
5.1
Table 9: Performance for the combined agent and the baseline when using learning rate of 1e-3. The combined agent works well even with high learning rates, whereas the baseline fails. Statistics computed over 10 seeds.
metric method
acrobot
finger turn
hopper hop
reacher
avg
/
baseline
27.8
50.1
0.0
1.4
19.8
/
combined
137.4
618.6
227.0
727.2
427.6
0
baseline 35.6
49.9
0.0
2.0
21.9
combined
33.9
176.6
24.1
151.0
96.4
28

---

Published as a conference paper at ICLR 2022
E EVALUATION VARIANCE
We now consider the variance arising from evaluating RL agents on a finite set of episodes. Similar the bias- variance decomposition, that depends upon the randomness of the algorithm, and one term that depends on the randomness from the finite set of episodes we evaluate on. We be a set training runs and S set of evaluation episodes sampled. The performance is measured by
will see that the variance here can be divided into one part
perf
ΣΣ TV [R||S|
iERJES
Let us calculate the variance of this quantity. We will assume that rewards from different runs are independent, and that rewards from the same training run are independent conditioned on that run. We now have
ΣΣ iERJES
Var
R
The variance of sum of independent variables is simply the sum of their variances. Thus we have
Var
|R|ST
rij
R
Var
S
Σ
réj
iER jES
jES
Let = E[ryy. We then have
Σ ri /1) }
Var
S
rig
Badg samples
S
JES
JES
Here we explicitly calculate the expectation over random training runs (alg) and the random episodes sampled (samples). We can condition the expectation upon the random training run and then inte- grate over this expectation. Furthermore, let μ4s E(rij i), i.e. the expected value an episode sampled from training run i. We then have
Var
= Eaig Esamptes
(15)
Tij P Σ jES
S
jES
Eolg
E Esamples
ray p4 + Mi -M)
jES
[
21
Eaig
Esamples
Σ rij 14
[
jeS
(m (m- (m= )
Σ jES
+ 2E somples
rif 1i
S
( (1) I
+ Esamples
The middle part vanishes since μ4 = E[rgli.] For the first term, given a fixed training run the terms Ti are independent. This gives
29

---

Published as a conference paper at ICLR 2022
Table 10: Sample variance and algorithm variance defined in eq. ( Some tasks have high
variance and others have low. However, sampling more episodes does not decrease variance consid- erable in the 10 training seeds and 10 episodes setting, see eq. )
method
walker
hopper hop finger turn acrobot
reacher
sample
1922.1
450.6
143763.0 55215.6
11182.9 5710.9
176211.9
alg 80002.6
9774.0
35231.7
(1s)
2
Σ riy - 15114/ )91
Ealg
E Bsamples
Σ jES
rij M4
Eaig
Esamples
|S|2
jES
1
Eais
Esamples
Σ
(ris -a:-m)))) 11)2/3]
1
Ealg Bsamples
(ris - (vi)) 12)210
1512
(5
jES
We then have
1
Var(perf)
Ealg (Hagg 12)31
+
Bulg Esamples (f Halg |2|i]]]
(3)
R
R
S
algorithm variance
sample variance
Here Matg = 14g. The first term what we are interested in measuring. The second term can large if the environment is noisy and we sample too few episodes. 'he second term is straightforward to estimate: run the algorithm N times and measure the performance over M samples. For each run, calculate the variance over the samples. Then average this variance over all runs.
In Table 10 we estimate these quantities over ten training runs and ten episodes for the baseline see, the environments have very different characteristies. Some have high
algorithm. As sample variance and some have very low sample variance. is common to evaluate with ten training seeds and ten episodes. Can improve the variance by evaluating more episodes, say 100? Even for the task with the highest sample noise (reacher), the relative improvement in the standard deviation of the sample variance, i.e.
V (o +
10 02 cample)) (G
100 cample 1.195
no larger than 20 %. Thus, we conclude that only minor gains can come from evaluating on more episodes. Other environments outside the five consider, however, might benefit more from using more samples.
30

---

Published as a conference paper at ICLR 2022
F EXTENDED RELATED WORK
There are plenty of important applications of RL. Examples include RL for medical testing (Bastani et al. 2021), autonomous vehicles (Bellemare et al. 2020), chemistry (Zhou et al. 2019 Liu et al., 2019). , robotics (Kober et al. 2013) and communications infrastructure (Liu et al. 2020). Many more applications remain. Many of these applications require a low variance, but
inference (Bjorck et al. 2021a) can also be important for c.g. autonomous robots. Below we discuss some
papers closely related to ours in detail.
Engstrom et al. 2019) studies how implementation level details affects performance from policy gradient methods. They specifically find that code-level optimizations can have significant im- pact on performance, often rivaling that of algorithmic innovations. This insight agrees with our experimental observations that small modifications can significantly influence performance.
Henderson et al. 2018) presents among the first studies of variability and reproducibility in deep reinforcement learning. They demonstrate that many minor details can have outsized effect on the outcomes seeds, environment randomness, and even hardware non-determinism. In our work, have demonstrated how improved implementations can significantly decrease variance and our improved agent suffers from much smaller variability between seeds.
Islam al. (2017) studies variance and reproducibility for continuous control with policy gradient methods. They discuss what metrics are suitable for comparing algorithms and highlight how hyper- parameter tuning significantly affects outcomes and thus fairness of evaluations. Our results show that variance in continuous control can be significantly decreased with minor fixes, but we do not study variance from hyperparameter tuning which an important direction.
Mania et al. (2018) demonstrates that random search is in fact competitive method for finding
policies for continuous control tasks. This strategy computationally cheap, and the authors present a large study of over 100 seeds, demonstrating that the variance is high. RL still suffers from high variance, but it is not clear if random search still is competitive compared to RL agents.
Clary et al. (2018) studies variability for the Atari benchmark and finds large variability. In light of this, it is proposed that post-training performance should be reported as a distribution rather than as point estimate. Our paper does not study the Atari suite, but this is important benchmark that can model discrete tasks rather than continuous robot control problems.
Lynnerup et al. (2020) presents study over variance for evaluating RL agents on real-world robots. is concluded that documenting hyperparameters and performing rigorous statistical analysis of the results is imperative. We only consider simulated tasks which easy reproduce as long as source code is available, reproducibility becomes much harder in the real world.
Colas et al. (2019) discusses hypotheses testing in the context of RL experiments. They make recommendations regarding significance testing and discuss violations of statistical assumptions. Statistical testing has been criticized in later research (Agarwal al., 2021), and we refrain from using it.
Chan al. (2020) proposes multiple metries to measure notations of reliability in reinforcement learning, and presents an open-sources library to help with this. They discuss the distinction between variability during or after training and the use of significance testing. We have refrained from using hypothesis testing as it suffers from some issues (Agarwal et al.] 2021).
Jordan et al. (2020) shows that hyperparameters significantly affect the outcome of RL experiments, and proposes an evaluation strategy that takes this into account. To limit the computational footprint, we have used fixed hyperparameters. While we do not study it, hyperparameter stability very important for practical applications, and developing RL agents with such stability is an outstanding challenge.
Agarwal et al. (022 studies variance and statistical significance for RL, focusing especially on Atari tasks where they provide experiments over 100 seeds which shows outlier runs. Among other things, they propose using inter- cererruuuattteeeeeeee means and performance profiles for measuring perfor- mance. We adopt the use of performance profiles, but since we are especially interested in outliers inter-quartile-meant is not suitable for purposes.
31

---

Published as a conference paper at ICLR 2022
Fujimoto al. 2018 discusses how variance can be harmful to Q-learning, and propose several fixes to it, culminating in the TD3 agent. Some these fixes have become very popular, especially double clipped Q-values. In Appendix B we revisit double clipped Q-values and find that it can hurt
performance for sparse tasks.
Anschel et al. 2017) shows that averaging Q-values improve the stability and improve perfor- mance for a DQN agent. This strategy increases the computational footprint, and such ensembling known often work well for supervised learning. The paper mainly focuses on Atari tasks, in contrast, we consider continuous control tasks which arguably are closer to real-world applications.
Nikishin et al. (2018), in similar spirit, demonstrates that averaging weights can improve the stability of learning. The authors demonstrate that this can improve performance on Atari tasks, although only a handful of seeds appears to be used. To the best of our knowledge, this method is not widely adopted in RL., and we do not experimentally study it.
Jia et al. (2200) considers using classical variance reduction methods such as SVRG Johnson & Zhang 2013) to improve deep Q-learning. The authors introduce a recursive framework for com- puting the stochastic gradient estimate and demonstrate that it can improve the performance of DQN. Again, this method is not widely adopted in RL to the best of our knowledge, and we do not empir- ically study
Chen et al. (018) considers RL as applied to dynamic environment, motivated by applications to online recommendation. They propose stratified sampling replay, which relies on a prior customer distribution to decrease variance and the use of approximate regret. These ideas improve perfor- mance for DQN agents for tip recommendation system, while this is an important setting, it is far from continuous control.
Greensmith et al. (2004) studies the use of variance reduction methods for policy gradient methods. Specifically, they study the of baseline methods, which significantly decrease the variance and often is adopted in deep reinforcement learning. The use of baselines for policy gradient meth- ods is distinct from the setup we consider.
Parisotto et al. (2020) considers the use of transformers for reinforcement learning and finds that naively using them leads poor performance. To address this, architectural modifications are pro- posed that lead to more stable training and improve performance. Studying transformer architectures important for RL, but in this study, we only consider classical CNN and MLP networks.
Kumar et al. (0199) considers stability for off-policy reinforcement learning, focusing on a setting where the off ofl-policy data fixed and there is no further interaction with the environment. In this setting, it is demonstrated that bootstrapping error is key driver of instability, method to address this issue is proposed. Off-policy reinforcement learning very important for practical applications, but not the focus of our work.
Mao al. 2019 considers RI. where a stochastic exogenous process influences the environment. In such a setting, a gradient policy method with standard state-dependent baseline suffers from high variance, and the authors consider fixes to this. The setting considered in this work is distinct from the continuous control tasks we use.
Gogianu et al. (202T) studies the use of spectral normalization (Miyato et al. 2018) for improving the performance of DQN for Atari tasks. They find that this method can stabilize learning when applicd to the penultimate layer. Due to the similarity with the normalization methods we consider, consider this strategy in Table
Wang et al. 2020) studies the behavior of a SAC agent when the entropy bonus is removed. They find that the tanh activations saturate which effectively eliminates exploration since SAC adds ex- ploration noise inside the tanh non-linearity. This argument does not apply to DDPG which adds the exploration noise outside the non-linearity. We study the proposed method Table and find that underperforms, especially for the acrobot task.
Salimans & Kingmal (2016) proposes weight normalization, which reparametrizes the weights by decoupling magnitude and direction. They demonstrate that this idea improves performance in DQN and other settings. Despite the paper being well-known, this strategy has not become widely adopted in RL to the best of our knowledge. Similarly, to the best of our knowledge, batch normalization
lloffe & Szegedy 2015 Bjorck et al. 2018) is typically preferred in supervised learning.
32


Reviews:
[{"review_id": "zKW070J_Vcc", "reviewer": "Reviewer_N91S", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As pointed out in the paper, variance in RL is one of the main challenges to efficient and relevant research today, and this paper is a well-written exploration of a possible cause of this variance from the DRQv2 algorithm. \nThe paper studies the performance of the DRQv2 algorithm on several continuous control tasks, arguing that the variance in RL performance over several runs is primarily driven from a bimodal distribution of training runs. \nSome of the runs learn well, but some get stuck early on and don't learn. Several possible reasons are proposed for this, including poor initialization, feature learning, and unforgiving environments. The authors then argue that saturating tanh nonlinearities are to blame. Two remedies are proposed: a penalty on the squared norm of the action, and a capacity reduction of the model where the features of the penultimate layer are normalized and a simple linear layer is used to produce the action. \n\nOverall I think the paper identifies an interesting area and an interesting idea, but there is currently a lack of rigour in a lot of the arguments and seems to have some incorrect statements. Furthermore, there is a lack of references to some existing work with quite similar approaches. These points are elaborated below\n\nComments:\n\n1. In equation 2 and 3, it is claimed that the action $a$ is produced as $a_\\theta(s) = \\tanh (\\mu_\\theta(s) + \\epsilon \\cdot \\sigma_t), \\epsilon \\sim {\\cal N}(0, 1)$. However, in DDPG, the action is generated from a clipped truncated normal which is centered at $\\mu = \\tanh(f(s))$ for some learned \\(f\\) mapping states to $\\mathbb{R}$. Indeed, in the authors' code this seems to be exactly the procedure followed in the `Actor` method in the drqv2.py file, unlike as described in eqs 2, 3. It's possible I may have misunderstood this, so could the authors comment on this discrepancy?\n\n2. The authors claim that the failure to learn is primarily due to runs that get 'stuck' early on, with figure 1 as evidence. However, it's quite hard to read figure 1 due to the large number of overlapping lines making it hard to tell the relative number of runs. Could the authors add a third row to this figure showing a histogram of the final return achieved by each run? The authors' claim can then be reinforced if the histogram is bimodal with many at zero return and the 'good' return. Continuing on figure 1, it is unclear to me how well this supports the argument that the final returns are bimodal, in particular with reference to the `hopper` environment where it appears the majority of the runs are still improving after 1,000 episodes. If we run for longer, do they all continue to improve?\n\n3. Section 4 argues that the cause of the stuck runs is saturation with the tanh nonlinearity. However, I think this section seems to leave out some important details which could help show convincingly that the 'stuck run' cause is the tanhs. Firstly, it would be useful to have a version of fig 3 that is done on a population level, like figure 1. This would allow the reader to compare the runs in figure 1 and 3, and verify that this behaviour happens consistently. (It may be too busy to show these all runs, but e.g. a large high-resolution plot showing 20 randomly-selected runs from the dataset of 40 seeds would be a great figure to eventually have in an appendix). Secondly, it would be useful to focus on the size and gradient of the final layer specifically. As it is, it's hard to judge what's happening when looking at the norm of l and w since they are taken over the whole actor's weights. Plotting the pre-tanh $a_\\theta^{\\text pre}$ would also be more useful than a flat saturated 1 value for the norm of the action. A logarithmic scale would also be useful for the gradient norm.\n\n4. In section 4, did the authors try initializing the last layer's weights to be much smaller than the other layers? It has been suggested previously that the last policy layer should be initialized to 100x the weight of the other layers [1], and it seems that this may combat the saturating nonlinearities.\n\n5. It is surprising to me that the $\\lambda$ value 1e-6 is sufficient, as this seems very small. Could the authors elaborate on why such a small value is sufficient to change the training dramatically?\n\n6. A similar idea, that of normalizing the features extracted for the agent in an off-policy algorithm, has been explored in quite a bit of detail for the DQN. In particular, [2] investigates weight normalization for DQN, and in [3] spectral normalization is applied in the layer before the action outputs, which seems quite similar in spirit to the normalization in this paper. [3] Also shows that this increases robustness with respect to hyperparameters. Can the authors comment on this work?\n\n7. At the moment the technique presented is only evaluated on DDPG. However, it seems that this saturating tanh problem could be present any other RL algorithm with bounded action range. If this method could be shown to consistently help with other algorithms, this would significantly increase the significance of this result. I appreciate that computation resources are not free, but if the authors could show even preliminary results that this method works with e.g. SAC, PPO, etc, this would be very significant.\n\nI am happy to reconsider my recommendation based on feedback from the authors and other reviewers.\n\nSummarizing:\n\nStrengths:\n+ Convincing argument about the importance of the tanh saturating in DDPG\n+ Thorough evaluation of a few different ways of addressing this problem\n\nWeaknesses:\n+ Missing some relevant related work on normalizing image features in off-policy RL\n+ Some possible incorrect statements about the action distribution\n+ Plots that could be made easier to read and interpret\n+ Limited experimental evaluation only on DDPG\n\n\n[1] What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study (Andrychowicz et al, ICLR 2021)\n\n[2] Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks (Salimans and Kingma, 2016)\n\n[3] Spectral Normalisation for Deep Reinforcement Learning: An Optimisation Perspective (Gogianu et al 2021)\n\n----------------------------------------------------------\nUpdate 2021-11-21\nBased on the authors' additional results, correction of some inaccuracies and comparison to previous work on normalization, I have raised my score to 6.", "summary_of_the_paper": "The paper studies the variance of the DDPG algorithm on continuous control tasks. They identify that the variance in rewards arises in part due to the failure of some runs to learn. They argue that the failure to learn is due to saturating tanh nonlinearities, and suggest two methods to address this problem. ", "summary_of_the_review": "The paper identifies an interesting phenomenon and has good proposed fixes, but there is currently a lack of rigour/convincing arguments and seems to have some incorrect statements. There is a lack of references to some existing work with similar approaches. The method has somewhat limited significance as it is only demonstrated to be useful with DDPG.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fnymOVY34l-", "reviewer": "Reviewer_ti7w", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n -  While existence of outlier runs is well known in deep RL, for example, see evaluation of 100 seeds on continuous control tasks in [1] (Figure 3)  and 100 seeds on on Atari 100k in [2] (Appendix), the empirical study on continuous control tasks to find what causes such runs is interesting. For example, it seems to rule out network initialization and feature learning. \n- The paper suggests a number of simple fixes for control tasks, such as normalizing feature activations in actor and critic, which are easy to implement and often lead to smaller variance on the 5 tasks in the paper.\n- The paper is well-written and easy to read.\n\nWeaknesses:\n\n- One of the main limitations of empirical study is that its conclusions are specifically tied to network architectures that use a tanh squashing function in the policy network. For example, policy gradient methods (such as PPO/TRPO) or DQN-style methods typically use `relu` activations and are unlikely to suffer from saturating nonlinearities. The same limitation also applies to the proposed fixes which try to prevent tanh from saturation. This limits the possible impact of this work.\n\n- The visualizations on walker walk seems somewhat misleading as this is the only task where the proposed fixes results in ~60x lower variance. This doesn’t seem to be true for other tasks such as finger turn where the improvements are significantly smaller. Same is true for the experiments for learning stability (can the authors possibly repeat it on other tasks too?) It is unclear if this is the case of showing evidence only on the tasks which support the hypothesis. \n\n- The fix which works well is combining feature normalization for both actor and critic but the rationale for normalizing features in the critic network is not explored in the paper (I believe that the critic network uses ReLU activations only). This makes me wonder if we actually understand why the proposed fix improves performance or reason behind poor outlier runs on tasks other than walker walk?\n\n- The use of a single benchmark with only a small subset of tasks with the high-variance issue casts doubt about the generality of the findings and proposed fixes. I feel lack of compute is not a solid argument to evaluate on only a small number of environments. To give an analogous example from deep RL, [3] show that methods proposed for tackling hard-exploration methods hurt performance on easy-exploration tasks. So, do the proposed fixes actually result in any performance improvement on the remaining tasks used by Yarats et. al (2021b) in the medium tasks. Furthermore, do the combined fixes in Section 4.2 generalize to harder tasks?\n\n- **(Missing result)** The paper introduction mentions reducing variance on a humanoid locomotion task by two orders of magnitude, however I couldn’t find this experiment anywhere.\n\nOther comments (Minor):\n\n- There is a distinction between self-supervised methods like BYOL  and contrastive methods like SimCLR. The authors interchange them throughout the paper which is technically incorrect. Furthermore,  CURL actually hurt performance compared to the baseline method on Atari and their reported improvement resulted from mismatch in evaluation protocols [2] and seems to be a poor example to cite.\n- Properly warm-starting / reusing features probably require lowering the learning rate for feature layers so that learning may not lose this initialization quickly after a few updates. Thus, it is unclear if feature learning experiments actually capture this or not. \n- Reporting mean and std is a specific kind of interval estimate, so the statement in related work about being different than [2] didn’t compile for me. \n- The results in Table 2 were somewhat overwhelming to read because of the plethora of numbers  across multiple rows and columns, maybe reporting aggregate statistics such as mean/median/ interquartile mean with CIs might be better visualization.\n- More detailed discussion about outlier runs in RL and discussing this in related work.\n- Spelling: intractable → intractable\n- While I appreciate the use of a large number of seeds, there is a trade-off between generalizable findings and statistically robust findings. Using a single task with 200 seeds is an extreme example of this.  As such, my opinion is that the conclusions in this work would be more generalizable if the authors used a large number of tasks but maybe slightly smaller number of seeds (maybe 30?) as that provides us with more diverse data points to validate our findings. \n\n[1] Mania, Horia, Aurelia Guy, and Benjamin Recht. \"Simple random search of static linear policies is competitive for reinforcement learning.\"  NeurIPS (2018).\n\n[2] Agarwal, Rishabh, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G. Bellemare. \"Deep reinforcement learning at the edge of the statistical precipice.\"  NeurIPS (2021).\n\n[3] Taiga, A.A., Fedus, W., Machado, M.C., Courville, A. and Bellemare, M.G., 2021. On Bonus-Based Exploration Methods in the Arcade Learning Environment. ICLR (2020).", "summary_of_the_paper": "This paper investigates sources of poor outlier runs in a DDPG-based agent on 5 image-based DM control tasks. The empirical analysis suggests that such outlier runs result from saturating tanh nonlinearities due to large-valued activations in the policy network. To prevent saturation of tanh, the paper proposes and evaluates fixes and shows that combining a bunch of them results in better performance with lower variance.", "summary_of_the_review": "Although I liked the empirical study, the analysis and the proposed findings may not generalize beyond the setup used by the paper (tanh nonlinearities) and lack of thorough evaluation on a large set of environments. Furthermore, there are other weakness such as missing results and fixes without much motivation with possibility of visualizing results on a cherry-picked environment. Overall, I feel the paper can be significantly improved by addressing some of these weaknesses. \n\n\n-------------------------------- Update after rebuttal -----------------------------------------------\n\nBased on the discussion with the authors, I feel that the paper can be accepted at the conference. I'd like to update my score from a 5 (weak reject) --> 7 (accept) but the ICLR review scale only allows me to pick a score of 6 or 8. Since a very related work (highlighted by reviewer cHfn is not discussed), I decided to go with a score of 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7Fp26CKw33a", "reviewer": "Reviewer_cHfn", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\nThe main strength of the paper is its focus on an important problem and studying it effectively. Being able to pinpoint the problem of variance to exploding activation of actions is definitely a success. Various experiments are conducted based on well-formed hypotheses trying to rule out other suspected sources.\n\nThe main weakness is not thoroughly discussing the result in table 2. Which approaches really worked? It is claimed that “Normalizing activations seems to stably improve performance and decrease rewards”. If I look at it closely, actor normalization achieves the second largest variance on finger turn. Actor+critic normalization does not achieve a low variance on that task either.\n\nAnother weakness is that the choice for combining different strategies isn’t justified. The penalty got mixed results from table 2 and so did the layer norm. Early contrastive learning, which is added in the combined fix, came out of nowhere. To me, it seems like different combinations were tried and this one somehow worked. Can you shed some light on it? How does Figure 4 look like when we only try normalization or penalty or a combination of just those two? What would the mean and the standard deviation for the combination of just these two be?\n \n", "summary_of_the_paper": "This paper is an empirical study of the issue of high variance in RL. Learning continuous control from pixels is adopted as the setup. The authors demonstrated that the failure of a few runs is responsible for much of the variance. To look for the source of the variance, they separately studied the impact of the initial and training-time randomization as well as variation in representation learning between runs, none of which were found to be majorly responsible for the variance. The authors hypothesized and isolated the exploding activation of the actions to be the main source of the variance, which cause some runs to saturate beyond salvation and fail at performing. The hypothesis is tested by employing several measures to counter this phenomenon. Employing normalization of the penultimate features turned out to be particularly helpful.\n", "summary_of_the_review": "It is a paper on an empirical study of an important question in RL with satisfactory claims and fixes. In the latter part of the paper, some thorough discussions are missing, and the choices made around the combined fixes are not well motivated, for which I would like to hear from the authors.\n\n*** Updated ***\n\nThe authors addressed all my concerns satisfactorily. They even addressed a last-minute question about a highly-related work that I didn't notice before. The authors clearly described the difference between that work and theirs. \n\nMore elaborately, the authors promptly and elaborately discussed a comparison to this newly discovered work. They not only showed that they can discuss this work in their submission fairly, but they can also utilize it to improve their submission. Moreover, their new results showed that the existing work was inferior to their proposed solution. The authors' explanation also made sense that the existing approach normalized the pre-tanh action, which limited the action space too restrictively.\n\nHence, I increased my score from 6 to 10. Thank you!", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5Kvz2Ovi7AZ", "reviewer": "Reviewer_J8Mk", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I believe the problem this paper concerns is very important to the area. High instability in the early training process is a serious issue that stops RL from being more widely applied. I like the authors eliminating possible factors that might cause high variance before finally identifying the main cause. The improvement of the combined agent seems solid. However, I do have the following concerns.\n\n1. It is not clear whether the similar variance-reducing techniques could be applied to other environments. For example, penalizing saturating actions only applies when the actions are continuous variables, which is not the case in Atari games or other environments with discrete action space.\n\n2. I don't fully understand the logic of testing environment sensitivity. The variance is guaranteed to reduce since the mean reward reduces. I think one should compare the variance of the original algorithm to the algorithm with a pre-trained good policy that may automatically eliminate the possibilities of reaching bad states, which may reduce the variance.\n\n3. Some techniques that are used to reduce variance are not new and are commonly used in previous literature like penalizing actions and learning rate warmup.\n\n-------------------------------- Update after rebuttal -----------------------------------------------\nI have read the insightful discussion on the newly-discovered work and I appreciate that the authors' responded quickly with new results and pointed out the novelty compared to the new paper. I support accepting the paper.", "summary_of_the_paper": "This paper gives a systematic evaluation on different factors that may cause the high-variance of actor-critic algorithm on various robotic control problems. They found that the main source of variance is from the numerical instability. The authors then propose four training techniques to mitigate the instability, which significantly reduced the variance.", "summary_of_the_review": "I work mostly on the theoretical side of RL so I am not sure how significant the experimental results are in this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers unanimously appreciated the quality of the experiments. The main point raised was about the related work by Wang et al. but that was addressed by the authors in the rebuttal. I thus encourage the authors to make sure that discussion is reflected in the final version of their work.
Decision: Accept (Poster)