Towards Deeper Deep Reinforcement Learning with Spectral Normalization
Authors: Johan Bjorck
Carla P Gomes
Kilian Q Weinberger
Venue: NeurIPS 2021 Poster (2021)
Keywords: reinforcement learning
continuous control
spectral normalization

Abstract: In computer vision and natural language processing, innovations in model architecture that increase model capacity have reliably translated into gains in performance. In stark contrast with this trend, state-of-the-art reinforcement learning (RL) algorithms often use small MLPs, and gains in performance typically originate from algorithmic innovations. It is natural to hypothesize that small datasets in RL necessitate simple models to avoid overfitting; however, this hypothesis is untested. In this paper we investigate how RL agents are affected by exchanging the small MLPs with larger modern networks with skip connections and normalization, focusing specifically on actor-critic algorithms. We empirically verify that naively adopting such architectures leads to instabilities and poor performance, likely contributing to the popularity of simple models in practice. However, we show that dataset size is not the limiting factor, and instead argue that instability from taking gradients through the critic is the culprit. We demonstrate that spectral normalization (SN) can mitigate this issue and enable stable training with large modern architectures. After smoothing with SN, larger models yield significant performance improvements --- suggesting that more ``easy'' gains may be had by focusing on model architectures in addition to algorithmic innovations. 


Towards Deeper Deep Reinforcement Learning with Spectral Normalization
Johan Bjorek;" Carla P. Gomes, Comell University
Kilian Q. Weinberger
Abstract
In computer vision and natural language processing, innovations in model architec- ture that increase model capacity have reliably translated into gains performance. In stark contrast with this trend, state-of-the-ar reinforcement learning (RL) algo- rithms often use small MLPs, and gains in performance typically originate from algorithmic innovations. It is natural to hypothesize that small datasets in RL necessitate simple models to avoid overfitting; however, this hypothesis is untested. In this paper we investigate how RL agents are affected by exchanging the small MLPs with larger modern networks with skip connections and normalization, fo- cusing specifically actor-critic algorithms. We empirically verify that naively adopting such architectures leads to instabilities and poor performance, likely contributing to the popularity of simple models in practice. However, show that dataset size is not the limiting factor, and instead argue that instability from taking gradients through the critic is the culprit. We demonstrate that spectral normalization (SN) can mitigate this issue and enable stable training with large modern architectures. After smoothing with SN, larger models yield significant performance improvements - suggesting that more "easy" gains may be had by focusing model architectures in addition to algorithmic innovations.
Introduction
In computer vision and natural language processing (NLP), competitive models are growing increas- ingly large, and researchers were often shallow [40] with performance dropping for excessively deep models [29]. However, ever since the introduction of batch normalization [31] and residual connections [29], performance has improved more or less monotonically with model scale [64]. As result, competitive models in computer vision and NLP are growing ever larger [10, 56], and further architectural innovations continuously researched [14].
train hallion-parameter models [13, 36). The earliest neural networks
In stark contrast with this trend, -fate-of-tte the-ar art (SOTA) reinforcement learning (RL) agents often rely on small feedforward networks [35, 37, 39] and performance gains typically originate from algorithmic innovations such novel loss functions [58, 62, 68) rather than increasing model capacity. Indeed, a recent large-scale study has shown that large networks can harm performance in RI [2]. It is natural suspect that high-capacity models might overfit in the low-sample regime common in RI evaluation. Imagenet contains over a million unique images whereas RI. is often evaluated in contexts with fewer environment samples [37, 39j. However, date, this overfitting hypothesis remains largely untested.
To address this, study the effects of using larger modern architectures in RL. By modern archi- tectures we mean networks with high capacity, facilitated by normalization layers [6, 31] and skip connections [29]. We thus depart from the trend in RL of treating networks as blxck-box function
"Conresoondence to: Johan Bjorck cojbl255@oomell.cdu>>>> cornell.eu>>
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

approximators. To limit the scope and compute requirements we focus on continuous control from pixels [65] with two actor-critic agents based on the Soft Actor-Critie (SAC) [25. 35] and Deep Deterministic Policy Gradients (DDPG) [43, 71] algorithms. Actor-critic methods form the basis of many SOTA algorithms for continuous control [35, 38, 39). As can be expected, we demonstrate that naively adopting modern architectures leads to poor performance, supporting the idea that RL does not benefit from larger models. However, we show that the issue is not necessarily overlitting, but instead, that training becomes unstable with deeper modern networks. We hypothesize that taking the gradient of the actor through the critic network creates exploding gradients [53] for deeper networks. We connect this setup with generative adversarial networks (GANs) [22], and propose to use simple smoothing technique from the GAN literature to stabilize training: spectral normalization [47].
We demonstrate that this simple strategy allows the training of larger modern networks in RL without instability. With these fixes, we can improve upon sfate-of-the-art RI. agents on competitive continu- ous control benchmarks, demonstrating that improvements in network architecture can dramatically affect performance in RL. We also provide performance experiments showing that such scaling can be relatively cheap in terms of memory and compute time in RL from pixels. Our work suggests that model scaling is complementary to algorithmic innovations and that this simple strategy should not be overlooked. We summarize our contributions as follows:
We verify empirically that large modern networks fail for two competitive actor-critic agents. We demonstrate dramatic instabilities during training, which casts doubt on overfitting being responsible.
We argue that taking the gradients through the critic is the cause of this instability. To combat this problem we propose to adopt spectral normalization [47] from the GAN literature.
We demonstrate that this simple smoothing method enables the use of large modern networks and leads to significant improvements for SOTA methods on hard continuous control tasks. We further provide evidence that this strategy is computationally cheap in RL from pixels.
2 Background
2.1 Reinforcement Learning
Reinforcement learning tasks are often formulated as Markov decision processes (MDPs), which can be defined by a tuple (S, A. r,r) [63]. For continuous control tasks the action space A and the state space S are continuous and can also be bounded. Each dimension of the action space might for example correspond to one joint of humanoid walker. At time step t, the agent is in a state S, ES and takes an action a A arrive a new state S S. The transition between states given an action is random with transition probability P S x S x A > [0, oo). The agent receives a reward r, from the reward distribution at each timestep f. The goal is typically to find a policy π S + A that maximizes expected discounted reward E E 7rrr). where 0 0<><1 is a pre-determined constant. In practice, it is common to measure performance by the cumulative rewards E. rr.
2.2 Actor-Critic Agents
Actor-critic methods have roots in Q-learning [69] and form the backbone of many recent state- of-the-art algorithms [35, 38, 39]. We focus on two popular and performant actor critic methods: SAC [25] and DDPG [43]. Given some features Φ(s) that can be obtained from any state E S, the actor-network maps each state-feature d(s) to a distribution To(s) over actions. The action distribution is known as the policy. For each dimension, the actor-network outputs an independent normal distribution, where the mean (s(S)) and possibly the standard deviation ( come from the actor-network. The action distribution is then obtained as
(o = (anh( + e ② (s)),
6~,
(1)
The tanh non-linearity is applied elementwise, which bounds the action space to [-1, 1ln for n = dA). Sampling e allows us to sample actions ag from the policy (s). DDPG typically adds the exploration noise outside the tanh non-linearity. To promote exploration, SAC instead adds the entropy of the policy distribution (TT(s))) times a parameter a to the rewards. The critic network outputs Q-value Q (a, s) for each state and action In practice, one can simply

---

concatenate the feature ((s) with the action and feed the result to the critic. The critic network is trained minimize the soft Bellman residual:
min (u(aa - I ((4)))) a (H(())]))
Here, rt is the obtained reward and [[[(au+1.S S1+1]] is the Q-value estimated by the target critic a network whose weights are the exponentially averaged weights of the critic. One can also use multi-step targets for the rewards [62]. The loss (2) is computed by sampling transitions from replay buffer [48]. Since the Q-values that the critic outputs measure expected discounted rewards, the actor should simply take actions that maximize the output of the critic network. Thus, to obtain gradient step for the actor, both SAC and DDPG sample an action ag from To (st) and then take derivatives of (as, s) with respect to 0, That is
V aQ 8)
(3)
00
As the critic differentiable neural network, the derivatives are taken through the Q-values (ag, s). In practice, the features (ss) can be obtained from convolutional network trained by backpropagating from (2). For further details, see Haarnoja et al. [25], Lillicrap al. [43].
3 Motivating Experiments
3.1 Experimental Setup
For experimental evaluation, we focus on continuous control from pixels, a popular setting relevant to real-world applications [33, 34]. Specifically we evaluate the DeepMind control suite [65], which has been used in [26, 27, 35, 39]. We use the tasks considered Kostrikov et al. [35] and evaluate after 000,00 samples, which has become common benchmark [35, 39, 41]. If learning crashes before 500,000 steps (this occasionally happens for modern networks). we report the performance before the crash. We will primarily focus the image augmentation based SAC agent of Kostrikov et al. [35], known as DRQ, and adopt their hyperparameters (listed in Appendix B). This agent reaches Krate-of-the-ee performance without any nonstandard gells-and-whisteee and has excellent open-source codebase. Unless specifically mentioned, all figures and tables refer to this agent. Its network consists of common convolutional encoder that processes the image into a fixed-length feature vector. Such feature vectors are then processed by the critic and actor-network, which are just feedforward networks. We will refer these feedforward networks as the heads. The convolutional encoder consists of four convolutional layers followed by a linear layer and layer normalization [6] and the heads consist of MLPss with two hidden layers that concatenate the actions and image features; see Appendix C for details. In Table 1 we also consider the augmentation-base Yarats al. [71], using the default hyperparameters proposed therein. Both agents use the same network design. In all experiments, we only vary the head architecture. We evaluation over ten seeds.
DDPG agent of
modern
900
avg. reward
mlp
600
300
0
cheetah
walker pendulum
reacher
walker
cartpole
cartzole
cartpole balence sparse
reacher
walker
swingup
run
run
spin
swingupp
hard
hop
stand
walt
balance
seioop sparse
easy
stand
Figure 1: Performance when using modern network architecture with skip connections [29] and normalization [6], averaged across ten seeds. Using modern architectures does not enable deeper networks; instead, performance often decreases sometimes catastrophically while increasing for a handful of tasks. This is in stark contrast with supervised learning where performance typically improves monotonically with model capacity.
3

---

3.2 Testing Modern networks
Two crucial architectural innovations which have enabled deep supervised models are residual connections [29] and normalization [31]. We test if adopting such modern techniques allows to successfully scale networks. Specifically, we use the feedforward network (and not the attention module) found in a Transformer block [67] and will refer to this architecture modern. Each block consists of two linear transformations W1; W2 with ReLu between and layer normalization [6] that is added to residual branch. I.e. the output of a layer is x wyRelu (1 norm(z)). Compared to the original architecture, normalization is applied before the feedforward blocks instead of after, which is now favored in practice [50, 70J. We use four transformer blocks for the critic and two for the actor. Results from these experiments are shown in Figure I, with one standard error given. We see that these modifications often decrease performance. Thus, actor-critic agents have seemingly not reached the level where model capacity improves performance monotonically, and tricks used supervised learning do not seem suffice. This is perhaps not very surprising as typically use simple feedforward networks.
RL agents
3.3 Testing Deeper Networks
Whereas naively adopting modern networks failed, we now investigate if simply making the MLP networks deeper a competitive strategy. To do this, we increase the number of MLP layers the actor and critic from two to four hidden layers while keeping the width constant. Results for the two configurations are shown in Figure 2, with one standard error given. We see that for some tasks, the performance improves, but for many others, it decreases. Simply scaling the network does not monotonically improve performance, as typically is the case in supervised learning. For practitioners, the extra computational burden might not make it worthwhile to increase network scale for such dubious gains.
4layers Zlayers
900
avg reward
600
300
ball
cartpote
cheetah
walker
finger
perdulum
reacher hand
hopper
hopper stand
walker
cartoate
carppote
reacher woingup
campole
walker
awingup
run
run
apan
sa nguu
hop
walc
balance
ballnce
easy
stand
Figure 2: The performance when making the networks deeper. Performance is averaged across seeds and plotted for 15 tasks from the DeepMind control suite [65]. Making the network deeper does monotonically improve performance is typically the case supervised learning. Instead, some tasks benefit, and some tasks suffer. As we shall see later. it is possible to improve the performance by stabilizing training, suggesting that deeper networks cause some instability.
3.4
Is it overfitting?
Both making the networks deeper and adopting modern methods such as normalization and residual connections do not monotonically improve performance, and can occasionally lead catastrophic drops in performance. It is natural to think that the small number of environment transitions available leads to overfitting for large models, which is known to be harmful in RL [76]. Indeed, since training and collecting samples are interleaved in RL, RL agents need to generate useful behavior when only 10 % of the training has elapsed, at which point only 10 % of the sample budget available. Such an overfitting hypothesis is relatively easy to probe. As per eq. (2), the critic is trained to minimize the soft Bellman residual. If overfitting truly was the issue, we would expect the loss to decrease when increasing the network capacity. We investigate this by comparing a modern network against smaller MLP an environment where the former performs poorly - cartpole-swing-up First, we study the critic loss as measured over the replay buffer which plays the role of training loss in supervised learning. Results averaged over five runs are shown in Figure The losses initially start on the same scale, but the modern network later has losses that increase dramatically suggesting

---

that overfitting is not the issue. Instead, training stability could be the culprit, as stable training should enable the network to reach lower training loss. To probe this, we simply plot the norm the gradients for the critic and actor networks and see that they indeed are larger for the large modern networks. Furthermore, the gradient magnitude increases during training and shows large spikes, especially for the actor. This suggests that training stability, rather than overfitting, is the issue.
critic
agent
mip
mlp modern
modern
300
20
log l
200
200
10
0-
10
20
10
20
actor
critic
16
mlp
mip
modern
modern
20
log [[V))
|| Vill
log
0-
20
steps (thousand)
steps (thousand)
Figure 3: Training dynamics when using small MLP deeper network with normalization and residual connections. Top left. The loss, as per eq. (2), of the critic during training. The loss does not decrease when making the network larger, suggesting that overfitting not the issue. Top right. The rewards obtained. The deeper network fails to learn and improve its performance. Bottom left & right. The gradient of the actor and critic during training. For the deeper network, training becomes unstable with occasional spikes and large and growing gradients. Notice the logarithmic scale.
4 Stabilizing with Spectral Normalization
To improve the stability of actor-critic algorithms, let us consider a simple model. We will assume that the action space has one degree of freedom, i.c. A R. We also assume that have some fixed features á which maps states to n--iimensional vectors, i.c. Φ(S) = R". The critic and actor both modeled MLPs with N layers and a nonlinearity (e.g. ReLu) applied elementwise after each layer. The weights are given as matrices wi or w for layer i in the actor and critic respectively. We then have y; f (w;z;) for layer i. If we let the functions representing the actor and critic be denoted by A and Q, and let denote vector concatenation, we have:
A(s) = (II f wf) o d(s) @(s) = Q(A(s,,s)) (II
/ o w o (A(s))l((s)
(4)
Recall that the Lipschitz constant of function f is the smallest constant such that f < C'||æ= yl for all It is straightforward to bound the Lipshitz constant of the critic, it made up of transformations with bounded constants themselves. The Lipschitz constant of the linear layer w; is the operator norm ||l;]] equal to the largest singular value dmax of wi- We have:
[[Q(s)] < (IIA(s)II + [l)(s) II ||]||]
(5)
Here ||]l|| is the Lipschitz constant of the function ʃ. which for Relu is 1. Equation (5) bounds the smoothness of the critic in the forward pass. A function that is I.-Lipschitz smooth also has gradient bounded by L. Thus, if we could ensure that the critic is L-smooth, that would also imply
5

---

that gradients eQ* being propagated into the actor, as per (3), are bounded. Equation (5) suggest das that the critic could be made smooth if the spectral norms of all layers are bounded. Fortunately, there is method from the GAN literature which achieves this: spectral normalization [47]. Spectral normalization divides the weight W for each layer by its largest singular value Umxx which ensures that all layers have operator norm The singular value can be expensive to compute in general, and to this end. spectral normalization uses two vectors u and v approximately equal to the right and left vectors of the largest singular value, and approximate max Rs u We. The forward pass becomes:
Wx
Wr
O max (W)
W
For this to work properly, the vectors and should be close to the vectors corresponding to the largest singular value. This is achieved via the power method [46, 47] by taking:
WT u// | ull vt Wu// Well
v
By repeating this procedure for all layers, we ensure that the spectral norms of all layers no larger than one. If that is the case, eq. (5) suggests that the critic should be stable in the forward pass. This would then bound the gradients being propagated into the actor as per eq. (3). Figure 3 has shown that exploding gradients are associated with poor performance, and we thus hypothesize that applying spectral normalization should stabilize learning and enable deeper networks.
5
Experiments
5.1 Smoothness Enables Larger Networks
We first investigate whether smoothing with spectral normalization allows to use deeper modern networks. To do this, we simply compare using the modern network defined in Section 3.2 without and with spectral normalization. Specifically, for both the actor and the critic, we apply spectral normalization to each linear layer except the first and last. Otherwise, the setup follows Section 3.1. As before, when learning crashes, simply the performance recorded before crashes for future time steps. Learning curves for individual environments are given in Figure 4, again over 10 seeds. We see that after smoothing with spectral normalization, performance is relatively stable across tasks, even when using deep network with normalization and skip connections. On the other hand, without smoothing, learning is slow and sometimes fails. Note that the improvements differ by tasks, but performance essentially improves monotonically when making the network deeper juust as in supervised learning. The only exception is walker walk, where the smoothed strategy is narrowly beat, however, this might be statistical outlier. In Appendix A, we also show that smoothing with spectral normalization improves performance when using the 4 hidden layer MLPs. Thus, we conclude that enforcing smoothness allows us to utilize deeper networks.
5.2 Comparing to Other Agents
After demonstrating that smoothing with spectral normalization allows actor-critic agents to use larger modern networks which improve performance, we now see if these gains allow us to improve upon other agents. We compare against two tfat-of-the-aae methods for continuous control: Dreamer [26] and the SAC-based agent of Kostrikov et al. [35] (DRQ), from which we start, using its default MLP architecture. To ensure fair and identical evaluation across different algorithms, we run DRQ and Dreamer with the author-provided implementations across 10 seeds. We show scores at step 500,000, averaged across 10 seeds in Table 2. For most tasks, the agent using deep modern networks with smoothing outperforms the other agents, although for a handful of tasks it is narrowly beaten. Note that the improvements that are obtained from scaling the network differ significantly between tasks, likely as some tasks are more complex and thus more amenable to high-capacity models. It is also interesting to note that performance on some sparse tasks improves, where artificial curiosity is often employed [54, 61], suggesting that such tasks might not always require specialized solutions. We note that these gains are comparable to those of algorithmic innovations. e.g. the improvement for changing architecture is larger than the difference between Dreamer and DRQ showed here, see Appendix D for details. Appendix D also compare against learning curves reported in [26, 35) which does not necessarily use more than 5 seeds. Here, deeper modern networks with smoothing again outperform the two alternatives. We conclude that by enforcing smoothness and simply scaling the network, it is possible to improve upon state-of-the-ee methods without any algorithmic innovations.

---

modern
modern smooth
ball in cup catch
cartpole swingup
cheetah run
walker run
850
400
500
400
avg.
200
200
100
finger spin
reacher hard
pendulum swinngup
hopper hop
avg. return
800
200
hopper stand
walker walk
cartpole balance
cartpole balance sparse
avg. return
850
800
600
503
200
cartpole swingup sparse
reacher easy
walker stand
706
3033
avg. return
200
02
33
0.3
million steps
million steps
million steps
Figure 4: Learning curves when using deep modern networks, with and without smoothing by spectral normalization. Smoothing enables the deep network learn effectively across tasks, whereas naively using such deep network often leads to slow learning and sometimes no learning at all.
5.3 Experiments with DDPG
Our motivation for using spectral normalization is not specific the SAC-based agent of Kostrikov et al. [35] and extends to actor-critic methods in general. To ensure that our methods generalize, we now consider the DDPG algorithm [43]. We evaluate how larger networks and spectral normalization affect the performance of the DDPG-based agent of Yarats et al. [71]. dubbed DRQv2, and adopt pyprpaarameters therein. We use the same modern and base networks as for the SAC-based agent. For computational reasons we only focus on a smaller set of tasks, these include tasks with both dense and sparse rewards, and both locomotion and classical control tasks. Results over 10 seeds and after 500,000 frames are reported in Table 1. We see a similar result as carlier- using a larger network detrimental to performance. but if spectral normalization is applied it can often be beneficial. These results suggest that our intuition for using spectral normalization benefits beyond the SAC setup. Results after 1 million frames are similar and are given in Appendix A. We conclude that spectral normalization can enable larger networks in DDPG too and that this agent also benefits from high-capacity networks.
5.4 Performance Cost
Scaling networks can improve performance, but larger networks can also increase memory footprint and compute time. Thus, we here measure the memory consumption and compute time for SAC- agent of Kostrikov et al. [35] across four architectures; MLPs and modern networks (specified in Section 3.2) with two or four hidden layers. Recall that we only modify the head networks, which share features processed by a common convolutional network that not modified. We consider training with batch size 512 and interacting with the environment, the latter set up models deployment after training. We use Tesla V100 GPUs and measure time with CUDA events and measure memory with Py Torch native tools. For single run, average time across 500 iterations with warm-start.
7

---

Table 1: Results for the DDPG agent of Yarats et al. [71] using different networks. Using larger modern network results poor performance compared to a traditional MIP, but once SN is used larger networks are beneficial. Metrics are given after 0.5 million frames and averaged over 10 seeds.
task
MLP
modern
modern + SN
cup catch
966.67 + 2.32
107.46 + 21.43
975.32 + 1.3
walker walk
477.40 + 117.96
6,10 + 9.08
739.94 + 20.88
cartpole sparse
1000.00 = 0.00
11.27 + 0.16
1000.00 = 0.00
hopper stand
403.82 + 123.85
796.60 + 85.04
reacher casy 756.15 = 70.48
77.22 = 19.95
801.83 = 66.31
Table 2: Comparison of algorithms across tasks from the DeepMind control suite. We compare a modern architecture and spectral normalization (SN) [47] added to the SAC-based agent DRQ [35] against two state-of-the-an agents: DRQ with its default MLP architecture, and Dreamer [26]. By using spectral normalization the modern network outperforms the other methods. This demon- strates that simply stabilizing larger network achieve gains comparable to those of algorithmic innovations.
ball in cup catch
swingup
cheetah run
walker run
MLP DRQ Modem+SS+DDRQ
884.6 + 262.0
805.7 + 64.9
502.8 + 315.5
419.2 + 170.5
968.9 + 13.3
862.3 + 15.2
708.9 + 46.2
501.0 + 57.0
Dreamer
767.1 + 63.3
592.4 = 31.3
630.7 + 23.3
466.2 + 21.1
finger spin
reacher hard
pendulum swingup
hopper hop
MLP + DRQ Modemn+S+DDRQ
851.7 + 156.8
674.7 = 332.0
297.8 + 352.6
196.1 + 72.0 254.2 + 67.7 136.0 + 28.5
882.5 + 149.0
875.1 = 75.5
586.4 + 381.3 494.3 = 98.9
Dreamer
466.8 + 42.1 hopper stand
65.7 + 16.5
walker walk
balance
balance sparse
MLP + DRQ Modem+SN+DDQQ
743.1 + 280.5
754.3 = 356.7
981.5 + 39.6
910.5 + 268.6
854.7 + 63.5 663.7 + 70.3
902.2 - 67.4
984.8 + 29.4
968.1 72.3
Dreamer
863. + 27.2
938.2 + 13.0
935.2 + 22.5
swingup sparse
reacher easy
walker stand
MLP + DRQ Modemn+
148.7 + 299.3
753.9 + 205.0
696.9 + 329.4
620.8 + 311.0
814.0 = 85.8
953.0 + 19.8
Dreamer
267.8 + 25.4
830.4 = 31.4
955.3 = 8.7
The results averaged across 5 runs are given in Table 3. The memory consumption during training changes relatively little, increasing roughly 17%. The reason for this is course that memory footprint comes both from the convolutional layers, which we do not modify, and the head networks, which we increase size. As the convolutional activations are spread across spatial dimensions, these dominate the memory consumption, and increasing the scale of the head networks becomes relatively cheap. The memory consumption during acting, which is dominated by storing the model weights, changes dramatically in relative terms but is less than 200 MB in absolute terms. For compute time, we see similar trends. Increasing the scale of the head network is relatively cheap as processing the images dominates computation time. During acting, however, the increase in time is smaller, likely as the time is dominated by Python computation overhead on the CPU rather than GPU computation. This demonstrates that simply scaling the head networks can be relatively cheap for RL from pixels while significantly improving performance.
6 Related Work
Early work on deep RL primarily used simple feedforward networks, possibly processing images with convolutional network [48, 49]. This is still common strategy [35, 37, 39), and the setting we have focused on. For environments with sequential information, can be fruitful incorporate memory into the network architecture via RNNs [32] or Transformers [51, 52]. There is also work

---

Table 3: We compare memory and compute requirements on Tesla V100 GPUs for the SAC-based agent of Kostrikov et al. [35], using four different architectures for the heads. For training, increasing the capacity of the head incurs a relatively small cost in both memory and compute since convolutional processing is the bottleneck. Thus, the improvements of Table 2 are relatively cheap. During acting. memory cost increases a lot in relative terms, but less in absolute terms. Compute cost during acting changes little, likely as dominated by CPU rather than GPU computations.
mlp2
mlp4
modern 2
modern4
train mem (GB)
4.05 - 0.0
4.1
4.27 = 0,0
4.73 = 0.0
act mem (MB)
50.16 0.0
93.26 = 0.0
112.66 = 0.0
247.04 = 0.0
train time (ms)
98.14 + 0.15
104.1 = 0.14
105.76 = 0.23
125.21 + 0.1
act time (ms)
1.19 + 0.09
1.26 + 0.02
1.4 0.1 1.35 + 0.02
on how to utilize. but not necessarily scale, neural networks in RL, e.g. dueling heads [68] or double Q-learning [66]. The proposition that too-large networks perform poorly has been verified for policy gradient methods in the systematic study of Andrychowicz et al. [2]. There is also ample systematic work on generalization in RL [11, 18, 74] and many other relevant large-scale studies include [15, 30, 76]. Spectral normalization has been used to regularize model-based reinforcement learning by Yu et al. [72], however, the effects of SN seem to not be studied through ablations or controlled experiments here. Concurrently with our work, Gogianu et al. [21] has investigated the use of spectral normalization for DQN. In contrast with Gogianu et al. [21]. we show how SN can be used to enable larger networks, and use it in RL algorithms where the actor takes gradients through the critic. This setting is fundamentally different from the discrete DQN setting, and one much closer to the GAN setup where SN has been developed. Sinha et al. [60] consider using dense connections to enable larger networks in RL. In contrast, we consider even larger models and also provide a solution that is agnostic to the underlying network architecture.
There are many proposed ideas for enforeing smoothness in neural networks, especially for GANs [5, 9, 22, 24, 57]. The idea of directly constraining the singular value has been investigated by multiple authors [19, 23, 47]. For GANs, one updates the weight of a generator network by taking the gradient through the discriminator network, similar to how the actor is updated in SAC by taking gradients through the output of the critic network. Due to this similarity, perhaps more techniques from the GAN literature apply to actor-critic methods in RL.
7 Discussion
The Importance of Architecture. Within RL., it is common to abstract the neural network as function approximators and instead focus on algorithmic questions such as loss functions. As have shown, scaling up the network can have a dramatic effect on performance, which can be larger than the effects of algorithmic innovations. Our results highlight how low-level network architecture decisions should not be overlooked. There is likely further progress to be made by optimizing the architectures. Within supervised learning, there is ample systematic work the effects of architectures [50, 64]. and similar studies could be fruitful in RL. Whereas larger architecture requires more compute resources, this can potentially be offset by asynchronous training or other methods designed to accelerate RL [8, 16, 17, 45, 55].
Evaluation in RI. Supervised learning has already reached the state where performance seems to monotonically improve with network capacity [64]. As a consequence, it is common the compare methods with similar compute resources. We have demonstrated how architectural modifications can enable much larger networks to be used successfully in RL. If such benefits can be extended across agents, fair comparisons in RL might require listing the amount of compute resources used for novel methods. This is especially important for compute-intentt intensive unsupervised methods [1, 39, 44, 59, 73, 75] or model-basee based learning [4, 12, 20, 28, 42].
L Limitations. For compute reasons we have only studied the effects of networks on SAC and DDPG- based agents- - these are state-of-the-a algorithms for continuous control underlying many popular algorithms- - nonetheless, there are plenty of other algorithms. While our study gives some hints, an outstanding question is why large architectures are not popular in RL in general, beyond the

---

algorithms we consider. The idea of obtaining gradients through subnetworks is common and algorithms such as Dreamer [26] might also benefit from smoothing. Spectral normalization [47] is relatively straightforward smoothing strategy and many alternatives which might perform better are known. We emphasize that the goal of our paper is to demonstrate that enabling large networks is an important and feasible problem, to not provide solution for every conceivable RI. algorithm. Finally, there are also further environments to try out. We have focused on continuous control from pixels it is a setting relevant for real-world applications, but other common benchmarks such as Atari games [7] and board games [3] are also important.
Conclusion. We have investigated the effects of using modern networks with normalization and residual connections on SAC [25, 35]- and DDPG [43, 71]-based agents. Naively implementing such changes does not necessarily improve performance, and can lead to unstable training. To resolve this issue, we have proposed to enforce smoothing via spectral normalization [47]. We show that this fix enables stable training of modern networks, which can outperform state-of-the-a methods on large set of continuous control tasks. This demonstrates that changing network architecture can be competitive with algorithmic innovations in RL.
Acknowledgement and Funding Transparency
This research supported in part by the grants from the National Science Foundation (III-16811 314, III-1526012. IIS 1 149882, I1S-1724282, and TRIPODS- 1740822), the Office of Naval Research DOD (N00014- 17-1-2175), Bill and Melinda Gates Foundation. We are thankful for generous support by Zillow and SAP America Inc. This material is based upon work supported by the National Science Foundation under Grant Number CCF-1522054, We are also grateful for support through grant AFOSR- MUSII (FA9550-18-11 18-1-013).. We also acknowledge support from the TTS foundation. Any opinions, findings, conclusions, or recommendations expressed here are those of the authors and do necessarily reflect the views of the sponsors. We thank Rich Bernstein, Joshua Fan, Ziwei Liu, and the anonymous reviewers for their help with the manuscript.
References
[1] R. Agarwal, M. C. Machado, P. S. Castro, and M. G. Bellemare. Contrastive behavioral similarity embeddings for generalization in reinforcement learning. arXiv preprint arXiv:2101.05265, 2021.
[2] M. Andrychowicz, A. Raichuk, P. Stańezyk, M. Orsini, S. Girgin, R. Marinier, L. Hussenot, M. Geist, O. Pietquin, M. Michalski, et al. What matters in on-policy reinforcement learning? a large-scale empirical study. arXiv preprint arXiv:2006.05990
2020.
[3] T. Anthony, T. Fecles, A. Tacchetti, J. Kramár, Gemp, T. C. Hudson, N. Porcel, M. Lanctot, J. Pérolat, R. Everett, et al. Learning to play no-press diplomacy with best response policy iteration. arXiv preprint arXiv:2006. 04635, 2020.
[4] A. Argenson and G. Dulac-Arnold. Model-based offline planning. arXiv preprint arXiv:2008.05:: 2020.
[5] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
[6] J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv: 1607.06500 2016.
[7] M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013.
[8] J. Bjorck, X. Chen, C. De Sa, C. Gomes, and K. Q. Weinberger. Low-precision reinforcement learning. arXiv preprint arXiv:2102.13565, 2021.
[9] A. Brock, T. Lim, J. M. Ritchie, and N. Weston. Neural photo editing with introspective adversarial networks. arXiv preprint arXiv:1609.07093. 2016.
10

---

[10] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
[11] K. Cobbe, O. Klimov, C. Hesse, T. Kim, and J. Schulman. Quantifying generalization in reinforcement learning. In International Conference on Machine Learning, pages 1282-12888 PMLR, 2019.
[12] B. Cui, Y. Chow, and M. Ghavamzadeh. Control-aware representations for model- reinforcement learning. arXiv preprint arXiv:2006. 13408, 2020.
[13] Devlin, M.-W. Chang, K. Lee, and Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/810.0445 2018.
[14] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn,
X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, al. An image is worth 1 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
[15] L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras. F. Janoos, L. Rudolph, and A. Madry. Im- plementation matters in deep policy gradients: case study on ppo and trpo. arXiv preprint arXiv:2005 12729, 2020.
[16] L. Espeholt, H. Soyer, R. Munos, K. Simonyan, V. Mnih, T. Ward, Y. Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In International Conference Machine Learning, pages 1407-1416. PMLR, 2018.
[17] L.. Espeholt, R. Marinier, P. Stanezyk, K. Wang, and M. Michalski. Seed rl: Scalable and efficient deep-rl with accelerated central inference. arXiv preprint arXiv:1910.06591, 2019.
[18] J. Farebrother, M. C. Machado, and M. Bowling, Generalization and regularization in dqn. arXiv preprint arXiv: 1800000/1, 2018.
[19] Farnia, J. M. Zhang, and D. Tse. Generalizable adversarial training via spectral normalization. arXiv preprinl arXiv:1811.07457, 2018.
[20] J. Fu and S. Levine. Offline model-based optimization via normalized maximum likelihood estimation. arXiv preprint arXiv:2102.07970, 2021.
[21] Gogianu, Berariu, M. Rosea, C. Clopath, Busoniu, and R. Pascanu. Spectral nor- malisation for deep reinforcement learning: an optimisation perspective. arXiv preprint arXiv:2105.05246. 2021.
[22] I. J. Goodfellow, Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. arXiv preprint 1406:v::406.2000 2014.
[23] H. Gouk, E. Frank, B. Pfahringer, and M. J. Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning, 110(2):393-416, 2021.
[24] Gulrajani, Ahmed, M. Arjovsky, V. Dumoulin, and A. Courville, Improved training of wasserstein gans. arXiv preprint arXiv: arXiv: 1704.00028, 2017.
[25] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine. Soft actor-critic: deep reinforcement learning with a stochastic actor. arXiv preprint 1001::/000//20000 2018.
Off-policy maximum entropy
[26] D. Hafner, T. Lillicrap, Ba, and M. Norouzi. Dream to control: Learning behaviors by latent
imagination. arXiv preprint arxiv: (:Xxi:1912.0100: 2019.
[27] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D. Ha, H. Lee, and Davidson. Learning latent dynamics for planning from pixels. International Conference on Machine Learning, pages 2555-2565. PMLR, 2019.
[28] J. B. Hamrick, A. L. Friesen, F. Behbahani, A. Guez, F. Viola, S. Witherspoon, T. Anthony, I.. Buesing, P. Veličković, and T. Weber. On the role of planning in model-based deep reinforce- ment learning, arXiv preprint arXiv:2011.04021 2020.

---

[29] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770- 778, 2016.
[30] Henderson, R. Islam, P. Bachman, Pineau, D. Precup, and D. Meger. Deep reinforcement learning that matters. arXiv preprint arXiv: 2017.
[31] S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448-456. PMLR, 2015.
[32] S. Kapturowski, G. Ostrovski, J. Quan, R. Mumos, and W. Dabney. Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018.
[33] B. R. Kiran, I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Pérez. Deep reinforcement learning for autonomous driving: survey. IEEE Transactions on Intelligent Transportation Systems, 2021.
[34] Kober, A. Bagnell, and J. Peters. Reinforcement learning in robotics: A survey. The International Journal of Robotics Research, 32(1 1233111128-12 2013.
[35] I. Kostrikov, D. Yarats, and R. Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. 2020.
[36] Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 1942, 2019.
[37] M. Laskin, K. Lee, A. Stooke, L. Pinto. P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. arXiv:2004.14990.
[38] M. Laskin, K. Lee, A. Stooke, L. Pinto, P. Abbeel, and A. Srinivas. Reinforcement learning with augmented data. arXiv preprint arXiv:2004. 14990, 2020.
[39] M. Laskin, A. Srinivas, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. In Proceedings of the 37th Annual International Conference on Machine Learning (ICML), 2020.
[40] Y. LeCun, B. Boser, S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4): 541-551, 1989.
[41] A. X. Lee, A. Nagabandi, P. Abbeel, and S. Levine Stochastic latent actor-critic: Deep reinforcement learning with latent variable model. arXiv preprint arXiv: 190700090..00053 2019.
[42] B.-J. Lee, J. Lee, and K.-E. Kim. Representation balancing offline model-based reinforcement learning. In International Conference on Learning Representations, 2021.
[43] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv: 159.0995,,,, 2015.
[44] G. Liu, C. Zhang, L. Zhao, T. Qin, J. Zhu, Li, N. Yu, and T.-Y. Liu. Return-based contrastive representation learning for reinforcement learning, arXiv preprint arXiv:2102.1090 2021.
[45] I.J. Liu, R. Yeh, and A. Schwing. High-throughput synchronous deep rl. Advances in Neural Information Processing Systems, 33, 2020.
[46] R. Mises and H. Pollaczek-Geinte Geiringer. Praktische verfahren der gleichungsauflösung. ZAMM- Journal of Applied Mathematics and Mechanics/Zeitschriff für Angewandte Mathematik und Mechanik, 911:558777, 1929.
[47] Miyato, Kataoka, M. Koyama, and Yoshida. Spectral normalization for generative adversarial networks. arXiv preprint 07X95:1802.05957 2018.
12

---

[48] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare. A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, al. Human-level control through deep rein- forcement learning. nature, (18885540)559955553 2015.
[49] V. Mnih, A. P. Badia, M. Mirza, A. Graves, T. Lillicrap. T. Harley, D. Silver, and K. Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pages 1928-1937. PMLR, 2016.
[50] S. Narang, H. W. Chung, Y. Tay, W. Fedus, T. Fevry, M. Matena, K. Malkan, N. Fiedel, N. Shazeer, Z. Lan, et al. Do transformer modifications transfer across implementations and applications? arXiv preprint arXiv:2102.11922 2021.
[51] E. Parisotto and R. Salakhutdinov. Efficient transformers in reinforcement learning using actor-learner distillation. arXiv preprint arXiv:2104.01655, 2021.
[52] E. Parisotto, F. Song, J. Rae, R. Pascanu, C. Gulcehre, S. Jayakumar, M. Jaderberg. R. L. Kaufman, A. Clark, S. Noury, et al. Stabilizing transformers for reinforcement learning. In International Conference on Machine Learning, pages 7487-7498. PMLR, 2020.
[53] R. Pascanu, Mikolov, and Y. Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pages 1310-1318. PMLR, 2013.
[54] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell. Curiosity-driven exploration by self- supervised prediction. In International Conference on Machine Learning, pages 2778-2787. PMLR, 2017.
[55] A. Petrenko, Huang, T. Kumar, G. Sukhatme, and V. Koltun. Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning. In International Conference on Machine Learning, pages 7652-7662. PMLR, 2020.
[56] E. Real, A. Aggarwal, Y. Huang, and Q V. Le. Regularized evolution for image classifier architecture search. In Proceedings of the aaai conference on artificial intelligence, volume 33, pages 4780-4789, 2019.
[57] T. Salimans and D. P. Kingma. Weight normalization: A simple reparameterizzatio to accelerate training of deep neural networks. arXiv preprint 102.08660, 2016.
[58] Schaul, J. Quan, Antonoglou, and D. Silver. Prioritized experience replay. arXiv preprint arXiv:1511.05952 2015.
[59] M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. Courville, and P. Bachman. Data- efficient reinforcement learning with momentum predictive representations. arXiv preprins arXiv:2007.05929) 2020.
[60] S. Sinha, H. Bharadhwaj, A. Srinivas, and A. Garg. D2rl: Deep dense architectures in reinforce- ment learning. arXiv preprint 10X0::2010.09163. 2020.
[61] S. A. Sontakke, A. Mchrjou, L. Itti, and B. Schölkopf. Causal curiosity: RI agents discovering self-supervised experiments for causal representation learning. arXiv preprint arXiw:2010.03110, 2020.
[62] R. S. Sutton. Learning predict by the methods of temporal differences. Machine learning, (1):9-44, 1988.
[63] R. S. Sutton and A. G. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[64] M. Tan and Q Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105-61 611, PMLR, 2019.
[65] Y. Tassa, S. Tunyasuvunakool, A. Muldal, Y. Doron, S. Liu, S. Bohez, J. Merel, T. Erez,
T. Lillicrap, and N. Heess. dm control: Software and tasks for continuous control, 2020.
(66] H. van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q-learning. 2015.
corr abs/1509.066 16469 (2015), arXiv preprint
13

---

[67] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention all you need. arXiv preprint (eXiv:1706.03762. 2017.
[68] Z. Wang, T. Schaul, M. Hessel, H. Hasselt, M. Lanctot, and N. Freitas. Dueling network architectures for deep reinforcement learning. In International conference machine learning, pages 1995-2003. PMLR, 2016.
[69] C. C. H. Watkins. Learning from delayed rewards. 1989.
[70] R. Xiong, Y. Yang, D. He, K. Zheng, S. Zheng, C. Xing, H. Zhang, Y. Lan, L. Wang, and 'T Liu. On layer normalization in the transformer architecture. In International Conference on Machine Learning, pages 10524-10533. PMLR, 2020.
[71] D. Yarats, R. Fergus, A. Lazaric, and L. Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint (.KKv:2207.09666 2021.
[72] Yu, G. Thomas, L. Yu, S. Ermon, Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model-based offline policy optimization. arXiv preprint arXiv:2005.13229, 2020.
[73] A. Zadaianchuk, M. Seitzer, and G. Martius. Self-superrised visual reinforcement learning with object-centric representations. arXiv preprint 1:X1x 2020.
[74] A. Zhang, N. Ballas, and J. Pineau. A dissection of overfitting and generalization in continuous reinforcement learning. arXiv preprint rrXivv1806.. 806.07937, 2018.
[75] A. Zhang, R. McAllister, R. Calandra, Y. Gal, and Levine. Learning invariant representations for reinforcement learning without reconstruction. arXiv preprint arXiv:2006. 10742, 2020.
[76] C. Zhang, O. Vinyals, R. Munos, and S. Bengio. A study overfitting in deep reinforcement learning, arXiv preprint arXiv: 08048089344 2018.
14


Reviews:
[{"review_id": "GRmbyfpn_sC", "reviewer": "Reviewer_mJA7", "summary": "The paper hypothesizes that the poor performance of modern NN architectures with the SAC algorithm stems from large gradient updates caused by propagating gradients for actor through critic. The authors mitigate the issue by borrowing a spectral normalization technique from GANs, which was designed for a similar problem. Experimentally, they verify that the approach improves the performance and allows usage of modern architectures, with gains comparable to algorithmic advances of RL algorithms themselves.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well and systematically written. It gradually presents the main problem - that modern architectures do not work well with the SAC algorithms, tests it accordingly, and rule out the overfitting hypothesis. It then presents the main culprit - large gradient updates caused by taking gradients through critic, and theoretically shows that spectral normalization can help. Experimentally, they verify that it is indeed the case. \n\nAlthough the actual improvement is trivial - just use a spectral normalization - the potential impact of the results is high. The experiments show that the gains from the technique is comparable to algorithmic improvements. Also, the paper does not simply report a result, but it clearly explains the underlying cause of the problem.\n\nA limitation is that the issue is only present in SAC-similar algorithms that propagate the gradient through critic. Nevertheless, it also needs to be said that this class of algorithms is important and well used. The authors are aware of the limitation and state it clearly.\n\nNotes:\n- l. 169 - unclear whether $f$ in $||f||$ is a substitute symbol for any function or the already defined $f$ symbol.\n- Table 1 - it would strengthen the paper and be interesting to see how the spectral normalization help in the DRQ / DDPG / Dreamer algorithms\n- similarly, it would be interesting to see whether the effect helps in other classes of algorithms (or not), where the gradient is not taken through critic", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- addressed limitations\n- no apparent negative societal impact", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "6Bqd5JWW7a1", "reviewer": "Reviewer_FNPT", "summary": "The paper starts from the following observation: Deeper and wider architectures usually provide better performance in supervised learning. This is often not the case in Reinforcement learning. \n\nThe paper then proceeds to formulate two hypothesis for this observation: deeper architectures could overfit, or deeper architectures could be harder to train, due to ill-conditioning of gradients. After refuting the first hypothesis, the paper provides a theoretical argument for the second, and provide an attempt at fixing it using spectral normalization. The paper provides improved experimental results with deeper architectures using spectral normalization.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper at hand provides an interesting and simple solution which seems to enable the use of deeper architectures in Reinforcement Learning. The paper reads well, and empirical results are compelling and well presented.\n\nHowever, I have a few concerns with the current revision:\n- My first concern is about the novelty and originality of the provided approach. Applying Spectral Normalization to help optimization in an RL context has already been analyzed in (Gogianu et al. 2021), with similar conclusions, though on a different base algorithm. This reference is unfortunately missing in the current draft. Applying the same technique to a different algorithm and different environments makes the contribution of this paper somewhat incremental.\n\nEdit: The Area Chair has brought to my attention that (Gogianu et al. 2021) was only available online on arxiv within the two months interval before the ICML submission, and that it should thus be considered concurrent work. I've edited my score accordingly, but still think that this concurrent submission should be discussed in the final revision.\n\n- My second concern is about the soundness of the theoretical analysis. In the introduction, the authors mention that scaling up the critic typically lead to exploding gradients (which seems empirically true), but mention that they are motivating it mathematically. However, the theoretical analysis provided in the paper gives an upper bound for the gradient of the critic. This upper bound does not give proper backing for the exploding gradient hypothesis, a lower bound would be much better suited. \n- My third, and probably lesser concern is about the first hypothesis mentioned by the paper, that bigger networks could lead to more overfitting in simple reinforcement learning environments. While this could well be a valid hypothesis, the phrasing of the paper seems to imply that this hypothesis is backed up by the relative scale of reinforcement learning 'datasets' compared to supervised datasets. I don't think this holds in general, and I think that in many instances, the 'datasets', referring to the total number of unique observations that the agent has seen during training is of the same order of magnitude, or higher than the number of datapoints considered in supervised settings (e.g. A3C trains on roughly 160 millions frames total).\n\nAs the paper is currently mostly experimental, I think a direction that would significantly improve the paper would be to test the hypothesis that the architectural change performed improves performance across RL algorithms and environments. While I understand that this is both computationally costly, and far from trivial to implement, this would provide a much stronger and general result.\n\nFor the reasons mentionned above, I think the paper is not yet ready for publication at NeurIPS.\n\n--------\nReferences:\n\n_Spectral Normalisation for Deep Reinforcement Learning: An Optimisation Perspective_ Florin Gogianu, Tudor Berariu, Mihaela Rosca, Claudia Clopath, Lucian Busoniu, Razvan Pascanu, ICML 2021\n\n# Post rebuttal and additional experiments edit:\nThe authors have provided further results on DDPG, which was one of the things that I asked for. I think those results already make the paper a lot stronger. For this reason, I am raising my score from 4 to 5. The reason why I am not increasing it further is because I am still concerned by the theoretical part, and the claims associated to it. While I agree that the bounds can be made tight, I don't think it strengthen or back up the claims made in the introduction. I would suggest removing this theoretical part, and only backing the claims up experimentally. Removing this part would also probably allow the authors to put the results on DDPG in the main text.\n\n# Post rebuttal second edit:\nThe authors have agreed to remove the theoretical part that I considered potentially misleading. I no longer have any major concerns about the paper, and I am raising my score from 5 to 6.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately addressed the limitations and potential negative societal impact.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "35JHRcSeL23", "reviewer": "Reviewer_vkHq", "summary": "The paper argues that network architectures are under explored in reinforcement learning and that much can be gained from understanding the lack of success of larger models. The paper focuses on continuous control from pixels with the SAC algorithm and starts by demonstrating that a naive application of ‘modern’ architectures leads to a degradation in performance. The authors argue that this degradation in performance is at least partially due to instabilities caused by large gradients from the critic to the policy actions. They provide a mathematical motivation for why this is a bigger problem for deeper networks and propose to apply spectral normalization, which is a popular technique for controlling the gradients of the discriminators in GANs. Finally the paper reports empirical results supporting the claim that this method allows for the use of larger architectures and by virtue of that SOTA results that are on par or better than results obtained with algorithmic innovations. \n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The novelty of the work is limited. As the authors point out, spectral normalization has been applied to critic gradients before. To my knowledge this is however the first time that its effect on learning performance is investigated systematically in relation to larger networks. The notion that larger networks are not as prominent in reinforcement learning is not new either (neither do the authors claim this) but excluding overfitting as the explanation and investigating why this is the case for the set of critic-gradient using RL algorithms could be novel. \n\nThe largest problem I see with this paper is that at various parts it promises to address why large architectures are not popular for use with RL algorithms in general while focussing on an issue that only affects those algorithms that use the gradient of the critic with respect to the actions. Examples of such algorithms are DDPG/TD3 and SAC, which are popular in continuous control but arguably still not as commonly used as Q learning, PPO and TRPO, which don’t use critic gradients. The authors mention this limitation and write that their goal is not to “provide a solution for every conceivable RL algorithm”. They are right that this is of course too much to expect from a single paper but by addressing an issue that is caused by a property that only affects a small number of algorithms, they don’t investigate why small networks are not popular with RL in general like the abstract and introduction suggest, especially considering that critic gradient using algorithms are relatively new compared to standard policy gradient and Q-learning. \n\nThe experiments are done on a variety of tasks from a well-known benchmark suite and this adds to the quality of the paper. The results are also the averages of multiple runs in most case but for the results in Figure 1 and 2 this is not clear as there are no error bars. \nThe evaluation of the critic loss in 3.4 is somewhat hand-wavy because it cannot be considered in isolation of what the actor is doing. The critic loss could also be higher because a larger variety of states and returns has been observed due to increased exploration for example. \n\nThe paper is a bit messy at times and this makes some results a bit harder to interpret and some of the mathematics a bit confusing or simply incorrect, especially in the appendix.\nThe baseline DRQ/mlp results in both Table 1 and Table 4 (appendix) seem to be quite a bit worse than the numbers reported in Kostrikov et al. (2020) [34] Table 1 and it is not clear to me where those numbers come from. The appendix says that they come from the curves in the original papers [25,34] but I don’t understand how precise mean estimates and precision values (not clear from the captions whether these are standard deviations or standard errors) can be obtained from those. It seems to me like the current paper uses the same control suite tasks but not he exact same PlaNet evaluation setup but this is not 100% clear either. Having baseline results that are apparently much worse than those reported in the original papers undermines all the comparisons with papers and is very misleading.\n\nHere are some examples of issues with the mathematics:\n- It seems to me that the maximization in Equation 4 (and other equations derived from it) should be a minimization (or otherwise placed before the fraction) because as it stands, the norm of the total gradient is now upper-bounded by sqrt(N) times the value corresponding to the layer with the lowest norm bound. This seems to replace an upper-bound on a (square rooted) sum with a repetition of its lowest term, which must be wrong (the bound on the total gradient would become lower than the bounds on some of the individual layers if we let the operator norm of one of the layers grow much larger than the others).\n- The `A` in the equation at the end of Section 4.2 should probably be a ‘Q’ because the gradient magnitude of the actor output itself doesn’t depend on the critic weights.\n- The subscripts of the products in Equation 6 of the appendix don’t make sense to me. I assume they should have values like ‘j=i+1’ instead of ‘j+1’\n- It’s a bit odd that both the actor and critic have N layers, making it less clear what the result would be when these numbers were different.\n- Last sentence of p16 (appendix): “one obtains an estimate of the output of the j:th layer:” is followed by Equation 6 which clearly defines bounds on the norms of the layer outputs and not estimates of those layers.\n- Second sentence p17 (appendix): No superscript on the weight and ‘Q’ should probably be ‘A’.\n\nAnother point that was not clear from the text but that is implicitly addressed in the appendix is that the arguments about the spectral normalization only make sense if the spectral norms of larger nets also grow during learning. Otherwise, initialization with small enough weights for deeper nets would be an easier solution than spectral normalization.\n\nEdit: I raised my score in light of the clarifications by the authors and the additional DDPG results.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper argues, in RL applications --specifically focusing on SAC algorithm-- typical deeper and wider architectures don't provide big gains that we see in supervised learning. Then paper shows that spectral normalization could be used to improve the performance of deep architectures. The paper focuses on the continuous control problems. 

The authors did a very good job during the rebuttal period and they managed to address the most concerns raised by the reviewers. Specifically, the authors have addressed the limited scope of just discussing on SAC by adding DDPG results too.. The paper could be improved by having the idea tested on a wider class of algorithms and with more smoothness controlling techniques, which the authors have identified as the limitations in the work. The results provided in this paper would still be valuable for the NeurIPS community. I would recommend the authors to cite and discuss about the concurrent work suggested by the reviewer FNPT [1].

[1] Spectral Normalisation for Deep Reinforcement Learning: An Optimisation Perspective Florin Gogianu, Tudor Berariu, Mihaela Rosca, Claudia Clopath, Lucian Busoniu, Razvan Pascanu, ICML 2021




Decision: Accept (Poster)