D2RL: Deep Dense Architectures in Reinforcement Learning
Authors: Samarth Sinha
Homanga Bharadhwaj
Aravind Srinivas
Animesh Garg
Venue:  (2021)
Keywords: Deep Reinforcement learning
Policy architectures

Abstract: While improvements in deep learning architectures have played a crucial role in improving the state of supervised and unsupervised learning in computer vision and natural language processing, neural network architecture choices for reinforcement learning remain relatively under-explored. We take inspiration from successful architectural choices in computer vision and generative modeling, and investigate the use of deeper networks and dense connections for reinforcement learning on a variety of simulated robotic learning benchmark environments. Our findings reveal that current methods benefit significantly from dense connections and deeper networks, across a suite of manipulation and locomotion tasks, for both proprioceptive and image-based observations. We hope that our results can serve as a strong baseline and further motivate future research into neural network architectures for reinforcement learning. The project website is at this link https://sites.google.com/view/d2rl-anonymous/home

Under review as a conference paper at ICLR 2021
D2RL: DEEP DENSE ARCHITECTURES IN REINFORCE- MENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
While improvements in deep learning architectures have played a crucial role in improving the state of supervised and unsupervised learning in computer vision and natural language processing, neural network architecture choices for rein- forcement learning remain relatively under-explored. We take inspiration from successful architectural choices in computer vision and generative modeling, and investigate the use of deeper networks and dense connections for reinforcement learning on variety of simulated robotic learning benchmark environments. Our findings reveal that current methods benefit significantly from dense connections and deeper networks, across suite of manipulation and locomotion tasks, for both proprioceptive and image-based based observations. We hope that our results can strong baseline and further motivate future research into neural net- work architectures for reinforcement learning. The project website is at this link https:/sttttooooo.ccco//veeee//s
INTRODUCTION
Deep Reinforcement Learning (DRL) a general purpose framework for training goal-directed agents in high dimensional state and action spaces. There have been plenty of successes from DRL for robotic control tasks, spanning across locomotion and navigation tasks, both in simulation and
in the real world (Schulman et al. 2015 Akkaya et al.) 2019 Kalashnikov et al.) 2018).
While the generality of the DRL framework lends itself to be applicable to a wide variety of tasks, one has to address issues such as the sample-efficiency and generalization of the agents trained with this framework. Sample-effiiiency fundamentally critical to agents trained in the real world, particularly for robotic control tasks. Baking in minimal inductive biases into the framework is one effective mechanism to address the issue of sample-efficiency DRL agents and make them more efficient.
The generality of the framework makes it difficult to control particular behaviours and inductive biases for DRL algorithms. Inductive biases important for learning algorithms, they are able to induce desirable behaviour in the learned agents. Recent work has sought to improve the sample efficiency of DRL by adding inductive bias of invariance, when learning from images, through techniques such as data augmentations (Laskin et al. 2020 Kostrikov et al. 2020) and contrastive losses (Srinivas et al. 2020). Similarly, another important inductive bias in DRL is the choice of the architectures for function approximators, for example how to parameterize the neural network for the policy and value functions. However, the problem of choosing architecture designs in DRL and robotics, for planning and control, has been largely ignored.
Modern computer vision and language processing research have shown the disproportionate advan- tage of the size and depth of the neural networks used (He et al. (0166] Radford et al.) wherein very deep neural networks can be trained such that they learn better and more generalizable representa- tions. Furthermore, recent evidence suggests that deeper neural networks can not only learn more complex functions but also have a smoother loss landscape (Rolnick & Tegmark 2017). Learning function approximators which enable better optimization and expressivity is an important inductive bias, which is greatly exploited in vision and language processing by using clever neural network architecture choices such as residual connections (He et al. 2016a), normalization layers (Santurkar jet al., 2018). and gating mechanisms (Hochreiter & Schmidhuber 1997). to name a few. would be ideal to incorporate similar inductive biases in modern DRI. algorithms in roboties in order to

---

Under review as a conference paper at ICLR 2021
Convolutional
St
Stc at)
encoder
at To
Qo (ats st)
Figure Visual illustrations the proposed dense-connections based D2RL modification to policy na and Q-value Q+(+) neural network architectures. The inputs are passed to each layer of the neural network through identity mappings. Forward pass corresponds to moving from left to right in the figure. For state-based envs, is the observed simulator state and there convolutional encoder.
allow for better sample efficiency that would significantly aid the deployment of real world robot learning agents.
In this paper, first highlight the problems that occur when learning policies and value func- tions using vanilla deep neural networks. Then we propose D2RL; an architecture that addresses these problems while benefiting from the utility of inductive biases added by more expressive func- tion approximators. We show how our proposed architecture scales effectively to wide variety of off-policy RL algorithms, for proprioceptive-feaure and image based inputs across diverse set of challenging robotic control and manipulation environments. Our approach is motivated by utilizing a form of dense-conneettios similar to the ones found in modern deep learning, such as DenseNet (Huang al. (017), SKip-VAE (Dieng et al. 2019) and U-Nets (Ronneberger et al.) 2015). We demonstrate that the proposed parameterization significantly improves sample efficiency of RL agents (as measured by the number of environment interactions required to obtain a level of performance) in continuous control tasks.
Our contributions can be summarized as:
1. We investigate the problem with increasing the number of layers used to parameterize policies and value functions.
2. We propose general solution based on dense-connections to overcome the problem.
We extensively evaluate our proposed architecture on diverse set of robotics tasks from propri- oceptive features and images across multiple standard algorithms.
2 RELATED WORK
Learning efficient representations for sample-efficient RL Several recent papers have sought to improve representation learning of observations for control. CURL (Srinivas et al. 2020) augments the usual RL loss with a contrastive loss that seeks to learn a latent representation which enforces the inductive bias of encodings of augmentations of the same image being closer in latent space than embeddings of augmentations of different images. RAD (Laskin et al.] 2020) and DrQ (Kostrikov al. 2020) showed that simple data augmentations like random crop, color jitter, patch cutout, and random convolutions can alone help improve sample-efficiency and generalization of RL from pixel inputs. Some other algorithms leam latent representations decoupled from policy learning, through a variational autoencoder based loss (Higgins et al., [0017] Hafner et al.] 2019 Nair et al. 2018)) OFENet (Ota al.) 2020) shows that learning higher dimensional feature space helps learn more informative representation when learning from states.
Inductive biases in deep learning Inductive biases in deep learning have been long explored in different contexts such as temporal relations (Hochreiter Schmidhuber 1997). spatial rela- tions (LeCun et al.] 1998 Krizhevsky al., 2012). translation invariance (Berthelot et al. 2019 He et al.] 2020 Chen et al., 2020 SSrinivas et al. 2020 Laskin et al., 2020) and learning contextual representations (Vaswani et al. 2017). These inductive biases are injected either directly through the network parameterization (LeCun et al, [998] Hochreiter & Schmidhuber [997] Vaswani et al,
2017] by implicitly changing the objective function (Berthelot et 2019 Srinivas et al. 2020)
2

---

Under review as a conference paper at ICLR 2021
Learning very deep networks Deep neural networks are useful for extracting features from data relevant for various downstream tasks. However, simply increasing the depth of a feed-forward neu- ral network leads to instability in training due issues such vanishing gradients (Hochreiter & Schmidhuber 1997). or a a loss of mutual information (He et al. 2016a). To mitigate this, residual connections were proposed which involve alternate path between layers through an identity map- ping (He et al. 6a). Skip- VAEs (Dieng et al., 2019) tackle a similar issue of posterior collapse the architecture of the VAE decoder. U- Nets (Ronneberger et al. 2015) consider a contractive path of convolutions and maxpooling layers followed by an expansive path of up-convolution layers. There are copy and crop identity mapping from layers of the contractive path to layers in the expansive path. Normalization techniques such as batch normalization are also important learning deep networks (loffe Szegedy 2015). Com- bining residual connections with batch normalization have been used to successfully train networks with 1000 layers (He et al. 2016b). Our proposed architecture closely resembles DenseNet (Huang et al.) ), which uses skip connections from feature maps of previous layers through concatena- tion, allowing for efficient learning and inference.
typical VAE training by adding skip connections
PRELIMINARIES
In this section, we describe the actor-critic formulation of RL algorithms, which serves as the basic framework for our setup. We then describe the Date-Processing Inequality which is relevant for explaining and motivating the proposed architecture.
3.1 ACTOR-CRITIC METHODS
Actor-critic methods learn two models, one for the policy function and the other for the value func- tion such that the value function assists in the learning of the policy. These are TD-learning (Tesauro
1995) methods that have an explicit representation for the policy independent of the value function. We consider the setting where the critic is an state-action value function Qo 8) parameterized by a neural network, and the actor (a)ss) is also parameterized by a neural network. Let the current state be s and ry denote the reward obtained after executing action a in state s, and transitioning to state s'. After sampling action a' (aa(a))) ss) in the next state s', the policy parameters are updated in the direction suggested by the critic Qo (a, 8)
f & + QoQa 8) V log (als).
The parameters 0 are updated using the TD correction A, +r++7008 7Qo(s' Qa a) as follows:
0 0 B.
Although the basic formulation above requires on-policy samples (s, a, 8', for gradient updates, a number of off off-policy variants of the algorithm have been proposed (Haarnoja et al. [011 Fujimoto (ct al.] (018)) that incorporate importance weighting in the policy's 's gradient update. Let the observed samples be sampled by behavior policy C(a|s), and (a(ass) be the policy to be optimized. The gradient update rule changes as
- o + B (a(s))
C(a|s) Qo(a, 8) S. logooalaass)
3.2 DATA- INEQUALITY
The Data processing inequality (DPI) states that the information content of a signal cannot be in- creased via a local physical operation. So, given a Markov chain X, + X2 - Xz. the mutual information (MI) between X1 and X2 is not less than the MI between X1 and X₃ i.c. MI(X;;
MI(X2;
vanilla feed-forwar neural network has successive layers depend only on the output of the previ- ous layer and so there is a Markov chain of the form X1 + X₂ + + X,, from the input X; to the final output X,, In practice the last layer X,, contains less information than the previous layer Xn-1- By using dense connections (Huang et al. 2017). we are able to overcome the problem of DPI, as the original input is simply concatenated with the intermediate layers of the networks. We postulate that using dense connections are also important when parameterizing policies and value networks in RL and robotics. By preserving important information about the input across layers explicitly through dense connections, we can achieve faster convergence in solving complex tasks.
3

---

Under review as a conference paper at ICLR 2021
3.3 IMPLICIT UNDER- PARAMETERIZATION IN DEEP Q-LEARNING
Kumar al. 2020) showed that using MLPs for function approximation of the policy and value functions in deep RL algorithms that use bootstrapping, leads to an implicituundeppaaecceriiiio phenomena that causes poor-er performance. Implicit under-pu paraneterrzatioononnn refers reduction in the effective rank of the feature, sranks (4) that occurs implicitly due to using MLPs for approx- imating Q-functions. This causes rank collapse for the feature matrix which are the weights of the penultimate layer of the Q network during training.
We believe that adding skip connections to the network architecture, as in D2RL could help alleviate this rank collapse issue, and improve performance. We empirically verify this in Section 5.2 and
Table 3
4 METHOD
In this section, we first show the issues with using deeper Multi-layered Perceptons (MLPs) to parameterize the policies and Q-networks in RL and robotics due to the Data Processing Inequality (DPI). We then propose a simple and effective architecture which overcomes the issues, using dense- connections. We will denote our proposed method as: Deep Dense architectures for Reinforcement Learning D2RL in the subsequent sections.
4.1 PROBLEM WITH DEEPER NETWORKS IN REINFORCEMENT LEARNING
To test whether we can use the expressive power of deeper networks in RL for robotic control, we train SAC agent (Haarnoja et al. 2018), while increasing the num- ber of layers in the MLP used parameterize the policy and the Q-networks, on the Ant-v2 environment of the OpenAl Gym suite (Brockman et al.) 2016). The results are shown in Fig. 2 The results suggest that by simply increasing the number of layers used to parameterize the networks, we are unable benefit from the inductive bias of deeper feature extractors in RL, like we see in com- puter vision. As we increase the number of layers in the network, the mutual information between the output and input likely decreases due to the non-linear transforma- tions used in deep learning as explained by the DPI (see Section (.2) Inereasing the number of layers from 2 to 4 significantly decreases the sample efficiency of the agent and furthermore, increasing the number of layers from 2 to 8 decreases the sample efficiency while also making the agent less stable during training. As expected, when instead decreasing the number of layers from 2 to I the vanilla MIPs are not sufficiently expressive to perform well on the Ant-v2 task. This simple experiment suggests that even though the expressivity of the function approx- imators is important, by simply adding more layers in vanilla MLP, the agent's performance and sample complexity significantly deteriorates. However, it is desirable to use deeper networks to increase the network expressivity and to enable better optimization for the networks.
One Layer
Eight Layers
Two Layers D2RL
Four Layers
6000
Epionde Rewards
4000
2000
0
3
# Steps (M)
Figure 2: The effect of increasing the num- ber of folly-connected layers paramcter- ize the policy and Q-Networks for Soft- Actor Critic (Harmoja el al.) 2018) on Ant- v2 in the OpenAl Gym Suite (Brockman) el 2016). It is evident that performance drops when increasing depth after 2 layers. However, our D2RL agent with 4 layers does not suffer from this, and performs better.
4.2 D2RL
Our proposed D2RL variant incorporates dense connections (input concatenations) from the input to each of the layers of the MLP used parameterize the policy and value functions. We simply concatenate the state or the state-action pair to each hidden layer of the networks except the last output linear layer, since that just linear transformation of the output from the previous layer. In case of pixel observations, we consider the states to be the encodings of a CNN encoder, as shown in Fig. I Our simple architecture enables us to increase the depth of the networks while also satisfying DPI. Fig is a visual illustration and we provide a PyTorch-liie Mokehh-lkke aseudo-code below to promote
4

---

Under review as a conference paper at ICLR 2021
2000
64C
(301
1030
30
5.0
244
00
10
6X2
10
22
28
15
22
10
002
168
100
(0) Clym Aut SAC
(b) Gyaa Cheetah SAC
Gym Humanoud SAC
(d) Gym Hopper SAC
(e) Glym Walker SAC.
Figure 3: OpenAT Gym benchmark environments with SAC. Comparison of the proposed D2RL and the baselines on suite of OpenAI-Gym environments. We apply the D2RL modification SAC (Haarnoja et al. 2018). The error bars with respect to random seeds. The results Humanoid in the Appendix.
clarity of the proposed method. We also include the actual PyTorch code (Paszke et al. 2019) for the policy and value networks to allow for fast-adoption and reproducibility in Appendix A
Sample state, action from the replay buffer state, acticn replay_buffer bfg_bbrrarrsrlllee
# Feed state, action into the first linear layer of Q-network input concatenate (state, acticn)
MLF (g input)
Concatenate the hidden representation with the input h concatenate (h, qinpput
Feed the concatenated representation into the second layer
MLP (h)
EXPERIMENTS
We experiment across suite of different environments, some of which are shown in Fig. 8 each simulated using the MuloCo simulator. We were unable to do real robot experiments due to COVID- 19 and have included 1-page statement along with the Appendix describing how method can conveniently scale to physical robots. Through the experiments, we aim to understand the following questions:
How does D2RL compare with the baseline algorithms in terms of both asymptotic perfor- mance and sample efficiency on challenging robotic environments?
Are the benefits of D2RL consistent across diverse set of algorithms and environments?
Is D2RL important for both the policy and value networks? How does D2RL perform with increasing depth?
5.1 EXPERIMENTAL DETAILS
To enable fair comparison between the current standard baselines and D2RL, simply replace the 2-layer MLPs that are commonly parameterize the policy and value function(s) in widely used actor-critic algorithms such SAC (Haarnoja et al. 2018 Yarats et al. 2019). TD3 (Fujimoto al.) 2018). DDPG (Qiu et al., 2019) and HIRO (Nachum et al. 2018). Instead, we use 4-layer D2RL to each of the actor-critic algorithms. Out-
parameterize both the policies and the value function(s) side of the network architecture, we do not change any hyperparameters, and use the exact same values as reported in the original papers and the corresponding open-source code repositories. The details of all the hyperparameters used are in Appendix C We perform ablation studies in Section 5.4 to i) investigate the importance of parameterizing both the policy and value function(s) using D2RL and ii) see how varying the number of layers of D2RL, affects performance. We also perform further experiments with ResNet style architecture and additional experiments with Hindsight Ex- perience Replay (Andrychowicz al. 2017) simpler manipulation environments which can be found in Appendix
5.2 EXPERIMENTAL RESULTS
The proposed D2RL variant achieves superior performance compared to the baselines state- based OpenAI Gym MuJoCo environments. We benchmark the proposed D2RL variant on a suite of OpenAl Gym (Brockman al. 2016) environments by applying the modification to standard
5

---

Under review as a conference paper at ICLR 2021
6200
6000
10920
3300
Eatecd Doverts
4000
7930
2200
Dslecd
2000
TD3
2520
T00
8000
2000
TD3
100
DORL
TD3-0201
0.0
0.2
00.4
0.6
1.0
0.0
0.2
5/4
0.6
0.5
1.0
0.0
0.2
004
0.6
1.0
0.5
0.2
0.6
10.6
086
Gyys
Glym Chestah TD3
Gym Happer TD3
Giym Waller TD3
Figure 4: OpenAl Gym benchmark environments with TD3. Comparison of the proposed variation D2RL and the baselines on a suite of OpenAl-Gyy A--Gym environments. We apply the D2RL modification to TD3 (Fujimoto etal. 2018). The error bars are with respect to random seeds.
From Images
From States
100K Step score
CURL
CURL-ResNet CURL-D2RL
SAC
SAC-ResNet SAC-D2RL
Finger, Spin
767 + 57
548 + 120
837 + 18
459 + 48
551 + 57
627 + 107
Cartpole, Swing
582 + 142
327 + 101
836 - 34
717 - 14
701 24
751 = 12
Reacher, Easy
538 233
526 79
754 + 168
752 121
626 = 112
675 = 203
Cheetah, Run
299 1 48
230 1 17
253 57
587 1: 58
633 1 28
721 1 43
Walker, Walk
403 1: 24
275 57
540 1 143
132 1 43
456 1 89
354 1 159
Ball in Cup, Catch
769 +
450 169
880 48
867 42
880
22
891 33
500K Step score
CURL
CURL-ResNet CURL-D2RL
SAC
SAC-ResNet SAC-D2RL
Finger, Spin
926 1: 45
896 1: 59
970 +: 14
899 1: 29
917 +: 21
961 +: 8
Cartpole, Swing
841 45
833 +:
859 8
884 1:2 2
885 4
885 2
Reacher, Easy
929 + 44
900 = 48
929 + 62
973 + 23
969 - 34
952 = 30
Cheetah, Run
518 + 28
459 = 108
386 + 115
781 - 65
807 = 69
842 = 75
Walker, Walk
902
807 134
931 24
959 16
972
964 14
Ball in Cup, Catch
959- 27
960 8
955 +
15
976 4: 12
970 + 19
972 + 13
Table 1: DeepMind control suite benchmark environments from images (CURL) and states (SAC), Results of CURL (Srinivas al. 2020), CURL-D2RL, SAC (Haarnoja et al. 2018). , and SAC-D2RI, the standard DM Control Suite benchmark environments. CURL (Srinivas al.) (220)) and CURL-D2RL, are trained purely with pixel observations while SAC (Haamoja et al, 2018) and SAC-D2RL are trained with proprioceptive features. The results for CURL were taken directly as reported by Srinivas et al. (2020). CURL-ResNet is the baseline that uses similar network as CURL-D2RA but with residual connections. Similarly. SeC-ResNet is the haseline that uses similar network as SAC-D2RI, hut with residual connections. The S.D. is over 5 random seeds.
100K Step Score
TD3
TD3-D2RL 500K Step Score
TD3
TD3-D2RL
Walker, Walk-Easy
67 1 8
69 11
Walker, Walk-Easy
79 10
77
1
Walker, Walk-Medium
38 + 8
60
+ 9
Walker, Walk-Medium
44 + 8
69
+ 6
Cartpole, Swing- -asy
96 + 24
142
+ 18
Cartpole, Swing-Easy
102 + 24
171
+ 32
Cartpole, Swing-Mediun
67 +
102
+ 13
Cartpole, -edig-Medium
68 2
138
16
Real World RL. suite environments from states. Results TD3 (Haamoja et al. 2018), and TD3- D221, on the Real World RI. suite environments after 500K environment steps over 5 seeds. We see that using D2RL we are able to perform better in environments with distractors, random noise and delays. These experiments show how D2RL is able learn robust agents.
Table 2:
RL algorithms, SAC (Haarnoja et al. 2018) and TD3 (Fujimoto et al., 2018). For all the environ- ments, namely Ant, Cheetah, Hopper, Humanoid, and Walker, the observations received by the agent are state vectors consisting of the positions and velocities joints of the agent. Additional details about the state-space and action-space are in the Appendix. From Fig. 4 we see that the proposed D2RL modification converges to significantly higher episodic rewards in most environments, and in others is competitive with the baseline. Also, these benefits can be seen across both the algorithms, SAC and TD3.
D2RL is more sample efficient compared to baseline state-of-the-art algorithms on both image- based and state-based DM Control Suite benchmark environments. We compare the proposed D2RL variant with SAC and state-of-the-ee paxel-based CURL algorithms the benchmark envi-

---

Under review as a conference paper at ICLR 2021
0.6
SAG
Complete
0.8
%y CeCl Completad
SAC-D2RL
Oeal Oeae Complated
0.6
0.4
0.6
0..4
0.2
0.2
SAC
HER
0.0
SAC-D2RL
0.0
0.0
HER.D2RL
0.0
1.0
7.5
10.0
12.5
Stepe
Stops
Stege
(a) Fetch Keach SAC
(b) Fesch Stade SAC
(e) Fetch Slide HER
200
Eptonds Rewerds
150
SAC-D2RL
SAC
% Coal Computted
0.8
stewards
3000
100
2000
0.4
1000
0.2
H1RO
SAC SAC-D2RL
HIRO-D2RL
0
0.0
0.5
1.0
1.5
2.0
2.5
of Steps (M)
of Staps (M)
Staps (M4
SAC
Block Join-Lift
Figure 5: Challenging selected manipulation and locomotion environments. Comparison of the proposed variation D2RL and baselines on suite challenging manipulation and locomotion environments. We apply the D2RL modification to the SAC (Haarnoja et al.) 2018). HER (Andrychowicz et al.] 2017). and IIROO (Nachum et al. 2018) algorithms and compare relative performance in terms of average episodic re- wards with respect to the baselines. The task complexity increases from Fetch Reach to Fetch Slide. Jaco Reach challenging due high-dimensional torque controller action space, AntMaze requires exploration to solve a temporally extended problem, and Furniture BlockJoin requires solving two tasks- join and lift sequen- tially. The error bars are with respect to 5 random seeds. Some additional results on the Fetch envs are in the Appendix.
ronments of DeepMind Control Suite (Tassa al. 2020). For CURL, and CURL-D2RL, we train using only pixel-based observations from the environment. For SAC and SAC-D2RL, we train using proprioceptive features from the environment. The action spaces are the same in both the cases, pix- els and state features based observations. The environments we consider are part of the benchamrk suite, and include Finger Spin, Cartpole Swing, Reacher Easy, Cheetah Run, Walker Walk, Ball in Cup Catch. Additional details are in the Appendix.
In Table T. we tabulate results for all the algorithms after 100K environment interactions, and after 500K environment interactions. To report the results of the baseline, we simply use the results as reported in the original paper (Srinivas et al. 2020). From this Table, we observe that the D2RL variant performs better than the baselines for both 100K and 500K environment interactions, and the performance gains are especially significant in the 100K step scores. This indicates that D2RL is significantly more sample-efficient than the baseline.
D2RL performs significantly better in challenging environments with various modalities of noise, system delays, physical perturbations and dummy-dimensions (Dulac-Arnold et al., 2020). Dulac-Arnold et al. (2019) propose a set of challenges in the DM Control Suite environ- ments that are more "realistic" by introducing the aforementioned problems. We train a TD3 agent Fujimoto et al. (2018) from states, on the "easy" and "medium" challenges for the walker-walk, and cartpole-swingup environments with and without D2RL. We present the results in Table 2 We see how the baseline TD3 agent gets significantly worse in the "medium" challenge compared to the "casy" version of the same environment. The agent trained with TD3-D2RL significantly out- performs the baseline TD3 agent on 3 of the 4 challenges, and the drop between the "easy" and "medium" challenges is significantly less severe, compared to the baseline. This experiment demon- strates how by using D2RL, are able to get significantly better performance environments which have been constructed to be more realistic by adding difficult problems that the agent must leam to reason with. The increased robustness to such problems further validates the general utility of D2RL in many different circumstances.
Verifying the alleviation of implicit underparameterization with D2RL. In Table 3 compare effective ranks of the feature matrices for D2RL and a normal MLP, with the TD3 algorithm after 1M interactions, corresponding to Fig. 4 We observe that the effective ranks for D2RL are higher
7

---

Under review as a conference paper at ICLR 2021
Walker2d-v2
Ant-v2
Hopper-v2
Cheetah-v2
1M Steps
TD3
TD3-D2RL
TD3 TD3-D2RL
TD3 TD3-D2RL
TD3
TD3-D2RL
Policy
153
161
165
178
145
159
157
161
Qentwwork
138
165
153
180
123
157
143
164
Table 3: OpenAl Gym benchmark environments with TD3. Comparison the proposed variation D2RL and the baselines on suite of OpenAl-Gym environments. We tabulate effective ranks of the feature matrices (learned weight matrices of the penultimate layer of the networks) for D2RL and normal MLP, with the TD3 algorithm after IM interactions, corresponding to Fig X The value is the average over 5 random sceds rounded off to the nearest integer. A loss rank of the matrix of weights of the penultimate layers (corresponding to learning features) leads to a loss in effectively expressivity of the network, and has been shown to correlate with poor performance (Kumar et al. 2020). Higher is better.
across environments. For this computation, we use the same formula of (ranks($) from (Kumar cet al. 2020), where is the learned weight matrix of the penultimate layer of the nwtwork (i.e. the feature matrix). Kumarr et al.] (2020) showed that lower effective rank of the feature matrix correlates negatively with performance, and we believe this might help explain some of the empirical observations of better performance of D2RL. Using skip connections as in D2RL leads to higher effective rank feature matrices compared to standard MLPs used in deep RL, as can be seen from Table 8
The sample efficiency and asymptotic performance of D2RL scale to complex robotic manip- ulation and locomotion environments. Additionally, we consider some challenging manipulation and locomotion environments with different robots, the details of which are discussed below:
Fetch- (Reach, Pick, Push, Slide): There are four environments, where Fetch robot tasked with solving the tasks of reaching goal location, picking an object and placing it at goal location, pushing puck to goal location, and sliding puck to goal location. The the Fetch-Slide envi- ronment, it is ensured that sliding occurs instead of pusing because the goal location is beyong the end-effector's reach. The observations to the agent consist of proprioceptive state features and the action space is the (x.y.z) position of the end-efficctor and the distance between the grippers.
Jaco-Reach: Jaco robot arm with three finger gripper tasked with reaching location location indicated by a red brick. The observations to the agent consist only of proprioceptive state features and the arm is joint torque controlled.
Ant-Maze: An Ant robot with four legs is tasked with navigating U-shaped maze while being joint torque controlled. This is challenging locomotion environment with a temporally-cxtendee task, that requires the agent to move around the maze reach the goal.
Baxter-Bloc Block Join and Lift: One arm of a Baxter robot with two fingers must be controlled to grasp a block, join it to another block and lift the combination above certain goal height. The observations to the agent consist of proprioceptive state features and the action space is the (x.y.z) position of the end-effector and the distance between the grippers.
For the Fetch-(Reach, Pick, Push, Slide} environments, we consider the HER AAnrychhwicz et al., 2017) algorithm (with DDPG (Qiu et al., 2019)) trained with sparse rewards that was shown to achieve state-of-the-art results in these environments. The plots for Fetch- Pick and Fetch-Push are in the Appendix, sue to space constraint here. In addition, we also show results with SAC on Fetch- Reach and Fetch-Slide trained using SAC agent. For Ant-Maze, we consider the hierarchical RL algorithm HIRO (Nachum et al. 2018) that was shown be successful in this very long horizon task. For Jaco-Reach and Baxter-Block Join and Lift, we consider the default SAC algorithm released with the environment codebase
com wwtp///cllcaaeeeooeeeeeeooeeeeea clvrai wwwwwa//fraattrrrrrerrrrrr ffunniuurrtrrr
The results are summarized in Fig. where we see that the proposed D2RL modification converges to higher episodic rewards and converges significantly faster in most environments. By performing wide range of experiments on challenging robotics environments, we further notice significantly better sample efficiency on all enviromments which suggests the wide generality and applicability of D2RL. Interestingly, Reco-Reach task in 3M environment steps, while SAC trained with D2RL policy and Q Q-networks is able to succesfully train agent and starts outperforming the SAC baseline early as 1M
also observe in 5d that SAC is unable to train the agent to perform the
8

---

Under review as a conference paper at ICLR 2021
environment steps. This shows how crucial parameterization is in some environments as simple 2-layer MLPs may not be sufficiently expressive or optimization using deeper network architectures may be necessary to solve such environments.
5.3 COMPARISONS WITH RESNET
Table ttabulates experiments with a ResNet-like MLP which utilizes residual connections instead of dense connections He et al. (20T6a). We experiment with images using a base CURL agent Srinivas et al. 2020), and directly from proprioceptive features using base SAC agent Haarnoja et al. (2018) on the DM Control Suite. We see that using D2RL clearly outperforms the ResNet agent on most
tasks from images and state features. Residual connections add the features of a previous layer to
the current layer instead of concatenating them (as done in D2RL). The same hyperparameters are used as D2RL for all experiments.
5.4 ABLATION STUDIES
Figure 6: Ablation studies with a SAC agent on the Ant-v2 env in the Open AI Gym suite. It is evident that the D2RL policy architecture applied both the policy and Q-value net- works achieves higher rewards than being ap- plied to either just the policy or just the value network. Also, in (b) deeper D2RL networks perform better. in contrast to vanilla MLP net- works in Fig.
6000
Sprodds Reweards
6000
3000
2000
B2RL,
2000
Towe Layars
Only Value Policy
Pour Layers Eight Layers
0.0
0.2
0.4
0.5
0.8
1.0
of Stops (ND
of Stops (N0
(al Inspostance of J2H
(b) Number of layers D2NL
In this section, we look answer the various components of using D2RL. We first analyze how the agent performs when only the policy or the value functions are parameterized as D2RL, while the other is vanilla 2-layer MLP. The results for training an SAC agent on Ant-v2 are present in Fig. 6al where we see that parameterizing both the networks as D2RL significantly outperforms when only one of the two use D2RL. However, one noteworthy observation can be made that when only the value functions are parameterized using D D2RL, the agent significantly outperforms the variant where only the policy is parameterized using a D2RL. This suggests that it may be more important to parameterize the value function, but more research is required to give a more conclusive statement.
Similarly we train the same agent but instead vary the number of layers used while parameterizing the policies and value functions using D2RL. The results in Fig. 6b show that even when 8 layer D2RL is used, the results are only moderately worse that when using 4 layers, even though it has twice the depth and therefore twice many parameters. These results are notably different from the results in Fig. 2 where as we increase the number of layers for vanilla MLPs to be greater than 2, see worsening results. The difference suggests that by using D2RL we able to circumvent the issue of DPI that may hinder the performance for vanilla MLPs, as we postulated.
DISCUSSION, FUTURE WORK, AND CONCLUSION
In this paper, investigated the effect of building better inductive biases into the architectures of the function approximators in deep reinforcement learning. We first looked into the effect of varying the number of layers to parameterize policies and value functions, and how the performance deteriorates as the number of layers increase. To overcome this problem, we proposed a generally applicable solution that significantly improves the sample efficiency of the state-of-the-art DRL baselines over a variety of manipulation, and locomotion environments with different robots, from both states and images. Studying the effect of network architectures has been long explored in computer vision and deep learning, and its benefits on performance have been well established. The effect of the network architectures, however, have not yet been studied in DRL and robotics. Improving the network architectures for variety of popular standard actor-critic algorithms demonstrates the importance of building better inductive biases in the network paramterization such that we can improve the performance for the otherwise identical algorithms. In future work, interested in building better network architectures and further improving the underlying algorithms for robotic learning.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Iige Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with robot hand. arXiv preprint arXiv:1910.07115. 2019.
Marcin Andrychowicz, Filip Wolski, Alex Ray, Jonas Schneider, Rachel Fong, Peter Welinder, Bob McGrew, Josh Tobin, OpenAl Pieter Abbeel, and Wojciech Zaremba. Hindsight experience re- play. In Advances in neural information processing systems, pp. 5048-5058, 2017.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach semi-supervised learning. In Advances in Neural Information Processing Systems, pp. 5049-5059, 2019.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.015400 2016.
Ting Chen, Simon Korbblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 2020.
Adji B Dieng, Yoon Kim, Alexander M Rush, and David M Blei. Avoiding latent variable collapse with generative skip models. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2397-2405, 2019.
Gabriel Dulac-Amold, Daniel Mankowitz, and Todd Hester. Challenges of real-world reinforcement learning. arXiv preprint arXiv: 12901, 2019.
Gabriel Dulac-Amold, Nir Levine, Daniel J Mankowitz, Jerry Li, Cosmin Paduraru, Sven Gowal, and Todd Hester. An empirical investigation of the challenges of real-world reinforeement learn- ing. arXiv preprint arXiv:2003. 1881, 2020.
Scott Fujimoto, Herke Van Hoof, and David Meger. Addressing function approximation error in actor-critic methods. arXiv preprint 1802.094777 2018
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. arXiv preprint arXiv: 0000000290, 2018.
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645 Springer, 2016b.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738. 2020.
Irina Higgins, Arka Pal, Andrei A Rusu, Loic Matthey, Christopher P Burgess, Alexander Pritzel, Matthew Botvinick, Charles Blundell, and Alexander Lerchner. Darla: Improving zero-shot trans- fer in reinforcement learning. arXiv preprint arXiv:
2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
10

---

Under review as a conference paper at ICLR 2021
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03166 2015.
Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang. Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, al. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. arXiv preprint arXiv: 10293, 2018.
llya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004. 13649, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Aviral Kumar, Rishabh Agarwal, Dibya Ghosh, and Sergey Levine. Implicit under-paraamttterizatoo inhibits data-efficient deep reinforcement learning. arXiv preprint arXiv:2010. /4498, 2020.
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Re- inforcement learning with augmented data, arXiv preprint WrXiv2200411499 2020.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(11) 1998.
Ofir Nachum, Shixiang Shane Gu, Honglak Lee, and Sergey Levine. Data-efficient hierarchical reinforcement learning. In Advances in Neural Information Processing Systems, pp. 3303-3313, 2018.
Ashvin V Nair, Vitchyr Pong, Murtaza Dalal, Shikhar Bahl, Steven Lin, and Sergey Levine Vi- sual reinforcement learning with imagined goals. In Advances in Neural Information Processing Systems, pp. 9191-9200, 2018.
Kei Ota, Tomoaki Oiki, Devesh Jha, Toshisada Mariyama, and Daniel Nikovski. Can increasing input dimensionality improve deep reinforcement learning? arXiv preprint arXiv:2003.0662,, 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. Advances in neural information processing systems, pp- 8026 8037,8037, 2019.
Chengrun Qiu, Yang Hu, Yan Chen, and Bing Zeng. Deep deterministic policy gradient (ddpg)- based energy harvesting wireless communications. IEEE Internet of Things Journal, 6(5):8577- 8588, 2019.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners.
David Rolnick and Max Tegmark. The power of deeper networks for expressing natural functions. arXiv preprint 1 2017.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch nor- malization help optimization? Advances in Neural Information Processing Systems, pp. 2483 2493, 2018
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pp. 1889-1897, 2015.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning. arXiv preprint arX1v22004.04136, 2020.

---

Under review as a conference paper at ICLR 2021
Yuval Tassa, Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, and Nicolas Heess. dm.control: Software and tasks for continuous control, 2020.
Gerald Tesauro. Temporal difference learning and td-gammon. Communications of the ACM, 38(3): 58-68, 1995.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, pp. 5998-6008, 2017.
Denis Yarats, Amy Zhang, Ilya Kostrikov, Brandon Amos, Joelle Pineau, and Rob Fergus. Im- proving sample efficiency in model-free reinforcement learning from images. arXiv preprint 1rX11:1910.01741, 2019.
12

---

Under review as a conference paper at ICLR 2021
A
PYTORCH CODE
import torch import torch.nn as nn import torch.nn. functional as F
LOG_SIG_MAX LOG_SIG_MIN -5
class Policy (nn. (odulle
def _init__sse___ num_inputs, num_actions, hidden_dim, action_space
None):
10
super (mper(Policyy self)._init_......
in satrden_ddimmmummiinee
self.linearl ninar (num_inputs, hidden_dim)
13
self.linear2 nn. (in_dim, hidden_dim)
14
w..inearrin_dim,, self.linear3 nn. (in_dim, hidden_dim)
15
self.lineard = (inineariin_diiii
hidden_dim,
1if.mmen__linest = mn.eine (hidden, dim, nam sum_accioos)
self.log_std_linee sineerrr_rr____ Lineir (hidden dim, num_actions)
18
19
def ferwa, statc):
20
X [.fcluusseff.1ncar (state))
terch. statel, dim-l)
..relu(se1f.1inear))
23
x terch, (1x, state], dim-1)
24
x = ff
25
= torch. cat [xx, statel, dim=1)
E (ellu
27
28
mean self.mean_linear((5
www.eannn
29
log_std = self self. selff.og_sstd_ineaarttt
log_std = torch clamp [td, min=LOG_SIG_MIN, max=1oG_SIG_MMN return mean, log_std
32
33
class (Network
34
def _nntt_ num_inputs, num_actions, hidden_dim, num_layers):
(Qeetok,,, self) i)
37
in in_dim num_ingut non.imootttttunnttttiiiitnhhh
18
self.11_1 nn. innerr (num_inguts+num_act
hidden_dim)
self.1l in.iinear (in dim, hidden_dimm
self.1l nn.iinear (in dim, hidden_diae
41
self.1l mn.Lnnear (in dim, hidden dim)
42
43
self.out1 nn Linear (hidden_dim, 1)
45
def ferward state, acticn):
46
xu torch, ([state, action], dim=1)
47
xl F rrelu((se1f.11_1(x)))
x1 corch.cat( xu], dim=1)
x1 F. reell((slf.111_(xx)) (x1))
50
x1 torch.cat xu], dim=1)
51
x1 3
52
x1 (Irc xu], dim=1)
x1 (sel (x1))
55
x1 self.out: (x1)
56
return x1
13

---

Under review as a conference paper at ICLR 2021
PyTorch code for stochastic SAC policy and Q-network (Haarnoja et al. 2018). The code provided can simply replace the policy and Q-newwrr for any current SAC implementation, or be adopted for other actor-critic algorithms such as TD3 (Fujimoto al. 2018 or DDPG (Qiu al. 2019).
B
ADDITIONAL EXPERIMENTS
1.0
Completed
0.8
Goal Comuperd
0.8
% Coall Complerdd
0.6
0.6
0.6
044
0.4
0.2
HER
0.2
HER
0.2
HER
0.0
0.0
0.0
2.5
5.0
7.5 of Staps (k)
10.0
12.5
0.0
2.5
5.0
7.5 10.0
12.5
0.0
2.5
5.0
7.5
10.0
12.5
of Stege tko
of Stege (k)
Fetch
Fesch Push
Fetch HER
Figure 7: Complete set of experiments with HER (Andrychowicz al. 2017 and DDPG QQuu et al 2019 the Fetch rohot. Using D2RL continues outperform the baseline the three environments considered.
C HYPERPARAMETERS AND ENVIRONMENT DETAILS
Table 4: Hyperparameters used for SAC in all the experiments. The hyperparameter values kept the across SAC and SAC-D2RL baselines environments.
Hyperparameter
Value
Hidden units (MLP)
256 for Gym
1024 for DM Control Yarats al. 2019
Evaluation episodes
10
Optimizer
Adam
Learning rate (Tp. Q+)
3e 4
Learning rate (a)
4
MiniBatch Size
256 RcLU
Non- con-lincarity
Discount
.99
Initial temperature
0.1
Fetch slide
bb) Jaco reach
(c) Ant maze
(d) Baxter Block join-lift
Figure Illustrations of some the challenging robotic control environments used our experiments. Fetch slide, Fetch robot arm with one finger must be controlled to slide a puck to goal location. In Jacko reach, a Jaco robot with a three finger gripper must be controlled to reach the red brick. In Ant maze, an Ant arm of Baxter robot with
with four legs must be controlled to navigate maze. In Baxter block join-lift, a finger gripper must be controlled join blocks and lift the combination above certain height.
14

---

Under review as a conference paper at ICLR 2021
Table 3: Hyperparamelers used for DMControl CURL experiments. The hyperparameter values are kept the same across CURL, CURL-D2RL, and CURL-ResNet basclines.
Hyperparameter
Value
Random crop
True
Ohservation rendering
(100, 100)
Observation downsampling
(84, 84)
Replay buffer size
100000
Initial steps
1000
Stacked frames
3
Action repeat
2 finger, spin; walker, walk
cartpole, swingup
otherwise
Hidden units (MLP)
1024 Yarats et al. 2019
Evaluation episodes Optimizer
10
Adam
(81,22 B2) - (So, TTp Q+)
(99.9999)
(31, B2) - (a)
(.5.9999
Learning rate (fo, Q+)
3e 4
Lcarning rate (a)
1e 4
Batch Size
512 (chectah), 128 (rest)
Q function EMA T
0.01
Critic target update freq
2
Convolutional layers
4
Number of filters
32
Non- con-lincarity
ReLU
Encoder EMA т
0.05
Latent dimension
50
Discount 7
.99
Initial temperature
0.1
Table 6: Consolidated details of all the environments used in experiments
Environment
Type
Controller
Inputs
Action dim.
Input dim.
Gym Cheetah
Locomotion
Joint torque
States
6
17
Gym Hopper
Locomotion
Joint torque
States
11
Gym Humanoid
Locomotion
Joint torque
States
376
Gym Walker
Locomotion
Joint torque
States
17
Gym Ant
Locomotion
Joint torque
States
8
111
Finger, Spin
Classical control
Joint torque
States/Images
2
9/ 84x84x3
Cartpole, Swing
Classical control
Joint torque
States/Images
5/ 84x84x3
Reacher, Easy
Classical control
Joint torque
States/Images
6/ 84x84x3
Cheetah, Run
Locomotion
Joint torque
States/Images
17 84x84x3
Walker, Walk
Locomotion
Joint torque
States/Images
24/ 84x84x3
Ball in Cup. Catch
Manipulation
Joint torque
States/Images
8/ 84x84x3
Fetch Reach
Manipulation Manipulation
EE position
States
10
Fetch Pick and Place
EE position
States
25
Fetch Push
Manipulation
EE position
States
4
25
Fetch Slide
Manipulation
EE position
States
3
25
Jaco Reach
Manipulation
Joint torque
States
45
Baxter Joinl
Manipulation
Joint torque
States
42
Ant Maze
Locomotion
Joint torque
States
8
30
15


Reviews:
[{"review_id": "m96Me7d-7qd", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission takes inspiration from work on deep learning architectures for visual tasks in order to make targeted model changes to deep reinforcement learning models. The authors show that by including “dense connections” (concatenating the state or state-action pair to the input of each hidden layer of the network) they are able to successfully train deeper networks. \n\nThe main idea behind the work is simple but effective, and their model surpasses the most of the presented benchmarks. The paper is also well written and presents a thorough set of experiments, making it a good submission for ICLR. \n\nPositives: \n* The main idea behind the architecture is fairly simple and the explanation is grounded in previous architectures (specifically densenet), making the experiments quite easy to understand.\n* The authors evaluate their method on a diver set of tasks, and their model outperforms the benchmark for the majority of tested conditions\n\nConcerns and Questions:\n* The results comparing the ResNet style architecture with the DenseNet style architecture are interesting, particularly because the ResNet architecture does not see the same benefit. An explanation of how to interpret this result would be helpful to readers (ie why does a residual connection in this setting not help with DPI?).\n* It is unclear why the authors chose to use a 4 layer D2RL. It looks like an experiment was done in 6b varying the number of layers, but perhaps introducing this earlier (ie as a direct comparison to Figure 2) would make this choice more clear.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "LZa1cAMS6E4", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a deep neural net structure for deep reinforcement learning methods (e.g., SAC) to replace the original fully-connected layers, by concatenating the state input into every hidden layers. The authors conduct experiments on OpenAI gym and MuJoCo environments and show that the proposed structure can further improve the performance of SAC or CURL. \n\nStrong points:\n1. The authors propose a method by concatenating the state features to every layer of the neural net to improve the performance of RL algorithm. The proposed method seems to have overcome the issue that purely adding more layers of fully-connected network can even harm the performance.\n\nWeak points:\n1. The biggest issue of this work is that the proposed method, regardless of the activation function, is similar to a special version of resnet. Stacking residual layers can make it possible to have skip connections from every layer to any layer after it. Thus, resnet has included the connection directly from feature input to each layer. Therefore, this method seems to lack enough technical innovation.\n2. The authors should compare with other types of neural net structures that aim to solve the \"depth\" problem. At least, resnet should be compared.\n\n\nMinor comments:\nWill this network structure also work for supervised learning problems? It seems this structure is independent from the RL setting.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4CvrinDpgzo", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors propose a neural network architecture that concatenates the input state with hidden state activations over multiple layers in order to train deeper networks in an RL setting. Whilst the work does improve over standard MLP in this setting, is seems like an incremental work that lacks real novelty.\n\nThe idea of residual connections or concatenation to improve stability of networks is not a new one. Although there is nothing technically wrong with this paper and there is an improvement over a vanilla network, I do not feel the work is enough for a publication at ICLR, the work is not novel enough and the authors should focus on bigger steps rather than incremental work.\n\nThe following changes would be required for me to up my rating:\n1. More ablations, particularly vs. resnet architectures, it would be good to see figure 2 with a resnet comparison.\n2. Analysis of why the standard MLP case fails, is the weight activation suitable, are there vanishing gradients? I find the discussion about DPI a bit hand-wavey.\n3. Comparisons on other environments such a Atari and even partially observable environments (DM-Lab, Habitat...)\n\n\nAs many of these changes are out of scope for a rebuttal, I would suggest that this publication is not ready for a venue such as ICLR and should either be greatly extended to a larger suite of scenarios such as Atari or submitted to more suitable conference.\n\nUpdate:\nI thank the authors for their significant updates to the paper.  Given the extended effort made by the authors, I am willing to raise my score to 5. My conclusion however remains the same, this work is not a significant advancement that we would expect to see at a conference such as ICLR.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2Sy4SSbXGwY", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\n**Summary**:\n\nThis paper investigates the effect of different network architecture in the context of reinforcement learning. It shows that by appending the input to each mid-layer's output, one can use a deeper network to get better learning performance. The idea is very similar to the residual connection or the skip connection. And I don't see too much novelty in applying such an idea in RL settings. \n\n\n##########################################################################\n\n**Strengths**:\n\nThe paper is well-written and provides PyTorch code that is easy to read and understand.\n\nIt experiments across a wide range of environments and tasks.\n\n##########################################################################\n\n**Weaknesses**:\n\nThe paper is investigating the use of skip connection in deeper networks in RL. The skip connection can be a summation operation like ResNet or a concatenation operation like DenseNet. This paper uses concatenation. Such an idea is not new to the learning community. There is nothing specific in RL that prevents one from using such standard techniques in the networks either. It is common to use skip connections in a deep network, even in RL [1, 2, 3]. The novelty of the paper is limited. And I would like to see a more systematic and thorough analysis of why this is a good choice people should choose and how it compares to other ways of skip connections, etc.\n\n\nFigures 3, 4, 5 show the results on a shallow (2 layers) network and a relatively deep (4 layers) network with skip connections. It is not clear whether the effect solely comes from a deep network or the skip connection. Even though Figure 2 shows that a deep network does not perform well in Ant-v2, this might not hold true in the other environments. Hence, I would like to see the learning curves of a deep network without skip connections. Also, since the network becomes deep and RL is sensitive to hyperparameters, it makes sense to tune the hyperparameters as well. We should compare the performance of two scenarios when each of them is best tuned.\n\nConcatenating the input to each mid-layer also makes each layer wider. What about using addition instead of concatenation? One can simply use a residual connection in each layer until the last one. Each layer will be `y=x+f(x)`, which is fairly common in many deep networks. How does it perform? And what about using the deep network with the same number of parameters without a skip connection? \n\n\nThe paper only shows the results with one kind of network width. As shown in [4], network architecture (both the width and depth) has a significant effect on the outcomes. I would like to see the effectiveness of D2RL on networks with a few different widths. \n\n\n\nMissing details:\n* Reward structure for the environments.\n* It is not clear how many skip connections are added in CURL. More details should be provided. \n\n[1] Espeholt, Lasse, et al. \"Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures.\" arXiv preprint arXiv:1802.01561 (2018).\n\n[2] Gupta, Saurabh, et al. \"Cognitive mapping and planning for visual navigation.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.\n\n[3]: Finn, Chelsea, and Sergey Levine. \"Deep visual foresight for planning robot motion.\" 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017.\n\n[4] Henderson, Peter, et al. \"Deep reinforcement learning that matters.\" arXiv preprint arXiv:1709.06560 (2017).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper shows that replacing fully connected layers by dense layers in the networks used by actors and critiques in RL can improve the results significantly.  The improvements for several RL techniques across several benchmarks are very nice.  That being said, replacing fully connected layers by dense layers is not particularly novel and it is not clear why dense layers instead of resnet layers works better.  The reviewers appreciate the addition of experiments confirming that dense layers work better than resnet layers.  This addresses an important concern of the reviewers.  However, at this point in deep learning, it is well-known that fully connected layers do not work well in general and therefore engineers are expected to use resnet, dense or highway style connections to improve performance when increasing the depth.  The fact that published baselines in OpenAI, TensorFlow and PyTorch do not use those improved networks is one thing, but this does not justify the publication of a paper.  The paper suggests that an RL-specific architecture will be proposed, but at the end of the day what is being proposed is not specifically for RL, but rather the addition of new connections to the inputs similar to the well-known dense architecture to augment fully connected layers in RL.  It is not clear why this works better than resnet connections.  Another alternative that was not considered are highway networks.  To strengthen the contribution of the paper, the authors are encouraged to provide an analysis of the possible approaches and to provide some insights.  
Decision: Reject