Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: In this work, the authors propose a neural network architecture that concatenates the input state with hidden state activations over multiple layers in order to train deeper networks in an RL setting. Whilst the work does improve over standard MLP in this setting, is seems like an incremental work that lacks real novelty.

The idea of residual connections or concatenation to improve stability of networks is not a new one. Although there is nothing technically wrong with this paper and there is an improvement over a vanilla network, I do not feel the work is enough for a publication at ICLR, the work is not novel enough and the authors should focus on bigger steps rather than incremental work.

The following changes would be required for me to up my rating:
1. More ablations, particularly vs. resnet architectures, it would be good to see figure 2 with a resnet comparison.
2. Analysis of why the standard MLP case fails, is the weight activation suitable, are there vanishing gradients? I find the discussion about DPI a bit hand-wavey.
3. Comparisons on other environments such a Atari and even partially observable environments (DM-Lab, Habitat...)


As many of these changes are out of scope for a rebuttal, I would suggest that this publication is not ready for a venue such as ICLR and should either be greatly extended to a larger suite of scenarios such as Atari or submitted to more suitable conference.

Update:
I thank the authors for their significant updates to the paper.  Given the extended effort made by the authors, I am willing to raise my score to 5. My conclusion however remains the same, this work is not a significant advancement that we would expect to see at a conference such as ICLR.
review_title: The work does empirically improve performance on a range of fully observable benchmarks. But lacks analysis and real novelty 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This submission takes inspiration from work on deep learning architectures for visual tasks in order to make targeted model changes to deep reinforcement learning models. The authors show that by including “dense connections” (concatenating the state or state-action pair to the input of each hidden layer of the network) they are able to successfully train deeper networks. 

The main idea behind the work is simple but effective, and their model surpasses the most of the presented benchmarks. The paper is also well written and presents a thorough set of experiments, making it a good submission for ICLR. 

Positives: 
* The main idea behind the architecture is fairly simple and the explanation is grounded in previous architectures (specifically densenet), making the experiments quite easy to understand.
* The authors evaluate their method on a diver set of tasks, and their model outperforms the benchmark for the majority of tested conditions

Concerns and Questions:
* The results comparing the ResNet style architecture with the DenseNet style architecture are interesting, particularly because the ResNet architecture does not see the same benefit. An explanation of how to interpret this result would be helpful to readers (ie why does a residual connection in this setting not help with DPI?).
* It is unclear why the authors chose to use a 4 layer D2RL. It looks like an experiment was done in 6b varying the number of layers, but perhaps introducing this earlier (ie as a direct comparison to Figure 2) would make this choice more clear.

review_title: novel and effective architecture for deep reinforcement learning 

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes a deep neural net structure for deep reinforcement learning methods (e.g., SAC) to replace the original fully-connected layers, by concatenating the state input into every hidden layers. The authors conduct experiments on OpenAI gym and MuJoCo environments and show that the proposed structure can further improve the performance of SAC or CURL. 

Strong points:
1. The authors propose a method by concatenating the state features to every layer of the neural net to improve the performance of RL algorithm. The proposed method seems to have overcome the issue that purely adding more layers of fully-connected network can even harm the performance.

Weak points:
1. The biggest issue of this work is that the proposed method, regardless of the activation function, is similar to a special version of resnet. Stacking residual layers can make it possible to have skip connections from every layer to any layer after it. Thus, resnet has included the connection directly from feature input to each layer. Therefore, this method seems to lack enough technical innovation.
2. The authors should compare with other types of neural net structures that aim to solve the "depth" problem. At least, resnet should be compared.


Minor comments:
Will this network structure also work for supervised learning problems? It seems this structure is independent from the RL setting.
review_title: Resnet should be compared

final_score: rating: 4: Ok but not good enough - rejection
review_content: ##########################################################################

**Summary**:

This paper investigates the effect of different network architecture in the context of reinforcement learning. It shows that by appending the input to each mid-layer's output, one can use a deeper network to get better learning performance. The idea is very similar to the residual connection or the skip connection. And I don't see too much novelty in applying such an idea in RL settings. 


##########################################################################

**Strengths**:

The paper is well-written and provides PyTorch code that is easy to read and understand.

It experiments across a wide range of environments and tasks.

##########################################################################

**Weaknesses**:

The paper is investigating the use of skip connection in deeper networks in RL. The skip connection can be a summation operation like ResNet or a concatenation operation like DenseNet. This paper uses concatenation. Such an idea is not new to the learning community. There is nothing specific in RL that prevents one from using such standard techniques in the networks either. It is common to use skip connections in a deep network, even in RL [1, 2, 3]. The novelty of the paper is limited. And I would like to see a more systematic and thorough analysis of why this is a good choice people should choose and how it compares to other ways of skip connections, etc.


Figures 3, 4, 5 show the results on a shallow (2 layers) network and a relatively deep (4 layers) network with skip connections. It is not clear whether the effect solely comes from a deep network or the skip connection. Even though Figure 2 shows that a deep network does not perform well in Ant-v2, this might not hold true in the other environments. Hence, I would like to see the learning curves of a deep network without skip connections. Also, since the network becomes deep and RL is sensitive to hyperparameters, it makes sense to tune the hyperparameters as well. We should compare the performance of two scenarios when each of them is best tuned.

Concatenating the input to each mid-layer also makes each layer wider. What about using addition instead of concatenation? One can simply use a residual connection in each layer until the last one. Each layer will be `y=x+f(x)`, which is fairly common in many deep networks. How does it perform? And what about using the deep network with the same number of parameters without a skip connection? 


The paper only shows the results with one kind of network width. As shown in [4], network architecture (both the width and depth) has a significant effect on the outcomes. I would like to see the effectiveness of D2RL on networks with a few different widths. 



Missing details:
* Reward structure for the environments.
* It is not clear how many skip connections are added in CURL. More details should be provided. 

[1] Espeholt, Lasse, et al. "Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures." arXiv preprint arXiv:1802.01561 (2018).

[2] Gupta, Saurabh, et al. "Cognitive mapping and planning for visual navigation." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.

[3]: Finn, Chelsea, and Sergey Levine. "Deep visual foresight for planning robot motion." 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2017.

[4] Henderson, Peter, et al. "Deep reinforcement learning that matters." arXiv preprint arXiv:1709.06560 (2017).

review_title: Official Blind Review #2

Meta-review: The paper shows that replacing fully connected layers by dense layers in the networks used by actors and critiques in RL can improve the results significantly.  The improvements for several RL techniques across several benchmarks are very nice.  That being said, replacing fully connected layers by dense layers is not particularly novel and it is not clear why dense layers instead of resnet layers works better.  The reviewers appreciate the addition of experiments confirming that dense layers work better than resnet layers.  This addresses an important concern of the reviewers.  However, at this point in deep learning, it is well-known that fully connected layers do not work well in general and therefore engineers are expected to use resnet, dense or highway style connections to improve performance when increasing the depth.  The fact that published baselines in OpenAI, TensorFlow and PyTorch do not use those improved networks is one thing, but this does not justify the publication of a paper.  The paper suggests that an RL-specific architecture will be proposed, but at the end of the day what is being proposed is not specifically for RL, but rather the addition of new connections to the inputs similar to the well-known dense architecture to augment fully connected layers in RL.  It is not clear why this works better than resnet connections.  Another alternative that was not considered are highway networks.  To strengthen the contribution of the paper, the authors are encouraged to provide an analysis of the possible approaches and to provide some insights.  
Decision: Reject