Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper argues that network architectures are under explored in reinforcement learning and that much can be gained from understanding the lack of success of larger models. The paper focuses on continuous control from pixels with the SAC algorithm and starts by demonstrating that a naive application of ‘modern’ architectures leads to a degradation in performance. The authors argue that this degradation in performance is at least partially due to instabilities caused by large gradients from the critic to the policy actions. They provide a mathematical motivation for why this is a bigger problem for deeper networks and propose to apply spectral normalization, which is a popular technique for controlling the gradients of the discriminators in GANs. Finally the paper reports empirical results supporting the claim that this method allows for the use of larger architectures and by virtue of that SOTA results that are on par or better than results obtained with algorithmic innovations. 

main_review: The novelty of the work is limited. As the authors point out, spectral normalization has been applied to critic gradients before. To my knowledge this is however the first time that its effect on learning performance is investigated systematically in relation to larger networks. The notion that larger networks are not as prominent in reinforcement learning is not new either (neither do the authors claim this) but excluding overfitting as the explanation and investigating why this is the case for the set of critic-gradient using RL algorithms could be novel. 

The largest problem I see with this paper is that at various parts it promises to address why large architectures are not popular for use with RL algorithms in general while focussing on an issue that only affects those algorithms that use the gradient of the critic with respect to the actions. Examples of such algorithms are DDPG/TD3 and SAC, which are popular in continuous control but arguably still not as commonly used as Q learning, PPO and TRPO, which don’t use critic gradients. The authors mention this limitation and write that their goal is not to “provide a solution for every conceivable RL algorithm”. They are right that this is of course too much to expect from a single paper but by addressing an issue that is caused by a property that only affects a small number of algorithms, they don’t investigate why small networks are not popular with RL in general like the abstract and introduction suggest, especially considering that critic gradient using algorithms are relatively new compared to standard policy gradient and Q-learning. 

The experiments are done on a variety of tasks from a well-known benchmark suite and this adds to the quality of the paper. The results are also the averages of multiple runs in most case but for the results in Figure 1 and 2 this is not clear as there are no error bars. 
The evaluation of the critic loss in 3.4 is somewhat hand-wavy because it cannot be considered in isolation of what the actor is doing. The critic loss could also be higher because a larger variety of states and returns has been observed due to increased exploration for example. 

The paper is a bit messy at times and this makes some results a bit harder to interpret and some of the mathematics a bit confusing or simply incorrect, especially in the appendix.
The baseline DRQ/mlp results in both Table 1 and Table 4 (appendix) seem to be quite a bit worse than the numbers reported in Kostrikov et al. (2020) [34] Table 1 and it is not clear to me where those numbers come from. The appendix says that they come from the curves in the original papers [25,34] but I don’t understand how precise mean estimates and precision values (not clear from the captions whether these are standard deviations or standard errors) can be obtained from those. It seems to me like the current paper uses the same control suite tasks but not he exact same PlaNet evaluation setup but this is not 100% clear either. Having baseline results that are apparently much worse than those reported in the original papers undermines all the comparisons with papers and is very misleading.

Here are some examples of issues with the mathematics:
- It seems to me that the maximization in Equation 4 (and other equations derived from it) should be a minimization (or otherwise placed before the fraction) because as it stands, the norm of the total gradient is now upper-bounded by sqrt(N) times the value corresponding to the layer with the lowest norm bound. This seems to replace an upper-bound on a (square rooted) sum with a repetition of its lowest term, which must be wrong (the bound on the total gradient would become lower than the bounds on some of the individual layers if we let the operator norm of one of the layers grow much larger than the others).
- The `A` in the equation at the end of Section 4.2 should probably be a ‘Q’ because the gradient magnitude of the actor output itself doesn’t depend on the critic weights.
- The subscripts of the products in Equation 6 of the appendix don’t make sense to me. I assume they should have values like ‘j=i+1’ instead of ‘j+1’
- It’s a bit odd that both the actor and critic have N layers, making it less clear what the result would be when these numbers were different.
- Last sentence of p16 (appendix): “one obtains an estimate of the output of the j:th layer:” is followed by Equation 6 which clearly defines bounds on the norms of the layer outputs and not estimates of those layers.
- Second sentence p17 (appendix): No superscript on the weight and ‘Q’ should probably be ‘A’.

Another point that was not clear from the text but that is implicitly addressed in the appendix is that the arguments about the spectral normalization only make sense if the spectral norms of larger nets also grow during learning. Otherwise, initialization with small enough weights for deeper nets would be an easier solution than spectral normalization.

Edit: I raised my score in light of the clarifications by the authors and the additional DDPG results.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper starts from the following observation: Deeper and wider architectures usually provide better performance in supervised learning. This is often not the case in Reinforcement learning. 

The paper then proceeds to formulate two hypothesis for this observation: deeper architectures could overfit, or deeper architectures could be harder to train, due to ill-conditioning of gradients. After refuting the first hypothesis, the paper provides a theoretical argument for the second, and provide an attempt at fixing it using spectral normalization. The paper provides improved experimental results with deeper architectures using spectral normalization.
main_review: The paper at hand provides an interesting and simple solution which seems to enable the use of deeper architectures in Reinforcement Learning. The paper reads well, and empirical results are compelling and well presented.

However, I have a few concerns with the current revision:
- My first concern is about the novelty and originality of the provided approach. Applying Spectral Normalization to help optimization in an RL context has already been analyzed in (Gogianu et al. 2021), with similar conclusions, though on a different base algorithm. This reference is unfortunately missing in the current draft. Applying the same technique to a different algorithm and different environments makes the contribution of this paper somewhat incremental.

Edit: The Area Chair has brought to my attention that (Gogianu et al. 2021) was only available online on arxiv within the two months interval before the ICML submission, and that it should thus be considered concurrent work. I've edited my score accordingly, but still think that this concurrent submission should be discussed in the final revision.

- My second concern is about the soundness of the theoretical analysis. In the introduction, the authors mention that scaling up the critic typically lead to exploding gradients (which seems empirically true), but mention that they are motivating it mathematically. However, the theoretical analysis provided in the paper gives an upper bound for the gradient of the critic. This upper bound does not give proper backing for the exploding gradient hypothesis, a lower bound would be much better suited. 
- My third, and probably lesser concern is about the first hypothesis mentioned by the paper, that bigger networks could lead to more overfitting in simple reinforcement learning environments. While this could well be a valid hypothesis, the phrasing of the paper seems to imply that this hypothesis is backed up by the relative scale of reinforcement learning 'datasets' compared to supervised datasets. I don't think this holds in general, and I think that in many instances, the 'datasets', referring to the total number of unique observations that the agent has seen during training is of the same order of magnitude, or higher than the number of datapoints considered in supervised settings (e.g. A3C trains on roughly 160 millions frames total).

As the paper is currently mostly experimental, I think a direction that would significantly improve the paper would be to test the hypothesis that the architectural change performed improves performance across RL algorithms and environments. While I understand that this is both computationally costly, and far from trivial to implement, this would provide a much stronger and general result.

For the reasons mentionned above, I think the paper is not yet ready for publication at NeurIPS.

--------
References:

_Spectral Normalisation for Deep Reinforcement Learning: An Optimisation Perspective_ Florin Gogianu, Tudor Berariu, Mihaela Rosca, Claudia Clopath, Lucian Busoniu, Razvan Pascanu, ICML 2021

# Post rebuttal and additional experiments edit:
The authors have provided further results on DDPG, which was one of the things that I asked for. I think those results already make the paper a lot stronger. For this reason, I am raising my score from 4 to 5. The reason why I am not increasing it further is because I am still concerned by the theoretical part, and the claims associated to it. While I agree that the bounds can be made tight, I don't think it strengthen or back up the claims made in the introduction. I would suggest removing this theoretical part, and only backing the claims up experimentally. Removing this part would also probably allow the authors to put the results on DDPG in the main text.

# Post rebuttal second edit:
The authors have agreed to remove the theoretical part that I considered potentially misleading. I no longer have any major concerns about the paper, and I am raising my score from 5 to 6.
limitations_and_societal_impact: The authors adequately addressed the limitations and potential negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper hypothesizes that the poor performance of modern NN architectures with the SAC algorithm stems from large gradient updates caused by propagating gradients for actor through critic. The authors mitigate the issue by borrowing a spectral normalization technique from GANs, which was designed for a similar problem. Experimentally, they verify that the approach improves the performance and allows usage of modern architectures, with gains comparable to algorithmic advances of RL algorithms themselves.
main_review: The paper is well and systematically written. It gradually presents the main problem - that modern architectures do not work well with the SAC algorithms, tests it accordingly, and rule out the overfitting hypothesis. It then presents the main culprit - large gradient updates caused by taking gradients through critic, and theoretically shows that spectral normalization can help. Experimentally, they verify that it is indeed the case. 

Although the actual improvement is trivial - just use a spectral normalization - the potential impact of the results is high. The experiments show that the gains from the technique is comparable to algorithmic improvements. Also, the paper does not simply report a result, but it clearly explains the underlying cause of the problem.

A limitation is that the issue is only present in SAC-similar algorithms that propagate the gradient through critic. Nevertheless, it also needs to be said that this class of algorithms is important and well used. The authors are aware of the limitation and state it clearly.

Notes:
- l. 169 - unclear whether $f$ in $||f||$ is a substitute symbol for any function or the already defined $f$ symbol.
- Table 1 - it would strengthen the paper and be interesting to see how the spectral normalization help in the DRQ / DDPG / Dreamer algorithms
- similarly, it would be interesting to see whether the effect helps in other classes of algorithms (or not), where the gradient is not taken through critic
limitations_and_societal_impact: - addressed limitations
- no apparent negative societal impact
review_title: null

Meta-review: This paper argues, in RL applications --specifically focusing on SAC algorithm-- typical deeper and wider architectures don't provide big gains that we see in supervised learning. Then paper shows that spectral normalization could be used to improve the performance of deep architectures. The paper focuses on the continuous control problems. 

The authors did a very good job during the rebuttal period and they managed to address the most concerns raised by the reviewers. Specifically, the authors have addressed the limited scope of just discussing on SAC by adding DDPG results too.. The paper could be improved by having the idea tested on a wider class of algorithms and with more smoothness controlling techniques, which the authors have identified as the limitations in the work. The results provided in this paper would still be valuable for the NeurIPS community. I would recommend the authors to cite and discuss about the concurrent work suggested by the reviewer FNPT [1].

[1] Spectral Normalisation for Deep Reinforcement Learning: An Optimisation Perspective Florin Gogianu, Tudor Berariu, Mihaela Rosca, Claudia Clopath, Lucian Busoniu, Razvan Pascanu, ICML 2021




Decision: Accept (Poster)