Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to combine differentiable neural architecture search (DARTS) with standard reinforcement learning (RL) frameworks by searching the model structures (convolutional cells) for the policy and value functions. The authors have applied the method to infinitely procedurally generated Procgen benchmark and demonstrated the benefits of DARTS in RL on search efficiency in terms of time and compute. In addition, the proposed method can be easily integrated with existing RL pipelines by simply replacing the image encoder with a DARTS supernet, compatible with both off-policy and on-policy RL algorithms. The authors further show that the supernet gradually learns better cells with more training iterations, leading to alternative architectures that can be highly competitive against manually designed policies and verify previous design choices for RL policies.
main_review: [Strength]

This paper aims to tackle interesting but less explored questions: while there have been extensive studies on searching network architectures for supervised learning, it is generally less understood whether and to which extent architecture search would benefit reinforcement learning pipelines.

This paper has provided extensive experimental results exploring this direction and studied how different design choices would affect the learning procedure and final results.


[Weakness]

While I like the direction this paper is going, I have concerns regarding the novelty of this paper and whether the experiments show the full power of differentiable neural architecture search (DARTS).

The proposed method is a direct application of neural architecture search on reinforcement learning problems without any new technical innovations. I'm not against straightforward A+B types of research (in this paper, A=RL, B=DARTS). For example, the nice thing about the proposed method is that it is simple and can be a plug-and-play module on top of existing RL frameworks. However, I typically expect the combined method to accomplish tasks that neither A nor B can do, or at least demonstrate significant performance gain over existing methods. Yet from Figure 3, it seems that the proposed method is on par (in Maze) or worse (in Bigfish and Dodgeball) than the IMPALA-CNN baseline [1], which makes me worry about the significance of the proposed method and its impact on the field. For example, should we prefer IMPALA-CNN over this paper's method since IMPALA-CNN is supposed to be easier to train as the network structures are fixed? The authors may want to include more discussions or comparisons to show how the proposed method is better and in which scenario we should choose it over other state-of-the-art approaches.

Continuing from my previous point, the IMPALA-CNN baseline also seems to have a smaller search cost, according to Table 3.

One great benefit of neural architecture search is that it allows a tradeoff between model accuracy and computational efficiency [2]. Are there any similar tradeoffs we can make in the RL setting, e.g., would smaller/sparser neural networks allow higher feedback control frequency such that control becomes easier? Showing examples demonstrating the unique benefit of neural architecture search can potentially make this paper much stronger and show potentials to enable new capabilities.

In the Heist task of Figure 8, why do random search and IMPALA-CNN seem to have a very good initial start but experience a sharp drop in performance at the beginning of the training processes? Why not directly keep the initial policy?



[Minor points]

The authors made a claim in the intro that "in RL, a bad architecture ... and thus a local optimum where the loss is zero while the policy is still poor." However, the statement may not be well-grounded without context, e.g., to which objective is the "optimum" referring? Is the "loss" defined over a very small replay buffer? If the loss is zero, why is it still a local optimum (assuming that the loss is nonnegative)? 

Typo in Page 7 first paragraph:
"RL time cost is partially is based on": multiple "is"


[1] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, Koray Kavukcuoglu, "IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures"
[2] Han Cai, Ligeng Zhu, Song Han, "ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware"


===============

[Post Rebuttal]

I thank the authors for the detailed feedback. I have read the reviews from other reviewers and decided to keep my score the same (5: marginally below the acceptance threshold).

As shared by most reviewers, the novelty of this paper is a bit limited, and as I have stated in my main review, I'm not against A+B types of research (in this paper, A=Reinforcement Learning, B=Differentiable Neural Architecture Search), but I typically have high expectations from such straightforward combination.

However, I do not think this paper has shown the full benefit of the combination. It will greatly strengthen this paper if the authors show what new capabilities are enabled by the proposed method but are not possible with either A or B.
summary_of_the_review: While I like the direction this paper is going, I have concerns regarding the novelty of this paper and whether the experiments show the full power of differentiable neural architecture search (DARTS). Therefore I currently lean towards the rejection side. Please see my main review for more details.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper studies the use of DARTS within the RL setting. Specifically, it investigates how DARTS can optimize perception modules for RL environments. RL-DARTS is optimised in a typical RL setting, using the standard RL loss function, in an end-to-end manner.
The proposed approach is evaluated using the Procgen benchmark, and compared to the IMPALA-CNN architecture. Several ablation studies are performed to further examine the contributions of the components of the proposed approach.

main_review: The authors give a nice overview of common issues of using DARTS within the RL framework, specifically compared to supervised learning settings. An interesting point is the non-stationarity of the collected data, which depends on the initialization, which does not occur in SL.

I found the explanation of the training procedure a bit unclear, and it would be beneficial to focus on this more, as this is one of the main contributions.

The performance does not seem to be better compared to the Vanilla benchmark in 4.2. I would like to see more discussion why this happens and maybe what are some potential improvements to supernet training and discretisation that could improve the overall performance.

Some relevant references which I think should be discussed, in order to provide better context for the proposed approach:
  - Luo, Renqian, Fei Tian, Tao Qin, Enhong Chen, and Tie-Yan Liu. "Neural architecture optimization." arXiv, 2018.
  - Stanley, Kenneth O., David B. D'Ambrosio, and Jason Gauci. "A hypercube-based encoding for evolving large-scale neural networks." Artificial life, 2009

summary_of_the_review: - Discuss additional references
- Provide some additional discussion how can the performance be improved
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper considers the problem of differentiable neural architecture search for RL applications. It applies existing DARTS to the RL setting and conducts extensive experimental studies to examine the performance and behaviors of RL-DARTS. Experimental results show that the supernet learns alternative architectures that are highly competitive against manually designed policies.
main_review: Strength: this paper discusses several main challenges of DARTS for RL. For example, the main challenge of applying DARTS to RL is that it needs to address the non-stationary data distribution arises due to the exploration process, where the data distribution depends on the network architecture used to generate the replay data. Another challenge is that it is unclear if DARTS’s default discretization procedure leads to a better discrete cell in the RL setting due to different optimization landscape in the RL case. In addition, the work also carried out extensive experiments to examine the behaviors of DARTS for RL.

Weakness. The main weakness of the paper is its lack of novelty and insufficient technical contribution. The work is mainly an application of standard DARTS to the RL setting, without additional contributions in the algorithm side. Although the unique challenges of DARTS for RL are pointed out earlier in the paper, the paper does not propose any solution to address these challenges. In contrast, experiments in later sections seem to indicate that these are indeed not the case, and standard DARTS technique developed in SL could be directly applied to the RL setting with minimal modification. For example, experimental results seem to suggest that end-to-end DARTS works well on such non-stationary data distribution even without any additional tweaks. (Likewise, the concerns about the discretization has also been shown to be not an issue here in experiments.) While it is indeed interesting to observe that existing DARTS from SL could be readily applicable to RL, it does not play a major contribution for the current paper.

In summary, this paper can be regarded as an extensive experimental investigations, which find out that standard DARTS developed in SL are actually applicable to RL. Although the findings are interesting, it could be fit better as a workshop paper.

- Minor comments
Need to give the precise definition / expression for the loss L(theta, alpha) in (2) in the RL-DARTS.

summary_of_the_review: This paper can be regarded as an extensive experimental investigations, which find out that standard DARTS developed in SL are actually applicable to RL. Although the findings are interesting, it could be fit better as a workshop paper.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors aim to automatically design Reinforcement Learning Q function approximator via NAS, specifically by adapting DARTS framework. They main motivation behind this is that as RL state spaces become more complex simple function approximator models, even predesigned DNNs prove to be sub-optimal, since supervised models are not always suitably designed to learn in an online fashion from reward signals rather than labels. Beyond that there are also existing RL bottlenecks such as reward sparsity, sample efficiency etc. The authors use the whole DARTS supernet at the function approximator. To bypass the bilevel optimization target in DARTS, which could prove to be intractable for RL context, authors propose to optimize the loss on cumulative returns of the RL problem by optimizing the loss on architecture and weight parameters from the replay buffer samples. 
main_review: 1 NAS in RL is a difficult problem. However has lot of promise. Designing suitable function approximators can affect convergence dynamics of deep RL. 

2. The proposed solution and its presentation, motivation and explanation is very lucid and clear. Notations are introduced succintly and resepctive intuitions are clearly stated as well. 

3. The experiments are indeed quite extensive and the authors have attempted to study the effect of apply end to end DARTS in RL from many possible dimensions including ablation studies on how the responses change in different kind of convolutions. 


Weaknesses/Concerns

1. Novelty is a concern. Tts not that people have not thought about NAS for RL. but prohibitive search space becomes a bottleneck in most cases. And the authors here solve that simply by taking DARTS and applying it to learning function approximators for deep RL. 
 
   1a.  One clear example of this is the fact that authors point out some interesting challenges wil RL earlier in the paper such as "RL training curves possess considerably more variance and noise than SL curves". However I do not see any change in formulation in section 3.2 to address that concern. 
 
   1b. Another challenge with RL which the authors also have hinted at is that unlike SL feedback data in RL replay buffer suffers from slight distributional shift over time, until enough amount of exploration is done. This is quite a big problem and RL-DARTS formulation does take care of this as well. 
 
 Inshort is is difficult to identify how the authors have extended DARTS formulation to suitably adapt to RL scenario. JUst stating that it was integrated with minimal code change is not sufficient. 

2. The main section needs to be expanded a bit more, the entire methodology is compressed into a few paragraphs. The formal algorithm provided here is also a surprise. It literally has 3 steps. A reader can already see these 3 things. An algorithm is meant to give the reader an idea of how to even implent the same without using any backbone codebase. For instance, this is an iterative setup. So the algo should outline at each itheration, how the components of the loss are computed. in what order the architecture parameters are updated etc... 

3. Diversity in domains is needed. Procgen is a great becnmark indeed. However no matter how we look at them and their different settings they are games. Other tasks such as robot locomition (simple version), or drone simulations, or treatment planning in clinical decision support systems will actually be able to realy test the validity of the claims made herein. 

summary_of_the_review: This indeed and interesting work / study and has a lot of merit. However Authors are strongly encouraged to address the concerns.
review_title: null

Meta-review: The paper studies network architecture search in the context of reinforcement learning. In particular it applies the DARTS method to the Procgen RL benchmark, and conducts extensive experimental evaluations. It identifies a number of issues that could potentially prevent DARTS from working well in the RL setting (such as nonstationarity and high variance), but in the end shows good performance without needing to modify DARTS substantially.

The reviewers agreed that a key strength of the paper is in its experiments. But they also identified a weakness in novelty: if a paper's main contribution is to combine two previously well-explored ideas (in this case, RL and DARTS) then there is a high bar for the quality of exposition and positioning, and the reviewers did not feel that this bar was met. (Though the authors' updates during the rebuttal period did help substantially with clarity and relationship to other methods -- thank you for these!)

Recommended decision: while the paper makes a worthwhile contribution, it does not in its current form rise to the level of novelty and general interest that is needed for publication in ICLR.
Decision: Reject