Venue: AutoML 2022 Track Main

Reviews:

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: The paper introduces the DARTS neural architecture search algorithm for the policies of reinforcement learning (RL) agents. In their experiments, the combination of the RL loss function with the updating of the architecture is explored. Second, the problems that can arise due to a possibly diverging RL agent. Finally, the discretization of the modules can lead to failure modes. The combination of DARTS and RL shows viable with several promising results.
overall_review: The authors provide a large number of experiments exploring how DARTS works in the reinforcement learning framework. The experiments provide the reader with a good overview of the success and failure cases of the method. The authors identify the two important differences in the optimization of the architecture between supervised en reinforcement learning. Additionally, the supplementary material contains an extensive review of the choices for the search space design. 

Moreover, the RL-DARTS method finds well-performing architectures improving over existing policy architectures and the random search baseline. Finally, it would be interesting to extend the random search baseline by comparing it with a more competitive version of random search. 

review_summary: The existing work provides an extensive analysis of DARTS for the architecture search for reinforcement learning policies. My only recommendation would be to extend the Random Search baseline with a more competitive version.
review_title: null

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: In this work, the authors extend the idea of Differentiable Architecture Search (DARTS) for the Reinforcement Learning domain and verify the viability of such an approach in the case of designing better neural network architecture for RL algorithms. They experimented with both discrete and continuous action space environments using the Procgen benchmark and the DeepMind Control Suite. They claim to achieve up to 250% performance improvement in some cases at the cost of 3 times greater computation time. They also performed ablation studies to answer a few insightful questions. 
overall_review: For years, RL models have been mainly hand-designed and tailored to particular environments. This work presents an approach to learn the model architecture via gradient-based Neural Architecture Search and contributes to the general AutoRL research. 

The paper clearly identified the key differences between Supervised Learning (SL) and RL. Then they attempted to answer whether these differences restrict the applicability of the DARTS to RL and answered negatively based on their experimental results. However, their novelty lies only in establishing the connection between RL and one existing DARTS method for SL by proposing some adaptations. I would like to see some effective modification of DARTS beyond the generic findings that gradient-based NAS can be applied to RL.  

In Figures 3 and 4, I would like to see the plots for at least 8 environments out of 16 environments from Procgen as opposed to only 4. Otherwise, the performance improvement is not motivating enough based on the limited numeric improvement presented in Table 1 and Table 5. In line 8, it is mentioned that up to 250% performance improvement can be achieved while line 65 suggests 150% performance gain. Thus, it causes confusion about the actual performance gain. Also, comparisons with other hand-designed modifications of IMPALA-CNN architecture such as [1] may be included to show the effectiveness of the approach. 

The ablation studies are well motivated and provide meaningful insights regarding the presented approach. The architectures discovered in different stages of training justify that actual learning is happening at the architecture level. 

[1] Nafi, N. M., Glasscock, C., & Hsu, W. (2021, October). Attention-based Partial Decoupling of Policy and Value for Generalization in Reinforcement Learning. In Deep RL Workshop NeurIPS 2021.
review_summary: The work deals with a crucial issue - the automated design of network architectures for RL. They empirically demonstrate that DARTS can be used to design a competitive RL architecture. This is an important finding that the community should know. Thus, I am voting for acceptance. However, there is still room for improvement in terms of the approach's efficacy across different environments.
review_title: null

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: This paper investigates how to apply DARTS, a differentiable neural archirecture search method that previously has only been used in supervised learning, to reinforcement learning. 
The authors argue that this is not a trivial application of DARTS to a new domain, and propose several key challenges due to the different optimization paradigm of RL compared to SL, including (1) nonstationary data collection process dependent on the policy itself, and (2) mismatch between the RL objective (also the objective for DARTS update) and the true goal of maximizing return in RL. 
The authors propose a slightly modified version of DARTS, which jointly optimizes both the model parameters and supernet weights. They compare to different baseline methods on different benchmarks, which illustrates that DARTS can work in RL domain. 
overall_review: Pos: 
1. This paper provides a first investigation on how DARTS applies to NAS to RL, which could be valuable for future work in (differentiable) NAS for RL. 
2. The paper includes detailed experiments to check different aspects of DARTS during the learning process, providing both quantitative and qualitative results which proves that DARTS indeed learns something meaningful in RL domain. 

Neg: 
1. As mentioned in the "Technical quality" part, I think the key challenges of applying DARTS to RL as mentioned by the authors in Section 3 is not well studied in the experimental section. I think a deeper look at these could significantly improves the quality of this paper, instead of just proving a fact that DARTS can work in RL domain. 
2. I don't see a clear advantage of RL-DARTS compared to the baselines w.r.t. the average performance, and the performance gap variance seems to be large across seeds and different tasks. It will be better if the paper can check the stability of RL-DARTS learning process and provide some insights on why this happens and how to improve it. 
review_summary: This paper takes a first step of applying DARTS to RL with large-scale model size, which is a good contribution to NAS in RL. The methodology novelty is limited, but the experimental evaluation is convincing for the fact that DARTS can indeed work in the RL domain. 

Novelty and contribution of the paper can be further improved if more insights can be provided on why DARTS is easy / hard to work on RL supported by carefully designed experiments.
review_title: null

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: The paper studies whether and to what extent the Differentiable Architecture Search (DARTS) is applicable to Reinforcement Learning (RL). They build a network called RL-DARTS, which applies the original DARTS to search architectures for RL algorithms such as PPO, Reinbow, SAC. For the experiments, they use Procgen environment for discrete action spaces and DM-control benchmark for continuous action space study. For Procgen benchmark, they performed both multi-game search and single-game search with PPO. In the multi-game search, they train and supernet across 16 games and achieved up to 250% performance over the IMPALA-CNN baseline, and in the single-game search, their algorithm can find game-specific architectures from scratch which outperforms IMPALA-CNN and random search. They have also performed experiments on training the supernet end-to-end with minimal hyperparameter tuning, which costs only 3 times more compute time to obtain extensive efficiency. Finally, they demonstrate that the discrete cell can also be improved on both quantitative and quantitative aspects with the default discretization, and they claim that the supernet and discrete cell rewards are correlated according to their experiments. In conclusion, they claim that jointly optimize both \theta and \alpha can provide concrete evidence that DARTS can be conducted efficiently in RL.

overall_review: Positive:
- Well-written and well structured.
- The Methodology is well explained.
- The experiments are well conducted to answer the authors' research questions.

Negative:
- A minor miss explanation is that SL states Supervised Learning, which might cause some confusion to people who are not familiar with it.
- The comparison of this work with the previous NAS for RL studies is limited and not very clear to me.


review_summary: In general, I think it is a well-written paper with comprehensive comparison experiments. According to their results, DARTS can indeed be applied to RL with some extra (3 times) computing time to obtain better performance. Although, in my opinion, the novelty and the performance improvement of the idea is not very significant, I think it is acceptable.
review_title: null

final_score: rating: 5: Accept, good paper
review_content: summary_of_contributions: This paper explores the use of Neural Architecture Search, more specifically the popular DARTS algorithm, for reinforcement learning.
More precisely, this paper conducts a thorough analysis of the performance and the behavior of the DARTS algorithm at the cell level when training in the standard ProcGen environment.
overall_review: This paper is well written and tackles the interesting question of using NAS approaches (DARTS in particular) for RL.
The analysis is well conducted and each claim is supported by multiple experiments and empirical results.
However, some results are still quite weak in the sense that the performance of the DARTS algorithm does not appear significantly better than standard RL approaches on most environments while the algorithm is reported to need 3 times more computation time.

This paper claims to be the first one to try to apply DARTS in the RL setting, and although I'm not specifically familiar with the DARTS literature, I did not find any counter-examples.
Moreover, the paper only focuses on the basic DARTS algorithm and not on all improvements proposed since, so I think the results are easily good enough to show that NAS approaches are relevant for RL.
review_summary: This paper seems to be the first one to apply DARTS to RL and to show that this is an interesting approach.
The results are not really better than standard RL approaches, but the discussion is well conducted and this work acts more as a proof of concept.
In consequence I would recommend to accept this paper.
review_title: null

Meta-review: metareview: All reviewers agree that this paper is relevant for the AutoML community and addresses a potentially significant open research problem. The experimental evaluation convinced the reviewers that DARTS can be applied to RL (with some additional experimental results added after the reviews). 

Pro: 
- This paper has high significance to the AutoML community, showing how to effectively use DARTS for RL.
- The paper includes potential failure cases and a discussion of future work, which could be a good starting point for follow-up work. The experiments / method itself can be a good benchmark / baseline for future work on DARTS for RL (especially given that code was provided and will be open-sourced).
- Their implementation can match or outperform the hand-designed IMPALA-CNN baseline.
- I was happy to see that the authors adequately addressed the reviewer's feedback and updated the paper accordingly.

Room for improvement: 
- More analysis on the generalisability of the learned architecture for more environments (some have been added in the rebuttal), or the efficacy of the approach across different domains 
- The paper would benefit from more insights into _why_ DARTS works (or not) in RL.
- There might be more effective ways to modify DARTS and make it work for RL.

Overall I get the sense that the empirical evaluation is extensive and convincing, and that it is fair to leave the rest for future work. I therefore recommend acceptance. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept