Venue: AutoML 2022 Track Main

Reviews:

final_score: rating: 6: Strong accept, should be highlighted
review_content: summary_of_contributions: This paper introduces BG-PBT, a novel algorithm for self-tuning in reinforcement learning. This algorithm improves PBT methods by adding architecture search using distillation in addition to hyperparameters tuning.
overall_review: This is a very good paper that introduces a novel self-tuning algorithm and reaches state-of-the-art performance in Brax environments. It is well written with an extensive literature review. The algorithm is rigorously benchmarked and analysed.
review_summary: Given the quality of the paper and the performance of the algorithm, this paper should be highlighted.
review_title: null

final_score: rating: 4: Marginally above the acceptance threshold (use sparsely)
review_content: summary_of_contributions: This paper proposes an approach that seeks to augment existing Population-based Training  (PBT) methods by a) improving hyperparameter search by incorporating recent advances in Bayesian Optimization (BO) for high-dimensional mixed-input search spaces, and b) improving neural architecture search (NAS) using generational training with on-policy distillation. This results in a unified approach for simultaneous hyperparameter and neural architecture search.
overall_review: Overall, the paper is well-written and the proposed method is clearly communicated. The method has some novelties (combines in a nontrivial manner things that work well in isolation) which are empirically validated fairly extensively.  The methodologies are highly relevant to the AutoML community. However, it is unclear whether they will have a significant and lasting impact given the lack of principled grounding.
review_summary: To summarize, I believe the positives outweigh the negatives in this contribution, and I would therefore weakly recommend acceptance of this paper. I believe it is highly relevant and will be of great interest to the AutoML community.
review_title: null

final_score: rating: 6: Strong accept, should be highlighted
review_content: summary_of_contributions: N/A for reproducibility reviewers
overall_review: N/A for reproducibility reviewers
review_summary: N/A for reproducibility reviewers
review_title: null

final_score: rating: 2: Reject, not good enough
review_content: summary_of_contributions: The author extends Casmopolitan (Wan et al., 2021) and PB2-Mix (Parker-Holder et al., 2021) by introducing the BG-PBT algorithm to jointly optimize hyperparameters and architecture of the value/policy network. The novelty lies in the explicit consideration of ordinal variables and the architecture search. They empirically evaluate their approach with the use of PPO in continuous control tasks of the BRAX library. They claim to provide the first approach for architecture search in population-based frameworks and that their approach creates an automatic architecture curriculum to improve significantly higher performance.   

overall_review: The paper considers the important area of AutoRL in combination with architecture search and argues that the architecture should be part of the HPO. The authors who the benefits of their approach in experiments on Brax where they outperformed other AutoRL approaches. 

My main criticism is that the paper is not the first work in AutoRL including neural architecture search into population-based optimization. The work “Sample-Efficient Automated Deep Reinforcement Learning” at ICLR 2021 as well as “Neural Architecture Evolution in Deep Reinforcement Learning for Continuous Control” at the Workshop on Meta-Learning, NeurIPS 2019, already jointly optimized hyperparameter and architecture parameter in a population-based fashion similar to PBT but focused on off-policy instead of on-policy RL. Some findings, like the decrease in the learning rate, are similar to the work above. It also does not consider this work as related work.  
Furthermore, it is unclear if it is beneficial to treat ordinal hyperparameters like the batch size as ordinal instead of as categorical parameters in a BO setting. How much does the optimization benefit from this? And how would a random search perform if the search space would include a learning rate schedule instead of keeping it fixed?  

review_summary: I suppose the authors are not aware of the related work mentioned above. However, since it exists, the paper lacks novelty due to existing work on joint hyperparameters and network parameter optimization in a population-based fashion. Additionally, I have doubts about the experiments and the benefit of considering some parameters as ordinal scales.
review_title: null

Meta-review: metareview: I would like to thank all reviewers for their constructive comments, which have improved the quality of the submission! 

In my view, the authors are able to fix the issues raised by WUDH & 98ZS; the mentioned related works do not settle several outstanding questions in AutoRL; some of these are well addressed by the present submission, as pointed out by reviewers xj4C & 98ZS, even if novelty claims need to be reduced.

While WUDH has perhaps most contributed to improving the quality of the manuscript, I believe they were successful in doing so, and hence I cannot support their original evaluation that the relevance of this work to AutoRL is small. 

Since the crucial concerns seem to have been addressed to a satisfactory degree, I hence recommend acceptance.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept