Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary：
In this paper, the authors intend to propose an efficient automated reinforcement learning (RL) framework. To achieve this goal, they integrate three technologies, i.e., evolutionary RL for hyperparameter search, evolvable neural network for policy network design, and shared experience replay for improving data usage. The paper uses a case study on MuJoCo to demonstrate the claimed advantages over baselines.

Some pros:
1. The motivation is good. The automation of reinforcement learning is beneficial to the research community, especially for researchers who are not familiar with RL but in need of it.
2. The framework is general and friendly to users. This framework is general and can be regarded as a plug-in module for a series of reinforcement learning methods. Besides, as we all know, both of the autoML and RL have a heavy computational burden, the adaption of evolvable neural network and shared experience replay greatly alleviate this dilemma. 
3. The organization of this paper is easy to follow. We can follow the authors from why they want to deal with the problem, to how they are inspired by existing work, and then to how the algorithms are design based on the questions to be answered and the existing technologies. 

Some cons:
1. The experiment is far from enough. Actually, this paper only has a case study. First, the author claims that the framework can optimize arbitrary off-policy RL algorithms, why only try on TD3? From the perspective of robustness, the authors need to compare more off-policy algorithms with and without the proposed method. Second, the paper claims there is no directly comparable approach for efficient AutoRL, which I do not agree with. In its own related work, many AutoRL baselines are listed, e.g., H. L. Chiang et al.  'Learning navigation behaviors end-to-end with autorl', F. Runge et al. 'Learning to design RNA'. For these baselines, they can either take the same exploration steps and compare performance with the proposed method, or compare the performance/computational cost when reaching the same performance. Anyway, the readers expect to see more comparisons with more baselines from more perspectives. Third, the authors claim that to tackle the non-stationarity of the RL problem, existing studies can substantially increase the number of environment interactions, implying the proposed framework has advantages on non-stationarity RL environments, but still, no experimental results are given.
2. Some technical details are missing. The logic is clear, the solution is reasonable, but the details are ignored. It is a good idea to keep the writing compact, but the lack of details may harm the readability of the paper. For example, since this is a general framework, how should we design hyperparameter settings of SEARL in initialization for different algorithms? In the training part, why should individual be trained for as many steps as frames have been generated in the evaluation phase, and why the training time could be reduced by using only a fraction j of the steps? 
review_title: A good paper with some flaws

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: This paper propose a population-based AutoRL framework for hyperparameter optimization of off-policy RL algorithms. The framework optimizes both the hyperparameters and also the neural architecture in a one-shot manner, e.g., search and train at the same time. A shared experience replay buffer is used across the population, which as demonstrated in the experiments, substantially increase the sample efficiency compared to PBT and random search. 

Strengths:
- The idea is simple and intuitively makes sense. By sharing the experiences across the population each experience sample gets re-used more often during the training hence the increase in sample efficiency.

Weakness:
- Only apply to off-policy RL. 
- Random search is not a very compelling baseline. PBT is not optimized for sample efficiency. It seems that the shared replay buffer can also be applied to PBT? What about differentiable HPO methods? Those are often shown to be much more sample efficient compared to evolution based approaches. 
- There is little discussions on what the search results look like, e.g., the learning rate schedule or the neural architecture found by the search method. How are they compared to the SOTA learning rate schedule or neural architecture? Or maybe not even SOTA just compare to some simple heuristics that gradually decrease the learning rate or increase the network size. It would be of limited value if the use of AutoRL can only achieve marginally better performance compared to those simple heuristics. 
review_title: Official Blind Review

final_score: rating: 7: Good paper, accept
review_content: Motivated by the sensitivity of RL algorithms to the choice of hyperparameters and the data efficiency issue in training RL agents, the authors propose a population-based automated RL framework which can be applied to any off-policy RL algorithms. In the framework, they optimise hyperparameters together with neural architectures. The authors use TD3 on MuJoCo environments as a showcase to demonstrate the advantages of the proposed method. They reduced the number of environment interactions significantly compared to baselines like random search and a modified population-based training algorithm.

Pros:
+ The authors propose an important novel component to PBT-like framework, which is sharing experience among agents in the population. This innovation itself leads to 10x improvement on sample efficiency;
+ Very clear description of the motivations, related work, the details of the training framework, and the experiments. The paper reading has been super pleasant;
+ Results of TD3 on MuJoCo environments, i.e., the ones in Figure 2, are very promising. In the five environments tested, SEARL algorithm shows an order of magnitude improvement on the sample efficiency.

Cons:
- The proposed sharing experience among agents in the population is limited to off-policy RL algorithm, as also noted by the authors;
- There are a few lines of AutoRL and AutoML research are missing in references or in discussions. For example, the architecture search lines of work from Zoph et al. , and for gradient-based meta parameters optimisation like "A Self-Tuning Actor-Critic Algorithm";
- For the main contribution in the paper, shared experience replay within the population, there's a recent work by Schmitt et al. called "Off-Policy Actor-Critic with Shared Experience Replay" which also demonstrates that sharing experience replay in off-policy learning can improve sample efficiency dramatically. Though they are not in the setting of meta learning hyperparameters or architectures, I think it worths discussions;
- From the ablations in Figure 5, it looks like the architecture adaption does not contribute much to the final performance. It might be because MuJoCo environments do not require complex neural architectures. To show that the neural architecture adaptation is a crucial part of the framework (which is a major difference between the proposed method and PBT), the authors might have to move to a complex domain of environments to demonstrate that.
review_title: Review

final_score: rating: 5: Marginally below acceptance threshold
review_content: # Summary

The authors in this paper propose to optimize the hyperparameters and also the neural architecture while simultaneously training the agent. They evaluate the proposed method with TD3 in the MuJoCo benchmark suite. 

Overall, the proposed method is well-motivated and well-written, and they provide enough experiment/implementation details to reproduce the results. More importantly, they provide the source code. 

# Strength

- Tuning architecture and hyperparameters with PTB for off-policy RL has not been studied before. Compared to baselines, the proposed method is much more sample-efficient.
- The evaluation on MuJoCo is comprehensive, especially the ablation study in Section 4.

# Weakness

- They only test on a single benchmark with one method, TD3. MuJoCo is arguably simple. Actually, the visual world in MuJoCo is quite limited, so the encoder of the RL agent does not have to be huge. It could be more convincing if the authors can test on another benchmark, e.g. ProcGen. 
- I think compared to computer vision tasks with huge neural networks, the search space for the architecture of RL models is much smaller, which can be observed in the ablation study. Without architecture adaption, there is no observable difference. It seems that combining the tuning of architecture and hyperparameters is not that useful.
review_title: Experiments are not convincing

Meta-review: This paper tackles a very important topic in deep RL, namely automatic (non-differentiable) hyper-parameter tuning. It does so by combining ideas from genetic algorithms and neural architecture search with shared experience replay in order to obtain the key property of sample efficiency.  The proposed solution is communicated clearly, and the results are compelling (often 10x improvements), as well as qualitatively interesting.

Unfortunately for the authors, their original submission contained only part of the intended results, hence the borderline scores by some reviewers. In the meanwhile, a second suite of experiments have been added, which I think are compelling enough evidence to validate the paper's approach.
Decision: Accept (Poster)