Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes to learn a “meta-policy” that selects the best policy among a set of policies, depending on the state. The idea of the authors is that each sub-policy could have a different structure or be trained with different hyper-parameters: the meta-policy can therefore be viewed as an hyper-parameter selection algorithm (we do not have to select the hyper-parameters, the meta-policy does it). The authors test their idea with the Deep Mind Control Suite and an experiment with the ANYmal simulation.
strengths_and_weaknesses: 
### Strengths
- Interesting idea
- A lot of different experiments are reported

### Weaknesses
- the sub-policies have to be chosen and might not be actually diverse; a simple baseline/variant would have been to use a set of sub-policies with random hyper-parameters (so that we do not have to choose, like in standard hyper-parameter tuning/parameter control algorithms)
- many important parts of the algorithms are unclear to me; in particular, it is never clearly stated that/if the sub-policies are trained simultaneously
- only 4 replicates (!), which is far from being enough to perform any statistics
- no real robotics experiments and most of the experiments (except maybe ANYmal) are actually not close to a robot
- the authors mention related work that does automatic parameter control/tuning, but there is no empirical comparison to this kind of technique
- no statistical analysis (p-value, etc.): on many plots, the variances seem to overlap, which strongly suggest that most differences are not statistically significant

summary_of_recommendation: Overall, this is a promising paper that describes a large amount of work. However, while each paragraph is well written, it found it difficult to follow the main story and the main ideas. 

In addition, I think the paper would be much stronger if the sub-policies were automatically generated (e.g., random hyper-parameters, grid search-like, etc.) instead of hand-picked.

I initially suggested a "weak accept" but I moved to "weak reject" when I finally found in the supplementary material that only 4 seeds were used. This is too little to conclude anything from the empirical comparisons, especially when many results look very similar.
issues: - The results critically depends on the choice of the sub-policies, which are hand-picked.

- Only 4 seeds were used, which is too little to perform any statistics and draw any conclusion in empirical work (most statisticians would say "at least 20, although more is almost always better). First, the number of seeds should be in the caption of each result figures (so that we have the elements to interpret the figures). Second, are the figures representing the mean/standard deviation? mean/standard error? median/95% CI? This should also be in the caption.


- The authors argue that their approach is faster/more data-efficient than alternatives (e.g., grid search); is it because all the competing sub-policies use the same samples? it is never clearly stated that/if all the sub-policies are trained together with the same samples. 

- The number of samples might be lower than learning each sub-policy independently, but all the rest of the computation (e.g., activating the networks) is the same as learning each of them independently; this should at least be discussed. 

- How is 4.1 different from the ref. 44 and from any other actor-critic algorithm? what is specific to the hierarchical nature of the problem here? The authors should highlight the critical novelties/changes here.

### Other issues
- The experiments on architecture variations (section 5.3, figure 7b) are probably not statistically significant (hard to say since the authors do not perform any statistical analysis). Please remove the claim that the proposed method is “slightly better” or provide some p-values.
- There is an important literature about “quality diversity” that aims at finding automatically a diverse set of policies, which seems to be very relevant for this paper (but overlooked by the authors). For instance, check:
  * Duarte M, Gomes J, Oliveira SM, Christensen AL. Evolution of repertoire-based control for robots with complex locomotor systems. IEEE Transactions on Evolutionary Computation. 2017 Jun 30;22(2):314-28.
   * Nilsson O, Cully A. Policy gradient assisted MAP-Elites. Proc. of GECCO 2021.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a robust hyperparameter selection method that learns diverse sub-policies with various distribution types and parameterization. The authors use relative policy optimization [44] for the policy improvement step with modifications that use a sum of constraints consisting of decoupled components, each only varying along its respective parameter. The authors also use the piece-wise constant pseudo-density during policy evaluation that combines continuous action distribution and discrete action distribution. For empirical analysis, the authors evaluated the method on deepmind DM control tasks as well as OpenAI gym tasks comparing to prior gradient-free optimization approaches that showed that the method can achieve robust and sample-efficient learning.
strengths_and_weaknesses: The paper addresses an important problem in robot learning, which is the hyperparameter optimization issue. The proposed framework is neat since it can automatically train multiple sub-policies with different configurations in a hierarchical way, which is novel. The algorithm is simple yet effective in terms of the experimental results in simulated locomotion tasks. The paper is also well written and easy to understand.

However, I do have a few concerns, which I will discuss as follows.

First, the applicability of the proposed method seems a bit limited since the authors only consider different distribution types and parameterization of the policy in RL for hyperparameter optimization, which only covers a very small portion of the hyperparameters that matter in RL. The experimental results also only focus on simulated locomotion tasks, which seem not comprehensive. I think it would be important to extend the method to handle a wider range of hyperparameters in RL, e.g. learning rate, architecture design, etc. It is also pivotal to test the method on robotic manipulation tasks and also real-world tasks, which are more realistic for robot learning.

Moreover, the algorithm appears a bit incremental, it seems to be almost a direct adaptation of the prior work [44] to the setting mixture of policies. The theoretical insight of such a direct adaptation is also missing.

================================

After reading the rebuttal, I appreciate that the authors conduct additional more realistic robotic experiments and also clarified my concerns on novelty of the paper. However, my concern on automatically selecting other hyperparameters still remains. The authors claim that their approach can automatically select hyperparameters and only show the use case for the distribution types. I think such a claim cannot be fully substantiated in the paper, therefore limiting the contribution of the paper. I will keep my score.

===============================

More updates: thank you for pointing me to the additional experiments on randomly sampled hyperparameters! I've increased my score to a 6. Though I still have a few questions as follows. It seems that NKCD does not outperform RHPO and is even a bit worse in several domains. It is not also clear to me if RHPO is a reasonable baseline for varying other hyperparameters. The authors should perhaps compare to the oracle, which is DDPG tuned with each of the varying hyperparameter. 
summary_of_recommendation: Given my two points on weaknesses of the proposed method discussed in the Strengths And Weaknesses section, I would vote for a weak reject.
issues: 1. Extend the method to handle more diverse hyperparameters in RL such as critic and policy learning rates, architecture choices etc.
2. Perform evaluations on robotic manipulation tasks and also real-world tasks.
3. Provide some theoretical insight of the method.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The manuscript is proposing a method for hierarchical RL using a diverse set of sub-policies and training a mixture of them. The main idea is modeling a parametric policy as a mixture of sub-policies and simultaneously learning a non-parametric policy and fitting the parametric policy to the non-parametric one. The algorithm is later evaluated on various learning to control tasks.
strengths_and_weaknesses: The consideration of the role of hyper-parameters and the hierarchical nature of the problem is appreciated. It is intuitive and fits the final application of RL for robotics better.

The proposed method is intuitive and mathematically sensible.

The results are very promising.

The major issue for me is the lack of proper coverage of related work. The paper clearly fits into the area of hierarchical RL as well as RL with options framework. However, the authors ignore this literature completely in their discussion of related work and empirical study.

The experimental study is not explained in detail. After reading the paper a few times, I am still confused about how many random trials are performed and what are the shaded areas in the figures. The baselines are also not justified. For example, the authors do not discuss why did they choose DDPG?
summary_of_recommendation: The method and the paper is sensible; however, I need a better justification for empirical study as well as an extended review of the relevant literature to recommend the paper for publication.

----

I thank the authors for their clarifications. It partially answers my concerns but the major issues remain. Hence, I will not change my score
issues: Related work covering Hierarchical RL and RL with options.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This works aims to combat the need for extensive hyperparameter tuning by proposing a method that can utilize a single trial to combine different control designs conditional on the task. This reduces impact of hyperparameters and design decisions and shows more robustness on different simulated tasks. 

The key contribution of this work seems to be reformulating the policy to be a mixture of experts style policy and then integrating this into the framework of Relative entropy regularized policy iteration

The key contributions of this work seem to be: 

1. Introducing a mixture policy distribution where each component has different parameterization and potentially hyperparameters. 
2. Modifying the KL regularized PI framework to add in the mixture policy. 
3. Enabling optimization of mixed continuous and discrete distributions by introducing a pseudo density function. 
4. Separating the constraints in KL regularized PI such that there is a separate constraint and corresponding lagrange multipliers per constraint. 

There is a detailed empirical analysis combining different components and noting how the combination does better than individual pieces. 
strengths_and_weaknesses: Strengths: 
1. The approach seems simple
2. It is nice to be able to reuse data across multiple hyperparameters and architectures for controllers, especially very heterogeneous controller modes. 
3. It is interesting that different controllers specialize to different parts of a trajectory. 
4. The range of empirical results on actually trying different hyperparameters is a plus

Weaknesses: 
1. Limited robotics experiments, results and discussion. 
2. The improvement on the domains is somewhat marginal, especially over RHPO. 
3. The figures and results are somewhat hard to parse without better signposting and labeling. 

summary_of_recommendation: Overall I think the paper has an interesting premise, but I am recommending for a weak reject because of the results not being very much better than the baselines in many domains, a lack of robotics results and not a very clear novel contribution over prior work. I think a combination of multiple of these factors makes me recommend a weak rejection, but I am willing to reconsider given more experiments across other domains showing bigger improvements, significantly more clarity in the writing or more discussion or presentation of robotics results. 
issues: The first and most glaring issue seems to be that this paper is entirely in simulation, with very little robotics. There is one domain which is the ANYmal robotics, but the rest is just standard mujoco tasks. This feels a bit underwhelming for a robotics conference, especially when the ANYmal task is just PD tracking. 

In the introduction, the words “synergy” is used excessively and is not adding much value. The overall introduction is a bit too vague right now to add much value on what is actually happening, would correct this. 

The paper describes the following phenomenon often: “This enables robust adaptation to a broad range of motion planning problems while reducing the necessity of manual parameter tuning and sequential experiment design” -> why is this?? I don’t see the clear reason why training a number of different models together makes this any  more robust? Could this be expanded more clearly? I’m assuming that it’s because multiple different hyperparameter models can be trained using the same data simultaneously and the best model is chosen automatically, but this could be spelled out a bit more for the reader. 

Overall, the presentation of the paper makes it very hard to distinguish what is novel and introduced in this paper versus just adopted from past papers. Now it is perfectly ok to take a lot from past papers, it just needs to be made more clear in Section 4, and the key contributions need to be highlighted. 

I think the methods section would also do well from a further discussion on why this framework helps with robustness and hyperparameter selections and such. 

It’s also not clear why this objective would yield to actual specialization and not just collapse like many mixture of experts frameworks have. Can this be elaborated on in the paper?

Also a discussion on how to pick the different components would be very useful, currently that whole thing is left in the air. Also are the components trained with different hyperparameters?

It seems like the actual improvement in Fig 4 is quite marginal besides in the ANYMal domain. Can other domains be presented where this makes more of a difference? There is currently only one datapointn that this actually matters. 

For the robustness plots, I think Fig 5 is actually a bit misleading. On closer inspection I can see that red is better than green on the quadruped, but the overall delta seems quite small and I was originally looking at just the error bars. I think more signposting should be done to indicate what to look for in the figures rather than “A is better than B”. 

I think the comparisons against HOOF are a bit moot because the base RL algorithm is quite different. Can HOOF be combined with the same base RL algorithm here for Fig 6?

What about the robustness to hyperaparameters of the whole learning procedure and not the individual components??

Can Fig 7 be labeled with what each different line means properly? Which is the homogenous, heterogenous and the individual components?

I think Fig 7 and 8 have pretty good analysis but on individual components at a time. What happens when multiple are varied together? Also the difference in performance is not really going between working and not working altogether, but just small differences in return. Are there other domains where these robustness benefits are highlighted more clearly.
review_title: null

Meta-review: After the discussion and rebuttal from the authors providing additional evidence for their method, some of the reviewers updated their scores to accept the paper. The remaining points of criticism are mostly regarding the related work section, which can be fixed in the final version of the paper. I encourage the authors to carefully read the reviews and address the points raised by the reviewers in the final version of the paper.
Decision: Accept (Poster)