Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: After rebuttal:

Thank you to the authors for responding to my review.

1) The title of the conference is "... on Learning Representations". As I stated in the review ("no, e.g., neural networks are employed"), neural networks are an *example* of, but do not subsume, all representation learning methods. Therefore, I agree that papers that do not cover neural networks are welcome at the conference. However, as I stated in the review, my evaluation of the method proposed in the submission is that it does not concern representation learning ("The employed features in Table 3 are handcrafted"). I believe this evaluation is defensible, but of course the final evaluation is up to the chairs. However, I note that the authors did not respond directly to my evaluation that the method is not engaging in representation learning.

2-7) As the other reviewer notes, the paper lacks clarity in many places, and does not sufficiently discuss prior work, including in postural control (there is one citation in the references that is not mentioned in the main text), hierarchical Bayesian optimization within or without a Gaussian processes framework (https://scholar.google.com/scholar?hl=fr&as_sdt=0%2C5&q=hierarchical+bayesian+optimization&btnG=), or experience replay (https://scholar.google.com/scholar?hl=fr&as_sdt=0%2C5&q=replay+machine+learning&btnG=). Therefore, it is difficult to ascertain the research contribution.

As such, I stand by my evaluation that this submission is not ready for publication at ICLR.

===========================

Before rebuttal:

The submission presents a hierarchical Bayesian optimization (HiBO) approach to solving a postural control task expressed as a proportional-derivative (PD) controller.

Strengths:
- The HiBO approach outperforms the non-hierarchical BO approach on the task of postural control.

Weaknesses:
- The paper does not make use of representation learning (no, e.g., neural networks are employed) and is therefore out-of-place at ICLR. The employed features in Table 3 are handcrafted.
- The task (simulating human postural control) is not well-situated in the context of prior work using HiBO for robotics, so the contribution remains unclear.
- It is not clear why this problem should be formulated as contextual policy search (i.e., to what the context variable refers).
- Only one baseline (Bayesian optimization (BO)) is reported. This baseline corresponds to the ablation of the HiBO method (i.e., the omission of the context variable), and so does not represent, more broadly, an alternative approach.
- The concept of "mental replay" is briefly introduced, but no reference is made to prior work in imagined rollouts, and no ablation study on the impact of this component is performed.

Minor points:
- It is unclear why the problem setting should be labeled as "psychological" postural control.
- There are several missing references ("?") in the text.
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: How to quickly learn control policies with minimized number of environment interactions have long been an important problem. To tackle this problem, this paper proposed a "hierarchical Bayesian optimization (HIBO)" algorithm to optimize the "feature parameter \phi" (which I don't know what that is) and the "policy parameter \theta" hierarchically. Under the formulation of maximizing reward J(\theta), the algorithm firstly uses EI to select \phi. Given the selected \phi, the algorithm selects the policy parameter \theta. The proposed algorithm is evaluated on a Humanoid Postural Balancing task, and shows achieves high rewards faster than the standard EI acquisition. However, the paper is awfully written such that I cannot understand what the "feature parameter \phi" is. Given my limited understanding, I think the paper should be rejected.

Strengths,
1, The paper deals with an interesting task: Humanoid Postural Balancing. A Humanoid is expected to learn how to balance as quick as possible to reduce the interactions with the environments, which suits well with Bayesian optimization.

Weakness,
1, The paper is awfully written. The problem statement subsection is unreadable. I don't see anywhere explaining how the states x_t, commands u_t, \theta, and feature \phi are related? What is \phi ? It is super wired why the feature parameter \phi is jointly maximized with the policy parameter.  Because I don't understand the formulation, I can hardly understand anything else.
2, From my very limited understanding on the formulation, the proposed HIBO is trivial.
3, The experiments are limited. The paper only conducts one experiment on the Humanoid control balancing. And they paper only compares with the EI acquisition, while the state-of-art acquisition MES should be also be compared with.
4, The proposed mental replay is not well justified, qualitatively or empirically.
review_title: Official Blind Review #1

Meta-review: The paper proposes hierarchical Bayesian optimization (HiBO) for learning control policies from a small number of environment interaction and applies it to the postural control of a humanoid. Both reviewers raised issues with the clarity of presentation, as well as contribution and overall fit to this venue. The authors’ response helped to clarify these issues only marginally. Therefore, primarily due to lack of clarity, I recommend rejecting this paper, but encourage the authors to improve the presentation as per the reviewers’ suggestions and resubmitting.
Decision: Reject