Venue: ICLR 2017 conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Thank you for an interesting read on an approach to choose computational models based on kind of examples given.

Pros
- As an idea, using a meta controller to decide the computational model and the number of steps to reach the conclusion is keeping in line with solving an important practical issue of increased computational times of a simple example. 

- The approach seems similar to an ensemble learning construct. But instead of random experts and a fixed computational complexity during testing time the architecture is designed to estimate hyper-parameters like number of ponder steps which gives this approach a distinct advantage.


Cons
- Even though the metacontroller is designed to choose the best amongst the given experts, its complete capability has not been explored yet. It would be interesting to see the architecture handle more than 2 experts.
review_title: null

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: A well written paper and an interesting construction - I thoroughly enjoyed reading it. 

I found the formalism a bit hard to follow without specific examples- that is, it wasn't clear to me at first what the specific components in figure 1A were. What constitutes the controller, a control, the optimizer, what was being optimized, etc., in specific cases. Algorithm boxes may have been helpful, especially in the case of your experiments. A description of existing models that fall under your conceptual framework might help as well.

In Practical Bayesian Optimization of Machine Learning Algorithms, Snoek, Larochelle and Adams propose optimizing with respect to expected improvement per second to balance computation cost and performance loss. It might be interesting to see how this falls into your framework.

Experimental results were presented clearly and well illustrated the usefulness of the metacontroller. I'm curious to see the results of using more metaexperts.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: This paper introduces an approach to reinforcement learning and control wherein, rather than training a single controller to perform a task, a metacontroller with access to a base-level controller and a number of accessory « experts » is utilized. The job of the metacontroller is to decide how many times to call the controller and the experts, and which expert to invoke at which iteration. (The controller is a bit special in that in addition to being provided the current state, it is given a summary of the history of previous calls to itself and previous experts.) The sequence of controls and expert advice is embedded into a fixed-size vector through an LSTM. The method is tested on an N-body  control task, where it is shown that there are benefits to multiple iterations (« pondering ») even for simple experts, and that the metacontroller can deliver accuracy and computational cost benefits over fixed-iteration controls.

The paper is in general well written, and reasonably easy to follow. As the authors note, the topic of metareasoning has been studied to some extent in AI, but its use as a differentiable and fully trainable component within an RL system appears new. At this stage, it is difficult to evaluate the impact of this kind of approach: the overall model architecture is intriguing and probably merits publication, but whether and how this will scale to other domains remains the subject of future work. The experimental validation is interesting and well carried out, but remains of limited scope. Moreover, given such a complex architecture, there should be a discussion of the training difficulties and convergence issues, if any.

Here are a few specific comments, questions and suggestions:

1) in Figure 1A, the meaning of the graphical language should be explained. For instance, there are arrows of different thickness and line style — do these mean different things? 

2) in Figure 3, the caption should better explain the contents of the figure. For example, what do the colours of the different lines refer to? Also, in the top row, there are dots and error bars that are given, but this is explained only in the « bottom row » part. This makes understanding this figure difficult.

3) in Figure 4, the shaded area represents a 95% confidence interval on the regression line; in addition, it would be helpful to give a standard error on the regression slope (to verify that it excludes zero, i.e. the slope is significant), as well as a fraction of explained variance (R^2). 

4) in Figure 5, the fraction of samples using the MLP expert does not appear to decrease monotonically with the increasing cost of the MLP expert (i.e. the bottom left part of the right plot, with a few red-shaded boxes). Why is that? Is there lots of variance in these fractions from experiment to experiment?

5) the supplementary materials are very helpful. Thank you for all these details.

review_title: Pondering on expert advice for control

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Pros (quality, clarity, originality, significance:):

This paper presents a novel metacontroller optimization system that learns the best action for a one-shot learning task, but as a framework has the potential for wider application. The metacontroller is a model-free reinforcement learning agent that selects how many optimization iterations and what function or “expert” to consult from a fixed set (such as an action-value or state transition function). Experimental results are presented from simulation experiments where a spacecraft must fire its thruster once to reach a target location, in the presence of between 1 and 5 heavy bodies.

The metacontroller system has a similar performance loss on the one-shot learning task as an iterative (standard) optimization procedure. However, by taking into account the computational complexity of running a classical, iterative optimization procedure as a second “resource loss” term, the metacontroller is shown to be more efficient. Moreover, the metacontroller agent successfully selects the optimal expert to consult, rather than relying on an informed choice by a domain-expert model designer. The experimental performance is a contribution that merits publication, and it also exhibits the use of an interaction network for learning the dynamics of a simulated physical system. The dataset that has been developed for this task also has the potential to act as a new baseline for future work on one-shot physical control systems. The dataset constitutes an ancillary contribution which could positively impact future research in this area.

Cons:
It's not clear how this approach could be applied more broadly to other types of optimization. Moreover, the REINFORCE gradient estimation method is known to suffer from very high variance, yielding poor estimates. I'm curious what methods were used to ameliorate these problems and if any other performance tricks were necessary to train well. Content of this type this could form a useful additional appendix.

A few critiques on the communication of results:

- The formal explication of the paper’s content is clear, but Fig.’s 1A and 3 could be improved. Fig. 1A is missing a clear visual demarcation of what exactly the metacontroller agent is. Have you considered a plate or bounding box around the corresponding components? This would likely speed the uptake of the formal description.

- Fig. 3 is generally clear, but the lack of x-axis tick marks on any subplots makes it more challenging than necessary to compare among the experts. Also, the overlap among the points and confidence intervals in the upper-left subplot interferes with the quantitative meaning of those symbols. Perhaps thinner bars of different colors would help here. Moreover, this figure lacks a legend and so the different lines are impossible to compare with each other.

- Lastly, the second sentence in Appendix B. 2 is a typo and terminates without completion.
review_title: Using metacontroller optimization produces more efficient learning on one-shot control task

Meta-review: This paper fairly clearly presents a totally sensible idea. The details of the method presented in this paper are clearly preliminary, but is enough to illustrate a novel approach.
Decision: Accept (Poster)