Title: Metacontrol for Adaptive Imagination-Based Optimization
Authors: Jessica B. Hamrick
Andrew J. Ballard
Razvan Pascanu
Oriol Vinyals
Nicolas Heess
Peter W. Battaglia
Venue: ICLR 2017
Keywords: Deep learning
Reinforcement Learning
Optimization

Abstract: Many machine learning systems are built to solve the hardest examples of a particular task, which often makes them large and expensive to run---especially with respect to the easier examples, which might require much less computation. For an agent with a limited computational budget, this "one-size-fits-all" approach may result in the agent wasting valuable computation on easy examples, while not spending enough on hard examples. Rather than learning a single, fixed policy for solving all instances of a task, we introduce a metacontroller which learns to optimize a sequence of "imagined" internal simulations over predictive models of the world in order to construct a more informed, and more economical, solution. The metacontroller component is a model-free reinforcement learning agent, which decides both how many iterations of the optimization procedure to run, as well as which model to consult on each iteration. The models (which we call "experts") can be state transition models, action-value functions, or any other mechanism that provides information useful for solving the task, and can be learned on-policy or off-policy in parallel with the metacontroller. When the metacontroller, controller, and experts were trained with "interaction networks" (Battaglia et al., 2016) as expert models, our approach was able to solve a challenging decision-making problem under complex non-linear dynamics. The metacontroller learned to adapt the amount of computation it performed to the difficulty of the task, and learned how to choose which experts to consult by factoring in both their reliability and individual computational resource costs. This allowed the metacontroller to achieve a lower overall cost (task loss plus computational cost) than more traditional fixed policy approaches. These results demonstrate that our approach is a powerful framework for using rich forward models for efficient model-based reinforcement learning.

TL;DR: We present a "metacontroller" neural architecture which can adaptively decide how long to run an model-based online optimization procedure for, and which models to use during the optimization.

Review discussion:
Thank you for an interesting read on an approach to choose computational models based on kind of examples given.
Pros
- As an idea, using a meta controller to decide the computational model and the number of steps to reach the conclusion is keeping in line with solving an important practical issue of increased computational times of a simple example.
- The approach seems similar to an ensemble learning construct.
But instead of random experts and a fixed computational complexity during testing time the architecture is designed to estimate hyper-parameters like number of ponder steps which gives this approach a distinct advantage.
Cons
- Even though the metacontroller is designed to choose the best amongst the given experts, its complete capability has not been explored yet. It would be interesting to see the architecture handle more than 2 experts.
We thank the reviewer for the comments and are glad that they enjoyed reading the paper! We agree this approach bears some similarities to ensemble learning methods.
However, we also note ours is quite distinct from typical ensemble learning approaches in that we use a different expert depending on the input, rather than aggregating or selecting from the results of multiple ensemble models.
We also agree that it would be interesting to see how our approach handles more than two experts.
Indeed, one can imagine many potential experts that might be available for a given problem beyond the MLP and interaction net experts used in the paper: for example, a “coarse” simulation expert (which uses an accurate dynamics model but which also uses a larger step size for time integration, ɛ, than the true simulation), a vanilla LSTM expert (rather than an interaction network), an “episodic” expert (which has an episodic memory to remember previous actions it has tried in the world), etc. Having more than just one or two experts might be particularly useful when those experts specialize in predictions of different parts of the state space, or at different levels of specificity.
For example, the metacontroller could potentially learn to execute a “coarse-to-fine” search by utilizing a number of simulation experts with different granularities of time integration, where the “coarser” experts have lower ponder costs. Exploring what policies the metacontroller learns when it has access to more diverse experts such as these is an important future direction.
This paper introduces an approach to reinforcement learning and control wherein, rather than training a single controller to perform a task, a metacontroller with access to a base-level controller and a number of accessory « experts » is utilized. The job of the metacontroller is to decide how many times to call the controller and the experts, and which expert to invoke at which iteration. (The controller is a bit special in that in addition to being provided the current state, it is given a summary of the history of previous calls to itself and previous experts.) The sequence of controls and expert advice is embedded into a fixed-size vector through an LSTM. The method is tested on an N-body  control task, where it is shown that there are benefits to multiple iterations (« pondering ») even for simple experts, and that the metacontroller can deliver accuracy and computational cost benefits over fixed-iteration controls.
The paper is in general well written, and reasonably easy to follow. As the authors note, the topic of metareasoning has been studied to some extent in AI, but its use as a differentiable and fully trainable component within an RL system appears new. At this stage, it is difficult to evaluate the impact of this kind of approach: the overall model architecture is intriguing and probably merits publication, but whether and how this will scale to other domains remains the subject of future work. The experimental validation is interesting and well carried out, but remains of limited scope.
Moreover, given such a complex architecture, there should be a discussion of the training difficulties and convergence issues, if any.
Here are a few specific comments, questions and suggestions:
1) in Figure 1A, the meaning of the graphical language should be explained.
For instance, there are arrows of different thickness and line style — do these mean different things?
2) in Figure 3, the caption should better explain the contents of the figure.
For example, what do the colours of the different lines refer to?
Also, in the top row, there are dots and error bars that are given, but this is explained only in the « bottom row » part. This makes understanding this figure difficult.
3) in Figure 4, the shaded area represents a 95% confidence interval on the regression line;
in addition, it would be helpful to give a standard error on the regression slope (to verify that it excludes zero, i.e. the slope is significant), as well as a fraction of explained variance (R^2).
4) in Figure 5, the fraction of samples using the MLP expert does not appear to decrease monotonically with the increasing cost of the MLP expert (i.e. the bottom left part of the right plot, with a few red-shaded boxes). Why is that? Is there lots of variance in these fractions from experiment to experiment?
5) the supplementary materials are very helpful. Thank you for all these details.
Thank you very much for your detailed comments! We have uploaded a new version of the paper with these issues addressed. We have also added a section in the appendix about training and convergence issues (Section D.3).
Briefly, we did encounter some difficulties with training (particularly with high numbers of ponder steps or with high ponder costs), which we largely attribute to the entropy penalty that we added to the REINFORCE loss.
Regarding the impact and scope of our work, we would like to emphasize that our approach can potentially be applied to *any* reinforcement learning or control task, and evaluating how well our approach works on a wide range of specific RL and control problems is an important direction for future research. If our metacontroller architecture does scale well to these other domains, it has the potential for significant impact.
For example, our approach may greatly reduce the amount of hand-tuning of architectures by researchers (by automatically identifying what, and how many, computations to perform). It also provides a means for addressing the important and open problem of how to best use inaccurate models.
Additionally, we note that although the domain we evaluated in this paper is constrained, it allowed us to experimentally test out hypotheses about our architecture and draw clear conclusions that would otherwise be difficult to draw in a more complex domain with confounding factors.
For example, by using an object-centric approach rather than training the network directly on pixel data, we were able to conclude that the five-planet scenes were genuinely more difficult to predict than the one-planet scenes. This would have been impossible to do if we had trained directly on pixel data because of confounds: with increasing numbers of planets, there are an increasing number of latent factors that need to be inferred, which also likely makes the task more difficult.
Further, by choosing the domain that we did, we were able to vary the number and location of the planets, thus manipulating the difficulty of the task and demonstrating that the metacontroller architecture does in fact learn to spend more resources on difficult scenes.
Here are responses to the specific questions:
1) The different thicknesses of the lines have no significance, and we have updated the figure so that all the lines are the same length. We have also updated the figure caption to explain differences in the styles (e.g. arrowhead vs. circle).
2) The different colors refer to different settings of 𝛕 (ponder cost). We have updated the figure and caption to make the different colors clearer.
3) All of the regression lines indeed have a slope and Pearson correlation that is significantly greater than zero. We have included these statistics in the figure.
4) Yes, there is some variance in the fractions from experiment to experiment which we believe are due to the entropy penalty added to the REINFORCE loss in order to discourage the policy from becoming deterministic. We have added some discussion about this point to the appendix in Section D.3.
5) Thank you, we are glad you found the supplemental material helpful!
Pros (quality, clarity, originality, significance:):
This paper presents a novel metacontroller optimization system that learns the best action for a one-shot learning task, but as a framework has the potential for wider application. The metacontroller is a model-free reinforcement learning agent that selects how many optimization iterations and what function or “expert” to consult from a fixed set (such as an action-value or state transition function). Experimental results are presented from simulation experiments where a spacecraft must fire its thruster once to reach a target location, in the presence of between 1 and 5 heavy bodies.
The metacontroller system has a similar performance loss on the one-shot learning task as an iterative (standard) optimization procedure.
However, by taking into account the computational complexity of running a classical, iterative optimization procedure as a second “resource loss” term, the metacontroller is shown to be more efficient.
Moreover, the metacontroller agent successfully selects the optimal expert to consult, rather than relying on an informed choice by a domain-expert model designer. The experimental performance is a contribution that merits publication, and it also exhibits the use of an interaction network for learning the dynamics of a simulated physical system. The dataset that has been developed for this task also has the potential to act as a new baseline for future work on one-shot physical control systems. The dataset constitutes an ancillary contribution which could positively impact future research in this area.
Cons:
It's not clear how this approach could be applied more broadly to other types of optimization.
Moreover, the REINFORCE gradient estimation method is known to suffer from very high variance, yielding poor estimates. I'm curious what methods were used to ameliorate these problems and if any other performance tricks were necessary to train well. Content of this type this could form a useful additional appendix.
A few critiques on the communication of results:
- The formal explication of the paper’s content is clear, but Fig.’s 1A and 3 could be improved. Fig. 1A is missing a clear visual demarcation of what exactly the metacontroller agent is. Have you considered a plate or bounding box around the corresponding components? This would likely speed the uptake of the formal description.
- Fig. 3 is generally clear, but the lack of x-axis tick marks on any subplots makes it more challenging than necessary to compare among the experts.
Also, the overlap among the points and confidence intervals in the upper-left subplot interferes with the quantitative meaning of those symbols. Perhaps thinner bars of different colors would help here.
Moreover, this figure lacks a legend and so the different lines are impossible to compare with each other.
- Lastly, the second sentence in Appendix B. 2 is a typo and terminates without completion.
Thank you very much for your review! We have uploaded a new version of the paper which addresses the issues you brought up.
We are not entirely sure what you have in mind when you mention the applicability to other types of optimization. Broadly, our approach applies to any form of iterative optimization procedure. The acquisition/selection function in our approach (i.e., the controller) does not directly correspond to that used in traditional iterative procedures (e.g. Bayesian optimization, gradient descent), but could easily be replaced by these methods as long as they are differentiable. Other iterative procedures such as value iteration could also be used with our approach, for example by using the manager in conjunction with a value iteration network (Tamar et al., NIPS 2016).
The only method we used to ameliorate problems with REINFORCE was the entropy term discussed in Section B.4. In general, we did not have many problems with REINFORCE because both the size of the action space and the length of the horizon were quite small (maximum 3 and 11, respectively). We believe the entropy term may have caused other problems, however, and have added a discussion about this and other convergence details in Section D.3.
Responses to the specific points indicated above:
a) Thank you for the suggestion of more clearly demarcating what is part of the metacontroller. We have updated Figure 1a to include a box around the metacontroller components (which is everything except the scene and the world).
b) We have added x-axis tick marks to Figure 3, as suggested. Unfortunately the confidence intervals in the top row of subplots are already at the thinnest width.
However, we have updated the line colors to match the color of the points to make them a little easier to tell apart. We have also changed the colors of the plot to make it clearer which lines correspond to which values of 𝛕 (ponder cost).
c) We have fixed this typo, thank you for catching it!
A well written paper and an interesting construction - I thoroughly enjoyed reading it.
I found the formalism a bit hard to follow without specific examples- that is, it wasn't clear to me at first what the specific components in figure 1A were. What constitutes the controller, a control, the optimizer, what was being optimized, etc., in specific cases. Algorithm boxes may have been helpful, especially in the case of your experiments. A description of existing models that fall under your conceptual framework might help as well.
In Practical Bayesian Optimization of Machine Learning Algorithms, Snoek, Larochelle and Adams propose optimizing with respect to expected improvement per second to balance computation cost and performance loss. It might be interesting to see how this falls into your framework.
Experimental results were presented clearly and well illustrated the usefulness of the metacontroller. I'm curious to see the results of using more metaexperts.
Thank you for your comments!
We agree that the formalism is difficult to understand without a specific example. To address this, we have updated the caption of Figure 1A to give specific examples of what all the components are. We have also provided algorithm boxes as suggested in the appendix.
As far as we are aware, there are no existing models outside the metareasoning framework that explicitly reason about all three of the following: *which* computations to perform (controller), *how long* to compute for (manager), and *how* to actually do the computation (manager + experts). There are, of course, many existing models that address one or more of these.
For example, in Bayesian optimization, the controller is the selection criterion (e.g., expected improvement, probability of improvement, UCB); there is no manager; and there is one expert, which is the function to be optimized. In gradient descent, the controller is the gradient step; there is no manager; and there is one expert, which is the gradient of the function to be optimized.
Thank you for pointing us to the idea of optimizing with respect to expected improvement (EI) per second. This is certainly a related idea, in that by minimizing for time cost our metacontroller is also essentially trying to improve as quickly as possible (though the metacontroller does not explicitly choose those actions which have the highest rate of improvement -- it performs this optimization over the entire sequence of actions).
We are also excited about doing more experiments with more experts in future work!

Meta-review:
role: PC
data: comment: This paper fairly clearly presents a totally sensible idea. The details of the method presented in this paper are clearly preliminary, but is enough to illustrate a novel approach.

Decision: Accept (Poster)