Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: 
This paper proposes a framework for meta learning neural acquisition functions for the Bayesian optimization of various underivable functions. The neural acquisition functions are learned using proximal policy optimization in an outer loop on different problems on the same domain, and the learned acquisition function can be deployed at test time in a practically vanilla Bayesian optimization procedure. The authors demonstrate the performance of the method through benchmarks on four problems.

I recommend that this paper be accepted for publication. The paper is well written and it proposes a novel direction for research. However, I think that the authors should look further inside their newly designed acquisition functions, not merely treat them as black boxes. Find below some questions and comments.


Due to the inclusion of the sample position x in the state tuple, I am curious as to what the authors think is the difference between their method and a learning-to-learn type of approach. Is the acquisition function learning to favor specific zones in the search space based on previous experiments? Some more experiments or insights on this would be useful to better understand what makes this method succesful.

Why was a categorical distribution used for the policy? These samples are located in D, aren't you getting rid of information by assuming they are completely independent? Aren't you also biasing the distribution by adding the local maxima to the set of ξ (Xi)?

Also, the right-most block in Figure 1 shows a continuous probability distribution, which is incorrect. If the distribution is indeed categorical, there is no continuity between points.


Minor mistakes:

- page 5, paragraph 3: "This choice does not penalize explorative evaluations which do not yield and immediate improvement" should read "an immediate improvement"
- Figure 4b, MetaBO-50 is missing

***********
Post rebuttal:
************

I have read the other reviews and the various replies by the authors. I'd say you did a good job in answering most questions and added a lot of valuable information in the appendices. I maintain my score.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: Summary: The authors propose a meta-learning based alternative to standard acquisition functions (AFs), whereby a pretrained neural network outputs acquisition values as a function of hand-chosen features. These neural acquisition functions (NAFs) are trained on sets of related tasks using standard RL methods and, subsequently, employed as drop-in replacements for vanilla AFs at test-time. 


Feedback:
Overall, the proposed method makes sense and would benefit from further experimental ablation. Using RL to automatically derive (N)AFs is a nice change of pace from the hand-crafted heuristics that dominate BO. I like the ideas at play here and hope that you will convince me to amend my score.

Results on synthetic functions presented in the body of the paper demonstrate that NAF outperforms, e.g., EI when transferring between homogenous tasks. In contrast, results when transferring between relatively heterogenous functions (Fig. 9) indicate that the aforementioned performance gain reflect NAFs ability to specialize. Two things remain unclear however: 
    a. What types of regularity are NAFs able to exploit?
    b. How quickly do NAFs benefits fall off as tasks become increasingly heterogenous?


Regarding (a), I am not yet convinced that NAFs learn representations that go beyond standard AFs combined with a prior over $x$. To help test this hypothesis, here is a sketch of a simple baseline algorithm:
  1. Fit, e.g., a Gaussian Mixture Model to the top $k=1$ designs $x^{*}_{i}$ on observed tasks $i \in [1, N]$, 
  2. Given a new task $f_{j}$, let log-likelihood $GMM(x)$ act as a 'prior' of sorts on $x$ 
  3. Use cross-validation to tune the scalar parameter $w$ of a new AF defined as the convex combination:

        GMM-UCB(x_{k}) = w * GMM(x_{k}) + (1 - w) * UCB(x_{k})
                                       = w * GMM(x_{k}) + (1 - w) * [\mu_{k} + \sqrt{\beta} * \sigma_{k}].

I suggest using UCB both because NAF could easily learn it from its inputs and because EI values often decay dramatically over the course of BO (I usually set UCB's confidence parameter to a fixed value $\beta = 2$). 

Further simplifying this idea, you could instead use an $\epsilon$-greedy style heuristic that, with probability $\epsilon$, samples without replacement from the set of historical minimizers and otherwise uses a standard AF. These baselines are comparatively straightforward and easily interpreted, so I hope that you will consider adding something along these lines.


Additionally, here are some questions/suggests to help probe (a-b):
  1. Another baseline: EI with multi-task GP? The cubic scaling should be fine for, e.g., 'xxx-20' multi-task variants.
  2. Extend experiments on functions drawn from GP priors (Fig 9):
      i. How does homogeneity (as enforced via the GP hyperprior) impact performance when transferring knowledge?
      ii. Rate of convergence suggests sampled tasks may be too easy; consider using Matern-5/2 and smaller lengthscales [*].
  3. What happens if you expand the task augmentation process to further include, e.g., flips and rotations?
  4. How do 'dimension-agnostic' versions of NAF (where $x$ is excluded from its input) perform on other synthetic tasks?
  5. Visualizing NAF (or the search strategies it produces) would be useful for building intuition.
  6. How were NAF input features chosen? Were alternatives such as also passing the 'best seen' value, considered?
  7. How easy to use are NAFs in comparison to alternative AFs (both in terms of training and test-time maximization)?
  8. Please report regret in log-scale (in appendix); currently, it is hard to tell what is going on in some places. Similarly, the tracked regret level in Figures 3 & 7 changes between tasks without explanation?


In summary, I genuinely want NAF to succeed but am not yet convinced of its performance. If you can provide empirical results to help extinguish my doubts, I will gladly change my assessment.


Nitpicks, Spelling, & Grammar: 
  - Some minor spelling and/or grammatical error, but the paper reads fairly well.
  - On [Chen et al., 2017]:  To the best of my knowledge, these RNN-based methods only require the gradient of the loss function. For example, using GP-based EI as the training signal only requires differentiating through EI + GP rather than through the target function $f$. Similarly, in cases where gradients are not available, the authors elude to use of RL algorithms such as REINFORCE.

[*] For Matern-5/2, just change the prior on your basis functions' weight parameters (https://github.com/metabo-iclr2020/MetaBO/blob/master/metabo/environment/objectives.py#L295) from standard normal to multivariate-t with 5 degrees of freedom.
review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: The authors present MetaBO, which uses reinforcement learning to meta-learn the acquisition function (AF) for Bayesian Optimization (BO) instead of using a standard constant AF. The authors shows that MetaBO enables transferring knowledge between tasks and increasing sample efficiency on new tasks. The paper is mostly clearly written and I am not aware of existing work on meta-learning the AF for BO. However, the approach is related to Chen et al, which is cited in the text but not used as a baseline. It is also not shown clearly enough how the performance of MetaBO depends on the number of training tasks and distance between training and test tasks. I therefore consider the paper as borderline.

Major comments
=============
1. The presented approach is very similar to Chen et al, which is discussed in the related work section but not used as a baseline. Although Chen et al assumed that f(x) is differentiable, their approach can be easily generalized to non-differentiable functions by using RL as Chen et al discussed in the last paragraph of section 2.1. Chen et al does not depend on a GP and is therefore more scalable. The source code is publicly available (https://github.com/deepmind/learning-to-learn) and you can also adapt your implementation by removing the GP part.

2. Global Optimization Benchmark Functions: How does the performance of MetaBO depend on the number of training samples (number of training tasks times the budget T)?

3. Figure 3: How does MetaBO generalizes to functions that are translated and scaled at the same time? This can be visualized as a heatmap with the scaling and translation on the x and y axis, and using the color to show the number of steps to reach a certain reward. How does the generalization performance depend on the noise level, where the noise can be sampled from standard normal distribution? Why does EI perform better if the function is translated more?

4. Simulation-to-Real task: How does the generalization performance of MetaBO depend on the distance between training and source tasks (x-axis: distance; y-axis: steps to reach a certain reward)? You sampled test tasks 10%-200% around the true parameters. Test tasks can therefore have identical or similar parameters than training tasks.

5. Simulation-to-Real task: How does the performance depend on the number of training tasks (x-axis: # training tasks; y-axis: steps to reach a certain performance)?

Minor comments
=============
6. Section 1, 2nd paragraph: The performance of BO also depends on the GP kernel and kernel hyper-parameters, not only the AF. Please mention this. Similarly, ‘no need to calibrate any hyperparameter’ in section 4 ignores GP hyper-parameters. Please clarify.

7. Section 2, 4th paragraph: A Neural Process (https://arxiv.org/abs/1807.01622) is another scalable alternative to a GP. Please cite.

8. Section 3, 2nd paragraph: Please cite standard AFs such as EI, PI, UCP. 

9. Section 4, last paragraph before ‘Training procedure’. The state s_t is undefined at this point. This section misses a clear description of the state, reward, and transition function of the MDB. Does the state s_t take previous function evaluations into account (e.g. via a RNN state), or only \mu and \sigma at the current step t? Does the state include the time step as described in the text and in the section about the value network in the appendix but not in table 1.

10. Section 4, ‘the state corresponds to the entire functions’.  It only depends on the first two moments (and the time step t?).

11. Section 4: replace ‘not to be available’ by ‘unavailable’.

12. Section 4: reference or describe ‘Sobol grid’.

13. Section 4: The approach to maximize the AF on grid points does not scale to high-dimensional search spaces. Please also clarify how global and local grid points were chosen. In particular, ‘local maximization’ is unclear. Also, ‘cheap approximation’ of the global maximum of f(x) is infeasible if the search space is high-dimensional. 

14. Please move figure 3 above figure 4.
review_title: Official Blind Review #2

Meta-review: This paper explores the idea of using meta-learning for acquisition functions. It is an interesting and novel research direction with promising results. 

The paper could be strengthened by adding more insights about the new acquisition function and performing more comparisons e.g. to Chen et al. 2017. But in any case, the current form of the paper should already be of high interest to the community

Decision: Accept (Spotlight)