Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Minor comments:
- page 7. “Then, observe that the same reasoning we used to develop the cfKG acquistion function
in (3.2) can be used when when we observe gradients to motivate the acquisition function…” - some misprints, e.g. double “when”
- The paper lacks theoretical analysis of convergence of the proposed modification of the knowledge gradient criterion.

Major comments:

Current approaches to optimisation of expensive functions are mainly based on Gaussian process model. Such approaches are important for Auto ML algorithms.

There are a lot of cases, when for an expensive function we can obtain measurements of its values with continuous fidelity by leveraging costs for evaluation vs. fidelity of the obtained values. E.g. as fidelity we can consider a size of the training set used to train a deep neural network.

The paper contains a some new algorithm to perform Bayesian optimisation of a function with continuous fidelity. Using modification of the knowledge gradient acquisition function the authors obtained black box optimisation method taking into account continuous fidelity. 

Due to some reason the authors forgot to take the cost function into account when formulating the algorithm 1 in 3.3.2 and corresponding formula (3.7).

So, the logic of the definition of q-cfKG is understandable, but the issue with the missing denominator, containing cost function, remains.

The approach, proposed in section 3.3.2, looks as follows:
- the authors used formulas from [Wu t al (2017) - https://arxiv.org/abs/1703.04389] 
- and include additional argument in the mean function of the Gaussian process.
However, in Wu t al (2017) they consider usual knowledge gradient, but in this paper they divide by the value of max(cost(z)), which is not differentiable.

Other sections of the paper are sufficiently well written, except 
- the section 3.3.2, 
- section with results of experiments: I was not able to understand how the authors defined cost function in sections 4.2 and 4.3 for their neural network and large scale kernel learning.

In principle, the paper contains some new results, but it should be improved before publishing.
review_title: a paper with some new results

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
Many black-box optimization problems are "multi-fidelity", in which it
is possible to acquire data with different levels of cost and
associated uncertainty.  The training of machine learning models is a
common example, in which more data and/or more training may lead to
more precise measurements of the quality of a hyperparameter
configuration.  This has previously been referred to as a special case
of "multi-task" Bayesian optimization, in which the tasks can be
constructed to reflect different fidelities.  The present paper
examines this construction with three twists: using the knowledge
gradient acquisition function, using batched function evaluations, and
incorporating derivative observations.  Broadly speaking, the idea is
to allow fidelity to be represented as a point in a hypercube and then
include this hypercube as a covariate in the Gaussian process.  The
knowledge gradient acquisition function then becomes "knowledge
gradient per unit cost" the KG equivalent to the "expected improvement
per unit cost" discussed in Snoek et al (2012), although that paper
did not consider treating fidelity separately.

I don't understand the claim that this is "the first multi-fidelity
algorithm that can leverage gradients".  Can't any Gaussian process
model use gradient observations trivially, as discussed in the
Rasmussen and Williams book?  Why can't any EI or entropy search
method also use gradient observations?  This doesn't usually come up
in hyperparameter optimization, but it seems like a grandiose claim.
Similarly, although I don't know of a paper that explicitly does "A +
B" for multi-fidelity BO and parallel BO, it is an incremental
contribution to combine them, not least because no other parallel BO
methods get evaluated as baselines.

Figure 1 does not make sense to me.  How can the batched algorithm
outperform the sequential algorithm on total cost?  The sequential
cfKG algorithm should always be able to make better decisions with its
remaining budget than 8-cfKG.  Is the answer that "cost" here means
"wall-clock time when parallelism is available"?  If that's the case,
then it is necessary to include plots of parallelized EI, entropy
search, and KG.  The same is true for Figure 2; other parallel BO
algorithms need to appear.
review_title: Incremental Technical Contribution, Weak Empirical Comparisons

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies hyperparameter-optimization by Bayesian optimization, using the Knowledge Gradient framework and allowing the Bayesian optimizer to tune fideltiy against cost.

There’s nothing majorly wrong with this paper, but there’s also not much that is exciting about it. As the authors point out very clearly in Table 1, this setting has been addressed by several previous groups of authors. This paper does tick a previously unoccupied box in the problem-type-vs-algorithm matrix, but all the necessary steps are relatively straightforward.

The empirical results look good in comparison to the competing methods, but I suspsect an author of those competitors could find a way to make their own method look better in those plots, too.

In short: This is a neat paper, but it’s novelty is low. I don't think it would be a problem if this paper were accepted, but there are probably other, more groundbreaking papers in the batch.

Minor question: Why are there no results for 8-cfKG and Hyperband in Figure 2 for SVHN?
review_title: Neat work of low novelty.

Meta-review: This paper combines multiple existing ideas in Bayesian optimization (continuous-fidelity, use of gradient information and knowledge gradient) to develop their proposed cfKG method.  While the methodology seems neat and effective, the reviewers (and AC) found that the presented approach was not quite novel enough in light of existing work to justify acceptance to ICLR.  Continuous fidelity Bayesian optimization is well studied and knowledge gradient + derivative information was presented at NIPS.  The combination of these things seems quite sensible but not sufficiently novel (unless the empirical results were *really* compelling).

Pros:
- The paper is clear and writing is of high quality
- Bayesian optimization is interesting to the community and compelling methods are potentially practically impactful
- Outperforms existing methods on the chosen benchmarks

Cons:
- Is an incremental combination of existing methods
- The paper claims too much
Decision: Reject