Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper presents a framework for conditional optimization, which is called ConBO. Authors also proposed a new acquisition function: hybird knowledge gradient, which is built on KG. Empirical experiments show their proposed method outperform many competing algorithms. 
main_review: I reviewed this paper at AISTATS. While I was glad to see that authors have added additional comparison experiments on multi-task and multi-fidelity methods (but only in the supplementary material, not int the main paper), a lot of goods points raised by reviewers there are still not addressed in this version. 

improvements:
1. many related work are covered in this version and this paper is well-organized

weaknesses:
1. Some related work is still missing.
2. There are insufficient updates in the main paper. Compared with the last version, all figures and empirical evaluation are exactly the same, without any changes. Although there are some extra experiments in the supplementary material.  Notably the comparison with [1] is still missing. This work is closely related to [1] and a comparison with [1] is recommended.
3. Given the proposed method doesn't provide sufficient improvements over KG, I would have expected stronger and extensive empirical evaluation. 






[1]. Krause, A., Ong, C.S.: Contextual Gaussian Process Bandit Optimization. In: Shawe-Taylor, J., Zemel, R.S., Bartlett, P.L., Pereira, F., and Weinberger, K.Q. (eds.) Advances in Neural Information Processing Systems 24. pp. 2447–2455. Curran Associates, Inc. (2011).


limitations_and_societal_impact: See the above review.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a new algorithm for a problem entitled "conditional Bayesian optimization." The goal is to search for a *series of input locations* that maximizes the performance of a given set of tasks. Unlike multi-fidelity optimization, the objective is to find good values for all tasks rather than the high-fidelity task. The main contribution is a Hybrid Knowledge Gradient (KG) algorithm that mixes two ways of computing KG: one that discretizes the input space (x, t) and computes KG for a discrete set of points,  and another one that uses the reparametrization trick to perform stochastic gradient descent over Monte Carlo samples. Formal guarantees about this method are also given. Empirical results show that the proposed method effectively reduces computation time and improves the quality of the solution.
main_review: **Strengths**

I think this paper explores a problem formulation that is not well-study, but it could be relevant. It is a subtle variation of multi-task learning that focuses on multiple solutions (on for each task). This distinguishing is a little bit tricky, and I was a little bit confused for a while. Fortunately, the authors give three different examples that illustrate the setting that they are considering. I also enjoy the experiment section of this paper. The authors discussed the studied problems adequately, and I found the visualization and presentation of the results very interesting.

**Weaknesses**

Originality. This paper would be more substantial if the main algorithmic contribution weren't so tight to computational tricks to compute Knowledge Gradient or if the problem under investigation wasn't similar to REVI's [22] formulation.  If I understood correctly, the main contribution is a speedup over REVI based on how KG is calculated. The theoretical analysis adds substance to the paper, but none of these results are ground-breaking.

Clarity. The presentation of section 2 could be a little bit more straightforward. There is unnecessary confusion about how information is captured by the inference procedure (the Gaussian process) and how decision-making is handled (construct an acquisition function). 

**Minors**

- 115 - "integrating over tasks multiplies the computational burden" multiples -> increases
- 130 - "non-trivial modification to be able to account for how a sample affects similar tasks." Wouldn't a kernel that captures the correlation between similar tasks be enough to accomplish this?
- 133 - location of the peak P(x | ...) -> mutual information between both distributions (or RV), you can remove the word peak here.
- 149 - I don't think you meant to say that squared exponential or Matern is a kernel that factorizes here.
- l89 - n_s? Shouldn't it be n_z? 

**After rebuttal**
 I appreciate the author's detail response and I would strongly recommend the authors to add this level of clarity and detail to the original manuscript. Thanks
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The authors tackle an underexplored problem they call conditional BO, which is a variant of multi-task BO in which one seeks to identify the optimum of each task, instead of the traditional multi-task BO seeks to identify the optimum of the average over all tasks. 

The authors introduce ConBO, which uses the knowledge gradient acquisition function to determine the next point to evaluate, and discuss its efficient computation through a combination of intelligent sampling and discretization. 

ConBO is evaluated on a variety of problems, including three good applications: hyperparameter optimization, ambulance base placement, and assemble-to-order policy selection. Results are promising, and suggest that ConBO outperforms a well-chosen set of sensible baselines.
main_review: Strengths:
* The applications were well suited to the problem at hand. 
* The discussion of prior work and baseline performance measures was very comprehensive. 
* The problem itself is both of theoretical interest and practical value. 
* I also appreciated the author’s efforts to speed up the acquisition function computation; this is too often ignored in the literature as of minimal importance. 

Weaknesses:
* The primary weakness, which the authors themselves point out, is that this problem of conditional BO is nearly identical to the problem of Bayesian quadrature; though the acquisition function itself is novel to my knowledge, the problem formulation is not. 
* The acquisition function is the expectation of the KG acquisition over tasks s $E_s[ KG(s)]$, where the expectation is taken w.r.t measure W(s). My intuition says this acquisition seeks to efficiently compute $E_s[ x(s)]$ where $x(s) = \max_s \mu(x)$ w.r.t. W(s), which seems to be precisely Bayesian quadrature (please correct me if my intuition is wrong!) 
* Thus, any method for Bayesian quadrature ought to work well, unless I am missing something. There don’t seem to be any Bayesian quadrature methods in the baselines, is there a reason why? 
* I think this paper can be greatly strengthened if the authors define different measures of success. Obviously there is no way to identify exactly x(s) for all s, so the authors have decided their measure of success is $E_s[ x(s) ]$, which is one of the most logical. But there are other measures of success for conditional BO that do not necessarily involve this expectation; for example, computing accurately the bounds $\max_s | x(s)|$. Perhaps these can be used to showcase the strengths of ConBO, though this would likely require significant revision. 

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The authors propose a method for conditional optimization problems: finding the optimum of each function in set of objective functions. In this setting, the method jointly selects the parameters to evaluate and the task to evaluate. The authors propose a theoretically-grounded variant of the Knowledge Gradient to address this class of problems and propose a new method of approximating the KG acquisition function. 
main_review: Strengths:
-	Elegant, theoretically-grounded formulation for conditional optimization
  -	Being myopically optimal in the value of information framework is a nice property
-	The hybrid KG results show that is a quite stable method for approximating KG.

Weaknesses:
-	The class of problems is not very well motivated. The CIFAR example is contrived and built for demonstration purposes. It is not clear what application would warrant sequentially (or in batches) and jointly selecting tasks and parameters to simultaneously optimize multiple objective functions. Although one could achieve lower regret in terms of total task-function evaluations by selecting the specific task(s) to evaluate rather than evaluating all tasks simultaneously, the regret may not be better with respect to timesteps. For example, in the assemble-to-order, even if no parameters are evaluated for task function (warehouse s) at timestep t, that warehouse is going to use some (default) set of parameters at timestep t (assuming it is in operation---if this is all on a simulator then the importance of choosing `s` seems even less well motivated). There are contextual BO methods (e.g. Feng et al 2020) that address the case of simultaneously tuning parameters for multiple different contexts (tasks), where all tasks are evaluated at every timestep. Compelling motivating examples would help drive home the significance of this paper.
-	The authors take time to discuss how KG handles the continuous task setting, but there are no experiments with continuous tasks
-	It’s great that entropy methods for conditional optimization are derived in Section 7 in the appendix, but why are these not included in the experiments? How does the empirical performance of these methods compare to ConBO?
-	The empirical performance is not that strong. EI is extremely competitive and better in low-budget regimes on ambulance and ATO
-	The performance evaluation procedure is bizarre: “We measure convergence of each benchmark by sampling a set of test tasks S_test ∼ P[s] ∝ W(s) which are never used during optimization”. Why are the methods evaluated on test tasks not used during the optimization since all benchmark problems have discrete (and relatively small) sets of tasks? Why not evaluate performance on the expected objective (i.e. true, weighted) across tasks?
-	The asymptotic convergence result for Hybrid KG is not terribly compelling
-	It is really buried in the appendix that approximate gradients are used to optimize KG using Adam. I would feature this more prominently.
-	For the global optimization study on hybrid KG, it would be interesting to see performance compared to other recent kg work (e.g. one-shot KG, since that estimator formulation can be optimized with exact gradients)

Writing:
-	L120: this is a run-on sentence
-	Figure 2: left title “poster mean” -> “posterior mean”
-	Figure 4: mislabeled plots. The title says validation error, but many subplots appear to show validation accuracy. Also, “hyperaparameters” -> hyperparameters
-	L286: “best validation error (max y)” is contradictory
-	L293: “We apply this trick to all algorithms in this experiment”: what is “this experiment”?
-	The appendix is not using NeurIPS 2021 style files
-	I recommend giving the appendix a proofread:
  - Some things that jump out
    -	P6: “poster mean”, “peicewise-linear”
    -	P9: “sugggest”

limitations_and_societal_impact: Limitations and societal impacts are discussed, but the potential negative societal impacts could be expounded upon.
review_title: null

Meta-review: This paper addresses a framework for conditional Bayesian optimization with a newly proposed hybrid KG. The method seems to be a variant of multi-task BO, where the optimum of each task is determined in contrast to the multi-task BO. The applications well suited to the problem are described to demonstrate the usefulness of the method. Reviewers agree that the paper is of theoretically interest and has practical value as well. However, a few issues were raised, details of which can be found in the reviewers' comments. During the discussion period, none of the reviewers were willing to champion this work. The paper still needs more substantial updates in particularly experiments. 


Decision: Reject