Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to augment Bayesian optimization (Gaussian processes based) formulation with human experts intervention on suggesting sample points (different from the Bayesian optimization suggestion) and human's good and bad sample regions (to be exact they are sample points instead of regions as it is formulated in the paper now).  The introduction of human intervention into BO is an interesting and potentially useful contribution. However, the nature of human intervention is not rigorous enough missing  formal  characterization of the exact requirement on human experts and practical verification of such human properties in human study. For example, we might need verifiable formal notion of what if human expert is wrong, what kind of wrongness, and how wrong the system can endure a human expert but still guarantee a bound of the optimality as well as the sampling process efficiency.     

strengths_and_weaknesses: Strength
	- The introduction of human expert recommended samples as well as good and bad region suggestions into the BO formulation

Weakness
	- Experiments lack real human experts human study
	- No formal treatment of human experts properties and how these properties will be factored into the proposed augmented BO
	- The BO formalization will be better off if it can also handle non-Gaussian processes BO formulation such as: Garnelo, Marta, et al. "Neural processes." arXiv preprint arXiv:1807.01622 (2018), and its follow-up work.
	

questions: - what if human expert is wrong?
- what kind of wrongness, and how wrong the system can endure a human expert but still guarantee a bound of the optimality as well as the sampling process efficiency?
limitations: Please improve the study with (1) a formal  characterization of the exact requirement on human experts and practical verification of such human properties in human study; (2) non-Gaussian processes BO formulation such as: Garnelo, Marta, et al. "Neural processes." arXiv preprint arXiv:1807.01622 (2018), and its follow-up work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a new method for human-AI collaboration in the context of Bayesian optimization, where the optimization algorithm can get occasionally help from a human expert with deeper knowledge about the underlying physical phenomenon, either in the form of a correction of the current recommendation within the search space or specifying good and bad regions within the search space. Authors provide a theoretical framework justifying their approach and an empirical validation of the method against a baseline method without expert feedback.

strengths_and_weaknesses: The paper contribution is clearly presented and its effectiveness in accelerating the optimization process is supported theoretically and validated both using synthetic and real world data. The  paper is articulated in a clear way and the overall quality is fair. 
The background section and results discussion do not report other works about collaborative human-AI in optimization processes, therefore an assessment of the paper's contribution originality and significance in the context of related work is hard. 

questions: - Could you better explain the statement at line 29, expanding more on why the nature of application ensure that there would be an expert running the optimization process?
- Are there other works in the literature introducing human feedback into Bayesian optimization or optimization processes? I would expand the background section by mentioning the state of the art of works including the human in the loop of optimization processes. 
- How would authors explain the better performance of HAT with recommendation correction than with bad/good regions feedback? (see Fig.2)
- In real-world experiments: which was the classification task? And, how was the feedback from humans provided in this case?
limitations: Authors don't address limitations of their work. 
The main limitation of including a feedback expert in the optimization process is that the performance relies heavily on the human true level of "experience" about the process under analysis, and besides such feedback goodness varies from expert and expert. I suggest authors to mention their view about this and other limitations as well as improvement perspectives, in the Conclusion section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a framework that makes use of expert's knowledge in improving the effectiveness of Bayesian optimization. The Human-AI Teaming (HAT) based Bayesian optimization framework takes expert inputs as constraints via the following two ways:
- Rectifying recommendations by making use of the correlation structure of the experimental system by updating the hyperparameters with the constraints that the acquisition function of the human model is better than the one of the AI model. 
- Specifying good and bad regions based on the current observation set by updating the hyperparameters that optimize the difference between the acquisition function of the good regions and the bad regions and achieve at least \Delta of the original likelihood. 
Furthermore, the paper also provides theoretical analysis that shows additional information from human experts enforced as constraints result in a narrower distribution, thus improving sample efficiency. 

Finally, the effectiveness of the proposed framework is tested using both synthetic data and real-world experiment of tuning the hyperparameters of SVM. 
strengths_and_weaknesses: Strength: 
- This paper is well written and flows well. 
- This paper provides theoretical analysis of the proposed method. 
- This paper provides an original idea of incorporating human experts knowledge into the Bayesian optimization procedure through constraitns
- The effectiveness of the proposed method is demonstrated in both synthetic and real-world experiments 

Weakness: 
- It's not clear whether the proposed method will achieve the optimal solution faster than the standard Bayesian optimization based on the presented results. 
questions: In the synthetic experiment section, Figure 2 shows the simple regret for 10*d+5 iterations for the different models. Why choose "10" here?  It seems that after certain iterations, although the human AI teamed method converge faster (i.e., big drops in simple regret in early iterations than standard BO procedure), they reach the optimal simple regret almost at the same iteration as the standard BO procedure. For example, Ackley 1D, human AI teamed model reached 0.0 regret around iteration 13 and standard BO around iteration 14. Since the iteration is capped at "10*d+5", it's not clear if human-AI will get to the global optimal faster than the standard BO procedure. The experiments does show that when the number of iterations (e.g. computation budget) is limited, using a human-AI teamed BO method will give better results.

Typos: 
- Line 278: "Algorithm 2" should be "Algorithm 1" 
limitations: The proposed method will improve the performance of Bayesian optimization method when the computation budget is limited; however, it does not seem to find the optimal solution in significant fewer iterations. One improvement to add in the experiment section is to provide a figure of the number of iterations of the optimal regret (after the optimizer is converged) for all the comparing models.
review_title: null

Meta-review: metareview: Reviewers found the contribution of introducing human feedback into Bayesian optimization novel, interesting, and sound. (It is worth noting that the author rebuttal was crucial in assuaging several reviewer concerns and confusions.) A more extensive human study would significantly strengthen the work, but that said, the paper should be an interesting contribution to NeurIPS as is.
confidence: Less certain
Decision: Accept