Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focuses on the optimization of a black-box expensive to evaluate function in the low data regime where the goal is to take advantage of offline related datasets. Specifically, when the offline and online data have different distributions, this work proposes to build a a generative model for pretraining a novel black-box optimizer based on the offline data. 
strength_and_weaknesses: Instead of a  heuristic for transitioning between low and high fidelity, it is interesting to see if the transition can be modeled and learned.

It is also interesting to see how this approach compares to multi-task learning where different tasks include different data distributions amongst offline and online datasets.
 This approach seems to have commonalities with Thompson Sampling (TS) and batch Thompson Sampling (q-TS) from Bayesian optimization literature. Although this work cites the related papers, a more through comparison/analysis can be valuable.

clarity,_quality,_novelty_and_reproducibility: "We haven’t included Hopper since the domain is buggy - we found that the oracle function used to evaluate
the task was highly inaccurate and noisy." Would the author please comment on why noise cannot be modeled and accounted for? For example, in a Gaussian process framework, observation noise can be incorporated. This is especially important since many real-world problems are corrupted with noise.
summary_of_the_review: This paper addresses and interesting problem. However, comparison to other related works can be improved. Without such comparison, an accurate judgement is hard to make.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies black-box optimization and proposes BONET, which is a generative pre-trained model from offline datasets. An autoregressive model on fixed-length trajectories is trained, and a sampling strategy is designed to synthesize trajectories from offline data using a simple heuristic of rolling out monotonic transitions.
strength_and_weaknesses: Strength:
1.	The idea to pre-train a generative model is interesting.
2.	The paper is well-structured.
Weakness:
1.	The setting that offline datasets are available is doubtful.
2.	Experiments are not convincing enough to illustrate the effectiveness of the proposed framework.
3.	Some statements are unclear and difficult to understand.


clarity,_quality,_novelty_and_reproducibility: This paper studies black-box optimization problems, which is not a new problem.  I have the following concerns for the authors to address:
1.	This paper assumes that offline datasets are available for pretraining, which is doubtful. It is unclear what types of black-box real-world applications satisfy this setting. The distributions of offline datasets may be different from online datasets, which deteriorates the performance of black-box optimizers.
2.	In the introduction part, the authors state that they can synthesize synthetic trajectories of offline points to mimic online points. The correctness of this statement requires further illustration. Again, the offline datasets and the online datasets may be differently distributed.
3.	Some figures are unclear to readers. For example, it is unclear what information Figures 1(b) and 1(c) convey.
4.	In the experiments, many recently proposed black-box baselines are missing. For example, the following black-box optimizers were published in top-tier conferences and should be compared with the proposed  BONET.
Explicit Gradient Learning for Black-Box Optimization
Black-Box Optimization with Local Generative Surrogates
Differentiating the Black Box: Optimization with Local Generative Surrogates

5.	On the Branin dataset, Table 1 is of little informative. Also, more baselines are required for comparison.
6.	On the DESIGN-BENCH tasks, several baselines are quite old, and Table 2 shows that the proposed BONET does not significantly outperform others. In fact, it is not the best optimizer for most tasks.
7.	Some typos should be corrected. For example, 
empirical observation relating to=> empirical observation related to

summary_of_the_review: This paper proposes an interesting method to deal with black-box optimization problems. However, it is unclear what real-world applications are suitable where off-line datasets are suitable for black-box optimization, the proposed method did not outperform baselines in the experiments, and some state-of-the-arts were not compared.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed BONET, a method to optimize expensive black-box function with offline data. BONET consists of three phases, 1) synthesize trajectories from offline data using a simple heuristic, 2) fit an autoregressive model based on the trajectories and regret budgets and 3) roll out the evaluation to output predictions for the maxima of the black-box function. Empirical study is conducted on one synthetic 2D example and 7 real-world tasks, which showed promising results. 
strength_and_weaknesses: Strength: 
* The paper is well written and easy to follow. 
* The proposed idea is neat and easy to use in practice. 


Weakness:
* The analysis regarding \hat{R} are for the Branin task, does it hold in general? 
The optimal value for \hat{R} seems to be 0 from Figure 4, how much value does it add when fitting the autoregressive model with regret information? 
* In phase one, what is the choice of N_B and how it affects the performance? 
* more clarification in the experimental section may improve the paper. For example, have a table of dimensionality for the 7 real world datasets, clarify the definition of normalised score in Table 2 and Table 3, and correct typos such as ''stitching'' -> ``stitching''. 
* It would be interesting to see more evidence in the benefits of the model in the case of high-dimensional multi-modal scenarios. It is not clear from the experiments that the problems are high-dimensional and multi-modal. 
* how does the model perform when observations are sparse? 
* is there a way to extend to the case when the functional evaluations are noisy?

clarity,_quality,_novelty_and_reproducibility: The paper is in general well written and easy to follow, the code is provided with an anonymous link. 
summary_of_the_review: The paper presented a neat idea in maximizing black-box functions using offline data only. Although the proposed method BONET is somehow based on exiting ideas such as importance reweighing and causally masked transformer, the simplicity of the proposed method brings value to be applicable in practice.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a new generative framework to pretrain the trajectory towards the optimal value of an objective function on an offline dataset, then use this generative model (more specifically, an autoregressive model) in evaluation to select data points so as to find the optimal value of the objective function. To synthesize the trajectories from the offline data, the paper presents a sampling strategy based on a simple heuristic of sorting the function values of the data points in the offline dataset and performing a binning process that weighs more on data points with higher function values. Experiments are conducted on a synthetic function and various real-world problems to understand the performance of the proposed approach.
strength_and_weaknesses: Strengths:
+ Overall, I think the idea of learning the trajectory towards the optimal value using offline dataset is interesting and might have potential.
+ The paper's writing is also clear and easy to understand.
+ The experiments include real-world problems.

Weaknesses:
I have various concerns regarding the paper. Please see my comments below.
+ The proposed sorting strategy seems to not take into account the fact that the observed data can be noisy. And the regrets in the trajectory training dataset also assume that we can know exactly the noiseless values of the observed data. But in practice, most of the time, the objective function values will be corrupted by noise. How will noise affect the performance of the proposed method?
+ The assumption of having an estimation of the objective function's optimal value f(x*) is very strong. How does the performance of the proposed method change w.r.t. this value? The sensitivity analysis conducted in Section C.5 is not enough to understand how this value affects the performance of the proposed method (it's just for one simple problem and the range of this estimated value is limited). Besides, what are the values f(x*) used in all the experiments? I seem to not be able to find this information in the main paper. Note that if the value f(x*) is known, there are other existing works developing new BO methods to use this known value to improve the efficiency of the optimization process, e.g., Knowing the what but not the where in Bayesian optimization by Nguyen and Osborne (ICML 2020).
+ A big concern is that there is a lack of comparison with warm-start BO methods, e.g., Initializing Bayesian Hyperparameter Optimization via Meta-Learning by Feurer et al (AAAI 2015). These warm-start BO methods also make use of the knowledge learned from similar datasets and then transfer this knowledge into a new optimization problem. It has been shown that these methods can help speeding up the BO process. And note that the learned datasets in these warm-start methods do not necessarily come from the same problem we need to optimize.
+ In the experiments, the offline datasets are generated from the same problem as the problem during evaluation, so I don't think the task of learning the trajectory towards the optimal value is challenging anymore. For the Branin function, the offline datasets are constructed from 5000 data points in the domain, this is really big, and to me, this offline dataset can already help to learn the objective function quite correctly. For other real-world tasks of Design-Bench, I'm not sure if I missed it but I can't find information regarding the size of the offline datasets of these tasks in the main paper. How much data used in the offline datasets of these tasks?
+ In Tables 2 & 3, what do 100th percentile comparative evaluation and 50th percentile evaluations mean? 

clarity,_quality,_novelty_and_reproducibility: The paper's writing is generally clear and easy to understand - please see my comments above.

Regarding the paper's quality, I have various concerns regarding the proposed method and the experimental evaluation - please see my comments above.

I think the main idea of the paper, learning the trajectory towards the optimal value using offline dataset, is instersting and new - please see my comments above.

The code is made available so I believe in the reproducibilty of the paper.
summary_of_the_review: Even though I think the main idea of the paper is interesting and new, I have various serious concerns regarding the rigorousness and soundness of the proposed method, and also the experimental evaluation. Please see my comments above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Authors introduce a method to optimize expensive black-box functions with offline data. The method consists of three phases, 1) synthesize trajectories from offline data using a simple heuristic, 2) fit an auto-regressive model based on the trajectories and regret budgets and 3) roll out the evaluation to output predictions for the maxima of the black-box function. Authors report encouraging results. However, the overall concern regarding this submission is the practical relevance of the proposed approach. Detailing realistic settings would help fixing ideas and convincing the readers that generating trajectories based on offline data is sensible and useful. While results are encouraging, the paper does not include in depth ablations studies that validate the assumptions and the pertinence of the different phases (but it does contain a number of other ablations studies related to the knowledge of the approximate optimum, the inclusion of noisce, etc.).

justification_for_why_not_higher_score: This work has interesting ideas and appears well executed, but the practical relevance is unconvincing and would require some work.
Decision: Reject