PABBO: Preferential Amortized Black-Box Optimization

Authors: Xinyu Zhang, Daolang Huang, Samuel Kaski, Julien Martinelli
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: bayesian optimization, preference learning, amortized inference, neural processes

Abstract:
Preferential Bayesian Optimization (PBO) is a sample-efficient method to learn latent user utilities from preferential feedback over a pair of designs. It relies on a statistical surrogate model for the latent function, usually a Gaussian process, and an acquisition strategy to select the next candidate pair to get user feedback on. Due to the non-conjugacy of the associated likelihood, every PBO step requires a significant amount of computations with various approximate inference techniques. This computational overhead is incompatible with the way humans interact with computers, hindering the use of PBO in real-world cases. Building on the recent advances of amortized BO, we propose to circumvent this issue by fully amortizing PBO, meta-learning both the surrogate and the acquisition function. Our method comprises a novel transformer neural process architecture, trained using reinforcement learning and tailored auxiliary losses.
On a benchmark composed of synthetic and real-world datasets, our method is several orders of magnitude faster than the usual Gaussian process-based strategies and often outperforms them in accuracy.

Decision: accept (spotlight)
Reviewer scores: 8, 8, 8, 8, 5

Reviews and discussion:
<thread>
creation date:
1730251001173

thread_type:
official_review

<thread_object>
creator: reviewer_do2x

<content>
summary:
This paper introduces the Preferential Amortized Black-Box Optimization (PABBO) approach, which learns acquisition function values for candidate pairs based on preferential feedback between pairs of designs. The method utilizes a transformer-based architecture and a reinforcement learning-based pretraining scheme.

strengths:
The PABBO architecture achieves end-to-end training for preference feedback and employs an application of amortization within a preferential Bayesian optimization (PBO) setting. The algorithm’s contributions are demonstrated through experiments against Gaussian process-based benchmarks.

weaknesses:
This paper seems to fall below the standard expected at ICLR for the following reasons:

1. A critical issue is that the paper lacks a theoretical foundation or even a mathematical intuition for its approach, focusing primarily on reporting numerical results without a deeper analytical context.

2. In terms of experiments, the explanation of the architecture design and the choice of hyperparameters requires more clarity and justification. I will outline these concerns in further detail in the questions below. 

3. Finally, the paper would benefit from additional polishing. For instance, in line 189, y_{i,1} and y_{i,2} appear without prior explanation. In line 219, the term “P” in “the rest of P queries” may be confusing for readers. In line 718, For in "M=50 For" should be for.

questions:
About the architecture:
1. Why did you choose the Gaussian distribution in equation (4)? What's the role of the Gaussian distribution in the algorithm's success for the out-of-distribution case in synthetic experiments of section 5.1 and human preferences in section 5.3? Are there any mathematical intuitions?
2. What's the influence of D^{ctxpred} in your training procedure? How do you determine the percentage of D^{ctxpred} and D^{tarpred}?

About the hyperparameters:
1. Why do you fix lambda=1 in all experiments? What's the influence of lambda?
2. In line 500, why do you choose gamma=0.5, 0.9, 0.98, 1.00? These choices seem unusual and may have been selected deliberately, potentially compromising the generality of the results.
3. What's the influence of query budget T?
4. What would high dimensions influence the algorithm? For example, what would happen if you chose the D to be very large in line 719? 

I would be willing to increase the score if the above questions could be well clarified.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730570730029

thread_type:
official_review

<thread_object>
creator: reviewer_x3e8

<content>
summary:
This paper introduces a new approach to preferential Bayesian Optimization, using large-scale pretraining to enhance inference speed and optimization performance. PABBO employs two distinct transformer heads: a Prediction Head that captures the underlying function determining preferences, and an Acquisition Head that selects the next query pair. Experimental results show that PABBO outperforms existing methods in accuracy and computational efficiency.

strengths:
1. The idea of utilizing meta-learning for Amortized Optimization in preferential Bayesian Optimization is new and reduces the cost of inference.

2. Experiment results show that PABBO outperforms other methods in the datasets selected in this paper.

weaknesses:
1. The pretraining in this method requires a large amount of data, which could be costly for certain tasks.

2. PABBO is currently limited to tasks with a fixed input dimensionality.

3. PABBO only supports pairwise preference queries, limiting its ability to handle queries with multiple options, which could reduce the efficiency of information gathering. While existing works like [1] could handle queries with multiple options.

4. Including more datasets in the experiments would provide a more comprehensive evaluation. For example, incorporating all acquisition functions and tasks from [1] and other related works could strengthen the comparison. 

Reference:
1. Astudillo, R., Lin, Z.J., Bakshy, E. and Frazier, P., 2023, April. qEUBO: A decision-theoretic acquisition function for preferential Bayesian optimization. In International Conference on Artificial Intelligence and Statistics (pp. 1093-1114). PMLR.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730748783340

thread_type:
official_review

<thread_object>
creator: reviewer_drx5

<content>
summary:
This paper presents PABBO a preference based Bayesian optimization approach using transformer architectures and deep sets approaches to amortize the inference time. The paper extend ideas of amortization from past works to preference based optimization and presents an end-to-end trainable approach.

The main idea behind the approach is to use a transformer and attention mechanisms on historical observed data and use them to make query predictions. Another prediction head from the same transformer produces the policy function which is used to suggest the future queries. Experimental results indicate that the proposed approach is faster compared to previous approach and achieves a smaller regret.

strengths:
Strengths:
- The paper presents a new approach to preference based BO using transformers which as several advantages - it can incorporate preferences, has amortized inference cost and can be trained end-to-end.
- The proposed approach is intuitive and a reasonable way to model preferences in BO, with the potential to scale to very large problems. The method uses neural networks instead of Gaussian processes which helps it scale to a large number of data points.
- Extensive experimentation shows that PABBO has a smaller inference cost while achieving a smaller regret compared to prior approaches.

weaknesses:
Weaknesses:
- While the proposed approach is faster than BO, the scale at which the experiments are performed (100 observations max), the gains are not fully realized since BO methods can perform quite comfortable at such small scales.
- The full potential of the algorithm is only realized in large experiment with thousands of observations. Such large scale experiments have not been presented in the paper.
- Preference based optimization is most useful when modeling users with varying preferences. The paper can benefit from the interesting extension of learning to model individual user preferences.

questions:
General questions
- How does preference based optimization work when users have varying preferences? Is it possible to incorporate personal user-preferences in this framework.

Specific questions
- Line 237, Is the same MLP applied to x_i,1, x_i,2 and l_i individuall? Why should l_i be encoded into the same embedding space as x_i,1 and x_i,2? Or does the statement mean to encode the concatenation of the triple using an MLP?
- Line 267, It seems that to train $\pi_\theta$ the reward used is given in line 305 defined as the maximum of the observations so far. This part is unclear to me.
  - How is this reward helping in learning a good policy?
  - It seems that the loss function is static given a history H_t, so it is not a RL setup but simply a supervised learning setup. Is any RL specific learning procedure being used here? Are the reward values r_t fixed observations or are they learnable and also backpropagated through to the prediction values y and consequently through the prediction head?
  - This step looks very similar to offline BO where a offline data set is used to learn an acquisition function or a function approximation. What is the relationship of this method to offline BO?
- Line 303, How is the query set constructed when computing $\pi_\theta$? Is it a random subset of all query pairs? Or is any specialized strategy used to locate the most promising queries?
- How is PABBO able to avoid sub-optimal solutions caused due to not exploring the whole optimization space?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730932470059

thread_type:
official_review

<thread_object>
creator: reviewer_xx1d

<content>
summary:
This paper explores preferential black-box optimization with an amortized inner-loop solver. In other words, RLHF but from the lens of Bayesian optimization instead of NLP.

strengths:
This work is well motivated. Aligning training objectives with downstream use makes sense (i.e. the explicit acquisition head) and amortization also makes sense if you expect to solve the same (or nearly the same) optimization problem many many times.

The paper is clear and well-written

The experiments hit pretty much all the key points I would expect

weaknesses:
This paper seems to treat simple and cumulative regret as interchangeable, although I am sure the authors know the difference. If the policy is optimized to minimize cumulative regret, why is cumulative regret not reported? On the other hand if simple regret is really what you care about, why are none of the baselines aimed at best-arm identification? For example, Thompson sampling is optimal w.r.t. cumulative regret, not simple regret. Top-two thompson sampling is optimal in the discrete case [1], although translating to the continuous case requires some thought (what epsilon is sufficiently large to constitute a different arm?).

I also feel that the experimental design could be more directed at identifying under what conditions this solution makes sense. What is the breakpoint when amortization starts paying off compared to direct search at test time? How far can you push this before it breaks, especially in terms of the initial data package?

It also seems very odd that preference tuning for language models is not mentioned even once, even though it is obviously the same problem and basically the same solution (ignoring low-level implementation choices). I can't tell if this is a deliberate choice that reflects the legal sensitivity around LLMs right now or a genuine oversight, however I see no reason not to draw the connection to preference tuning work.

I generally try to avoid policing language, but sentences like "... amortization ... has emerged as an almost magical bullet solution" clearly crosses the line from excitement to needless hype. Amortization is not anything close to a magic bullet. You know that and I know that. It is just moving compute around from test time to train time. Of course science involves some marketing, but language and framing like this hurts our credibility with more staid scientific disciplines, people that you presumably would like to take you seriously. Write like a scientist, not a salesman. It's ok to be excited, just don't get carried away!


References

- [1] Russo, D. (2016, June). Simple bayesian algorithms for best arm identification. In Conference on Learning Theory (pp. 1417-1418). PMLR.

questions:
Can you add a reasonable interpretation of top-two thompson sampling in the continuous case to your experiments by the end of the rebuttal period? 

Are you warm-starting the amortized solver training as the data changes? Do you lose performance because of that choice? How expensive does the test-time search have to be to warrant amortization? How carefully have you thought about the FLOPS involved here?

How much can you push this? I don't just want to see rosy results, I want to know when this breaks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731016300504

thread_type:
official_review

<thread_object>
creator: reviewer_tep5

<content>
summary:
This work introduces a novel approach to Preferential Bayesian Optimization (PBO) by fully amortizing the optimization process. Traditional PBO methods rely on GPs for modeling user preferences between design pairs, requiring extensive computational resources due to approximate inference for non-conjugate likelihoods. PABBO addresses this challenge by employing a transformer-based neural architecture and reinforcement learning to learn both the surrogate model and acquisition function end-to-end, significantly accelerating the optimization process. By pre-training on synthetic and real-world datasets, PABBO achieves several orders of magnitude in speed improvement while often surpassing GP-based methods in optimization quality.

strengths:
- The paper is well written and easy to understand for the most part.
- The proposed method is a novel and interesting application of amortized learning/BO for PBO setting. The empirical results seem superior than traditional PBO methods and acquisition functions.
- Nice set of experimental evaluation and ablation studies — though I’d love to see a more comprehensive experimentation section (see weaknesses below) for methods that are hard to provide theoretical guarantees.

weaknesses:
- Evaluation on harder problems are very limited. Most test functions are of very small dimensions and may not resemble real-world optimization tasks. The only experiments with moderate dimensionalities are 6 and 9-dims respectively and are all from the HPO-B. This leaves the scalability of the proposed methods in question.
- No ablation studies on meta-learning training set. It is plausible that amortized learning methods are highly sensitive to the selection of pre-training data. There’s no justification or ablation on the choice of pre-training data for PABBO and it’s unclear how robust the method is to a different set of pre-training data.
- The performance advantage vs baseline methods are not significant in many cases.
- While it is true that estimating the true posterior of a preference model’s posterior might be expensive, variational inference or Laplace approximation with some clever implementation can go quite far and allows for gradient-based optimization for acqf value. On the other hand, for larger models like PABBO1024, the inference time advantage of amortized learning might be diminishing when compared to GP-based model according Figure 3 and 5.

questions:
- S in query set is from a Sobol sample. How big is sufficient? How would this affect the optimization quality as even the largest experimented S=1024 might not be sufficiently for moderately high-dimensional spaces, limiting the scalability of this PABBO.
- How does PABBO work for higher dimensional problems? All experiments presented are of very small dimensions and may not resemble real-world optimization tasks.
- Algorithm 1 seems to only describe the meta-learning/pre-training part of PABBO as we are still updating the PABBO model at the end of the loop. What’d be the algorithm for applying PABBO on new, unseen dataset? I’d assume that’d be the same as the inner loop of the algo, but it’d be nice to have some clarification from the author.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732043494340

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for their careful consideration and the time they dedicated to our submission. The reviewers agreed on the following strengths of the paper:

**Methodology:** ''The proposed method is a novel and interesting application of amortized learning/BO for PBO setting'' (Tep5, x3e8), ''The proposed approach is intuitive and a reasonable way to model preferences in BO'' (dRx5). ''Aligning training objectives with downstream use makes sense  and amortization also makes sense if you expect to solve the same (or nearly the same) optimization problem many many times.'' (Xx1d).

**Relevance:** ''This work is well-motivated'' (Xx1d).

**Empirical evaluation:** ''The experiments hit pretty much all the key points I would expect'' (Xx1d), 
''Nice set of experimental evaluation and ablation studies`` (Tep5),
''Extensive experimentation shows that PABBO has a smaller inference cost while achieving a smaller regret compared to prior approaches''. (dRx5, x3e8)

**Clarity/Presentation:** ''well-written'' (Tep5, Xx1d),  ''easy to understand for the most part'' (Tep5).

--------

**Additional experiments:**

All reviewers made remarks that can be answered partly by carrying out further experiments. These include but are not limited to, ablation studies related to PABBO's hyperparameters, high-dimensional settings, higher number of iterations, and varying the composition of the pre-training dataset.
We will try to carry out all experiments in the allocated time, and we thank the reviewers for giving us actionable insights. These can only improve the current submission.

This being said, due to limited computational resources, some performing additional experiments takes time, and some of them might not terminate in time. In the interest of making the most of the discussion period, we would still like to engage in a fruitful discussion with the reviewers. Therefore, we begin by answering the questions that do not require additional experiments in separate comments.

--------

**Updated PDF of the submission:**

Finally, as authorized by ICLR, we are currently preparing a revised PDF and will let you know as soon as it is fully ready.
Currently, the PDF includes an additional algorithm describing how PABBO operates at *test-time*, as suggested by Reviewer Tep5. (Appendix A.1), and a justification for why we mainly report single regret during the experiments. Additionally, The PDF will:

- Contain the additional experiments performed, in a dedicated section in the appendix.
- Include an extended related work section in the appendix, focusing on preference tuning for large language models. While this area was not a primary focus of our study, we recognize its relevance to our research and its potential to inspire extensions of our method, and we thank the reviewer Xx1d for pointing us toward this literature.
- Elaborate on the embeddings used by PABBO, specifically the use of component-wise addition. 
- Add cumulative regret plot, as suggested by reviewer Xx1d, in the appendix.
- Fix the typos identified by Reviewer do2x, and generally employ a more measured tone, as suggested by Reviewer Xx1d.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732367631174

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Comment 2 - Updated PDF and experiments

comment:
We have now edited the PDF. **Changes are highlighted in red**. They are either from reviewer requests or can be seen as the output of the engaged discussions. Furthermore, multiple experiments requested by reviewers have been added to the appendix. For commenting on the results, we respond in individual comments. Overall, changes include:

- A more measured tone regarding the sentence L45 as suggested by (Xx1d).

- Definition of $y_{i,1}$ and $y_{i,2}$ L189 (do2x).

- Justification on why using a Gaussian Likelihood for Equation 4 L277 (do2x).

- An algorithm to describe how PABBO operates at test-time, presented in Appendix A.1.1 
and referred to in the main text at L299.

- A paragraph of related work on preference tuning for language models L351 (Xx1d).

- A brief description of how the baseline qTS works L376 (Xx1d).

- A sentence elaborating on the fact that while we use (discounted) cumulative regret during pre-training, our main interest is simple regret, although for completeness, both are reported. L402 (Xx1d).

- A supplementary section Appendix A.1.2 providing additional justifications on our transformer architecture, namely the use of three separate embeddings for $x_{i,1}, x_{i,2}$ and $l_i$, with a component-wise-sum to obtain the token embedding for $x_{i,1},x_{i,2},l_i$ (dRx5).

- An additional experiment on the composition of the pre-training dataset in Appendix A.2.1, Figure 6 (Tep5).

- Cumulative simple regret plots for all test functions in Appendix A.2.2, Figure 7 (Xx1d).

- An additional experiment studying the impact of the loss hyperparameter $\lambda$, controlling how important the prediction task should be, in Appendix A.2.3, Figure 8 (do2x).

- An additional experiment on the discount factor $\gamma$ that uses a larger grid for $\gamma$, and also reports cumulative simple regret, in Appendix A.2.4, Figure 9 (do2x).

- A higher dimensional experiment using HPO bench and search space No. 5971, which is $16$-dimensional, in Appendix A.2.5, Figure 10 (Tep5).

- The addition of another baseline, Multinomial Predictive Entropy Search (MPES), and its evaluation on synthetic examples, presented in Appendix A.2.6, Figure 11 (x3e8).

- An additional test function, the 6-Dimensional Ackley function. In particular, for this example, to address the concerns regarding query set size scalability (Tep5), we include a ''batch'' version of PABBO. This experiment is available in Appendix A.2.7, Figure 12 (x3e8).

- An additional experiment on the query budget $T$ which considers different values of $T$, and also compares two versions of PABBO: with or without giving $t/T$ as an input to the acquisition head (see Figure 2 right). This experiment is available in Appendix A.2.8, Figure 13 (do2x).

- An additional experiment on the percentage of $D^{\text{ctxpred}}$ and $D^{\text{tarpred}}$. We fix the $D^{\text{ctxpred}}$ and run the prediction task with different $D^{\text{tarpred}}$. This experiment is available in Appendix A.2.9, Figure 14 (do2x).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work proposes an approach to generating amortized policies for preferential Bayesian optimization (PBO) using transformer neural processes and reinforcement learning.  In their experiments, they appear to be able to achieve parity with established methods for PBO.  The meta-training procedure allows the authors to pre-train off of relevant priors which can lead to improved sample complexity of the algorithms (e.g., meta-tasks from HPO-B benchmarks).

Reviewers were overall quite positive, finding the paper to be well-written, clearly motivated, and well-executed.

All reviewers expressed some amount of skepticism of the results that could be addressed via more rigorous experiments and modest framing.  This is interesting work that will be published and read, so the authors have the opportunity to demonstrate what the limitations are (Xx1d). Exposing such limitations will ultimately benefit the community and will help stimulate future directions for this line of work.

Some reviewers expressed concerns that runtime improvements from PABBO may be overstated (Tep5) (e.g., S=1024 appears necessary, for the highD problems, but illustrations of runtime comparisons in the MT focus on the case of S=256 (Fig3)—which is orders of magnitude faster than S=1024 (Fig 5c)).   Reviewer Xx1d asked for further contextualization of the runtimes.

There are a few remaining gaps.  First, the authors do not run enough replicates to demonstrate whether certain methods are statistically significantly better than others (see cf., review+discussion w/ Tep5).  In fact, it’s somewhat ambiguous what statistics are reported in the figures: are the authors plotting ± 1 standard deviation, or ±1 std of the mean (standard error of the mean, SEM).  It is common practice to report the 95% CI for the mean (i.e., 1.96*std/sqrt(num replications)).  Please update your figures to use at least 30 replicates and report 1.96*SEM for the intervals.

Reviewer Tep5 also asked about sensitivity to the pre-training and whether the advantages of PAABO in terms of sample complexity/runtime would hold if alternate models (such as the pairwise kernel+VI from Astudillo et al.’s qEUBO paper) were used.  The responses from the authors were not satisfactory to the reviewer (as well as the AC).  Addressing these points in the CR will strengthen the work.

I have also read the paper in some detail, a few brief comments:
- The updated PDF does not adequately explain how qTS works.  It sounds like you are optimizing the sample path, vs doing discrete sampling (say, using the same S used by PAABO).  Please clarify the exact procedure, and if optimizing sample paths, describe what method you are using (e.g., RFFs, pathwise conditioning, etc).
- Why is only S=1024 used in the higherD (HPO-B) examples in Fig4, but it is evaluated with lower cardinality S for the lowerD examples (Candy/Sushi).  The authors discuss how larger S might be necessary for higherD problems (see comments from Tep5). Including these lower S’s in this plot, or having a more in-depth ablation could help make this point better).  Reporting the wall time of the methods here would also be instructive, since the wall time improvements over GPs may be much thinner with S=1024.
- The superior sample complexity of PAABO is by far most apparent on the HPO-B datasets where (to the best of my understanding), PAABO the model / policy is pre-trained on the meta-task itself.  Does PAABO’s improvement lie in learning a better policy, or having a better prior?  Fig6 from the rebuttal does not address Reviewer Tep5’s point.  Running the HPO-B benchmarks with the standard prior used for the synthetic functions would address this question.
- The motivation for the HPO-B datasets is not very clear, since it's not obvious why one would want to do PBO with HPO datasets.  Some additional discussion here would be helpful.  It would also be nice to see more standard or real-world motivated PBO problems from previous works (Tep5 gives some suggestions).