PABBO: Preferential Amortized Black-Box Optimization
Authors: Xinyu Zhang
Daolang Huang
Samuel Kaski
Julien Martinelli
Venue: ICLR 2025 Spotlight (2025)
Keywords: Bayesian optimization
preference learning
amortized inference
neural processes
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We present the first fully amortized preferential black-box optimization framework, featuring a novel transformer-based neural process architecture.

Abstract: Preferential Bayesian Optimization (PBO) is a sample-efficient method to learn latent user utilities from preferential feedback over a pair of designs. It relies on a statistical surrogate model for the latent function, usually a Gaussian process, and an acquisition strategy to select the next candidate pair to get user feedback on. Due to the non-conjugacy of the associated likelihood, every PBO step requires a significant amount of computations with various approximate inference techniques. This computational overhead is incompatible with the way humans interact with computers, hindering the use of PBO in real-world cases. Building on the recent advances of amortized BO, we propose to circumvent this issue by fully amortizing PBO, meta-learning both the surrogate and the acquisition function. Our method comprises a novel transformer neural process architecture, trained using reinforcement learning and tailored auxiliary losses.
On a benchmark composed of synthetic and real-world datasets, our method is several orders of magnitude faster than the usual Gaussian process-based strategies and often outperforms them in accuracy.

Published as a conference paper at ICLR 2025
PABBO: PREFERENTIAL AMORTIZED BLACK-BOX OPTIMIZATION
Xinyu Zhang", Daolang H Samuel Kaski Department of Computer Science Aalto University x xinyu. wwwnggaalto........
Julien Martinelli
Inserm Bordeaux Population Health
Vaccine Research Institute
Université de Bordeaux
Inria Bordeaux Sud-ouest
ABSTRACT
Preferential Bayesian Optimization (PBO) is a sample-efficient method to learn pair of designs. It relies
latent user utilities from preferential feedback statistical surrogate model for the latent function, usually a Gaussian process, and acquisition strategy to select the next candidate pair to get user feedback on. Due to the non-conjugacy of the associated likelihood, every PBO step requires a significant amount of computations with various approximate inference techniques. This computational overhead is incompatible with the way humans interact with computers, hindering the use of PBO real-world cases. Building on the recent advances of amortized BO, we propose to circumvent this issue by fully amortizing PBO, meta-learning both the surrogate and the acquisition function. Our method comprises novel transformer neural process architecture, trained using reinforce- ment learning and tailored auxiliary losses. On a benchmark composed of synthetic and real-world datasets, our method is several orders of magnitude faster than the usual Gaussian process- based strategies and often outperforms them in accuracy.
INTRODUCTION
Learning the latent utility function of given user from its feedback is a task that is becoming increasingly important in the era of personalization, with applications ranging from the optimization of visual designs (Koyama et al., 2020), thermal comfort (Abdelrahman & Miller, 2022), proportional integral controller (Coutinho et al., 2024), or robotic gait (Li et al., 2021). As humans are typically better at comparing two options rather than assessing their absolute value (Kahneman & Tversky, 2013), current solutions often leverage data based on preferential feedback, the outcome of a pairwise comparison also referred as duel (Chu & Ghahramani, 2005).
Preferential Bayesian Optimization (PBO, González et al. (2017)) is the gold standard method for optimizing black-box functions from sequential duel feedback. To learn the latent utility of human subject, PBO operates a Bayesian framework and classically relies on Gaussian Process (GP) prior (Rasmussen & Williams, 2006) and probit logit likelihood (Brochu et al., 2010). Due to the non-conjugacy of the preferential likelihood, approximation techniques are required for posterior inference (Takeno et al., 2023). The obtained posterior is then plugged into acquisition function, which selects the next pair of candidate designs to display to the user, in a way that balances exploration and exploitation (Garnett, 2023). Approximating the posterior and maximizing the acquisition function results in a significant computational overhead, which may hinder the use of PBO in real-world cases, as the user would have wait an extended period of time before each iteration.
To circumvent this issue, amortization, or pre-computing solutions and learning an ML model to approximate them, has emerged as a promising solution, effectively enabling rapid and still accurate on-line computation with sophisticated models, trading off the speed to off-line computations. Amortization has been successfully applied to simulation-basee inference (Cranmer et al., 2020; Gloeckler et al., 2024), sequential Bayesian experimental design (Foster et al., 2021), and more relevantly, vanilla Bayesian optimization (Chen et al., 2017; Maraval et al., 2023; Song et al., 2024).
Equal contribution.

---

Published as a conference paper at ICLR 2025
Pre-training stage
Fast and reliable candidate pairs generation
Pre-training Datasets
PABBO
Prediction Head
Acquisition Head
PABBO
Saadkm 42UBO
423
000
Simple regret
Transformer
20
Data Embedder
20f
Inferencc time (8)
generate
update history
100
query
Candidate pairs
Preference
Interaction steps
15
>0
30
Preference Feedback
End-to-Bnd End training
Method
Amortization
Reference
BO
(Gamett, 2023)
gEUBO. qEL QNEI
(Astudillo al., 2023; Süiola al., 2021)
qTS. MPES
x
(Hemandez-Lobato al., 2017: Nguyen et al., 2020)
PFN
X
(Müller et al., 2023)
NAP
X
(Maraval et al., 2023)
PABBO
This work
Figure I: Leveraging synthetic and existing BO datasets, PABBO learns an acquisition policy in an end-to-end manner, thus directly amortizing the design proposal step. Contrary to existing methods, at inference time, PABBO only relies on preference data. Our method outperforms GP-based strategies and dramatically speeds up the optimization.
Beyond speed gains, interesting characteristic of amortization with regards to BO and PBO is that can jointly model the probabilistic surrogate and the acquisition function, two modules that are usually treated independently in BO. This provides two powerful advantages: rather than guessing which combination of GP kernel and acquisition function works best for given task through various heuristics, these are now learned from data. In the end, the model will directly predict the acquisition function values. However, current and-to-end black-box optimization methods predict acquisition function values for single designs. Translating this task the preferential framework requires predicting acquisition values for pairs of designs, suggesting novel network architectures and training procedures. To the best our knowledge, amortization has not yet been employed in a PBO setting.
Contributions.
We introduce PABBO, the first end-to-end framework for Preferential Amortized Black-Box Optimization (Figure 1). PABBO directly outputs acquisition function values for candidate pairs, conditioned on observed data, and handles preferential feedback through a novel transformer- based architecture.
We present a reinforcement learning-based pre-training scheme to directly learn acquisition function values for pairs. Due to the sparsity of the rewards, in similar spirit as Maraval et al. (2023), we add an auxiliary loss, but based on binary classification instead of regression, due to the preferential nature of the data. Figure I displays the PABBO workflow.
On set of synthetic and real-world examples, PABBO provides speedups of several orders of magnitude against GP-based methods, often outperforming them, even on unseen examples during pre-training, thus demonstrating its impressive in-context optimization capability. We conclude by illustrating the robustness of PABBO through a series of ablation studies.
2

---

Published as a conference paper at ICLR 2025
2 PRELIMINARIES
2.1
PREFERENTIAL BLACK- -Bx OPTIMIZATION
The goal of preferential black-box optimization (PBBO) is to determine the optimal solution x" arg m f()) using the duel feedback æ x', signifying preference of over x'. The preference structure is governed by the latent function f - R deñieed over the subset X TR:
x > x' = f(x) + f(x') +E',
where and e' are i..dd realizations of the normal distribution (/0, o
By definition of PBBO, we only have access to a dataset made of the human observations m duel feedbacks [4:) :=1 {æ, > æ rr)::- with a and a being the winner and the loser of the duel, respectively, and a binary variable recording the location of preferred instance in a query. No information about the latent function is provided, e.g., its analytic form, convexity, regularity.
2.2 AMORTIZATION
Amortization is data-driven approach that involves training a model, often a neural network, on large set of pre-collected tasks. The goal is to enable the model capture and leverage shared information across these tasks, thus learning how to make predictions efficiently on similar tasks future. This process allows the model to perform approximate inference rapidly during the inference stage, since knowledge from related tasks encountered during training has already been internalized.
Recently, amortization has been introduced to BO, either amortizing the surrogate model (Müller et al., 2023; Chang et al., 2024), the acquisition function (Swersky et al., 2020; Volpp et al., 2020), or directly end-to-end training system (Chen et al., 2017; 2022; Yang et al., 2024; Maraval et al., 2023). Our work aims to employ a fully end-to-end trained system that learns how to optimize functions directly from preference data.
architecture is crucial to efficiently utilize data in amortized learning. Conditional neural processes (CNPs; Garnelo al. 2018), meta-leamiin framework built on Deep Sets (Zaheer et al., 2017), are particularly well-suited for tasks like BO (Maraval et al., 2023; Müller al., 2023), where historical queries are permutation-invarian processes (Nguyen & Grover, 2022; Müller et al., 2022; Chang et al., 2024) extend this idea by leveraging the attention mechanism to more effectively capture interactions between points.
Designing a task-specific
Transformer-based neural
2.3 REINFOR LEARNING
Reinforcement Learning (RL) has recently found success in the Black- -ox Optimization realm (Volpp al., 2020; Hsich al., 2021; Maraval et al., 2023), due to its ability to yield non-myopic acquisition strategies. This leads to better performances since these strategies consider the influence of future evaluations up to a given horizon determined by the available budget T. RI, problem is defined as Markov Decision Process (MDP) M = (S A. P, R, 7), S and A stand for the state and action spaces, respectively, P S x Ax - [o, 1] is the state transition model, and R is the reward function which encompasses a given optimization goal. The goal is to learn a probability distribution over stateaaction pairs π (a))s;), referred to as a policy, such that v maximizes the discounted expected returns with discount factor [0, 1]. which defines the degree of myopia of the policy.
3 PREFERENTIAL AMORTIZED BLACK-BOO OPTIMIZATION (PABBO)
To solve the problem introduced in Section 2.1 in an amortized manner, PABBO proceeds as illustrated in Figure 2. In brief, PABBO directly outputs acquisition values for candidate pairs through the policy na. which is parameterized by transformer backbone and acquisition head. To stabilize the training, we additionally employ a prediction head for an auxiliary preference prediction task. Section 3.1 describes the MDP we employ for learning the policy æT, Section 3.2 details the conception of the pre- pre-training dataset accordingly to the neural process framework, and Section 3.3 focuses on the transformer architecture and the loss functions involved in its actual training.
3

---

Published as a conference paper at ICLR 2025
(1teranet) (pwn)) Mi-9 Prediction
Policy By
Acquisition head.
Trans Trunsformer former Black fun
State Mr
Acquisition
Action (aa
2
177
AAO)
Fndoddder feets
Transformer Block fise
Paa
Prediction Set
Prediction Head
Reward y
gaappedd
gr
Dioigroll (imppodd
Data Embedder
1 2r1 Contest Set
Shuffle
go/ U (eyypeo Prediction Set
2090 Query Set
Ausilaar (arpard) phurpost)
BCE (1.
Audiliay Prediction Task
Policy learning
Flowehart of PABBO (left), together with a zoom on our proposed transformer block (right). We use reinforcement learning to guide the acquisition head in proposing valuable query pairs, and apply an auxiliary BCE loss to the prediction head to stabilize the training of the transformer.
Figure 2:
3.1 POLICY LEARNING
Unlike current PBO methods, we leverage RL to learn a non-myopic policy zg. For a single task, our RL problem is formulated as follows:
States: 8t [He, I, 7], where Hi = {(0,,,, 4,), contains the optimization history made of (1 1sttep queries and preferences.
Actions: O₃ (æx.1 Zt,2), the next query pair.
Rewards: Tz maxisist max 1 31 the best objective function values observed far with Vi,1 and Vi.2 being associated to inputs 21,1 and 2,,2, respectively.
State transitions consist in updating the optimization history with the selected next query and preference feedback, He+1 = Hi U {(x,,1, 22,2,4) We seek policy maximizing the rewards along the entire optimization trajectory while performing well on average across tasks sampled from Plasks Rewards denote the maximal objective function values observed at step 1. Such values are always available in synthetic pre- pre-trainigg datasets, like GP draws. the latent function values unavailable during pre- pre-training, ranking data can be used to obtain a score for each point as the reward. We show in Section 5.4 that this substitution does not greatly alter performance.
3.2 PRE-TRAINING DATA ARCHITECTURE
The PABBO workflow involves pre-training phase, where policy is learned on all meta-tasks, and an inference phase, where the trained model is applied to predict acquisition values on a new task directly. Our approach effectively amortizes the optimization process over multiple tasks, allowing rapid inference by leveraging the shared structure learned during pre- gre-training. The pre-training dataset for each meta-task features three parts:
A context set ((e) = {( 2.2 ( ME containing all past queries and preferences.
A query set D(4) = { " consisting of S locations that the model considers for the next step, which will be further combined into candidate query pair Q (xx (e))))
a.()) 2x E , j} of size M. In a training iteration, we randomly sample S query points, and candidate query pairs are chosen from all possible pairwise combinations within this set. At inference time, query points are sampled across the search space from quasi-random Sobol sequence, and all possible pairs from the full set of combinations are considered. S should be set to a sufficiently large value and scaled with data dimension to ensure adequate exploration.
A prediction set D(2) = (< 42 (P) ) containing N queries and preferences. Sim-
(p) (p)
ilarly, we generate (()) by sampling pairwise combinations of a set of random points. This prediction set is used to compute auxiliary binary classification loss: for each op- timization step, we randomly sample a subset of D(p) which refers as prediction context set (ecxpred)
{ ((x(pre)))) 22.2 f
(ctxpred) (etspred) f(ctspeed) ) ( and all the individual points from the
4

---

Published as a conference paper at ICLR 2025
remaining
P N (etspred) pairs are collected as prediction target set D((urved)) The model is required to estimate the function values for each point (turpredd conditioned on D(cixpred) and further recover the preference relationship of pair.
(ar) (ar (tarpred) )i=l |
It is worth mentioning that no reward information can leak from the query set to the prediction set as entirely different instances are assigned to each set.
3.3 MODEL ARCHITECTURE
PBBO differs from standard BO in two aspects: (1) preferential outcomes instead of exact latent function values are observed, and (2) a pair of points is expected as the next query instead of a single point. To address these challenges, PABBO leverages novel data encoding and decoding structures, efficiently handling pairwise preference data. At the center of PABBO'S 's architecture lie conditional neural processes, class of models quite amenable for handling the sets that naturally arise due to the permutation invariance of the optimization history. Specifically, the PABBO model architecture, presented in Figure 2, consists of four main components: the data embedder femb, transformer block film, prediction head fp. and the acquisition head fa-
Firstly, inputs from all parts of the dataset are mapped into shared embedding space via the data embedder femh. The first and second instance within a pair I. F22 and their corresponding preference l are encoded separately with individual ML.ss into embeddings of the same size and added up as E = E( @ E( @ E For parts D(g) and ((tarpred) containing only a single instance 2;, E = E(m.), This gives us embeddings E(e) E((), E(etxpred) and (tarpred) corresponding to D(o), D(9), D (stxpred) and ((tarpedd) respectively. More details and a discussion related to the choice of this particular architecture can be found in Appendix A.
Next, embeddings are passed through the transformer block fifm. which captures the interactions between different parts of a meta-dataset. Dependencies between elements are controlled with masking in self-attention layers of fimm (see Figure S1). Precisely, D(e), the observations collected by the current step, can be attended by other elements in Díc) and all the query points in D(4), while each query point can only be attended by itself. Our interest is the Transformer outputs for query set A (4) [um (E(+)). The same rule applies for the prediction context set (eexxprod) and prediction target set D (tarpede)) leading to the transformer outputs A (tarpred) = fim (E (trrped)). These outputs = A1 A (turpeed) are then processed with different decoders according to their specified task.
Subsequently, ( are combined into pairs {(A(s) (4) A((0) (a/
(2) E Q)) according to the candidate query pair set Q". Each pair (2(9), (4) passes through the acquisition head fa along
with current step and budget T to predict the acquisition function value q. fa(A( (q) > ,T,T) for associated query pair. This enables creating policy for proposing the next-step query (It,1, 21,2) from Q, mapping outputs of fa. (qr
((a])) 2 )EQ), to a categorical distribution via SoftMax:
(9) (4)
To( (2) x He. T,
Exp 4:.j
(2)
En Eq exp 4m
As defined in Section 3.1, the best objective function value obtained so far is chosen the immediate reward when training this policy, since the goal is to find the global optimum as efficiently as possible. To further pursue non-myopic solution, we consider the cumulative reward across the entire query trajectory, incorporating a discount factor 7, which controls the rate at which future rewards are discounted. The loss writes as the negative expected reward over the entire optimization trajectory:
((4)
(o) (g) rr log To(( 1. 22. 144, t, T)).
1=1
In addition to the forward process fa. reward sparsity is handled by introducing an auxiliary path leading to the prediction head fp. encouraging the learning of latent function shape from preferences. For each prediction target point x; (tarpred) (tarprred) .p parameterizes a Gaussian distribution:
p [2 (tarpeed) ((rpp)))))
= =N (larpred), (mi o;) fp (>. (lappee)
(4)
To reduce computational costs from evaluating all the possible pairwise combinations between query points, which are of size O(S²). ) we randomly sample M S pairs for each meta-task during pre-training.
5

---

Published as a conference paper at ICLR 2025
Using independent Gaussian likelihoods is common in the Neural Processes community (Garnelo et al., 2018). Correlations can be accounted for if the downstream task requires it (Markou et al., 2022). A similar choice is often found in PBO, using the probit likelihood (Brochu et al., 2010).
(tarpred) that constitute query pair in D(P), the predicted latent At any two target points a;; (tarpred) and x;
function values Vi (tarpred) and yj are obtained by sampling from p. thus capturing the randomness in the process as described by Equation| 1. with the Sigmoid function. As mentioned, at each optimization step, we generate a prediction task with different contexts by shuffling the prediction set D(e), creating different splits of Distxpred) and
(tarpred)
The final preference is [(upped) := o(y, ((arpned) 1); ((uppeed)
D(tuppred) to fully utilize the data. fp is trained by minimizing the binary cross entropy (BCE) loss between predicted preferred location (atppedd and the ground truth (tarpreed) on all the prediction tasks:
P
((r)
(urpred) 4,
log( log(( (,2
((urpred)) (1
((aappeed) 4,,
) log(1 - [(tupped)) )]
(5)
1=1
ftarppodd and ((rappred) where L2,
represent the ground truth and predicted preference between a possible pair of prediction target points at step t, respectively.
44,1
The final objective C for policy learning combines the two losses: = C(a) + XCP), We fix À 1 in all the experiments, In practice, we warm up the model with only the prediction task by setting C C(P) in the initial training steps, the rationale being that the policy can only perform well once the shape of the latent function has been learned. Algorithm I describes the complete meta-learning training procedure, and Algorithm S1 describes how PABBO operates at test-time on new task.
Algorithm 1 Preferential Amortized Black-box Optimization (PABBO)
Require: K meta-tasks, candidate query pair set size M, discount factor y, query budget T for each training iteration do initialize D(P), D(a), sample Q of size M, set Hi + {0}
for t = ,,...,T
T do
>> start policy learning > sample next query pair from policy comparison of two sampled query points & collect immediate reward update history b update query set auxiliary prediction task b update prediction dataset
(It,1. x1,2) ~ ( H,,t,TT T)
4 Tt,1 > D1.2
r. =max max(11. 21
He+1 + H4 U ( 21 b+)
Q - (((2,,,,,1,)))
infer prediction target preference (topred D(P) + Dietspred) U ((urpred)
end for Update PABBO using L := ((1) + AC (Equations 3 and 5) end for
RELATED WORK
Preferential Black-Box Optimization. While PBBO has been tackled using techniques like dueling bandits (Bengs et al., 2021) or Reinforcement Learning (Myers et al., 2023; Rafailov et al., 2024), the bulk of the work points the Preferential Bayesian Optimization field. The latter mostly hinges on an approximate GP surrogate, with vastly different results depending on the approximate inference scheme (Takeno et al., 2023), and several dedicated acquisition functions have been proposed. These include dueling Thompson sampling, or EUBO, a decision-theoretic AF (Lin et al., 2022). Unlike standard BO, deriving regret bounds for PBO is cumbersome, Only one work has done so (Xu et al., 2024), although convergence can be demonstrated, e.g. for dueling Thompson sampling (Astudillo et al., 2024). Lastly, Bayesian Neural Networks were also trialed (Huang et al., 2022).
PBO has been extended to various settings: handling a batch of queries q > 2 (Siivola et al., 2021; Astudillo et al., 2023), heteroscedastic noise (Sinaga et al., 2024) and mujtii-oeeective optimiza- tion (Astudillo et al., 2024). On another note, preferential relations 21 I2 have been leveraged to enhance vanilla BO loops: Hvarfner et al. (2024) investigate a user-defined prior that is integrated to the GP surrogate, whereas Adachi al. (2024) allow the prior be iteratively updated.

---

Published as a conference paper at ICLR 2025
Amortization and Meta-Learning. Our architecture is closely related to Conditional Neural Pro- cesses (CNPs) (Garnelo et al., 2018; Kim et al., 2019; Jha et al., 2022; Huang et al., 2023), and its more recent variant, Transformer-batt baaemeer-baeddddd Neural Processes (Nguyen & Grover, 2022; Müller et al., 2022; Chang et al., 2024). CNPs have primarily focused on amortizing the predictive posterior distri- bution via supervised learning on a collection of datasets, whilst our approach employs reinforcement learning to train an optimization policy.
In recent years, multiple works have explored training neural networks directly amortize black-box optimization (Chen et al., 2017; Yang et al., 2024; Maraval et al., 2023; Amos, 2023; Song et al., 2024; Chen et al.. 2022). These methods typically rely on direct function utilities for training. Recent advancements also include the use of large language models to assist in Bayesian Optimization (Liu et al., 2024; Chen et al., 2024; Aglietti et al., 2024). However, to the best of our knowledge, no existing work has yet attempted to directly amortize optimization based on preferential feedback data.
Besides, since PBBO can be viewed as a sequential experimental design (SED) problem (Rainforth et al., 2024), our work is also related to research amortized SED, This line of work primarily revolves around Deep Adaptive Design (DAD; Foster et al. 2021), which develops a comprehensive framework to amortize experimental design. Subsequent work has extended this approach to settings with implicit likelihoods (Ivanova et al., 2021) and has leveraged reinforcement learning to further improve performance (Blau et al., 2022; Lim et al., 2022). Our work can be seen specialized application of SED with specific downstream objectives (Huang et al., 2024).
Preference tuning for language models. Finally, recent advancements in preference learning have further expanded its application in tuning the language models, particularly in tasks such as reinforcement learning with human feedback (RLHF) align large-scale models with user preferences. Christiano et al. (2017) proposes a framework for guiding policy optimization in RL with human preference feedback. which has been applied to several natural language tasks (Zaheer et al., 2017: Stiennon et al., 2020). While those works focus on relatively small models and specific tasks, InstructGPT (Ouyang et al., 2022) extends with user instructions across wide range of tasks. Recently, Rafailov et al. (2024) proposes Direct Preference Optimization (DPO), which shows that language models can be directly trained on preference data without explicit reward modeling or RL optimization, significantly simplifying the preference learning pipeline. While PABBO and these works both leverage preference feedback, they serve fundamentally different purposes: language model alignment aims to steer model behavior towards desired outputs in text spaces, whereas PABBO focuses on efficient optimization of black-box functions, requiring different architectural designs and optimization strategies.
large-scale models like GPT-3, allowing alignment
EXPERIMENTS
We evaluate PABBO on synthetic functions, described in Section 5.1, and against real-world examples from the BO literature: hyperparameter optimization (Section 5.2) and human preferences datasets (Section 5.3). Subsequently, Section 5.4 presents a series of ablation studies: replacing the latent values in the rewards by pure rankings, examining different discount factor values y in Equation 3, and varying the number of query locations on which the acquisition function values are predicted.
Baselines. Our approach is benchmarked against five GP-based acquisition strategies from the PBO realm: qEUBO (Astudillo et al., 2023), qEI/ / QNEI (Siivola et al., 2021), qTS (Hernández-Lobrr et al., 2017) and MPES (Nguyen et al., 2020). We also include a strategy where candidate pairs are acquired at random. While QEUBO is a principled preferential acquisition strategy, qEI and qTS are effectively batch BO strategies used in PBO setting, solution commonly employed in the field. For instance, TTS samples two draws from the posterior, maximizes each draw, and provides each maximizer a candidate pair for comparison. Further details can be found in Appendix D.3.
As performance metric, we consider the simple regret f(x=* - f(mbbest where best argmmax max f(xx). For pre-training, the (discounted) cumulative simple regret was used (Equation 3), sending stronger and less sparse signal than simple regret. But because our goal is human preferences optimization, e.g., finding the best sushi product for given user, we report simple regret at inference time. However, for completeness, Figure S2 reports cumulative simple regret.
Implementation. PABBO is implemented using PyTorch (Paszke et al., 2019). Hyperparameter settings can be found in Appendix D.1. For qEUBO, qBI, qNEI, we used the implementation from
7

---

Published as a conference paper at ICLR 2025
GP2D
Forrester1D
Beale2D
Bramin2D
10
107
10
Smupl Regret
10
10--
10-1
10
10*-
10P
10-22
10F
10
20
30
10
20
30
10
20
30
150
150
200
Inferetiee time [6)
130
100
100-
100-
100
100
50
30-
50
6-
0-
10
10 20
20
10
0
20
Iterations
@EUDO
qEI
GTS
ONEI
Figure 3: Simple regret and inference time on synthetic examples. Mean with 95% confidence intervals computed across 30 runs with random starting pairs. PABBO consistently achieved the lowest simple regret across all tasks, except for GP cases where it performs comparably to qNEI, while offering a 10x speedup.
the BoTorch library (Balandat et al., 2020). For qTS and MPES our implementation is described in Appendix D.3. The code to reproduce experiments is available at https: / /tgtttbttbb/////////////// com/ xinyuzc/PABBO.
5.1 SYNTHETIC FUNCTIONS
We begin by benchmarking PABBO on two types of synthetic functions. The first type involves 30 draws from the same GPs that were used to build the pre-training dataset (See Appendix C for the data generation process). We refer to this case as the in-distribution. The second type refers to classical test functions: the Forrester, Branin, and Beale functions. These functions were not part of the pre-training dataset used by PABBO, hence we refer to this case as out-of- -fitiiibiiiiii
At each training epoch, we generate B different GP samples, which a quadratic bowl added to increase the chances of a draw exhibiting global optimum. Then, we draw 200 x D points, where D the dimension æ., half of which for the auxiliary prediction task and the rest for the optimization task. For synthetic functions, the query set size was set to S = 256.
PABBO consistently ranks either as best or second best method in terms of simple regret (Figure 3). While this is expected for the in-distribution case, given its low dimensionality compared to our model architecture, the out-nf-distribution setting, PABBO also achieves the lowest simple regret across all three examples. Of note, the random strategy often outperforms certain GP baselines. Regarding inference speed, PABBO offers an average 12-fold speed-up over the fastest GP-based strategy across 5 problems, consistently over the whole trial.
Additionally, PABBO successfully recovers at least one of the three global optima of the Branin function (Figure S11, left panel), and correctly identifies the global optimum of the Forrester function (right panel). Beyond convergence to an optimum, PABBO also accurately learned the shape of each function from preference data, a modality that only allows identification up monotonic transformation.
5.2 HYPERPARAMETER OPTIMIZATION
Next, we consider hyperparameter optimization (Pineda Arango et al., 2021). This challenge emulates the task an MI. expert would face when optimizing the hyperparameter of given model. In real-
8

---

Published as a conference paper at ICLR 2025
Searc spein No. 200
Seadc spex No. 3035
beanc space No. 0652
Caudyy
Sodi
20
31
73 302 31
350
Iteralio
30
200
50
300
21BBO Starkes
<EUBO
025 -080 SIPE5
Figure Simple regret on different search spaces from the HPO-B benchmark and human preferences datasets. Mean with 95% confidence intervals computed across 30 runs with random starting pairs. Attached each PABBO baseline is a number corresponding S, the size of the query set. On average, PABBO ranks first on HPO-B tasks and second for human preferences datasets.
world scenarios, experts often find it easier to express preferences between pairs of hyperparameter configurations, rather than assigning precise scalar utilities. The HPO-B benchmark contains multiple search spaces, each corresponding to the optimization of different MI. models.
Each search space contains multiple meta-datasets from different meta-tasks, and each meta-dataset collects list of hyperparameter configurations with their response values, {(T;, 34 })){== allowing us simulate preference for any two configurations given their responses. All the meta-datasets within a search space have been categorized into three splits: meta-train meta- meta-validation, and meta-test.
We choose three representative search spaces and pre-train our model on the meta-train split associated with each space, further splitting each set into two equal-sized, non-overlapping parts beforehand for the auxiliary prediction and optimization tasks (see Appendix D.2). For evaluation, we take all points from each meta-dataset of the meta-test split of the search space, allowing us to assess the average performance of our model and the baselines across all meta-datasets.
Overall, PABBO achieves the best performance, closely followed by the QNEI baseline (Figure 4, first three panels). is worth noticing that on these higher-dimensional problems, all methods now clearly outperform the random acquisition strategy.
5.3 HUMAN PREFERENCES
We experiment with our model pre-trained with synthetic GP samples on two real-world human preference datasets: the Candy" and Sushi" datasets. The Candy dataset provides the full ranking of 85 different candies from pairwise preference via online studies. Each candy object contains 2 continuous features, asking users to rate 100 different types of sushi obtained by averaging across users. Each sushi contains 4 continuous features. Following Siivola al. (2021), we generalize both datasets to continuous space by performing linear extrapolation, and the out-of-bound values are filled with their nearest neighbors in the original dataset.
reent and pri ceperceer.. The Sushi dataset collects a preference score by five-point-scale. The overall score for each sushi
For both examples, report the simple regret achieved by model when using query of size S = 512. (Figure 4, fourth and fifth panels). On these challenging out-of-distribution preference learning tasks, PABBO remains competitive with qNEI and MPES, achieving the second and third place for the candy and sushi datasets, respectively. Both problems are characterized by a large variance in the results for all baselines.
5.4 ABLATION STUDY
This last experimental section investigates whether certain modeling choices and hyperparameter values may have significant impact on the performance of PABBO. A set of more detailed ablation studies can be found in Appendix B.3.
Access to latent values during pre-training. Even though latent values are always accessible for synthetic datasets used for pre-tras me-training. this might not be the case for real-world preferential meta datasets. simple solution for proposing rewards without access the utility is rankings from a fully ranked set to approximate the latent function values. We trained a models using a pre-training dataset of GP samples, substituting classical rewards for each meta-tas
Available powrradd--powr--ankkig Available http://wwww.amiihimaaaamm net/sushi/

---

Published as a conference paper at ICLR 2025
(a) Ablation Reward model
(a) Abiation Discount factor
(c) Ablation Queeyy sot size
Wibes Koskann
10a
10
Staple roppeet
1.00
Simple rAgUMe
101
Sumple
30-1
10
20-1
107
3304
1077
10-2
20 10 1
10
20
20
Iterations
Figure 5: Ablation studies. Mean with 95% confidence intervals computed across 30 runs with random starting pairs. On low dimensional examples, PABBO maintains similar performance when substituting latent function values for rankings, and performs as expected when varying the discount factor y and query set size S.
by the rankings of corresponding samples. On the 1-dimensional Forrester function, this alternative version of PABBO still yields decent performances (Figure 5())).
Discount factor 7. PABBO learns a policy πe by taking the future discounted rewards into account across a time horizon T. smaller discount factor prioritizes immediate queries, giving more weight to the early steps in the optimization process. In contrast, as y approaches 1, the future queries become increasingly important, reflecting a more balanced consideration of both immediate and future rewards. We test with different discount factors the 1-dimensional Forrester problem. The results in Figure 5(b) align with our expectation. When y is small, we observe faster convergence in the early stages. However, since the future rewards are not sufficiently considered, the average performance over larger steps is less favorable compared to using a larger which balances both immediate and future rewards, resulting in better long-term optimization outcomes.
Query set size S. At inference time, a continuous search space is discretized into S query locations sampled from quasi-random Sobol sequence. used build set of candidate query pairs consisting of all possible pairwise combinations, resulting in O(5²) query pairs. S introduces a trade-off off between performance and inference time: denser grid guarantees adequate exploration of the search space, but longer inference times. For the 2-dimensional Branin problem, increasing the grid size S E [128, 256, 512, 1024] leads to increased inference time, a consequence of the need to embed an increasing number of candidate pairs (Figure 5(c)). In terms of simple regret, grid of size S = 1024 achieves the best result, whereas S = 256 offers the best speed/accuracy tradeoff.
6 DISCUSSION
We introduced PABBO, a method to conduct preferential black-box optimization in a fully amortized manner. Leveraging an offline training phase on both synthetic and existing datasets, PABBO achieves considerably reduced inference time jointly with top-ranking performances compared to GP alterna- tives, based on our benchmarks. One way explain this gap is that our pre-training dataset contains latent function values, which are always accessible for synthetic datasets, thus giving PABBO a significant advantage over preferential GPs. This being said, dedicated ablation study demonstrated that replacing latent function values with pure rankings did not significantly affect PABBO's 's results.
Due to the model architecture and the difficulty of the task, learning to optimize from preferential data, PABBO requires large amounts of pre-collected data. While synthetic datasets might be expressive enough, specific applications falling out-of-distribution might pose an issue. More generally, this raises the question of how the pre-training dataset should be constructed. A current limitation of our approach is the size of the query set, S, which scales with task dimensionality, increasing inference times (Figure 5). Preliminary results suggest parallclization may help mitigate this issue. Additionally, a promising solution for high- -igh-iimensioaal tasks is developing a dimension-agnosto architecture that can handle inputs of varying dimensions. Currently, PABBO dimensionality, requiring separate models for tasks with different dimensions. Such a strategy could leverage low-dimensional tasks for high-dimensional ones. This is typically useful if the objective possesses a hidden owwdiimesiional structure, or decomposes additively. Finally, PABBO directly outputs acquisition values for a candidate pair, modeling the user's latent goal, but does not account for the fact that ultimately, a human is performing the comparison. Introducing dedicated user models like the one presented by Sinaga et al. (2024) could be an interesting extension to this work.
only handle tasks with the same
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
XZ, DH and SK were supported by the Research Council of Finland (Flagship programme: Finnish Center for Artificial Intelligence FCAI and decision 341763). SK was also supported by the UKRI Turing AI World-Leading Researcher Fellowship, [EP/W002973/1]. The authors wish to thank Aalto Science-IT project, for the computational and data storage resources provided.
REFERENCES
Mahmoud M Abdelrahman and Clayton Miller. Targeting occupant feedback using digital twins: Adaptive spatial-temporal thermal preference sampling to optimize personal comfort models. Building and Environment, 2022.
Masaki Adachi, Brady Planden, David Howey. Michael A. Osborne, Sebastian Orbell, Natalia Ares, Krikamol Muandet, and Siu Lun Chau. Looping in the human: Collaborative and explain- able Bayesian optimization. In Proceedings of The 27th International Conference on Artificial Intelligence and Statistics, 2024.
Virginia Aglietti, Ira Ktena, Jessica Schrouff, Eleni Sgouritsa, Francisco JR Ruiz, Alan Malek, Alexis Bellot, and Silvia Chiappa. Funbo: Discovering acquisition functions for bayesian optimization with funsearch. arXiv preprint 0rXiv:2406.04824, 2024.
Brandon Amos. Tutorial on amortized optimization. arXiv preprint arXiv:2202.00666. 2023.
Raul Astudillo, Zhiyuan Jerry Lin, Eytan Bakshy, and Peter Frazier. qeubo: decision- theorretc ac- quisition function for preferential bayesian optimization. In Proceedings The 26th International Conference on Artificial Intelligence and Statistics, 2023.
Raul Astudillo, Kejun Li, Maegan Tucker, Chu Xin Cheng, Aaron D. Ames, and Yisong Yue. Preferential multi-objective bayesian optimization. arXiv preprint arXiv:2406. 14699, 2024.
Maximilian Balandat, Brian Karrer, Daniel Jiang, Samuel Daulton, Ben Letham, Andrew G Wilson, and Eytan Bakshy. BoTorch: framework for efficient monte-carlo bayesian optimization. In Advances in Neural Information Processing Systems, 2020.
Viktor Bengs, Robert Busa-Fekete Adil El Mesaoudi Paul, and Eyke Hüllermeier. Preference-basee online learning with dueling bandits: A survey, Journal of Machine Learning Research, 2021.
Tom Blau, Edwin V Bonilla, ladine Chades, and Amir Dezfouli. Optimizing sequential experimental design with deep reinforcement learning. In International conference machine learning, 2022.
Eric Brochu, Vlad M. Cora, and Nando de Freitas. tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint arXiv: 1012.259, 2010.
Paul E Chang, Nasrulloh Loka, Daolang Huang, Ulpu Remes, Samuel Kaski, and Luigi Acerbi. Amortized probabilistic conditioning for optimization, simulation and inference, arXiv preprint arXiv:2410. 15320, 2024.
Guojin Chen, Keren Zhu, Seunggeun Kim, Hanqing Zhu, Yao Lai, Bei Yu, and David Z Pan. Llm- enhanced bayesian optimization for efficient analog layout constraint generation. arXiv preprint arXiv:2406.05225,, 2024.
Yutian Chen, Matthew W Hoffman, Sergio Gómez Colmenarejo, Misha Denil, Timothy P Lillicrap. Matt Botvinick, and Nando Freitas. Learning to learn without gradient descent by gradient descent. In International Conference on Machine Learning, 2017.
Yutian Chen, Xingyou Song. Chansoo Lee, Zi Wang, Richard Zhang, David Dohan, Kazuya Kawakami, Greg Kochanski, Arnaud Doucet, Mare' aurelio Ranzato, et al. Towards learning universal hyperparameter optimizers with transformers. Advances in Neural Information Process- ing Systems, 2022.
Marc'aurelio
11

---

Published as a conference paper at ICLR 2025
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems. 30, 2017.
Wei Chu and Zoubin Ghahramani. Preference learning with Gaussian processes. In Proceedings of the 22nd international conference on Machine learning, 2005.
João P.L. Coutinho, Ivan Castillo, and Marco S. Reis. Human-in-the-loyy controller tuning using preferential bayesian optimization. PFAC-PupersOnnLine, 2024.
Kyle Cranmer, Johann Brehmer, and Gilles Louppe. The frontier of simulation based inference. Proceedings of the National Academy of Sciences, 2020.
Adam Foster, Desi Ivanova, llyas Malik, and Tom Rainforth. Deep adaptive design: Amortizing sequential bayesian experimental design. In Proceedings of the 38th International Conference on Machine Learning, 2021.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Tch, Danilo Rezende, and SM Ali Eslami. Conditional neural processes, In International conference on machine learning, 2018.
Roman Garnett. Bayesian Optimization. Cambridge University Press, 2023.
Manuel Gloeckler, Michael Deistler, Christian Dietrich Weilbach, Frank Wood, and Jakob H. Macke. All-in-one simulation-based inference. Forty-first International Conference on Machine Learn- ing. 2024.
Javier González, Zhenwen Dai, Andreas Damianou, and Neil D Lawrence. Preferential bayesian optimization. In International Conference on Machine Learning, 2017.
José Miguel Hernández-Lobato, James Requeima, Edward O Pyzer-Knapp and Alán Aspuru-Guzii Parallel and distributed thompson sampling for large-scale accelerated exploration of chemical space. In International conference machine learning. 2017.
Bing-Jing Hsich, Ping-Chur Hsich, and Xi Liu. Reinforced few-shot acquisition function learning for bayesian optimization. In Advances in Neural Information Processing Systems, 2021.
Daolang Huang, Louis Filstroff, Petrus Mikkola, Runkai Zheng, and Samuel Kaski. Bayesian optimization augmented with actively elicited expert knowledge. arXiv preprint 087442, 2022.
Daolang Huang, Manuel Haussmann, Ulpu Remes, ST John, Grégoire Clarté, Kevin Luck, Samuel Kaski, and Luigi Acerbi. Practical equivariances via relational conditional neural processes. Advances in Neural Information Processing Systems, 36, 2023.
Daolang Huang, Yujia Guo, Luigi Acerbi, and Samuel Kaski. Amortized bayesian experimental design for decision-making. Advances in Neural Information Processing Systems, 37, 2024.
Carl Hvarfner, Frank Hutter, and Luigi Nardi. A general framework for user-guided bayesian optimization. In The Twelfth International Conference on Learning Representations, 2024.
Desi R Ivanova, Adam Foster, Steven Kleinegesse, Michael U Gutmann, and Thomas Rainforth. Implicit deep adaptive design: Policy-based experimental design without likelihoods. Advances in neural information processing systems, 2021.
Saurav Jha, Dong Gong, Xuesong Wang, Richard E Turner, and Lina Yao. The neural process family: Survey, applications and perspectives. arXiv preprint arXiv:2209.0051 2022.
Daniel Kahneman and Amos Tversky. Prospect theory: An analysis of decision under risk. In Handbook of the fundamentals of financial decision making: Part I. World Scientific, 2013.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garneoo, Ali Eslami. Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference on Learning Representations, 2019.
12

---

Published as a conference paper at ICLR 2025
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential gallery for interactive visual design optimization. ACM Transactions on Graphics (TOG), 2020.
Kejun Li, Maegan Tucker, Erdem Bıyık, Ellen Novoseller, Joel W Burdick, Yanan Sui, Dorsa Sadigh, Yisong Yue, and Aaron D Ames. Roial: Region of interest active learning for characterizing exoskeleton gait preference landscapes, In 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021.
Vincent Lim, Ellen Novoseller, Jeffrey Ichnowski, Huang Huang, and Ken Goldberg. Policy- based bayesian experimental design for non-differentiable implicit models. arXiv preprint SxXiv:2203.04272.. 2022.
Zhiyuan Jerry Lin, Raul Astudillo, Peter Frazier, and Eytan Bakshy. Preference exploration for efficient bayesian optimization with multiple outcomes. In International Conference on Artificial Intelligence and Statistics, 2022.
Tennison Liu, Nicolás Astorga, Nabeel Seedat, and Mihaela van der Schaar. Large language models to enhance bayesian optimization. arXiv preprint arXiv:2402.03921, 2024.
Alexandre Maraval, Matthieu Zimmer, Antoine Grosnit, and Haitham Bou Ammar. End-to-end meta-bayesian optimisation with transformer neural processes. In Advances in Neural Information Processing Systems, 2023.
Stratis Markou, James Requeima, Wessel Bruinsma, Anna Vaughan, and Richard E Turne.. Practical conditional neural process via tractable dependent predictions. In International Conference on Learning Representations, 2022.
Samuel Müller, Noah Hollmann, Sebastian Pineda Arango, Josif Grabocka, and Frank Hutter. Transformers can do bayesian inference. In International Conference on Learning Representations, 2022.
Samuel Müller, Matthias Feurer, Noah Hollmann, and Frank Hutter. Pfns4bo: In-context learning for bayesian optimization. In International Conference on Machine Learning, 2023.
Vivek Myers, Erdem Biyık, and Dorsa Sadigh. Active reward learning from online preferences. In 2023 IEEE International Conference on Robotics and Automation (ICRA), 2023.
Quoc Phong Nguyen, Sebastian Tay, Bryan Kian Hsiang Low, and Patrick Jaillet. Top-A ranking bayesian optimization. arXiv preprint arxiv:2012 10688, 2020.
Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty meta learning via sequence modeling. In International Conference on Machine Learning, 2022.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vetoto, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, nigh-performance deep learning library. In Advances in Neural Information Processing Systems 32, 2019.
Sebastian Pineda Arango, Hadi Jomaa, Martin Wistuba, and Josif Grabocka. Hpo-b: large-scale reproducible benchmark for black-box hpo based on openml. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
Tom Rainforth, Adam Foster, Desi R Ivanova, and Freddie Bickford Smith. Modern bayesian experimental design. Statistical Science, 2024.
13

---

Published as a conference paper at ICLR 2025
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning. MIT Press, 2006.
Eero Siivola, Akash Kumar Dhaka, Michael Riis Andersen, Javier González, Pablo García Moreno, and Aki Vehtari. Preferential batch bayesian optimization. In 2021 IEEE 31st International Workshop on Machine Learning for Signal Processing (MLSP), 2021.
Marshal Arijona Sinaga, Julien Martinelli, Vikas Garg, and Samuel Kaski. Heteroscedastic preferen- tial bayesian optimization with informative noise distributions. arXiv preprint arXiv:2405.1465 146577 2024.
Lei Song, Chenxiao Gao, Ke Xue, Chenyang Wu, Dong Li, Jianye Hao, Zongzhang Zhang, and Chao Qian. Reinforced in-context black-box optimization. arXiv preprint arXiv:2402.17423 2024.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
Kevin Swersky, Yulia Rubanova, David Dohan, and Kevin Murphy. Amortized bayesian optimization over discrete spaces. Conference on Uncertainty in Artificial Intelligence, pp. 769-778. PMLR, 2020.
Shion Takeno, Masahiro Nomura, and Masayuki Karasuyama. Towards practical preferential Bayesian optimization with skew Gaussian processes. In Proceedings of the 40th International Conference Machine Learning, 2023.
Michael Volpp. Lukas Fröhlich, Kirsten Fischer, Andreas Doerr, Stefan Falkner, Frank Hutter, and Christian Daniel. Meta-leaming acquisition functions for transfer learning in bayesian optimization. In International Conference on Learning Representations, 2020.
Wenjie Xu, Wenbin Wang, Yuning Jiang, Bratislav Svetozarevic, and Colin Jones. Principled preferential bayesian optimization. In Forty-first International Conference Machine Learning, 2024.
Adam X. Yang, Laurence Aitchison, and Henry Moss. MONGOOSE: Path-wise smooth bayesian optimisation via meta-learning. In ICML 2024 Workshop Structured Probabilistic Inference & Generative Modeling, 2024.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. Advances in neural information processing systems, 2017.
14

---

Published as a conference paper at ICLR 2025
Appendix
The appendix is organized follows:
Section A provides details about the inner mechanisms of PABBO. It contains algorithm for test time inference, an illustrative figure for the masks employed in self-attention layers, discussion different modeling choices for PABBO'S architecture, and description of our Batch PABBO extension for high-dimensional scalability.
Section contains additional experiments and analyses:
Section B.1 presents cumulative regret results across all experiments from the main text.
Section B.2 evaluates PABBO'S performance additional high high-dimensional test func- tions, including 6D Ackley, 6D Hartmann, and a 16D HPO-B task.
Seccion B.3 provides comprehensive ablation studies on pre-training dataset composi- tion, auxiliary loss weight, discount factor, and query budget effects.
Seccion B.4 shows visualization demonstrates about PABBO'S 's capabilities.
Section C gives the procedure for generating the synthetic datasets used by PABBO during meta-training.
Section provides additional details regarding the experiment section. Specifically, a table for PABBO'S hyperparameters is provided, along with description of the HPO-B datasets, the baselines employed, and the hardware used for training and inference.
A FURTHER DETAILS ON PABBO
A.1 TEST-TIME ALGORITIIM
Algorithm S1 PABBO- test time algorithm
Require: Query set size S, query budget T, initial observations Hz (possibly empty) Initialize D(g) with quasi-random samples
Q { x,) Vii,j {1, , s}3,
for t = 1,.... do
((c) + H+
update context set embed context and query Transformers output for query set conditioning on context set t. T) V,,, Q} b predict acquisition function values for all query
((e) Sembl (D(e)) E(a) fembl (D(
(v) Sum (F()), E(el)
(A (o) .1. (0)
pairs
T | Hr, 1, T) = Categorical(
form policy Categorical distribution sample next query pair from policy observe binary preference
(It,1, ~ Ne
4,
4 = 0t,1 Ft.2
H4+1 + He U ( (1 2r,2, lt)
update history
Q - (1((11,1 x1,2))
update query set
end for
A.2 ARCHITECTURE DETAILS
PABBO employs separate MLPs for 15,1, and 4, mapping each of them to separate embeddings of Transformer input dimension as E( ((,,,)) and E,(,), with @ denoting the component addition. When working on the context set D(e) and prediction context set (ccxxpred) which consists of duels (x 2,,2, 4,), the token embedding is obtained by adding the individual embeddings up, ie. E = 1) + ((.22) + E(4), When processing a single design from the query set D(4) or prediction target set (tarpeed) we directly the individual embedding for that design the token
15

---

Published as a conference paper at ICLR 2025
embedding, i.e., E E (.L1)) Because (I,,1, 2i,, 44) forms single atomic element in context set, Transformer architecture requires all elements its input sequence to live the same embedding space: pass these preference observations to the transformer, need to make sure their dimensions are aligned.
Given that PABBO encodes elements separately, for the same type of elements, like æ, shared embedder can be used across all three sets. This enables parameter sharing for the embedders, as the same embedder processes all æ-type input, regardless of which set they belong Model complexity benefits from this choice as the need to train separate embedding functions for each set is avoided. Besides, separate embeddings confer distinct semantic meanings features and preference labels I. Thus, separate MLPs can specialize in their respective tasks: feature embedders can focus on capturing spatial relationships, and preference embedders can focus on binary relationship encoding. This leads an increased representation power.
On a similar note, one could have settled for using the same embedding for æ;,1 and 2;,2- However, in the particular case of preferential feedback, upon summing (( and E g( the order information would have been lost: the network would not know whether æ,,1 22 the opposite. Therefore, we opted for separate embedders for p( and ((
The interactions between different elements in transformer architecture controlled through attention masking, which determines which elements can attend to each other. Figure S1 illustrates examples of these masks in self-attention layers. These carefully designed masks ensure that model processes information a manner that preserves the causal structure of the preferential optimization problem.
@berrved pair from
Observed p frw
Query Iortion from
Query lecaion from (700
Obuxreed Dicbanad)
Ditissred)
Ditappeea; (urgral)
Obsrreed pair from DP?
piettpred)
Obsereed pair fron peo
Othersed (iteeend)
Teges fro D(trpestt
DN)
Thrgen from (trrpas))
Figure S1: Examples of masks in self-attention layers. Colored squares represent the elements from the left column can attend the elements the top in the self-attention layers of the Transformer block Jim- Left: An example of mask for the acquisition process, involving samples from D(e) and 2 points from D(4), Right: Mask example for the auxiliary prediction process, with 2 samples from (ectppred) and 2 points from (tarpeed)
A.3 BATCH PABBO FOR -IGEDSIMENION SCALABILITY
To address scalability challenges in high-dimensional spaces, we developed Batch PABBO, a par- allelized extension of our standard approach. While the original PABBO processes single query set size S, Batch PABBO evaluates separate query sets of size S simultaneously, selecting the best candidate pair from the combined pool of B x S candidates: at each optimization step, B candidate pairs are independently proposed from the query sets, and the one with the highest predicted acquisition function value is selected as the next query. This approach effectively increases the exploration coverage while maintaining manageable computational requirements. We show the results of batch PABBO in additional high-dimensional experiments (Appendix B.2).
16

---

Published as a conference paper at ICLR 2025
B ADDITIONAL EXPERIMENTS
B. CUMULATIVE REGRET RESULTS
While our main evaluation focuses on simple regret since our primary applications (such as human preference optimization) prioritize finding the optimal solution rather than the path taken, we also analyze the cumulative regret performance of PABBO. This metric is particularly relevant during training, where we use the (discounted) cumulative simple regret as reward function to provide stronger learning signals (Equation 3). Figure S2 shows the cumulative simple regret across all experiments presented in the main text.
GP 30 draws
2D GP draws
Forretter ID
Branin
10
10
101
10
1OP
10
100
10P
10
20
50
10 Search Space No.
Cunmlative Simple Regret
Boale2D
Search Space No. 7609
Scarch Space No. 5630
109
100
109
105
100
10p
10P
10P
10
20
50 Sushi
100
50
100
Candy
PABBO
TTS
101
Random
9NEI
qEUBO
MPES
qEI
10"
10
100
60
100
Iterations
Figure S2: Cumulative simple regret plots for all experiments carried out in the main text. Mean with 95% confidence intervals computed across 30 runs with random starting pairs. PABBO achieves competitive cumulative regret, ranking second on average across all tasks.
B.2 HIGH-DIMENSIONAL FUNCTION EVALUATION
To further evaluate PABBO'S 's performance in nigher-dimensiont spaces, we conducted additional experiments on three challenging benchmarks: the 6-dimensional Ackley function, the 6-dimensional Hartmann function, and 16-dimensional search space from the HPO-B benchmark. On the Ackley function (Figure S3), we also tested our Batch PABBO (Appendix A.3) which allows PABBO to evaluate multiple query sets simultaneously. The experiments reveal varied performance across different functions. While PABBO performs well on the Ackley function (Figure S3) and the 16- dimensional HPO-B task (Figure S5), it shows limitations the Hartmann function (Figure S4). This performance gap on Hartmann likely stems from its multimodal landscape with many local optima, which differs significantly from the functions in our pre-training dataset. This observation suggests that incorporating more diverse function types during pre-trate pre-training could improve performance on such challenging landscapes.
B.3 FURTHER ABLATION STUDIES
To provide deeper insights into PABBO'S performance characteristics and design choices, we conduct series of additional ablation studies exploring various aspects of our model.
17

---

Published as a conference paper at ICLR 2025
20
Cunulaaive Simple Rogret
1200
1000
Simple Regret
15+
500-
10
800
400-
0-
30
0
10
Iterations
PABBO 1024
PABBO = 256
Random
<EUBO
MPES
PABBO 512
PABRO Batch B 5, 256
TTS
qBI
NNRI
Figure S3: Simple regret and cumulative simple regret on the 6-dimensional Ackley function. Mean with 95% confidence intervals computed across 30 runs with random starting pairs. Using paralleliza- tion, we can mimic a larger query set size for PABBO, thus achieving faster convergence than non-batch PABBO versions, for significantly faster cumulative inference time (13s compared to 40s for PABBO = 1024).
3.0
Cumulative Simple Regret
2.5
Simple Regret
2.0
100
1.5
1.0
50
0.5
10
40
50
60
10
50
Itcrations
PABBO
Random
QTS
<EUBO
gEl
MPES
gNEI
Figure S4: Simple regret and cumulative simple regret on the 6-dimensional Hartmann function.
Mean with 95% confidence intervals computed across 30 runs with random starting pairs. We used query set size S = 1024 for PABBO. PABBO does not match the performance most GP-based baselines this example.
B.3.1 PRE-TRAINING DATASET COMPOSITION
The composition of the pre-training dataset is critical for the generalization capabilities of our model. To quantify this effect, we compare two variants: PABBO RBF, pre-trained exclusively on GP draws sampled using RBF kemel,, and the standard PABBO, treitrieed diverse mixture of kemnlss as described in Appendix C.
As shown in Figure S6, training using only the RBF kernel. This confirms our hypothesis that a more diverse pre-training dataset enhances the model's ability to generalize various function landscapes.
mixture of kernels leads to better performance compared to
18

---

Published as a conference paper at ICLR 2025
Search Space No. 5971
10-1
Chumllativ Simple Regvet
204
Stuple Regret
10:3
10k-
10-1
0
20
60
90
100
20
60
180
Iterations
Itraations
PABBO
Rendom
QEUBO
gEI
GT8
ANEI
Figure S5: Simple regret and cumulative simple regret on 16- 16-dimensional search space from the HPO-B benchmark. Mean with 95% confidence intervals computed across runs with random starting pairs. We used query set size S = 1024 for PABBO. PABBO ranks 2nd in ferms of simple regret and 3rd terms of cumulative simple regret.
Coumllative Simple regpet
Simple reggees
101
Iterations
Iterations
PABBO
Figure S6: Ablation study on the composition of the pre-training synthetic dataset on 30 2D GP draws, with query set size S 256. Mean with 95% confidence intervals computed across runs with random starting pairs. PABBO RBF is pre-trained on GP draws sampled using an RBF kernel, whereas PABBO uses the procedure described in Section C. Both baselines use lengthscales I ~ ([1.5, 2)). As expected, pre-training on dataset coming from mixture of kernels rather than only RBF kernel yields better results, due to more pronounced diversity the synthetic samples.
B.3.2 LOSS WEIGHT AND EFFECT OF WARM-UP PHASE
explore impact of the loss weight hyperparameter À in the combined function, as well as the effect of the warm-up phase where the model is initially trained using only the prediction task.
The results Figure S7 reveal that the warm-up phase plays crucial role PABBO'S performance. This confirms our intuition that allowing the model to first learn the shape of the latent function stabilizes the subsequent policy learning process. Interestingly, once the warm-up phase is included, notice that the specific value of X has only mild effect the overall performance.
B.3.3 DISCOUNT FACTOR IN REINFORCEMENT LEARNING
We conduct more comprehensive ablation study by considering denser grid of y values, spaced between 0 to 1, with results shown Figure S8. The conclusion aligns with the main text: smaller values prioritize immediate rewards, leading to slightly faster convergence in the early stage but worse performance in the long term. Additionally, we observe that y < 0.5 deteriorates performance. it might be that sparse reward signals hinder the model convergence. Based these findings, recommend using a close to when the primary goal is minimizing final regret. For scenarios where early-stage performance is crucial, a a between 0.5 and 1.0 can be considered.
19

---

Published as a conference paper at ICLR 2025
Cauullativ Simple Regret
2D GP 30 draws
Simple Regret
109
10'
10
15
20
25
30
10
15
20
25
30
Cunulative Simple Regrot
Simple Regret
101
Branin 2D
10P
109
10
15
20
10
16
Cuumlative Simple Rogret
109
Simple Regret
108
10!
Beale 2D
104
10"
10a
15
20
25
10
15
20
25
Iterations
Iterations
w/ burnin plase
w/ buznin pluse
À w/a buznin plase
X 1 w/ burain plase
A = 0 wy'o burnin pluse A = 2 w/o buznin pluse
Figure S7: Ablation study on the loss weight hyperparameter i L ( XL (Equations and 5) on three 2D functions, with query set size = 256. During the warm-up phase, PABBO is trained using only the prediction task [(P). We observe a stark contrast depending on the presence or not of the warm-up phase, which validates the usefulness of the prediction task, mainly tool to stabilize training. Results are mildly affected by different values of X.
B.3.4 QUERY BUDGET AND TEMPORAL INFORMATION
We now explore how the query budget T used during pre-training affects performance, as well as the impact of passing temporal information i/T to the acquisition head. The results are shown in Figure $9. Our results indicate that including t/T as imput to the acquisition head improves the performance, which means the temporal information t/T helps the model better capture the progression of the optimization process, leading to more informed acquisition decisions. Additionally, we observe that when t/T is included, a larger query budget T is associated with faster convergence and the lowest cumulative simple regret.
B.3.5 ABLATION ON THE PERCENTAGE OF CONTEXT AND TARGET PREDICTION SET
In the main experiments, we set total number of samples for the prediction set and randomly sample (esxpred and assign the remaining pairs (tarpred) To examine the effect of the percentage of context and target prediction set, we train different PABBO for prediction task, when setting (ettpped) 10 with varying (ttapree) [1, 10, 100). The result shown in Figure S10.
20

---

Published as a conference paper at ICLR 2025
Canmelative Simppe Regret
10%
Sinple Regret
10-1
10-
10"-2
20
Iterations
Finel commlotive simple regrst
Iterations
Fiua siaple roges
10f
10-1
10-4
000
03
0.5
0.98
1.0
0.0
022
048
055
0.
Figure S8: Ablation Study on the discount factor y (Equation 3) on the ID Forrester function, with a query size S 256. Mean with 95% confidence intervals computed across 30 runs with random starting pairs. As expected, an higher value of y leads to a lower cumulative simple regret. The picture is nuanced for simple regret, even though of iterations, the conclusion appears to be similar.
progress towards large number
Cuamlaaive Rogre
Stappe Rogeet
10-1
10l
10
20 30
60
Iterations
T 1/
T- pensoved
T 16 64
soncodd
Figure S9: Simple regret and cumulative simple regret on the 1-dimensional Forrester function depending on the query budget T and whether or not t/T is passed input to the acquisition head fa- Mean with 95% confidence intervals computed across 30 runs with random starting pairs. The variance is mostly driven by whether not 1/" is passed to the acquisition head or not (see Figure 2, right). When passing t/T, a large budget T is associated with the fastest convergence and the lowest simple cumulative regret.
B.4 EXAMPLE OF INFERENCE FUNCTION SHAPE AND OPTIMUM ON TEST FUNCTIONS
To provide qualitative insight into PABBO'S learning capabilities, Figure S11 visualizes the function landscapes learned by our model after 30 optimization steps on two test functions. For the 2D Branin function (left), PABBO successfully identifies one of the three global optima. For the ID Forrester function (right), PABBO accurately captures both the overall shape and the location of the global optimum.
21

---

Published as a conference paper at ICLR 2025
Validation Accuracy
0.8
0.7
0.6
N N_tarpred_1
N _tarpped_10
N Nttarpred_100
0.5
50
100
150
200
Iterations
Figure S10: Validation accuracy 128 1-dimensional GP samples. Increasing the size of (itarpred) provides more loss terms during training, thereby enriching the training signal, and resulting in faster convergence of the network.
100
-100
100 Z
300
-5
300
-00
-15
15
0.00
0.50
0.75
X
-10
Y
-5
X
15
true optimun
Inference optimum
queries
prediction
Figure S11: Optimum inference and function shape on test functions after 30 optimization steps for the 2D Branin function (left) and 1D Forrester function (right) trialed in Section 5.1.
C GENERATIVE PROCESS OF SYNTHETIC DATASET
To generate the D-dimensional synthetic data in Section 5.1, we first sample from GP, where
The kernel are equally sampled from the RBF, Matérn-1/ /2, Matérn-3/2 and Matérn-5/2 ker- nels, with the kernel standard deviation ~ ~((.0111 2]) and lengthscale for each dimension dd (1/3,, 0.75) truncated to |0.05, 2].
Then sample function mean as the maximal value of m observations from the Normal distribution (0, o²). To account for very high optimum, exp(1) is added to the mean with probability of 0.1.
22

---

Published as a conference paper at ICLR 2025
0.0
-2.5
2.5
-0.5
1
-3.5
-1.5
lM
0.5
0.0
-0.5
4.50
4.25
-5
4,00
4.0
-2
-1.0
3.8
-1.
3.6
66
-25
Figure S12:
Example of I-dimensional synthetic data generated using the protocol defined in Appendix C. The orange dot in each plot indicates the optimum of the sampled function.
With the defined prior, we randomly sample an optimum (x", 0) inside [] and total N - context points, where N 100 D. The context set sampled from GP posterior conditioned on the optimum by first sampling M - points conditioned on the optimum, and the rest N M points conditioned on these points and the optimum. 50 for 1-dimensional data and 100 when D = 2,3.4
To ensure the existence of global optimum, additionally add quadratic bowl at æ="
Finally, the function value y - (ly) - ell) The offset Ay ~U[- makes the maximal value y = (0 + Ag) E [-5,,5) 5J.
x1/2+
Figure S12 shows 16 randomly generated 1-dimensional synthetic functions.
23

---

Published as a conference paper at ICLR 2025
D TRAINING AND EXPERIMENTAL DETAILS
D.1 PABBO HYPERPARAMETERS
Model Architecture
Embedding dimension Transformer
Point-wise feed-forward dimension Transformer
Number of self-attention layers in Transformer
Number of self-attention heads in Transformer
Number of hidden layers in data embedders
Number of hidden layers in decoder (fp: fa)
Hidden layer dimension of decoder (fp.
128
Training
Number of iterations for warm-up Total number of training iterations Horizon of episodes (?) Learning rate warm-up Learning rate after warm-up Learning rate decay
3000
8000
64
1-10-3
3
3-10-6
Cosine decay to 0 over 8000 iterations
Number of meta-task in batch during warm-up (B)
128
Number of meta-tasks in batch after warm-up (B)
16
Number of trajectories from meta-task for policy learning
Weight on auxiliary loss (A)
discount factor (7)
0.98
Size of query set (S)
min(300, 100 D)
Size of candidate query pair set (M)
min(300, 100
Size of prediction set (NV
min(300, 100 DD)
Maximal size prediction context set (N (stpreed)
50(D
100 < D < 4), 200(D 4)
Environment
Number of initial pairs during training Number of initial pairs during evaluation Observation noise of duel feedback (anoise)
0.0
Table S1: Hyperparameter settings used for PABBO. Most of them remain consistent across all the tasks. Additionally, the size of meta-datasets was scaled according to the search space dimension.
D.2 HPO DATASET DESCRIPTION AND EXPERIMENTAL DETAILS
As mentioned in Section 5.2 and Appendix B.2, experimented on four high-dimensional search spaces from the HPO-B benchmarks (Pineda Arango et al., 2021), each corresponding to the hyper- paramers of certain model: No. 5636 for rpart (29), No. 5859 for rpart (31), No. 7609 for ranger (1 6). and No. 5971 for xgboost (6). Both No. 5636 and No. 5859 are 6-dimensional spaces, No. 7609 is 9-dimensional space, and No. 5971 is 16-dimensional problem. Meta-datasets within each space are divided into three splits: meta-train, meta- mnta--alidattin, and meta-test.
An individual model trained on the meta-train split of each search space. Each meta-dataset is equally divided beforehand into parts from which sample either prediction set D(P) or query set ((4), to prevent any information leak from rewards. The only exception happens when meta meta-dataset has too few data points: we fit a GP to the original data points and sample from the posterior to generate either D(v) D(4), described in Section 3.2, we sample D(P) containing N queries and preferences and D(a) with S query points from each meta meta-dataset at the beginning of each training iteration.
D.3 DETAILS OF BASELINES
The PBO baselines are implemented based on PairwiseG? model class from BoTorch. For QEUBO, qEI and qNEI, BoTorch provides ready-to-use acquisi- tion class @wxpettadtii
and For qTS. we sample two draws from the GP posterior at: (1) locations from quasi-random Sobol sequences for continuous search spaces, or (2) locations of all candidates for discrete search spaces. The maximum of each draw chosen the
qExpectedImproveett
24

---

Published as a conference paper at ICLR 2025
next query. For MPES, we follow the implementation based (Nguyen et al., 2020), which available at bttpa://github wtw./aalastttiileeeaaaea. htt/a//1///////////// /EUBo/ ma meiinn/eee acquisition functions. Finally, we select pair of random points for continuous spaces and a random pair of possible candidates for discrete spaces as random strategy. Baseline inputs are all normalized to (0, 1).
D.4 HARDWARE
We train our model using up Tesla V100-SXM2-32GIB GPUs, with training time of roughly 10, 25, and 29 hours for 1-, 2-, and 4-dimensional synthetic data respectively. There is significant potential to shorten this pre-training process by creating the training samples beforehand, rather than generating them online. For the HPO-B tasks on search spaces No. 5636 and No. 5859 (6 dimensions) and the search space No. 7609 (9 dimensions). training takes approximately 21.5 hours per task. We evaluate all the models on 2x64 core AMD EPYC 7713 @2.0 GHz.
25


Reviews:
[{"review_id": "VdQjmh82tz", "reviewer": "Reviewer_Tep5", "summary": "This work introduces a novel approach to Preferential Bayesian Optimization (PBO) by fully amortizing the optimization process. Traditional PBO methods rely on GPs for modeling user preferences between design pairs, requiring extensive computational resources due to approximate inference for non-conjugate likelihoods. PABBO addresses this challenge by employing a transformer-based neural architecture and reinforcement learning to learn both the surrogate model and acquisition function end-to-end, significantly accelerating the optimization process. By pre-training on synthetic and real-world datasets, PABBO achieves several orders of magnitude in speed improvement while often surpassing GP-based methods in optimization quality.", "questions": "- S in query set is from a Sobol sample. How big is sufficient? How would this affect the optimization quality as even the largest experimented S=1024 might not be sufficiently for moderately high-dimensional spaces, limiting the scalability of this PABBO.\n- How does PABBO work for higher dimensional problems? All experiments presented are of very small dimensions and may not resemble real-world optimization tasks.\n- Algorithm 1 seems to only describe the meta-learning/pre-training part of PABBO as we are still updating the PABBO model at the end of the loop. What’d be the algorithm for applying PABBO on new, unseen dataset? I’d assume that’d be the same as the inner loop of the algo, but it’d be nice to have some clarification from the author.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well written and easy to understand for the most part.\n- The proposed method is a novel and interesting application of amortized learning/BO for PBO setting. The empirical results seem superior than traditional PBO methods and acquisition functions.\n- Nice set of experimental evaluation and ablation studies — though I’d love to see a more comprehensive experimentation section (see weaknesses below) for methods that are hard to provide theoretical guarantees.", "weaknesses": "- Evaluation on harder problems are very limited. Most test functions are of very small dimensions and may not resemble real-world optimization tasks. The only experiments with moderate dimensionalities are 6 and 9-dims respectively and are all from the HPO-B. This leaves the scalability of the proposed methods in question.\n- No ablation studies on meta-learning training set. It is plausible that amortized learning methods are highly sensitive to the selection of pre-training data. There’s no justification or ablation on the choice of pre-training data for PABBO and it’s unclear how robust the method is to a different set of pre-training data.\n- The performance advantage vs baseline methods are not significant in many cases.\n- While it is true that estimating the true posterior of a preference model’s posterior might be expensive, variational inference or Laplace approximation with some clever implementation can go quite far and allows for gradient-based optimization for acqf value. On the other hand, for larger models like PABBO1024, the inference time advantage of amortized learning might be diminishing when compared to GP-based model according Figure 3 and 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yDQOBcgj6y", "reviewer": "Reviewer_Xx1d", "summary": "This paper explores preferential black-box optimization with an amortized inner-loop solver. In other words, RLHF but from the lens of Bayesian optimization instead of NLP.", "questions": "Can you add a reasonable interpretation of top-two thompson sampling in the continuous case to your experiments by the end of the rebuttal period? \n\nAre you warm-starting the amortized solver training as the data changes? Do you lose performance because of that choice? How expensive does the test-time search have to be to warrant amortization? How carefully have you thought about the FLOPS involved here?\n\nHow much can you push this? I don't just want to see rosy results, I want to know when this breaks.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This work is well motivated. Aligning training objectives with downstream use makes sense (i.e. the explicit acquisition head) and amortization also makes sense if you expect to solve the same (or nearly the same) optimization problem many many times.\n\nThe paper is clear and well-written\n\nThe experiments hit pretty much all the key points I would expect", "weaknesses": "This paper seems to treat simple and cumulative regret as interchangeable, although I am sure the authors know the difference. If the policy is optimized to minimize cumulative regret, why is cumulative regret not reported? On the other hand if simple regret is really what you care about, why are none of the baselines aimed at best-arm identification? For example, Thompson sampling is optimal w.r.t. cumulative regret, not simple regret. Top-two thompson sampling is optimal in the discrete case [1], although translating to the continuous case requires some thought (what epsilon is sufficiently large to constitute a different arm?).\n\nI also feel that the experimental design could be more directed at identifying under what conditions this solution makes sense. What is the breakpoint when amortization starts paying off compared to direct search at test time? How far can you push this before it breaks, especially in terms of the initial data package?\n\nIt also seems very odd that preference tuning for language models is not mentioned even once, even though it is obviously the same problem and basically the same solution (ignoring low-level implementation choices). I can't tell if this is a deliberate choice that reflects the legal sensitivity around LLMs right now or a genuine oversight, however I see no reason not to draw the connection to preference tuning work.\n\nI generally try to avoid policing language, but sentences like \"... amortization ... has emerged as an almost magical bullet solution\" clearly crosses the line from excitement to needless hype. Amortization is not anything close to a magic bullet. You know that and I know that. It is just moving compute around from test time to train time. Of course science involves some marketing, but language and framing like this hurts our credibility with more staid scientific disciplines, people that you presumably would like to take you seriously. Write like a scientist, not a salesman. It's ok to be excited, just don't get carried away!\n\n\nReferences\n\n- [1] Russo, D. (2016, June). Simple bayesian algorithms for best arm identification. In Conference on Learning Theory (pp. 1417-1418). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "twkcDfKVr5", "reviewer": "Reviewer_dRx5", "summary": "This paper presents PABBO a preference based Bayesian optimization approach using transformer architectures and deep sets approaches to amortize the inference time. The paper extend ideas of amortization from past works to preference based optimization and presents an end-to-end trainable approach.\n\nThe main idea behind the approach is to use a transformer and attention mechanisms on historical observed data and use them to make query predictions. Another prediction head from the same transformer produces the policy function which is used to suggest the future queries. Experimental results indicate that the proposed approach is faster compared to previous approach and achieves a smaller regret.", "questions": "General questions\n- How does preference based optimization work when users have varying preferences? Is it possible to incorporate personal user-preferences in this framework.\n\nSpecific questions\n- Line 237, Is the same MLP applied to x_i,1, x_i,2 and l_i individuall? Why should l_i be encoded into the same embedding space as x_i,1 and x_i,2? Or does the statement mean to encode the concatenation of the triple using an MLP?\n- Line 267, It seems that to train $\\pi_\\theta$ the reward used is given in line 305 defined as the maximum of the observations so far. This part is unclear to me.\n  - How is this reward helping in learning a good policy?\n  - It seems that the loss function is static given a history H_t, so it is not a RL setup but simply a supervised learning setup. Is any RL specific learning procedure being used here? Are the reward values r_t fixed observations or are they learnable and also backpropagated through to the prediction values y and consequently through the prediction head?\n  - This step looks very similar to offline BO where a offline data set is used to learn an acquisition function or a function approximation. What is the relationship of this method to offline BO?\n- Line 303, How is the query set constructed when computing $\\pi_\\theta$? Is it a random subset of all query pairs? Or is any specialized strategy used to locate the most promising queries?\n- How is PABBO able to avoid sub-optimal solutions caused due to not exploring the whole optimization space?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Strengths:\n- The paper presents a new approach to preference based BO using transformers which as several advantages - it can incorporate preferences, has amortized inference cost and can be trained end-to-end.\n- The proposed approach is intuitive and a reasonable way to model preferences in BO, with the potential to scale to very large problems. The method uses neural networks instead of Gaussian processes which helps it scale to a large number of data points.\n- Extensive experimentation shows that PABBO has a smaller inference cost while achieving a smaller regret compared to prior approaches.", "weaknesses": "Weaknesses:\n- While the proposed approach is faster than BO, the scale at which the experiments are performed (100 observations max), the gains are not fully realized since BO methods can perform quite comfortable at such small scales.\n- The full potential of the algorithm is only realized in large experiment with thousands of observations. Such large scale experiments have not been presented in the paper.\n- Preference based optimization is most useful when modeling users with varying preferences. The paper can benefit from the interesting extension of learning to model individual user preferences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fsmQFEFrCj", "reviewer": "Reviewer_x3e8", "summary": "This paper introduces a new approach to preferential Bayesian Optimization, using large-scale pretraining to enhance inference speed and optimization performance. PABBO employs two distinct transformer heads: a Prediction Head that captures the underlying function determining preferences, and an Acquisition Head that selects the next query pair. Experimental results show that PABBO outperforms existing methods in accuracy and computational efficiency.", "questions": "NA", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of utilizing meta-learning for Amortized Optimization in preferential Bayesian Optimization is new and reduces the cost of inference.\n\n2. Experiment results show that PABBO outperforms other methods in the datasets selected in this paper.", "weaknesses": "1. The pretraining in this method requires a large amount of data, which could be costly for certain tasks.\n\n2. PABBO is currently limited to tasks with a fixed input dimensionality.\n\n3. PABBO only supports pairwise preference queries, limiting its ability to handle queries with multiple options, which could reduce the efficiency of information gathering. While existing works like [1] could handle queries with multiple options.\n\n4. Including more datasets in the experiments would provide a more comprehensive evaluation. For example, incorporating all acquisition functions and tasks from [1] and other related works could strengthen the comparison. \n\nReference:\n1. Astudillo, R., Lin, Z.J., Bakshy, E. and Frazier, P., 2023, April. qEUBO: A decision-theoretic acquisition function for preferential Bayesian optimization. In International Conference on Artificial Intelligence and Statistics (pp. 1093-1114). PMLR.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zi7UhjuLwx", "reviewer": "Reviewer_do2x", "summary": "This paper introduces the Preferential Amortized Black-Box Optimization (PABBO) approach, which learns acquisition function values for candidate pairs based on preferential feedback between pairs of designs. The method utilizes a transformer-based architecture and a reinforcement learning-based pretraining scheme.", "questions": "About the architecture:\n1. Why did you choose the Gaussian distribution in equation (4)? What's the role of the Gaussian distribution in the algorithm's success for the out-of-distribution case in synthetic experiments of section 5.1 and human preferences in section 5.3? Are there any mathematical intuitions?\n2. What's the influence of D^{ctxpred} in your training procedure? How do you determine the percentage of D^{ctxpred} and D^{tarpred}?\n\nAbout the hyperparameters:\n1. Why do you fix lambda=1 in all experiments? What's the influence of lambda?\n2. In line 500, why do you choose gamma=0.5, 0.9, 0.98, 1.00? These choices seem unusual and may have been selected deliberately, potentially compromising the generality of the results.\n3. What's the influence of query budget T?\n4. What would high dimensions influence the algorithm? For example, what would happen if you chose the D to be very large in line 719? \n\nI would be willing to increase the score if the above questions could be well clarified.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The PABBO architecture achieves end-to-end training for preference feedback and employs an application of amortization within a preferential Bayesian optimization (PBO) setting. The algorithm’s contributions are demonstrated through experiments against Gaussian process-based benchmarks.", "weaknesses": "This paper seems to fall below the standard expected at ICLR for the following reasons:\n\n1. A critical issue is that the paper lacks a theoretical foundation or even a mathematical intuition for its approach, focusing primarily on reporting numerical results without a deeper analytical context.\n\n2. In terms of experiments, the explanation of the architecture design and the choice of hyperparameters requires more clarity and justification. I will outline these concerns in further detail in the questions below. \n\n3. Finally, the paper would benefit from additional polishing. For instance, in line 189, y_{i,1} and y_{i,2} appear without prior explanation. In line 219, the term “P” in “the rest of P queries” may be confusing for readers. In line 718, For in \"M=50 For\" should be for.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)