Title: Bandits with Preference Feedback: A Stackelberg Game Perspective
Venue: NeurIPS 2024
Accepted: yes
Keywords: Preference Feedback

Duelling Bandits

Reproducing Kernel Hilbert Space

Stackelberg Games

Exploration-exploitation dilemma
Abstract: Bandits with preference feedback present a powerful tool for optimizing unknown target functions when only pairwise comparisons are allowed instead of direct value queries. This model allows for incorporating human feedback into online inference and optimization and has been employed in systems for tuning large language models.
The problem is fairly understood in toy settings with linear target functions or over finite small domains that limits practical interest.
Taking the next step, we consider infinite domains and kernelized rewards. In this setting, selecting a pair of actions is quite challenging and requires balancing exploration and exploitation at two levels: within the pair, and along the iterations of the algorithm.
We propose MaxMinLCB, which emulates this trade-off as a zero-sum Stackelberg game and chooses action pairs that are informative and have favorable reward values. MaxMinLCB consistently outperforms algorithms in the literature and satisfies an anytime-valid rate-optimal regret guarantee. This is owed to our novel preference-based confidence sequences for kernelized logistic estimators, which are of independent interest.

Bandits with Preference Feedback: A Stackelberg Game Perspective

Abstract

Bandits with preference feedback present a powerful tool for optimizing unknown target functions when only pairwise comparisons are allowed instead of direct value queries. This model allows for incorporating human feedback into online inference and optimization and has been employed in systems for fine-tuning large language models. The problem is well understood in simplified settings with linear target functions or over finite small domains that limit practical interest. Taking the next step, we consider infinite domains and nonlinear (kernelized) rewards. In this setting, selecting a pair of actions is quite challenging and requires balancing exploration and exploitation at two levels: within the pair, and along the iterations of the algorithm. We propose MAXMINLCB, which emulates this trade-off as a zero-sum Stackelberg game, and chooses action pairs that are informative and yield favorable rewards. MAXMINLCB consistently outperforms existing algorithms and satisfies an anytime-valid rate-optimal regret guarantee. This is due to our novel preference-based confidence sequences for kernelized logistic estimators.

1 Introduction

In standard bandit optimization, a learner repeatedly interacts with an unknown environment that gives numerical feedback on the chosen actions according to a utility function f . However, in applications such as fine-tuning large language models, drug testing, or search engine optimization, the quantitative value of design choices or test outcomes are either not directly observable, or are known to be inaccurate, or systematically biased, e.g., if they are provided by human feedback [Casper et al., 2023]. A solution is to optimize for the target based on comparative feedback provided for a pair of queries, which is proven to be more robust to certain biases and uncertainties in the queries [Ji et al., 2023].

Bandits with preference feedback, or dueling bandits, address this problem and propose strategies for choosing query/action pairs that yield a high utility over the horizon of interactions. At the core of such algorithms is uncertainty quantification and inference for f in regions of interest, which is closely tied to exploration and exploitation dilemma over a course of queries. Observing only comparative feedback poses an additional challenge, as we now need to balance this trade-off jointly over two actions. This challenge is further exacerbated when optimizing over vast or infinite action domains. As a remedy, prior work often grounds one of the actions by choosing it either randomly or greedily, and tries to balance exploration-exploitation for the second action as a reaction to the first [Ailon et al., 2014, Zoghi et al., 2014a, Kirschner and Krause, 2021, Mehta et al., 2023b]. This approach works well for simple utility functions over low-dimensional domains, however does not scale to more complex problems.

Aiming to solve this problem, we focus on continuous domains in the Euclidean vector space and complex utility functions that belong to the Reproducing Kernel Hilbert Space (RKHS) of a potentially non-smooth kernel. We propose MAXMINLCB, a sample-efficient algorithm that at every step chooses the actions jointly , by playing a zero-sum Stackelberg (a.k.a Leader-Follower) game. We choose the Lower Confidence Bound (LCB) of f as the objective of this game which the Leader aims

to maximize and the Follower to minimize. The equilibrium of this game yields an action pair in which the first action is a favorable candidate to maximize f and the second action is the strongest competitor against the first. Our choice of using the LCB as the objective leads to robustness against uncertainty when selecting the first action. Moreover, it makes the second action an optimistic choice as a competitor, from its own perspective. We observe empirically that this approach creates a natural exploration scheme, and in turn, yields a more sample-efficient algorithm compared to standard baselines.

Our game-theoretic strategy leads to an efficient bandit solver, if the LCB is a valid and tight lower bound on the utility function. To this end, we construct a confidence sequence for f given pairwise preference feedback, by modeling the noisy comparative observations with a logistic-type likelihood function. Our confidence sequence is anytime valid and holds uniformly over the domain, under the assumption that f resides in an RKHS. We improve prior work by removing or relaxing assumptions on the utility while maintaining the same rate of convergence. This result on preference-based confidence sequences may be of independent interest, as it targets the loss function that is typically used for Reinforcement Learning with Human Feedback.

Contributions Our main contributions are:

We propose a novel game-theoretic acquisition function for pairwise action selection with preference feedback.

We construct preference-based confidence sequences for kernelized utility functions that are tight and anytime valid.

Together this creates MAXMINLCB, an algorithm for bandit optimization with preference feedback over continuous domains. MAXMINLCB satisfies O ( γ T √ T ) regret, where T is the horizon and γ T is the information gain of the kernel.

We benchmark MAXMINLCB over a set of standard optimization problems and consistently outperform the common baselines from the literature.

2 Related Work

Learning with indirect feedback was first studied in the supervised preference learning setting [Aiolli and Sperduti, 2004, Chu and Ghahramani, 2005]. Subsequently, online and sequential settings were considered, motivated by applications in which the feedback is provided in an online manner, e.g., by a human [Yue et al., 2012, Yue and Joachims, 2009, Houlsby et al., 2011]. Bengs et al. [2021] surveys this field comprehensively; here we include a brief background.

Referred to as dueling bandits, a rich body of work considers (finite) multi-armed domains and learns a preference matrix specifying the relation among the arms. Such work often relies on efficient sorting or tournament systems based on the frequency of wins for each action [Jamieson and Nowak, 2011, Zoghi et al., 2014b, Komiyama et al., 2015, Wu and Liu, 2016, Falahatgar et al., 2017]. Rather than jointly selecting the arms, such strategies often simplify the problem by selecting one at random [Zoghi et al., 2014a, Zimmert and Seldin, 2018], greedily [Chen and Frazier, 2017], or from the set of previously selected arms [Ailon et al., 2014]. In contrast, we jointly optimize both actions by choosing them as the equilibrium of a two-player zero-sum Stackelberg game, enabling a more efficient exploration/exploitation trade-off.

The multi-armed dueling setting, which is reducible to multi-armed bandits [Ailon et al., 2014], naturally fails to scale to infinite compact domains, since regularity among 'similar' arms is not exploited. To go beyond finite domains, utility-based dueling bandits consider an unknown latent function that captures the underlying preference, instead of relying on a preference matrix. The preference feedback is then modeled as the difference in the utility of two chosen actions passed through a link function. Early work is limited to convex domains and imposes strong regularity assumptions [Yue and Joachims, 2009, Kumagai, 2017]. These assumptions are then relaxed to general compact domains if the utility function is linear [Dudík et al., 2015, Saha, 2021, Saha and Krishnamurthy, 2022]. Constructing valid confidence sets from comparative feedback is a challenging task. However, it is strongly related to uncertainty quantification with direct logistic feedback, which is extensively analyzed by the literature on logistic and generalized linear bandits [Filippi et al., 2010, Faury et al., 2020, Foster and Krishnamurthy, 2018, Beygelzimer et al., 2019, Faury et al., 2022, Lee et al., 2024].

Preference-based bandit optimization with linear utility functions is well understood and is even extended to reinforcement learning with preference feedback on trajectories [Saha et al., 2023, Zhan et al., 2023, Zhu et al., 2023, Ji et al., 2023, Munos et al., 2023]. However, such approaches have limited practical interest, since they cannot capture real-world problems with complex nonlinear

utility functions. Alternatively, Reproducing Kernel Hilbert Spaces (RKHS) provide a rich model class for the utility, e.g., if the chosen kernel is universal. Many have proposed heuristic algorithms for bandits and Bayesian optimization in kernelized settings, albeit without providing theoretical guarantees Brochu et al. [2010], González et al. [2017], Sui et al. [2017], Tucker et al. [2020], Mikkola et al. [2020], Takeno et al. [2023].

There have been attempts to prove convergence of kernelized algorithms for preference-based bandits [Xu et al., 2020, Kirschner and Krause, 2021, Mehta et al., 2023b,a]. Such works employ a regression likelihood model which requires them to assume that both the utility and the probability of preference, as a function of actions, lie in an RKHS. In doing so, they use a regression model for solving a problem that is inherently a classification. While the model is valid, it does not result in a sample-efficient algorithm. In contrast, we use a kernelized logistic negative log-likelihood loss to infer the utility function, and provide confidence sets for its minimizer. In a concurrent work, Xu et al. [2024] also consider the kernelized logistic likelihood model and propose a variant of the MULTISBM algorithm [Ailon et al., 2014] which uses likelihood ratio confidence sets. The theoretical approach and resulting algorithm bear significant differences, and the regret guarantee has a strictly worse dependency on the time horizon T , by a factor of T 1 / 4 . This is discussed in more detail in Section 5.

3 Problem Setting

Consider an agent which repeatedly interacts with an environment: at step t the agent selects two actions x t , x ′ t ∈ X and only observes stochastic binary feedback y t ∈ { 0 , 1 } indicating if x t ≻ x ′ t , that is, if action x t is preferred over action x ′ t . Formally, P ( y t = 1 | x t , x ′ t ) = P ( x t ≻ x ′ t ) , and y t = 0 with probability 1 -P ( x t ≻ x ′ t ) . Based on the preference history H t = { ( x 1 , x ′ 1 , y 1 ) , . . . ( x t , x ′ t , y t ) } , the agent aims to sequentially select favorable action pairs. Over a horizon of T steps, the success of the agent is measured through the cumulative dueling regret

which is the average sub-optimality gap between the chosen pair and a globally preferred action x ⋆ . To better understand this notion of regret, consider the scenario where actions x t and x ′ t are both optimal. Then the probabilities are equal to 0 . 5 and the dueling regret will not grow further, since the regret incurred at step t is zero. This formulation of R D ( T ) is commonly used in the literature of dueling Bandits and RL with preference feedback [Urvoy et al., 2013, Saha et al., 2023, Zhu et al., 2023] and is adapted from Yue et al. [2012]. Our goal is to design an algorithm that satisfies a sublinear dueling regret, where R D ( T ) /T → 0 as T →∞ . This implies that given enough evidence, the algorithm will converge to a globally preferred action. To this end, we take a utility-based approach and consider an unknown utility function f : X → R , which reflects the preference via where s : R → [0 , 1] is the sigmoid function 1 , i.e. s ( a ) = (1 + e -a ) -1 . Referred to as the Bradley-Terry model [Bradley and Terry, 1952], this probabilistic model for binary feedback is widely used in the literature for logistic and generalized bandits [Filippi et al., 2010, Faury et al., 2020]. Under the utility-based model, x ⋆ = arg max x ∈X f ( x ) and we can draw connections to a classic bandit problem with direct feedback over the utility f . In particular, Saha [2021] shows that the dueling regret of (1) is equivalent up to constant factors, to the average utility regret of the two actions, that is ∑ T t =1 f ( x ⋆ ) -[ f ( x t ) + f ( x ′ t )] / 2 .

Throughout this paper, we make two key assumptions over the environment. We assume that the domain X ⊂ R d 0 is compact, and that the utility function lies in H k , a Reproducing Kernel Hilbert Space corresponding to some kernel function k ∈ X × X → R with a bounded RKHS norm ∥ f ∥ k ≤ B . Without a loss of generality, we further suppose that the kernel function is normalized and k ( x , x ) ≤ 1 everywhere in the domain. Our set of assumptions extends the prior literature on logistic bandits and dueling bandits from linear rewards or finite action spaces, to continuous domains with non-parametric rewards.

While our theoretical framework targets euclidean domains, our methodology may be used on general domains of text or images, given vector embeddings obtained via unsupervised learning. Solving a bandit problem on top of embeddings from a pretrained language model is common practice in

further fine-tuning of such models [e.g., Nguyen et al., 2024, Mehta et al., 2023a], and is further demonstrated in our Yelp experiment (c.f Section 6.3). Lastly, we highlight that our results may be smoothly extended to contextual bandits with stochastic context, by simply modifying the signature of the kernel function to k ′ ( x , x ′ , z ) : X × X × Z → R , where z ∈ Z denotes the context. This setting accommodates applications in active learning for fine-tuning of large language models, where the context is the prompt and the two actions are two alternative responses.

4 Kernelized Confidence Sequences with Direct Logistic Feedback

As a warm-up, we consider a hypothetical scenario where x ′ t = x null for all t ≥ 1 such that f ( x null ) = 0 . Therefore at every step, we suggest an action x t and receive a noisy binary feedback y t , which is equal to one with probability s ( f ( x t )) . This example reduces our problem to logistic bandits which has been rigorously analyzed for linear rewards [Filippi et al., 2010, Faury et al., 2020]. We extend prior work to the non-parametric setting by proposing a tractable loss function for estimating the utility function, a.k.a. reward. We present novel confidence intervals that quantify the uncertainty on the logistic predictions uniformly over the action domain. In doing so, we propose confidence sequences for the kernelized logistic likelihood model that are of independent interest for developing sample-efficient solvers for online and active classification.

The feedback y t is a Bernoulli random variable, and its likelihood depends on the utility function as s ( f ( x t )) y t [1 -s ( f ( x t ))] 1 -y t . Then given history H t , we can estimate f by f t , the minimizer of the regularized negative log-likelihood loss

where λ > 0 is the regularization coefficient. The regularization term ensures that ∥ f t ∥ k is finite and bounded. For simplicity, we assume throughout the main text that ∥ f t ∥ k ≤ B . However, we do not need to rely on this assumption to give theoretical guarantees. In the appendix, we present a more rigorous analysis by projecting f t back into the RKHS ball of radius B to ensure that the B -boundedness condition is met, instead of assuming it. We do not perform this projection in our experiments.

Solving for f t may seem intractable at first glance since the loss is defined over functions in the large space of H k . However, it is common knowledge that the solution has a parametric form and may be calculated by using gradient descent. This is a direct application of the Representer Theorem [Schölkopf et al., 2001] and is detailed in Proposition 1.

Proposition 1 (Logistic Representer Theorem) . The regularized negative log-likelihood loss of (3) has a unique minimizer f t , which takes the form f t ( · ) = ∑ t τ =1 α τ k ( · , x τ ) where ( α 1 , . . . α t ) =: α t ∈ R t is the minimizer of the strictly convex loss

with k t ( x ) = ( k ( x 1 , x ) , . . . , k ( x t , x )) ∈ R t .

Given f t , we may predict the expected feedback for a point x as s ( f t ( x )) . Centered around this prediction, we construct confidence sets of the form [ s ( f t ( x )) ± β t ( δ ) σ t ( x )] , and show their uniform anytime validity. The width of the sets are characterized by σ t ( x ) defined as

Theorem 2 (Kernelized Logistic Confidence Sequences) . Assume f ∈ H k and ∥ f ∥ k ≤ B . Let 0 < δ < 1 and set where κ = sup a ≤ B 1 / ˙ s ( a ) , with ˙ s ( a ) = s ( a )(1 -s ( a )) denoting the derivative of the sigmoid function, and K t ∈ R t × t is the kernel matrix satisfying [ K t ] i,j = k ( x i , x j ) . Our first main result shows that for a careful choice of β t ( δ ) , these sets contain s ( f ( x )) simultaneously for all x ∈ X and t ≥ 1 with probability greater than 1 -δ .

Function-valued confidence sets around the kernelized ridge estimator are analyzed and used extensively to design bandit algorithms with noisy feedback on the true reward values [Valko et al., 2013, Srinivas et al., 2010, Chowdhury and Gopalan, 2017, Whitehouse et al., 2023]. However, under noisy logistic feedback, this literature falls short as the proposed confidence sets are no longer valid for the kernelized logistic estimator f t . One could still estimate f using a kernelized ridge estimator and benefit from this line of work. However, as empirically demonstrated in Figure 1a, this will not be a sample-efficient approach.

Proof Sketch. When minimizing the kernelized logistic loss, we do not have a closed-form solution for f t , and can only formulate it using the fact that the gradient of the loss evaluated at f t is the null operator, i.e., ∇L ( f t ; H t ) : H → H = 0 . The key idea of our proof is to construct confidence intervals as H -valued ellipsoids in the gradient space and show that the gradient operator evaluated at f belongs to it with high probability (c.f. Lemma 8). We then translate this back into intervals around point estimates s ( f t ( x )) uniformly for all points x ∈ X . The complete proof is deferred to Appendix A, and builds on the results of Faury et al. [2020] and Whitehouse et al. [2023].

Logistic Bandits. Such confidence sets are an integral tool for action selection under uncertainty, and bandit algorithms often rely on them to balance exploration against exploitation. To demonstrate how Theorem 2 may be used for bandit optimization with direct logistic feedback, we consider LGP-UCB, the kernelized Logistic GP-UCB algorithm. Presented in Algorithm 2, it extends the optimistic algorithm of Faury et al. [2020] from the linear to the kernelized setting, by using the confidence bound of Theorem 2 to calculate an optimistic estimate of the reward. We proceed to show that LGP-UCB attains a sublinear logistic regret, which is commonly defined as

To the best of our knowledge, the following corollary presents the first regret bound for logistic bandits in the kernelized setting and may be of independent interest.

Corollary 3. Let δ ∈ (0 , 1] and choose the exploration coefficients β t ( δ ) as described in Theorem 2 for all t ≥ 0 . Then LGP-UCB satisfies the anytime cumulative regret guarantee of

5 Main Results: Bandits with Preference Feedback

We return to our main problem setting in which a pair of actions, x t and x ′ t , are chosen and the observed response is a noisy binary indicator of x t yielding a higher utility than x ′ t . While this type of feedback is more consistent in practice, it creates quite a challenging problem compared to the logistic problem of Section 4. The search space for action pairs X × X is significantly larger than X , and the observed preference feedback of s ( f ( x t ) -f ( x ′ t )) conveys only relative information between two actions. We start by presenting a solution to estimate f and obtain valid confidence sets under preference feedback. Using these confidence sets we then design the MAXMINLCB algorithm which chooses action pairs that are not only favorable, i.e., yield high utility, but are also informative for improving the utility confidence sets.

5.1 Preference-based Confidence Sets

We consider the probabilistic model of y t as stated in Section 3, and write the corresponding regularized negative loglikelihood loss as

This loss may be optimized over different function classes and is commonly used for linear dueling bandits [e.g., Saha, 2021], and has been notably successful in reinforcement learning with human feedback [Christiano et al., 2017]. We proceed to show that the preference-based loss L D k is equivalent to L L k D , the standard logistic loss (3) invoked with a specific kernel function k D . This will

allow us to cast the problem of inference with preference feedback as a kernelized logistic regression problem. To this end, we define the dueling kernel as

for all ( x 1 , x ′ 1 ) , ( x 2 , x ′ 2 ) ∈ X × X , and let H k D be the RKHS corresponding to it. While the two function spaces H k D and H k are defined over different input domains, we can show that they are isomorphic, under simple regularity conditions.

Proposition 4. Let f : X → R . Consider a kernel k and the sequence of its eigenfunctions ( ϕ i ) ∞ i =1 . Assume the eigenfunctions are zero-mean, i.e. ∫ x ∈X ϕ i ( x )d x = 0 . Then f ∈ H k , if and only if there exists h ∈ H k D such that h ( x , x ′ ) = f ( x ) -f ( x ′ ) . Moreover, ∥ h ∥ k D = ∥ f ∥ k .

Proposition 4 allows us to re-write the preference-based loss function of (6) as a logistic-type loss

The proof is left to Appendix C.1. The assumption on eigenfunctions in Proposition 4 is primarily made to simplify the equivalence class. In particular, the relative preference function h can only capture the utility f up to a bias, i.e., if a constant bias b is added to all values of f , the corresponding h function will not change. The value of b may not be recovered by drawing queries from h , however, this will not cause issues in terms of identifying arg max of f through querying values of h . Therefore, we set b = 0 by assuming that eigenfunctions of k are zero-mean. This assumption automatically holds for all kernels that are translation or rotation invariant over symmetric domains, since their eigenfunctions are periodic L 2 ( X ) basis functions, e.g., Matérn kernels and sinusoids.

that is equivalent to (3) up to the choice of kernel. We define the minimizer h t := arg min L L k D ( h ; H t ) and use it to construct anytime valid confidence sets for the utility f given only preference feedback. Corollary 5 (Kernelized Preference-based Confidence Sequences) . Assume f ∈ H k and ∥ f ∥ k ≤ B . Choose 0 < δ < 1 and set β D t ( δ ) and σ D t as in equations (4) and (5) , with k D used as the kernel function. Then,

where h t = arg min L L k D ( h ; H t ) .

Corollary 5 gives valid confidence sets for kernelized utility functions under preference feedback and immediately improves prior results on linear dueling bandits and kernelized dueling bandits with regression-type loss, to kernelized setting with logistic-type likelihood. To demonstrate this, in Appendix C.3 we present the kernelized extensions of MAXINP (Saha [2021], Algorithm 3), and IDS (Kirschner and Krause [2021], Algorithm 4) and prove the corresponding regret guarantees (cf. Theorems 15 and 16). Corollary 5 holds almost immediately by invoking Theorem 2 with the dueling kernel k D and applying Proposition 4. A proof is provided in Appendix C.1 for completeness. Comparison to Prior Work. A line of previous work assumes that both f and the probability s ( f ( x )) are B -bounded members of H k . This allows them to directly estimate s ( f ( x )) via kernelized linear regression [Xu et al., 2020, Mehta et al., 2023b, Kirschner and Krause, 2021]. The resulting confidence intervals are then around the least squares estimator, which does not align with the logistic estimator f t . This model does not encode the fact that s ( f ( x )) only takes values in [0 , 1] and considers a sub-Gaussian distribution for y t , instead of the Bernoulli formulation when calculating the likelihood. Therefore, the resulting algorithms require more samples to learn an accurate reward estimate. In a concurrent work, Xu et al. [2024] also consider the loss function of Equation (6) and present likelihood-ratio confidence sets. The width of the sets at time T , scales with √ T log N ( H k ; 1 /T ) where the second term is the metric entropy of the B -bounded RKHS at resolution 1 /T , that is, the log-covering number of this function class, using balls of radius 1 /T . It is known that log N ( H k ; 1 /T ) ≍ γ T as defined in Theorem 2. This may be easily verified using Wainwright [2019, Example 5.12] and [Vakili et al., 2021, Definition 1]. Noting the definition of β D t , we see that likelihood ratio sets of Xu et al. [2024] are wider than Corollary 5. Consequently, the presented regret guarantee in this work is looser by a factor of T 1 / 4 compared to our bound in Theorem 6.

Algorithm 1 MAXMINLCB

Play the most potent pair ( x t , x ′ t ) according to the Stackelberg game

Observe y t and append history.

Update h t +1 and σ D t +1 and the set of plausible maximizers

end for

5.2 Action Selection Strategy

Let x ′ ( x ) = arg min x ′ ∈M t LCB t ( x , x ′ ) denote a response function. We propose MAXMINLCB in Algorithm 1 for the preference feedback bandit problem that selects x t and x ′ t jointly as

where the lower-confidence bound LCB t ( x , x ′ ) = s ( h t ( x , x ′ )) -β D t σ D t ( x , x ′ ) presents a pessimistic estimate of h and M t = { x ∈ X|∀ x ′ ∈ X : s ( h t ( x , x ′ )) + β D t σ D t ( x , x ′ ) ≥ 0 . 5 } is the set of potentially optimal actions. The second action is chosen as x ′ t = x ′ ( x t ) . Equation (7) forms a zero-sum Stackelberg (Leader-Follower) game where the actions x t and x ′ t are chosen sequentially [Stackelberg, 1952]. First, the Leader selects x t , then the Follower selects x ′ t depending on the choice of x t . Importantly, due to the sequential nature of action selection, x t is chosen by the Leader such that the Follower's action selection function, x ′ ( · ) , is accounted for in the selection of x t . Sequential optimization problems are known to be computationally NP-hard even for linear functions [Jeroslow, 1985]. However, due to their importance in practical applications, there are algorithms that can efficiently approximate a solution over large domains [Sinha et al., 2017, Ghadimi and Wang, 2018, Dagréou et al., 2022, Camacho-Vallejo et al., 2023].

MAXMINLCB builds on a simple insight: if the utility f is known, both the Leader and the Follower will choose x ⋆ yielding an objective value 0 . 5 for both players, and zero dueling regret. Since MAXMINLCB has no access to f , it leverages the confidence sets of Corollary 5 and uses a pessimistic approach by considering the LCB instead. There are two crucial properties of the Follower specific to this game. First, the Follower can not do worse than the Leader with respect to the LCB t . In any scenario, the Follower can match the Leader's action which results in LCB t ( x t , x ′ t ) = 0 . 5 . Second, for sufficiently tight confidence sets, the Follower will not select sub-optimal actions. In this case, the Leader's best action must be optimal as it anticipates the Follower's response and Equation (7) recovers the optimal actions. Therefore, the objective value of the game considered in Equation (7) is always less than, or equal to the objective of the game with known utility function f , i.e., LCB t ( x t , x ′ t ) ≤ 0 . 5 = f ( x ⋆ , x ⋆ ) and the gap shrinks with the confidence sets. Overall, the Stackelberg game in Equation (7) can be considered as a lower approximation of the game played with known utility function f .

The primary challenge for MAXMINLCB is to sample action pairs that sufficiently shrink the confidence sets for the optimal actions without accumulating too much regret. MAXMINLCB balances this exploration-exploitation trade-off naturally with its game theoretic formulation. We view the selection of x t to be exploitative by trying to maximize the unknown utility f ( x t ) and minimizing regret. On the other hand, x ′ t is chosen to be the most competitive opponent to x t , i.e., testing whether the condition LCB t ( x t , x ′ t ) ≥ 0 . 5 holds. Note that LCB t is pessimistic concerning x t making it robust against the uncertainty in the confidence set estimation. At the same time, LCB t is an optimistic estimate for x ′ t encouraging exploration. In our main theoretical result, we prove that under the assumptions of Corollary 5, MAXMINLCB achieves sublinear regret on the dueling bandit problem.

0

250

500

750

1000

1250

1500

1750

2000

Time Step:

T

0

100

200

300

400

500

Regret:

R L

(

T

)

Ind-UCB

LGP-UCB

(

Ours

)

GP-UCB

0

200

400

600

800

1000

Time Step:

T

0

25

50

75

100

125

Regret:

R D

(

T

)

MaxMinLCB

(

Ours

)

MaxInP

Doubler

RUCB

IDS

(a) Logistic feedback

(b) Preference feedback

Theorem 6. Suppose the utility function f lies in H k with a norm bounded by B , and that kernel k satisfies the assumption of Proposition 4. Let δ ∈ (0 , 1] and choose the exploration coefficient β D t ( δ ) as in Corollary 5. Then MAXMINLCB satisfies the anytime dueling regret of

where γ D T is the T -step information gain of kernel k D and C 3 = (8 + 2 κ ) / √ log(1 + 4( λκ ) -1 ) .

The proof is left to Appendix C.2. The information gain γ D T in Theorem 6 quantifies the structural complexity of the RKHS corresponding to k D and its dependence on T is fairly understood for kernels commonly used in applications of bandit optimization. As an example, for a Matérn kernel of smoothness ν defined over a d -dimensional domain, γ T = ˜ O ( T d/ (2 ν + d ) ) [Remark 2, Vakili et al., 2021] and the corresponding regret bound grows sublinearly with T .

Restricting the optimization domain to M t ⊂ X is common in the literature [Zoghi et al., 2014a, Saha, 2021] despite being challenging in applications with large or continuous domains. We conjecture that MAXMINLCB would enjoy similar regret guarantees without restricting the selection domain to M t as done in Equation (7). This claim is supported by our experiments in Section 6.2 which are carried out without such restriction on the optimization domain.

6 Experiments

Our experiments are on finding the maxima of test functions commonly used in (non-convex) optimization literature [Jamil and Yang, 2013], given only preference feedback. These functions cover challenging optimization landscapes including several local optima, plateaus, and valleys, allowing us to test the versatility of MAXMINLCB. We use the Ackley function for illustration in the main text and provide the regret plots for the remainder of the functions in Appendix E. For all experiments, we set the horizon T = 2000 and evaluate all algorithms on a uniform mesh over the input domain of size 100 . Additionally, we conducted experiments on the Yelp restaurant review dataset to demonstrate the applicability of MAXMINLCB on real-world data and its scaling to larger domains. All experiments are run across 20 random seeds and reported values are averaged over the seeds, together with standard error. The environments and algorithms are implemented 2 end-to-end in JAX [Bradbury et al., 2018].

6.1 Benchmarking Confidence Sets

Performance of MAXMINLCB relies on validity and tightness of the LCB. We evaluate the quality of our kernelized confidence bounds, using the potentially simpler task of bandit optimization given logistic feedback. To this end, we fix the acquisition function for the logistic bandit algorithms to the Upper Confidence Bound (UCB) function, and benchmark different methods for calculating the confidence bound. We refer to the algorithm instantiated with the confidence sets of Theorem 2 as LGP-UCB (c.f. Algorithm 2). The IND-UCB approach assumes that actions are uncorrelated, and

±

±

±

±

±

±

maintains an independent confidence interval for each action as in Lattimore and Szepesvári [2020, Algorithm 3]. This demonstrates how LGP-UCB utilizes the correlation between actions. We also implement LOG-UCB1 [Faury et al., 2020] that assumes that f is a linear function, i.e., f ( x ) = θ T x to highlight the improvements gained by kernelization. Last, we compare LGP-UCB with GP-UCB [Srinivas et al., 2010] that estimates probabilities s ( f ( · )) via a kernelized ridge regression task. This comparison highlights the benefits of using our kernelized logistic estimator (Proposition 1) over regression-based approaches [Xu et al., 2020, Kirschner and Krause, 2021, Mehta et al., 2023b,a]. Figure 1a shows that the cumulative regret of LGP-UCB is the lowest among the baselines. GP-UCB performs closest to LGP-UCB, however, it accumulates regret linearly during the initial steps. Note that GP-UCB and LGP-UCB differ in the estimation of the utility function f t while estimating the width of the confidence bounds similarly. This result suggests that using the logistic-type loss (3) to infer the utility function is advantageous. As expected, IND-UCB converges at a slower rate than LGP-UCB and GP-UCB due to ignoring the correlation between arms while LOG-UCB1's regret grows linearly as the Ackley function is misspecified under the assumption of linearity. We defer the results on the rest of the utility functions to Table 2 in Appendix E and the figures therein.

6.2 Benchmarking Acquisition Functions

In this section, we compare MAXMINLCB with other utility-based bandit algorithms. To isolate the benefits of our acquisition function, we instantiate all algorithms with the same confidence sets, and use our improved preferred-based bound of Corollary 5. Therefore, our implementation differs from the corresponding references, while we refer to the algorithms by their original name. We consider the following baselines. DOUBLER and MULTISBM [Ailon et al., 2014] choose x t as a reference action from the recent history of actions and pair it with x ′ t which maximizes the joint UCB (cf. Algorithm 5 and 6). RUCB [Zoghi et al., 2014a] chooses x ′ t similarly, however, it selects the reference action uniformly at random from M t (Algorithm 7). MAXINP [Saha, 2021] also maintains the set of plausible maximizers M t , and at each time step, it selects the pair of actions that maximize σ D t ( x , x ′ ) (Algorithm 3). IDS [Kirschner and Krause, 2021] selects the reference action greedily by maximizing f t , and pairs it with an informative action (Algorithm 4). Notably, all algorithms, with the exception of MAXINP, choose one of the actions independently and use it as a reference point when selecting the other one. Figure 3 illustrates the differences in action selection between UCB, maximum information, and MAXMINLCB approaches. We note that POP-BO [Xu et al., 2024] and MULTISBM only differ in the estimation of the confidence set. Since we deploy the same confidence set for all acquisition functions, the two algorithms are equivalent and we use MULTISBM in our results, however, comparisons hold for POP-BO as well.

Figure 1b benchmarks the algorithms using the Ackley utility function, where MAXMINLCB outperforms the baselines. All algorithms suffer from close-to-linear regret during the initial stages of learning, suggesting that there is an inevitable exploration phase. Notably, MAXMINLCB, IDS, and DOUBLER are the first to select actions with high utility, while RUCB and MAXINP explore for longer. Table 1 shows the dueling regret for all utility functions. MAXMINLCB consistently outperforms the baselines across the analyzed functions and achieves a low standard error, supporting its efficiency in balancing exploration and exploitation in the preference feedback setting. Only MULTISBM shows comparable performance to MAXMINLCB and even outperforms it on the Matyas function which is a relatively smooth function posing a simple optimization problem. However, MAXMINLCB achieves lower regret on the Ackley and Eggholder functions which obtain many local optima and steeper

0

200

400

600

800

1000

Time Step:

T

0

25

50

75

100

125

150

Regret:

R D

(

T

)

MaxMinLCB

(

Ours

)

MaxInP

Doubler

RUCB

IDS

gradients showing that MAXMINLCB is preferable for challenging optimization problems. Other acquisition functions work well only in certain cases, e.g., IDS achieves the smallest regret for optimizing Matyas, while RUCB excels on the Branin function. This indicates the challenges each utility function offers and the performance of the action selection is task dependent. The consistent performance of MAXMINLCB demonstrates its robustness against the underlying unknown utility function.

6.3 Real-world Experiment

To demonstrate the scalability and applicability of MAXMINLCB, we conduct an experiment on the Yelp dataset of restaurant reviews, which consists of 275 restaurants and 20 users after the pre-processing stage. For each user, we want to find the restaurants that best fit their preference, via making sequential recommendations and receiving comparative feedback. We define the action space X by assigning to each restaurant their respective 32 -dimensional embedding of their reviews, i.e., X ⊆ R 32 . The dataset provides utility values for users in the form of ratings on the scale of 1 to 5 , however, not all users rated every restaurant.

We estimate missing ratings using collaborative filtering [Schafer et al., 2007]. Further details on the data processing are deferred to Appendix D.1. Figure 2 shows that the results of this larger problem align with previous conclusions. MAXMINLCB remains the best-performing algorithm with MULTISBM following second. The cumulative regret is also reported in Table 1. Note that neither of the algorithms is tuned or modified for this experiment. These results are only intended to demonstrate that 1) the computations easily scale, and 2) the kernelized approach is still applicable in a text-based domain, by using high-quality vector embeddings.

7 Conclusion

We addressed the problem of bandit optimization with preference feedback over large domains and complex targets. We proposed MAXMINLCB, which takes a game-theoretic approach to the problem of action selection under comparative feedback, and naturally balances exploration and exploitation by constructing a zero-sum Stackelberg game between the action pairs. MAXMINLCB achieves a sublinear regret for kernelized utilities, and performs competitively across a range of experiments. Lastly, by uncovering the equivalence of learning with logistic or comparative feedback, we propose kernelized preference-based confidence sets, which may be employed in adjacent problems, such as reinforcement learning with human feedback. The technical setup considered in this work serves as a foundation for a number of applications in mechanism design, such as preference elicitation and welfare optimization from multiple feedback sources for social choice theory, which we leave as future work.

Acknowledgments and Disclosure of Funding

We thank Scott Sussex for his thorough feedback. This research was supported by the European Research Council (ERC) under the European Union's Horizon 2020 research and Innovation Program Grant agreement No. 815943. Barna Pásztor was supported by an ETH AI Center doctoral fellowship, and Parnian Kassraie by a Google PhD Fellowship.

References

Yasin Abbasi-Yadkori. Online learning for linearly parametrized control problems . PhD thesis, University of Alberta, 2013.

Nir Ailon, Zohar Karnin, and Thorsten Joachims. Reducing dueling bandits to cardinal bandits. In International Conference on Machine Learning , pages 856-864. PMLR, 2014.

Fabio Aiolli and Alessandro Sperduti. Learning preferences for multiclass problems. Advances in neural information processing systems , 17, 2004.

Sheldon Axler. Measure, integration & real analysis . Springer Nature, 2020.

Viktor Bengs, Róbert Busa-Fekete, Adil El Mesaoudi-Paul, and Eyke Hüllermeier. Preference-based online learning with dueling bandits: A survey. The Journal of Machine Learning Research , 2021.

Alina Beygelzimer, David Pal, Balazs Szorenyi, Devanathan Thiruvenkatachari, Chen-Yu Wei, and Chicheng Zhang. Bandit multiclass linear classification: Efficient algorithms for the separable case. In International Conference on Machine Learning , pages 624-633. PMLR, 2019.

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, et al. JAX: composable transformations of Python+ NumPy programs, 2018. URL http://github.com/google/ jax .

Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika , 1952.

Eric Brochu, Vlad M Cora, and Nando De Freitas. A tutorial on bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. arXiv preprint , 2010.

José-Fernando Camacho-Vallejo, Carlos Corpus, and Juan G Villegas. Metaheuristics for bilevel optimization: A comprehensive review. Computers & Operations Research , page 106410, 2023.

Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, et al. Open problems and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learning Research , 2023.

Bangrui Chen and Peter I Frazier. Dueling bandits with weak regret. In International Conference on Machine Learning , pages 731-739. PMLR, 2017.

Sayak Ray Chowdhury and Aditya Gopalan. On kernelized multi-armed bandits. In International Conference on Machine Learning , pages 844-853. PMLR, 2017.

Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems , 30, 2017.

Wei Chu and Zoubin Ghahramani. Preference learning with gaussian processes. In Proceedings of the 22nd international conference on Machine learning , pages 137-144, 2005.

Mathieu Dagréou, Pierre Ablin, Samuel Vaiter, and Thomas Moreau. A framework for bilevel optimization that enables stochastic and global variance reduction algorithms. Advances in Neural Information Processing Systems , 35:26698-26710, 2022.

Miroslav Dudík, Katja Hofmann, Robert E Schapire, Aleksandrs Slivkins, and Masrour Zoghi. Contextual dueling bandits. In Conference on Learning Theory , pages 563-587. PMLR, 2015.

Moein Falahatgar, Alon Orlitsky, Venkatadheeraj Pichapati, and Ananda Theertha Suresh. Maximum selection and ranking under noisy comparisons. In International Conference on Machine Learning , pages 1088-1096. PMLR, 2017.

Louis Faury, Marc Abeille, Clément Calauzènes, and Olivier Fercoq. Improved optimistic algorithms for logistic bandits. In International Conference on Machine Learning , pages 3052-3060. PMLR, 2020.

Louis Faury, Marc Abeille, Kwang-Sung Jun, and Clément Calauzènes. Jointly efficient and optimal algorithms for logistic bandits. In International Conference on Artificial Intelligence and Statistics , pages 546-580. PMLR, 2022.

Sarah Filippi, Olivier Cappe, Aurélien Garivier, and Csaba Szepesvári. Parametric bandits: The generalized linear case. Advances in neural information processing systems , 2010.

Dylan J Foster and Akshay Krishnamurthy. Contextual bandits with surrogate losses: Margin bounds and efficient algorithms. Advances in Neural Information Processing Systems , 31, 2018.

Saeed Ghadimi and Mengdi Wang. Approximation methods for bilevel programming. arXiv preprint arXiv:1802.02246 , 2018.

Javier González, Zhenwen Dai, Andreas Damianou, and Neil D Lawrence. Preferential bayesian optimization. In International Conference on Machine Learning , pages 1282-1291. PMLR, 2017.

Neil Houlsby, Ferenc Huszár, Zoubin Ghahramani, and Máté Lengyel. Bayesian active learning for classification and preference learning. arXiv preprint arXiv:1112.5745 , 2011.

Kevin G Jamieson and Robert Nowak. Active ranking using pairwise comparisons. Advances in neural information processing systems , 24, 2011.

Momin Jamil and Xin-She Yang. A literature survey of benchmark functions for global optimisation problems. International Journal of Mathematical Modelling and Numerical Optimisation , 4(2): 150-194, 2013.

Robert G Jeroslow. The polynomial hierarchy and a simple model for competitive analysis. Mathematical programming , 32(2):146-164, 1985.

Xiang Ji, Huazheng Wang, Minshuo Chen, Tuo Zhao, and Mengdi Wang. Provable benefits of policy learning from human preferences in contextual bandit problems. arXiv preprint arXiv:2307.12975 , 2023.

Johannes Kirschner and Andreas Krause. Bias-robust bayesian optimization via dueling bandits. In International Conference on Machine Learning . PMLR, 2021.

Johannes Kirschner, Tor Lattimore, and Andreas Krause. Information directed sampling for linear partial monitoring. In Conference on Learning Theory , pages 2328-2369. PMLR, 2020.

Junpei Komiyama, Junya Honda, and Hiroshi Nakagawa. Optimal regret analysis of thompson sampling in stochastic multi-armed bandit problem with multiple plays. In International Conference on Machine Learning , pages 1152-1161. PMLR, 2015.

Wataru Kumagai. Regret analysis for continuous dueling bandit. Advances in Neural Information Processing Systems , 30, 2017.

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Peter D Lax. Functional analysis , volume 55. John Wiley & Sons, 2002.

Junghyun Lee, Se-Young Yun, and Kwang-Sung Jun. Improved regret bounds of (multinomial) logistic bandits via regret-to-confidence-set conversion. In International Conference on Artificial Intelligence and Statistics , pages 4474-4482. PMLR, 2024.

Viraj Mehta, Vikramjeet Das, Ojash Neopane, Yijia Dai, Ilija Bogunovic, Jeff Schneider, and Willie Neiswanger. Sample efficient reinforcement learning from human feedback via active exploration. arXiv preprint , 2023a.

Viraj Mehta, Ojash Neopane, Vikramjeet Das, Sen Lin, Jeff Schneider, and Willie Neiswanger. Kernelized offline contextual dueling bandits. arXiv preprint , 2023b.

Petrus Mikkola, Milica Todorovi´ c, Jari Järvi, Patrick Rinke, and Samuel Kaski. Projective preferential bayesian optimization. In International Conference on Machine Learning . PMLR, 2020.

Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprint , 2023.

Tung Nguyen, Qiuyi Zhang, Bangding Yang, Chansoo Lee, Jorg Bornschein, Yingjie Miao, Sagi Perel, Yutian Chen, and Xingyou Song. Predicting from strings: Language model embeddings for bayesian optimization. arXiv preprint , 2024.

Aadirupa Saha. Optimal algorithms for stochastic contextual preference bandits. Advances in Neural Information Processing Systems , 34:30050-30062, 2021.

Aadirupa Saha and Akshay Krishnamurthy. Efficient and optimal algorithms for contextual dueling bandits under realizability. In International Conference on Algorithmic Learning Theory . PMLR, 2022.

Aadirupa Saha, Aldo Pacchiano, and Jonathan Lee. Dueling rl: Reinforcement learning with trajectory preferences. In Proceedings of The 26th International Conference on Artificial Intelligence and Statistics . PMLR, 2023.

J Ben Schafer, Dan Frankowski, Jon Herlocker, and Shilad Sen. Collaborative filtering recommender systems. In The adaptive web: methods and strategies of web personalization , pages 291-324. Springer, 2007.

Bernhard Schölkopf, Ralf Herbrich, and Alex J Smola. A generalized representer theorem. In International conference on computational learning theory , pages 416-426. Springer, 2001.

Ankur Sinha, Pekka Malo, and Kalyanmoy Deb. A review on bilevel optimization: From classical to evolutionary approaches and applications. IEEE transactions on evolutionary computation , 22(2): 276-295, 2017.

Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proceedings of the 27th International Conference on International Conference on Machine Learning , 2010.

Heinrich von Stackelberg. Theory of the market economy . Oxford University Press, 1952.

Yanan Sui, Vincent Zhuang, Joel W Burdick, and Yisong Yue. Multi-dueling bandits with dependent arms. arXiv preprint arXiv:1705.00253 , 2017.

Shion Takeno, Masahiro Nomura, and Masayuki Karasuyama. Towards practical preferential bayesian optimization with skew gaussian processes. In International Conference on Machine Learning , pages 33516-33533. PMLR, 2023.

Maegan Tucker, Ellen Novoseller, Claudia Kann, Yanan Sui, Yisong Yue, Joel W Burdick, and Aaron D Ames. Preference-based learning for exoskeleton gait optimization. In 2020 IEEE international conference on robotics and automation (ICRA) . IEEE, 2020.

Tanguy Urvoy, Fabrice Clerot, Raphael Féraud, and Sami Naamane. Generic exploration and k-armed voting bandits. In International Conference on Machine Learning . PMLR, 2013.

Sattar Vakili, Kia Khezeli, and Victor Picheny. On information gain and regret bounds in gaussian process bandits. In International Conference on Artificial Intelligence and Statistics , pages 82-90. PMLR, 2021.

Michal Valko, Nathaniel Korda, Rémi Munos, Ilias Flaounas, and Nelo Cristianini. Finite-time analysis of kernelised contextual bandits. arXiv preprint arXiv:1309.6869 , 2013.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press, 2019.

Justin Whitehouse, Zhiwei Steven Wu, and Aaditya Ramdas. Improved self-normalized concentration in hilbert spaces: Sublinear regret for gp-ucb. arXiv preprint arXiv:2307.07539 , 2023.

Huasen Wu and Xin Liu. Double thompson sampling for dueling bandits. Advances in neural information processing systems , 29, 2016.

Wenjie Xu, Wenbin Wang, Yuning Jiang, Bratislav Svetozarevic, and Colin N Jones. Principled preferential bayesian optimization. arXiv preprint arXiv:2402.05367 , 2024.

Yichong Xu, Aparna Joshi, Aarti Singh, and Artur Dubrawski. Zeroth order non-convex optimization with dueling-choice bandits. In Conference on Uncertainty in Artificial Intelligence . PMLR, 2020.

Yisong Yue and Thorsten Joachims. Interactively optimizing information retrieval systems as a dueling bandits problem. In Proceedings of the 26th Annual International Conference on Machine Learning , 2009.

Yisong Yue, Josef Broder, Robert Kleinberg, and Thorsten Joachims. The k-armed dueling bandits problem. Journal of Computer and System Sciences , 2012.

Wenhao Zhan, Masatoshi Uehara, Nathan Kallus, Jason D Lee, and Wen Sun. Provable offline reinforcement learning with human feedback. arXiv preprint , 2023.

Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons. In International Conference on Machine Learning , pages 43037-43067. PMLR, 2023.

Julian Zimmert and Yevgeny Seldin. Factored bandits. Advances in Neural Information Processing Systems , 31, 2018.

Masrour Zoghi, Shimon Whiteson, Remi Munos, and Maarten Rijke. Relative upper confidence bound for the k-armed dueling bandit problem. In International conference on machine learning . PMLR, 2014a.

Masrour Zoghi, Shimon A Whiteson, Maarten De Rijke, and Remi Munos. Relative confidence sampling for efficient on-line ranker evaluation. In Proceedings of the 7th ACM international conference on Web search and data mining , 2014b.

Contents of Appendix

A Proofs for Bandits with Logistic Feedback

̸

We have written the equations in the main text in terms of kernel matrices and function evaluations, for easier readability. For the purpose of the proof however, we mainly rely on entities in the Hilbert space. Consider the operator ϕ : X → H which corresponds to kernel k and satisfies k ( x , · ) = ϕ ( x ) . Then by Mercer's theorem, any f ∈ H k may be written as f = θ ⊤ ϕ , where θ ∈ ℓ 2 ( N ) and has a B -bounded ℓ 2 norm. For a sequence of points x 1 , . . . , x t ∈ X , we define the (infinite-dimensional) feature map Φ t = [ ϕ ( x 1 ) , · · · , ϕ ( x t )] ⊤ , which gives rise to the kernel matrix K t : R t → R t and the covariance operator S t : H k →H k , respectively defined as K t = Φ t Φ ⊤ t and S t = Φ ⊤ t Φ t . Let I t denote the t -dimensional identity matrix, and I H be the identity operator acting on the RKHS. Then it is widely known that the covariance and kernel operators are connected via det( I H + ρ -2 S t ) = det( I t + ρ -2 K t ) for any t ≥ 1 and ρ = 0 . For operators on the Hilbert space, det( A ) refer to a Fredholm determinant [c.f. Lax, 2002]. Lastly, in the appendix, we refer to the true unknown utility function as f ⋆ ( x ) = ϕ ⊤ ( x ) θ ⋆ . In the main text, the true utility is simply referred to as f .

To analyze our function-valued confidence sequences, we start by re-writing the logistic loss function

which is strictly convex in θ and has a unique minimizer θ t which satisfies

where g t ( θ ) : H → H is a linear operator defined as

In the main text, we assumed that minimizer of L satisfies the norm boundedness condition. Here, we present a more rigorous analysis which does not assume so. Instead, we work with a projected estimator defined via

where V t = S t + κλ I H and θ t is the minimizer of L ( θ ; H t ) . Roughly put, θ P t ∈ ℓ 2 ( N ) is a norm B -bounded alternative to θ t , who also satisfies a small ∇L , and therefore, is expected to result in an accurate decision boundary. We will present our proof in terms of θ P t . This also proves the results in the main text, since θ P t = θ t if θ t itself happens to have a B -bounded norm, as assumed in the main text. Our analysis relies on a concentration bound for H -valued martingale sequences stated in Abbasi-Yadkori [2013] and later in Whitehouse et al. [2023]. Below, we have adapted the statement to match our notation.

Lemma 7 (Corollary 1 Whitehouse et al. [2023]) . Suppose the sequence ( x t ) t ≥ 1 is ( F t ) t ≥ 1 -adapted, where F t := σ ( x 1 , . . . , x t , ε 1 , . . . , ε t -1 ) and ε t are i.i.d. zero-mean σ -subGaussian noise. Consider the RKHS H corresponding to a kernel k ( x , x ′ ) = ϕ ⊤ ( x ) ϕ ( x ′ ) . Then, for any ρ > 0 and δ ∈ (0 , 1) , we have that, with probability at least 1 -δ , simultaneously for all t ≥ 0 ,

where V t = S t + ρ 2 I H .

The following lemma, which extends Faury et al. [2020, Lemma 8] to H -valued operators, expresses the closeness of θ t and θ ⋆ in the gradient space, with respect to the norm of the covariance matrix.

Lemma 8 (Gradient Space Confidence Bounds) . Set 0 < δ < 1 . Under the assumptions of Theorem 2

where V t = S t + κλ I H .

Proof of Lemma 8. Recall that g t ( θ ) = ∑ τ ≤ t s ( θ ⊤ ϕ ( x τ )) ϕ ( x τ ) + λ θ . Then it is straighforward to show that

Since θ t is a minimizer of L t , it holds that g t ( θ t ) = ∑ τ ≤ t y τ ϕ ( x τ ) . This allows us to write,

where ε τ = y τ -s ( ϕ ⊤ ( x τ ) θ ⋆ ) is a history dependent random variable in [0 , 1] due to the data model. To bound the first term, we recognize that any random variable in [0 , 1] is σ sub-Gaussian with σ = 0 . 5 and apply Lemma 7. We obtain that for all t ≥ 0 , with probability greater than 1 -δ

since γ t ( ρ ) = sup x 1 ,..., x t 1 2 log det( I t + ρ -2 K t )) . To bound the second term in (9), note that S t = Φ ⊤ t Φ t is PSD and therefore V t ≥ κλ I H . Then

concluding the proof.

The following lemma shows the validity of our parameter-space confidence sets.

Lemma 9. Set 0 < δ < 1 and consider the confidence sets

Then,

Proof of Lemma 9. We check if θ ⋆ ∈ Θ t ( δ ) by bounding the following norm

Lastly, we prove a formal extension of Theorem 2.

Theorem 10 (Theorem 2 - Formal) . Set 0 < δ < 1 and consider the confidence sets E t ( δ ) ⊂ H k

Then, simultanously for all x ∈ X , f ∈ E t ( δ ) and t ≥ 0

with probability greater than 1 -δ , where

Proof of Theorem 10. For simplicity in notation let us define

Suppose ⊤

where the third to last inequality holds with probability greater than 1 -δ , but the rest of the inequalities hold deterministically.

Given the confidence set of Theorem 2, we give extend the LGP-UCB algorithm of Faury et al. to the kernelized setting (c.f. Algorithm 2) and prove that it satisfies sublinear regret.

Proof of Corollary 3. Recall that if x t is the maximizer of the UCB, then

Then using Theorem 10, we obtain the following for the regret at step t ,

Algorithm 2 LGP-UCB

Initialize Set ( β t ) t ≥ 1 according to Theorem 2.

Choose an optimistic action via

Observe y t and append history.

Calculate f acc. to Proposition 1 and update σ acc. to Theorem 2.

t t end for with probability greater than 1 -δ for all t ≥ 0 . Which allows us to bound the cumulative regret as,

B Helper Lemmas for Appendix A

Lemma 11 (Mean-Value Theorem) . For any x ∈ X and θ 1 , θ 2 ∈ ℓ 2 ( N ) it holds that where

Proof of Lemma 11. For any differentiable function s : R → R by the fundamental theorem of calculus we have

We change the variable of integration to ν = ( z -z 1 ) / ( z 2 -z 1 ) , then z = νz 2 +(1 -ν ) z 1 and re-writing the integral in terms of ν gives,

Letting z 1 = θ ⊤ 1 ϕ ( x ) and z 2 = θ ⊤ 2 ϕ ( x ) concludes the proof.

Lemma12 (Gradients to Parameters Conversion) . For all t ≥ 0 and norm B -bounded θ 1 , θ 2 ∈ ℓ 2 ( N ) .

Proof of Lemma 12. We prove the lemma through an auxiliary operator G t ( θ 1 , θ 2 ) operating on H k

where α is defined in (10).

Step 1. First we establish how we can go back and forth between the operator norms defined based on G t and V t . Recall that κ = sup z ≤ B 1 ˙ s ( z ) . Therefore, κ -1 ≤ ˙ s ( z ) for all z < B , implying that α ( x ; θ 1 , θ 2 ) ≥ ∫ 1 0 κ -1 d ν = κ -1 . We can then conclude,

Step 2. Now by the definition of g t ( θ ) ,

Therefore,

Step 3. Putting together the previous two steps, we can bound the V t -norm over the parameters to the V -1 t role in the gradients,

concluding the proof.

The following two lemmas are standard results in kernelized bandits [Srinivas et al., 2010, Chowdhury and Gopalan, 2017, e.g.,]. We include it here for completeness.

Lemma 13. Let σ t be as defined in (4) . Then √ λκ ∥ ϕ ( x ) ∥ V -1 t = σ t ( x ) , for any x ∈ X .

Proof of Lemma 13. We start by stating some identities which will later be of use. First note that

Moreover, by definition of k t we have

which allow us to write

which gives and obtain

Given the above equation, we conclude the proof by the following chain of equations:

To obtain the third equation we have used the fact that for bounded operators on Hilbert spaces, the inverse of the adjoint is equal to the adjoint of the inverse [e.g., Theorem 10.19, Axler, 2020]. Re-ordering the equation above we obtain σ 2 t ( x ) = λκ ∥ ϕ ( x ) ∥ 2 V -1 t , concluding the proof.

Lemma 14 (Controlling posterior variance with information gain) . For all T ≥ 1 ,

Proof of Lemma 14. By Srinivas et al. [2010, Lemma 5.3],

Following the technique in Srinivas et al. [2010, Lemma 5.4], since σ 2 t ≤ 1 , then ( λκ ) -1 σ 2 t ∈ [0 , ( λκ ) -1 ] . Now for any z ∈ [0 , ( λκ ) -1 ] , z ≤ C log(1 + z ) where C = 1 / ( λκ log(1 + ( λκ ) -1 )) . We then may write,

Putting both together proves the first inequality of the lemma. As for the dueling case, we can easily check that σ D t ≤ 2 , and a similar argument yields the second inequality.

C Proofs for Bandits with Preference Feedback

This section presents the proof of main results in Section 5, and our additional contributions in the kernelized Preference-based setting.

C.1 Equivalence of Preference-based and Logistic Losses

We start by establishing the equivalence between the logistic loss (3) and dueling loss (6).

Proof of Proposition 4. By Mercer's theorem, we know that the kernel function k has eigenvalue eigenfunction pairs ( √ λ i , ˜ ϕ i ) for i ≥ 1 where ˜ ϕ i are orthonormal. Then k ( x , x ′ ) = ∑ i ≥ 1 ϕ i ( x ) ϕ i ( x ′ ) with ϕ i ( x ) = √ λ i ˜ ϕ i ( x ) . Now applying the definition of k D , it holds that k D ( z , z ′ ) = ∑ i ≥ 1 ψ ⊤ i ( z ) ψ i ( z ′ ) where ψ i ( z ) = √ λ i ( ϕ i ( x ) -ϕ i ( x ′ )) . It is straighforward to check that ψ i are the eigenfunctions of k D , however, they may not be orthonormal. We have,

where b i = ∫ ˜ ϕ i ( x )d( x ) . By the assumption of the proposition, we have b i = 0 . However, this assumption holds automatically for all kernels commonly used in applications, e.g. any translation invariant kernel, over many domains, since ˜ ϕ i for such kernels are a sine basis.

Now since f ∈ H k , it may be decomposed f = ∑ i ≥ 1 β i ϕ i and ∥ f ∥ 2 k = ∑ i ≥ 1 β 2 i ≤ ∞ . And set the difference function to h ( x , x ′ ) = ∑ i ≥ 1 β i ψ i ( z ) . We can then bound the RKHS norm of h w.r.t. the

̸

Now by Mercer's theorem, h ∈ H k D since it is decomposable as a sum of k D eigenfunctions, and attains a B -bounded k D -norm which we showed to be equal to ∥ f ∥ k . The other direction of the statement is proved the same way.

Proof of Corollary 5. Consider the utility function f and define h ( x , x ′ ) := f ( x ) -f ( x ′ ) . Then by Proposition 4, h is in RKHS of k D with a k D -norm bounded by B . We may estimate h by minimizing L L k D ( · ; H t ) . Now invoking Theorem 2 with the dueling kernel we have,

concluding the proof by definition of h .

C.2 Proof of the Preference-based Regret Bound

Recall Corollary 5, which states

with high probability simultaneously for all ( x , x ′ ) and t ≥ 1 . For simplicity in notation in the rest of this section, we define ω t ( x , x ′ ) := β D t ( δ ) σ D t ( x , x ′ ) and

Note that ω t ( x , x ′ ) = ω t ( x ′ , x ) by the symmetry of the dueling kernel k D . Furthermore, recall the notation h ( x , x ) = f ( x ) -f ( x ) .

Proof of Theorem 6. Step 1: First, we connect the term of x ′ t in the dueling regret defined in Equation (1) to that of x t . Note that both s ( f ( x ∗ ) -f ( x ′ t )) and s ( f ( x ∗ ) -f ( x t )) are greater than 0 . 5 due to the optimality of x ⋆ and the sigmoid function s is concave on the interval [0 . 5 , ∞ ) . Using the definition of concavity, we get

where the second line comes from the derivative of the sigmoid function, ˙ s ( x ) = s ( x )(1 -s ( x )) = s ( x ) s ( -x ) , and in the last line we use s ( f ( x t ) -f ( x ∗ )) ≤ 0 . 5 . Using Equation (15), we can upper bound the dueling regret in Equation (1) as

Step 2 : Next, we show that the single-step regret is bounded by ω ( x , x ) .

First, consider the term s ( f ( x ) -f ( x t t t ′ t ⋆ ))

In the last inequality, we used that x t ∈ M t implying that 0 . 5 -s ( h t ( x t , x ⋆ )) ≤ ω t ( x t , x ⋆ ) . Combining Equation (16) and Equation (17) implies that

̸

Now, we bound ω t ( x t , x ⋆ ) by ω t ( x t , x ′ t ) . If x t = x ⋆ , then ω t ( x t , x ⋆ ) = 0 ≤ ω t ( x t , x ′ t ) . If x ′ t = x ⋆ , then the two expressions are equivalent. Now, assume that x t = x ∗ and x ′ t = x ⋆ and consider ω t ( x t , x ⋆ ) .

Case 1: Assume that UCB t ( x t , x ⋆ ) ≤ UCB t ( x t , x ′ t ) . Then, where we used the assumption in the first inequality and the definition of x ′ t in the second inequality. Case 2: Assume that UCB t ( x t , x ⋆ ) ≥ UCB t ( x t , x ′ t ) . First note the following connection between LCB t and UCB t .

̸

where we used the equality s ( z ) = 1 -s ( -z ) in the second equality. Using Equation (20), the assumption of Case 2 can be rewritten as UCB t ( x t , x ⋆ ) ≥ UCB t ( x t , x ′ t ) , implying that

Similarly, LCB t ( x t , x ′ t ) ≤ LCB t ( x t , x ⋆ ) implies

Combining Equation (21) and Equation (22), we get

Combining Equation (19) and Equation (23), we can rewrite the first term in Equation (18) to get

It remains to bound the second term of Equation (24). By the Mean-Value Theorem, ∃ z ∈ [0 , h ( x t , x ′ t )] such that

Now since κ = sup z ≤ B 1 / ˙ s ( z ) then,

Next, we consider the term s ( h ( x t , x ′ t )) -0 . 5 in Equation (25). Note that x t , x ′ t ∈ M t implies that

Additionally note that LCB t ( x t , x t ) = 0 . 5 for all x t , therefore, by the definition of x ′ t , LCB t ( x t , x ′ t ) ≤ 0 . 5 . It implies that

From Equation (26) and Equation (27), it follows that

furthermore, and similarly

From Equation (28) and Equation (29), it follows that

Combining Equation (30) with Equation (25), it follows

Using Equation (31) in Equation (24) and the fact that s ( f ( x ∗ ) -f ( x t )) ≤ 1 , we get

Therefore, for the cumulative dueling regret it holds

with probability greater than 1 -δ for all T ≥ 1 .

C.3 Extending Algorithms for Linear Dueling Bandits to Kernelized Setting

Maximum Informative Pair Algorithm. Proposed in Saha [2021] for linear utilities, the MAXINP algorithm similarly maintains a set of plausible maximizer arms, and picks the pair of actions that have the largest joint uncertainty, and therefore are expected to be informative. Algorithm 3 present the kernelized variant of this algorithm. Using Corollary 5, we can show that the kernelized MAXINP also satisfies a ˜ O ( γ T √ T ) regret.

Theorem 15. Let δ ∈ (0 , 1] and choose the exploration coefficient β D t ( δ ) as defined in Corollary 5. Then MAXINP satisfies the anytime dueling regret guarantee of

where γ D T is the T -step information gain of kernel k D and C 2 = 4 / √ log(1 + 4( λκ ) -1 ) .

Algorithm 3 MAXINP- Kernelized Variant

Play the most informative pair via

Observe y t and append history.

Update h t +1 and σ D t +1 and the set of plausible maximizers

end for

Proof of Theorem 15. When selecting ( x t , x ′ t ) according to Algorithm 3, we choose the pair via

where action space is restricted to M t and therefore, where we have used the identity s ( -z ) = 1 -s ( z ) . Simultaneously for all t ≥ 1 , we can bound the single-step dueling regret with probability greater than 1 -δ

where for the first inequality we have invoked Corollary 5. Then for the regret satisfies

with probability greater than 1 -δ for all T ≥ 1 .

Dueling Information Directed Sampling (IDS) Algorithm. To choose actions at each iteration t , MAXINP and MAXMINLCB require solving an optimization problem on X × X . The Dueling IDS approach addresses this issue and presents an algorithm which requires solving an optimization problem on X × [0 , 1] and is computationally more efficient when d 0 > 1 . This work considers kernelized utilities, however, assumes the probability of preference itself is in an RKHS and solves a kernelized ridge regression problem to estimate the probability s ( h ( x , x ′ ) . In the following, we present an improved version of this algorithm, by considering the preference-based loss (6) for estimating the utility function. We modify the algorithm and the theoretical analysis to accommodate this.

Consider the sub-optimality gap ∆( x ) := h ( x ⋆ , x ) for an action x ∈ X . We may estimate this gap using the reward estimate maximizer ˆ x ⋆ t := arg max x ∈X f t ( x ) . Suppose we choose ˆ x ⋆ t as one of the actions. Then u t shows an optimistic estimate of the highest obtainable reward at this step:

where ˜ β t is the exploration coefficient. We bound ∆( x ) by the estimated gap

for t ≥ 1 do

Initialize Set ( β t ) t ≥ 1 according to Theorem 2.

for t ≥ 1 do

Find a greedy action via fixing any point x null ∈ X and maximizing

Update u t and ˆ ∆ t ( x ) acc. to (34).

Find an informative action and the probability of selection via

Draw α t ∼ Bern( p t ) .

if α t = 1 then choose pair ( x t , x ′ t ) = ( x (1) t , x (2) t ) else choose ( x t , x ′ t ) = ( x (1) t , x (1) t ) . Observe y t and append history. Update h t +1 and σ D t +1 .

end for and show its uniform validity in Lemma 17. We can now propose the Kernelized Logistic IDS algorithm with preference feedback in Algorithm 4, as a variant of the algorithm of Kirschner and Krause.

Theorem 16. Let δ ∈ (0 , 1] and for all t ≥ 1 , set the exploration coefficient as ˜ β t = β D t ( δ ) /L . Then Algorithm 4 satisfies the anytime cumulative dueling regret guarantee of

Proof of Theorem 16. Our approach closely follows the proof of Kirschner and Krause [2021, Theorem 1]. Let P ( · ) show the set of continuous probability distributions over a domain. Define the expected average gap for a policy µ ∈ P ( X × X )

and the expected information ratio as

Algorithm 4 draws actions via µ t = (1 -p t ) δ ( x (1) t , x (1) t ) + p t δ ( x (1) t , x (2) t ) , where δ ( x , x ′ ) denotes a Direct delta. Then by Kirschner et al. [2020, Lemma 1],

which allows us to bound the regret with probability greater than 1 -δ as

since s ( · ) with its domain restricted to [ -2 B, 2 B ] is L -Lipschitz. It remains to bound Ξ t ( µ t ) , the expected information ratio for Algorithm 4. Now by definition of µ t

and similarly

allowing us to re-write the expected information ratio as

Now consider the definition of u t and let z t denote the action for which u t is achieved, i.e. z t = arg max h ( x , ˆ x ⋆ t ) + ¯ β t ( δ ) σ D t ( x , ˆ x ⋆ t ) . Then

therefore using the above chain of equations we may write

where last inequality holds due to the following argument. Recall that k ( x , x ) ≤ 1 , implying that σ D t ( x , x ′ ) 2 ≤ 4 and therefore log(1 + σ D t ( x , x ′ ) 2 ) ≥ log(1 + ( λκ ) -1 ) σ D t ( x , x ′ ) 2 / 4 , similar to Lemma 14. To conclude the proof, from (35) and (36) it holds that

with probability greater than 1 -δ , simultaneously for all T ≥ 1 .

C.4 Helper Lemmas for Appendix C.3

Lemma 17. Let 0 < δ < 1 and f ∈ H k . Suppose sup a ≤ B ˙ s ( a ) = L and sup a ≤ B 1 / ˙ s ( a ) = κ . Then

Proof of Lemma 17. Note that for any three inputs x 1 , x 2 , x 3

Therefore, from the definition of the estimated gap get

Then going back to the definition of the true gap we may write

with probability greater than 1 -δ .

Lemma 18. Assume f ∈ H k . Suppose sup a ≤ B 1 / ˙ s ( a ) = κ . Then for any 0 < δ < 1

where

Proof of Lemma 18. This lemma is effectively a weaker parallel of Corollary 5. We have

where the third to last inequality holds with probability greater than 1 -δ , but the rest of the inequalities hold deterministically.

D Details of Experiments

(1

,

2)

(1

,

3)

(2

,

1)

(2

,

3)

(3

,

1)

(3

,

2)

Actions:

(

x

,

x

′

)

0

.

0

0

.

5

1

.

0

P

[

x

/follows

x

′

]

MaxMinLCB

Optimism

True Prob.

Estimated Prob.

Confidence Set

Regret

Test Environments. We use a wide range of target functions common to the optimization literature [Jamil and Yang, 2013], to evaluate the robustness of MAXMINLCB. The results are reported in Table 1 and Table 2. Note that for the experiments we negate them all to get utilities. We use a uniform grid of 100 points over their specified domains and scale the utility values to [ -3 , 3] .

Ackley: X = [ -5 , 5] d , d = 2

Branin: X = [ -5 , 10] × [0 , 15]

Eggholder: X = [ -512 , 512] 2

Hölder: X = [ -10 , 10] 2

Matyas: X = [ -10 , 10] 2

Michalewicz: X = [0 , π ] d , d = 2 , m = 10

Rosenbrock: X = [ -5 , 10] 2

Acquisition Function Maximization. In our computations, to eliminate additional noise coming from approximate solvers, we use an exhaustive search over the domain for the action selection of LGP-UCB, MAXMINLCB, and other presented algorithms. For the numerical experiments presented in this paper, we do not consider this as a practical limitation. Due to our efficient implementation in JAX, this optimization step can be carried out in parallel and seamlessly support accelerator devices such as GPUs and TPUs.

Hyper-parameters for Logistic Bandits. We set δ = 0 . 1 for all algorithms. For GP-UCB and LGP-UCB, we set β = 1 , and 0 . 25 for the noise variance. We use the Radial Basis Function (RBF) kernel and choose the variance and length scale parameters from [0 . 1 , 1 . 0] to optimize their performance separately. For LGP-UCB, we tuned λ , the L 2 penalty coefficient in Proposition 1, on the grid [0 . 0 , 0 . 1 , 1 . 0 , 5 . 0] and B on [1 . 0 , 2 . 0 , 3 . 0] . The hyper-parameter selections were done for each algorithm separately to create a fair comparison.

Hyper-parameters for Preference-based Bandits. We tune the same parameters of LGP-UCB for the preference feedback bandit problem on the following grid: λ ∈ [0 , 0 . 1 , 1] , B ∈ [1 , 2 , 3] , and [0 . 1 , 1] for the kernel variance and length scale. The same hyper-parameters are tuned separately for every baseline .

Pseudo-code for Baselines. Algorithm 5, Algorithm 6, and Algorithm 7 described the baselines used for the benchmark of Section 6.2. MAXINP and IDS are defined in Algorithm 3 and Algorithm 4,

respectively, in Appendix C.3 alongside with their theoretical analysis. We note that DOUBLER includes an internal for-loop, therefore, we adjusted the time-horizon T such that it observes the same number of feedback y t as the other algorithms for a fair comparison.

Computational Resources and Costs. We ran our experiments on a shared cluster equipped with various NVIDIA GPUs and AMD EPYC CPUs. Our default configuration for all experiments was a single GPU with 24 GB of memory, 16 CPU cores, and 16 GB of RAM. Each experiment of the 11 configurations reported in Section 6.2 ran for about 12 hours and the experiment reported in Section 6.1 ran for 5 hours. The total computational cost to reproduce our results is around 140 hours of the default configuration. Our total computational costs including the failed experiments are estimated to be 2-3 times more.

D.1 Yelp Experiment

We filter the Yelp data 3 for restaurants in Philadelphia, USA with at least 500 reviews and users who reviewed at least 90 restaurants. The final dataset includes 275 restaurants, 20 users, and a total of 2563 reviews. We define the action space X by assigning to each restaurant their respective 32 -dimensional embedding of their reviews, i.e., X ⊆ R 32 . For each restaurant, we concatenate all reviews in the filtered dataset and we use the TEXT-EMBEDDING-3-LARGE OpenAI embedding model 4 to retrieve the embeddings. The Yelp dataset provides utility values for users in the form of ratings on the scale of 1 to 5 , however, not all users rated every restaurant. We use collaborative filtering to estimate the missing reviews [Schafer et al., 2007]. For each user, these values are used as the utilities for restaurants during the simulation. Experiments are conducted separately for each user, therefore, utility values are not aggregated but the action space is identical for each experiment.

Note that we do not assume any explicit functional form of the utility functions f that we calibrate to this data. Instead, the actions space X and utility values are derived separately from the dataset. Regardless, the results presented in Section 6.3 show that our kernelized method achieves good performance on this task.

E Additional Experiments

In this section, we provide Table 2 that details our logistic bandit benchmark, complementing the results in Section 6.1. Figure 4 and Figure 5 show the logistic and dueling regret of additional test functions, complementing the results of Table 1.

±

±

±

±

0

250

500

750

1000

1250

1500

1750

2000

Time Step:

T

0

50

100

150

200

Regret:

R L

(

T

)

Ind-UCB

LGP-UCB

(

Ours

)

GP-UCB

0

250

500

750

1000

1250

1500

1750

2000

Time Step:

T

0

100

200

300

Regret:

R D

(

T

)

MaxMinLCB

(

Ours

)

MaxInP

Doubler

RUCB

IDS

0

250

500

750

1000

1250

1500

1750

2000

Time Step:

T

0

100

200

300

400

500

Regret:

R L

(

T

)

Ind-UCB

LGP-UCB

(

Ours

)

GP-UCB

0

250

500

750

1000

1250

1500

1750

2000

Time Step:

T

0

50

100

150

200

250

Regret:

R D

(

T

)

MaxMinLCB

(

Ours

)

MaxInP

Doubler

RUCB

IDS

0

250

500

750

1000

1250

1500

1750

2000

Time Step:

T

0

100

200

300

400

500

Regret:

R L

(

T

)

Ind-UCB

LGP-UCB

(

Ours

)

GP-UCB

0

250

500

750

1000

1250

1500

1750

2000

Time Step:

T

0

50

100

150

200

Regret:

R D

(

T

)

MaxMinLCB

(

Ours

)

MaxInP

Doubler

RUCB

IDS

0

250

500

750

1000

1250

1500

1750

2000

Time Step:

T

0

50

100

150

200

Regret:

R L

(

T

)

Ind-UCB

LGP-UCB

(

Ours

)

GP-UCB

0

250

500

750

1000

1250

1500

1750

2000

Time Step:

T

0

25

50

75

100

125

Regret:

R D

(

T

)

MaxMinLCB

(

Ours

)

MaxInP

Doubler

RUCB

IDS

0

250

500

750

1000

1250

1500

1750

2000

Time Step:

T

0

100

200

300

400

500

Regret:

R L

(

T

)

Ind-UCB

LGP-UCB

(

Ours

)

GP-UCB

0

250

500

750

1000

1250

1500

1750

2000

Time Step:

T

0

100

200

300

Regret:

R D

(

T

)

MaxMinLCB

(

Ours

)

MaxInP

Doubler

RUCB

IDS

0

250

500

750

1000

1250

1500

1750

2000

Time Step:

T

0

20

40

60

80

100

Regret:

R L

(

T

)

Ind-UCB

LGP-UCB

(

Ours

)

GP-UCB

0

250

500

750

1000

1250

1500

1750

2000

Time Step:

T

0

20

40

60

80

100

Regret:

R D

(

T

)

MaxMinLCB

(

Ours

)

MaxInP

Doubler

RUCB

IDS

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We provide theoretical guarantees of the proposed algorithms in Section 4 and Section 5. In Section 6, we provide the results for our numerical experiments supporting our claims.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper is of theoretical nature. All theoretical assumptions are presented in the text, and we discuss how limiting they are. This is mainly in the Problem Setting section, or theorem statements.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Proposition 1, Theorem 2, Corollary 3, Proposition 4, Corollary 5, Theorem 6 describe our theoretical proofs with detailed explanation of the assumptions.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We describe the experimental setting in Section 6 while providing further details on the hyperparameter selection, pseudocode, and utility function definition in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We include the code used to carry out the experiments in the Supplementary Material.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide comprehensive explanations of the setting, hyperparameters, algorithms, and functions used in Section 6 and in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report all results in our figures and tables as the average and standard error over 20 random seeds.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We describe computation resources and time used for the experiments in Appendix D.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the NeurIPS Code of Ethics and conducted the research following the guidelines.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The research presented in this paper is theoretical without any immediate societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: To the best of the authors' knowledge, there is no such risk involved.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: Assets used to conduct the research presented in this paper are always cited and properly credited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers bandits with preference feedback. It first constructs a novel confidence set that covers the ground truth with high probability. Then from a Stackelberg game perspective, it proposes an efficient algorithm that enjoys tighter regret bound than SOTA.\n\nSTRENGTHS:\n1. The technique used to construct the confidence set is interesting. The resulting confidence set is tighter.\n2. The Stackelberg game perspective is interesting, and allows the author(s) to design the algorithm with better exploration-exploitation trade-off as demonstrated in the experiment.\n\nWEAKNESSES:\nThe major concern is the practical applicability of the algorithm. Seems that the proposed algorithm can hardly scale up to a higher dimension (e.g., dimension equals to 7). In the proposed algorithm, a complicated sequential optimization problem needs to be solved. Notably, the experiment only considers two-dimensional problem, in sharp contrast to recent works (e.g., 12-dimensional problem considered in Xu et al. [2024]).\n\nQUESTIONS:\n1. In the abstract, the review claims that the regret bound is 'rate-optimal'. Is there a matching lower bound?\n2. Can the results be generalized to other link functions?\n3. In line 168, should it be $\\sup_{|a|\\leq B}$ instead of $\\sup_{a\\leq B}$?\n4. Could the authors explain the linear growth of multiSBM in Fig. 1 (b)?\n\nLIMITATIONS:\nSome major comments:\n1. The paper discusses the comparisons to [Xu et al. 2024]. Seems that the theoretical gain of $T^{\\frac{1}{4}}$ mainly comes from tighter confidence set rather than the design of the algorithm. If the algorithm POP-BO in [Xu et al. 2024] is equipped with this tighter confidence set, it could also achieve the same regret bound. To see which algorithm is empirically better, it would be interesting to compare the proposed algorithm with POP-BO equipped with the tighter confidence set in this paper.\n\nBesides the points mentioned above, here are some minor comments:\n1. In line 109, I guess it should be {$0, 1$} instead of $[0, 1]$.\n2. In line 178 to 179, typo in \"ridge estimator estimator\".\n3. In line 574, abuse of the notation $s$.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers novel game-theoretic acquisition function for pairwise action selection with preference feedback. It is tailored to the setting with infinite domains and nonlinear kernelized rewards. The preference-based confidence sequences for kernelized utility functions are shown to be tight and anytime valid. The proposed algorithm MAXMINLCB is shown to satisfy a sublinear regret. Various simulations were conducted to showcase the advantage of the proposed method.\n\nSTRENGTHS:\n1. Although preference-based bandit optimization with linear utility functions is fairly well understood, such approaches cannot capture real-world problems with complex nonlinear utility functions. This paper aims to close this gap. The considered problem is timing and interesting. \n\n2. The technical contribution is non-trivial. Although there have been attempts to prove convergence of kernelized algorithms for preference-based bandits, such works employ a regression likelihood model which requires them to assume that both the utility and the probability of preference lie in an RKHS. Moreover, a sample-efficient algorithm is lacking for such approaches. In contrast, this work uses a kernelized logistic negative log-likelihood loss to infer the utility function, and provide confidence sets for its minimizer.\n\n3. Some theoretical result, like Kernelized Logistic Confidence Sequences in Theorem 2, is also of independent interest. \n\n4. In spite of a theoretical paper, it is well written and is easy to follow.\n\nWEAKNESSES:\n1. In practice, how to determine the hyper-parameters like $\\gamma_t$, $L$, and $B$ in (5)? Is there any data-driven way to select them?\n\n2. In the main Theorem 6, the regret bound is $\\gamma_T^{D}\\sqrt{T}$. The term $\\gamma_T^{D}$ is the T-step information gain of kernel, which is also a function of $T$. The authors claim that this rate improves that of Xu et al. (2024) by a factor of $T^{1/4}$. However, the cumulative regret bound in Theorem 5.2 of Xu et al. (2024) is of a similar order. Xu et al. (2024) also provided explicit regret upper bounds for various common kernels in Theorem 5.5. Hence, it is also interesting to provide an explicit form of $\\gamma_T^{D}$ for some common kernels, and to compare these regret upper bounds in a fair way. \n\n3. In Figure 1, the authors presented the result for the Ackley function, which shows a clear advantage of the proposed method. However, in more extensive simulations (e.g., Matyas function in Figure 6, ) in the appendix, the proposed method is outperformed by the competitors. It is helpful to provide some discussion on these results, and offer some insights on when the proposed method would work well. This is helpful for practitioners.\n\nQUESTIONS:\nsee Weakness\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

comments: SUMMARY:
The paper examines the problem of bandit optimization with preference feedback in large domains and nonlinear (kernelized) rewards. It introduces MAXMINLCB, which adopts a game-theoretic approach to action selection under comparative feedback. Additionally, it proposes kernelized preference-based confidence sets, which can be utilized in related problems.

STRENGTHS:
(1) Rather than jointly selecting the arms in dueling bandits, the proposed method jointly optimizes both actions by choosing them as the equilibrium of a two-player zero-sum Stackelberg game. This approach enables a more efficient exploration/exploitation trade-off.

(2) The regret guarantee presented in this paper is tighter by a factor of  $T^{1/4}$ compared to Xu et al. (2024).

WEAKNESSES:
(1) Although the paper uses a kernelized logistic model to approximate the rewards, this approach may remain too simplistic for capturing the complexity of rewards in real-world applications.

(2) The paper lacks a comparison in the experiments with the related work by Xu et al. (2024).

(3)  In real applications, it is more common to rank between two state-action pairs. However, the paper does not consider contextual information and solely focuses on the multi-armed setting, which is less interesting and useful.

QUESTIONS:
(1) Can you implement the algorithms compared in Section 6.2 using the original confidence sets from the references?

(2) Can this work be extended to contextual bandits with preference feedback?

LIMITATIONS:
Please see weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers bandit optimization with preference feedback over continuous action spaces and kernelized reward function. The goal in this problem is to minimize the dueling regret against an optimal action over a finite time-horizon. Previous works on this problem are either restricted to finite action spaces or linear reward functions. The proposed algorithm casts the problem as kernalized logistic regression and designs confidence sets for the relative preference between two actions. It then proposes an action selection strategy based on a game theoretic Leader-Follower formulation that utilizes these confidence intervals. The paper provides a regret bound as well as empirical evaluation for the proposed algorithm.\n\nSTRENGTHS:\nThe main contributions of the paper are two-fold:\n\n1. Expanding the existing literature on dueling bandits by studying kernelized reward functions under infinite and continuous action sets. This requires new techniques to bound the confidence intervals. \n\n2. Proposing a principled game-theoretic approach to action selection in dueling bandits that can be of further interest.\n\nIn my opinion these are two important contributions to the literature. Since these ideas are likely to be relevant to other learning problems with preference feedback such as RLHF, I think that the results in this paper have a good scope. The paper is well-written and the contributions are clear.\n\nWEAKNESSES:\nExperimental evaluation can include other algorithms that are known to perform better than RUCB such as RMED (Komiyama et al., 2015) and Double Thompson Sampling (Wu and Liu, 2016).\n\nQUESTIONS:\nIt would be interesting to see if the current approach can be extended to other link functions beyond the sigmoid such as probit.\n\nLIMITATIONS:\nThe paper can include a section of limitations of the current work in terms of social impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-23"}