Title: Tackling Biased Evaluators in Dueling Bandits
Venue: NeurIPS 2025
Accepted: yes
Keywords: Dueling Bandits; Evaluation Bias
Abstract: In dueling bandits, an agent explores and exploits choices (i.e., arms) by learning from their stochastic feedback in the form of relative preferences. Prior related studies focused on unbiased feedback. In practice, however, the feedback provided by evaluators can be biased. For example, human users are likely to provide biased evaluation towards large language models due to their heterogeneous background. In this work, we aim to minimize the regret in dueling bandits considering evaluators’ biased feedback. We begin with a benchmark case where evaluators’ bias information is known. Solving the known-bias case is nontrivial, because the bias cannot be easily decoupled from the feedback. We overcome this challenge and propose an unbiased arm performance estimator and a bias-sensitive dueling bandits algorithm. We manage to analyze the regret, dealing with the complex form of the estimator, and show that the feedback either matching or opposing the ground-truth reduces the regret. Then, we study the case where evaluators’ bias information is unknown. The associated estimator can hardly be solved in closed-form due to the non-convexity of the estimator solving problem. We address this challenge and propose an extended bias-sensitive algorithm by incorporating block coordinate descent. This algorithm is proven to achieve the same order of regret (as in the known bias case) with a bounded error. Experiments show that when compared with baselines, our algorithms reduces the regret by up to 86.9%.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the previously underexplored problem of biased evaluators in the dueling bandits setting, where comparisons between arms may be corrupted. The authors first consider a benchmark setting where evaluator biases are known, and propose a bias-sensitive UCB algorithm with a novel unbiased estimator. They then extend this to the more realistic setting where the bias is unknown, proposing an iterative estimator. The paper provides theoretical regret bounds in both cases, and demonstrates that the latter has the same order of bound on the regret as the former. Extensive experiments demonstrate that the proposed methods significantly outperform standard dueling bandit algorithms, especially when evaluator bias is heterogeneous. The proposed techniques are general and can be incorporated into existing dueling bandit frameworks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- **Quality:** The paper’s main claims are supported by rigorous and well-structured proofs, demonstrating a solid theoretical foundation. Additionally, the paper includes a thorough experimental evaluation that demonstrates the superiority of the proposed methods over existing baselines and provides insight into the behavior of the key parameters introduced throughout the study.\n- **Clarity:** The authors clearly motivate the study of biased evaluators through a compelling example involving LLM selection. The proposed algorithms and accompanying proof techniques are presented with intuitive explanations in the main text, and their effectiveness is validated through a comprehensive and rigorous experimental evaluation.\n- **Significance:** This work advances the state of the art by introducing techniques that enable effective learning in environments with biased feedback. Beyond proposing novel algorithms, the authors demonstrate that their methods can be integrated with existing approaches, which enhances the significance of this work. \n- **Originality:** This is the first work to address the dueling bandits problem under biased feedback. Although the authors build on established convex optimization techniques, they employ them in a novel manner to construct provably unbiased estimators with theoretical regret guarantees.\n\n**Weaknesses:**\nRather than separating into broad quality, clarity, significance and originality categories, I will outline my main concerns in a more detailed manner below.\n\n- The primary concern with this work lies in the significance of the regret bounds presented in Theorems 1 and 2. As noted in Corollary 1, when the bias can take on multiple values (i.e., $|\\mathcal{B}|>1$), the cumulative regret becomes linear in $T$ which is no better than the regret of a naive algorithm that selects two arms at random in each round. This issue is further exacerbated in Theorem 2, where an additional error term of the form $O(\\sum^{T} \\xi(t))$ appears. Unfortunately, the authors do not provide a theoretical bound for this term; instead, they offer only empirical evidence suggesting that it converges to a small value $T\\epsilon$, which remains linear in $T$. As a result, this limitation substantially weakens the theoretical contribution of the paper.\n- This concern is further amplified by the absence of a lower bound that would justify the observed linear cumulative regret in the presence of multiple biased evaluators. Moreover, the experimental results do not clearly indicate whether the cumulative regret grows sublinearly with the number of rounds. This raises the question of whether the linear regret is fundamentally unavoidable in this setting, or if a more refined analysis would yield improved bounds. Without addressing this, the theoretical contribution remains incomplete.\n- The authors motivate the biased evaluators setting with an example involving a company acting as the agent, multiple LLMs as arms, and users as evaluators. However, the framework assumes the existence of a ground-truth pairwise preference matrix $p_{ij}$, which is not clearly justified in this context. Given that preferences over LLM outputs are inherently subjective and user-dependent, it is unclear why such a universal ground-truth should exist. This raises questions about the practical usability of this model.\n- In lines 143–152, the authors state that their analysis covers both average and weak regret. However, since weak regret is always upper bounded by average regret, it seems the provided bounds effectively only analyze the latter. As a result, the regret bound, while technically applying to both, may be loose for the weak regret. Notably, in the non-biased dueling bandits setting, it is known that cumulative weak regret can be independent of $T$ [1], whereas the bound presented here remains $O(\\sqrt{T})$, even under similar conditions. This observation is further supported by the experimental results, which show that weak regret is significantly lower than average regret.\n- Returning to the regret upper bound in Theorem 2, the inclusion of the parameter $\\xi(t)$ renders the bound somewhat incomplete, as $\\xi(t)$ lacks a closed-form expression or meaningful interpretation in simple terms. Moreover, the authors do not provide a theoretical guarantee that the resulting term $O(\\sum^{T} \\xi(t))$ is bounded, relying instead on empirical evidence presented in Appendix I. \n\nOther concerns – \n- For a given evaluator $m$, the bias parameter $\\eta_{m}$ in Equation (3) is assumed to be uniform across all arm pairs.\n- In Appendix J.1, the authors describe how their techniques can be integrated into existing dueling bandit algorithms to handle biased evaluators and verify the effectiveness of this approach through experiments. However, they do not provide any accompanying theoretical analysis to support these extensions.\n- The regret upper bounds in Theorems 1 and 2 are lengthy, involve numerous parameters, and are difficult to interpret. It would be preferable to move the full expressions to the appendix and present a simplified or more interpretable version in the main text.\n- Additional proofreading would improve the paper. For example, “pairs” should be used instead of “pair” in line 234. The notations $p_{ij}(t)$ and $\\eta_m(t)$, as used in line 240 and elsewhere, should be without $t$. In line 253, “update” should replace “updates.” In line 264, the word “for” should be removed. The definition of HF in line 281 is unclear. Line 287 appears to refer to “Assumption 3,” but likely means “Definition 3.” Finally, the parameter $t_0$ in Equation (21) seems to be used without being defined in the main paper.\n\n[1] Saad, El Mehdi, et al. \"On weak regret analysis for dueling bandits.\" Advances in Neural Information Processing Systems 37 (2024): 40752-40791.\n\nQUESTIONS:\nAs noted earlier, my primary concern and the reason for the current score is related to the regret bounds. I would appreciate clarification on the following points:\n1. Based on your analysis and the observed behavior of cumulative regret over time in the experiments, do you believe that the linear dependence on $T$ in the regret upper bounds of Theorems 1 and 2 is fundamentally unavoidable when multiple evaluators are involved? If so, could you provide some theoretical or empirical evidence to support this? Conversely, if this dependence might be removable, what do you see as the main technical obstacle? Do you have any ideas or directions for how this dependence could potentially be mitigated?\n2. What is the justification for assuming the existence of a ground-truth preference matrix $p_{ij}$? For instance, in the context of LLM selection—where feedback is inherently subjective—what would this ground truth represent?\n3. Could you elaborate further—beyond the brief explanation in lines 71–72—on how you address the challenges involved in deriving the regret bounds?\n4. You mention that your proposed estimators can be readily incorporated into existing algorithms to improve their robustness under biased evaluation. How challenging do you anticipate it would be to derive regret bounds for these modified algorithms? Can the techniques used to prove Theorems 1 and 2 be adapted to this setting, or would each algorithm require a distinct and potentially more complex analysis?\n5. In Lemma 3, you derive the true confidence radius for the unknown bias setting, which involves the parameter $\\eta_m$ and thus cannot be computed exactly by Algorithm 2. However, why not substitute $\\eta_m$ with its estimate $\\hat{\\eta}_m$ in this expression, rather than relying on the less precise approximation $\\hat{r}$ used in Algorithm 2? Could this substitution lead to improved empirical performance or tighter regret bounds?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider the duelling bandit problem with finite arms and biased evaluators.\nIn particular, they assume that there is a finite set of evaluators that are either provide feedback according to the true preferences or the contrary in pairwise comparisons. Each evaluator provides good feedback with certain probability (their bias variable).\nIn the first part, the authors derive a no-regret algorithm for the case when the evaluators' biases are known.\nIn the second part, they extend the case where both the preferences and the biases are unknown.\nFinally, the authors provide experimental evidence that their proposed algorithms outperform algorithms that disregard evaluator biases and their confidence estimation can be incorporated in many other algorithms to improve their performance.\n\nSTRENGTHS AND WEAKNESSES:\nI found the following points to be strengths of this paper:\n1. General preference case not restricted to the Bradley-Terry or Logistic case. This means it is applicable to complex preferences that might not be transitive or follow a reward model.\n2. The results on estimating the pairwise comparison probabilities and biases are general and can be incorporated in many other algorithms\n3. The investigation of biases (especially in human feedback) is more and more crucial given their use in training frontier models and deployed to the general public\n\nWeaknesses:\n1. Considering finite arms and set of evaluators limit the applicability of these results to applications such as training LLMs which the authors use as a motivating example\n2.  I found the motivating example of a company choosing LLMs for deployment to be a bit misleading. In natural language, biases are often contextual (e.g. depends on the question posed). Accordingly some models might be better for certain contexts than others and hard to find a single best option. Additionally, these contexts are infinite (but structured) and the proposed approach can not be easily extended to this case.\n3. In the general preference model setting, the definition of optimality is non-trivial. The authors choose the Borda winner as their notion of optimality but do not discuss its advantages and disadvantages such as neglecting minorities.\n4. While the experiments provide some experimental evidence its is limited to 10 arms and 10 evaluators and a single preference model. Further experiments in larger settings are required to demonstrate the applicability of their results.\n\nQUESTIONS:\n1. Why do the authors choose the Borda Score as their notion of optimality in the general preference setting? I would encourage them to discuss strengths and weaknesses of this design choice. E.g. how does it represent minorities in the annotator pool?\n2. In Line 222-223, the authors claim \"This sublinearity result is consistent with and generalizes those existing works on dueling bandits\nwithout considering evaluators’ bias (e.g., [13])\" could they provide an elaboration on this claim?\n3. The authors use the term \"highly coupled\" multiple times. Could they specify what they mean exactly? I found this fairly ambiguous.\n4. Line 264-265, the authors claim \"We empirically show that performing such updates twice leads to the best performance\" but I could not find these empirical evaluations. Could you point me to them?\n5. Could the authors also report the cumulative regret curves in the Experiments section corresponding to Table 1 and 2 instead of just reporting the values at a specific time-step?\n6. Could the authors elaborate why they choose RC, RUCB, DT, and MBTW as comparison algorithms and omit other works such as Double, MultiSBM [Ailon et al. 2014], MaxInP [Saha, 2021] and MaxMinLCB [Pasztor et al. 2024] that show good performance on the standard duelling bandit problem?\n\nLIMITATIONS:\nPlease see the Weaknesses section. I have listed my concerns there.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors tackle the problem that the binary feedback in a dueling bandit scenario, e.g. given by a human evaluator, might be biased. In their setting, the bias refers to the probability that the expected feedback from a specific evaluator coincides with the ground truth pairwise winning probabilities. In the paper, two settings, with known and unknown bias, are covered. For both, they provide a closed form to estimate the pairwise winning probabilities between the arms (and resp. the bias of the evaluators) and propose UCB-based algorithm to solve the regret minimization problem, while they focus on average and weak regret. In a theoretical analysis, the authors provide regret bounds for both cases, which is in the case of unknown bias far from trivial due to the non-convexity of the problem. In an experimental study, it is shown that their methods suffer less regret in comparison to classical dueling bandit algorithms as baselines and moreover, that these baselines could be extended and improved by their approach for the bias estimation.\n\nSTRENGTHS AND WEAKNESSES:\nThe topic is motivated well with a recently relevant application of the output selection of LLMs. Although I did not read the proofs in the appendix in full detail, the biggest strength of the paper is that the proposed approach is theoretically grounded and provided with a regret analysis for weak as well as for average regret that generalizes over the known regret bounds for the case of no existing bias. The whole paper is well written and easily understandable. However, a weakness is the experimental study, which covers to the best of my understanding only synthetic data and only one instantiation of the distribution from which the biases are sampled. Here, a more detailed sensitivity analysis of how the methods behave, for example, for no bias at all or for different beta distributions from which the biases are sampled, would be insightful.\n\nQUESTIONS:\nQuestions:\n- I am wondering how the weak and average regret can have the same regret bounds. As the name suggests, the weak regret is usually lower than the average regret (assuming that not two times the same arm is chosen during a time step), so intuitively I would expect a tighter regret bound for the weak regret. Can you discuss how tight the bounds are? What is about the strong regret?\n\n- Usually, in dueling bandit methods like in Relative UCB, only the first arm is chosen as the one that maximizes the upper confidence bound, and the second arm is chosen as the one that is the \"best competitor\" for the first chosen arm where the winning probability of the first arm is minimized. Formally, the first chosen arm would be $x_1(t) = max_i UCB_i(t)$ and the second chosen one $x_2(t) = min_j UCB_{i,j}(t)$. Have you tried out this approach or do you have a reason why you don't do it like this?\n\n- How does your method with unknown bias behave if no bias is present? Does it correctly recognize that there is no bias? Can it keep up with the baselines like RUCB or double thompson sampling? Does the difference between your methods and the baselines become bigger with larger bias? For this, some experiments with different beta distributions to sample the bias would be nice\n\nLIMITATIONS:\nThe authors mentioned some limitations, like the assumption that the bias is fixed over the whole time horizon and the lack of experiments on real-world human data. Other potential limitations (maybe a slower convergence of the estimated pairwise probabilities if no bias is present?) are not made clear and cannot be derived by the reader from the shown experiments.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-18"}