Title: Learning Across the Gap: Hybrid Multi-armed Bandits with Heterogeneous Offline and Online Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: online learning

multi-armed bandit

dueling bandit

hybrid feedback
Abstract: The multi-armed bandit (MAB) is a fundamental online decision-making framework that has been extensively studied over the past two decades. To mitigate the high cost and slow convergence of purely online learning, modern MAB approaches have explored _hybrid_ paradigms that leverage offline data to warm-start online learning. However, existing approaches face a significant limitation by assuming that the offline and online data are homogeneous—they share the same feedback structure and are drawn from the same underlying distribution. This assumption is often violated in practice, where offline data often originate from diverse sources and evolving environments, resulting in feedback heterogeneity and distributional shifts. In this work, we tackle the challenge of learning across this offline-online gap by developing a general hybrid bandit framework that incorporates heterogeneous offline data to improve online performance. We study two hybrid settings: (1) using reward-based offline data to accelerate online learning in preference-based bandits (i.e., dueling bandits), and (2) using preference-based offline data to improve online standard MAB algorithms. For both settings, we design novel algorithms and derive tight regret bounds that match or improve upon existing benchmarks despite heterogeneity. Empirical evaluations on both synthetic and real-world datasets show that our proposed methods significantly outperform baseline algorithms.


Peer reviews:
comments: SUMMARY:
This paper studies hybrid stochastic MAB. That is, warm starting the online interaction stage with offline dataset. this paper considers that the offline and online feedback are of different types, one is absolute signal and the other is pairwise comparison. The main challenge is how to fuse these two types of feedback

In both settings, there are algorithms proposed that leverege offline dataset whenever it is helpful, and does not suffer from it otherwise. Lower bounds are also provided. Lastly, experiments validate the effectiveness of the proposed algorithms.

STRENGTHS AND WEAKNESSES:
Strengths

1. It's a well-motivated problem due to RLHF
2. Algorithm design is not obvious or trivial
3. Remarks which interpret the results are very informative and sharp
4. Lower bound results complement upper bound results.

Weaknesses

1. The performance depends on hyper-parameter tuning, which further relies on some unknown knowledge.

QUESTIONS:
1. Does the offline data never hurt the online regret, even if they are not helpful at all?
2. "closely match" in line 279 is not precise enough. I expect more discussion on the gap between upper and lower bounds
3. Now two settings have different types of alg. design (UCB vs. elimination). What's the challenge to unify them?

typos

1. $\\mathcal{C}_t$ in Eq. (5) should be $\\mathcal{C}^{\\mathrm{on}}_t$?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the hybrid multi-armed bandit setting, where the learner has access to an offline dataset before interacting with an online learning environment. Interestingly, this setting considers the heterogeneous across the offline versus online: one is the preference relative data, the other is the typical reward data.\n\nEven though Agnihotri et al., 2024 have studied a similar setting previously, in this paper, the authors also analyze the distribution shift problem, where the offline dataset may not be informative to the online phase. I think that the warm-start setting with heterogeneous data and distribution shift is a very interesting and practical problem.\n\nThe authors provide the instance-dependent and instance-independent bounds for both the lower bound and the upper bound of their proposed algorithms, for both of their settings. They also provided an experiment section to demonstrate the effectiveness of their algorithms.\n\nSTRENGTHS AND WEAKNESSES:\nQuality: The submission is technically sound. The theorem's results make sense to me, although I haven't checked all the proofs.\nThe idea of using pessimism to address mis-specification (or distribution shift in this case) is valid and well-known on a high level, but its application to construct the confidence set in this paper is novel.\n\nClarity: The submission is clearly written and organized. Still, it would help a lot if the authors gave a high-level and intuitive explanation for their algorithms and analysis. For example, I still don't have a good high-level understanding of the exploration term in equations 4 and 10, as well as the significance of the terms in the upper and lower bounds.\n\nAnother suggestion on clarity is on the Remarks. I think they are all important discussions, but they may be written in a better way. For example, Theorem 1b bounds include the term Saving(a_i, a_j). I understand that keeping it like this helps generalize different amounts of saving, which is problem-dependent, but it makes the reader have a harder time evaluating whether the saving can make a difference to the bound or not. Instead of making a remark like the one in the paper, maybe you can just add a corollary that, assuming x and y, the upper bound is z.\n\nSignificance and Originality: This paper analyzes a practical setting, with novel analysis and algorithms. I think there's a significant contribution from this compared to the previous work. I think the related work section shows good and relevant previous and related work.\n\nNotes:\n- I'm not sure if the instance-dependent upper-bound and lower-bound in Theorems 1 and 2 \"closely match\" or are tight.\n- I can't comment on the correctness of the proof of the significance (impact) of the bounds (about how much this paper improves on the previous results)\n\nQUESTIONS:\nAlready discussed above.\n\nLIMITATIONS:\nThe author should discuss the limitations clearly in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies how users can utilize two different types of heterogeneous data that arise within a single game. In particular, motivated by the recent emergence of RLHF, it investigates how two commonly encountered observation types—preference-based observations and conventional direct bandit rewards—can coexist and complement each other. The paper further explores how such offline data can be mutually beneficial. Moreover, it presents the results in a highly general setting that allows for heterogeneity not only in observation types but also in the underlying data distributions.\n\nSTRENGTHS AND WEAKNESSES:\nThis is a personal opinion as a reviewer, but I found this paper to propose an interesting approach to fusing two types of data that, while intuitively complementary, are not straightforward to combine. The significance and novelty of the topic addressed in this work are, at least from my personal perspective, quite clear.\n\nMoreover, the paper is formulated in a very general framework that can accommodate heterogeneous cases that are likely to arise in practice. By employing a \"minimum UCB\" approach, the authors propose a mechanism that naturally discards offline data when its quality is not sufficiently helpful for online learning. I found this to be a particularly interesting idea.\n\nRegarding **soundness**, I unfortunately was not able to carefully verify all the proofs. However, assuming the results are correct, the fact that the authors provide a matching lower bound (Theorem 2) to their upper bound strongly supports the optimality of their results. The logical flow and the implications drawn by the authors seem reasonable to me, and while my confidence is not high due to limited verification, I intuitively believe there is no major issue with the soundness. I plan to examine the validity of the proofs more closely during the rebuttal period.\n\n---\n\n**Weaknesses**:\n\nA few points remain unclear:\n\n1. The authors appear to have made efforts to clearly delineate their contributions. However, the results themselves depend on many variables and are presented in quite a complex form (e.g., involving a prior upper bound referred to as $V$, or a quantity represented as $\\tau^*$, which arises from some complicated optimization). While it is clear that the \"saving\" offers some degree of benefit, it is difficult to assess how impactful or innovative this actually is. It would be helpful if the authors could provide a carefully chosen, perhaps even \"cherry-picked,\" example that most dramatically illustrates the benefit—preferably in an environment where offline data can dramatically help, such as a homogeneous setting where greedy algorithms suffice online, and then quantify how many offline samples are needed to guarantee such behavior.\n\n2. Remark 4 is difficult to understand. Is the author claiming that there are situations in which preference-based feedback is more useful than direct reward observations? While I appreciate the authors' honesty in noting this, I think the explanation—that it is due to subgaussianity and the BT model—could be expanded. A mathematical example or deeper explanation of why preference feedback is more helpful in such cases would be much appreciated.\n\nQUESTIONS:\nPlease check the Strength & Weakness section above.\n\nLIMITATIONS:\nNo.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper is the first work to analyze hybrid MAB settings with heterogeneous offline-online feedback, addressing two scenarios:
- Offline preference data $\to$ online absolute rewards
- Offline absolute data $\to$ online dueling bandits

This paper proposes two algorithms HybUCB-AR and HybElimUCB-RA for Offline absolute data + online dueling bandits and Offline preference data + online stochastic bandits, respectively.

Empirical results show that both algorithms outperform state-of-the-art online methods when using heterogeneous offline data

STRENGTHS AND WEAKNESSES:
Strengths: 
- Providing both theoretical foundations and practical performance improvements while maintaining safety guarantees.
- Although I didn't go through all the proof details, the theoretical part looks correct and well-organized.

Weaknesses:
- The results seem to be a heterogeneous version of [1]. While in practice considering heterogeneous data type is useful, the novelty in theoretical aspect is not enough.
- The algorithm follows the standard UCB-based exploration technique and incorporate the information from offline dataset by taking the minimum of two UCB rewards, which is not surprising. And the algorithms require knowledge of the bias for each data, which is not practical enough.

[1] Wang Chi Cheung, Lixing Lyu. Leveraging (Biased) Information: Multi-armed Bandits with Offline Data.

QUESTIONS:
See weaknesses.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-03