Towards Domain Adaptive Neural Contextual Bandits

Authors: Ziyan Wang, Xiaoming Huo, Hao Wang
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: Domain Adaptation, Deep Learning, Adversarial Learning

Abstract:
Contextual bandit algorithms are essential for solving real-world decision making problems. In practice, collecting a contextual bandit's feedback from different domains may involve different costs. For example, measuring drug reaction from mice (as a source domain) and humans (as a target domain). Unfortunately, adapting a contextual bandit algorithm from a source domain to a target domain with distribution shift still remains a major challenge and largely unexplored. In this paper, we introduce the first general domain adaptation method for contextual bandits. Our approach learns a bandit model for the target domain by collecting feedback from the source domain. Our theoretical analysis shows that our algorithm maintains a sub-linear regret bound even adapting across domains. Empirical results show that our approach outperforms the state-of-the-art contextual bandit algorithms on real-world datasets. Code will soon be available at https://github.com/Wang-ML-Lab/DABand.

Decision: accept (poster)
Reviewer scores: 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730424524391

thread_type:
official_review

<thread_object>
creator: reviewer_mn7w

<content>
summary:
This paper studies the domain transfer in (neural) contextual bandits. The authors provide a decomposition of the domain transfer in contextual bandits, and proposed an algorithm leveraging the adversarial training in domain-transfer. Theoretical results are presented understanding this decomposition and empirical evaluation are performed to validate the performance of the algorithm

strengths:
- The authors provide a theoretical justification on the decomposition for the domain-transfer issue for contextual bandits. 
- The authors provide empirical evaluation demonstrating the performance of the algorithm.

weaknesses:
- (Minor) Definition 3.6 is hard to understand for me. In particular, are $\mathbf x$ and $x$ referring to the same thing? Are the authors implicitly assumes $h \in \mathcal H$ and $x \in \mathbb R^n$ in this case? It would be helpful if the authors could rephrase the mathematical formulation on this. 
- In Theorem 3.1 needs further justification. For example, I wonder why the sublinear in $R_S$ will lead to the sublinear result in $R_T$. I checked the proof for Theorem 3.1 in appendix but found nowhere explicitly discuss this part. In addition, why the data-divergence term is sublinear? 
- The adversarial training paradigm adapted in controlling the data divergence term in the proposed algorithm requires better justification: why the hypothesis class here is the same with the hypothesis class used in contextual bandits?
- It lacks some discussions on the performance analysis in linear contextual bandits or discussions in related literautre. The authors might want to provide more discussion on the theoretical insight of this. 

- (Minor) Section 4.1 title: eq (16) -> eq (5)

questions:
- Besides the presentation in Theorem 3.1, are there any regret scale for linear contextual bandits to help me better understand the performance of the algorithm?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730714423143

thread_type:
official_review

<thread_object>
creator: reviewer_mpvh

<content>
summary:
The paper introduces an algorithm for domain adaptation in contextual bandits. This method aims to adapt a neural contextual bandit algorithm from a low-cost source domain to a high-cost target domain, to address the challenge of distribution shifts between the two domains. The theoretical analysis conducted demonstrates that DABand achieves a sub-linear regret bound in the target domain, which is interesting. Empirical results are presented using real-world datasets, which show that DABand outperforms existing contextual bandit methods. The proposed work has broader implications, including potential insights into reinforcement learning with human feedback algorithms.

strengths:
(1) The proposed problem setting of contextual bandits in a domain adaptation scenario is interesting and challenging.

(2) The method utilizes unlabeled data from both source and target domains for effective representation learning and alignment across different domains.

(3) The algorithm is capable of attaining a sub-linear regret bound in the target domain by solving an online network lasso problem with time-dependent regularization.

weaknesses:
The method leverages unlabeled data from both source and target domains to learn robust representations and aligns them effectively across different domains, enabling efficient domain adaptation. While the proposed algorithm builds upon the NeuralLinUCB framework, it introduces an adaptation in the loss function specifically tailored for updating the neural network. This loss function integrates insights from classic domain adaptation techniques, and thereby has some similarity with existing methods.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731477891222

thread_type:
official_review

<thread_object>
creator: reviewer_2moo

<content>
summary:
The manuscript introduces a domain adaption algorithm for contextual bandit. It further proves that the proposed method achieves a sub-linear regret bound. Empirical experiments conducted on real-world data shows the the proposed method outperforms existing state-of-the-art contextual bandit methods for cross domains.

strengths:
1. The design of the new algorithm originates from an observation that leveraging data across domains leads to sub-linear regrets, which makes the whole method simple, yet elegant.

2. Theoretical proof has been provided to support the performance of the method.

3. The algorithm is extensively tested on three datasets.

weaknesses:
It would be helpful to elaborate on why the data divergence term is sub-linear in the proof.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732134403184

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Global Response

comment:
We thank all reviewers for their valuable comments. 

We are glad that they found our problem setting ``"interesting and challenging"`` (MPVh), our method ``"simple, yet elegant"``/``"outperforms existing contextual bandit methods"``/``"has broader implications"`` (2Moo, MPVh), and our algorithm ``"extensively tested"``/``"provide empirical evaluation"`` (2Moo, Mn7w). They also acknowledged that ``"theoretical justification"``/``"theoretical analysis"`` ``"has been provided"``, is ``"interesting"``, and ``"support the performance of the method"`` (2Moo, MPVh, Mn7w). 

Below we address the reviewers' questions one by one in detail. We have cited all related references and **included all discussions/results below in our revision** (with the changed part marked in blue).
</content>
</thread_object>

</thread>


Metareview:
metareview:
The reviewers generally appreciate the novel algorithm for domain adaptation in bandits. I only have one more thing to add. Domain adaptation ultimately is a form of distributional shift. Another approach to deal with is through distributional robustness (see Distributionally Robust Batch Contextual Bandits, Si et al). The authors should discuss the merits and drawbacks of the current work with respect to this approach. But overall a good paper.