Towards Domain Adaptive Neural Contextual Bandits
Authors: Ziyan Wang
Xiaoming Huo
Hao Wang
Venue: ICLR 2025 Poster (2025)
Keywords: Domain Adaptation
Deep Learning
Adversarial Learning
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We introduce the first general domain adaptation method for contextual bandits.

Abstract: Contextual bandit algorithms are essential for solving real-world decision making problems. In practice, collecting a contextual bandit's feedback from different domains may involve different costs. For example, measuring drug reaction from mice (as a source domain) and humans (as a target domain). Unfortunately, adapting a contextual bandit algorithm from a source domain to a target domain with distribution shift still remains a major challenge and largely unexplored. In this paper, we introduce the first general domain adaptation method for contextual bandits. Our approach learns a bandit model for the target domain by collecting feedback from the source domain. Our theoretical analysis shows that our algorithm maintains a sub-linear regret bound even adapting across domains. Empirical results show that our approach outperforms the state-of-the-art contextual bandit algorithms on real-world datasets. Code will soon be available at https://github.com/Wang-ML-Lab/DABand.

Published as a conference paper at ICLR 2025
TOWARDS DOMAIN ADAPTIVE NEURAL CONTEXTUAL BANDITS
Ziyan Wang', , Xiaoming Huo', Hao Wang²
Georgia Institute of Technology' Rutgers University? (wzy, huc) ) @egttchh. edu hw4880cs. rut.grs..ed. edu?
ABSTRACT
Contextual bandit algorithms are essential for solving real-world decision making problems. In practice, collecting contextual bandit's feedback from different domains may involve different costs. For example, measuring drug reaction from mice (as a source domain) and humans (as a target domain). Unfortunately, adapting contextual bandit algorithm from source domain to target domain with distribution shift still remains a major challenge and largely unexplored. In this paper, we introduce the first general domain adaptation method for contextual bandits. Our approach learns a bandit model for the target domain by collecting feedback from the source domain. Our theoretical analysis shows that our algorithm maintains a sub-linear regret bound even adapting across domains. Empirical results show that our approach outperforms the state-of-the-ar contextual bandit algorithms on real- real-world datasets. Code will soon be available at https: g rithub... com/ Wang-ML MML-g-ML-LLbbbb HLL-Lab/DABBnd.
INTRODUCTION
Contextual bandit (CB) algorithms have shown great promise for naturally handling explo- ration/exploitation trade-off problems with optimal statistical properties. Notably, LinUCB (Li al., 2010) and its various adaptations (Yue Guestrin, 2011; Agarwal et al., 2014; Li al., 2016; Kveton et al., 2017; Foster et al., 2018; Korda et al., 2016; Mahadik et al., 2020; Zhou et al., 2019), have been shown to be able learn the optimal strategy when all data come from the same domain. However, these methods fall short when applied to data from reaction prediction model trained by collecting feedback from mice (the source domain) may not work for humans (the target domain).
domain. For example, drug
So, how does one effectively explore a high-cost target domain by only collecting feedback from a low-cost source domain, e.g., exploring drug reaction in humans by collecting feedback from mice or exploring real-world environments by collecting feedback from simulated environments? The challenge of effective cross-domain exploration is multifaceted: (1) The need for effective exploration in both the source and target domains depends on the quality of representations learned in the source domain, which in turn requires effective exploration; this leads chicken-anc warkeen--nd-eege and-egg dilemma. (2) Aligning source-domain representations with target-domain representations is nontrivial in bandit settings, where ground-trui -ttnhhhhhhhhh may still be unknown if the action incorrect. (3) Balancing these aspects effective exploration and accurate alignment in bandit settings is also nontrivial.
To address these challenges, as the first step, we allow our method to simultaneously perform effective exploration and representation aligument, leveraging unlabeled data from both the source and target domains. Interestingly, our theoretical analysis reveals that naively doing so leads sub-optimal accuracy/regret (verified by empirical results) and naturally leads to additional terms in the target-domain regret bound. We then follow the regret bound derived from our analysis to develop an algorithm that adaptively collect feedback from the source domain while aligning representations from the source and target domains. Our contributions are outlined as follows:
We identify the problem of contextual bandits across domains and propose contextual bandits (DABand) as the first general method to explore high-cost target domain while only collecting feedback from low-cost source domain.

---

Published as a conference paper at ICLR 2025
Our theoretical analysis shows that method can achieve sub-linear regret bound in the target domain.
Our empirical results on real-world datasets show DABand significantly improve perfor-
mance over the state-of-thh-ard contextual bandit methods when adapting across domains.
2 RELATED WORK
Contextual Bandits. In the realm of adaptive decision decision-making, contextual bandit algorithms, epitomized by LinUCB (Li et al., 2010), have carved a niche in efficiently balancing the exploitation- exploration paradigm leveraging estimation reward and uncertainty (Wang et al., 2022; 2016); they have been influential across spectrum of use cases, notably in complex adaptive systems such as recommender systems (Li et al., 2010; Wang et al., 2022). These algorithmic frameworks, along with their myriad adaptations (Yue & Guestrin, 2011; Agarwal et al., 2014; Li et al., 2016; Korda et al., 2016; Kveton et al., 2017; Foster et al., 2018; Zhou et al., 2019; Mahadik et al., 2020; Xu et al., 2020), have decisively outperformed conventional bandit models that operate devoid of contextual awareness (Auer, 2002). This superiority is underpinned by theoretical models, paralleling the insights in (Auer, 2002), where LinUCB variants are validated to conform to optimal regret boundaries in targeted scenarios (Chu et al., 2011). However, a discernible gap in these methodologies is their reduced efficacy in a new domain, particularly when getting feedback involves high cost. Our proposed DABand bridges this gap by adeptly aligning both source-domain and target-domain representations in the latent space, thereby reducing performance drop when transfer across different domains.
Domain Adaptation. The landscape of domain adaptation has been extensively explored, as evi- denced by breadth of research (Pan & Yang, 2009; Pan al., 2010; Long al., 2018: 88; Saito et al., 2018; Sankaranarayanan et al., 2018; Zhang et al., 2019; Peng et al., 2019; Chen et al., 2019; Dai al., 2019; Wang al., 2020; Nguyen-Meidine et al., 2021; Xu al., 2023). The primary objective of these studies has been the aligmment of source and target domain distributions to facilitate the effective generalization of models trained on labeled source data unlabeled target data. This alignment is conventionally attained either through the direct matching of distributional statisties (Pan et al., 2010; Tzeng et al., 2014; Sun Saenko, 2016 6; Peng et al., 2019; Nguyen- Meiedne et al., 2021) or via the integration of an adversarial loss (Ganin et al., 201 6b; Zhao et al., 2017; Tzeng et al., 2017; Zhang et al., 2019; Kuroki al., 2019; Chen al., 2019; Dai al., 2019; Wang et al., 2020; Xu et al., 2022; Liu et al., 2023; Shi & Wang, 2023). The latter, known adversarial domain adaptation, has surged in prominence, bolstered by its theoretical foundation (Goodfellow et al., 2014; Zhao et al., 2018; Zhang al., 2019; Zhao al., 2019), the adaptability of end-to-end training in neural network architectures, and its empirical efficacy. However, these methods only work in offline settings, and assumes complete observability of labels in the source domain. Therefore they are not applicable to our online bandit settings.
Distribution Shift in Bandits. A related study addresses the challenge of policy learning in contextual bandits using historical observational data, particularly when there is shift in the environment (Si al., 2023). The authors propose a distributionally robust approach to policy evaluation and learning, ensuring that the learned policy remains effective even under worst-case environmental shifts. However, their focus distinct from ours. Their goal is to ensure policy robustness against environmental shifts between training and deployment within the same domain, whereas our proposed DABand focuses on adapting policies from a source domain to a different target domain, particularly when direct feedback from the target domain is limited or costly.
Domain Adaptation Related to Bandits. There is also work related to both domain adaptation and bandits. Specifically, Guo et al. (2020) propose domain adaptation method using a bandit algorithm to select which domain to use during training.
We note that their goal and setting is different from our DABand's. (1) Guo et al. (2020) focuses on improving accuracy in a typical, offline domain adaptation setting. In contrast, our DABand focuses on minimizing regret in online bandit setting. (2) Guo et al. (2020) assumes complete access to ground-truth labels in the source domain, while in our bandit setting, ground-trutt may still be setting.
unknown if the action is incorrect. Therefore their work is not applicable
2

---

Published as a conference paper at ICLR 2025
3 THEORY
In this section, we formalize the problem of contextual bandits across domains and derive the corresponding regret bound. We then develop our DABand inspired by this bound in Sec. All proofs of lemmas, theorems can be found in the Appendix.
for k N+ The Euclidean norm of a vector X E Rd is denoted as ||x|| | = vx'xx. We denote the operator norm and Frobenius norm of a matrix W E R || and ||W| [|,, respectively. Given semi-definite matrix A E Rdxd and a vector Rd. we denote the Mahalanobis norm as ||x||A vxx Axx. For function f of a parameter T. we denote as (T)( the terms growing in the order of (T),, ignoring constant factors. We assume A2 = = |Avl = K. We use (,, > to denote the inner product of two vectors. We denote as ap, the action in domain D,
Notation. We use [k] denote the set (1.2.-... k).
all the action spaces are identical and with cardinality of K, i.e., |Ail
and omit the subscript D when the context is clear, e.g.,
2D 2,,; = x 1.
3.1 PRELIMINARIES
Typical Contextual Bandit Setting, Suppose we have N samples from an unknown domain D, which is represented {x? a, [e(])] Here, Xi.a E 'D denotes the context for action A, and of E A, is the ground-truth action that will receive the optimal reward. We denote as fi,e the received reward in round after performing action a and â; E A; the chose action by bandit algorithm in round i. The goal in typical contextual bandits is to learn a policy of choosing actions ā in each round to minimize the regret after N rounds:
Σ 1=1 T.a.
R = i=1 Tr.a
LinUCB, LinUCB (Li et al., 2010) a classic method for the typical contextual bandit setting. It assumes that the reward is linear to its imput context, i.e., X;,a- During each round i, the agent selects an arm according to historical contexts and their corresponding rewards, Tí,a.
i = argmax {xTna 0, + [xx.allA21]
aE.A;
where - X X a,a with 7 0 and x the historical selected action's 's context, Xi,a the context for the candidate action a in round i, 0, is the bandit parameter in round and >> 0 is a hyperparameter that adjusts the exploration rate.
LinUCB with Representation Learning. In this paper, we use Neural LinUCB (Xu et al., 2020; Wang et al., 2022) as backbone model to handle high-dimensional context with representation learning. We assume that there exist a ground-truth encoder 6² to encode the raw context X,,a into latent-space representation (encoding) o (x,,a). Subsequently, the reward for the context x;,a is then r(x,,a) (0", o" (x) plus some stochastic noise e. Furthermore, we use the same setting as Xu et al. (2020) where the ground-truth rewards are restricted to the range of [0. 1]. Additionally, we further bound predicted rewards by normalizing ||| 1 and [ 1.
In the single-domain typical setting, the goal is to learn an encoder T and the contextual bandit parameter 0, such that our estimated reward f(x;,a) (Ō, ((x,,)) can be close to the ground-truth reward r(xi,a) One can then use this form derive policies minimize the regret in Eqn. (1).
For simplicity, in Definition 3.1 below, we further denote as fo the (ground-truth) labeling function for domain D and h H a hypothesis such that is parameterized by db. 0b. and h is parameterized by G, 0.
3.2 OUR PROSS-DOMANE CONTEXTUAL BANDIT SETTING
Typical contextual bandits operate in a single domain D. In contrast, our cross-domain bandit setting involves a source domain S and a target domain T. In this setting, feedback (reward) from the source domain, but not from the target domain. Specifically: (1) We assume low-cost source domain (experiments mice), where for each round i, has access
only collect
3

---

Published as a conference paper at ICLR 2025
to the contexts {x [e((K] for each candidate actions, chooses one action a, and receives reward S. ".a." (2) Additionally, we assume a high-cost targer domain where collecting feedback (reward) is expensive (experiments on humans); therefore, one only has access to the contexts fx ke((k] for each candidate actions for each round, but cannot collect feedback (reward) r.,a, for any action in the target domain. The goal in our cross-domain contextual bandits is to learn policy of choosing actions 0 in the target domain to minimize the target-domain zero-shot regret for (hypothetical) future rounds:
Ry = -.
T 2,0 -
Difference between Eqn. (1) and Eqn. (2). In the typical setting, Eqn. (1) uses different updated policies for each round; this is also true for the source domain. In contrast, the target regret in Eqn. (2) use the same fixed policy obtained from the source domain for N (hypothetical) rounds because one cannot collect feedback from the target domain. See the difference between Definition 3,4 and Definition 3.5 below for a more formal comparison.
3.3 FORMAL DEFINITIONS OF ERROR
Definition 3.1 (Labeling Function and Hypothesis). We define the labeling function fD and the hypothesis h for domain 'D as follows:
(x(x r(xx,x) = (0:: (x(xa)) +
M(xP,)) =(xx,,) = (0,x(x),a)) +
where 0b is the optimal predictor for domain D. 0 denotes the estimated predictor, and e is the random noise.
Next, analyze how well our hypothesis h estimates the labeling function f (i.e., ground-truth hypothesis). Definition 3.2 below quantifies the closeness between two hypotheses by calculating the absolute difference in their estimated rewards.
Definition 3.2 (Estimated Error between Two Hypotheses). Assuming all contexts xLa, are from domain D, the error between two hypotheses h1, h2 E H on domain given selected actions (ā: E [K] is eD(1i, 1b)==>> (1i(=22a)-oox(aa)
= (10..0112aa))) - (22, S2 (xPaa)))
(3)
Note that the domain D above can be the source domain S or the target domain T. We then define the error of our estimated hypothesis in these domains.
Definition 3.3 (Source- and Carget-Domaile Error). With N samples from the source domain S, and is denoting the labeling function for S, the source-domate error then
es(Js. h1) h) > (1s(((s))))))))
Mixe(a))) ((stal))))M((aaa) = -.",, (105(100666GE -0.00x(a)))
where x., comes from Furthermore, 0N = arg max, ((x(.) and at = arg maxo (x( For simplicity, we shorten the notation (s(Js,)) to (s(h) and similarly use (h) for domain T. Assuming x!, comes from T, the estimated error for the target domain is then:
n)) - (IF((TTa) (15((xla)-h(()))))) h(sxTai)) = (107.00((Taa) @0, (0.0xaaa)))
3.4 SOURCE REGRET AND TARGET REGRET
Below define the regret for the source and target domains. Definition 3.4 (Source Regret). Assuming xx_ comes from domain S, the source regret (i.e., the regret in the source domain) is
Rs Σ/1 (Is(x(nt)) fs(xfaa))
-'"
(03, 03(x (3(x(x)))))
(4)
4

---

Published as a conference paper at ICLR 2025
The goal in cross- contextual bandits is learn a policy of choosing actions ã in the high-cost target domain (e.g., human experiments) by collecting feedback only in the source domain (e.g,, mouse experiments). Formally, would like to minimize the target regret defined below. Definition 3.5 (Target Regret and Problem Formulation). Denoting the estimated hypothesis as h Ō 0 the target regret we aim to minimize defined as
RT RT= = ===, (F(xTTa;) - fr fr(x(a))
argmaxh(x].. + all ((l0(xxT,) 11
h argmin csh),
h
where A - I i,as (x,x) denotes the context matrix accumulated by the selected context features in the source domain.
(x
3.5 CROSS-DOMAIN ERROR BOUND FOR REGRESSION
Prior introducing final regret bound, it is necessary define additional component. A funda- mental challenge in general domain adaptation problems is to manage the divergence between source and target domains. Unfortunately, previous domain adaptation theory only covers classification tasks (Ben-David et al., 2010; Zhang et al., 2019). In contrast, the problem of contextual bandits is essentially reward regression problem. To address this challenge, we introduce the following new definition to bound the eтror between the source and target domains.
Definition 3.6 (H&H Hypothesis Space for Regression). For hypothesis space H. the symmetric difference hypothesis space H HAH is the set of hypotheses s.t.
9 HAH
g(x) [1(x) h'(x)).
We then can further define the Divergence for H&H Hypothesis Space:
dn T) 2 sup M.EE
[x~sslh(x)) h'(x)]l - EZwT(lh(x)) h'(x)lll
Definition 3.7 (Optimal Hypothesis). The optimal hypothesis, denoted h. balance the estimated error for both source and target domain. Formally,
h = arg min hEH
(s(h) + ch(),,
and we define the minimum estimated error for the optimal hypothesis hi" as
- = Es(")) + (T(").
3.6 FINAL REGRET BOUND
With all the definitions of source/target regret in Definition 3.4 and Definition 3.5, we can then bound the target regret using Theorem 3.1 below.
Theorem 3.1 (Target Regret Bound). Denoting the contexts from the source domain S as N (xf,a (E()) the associated ground-truth action as {aT)N Tiz1, and the contexts from the tar- get domain T as (x([ahetMM)aEI)) the upper bound for our target regret Ry is
(07, (07,07 (x )) 107, -00.707 (xTxx))
Rs Source Regret
2. (h)
+ N (NANRSS (S,
++C
Regression Error
Data Divergence
Constont
N
(10,
Σ
1 * ((0,0x((a))))
(5)
Predicted Rewards
where Rs. (h( and dnan the source regret, source-domain error, and H.H divergence defined Definition 3.4, Definition 3.3, and Definition 3.6, respectively. 1 is a constant independent the problem and and C constant which can be ignored (see the Appendix for more details).
5

---

Published as a conference paper at ICLR 2025
Here provide several observations (for more analysis, please refer to the Appendix):
(1) Since Rs is sub-linear w.r.t. the number of samples in the source domain, so is RT.
(2) Theorem 3.1 bounds the target regret using the source regret, enabling the exploration of the target domain by only collecting feedback (reward) from the source domain.
(3) Typical generalization bounds in DA naively minimize the source regret Rş while aligning source and target data in the latent space (i.e., minimizing dHAH). This is not sufficient. as we need two additional terms, i.c., the regression error (s) and the predicted reward
Σ (1,0 (0,0(x(a + Σ 1a o =āi] (00, (00,,(xxf.. (xx.,a))) as regularization terms.
(4) Minimizing the regression error (s()) encourages accurate prediction of source rewards.
(5) Minimizing the predicted reward Σ
)1) E 1a * @el (1
regularizes the model to avoid overestimating rewards.
Note that Theorem 3.1 nontrivial. While it does resemble the generalization bound domain adaptation, there are key differences. As mentioned in Observation (3) above, our target regret bound includes two additional crucial terms not found in domain adaptation. Specifically:
Σ (103.3((5(a)) ((. (( (i...
Regression Error in the Source Domain.
es(h) in Theorem 3.1) defines the difference between the true reward from selecting action ā and the estimated reward for this action.
Predicted Reward. ΣN (0, oxx ) + I[a; # a.:] (100, serves as
regularization term regularize the model to avoid overestimating rewards.
The results of the ablation study Table in Sec. 5 highlight the significance of these two terms. See Appendix G.4 for more discussion novelty well key differences between our DABand and classic domain adaptation.
4 METHOD
With Theorem 3.1, can then design our DABand algorithm obtain optimal target regret. We discuss how to translate each term in Eqn. (5) to a differential loss term in Sec. 4.1 and then put them together in Sec. 4.2.
4.1 FROM THEORY TO PRACTICE: TRANSLATING THE BOUND IN EQN. (5) TO DIFFERENTIABLE LOss TERMS
Source Regret. Inspired by Neural-LinUCB (Xu et al., 2020), we use LinUCB (Li et al., 2010) to update 9 and the following empirical loss function when updating the encoder parameters by back-propagation in round
C. = Σ a=1
K (0 (00(()))) - r rxx
(6)
where 0 is the contextual bandit parameter, and is the encoder shared by the source and target domains, i.e., we set Øs = &. during training.
Insights. From Eqn. (6), a fundamental difference between classification problems and bandit problems becomes apparent. In classification tasks, both the ground-truth label and the estimated label are accessible. However, in bandit problems, we only know the estimated label and reward. If the estimated rewards align with our estimated label's correctness, it suggests knowledge of the ground-truth label for that feature. If not, we can only deduce that the estimated label is inaccurate, without identifying the correct label among the other candidates. This uncertainty significantly amplifies the complexity of contextual bandit problems.
Regression Error. We use L1 loss optimize the regression error, defined the difference between the true reward from selecting action à; and the estimated reward for this action (see Definition 3.3 for detailed explanation of this regression error). Minimizing the regression error term directly

---

Published as a conference paper at ICLR 2025
Algorithm DABand Training Algorithm
Input: regularization parameter 7 > 0, number of total steps N, episode length H, exploration parameters a.
2: Output: parameters of the model: PN. 0N, AN. by-
3: Initialization: Ag yI, bo 0, and for 8g and do initialized following (Xu et al., 2020).
4: for 1,... N do
5: Obtain { (x(xa [E[K] and { aE[K] from the source and the target domain, respectively.
6: Choose an action ã argmax (x 0,_1 | 123
7: Get the reward r, r(x,,,, based on the selected action āz Update bandit parameters:
8:
A A
Ø (x
9:
b, b, trs
2 (x
0: A,'b;
if mod (i. H= then for j 1,.... H do
11:
12:
Calculate AABddd Eqn. (11).
13:
Use the Adam optimizer (Diederik, 2014) to update encoder 6; and discriminator by back-propagation to solve the minimax optimization in Eqn. (10).
14:
end for
else
16:
Ø = -
end if
18:
end for
Output: SN, ŌN AN,
19:
encourages accurate prediction of source rewards using L, loss. Specifically, we
S (h)
(0's rh' (x (0, ( (x)
where (0', 03 (x(,,)) regression term above directly encourages accurate prediction of source rewards.
)) is the reward received by the agent in the source domain. Minimizing the
Data Divergence. The data divergence term in Eqn. (5) leads to following loss term:
div
max N 5 (CD(g(((x)) 0)] + E [Co(o(o(x),, 1)]),
(8)
where ES and ET denote expectations over the data distributions of (x, a) in the source and target domains, respectively. is a discriminator that classifies whether x is from the source domain or the target domain. LD is the binary classification accuracy, where labels and indicate the source and target domains, respectively. In practice, we use the cross-entropy loss as differentiable surrogate loss. As in Ganin al. (2016b), solving the minimax optimization ming max, Ldiv is equivalent to aligning source- and target-domain data distributions the latent (encoding) space induced by the encoder ő.
Predicted Reward. The predicted reward term in Eqn. (5). i.e., ΣN (00
ΣN 1aa + àr] ((x.)))) can be translated to the loss term ΣN Creg, where for each round we have
c
(0, 2l ( ) [0 + ad (100 ((000((xa)))))
Minimizing the predicted reward e regularizes the model avoid overestimating rewards.
Constant Term. In Eqn. (5), 1b + the constant term, where e is defined Definition 3.7, and C contains small constants which can be ignored (see the Appendix for more details).
7

---

Published as a conference paper at ICLR 2025
Table 1: Accuracy on the target domain for the DIGIT dataset. Note that in this zero-shot target regret setting, the accuracy ACC = 1- - N Ry, where Ry is the target regret.
Menics
Asc Per Clas
Ase
Tod Assumay
Avcrage
LINUCB CLi et 2010)
0.3667885
0,5002
0.3811-06
0.315
0,3445
9,4229um0
0.455
0,5459:2/2
0.34998311
1.3906
LINUCB-P at. 2010
0.36454882
063984854
0,315t002
02709
0.228_
a00
0
0.786.UM
0.12732212
0.29924413
0.3060
NLIJUCB
2020)
0.78
0,3281022
0.356500
04312 0.366.00M
0,4766uve
04004.0v
0,0134.310
0.36131843
0.38161834
NLNvICB-P Dluctal,
025884653
0,223466
0.388500
0.297--00
0.00/
0818000000
0.21466301
0.20064332
DABAND (OURS)
0.600
0,682
06188.00
0.3703-0u
0,6005.0r6
0.5534
0,6010.000
0.5799-0m
0.6266-20
0.5023
0.60024333
OURS NLIUUCH
+0,1100
+0.3454
+1.26119
-0.2345
-0.1244
+0.1715
+0.1231
+0.1410
+8.2186
Table Accuracy on the target domain for the VisDA17 dataset. Note that in this zero-shot target regret setting, the accuracy ACC = - RT, where Ry is the target regret.
Telt MANRINY
occeplone
beyde
1ea
kafe
plax
Irae
rick
Awcrage
LISEIC oL 3500
NiA
NiA
NDA
MEA
NAA
NA
NIA
NIA
NS
XIA
X/A
LOHUCK et al. 2010
DAG10LMC
DOWMLLEO
01 011300005
0.648 Lo2
0.0992000
SIMIDUM:
0.000LM
0.59011000
004602012
0.1 8.1145
MLIUUC Ole 20019
66300-440
0.000
20070
SU6TOAA
01892
40000
11000
NUSUICH
00000
68222.000
(00)000
00130:306
0.003
4.2 PUTTING EVERYTHING TOGETHER: DABAND TRAINING ALGORITHM
Putting all non-constant loss terms above together, we have the final minimax optimization problem
ming maxg L DAAand
(10)
where the value function (objective function):
DABand
(c..s Rs + c,5(h)
+ +2-2,5(h)
2z + A Cdie
(1I)
Note that we need to alternate between updating Ō using LinUCB Li et al. (2010) and updating the encoder & with the discriminator g in Eqn. (10).
Formally, our method, described in Alg. 1, operates as follows: In each iteration, we access the context in the original feature space (i.e., Rd)) for each action from both source and target domains, denoted as (= aelk and {x, [c[K] respectively. Subsequently, we compute the latent representation using the encoder <6-) and select the action ā; for iteration based on the LinUCB selection rule. The bandit parameters (i.e., 0, A. b) are updated in each iteration. The weights in é are updated every episode of length H (i.e., a batch with past iterations) using our objective function in Eqn. (10), following the same rule as in Neural-LinUCB (Xu et al., 2020).
5 EXPERIMENTS
In this section, we compare DABand with existing methods on real-world datasets.
Datasets. To demonstrate the effectiveness of our DABand, We evaluate our methods in terms of prediction accuracy and zero-shot target regret on three datasets, i.e., DIGIT (Ganin et al., 201 2016a), VisDA17 (Peng et al., 2017), and S2RDA49 (Tang & Jia, 2023). See details for each dataset in Appendix B.
Baselines. We compare our DABand with both classic and -fate-of-the-arr contextual bandit al- gorithms, including LinUCB (Li et al., 2010), LinUCB with principle component analysis (PCA) pre-processing (i.e., LinUCB-P), Neural-LinUCB (Xu et al., 2020) (NLinUCB), and Neural-LinUUB variant that incorporates PCA (i.c., NLinUCB-P) Details for each baselines and discussion are in Ap- pendix C. Note that domain adaptation baselines are not applicable to our setting because it only works in offline settings and assumes complete observability of labels in the source domain (see Ap- pendix C for details).
Zero-Shot Target Regret (Accuracy). We evaluate different methods on the zero-shot target regret setting in Definition 3.5, where all methods can only collect feedback from the source domain, but not the target domain. In this setting, accuracy is equal to 1 minus the average target regret, i.e.,
8

---

Published as a conference paper at ICLR 2025
Table 3: Accuracy the target domain for the S2RDA49 dataset. Note that in this zero-shot target regret setting, the accuracy ACC = =1 - Rr, where Ry is the target regret.
Mentic
Arc Per Cloa
Toi Ascurayy
acroplane
Heycle
bau
Inii
motrocycle
plant
skatebourd
Irain
Average
LINUCB Cla al, 2010)
NIA
NIA
NAA
MAA
NA
NA
NiA
NAA
NiA
N/A
NiA
LENUCB-P P PLi 2010)
0,14504622
0.20011233
00810uom
08240
00070
0,1060
0.0090.am
0,0950.202
0.12504332
0.10294832
NLNOCB ( al, amtn
0.1210462>
0,010,000
00800-0e
0,220enie
6600
0.230
0,0094231
0 01106
NLNVICB.P Dfuct
000004000
0.000a66)
0.000.000
0,700
6.00
0.0000.00
00000
0.00002000
0.1150
041154931
DABAND (OURS)
0,844200
0.410
11
0.3933
OurS NLaNUCB
(0.151
60.5208
+0.5334
00.146
000140
+0.7340
+0.2969
+0.1255
4
+0,2817
+0.2805
DAbund
DASund
1750
DABund
(onukibve hojee
1.000-
1200
NLinUCS
LinUCe F
NLinUCE:
Comulatte Sagwe
1500
1250
Un.KB P
NLnUKe-
NLnuke
Camulaiv Rogeer
1250
NURLCS
NUHACS-P
UruCBIP
Linuce
1203
3000
000
750
400-
503-
500
230
300
750
1250 3500 3750 2003 found
1000
230 0
300
750
1000
1250
1500 1750 2000
0
150
500
750 2000
1250 found
1305 3750 2000
Round
Figure 1: The cumulative regrets of different methods for 1,920 rounds on DIGIT (left), VisDA17 (middle), and S2RDA49 (right). Results are averaged over 5 runs. LinUCB is not reported for VisDA17 and S2RDA49 due to out-of-memory issues.
ACC = 1- रे Rr. where Ry is defined in Eqn. (2) and Definition 3.5. Therefore higher accuracy indicates lower target regret and better performance.
ACC=
We report the per-class accuracy and average accuracy of different methods in Table 1, Table 2 and Table for DIGIT, VisDA17 and S2RDA49, respectively. Our DABand demonstrates favorable performance against NLinUCB, leading contextual bandit algorithm with representation learning. Despite NLinUCB's s superior representation learning capabilities through neural networks (NN) over LinUCB yielding impressive performance in single-domain applications its efficacy in cross-domain tasks is heavily contingent on the disparity between the source and target domains. Typically, the strengths of NN, which include enhanced representation learning (feature extraction), may inadvertently amplify this domain divergence, potentially leading to overfitting on the source domain. Conversely, our DABand algorithm not only improves accuracy but also adeptly addresses the challenges posed by domain divergence. Note that LinUCB encounters an out-of-memory issue due to the nigh-dimensionalityy of the data in VisDA17 and S2RDA49, marked as "N/A" in Table and Table 3, respectively.
Performance Analysis of LinUCB and NLinUCB. Our target regret bound in Eqn. (5) consists of 5 terms, i.e., source regret, regression error, data divergence, constant, and predicted reward. After applying linear contextual bandits on the source domain, the source regret enjoys a sub-linear rate. However, in the linear model, the encoder @(x) = x is an identity function and therefore fail to align dource-domain and target-domain contexts, leading in Eqn. (5). This is the main reason why linear contextual bandits, such as LinUCB, perform poorly in the cross-domain contextual bandit setting.
unbounded, large data divergence term
Note that even contextual bandits using a nonlinear encoder, e.g., Neural-LinUUCB (NLin fail in this case because their encoders @(x) are not learned to align source-domain and target-domain contexts; they therefore will still lead to an unbounded, large data divergence term in Eqn. (5), and subsequently poor performance in the cross-domain contextual bandit setting.
Continued Training in Target Domains and Cumulative Regret. Besides zero-shot target regret, we also evaluate how different methods perform when one continues their training in the high-cost target domain and starts to collect feedback (reward) after the model is trained in the source domain. A good cross-domain bandit model can provide head-start in this setting, therefore enjoying significantly lower cumulative regret in the target domain and saving substantial cost in collecting feedback in the high-cost target domain.
Fig. 1 shows the cumulative regrets of different methods on DIGIT, VisDA17 and S2RDA49, respec- tively. We show results averaged over five runs with different random seeds. Our DABand consistently

---

Published as a conference paper at ICLR 2025
surpasses all baselines for all rounds in terms of both cumulative regrets and their increase rates, demonstrating DABand's ' potential to significantly reduce the cost in high-cost target domains.
Ablation Study. To verify the effec- tiveness each term our DABand's regret bound (Theorem 3.1, we re- port the accuracy of our proposed DA- Band after removing the Regression Error term and/or the Predicted Re- ward term (during training) in Table for DIGIT, VisDA17 and S2RDA49. Results on all datasets show that re- moving either term will lead to per- formance drop. For example, in VisDA17, our full DABand achieves accuracy of 0. 0646;2; the accuracy drops to 0.4238 after removing the Regression Error (R) term, and further drops to 0.4088 after removing the Predicted Reward (P) term. These results verify the effectiveness of these two terms our DABand.
Table 4: Results of the ablation study in terms of accuracy (higher better). Note that the accuracy ACCC = RT, where Ry is the target regret. "R" and "p" is short for "Regression Error" and "Predicted Reward", respectively.
Datasets
w/o & P
w/o R
w/o P
DABAND (FULL.)
DIGIT
0.5676
0.5682
0.5768
0.6002
VISDA17
0.4088
0.4096
0.4304
0.4644
S2RDA49
0.3691
0.3694
0.3719
0,3923
CONCLUSION
In this paper, we introduce a novel domain adaptive neural contextual bandit algorithm, DABand, which adeptly combines effective exploration with representation alignment, utilizing unlabeled data from both source and target domains. Our theoretical analysis demonstrates that DABand can attain sub-linear regret bound within the target domain. This marks the first regret analysis for domain adaptation in contextual bandit problems incorporating deep representation, shallow exploration, and adversarial alignment. We show that all these elements are instrumental in the domain adaptive bandit setting on real-world datasets. Moving forward, interesting future research could be to uncover more innovative techniques for aligning the source and target domains (mentioned in Appendix F), particularly within the constraints of: (1) bandit settings, as opposed to classification settings, and (2) the high- ligh-dimensionll and dense nature the target domain, contrasted with the sparse and simplistic nature of the source domain. It would also be interesting explore the effect of improved uncertainty quantification (Wang & Wang, 2023), potentially via Bayesian deep learning (Wang Yeung, 201 6; 2020; Huang et al., 2019), DABand's performance.
ACKNOWLEDGMENTS
We sincerely appreciate the generous support from the reviewers/AC for the constructive comments to improve the paper. ZW and XH are partially sponsored by a subcontract of NSF grant 2229876, the A. Russell Chandler III Professorship Georgia Institute of Technology, and NIH-sponsored Georgia Clinical & Translational Science Alliance. HW partially supported by the Microsoft Research AI & Society Fellowship, NSF Grant IIS-2127918. NSF CAREER Award IIS-2340125, NIH Grant IR01CA297832, and the Amazon Faculty Research Award. This research is also supported by NSF National Artificial Intelligence Research Resource (NAIRR) Pilot and the Frontera supercomputer, funded by the National Science Foundation (award NSF-OAC 1818253) and hosted at the Texas Advanced Computing Center (TACC) The University of Texas at Austin. Finally, we extend our gratitude to the Center for AI Safety (CAIS) for providing the essential computing resources that made this work possible.
REFERENCES
Alekh Agarwal, Daniel Hsu, Satyen Kale, John Langford, Lihong Li, and Robert E. Schapire. Taming the monster: A fast and simple algorithm for contextual bandits. In ICML, pp. 1638-1646, 2014.
Peter Auer. Using confidence bounds for exploitation-exxllrraion trade-offs. JMLR, 2002.
10

---

Published as a conference paper at ICLR 2025
Andrei Barbu, David Mayo, Julian Alverio, William Luo, Christopher Wang, Dan Gutfreund, Josh Tenenbaum, and Boris Katz. Objectnet: A large-scale cantroondoleed dataset for pushing the limits of object recognition models. Advances in neural information processing systems, 32, 2019.
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning, 79:151 19:151- 2010.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information- 3d model repository. arXiv preprint arXiv: 111..05122012221 2015.
Ziliang Chen, Jingyu Zhuang, Xiaodan Liang, and Liang Lin. Blending-target domain adaptation by adversarial meta- adaatatapttiinnnnn networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2019.
Wei Chu, Lihong Li, Lev Reyzin, and Robert Schapire. Contextual bandits with linear payoff functions. In AISTATS, pp. 208-214, 2011.
Shuyang Dai, Kihyuk Sohn, Yi-Hsuan Tsai, Lawrence Carin, and Manmohan Chandraker. Adaptation across extreme variations using unlabeled domain bridges. arXiv preprint (rXiv:1906.02238. 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Ieee, 2009.
Kingma Diederik. Adam: method for stochastic optimization. (No Title), 2014.
Dylan J. Foster, Alekh Agarwal, Miroslav Dudik, Haipeng Luo, and Robert E. Schapire. Practical contextual bandits with regression oracles. In ICML, pp. 1534-1543, 2018.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural networks. Journal of machine learning research, 17(59):1-35, 2016a.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. JMLR, 17(1):2096-200 20300, 2016b.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Fart Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS, pp. 2672-2680, 2014.
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. Multi-source domain adaptation for text classifi- cation via distancenet-banddit. In Proceedings of the AAAI conference on artificial intelligence, 2020.
Hengguan Huang, Hao Wang, and Brian Mak. Recurrent poisson process unit for speech recognition. In AAAI, volume 33, pp. 6538-6545, 2019.
Nathan Korda, Balazs Szorenyi, and Shuai Li. Distributed clustering of lincar bandits in peer to peer networks. In ICML., pp. 1301-1309, 2016.
Seiichi Kuroki, Nontawat Charoenphakdee, Han Bao, Junya Honda, Issei Sato, and Masashi Sugiyama. Unsupervised domain adaptation based on source-guided discrepancy. AAAI, pp. 4122-4122 2019.
Branislav Kveton, Csaba Szepesvári, Anup Rao, Zheng Wen, Yasin -adasii and S. Muthukr- ishnan. Stochastie low-rank bandits. CoRR, abs/1712.04644, 2017.
Lihong Li. Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In WWW. pp. 661-670, 2010.
Shuai Li, Alexandros Karatzoglou, and Claudio Gentile, Collaborative filtering bandits. SIGIR, pp. 539-548, 2016.
11

---

Published as a conference paper at ICLR 2025
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings. Part V 13, pp. 740-755. Springer, 2014.
Tianyi Liu, Zihao Xu, Hao He. Guangyuan Hao, Guang- He Lee, and Hao Wang. Haxonomy-structum domain adaptation. In ICML, 2023.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I. Jordan. Conditional adversarial domain adaptation. In NIPS, pp. 1647-1657, 2018.
Kanak Mahadik, Qingyun Wu, Shuai Li, and Amit Sabne. Fast distributed bandits for online recommendation systems. In SC, pp. 1-13, 2020.
Le Thanh Nguyen-Meidine, Atif Belal, Madhu Kiran, Jose Dolz, Louis-Antoine Blais-Morin and Eric Granger. Unsupervised multi multi-target domain adaptation through knowledge distillation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1339- 1347, 2021.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. TKDE, 22(10):1345-133) 1359, 2009.
Sinno Jialin Pan, Ivor W Tsang, James T Kwok, and Qiang Yang. Domain adaptation via transfer component analysis. TNN, 2292):199-210, 2010.
Xingchao Peng, Ben Usman, Neela Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.0692: 2017.
Xingchao Peng, Qinxun Bai, Xide Xia, Zijun Huang, Kate Saenko, and Bo Wang. Moment matching for multi-source domain adaptation. In Proceedings the IEEE/CVF International Conference Computer Vision pp. 1406-1415, 2019.
Viraj Prabhu, Shivam Khare, Deeksha Kartik, and Judy Hoffman. Sentry: Selective entropy opti- mization via committee consistency for unsupervised domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8558-8567, 2021.
Esteban Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan, and Vincent Vanhoucke. Youtube- boundingboxes: A large high-precision human-annotated data set for object detection in video. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5296-5305, 2017.
Kuniaki Saito, Kohei Watanabe, Yoshitaka Ushiku, and Tatsuya Harada. Maximum classifier discrepancy for unsupervised domain adaptation. In CVPR, pp- 3723-3732, 2018.
Swami Sankaranarayanan, Yogesh Balaji, Carlos D. Castillo, and Rama Chellappa. Generate to adapt: Aligning domains using generative adversarial networks. In CVPR, pp. 8503-8512, 2018.
Haizhou Shi and Hao Wang. A unified approach domain incremental learning with memory:
Theory and algorithm. In NeurlPS, 2023.
Nian Si, Fan Zhang, Zhengyuan Zhou, and Jose Blanchet. Distributionally robust batch contextual bandits. Management Science, 69 69110):577--5793, 2023.
Baochen Sun and Kate Saenko. Deep CORAL: correlation alignment for deep domain adaptation. In ICCV workshop on Transferring and Adapting Source Knowledge in Computer Vision (TASK-CV) pp. 443-450, 2016.
Hui Tang and Kui Jia. A new benchmark: On the utility of synthetic data with blender for bare super- vised learning and downstream domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15954-15964, 2023.
Eric Tzeng, Judy Hoffman, Ning Zhang, Kate Saenko, and Trevor Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv: 1412.3474 2014.
Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In CVPR, pp. 7167-7176, 2017.
12

---

Published as a conference paper at ICLR 2025
Hao Wang and Dit- Yan Yeung. Towards bayesian deep learning: A framework and some existing methods. TDKE, 28(12):3395-34 3408, 2016.
Hao Wang and Dit-Yan Yeung. A survey on bayesian deep learning. CSUR, 53(5):1-37, 2020.
Hao Wang, SHI Xingjian, and Dit- Yan Yeung. Natural-parameter networks: A class of probabilistic neural networks. NIPS, pp. 118 118-126, 2016.
Hao Wang, Hao He, and Dina Katabi. Continuously indexed domain adaptation. arXiv preprint 200.:20007.01877, 2020.
Hao Wang, Yifei Ma, Hao Ding, and Yuyang Wang. Context uncertainty in contextual bandits with applications to recommender systems. In AAAI, 2022.
Hengyi Wang, Shiwei Tan, Zhiqing Hong, Desheng Zhang, and Hao Wang. Variational language concepts for interpreting foundation language models. In EMNLP, 2024a.
Hengyi Wang, Shiwei Tan, and Hao Wang. Probabilistic conceptual explainers: Towards trustworthy conceptual explanations for vision foundation models. In ICML, 2024b.
Ziyan Wang and Hao Wang. Variational imbalanced regression: Fair uncertainty quantification via probabilistic smoothing. In NeurlPS, 2023.
Pan Xu, Zheng Wen, Handong Zhao, and Quanquan Gu. Neural contextual bandits with deep representation and shallow exploration. arXiv preprint arXiv:2012.01780, 2020.
Zihao Xu, Guang-He He Lee, Yuyang Wang, Hao Wang, al. Graph-relational domain adaptation. In ICLR, 2022.
Zihao Xu, Guangyuan Hao, Hao He, and Hao Wang. Domain indexing variational bayes: Interpretable domain index for domain adaptation. In ICLR, 2023.
Yisong Yue and Carlos Guestrin. Linear submodular bandits and their application to diversified retrieval. In NIPS, pp. 2483-2491, 2011.
Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael I Jordan. Bridging theory and algorithm for domain adaptation. arXiv preprint arXiv: 1904.05001, 2019.
Han Zhao, Shanghang Zhang, Guanhang Wu, José M. F. Moura, João Paulo Costeira, and Geoffrey J. Gordon. Adversarial multiple source domain adaptation. In NIPS, pp. 8568 (168-8579,25 2018.
Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey J. Gordon. On learning invariant representations for domain adaptation. In ICML, pp. 7523-7532, 2019.
Mingmin Zhao, Shichao Yue, Dina Katabi, Tommi S. Jaakkola, and Matt Bianchi. Learning sleep stages from radio signals: A conditional adversarial architecture. In ICML, pp. 4100-4109, 2017.
Dongruo Zhou, Lihong I.i, and Quanquan Gu. Neural contextual bandits with upper confidence bound-based exploration. arXiv preprint arXiw:1911.04442 2019.
13

---

Published as a conference paper at ICLR 2025
A PROOFS
A. PROOFS FOR LEMMAS
Lemma A.l. For any hypotheses h, h H and A is selected by either h or I',
N
les(h, h() - (T (()
dH.H(S,T).
2
Proof. By definition of HAH (as defined in Def. 3.6), we have
Def.
dnau(S,7)
2
sup A.
[xxss| - 12 (x)] - 1xx (0bx)111(())--)))))))))())) wx)]lll
2
sup
N Ex-xxx((((() 11' M(xxI] N - (xx-7(11(xx) -'
N
AAK'H
Def. 3.3
2
sup
hs( =T((,,1)
N
AATEH
2
IV
h') -s(h,,))ee(bb,l) cT(
N
Lemma A.2. Given three hypotheses h2 Nu by H. have the following triangle inequality:
eD(b,, h2) S ( ha) + ( ha)-
(12)
Proof. The hypothesis difference
ep(1,,12)
h1 (2 - k2)
Def. 3.2
h1(x )) - k2( + 1)((1)) - -1a(0
Σ
h1(2 ( - hs(xD)
+ Σ 1-- hs(xP) - 122(1))
hb) Es(h ha).
Lemma A.3. Let H be a hypothesis space.
Then for every h H:
(T))) S (s() + (S +
(13)
where va is defined in Definition 3.7.
Proof. By definition in Definition 3.3 in main paper, we start from cT(h) = CT (h, fr), then we have
((h)
()T,,h)
Lm. A.2 S
G(fT,h" h²) + (h)
((h))) + (s(h leT(h, h²) Es(h,, h²))
Lm. A.I
er(h")+ + ts + N
S
2 dnan(S,7)
N
e7(h") cs(h) =s(("²) dnan((,T) N <s(h) + 3 (HAHHSTT)
14

---

Published as a conference paper at ICLR 2025
Lemma A.4. Denoting the contexts from the target domain T as {x1 (et]) and the associated ground-truth action as {a LN the estimated regret (i.e., using the trained model 0 from the source domain) for the target domain is
N Σ -1000
N
N + a RT
(0,
- E ((0x +
where ki is max {2 p|0oxxTaa A,= | )
+ + -110xaaa)1
Proof. By definition, we have
Σ (10,0xxTe - 10, (x)
Σ
@ oi 0, (xT(x + a |1x0xxaaa A73 - -a1 |1 + =a1( - -a 2110
E
(1
a | > Ilazt))
(0,00(TTa))) |
Σ (11 1.2 - -1100xTTo2
(1 0 0.0xx([)))) )) | (1att)) ( (0.00((Ta))) + +1(0xTaa))1
a max | 11 173 | |
(14)
(xxxTa) |11 )) + Σ a max { | |
(15)
(1 T(x
11
a max 2 p|
||00oTaa)l + ||0csTai))
A,3
=> 0(xx
Σ x xT.
(16)
where the inequality in Eqn. (14) is based on the fact that for any a. > 0, la bl < max(a,b), 6}. The inequality in Eqn. (15) is derived from the definition of ā, since ā se-
lected from the maximum value of (0,0(xTTa)) + a [o(xTa A. Iae.) therefore we guarantee that
(0,00x + | lA
(0 (0.0(x(TT))) + | A.l ). In addition, Eqn. (16)
holds by definition as we define s? as max
{2 22100(xTa) (2a A73 |0(xTTnr
A23 +
+110xxaa)
Next, we discuss the property of KT. in the following lemma. Lemma A.5. For any arbitrary domain D, we denote
K max [2
| 1l:: + +1[00xpa)
||
Then, by our DABand algorithm.
A=
A
as the time step i + Do, we have x. + O.
Proof. By definition of A, its singular value increase as i increase, which means that the singular values in its inverse matrix, A-1, will decrease to O. Therefore, for any new array u, ||
15

---

Published as a conference paper at ICLR 2025
V uA u²" + O. Then when + co, we have
lim No max { (2 ||a3; [
A72 |
|
}
+
O.
Lemma A.6. Denoting the contexts from the source domain S as (xM,)) the associated ground-truth action as {az (a]):= we have the following inequality:
Σ (,xxxx) 18", (00 S cs(h) + Σ 1 * ā.]
(((x)())())1)) + ki))
where ost is defined in Lemma A.5.
Proof. By definition, we have
(00, (xx. 10''0"0000000000""""""0""""" (x{(a)))
[1 a.l (10, 06x 10 (x 111) iaa ail ((x,x)) (0", ,0 (x(x)))
Σ [1 a.l (0. 0(x + 1a; * a.1 10.000x )) - 01)
W
Σ [iaa a: (0,00 +aa * a.) ( (0, -0, 0( 1)
Σ [1o0 a.l 0 11) + 1a # a.l ((x - ol) + +1 # a.l (1 x(x{ (x,0(x(a))))
L.m. A.4
(106 و3 (x )) 0, 6( 11) + Iaa + a.l @(0,0 1) ta. k².
N
cs(h) + > iaa a.l ( +
A.2 PROOF FOR THE TARGET REGRET BOUND
Theorem A.1 (Target Regret Bound). Denoting the contexts from the source domain S as {x J.S.N [c the associated ground-truth action as {at and the contexts from the target domain T as (iee),))))))) the upper bound for our target regret Ry is
Ry Φ
Σ
(0 (i
(0r, 05 (x..x.)
Rs.
2 ch(
N (HAAMSST)
+
Source Regret
Regresssion Frrar
Divergence
Constant
Σ (00,00xx7
1
1la + + a.] ((0.0(xfa))))
Predicted Rewards
where Rs, (s()), and dusu are the source regret, source-domain error, and H&H divergence defined in Definition 3.4, Definition 3.3, and Definition 3.6, respectively. ap is a constant independent to the problem and and a constant which can be ignored.
Proof. Please check the full proof in the next page.
16

---

Published as a conference paper at ICLR 2025
By the definition of the Target Regret Bound, we have
RT
100 .6 (x[ox)) (0 x (x(a))
(100000 10'xxx0(x (xx[a)) )} 10"00 (xxxx -1 (0.0)x(a)))
(1100 8² 100000 )} (0, -xxx(aa)- 0(xTa,
111) + Σ (000 - (0, -0,
i=1
N
VI
G())+ + Σ 1100'0('' (xTxxt))) (0, x(x )) + (0, x(xT(x)) )) - (0, (110x))22)
N
(h()) + Σ (10 )) (0, ((xxx)))) )1/
+ Σ (1 100 . (xxTat)) )) - (0, ((xT(a))
(b()) + Σ (10, )) 0,
-0 ((xTa) 1)
+ Σ ( 10 10',0" 8i
)} -( 2(x
Σ ( @0, 2 11
Σ (110r (xx(::) - (0, )1
+
1=1
eT( + Σ (100, - @0 ((xTaa)1)
+ Σ (1000 07 -0 0(xc(at)1)
(0,
N
N
Σ (1ro of (xTxx)) )) - (0, ((xTao))1 - -Σ
(6, (100)
+
(h))+ + N
(S.T) + Σ (10, - (0, ((xsTa))))
2
n +
(100 o" (xi )) - (0, ((x0(x) + 10", o (xffa)) - -100, o0 (x((a)))
L.m.. A.3 S
<b( + tb + 1/225 )dnAA(S.T) (5+ N
N
2 (S,
Σ
N
(0. 3(xx(x)))
N
+ (0000000 8² (xx 1) - 10", (x{xx))))
+ +
( (10 (0,0(x((;)) )) - 00,,0"(20 (xx
1=1
L.m. A.4 $
<s(h) + + + (S, +) Σ ((0.0(xTa.))) + Σ a no
N
N +Rs + Σ (, )) - 10"00"
-10))))000000000 (x((x)))
L.m. A.6 5
N (s(h) + t + Ndnan (S,))++) Σ (10 (2)(TTaaa)))) + Σ T.
i=1
+Rs + cs(h) + Σ 1 + aā, ((00 )3 + to mm))
L.m.A.5
Rs
+
2 2.
+ NN. -.dnnnn T) + +
Source Reyre!
Regression Error
Data Divergence
Constant
N
( 1aa A a.]
Σ ((0A6
Predicted Rewards
17

---

Published as a conference paper at ICLR 2025
where C= = Ei= a .T. + 1a; # ā,l (Eia a. n².) is a small number which can be ignored as (e..))))
i + oo (i.e., see Lemma A.5).
B DATASETS
To evaluate the effectiveness of our DABand, for each dataset, we treat the "easy" domain as the low-cost source domain and the more challenging one as the high-cost target domain. This allows us to demonstrate the efficacy of our method by adapting from simpler to more complex domain within a contextual bandit setting.
DIGIT. Our DIGIT dataset consists of MNIST and MNIST-M. MNIST is gray-scale hand-written digit dataset, while MNIST-M (Ganin et al., 20 2016a) features color digits. In this paper, MNIST serves as our source domain, and MNIST-M as target domain, offering a more challenging adaptation path. The MNIST digits are gray-scale and uniform in size, aspect ratio, and intensity range, in stark contrast to the colorful and varied digits of MNIST-M. Therefore, adapting from MNIST to MNIST-M presents a greater challenge than adapting from MNIST-M to MNIST, where the target domain is less complex.
/1. The VisDA-2017 image classification challenge (Peng et al., 2017) addresses a domain adaptation problem across 12 classes, involving three distinct datasets. The training set consists of 3D rendering images, whereas the validation and test sets feature real images from the COCO (Lin al., 2014) and YouTube Bounding Boxes (Real et al., 2017) datasets, respectively. Ground truth labels were provided only for the training and validation sets. Scores for the test set were calculated by a server operated by the competition organizers. For our purposes, we use the training set as the source domain and the validation set as the target domain.
S2RDA49. The S2RDA49 (Synthetic-to-Be (o-thet is a new benchmark dataset (Tang & Jia, 2023) constructed in 2023. This dataset contains 49 classes. The source domain (i.e., the synthetic domain) synthesized by rendering 3D models from ShapeNet (Chang et al., 2015). The used 3D models are in the same label space as the target/real domain, and each class has 12K rendered RGB images. The target domain (i.e., the real domain) of S2RDA49 contains 60535 images from 49 classes, collected from the ImageNet validation set (Deng et al., 2009), ObjectNet (Barbu et al., 2019), VisDA2017 validation set (Peng et al., 2017), and the web. this paper, we select the only 10 class that matches the VisDA 17 dataset.
C BASELINES
We compare our DABand with both classic and state-of- -he----rtth-tttt- contextual bandit algorithms. We start with LinUCB (Li et al., 2010), a typical baseline for linear contextual bandit problems. To enable comprehensive comparisons, we extend our evaluation to include LinUCB augmented with pre-processed features using principle component analysis (PCA), referred to as LinUCB-P. In LinUCB-P, we first apply PCA to fit on training data (without labels/feedback) in both source and target domain. Then, with transformed, lower-dimensional context data, LinUCB is performed to whether this pre-all rre-alignment procedure can transfer the knowledge the target domain. Another pivotal baseline in our study is Neural-LinUCB (Xu et al., 2020) (NLinUCB), which utilizes several layers of fully connected neural networks to dynamically process the original features the beginning of every iteration. Similar to LinUCB-P, we introduce a NLinUCB variant that incorporates PCA, i.e., NLinUCB-P. Note that domain adaptation baselines are not applicable our setting. Specifically, domain adaptation methods only work in offline settings, and assume complete observability of labels in the source domain. contrast, contextual bandit an online setting where the oracle revealed only when correctly predicted. Therefore domain adaptation methods are not applicable to our online bandit settings.
D IMPLEMENTATION DETAILS
In this section, we provide detailed insights into the implementation of our approach, applied two distinct datasets: DIGIT and VisDA17. We use Pytorch to implement our method, and all experiments servers with NVIDA A5000 GPUs.
18

---

Published as a conference paper at ICLR 2025
DIGIT. Within the DIGIT dataset framework, we the MNIST dataset the source domain and MNIST-M as the target domain. To ensure compatibility between the datasets, we standardize the channel size of images in the source domain (cs 1) to align with that in the target domain (eT = 3).. Each image undergoes normalization and is resized to 28 x 28 pixels with 3 channels to accommodate the format requirements of both domains. Then, an encoder is utilized to diminish the data's dimensionality to a more manageable latent space. Following this reduction, the data processed through two fully connected neural network layers, ending in the final latent space necessary for loss computation as delineated in main paper. For the optimal hyperparameters, we set the learning rate to 1 X e with chosen from {1.0, 5.0, 10.0, 15.0, 20.0} and kept the same for all experiments. Additionally, we set the exploration rate to 0.05.
VisDA17. We use the VisDA17 dataset's training set as the source domain, with the validation set functioning as the target domain. We adhere to preprocessing steps established by (Prabhu et al., 2021) to ensure uniformity across domains: Each image normalized and resized to 224 x 224 pixels with 3 channels, matching the requisite specifications for both source and target domains. For hyper- parameters, the learning rate of x e is applied, with À is chosen from {1.0, 5.0, 10.0, 15.0, 20.0} and then kept the same for all experiments. We set the exploration rate a to 0.05.
S2RDA49. We selects 10 classes from the original 49 classes since it matches the target domain samples in VisDA17 (Peng et al., 2017). We adhere to preprocessing steps established by (Prabhu et al., 2021) to ensure uniformity across domains: Each image is normalized and resized to 224 x 224 pixels with 3 channels. For hyperparameters, the learning rate of e is applied, with À chosen from {1.0, 5.0, 10.0, 15.0, 20.0} and then kept the same for all experiments. We set the exploration rate a to 0.01.
Table 5: Results of the ablation studies in terms of accuracy (higher is better). Note that the accuracy ACC = 1 x Ry, where Ry is the target regret. "R", "P" and "D" are short for "Regression Error", "Predicted Reward" and "Data Divergence" (i.e., adversarial loss and the discriminator), respectively.
Datasets
w/o R&P&
w/o R&PP
w/o R&D
W/o P&DD
W/o
w/o P
w/o D
DABAND (PULL.
DIGIT
0.38162004
0.5676200z
0.35442001 0.
0.5682=am
0.5768=00m
0.364920000
0.6002=000
VISDA17
2.000
0.4088
0.09362001 0.10101002
0.4304 0.4096
20000
0.4644-000
S2RDA49
100
0,369 2002
0.09182001
0.1032a001
0.3719 0.3694
0.1121
0,3923-0:0
E FULL ABLATION STUDIES
The full ablation study is shown in Table 5. Furthermore, we ran the corresponding hypothesis tests, and the p values are in the range of (3.201 x 10-21, 1.504 x 10-2), much lower than the threshold of 0.05 and therefore verifying the significance of DABand's performance improvement.
F LIMITATION
This work contains several limitations. Specifically, we highlight below:
Intuitive Perspective. The bandit algorithm indeed enjoys interpretability (Wang et al., 2024b;a) and the accuracy of its closed-form updates (Li et al., 2010), but this is true only because it typically employs a linear model.
Unfortunately, linear models do not work very well for real-world data, which is often high- dimensional. For example, the empirical results for LinUCB and LinUCB-P in Table and Table show that such linear contextual bandit algorithms significantly underperform state-of-the-ard neural bandit algorithms, which use back-propagation (similar results are shown in (Zhou et al., 2019; Xu et al., 2020)). In summary deep learning models with propppropagttion is necessary due to the following reasons:
High- Dinnensammiiinnl Data. To enhance performance in real-world high-dimensional data (e.g., back-propagee propagation) is necessary, though this may sacrifice portion of the algorithm's explanatory power.
images), the integration of deep learning (and back-
19

---

Published as a conference paper at ICLR 2025
Covariate Shift and Aligning Source and Target Domains in the Latent Space. In settings where there is covariance shift between the source and target domains, deep (nonlinear) encoder is required to transform the original context into latent space where source-domain encodings and target-domain encoders can align. This also necessitates deep learning models with pack-propagation.
Indeed, there is trade-off between interpretability and performance. This would certainly be an interesting future direction, but it is out of the scope of this paper.
Theoretical Perspective. Our Theorem 3.1 is sharp, as all the inequalities are based on lemmas in the paper and the Cauchy inequality. Identifying the criteria under which the target regret bound reaches equality as well as how one can achieve it in practice would be interesting future work.
Empirical Perspective. The performance DABand largely relies on the alignment quality between the source and target domains. Therefore, for two domains that cannot be aligned (for example, most domain adaptation tasks are predefined, and the data for both domains is pre-processed and cleaned, not original weal-world data), it is challenging to evaluate how DABand can still transfer knowledge across different domains. Furthermore, it is still unknown whether, we increase the domain shift in dataset from another domain, our DABand can still perform well. These issues are beyond the scope of this paper, but they represent interesting areas for future work.
G DISCUSSIONS
G.1 SUBLINEARITY FOR THE TARGET REGRET BOUND
To see the data divergence term is sub-linear, note that our target regret bound Eqn. (5) can be divided into two parts:
Sublinear
100
Reference:
Reference: y 0.65x
Source Regret
Curulativv Results
Total Source Regret: In Neural- LinUCB (NLinUCB) (Xu et al., 2020),
Regression Error
60
Predicted Rewards
this is shown to be O(VN), which is sub-linear.
Data Divergence
Sum of others
40
Sum of Other Terms (Including the Data Divergence): These terms can be directly optimized convergence and minimized to small value us- ing SGD variants such Adam. As shown in Corollary 4.2 of the Adam paper (Diederik, 2014), the Adam opti- mizer enjoys average regret R(N)/N of 0 (+))) (here N is equivalent to T in the Adam paper), which leads to sub-line linear total regret O(VN), Since all other terms are directly optimized using Adam, they also enjoy a sub-linear regret.
20
40
60
80
100
Batch per Rounds
Figure 2: Sub-linearity for each term in Eqn. (5).
Therefore, our target regret bound is also sub-linear. Moreover:
Key Difference between the Source Regret and Other Terms. Note that our target regret bound in Eqn. (5), the source regret term Rs increases monotonically with respect to and cannot be directly minimized. In contrast, all other terms, e.g., the data divergence term N dHaH(S, T) can be directly minimized since all related data is training data and is already known. This why we minimize the corresponding loss terms like Eqn. (7), Eqn. (8) and Eqn. (9) during training.
The data divergence term can be minimized to a small value as N + the source and target domains perfectly aligned after training, the data divergence becomes 0. Therefore, the cumulative sum of the data divergence term over all rounds will be sub-linear.
cc. For example,
20

---

Published as a conference paper at ICLR 2025
G.2 DIFFERENCE BETWEEN BANDIT AND CLASSIFICATION SETTINGS
In the classification setting, given sample x, model predicts its label 7. Since its ground-truth label is known, one can directly apply the cross-entropy loss to perform back-propagation and update our model. In contrast, in bandit settings, for each sample x. our model predicts an action ā by optimizing the estimated rewards. One then submits this predicted action to the environment and receives feedback that only indicates whether the predicted action is the optimal action a* or not. If not, we are informed that our prediction is incorrect, yet we do not receive information on what the correct (optimal) action should be. Moreover, during training. we only train on 300 episodes for DIGIT and VisDA17 and 100 episodes for S2RDA49, Each episode contains 64 samples. This implies that compared with those DA methods which train on multiple epochs, in our settings, all of the models only see sample once. Such complexity makes the bandit setting much more challenging.
Why Simultaneously Training Source and Target Domains is Needed and Helpful. It is also noteworthy that our DABand is an end-to-end model that enables two-way feedback between source and target domains:
Collecting source-domain rewards helps DABand to learn a better encoder (which transforms raw contexts into encodings in the latent space) because the reward signals help the encoder extract more relevant encodings (embeddings) from target-domain contexts, thereby reducing the target-domain regret.
Source to Target:
After aligning source-domain and targef-domain encodings in the shared latent space (i.e., minimizing the data divergence term in Eqn. (5)), the information from the target-domain context can then help the source domain to explore arms that are more relevant to the target domain. Ultimately, this also helps reduce the target-domain regret even without collecting target-domain rewards.
Target to Source:
G.3 IMPORTANCE OF BANDITS AND DABAND
Bandit algorithms are designed to navigate environments where feedback is sparse, costly, and indirect. By efficiently learning from limited feedback identifying not just when prediction wrong but adapting without explicit guidance on the right choice bandit algorithms offer a strategic advantage in dynamically evolving settings. Our DABand exemplifies this by leveraging a low-cost source domain to improve performance in a high-cost target domain, thereby reducing cumulative regret (improving accuracy) while minimizing operational costs. DABand not only reduces the expense associated with acquiring and labeling vast datasets but also capitalizes on the intrinsic adaptability of bandit algorithms to learn and optimize in complex, uncertain environments.
G.4 NOVELTIES RESTATEMENT
Theoretical Novelty. In general, DABand is the first work to perform contextual bandit in domain adaptation setting, i.e., adapt from source domain with feedback to a target domain without feedback. Moreover, theoretical analysis presents two major technical challenges/novelties below:
Generalization of HAH Distance Regression. In the original multi-domain general- ization bound (Ben-David et al., 2010), the HAH divergence between source and target domains is derived for classification models. However, contextual bandit is essentially a reward regression problem, and therefore necessitates generalizing the H HAH distance from the classification case to the regression case. This presents a significant technical challenge, and leads to a series of modifications in the proof.
Decomposing the Target Regret into the Source Regret with Other Terms. The subse- quent major challenge our DABand addresses involves decomposing the regret bound for the target domain (i.e., the target regret) into source regret term and other terms to upper-bound the target regret. This is necessary because we do not have access to feedback/reward from the target domain (we only have access to feedback/teward in the source domain). This process requires a nuanced understanding of the interplay between source and target domain dynamics within model's framework.
21

---

Published as a conference paper at ICLR 2025
Other Technical Novelties. Moreover, contribution goes beyond the theoretical analysis itself. Specifically,
We identify the problem of contextual bandits across domains and propose domain domain-adaptive adaptive contextual bandits (DABand) as the first general method to explore high-cost target domain while only collecting feedhack from a low-cost source domain.
Our theoretical analysis shows that method can achieve sub-linear regret bound in the target domain.
Our empirical results on real-world datasets show our DABand significantly improves performance over the state-of-the-art contextual bandit methods when adapting across domains.
G.5 DISCUSSION ON ZERO-SHOT TARGET REGRET BOUND
G.5.1 SIGNIFICANCE OF THEOREM 3.1
Note that Theorem 3.1 is nontrivial. While it does resemble the generalization bound in domain adaptation, there are key differences. As mentioned in Observation (3) in Sec. 3.6, our target regret bound includes two additional crucial terms not found in domain adaptation. Specifically:
(105 (3 ) (0, Ma
which 1, )1)
Regression Error in the Source Domain. Σ
defines the difference between the true reward from selecting action ã. and the estimated reward for this action.
Predicted Reward. Σ ((0,0(xTa))))
+ 1|a; + a.l ((f.,(x()) which serves regularization term regularize the model to avoid overestimating rewards.
The results of the ablation study in Table 4 in Sec. 5 highlight the significance of these two terms. Please also refer "Technical Novelty" of Sec. G.4 above for discussion key novelty/challenges in deriving Theorem 3.1.
G.5.2 TESTING PHASE RATHER THAN TRADITIONAL BANDIT LEARNING SETTINGS
Our regret bound in Theorem 3.1 zero-shot regret bound for the target domain (with empirical results in the Zero-Shot Target Regret paragraph of Sec. 5), which corresponds to testing phase.
However, we would like to clarify that extending this bound to handle continued training the target domain (corresponding to the Continued Training in Target Domains and Cumulative Regret paragraph Sec. 5) is straightforward. At high level, we can have
Riatal R
Reero-shot Reontinued
with the first term handled by our DABand's Theorem 3.1 and the second term handled by typical single-domain contextual bandit.
G.5.3 SCALE OF SOURCE REGRET AND PREDICTED REWARDS
Similar to Nenral-LinUCB (NLinUCB). in our DABand, the true reward is restricted to the range of [0, we mentioned in See. 3.1); therefore it will make the source regret unbounded.
Furthermore, our DABand algorithm tries to minimize the source regret while aligning the source and target domains. The minimization of the source regret is theoretically guaranteed, as discussed in recent work such as Neural-LinUCB (NLinUCB) (Xu et al., 2020).
G.6 ARE THE COMPARISONS FAIR?
Leveraging Target-Domain Contexts. A lot of our baselines do leverage the contexts of the target domain. For example, baseline LinUCB-P starts by performing PCA jointly both source- domain and target-domain contexts and then perform LinUCB. Therefore LinUCB-P does leverage the contexts of the target domain. Similarly, NLinUCB-P starts by performing PCA jointly both
22

---

Published as a conference paper at ICLR 2025
source-domain and target-domain context and then perform NLinUCB. It therefore also leverages target-domain contexts.
Seeing Target-Domain Rewards. Note that in our domain adaptive bandit settings:
During the zero-shot phase, target rewards are not visible for all methods (include both baselines and our DABand); it is therefore fair comparison.
During the continued- -rriinndd-taaning phase, all methods (include both baselines and our DABand) start see the target reward update their parameters; is therefore also fair comparison.
G.7 WHY MINIMIZE THE PREDICTED REWARD ON THE SOURCE DOMAIN?
While the pedicted-rewart term is naturally derived from theoretical analysis, do find interesting insights when examining this term and its relation to our model. Specifically:
Indirect Regularization on Bandit Parameters and the Encoder Φ(-). One can see this term an L1 regularization term. It does not directly regularize the bandit parameter 0: however, minimizing the LI norm the predicted reward (i.e., a K -dddmmnnsional vector for a K-arm bandit) does indirectly regularize the bandit parameter 0 and the encoder Φ(-), thereby preventing the LI norm of the predicted reward from getting to large.
Smaller Predicted Rewards for Smaller Variance and Better Stability. Furthermore, this regularization can help avoid predicting high rewards for arms in the bandit.
Note that for the bandit algorithm achieve low regret, predicting large rewards are not necessary. This is because one uses the argmax operation (i.e., Line 6 in Alg. 1) to select the best arm; an arm & with a small predicted reward can still be selected as long as all other arms have even smaller predicted rewards.
Too higher predicted rewards are not desirable because they increase the model's sensitivity, leading to higher variance and subsequently increasing the generalization error (i.e., the target regret's 's bound).
G.8 CLARIFICATION OF DABAND'S CONTRIBUTIONS
Our DABand is the first general method to explore a target domain while only collecting feedback from the source domain, regardless of linear nonlinear assumptions. No prior methods have explored this setting of exploring target domain while only collecting feedback from the source domain.
Therefore, DABand's contribution is two-fold: (1) DABand is the first general method to explore a target domain while only collecting feedback from the source domain; (2) DABand is also the first general method in this domain-adaptive bandit setting that works even under general nonlinear assumptions.
G.9 IMPORTANCE OF THE DISCRIMINATOR.
Our bandit setting aims explore target domain while only collecting feedback from the source domain. All reward feedback in the target domain is unknown, making this approach very challenging. Therefore, using discriminator to align representations for both domains is necessary. If we ignore the discriminator, the method will not work in our settings. This is also evidenced by the whole results for ablation studies in Table 5.
G.10 How THE TERMS IN THE REGRET DECAY AS N INCREASES.
To see how the terms in the regret decay:
Decay in the Source Regret. Most of the decay occurs the first term (i.c., Source Regret R_S)), which has been discussed in the NLinUCB paper (Xu et al., 2020). Specifically the cumulative source regret is O( ((VN). Then, dividing both sides of Eqn. (5) by N, will get the average source regret term with O(vN/N), which does decay with N.
23

---

Published as a conference paper at ICLR 2025
Decay in the Data Divergence Term. Decay in the Data Divergence Term. The data divergence term, (8 T), can actually be further decomposed into
an empirical term that estimates the divergence using N source-domain contexts and N target-domain contexts, and term related to N with complexity (1log(2N))M) (Ben-David et al., 2010). Therefore, the second term with (log((NN/N) does also decay as increases.
G.11 UNDERSTANDING THE BOUND IF THE SOURCE AND TARGET DOMAINS ARE EQUIVALENT
Even if the input distributions of the two domains match, it does not imply that the relationship between the input context x and the predicted reward is the same for them, since the data divergence term is only responsible for aligning the distribution of (input); it is irrelevant to the output and rewards. This why other terms are needed to characterize the difference the relationship between the context x and the reward in the source and target domains. The data divergence term alone is not sufficient.
G.12 CLARIFICATION ON THE PROBLEM SETTING.
Simultaneous Observation. In our setting, source-domain contexts and target-domain contexts are simultaneously observed, with only the reward of the source domain being observed.
Target-Domain Contexts Available Even Before Running Alg. 1. Note that in practice, target- domain contexts are usually available even before Algorithm starts and before observing any reward from the source domain. (This makes it possible train our DABand by simultaneously using source-domain and target-domain contexts.)
For example, in the case of testing drug reactions on mice (source domain) and humans (target domain), usually one already has the human subjects' genomics, demographic, and other data as target-domain contexts, even before testing the drug on mice (i.e., the source domain) and collecting rewards.
This is because these human genomics/demographicc data are easy to collect at a low cost; in contrast, testing the drug on humans (i.e., the target domain) and collecting rewards involves extremely high costs, due to the risks of fatality, side effects, and the enormous costs of conducting clinical trials.
H POTENTIAL IMPACT OF DABAND
Our DABand has many potential real-world applications, especially when obtaining responses is costly. For instance, in testing drugs, and then use DABand to obtain the zero-shot hypothetical regret bound for responses in humans. In another scenario, and then transfer knowledge by aligning the divergence between drug A and drug B. If both regrets reveal acceptable performance, we might not need excessive costs for back-and-forth testing, which significantly speeds up the process and reduces costs.
construct responses from mice for new drug A.
collect data (responses) humans for another published, similar drug B,
24


Reviews:
[{"review_id": "lTZ3kkk5jZ", "reviewer": "Reviewer_2Moo", "summary": "The manuscript introduces a domain adaption algorithm for contextual bandit. It further proves that the proposed method achieves a sub-linear regret bound. Empirical experiments conducted on real-world data shows the the proposed method outperforms existing state-of-the-art contextual bandit methods for cross domains.", "questions": "See weakness.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The design of the new algorithm originates from an observation that leveraging data across domains leads to sub-linear regrets, which makes the whole method simple, yet elegant.\n\n2. Theoretical proof has been provided to support the performance of the method.\n\n3. The algorithm is extensively tested on three datasets.", "weaknesses": "It would be helpful to elaborate on why the data divergence term is sub-linear in the proof.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W5hMTEuP2j", "reviewer": "Reviewer_MPVh", "summary": "The paper introduces an algorithm for domain adaptation in contextual bandits. This method aims to adapt a neural contextual bandit algorithm from a low-cost source domain to a high-cost target domain, to address the challenge of distribution shifts between the two domains. The theoretical analysis conducted demonstrates that DABand achieves a sub-linear regret bound in the target domain, which is interesting. Empirical results are presented using real-world datasets, which show that DABand outperforms existing contextual bandit methods. The proposed work has broader implications, including potential insights into reinforcement learning with human feedback algorithms.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The proposed problem setting of contextual bandits in a domain adaptation scenario is interesting and challenging.\n\n(2) The method utilizes unlabeled data from both source and target domains for effective representation learning and alignment across different domains.\n\n(3) The algorithm is capable of attaining a sub-linear regret bound in the target domain by solving an online network lasso problem with time-dependent regularization.", "weaknesses": "The method leverages unlabeled data from both source and target domains to learn robust representations and aligns them effectively across different domains, enabling efficient domain adaptation. While the proposed algorithm builds upon the NeuralLinUCB framework, it introduces an adaptation in the loss function specifically tailored for updating the neural network. This loss function integrates insights from classic domain adaptation techniques, and thereby has some similarity with existing methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ghtz6FoEx6", "reviewer": "Reviewer_Mn7w", "summary": "This paper studies the domain transfer in (neural) contextual bandits. The authors provide a decomposition of the domain transfer in contextual bandits, and proposed an algorithm leveraging the adversarial training in domain-transfer. Theoretical results are presented understanding this decomposition and empirical evaluation are performed to validate the performance of the algorithm", "questions": "- Besides the presentation in Theorem 3.1, are there any regret scale for linear contextual bandits to help me better understand the performance of the algorithm?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The authors provide a theoretical justification on the decomposition for the domain-transfer issue for contextual bandits. \n- The authors provide empirical evaluation demonstrating the performance of the algorithm.", "weaknesses": "- (Minor) Definition 3.6 is hard to understand for me. In particular, are $\\mathbf x$ and $x$ referring to the same thing? Are the authors implicitly assumes $h \\in \\mathcal H$ and $x \\in \\mathbb R^n$ in this case? It would be helpful if the authors could rephrase the mathematical formulation on this. \n- In Theorem 3.1 needs further justification. For example, I wonder why the sublinear in $R_S$ will lead to the sublinear result in $R_T$. I checked the proof for Theorem 3.1 in appendix but found nowhere explicitly discuss this part. In addition, why the data-divergence term is sublinear? \n- The adversarial training paradigm adapted in controlling the data divergence term in the proposed algorithm requires better justification: why the hypothesis class here is the same with the hypothesis class used in contextual bandits?\n- It lacks some discussions on the performance analysis in linear contextual bandits or discussions in related literautre. The authors might want to provide more discussion on the theoretical insight of this. \n\n- (Minor) Section 4.1 title: eq (16) -> eq (5)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)