Cross-Domain Off-Policy Evaluation and Learning for Contextual Bandits
Authors: Yuta Natsubori
Masataka Ushiku
Yuta Saito
Venue: ICLR 2025 Poster (2025)
Keywords: Off-Policy Evaluation
Off-Policy Learning
Importance Weighting
Cross Domain
Primary area: reinforcement learning
TL;DR: We propose a new estimator and off-policy gradient method to deal with unsolved challenges in Off-Policy Evaluation and Learning by effectively leveraging both the target and source domain data

Abstract: Off-Policy Evaluation and Learning (OPE/L) in contextual bandits is rapidly gaining popularity in real systems because new policies can be evaluated and learned securely using only historical logged data. However, existing methods in OPE/L cannot handle many challenging but prevalent scenarios such as few-shot data, deterministic logging policies, and new actions. In many applications, such as personalized medicine, content recommendations, education, and advertising, we need to evaluate and learn new policies in the presence of these challenges. Existing methods cannot evaluate and optimize effectively in these situations due to the notorious variance issue or limited exploration in the logged data. To enable OPE/L even under these unsolved challenges, we propose a new problem setup of Cross-Domain OPE/L, where we have access not only to the logged data from the target domain in which the new policy will be implemented but also to logged datasets collected from other domains. This novel formulation is widely applicable because we can often use historical data not only from the target hospital, country, device, or user segment but also from other hospitals, countries, devices, or segments. We develop a new estimator and policy gradient method to solve OPE/L by leveraging both target and source datasets, resulting in substantially enhanced OPE/L in the previously unsolved situations in our empirical evaluations.

Published as a conference paper at ICLR 2025
CROSS-DOMAIN OFF-POLICY EVALUATION AND LEARNING FOR CONTEXTUAL BANDITS
Yuta Natsubori
Masataka Ushiku
Hakuhodo DY Holdings, Inc. yuta. nat.ubbrr @hakuhodo. co jp
Hakuhodo DY Holdings, Inc.
masataka .sh ku@ha wwwodaukuoooo.oooo co jp
Yuta Saito
Cornell University
ys5228cornell.edl
ABSTRACT
Off-Policy Evaluation and Learning (OPE/L) contextual bandits rapidly gain- ing popularity in real systems because new policies be evaluated and learned securely using only historical logged data. However, existing methods in OPE/L cannot handle many challenging but prevalent scenarios such few-shot data, deterministic logging policies, and new actions. In many applications, such as personalized medicine, content recommendations, education, and advertising, we need to evaluate and learn new policies in the presence of these challenges. Exist- ing methods cannot evaluate and optimize effectively in these situations due to the notorious variance issue or limited/no exploration in the logged data. To enable OPE/L even under these unsolved challenges, we propose a new problem setup of Cross-Domain OPE/L, where we have access not only to the logged data from the targel domain which the new policy will be implemented but also to logged datasets collected from other domains. This novel formulation is widely appli- cable because we can often use historical data not only from the target hospital, country, device, or segments. We develop a new estimator and policy gradient method to solve OPE/L by leveraging both target and source datasets, resulting in substantially hanced OPE/L in the previously unsolved situations in our empirical evaluations.
segment but also from other hospitals, countries, devices,
INTRODUCTION
Many decision-making systems (e.g., recommendation, medication, budget allocation) interact with the environment through a contextual bandit process in which a policy observes the context, takes action, and obtains rewards. Off-Policy Evaluation and Learning (OPE/L) has gained attention as technique for estimating and learning new policies without deploying them, using only historical logged data. We can find many real applications of these techniques since they do not require costly and risky online A/B testing and exploration to enable the data-driven policy evaluation and learning lifecycle (Mehrotra et al. 2018 Gilotte et al. 2018 Saito et al. 2021 Kiyohara et al. 2024a).
While recent advances have led to the development of a number of estimators and policy gradient methods (Saito & Joachims] 2021] Uehara et al. 2022),, most of these are based on inverse propen- sity scoring (IPS), reward regression, or their mixture (Dudik et al. 2014; Wang al. 2017 Su et al, 2020a). These estimators heavily rely on theoretical assumption called common support to provide low-bias estimate. Due to the common support assumption, we can only evaluate and learn new policies regarding actions that have already been sufficiently explored by the logging pol- icy (Sachdeva et al.) 2020 Felicioni et al. 2022). Therefore, challenging but realistic cases where the logging policy is completely deterministic or there are new actions, existing methods simply cannot evaluate and choose under-explore and new actions at all due to the lack of their reward in- formation in the historical data (Sachdeva et al. 2020). In addition, the use of importance weighting often causes severe variance issues, particularly when the sample size is small, such as few-shot data
and the action space is large (Saito & Joachims 2022 Cief al. 2024a Sachdeva et al. 2024

---

Published as a conference paper at ICLR 2025
Conventional OPE/L
Cross-Domas OPE/L (Ours)
Hospital Gargeer)
Hospital A Larreer
Hosphal (sounce 1
Hosphal
Hespiall D
Lournee 2)
Coounne 3)
noW
losped data
new polcy
logged dara
Dew polky
Juaged source domais
π
π
T
D Ti
2
X A
19 M
Figure 1: Comparison of Conventional OPE/L and Cross-Domain OPE/L (Ours).
There have been several previous efforts to enable OPE (Felicioni et al. 2022) and OPL. (Sachdeva et al. 2020) under the violation of common support (also known support deficiency). ) but they cannot handle deterministic logging policies and completely new actions. It might also be useful to use some structure in the action or reward space to relax the requirement of common support, as studied by [Saioo & Joachims (2022); Saito et al. (2023): Cief et al. (2024a); [Taufiq et al. (2023); Sachdeva al. (2024); Kiyohara et al. (2024b) but this useful structure not always learnable.
To enable effective OPE/L even with deterministic logging policies and new actions, we propose a new problem formulation called Cross-Domale OPE/L where we aim to evaluate and optimize the value of new policies under the target domain, but have access to historical logged data collected previously in source domains (as depicted in Figure D.. There are many cases where we have access to such multiple-logged datasets. In the medical field, example scenario is when the impact of treatment on a patient's 's progmosis is recorded in several hospitals of different sizes and with different patient demographics. Another example is in recommender systems where new content function may become available in certain countries or for subset of users (c.g., active members), and these can be source domains when aim perform OPE/I. regarding new countries and other users, such as relatively new members. If we have access to these logged datasets collected in source domains with more data and exploration in the action space, we can leverage these data to perform OPE/L more effectively towards the target domain even when it has less logged data, a deterministic logging policy, and new actions that existing estimators cannot address.
After formally presenting the new formulation, we propose a new estimator, called Cross-domatt Off-Policc Evaluation (COPE) and respective policy gradient method to solve OPE/L by effec- tively leveraging useful information from source domains to estimate and optimize the value of new policies in the target domain. To achieve effective transfer from source domains, our methods are based on a decomposition of the expected reward function into the domain-cluster effect and domain-specific effect. The domain-cluster effect is a component in the reward function that do- mains in the same cluster have in common, while the domain-specific effect represents the causal effect not modeled merely by the domain-cluster effect. The COPE estimator unbiasedly estimates the comain-cluster effect by applying multiple importance weighting (Owen) 2013 [Agarwal et al.) 2017) using data not only from the target domain but also leveraging data from the source domains that are in the same cluster the target domain. It also deals with the domain-specific effect via reward regression using the logged data from the target domain. which reduces the bias of the esti- mator depending on the regression accuracy. We also extend the COPE estimator as a policy gradient estimator to perform OPL using both target and source data, enabling us to perform policy learning even under completely deterministic logging and new actions involved. Theoretical analysis shows that COPE has analyzable bias and, in particular, that COPE can be unbiased even when the target domain has a deterministic logging policy and new actions. Empirical evaluation using real-world recommendation dataset demonstrates that COPE outperforms existing estimators and policy learn- ing methods by properly leveraging data from both the target and source domains, particularly when there are few target data and many actions that are not previously explored in the target domain.
Key Related Work. This section differentiates our work from two closely related studies. A more comprehensive overview of related work is available in Appendix A
First, we discuss the work by Uehara et al. (020). which studied OPE/L under covariate shift, where the context distributions differ between the logged data (2:)) and the evaluation environ- ment (peval (z)), while the reward distribution p(r jx, a) remains unchanged. Under this setup, Uchara
et al. (0000) proposes methods to estimate the value of new policies when deployed in the evalua-
2

---

Published as a conference paper at ICLR 2025
tion environment, using only the logged data collected under the historical distribution p (z). In contrast, we aim to estimate the same estimand, but leveraging data from both target and source domains (where there could be multiple source domains), with each domain having its own unique context and reward distributions. By leveraging this new setup, our primary goal is to address chal- lenging scenarios in the target domain, such new actions, deterministic logging, and extremely small logged data, which clearly differ from the motivations ofUuaa et al. (2020).
We also distinguish our contributions from those offSa al. (2023). which developed the OffCEM estimator to handle OPE in large action spaces within a single domain setup. Although our main idea of reward function decomposition is inspired by OffCEM, its application solve the problem of Pross-Domain OPE/L would not be possible without our unique formulation. Moreover, we propose an extension of our estimator to an OPL method, whereas Saito et al. (2023) focused solely on the OPE problem. Therefore, our work is the first to formulate the problem of cross-domain OPE/L and leverage the respective version of reward function decomposition to solve non-trivial challenges, such as new actions and deterministic logging, and thus offers several unique contributions from both methodological and empirical angles.
Finally, we discuss an important distinction between our work and those that address limited overlap or deficient support issues (Hansen 2008; Sachdeva et al.) 2020; Wu & Fukumizu 22021] Felicioni et al. 2022). Limited overlap refers to situation where certain actions that can be taken under new policy have zero probability of being observed under the logging policy. Limited overlap is problematic because, without any data about actions in the logged data, importance weighting tech- niques become biased. It is important to note that our work aims to address even more challenging scenarios compared the general limited overlap problem, namely, completely deterministic log- ging and new actions. Deterministic logging refers to situation where the logging policy selects a specific action with probability one, i.e., there is no stochasticity. This more difficult because the typical limited overlap issue still allows the logging policy to be stochastic. Completely new actions present an even greater challenge, as they refer to actions a that have zero probability of previous work
being observed in the logged data for any context. To the best of our knowledge, specifically addresses the issues of completely deterministic logging and new actions. We tackle these extremely challenging scenarios by newly formulating the problem of cross-domain OPE/L.
2 CONVENTIONAL FORMULATION
We begin with describing the conventional formulation of OPE in the contextual bandit setting. Here, a decision maker repeatedly observes a context x E X' drawn from an unknown distribution p(z). Given a context z, a fixed and potentially stochastic policy x[a|z)) selects an action from a finite action space denoted by A. Reward r is then observed following an unknown distribution p(r|z, and we use q(2r, a) == E[r|z, uj to denote the expected reward given context and action. We define the performance measure of policy æ by the expected reward under its deployment as
V(T) = EE (eppe))x(llabpcrle... [r] = (ala)) (g(x, a)],
(1)
which is often called the policy value. The logged bandit data we can use to perform OPE/L can be denoted by 'D : {(x,, as r;) r=)] which contains n independent observations drawn from the data distribution induced by the logging policy To. i.e., p(D) II. p(x;) mo (a, (r. )r(r))rrrr))rr (r,, a.).
The aim of OPE is develop an estimator V that can accurately estimate the policy value of new policy л using only D. We measure the accuracy of V by the mean squared error (MSE) defined
MSE(V D)) = (V (T;; D) = Bias(V (T:: D)) + [D[(V D))]
where Epcd) [:] takes the expectation over the distribution of D.
Limitations of Existing Estimators. As an existing method for OPE, we first describe IPS (Horvitz & TToomsoon 1952,, which estimates the policy value by re-weighting the rewards as
T(a.lx;)) VIPS(T;; ; D) = )o(a)) ri va
1
Σ
n
w wx, ar)r
(2)
where w(z, a) == h(alx)/mm /O (a unbiased, i.e., (Fpp))/Virs(( D) V(T), under the common support condition.
is called the importance weight. It is widely known that IPS is
3

---

Published as a conference paper at ICLR 2025
Condition 2.1 (Common Support), The logging policy To is said to have common support for policy π if (aa2)) > 0 = x > 0 for all a E A and x E X.
This assumption for unbiasedness of IPS is never satisfied in situations when the logging policy mo is deterministic or there are new actions that were not available when the logged data was collected. Under the violation of Condition 2.1 IPS produces the following bias (Sachdeva et al. 2020)
(Bias(ViPs(T:: D)) = Ep(r)]
Σ
T(a)z)g(2,a))
(3)
where Mo(a, mo) :- {a E A To(a[z) = 0} is the set of deficient actions for context x under mo- This bias is due to the fact that IPS cannot evaluate the actions that are not explored by the logging policy. To deal with this bias unavoidable for Sachdeva et al. (2020) suggests using a reward regression model ((zz a) g a) in the form of Doubly Robust (DR):
(D D.q) = 1 Σ {w(z, (r; - ((x,, + E (ada,,(((
(4)
While the bias caused by support deficiency and the variance can be smaller if ((xr, a) is accu- rate (Dudík et al., [0144] SSachdeva et al.] (020), it is almost impossible to achieve such accuracy when only a small (like few-shot) dataset is available and there are new actions in the target domain.
3 CROSS-DOMAIN OFF-POLICY EVALUATION AND LEARNING
To achieve yet effective OPE/L even with no exploration and few-shot data in the target environment, this section proposes new OPE problem that aims to estimate the value of a new policy, but we can have access not only to the historical data collected from the domain in which the new policy will be implemented (target domain), but also the datasets from other domains (source domains) that have varying data generation processes (DGPs). We also develop an estimator to effectively leverage logged data from both the target and source domains and extend it policy gradient method.
Here, we introduce K different domains {k] where the logged dataset is collected under po- tentially different logging policies in each domain. One out of K domains is the target domain T. where we are interested in deploying a new policy, and the rest are source domains S. The DGP within each domain & can be formulated as a respective contextual bandit process, i.c. we first observe the context x* E X' from an unknown (x)) Given context æ², logging policy n. (a/xx) chooses action a², and then the reward , observed from an unknown p (rlæ, a). Following this process, we observe logged bandit data denoted by D == ((at, at (f)) ~ p(D*)) = IT> (x:) (a (pp (ra [2; az) for each domain, where n is the sample size of domain k. Note that we sometimes use D U D* to denote available logged data across all domains.
We are now interested in evaluating a new policy T<a)z) (z) in the target domain whose policy value is defined using the target distributions, (p" (z), (rla, a)). follows.
V (T) E," (1))((l)))) (r(taa) [r] (s) a)),
where qexgpT((x)))) (z := (riz.a)) [r|a,, a] denotes the expected reward in the target domain given context and action Note here that we can regard the conventional formulation described in Section Das
a special case where there exists only the target domain. Thus, IPS and DR can readily defined in our formulation by using only the target domain data D as VIPs( D")) and (T; DV" q.").
In contrast, a possible approach to perform the existing estimators with source domain data is to naively integrate the datasets from all domains when performing IPS or DR as below.
1
(T;;
(T:
w (a) D.
(6)
N
1=
K
1
VTR-ALL(T;:D))
(
wh
at af) ((a))aa)) + Enja 10(,
(7)
N
4

---

Published as a conference paper at ICLR 2025
where w" a) =(alz)/=( and N := = n². IPS-ALL and DR-ALL use information
explored in source domains to estimate the policy value in the target domain with smaller variance by using more data. However, they completely ignore the differences in DGPs across different domains, possibly introducing substantial bias as we will demonstrate in our experiments.
3.1 THE COPE ESTIMATOR
This section proposes a new estimator called COPE, which effectively integrates logged bandit data from target and source domains to estimate the value of new policies in the target domain even with less data and no exploration within the target domain. The key to deriving our estimator how to pool information from source domains to deal with the aforementioned challenging scenarios while not introducing much bias by taking into account the differences in DGPs. We achieve this by the following decomposition of the expected reward function.
L (x, a)
g(x, a, ((()))
+ Mx,a,K
domain cluster fffcct
domain domain--pcccfic effect
where (k) is a function to cluster similar domains. We can, for example, obtain clustering func- tion by heuristically performing off-the-shelff clustering algorithm based on the empirical average of the rewards of each domain (F := En rs/ng) as the domain embedding, which generally works well in our experiments. The domain-cluster effect g(z, o(k)) in Eq. (8) is factor of the expected reward function that domains in the same cluster have in common, while the domain- specific effect h(v, a, k) is the effect not solely modeled by the domain-cluster effect and thus is dependent on each individual domain k. For example, in medical problem, the cluster effect could capture the shared effect of medical treatment within similar hospitals, while the domain-specific ef- fect models how effective an action is for a particular hospital compared to the average in the cluster. Based on this decomposition, we design new estimator by applying different estimation strategies between the Comain-clussee (g) and specific (h) effects. Specifically, our estimator unbiasedly es- timates the domain-cluster effect by applying the multiple importance weighting technique (Owen 2013 Agarwal et al. 2017) and the domain- specific effect by reward regression as
T(a (
VeOPe (T; DD(T)
(1 (o)t,aa))
(TT
DOLT) (at (x+)
REd(T) 1=1
((a bo da (xf
EAA
where DeT(T represents the logged data of domains belonging the same cluster as the target domain (i.e., (TT), which call the target cluster), and no(T)) = Exeecry n² is the total sample size of logged data within the target cluster (T)). It should also be noted that we use the joint distribution of context and action under the logging policy averaged within the target cluster ((T) in the denominator of the first term of the COPE estimator, which is formally defined as
p (aT))
Σ KEO(T)
n
(a/x),
(10)
(2
which results in what is called multiple importance weighting Owen 2013 Agarwal et al. 2017), i..., 2(a")=")/p<(() (a*12=)) in COPE deal with varying distributions in the target cluster. In our experiments, we estimate the weight based only on observable logged data using techniques from density ratio estimation (Sugiyama et al. 2012. Kanamori et al. 2012). as detailed in Appendix C
Our estimator is expected to substantially outperform the naive application of existing estimators when there is only limited/no exploration and less data in the target domain. First, COPE has a lower bias than IPS and DR by estimating the value of deficient and new actions by using data from source domains that are in the same cluster as the target domain, i.e., (TT). COPE also provides substantial variance reduction in the source domains, while IPS and DR use only the target domain data. Moreover, CÓPE has lower bias than IPS-ALL and DR-ALL because it does not naively integrate data from all domains IPS-AL.I
of few-shot data in the target domain by transferring information from
5

---

Published as a conference paper at ICLR 2025
and DR-ALL, but rather applies multiple importance weighting within the data in the target cluster, with adjustments for the varying context and action distributions across different domains.
In the following, analyze the statistical properties of COPE and discuss its intriguing interpreta- tion as strict generalization of existing estimators.
Condition 3.1 (Common Cluster Support). The logging policy oh is said to have common cluster for all A and x E X.
support for policy x if x(a2)) > 0 = (o(T)
Note that, when Condition 2.1 true, Condition 3.1 is always true, indicating that Condition 3.1 more relaxed. In particular, when the logging policy is deterministic or there are new actions in the target domain, Condition 2.1 never satisfied, while Condition 3.I can still be satisfied when there is some exploration in the target cluster (TT) (not necessarily in the target domain).
Based on this milder support condition, we provide the bias of COPE as below. Theorem 3.1 (Bias of COPE). Under Condition
3.1 COPE has the following bias.
E (1)
K(
nk (xx) T (ala) no(T) p (T(x) (a(o)
A A
- Aholoral/
(I1)
kEA(T)
(k)
(z, a) is an estimation error of the regression model given context and action a for domain k. Note that ExEO(TY 17(k) 1. See Appendix B.2 for the proof.
where A (2, gt, a)
Theorem suggests that the bias is characterized by the difference between the weighted average of the regression error in the target cluster (keerry (kk) & (g a)) a)) and the error in the target domain (A (1; a)). Therefore, if A (1, (e, a) R A 9.9 (x, a) = g² a) - (I, a) x (², a) 4.4
9.9
qT (2x, a) holds for vk ((T) (i.e., q² (x, a) accurately preserves the relative reward differences of actions, (xx a) -4"
4f'(z (I. a). within (T))). the bias of COPE becomes small. This is because COPE already unbiasedly estimates the domain-cluster effect via its multiple importance weighting, and thus it is sufficient for the regression model to estimate only relative reward differences of actions between domains in d(T) and the target domain T' to make the global estimate low-bias. More formally, Theorem 3.1 Iippies the unbiasedness of COPE under the CPC condition.
Condition 3.2 (Conditional Pairwise Correctness; CPC). A regression model (² a) and domain clustering function (k) satisfy conditional pairwise correctness if the following holds true:
(x, - q" [(, = (x, - 9" (x,
for all x E X, a E A, and s.t. & E (T)).
Corollary 3.1. Under Conditions 3.1 and 3.2 (xx). See Appendix B.2 for the proof.
CОРE is unbiased, i.e., Exocr) [COOP((T; De(T))
Condition 3.2 requires that the regression model should only correctly preserve the relative reward difference q (x, a) -q"'x)""""rr a) between domains in ((T)) and the target domain T. Therefore, it does necessitate an accurate estimate of the absolute value of the reward function, which is general estimators such as the direct method (DM). Note that CPC needs only an accurate estimation of the relative reward difference within the target cluster, and thus does not require anything about source domains outside the particular cluster.
requirement for regression-basee
It is important clarify that we do not expect Condition 3.2 to hold in practice. is simply a condi- tion that helps to understand when COPE can be unbiased. This is why we first present Theorem 3. which characterizes the bias of COPE without assuming Condition 3.2. It is also important to note that in challenging cases, such as deterministic logging and the presence of actions, existing estimators, including IPS and DR, produce much bias, but our estimator leverages source domain data and is much more robust the bias arising due those challenges,
will demonstrate.
is also worth mentioning that the size of the target cluster plays a crucial role in deciding the bias- variance tradcoff of COPE. When (o(T) is small, the bias of the estimator becomes small. This because decreasing the cluster size makes CPC milder. In the extreme case where (o(T) = 1, COPE becomes unbiased irrespective of the accuracy of the regression model because CPC requires

---

Published as a conference paper at ICLR 2025
nothing in that case. In contrast, the variance of COPE is likely to increase with small cluster because it leads to higher variation of its importance weights.
Here, we also provide an intriguing interpretation of COPE spectrum of DR and DR-ALL, with the number of domains in the cluster [o(Z)) being lever of the tradcoff. More specifically, in the extreme case with (T) = 1, meaning that there is only the target domain in the cluster, COPE reduces to DR because the averaged joint distribution px(T) (a)zz))) reduces the logging policy of the target domain, i.e., no' (a]z)). On the other hand, when [o(T) = K, meaning that the cluster ((T) contains all K domains, COPE reduces DR-ALL.. As strict generalization of different versions of DR, COPE never performs worse than these existing methods with good (if not perfect) tuning of the cluster size Jo(T)). Note that the cluster size [o(T) is a hyperparameter of COPE, and we can tune it based only on available logged data by using recent advancements in the relevant literature (Su et al. 2020b; Udagawa et al. 2023 Felicioni et al. 2024 Cief al. 2024b).
3.2 EXTENSION TO
OFF-POLICY LEARNING
In addition to the OPE counterpart, we can formulate the problem of learning new policy to opti- mize the expected reward in the target domain as max, V (mT) where 0 E R is the policy param- eter. A typical approach to solving the policy learning problem is the policy-based approach, which updates the policy parameter by iterative gradient ascent as 0.41 f- 0, + V V (Tg). Since the true gradient VoVT(T)) = E,," (=)=a(a|=))) (zz, V. log To [2)] is unknown, we need to estimate it from observable data, which has been done by applying IPS or DR in the existing literature (Su al. 2019: Metelli al., 202T). However, similarly to the case with OPE, under the violation of common support with deterministic logging and new actions, policy gradient estimators based on IPS and DR produce bias. Moreover, in the presence of new actions that become newly available when learning new policy for the target domain, these conventional methods cannot evaluate and choose such new actions at all, even though some of them might have high expected rewards.
To solve these seemingly intractable problems in OPL, we can indeed readily extend COPE as a estimator to learn a new policy that optimizes the policy value of the target domain.
To(a lot) (r+ -(aa V (xt) po(T)
Vø VOPPE- (T Do(T))== (D)T:)
nO(T)
(a/lat)
(EA(T)
n
o (o (o7 T Velogma(a" IxT))
(12)
n2
As already discussed, our policy-gradienn estimator in Eq. 12 particularly uses data from source domains that belong to the target cluster through multiple importance weighting, enabling to even learn the value of actions with little or no previous exploration.
EMPIRICAL ANALYSIS
Dataset. This section empirically demonstrates the advantages of COPE and COPE-PG against existing ideas on a real-world public dataset called KuaiRec (Gao et al. 2022) collected from a recommendation system
video-sharing app. The small matrix of the dataset consists of 1,411 users (denoted as u E U)) 3,327 items, and 4,676,570 interactions, with a density of 99.6%, which enables OPE/L experiments without synthetic reward functions. We randomly select 30 actions that have at least one interaction with all users for our experiments. We use the user features and watch ratio recorded in the original data the context T., and expected reward g( a), respectively.
Setup. To simulate our problem of cross-domain OPE/L, we sample users from a domain-specific exp(a=-p(w) distribution (u) for each domain k., which defined as p²(n) = E.reu exp(a=-q(x))) where q(u) := (1/1-A) EnEA ((2,,00 a) and a is parameter that controls the user distribution in each domain. A domain with a large positive a has a higher density of users with larger q(u). We use different values of 8 for different domains vary the context distributions across domains.
7

---

Published as a conference paper at ICLR 2025
DM(T)
DM(ALL)
IPS(T)
IPS(ALL)
DR(T)
DR(ALL)
COPE
MSE (log scall))
Squared sialee
Variance (lag scale)
33
10
300
03
6.1
030
ratio actions the target domaln
ratio actions target domain
of actions the target domain
Figure 2: MSE(left), Squared Bias(center), and Variance(right) with varying ratios of new actions in the target domain.
MSE (log-scale)
Squared Bias(log-scale)
Variance (log-scale) scale)
30
231
200
numbbr of users with determinstic logging
rumber users with deterministic logging
number of users with determirisiic logging
Figure 3: MSE(left), Squared Bias(center), and Variance(right) with varying numbers of users whose logging policy is deterministic in the target domain.
MSE (log-scale)
Squared Biastlog-scale)
Variance (log-scale)
10
321
10
12
per-action logged data sizes in the target
per-action logged data sizes in the Larget
per-action logged data sizes in the target
Figure 4: MSE(eeft, Squared Bias(center). and Variance(right) with varying (per-action) logged data sizes in the target domain.
We sample an action at based on the domain-specific logging policy = which is defined as below.
exp(8* (g(#u, a) + Mu,a))
T (a xa) == En'es exp(3* (g(In, + m))a)))
(13)
where 8h and 7a.o are sampled from uniform distribution within range [-2.0, 2.0] and [-3.0, 3.0], respectively. In contrast, the new policy æ is defined via the epsilon- serioon-gredyy rule as
x (a x) = (1 - e) [[a argmax (x a')} (1)+++2) t/|A),
(14)
where e E [0, 1] controls the quality of and we set 0.2 as default. We sample the reward n from a normal distribution with mean q( q(zx. a) and standard deviation a = 1. Iterating this procedure n times in each domain generate D with n independent copies of (u', x at ).).
Baselines. We compare COPE with IPS(T), IPS-ALL, DR(T), and DR-ALL where IPS(T) and DR(T) perform off-policy estimation using only the target domain data DT We also include the Direct Method (DM) (TT D" q²")) and DM-ALL VDDM-LLL (T; D, 4)) as baselines, which are rigorously defined in the appendix. We use Random Forest (Breiman) (000) implemented in scikit- leamn (Pedregosa et al. 201T) along with 3-fold cross-fitting (Newey & Robins 2018) to obtain q" (z. a) for DR and DM, and q(z, a) for DR-ALL, DM-ALL, and COPE. In addition, for COPE, we use (o(T) = 4. where we define the target cluster @(T) by the set domains for which the difference in the empirical average of the rewards, |a - is small.
8

---

Published as a conference paper at ICLR 2025
Results: Cross-Domain OPE. The following reports and discusses the MSE, squared bias, and variance of the OPE estimators computed over 200 sets of logged data, each replicated with different seeds. The shaded regions in the MSE plots represent the 95% confidence intervals estimated via bootstrap. Note that we set K = 10 for the number of domains and n* = 100 for the logged data size of each domain as the default experimental parameters.
First, we compare the estimators under varying ratios of new actions in the target domain, R // E {0, 0.2, 0.4, 0.6, 0.8], where U4 :- {a E A mi (a|æ) 0, Vrr in Figure 2 The results indicate that estimators using only logged data from the target domain, particularly IPS(T) and DR(T), pro- duce larger bias as the ratio increases, while estimators that naively use logged data from all domains, such as ALL, DR-ALL, and DM-ALL, exhibit consistently high bias due to their inability to deal with differences in DGPs. In contrast, COPE consistently performs the best without being af- fected by the presence of new actions and producing much bias. Note that the MSEs of IPS and DR do not deteriorate rapidly as the number of new actions increases. This is because, although the bias of IPS and DR increases with new actions, their variance decreases as the number of supported actions decreases (a similar phenomenon is observed in [aito & Joachims (022)).
Next, we compare the estimators under varying numbers of users whose logging policy is deter- ministic. We applied a logging policy that deterministically selects a single action to a subset of the users, while we apply a stochastic logging policy in Eq. (13) to the rest. Figure 3 shows that COPE is robust against an increasing number of users having a deterministic logging policy and maintains small bias even when a deterministic policy is applied to about 70% (= 1000/[44] of all users. On the other hand, IPS(T) and DR(T) exhibit an increasing bias as the number of users with a deterministic logging policy grows, which is consistent with Eq. (3). This is similar to the pattern observed with the increasing ratio of new actions and demonstrates the larger bias reduction by COPE in harder scenarios with more new actions and users with deterministic logging.
We also compare the estimators under varying per-action data sizes (n² /|A in Figure 4 We can see that COPE achieves the most accurate estimation, particularly with smaller logged data sizes in the target data. Specifically, when there is only a single data point per action, i.e., n². /|A| = 1,
COРE outperforms the best baseline (DM) by a substantial margin MSE((oXeTT)
MSE(VCOPE) 6.79). Note also that the best baselines are different for different n²= /|A), while COPE generally performs the best in any per-action data size. This powerful and stable behavior of COPE is due to its variance reduction by using data from multiple domains while accounting for varying DGPs avoid producing bias.
-Dosai OPL. We now compare the estimators in terms of their effectiveness in OPI. when used estimate the policy gradient VoVT(TT) (TT). In the following, we report the target policy values relative to those of the logging policy, computed over 350 sets of logged data, each replicated with different seeds. The default configurations are the same as those in the OPE experiments.
Results:
Cross-
We first compare OPL methods under varying ratios of new actions in the target domain in Fig- 5re (left). The results indicate that COPE-PG generally outperforms other methods without being affected by the presence of new actions. Additionally, Figure 5 (center) illustrates the policy value within new actions, EpT(w)) Eaeuz To (0(a(r)q(z, a)] (Figure (rghh)) reports the normalized version: E (a) Eneuy Ta (a(a)z) a)/ / Eneuy Ta (a(x)))) We observe that methods using only logged data from the target domain, such as Reg-based(T), IPS-PG(T), and DR-PG(T), are not able to gain pol- icy values from new actions, which is reasonable. In contrast, methods that additionally leverage logged data from the source domains, such as IPS-ALL, DR-ALL, and COPE-PG, develop policies that select new actions relatively well, with COPE-PG achieving particularly efficient selection of new actions, even though there is exploration of such actions in the target domain.
We then compare the OPL methods under varying numbers of users with a deterministic logging policy in ( We observed that the policy value of IPS-PG(T) and DR-PG(T) deteriorates as the degree of determinism increases. This is due to the fact that under deterministic logging policies and severe common support violations, they become substantially biased. On the other hand, COPE-PG consistently maintains the best policy value, unaffected by the increasing number of users with deterministic logging policy due to its milder support condition (Condition 3.1).
We also compare the OPL methods under varying per-action training data sizes (n² /|AD in the target domain in Figure As we can observe from the figure, most existing methods show that the policy value becomes higher as the training data size increases, but COPE-PG generally

---

Published as a conference paper at ICLR 2025
Reg-based(T)
Reg-based(ALL)
IPS-PG(T)
IPS-PG/ALL)
DR-PG(T)
DR-PGIALL)
COPE-PG
relative policy value
policy valuee within new actions
noraalzedd policy
10
1.25
1.00
0.95
140
0.
ratio of new actions in the target domain
ratio of new actions in the target domain
radlo of new actions the target domain
Figure 5: Comparison of the (left) the test policy values V2 (normalized by V(To)), (center) the test policy values within new actions, (rigt) the normalized test policy values within new actions, under varying ratios of new actions in the target domain.
relative policy value
relative policy value
relative policy value
1.45
1.40
1335
1.15
1.10
number of users with a deterministic logging
per-action training data sires the target
size of the target cluster
Figure 6: Comparison of the test policy values V (T)) (normalized by V(TT)) of the OPL methods under (Ieft) varying numbers of users whose logging is deterministic, (center) varying (per-action) training data sizes in the target domain, (right) varying sizes of the target cluster.
performs the best. It is particularly impressive to the stable performance of COPE-PG, while no baseline methods generally perform well for varying training data sizes. Specifically, IPS-PG(ALL) and DR-PG(ALL) are the second best, following COPE-PG when n² /| 1, while they perform even worse than IPS-PG(T) and DR-PG(T) when n² /IAI = 30. In contrast, IPS-PG(T) and DR- PG(T) perform similarly COPE-PG when n² /| 30 while they perform substantially worse when n² /|A| = 1. Given the stable performance of COPE-PG, there is no particular reason to prioritize other methods over ours in any n² /| in terms of the OPL effectiveness.
We finally evaluate the effectiveness of COPE-PG with varying sizes of the target cluster [o(T') (a key hyperparameter of COPE-PG) in Figure o (right). Note that the baseline methods are inde- pendent of this parameter, so their results remain flat. The figure shows that the size of the target cluster indeed affects the performance of COPE-PG, with the best performance observed at moder- ate cluster sizes, such 6 and 8. However, it should also be noted that COPE-PG still outperforms the baseline methods even with overly small (((T)) = 2) or large ((x(T) 10) cluster sizes, demonstrating the robustness of our method to potential failures in tuning of the parameter.
5 CONCLUSION AND FUTURE WORK
This paper studied OPE/L in challenging scenarios such with deterministic logging policies, new actions, and few-shot logged data, which existing formulations and methods cannot address. To solve those challenging issues, we formulated the problem of cross-domain OPF/I. and proposed novel estimator and pplicy-gradiem method based on a reward function decomposition to leverage data explored in source domains without introducing much bias. particular, our methods are able to evaluate and optimize new policies even with deterministic logging policies or new actions in the target domain with analyzable bias.
As future work, even though the heuristic domain clustering of using the empirical averaged rewards as domain embeddings worked satisfactorily in our experiments, it would be valuable to develop more principled method to perform clustering of domains where we can possibly apply techniques from relevant work in OPE (Peng et al. 2023 (aachdvva al. 2024, Kiyohara et al. 2024b).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Aman Agarwal, Soumya Basu, Tobias Schnabel, and Thorsten Joachims. Effective evaluation us- ing logged bandit feedback from multiple loggers. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 687-696, 2017.
Leo Breiman. Random forests. Machine learning, 45:5-32, 2001.
Matej Cief, Jacek Golebiowski, Philipp Schmidt, Ziawasch Abedjan, and Artur Bekasov. Learning action embeddings for off-policy evaluation. In European Conference on Information Retrieval, pp. 108-122. Springer, 2024a.
Matej Cief, Michal Kompan, and Branislav Kveton. Cross-validated off-policy evaluation. arXiv preprint arXiv:2405. 15332, 2024b.
Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. Doubly robust policy evaluation and optimization. 2014.
Nicolo Felicioni, Maurizio Ferrari Dacrema, Marcello Restelli, and Paolo Cremonesi. Off-policy evaluation with deficient support using side information. Advances in Neural Information Pro- cessing Systems, 35, 2022.
Nicolò Felicioni, Michael Benigni, and Maurizio Ferrari Dacrema. Autoope: Automated off-policy estimator selection. arXiv preprint arXiv:2406. /8022, 2024.
Germano Gabbianelli, Gergely Neu, and Matteo Papini. Importance-weigghee offline learning done right. In Proceedings of the 35th International Conference on Algorithmic Learning Theory, volume 237, pp. 614.634. PMLR, 2024.
Chongming Gao, Shijun Li, Wenqiang Lei. Jiawei Chen, Biao Li, Peng Jiang, Xiangnan He, Jiaxin Mao, and Tat-Seng Chua. Kuairec: A fully-observed dataset and insights for evaluating recom- mender systems. In Proceedings of the 31 ACM International Conference on Information and Knowledge Management, pp. 540-550, 2022.
Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham, and Simon Dollé. Offline testing for recommender systems. Proceedings of the Eleventh ACM International Conference Web Search and Data Mining, pp. 198-206, 2018.
Ben B Hansen. The prognostic analogue of the propensity score. Biometrika, 95(2):481-488 2008
Daniel G Horvitz and Donovan J Thompson. A generalization of sampling without replacement from finite universe. Journal of the American Statistical Association, 47(260):663-633 -655, 1952.
Takafumi Kanamori, Taiji Suzuki, and Masashi Sugiyama. Statistical analysis of kernel-based least-
squares density-ratio estimation. Machine Learning, 86:335-367, 2012.
Haruka Kiyohara, Masatoshi Uehara, Yusuke Narita, Nobuyuki Shimizu, Yasuo Yamamoto, and Yuta Saito. Off-policy evaluation ranking policies under diverse user behavior. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, pp. 1154 1163, 2023.
Haruka Kiyohara, Ren Kishimoto, Kosuke Kawakami, Ken Kobayashi, Kazuhide Nakata, and Yuta Saito. Towards assessing and benchmarking risk-return tradeoff of off-policy evaluation. In The Twelfth International Conference on Learning Representations, 2024a.
Haruka Kiyohara, Masahiro Nomura, and Yuta Saito. Off-policy evaluation slate bandit policies via optimizing abstraction. In Proceedings of the ACM Web Conference 2024, pp. 3150-3161, 2024b.
Jinxin Liu, Ziqi Zhang, Zhenyu Wei, Zifeng Zhuang, Yachen Kang, Sibo Gai, and Donglin Wang. Beyond ood state actions: Supported cross-domain offline reinforcement learning. In Proceedings of the 38th AAAI Conference on Artificial Intelligence, volume 38, pp. 13945-13953, 2024.
Ben London and Ted Sandler. Bayesian counterfactual risk minimization. Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 4125-4133. PMLR, 2019.
11

---

Published as a conference paper at ICLR 2025
Rishabh Mehrotra, James McInerney, Hugues Bouchard, Mounia Lalmas, and Fernando Diaz. To- wards a fair marketplace: Counterfactual evaluation of the trade-off between relevance, fairness & satisfaction in recommendation systems. In Proceedings of the 27th ACM International Con- ference on Information and Knowledge Management, pp. 2243-2251, 2018.
Alberto Maria Metelli, Alessio Russo, and Marcello Restelli. Subgaussian and differentiable impor- tance sampling for off-policy evaluation and learning. Advances in Neurul Information Processing Systems, 34, 2021.
Whitney K Newey and James R Robins. Cross-fitting and fast remainder rates for semiparametric estimation. arXiv preprint arXiv:1801.09138 2018.
Art B Owen. Monte carlo theory, methods and examples, 2013.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Scikit-learn: Machine learning in python. The Journal of Machine Learning Research, 12:2825-288 2011.
Jie Peng, Hao Zou, Jiashuo Liu, Shaoming Li, Yibao Jiang. Jian Pei, and Peng Cui. Offline policy evaluation in large action spaces via outeome-oriented action grouping. In Proceedings of the ACM Web Conference 2023, pp. 1220-1223 1230. ACM, 2023.
Noveen Sachdeva, Yi Su, and Thorsten Joachims. Off-policy bandits with deficient support. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 965-975. ACM, 2020.
Noveen Sachdeva, Lequn Wang, Dawen Liang, Nathan Kallus, and Julian McAuley. Off-policy evaluation for large action spaces via policy convolution. In Proceedings of the ACM on Web Conference 2024, pp. 3576-3585 ACM, 2024.
Yuta Saito and Thorsten Joachims, Counterfactual learning and evaluation for recommender sys- tems: Foundations, implementations, and recent advances. In Proceedings of the 15th ACM Conference on Recommender Systems, pp. 928-830. ACM, 2021.
Yuta Saito and Thorsten Joachims. Off-policy evaluation for large action spaces via embeddings. In Proceedings of the 39th International Conference on Machine Learning, pp. 19089-19122. PMLR, 2022.
Yuta Saito, Takuma Udagawa, Haruka Kiyohara, Kazuki Mogi, Yusuke Narita, and Kei Tateno. Evaluating the robustness of off-policy evaluation. In Proceedings of the 15th ACM Conference on Recommender Systems, pp. 114-123. ACM, 2021.
Yuta Saito, Qingyang Ren, and Thorsten Joachims. Off-policy evaluation for large action spaces via conjunct effect modeling. In Proceedings of the 40th International Conference Machine learning, pp. 29759. PMLR, 2023.
Otmane Sakhi, Imad Aouali, Pierre Alquier, and Nicolas Chopin. Logarithmic smoothing for pes- simistic off off-policy evaluation, selection and learning. arXiv preprint arXiv:2405.14335, 2024.
Yi Su, Lequn Wang, Michele Santacatterina, and Thorsten Joachims. Cab: Continuous adaptive blending for policy evaluation and learning. In Proceedings of the 36th International Conference on Machine Learning, volume 97, pp. 6005-6014, PMLR, 2019.
Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, and Miroslav Dudík. Doubly robust off-policy evaluation with shrinkage. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 9167-9176. PMLR, 2020a.
Yi Su. Pavithra Srinath, and Akshay Krishnamurthy. Adaptive estimator selection for off-policy evaluation. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pp. 9196 9205. PMLR, 2020b.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
12

---

Published as a conference paper at ICLR 2025
Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. In Proceedings of the 32nd International Conference on Machine Learn- ing, volume 37, pp. 814-823, PMLR, 2015a.
Adith Swaminathan and Thorsten Joachims. The self-normalized estimator for counterfactual learn- ing. Advances in Neural Information Processing Systems, 28, 2015b.
Muhammad Faaiz Taufiq, Arnaud Doucet, Rob Cornish, and Jean-Francon Ton. Marginal density ratio for off- offlpoliey evaluation in contextual bandits. Advances in Neural Information Processing Systems, 36, 2023.
Takuma Udagawa, Haruka Kiyohara, Yusuke Narita, Yuta Saito, and Kei Tateno. Policy-adaptive estimator selection for off-policy evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 10025-10033. AAAI Press, 2023.
Masatoshi Uehara, Masahiro Kato, and Shota Yasui. Off-policy evaluation and learning for external validity under a covariate shift. Advances in Neural Information Processing Systems, 33, 2020.
Masatoshi Uehara, Chengchun Shi, and Nathan Kallus. A review of off-policy evaluation in rein- forcement learning. arXiv preprint arXiv:2212.06355. 2022.
Yu-Xiang Wang, Alekh Agarwal, and Miroslav Dudik. Optimal and adaptive off-policy evaluation in contextual bandits. In Proceedings of the 34th International Conference Machine Learning, pp. 3589-3597. PMLR, 2017.
Pengzhou Wu and Kenji Fukumizu. --intact-vae: Identifying and estimating causal effects under limited overlap. arXiv preprint 11x0v:21 2021.
Zhiming Yang. Haining Gao, Dehong Gao, Luwei Yang, Libin Yang, Xiaoyan Cai, Wei Ning, and Guannan Zhang. Mlora: Multi-domain low-rank adaptive network for ctr prediction. In Proceed- ings of the 18th ACM Conference on Recommender Systems, pp. 287-297, 2024.
13

---

Published as a conference paper at ICLR 2025
A RELATED WORK
Off-policy evaluation (OPE) and learning (OPL) have gained particular attention in contextual ban- dit settings as they offer a safe and cost-efficient alternative to online A/B testing (Mehrotra et al.] 2018 Gilotte et al. 2018 Saito et al., 2021 Kiyohara et al., (024a) and are of high practical rel- evance. While several practical estimators and policy gradient methods (most of which are based on importance weighting) already exist (Saito & Joachims) 2021 Uehara al. [022 Dudik al.) 2014] Wang et al. 2017 Su et al. (020a) for each of OPE or OPL, the effectiveness of these meth- ods can decline in challenging but realistic scenarios. First, high bias may occur when the logging policy fails to satisfy the common support condition (Sachdeva et al.) 2020), which becomes even more severe when the logging policy is deterministic or when there works have addressed violations of common support (also known as support deficiency) by exploit- ing additional information about actions, restricting the action space, applying reward extrapolation, or limiting the policy space, but they cannot handle deterministic logging policies or entirely new actions (Felicioni et al., 2022, Saacddvaa et al.] 2020). It may also be helpful to use some structure in the action or reward space to relax the common support requirement, as explored by [Saito & Joachims (2022); Saito et al. (2023); Cief et al. (2024a); (Taufiq et al. al.2)) [Sachdeva et al. (2024): Kiyohara et al. (0024b). For example, [Saito & Joachims (2022) leverage additional information about the actions in the form of action embeddings. and Kiyohara et al. (023) define importance weights in fow-dimensions slate abstraction space. However, this useful structure is not always learnable. In addition, existing estimators and methods based on importance weighting face sig- nificant variance issues, particularly when the logged dataset is small and the action space is large (Saito & Joachims) 2022 Cief et al. 2024a [aachdeva et al. (0224). Weight clipping (Swaminathan) & &Joachims) 2015a Su et al. 2020a) or normalization (S (Swaminathan & Joachims 2015b) might be applied to mitigate variance, but they often introduce a substantial amount of bias in estimation.
new actions. Several recent
Another line of work is OPE/L under covariate shift, as researched by Uehara et al. (2020). This re- search focuses on cases where the context distributions differ between the logged data (p (x)) and the evaluation environment (pttal (x)). while the reward distribution p(r[ær, a) remains unchanged. Under this setup, [eehara et al. (2020) propose doubly robust and efficient estimators by leveraging an estimator of the density ratio between the historical and evaluation data distributions. In con- trast, we aim to estimate the same estimand using data from both target and source domains (which may include multiple source domains), with each domain having unique context distributions (xr) and reward distributions p² (rlæ, a). By leveraging this new setup, our primary goal is to address challenging scenarios in the target domain, such as new actions, deterministic logging policies, and extremely small logged data, which clearly differ from the motivations of Uehara et al. (2020).
Finally, we mention the work by Cief et al. (2024b) and Liu et al. 2024), which formulate the problem of cross-domann offline reinforcement learning (offline RL). The goals of their formulations are somewhat related to ours, as they aim to learn a policy that performs well in the target domain by leveraging data from both target and source domains. However, they focus on the offline RL setup and consider only shifts in transition dynamics. In contrast, we focus on the contextual bandit setup and address varying context and reward distributions across domains. We also analyze and experiment with the problem of OPE, while Cief et al. (2024b) and Liu et al. (2024) focus solely on policy learning. Furthermore, it is important to note that we explore the problem of crosi- OPE/OPL to tackle the unsolved issues of new actions, deterministic logging, and limited logged data, whereas previous work more generally considers the cross-domain setup. Note that we did not compare pessimistic OPL methods (Swaminathan & Joachims) 2015a London & Sandler, 2019 Gabbianelli et al. 2024 Sakhi et al.) 2024) because they are not relevant to our context. Our main motivation is to solve the prevalent problem of (completely) deterministic logging and new actions, issues that pessimistic techniques do not aim to address. However, our proposed method could easily be combined with a pessimistic approach if one wants to do so.
14

---

Published as a conference paper at ICLR 2025
B
OMITTED PROOFS
B.1 PROOF OF THEOREM 3.1
Proof. The bias of COPE under Condition 3.1 is derived as
Bias(VooPP(T;; D)(T)) E[VcorF(TT ;; D)( (T)
(oooetttll polT (a(T((a(aa)))
(r a(zt,a))
(o(T)
kEO(T)
i=1
12"
B (ela) $60 a²)]
n
n
(aa
E B (o)a((aa))) n²(T)
po(T) (ab (rr 0(* aa))
KEO(T)
+ (1))
x(al="))(x",
aEA
- (apr (2,
n
Σ AEO(T)
no(T) E (orn
Ak (x, a)]
polt)
aEA
Σ aEA'
(c)m(())))
(a())
n
Σ E Σ EKEAT) a² A (2) noury (1)6((aa)))
EE KEO(T) aE
TT -E (1))(())))))))))))))))
ZEA'
n² kEO(T) aEd EkEo(T) (x) (a) noाry (x)) (1))
T(a22)
(x)( Σ E Σ
TEX
Σ a
(
n kEO(T) ELEOCTy nhi (a)a)(()))) norry (1
Σ
PTT(a (x) Σ [(a|x)(( Σ
ZEN
aEA
(x,
Σ
no(T) (x() po(t) (alx) A A (I, a)) - -Aiolesa)
n (xx) & (alz))
= (+)(()((aaa))
kEO(T)
15

---

Published as a conference paper at ICLR 2025
B.2 PROOF OF COROLLARY 3.1
Proof. Starting from the final expression B.1 the bias of COPE under the additional Condition 3.2 calculated
|
n p (x) म (a|x)
BpT (1))(((a)))
4 4 (
- a)
no(T) p² (x) (a(z))
KEOCY)
n (x) (a|xx)
A (1,0)
AT
q. a)
Condition
no(T) p" (x) (a(x)
KEO(T)
A (2,(2, (s)T())(()))))
n* (x) )
-11
no(T) p" (x) (a(x)
kEO(T)
0,
which concludes the proof.
B.3 BIAS OF THE BASELINE METHODS
Here, we can derive the bias of IPS-ALL by calculating its expectation as
K
1
Ep EDIViPS-ALLE (T: D)]
E (a Iw (at at aat)aa (z,a))
N
(or (2) w w (at,,tt) a²)= <" x))
N
ma(a x)
K
N (n..))
Therefore, the bias of IPS-ALL
Σ n²
Bias Bias(Vips(T;D))) D))
E Epr(a)maaanll (z,
We observe that they no longer guaranteed remain unbiased in our setup. This bias becomes significant when the reward distributions across domains differ substantially because the expectation of IPS-ALL characterized by the weighted sum the policy values of all the domains involved. In contrast, the bias becomes zero when the reward distributions across all domains are identical (which no longer cross-domain setup). Based on this derivation and Theorem 3. we can compare the bias of COPE and that of IPS/DR-ALLL as follows:
CОРE performs better when the distributions across domains are substantially different and also when the pairwise estimation by áxx a) is accurate.
IPS/DR-ALL performs better when the distributions across domains similar, i.e., when the cross-domain setup is close to a typical single-domain setup.
When the setup is close to the typical single-domaii scenario, effective estimators such as DR and its extensions already well-known. Therefore, focused more novel and challenging scenarios where the distributions across domains are not necessarily similar, as baseline methods tend to fail in these cases as shown above and in our experiments.
In practice, we do not precisely know which of COPE and baselines are better. This is not the case only for our setup, but also for the other OPE problems. A typical example is that never know
16

---

Published as a conference paper at ICLR 2025
DM(T)
DM(ALL)
IPS(ALLI
DR(T]
DR(ALL)
MSE (log-scale)
Squared Biastlog-scale)
Variance (log-scale)
147
20
size of the target cluster
size the target cluster
size of the target cluster
Figure 7: MSE(left), Squared Bias(center), and Variance(right) with with varying size of the target cluster.
which of IPS and DM is better, because the comparison depends many (potentially unknown) parameters such as the reward noise and logged data size. This is why there exists an orthogonal line of work around estimator selection in OPE (Su et al.] [020b] [dagawa et al. 2023 Cief al. 2024b), and we can use those methods to identify the appropriate estimator tailored to the specifiic problem when we use OPE general.
C DENSITY RATIO ESTIMATION
Our implementation in the experiments relies on one of the most standard methods, uncon- strained Saast-Suuarss Importance Fitting (ul UIF), perform density ratio estimation proposed in (Kanamori et al. 2012; Sugiyama et al. 2012). The method considers minimizing the following squared error between and where is estimator for the ratio and w<x) p (x)/p" (x)) the ratio we need to estimate in order use p0(2 (x, a) in Eq. 10
[ (2) - w()))² = (m)(((())) 2 + (e)(((((()))
The first term of the equation does not affect the result of the optimization and we can ignore i.e., the density ratio is estimated through the following minimization problem:
arg min SES
- (B)T()(((()))) 2
To solve the empirical version of this optimization, Kanamori al. (2012) kernel-based hypotheses to estimate the density ratio nonparametrically with a regularization term as usual. https://gihbbb.co/ooxxoo////
-py) which we relied in our experiments, is one of the well- known public implementations of the method.
D ADDITIONAL EXPERIMENT SETUP AND RESULTS
D.1 ADDITIONAL RESULTS ON WORLD DATA
This section compares the OPE estimators under varying sizes of the target cluster [o(T) E (2, 4, 6, 8, 10) (a key hyperparameter of COPE) Figure 7 Note that the baseline estimators are independent of this parameter, so their results remain constant. We observe that COPE, when using a smaller target cluster size, has a small bias but a large variance. This is because the smaller target cluster size reduces the bias caused by differences in the distribution across domains. In contrast, the smaller size also decreases the amount of available logged data, leading to an increase variance. Additionally, the figure suggests that COPE with larger target cluster size exhibits higher variance. This occurs because as the number of domains included in the COPE estimation increases, larger variances more likely arise from the density ratio terms. Nevertheless, the results show that COPE outperforms the baseline estimators, even with small ([o(T) = 2) or large ([o(T) = 10) cluster sizes, as seen in the OPL. experiment.
17

---

Published as a conference paper at ICLR 2025
Moreover, we exclude q(u) from the definition of the user distribution p" (u). following ob- servation that this might favor our methods. Instead, we define a linear function of the user feature (u) In. where the coefficient vector 3 is sampled from normal distribution separately for each domain k. We then define the user distribution p² (u) for each domain as (x))
(") == E.'eu (w']]" This procedure follows the multi-domain recommendation literature, which uses user features to define the domains in their experiments (Yang et al. 2024). More im- portantly, this approach avoids any dependence of the distribution on q(u). We conducted both OPE and OPL experiments based on this new user distribution, and the results are summarized in Ta- bles to 4 The results indicate that our method mostly outperforms the baseline methods that use only the target domain data all the data across target and source domains in both OPE and OPI. experiments (bold fonts indicate the best method, if the best and second best are not significantly different, we use bold fonts for both), even with p (u) independent of ((u).
Furthermore, we can extend the definition of the user distribution as:
exp(Aς(u) + +1 (1- pm(u (u) Ew'eu Eweu (1-
where À controls the degree to which the user distribution depends on q(u). When À = 1, the setup reduces to our original experiment. Table [ shows the result of the OPL experiment for vary- ing À values. We should mention here that the difference between the proposed method and the baselines decreases when À is small, which is reasonable. These additional observations, however, demonstrate the robustness of COPE to varying user distributions, even with a simple clustering procedure. Nevertheless, we believe a more principled and effective approach to domain clustering for our method likely exists. As mentioned in Section 5, we consider the development of a clustering algorithm tailored to Cross-Domain OPE and OPL to be a valuable direction for future research.
D.2 SYNTHETIC EXPERIMENT
This section empirically demonstrates the advantages of COPE over existing approaches using syn- thetic data. The benefits of using synthetic data include the ability to accurately compare estimators based on ground-truth policy values and the flexibility to control experiment configurations in order to test a wide range of scenarios. To generate the synthetic data, we first randomly cluster a total of K = 30 domains, corresponding to the definition of (kk). We then sample a 5-dimensional domain embedding c² for each domain k from the standard normal distribution. We also sample 10- dimensional context vectors x² from normal distribution with mean μ4 and a standard deviation of a₂ 1. The mean parameter μl of the context distribution is sampled from a uniform distribution within the range [-1.0, 1.0].
We synthesize the expected reward function by using domain embedding e as an input as below.
q( a, X) >g(a, a, c) + h(,, a. e),
(15)
where À is an experiment parameter that controls the influence of the domain cluster effect (g) com- pared to the domain- speaciic effect (h), where we use À = 0.5 as the default parameter. Specifically, used the following functions g(.,.-) (domain-cluster effect) and h(),,,)) (domain-specific effect), respectively.
g(z, a, c) == TM one TT,.x + one + 9- one, h(I, a)r,a,e) == -_e e x. MM,ce + one hot Maree,
where ooe_hot, is the one-hot encoding of the action and hot is the one-hot encoding of the domain-clusted c. M Og.c: fa,e, 6g are parameter matrices or vectors in domain-cluster effect function sampled from a uniform distribution with range [-1, separately for each given domain cluster c. Also, Mz,e, Mae parameter matrices in domain-speciff effect function sampled from a uniform distribution with range 1-1, 1), and 0. is parameter vector sampled from a uniform distri- bution with range |-10, 10] separately for each given domain cluster
Based on the expected reward function, we sample the reward y* from a normal distribution with mean q(z, a, e; X) and standard deviation o, =
18

---

Published as a conference paper at ICLR 2025
Table I: MSE in OPE for varying ratios of new actions in the target domain
ratios of new actions
0.0
0.2
0.4
0.6
0.8
COPE(Ours) 1.1641
1.1647
0.8066
1.0663
1.0801
DR(Target Domain)
4.3283
4.2225
2.3985
2.9498
3.7964
DR(ALL Domain)
2.7582
2.2401
3.5354
3.3172
1.4530
IPSrTarget Domain)
5.6442
4.9797
3.6800
4.8295
6.5302
IPS(ALL Domain) 2.1763
2.2538
2.2006
2.1881
2.9360
Table 2: MSE in OPE for varying numbers of users with deterministic logging in the target domain
numbers of users with deterministic logging
200
400
600
800
1000
COPE(Ours)
1.3238
1.3120
1.2926
1.2125
1.2172
DR(Target Domain)
2.3200
2.8398
3.2755
3.7992
4.4499
DR(ALL Domain)
2.5240
2.7915
3.0358
2.9843
3.3314
IPSgTa Domain)
3.2026
3.9641
4.6112
5.6947
7.0049
IPS(ALL Domain)
2.1561
2.1630
2.1906
2.2188
2.2076
Table 3: Test (Relative) Policy Value in OPL for varying ratios of new actions in the target domain
ratios of new actions
0.0
0.2
0.4
0.6
0.8
COPE-PG(Ours) 1.4463
1.4060
1.4389
1.3884
1.4090
DR-PG(Target Domain)
1.3518
1.3220
1.3364
1.2663
1,3339
DR-PG(ALL Domain)
1,4308
1.3947
1,4064
1.3420
1.4403
IPS-PG(Target Domain)
1.3119
1,2755
1.2899
1.2389
1.3350
IPS-PG(ALL Domain) 1.3588
1.3198
1.3267
1.3134
1.3246
Table 4: Test (Relative) Policy Value in OPL for varying numbers users with deterministic logging in the target domain
numbers of users with deterministic logging
200
400
600
800
1000
COPE-PG(Ours)
1.4408
1.4531
1.4769
1.4669
1.4139
DR-PG(ALL Domain)
1.4477
1.4180
1.3954
1,4059
1.5115
DR-PG(TTrget Domain)
1.3101
1.3014
1.3091
1.3596
1.3013
IPS-PG(ALL Domain)
1.3381
1.3003
1.3149
1.3625
1.4651
IPS-PGGTarget Domain)
1.2795
1.2770
1.2905
1.2765
1.2467
Table 5: Test (Relative) Policy Value in OPL for varying À values
lambda values
0.0
0.2
0.4
0.6
0.8
COPE-PG(Ours) 1.4713
1.4646
1.4739
1.4502
1.4780
1.4404
DR-PGGALL Domain)
1.3812
1.3558
1.3475
1.3592
1.3042
1.3657
DR-PG(Target Domain)
1.4639
1.4334
1.3823
1.4374
1.3956
1.2799
IPS-PG(ALL Domain)
1,3553
1.3010
1.3133
1.2869
1.2770
1.2755
IPS-PG(Target Domain) 1.3629
1.3580
1.3572
1.3480
1.4390
1.3889
We define the logging policy T for each domain & by applying softmax to the expected reward function q(z, a, as
exp(s* (g(2, a, A) nr.a)
(a|r) EveA exp(3* (q(x, a', e, + +2,a)
(16)
where 8* and 7z.a are sampled from uniform distribution within range (-0.5,0.5) In contrast, the evaluation policy æ is defined as
क (a(#:) (1 e) I[a argmax g(x, a', e; A)) +
(17)
A
where [0, 1] controls the quality of and 0.2 default.
19

---

Published as a conference paper at ICLR 2025
D.2.1 BASELINE ESTIMATORS
Below, define the baseline estimators compared in our (both synthetic and real-world) experi- ments.
Direct Method (DM). The DM estimator, which uses only logged data from the target domain, is defined as follows:
n Σ T2
(T; D (T)
(ala) ( (Iz, a)l,
where s/ (x, a) estimates o² (x, a) based on logged bandit data from the target domain. Additionally, the DM-ALL estimator, which uses logged data from both the target and source domains, is defined as follows:
K
n
VDMMALL (T; D,
E
[(a a)l,
N
where ((x, a) estimates (xx, a) based on all domains logged bandit data.
Inverse Propensity Score (IPS). The IPS estimator, which uses only logged data from the target domain, is defined as follows:
1 Σ To (a,12z)) T3
1 71 Σ " (I:,
VIPS(T; D²)
where w² (x, == T(a|z)/ mō (a/x) is importance weight of the target domain. The IPS-ALL estimator, which uses logged data from both the target and source domains, is defined as follows:
<aalatt) To (a+
1
AIPS-AL =
where wn (x, 7 (a/x)// (a(()) is importance weight of each domain and N == Σ n².
Doubly Robust (DR). The DR estimator, which uses only logged data from the target domain, is defined as follows:
(D (T Σ fw(xs, - g" (x az)) + Es(a m.) m.)la"
(z; a)]}-
The DR-ALL estimator, which uses logged data from both the target and source domains, is defined follows:
Q.
(a af) - (( aff)) + B (a
N
D.2.2 EXPERIMENT RESULTS ON SYNTHETIC DATA
In the following, we report and discuss the MSE, squared bias, and variance of the estimators com- puted over 150 sets of logged data, each replicated with different seeds. Note that the default pa- rameters in our synthetic validation are |A| = 20, K = 30, n* 100, and | = 9.
First, we compare the estimators under varying ratios of new actions in the target domain, I/ E {0, 0.2, 0.4, 0.6, 0.8}. where 266" := E A (a 0, Var (see Figure g). The results show that COPE consistently outperforms the other estimators without being affected by the presence of new actions. As in the real-world data experiment, the estimators using only logged data from the target domain, particularly IPS(T) and DR(T), produce large bias as the ratio in- creases. Estimators that naively logged data from all domains, such IPS(ALL), DR(ALL), and DM(ALL), exhibit consistently high bias due to their inability to handle differences DGPs.
Second, compare the estimators under varying percentages of samples with deterministic log- ging, as shown Figure 9 the experiment, we calculate the percentages (20%. 40%, 60%, 80%) of particular one-dimensional context in the target domain. Then, if the value of the context in the
20

---

Published as a conference paper at ICLR 2025
DM(T)
DM(ALL)
IPS(T)
IPS(ALL)
DR(T]
DR(ALL)
COPE
MSSE (log-scale)
Squared Blastlog-scale)
Variance (log-scale)
100
0.8
344
0.1
0.1
62
641
60
5.0
0.2
14
06
ratio of new actions in the target domain
ratio of new actions target domain
ratio of new actions in the target domain
Figure 8: MSF(eeft) Squared Bias(center), and Variance(right) with varying ratios of new actions in the target domain.
MSE (log-scale)
Squared Blas(log-scale)
Variance(log scole)
13
of semples with deterministic logging
% of samples with deterministic logging
16 samples with deterministic logging
Figure 9: MSE(left), Squared Bias(center), and Variance(right) with varying percentages of sam- ples with deterministic logging.
MSE (log-scale)
Squared Blas (log-scale)
Varlance (log-scale)
32
10"1
32
220 125 250 275
100 125
175
200
320
150
logged data size the target domainn
logged data size in the target domain
logged data size the target domaln
Figure 10: MSE(left), Squared Bias(center), and Variance(right) with varying logged data sizes in the target domain.
target logged data is smaller than the percentile point, we apply deterministic logging policy; oth- erwise, we apply a probabilistic logging policy defined by Eq. (16). The results indicate that COPE is robust against increasing degree of deterministic logging policy and maintains small bias. In contrast, IPS(T) and DR(T) show increasing bias as the percentile of the target domain context grows, similar to the real-world OPE experiment with varying numbers of users whose logging is deterministic in the target domain.
Finally, we compare the estimators under varying logged data sizes of the target domain (n²) in Figure I0 We observe that most existing estimators show a decrease in MSE as the logged data size in the target domain increases, but COPE generally achieves the most accurate estimation. Notably, when n 50, COРE outperforms the best baseline DM(T) and the second-ber -eet bascline DR(T) by MSE( ASE((DD((T) DR(T)) CSP(VC 3.29).
substantial margin VSE(V DM(T) 2.46,
MSE(YCOPP)
21


Reviews:
[{"review_id": "dT40ACxrG5", "reviewer": "Reviewer_NVrf", "summary": "Cross-Domain Off-Policy Evaluation and Learning (OPE/L) addresses challenges in contextual bandits, such as few-shot data, deterministic logging, and new actions, by leveraging logged data from multiple domains to evaluate and learn new policies more effectively. This paper introduces a new estimator and policy gradient method that improves OPE/L performance, both theoretical analysis and experimental results are presented.", "questions": "Please refer to the above Weaknesses section.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper studies an interesting and practical problem, with both theoretical analysis and experimental results.\n2. This paper is mostly clearly written, and most parts are easy to understand.\n3. A suite of experimental results are provided after the theoretical analysis.", "weaknesses": "1. I feel the theoretical results are a little bit weak:\n\n1a I am curious about the condition 3.2, is this a common and reasonable assumption? It is better to provide more insightful comments on when this assumption will hold in practice.\n\n1b It is hard for me to understand the paragraph in line 314-320. Why is the estimator unbiased when its cluster size is equal to 1? Intuitively, more clusters in the target cluster $\\phi(T)$ mean more logged data come from the same cluster as the new data in the target domain, which will help reduce the bias? Please expand the discussion on this paragraph.\n\n1c I also feel curious about using the off-the-shelf clustering algorithm based on empirical average. How does it work in your theoretical analysis? If the rewards are not very symmetrically distributed, i.e. heavy-tailed distributed, then the empirical mean will lead to terrible estimation, so how to deal with this issue in practice?\n\n1d is there a final regret bound in this paper in terms of the magnitude of $T$ like other bandit works. e.g. $O(\\sqrt{T})$\n\n2. Since this paper studies a very niche area of contextual bandits, it is better to give more real-world applications to help readers understand its significance.\n\n3. It is better to report the running time of the proposed algorithm along with baselines to validate the efficiency of the proposed method.\n\nRemark: I checked the codes in the supplementary material and most notes are written in non-English (I guess it is Japanese). Please use English for everything in your source codes, since other languages may lead to violation of double-blind reviewing policy.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tLEzT7uVbH", "reviewer": "Reviewer_KgjZ", "summary": "This work studies off-policy estimation and learning (OPE/OPL) for the cross-domain scenario, where logged bandit feedback for different domains is leveraged to improve the offline estimation and learning for policies on a target domain. The proposed estimator COPE relies on a reward decomposition inspired by previous work. Its bias and variance were studied for OPE and a straightforward policy gradient approach was derived for OPL. Experiments show that the estimator performs well on cross-domain scenarios with support deficiency in both OPE and OPL", "questions": "- What are the challenges that make the cross-domain application more difficult/particular compared to the large action space scenario? If it is just deterministic policies/deficient support, then it is not different than the case of large action spaces.\n- What is the originality of the bias/variance study of the COPE estimator?\n- How can you be sure that your OPL method will converge to the right policy?\n- Is there a reason why you omit the whole OPL literature based on pessimism?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written and pleasant to read.\n- The experimental section shows favorable results for the proposed solution.", "weaknesses": "- If the cross-domain application in OPE/OPL is novel, the motivation and contribution of the work is oversold:\n   + Hospitals for example rarely share data of patients with centralised entities, and learning on this type of sensitive data is primarily done within the framework of _federated learning_ [3]. The described framework where the learner has access to the data of all hospitals in a centralised manner is far from reality. \n   + The MIPS [1] and OffCEM [2] estimator for example are based on the same reward decomposition, and treat the large action space scenario, where support deficiency happens all the time, as the logging policy cannot cover all the action space. For example, this sentence in the work: \"However, our work differs from these studies in motivation, as we formulate the cross-domain OPE/OPL problem to solve non-trivial challenges such as new actions and deterministic logging\" lacks support, as MIPS and OffCEM already solve these non-trivial challenges. See for example Section 3 of [1] where one of the main motivations is going beyond the common support assumption.\n\n- The derivation technique of the estimator is not novel. The COPE estimator is based on the same technique of [1, 2], with little adjustment  to the new application. Its bias/variance tradeoff is also studied in the same way as [2], which makes the theoretical contribution unoriginal.\n\n\n- The work claims to study OPL, which was not done in [1, 2]. Their proposed OPL method is based on the derivation of a policy gradient, which is basically computing the gradient of the proposed estimator and using it in first order optimisation methods. This contribution is also straightforward and lacks studying in the paper.\n\n- The paper omits a large spectrum of OPE/OPL contributions based on the pessimism principle. This principle was heavily studied for OPL specifically and was proven to be optimal, contrary to optimising the estimator directly. The following lines of work should be incorporated to the related work and discussed, the following papers can be a good start:\n\n   + Bayesian Counterfactual Risk Minimization. Ben London, Ted Sandler Proceedings of the 36th International Conference on Machine Learning, PMLR 97:4125-4133, 2019.\n   + Confident Off-Policy Evaluation and Selection through Self-Normalized Importance Weighting. Ilja Kuzborskij, Claire Vernade, Andras Gyorgy, Csaba Szepesvari Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, PMLR 130:640-648, 2021.\n   + PAC-Bayesian Offline Contextual Bandits With Guarantees. Otmane Sakhi, Pierre Alquier, Nicolas Chopin Proceedings of the 40th International Conference on Machine Learning, PMLR 202:29777-29799, 2023.\n   + Importance-Weighted Offline Learning Done Right. Germano Gabbianelli, Gergely Neu, Matteo Papini Proceedings of The 35th International Conference on Algorithmic Learning Theory, PMLR 237:614-634, 2024.\n   + Logarithmic Smoothing for Pessimistic Off-Policy Evaluation, Selection and Learning. Otmane Sakhi, Imad Aouali, Pierre Alquier, Nicolas Chopin.\n\n\n\n\n\n\n\n\n\n[1] Off-Policy Evaluation for Large Action Spaces via Embeddings. Yuta Saito, Thorsten Joachims.\n\n[2] Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling. Yuta Saito, Qingyang Ren, Thorsten Joachims\n\n[3] Rieke, N., Hancox, J., Li, W. et al. The future of digital health with federated learning. npj Digit. Med. 3, 119 (2020). https://doi.org/10.1038/s41746-020-00323-1", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "890doaCbzT", "reviewer": "Reviewer_8D6L", "summary": "This paper focuses on addressing the problem of OPE/L when the common support assumption is not met. It introduces a new estimator based on the Common Cluster Support assumption and multi-source data to tackle this issue. The proposed approach is validated through semi-simulated experiments.", "questions": "see above", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The challenges of OPE/L with multi-source data and violations of the common support assumption are significant and relevant in practical contexts.\n\n- The proposed estimator aligns well with the motivations of this paper and provides theoretical support for its unbiasedness.", "weaknesses": "- This paper does not compare with significant literature focused on the limited overlap problem. For instance, the assumptions in Condition 3.1 and the subsequent proof of unbiasedness are fundamentally similar to those in [1, 2], which also assume that while the original x may not satisfy the overlap assumption, there exists a score or representation that does. The unique contribution of this paper lies in its application to the cross-domain OPE/L setting.\n\n- However, the absence of a real cross-domain experimental validation, relying instead on semi-simulated experiments, limits the contribution of this work.\n\nIf the authors clearly outline the unique challenges of limited overlap in the context of multi-source data compared to conventional scenarios, and explain the specific efforts made to address these challenges, I would consider increasing my score.\nAdditionally, if the authors provide evaluation results of their method in real-world settings, I would also reconsider my score.\n\n---\n[1] Hansen B B. The prognostic analogue of the propensity score[J]. Biometrika, 2008, 95(2): 481-488.\n\n[2] Wu P, Fukumizu K. $\\beta $-Intact-VAE: Identifying and Estimating Causal Effects under Limited Overlap[J]. arXiv preprint arXiv:2110.05225, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hYWQwQrvng", "reviewer": "Reviewer_A29Z", "summary": "This paper tackles the challenging problem of Cross-Domain Off-Policy Evaluation and Learning (OPE/L) for contextual bandits. It focuses on addressing practical issues frequently encountered in real-world applications, such as deterministic logging policies, few-shot data in the target domain, and the presence of new actions. The authors propose COPE, a novel estimator that leverages data from multiple source domains to improve evaluation and learning in a target domain. COPE decomposes expected rewards into domain-cluster and domain-specific effects, aiming to reduce bias and variance. The paper includes an empirical evaluation on a real-world dataset and provides the code for reproducibility.", "questions": "- Can the authors provide a theoretical comparison of COPE with the existing methods in the setting of deterministic logging policies?\n\n- Can the authors provide a more detailed explanation of the clustering function, its practical implementation, and its impact on the performance of COPE? What guidelines can be offered for choosing appropriate clustering strategies in different domains?\n\n- What are the practical implications and limitations of the theoretical assumptions in real-world scenarios? How can these assumptions be verified in practice?\n\n- Can the authors provide an analysis of the variance of the COPE estimator? How does the variance compare to existing methods?\n\n- What is the bias of COPE when the conditions for unbiasedness are not met? Is it possible to characterize or bound this bias?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses the important problem in OPE/L for contextual bandits. In particular, the setting where there are deterministic logging policies is often overlooked, despite being a scenario highly relevant to practical deployments. This focus significantly enhances the paper's relevance to real-world applications\n\n- The proposed COPE estimator introduces an interesting approach to leveraging cross-domain data, building upon and extending existing ideas in the field. The connections and distinctions with related estimators, such as OffCEM by Saito et al., are noteworthy\n\n- The empirical evaluation is comprehensive, including ablation studies that provide insights into the contributions of different components of the proposed method. The provided code further strengthens the paper's contribution by ensuring reproducibility", "weaknesses": "- While the experimental evaluation on the chosen dataset is comprehensive, I believe the paper would benefit from evaluation on an additional dataset. This would improve the paper, evaluating COPE across different data distributions and characteristics.\n\n- The clarity of the paper could be improved, particularly in explaining the technical details of the COPE estimator, the clustering approach, and the underlying assumptions. More precise definitions and a step-by-step explanation of the methodology would enhance understanding and reproducibility.\n\n- The novelty of the approach is present, however COPE is very similar to OffCEM, as also mentioned in the paper \n\n- The theoretical analysis, which I believe is already interesting, could be strengthened. Specifically, an analysis of the variance of COPE would be valuable. Furthermore, exploring the estimator's bias when the various analyzed conditions are not satisfied would provide a more complete picture of its performance characteristics.\n\n- The theoretical assumptions, such as Conditional Pairwise Correctness (CPC), while providing a foundation for the analysis, might be difficult to verify or guarantee in practice. \n\nMinor:\ntypo at line 081: Cross-domain Off-Policy EvaluatOIn (COPE)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)