Distributionally Robust Policy Evaluation under General Covariate Shift in Contextual Bandits
Authors: Yihong Guo
Hao Liu
Yisong Yue
Anqi Liu
Venue: Accepted by TMLR ()

Abstract: We introduce a distributionally robust approach that enhances the reliability of offline policy evaluation in contextual bandits under general covariate shifts. Our method aims to deliver robust policy evaluation results in the presence of discrepancies in both context and policy distribution between logging and target data. Central to our methodology is the application of robust regression — a distributionally robust technique tailored here to improve the estimation of conditional reward distribution from logging data. Utilizing the reward model obtained from robust regression, we develop a comprehensive suite of policy value estimators, by integrating our reward model into established evaluation frameworks, namely direct methods and doubly robust methods. Through theoretical analysis, we further establish that the proposed policy value estimators offer a finite sample upper bound for the bias, providing a clear advantage over traditional methods, especially when the shift is large. Finally, we designed an extensive range of policy evaluation scenarios, covering diverse magnitudes of shifts and a spectrum of logging and target policies. Our empirical results indicate that our approach significantly outperforms baseline methods, most notably in 90% of the cases under the policy shift-only settings and 72% of the scenarios under the general covariate shift settings.

Published in Transactions on Machine Learning Research (05/2024)
Distributionally Robust Policy Evaluation under General Co- variate Shift in Contextual Bandits
Yihong Guo*
yguo800rjuu edu
Deportment of Computer Science, Johns Hopkins University
Hao Liu'
Milu3@caltcche edu
Department of Compuling and Mathematical Sciences, Caltech Yisong Yue
yyue@caltech. edu
Department of Computing and Mathematical Sciences, Caltech
Anqi Liu
aliu@cs.jhuu edu
Department of Computer Science, Johns Hopkins University
Reviewed on OpenReview: https: // openreview. net/ forum? id= 87PReNNELuw
Abstract
We introduce distributionally robust approach that enhances the reliability of offline policy evaluation in contextual bandits under general covariate shifts. Our method aims to deliver robust policy evaluation results in the presence of discrepancies in both context and policy distribution between logging and target data. Central to our methodology is the application of robust regression - a distributionally robust technique tailored here to improve the estimation of conditional reward distribution from logging data. Utilizing the reward model obtained from robust regression, we develop comprehensive suite of policy value estimators, by integrating our reward model into established evaluation frameworks, namely direct methods and doubly robust methods. Through theoretical analysis, we further establish that the proposed policy value estimators offer finite sample upper bound for the bias, providing a clear advantage over traditional methods, especially when the shift is large. Finally, we designed an extensive range of policy evaluation scenarios, covering diverse magnitudes of shifts and a spectrum of logging and target policies. Our empirical results indicate that our approach significantly outperforms bascline methods, most notably in 90% of the cases under the policy shift-only settings and 72% of the scenarios under the general covariate shift settings.
1 INTRODUCTION
Contextual bandits are online learning algorithms where a policy learner repeatedly observes context, takes an action, and receives a reward only for the selected action (Langford & Zhang 2007). It has wide range of real-world applications, including recommendor systems (Li ct al. 2010 Yue et al.) 2012 Joachims
let al. 2021). online advertising (Tang et al.
2013
Bottou et al. 2013 experimental design Krause & Ong.
2011), and medical interventions (Lei al. 2014
Tewari Murphy 2017). Evaluating policy performance
is essential for these applications, but assessing the value through direct online deployment can be costly and impractical. This motivates the important task of policy evaluation from historical, pre-collected data.
The key challenge in policy evaluation using pre-collected data lies in addressing the counterfactual reasoning that arises from the divergence between the distributions of the logging and target data. Specifically, the actions selected by the target policy may not be covered in the pre-collected logging data, and shifts in the context distribution can further complicate policy evaluation. When the distribution shift only occurs in the policy distribution, we recover the classic setting of off off-policy evaluation (OPE). Existing OPE approaches
"These authors contributed equally to this work and listed alphabetical order.

---

Published in Transactions on Machine Learning Research (05/2024)
can be divided into three main categories: (1) inverse propensity scoring (IPS), which uses importance weights
mdjustment (Horvitz& & Thompson 1952 Swaminathan & Joachims 2015b : (2) direct methods (DM), which
directly regress the value of a target policy by learning a conditional reward model; and (3) doubly robust
(DR) methods, which integrate DM and IPS (Bang & Robins 2005 Dudik et al., 2011 Wang et al. 2017:
Su el. al. 2019a Dudik et al. 2014), achieving asymplotic consistency if either DM or IPS is consistent.
both DM and DR methods, high-quality reward estimation for each context-action pair the target data can be beneficial. However, compared to the rich literature on improving the IPS component (Swaminathan
& Joachims 2015b Wang et al.) 2017 Su et al. (019a), less work has explored how to integrate carefully designed conditional reward model in OPE (Farajtabar et al. 2018).
Beyond the standard setting of policy shift discussed above, very few studies have explored distribution shift in the context variables. [Uehara et al. (2020) suggested incorporating the context distribution into IPS estimation for both TPS and DR methods when context shift occurs and may suffer from similar issues like high variance as traditional OPE methods. However, their approach does not address how a shift in context. distribution might influence reward estimation. Recently, dist rinuuill robust methods have been introduced to enhance the robustness of the evaluation by constructing KL-divergence uncertainty sets to capture potential distribution shifts (Si ct al. 2020 Kallus et al. 2022 in the joint distribution of context, action, and reward. However, the substantially large uncertainty set that characterizes the shift may lead to over-conservativenee in practice.
In this paper, we tackle the problem of policy evaluation through the lens of covariate shift (Shimodaira 2000). Covariate shift refers to the modeling of a dependent variable when the marginal distribution of the covariates differs between training and test time but the conditional output distribution stays the same. Policy evaluation can be interpreted as learning under the covariate shift, where the depen- dent variable is the reward, the covariates are the contexts and actions, and we are interested in learning a robust conditional reward model f(x; a) ~ P(r|æ, a) that performs well under distribution shift. Specifically, we study the scttings where shifts can occur in policy distribution alone (the standard sctting), and both policy and context distributions, which we refer to as Policy Shift (PS) and General Covuriate Shift (GCS), respectively. Building on recent progress in robust regression under covariate shift (Chen et al. 2016 Liu ct al. 2019), we introduce novel family of distributionally robust policy evaluation estimators, which is cre- ated by scamlessly integrating the conditional reward model, obtained from robust regression, into existing DM and DR methods. Our contributions are as follows:
the covariates
We formulate the task of policy evaluation using the framework of covariate shift and incorporate robust regression method into the policy evaluation setting where shifts can happen in both policy and context distribution. Leveraging base distribution, the resulting shift-aware conditional reward model allows for conservative evaluation of policy value in scenarios where the context-action pair in target data is not encompassed by the pre-collected logging data.
We develop direct methods based on the proposed shift-aware conditional reward model, which can be casily integrated into various doubly robust mothods. We establish the finite sample upper bound results for the bias of the proposed method, demonstrating the proposed method's robustness under large distribution shifts, which is further validated in the experiments.
We conduct extensive experiments spanning 360 policy shift conditions and 1260 total conditions for various combinations of logging, target policies, and covariate shifts on data. In our comparison against strong bascline methods on standard benchmark datasets, our method demonstrates superior performance under distribution shifts across various conditions, out purperromiing the baseline methods in over 90% of cases under policy shifts, and over 72% of the instances in general covariate shift settings.
2 Related Work
Off- Pfllcy Evaluation for Contextual Bandits. Classic methods for OPE include direct methods (DM), inverse propensity scoring (IPS), and doubly robust (DR) methods that combine DM and IPS. DM can suffer from significant bias Dudik et al. (0011). IPS, though unbiased, exhibits high variance. DR methods (Dudik Farajtabar et al. 2018) balance the strengths of both. If either TPS
et al.
2011 2014 Wang et al.; 2017
2

---

Published in Transactions on Machine Learning Research (05/2024)
or DM is unbiased, DR methods are guaranteed to be unbiased (Dudik et al. 2011 Much of the follow-up work on DR methods has focused on mitigating the detrimental effects of variance or rxtreme probabilities from the IPS component. For example, SnDR utilizes SnIPS (Swaminathan & Joachims 2015b SWITCH estimator truncates IPS using& a threshold (Wang et al., 2017), and DR with Optimistic Shrinkage (DRoS) finds a weighting stralegy by optimizing a sharp bound on mean squared error (Su et al. 2019a). While much of the prior work has emphasized improving the IPS component of DR estimators and their asymptotical guarantees, our research targets enhancing the robustness of conditional reward estimators, especially under large shifts and limited data. Our key insight is to view this problem as covariate shift problem and utilize robust regression to train the conditional reward estimator.
Covariate Shift and Distributional Robustness. Covariate shift has been tackled through methods such as importance weighting (Shimodaira 2000 Sugiyama et al., 2007) and kernel mean matching (Gretton et al, 2009) in machine learning. In policy evaluation area, [eehara et al. (2000) introduced a. DR-based estimator under covariate shift that uses density ratio estimator between the source and target distributions. Their method focuses on the propensity score weighting formulation, and the IPS component in the estimator introduces high variance when extreme weights exist. In contrast, we tackle the problem orthogonally, utilizing a distributionally robust direct method that can be effortlessly combined with their method.
There are several studies exploring distributionally robust optimization (DRO) (Ben- Tal et al., 2009: Slowik & Bottou 2022 Blanchet et al. 2021) in policy evaluation. SS et al. (2020) and Kallus et al. (2022) augmented evaluation robustness by conservatively estimating reward mapping with KL-divergence uncertainty sets. However, in these methods, the uncertainty set covers all possible shifts in the joint distribution of context, action, and reward, which does not characterize the covariate shift. specifically. As a result, the estimator can be overly robust and less effective. Another line of work utilize DRO for estimating confidence interval in the off-policy setting (Karampatziakis et al.) 2020 Dai et al. 2020 Faury et al.) 2020), following a generalized empirical likelihood approach (Duchi et al. 2021). The differences from our approach lie in the formulation of the DRO problem. Specifically, they do not use the conditional distribution of the target variable (the reward in our case) as the adversarial player, and the uncertainty set constraint is formulated by the perturbation of empirical distribution of the data with distance measured using f-divergence, rather than using feature matching as in our case. Robust regression (Chen et al., 2016 Liu & Ziebart 2017) tailored distributional robustness methods specifically for the setting of covariate shift. It constructs a predictor that approximately matches training data statistics but is herwise the most uncertain on the testing distribution by minimizing the worst-case expected target loss and obtaining a parametric form of the predicted output labels' probability distributions. We are the first to employ robust regression methods for policy evaluation, enabling the refinement of direct met and the further enhancement of doubly robust. estimators.
Detailed Related Work. We provide more detailed review of related work in Appendix A
3 Problem Description
Policy Evaluation for Contextual Bandits. In contextual bandits, a policy π iteratively observes a context æ, takes an action a, and receives a scalar reward ra,a- Assuming that the contexts x are independent and identically distributed (i.idd),, the target object, the value of the target policy, can be expressed as:
VT =
where xc ~ (a)) denotes some known exogenous larget context distribution (e.g., profiles of users), and a ~ <(aaz) denotes the stochasticity of the known target policy. Ususally, we assume access to a pre-collected dataset of n tuples of the form: S = {(xr, a, r)),,, where x ~ P.(x), a ~ ((aaz), and r is the reward observed. Then given S and n, the goal of policy evaluation is to reliably estimate V = V of VT in Equation In traditional OPE, the goal is to estimate VT offline using pre-collected historical data from some other (possibly unknown) logging policy 8(a(z) on the context distribution a ~ (xx), with the assumption that (x) = (z)). When we consider context distribution shift, we further assume (.(x) * P((x). Here, s and t stand for source and target distribution, respectively.
Policy Evaluation as Covariate Shift.
3

---

Published in Transactions on Machine Learning Research (05/2024)
A
Covariate shift is a special case of distribution shift where only the distribution of the covariates changes, but the conditional out- put distribution remains unaltered (Shimodaira) (200). In the setting of this paper, our goal is to evaluate policy π over a po- tentially different context distribution Pt(ar), as depicted in the data-generating process in Figure 1 The assumption can be ex- pressed as (x,,a) a) + a)) and Pa(rla, a) Pa(rla,, a). Note we use "logging" and "source" distribution interchangeably, Based on this data-generating process, the joint distribution of covari- ates (æ and a in this case) can be described as shifting from (a) = (a(x)) ((())) to Pr( a) = (:(a))((aaaa) We can fur- ther assume that the distribution shift problem of policy evalua- tion can be docomposed into two different mechanisms (Schölkopf eet al. 2012 Peters et al. 2017): (1) Context distribution shift, and (2) Policy shift. General covariate shift (GCS) refers to the setting when both context distribution shift and policy shift occur. As the ground truth conditional reward distribution P" P"(r|æ,a) a), that we aim to learn, remains unchanged, we can formulate the reward es- timation problem in policy evaluation as a covariate shift problem. To simplify the notations, We omit x ~ P(æ) sometimes in the and P(.,a,77 P((x, (
Aa
La(a [x)
x
R
P.
P.0)
Data generating process for policy evaluation under general covari- ale shift. X, A, and R represent con- text, action, and reward variables, re- spectively. Logging data distribution Po (=))((azzz) is marked with solid pur- ple lines, and target data distribution ((a))((a)))) is marked with dashed red lines. Black lines indicate reward is generated by both contexts and ac- tions.
Figure 1:
OPE setting. We further define Pa(z,,,r) P. (P (rlza)))) a)
4 Distributionally Robust Policy Evaluation under General Covariate Shift
In this section, we formally present our robust regression approach for reward prediction from logging data in policy evaluation, which is constructed from a distributionally robust learning framework. The high-level goal is to estimate a reward model f(x,, a) that can be robust to the worst-casc data-gencrating distribution that can occur to maximize a loss function on the larget data. We present the formulation and derived predictive form in Section 4.1 and the parameter learning algorithm in Section 4.2)
4.1 The Distributionally Robust Learning Formulation
ERM v.s. DRL. Traditional supervised learning frames reward estimation as an empirical risk minimization problem (ERM). ERM sceks to minimize the following empirical risk on the logging data:
foup(2, a) = arg min (p.a))))) [c( f(z, a))] = arg min |
1
Σ C(r,, f(z;; iES
(2)
where L is a loss function that measures the disparity between the actual reward and prediction from the model. However, this approach is susceptible to bias under covariate shift, as the logging data re- flect reward information solely within the scope of the logging context and policy. To address this, the Distributionally Robust Learning (DRL) framework provides an alternative perspective by mod- eling the problem as a two-player minimax game over the target data, which involves a predictor player minimizing and an adversarial player maximizing the expected target loss as follows: f(x, a)
f(x,a) is the predictor player and ~ g( a) is the adversarial player. The adversarial player is confined to reside in a constrained set Es; defined on the training data S. Covariate shift results in mismatch between the expected loss we are optimizing (defined on Pz(x, a)) and the coistrained set for the adversary (defined on Pa(x, a)). The choice of the loss function L and the constrained set Σs shape the specific derived form of the predictor f(x,a)) Both f and g are valid conditional distributions of r given the context and the action (x, a).
arg f(m.)) maxg(z.0)EEs Pa,,))(( ((rr, where
DRL Formulation for Robust Regression. In this paper, we incorporate the following relative loss function for the reward estimation:
C(r, f) = Em P ((,.)))) [- log + log fo
(3)
4

---

Published in Transactions on Machine Learning Research (05/2024)
Algorithm 1 Stochastic Gradient Descent for Robust Regression under General Covariate Shift
Training data points {( 0z; r:)}, logging policy B(a)æ)), target policy T(alz), source data distri- bution (z)), target data distribution P((x), 0 with initialization, SGD optimizer Opt, learning rate Ir, epoch number T.
Input:
0 < random initialization, epoch 0;
While epoch < T
For each mini-batch
Compute (o a) and oz a) following Equation 5
Compute gradients for following Equation G and Equation 7
Stochastic Gradient descent on 0 using Opt.step(Ir); Output: Trained model parameters O.
where fo is predefined base predictor. This loss function represents the conditional log-loss difference between predictor f and base conditional distribution fo with respect to g on the target data distribution. Given a fixed g. f aims to be as close as to g if possible. If this is unachievable, ʃ will be equal to fo- The motivation for introducing the base distribution fo is that the logging data is not always informative in minimizing the loss function on the target data. Therefore, there should be default choice of prediction when logging data does not cover the target data distribution well enough. To further shape this process, we incorporate the following constrain Eg (used in the definition of f(x, a)) to regulate the adversary g:
Eg = Paa. (r a)
Σ FES
S v, P.a. (rae a)]
1 Σ [
riblaa a) S S
S
SES
(4)
where Φ(x, a) is feature vector and 77 is the slack term. In general, Φ(x,a) a) can be any feature functions. In this work, we set it to be the concatenation of and For simplicity, we use the same slack for the two constraints in Equation 4 This constraint indicates that the expectation of quadratic function in r under the adversary conditional probability distribution g must be close to the empirical expectation (i.e., the average) of that function within the logging data. We choose this quadratic function because it leads to a Gaussian distribution predictor, which is an exponential family distribution with a quadratic form.
Derived Predictive Form. If we also choose the base distribution to be Gaussian as So Nipo, o we have the following form for the predictor f: fo(o,,) (No a), oz( a)) (The full derivation can be found in Appendix 9
og( a) (2W(z, a)O, - a = oz (-2W(x, a a))x((a, +Mo07?)
(5)
where 9_ and 9, are components in the full parameter θ 10., 0w], and Oz is vector with the same dimension as Φ. W(x,a) is the density ratio, which equals to Pilz,a] under general covariate shift, and reduces to [(aa=) in the OPE setting as the context distribution does not shift. The derivation involves using [[a[ø)
(.(m.a)
strong duality to switch the min and max problem and applying the Lagrangian multiplier 0 to convert the constrained problem of Equation 3 and Equation into an unconstrained one.
The Role of the Density Ratio W(x,a) a) and the Base Distribution fo- The density ratio W(z, a) is defined as the ratio of the source density (a a) over the larget density (x a). For cases where the magnitude of (a a) is significantly smaller than Pt(æx, a), the corresponding (x,, a) occurs infrequently in the logging data but frequently in the target data. Under such cases, the estimator will rely more on the base distribution fo rather than the logging data, W(x, a) becomes closer to zero in Equation 5 In other words, the estimator heavily depends on the base distribution when there is a large discrepancy between the logging and target data. On the other hand, if the logging data is well-covered, the estimator becomes more confident. in its predictions, with μi relying heavily on the learned parameters O₂ and 0,. Intuitively, the base distribution will affect the result more when the distribution shift is considerably large.
The Role of Gaussian Predictive Form. We assume a Gaussian base distribution and quadratic feature constraint, which results in a Gaussian conditional target distribution p(r|æ, a). While this Gaussian
5

---

Published in Transactions on Machine Learning Research (05/2024)
assumption is generally valid for real-world examples and widely applied in ML literature, synthetic examples to violate such assumption. For instance, can define the conditional target distribution as an exponential family with cubic terms over the exponential, corresponding to a cubic function feature constraint. In such cases, the standard bias and variance tradeoff analysis applics: using a model with Gaussian assumption this situation leads to an overly simplified model, resulting in high bias and low variance. However, practice, when little is known about the underlying data generation process, the robust regression model under Gaussian assumption is preferred due to its nice implementation properties and broad applicability. Further discussions the Gaussian assumption is provided in Appendix C.1
construct
4.2 The Proposed Algorithm for Parameter Learning
We summarize the proposed robust. regression approach for reward prediction in Figure 2 (a). After deriving the prodictive form, we here present how to learn model parameters 0. With the predictive form of f in Equation 5 we can plug back into the Lagrangian form of the constrained optimization problem Equation and Equation 4 and obtain the target loss be equivalent as maximizing the log-likelihood over the target distribution: (())= = Ep.(.,,.)) log fo ai) Given the form of f, this objective is convex w.r.t 0. By taking the derivative of ((0) w.r.t the 0, and 0z. we can obtain the following gradient form, where Shatch is the set for data mini-batch. Standard gradient-based learning can then be applied for learning the Detailed derivation are Appendix
parameters. The training procedure summarized in Algorithm !
Ve,C(0)
Σ
(ug( a) =z) - -²))
(6)
Shatch
(E
Vo, ((0)
1
Σ
(uo(z,a) r) r)) x.
(7)
SSbtth
(2.a.))EShssch
5
Methods
In this section, we first provide an overview of how our robust regression approach can be integrated into existing policy evaluation methods reward model. A detailed comparison between classical OPE and the setting of policy evaluation under general covariate shift considered in this paper is illustrated in Figure 2 (b). We then establish upper bounds for the bias of these methods. We will refer to our methods using the notation PS and GCS as suffxxes, representing policy shift and general covariate shift, respectively. Specifically, the proposed methods DM-PS, DR-PS, DM-GCS, DR-GCS, SnDR-PS and SnDR-GCS are formulated as follows. We also investigate the integration of our robust regression approach into other OPE methods including SWITCH (Wang et al. 2017) and DRoS (Su et al. 2020) in Appendix E.4
DM-PS. In the OPE setting, the context distribution P(æ=) does not shift, reducing the density ratio p(a(=)
W(z,a) a) to =(a]m)) We onhance standard DMs by incorporating the mean value predictor estimated from robust regression, denoted no (x, a), yielding the "Direct Method with Policy Shift" (DM-PS) estimator.
VOMPS |s| EnEs East [1PS(2z (2,,))]
DR-PS. We apply DM-PS to be the reward model part in DR framework, yielding the "Doubly Robust estimator with Policy Shift" (DR-PS) estimator.
PS ((a)) (1- Me Aaaa))
VDR-PS- [s] E(aar)s
VDM-PS.
(9)
DM-GCS. Under general covariate shift, set W(x, a) to be (a(aa) the standard DM to get "Direct Method with General Covariate Shift" (DM-GCS) estimator. We also plug
[. in robust regression and plug it into
W(x, = (m(aa))) into IPS to implement IPS-GCS as baseline. More details can be found in Appendix B (((a)))
VDM-GGS [s| Exes Ensa [ng
GOS (2,a))
(10)
6

---

Published in Transactions on Machine Learning Research (05/2024)
Regression
Traditional OPE
Policy Evaluation Under GCS
Constroined
8
WiX.A)
IPS
WIX.A) Density Rutio
IPS-GCS
Logging
Density
Target Duta
WWX, A)
Base Distribution
Target
A
Data
K
DR
DR-GCS
Logging Data
Logging
P(R|X.A)
DM
Reward Estimate
Reward Estimate
DM-GCS
Estimator
(a) The Robust Regression Method
(b) Comparison between OPE and the proposed approach
Figure (a): Description of the robust regression method. The density ratio W(ar, a) estimated from the logging data and the target data, The fcature set Σ is obtained from logging data constrain the adversarial player g. The estimator f is estimated by laking minimax loss over target data distribution, with density ratio W(x, a), base distribution fo. and adversarial player 9. (b): Comparision between the setting of classical off-policy evaluation (OPE) and policy evaluation under general covariate shift. There are two types of information derived from the data: (1) density ratio information, obtained from the covariate (context and action) distribution from both logging and larget data and used by the inverse propensity score (IPS) method (green arrow), and (2) pairing information between covariate and reward from the logging data, used by the dircct method (DM) (red arrow) to estimate the reward conditional distribution. Traditional DM relies solely on the logging data, whereas the proposed DM-PS and DM-GCS methods also utilize the density ratio information in the robust regression.
DR-GCS.
We apply DM-GCS to be the reward model part in DR and apply IPS-GCS to the IPS part of DR to obtain the "Doubly Robust estimator with General Covariate Shift" (DR-GCS) estimator.
GCS (m.a)) (r Mg
DR--CCS 1sT E(M.EES
+ VDM-GGS.
(11)
P. (.
SnDR-PS/SnDR-GER Similar to standard DR, following the idea of SnIPS, we can further control the variance of the IPS component and obtain the nelf-normalized DR-PS (SnDR-PS) and ceff-normalized DR- GCS (SNDR-GOS), by normalizing over the sum of the importance weights.
(r Me (x, (a)=(( x(ol
SDDR-PS
+ VDM-PS.
(12)
S
3(a(x) Σ (ir,a,reS 3(a(z))
(a.a.))
(r- ng
GOS (x,, (r
V VDDR-GGS
Σ
DM-GCS.
(13)
Pa a) E E(m,r))s (=,
5.1 Theoretical Results
5.1.1 Bias Analysis
Expected Error. We first establish the upper bound for the robust regression- based reward estimation's 's expected squared error on the target distribution as follows:
Theorem 1. Assuming the density ratio of the target over the source distribution P.(a.a) is upper bounded by a constant C for (z, a) in the source distribution, the base distribution is chosen in the way that pp {o)2] over the larget distribution is upper bounded by conslant H, and source distribution has i..dd. data
Pe(a.u)

---

Published in Transactions on Machine Learning Research (05/2024)
samples. Assuming there are percentage of samples in the target distribution that have non-zero density P.(m.)) ratio W(x, a) P((a,a)) the expected squared error in the taryet data distribution for the robust regression estimalor upper bounded as follows with probability at least 1 6:
E (( [[r ((x, I <m of + 4M)R(F) + 3M2 log + m)
where F is the function class of I with sup s.p,, (z, a)Is M and a Rademacher complerity of R, and in upper bounds the gradient of optimization algorithm when it converges, as shown in Equation 31
Bias Analysis for DM Methods. Based on the analysis of the expected error as above, we obtain the lollowing upper bound for the bias of the proposed direct method estimators, DM-PS/DM-GCS.
Corollary 1. Under the same assumptions as Theorem 1 the bias of the robust regression direct method estimators over the torget data distribution are upper bounded as follows with probability at least 6:
1/2
Br((a.))) I(r ()),
mC
Ei o' (as, + mm 4M5R(F) + 3MM2
leg
+
+1--m)
=e (14)
22
Bias Analysis for DR Methods. We further analyze the bias of the proposed doubly robust cstimators, DCSSS/DR-GCS. We have the following upper bound.
Theorem 2. Under the same assumptions as Theorem 1 the bias of the robust regression doubly robust
estimalors over the larget distribution are upper bounded as follows with probubility al least 1 - 6:
E p. POR-PS/DR-COS
vT] S -m + 2CMMA(F)++ +
log 3
+c.
(15)
The proofs of Theorem I and 2 can be found in Appendix D
Robustness under Large Shift. The above bias analysis shows that the proposed methods remain robust under large distribution shifts. Our bias analysis results consist of two parts, the first term is based on data with a non-zero density ratio where we reweight the target distribution into source distribution and use standard Rademacher complexity. The second term refers to the bounded error of the base distribution for target data that do not share support with the source data. In this case, the m value, which is the percentage of target data points well-supported by the source distribution, represents the magnitude of the distribution shift. When the shift becomes larger, more proportion of the data will have a density ratio close to 0, and becomes smaller, corresponding to cases where the prediction relies more heavily the base predictor fo- Note that traditional ERM methods can have arbitrarily bad performance when there is a severe non-overlapping issue between the source and target data distribution, such as large part of the target distribution is uncovered by the source distribution. Reweighting- the importance weights will have a zero denominator in a subset of data points. Our methods can effectively alleviate the issuc as the bias of the proposed mothods, which shown in Equation [4 and Equation [5] will lean towards utilizing the base predictor fo and have a bounded error as m becomes closer to 0. This makes our method more favorable than other methods under large shift. settings, as further illustrated in the experiments section. In particular, demonstrate how our method performs compared with other methods under different scales of shifting in Section 6.2 The experimental results show that our method remains robust and outperforms other methods, especially when the shift. becomes large, as shown in Figure Besides, we also provide a consistency analysis result for the settings where the density ratio is bounded away from Appendix D.I
methods also become ineffective as
6 Experiments
6.1
Experiment Settings
Datasets. In line with the experimental settings employed in previous studies (Dudik et al. 2014 Wang et al.
2017 Farajtabar et al. 2018 Su et al. 2019b a) we conduct experiments on 9 UCI datasets by
"The code for the experiments available httpe///tt/tt////tttbboooo https://wwchlll.ooo////ooggg///s under- Cnnterr-Ceeeeaal-oooaaattthhhiti Conteeetllnnnt
Robusttouutttooo Policc--llleae Evaluation
8

---

Published in Transactions on Machine Learning Research (05/2024)
transforming the classification problems to the contextual bandits setting. Specifically, the context is set to be the input feature of the dataset, while the policy is a classification model and the action is the predicted class from the model. The reward is 1 if the predicted class matches the ground truth label and 0 otherwise. More details about how the dataset is created can be found in Appendix and the statistics of these
datasets are summarized in Table 1 as follows.
Table 1: Dataset Statistics.
DATASET
GLASS
Ecoli
VEHICLE
YEAST
PAGEBLoK
OPTDIG
SATIMAGE
PENDIGUIS
LETTER
ACTIONS
6
10
10
10
26
FEATURES
9
7
18
103
10
64
36
16
16
INSTANCES
214
336
846
1484
5473
5620
6435
10992
20000
Logging Policies. We conducted experiments using three distinct categories of logging policies, encompass- ing 10 different policies. Specifically, we employed the softened policy, which is in line with previous off-policy evaluation (OPE) work (Farajtabar et al.) 2018 Su ct al.) 2019a) ) as well as Tweak-1 (p) policy and Dirichlet (7) policy, which were inspired by prior research on label shift (Lipton el al.) 2018) and offline contextual bandit optimization (Yang et al. 2023). 1) Softened policy: Softened policy is defined as (((,c)(a)z)) X+ Çu
if aa = (Z)), where u ~ Uniform( -0.5, 0.5) and & is a deterministic policy trained by logistic regression. The remaining probability mass, 1 (u) is evenly distributed among classes a A v0(x). 2) Tweak-1 (p): One class accounts for probability, and the other classes evenly share the remaining 1 - p. 3) Dirichlet (7): A fixed probability distribution generated by Dirichlet distribution parameterized by 7. Compared with the softened policy, the Tweak-1 and Dirichlet policies create more probability values close to 0 or 1 among the arms, resulting in a larger distribution shift between the logging policy and the target policy.
Target Policies. We evaluate our method across various combinations of logging policies and three distinct types of target policies. 1) Softened target policy: We use the softened policy with parameters (A, (0.9,0) 2) Softened perfect policy: We further apply the softened policy techniques to modily a perfect policy, which consists of a 100% accuracy classification model. 3) Softened diverse perfect policy: We set the target policy value for each class as a random permutation of the sequence [A. 2.....1] 1], where n represents the number of classes. This can be viewed as a discrete version of uniform sampling from the interval [0.1].
Covariate Shift on the Context. In our study, we examine two different types of context shifts within the data. We uniformly sample context for the target data, while manipulating the distribution of the context in the logging data in the following two ways to create covariate shift. 1) Gaussian covariate shift: We first apply Principal Component Analysis (PCA) to reduce the dimension. We then define a Gaussian distribution on the first principal component as the new data distribution. 2) Tweak-1 covariate shift (w): Following the logic of the 1 (p) logging policy, we assign specific class of data a higher weight w and all other classes with weight 1, and then do weighted sampling. Detailed information about the policies and data generation process can be found in Appendix E.1
Estimation of the Density Ratio. We consider both the density ratio W known and unknown conditions in our experiments. For W unknown cases, we need to estimate them. Specifically, in policy shift settings, we estimate the logging policy 3 with logistic regression model, where we use the context x in the logging data as features and the corresponding action a as labels. For general covariate shift case, we also need to F. We leverage discriminative classifier to obtain P(eaugesje) using both the logging and the (arget context. For instance, we train a logistic regression model to classify whether the context æ comes from the logging data or the target data. Then, according to Bayes' theorem, we can obtain the P.(m) by Pi(w) Pi(z) (when the source and target data has the same size, i.c.
estimate (a(m)
P(source|a)) ((sourec)) ((targe))
r.(#)
P(target/zz) x) P P(source) P(target)) BBckel et al. 2007
P(x)
P(a)
Basclines. For our experimental comparisons, we consider several bascline methods, including Direct Method (DM), Inverse Propensity Score (IPS), Doubly Robust (DR) (Robins & Rotnitzky 1995 Bang Doubly Robust (SnDR) [Swaminathan & Joachims 2015b), More Robust Doubly Robust Estimator (MRDR) (Farajtabar et al. 2018), and Distribu-
& Robins 2005 Dudik et al. 2011), SnIPS, Self-NNrrmalize
tionally Robust Policy Evaluation (DRPE) (Si et al. (020). We include a variant of our proposed methods
DM (R), shown in Equation 23 baseline by setting the density ratio in the robust regression to 1. More
9

---

Published in Transactions on Machine Learning Research (05/2024)
1.0
1,0
0.8
0.8
0.8
0.8
0.8
0.8
0.6
0.4
0.4
0.4
04
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.2
0.4
0.8 0.8 1.0
1.4
0.0
0.2
0.8
0.8
12 1.4
0.0
0.2 0.4
0.8
0.8
1.0
1.2
1.4
0.0
0.2 044
0.8 0.8 1.0
1.4
(A] Logging pollcy known
(b) Logging policy unknown
Logging policy known
(d) Logging policy unknown
Comparison Betweon Familles
Comparison Between Estimators
SnIPS
DMIR) SHOR(R)
DM-PS
MRDR DRPE
SnIPS
- DM famly
DM(R) familyy
DM-PS family
DM
SrDR: PS
SnDR
CDF of MSE wil.. SNIPS, Policy Shift Experiment
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.0
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.
0.2
0.0
0.0
0.0
0.0
0.2
088
144
0.0 02
08
12
0.0
04
08 1.0
1.4
0.0
122 02
08 088
(e) WIX, known
(1) MX/ a) unknown
(8) WDX, af known
(h) (h) a) unknown
Comparison Between Familles
Comparison Between Estimators
SeIPS-3CS
DM famil
DM-PS family
SrIPS-GCS
DM()
SxDR-P8
MRDR
SnIPS
DM(R) family
DM-GCS famiy
DM
SnDR[R)
DM-GCS
DRPE
SnDR
DM-PS
SxDR-0CS
CDF MSE wirl SNIPS-GCS, Ganeral Coveriste Experiment
Figure 3: First row: Cumulative Distribution Function (CDF) of relative MSE w.r.t SnIPS when only policy shifts exist across 360 conditions. Second row: CDF of relative MSE w.r.t SnIPS- GCS when general covariate shifts exist across 1260 conditions. Subfigures (a)/(e) and (b)/(f) are comparisons between different families of methods, while (c)/(g) and (d)/(h) are comparisons between individual estimators. Better performance is indicated by CDF curves positioned in the top-left corner, signifying a relatively lower MSE. In the first row, among the evaluated methods, the DM-PS family is the best performer in known and unknown logging policy cases. In the second row, in the context of known a), the DM-GCS family performs best, while in scenarios with unknown W(x, a), the DM-PS family prevails.
details about the baselines, including SnIPS, SnDR, IPS-GCS, and DM(R), can be found in Appendix B Ad-
ditionally, we provide the further comparison with other OPE methods in Appendix E.4 including SWITCH
(Wang et al. 2017] and Doubly robust estimator with Shrinkage (DRoS) (Su et al. 2020), demonstrating the advantage of our method when integrated with various kinds of OPE approaches.
To better compare our methods with baselines, we consider the corresponding families. Each DM estimator family comprises DM itself, DR, and SnDR employing the corresponding DM. Fach family contains the same number of estimators for fair comparison. For instance, the DM-PS family includes DM-PS, DR-PS, and SnDR-PS. We select the best-performing method from each family for comparison in our experiments.
Evaluation Metric. We measure the performance of each estimator using Mean Squared Error (MSE), which is defined as E[(V - (())) where V is the value calculated from the cstimator and V(T) is the ground truth value of the larget policy on the larget data distribution, both computed on test set which consists of 25% of the dataset.
6.2 Experiment Results
We present the results for various combinations of logging policies, target policies, and covariate shifts data, encompassing 360 conditions for the policy shift-only experiments and 1260 conditions for the general covariate shift experiments. Following the established setting in previous studies (Wang et al. 2017 Su et al. 2019a), we show the Cumulative Distribution Function (CDF) of relative MSE with respect to SnIPS for the policy shift and CDF of relative MSE with respect to SnIPS-GCS for the general covariate shift as the evaluation metric in Figure 3 Our analysis covers both cases where the logging policy is known (Subfigures (a), (c), (e), and (g)) and unknown (Subfigures (b), (d), (f) and (h)).
10

---

Published in Transactions on Machine Learning Research (05/2024)
Table 2: The number of conditions where each family is the best under general covariate shift. DM-PS and DM-GCS families outperform baseline methods in over 72% of the conditions. When comparing the settings of (a a) known with P. (Z a) unknown, the DM-GCS family outperforms DM-GCS family among known settings while the DM-PS family excels among unknown settings.
SnIPS-GCS
DM
DM(R)
DM-PS
DM-GCS
Pa a) known Pa(a, a) unknown
40
148
93
372
607
60
265
84
690
161
Table 3: The number of conditions where DM-PS or SnDR-PS is the best under different Tweak-1 logging policies. From left to right, the shift becomes larger. This suggests that SnDR-PS's advantage can be weakened by the worsened performance of the IPS component and disappear, as SnDR-PS consists of the DM-PS and the IPS component. Tweak-1 0.91)
Tweak-1 (p- = 0.95)
Tweak-1 1 (p 0.99)
DM-PS
6
8
18
SnDR-PS
30
28
18
From Figure 3 (a) and (b), which compare the performance across different method familics under policy shift, we can see that the DM-PS family consistently outperforms other methods substantially. In Table inn in Appendix E.2 we further present the number of conditions where our proposed method families perform optimally. The DM-PS family emerges as the top performer in both known and unknown logging policies, outperforming other baseline methods in over 90% of the conditions. For general covariate shift settings, when the density ratio W(2, a) is known (Figure 3 (e)), the DM-GCS family exhibits the best performance. When W(x,a) is unknown (Figure 3 (f)). the DM-PS family leads in performance. In both scenarios (Figure (e) and (f)), both DM-GCS and DM-PS families outperform SnIPS- GCS in over 70% of the examined conditions. In (c), (d), (x), and (h), DM-PS and DM-GCS significantly outperform MRDR and DRPE. Further comparisons with advanced OPE methods such as SWITCH and DRoS are provided in Appendix E.4
General Performance Comparison:
Comparison between DM-PS and DM-GCS: When comparing DM-PS and DM-GCS in Figure 3 (g) and (h) well as in Table 2 observe that DM-PS and its family perform better when the density ratio W(2, a) is unknown, while DM-GCS and its family excels when W(x, a) is known. These results can be attributed to that the GCS setting introduces the disparity between (x) and (x(),, creates larger distribution shifts and harder-to-estimate density ratios, and destabilizes the training of DM-GCS and the IPS component within DR methods. We further demonstrate this by comparing SnIPS and SnIPS-GCS in Table 5 in Appendix E.3 Also, the relatively weaker performance of DM-GCS when W(x, a) is unknown can be attributed to the estimation of (2 a), which requires a concurrent estimation of both P, (z) and (()).). The inherent difficulty in accurately estimating (z),, which is typically more complex than estimating the logging policy, negatively affects the performance of the DM-GCS family.
Comparison between DM Methods and DR Methods: In Figure 3 (c) and (d), by comparing the DM- PS with DM(R) and DM, we see that our method shows the best performance, highlighting the advantages of accommodating covariate shifts in reward estimation. We also observe that the SnDR-PS has the best performance. However, in the left corner, we see instances where the DM-PS outperforms the SnDR- PS, indicating that although SnDR-PS generally outperforms the DM-PS, as the shift becomes larger, its advantage weakened by the worsened performance of IPS component. This further demonstratod in Table 3 where we show the number of conditions where DM-PS and SnDR-PS are the best under different Tweak-1 1 logging policies. A larger p represents larger shift. One can see that when the shift becomes larger, the SnDR-PS's advantage becomes smaller. Furthermore, in the general covariate shift setting, we can still observe that DM-PS outperforms SnDR-PS in many conditions shown in Figure 3 (c) and (d), a pallern that is also seen between DM-GCS and SuDR-GCS. This can be explained by the negalive effect on performance when incorporating the unreliable IPS into the SnDR method when the distribution shift is large. Additionally, when the distribution shift of P, (x:) occurs, obtaining a reliable estimation of ((aax) becomes increasingly challenging, consequently affecting the estimation of the density ratio W(z,), a), and harming the performance of SnDR methods.
11

---

Published in Transactions on Machine Learning Research (05/2024)
The Effect of Different Shift Magnitudes: We analyze the performance of DM-PS and DM-GCS under various scales of policy and covariate shifts to show how shifts affect the performance of the DM methods.
1. Policy Shift. Consider two types of softened shift logging policies: small shifts, such as ((..55,0) (a/æ) and ((0.7,0.))a where logging policy deviations from the target policy are modest; and large shifts, represented by T(0.5,0.))(a(;;) and ((0.1.0)aa()) that exhibit more substantial deviations. The Tweak-1 and Dirichlet policies are treated even larger shift logging policies due to more probability values close to 0 or 1. Figure (a.) illustrates the proportion of each DM method achieving the best performance across these conditions, revealing that DM-PS consistently surpasses other methods across diverse logging policy scenarios. DM-PS's advantage grows as the shift enlarges, demonstrating its excellence in accommodating policy shifts in reward estimation and aligning well with our theoretical results in Section 5.1.1
2. General Covariate Shift. Figure 4 (b) showcases the CDF of the relative MSE of DM-GCS with respect DM-PS in Gaussian covariate shift scenarios, with larger w indicating a larger covariate shift. As the covariate shift increases, DM-GCS exhibits widening advantages over DM-PS, indicating the benefits of accommodating covariate shifts in reward estimation.
Collectively, these analyses highlight the superior performance of DM-PS and DM-GCS in handling varying scales of covariate shifts in reward estimation. They consistently exceed other methods, particularly as shifts become more pronounced.
TwiSk 1. 15
Tweak 1. 12
Softened, shift
0.8
Tweak (alak-m..s
Taeee
Softened, large shift
0.6
Treek
Tweak
DM-PS
Tweak-1
0.4
DM(R)
DM
Dirichlet
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
(a) Policy shift
(b) General covariate shift.
Figure 4: (a) The proportion of each DM method achieving the best performance across conditions. As the covariate shift increases, the advantages of DM-PS become more pronounced, signifying the importance of considering covariate shifts reward estimationn (b) The CDF of the relative MSE of DM-GCS with respect DM-PS in Gaussian covariate shift cases. As the Gaussian covariate shifts on context increase, DM-GCS demonstrates more advantages.
7 CONCLUSION AND FUTURE WORK
We introduced a distributionally robust approach for policy evaluation under general covariate shifts in contextual bandits using robust regression. Utilizing this approach, we developed a novel family of policy evaluation methods for not only the policy shift setting but also the general covariate shift setting. We showed how the reward model from such distributionally robust reward estimators can be integrated into direct methods and doubly robust policy estimators. Theoretically, we established the finite sample upper bounds the bias of the proposed methods, providing advantages in large shift. situations. Empirically, conducted extensive studies across wide range of varying distribution shift conditions, demonstrating the superior robustness of our method. Future work includes the investigation of applying the distributional robustness idea to the OPE in reinforcement learning, offline, and safe reinforcement learning.
Acknowledgement
This work is partially supported by the JHU-Amazon AI2AI faculty award, the Discovery Award of the Johns Hopkins University, and seed grant from JHU Institute of Assured Autonomy.
12

---

Published in Transactions on Machine Learning Research (05/2024)
References
Susan Athey. Machine learning and causal inference for policy evaluation. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 5-6. ACM, 2015.
Heejung Bang and James M Robins. Doubly robust estimation in missing data and causal inference models. Biometrics, 61(4):962-973 2005.
Aharon Ben-Tai Tal, Laurent El Ghaoui, and Arkadi Nemirovski. Robust optimization, volume 28. Princeton university press, 2009.
Steffen Bickel, Michael Brückner, and Tobias SchefTer. Discriminative learning for differing training and test distributions. In Proceedings of the 24th international conference Machine learning, pp. 81-88, 2007.
Jose Blanchet, Karthyek Murthy, and Viet Anh Nguyen. Statistical analysis of wasserstein distributionally robust estimators. In Tutorials in Operations Research: Emerying Optimization Methods and Modeling Techniques with Applications, pp. 227-254. INFORMS, 2021.
Léon Bottou, Jonas Peters, Joaquin Quinonero-Candela, Denis X Charles, D Max Chickering, Elon Portugaly, Dipankar Ray, Patrice Simard, and Ed Snelson. Counterfactual reasoning and learning systems: The example of computational advertising. The Journal of Machine Learning Research, 14(1):3207-3260, 2013.
Xiangli Chen, Mathew Monfort, Anqi Liu, and Brian D Ziebart. Robust covariate shift regression. In Arlificial Intelligence and Stalisties, 2016.
Bo Dai, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. Coindice: Off- policy confidence interval estimation. Advances in neural information processing systems, 33:9398 13:9398-9411, 2020.
John C Duchi, Peter W Glynn, and Hongseok Namkoong. Statistics of robust optimization: A generalized empirical likelihood approach. Mathematics of Operations Research, 46(3):946-969, 2021.
Miroslav Dudik, John Langford, and Lihong Li. Doubly robust policy evaluation and learning. In Interna- tional Conference on Machine Learning (ICML), 2011.
Miroslav Dudik, Dumitru Erhan, John Langford, Lihong Li, et al. Doubly robust policy evaluation and optimization. Statistical Science, 29(4):485-51) 511, 2014.
Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavanzadeh. More robust doubly robust off-policy evaluation. In International Conference on Machine Learning (ICML), 2018.
Rizal Fathony, Anqi Liu, Kaiser Asif, and Brian Ziebart. Adversarial multiclass classification: A risk mini- mization perspective. In Advances in Neural Information Processing Systems, pp. 559 567, 2016.
Louis Faury, Ugo Tanielian, Elvis Dohmatob, Elena Smirnova, and Flavian Vasile. Distributionally ro- bust. counterfactual risk minimization. In Proceedings of the AAAI Conference on Arlificial Intelligence, volume 34, pp. 3850-3857, 2020.
Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, Bernhard Schölkopf, et al. Covariate shift by kernel mean matching. Dataset shift in machine learning, 3(4):5, 2009.
Daniel G Horvitz and Donovan J Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American statistical Association, 47(260):6633-6 685, 1952.
Thorsten Joachims, Ben London, Yi Su, Adith Swaminathan, and Lequn Wang. Recommendations as treatments. Al Magazine, 42(3):19 30, 2021.
Fredrik Johansson, Uri Shalit, and David Sontag. Learning representations for counterfactual inference. International conference on machine learning (ICML), 2016.
13

---

Published in Transactions on Machine Learning Research (05/2024)
Nathan Kallus, Xiaojie Mao, Kaiwen Wang, and Zhengyuan Zhou. Doubly robust distributionally robust off-policy evaluation and learning. In International Conference on Machine Learning, pp- PMLR, 2022.
Joseph DY Kang, Joseph Schafer, et al. Demystifying double robustness: A comparison alternative strategies for estimating a population mean from incomplete data. Statistical science, 22(4)5223-539 2007.
Nikos Karampatziakis, John Langford, and Paul Minciro. Empirical likelihood for contextual bandits. Ad- vances in Neural Informalion. Processing Systems, 33:9597-96077 2020.
Andreas Krause and Cheng S Ong. Contextual gaussian process bandit optimization. Newral Information Processing Systems (NeurIPS), 2011.
John Langford and Tong Zhang. The epoch-greedy greedy algorithm for contextual multi-armed bandits. In Neural Information Processing Systems (NeurIPS), 2007.
Huitian Lei, Ambuj Tewari, and Susan Murphy. An actor-critic contextual bandit algorithm for personalized interventions using mobile devices. Newral Informalion Processing Systems (NeurIPS), 2014.
Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. In The Web Conference (WWW), 2010.
Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference machine learning, pp. 3122 3130. PMLR, 2018
Anqi Liu and Brian D Ziebart. Robust covariate shift prediction with general losses and feature views. arXiv preprint xrXiv:1712.1004:3 2017.
Anqi Liu, Guanya Shi, Soon-Jo Chung, Anima Anandkumar, and Yisong Yue. Robust regression for safe exploration in control. arXiv preprint arXiv:1906. 05819, 2019.
Hongscok Namkoong, Ramtin Keramati, Steve Yadlowsky, and Emma Brunskill. Off-policy policy evaluation for sequential decisions under unobserved confounding. arXiv preprint arXiv:2003. 05623, 2020.
Michael Oberst and David Sontag. Counterfactual off-policy evaluation with gunbel-max structural causal models. arXiv preprint 1XX::1995.05824, 2019.
Judea Pearl and Elias Bareinboim. External validity: From do-calculus to transportability across populations. In Probabilistic and causal inference: The works of Judea. Pearl, pp. 451-482. ACM, 2022.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
James M Robins and Andrea Rotnitzky. Semiparametric efficiency in multivariate regression models with missing data. Journal of the American Statistical Association, 90(429):122 122-11992 1995.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On cansal and anticausal learning. arXiv preprint arXiu:1266. 6471, 2012.
Uri Shalit, Fredrik Johansson, and David Sontag. Estimating individual treatment effect: gencralization bounds and algorithms. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 3076-3085 JMLR. org, 2017.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- lig-likeldhood function. Journal of statistical planning and inference, 90(2):227 244, 2000.
Nian Si, Fan Zhang, Zhengyuan Zhou, and Jose Blanchet Distributionally robust policy evaluation and learning in offine contextual bandits. In International Conference on Machine Learning, pp. 8884-8894 PMLRR 2020.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Agnieszka Slowik and Léon Bottou. On distributionally robust optimization and data rebalancing. In International Conference on Artificial Intelligence and Stalistics, pp. 1283-1297. PMI.R, 2022.
Yi Su, Maria Dinakopoulou, Akshay Krishnamurthy, and Miroslav Dudik. Doubly robust off ofi-policy evalu- ation with shrinkage. arXiv preprint arXiu:1907.000 000700962, 2019a.
Yi Su, Lequn Wang, Michele Santacatterina, and Thorsten Joachims. Cab: Continuous adaptive blending for policy evaluation and learning. In International Conference on Machine Learning, pp. 6005 6014. PMLR, 2019b.
Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, and Miroslav Dudik. Doubly robust off-policy evalu- ation with shrinkage. In International Conference on Machine Learning, pp. 9167 9176. PMLR, 2020.
Masashi Sugiyama, Matthias Krauledat, and Klaus-Robert MÅžller. Covariate shift adaptation by impor- lance weighted cross validation. Journal of Machine Learning Research, 8(May):985-100) 1005, 2007.
Adith Swaminathan and Thorsten Joachims. Batch learning from logged bandit leedback through counter- factual risk minimization. Journal of Machine Leorning Research, 16(1):1731-1755) 2015a.
Adith Swaminathan and Thorsten Joachims. The self-normalized estimator for counterfactual learning. Tn Neural Information Processing Systems (NeurIPS), 2015b.
Zhiqiang Tan. A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association, 14166)6):111111117 2006.
Liang Tang, Romer Rosales, Ajit Singh, and Deepak Agarwal. Automatic ad format selection via contextual bandits. In ACM International Conference Information and Knowledge Management (CIKM), 2013.
Ambuj Tewari and Susan A Murphy. From ads to interventions: Contextual bandits in mobile health. Mobile health: sensors, analylic melhods, and applications, pp. 495 517, 2017.
Philip Thomas and Emma Brunskill. Data-cfficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning (ICML), 2016.
Masatoshi Uehara, Masahiro Kato, and Shota Yasui. Off-policy evaluation and learning for external validity under a covariate shift. Advances in Neural Information Processing Systems, 33:49-65 61, 2020.
Joel Veness, Mare Lanctot, and Michael Bowling. Variance reduction in monte-carlo tree search. In Advances Neural Information Processing Systems, pp. 1836-1844, 2011.
Yu-Xiang Wang, Alekh Agarwal, and Miroslav Dudik. Optimal and adaptive off-policy evaluation in con- textual bandits. In International Conference on Machine Learning (ICML), 2017.
Zhouhao Yang, Yihong Guo, Pan Xu, Anqi Liu, and Animashrce Anandkumar. Distributionally robust policy gradient for oflline contextual bandits. In International Conference on Artificial Intelligence and Statistics, pp. 6443-6462. PMLR, 2023.
Yisong Yue, Sue Ann Hong, and Carlos Guestrin. Hierarchical exploration for accelerating contextual bandits. Tn International Conference on Machine Learning (ICML.), 2012.
15

---

Published in Transactions on Machine Learning Research (05/2024)
A Related Work
Off-Policy Evaluation for Contextual Bandits. Classic approaches for OPE (which only handle the policy shift setting) can be summarized into three calegories: direct methods (DM), inverse propensity scoring (IPS), and doubly robust (DR) methods that combine DM and IPS. DMs aim to directly learn the mapping from context and action to reward, but they can suffer from significant bias (Dudik et al. 2011] due to the divergence between the target and logging policies. IPS tackles the problem using the idea of important weighting between policy distribution. While unbiased, they exhibit high variance, especially under large policy shifts, due to unstable estimations of the density ratio (Dudik et al. 2011). A more recently proposed approach is the nelf-normalized IPS estimator (SnIPS) (Swaminathan & Joachims 2015a), which offers improved accuracy over standard IPS when fluctuations in weights dominate that of the rewards.
DR methods (Dudik et al. 2011 2014 Wang et al. 2017 Farajtabar et al. 2018 strike a balance between the biased, low-variance DM and the unbiased, high-variance IPS by incorporating both. It can also be interpreted as using control variates within an IPS framework (Thomas & Brunskill 2016 Veness et al.
2011). If either TPS or DM is unbiased, DR methods are guaranteed to be unbiased (Dudik et al.] [0011] Much of the follow-up work on DR methods has focused on mitigating the detrimental effects of variance extreme probabilities from the IPS component. For example, SnDR, utilizes SnIPS (Swaminathan & Joachims 2015b), SWITCH estimator truncates IPS using a threshold (Wang et al. 2017), and DR with Optimistic Shrinkage (DRoS) finds weighting strategy by optimizing sharp bound on mean squared error (Su et al.] 2019a). While much of the prior work has emphasized improving the IPS component of DR estimators and their asymptotical guarantees, our research targets enhancing the robustness of conditional reward estimators, especially under large shifts and limited data. Our key insight is to view this problem as a covariate shift problem and utilize robust regression train the conditional reward estimator. Our proposed method can also be naturally incorporated into existing methods like DMs and DRs by offering distributionally robust reward estimator.
Covariate Shift and Distributional Robustness. Covariate shift is specific type of distribution shift, and has been extensively studied in the machine learning community (Shimodaira) 2000). Various methods have been developed to mitigate this challenge, including importance weighting (Shimodaira 2000
Sugiyama et al. (007) and kernel mean matching (Gretton al.) 2009). In the context of policy evaluation,
Uehara et al. (020)) introduced a DR-based estimator under covariate shift that uses a density ratio estimator between the source and target distributions. However, their method focuses the propensity score weighting formulation, and the IPS component in the estimator introduces high variance when extreme weights exist. In contrast, we tackle the problem orthogonally in this paper, utilizing a distributional robust direct method that can be effortlessly combined with their method.
Distributionally robust optimization (DRO) (Ben-Tal et al., 2009 Slowik & Bottou 2022 a prevalent approach to handling distribution shifts, introduces the idea of optimizing with respect to the most adverse scenarios over a defined family of distributions. This has led to different variants such as Wasserstein DRO 2021). A few studies have explored distributional robustness in policy evaluation. Si et al.] (2200) and Kallus et al. (2022) augmented evaluation robustness by conservatively estimating reward mapping with KL-divergence uncertainty sets. However, in these methods, the uncertainty set covers all possible shifts in the joint distribution of context, action, and reward, which does not characterize the covariate shift specifically. As a result, the estimator can be overly robust and less effective. Another line of work utilize DRO for estimating confidence interval in the off-policy setting (Karampatziakis et al.] 2020: Dai al., 2020 Faury al.] 2020), following a generalized empirical likelihood approach (Duchi et al, 2021j. The differences from our approach lie in the formulation of the DRO problem. Specifically, they do not use the conditional distribution of the target variable (the reward in case) as the adversarial player, and the uncertainty set constraint is formulated by the perturbation of empirical distribution of the data with distance measured using (-divergence, rather than using feature matching as in our case. Robust regression (Chen et al., 2016 Liu & Ziebart 2017) tailored distributional robustness methods specifically for the sctting of covariate shift. constructs a predictor that approximately matches training data statistics but is otherwise the most uncertain on the testing distribution by minimizing the worst-case expected target loss and obtaining a parametric form of the predicted out labels' probability distributions. We are the
(Blanchet et al.)
16

---

Published in Transactions on Machine Learning Research (05/2024)
first to employ robust regression methods for policy evaluation, enabling the refinement of direct methods and the further enhancement of doubly robust estimators.
Causal Inference. The task explored in this paper has close ties to causal inference (Athey 2015 Joachims et al. 2021). core challenge in evaluating the individual treatment effect (ITE) and the average treat- ment effect. (ATE) causal inference the evaluation of a counterfactual policy. TPS and DR have been investigated for ATE estimation using linear models (Tan 2006 Kang al., (007)) and recent advancements have leveraged methods from domain adaptation and deep representation learning (Johansson al.) 2016). In particular, ITE estimation is challenged by the specific type of distribution shift where the context dis- tribution changes between treatment and control groups (Shalit et al.] 2017). There has also been work on using causal models to enhance off-policy evaluation results (Oberst & Sontag, 2019 Namkoong et al. 2020j. Besides, the setting of covariate shift has been studied in the task randomized controlled trials (RCT) causal inference as an issue of lacking external validity (Pearl & Barcinboim, 2022). However, these works mainly focus on devising new identification strategics of causal estimands. In this work, we focus on developing robust learning methods for counterfactual reasoning from the perspective of covariate shift.
B Description of More Baseline Methods
DM. Given reward predictor f, direct method estimate VDM
VDM = |S|
Σ DES
Eann [f(x, a)].
(16)
IPS. Inverse propensity scoring (IPS) uses important weighting the entries in S to reflect the relative probabilities of choosing some action by the target policy versus the logging policy 8::
VIPS
1
Σ
((aa=)
(17)
S
B(alæ)
(2,0.))S
where 3 is the logging policy (estimated (3 is unknown).
DR. The basic formulation of the doubly robust estimator is:
- f(x, (a))((
VDR = VDM +
Σ
(18)
S
B(a|z)
(n,a,rieS
SnIPS. The self-normalized IPS Swaminathan & Joachims tance weights. follows the form as below:
2015b
normalized by the sum of the impor-
Eina,,) 7(a/m]
VRIPS
3[a s+
(19)
E(ear)es
B(amm)
SnDR. The self-normalized doubly robust estimator normalizes the importance weights following the idea of SnIPS. The formulation of the SnDR is:
VDDDR Einames (o=)
d(ola)
VDM-
(20)
E 3(amm)
IPS-GCS. When general covariate shift exists, the density ratio of IPS general covariate shift" (TPS-GCS) is calculated by:
Pa(ao.)
And the "IPS under
(a
1
P1(2, P,(x,a)'
Vips-Gos
(21)
S
17

---

(R) Published in Transactions on Machine Learning Research (05/2024)
where o)) = (((x))((az)) and (x = (()))(())))).
DM (R). We further obtain the baseline method DM (R) by setting W( = 1 in the robust regression framework. Such a framework does not consider covariate shift information. The solution of DM (R) is
formulated as follows:
(R) Mo (a. a) = =0² a) (- 20 a) + 2o00²).
2( 8 (x, a) (20, =0²
(22)
We then plug the mean estimates from Equation 22 into the direct method and obtain:
VDM (R)
Easo
(40 (10 (x,
(23)
SES
C Derivation of Robust Regression Model
We here show the derivation of the robust regression model. According to Chen et al. 2016 if we have the constraints for the adversary g as follows,
Es
Ez P. (g (r,,
5(r, 5 x. a) Snj
(24)
S
iES
then the solution of the minimax formulation has the parametric form:
(=(x))((aaz) o 5 (T,
aa( So(r|ær, a) exp (- P((x)) P,(x,a) 0" 8(, (1.,,aa) fo(rl a) exp (- P.(xx
(25)
If we set the function ô in a specific way such that it has a quadratic form and assuming the base distribution fo NHo, ou), obtain Gaussian distribution. We provide the necessary details here for the derivation of the mean and variance from this special form and refer more details to Chen et. al. (2016).
If our 8 == vector (Ir, ( a) [r, ((x, a)), it corresponds to the following constraints in Σ:
Σg
P. (r(2,
r
S
n
S
iES
Ba (0en a)]
r a)
Eng
(26)
S
SE
With strong duality and the constraint set for the adversary g in Equation 24 the solution of loss function Equation 3 is equivalent to the solution of the following minimization problem:
min D
such that: E (.. =
where c c= Σ, 8(ri, 8(r,, u:). Let Po be defined as:
fo fo(rlæ, a) exp (-W(x, a² 8(r, 8(r, a))
Z(x, a,
18

---

Published in Transactions on Machine Learning Research (05/2024)
where (xx, a, 8) denote the normalization term. Moreover, let
(r,, æ a) -W a)o" ö(r, æ. a) log Z 0),
and use the A as the Lagrangian parameters, the Lagrangian of the optimization problem is written as:
(()) D n. a. Ma.)) 0" (Br,(maa) x.a)] -c) L
ff P((2))(((()) / a(
a) log( + 0T 11
50
rla.
= P((2))(a())
/ (-
- Pr(2))((a(2r)
f(rlz,, a) log Z(xr, a. 0)
1/2,0
a,a
2,0
(1 (.(x))((a(x))
/
+ TT
x,
æ2a
1/2,0
lo a c²c
If we take the maximum of ((0), we have the following result:
lo Z(, 0Jc
arg max arg max
m,a
(x(x))((ax)) / ()
((0)))(()))1182((00000 0T
arg max
(o 20, 0²
arg max
/ (*(r(,, (188 Z(x,a,0) r|e,a
arg max
Pr(x))((aa))
W(x, a(xaa)0" (),,)), 3(1,
log fo(r
arg max E az
If we use parameters 0 = 10., Om, On. Omn] as the Lagrangian parameters (here Oz is vector with the same dimension Φ can derive the following predictive form for fo(r) (note that Oun is only relevant to Φ(0.0) a) but not we do not need to learn it explicitly):
P2(x, a) TT 8(1,, (e,, So exepp
P.(xx.)) a)
fa(r)æ,a) a) x Soexp
(0, 20,r a) + Omn ()))
P((
1
(r-mop) n0)2
(2( P1(2, a)
V2T00
exp
2
of
exp
(0, + +20,rp(z..)) + Mun
(27)
x exp
20
+
0r
2 + (-2
P,(x,, a) ax)
0n (( a) + //)))) exp
1 (r 12)
2
(28)
0
where
(0 a) = Pa a)
0r 000 -2
(29)
P((x,
(- Pz a) Pr(x, a)
Me(æ, a) = (2.20)) a)
0 1, a) + moop²)
(30)
C.1
Gaussian Predictive Form
Here we provide an example that the data generated from cubic form distribution instead of Gaussian distribution. In this scenario, we adjust the robust regression model by incorporating the cubic function
19

---

Published in Transactions on Machine Learning Research (05/2024)
feature constraint instead of the quadratic constraint. Denote the covariate variable, y as the target variable, fo(y/az) as the prior and ((x, to be the cubic function feature constraint. We can obtain the derived predictive model form of p(ylæ)) x fo(y/z) exp(= y)), where 0 is the model parameter and w(x) is the density ratio. This cstimator has a hard-to-track normalization denominator for the predictive model distribution, so we need lo sample from this distribution to oblain the prediction of y from such a model, which will sacrifice the nice implementation convenience of our robust regression model, and we may need to tailor specific sampling methods to efficiently obtain the prediction from such an estimator.
We conduct experiments under settings where p(ylz) is not Gaussian by generating synthetic data (z, y) such that p(ular) & exp(y² - 2y²x - 3yz - 4y² + 5yr + 6y) following a cubic feature constraint assumption. Specifically, we created the distribution shift by generating x following Ztrain ~ NN,, 1) and Trest ~ (1,, 2). There are 500 data points in both the train and test datasets. Since the cubic feature constraint does not result in Gaussian distribution prodictor model like the quadratic feature setting and the denominator of such distribution is intractable, we use discrete target y by sampling y E {0,1} from the aforementioned prediction ristribution in this experiment, for both the data generation and getting predictions from the robust regression model.
The cubic form robust regression achieved a mean squared error (MSE) of 0.31(#0.0002).. The quadratic form robust regression received an MSE of 0.42(+0.0003), and the linear regression model received an MSE of 0.49(=1.2 x 10-1"). These results are averaged over 10 runs. These results suggest that a more tailored feature set could improve the performance in synthetic settings and using a simpler hypothesis class could induce bias. However, given the original model for regression without Gaussian assumption, p(yla:) x exp(-w(z)f0oxr; v)), can be less convenient for making predictions (which is the reason need to use sampling to get the y in this synthetic experiments) and the broad applicability of Gaussian data assumptions, a robust regression model with Gaussian assumptions is still preferred in practice.
D Proof of Bias Analysis
We prove the bias analysis results of the proposed robust regression estimators under large distributional shift, building upon established results from Liu et al. (2019). We assume the selection of base distribution WN oī), such that the expected squared error over the targel distribution (x,,,,,))) )o) is upper bounded by some constant H. Typically, this condition is achieved in practice by appropriately sctting the base distribution, such as to be the empirical mean of the datasct.
Expected Error.
Our analysis begins with establishing a generalization bound for the expected squared error of the proposed robust regression-basee direct method over the target distribution. We divide the target distribution data into two disjoint subsets based on whether it shares support with the source distribution, which is indicated by the value of its density ratio between the source and target distribution. Specifically, for the target distribution data points (x. with W(x, a) -> O, which means they do not share support with the source distribution, the robust regression estimator is reduced to the base estimator based on the formulation. Consequently, under the aforementioned assumption, the following inequality holds:
E.((a,,))) [(r mbo < H.
For the remaining data points in the target distribution with W(r, a) > 0, we assume that Pr(m.)) P[(a,)) the density ratio of target over the source distribution, is upper bounded by some constant C. Drawing upon Theorem 1 from Liu et al. 2019), we derive the following upper bound for the expected squared error over the target distribution:
log 2
E En(maar) 5
2 (x ((2,0) + 3M² 1=1
2n
20

---

Published in Transactions on Machine Learning Research (05/2024)
where FF is the function class of f with supp,reeF - (2(x,a))5 M, and with Rademacher com- plexity of (FF). Assuming the training process converges satisfactorily, we can upper bound the training gradient and derive the following results:
r² (u(x a)² + (xr S m:
71 r - 1(x a) 1 T
(31)
where is some small constant and is the lower bound of all the features in the context and action pair representation f(x, a). Without loss of generality, we assume that there are n iidd sampled data samples in training and m percentage samples have a non-zero density ratio W(x, a) in the target data distribution. We have the following upper bound for the expected error the target distribution holds with probability at least 1 - 6:
log 2n
Ep..aa ((2r,a)) & mC
2 (x, a4) + + 4MM(FF) + 3M²
+ m)
72
111
Bias Analysis of DM. Building upon our analysis the expected squared error of the proposed direct method using robust regression as above, we establish an upper bound for the bias of the robust regression- based direct method estimator. The following bound for the bias of DM-PS/DM-GCS holds with probability at least 1 - 5:
1/2
n
log 3
Er ((xxa))] < mC
0x) +m + 4MM((F) +M2
+ (1 - m)) H
2n
Bias Analysis of DR. We then analyze the bias of the proposed DR-PS and DR-GOS estimators. Similarly, assuming the training process converges satisfactorily, we have the following lemma by upper bounding the training gradient and using Rademacher Complexity.
The expectation of the In distance belween the reward and the meam estimator from robust regression over the source distribution satisfies the following with probability least 1- 8:
Lemma 1.
log 2 A
E..( [I a( )))
Th + (2MR(F)+ +3M
(32)
2n
where is the lower bound of all the features in the contezt and action pair representation f(z, a).
Given that Equation 32 in Lemmal holds, we establish the following bound for the bias of DR-PS/DR-GCS:
E E - Val s E (p,(ma)) (C(r--((a)) a))l + (P, |r (((x
(33)
< Cm i
log +
+ 2CM5A(F) + 3CMM
(34)
2n
D.1 Consistency
We here analyze the asymptotic convergence of the proposed DM-PS/DM-GCS and DR-PS esti-
mators using the asymptotic performance of robust regression method (Chen et al. 2016 Fathony et al., 2016). We have the following conclusions.
Proposition 1. Assuming (P a)) is bounded away from zero, W(x, a) is accurately estimated in training, & is chosen to be universal kernel, then DM-PS/DM-GCH is consistent.
21

---

Published in Transactions on Machine Learning Research (05/2024)
Remarks. This proposition can be proved by directly applying the consistency results of the distributionally robust learning (Fathony et. al. 2016). Here, the estimator will converge to zero error over the target distribution under the distribution shift. Note that the result is applicable for all loss functions.
We then analyze the asymptotic behavior of DR-PS/DR-GGO estimators.
Proposition 2. If DM-PS/DM-GCS is consistenl, then PC-PS/DD-GG5 is consistend.
Remarks. This proposition is obvious since we formulate the DR-PS/DR-GCS method as doubly robust methods with robust regression-based reward models. It demonstrates that using robust regression as a reward model does not hurt the asymptotic consistency of the doubly robust methods. Despite we need more assumptions for DM-PS/DM-GCS to be consistent, such as accurate W(z, a) and expressive feature representation Φ, it does not mean that DR-PS/DR-GE is weaker than general DR since we use a specific distributionally robust model for the reward model, while traditional DR analysis treats the reward model as black box.
E
Experimental details
We here describe the logging policy, target policy, and covariate mentioned in Section 6. in detail and the specific parameters we used. We also provide experiment results on cach dataset. The code for the experiments can be found al
wariate-Shife Shift-in in Contextual-
E.1 Data Generation
Logging Policies. We further describe our logging policy here. Firstly, we employed the softened policy, which is in line with previous work of OPE (Farattabar et al.] 2018 Su et al.] 2019a). The softened policy is defined by two parameters, denoted as (), and a trained deterministic policy using logistic regression. Additionally, inspired by previous research on label shift (Lipton al.) 2018) and offline contextual bandit optimization (Yang et al., 2023), we generated the other two types of logging policies, Tweak-1 (p) and Dirichlet (y), by creating biased "action label distributions".
Softened policy, (a)x)) Given a deterministic policy & and parameters (A, {), the softened policy is defined as (alar) = À + Çu if a ((x)) and other classes evenly share the 1 (A + +C),, where u ~~ Uniform(-0.5, 0.5). The deterministic policy (xx) is trained by logistic regression on 10% of training data. In our paper, we consider four different softened policies, which are ((oo9s.0)(((2)) ( 7(0.5,0.1(aa/z and (a)x).))())z).
Tweak-1 (p). We allocate a probability of p to one class while the remaining classes equally divide the residual probability of 1-pp. A higher p indicates a larger distribution shift of the logging dataset since more probability values close to or exist when P large. We evaluate this setting with p values of 0.91, 0.95, 0.99.
Dirichlet (7). Dirichlet (7) represents a Dirichlet probability distribution parameterized by A smaller ? implies larger shift of the logging dataset since there are more probability values close to 0 or 1. We generate one logging policy randomly and fix that across all experiments. Our study considers y values of 1.0, 0.5, and 0.1. Note that with Dirichlet (0.1), the probability for some classes becomes low that they may not be sampled. Hence, for Dirichlet (0.1), we utilize logging policy of 0.95 x Dirichlet(O.1) + 0.05 Uniform.
Target Policies. We test. method various combinations of logging policies and target policies. The target policies are follows:
Softened policy, (o.0,,0) (alæ). We first obtain deterministie policy & by training a logistic regression on the entire training set. Then we set (A = 0.9, = 0) and obtain the softened policy.
Softened perfect policy, [pperfeet,)) Given access the ground label when generating the policy, we can have the perfect policy, Eperfect, which means that for every x, the policy always chooses the action
22

---

Published in Transactions on Machine Learning Research (05/2024)
that receives the highest reward. Following the idea of softened policy, we can soften the peerfect with (A, = (2 to obtain "(perfect,A)- Our paper considers X = 0.9, 0.7, 0.5, respectively.
Diverse softened perfect policy. We create a case where the performance of policy evaluation will be substantially affected by the extent of the general covariate shift. As the covariate shift of the data we consider would generally make the target data only focus on a subset of all the classes in a dataset, we make the larget poliey value vary across different classes. Specifically, we set the targel policy value for each class as random permutation of [ 2, ... 1), where is the number of arms. This can be viewed as discrete version of uniform sampling from [0.1].
Covariate Shift on the Data. We consider various context shifts in the experiments. We uniformly sample context for the target data while we consider the following choices for the scale of the distribution shift of the logging data distribution.
Gaussian covariate shift. We first perform PCA on the dataset and select the first component We then define a Gaussian distribution with mean as min(c)+ meen(()) and standard deviation as bb where (0,inf) and b e (0, We then sample logging data based on the generated Gaussian distribution. Smaller b means smaller variance in the generated Gaussian distribution and a larger shift. Also, when a > 1, larger a means larger shift, when < 1, smaller means a small shift. In our experiments, we consider the settings of (a, b) = (1.5, 3), (2, 2), (1.5, 2), (0.6, 2), respectively.
std(e)
1 covariate (w). Following the idea of the Tweak- - (p)) logging policy, we assign one class of data with a higher weight and other classes with weight 1. We can obtain a probability distribution for each data by normalizing the weight to 1. A larger mcans larger shift. Our paper considers the sctting of = 15, 12,9, 6, 2,2 respectively.
When only policy shifts exist, we consider 9 x 10 4 = 360 conditions derived from 9 datasets, 10 logging
policies, and 4 target policies. When general covariate shifts exist, we consider 9 x 7 x 10 x 2 = 1260 conditions derived from 9 datasets, 7 logging policies, 10 covariate shifts, and 2 target policies. Specifically, the 7 logging policy are T(0.95,0.1) (4llz)))) TT..7.0 110/7,..)((//r)) Tweak-1 (p = (p= 0.91), Dirichlet (1.0) and Dirichlet (0.1). The two target policies are TTO..,0) (a|z) and softened perfect policy (( (a)z).
(p = 0.95), Tweak-1
Dataset Generation. Based on the aforementioned settings, the datasct generation and evaluation process is as follows:
1. Data Split: We randomly split. the original dataset into 75% training set Drn and 25% test set Drs.
2. Context and Policy Distribution Generation: We generate logging policy 8 and target policy a given the policy parameters. If under the general covariate shift (GCS) setting, we further generate a logging context distribution Pz(
3. Target Policy Value Calculation: We compute the ground truth of the target policy a's value VT the entire test set Drs, by sample the action a for each context æ in Drs and get the corresponding reward r.
4. Logging Dataset Generation: If under the GCS selting, we sample context æ from the training set. DTR based on the logging context distribution (x). Otherwise, we sample the logging context æ uniformly. We then sample the action a for each context æ using the logging policy 3 and obtain the corresponding reward r create the logging dataset DoR-Logging-
5. Model Training: We train the reward predictors, inchuding DM, DM(R), DM-PS, and DM-GCS, using (n, a)) in the logging dataset Drn.ingging
6. Evaluation: We sample another logging dataset Drs.Logging following the Logging Datasel Generation process in step 4 from the test set Drs for estimating the importance sampling part of the estimators. All methods' final estimated policy value V is calculated on Drs-Logging-
Hyperparameters. The base distribution for robust regression is a Gaussian distribution with mean 0.6 and variance = 1. 0 is updated with SGD. We tune the hyperparameters with grid search the learning
23

---

Published in Transactions on Machine Learning Research (05/2024)
rate in [0.001,0.0005). We also set the learning rate decay for the learning of 8, where the learning rate is multiplied by 10+Vi-I at i-th epoch. The batch size is searched in [8, 32, 64, 256].
10
E.2 Comparison of DM-PS Family and Other Baselines under Policy Shift
SnIPS
DM
DM(R)
DM-PS
((a known 3(a[z) unknown
10
334
17
13
327
Table 4:
The number of conditions where each family the best under policy shift. DM-PS family outper-
forms baseline methods in over 90% of the conditions.
E.3 Comparison of SnIPS and SnIPS-GCS under different covariate shift
We also analyze the impact of various types of covariate shifts on the performance of SnlPS and SnIPS- ACCSS in Table aas as below. Specifically, SnIPS exhibits superior performance under the Gaussian covariate shift, which generally creates larger covariate shift compared to the Tweak-1 covariate shift, outperforming SnIPS-GCS across an extensive range of conditions. While under the Tweak-1 shift, SnIPS-GCS demonstrates enhanced performance in larger number of scenarios than SnIPS. This observation demonsttrtes that large covariate shifts create barder-to-estimate density ratios and negatively affect the performance of SnIPS-GCS, compared SnIPS.
Table 5: The number of conditions where each estimator is the best when Pa(x, a) is known.
covariate shift
Targel policy
SulPS
SnIPS-GCS
(10.9.0)(a))) x)
870
390
Gaussian covariate
Diverse Softened Perfect policy
945
315
T(0.9.0) (a x)
404
856
Tweak-1 covariate
Diverse Softened Perfect policy
240
1020
E.4 Further Comparison with Advanced OPE Methods
We present the result of our method applied to and compared with other advanced OPE approaches such as SWITCH (Wang al. (017)) and DRoS (Su et al.) (020) Figure 5 Our method can be integrated into SWITCH and DRoS by replacing their reward estimator with our estimators, DM-PS and DM-GCS. Incorporating DM-PS into SWITCH and DRoS results in SWITCH-PS and DRoS-PS, respectively. Sim- ilarly, applying DM-GCS to SWITCH and DRoS results in SWITCH-GCS and DRoS-GCS. As shown in Figure 5) SWITCH-PS and DRoS-PS consistently outperform the original SWITCH and DRoS methods, respectively, across all cases, including both policy shift and general covariate shift cases. Additionally, in the general covariate shift setting, the performance DRoS also further improved by DM-GCS. These results demonstrates the advantage of integrating DM-PS and DM-GCS into SWITCH and DRoS.
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
1.0
(8) PS, Logging poicy known
(b) PS, Logging policy unknown
(6) GCS. WDx known
GCS, Htx, a) unknown
SrIPS
SWITCH
SHOR-PS SWITCH-PS
DRoS-PS
DRoS
SnDR SrOR-PS
SnDR-GCS .--
DRoS-GCS
SWITCH
SWICCH
Figure 5: Subfigures (a) and (b) present the experiment results under policy shift. Subfigures (c) and (d) present the experiment results under general covariate shift. DM-PS and DM-GCS enhance the performance of SWITCH and DRoS.
24

---

Published in Transactions on Machine Learning Research (05/2024)
E.5
Experiments Results on Each Dataset
We show the experiment results each dataset under policy shift and general covariate shift in Figure t and Figure 7 as below. Specifically, each row corresponds to the result of each dataset. In each row, subfigures (a) and (b) are comparisons between different families of methods, while (c) and (d) are comparisons between individual estimators. Subfigures (a) and (c) display the results when the {(a(z) known policy shift conditions or W(x, a) is known in general covariate shift conditions, while (b) and (d) present the results when they unknown.
25

---

Published in Transactions on Machine Learning Research (05/2024)
SnIPS
DM(R)
DM-PS SnDR-PS
MRDR
SniPS
DM family
DM(R) family
DM-PS family
DM
SnDR(R)
DRPE
0.0
0.0
0.0
0.0
(8) Logging policy known
(b) Logging policy unknown
(c) Logging policy known
(d) Logging policy unknown
Comparisan Between Families
Comparisan Between Estimatoos
vehicle
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
0.6
0.8
0.8
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.2
0.4
0.6 0.8
1.0
1.2
1.4
0.0
0.2
0.4
0.8
0.8
1.0
12
1.4
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
0.0
0.2 0.4
0.8
0.8
1.0
1.2
1.4
(a) Logging policy known
Logging policy unknown
[c] Logging policy known
(d) Logging policy unknown
Comparison Between Families
Comparison Between Estimators
ecoli
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.4 0.6
0.0
022
12
(a) Logging policy known
(b) Logging policy unknown
[c] Logging policy known
(d) Logging policy unknown
Comparison Between Families
Comparison Between Estimators
glass
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.8
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
0.0
0.2
0.4
0.8
0.8
1.0
1.2
1.4
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
0.0
0.2
0.4
0.6
0.8
1.0
12
14
(a) Logging policy known
Logging policy unknown
(e) Logging policy known
(d) Logging policy unknown
Comparison Between Families
Comparison Between Estimators
yeast
1.0
1.0
1.0
0.8
0.8
0.0
0.6
0.0 0.2
1.2
(8) Logging pollcy known
(b) Logging policy unknown
(6) Logging policy known
(d) Logging policy unknown
Comparison
Comparison
pageblock
26

---

Published in Transactions on Machine Learning Research (05/2024)
1.0
1.0
0.8
0.8
0.2 0.4 0.6
0.0
0.2
(8) Logging polloy known
(b) Logging policy unknown
(6) Logging policy known
(d) Logging pollcy unknown
Comparison Between Families
Comparison Between Estimators
optdigits
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
0.6
0.8
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.2
0.4
0.6
08
1.0
1.2
1.4
0.0
0.2
0.4
0.6
0.8
1.0
12
144
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
0.0
0.2
0.4
0.6
0.8
1.0
1.2
144
Logging pollcy known
(b) Logging policy unknown
Logging policy known
(d) Logging policy unknown
Comparison Between Families
Comparison Between Estimators
satimage
0.4
0.0
0.0
0.2
1.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.8
1.0
1.2
1.4
0.0
0.2
0.4
0.6
0.8
Logging pollcy known
(b) Logging policy unknown
(c) Logging policy known
(d) Logging policy unknown
Comparison
Comparison Between Estimators
pendigits
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.2
12
1.4
0.0
0.2 0.4
0.8
0.8
1.0
12
144
0.0
0.8
1.0
1.2
1.4
0.0
0.2
0.4
0.8
0.8
(8) Logging policy known
(b) Logging policy unknown
(c) Logging policy known
(d) Logging policy unknown
Comparison Between Families
Comparison Between Estimators
letter
Figure 6: Experimental results of different datasets under policy shifts.
27

---

Published in Transactions on Machine Learning Research (05/2024)
SnIPS-GCS SniPs
DM family
DM-PS family
SnIPS-GCS
DM(R)
SnDR-PS
MRDR
DM(R) family
DM-GCS family
DM
SnDR(R)
DM-GCS
DRPE
SnDR
DM.PS
SnDR-GGS
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0 0.2
0.2
0.8
0.8
1.0
12
1.4
0.0
0.2
0.0
0.8
0.8
1.0
(a) wixx, a) known
(6) WX a) unknown
(6) YX a) known
(d) W/x, a) unknown
Comperison Between Familles
Comperison Between Estimetors
vehicle
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
0.6
0.6
0.8
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.2
0.4
0.8 0.8
1.0
1.2
1.4
0.0
0.2
0.4
0.6
0.8
1.0 1.2 1.4
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
0.0
0.2
0.4
0.8
0.8
1.0
1.2
1.4
WXx, a) known
WVX, 8) unknown
Wix, a) known
(d) WXx, unknown
Comparison Between Families
Comparison Between Estimators
ecoli
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.2
0.4
0.8 0.8
12
1.4
0.0
0.2
(a) WVX, a) known
(b)
WVx, a) unknown
(c) WIx, a) known
(d) WDC, #) unknown
Companison Belween Families
Comparison Between Estimators
glass
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.2
0.1
0.6 03
1.0 1.2 1.4
0.0
0.2
0.4
0.5
0.8
1.0
1.2 14
0.0
0.2
0.1
0.6
0.8
1.0
1.2
1.4
0.0
0.2
0.4
0.
0.8
1.0
1.2
14
VVX, a) known
wix, a) unknown
WIX, a) known
(d) WXX, unknown
Comparison Between Families
Comparison Between Estimators
yeast
1.0
1.0
1.0
1.0
0.8
0.8
0.0
0.2
0.4 0.6
1.2 1.4
0.0
0.2
0.4 06
(8) YVX, a) known
(b) WLx, a) unknown
(6) WDx, a) knownn
(d) WXx, 2) unknown
Comparison Between Families
Comparison Between Estimators
pageblock
28

---

Published in Transactions on Machine Learning Research (05/2024)
1.0
1.0
1.0
0.8
0.8
0.4
0.2
0.2
0.4 0.6 0.8
122
0.0
0.2
0.4
(8) WVX, a) known
(b) Wtx, a) unknown
[0] WIX, a) known
(d) WVX, a1 unknown
Comparison Between Families
Comparison Between Estimators
optdigits
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.8
0.6
0.8
0.6
0.4
0.4
0.4
0.4
0.2
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.0
0.2
0.4
0.6 08
1.0 1.2 1.4
0.0
0.2
0.4
0.6
0.8
1.0
12 144
0.0
0.2
0.4
0.6
0.8
1.0
1.2
1.4
0.0
022
0.4
0.6
0.8
1.0
1.2
14
(8) yttx, a) known
(b) yitx, a) unknown
(c)
W/x. a) known
(d) wX(x a) unknown
Comparison Between Families
Comparison Between Estimators
satimage
0.4
0.0
0.0 0.2
0.4
0.6
1.2
0.0
0.2
0.4
0.6
0.8
1.0 1.2
0.2
0.6
0.8
1.0
1.2
1.4
0.0
0.2
0.4
0.
0.8
1.2
(a) WXx, a)
(b) vix, a) unknown
[c)
Wix, a)
(d) WX, unknown
Comparison Between Families
Comparison Between Estimators
pendigits
1.0
1.0
1.0
1.0
0.8
0.8
0.8
0.8
0.6
0.6
0.6
0.6
0.4
0.4
0.4
0.4
0.2
02
0.2
0.2
0.0
0.0
0.0
0.0
0.2
0.4 0.8
12 1.4
0.0
0.2
0.4
0.8
0.8 1.0
12 1.4
0.0
0.2
0.6
0.8
1.0
1.2
1.4
0.0
0.2
0.4
088
088
12 1.4
(a) Wtx. a) known
(b) YVx a) unknown
[c) WXX, a) known
(d) WVX, a) unknown
Comparison Between Families
Comparison Between Estimators
letter
Figure 7: Experimental results of different datasets under general covariate shifts.
29


Reviews:
[{"review_id": "TnD7xr2pk4", "reviewer": "Reviewer_cCYM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The author(s) presented a detailed review of related works on offline policy evaluation.\n1. The author(s) described how the difference between source and target distributions influence the density ratio $W(x,a)$ and hence the evaluation.\n\nWeaknesses:\n1. Algorithm 1 assumes that the contexts are drawn from Gaussian distribution. If this assumption does not hold, what should we expect theoretically and empirically? For example, what if source or target distribution is not Gaussian?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposed a method to improve offline policy evaluation in contextual bandits. It discussed existing works and introduced Algorithm 1 where both source and target distributions are Gaussian distributions. I provided bias analysis for DM and DR methods and presented experimental results.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the 'Weaknesses' part in the 'Strengths and Weaknesses' section.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SSBLWzttWe", "reviewer": "Reviewer_FLHu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n- This paper is well-written and easy to read, it is interesting to look the OPE problem from the covariate shift perspective and the utilization of robust regression to improve DM in OPE is intuitive and well-justified. \n- The method is simple to implement, and it introduces improvements without significantly increasing computational complexity over existing baselines.\n- Meanwhile, the empirical results across various conditions (such as dataset, magnitude of policy/covariate shift, known/unknown density estimation, etc) is comprehensive, demonstrating the effectiveness of the advanced DM estimator, compared with the vanilla ones.\n\nWeakness:\n\n- The bias analysis for the regression estimator is interesting, but it does not provide much insight comparing only looking at the closed form solution in Equation (5). Theorem 2 on the upper bound of DR is confusing, ideally it should have a much simpler form by utilizing the \"doubly robust\" unbiased nature of the estimator, as here only need to bound the unsupported part?\n\n- The justification of the constant prior is missing, is it for a simpler form of the estimator only? Ideally the prior could be based on any existing DM estimators and improve upon that.\n\n- How is the optimality of the proposed robust regression estimator, say under a specific class of reward distributions?\n\n- The empirical results seems missing some important baselines. Examples could include MRDR, variations of reweighting on the DM training loss function, localized doubly robust DROPE, and other general OPE estimators like SWITCH, DRoS, clipping, etc. This would provide a more comprehensive comparison and a clearer understanding of how the proposed reward estimator improvements compare with a broader range of existing methods. It would also be great to see how the reward estimators improvement further translate into the improvement of these advanced estimators.\n\nReference:\n- More Robust Doubly Robust Off-policy Evaluation. Mehrdad Farajtabar, Yinlam Chow, Mohammad Ghavamzadeh\n- Optimal and Adaptive Off-policy Evaluation in Contextual Bandits. Yu-Xiang Wang, Alekh Agarwal, Miroslav Dudik\n- Doubly robust off-policy evaluation with shrinkage. Yi Su, Maria Dimakopoulou, Akshay Krishnamurthy, Miroslav Dudík\n- Distributionally Robust Batch Contextual Bandits. Nian Si, Fan Zhang, Zhengyuan Zhou, Jose Blanchet\n- Doubly Robust Distributionally Robust Off-Policy Evaluation and Learning. Nathan Kallus, Xiaojie Mao, Kaiwen Wang, Zhengyuan Zhou", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies OPE for contextual bandits, under general covariate shift conditions (both the covariate distribution and policy are changing). It mainly focus on how to improve the reward model part among the OPE components. Specifically, it views the OPE problem as covariate shift, and utilizes robust regression, a distributionally robust technique to improve the estimate of the reward distribution from logged data. The more advanced reward model (RM) estimator could be used as a plug-in estimator for both DM, DR, snDR, etc. Extensive semi-synthetic experiments conducted on contextual bandits showcase the method's efficacy compared to traditional DM approaches.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the Weakness Section.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SQYmXEKEmc", "reviewer": "Reviewer_ATLF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\nThe technical setting studied, as well as the algorithms developed, are novel, to my knowledge.\n\nSpecifically, distributionally robust optimization approaches are developed for the off-policy evaluation objective. Their derived predictive forms is also novel, in my understanding, as well as stochastic gradient algorithms based upon them.\n\nExperiments clearly demonstrate the merits of the proposed techniques for the setting of covariate shift.\n\n Weaknesses:\n\nConvergence analysis seems to mostly follow from prior techniques, or otherwise it is not very clear what innovative steps are deployed here to establish the main theoretical results.\n\nThe class of distributional drifts that the method can address seem to require a specific model of distributional robustness that seems to be not specifically tethered to the dimensions in the covariate where the drift is largest, but instead defines a coarser notion of distributional robustness. It would be nice to see how a prior could be weighted to better target the robustness of the resultant estimator.\n\nIf one does not impose structural such that the derived predictor is Gaussian (aka non-quadratic losses), how much more predictive power does one gain? Are there situations where the Gaussian form is not valid? I would like to see some experimental studies with respect to classes of mismatch distributions, to better understand when this modeling assumption is/is not valid.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studies methods to incorporate covariate shift into off policy evaluation, especially policy shift and context shift. Extensions of doubly robust estimators (which fuse together importance weighting and control variates) are developed for these settings, and their finite-sample analysis is provided. Experiments demonstrate the merits of the proposed methods relative to prior art especially in the context of covariate shift.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See response to previous field.", "broader_impact_concerns": "See response to previous field.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jspOMJ5cQR", "reviewer": "Reviewer_mVrZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**  \nIn general, this paper did a good job in posing the research problem and articulating how they reached the solution. While the formulation of covariate shift in the contextual bandit setting may resemble that in the classification setting, I find the work to be original and novel.  The paper is clearly written and well organized. Notably, the authors implement the reward estimation methods into different modeling framework, and provide the corresponding error bound.\n\n**Weaknesses**\n- It is not intuitive to understand why the parameters $\\theta$ are divided into $\\theta_r$ and $\\theta_x$ in Eq. (5). I would suggest the authors to further elaborate this in the *main* paper.\n-  The authors claim that they conduct experiments under 1260 total combinations of datasets, logging policies, covariate shifts, and target policies. However, this claim appears somewhat exaggerated. Have the authors provided experimental results for all 1260 combinations, and does the performance consistently exhibit the same trend across this entire set of combinations?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work considers the distribution shift in the contextual bandit setting, recovering the classic setting of off-policy evaluation. The authors interpret the policy evaluation as a bandit equivalent version of covariant shift, where the covariates are the contexts and actions and the dependent variable is the reward. Borrowing the ideas from distributional robust optimization, the authors propose a robust reward estimator and provide justifications about the error bounds. This work also conduct implausible number of combinations of covariate shift settings to evaluate their methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I recommend that the authors comprehensively present all the experimental results obtained for each combination of covariate shift settings.", "broader_impact_concerns": "While this paper does not present a Broader Impact Statement, I don't foresee any societal impacts that should be addressed in this context.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this submission, the authors study OPE for contextual bandits under general covariate shift. Specifically, the authors exploit distributionally robust regression for reward estimation, which will be plugged for existing OPE estimator, e.g., DM, DR, and snDR. 

The authors did comprehensive evaluation the proposed methods to justify the claim and provided theoretical analysis for the proposed method. Although the novelty of the proposed method is not significant, the TMLR does not include novelty and significancy as a criterion. However, there are several issues raised by the reviewers need to be address: 
 
- The important baselines suggested by reviewer FLHu should be included in the final version. 

-  The method derivation is based on Gaussian assumption (reviewer cCYM and ATLF). The limitation and potential extension beyond this Gaussian assumption should be included in the final version. 

-  There are existing methods exploiting distributionally robust optimization for confidence interval of OPE (under the name "empirical likelihood") [1, 2, 3], which should be also discussed. 

[1] Nikos Karampatziakis, John Langford, and Paul Mineiro. Empirical likelihood for contextual bandits. arXiv
preprint arXiv:1906.03323, 2019.

[2] Faury, Louis, Ugo Tanielian, Elvis Dohmatob, Elena Smirnova, and Flavian Vasile. "Distributionally robust counterfactual risk minimization." In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 04, pp. 3850-3857. 2020.

[3] Dai, Bo, Ofir Nachum, Yinlam Chow, Lihong Li, Csaba Szepesvári, and Dale Schuurmans. "Coindice: Off-policy confidence interval estimation." Advances in neural information processing systems 33 (2020): 9398-9411.
Decision: Accept (poster)