Marginal Density Ratio for Off-Policy Evaluation in Contextual Bandits
Authors: Muhammad Faaiz Taufiq
Arnaud Doucet
Rob Cornish
Jean-Francois Ton
Venue: NeurIPS 2023 poster (2023)
Keywords: contextual bandits
variance reduction
off-policy evaluation
TL;DR: We propose an off-policy estimator with reduced variance compared to IPW and DR methods, by considering only the variations in the outcome instead of the variations in the policies.

Abstract: Off-Policy Evaluation (OPE) in contextual bandits is crucial for assessing new policies using existing data without costly experimentation. However, current OPE methods, such as Inverse Probability Weighting (IPW) and Doubly Robust (DR) estimators, suffer from high variance, particularly in cases of low overlap between target and behaviour policies or large action and context spaces. In this paper, we introduce a new OPE estimator for contextual bandits, the Marginal Ratio (MR) estimator, which focuses on the shift in the marginal distribution of outcomes $Y$ instead of the policies themselves. Through rigorous theoretical analysis, we demonstrate the benefits of the MR estimator compared to conventional methods like IPW and DR in terms of variance reduction. Additionally, we establish a connection between the MR estimator and the state-of-the-art Marginalized Inverse Propensity Score (MIPS) estimator, proving that MR achieves lower variance among a generalized family of MIPS estimators. We further illustrate the utility of the MR estimator in causal inference settings, where it exhibits enhanced performance in estimating Average Treatment Effects (ATE). Our experiments on synthetic and real-world datasets corroborate our theoretical findings and highlight the practical advantages of the MR estimator in OPE for contextual bandits.

Marginal Density Ratio for Off-Policy Evaluation in Contextual Bandits
Muhammad Faaiz Taufiq"
Arnaud Doucet Department of Statistics University of Oxford
Department of Statistics University of Oxford
Rob Cornish
Jean-François Ton
Department of Statistics University of Oxford
ByteDance Research
ByteDance
Abstract
Off-Policy Evaluation (OPE) in contextual bandits is crucial for assessing new policies using existing data without costly experimentation. However, current OPE methods, such as Inverse Probability Weighting (IPW) and Doubly Robust (DR) estimators, suffer from high variance, particularly in cases of low overlap between target and behavior policies or large action and context spaces. In this paper, we introduce a new OPE estimator for contextual bandits, the Marginal Ratio (MR) estimator, which focuses on the shift in the marginal distribution of outcomes Y instead of the policies themselves. Through rigorous theoretical analy sis, we demonstrate the benefits of the MR estimator compared to conventional methods like IPW and DR in terms of variance reduction. Additionally, we establish a connection between the MR estimator and the state-of-the-ar Marginalized Inverse Propensity Score (MIPS) estimator, proving that MR achieves lower variance among generalized family of MIPS estimators. We further illustrate the utility of the MR estimator causal inference settings, where exhibits enhanced performance in estimating Average Treatment Effects (ATE). Our experiments on synthetic and real-world datasets corroborate our theoretical findings and highlight the practical advantages of the MR estimator in OPE for contextual bandits.
Introduction
In contextual bandits, the objective is to select action A, guided by contextual information X, to maximize the resulting outcome Y. This paradigm is prevalent many real-world applications such as healthcare, personalized recommendation systems, online advertising [13] The objective is to perform actions, such as prescribing medication or recommending items, which lead to desired outcomes like improved patient health or higher click-through rates. Nonetheless, updating the policy presents challenges, as naïvely implementing a new, untested policy may raise ethical or financial concerns. For instance, prescribing a drug based on a new policy poses risks, as it may result in unexpected side effects. As a result, recent research [4] has concentrated on evaluating the performance of new policies (target policy) using only existing data that generated using the current policy (behaviour policy). This problem is known as Off- Off-Policy Evaluation (OPE).
Current OPE methods in contextual bandits, such as the Inverse Probability Weighting (IPW) [22 and Doubly Robust (DR) [33 estimators primarily account for the policy shift by re-weighting the
Part of the work was done during an internship at ByteDance Research. Correspondence to muhammad. taufiq iesttttss............. ox. ac. uk and j yeadfrancois@@yteddccc.oom
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

data using the ratio of the target and behaviour polices to estimate the target policy value. This can be problematic it may lead to high variance in the estimators in cases of substantial policy shifts. The issue is further exacerbated in situations with large action or context spaces [14], since in these cases the estimation of policy ratios is even more difficult leading to extreme bias and variance.
In this work we show that this problem of high variance in OPE can be alleviated by using methods which directly consider the shift in the marginal distribution of the outcome Y resulting from the policy shift, instead of considering the policy shift itself (as in IPW and DR). To this end, we propose a new OPE estimator for contextual bandits called the Marginal Ratio (MR) estimator, which weights the data directly based on the shift in the marginal distribution of outcomes Y and consequently is much more robust to increasing sizes of action and context spaces than existing methods like IPW or DR. Our extensive theoretical analyses show that MR enjoys better variance properties than the existing methods making highly attractive for a variety of applications in addition to OPE. One such application is the estimation of Average Treatment Effect (ATE) in causal inference, for which we show that MR provides greater sample efficiency than the most commonly used methods.
Our contributions in this paper are as follows:
Firstly, we introduce MR, an OPE estimator for contextual bandits, that focuses on the shift in the marginal distribution of rather than the joint distribution of (X, A, Y). We show that MR has favourable theoretical properties compared existing methods like IPW and DR. Our analysis also encompasses theory on the approximation errors of our estimator.
Secondly, we explicitly lay out the connection between MR and Marginalized Inverse Propensity Score (MIPS) [14), a recent state-of-the-ae contextual bandits OPE method, and prove that MR attains lowest variance among a generalized family of MIPS estimators.
Thirdly, we show that the MR estimator can be applied in the setting of causal inference to estimate average treatment effects (ATE), and theoretically prove that the resulting estimator is more data-efficient with higher accuracy and lower variance than commonly used methods.
Finally, we verify all our theoretical analyses through a variety of experiments on synthetic and real-world datasets and empirically demonstrate that the MR estimator achieves better overall performance compared to current state-of-the tort methods.
Background
2.1 Setup and Notation
We consider the standard contextual bandit setting. Let X X' be a context vector (e.g., user features), A E A denote an action (c.g., recommended website to the user), and Y E y denote a scalar reward or outcome (c.g., whether the user clicks on the website). The outcome and context are sampled from unknown probability distributions p(y | a) and p(x:) respectively. Let D == {(x;, a; } be historically logged dataset with n observations, generated by (possibly unknown) behaviour policy (a z). Specifically, 'D consists of i.i.d. samples from the joint density under behaviour policy,
P= (I, a, y) ply x. a) xa xx) p(z).
We denote the joint density of (X, A, Y) under the target policy as
P- (Ir, a, y) == ply x. a) x (a x)) p(x).
(2)
Moreover, we use (y) to denote the marginal density of Y under the behaviour policy,
P- (v) f
JAxx Pe (x. a, y) da
and likewise for the target policy n¹ Similarly, we use E, and Eg- to denote the expectations under the joint densities pa (I. a, v) and px (x., a, respectively.
Off-policy evaluation (OPE) The main objective of OPE is estimate the expectation of the outcome Y under a given target policy x². i.e., E,, [Y], using only the logged data D.
Throughout this work, we assume that the support of the target policy n" is included in the support of the behaviour policy n². This is to ensure that importance sampling yields unbiased off-policy estimators, and is satisfied for exploratory behaviour policies such as the e-greedy policies.

---

Assumption 2.1 (Support), For any 2 X A. (a > 0 = x²(a x)) > 0.
2.2 Existing off-policy evaluation methodologies
Next, will present some of the most commonly used OPE estimators before outlining the limita- tions of these methodologies. This motivates our proposal of an alternative OPE estimator.
The value of the target policy can be expressed as the expectation of outcome Y under the target data distribution px- (z, a, y). However in most cases, we do not have access to samples from this target distribution and hence we have to resort to importance sampling methods.
Inverse Probability Weighting (IPW) estimator One way to compute the target policy value, Eg [Y]. when only given data generated from Pa (I, a, is to rewrite the policy value as follows:
B. [x] = / pp (2e, y) dy do de
Px (z,, a.
P. (z, a.)) Pab Pao a, 2) dy B (Y p(A,
p(a.2)
where p(a, x)- Pe- (u..a))
P.s(2,0.9) (aa given the factorizations in Eqns.
(ala)
This leads to and 2
the commonly used Inverse Probability Weighting (IPW) 122 estimator:
Opw Σ x1, Ve-
When the behaviour policy known, IPW an unbiased and consistent estimator. However, suffer from high variance, especially as the overlap between the behaviour and target policies decreases.
Doubly Robust (DR) estimator To alleviate the high variance of IPW, K proposed a Doubly Robust (DR) estimator for OPE, DR uses an estimate of conditional mean jı(a, x) EE
A a] (outcome model), control variate to decrease the variance of IPW. It is also doubly robust in that yields accurate value estimates if either the importance weights p(a, x) or the outcome model fi(a,20 2) is well estimated [5]. The DR estimator for Ex, [Y] can be written as follows:
a Σ
0DR
pla, 2;) (ys ((a,, + f)(=)),
where 0i(=T) n Ei EvEA Maa' xx) (a' x2) E. (A X)). Here, n((T") referred as the Direct Method (DM) uses (aaa,)) directly estimate target policy value.
2.3 Limitation of existing methodologies
To estimate the value of the target policy π², the existing methodologies consider the shift in the joint distribution of (X, A, Y) as result of the policy shift (by weighting samples by policy ratios). As we show Section 3.T considering the joint shift can lead inefficient policy evaluation and high variance especially the policy shift increases [16]. Since our goal is to estimate E,, [Y), will show in the next section that considering only the shift in the marginal distribution of the outcomes Y from P, (Y) to px (Y), leads to more efficient OPE methodology compared to existing approaches.
To better comprehend why only considering the shift the marginal distribution advantageous, let us examine an extreme example where we assume that Y II A X, i.e., the outcome Y for a user X independent of the action A taken. this specific instance, F [Y] = E_ 1/n >" Vis indicating that an unweighted empirical mean serves as a suitable unbiased estimator of E,, [Y].
However, IPW and DR estimators use policy ratios p(a, x) = " (alxz)
[aaaa as importance weights. In case
of large policy shifts, these ratios may vary significantly, leading to high variance in IPW and DR.
In this particular example, the shift in policies is inconsequential as it does not impact the distribution of outcomes Y. Hence, IPW and DR estimators introduce additional variance due to the policy ratios when they are not actually required. This limitation is not exclusive to this special case; in general, methodologies like IPW and DR exhibit high variance when there low overlap between target and behavior policies [16] even if the resulting shift in marginals of the outcome Y is not significant.
3

---

Therefore, we propose the Marginal Ratio (MR) OPE estimator for contextual bandits in the subse- quent section, which circumvents these issues by focusing on the shift the marginal distribution of the outcomes Y. Additionally, we provide extensive theoretical insights on the comparison of MR to existing state-of-the-arr methods, such as IPW and DR.
3 Marginal Ratio (MR) estimator
Our method's key insight involves weighting outcomes by the marginal density ratio of outcome Y:
Rg M = / y yp (v) (y) S
Px (1)
Pa (v) P (v) dy = E [V (Y)),
where w(y) = P-- [v) This leads to the Marginal Ratio OPE estimator: p(r(a)
0MR == 1 Σ w(y)) y:-
In Section 3.1 we prove that by only considering the shift in the marginal distribution of outcomes, the MR estimator achieves lower variance than the standard OPE methods. In fact, this estimator does not depend on the shift between target and behaviour policies directly. Instead, it depends on the shift between the marginals pa (y) and px (v)
Estimation of w(y) When the weights wyy) are known exactly, the MR estimator is unbiased and consistent. However, in practice the weights w(y) are often not known and must be estimated using the logged data D. Here, we outline an efficient way to estimate w(y) by first representing as conditional expectation, which can subsequently be expressed as the solution to regression problem.
Lemma 3.1. Let w(y) = (v)(v) and pla, x)) Pes(a)
<'(a|z) then w(y) E, (o(A, Y = w). and, "'(aaa'
w = arg min E, ()(AA 1(
(3)
Lemma 3.1 allows us to approximate w(y) using parametric family {So : + R neural networks) and defining (yy) = fo- (v), where o² solves the regression problem in Eq. (3)
} (e.g.
Note that MR can also be estimated alternatively by directly estimating h(y) = w(v) y using similar regression technique above and computing 0MR = Ei= h(y:). We include additional details along with empirical comparisons in Appendix
3.1 Theoretical analysis
Recall that the traditional OPE estimators like IPW and DR use importance weights which account for the the shift in the joint distributions of (X, A, Y). In this section, we prove that by considering only the shift in the marginal distribution of Y instead, MR achieves better variance properties than these estimators. Our analysis in this subsection assumes that the ratios p(a, x) and w(y) are known exactly. Since the OPE estimators considered are unbiased in this case, our analysis of variance is analogous that of the mean squared етror (MSE) here. We address the case where the weights are not known exactly in Section 3.1.2] First, we make precise our intuition that the shift in the joint distribution of (X, A, Y) is 'greater' than the shift in the marginal distribution of outcomes Y. We formalise this using the notion of f-iivergences.
Proposition 3.2. Let f (0) the f-divergence between distributions P and Q. Then,
oo) + be a convex function with f(1) = 0, and (PI(Q) denotes
Dy (pe. (x, v) P. (I, a,y) 9)) > Dy (px- (1) (1))
Intuition Proposition 3.2 shows that the shift in the joint distributions is at least as 'large' as the shift in the marginals of the outcome Y. Traditional OPE estimators, therefore take into consideration contrast,
more of a distribution shift than needed, and consequently lead inefficient estimators. the MR estimator mitigates this problem by only considering the shift in the marginal distributions of outcomes resulting from the policy shift. This provides further intuition on why the MR estimator has lower variance compared to existing methods.

---

Proposition 3.3 (Variance comparison with IPW estimator). When the weights p(a, and w(y) are known exactly, we have that Var," [OMi]] I Var,, [0 In particular,
1 E [Var (ø(A, x) y 2 0.
, [0pp]]
VVr,z [0MR]
71
Intuition Proposition 3.3 shows that the variance MR estimator is smaller than that of the IPW estimator when the weights are known exactly. Moreover, the proposition also shows that the difference between the two variances will increases as the variance Var [p(A, X) YY] increases. This variance likely to be large when the policy shift between म. and " is large, or when the dimensions of contexts X and/or the actions A is large, and therefore in these cases the MR estimator will perform increasingly better than the IPW estimator, A similar phenomenon occurs for DR as we show next, even though in this case the variance of MR is not in general smaller than that of DR. Proposition 3.4 (Variance comparison with DR estimator). When the weights pla, x) and w(y) are known exactly and x)) = E[ X, A}, we have that,
1 E [Var (o(A, IY V [o(A, X) (X,, X]]-
Var, [QDR] Var. [OMR]
Intuition Proposition
3.4 shows that X average, the variance of the MR estimator will be less than
Var,* (
greater than
Var,. [p(A, X) ((A,X) that the DR estimator. Intuitively, this will occur when the dimension of context space is high because in this case the conditional variance over X and A, Var, X) Y] is likely to be greater than the conditional variance over A, Var.* [p(A, X) ((A, X]. Our empirical results in Appendix
F2 are consistent with this intuition. Additionally, with other extensions of DR, such as Switch-DD DR [3] and DR with Optimistic Shrinkage (DRos) [I7] in Appendix B and show that similar intuition applies for these results. We emphasise that the well known results in [5] which show that IPW and DR estimators achieve the optimal worst case variance (where the worst case is taken over class of possible outcome distributions Y X, A) not at odds with our results presented here (as the distribution of Y X, A is fixed in our setting).
also provide theoretical comparisons
3.1.1 Comparison with Marginalised Inverse Propensity Score (MIPS) 14
In this section, we compare MR against the recently proposed Marginalised Inverse Propensity Score (MIPS) estimator [14, which uses marginalisation technique to reduce variance and provides a robust OPE estimate specifically in contextual bandits with large action spaces. We prove that the MR estimator achieves lower variance than the MIPS estimator and doesn't 't require new assumptions.
MIPS estimator As we mentioned earlier, the variance of the IPW estimator may be high when the action A is high dimensional. To mitigate this, the MIPS estimator assumes the existence of (potentially lower dimensional) action embedding E, which summarises all 'relevant' information about the action A. Formally, this assumption can be written follows:
Assumption 3.5. The action has no direct effect on the outcome Y, i.i... Y TL A,
For example, in the setting of recommendation system where A corresponds to the items recom- mended, E may correspond to the item categories. Assumption 3.5 then intuitively means that item category É encodes relevant information about the item which determines the outcome Y. Assuming that such action embedding E exists,
[14 prove that the MIPS estimator ÓxIps. defined as @MPS = 1 Σ Px- (es,
1 Par (e; xz) i=1
1=1 Par (es, x;)
Pat x.)) y/s
provides unbiased estimator of target policy value E, [Y]. Moreover, Var, [OMuP]] I Var,, [0pw]]
Intuition The context-embedding pair (X, E) can be seen as representation of the context-action pair (X, A) which contains less 'redundant information' regarding the outcome Y. Intuitively, the MIPS estimator, which only considers the shift in the distribution of (X, E) is therefore more efficient than the IPW estimator (which considers the shift in the distribution of (X, A) instead).
(X,A)
(X, F))
Figure 1: Bayesian network correspond-
ing to Assumption 3.5
5

---

MR achieves lower variance than MIPS Given the intuition above, we should achieve greater variance reduction as the amount of redundant information in the representation (X, E) decreases. We formalise this in Appendix D and show that the variance of MIPS estimator decreases as the representation gets closer to Y in terms of information content. As a result, we achieve the greatest variance reduction by considering the marginal shift in the outcome Y itself (as in MR) rather than the shift in the representation (X, B)) (as in MIPS). The following result formalizes this finding.
Theorem 3.6. When the weights (V)), Put (e.a) and p(a, .) are known exactly, then under Assumption (e,(e,z)
3.5
E.za [0NR] = Z [0MIPs] = -x Y and Var,* [QMRR] S Var," [0MIs)] S Var [0pp]]
This analysis provides link between the MR and MIPS estimators in the framework of contextual bandits, and shows that the MR estimator achieves lower variance than MIPS estimator while not requiring any additional assumptions (e.g. Assumption as in MIPS). We also verify this empirically in Section 5.1 by reproducing the experimental setup in 14) along with the MR baseline.
3.1.2 Weight estimation error
Our analysis so far assumes prior knowledge of the behavior policy TO and the marginal ratios w(y). However, in practice, both quantities are often unknown and must be estimated from data. To this end, assume access to additional training dataset Dr {<a! a. v/")
) ) (for weight estimation), in addition to the evaluation dataset D = { az yr hi=1 (for computing the OPE estimate). The estimation of (y) involves a two-step process that exclusively utilizes data from Da:
(i) First, we estimate the policy ratio p(a, = " (a(ax)
[a] This can be achieved by estimating
the behaviour policy 8h, and defining p(a, xx) = TPaa)) Alternatively, ((a, can also be estimated directly by using density ratio estimation techniques in [8].
(ala)
(ii) Secondly, we estimate the weights (1) using Eq.
3 with instead of p.
In practice, one may consider splitting Du for each estimation step outlined above. Moreover, each approximation step may introduce bias and therefore, the MR estimator may have two sources of bias. While classical OPE methods like IPW and DR also suffer from bias because of ý estimation, the estimation of (y) is specific to MR. However, we show below that given any policy ratio estimate if vv(y)) approximates F. (Ā(A, A, X) Y = y] 'well enough' (i.e., the estimation step (ii) shown above is 'accurate enough'), then MR achieves lower variance than IPW and incurs little extra bias.
Proposition 3.7. Suppose that the IPW and MR estimators are defined as,
OIPW = M(a,,
1 TE
Σ
n
(4) yss and 0MR
Σ
((y)) Ve.
n
and define the approximation error as E = wY) (Y),, where wYY = E (,(, | we have that, (ias( - (fpp)) ) = E.A /e VJ. Moreover,
YJ. Then
Var, [0rw] Var, (BaRk] (F. [ar,, (M(,, MI -Var, - (Co (4)
The term defined in Proposition 3. denotes the error of the second approximation step outlined above. As a direct consequence of this result, we show in Appendix C that as the error e becomes small (specifically as [c2] 0), the difference between biases of MR and IPW estimator becomes negligible. Likewise, the terms Var Y) and Cov (O)(w(Y)Y.Cc CY) in Eq. { will also be small and as result the variance of MR will be lower than that of IPW (as the first term is positive).
Intuition
In fact, using recent results regarding the generalisation error of neural networks [9], we show that when using 2-layer wide neural networks to approximate the weights (vv), the estimation error declines with increasing training data size m. Specifically, under certain regularity assumptions we obtain E.,, [c2] = O(m -2/3). Using this we show that as the training data size m increases, the biases of MR and IPW estimators become roughly equal with high probability, and
1 E Var (((A Y]] + O(m 11/3).))
Var,+ [61p]] a [0MR]
n

---

Therefore the variance of MR estimator falls below that of IPW for large enough m. The empirical results shown in Appendix F2 are consistent with this result. Due space constraints, the main technical result has been included in Appendix
3.2 Application to causal inference
Beyond contextual bandits, the variance reduction properties of the MR estimator make it highly useful in a wide variety of other applications. Here, we show one such application in the field of causal inference, where MR can be used for the estimation of average treatment effect (ATE) [20] and leads to some desirable properties in comparison to the conventional ATE estimation approaches. Specifically, we illustrate that the MR estimator for ATE utilizes the evaluation data D more efficiently and achieves lower variance than state-of aart--rrttttttttt- ATE estimators and consequently provides more accurate ATE estimates. To be concrete, the goal in this setting estimate ATE, defined follows:
ATE == (1) Y())--
Here Y()) corresponds to the outcome under a deterministic policy aa x)) == 1(a a). Hence any OPE estimator can be used to estimate Y (a)] (and therefore ATE) by considering target policy " TT. An important distinction between MR and existing approaches (like IPW or DR) is that, when estimating B[Y (a)], the existing approaches only use datapoints in D with A = a. To see why 1(4=a)
this is the case, we note that the policy ratios x (=4
**(A[) **[A[X) are zero when A + a. In contrast,
the MR weights P+) not necessarily zero for datapoints with A a, and therefore the MR
P_)(F)
estimator uses all evaluation datapoints when estimating (a)].
As such show that MR applied ATE estimation leads to smaller variance than the existing approaches. Moreover, because MR is able to use all datapoints when estimating (a)). MR will generally be more accurate than the existing methods especially in the setting where the data is imbalanced, i.e., the number of datapoints with A- = a is small for a specific action a. In Appendix we formalise this variance reduction of the MR ATE estimator compared IPW and DR estimators, by deriving analogous results to Propositions 3.3 and 3.4 In addition, we also show empirically in
B
Section 5.3 that the MR ATE estimator outperforms the most commonly used ATE estimators.
4 Related Work
Off-Policy evaluation is a central problem both in contextual bandits 13 5 21 6,7 8 23
and RL (24 Existing OPE methodologies can be broadly categorised into Direct Method (DM), Inverse Probability Weighting (IPW), and Doubly Robust (DR). While DM typically has low variance, it suffers from high bias when the reward model is misspecified (28). On the other hand, IPW [12 and DR 13 5 use policy ratios as importance weights when estimating policy value and suffer from high variance as overlap between behaviour and target policies increases or as the action/context space gets larger 29 To circumvent this problem, techniques like weight clipping or normalisation [ 30 31 are often employed, however, these can often increase bias.
In contrast to these approaches, 14 propose MIPS, which considers the marginal shift in the distribution of a lower dimensional embedding of the action space. While this approach reduces the variance associated with IPW, we show in Section 3.1. that the MR estimator achieves lower variance than MIPS while not requiring any additional assumptions (like Assumption 3.5).
In the context of Reinforcement Learning (RI.), various marginalisation techniques of importance weights have been used to propose OPE methodologies. [2] 25 26 use methods which considers
the shift in the marginal distribution of the states, and applies importance weighting with respect to this marginal shift rather than the trajectory distribution. Similarly,
[2] use marginalisation for OPE in deep RL, where the goal is to consider the shift in marginal distributions of state and
action. Although marginalization is key trick of these estimators, these techniques do not consider the marginal shift in reward as in MR and are aimed at resolving the curse of horizon, a problem specific to RL. Apart from this, 33 propose a general framework of OPE based on conditional expectations of importance ratios for variance reduction. While their proposed framework includes reward conditioned importance ratios, this is not the main focus and there is little theoretical and empirical comparison of their proposed methodology with existing -hee-----hhe-rr methods like DR.
7

---

M52
107
107
107
10a
(0un) 17"4
10
2033
4003
0003
9303
12000
2000
4000
6000
2000
13000
0.0
022
0.4
0.6
0.3
0.0
0.2
0.1
0.5
03
10
(a) Results with varying size of evaluation dataset r1.
(b) Results with varying a²
Figure 2: Results for synthetic data experiment. In
2a have a" 0.8 and in 2b have 800.
Finally we note that the idea of approximating the ratio of intractable marginal densities via leveraging the fact that this ratio be reformulated the conditional expectation of ratio of tractable densities a standard idea in computational statistics [34] and has been exploited more recently to perform likelihood-free inference [35] In particular, while (34] typically approximates this expectation through Markov chain Monte Carlo, [35 uses regression instead, however without any theory.
Empirical Evaluation
In this section, we provide empirical evidence support theoretical results by investigating the performance of our MR estimator against the current state-of-the-arl OPE methods. The code to reproduce experiments has been made available at: Ctthub.com/kaniI2/MMM-0000
5.1 Experiments synthetic data
For our synthetic data experiment, we reproduce the experimental setup for the synthetic data experiment in [14] by reusing their code with minor modifications. Specifically, X' C TR4, for various values of d as described below. Likewise, the action space A = .., no - 1), with na taking a range of different values. Additional details regarding the reward function, behaviour policy T², and the estimation of weights (y)) have been included in Appendix F.2 for completeness.
Target policies
To investigate the effect increasing policy shift, we define a class of policies,
(alx) = a" I(a = arg max q(1, d')) + AAI where q(x, a) = E| X = = A a),
a'EA
where a* E [0, 1] allows us to control the shift between x and n² In particular, as we show later, the shift between x² and n² increases as a" + 1. Using the ground truth behaviour policy T², we generate a dataset which is split into training and evaluation datasets of sizes m and n respectively.
Baselines We compare our estimator with DM, IPW, DR and MIPS estimators. Our setup includes action embeddings E satisfying Assumption 3.5 and so MIPS is unbiased. Additional baselines have been considered in Appendix F.2 For MR, we split the training data to estimate # and (hy). whereas for all other baselines use the entire training data to estimate 7t for fair comparison.
Results We compute the target policy value using the n evaluation datapoints. Here, the MSE of the estimators is computed over 10 different sets of logged data replicated with different seeds. The results presented have context dimension 1000, number of actions n. = 100 and training data size 5000. More experiments for variety of parameter values are included in Appendix F2
Varying number of evaluation data n In Figure 2a we plot the results with increasing size of evaluation data increases. MR achieves the smallest MSE among all the baselines considered when n is small, with the MSE MR being at least an order of magnitude smaller than every baseline for 500. This shows that MR is significantly more accurate than the baselines when the size of the evaluation data small. As n -> oo, the difference between the results for MR and MIPS decreases. However, MR attains smaller variance and MSE than MIPS generally, verifying our analysis in Section Moreover, Figure 2a shows that while the variance MR greater than that of DM, still achieves the lowest MSE overall, owing the high bias of DM.

---

Varying a' As a" parameter of the target policy increases, so does the shift between the policies x" and π as illus- trated by the figure on the right, which plots the KL-divergence DRi. (T* Th ) as a function of Figure 2b plots the results for increasing policy shift. Overall, the MSE of MR estimator lowest among all the baselines. Moreover, while the MSE and variance of all estimators increase with increasing a" the increase in these quantities is lower for the MR estimator than for the other baselines. Therefore, the relative performance of MR estimator improves with increasing policy shift and MR remains robust to increase in policy shift.
16
12
4
0
0.0
0.2
0.4
0.6
0.8
1.0
a
Additional ablation studies In Appendix F.2 we investigate the effect of varying context dimen- sions d, number of actions and number of training data m. In every case, we observe that the MR estimator has a smaller MSE than all other baselines considered. In particular, MR remains robust to increasing n. whereas the MSE and variance of IPW and DR estimators degrade substantially when na 2 2000, Likewise, MR outperforms the baselines even when the training data size m is small.
Table 1: Mean squared error of target policy value with standard errors over 10 different seeds for different classification datasets. Here, number of evaluation data n 1000, and a² 0.6.
Dataset
Digics
Letter
OpiDigits
PenDigits
Suilmage
Minstt
CIFAR 100
0.0011:5000015
0.086-
0.001
0.0520=200001
0.1109:000014
0,0000+-00001
DR
0 0400
35.08541177 17 768
00464_0.0000
0. 0.2347:1:0 1404
0.05502000000
0:26172.00000 30
3823 2033
DR
0.0047:2.0025
0.23633--000866
0.088-00-0025
0.0138:: 0029
0,0078 0.0078 000000
02151:20.0051
0,262840 1087
JPW
45.25331:22.00
0.844-0.0056
0.1342-10.0311
0.3359:00:011
9:122777
SwitchDR
0.008::1000012
0.057-0000447
0.0142:1:00000 0090
0.00000001
0.3750:200000:
1,1644::0 8327
MR (Qurs)
0.0034::0.0001
0.0018::0.0004
0.0066=110002
0.0008:100000
0.8016::000003
0.0121:2000019
0.0007
5.2 Experiments on classification datasets
Following previous works on OPE in contextual bandits 13
22 36 5 we transform classification
datasets into contextual bandit feedback data in this experiment. We consider five UCI classification datasets 37 as well Mnist 38 and CIFAR-IO0 [39] datasets, each which comprises { a } where 2; E X' are feature vectors and at are the ground-truth labels. In the contextual bandits setup, the feature vectors H; are considered to be the contexts, whereas the actions correspond to the possible class labels. For the context vector 2; and the action dj the reward y; is defined as y/ = :14 ). i.e., the reward is when the action the same the ground truth label and 0 otherwise. Here, the baselines considered include the DM, IPW and DR estimators as well as Switch-DR (5] and DR with Optimistic Shrinkage (DRos) I7. We do not consider MIPS baseline here as there is no natural embedding E of A. Additional details are provided in Appendix F.3
In Table we present the results with number of evaluation data n 1000 and number of training data m = 500. The table shows that across all datasets, MR achieves the lowest MSE among all methods. Moreover, for the Letter and CIFAR-100 datasets the IPW and DR yield large bias and variance arising from poor policy estimates म². Despite this, the MR estimator which utilizes the same A for the estimation of u (v) leads much more accurate results. We also verify that MR outperforms the baselines for increasing policy shift and evaluation data n in Appendix F3
5.3 Application to ATE estimation
In this experiment, we investigate the empirical performance of the MR estimator for ATE estimation.
Twins dataset We use the Twins dataset studied in 40 which comprises data from twin births in the USA between 1989-19911 The treatment a corresponds to being born the heavier twin and the outcome Y corresponds to the mortality of each of the twins in their first year of life. Specifically, (1) corresponds to the mortality of the heavier twin (and likewise for (()). To simulate observational study, we follow similar strategy as in 40 to selectively hide one of the two twins as explained in Appendix [.4] We obtain a total of 1 11,984 datapoints, of which 5000 datapoints are used to train the behaviour policy 70 and outcome model g(x,a). a).

---

Table Mean absolute ATE estimation error CATE with standard errors over 10 different seeds, for increasing number of evaluation data 72.
200
1600
3200
DM
0.092=00,015
0:0922200003
0.002::00004
0.099:2000004
200022
0.065-10.099
0.01
0.100=0.011
0,089:200006
0.93::0.002
0:087=0.004
IPW
0.092200.024
0.088:200014
0.06::0.007
0.067:50.007
SwitcDDR
0.102
0.065-10.009
00000000005
0.069:20.004
0.065-200007
0.061:200005
0.001
Here, we consider the same baselines as the classification data experiments in previous section. For our evaluation, we consider the absolute error in ATE estimation, CXTE. defined as: EXTE = ATE ATT Here, O denotes the value of the ATE estimated using n evaluation datapoints. We compute the ATE value using the evaluation datapoints, over 10 different sets of observational data (using different seeds). Table 2 shows that MR achieves the lowest estimation error CAIE for all values of n considered here. While the performance of other baselines improves with increasing n, MR outperforms them all.
6 Discussion
In this paper, proposed OPE method for contextual bandits called marginal ratio (MR) estimator, which considers only the shift in the marginal distribution of the outcomes resulting from the policy shift. Our theoretical and empirical analysis showed that MR achieves better variance and MSE compared to the current state-of of-thee-rt methods and is more data efficient overall. Additionally, we demonstrated that MR applied to ATE estimation provides more accurate results than most commonly used methods. Next, we discuss limitations of our methodology and possible avenues for future work.
Limitations The MR estimator requires the additional step of estimating w(y) which may introduce an additional source of bias in the value estimation. However, (v) can be estimated by solving simple 1d regression problem, and as we show empirically in Appendix F.] MR achieves the smallest bias among all baselines considered in most cases. Most notably, our ablation study in Appendix F22 shows that even when the training data is reasonably small, MR outperforms the baselines considered.
Future work The MR estimator can also be applied to policy optimisation problems, where the policy. This approach has been used in
data collected using an 'old' policy is used to learn Proximal Policy Optimisation (PPO) [1 for example, which has gained immense popularity and has been applied to reinforcement learning with human feedback (RLHF) [4)). We believe that the MR estimator applied to these methodologies could lead to improvements in the stability and convergence of these optimisation schemes, given its favourable variance properties.
Acknowledgements
We would like to thank Jake Fawkes, Siu Lun Chau, Shahine Bouabid and Robert Hu for their useful feedback. We also appreciate the insights and constructive criticisms provided by the anonymous reviewers. MFT acknowledges his PhD funding from Google DeepMind.
References
[1] Lihong Li, Wei Chu, John Langford, and Robert Schapire. A contextual bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference
on World Wide Web, www '10, page 661-670, New York, NY, USA, 2010, Association for Computing Machinery. ISBN 9781605587999 doi: 10. 1115//177299..77777555 URL
114./117111179 1772758
[2] Hamsa Bastani and Mohsen Bayati. Online decision making with high-dimensional covariates. Operations Research, 68, 11 2019, doi: 10. 00.1287/prr..2019.1900
10

---

[3] Xiao Xu, Fang Dong, Yanghua Li, Shaojian He, and Xin Li. Contextual-bandee based personal- ized recommendation with time-varying user interests. Proceedings of the AAAI Conference on Artificial Intelligence, 34:6518 8-215,, 04 2020. doi: 10. 1a.16669aaii.44iiia x34.04.6125..
[4] Adith Swaminathan and Thorsten Joachims. Counterfactual risk minimization: Learning from logged bandit feedback. In Proceedings of the 32nd International Conference on International Conference on Machine Learning - Volume 37, ICML' 15, page 814-823. JMLR.org, 2015.
[5] Yu-Xiang Wang, Alekh Agarwal, and Miroslav Dudík. Optimal and adaptive off-policy evalua- tion in contextual bandits. In Proceedings of the 34th International Conference Machine Learning- Volume 70, ICML'17, page 5589-3597. JoLR.org, 2017.
[6] Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off- off-policy evaluation. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1447-1456. PMLR. 10-15 Jul 2018. URL JtRL/htts////roceeeeee
mlr. www.prresssveeee htta////aaaaaaaaaaaaaaaaaaaaaaaa 18ar html
[7]
Yi Su, Lequn Wang, Michele Santacatterina, and Thorsten Joachims. CAB: Continuous adaptive blending for policy evaluation and learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6015-6014. PMLR, 09-15 Jun 2019. URL httpps://proceedingg mlr ppwss/v911 vetee//v7///suu
/tness//97/su9a... ss19a, html
[8] Alberto Maria Metelli, Alessio Russo, and Marcello Restelli. Subgaussian and differ- entiable importance sampling for off-policy evaluation and learning. In M. Ranzato, A. Beygelzimer, Y. Dauphin. P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 8119-8132. Curran Associates, /apperrriieeeeppp/////////// pappr//022////rle
Inc., 2021. URL httpstttstttttt/tt http::t//p/oeeetannnngeee/eeeee hewp:/eeeeddiinnnnnneeeeeeeeeeee neurips. htcppappr___fffe pdf
[9] Anqi Liu, Hao Liu, Anima Anandkumar, and Yisong Yue. Triply robust off-policy evaluation, 2019. URL httpstsss: http::::/aaaxii..org///////1..1.
[10] Masashi Sugiyama and Motoaki Kawanabe. Machine Learning in Non-Stationary Environments: Introduction to Covariate Shift Adaptation. The MIT Press, 2012. ISBN 9780262017091. URL
http: http::/wwww..ssrrrr. jstorr org/ httgg//aall//j_j wtt.aa/ll///.ttttttttt
[11] Adith Swaminathan, Akshay Krishnamurthy, Alckh Agarwal, Miroslav Dudík, John Langford, Damien Jose, and Imed Zitoumi. Off- Off-policy evaluation for slate recommendation. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 3635-3645, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 10860964..
[12] D. G. Horvitz and D. Thompson. A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47(260):663-683 1952. ISSN 01621459. Uttptttp: http:://wwww. jstor www.ooo..rrr//ooe wtggsttable/2287883..
[13] Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. Doubly robust policy evaluation and optimization. Statistical Science, 29(4):485-5) -11, 2014. ISSN 08834237, 21688745. URL
jstor. httgg/stabbl/432884999
[14) Yuta Saito and Thorsten Joachims. Off-policy evaluation for large action spaces via embeddings. In Proceedings of the 39th International Conference on Machine Learning, pages 19089-19122. PMLR, 2022.
[15] Nan Jiang and Lihong Li. Doubly robust off off-policy value evaluation for reinforcement learn- ing. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference Machine Learning, volume 48 of Proceedings of Machine Learn- ing Research, pages 652-661, New York, New York, USA, 20-22 Jun 2016. PMLR. URL https::/prooeeedingg mlr prwsss/44//iaagl6. html
[16] Fan Li, Laine E Thomas, and Fan Li. Addressing Extreme Propensity Scores via the Overlap Weights. American Journal of Epidemiology, 185(1)250--577, 09 2018. ISSN 0002-9262. doi: 10. 10.1093/aje/www201. htLps:ts:: org/ 10. 1093/ kwy201

---

[17] Yi Su, Maria Dimakopoulou. Akshay Krishnamurthy, and Miroslav Dudík. Doubly robust off off-policy evaluation with shrinkage. In Proceedings of the 37th International Conference on Machine Learning, ICMI1'0.. Jor,.org, 2020.
[18] Arjun Sondhi, David Arbour, and Drew Dimmery. Balanced off-policy evaluation in general action spaces. In Silvia Chiappa and Roberto Calandra, editors, Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 2413-2423, PMLR, 26-28 Aug 2020. URL https //proceedings.mi/
2rress/v00/sonddiie
html
[19] Jianfa Lai, Manyun Xu, Rui Chen, and Qian Lin. Generalization ability of wide neural networks on R, 2023. https://///ax
[20] Judea Pearl. Causality. Cambridge University Press, 2 edition, 2009. doi: 10.1017/ CBO9780511803116
[21] Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite- horizon off- offlpo estimation. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: / http://ppoeeeen..ce/ec www.eeirnnnnnnnnnnnneeeee..eeeee httppscc/papppe cc/paper/2018/
pdf
[22] Nathan Kallus, Yuta Saito, and Masatoshi Uehara. Optimal off-policy evaluation from multiple logging policies. In International Conference on Machine Learning, pages 5247-5256. PMLR, 2021.
[23] Yuta Saito, Aihara Shunsuke, Matsutani Megumi, and Narita Yusuke. Open bandit dataset and pipeline: Towards realistic and reproducible off- off-policy evaluation. arXiv preprint arXiv:2008.07146, 2020.
[24] Philip S. Thomas and Emma Brunskill. Data-efficient off- teff-polic policy evaluation for rein- forcement Icarning. In Proceedings of the 33rd International Conference on International Conference Machine Learning Volume 48, ICML 16. page 2139-21111101 JMLR.org. 2016.
[25] Tengyang Xie, Yifei Ma. and Yu-Xiang Wang. Towards optimal off- off-policy evaluation for reinforcement learning with marginalized importance sampling. In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. / neurips. wwtppppppppppp////////// (tt.a////...00202222011000011010 Paper pdf
[26] Nathan Kallus and Masatoshi Uehara. Double reinforcement learning for efficient off- evaluation in markov decision processes. J. Mach. Learn. Res., 21(1), jun 2022. ISSN 1532. 4435.
[27] Yao Liu, Lierre-Luc Bacon, and Emma Brunskill. Understanding the curse of horizon in off- policy evaluation via conditional importance sampling. In Proceedings of the 37th International Conference on Machine Learning, ICML'20. JMLR.org, 2020.
[28] Cameron Voloshin, Hoang Minh Le, Nan Jiang, and Yisong Yue. Empirical study of off- policy policy evaluation for reinforcement learning. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021. URL https:://openreviee..nett Pttt://orrmmdddIIIiiiiia
[29] Noveen Sachdeva, Yi Su, and Thorsten Joachims. Off-policy bandits with deficient support. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20. page 975, New York, NY, USA, 2020. Association for Computing Machinery. ISBN 9781450379984, doi: 10.114553394488634033 340344444844443399 URL/htts:://do org/ 10. 10.1145/3994486 3403139
[30] Adith Swaminathan and Thorsten Joachims. The self-normalized estimator for counter- factual learning. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett, edi- tors, Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL htt.:htst//prrceedingge htwp:r/rceeeiiinnneeeirieerpepee cc/ wawperfiles/e/epeeppppppppeeeeee paailes/ppprr/21///iie// 33007 Paper, pdf
12

---

[31] Ben London and Ted Sandler. Bayesian counterfactual risk minimization. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Reseurch, pages 4125- URL hattps://proceedingg. .ml............................ w.w...eess.........m.m.........e https///////oooooon /1ndoo1dd
4133. PMIR, 09-15 Jun 2019.
html
[32] Scott Fujimoto, David Meger, and Doina Precup. A deep reinforcement learning approach to marginalized importance sampling with the successor representation. In Marina Meila and Tong Zhang. editors, Proceedings of the 38th International Conference on Machine Learning. volume 139 of Proceedings of Machine Learning Research, pages 3518 8--3929. PMLR, 18-24 Jul 2021. URL httphttt///ooceeeddnneee mlr. wwwpppprsssseepeepeeepp wwwp///ffffooool..11o html
[33] Mark Rowland, Anna Harutyunyan, Hado Hasselt, Diana Borsa, Tom Schaul, Rémi Munos, and Will Dabney. Conditional importance sampling for off- learning. In International Conference on Artificial Intelligence and Statistics, pages 45-55. PMLR, 2020.
[34] Xiao-Li Meng and Wing Hung Wong. Simulating ratios of normalizing constants via a simple identity: theoretical exploration. Statistica Sinicu, pages 831-860, 1996.
[35] Johann Brehmer, Gilles Louppe, Juan Pavez, and Kyle Cranmer. Mining gold from implicit models to improve likelihood free inference. Proceedings of the National Academy of Sciences, 117(10):5242-22) 2020.
[36] Mehrdad Farajtabar, Mohammad Ghavamzadeh, and Yinlam Chow. More robust doubly robust off-policy evaluation. 2018.
[37] Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL /ttp:///cccevv ics. uci. edu/ml
[38] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 1911)144 2012.
[39] Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
[40] Christos Louizos, Uri Shalit, Joris Mooij, David Sontag. Richard Zemel, and Max Welling. variable models. In Proceedings of the 3 International Conference on Neural Information Processing Systems, NIPS' 17, page 6449-6459, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964,
Causal effect inference with deep latent-variable
[41] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms, 2017.
[42] Nathan Lambert, Louis Castricato, Leandro von Werra, and Alex Havrilla. Illustrat- ing reinforcement learning from human feedback (rlhf). Hugging Face Blog, 2022.
https//uuggingfacc..oobog/////
[43] Junhong Lin, Alessandro Rudi, Lorenzo Rosasco, and Volkan Cevher. Optimal rates for spectral algorithms with least-squares regression over hilbert spaces. Applied and Computa- tional Harmonic Analysis, 48(3):868 -900, 2020. ISSN 1063-5203. doi: https///do.org//00
1016/ 009. URL https::/www.scitt http///dww.ciieeeddrreee.. com///ciiinnee///riceee. p11/$10635203188001778
[44] James Robins. A new approach to causal inference in mortality studies with sustained exposure period-applicationn to control of the healthy worker survivor effect. Mathematical Modelling, 70)): 1986. ISSN 0270-0255, doi: http:///oi.oogooooooooo/o////2/0 URL hwl.ittpss//ww..iieeddeeee wttp://wcccccc.coooooccccocoooop
[45] Leo Breiman. Random forests. Machine Learning 45(1):5-32, 2001.
doi: 10.1023/A:
1010933404324. URI.https: :/tb:p::///o// http://doooooogg/// 1023 wttp:////0.033///.11103333000000 10199334444111103444443433
[46] Whitney K. Newey and James R. Robins. Cross fitting and fast remainder rates for semipara-
metric estimation, 2018, URL Iittp:t///// http::tttt//aatttee
13

---

A Proofs
Proof of Lemma 3.7
First, we express the weights w(y) as the conditional expectation follows:
Pa- (y)
w(()
(x)
Pr (r, da
.x.AA Pnt
J Pr- (r, (ao a y) da dr
Pay(y) Pau y)
J Pr- (r-
X Pab (2, Pao (z. a y) da dr
Jr pla, x) Pa (2, v) da Ex,lo(A, Y y/.
X.
where x) (z,a,) x (ala) Since conditional expectations can be defined as the solution
P_ (x..,)) *aaa=)
of regression problem, the result follows.
Proof of Proposition 3.2
We have
Pa Px (X, A, Y) ]
Ps (X, A, x
Dy (px- (x, a, 4) Pa (x, 4)) ES
n (A | X)
E
7 X)
T" (A (AI XX)
B
Y
< X)
4 T]]
(A X)
2 E
f
E
(Jensen's inequality)
x)
Pr-
E
Ps+ (Y)
Dy (px- (y) (1pp++()))
Proof of Proposition 3.3
Since E,, [0pww] E,, [ŌMR] = Ex [Y], we have that,
Var [0pp]] Var [QMR] = E [0mew] Ey, [QMk])
1
(E, x) y² ETo (2 y2)
71
1
(E,, (p x)2 | Exp x2))
(E,, [e | - E [E. | x2))))
1
[Var [p(A, X) Y] y²]
n
In the second last step above, use the fact that w(y) = Ex (p( A, X) Y v).
14

---

Proof of Proposition 3.4 Let ja(a = B|y = x a] denote the outcome model in DR estimator. Then, using multiple applications of the law of total variance we get that
n Var.=10Dxx 3a p(A, (Y ((A,)) Σ ((d'.)) x xx)
a'EA
Var," [o(A, X) (Y - ()), + E X]]
EE,'/Var,,
X] X,
+ Var, (B, (Y
X] X,
=E,=lo(A, |X,]]]]
+ Va (P,,, X) (Y +((A.X))+ ( ( X) X X,
[E,
[X,
+ B. (0 + Var,+ [o,, (J(AAX) -)(
X) A(4, (44, X) x]]
[A x)) Var(Y |X,
+ Var, [E,
- I ( ((4, X] X]
+ E,, Var,+ (A(A, X) (H(A, - M(, + (p ((A,X) X XI]
E (p(1, X) Var X,] + Var,, [ [p( (A, m(A, X) [X]
+ E,, Var + (p(A, X) ()
XX]
2 E (MA, VarlY X, A]] + Var B (o A, X) ((A, XI].
Using this, we get that
n( Var, [dDR] Var, (RR])
2 En [,,, Again, using the law of total variance,
X, A]] + Var ()) (A, X]] - Var (ur Y..
Var,. [m(, X) = E.. [p( X) Y X, A]] + Var E ()), X, A]]
X)
= X)) Varl Y X]] + Var, X) XX]
[,), 2
VarlY [Y|
X, + Var,* EE. (p X) e (A, X]]
+ E [Var, 2 (p(A, X) ((A, X) X]]
Rearranging and substituting back into the expression earlier, we get that
T2( Var,, [PDR] Var, [0MR])
Var [p(A, X) Y E,, [Var + [p), X]] Var [ar.+ Y].
Now, from Proposition 3.3 we know that n Var,. [0pw]
Var [@MR]) = Var, Iar,, Var (Y = E (Var [p(A, X) Y1 y2].
Therefore.
n( VaVa [DDR] - Var,* [OMR])
2 E,, [ar, [p(A, Y| Y-] - E, (Var, ((AA,) ((44,) ((A,X) X]] = E. \Var [p(A, X), Y Y] ()A, p(A, X]].
Proof of Theorem
3.6 This result follows straightforwardly from Proposition D.4 in Appendix D
Proof of Proposition 3.7
Bias (f1Pw) ) = Ex, E. [Y]
E [E,+ Y] E,, [Y]
_x, - Ext E.- [Y]
Bias((@R))- E
15

---

Next, to prove the variance result, we first use the law of total variance to obtain
Var ge [01pw]
Var, (O,A, Y]
71
1
(Var,, [ [Ā(A, Y] + E. Var, (p(A, Y]])
71
(Var, (V) YJ + 18 Var I X) Y]]).
Moreover, using the fact that wYY = -Y) get that,
Var, [MR]
Varzy [10 (
71
(ar=:((@(()) +
(Var Y] + Var YJ (Y
Putting together the two variance expressions derived above, we get that
Var x [0Pw] - Var,* [0MR]
(E+ Var [X - -0v -
B Comparison with extensions of the doubly robust estimator
In this section, theoretically investigate the variance of MR against the commonly used extensions of the DR estimator, namely Switch-DR [5] and DR with Optimistic Shrinkage (DRos) 17. high level, these estimators seek reduce the variance of the vanilla DR estimator by considering modified importance weights, thereby trading off the variance for additional bias. Below, we provide the explicit definitions of these estimators for completeness.
Switch-DR estimator The original DR estimator can still have a high variance when the importance weights large due to a large policy shift. Switch-DR (5) aims circumvent this problem by switching to DM when the importance weights are large:
@SwitchDR
pla, xx) (y z;la:.;))1((po;; v.) S <) n((T))
n
where 2 0 is hyperparameter, Ma Y X x, A is the outcome model, and
i(TT)
Ala', ' x.)) ~ E, | X)]
1=1 a'EA
where of x;))
Doubly Robust with Optimal Shrinkage (DRos) DRos proposed by [17 uses new weights Zx(a; which directly minimises sharp bounds on the MSE of the resulting estimator,
1 Σ pa(a; xi)) (y: fa(a,, xz)) + i ")
n
where À 2 0 a pre-defined hyperparameter and Pu is defined as
A
,x(a,z) P(a,x) + p(a,x).
When À = 0, pi(a, æ)) leads to DM, whereas as X oc, (2, + p(a,2) leading to DR.
16

---

More generally, both of these estimators can be written as follows:
6R = 1
p( x;) (yt (ia;, zi)) f(TT).
1=1
Here, when p(a, x)) = x) 24) S T), we recover the Switch- estimator and likewise when p(a, x) = Px (2, x), we recover DRos.
B.1 Variance comparison with the DR extensions
Next, we provide a theoretical result comparing the variance of the MR estimator with these DR extension methods.
Proposition B.1. When the weights wyy) are known exactly and the outcome model exact, iee, ((aa.)) = (() = E|y X = z, A a) in the DR estimator ffR defined above,
Var,* [0 Var.. [0MR] 2 E [Var (p(A, LJ Y Var,o ()A,,) x]] A,
where A = n E (A) (2(X)))(((AAAA (A, VarlY A]]
Proof of Proposition B. that
Using the fact that ji(a, x) = u(a, x) and the law of total variance, we get
n1 Var," [0
Var, ()(A X) (Y ((A,))) + Σ Mla' X)T"
X)]
E'EA
B Var, [,(A, (Y - ((A, B. [A(A,, X) X]]
Var, (P(A,) - =(A, x(4 B Iu(A, X]]
Var,+ (E++,1EEEpA(AX (X- (() E. (A, X) X] X, A]]
+
+ E, (m(A, X) X) | X, X,
Var,+ [E,, E.+ VarlY AX)
Var,* [EE,. [p(A, [M
X]] + (2= (A, VarlY AI]
E (A, (A, VarlY X[
(E.,: ((A, X]] + Ep VarlY X, -nA.
Again, using the law of total variance can rewrite the second term the RHS above as,
E (A X) Var X, A]]
Var, [p(A, X) Y Var, (((4,))
Var [E,,,,,A, Y] + Ex [aa, (V Y
Var=+ ((A, X))
Var,+/u(()) Y| + E, Var,, (p(A, Y| Var, (o(A, ((A, ((A
Var, + EE, Var, (p(A, y| () M(A,
Putting this together, we get that
Var [
n Var,+ [MRR] + E Var, (p(A, Y Var ())
+ Var, E X)) AA X]] n A Y] - Io(A ((A, X]] n1, Ear
n Var, [OMR] + L [aar, (p(A,
where in the last step above, we again use the law of total variance. Rearranging the above leads us to the result.
17

---

Intuition Note that for both of the DR extensions under consideration, the modified ratios p(a. 2) satisfy 0 p(a,x) S p(a. 2) and hence A 2 0 (using the definition of A in Proposition B.1 When the modified ratios ((a.z) 2) are "close' to the true policy ratios p(a, 2.), then using the definition of A. we have that A 0. In this case, the result above provides a similar intuition to Proposition 3.4 the main text. Specifically, in this case we have that if Var,* [p(A, X) Y Y is greater than Var,. [p(A, X) [AA,, X] on average, the variance of the MR estimator will be less than that of the DR extension under consideration. Intuitively, this will occur when the dimension of context space X' is high because in this case the conditional variance over and A, Var," [p(A, (A, X) Y [Y] likely to be greater than the conditional variance over A, Var,, [p(4, (AA, XX X).
In contrast if the modified ratios p(a, z) differ substantially from p(a. x), then A will be large and the variance of MR may be higher than that of the resulting DR extension. However, this comes at the cost of significantly higher bias in the DR extension and consequently MSE of the DR extension will be high in this case.
C Weight estimation error
In this section, we theoretically investigate the effects of using the estimated importance weights (v)) rather than p(a, r) on the bias and variance of the resulting OPE estimator. Further to our discussion in Section 3.1.2 we focus in this section on the approximation error when using a wide neural network to estimate the weights (t)) To this end, we use recent results regarding the generalization of wide neural networks [19] to show that the estimation error the approximation step (ii) in the
Section 3.1.2 declines with increasing number of training data when (()) is estimated using wide neural networks. Before providing the main result, we explicitly lay out the assumptions needed.
C.1 Using wide neural networks to approximate the weights (3)
Assumption C.I. Let w == f ÍĀ X) Y = v)- Suppose 0 E H4 and || I S R for some constant R, where Hi is the reproducing kernel Hilbert space (RKHS) associated with the Neural Tangent Kernel K₁ associated with 2 layer neural network defined on R.
Assumption C.2. There exists an M [0, oo) such that ,, ( (|| M)
Assumption C.3. p(a,, 24) satisfies
Maa =(y)) + mh-
iid where V/s ~ (0,022) for some a >> O.
Theorem C.4. Suppose that the IPW and MR estimators are defined as,
Fiuw Σ plas, 24) Ves and 0MR
n
1
Σ Wm (y:) Va-
n
where wm (y) is obtained by regressing to the estimated policy ratios pla, x) using m i.i.d. training samples Dr = {(x!) ab ymm i.e., by minimising the loss
((Ø) = -EX.AYY) (O(A, - (300))f(())))
Suppose Assumptions C.1 C.3 hold, then for any given d E (0, fo is two-layer neural network with width & that is sufficiently large and stops the gradient flow at time t, x m2/3, then for sufficiently large m. there exists a constant C₁ independent of and m, such that
Bias( (BMR) Bias(@1ew)) S C1 m 1/3 log
holds with probability at least (1 88))) (0(11-xx(1))))) Moreover. there exist constants C2, C's independent of and m such that
n <Var [01p]] Var (DMRR)
2 E.p. [Var, + (A,,, Y] -C2 2/3 log²
C3 m 1/3 log
20
holds with probability at least (1 - 8)(1 (1)) Here, the randomness comes from the joint distribution of training samples and random initialization of parameters in the neural network fo.
18

---

Proof of Theorem C.4
The proof of this theorem relies on [9) Theorem 4.1]. Recall the definition
( == En, (p A. X) y.. We can rewrite our setup in the setting of [19 Theorem 4.1], by
relabelling p(a, x) in our setup as y in their setup and relabelling y in our setup as x in their setup.
Then, given 6 E (0, 1). from [19 Theorem 1. it follows that under Assumptions C.1 C.3
that there
exists constant C' independent of 8 and m. such that
B [c2] C 2/3 log² 6
(5)
8
holds with probability at least 3.7 that
where == úmm (Y)) Recall from Proposition
[Bias(OAR) Bias = |EEx+ Y]I-
From this it follows using Cauchy-Schwarz inequality that,
(0aR)( - Bias (f1rw) = |E - (Ea+|C"1,,t (x,2)) 1/2 Combining the above with Eqn. ( it follows that,
Bias MMR) Bias (f(w)) ) - C²/²2 m m1/3
1/3 log σ (B (x2))/22) = C1 m 1/3 log
holds with probability at least (1 (1 - oz))) where Ci C2/22 (E.. (Y2))1/23 Next, prove the variance result, we recall from Proposition 3.7 that
ni( o(Var... [Orww] Var,* (0MR)) E (,AA, X) Y? Var Y 2 Cov(e Y, YY)
Now note that, under Assumption C.2
Var (e Y| S [,, (())) < 1,,"" S CM2 m 2/3 log 2 6 8
C2 m 2/3 log 6 3
holds with probability at least (1 8) 0x (1)), where C2 CM. Similarly, we have that with probability at least (1 - () - (1)),
Y, (x)) = [e (Y) y² Ex+ (e
S |e (V) 1 + Rg (c B (V) Y]I
S (Enz (c (ū 11) 1/2 + (B |<" (E Yl
(B,+ (1)
x ( (E,,
(B, (1)
6 1/3 log 8¹
S SC3m m -1/3
where C3- = C (E.., (Y)) 11)11/2 +
(ύ)) Y]I. and we use Cauchy-Schwar inequality in the third step above. Putting this together, we obtain the required result.
Intuition This theorem shows that as the number of training samples m increases, the biases of MR and IPW estimators become roughly equal, whereas the variance of MR estimator falls below that of the IPW estimator. The empirical results shown in Appendix F.2 are consistent with this result. Moreover, in Theorem C.4 the estimated policy ratio ò(a, .)) is fixed for increasing m, i.e., do not update ò(a, 2) as more training data becomes available. While this may seem as a disadvantage for the IPW estimator, we point out that the result also holds when the policy ratio is exact (i.e.. p(a,x) = p(a, z)) and hence the IPW estimator is unbiased.
[heorem 4.1] suppose that the data has the relationship shown However, the theorem relies on Corollary 4.4 in [43), which requires a strictly weaker assumption (Assumption in 43). Therefore, we can relax Assumption C33 to the following assumption.
Relaxing Assumption C,3
in Assumption C.3
Assumption C.5. There exists positive constants Q and M such that for all / 2 with 1 E N
(,A, YJ
2 u1 1 Q²
Pa -lmost surely.
It is easy to check that Assumption C.5 is strictly weaker than Assumption C.3 and is also satisfied if the policy ratio i(A, X) is almost surely bounded. For simplicity, we use the stronger assumption in
our Proposition C.4
19

---

D Generalised formulation of the MIPS estimator
[4
As described in Section 3.1.I the MIPS estimator proposed by 14 assumes the existence of action embeddings E which summarise all relevant information about the action A, and achieves lower variance than the IPW estimator, To achieve this, the MIPS estimator only considers the shift in the distribution of (X, E)) result of policy shift, instead of considering the shift in (X, A) (as in IPW estimator). In this section, representations R of the context-action pair (X, A), which encapsulate all relevant information about the outcome Y. The MIPS estimator special case of this generalised setting where the representation R is of the form (X, F.).
show that this idea can be generalised to instead consider general
Generalised MIPS (G-MIPS) estimator Suppose that there exists an embedding of the context- action pair (X, A). with the Bayesian network shown in Figure Here, I may be lower-dimensional representation
(X, A) pair which contains all the information necessary to predict the outcome Y. This corresponds to the following conditional independence assumption:
Assumption D.I. The context-action pair (X, A) has no direct effect on the outcome Y given R, i.e. y IL (X, R.
As illustrated in Figure 3 Assumption D.I means that the embedding fully mediates every possible effect of (X, on Y. The generalised MIPS estimator fg. MIPS of target policy value, E, (Y), is defined as
(X
R
Figure 3: Bayesian network correspond-
0G-MIPS = Σ Pr (ri) P (ri)rs Dir
1 "
ing to Assumption D.I
where Pa (r) denote the density of under the behaviour policy (likewise for Pa- (r)). Under assumption D.I
0G-MIPS provides unbiased estimator of target policy value. Similar to Lemma 3.1 the density ratio Pet can be estimated by solving the regression problem (r)(r)
(A X) T X)
arg min E
S [
(6)
D.1 Variance reduction of G-MIPS estimator
By only considering the shift in the embedding R, the G-MIPS estimator achieves a lower variance relative to the vanilla IPW estimator. The following result, which is straightforward extension of [4 Theorem 3.6], formalises this.
Proposition D.2 (Variance reduction of G-MIPS), When the ratios p(a, x)) and Ps (r)
p.)(r) are known
exactly then under Assumption D.1 we have that Eg [0pPw] E., [0+MMMMS
Eg [Y]. Moreover,
Var, [1pw]] Var [GGMMPs] 2 E, E
Var, [p(A,
R 2 0.
Proof of Proposition D.2
The following proof, which is included for completeness, straightfor-
ward extension of [14 Theorem 3.6].
n , [0prw]
Var (Bnaus))
XA (AIX)
Pr
(R)
Var,,*
Y
Var
Y
x
P
(R)
(A
T² (A Y YR *((A)x
Pr
R
Var,
E
Y R
E
Var
Var
E
Y
R
n (AIX)
P
R
Par RR)
Var
Y
Pa+ (I2)
20

---

Now using the conditional independence Assumption
D.1
the first term on the RHS above becomes,
(A(X) T (A[X)
(A{X)
Var
E
Y
R
,
R
Ex [2]
Px-
R
1 [Y [2]]
Pa
(t)
where in the last step above we use the fact that
(A(X)
Px (R)
B
R
T6 (AX)
prt(R) (R)
Putting this together, we get that
n Var, [01p]]
Var,, (8MUs))
(AIX)
Pr R
Ex
V
Y R
E,d
Var
Y R
Pe. (R)
Since we have that
(A[X) (A[x)
(AIX) x4
Pr (R2)
E
y
R
Egg
R
T [Y
E.g (Y
Pxb Pxa(R) (R)
Eq.
becomes,
T (AIX) (A(X)
Pr- R
E
Var n
Y R
E,p
Var
Y R
pxb R)
(AIX)
Px- R
E
E
R
R
Pope (R)
(AIX)
Per (Ht)
Ezb
Ex''
12
[Y218]]
EE [
P (R)
(AIX)
(A|x)
Esb
En [Y
E
R
(AIX)
(AA
(AAA)) *((A))
Exb
Exb Y
R
Varga
R
Intuition Here, R. contains all relevant information regarding the outcome Y. Moreover, intuitively R can be thought of as the state obtained by 'ilttering out' relevant information about Y from (X, A). Therefore, R contains less 'redundant" information regarding the outcome Y compared to the covariate-action pair (X,A) A). As a result, the G-MIPS estimator which only considers the shift in the marginal distribution of R due to the policy shift is considers the shift the joint distribution of (X, A) instead. In fact, as the amount 'redundant' information regarding decreases in the embedding R, the G-MIPS estimator becomes increasingly efficient with decreasing variance. We formalise this as follows:
efficient than the IPW estimator, which
Assumption D.3. Assume there exist embeddings R(), R( of the covariate-action pair (X, with Bayesian network shown in Figure This corresponds to the following conditional independence assumptions:
R I1 (X, R
and
Y IT (x,))
X, R(2)
21

---

(X,A)
R
R(2)
Y
Figure 4: Bayesian network corresponding to Assumption D.3
We can define G-MIPS estimators for these embeddings to obtain unbiased OPE estimators under Assumption D.3 as follows:
Px (r:1)) Pa (1())) Vir
(
1
G MIPS
71 i=1
(2(5)) for [1,2]. Here, (r())
is the ratio of marginal densities of R under target and behaviour م
policies. We next show that the variance of G-MIPS decreases with increasing
Proposition D.4. When the ratios p(a, w(y) and Px (166))
Pes(F()) are known exactly for j E f1, 21, then
under Assumption D.3 we get that
E b [0PPw] ET (0 M.PPs
E, (0G2) MIPS E,z [OMR] ET [Y].
Moreover,
Var [0pww] Var (6(1)
(-MMPs] Var, (0(2) @.MMPs 2 Var, [0Aa]]
Proof of Proposition D.4 on We define R(()= (X, A) and ((0) GAMPPS defined
First, we prove that the G-MIPS estimators are unbiased using induction
1 Px- (r: (0) 1= Pu (r (0) Vi:
(0)
G-MIPS
=
recovers the IPW estimator Õppw. When O, we know that 0:0) MIPS Õipw is unbiased. Now, assume that E, (0.0 = Ex, [Y].
Conditional on RU), R(++) does not depend on the policy. Therefore,
Pa- (r(1))
Ps. (r(i)) p(rli+1) ((مي Pa. (r(1), (,(1),,,++)) ,(,+1))
(r(1)
Pa (r(0) p(r(++1) r(6)) Px (r((), ,((+1))*
And therefore,
Pr- (x-5i+-1)
Pr. (r(1), ,.li+1)/
(es) Prr (r(i), r(++ p,(((() r(t+1) dr
Par (x(0+1))
Pa (r(5)
Pao (ro(r()) (r(ر) r(ا+1) dr
((t) Pro (r()))
(1(
R(++1 = ,0+11
E
P R())
22

---

Using this and the fact that RG)
11 Y
R(++), we get that
pr. (1e(++1))
gg
Eng
Y
G-MIPS
Prr (RG+1))
Px- (R(i+1))
En
Ex | 1yR
(R(++))
P
Pr. (R())
E
Enp
R(++1)
Ex aa (orl1/RRttt
[16+111
Px (x
Pr. (R(i)
XR6+1)
Exs
Px (RS)
P (R(i))
E
py (R()) Y
E
G-MIPS
= E,, [A].
Next, to prove the variance result we consider the difference
Varg* g
10
G-MIPS
Var, 10
Px (RG))
Px. (RG-1))
V
Y
Var
Pr (126))
Pro (1(641))
Psr (R(i))
Px (B(())
126+1)
R((+2))
Varzo
Exp
+ Bx
V
Px (RG))
pxo (125))
Px (R(-11)
Px (R(5+-11)
Var [V (20 (
Varg*
B. [V 18G+11]
Pa (R(++))
Pao (1G6+JJ)
where in the last step we use the law of total variance. Now, using the fact that RG) 1L Y PRR (iie we can rewrite the expression above as
Pa- Pr- (R())
Pa- (a
Var
R(+1)
E
+ Egg
Var
Pan Y (R++))
Px (RO2)
p-- (R(++))
B (Y RDOtij]
P (+((t)))))))
Var (Y (U++11))
Var
Eng
b 1)
Pab ((((())))))
Pr- (R(())
Y R(5+2)
Pa- (1Q6+1)
Var,o|Y R
E
Var à
En
Pzt (R0))
Pa (PG++1)
Moreover, again using the conditional independence R()) 11 Y RO+1), we can expand the first term in the expression above as follows:
Pr. (R()) Y 1((+11)
p (R())
E
Var,
E
E
R((+1)
Exo ( (11tii+1111
Pr (RGT)
p². (RG)
Pr- (R())
Esa
R(i+1)
E ( NYY(R6+2)1
P (RG))
Px (RG))
> Exp
E
Pao (R())
par (R(++))
Be (M(R(+22)
Px (R(++))
Pnr Pm (R(0+2)
2
Ex
Var |Y| |8++1)
Pa (RU-1))
23

---

Here, to get the inequality above, we use the fact that E[X²]] 2 (E[X))². Putting this together, we get that Var.+ G-Murs , G-MIPS 2 O.
Moreover, the result Var G.MPss 2 Var,' [QwR] follows straightforwardly from above by defining R(3) = Y. Then, the embeddings satisfy the causal structure
R R(1) R R($) Y. Using the result above, we know that Var 16 G-MIPs 2 Var ((()) G-MIP)) But now it is straightforward to see that ((3) G-MIP = ÖMR. and the result follows.
Intuition Here, R(i+1) can be thought as the embedding obtained by 'filtering out' relevant infor- mation about Y from R(i), As such, the amount of 'redundant" information regarding the outcome Y decreases successively along the sequence < (X, A)), R(),, R( As result, the G-MIPS es- timators which only consider the shift in the marginal distributions RD5) due policy shift become increasingly efficient with decreasing variance as increases. Define the representation R(3) = Y. then the corresponding G-MIPS estimator reduces to the MR estimator, i.c., ((3) G-MIPS = 0MR. More- over, this estimator has minimum variance among all the G-MIPS estimators (0 ( Josjsk. as the representation R( contains precisely the least amount of information necessary to obtain the outcome Y. In other words, Y itself serves as the "best embedding' of covariate-action pair R()) which contains all relevant information regarding Y. We verify this empirically in Appendix R2 by reproducing the experimental setup in [1] along with the MR baseline. Additionally, the MR estimator does not rely on assumptions like D.I for unbiasedness.
In addition to this, solving the regression problem in Eq. (6) will typically be more difficult when R is higher dimensional (as is likely to be the case for many choices embeddings R), leading to high bias. In contrast, for MR the embedding R = Y is one dimensional and therefore the regression problem is significantly easier to solve and yields lower bias. Our empirical results in Appendix confirm this.
D.2 Doubly robust G-MIPS estimators
Consider the setup for the G-MIPS estimator shown in Figure 3 In this case, we can derive doubly robust extension of the G-MIPS estimator, denoted as GM-DR, which uses an estimate of r] as a control variate decrease the variance of G-MIPS
the conditional mean (((r) E|Y estimator. This can be explicitly written as follows:
a Pa (rz)
ÖDM DR =
n Pxb (rs) (yi - ji(r:)) + v)(m²)
where fi(==) = Et EMER ((r') P= (r' x3)) is the analogue of the direct method. Here, R. denotes the space of the possible of the representations F Moreover, given the density p(r | a). we can compute p.r (r x)) using
Pr- (r 2))
Σ plr (2,a) z). a'EAA
It is straightforward to extend ideas from 13 to show that estimator dDM.dr is doubly robust in that
it will yield accurate value estimates if either the importance weights 2x- (r) the outcome model ū(r)) is well estimated.
(r)(r)
There is analogous DR extension of the MR estimator A consequence of considering the embedding
(as in MR) is that in this case we do not have an analogous doubly robust extension above. To why this is the case, note that when R Y. we get that ju(r) E Rt r] E|Y Y y] = y. If we substitute this ji(r) in (8), we are simply left with 7(mr") on the right hand side (as the first term cancels out). This means that the resulting estimator does retain
the doubly robust nature as we no longer obtain an accurate estimate if either the outcome model or the importance ratios well estimated.
the EvER be replaced with J.reR dr' when R is continuous

---

E Application to causal inference
In this section, we investigate the application of the MR estimator for the estimation of average treatment effect (ATE). In this setting, we suppose that A {0, 11, and the goal is to estimate ATE defined as follows:
ATE = (1) - Y())
Here, we use the potential outcomes notation to denote the outcome under a deterministic policy (a xx 1(a' a.) as (a)).
Specifically, the IPW estimator applied to ATE estimation yields:
ATE₁PW
(ATE(ai n32) x Vi.
1=1
where
1 [a 1) 1(a
PATE x)
x (a)
Similarly, the MR estimator be written
ATEMR (A)E(y)) x yis
where
WATE(Y) Par (v) Porcos (v) Pru
and 7 (a) (a' I²) = 1(d' = a) for a E 10.
Again, using the fact that (V) E[PATE () YJ. we can obtain WATe by minimising a simple mean-squared loss:
WATE arg min By 1(A = 1) 1(A=0) = 0) - -(M)
(AIX)
Proposition E.1 (Variance comparison with IPW ATE estimator). When the weights PATE(a, x) and WATE(y) are known exactly, we have that Var[ATEsat I Var Specifically,
Var [TE
Var[ATEMR]
E Var (PATE r 0.
Proof of Proposition E.1 We have
Var Var[ATE
Var Var[ATEMRl (Var(pAnE(A, VarfwArE (Y)
(9)
Using the tower law of variance, we get that
Var(pATE(A, = VErTEE (A X) Y Y] + (AA, Y]]
Va (A, X) Y Y + E[Var|pATE(A (A, Y Y
Var (Y) Y] + E ((Va(((AT(AAAAA YJ Y|y
Putting this together with 9 we obtain,
Var ATE Var[ATEMR] = VaVaaaaa Y|
which straightforwarddy leads to the result.
Given the above definitions, the IPW estimator for (a)( would only consider datapoints with A = as it weights the samples using the policy ratios 1(A (AIx* which are only non-
zero when a. This however not case with the MR estimator, as uses the weights Px- 1 (Y') which are not necessarily zero for A * a. Therefore, MR uses all evaluation
25

---

datapoints D when estimating B|Y (a)). The MR estimator therefore leads to more efficient use of evaluation data in this example. Likewise, the doubly robust (DR) estimator applied to ATE estimation yields,
1
n 72
1
A
(ATe(a;, x;)) (y: (aa, 2;)) Σ 72
2; ((0,
where ii(a, 2) se E|Y = a}. Like in classical off-policy evaluation, DR yields an accurate estimator of ATE when either the weights PATE(a, the outcome model i.e., ((,, E[Y X = 2, A a). are well estimated. However, despite this doubly robust nature of the estimator, we show that the variance of the DR estimator may be higher than that of the MR estimator in many cases. The following result formalises this variance comparison between the DR and MR estimators, and is analogous to the result in Proposition 3.4 derived for classical off-policy evaluation.
Proposition E.2 (Variance comparison with DR ATE estimator). When the weights PxIe(a, x) and (ATe(y) are known exactly.
Var ATEER] (ATTAT 2 - [Var (A, Y Y] - [PATE(A(AA X))(A, |X]]
where () B|Y A]
Proof of Proposition E.2
Using the law of total variance, we get that
n Var Var[ATE
Var[PATE X) +((,,)) A(O,
Var[EE[AATE (A -
M(, (Y
x)) (A(1, (00,))) [X,
Ra[ (ATTE A, () -M,, + -A(0, X)) X, XX, Var[/ATT (A, X) (n(A, X) -)(A,X))) + (ū(1,)) -A(0, X))]
+ EE X) Var(Y X.
Again, using the law of total variance we can rewrite the first term on the RHS above as,
Var[PATE (A, ( + (f(L, -(X X))]
Var[E[pATE ( A, ((AX))) (A(1, (00, X)) X]]
+ EE[aa[[ (A, X) (μ(A, (x(4,)))MAAX)) (i(1, M(0, X)) [X]
Var(E[[AATe X))
+ (A(L, A(0, X)) X]]
Var[E[PATE(AA X) (H(A,X) A(4,X)) + (ATE(A,
((A,)) ((A, []]
Var[B[pATT( A, X) (AA X]].
where, in the second last step above we use the fact that
(A X) X] = (), x).
Putting this together, we get that
n Var(ATEpRl 2
[ EE (A, X.
26

---

Therefore,
n [ATEER Var(ATEMRI)
VarEEE[PATE (A, X) (AA, X)) X + PI Var XX - Var ATEE (Y) Y]
Var[IE[PATE (A, X) (A(A,
X]] + VV X) X, A]] VarlwATE (Y)
Var[[E[PATE (A, X)) (),
X]] + VaripATE (A, X) Y] (A)( Y X,
Var(WAATE (V)) (Y)Y)
Var[EE[PATE (4, X)) (x) | X
VATE (A, VarEpA Y] E/Var(pNTT (A, y]
Ba (X) Y | XX]]] VarlWArE Y) Y]
Var[B[UPATE (,, [X] + VarlWATE (Y) Y] E| (A, fA
VarTEE/ATTE -Arr
Var[WATE Y]
X - Var[E[ X) X, A|] El E/Va (A, | Y
Var[pATE (A, XX) (A), V X) ((
X]] (Mt ((A
(AVV (A, Y]
E Var (P (A,X) Y] y² Var (ATEE ()) xx]
Proposition E.2 shows that if Var Y PATE (A, X) Y] is greater than Var [AATR (A, X) ()x()))(AAA))))) (A, X) X] on average, the variance of the MR estimator will be less than that of the DR estimator. Intuitively, this is likely to happen when the dimension of context space X' is high because in this case, the conditional variance over X and A, Var PATE(A, X) Y] likely be greater than the conditional variance over A, Var [PATE ())(,, X].
F Experimental Results
In this section, we provide additional experimental details for the results presented in the main text. We also include extensive experimental results to provide further empirical evidence in favour of the MR estimator.
Computational details We ran our experiments on Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz with 8GB RAM per core. We able to use 150 CPUs in parallel iterate over different configurations and seeds. However, we would like to note that for each run our algorithms only requires CPU and at most 30 minutes run as our neural networks are relatively small. Throughout our experiments, whenever the outcome Y was continuous, we used fully connected neural network with three hidden layers with 512, 256 and 32 nodes respectively (and ReLU activation function) to estimate the weights vv))). On the other hand, when the outcome is discrete we can directly estimate (v) (( (A, X) Y y] by calculating the sample mean of p(A, X) on samples with y. Additionally, for each configuration of parameters in our experiments, we ran experiments for 10 different seeds (in {0, 1.....9)).
F1 Alternative methodology of estimating MR
In addition to the OPE baselines like IPW, DM and DR estimators considered in the main text, we also include empirically investigate an alternative methodology of estimating MR. Below describe this methodology, denoted as 'MR (alt)', in greater detail:
F1.1
MR (alt)
Recall our definition of MR estimator:
1 n
ÔMR :
w(nn) yi-
71 i=1
27

---

In the main text, we propose estimating the weights w(y) first and using this to estimate θNR using the above expression. Alternativly,, we can estimate h(y) == y wyy) using
500) 27
(AAX) (A(()
h = arg min E y
Subsequently, the MR estimator can be written as:
ÔMR
TO Σ h(y)).
We refer this alternative methodology as 'MR-alt" and compare it empirically against the original methodology (which we simply refer to as 'MR'). In general, it is difficult to say which of the two methods will perform better. Intuitively speaking, in cases where the behaviour of the quantity ('(A))
="[] with varying Y is 'smoother' than that of ='(A(X), >'] the alternative method is expected to
perform better. Our empirical results in the next sections show that the relative performance of the
two methods varies for different data generating mechanisms.
F.2 Synthetic data experiments
Here, include additional experimental details for the synthetic data experiments presented in Section 5.1 for completeness. For this experiment, we use the same setup as the synthetic data experiment in [4]] reproduced by reusing their code with minor modifications.
Setup Here, we sample the d-dimensional context vectors z from standard normal distribution. The setup used also includes 3-dimensional categorical action embeddings E E, which are sampled from the following conditional distribution given action A
p a) II exp (aa,es))
Eeee, exp ("la,e')
which is independent of the context X. {aa.es) is set parameters sampled independently from the standard normal distribution. Each dimension of has cardinality of 10, i.e., & {1, ...,, 10).
Reward function
The expected reward is then defined as:
q(x, E)
Σ mhk (aJ M Fee + + x + 0 xee)s
where M,, 0g and dg are parameter matrices or vectors sampled from uniform distribution with range [-,, Zec is context vector corresponding to the k-th dimension of the action embedding, which is unobserved to the estimators. 77 specifies the importance of the k-th dimension of the action embedding, sampled from Dirichlet distribution so that D 17
Behaviour and target policies to q a) [Xg(XX E) A
The behaviour policy T is defined by applying the softmax function
X 2²
exp (-)(xr)))
la x) EN'E exp exp(pxpp) (-q(x, a'))
For the target policy, we define the class of parametric policies,
1- a
" (a|z) a" L(a arg max d')) d'a
[A]
where [0, controls the shift between the behaviour and target policies. shown in the main text, as ax 1. the shift between behaviour and target policies increases.
28

---

10000 Baselines In the main text, we compare MR with DM. IPW. DR and MIPS estimators. In addition to these baselines, here we also consider Switch-DR and DR with Optimistic Shrinkage (DRos) 17 Following [14], we use the random forest 45 along with 2-fold cross-fitting [46] to obtain
e(),e) for DR and DM methods. To estimate P=* x; e) for MIPS estimator, we use logistic
regression. We also include the results for MR estimated using the alternative methodology described in Section F.1.I We refer to this as 'MR (alt)'.
Estimation of behaviour policy R and marginal ratio ( We do not assume that the true behaviour policy T² is known, and therefore estimate R" using the available training data. For the MR estimator, we estimate the behaviour policy using a random forest classifier trained on 50% of the training data and use the rest of the training data to estimate the marginal ratios (v)) using multi-layer perceptrons (MLP). Moreover, for a fair comparison we use a different behaviour policy estimate H for all other baselines which is trained on the entire training data.
DRos
IPW
MIPS
SwitchDR
MR (aft)
MR
Squaree Rias
Varience
101
10
10-
10
10"
10
10-1
10-7
10
10
30
2000
6000
8000 10000
2000
5000 8000 10000
4000
0000
8000 10000
Number of evaluation data
Number of evaluation data
Number of evaluation data
(a)
1000, na
100, a" 0.8
Squared Rlas
Variance
107
10
10"4
10r3
10
20
10a
4000
10000
Number evalustion data
Number of evaluetion
Number evaluation data
(b)
5000, 1a. 250, a" 0.8
MSE
Squared Bias
Variance
10a
102
10
10-2
10-2
10-1
10
2000
8000
2000
5000
10000
4000
6000
evallation
of
Number data
(c) = 5000, Yt, = 250, a = 1.0
Figure 5: MSE with varying size of evaluation dataset for different choices of parameters.
F.2.1 Results
For this experiment, the results are computed over 10 different sets of logged data replicated with different seeds, and in Figures - we use a total of m 5000 training data.
shows that MR outperforms the other baselines, in terms of MSE and squared bias, when the number of evaluation data n < 1000. Additionally, we observe that in this experiment, MR estimated using our original methods ('MR'), yields better results than the alternative method of estimating MR ('MR (alt))). Moreover, while the variance of DM is lower than that of MR, the DM method has a high bias and consequently a high MSE. We note that while the difference between MSE and variance of MIPS and MR estimators decreases with increasing evaluation data size, MR still outperforms MIPS in terms of both MSE and variance.
Varying size of evaluation data Figure 5
29

---

10
10"
100
10
10
10-
(a)
100, Ra 100, 7
100
MSE
Squared Bias
Verfance
10
20-1
10
10
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
0.5
0.8
1.0
(b) d 100,
250,
100
MSE
Squared
Variance
10
201
10-
10"
10-
0.0
0.2
0.4
1.0
0.0
0.2
0.4
0.8
1.0
0.0
0.2
0.4
0.8
1.0
(c) d 1000, na 250, n
100
Figure 6: MSE with varying a* for different choices of parameters.
Varying a" Figure showss the results with increasing policy shift. It can be seen that overall MR methods achieve the smallest MSE with increasing policy shift. Moreover, the difference between MSE and variance of MR and IPW/DR methods increases with increasing policy shift, showing that MR performs especially better than these basclines when the difference between behaviour and target policies is large. Similarly. we observe in Figure 6 that as the shift between the behaviour and target policy increases with increasing a" so does the difference between the MSE and variance of MR and the MIPS estimators. This shows that generally MR outperforms MIPS estimator in terms of variance and MSE, and that MR performs especially better than MIPS the difference between behaviour and target policies increases.
Varying d and Figures 7 and 8 show that MR outperforms the other baselines as the context dimensions and/or number actions increase. In fact, these figures show that MR significantly robust to increasing dimensions of action and context spaces, whereas baselines like IPW and DR perform poorly in large action spaces.
Varying m Figure 10 shows the results with increasing number training data m. We again observe that the MR methods 'MR' and 'MR (alt)' outperforms the other baselines in terms of the MSE and squared bias even when the number of training data is low. Moreover, the variance of both the MR estimators continues improve with increasing number of training data.
In this experiment, we observe that overall 'MR (alt)' performs worse than the original MR estimator ('MR² in the figures). However, we observe in Appendix F.5 this does not happen consistently across all experiments, which suggests that the comparative performance the two MR methods depends on the data generating mechanism.
30

---

PW
SwitchD3
(alt)
Squared
Variance
304
10.
10"
307
10
10
10"4
10"
2000 4000
a000
10000
2000
4000
8000
10000
4000
10000
Context dimension df
Context dimension d
Context dimension d
(a) Tig
20, 200,
Squared Diaz
Variance
107
10"
10"
10
10-4
10-1
10
8000
10000
Context dimensian
Context dimension
Context dimension
(b) Yh..
100, 200, a
0.8
MSE
Squared Biss
Varience
10.
109
10
1.0
10-1
10
10
10"1
2000
4000
6000
B000
10000
2000
4000 dimension
5000
8000
10000
2000
4000
5000
8000
10000
dimension
dimension
(c) na
250,
200, a 0.8
Figure MSE with varying context dimensions d different choices of parameters.
Table 3: Mean-squared error results with 2 standard errors for synthetic data setup considered in Section [5_ with d = 5000, n, = 50, a* 0.8. We use fixed budget of datapoints (denoted by N) for each baseline and in the case of MR we use m = 2000 of the available datapoints to estimate (yy and the rest of data to evaluate the MR estimator (i.e. n = N 2000 for MR). In contrast, for IPW and MIPS since the importance ratios are already known, we use all of the N datapoints for evaluation of the off off-policy value (i.e. n for IPW and MIPS).
N
2800
3200
6400
10000
12000
GT weights p(a. x) and estimated reward model ((a, x) (mr 2000 used for training (a, 2) and N 2000 used evallaation)
DM
0.137:10.028
0.009100011
0.103-10012
0.093:20.010
0.009-1:00000
DR
0.227:20.065
100005
0:08:100022
0.024:00.11
0.015:1:0015
DRos
0.128200027
0077-00011
004920001
0.06320.014
00011100116
GT weights Gill of datepoints msed evalusioo)
IPW
0.236:20.162
0.237:20.022
D.06--00116
0.662200021
0067:00022
0.02520.011
0,025200011
0.044-10014
0.044-10014
Estimated weights to (a)) (i 2000 used for training and n. N 2000 used evalustion)
MR (Ouns)
0.045 000015
0.042:00004
0.008:10.020
0,049
0.017 1.0016
F.2.2 Known policy ratios p(a, 2)
Our previous setting of unknown importance policy ratios p(a, x) captures wide variety of real- world applications, ranging from health care to autonomous driving. In addition, to demonstrate the utility MR in settings with known p(a, x).p(e a, x)) and unknown w(y) (for our proposed method, MR), we have conducted additional experiments. Here, we use fixed budget of datapoints (denoted by N) for each baseline and for MR we allocate m 2000 of the available datapoints to estimate wi and use the remaining for evaluating the MR estimator (i.e., n N 2000 for MR). In contrast, IPW and MIPS (since the importance ratios are already known), we use all of the N datapoints to evaluate the off- off-policy value (i.c. n N for IPW and MIPS).
31

---

MIPS
SwitcnDR
Squared
10
10"
10"
100
10-4
10-4
20-1
1000
1500
2000
2000
500
1000
1500
Number of Actions
Number Actions
Number Actions
(a)
1000, Ti 100,
MSE
Squared Bias
Varience
10
10
107
10a
30-1
10:2
Number
Number Actions
Number Actions
(b)
1000, n 100, ax
0.8
MSE
Squared Bias
Varience
10a
10
10'
10-
10-1
107
10:2
10"
500
1000
1500
2000
500
-000
1500
2000
500
1000
1,000
2000
(c) d = 1000,
100, a 1.0
Figure MSE with varying number of actions n, for different choices of parameters.
The results included in Table show that MR achieves the smallest MSE among the basclines for N 6400. However, observe that the MSE of IPW, DR and MIPS (with true importance weights) falls below that of MR (with estimated weights w) when the data size N large enough (i.e., N 2 10, 000). This is be expected since IPW, DR and MIPS are unbiased (i.e., ground truth importance ratios p(a, x)) whereas MR uses estimated weights (1) (and hence may be biased). MR still performs the best when N < 6400.
Experiments on classification datasets
Here, conduct experiments four classification datasets, OptDigits, PenDigits, SatImage and Letter datasets from the UCI repository (37] the Digits dataset from scikit-learn library, as well as
the Mnist 38 and CIFAR-100 datasets 39
Setup Following previous works B 22 36 51, the classification datasets transformed to
contextual bandit feedback data. The classification dataset comprises {z,, a Lno Ji=1: where E X' are feature vectors and af A are the ground-truth labels. In the contextual bandits setup, the feature vectors I; are considered to be the contexts, whereas the actions correspond to the possible class of labels. We split the dataset into training and testing datasets of sizes m and respectively. We present the results for a range of different values of m and n.
Reward function A as:
Let X be a context with ground truth label A we define the reward for action
y= 1(A= A
Behaviour and target policies Using the m training datapoints, first train classifier f RIA which takes as input the feature vectors z; and outputs vector of softmax probabilities over
32

---

labels, i.e. the a-th component of the vector f(r), denoted as (f(z)) corresponds to the estimated probability P(4 a X z).
Next, we use f to define the ground truth behaviour policy,
(a x) = (f(z))
For the target policies, we use f to define a parametric class of target policies using a trained classifier f:
a² [A]
π x) a² (a = arg
a'E.A
where a" [0, 1]. A value of a" close to leads to near-deterministic and well-performing policy. As decreases, the policy gets increasingly worse and "noisy". In this experiment, we consider target policies x* x" for a" E {0.00,00 0.4.0.0..0.4....... 1.0).
Using the behaviour policy defined above, we generate the contextual bandits data described with training and evaluation datasets of sizes and respectively.
Estimation of behaviour policy 7 and marginal ratio vi(y) We do not assume that the behaviour policy x is known, and therefore estimate it using training data. To estimate the behaviour policy Rb we train random forest classifier using the training data. This estimate of behaviour policy is used for all the baselines in our experiment. Since the reward is binary, we can estimate the marginal ratios ri( E,, [Ā(A, (A, Y yl by directly estimating the sample mean of p( X) for datapoints with Y = y. We re-use the m training datapoints to estimate this sample mean.
Baselines We compare our estimator with Direct Method (DM), IPW and DR estimators. In addition, we also consider Switch-DR [5] and DR with Optimistic Shrinkage (DRos) [17]. To estimate q(z, a) for DM and DR estimators, binary). Moreover, because of the binary nature of Y, the alternative method of estimating MR yields the same estimator the original method, therefore we do not consider the two separately here. Additionally, in this experiment, we do not include MIPS (or G-MIPS) baseline, as there is no natural informative embedding E of the action A.
random forest classifiers (since reward Y is
F3311 Results
For this experiment, we compute the results over 10 different sets of logged data replicated with different seeds. Figures 11 17 show the results corresponding each baseline for the different datasets. It can be seen that across all datasets, the MR achieves the smallest MSE with increasing evaluation data size n. Moreover, across all datasets, MR attains the minimum MSE with relatively small number of evaluation data (n < 100).
Unlike the experiments in Section 5.1 we observe that the KL-divergence between target and behaviour policy decreases as a* increases (see Figure 9b. Therefore, as a increases the shift between target and behaviour policies decreases. Figures II 16 show that a* increases, the difference between the MSE, squared bias and variance of MR and the other baselines decreases. This confirms findings from earlier experiments that MR performs especially better than the other baselines when the difference between behaviour and target policies is large.
Moreover, the figures also include results with increasing number of training data m. It can be seen that MR out-performs the baselines even when the number of training data m is small (m = 100). Moreover, the relative advantage of MR improves with increasing
E.4 Application to Average Treatment Effeet (ATE) estimation
In this subsection, we provide additional details for our experiment applying MR to the problem of ATE estimation presented in the main text. We begin by describing the dataset being used in this experiment.
Twins dataset We use the Twins dataset studied by 40 which comprises data from twin births in the USA between 1989-1991, The treatment a corresponds to being born the heavier twin and the outcome Y corresponds to the mortality of each of the twins in their first year of life. Since
33

---

16
12
Dxa(n ||
0.0
0.2
0.4
0.6
0.8
1.0
Figure 9: KL divergence (TT("T) -) with increasing o' for the classification data experiments. Here, we only include the results for specific choice of parameters for the Letter dataset. We observe similar results for other datasets and parameter choices.
DM
DA
DRos
JPW
MIPS
SwitchDR
MR (ait)
MR
MSE
Squared Dias
10'
Varience
10a
1.0
10
10"
10P
10-4
10"4
2000 4000 6000
8000
10000
2000
4030 5000 8000 10000
2000
4000
6000
8000
10000
Number of training data
Number of training data
Number of training data
(a) d 1000, 10, a
0.8
Squared Bias
10
10-
10
10-
10a
10-1
2000 4000 6000
8000
10000
2000 5000
10000
4000 Number of training data
6000
10000
Number training data
Number of training data
(b)
1000. Y 200,
0.8
Squared Binz
Varience
10
10
30-4
10
10
10
10-1
30-2
Number training 10000
Number training
Number
(c) d 1000, n 800, a 0.8
Figure 10: MSE with varying number of training data for different choices of parameters.
the data includes records for both twins, their outcomes would be considered as the two potential outcomes. Specifically., Y (1) corresponds to the mortality the heavier twin (and likewise for (0)). Closely following the methodology of [40], we only chose twins which are the same sex and weigh less than 2kgs. This provides us with a dataset of 1984 pairs twins.
The mortality rate for the lighter twin is 18.9% and for the heavier twin is 16.4%, leading to the ATE value being A = 2.5%. For each twin-pair we obtained 46 covariates relating to the parents, the pregnancy and birth.
Treatment assignment To simulate an observational study, we selectively hide one of the two twins by defining the treatment variable A which depends the feature GESTATIO. This feature, which takes integer values from 0 to 9, is obtained by grouping the number of gestation weeks prior to birth into 10 groups Then sample actions A as follows,
A X Bern Bemm(Z/10) Z/10),
34

---

DM
SwitchDR
Squared
Varionce
10
10
10
10-1
10"
30-1
10-
10
10
10
30
3000
4000
1000
2000
No of evallation data
No of evaluation data
No ovaluation data
(a) Results with varying for and m 1000
MSE
Squared Bies
Variance
10-3
30'7
10
10
10'
30-4
(h) Results with varying a for m n 1000
MSE
Squared Blas
Varibnce
10"
-0-1
10"
10
10
1508 training data
1000
2000
506
1000
1506
2000
500
1000
1500
2000
training data
of training data
(c) Results with varying m for n 1000 and a² 0.6
Figure 11: Results for OptDigits dataset
where Z is GESTATIO, and X are all the 46 features corresponding to twin pair (including GESTATIO).
Using the treatment assignments defined above, we generate the observational data by selectively hiding one of the two twins from each pair. Next, we randomly split this dataset into training and evaluation datasets of sizes and respectively. In this experiment, we consider 5000 training datapoints.
Baselines Recall that ATE estimation can be formulated the difference between off- off-policy values of deterministic policies r(1) := 1 (AA = 1) and (0) = 1(4 A = 0). Therefore, any OPE estimator can be applied ATE estimation. this experiment, we compare our estimator against the baselines considered in our OPE experiments in Section F3 This includes the Direct Method (DM), IPW and
DR estimators as well as Switch-DR (5) and with Optimistic Shrinkage (DRos) 177. estimate ((x, a) for DM and DR estimators, we use multi-layer perceptrons (MLP) trained on the training
datapoints, Additionally, we estimate the behaviour policy d using random forest classifier trained on the full training dataset.
Since the outcome in this experiment is binary, we estimate the weights (у) = E, (pA, Y y] directly by estimating the sample mean of ((A, X) for datapoints with Y y. This means that the alternative method of estimating MR yields the same value as the default method. We therefore do not consider these estimators separately. Additionally, since there is no natural embedding R of the covariate-action space which satisfies the conditional dependence Assumption D.I consider the G-MIPS MIPS) estimator either.
we do not
Performance metric defined
For our evaluation, we consider the absolute error in ATE estimation, 6XIE.
EATE = 1 (n)
OATEL
35

---

DM
SwitchDR
Varionce
10
10"
10"1
10
10-1
10'1
10"1
10
30
3000
4000
2000 No ovaluation data
1000
No of evaluation data
No of evaluation data
(a) Results with varying for and m
1000
107
MSE
10
Squared Bies
Variance
30-4
10"
10
10'
10"
30-4
(h) Results with varying a for m n 1000
MSE
Squared Blas
Varibnce
10"
30-1
10
10
30-2
1000 training data
1500
2000
500
1000
1506
2000
500
1000
1500
2000
training data
training data
(c) Results with varying for a 0.6 and 1000
Figure 12: Results for PenDigits dataset
Here, 0/m) denotes the value of the ATE estimated using evaluation datapoints. For example, for IPW estimator, the ((m) can written as:
0) 00
1(a ((a =10=1) 11 (aa==0) =0)
((m) ATE
1TE
Σ
2a; xza) ys.
i=1
All results for this experiment are provided in the main text.
F.5 Additional synthetic data experiments
In addition to the synthetic data experiments provided in Section 5.1 synthetic data setup to obtain further empirical evidence in favour of the MR estimator, and also compare it against the generalised version of the MIPS estimator (described as G-MIPS in Appendix DD.. Here, we use similar setup to [4] (albeit without action embeddings E)) where the d-dimensional context vectors x are sampled from standard normal distribution. Likewise, the action space is finite
we also consider an additional
and comprises n, actions, i.e. A = {O...., nl - 1}, with n, taking range different values. The reward function is defined as follows:
Reward function follows:
The expected reward q( a) E x, for these experiments defined as
q(x, a) sin ||rl|))
The reward Y is obtained by adding normal noise random variable to g(a, a)
Y = qXx +
where (0, 0.01). Here, it can be seen that conditional on R = (lIX]2,, A), the reward Y does not depend on (X, A), i.e., the embedding R satisfies the conditional independence assumption Y 1L (X R.
36

---

DM
SwitchDR
Squared
10
10"
107
10-1
10
10
10-
30-1
3000
4000
4000
1000
2000
No of evaluation data
No of evaluation data
No evaluation data
(a) Results with varying for and m Squared Bies
1000
MSE
Variance
10
10
10
10-1
10
10-1
(b) Results with varying a for n
1000
MSE
Squared Blas
Variance
100
10
10-3
30
10"4
10'
10
10a
1508 training data
1000
2000
500
1000
1506
2000
500
1000
1500
2000
training data
of training data
(c) Results with varying for a 0.6 and 1000
Figure 13: Results for SatImage dataset
Behaviour and target policies We first define behaviour policy by applying softmax function to q(z, a) as
exp (xp(((2,0)
ra x)) Evres exp (q(2, a'))
Just like in Section 5.1 to investigate the effect of increasing policy shift, we define class of policies,
(a|z) = a (a = arg max q(x, d')) + LA e'EA
where q(2, = E X = A = a),
where a* [0, allows to control the shift between T² and n² Again, the shift between n² and r increases a* - 1. Using the ground truth behaviour policy n², we generate dataset which is split into training and evaluation datasets of sizes m and respectively.
In Figures 18 21 we present the results for this experimental setup for different choices of paramater
configurations.
Estimation of behaviour policy 7 and marginal ratio ((y) For the MR estimator, we estimate the behaviour policy using a random forest classifier trained on 50% of the training data and use the rest of the training data to estimate the marginal ratios w()) using multi-layer perceptrons (MLP). Moreover, for a fair comparison we use a different behaviour policy estimate 76 for all other baselines which is trained on the entire training data.
Additional Baselines In addition to the baselines considered in the main text (Section 5.1 we also consider Switch-DR [5] and DR with Optimistic Shrinkage (DRos) [17]. addition, we also include the results for MR estimated using the alternative method ("MR (alt)') outlined in Section F.I.I For
37

---

DM
SwitchDR
Squared
Varionce
10-
10
10"
20-4
10
10
10
10-4
3000
4000
2000 No evaluation data
1000
No of evallation data
No of evaluation data
(a) Results with varying for and m Squared Bies
1000
MSE
Variance
107
101
102
10
10-1
10'1
(h) Results with varying a for m 7
1000
MSE
Squared Blas
Varionce
10a
10-
101
10-1
10
-0-
1508 training data
1000
2000
506
1000
1506
2000
500
1000
1500
2000
training data
of training data
(c) Results with varying for a 0.6 and
1000
Figure 14: Results for Letter dataset
the G-MIPS estimator (defined in Appendix D) considered here, we use R = (a, ||0||2 To estimate ((x, a) for DM and DR estimators, multi-layer perceptrons (MLPs).
E.5.1
Results
For this experiment, the results are computed 10 different sets of logged data replicated with different seeds, and Figures [8] 21 we use a total of m = 5000 training data.
Varying n Figure 18 shows that MR outperforms the other baselines, terms of MSE and squared bias, when the number of evaluation data < 1000. Additionally, observe that in this experiment, MR esitmated using alternative methods, MR (alt), yields better results than the original method of estimating MR. Moreover, while the variance of DM is lower than that of MR, the DM method has high bias and consequently a high MSE
Varying a* Figure 19 shows the results with increasing policy shift. can seen that overall MR methods achieve the smallest MSE with increasing policy shift. Moreover, the difference between MSE and variance of MR and IPW/DR methods increases with increasing policy shift, showing that MR performs especially better than these basclines when the difference between behaviour and target policies is large.
Varying d and na Figures 20 2nd show that MR outperforms the other baselines as the context dimensions and/or number of actions increase. In fact, Figure 21 shows that MR is significantly robust to increasing action space, whereas baselines like IPW and DR perform poorly large action spaces.
'It is easy to see that in our setup. the embedding ft = (a, [ satisties the conditional independence assumption I1 (X, Rt needed for G-MIPS estimator be unbiased
38

---

DM
SwitchDR
Squared
10"3
10"
10'1
10-
10
3000
4000
2000 No evaluation data
1000
No of evallatioo data
No of evaluation data
(a) Results with varying for and m 1000
MSE
Squared Bies
Variance
-0
10"
10-1
10
10"
30
10.14
(h) Results with varying a for m n
1000
MSE
Squared Blas
Variance
10"1
10
10
10
10
500
1508 training data
1000
2000
500
1000
1506
2000
500
1000
1500
2000
training data
of training data
(c) Results with varying m for a 0.6 and
1000
Figure 15: Results for Mnist dataset
Varying m Figure 22 shows the results with increasing number training data m. We again observe that the MR methods 'MR' and 'MR (alt)' outperforms the other baselines in terms of the MSE and squared bias even when the number of training data is low. Moreover, the variance of both the MR estimators continues to improve with increasing number of training data.
Unlike our experimental results in Section estimator overall. This shows that one of these two methods is not better than the other consistently in all cases, and their relative performance depends the dataset under consideration.
F.2 'MR (alt)' performs better than the original MR
F.6 Self-normalised MR estimator
Seff-normalization trick has been used in practice to reduce the variance off-policy estimators 30 This technique is also applicable to the MR estimator, and leads the self- nef-nnommllize MR estimator (denoted as 6SNMR) defined as follows:
w(i)) j=1 W(,))
8NNMR
Ye.
We conducted experiments to investigate the effect of celf-normalisation on the performance of the IPW, DR and MR estimators. Figure shows results for three different choices of parameter configurations. Overall, we observe that in all settings, the MR and self-normalised MR (SNMR) estimator outperform all other baselines including the self-normalised IPW and DR estimators (denoted as SNIPW and SNDR respectively). Moreover, in some settings, where the importance ratios achieve very high values, self-normalisation can reduce the variance and MSE of the corresponding estimator (for example, Figure 23b). However, we also observe cases in which nelf-oormaiizaiion does not significantly change the results (Figure 23ah, or may even slightly worsen the MSE of estimators (Figure 23ch
39

---

DM
SwitchDR
10a
10"1
10-
10"
10
10"4
10
10
1000
10-
500
1000
200
800
No of evaluation data
No of evaluation data
No evaluation data
(a) Results with varying n
0.2 and m
MSE
Squared Bies
Veriance
10
10
10
10-3
10-1
(b) Results with varying a for 500 and m 1000
MSE
Squared Bias
Varlance
30
10-1
10"2
1.0
100
800
1000
200
600
1000
400
800
training
training
training
(c) Results with varying m for a 0.6 and 500
Figure 16: Results for Digits dataset. Note that compared other datasets we consider smaller maximum dataset sizes m. here as the total number of available datapoints was 1797.
40

---

DM
DR
DRos
IPW
SwitchDR
Squared
Variance
10
10"
20
10
10
400
6010
1000
600
1000
200
1000
No of evalustion data
No of evalustian data
No of evaluation data
Results with varying for 0.4 and
2000
107
10
Squared
Veriance
10a
10a
10-1
10'1
(b) Results with varying ax for 100 and m 2000
MSE
Squared Blas
Variance
10a
10
10
10-39
30
10-
10-21
30
1000 1500 2000 2500 3000 3500 4000 of training data
1000 1500 2000 2500 3000 3500 4000
1000
1500 2000 2500 3000 3500 4000 training data
training
(c) Results with varying for a 0.4 and 100
Figure 17: Results for CIFAR- 100 dataset.
DR
DRos
G-MIPS
IPW
SwitchDR
(elt)
MR
MSE
Squarcc Bias
Variance
100
10-9
10
10
30
10
10"
10
No evalustion deta
of evalustion
evelustion data
(a)
1000, n1.
100,
0.4.
MSE
Squa rod Bias
Variance
100"
20-
10
10
30
10"
10
No evalustion deta
of evalustion
evelluaiio data
(b) d 10000, n. 100, 0.4.
Figure 18: Results with varying size of evaluation dataset
41

---

DM
G-MIPS
SwitchDR
MR (alt)
MR
MSE
Variance
30-1
10
10
10
10
10
30-
10
10"
10a
1.0
0.0
1000, 7aa
100,
100.
MSC
10
Squared bies
Verience
10
10
(b) d = 10000, nh = 100, n = 100.
Figure 19: Results with varying a².
SwitchDR
Squaated
10
10
10
10
10
10
7080
4000
ARBM
8300
10000
2000
4000
6000 IINO 10000
2000
41000
6000
8000
16000
Context dimensions
Context dimensiors
Context dimensions
100,
100, a
0.4.
Squared blos
Variance
10-1
10
30
10
10-
10-4
10
10-1
2000
6000
8000
2008
6600
10000
40000
6000
16000
Context dimensions
Context dimensions
Context dimensions
The 500, n 100, a 0.4.
Figure 20: Results with varying context dimensions d.
42

---

MIPS
SwitchDR
(allt)
Bies
Veriance
10
10a
10"
10
10'4
1000 2000
3,00
4000
5000
1000
2000
3000
4000
5003
1000
2000
3000
4000
5000
Number of Actions
Number of Actions
Number of Actions
d 100, 100,
0.2.
Squared Bies
Veriance
10
10
10
10
1000 2000 3000
4000
5000
1000
2000
3000
4000
5000
1000
2000
3000
4000
5000
Number of Actions
Number of Actions
Number of Actions
(b) d 100, 100, a² 0.4.
Figure 21: Results with varying number of actions flg-
DM
DR
DRos
G MIPS
IPW
SwitcaDR
MR (alt)
MR
MSE
Souared Dias
Varience
10
10-
10
10-
2000 4000 6000
8000
10000
2000 4030 5000 B000 10000
2000 4000
6000
6000 10000
Number of training data
Number of training data
Number of training data
(a) d 5000.
100,
10, a 0.2.
Squared Diss
10-
10
10a1
10
2000 6000
8000 10000
2000 5000
10000
4000 5000
10000
Number training data
Number of training data
Number of training data
(b) 5000. 7 100, na 10, a 0.4.
Figure 22: Results with varying number of training data
43

---

JPW
SNDR
SNIP
SNMR
10
10
10
0.0
0.2
0.4
0.5
08
100
0.0
02
0.4
0.6
0.8
10
0.0
02
0.4
0.6
0.8
100
(a) d 10000, 200,
20,
5000.
MSE
Squored bias
30
10-1
10
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
1.0
(b) = 5000,
200, 7la
20, m
1000.
MSE
Squered bios
Veriance
10
10'1
10
(c) 10000, n 200, Bo. 20, m 5000.
Figure 23: Results for nelf-normalised estimators with varying target policy shift a for synthetic data setup considered in Section 5.1 Here, "SN" denotes self-normalised estimators.
44


Reviews:
[{"review_id": "A84yai6ATB", "reviewer": "Reviewer_2CGe", "summary": "This paper addresses the problem of off-policy evaluation (OPE) using a variant of the inverse propensity score (IPS) estimator on logged bandit data. The authors specifically aim to reduce the variance of IPS by focusing on the shift in the marginal distribution of rewards instead of the policies themselves, resulting in a new estimator called the Marginal Ratio (MR) estimator. The proposed estimator is extensively analyzed, including comparisons with other related works such as IPS, DR, and MIPS. Furthermore, the authors demonstrate the utility of the MR estimator in causal inference for estimating average treatment effects (ATE). The effectiveness of the MR estimator in OPE is evaluated through a standard experimental setup, involving both synthetic and semi-synthetic data.\n\n# POST-REBUTTAL:\nI have read the author’s rebuttal and the authors addressed all my concerns.", "questions": "- In terms of computational efficiency, how does the efficiency of the MR estimator compare to IPS?\n- Is there an efficient method to optimize the MR estimator with respect to the policy $\\pi$ (i.e., in the context of off-policy learning) considering that estimating $w(y)$ is required in each iteration during the optimization process?", "limitations": "The authors adequately addressed limitations in Section 6.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The clarity and quality of writing in this paper are solid. The authors not only address the standard contextual bandit setting but also extend their work to causal inference for estimating average treatment effects (ATE). The overall idea of considering the shift in the marginal distribution of outcomes instead of the policies themselves is simple and very interesting. The theoretical concepts are effectively explained, and the proposed estimator is comprehensively compared to standard estimators such as IPS, DR, and MIPS. To the best of my knowledge, the derived guarantees are original contributions. The authors employ a standard experimental setup to validate their claims and showcase the favorable performance of their estimator through an extensive set of experiments.", "weaknesses": "**Computational Efficiency:**\n\nThe proposed MR estimator introduces an additional step of estimating $w(y)$, which may pose computational challenges compared to IPS, where the importance weights $\\pi/\\pi_0$ are readily accessible. While this might be manageable in the context of off-policy evaluation, where a policy $\\pi$ is given and the goal is to estimate its value using $\\hat{\\theta}$ through IPS, DR, MIPS, or MR, it may become problematic in off-policy learning. In off-policy learning, we optimize $\\hat{\\theta} = \\hat{\\theta}(\\pi)$ with respect to $\\pi$ to find the policy that maximizes the reward. Then, performing a regression to estimate $w(y)$ in each optimization step with respect to $\\pi$ can be computationally demanding. This presents a notable challenge.\n\n**Experiments:**\n\nThe experimental setup presented in the paper is solid. However, it would be beneficial to incorporate more complex computer vision datasets such as Fashion-MNIST and CIFAR-100, as seen in recent studies (e.g., [1, 2]). These datasets feature larger action sets and contexts when converted to contextual bandit instances.\n\n**Related Work:**\n\nWhile not a weakness, it would be valuable to include citations for two recent papers [1, 2] that propose new corrections for the importance weights to reduce variance. In [1], the authors suggest clipping only the propensity scores $\\pi_0$ (as $\\rm{max}(\\pi_0, 0)$) instead of the importance weight $\\pi/\\pi_0$. Additionally, [2] introduces a smooth regularization of the importance weights by incorporating $\\pi/\\pi_0^\\alpha$, where $\\alpha \\in [0, 1]$. While it is not necessary to directly compare the proposed approach with these papers, acknowledging their existence and contributions would be appropriate.\n\n[1] Ben London and Ted Sandler. Bayesian counterfactual risk minimization. In International Conference on Machine Learning, pp. 4125–4133. PMLR, 2019. Preprint version arXiv:1806.11500.\n\n[2] Imad Aouali, Victor-Emmanuel Brunel, David Rohde, and Anna Korba. Exponential Smoothing for Off-Policy Learning. arXiv preprint arXiv:2305.15877, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their thoughtful comments on our contributions and soundness of our approach. Below we respond to the questions raised:\n\n> In terms of computational efficiency, how does the efficiency of the MR estimator compare to IPS?\n\nIn the setting of off-policy evaluation, computing the MR estimator involves an additional step of estimating the weights $\\hat{w}(y)$ (compared to the IPW estimator). When the outcome $Y$ is continuous, the estimation of weights $\\hat{w}(y)$ can be achieved by solving a simple regression over functions $f:\\mathbb{R}\\rightarrow \\mathbb{R}$ as we show in Lemma 3.1:\n$$\nw = \\arg\\min_{f} E_{\\pi^b}[(\\hat{\\rho}(A, X) - f(Y))^2].\n$$\nThis is a simple scalar to scalar regression, which does not impose any significant computational overhead, and in practice $\\hat{w}(Y)$ is well estimated given a moderately sized training data (around 1000 to 2000 datapoints). See Figure 10 in the appendix, for example. In the case when $Y$ is discrete, this estimation becomes even more straightforward as we can directly use the formulation $w(y) = E_{\\pi^b} [\\rho(A, X)\\mid Y=y]$ and estimate $w(y)$ by computing the empirical mean of $\\hat{\\rho}(A, X)$ over datapoints where $Y=y$.\n\n> The proposed MR estimator introduces an additional step of estimating $w(y)$, which may pose computational challenges compared to IPS, where the importance weights $\\pi/\\pi_0$ are readily accessible. While this might be manageable in the context of off-policy evaluation, where a policy $\\pi$ is given and the goal is to estimate its value using $\\hat{\\theta}$ through IPS, DR, MIPS, or MR, it may become problematic in off-policy learning. In off-policy learning, we optimize $\\hat{\\theta} = \\hat{\\theta}(\\pi)$ with respect to $\\pi$ to find the policy that maximizes the reward. Then, performing a regression to estimate \n$w(y)$ in each optimization step with respect to $\\pi$ can be computationally demanding. This presents a notable challenge.\nIs there an efficient method to optimize the MR estimator with respect to the policy $\\pi$ (i.e., in the context of off-policy learning) considering that estimating $w(y)$ is required in each iteration during the optimization process?\n\nWe emphasise that this paper mainly addresses the problem of off-policy evaluation. We agree that directly applying our methodology to off-policy learning will indeed incur additional computational overhead since we need to run additional optimisation to update our ratios $\\hat{w}(y)$ after every policy update. However, one possible way to avoid this computational overhead is to use ideas from transfer learning literature [Lu et al., 2021]. Specifically, Section 5.1 of [Lu et al., 2021] proposes a methodology which can be used to combine the two optimisation problems into one, and hence could be used to avoid running regression separately after each optimisation step. We reiterate, however, that our paper is specific to off-policy evaluation and extending this to off-policy learning is an interesting avenue for future research.\n\n> The experimental setup presented in the paper is solid. However, it would be beneficial to incorporate more complex computer vision datasets such as Fashion-MNIST and CIFAR-100, as seen in recent studies (e.g., [1, 2]). These datasets feature larger action sets and contexts when converted to contextual bandit instances.\n\nWe thank the reviewer for their suggestion. However, like the fashion-MNIST dataset, MNIST dataset included in our experiments also comprises 10 labels and $28 \\times 28$ images, and therefore the size of the action and context spaces is the same for both datasets. Additionally, our synthetic data setup also includes experiments with significantly larger actions spaces (of sizes up to 5000). We will consider adding additional experiments on the image data like CIFAR-100 as suggested in the updated version of our paper.\n\n> While not a weakness, it would be valuable to include citations for two recent papers [1, 2] that propose new corrections for the importance weights to reduce variance.\n\nThank you for this suggestion, we will cite these works in the updated version of our paper.\n\nWe hope our clarifications have addressed the reviewer's concerns, and kindly ask them to consider increasing their score.\n\nNan Lu, Tianyi Zhang, Tongtong Fang, Takeshi Teshima, Masashi Sugiyama. Rethinking Importance Weighting for Transfer Learning, 2021."}, {"review_id": "sMan7ut21A", "reviewer": "Reviewer_ceqD", "summary": "The paper looks at the problem of off-policy evaluation (OPE) in contextual bandits. \nThe problem consists of estimating the expected reward obtained from a policy different from the one that collected the data.\nThe authors first analyze classic OPE estimators used for this problem, such as Inverse Probability Weighting (IPW) and Doubly Robust (DR). The authors notice how such estimators have high variance (especially IPW), and this is due to their focus on policy shift. The authors propose a novel estimator, called Marginal Ratio (MR) estimator, which focuses on the shift of the marginal distribution of the rewards instead. The authors also theoretically show that, whenever the marginal ratio is known exactly, MR has lower or equal variance compared to IPW. The authors also derive a relationship between the variances of MR and DR. In this case, however, they show that it is not true that the variance of MR is always lower or equal to the one of DR. Furthermore, the authors show that MR has lower variance than Marginalized IPS (MIPS, a recently proposed estimator), under the same assumption used by MIPS. ", "questions": "Taken from the above section:\n\n- Do you think that the Self-Normalization trick is applicable to MR?", "limitations": "The authors have addressed the limitations of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Simple and interesting idea with a novel point of view\n- Clear presentation\n- Careful experimental analysis (also considering the appendix) and strong empirical performance\n", "weaknesses": "- Since the problem with IPW and DR is high variance, a really simple way to reduce the variance for such estimators is the Self-Normalization trick [1]. I would have preferred to see the empirical comparison also against the Self-Normalized variant of IPW and DR.\n- There is no investigation on whether the Self-Normalization trick is applicable to the proposed MR. This could further improve the performance of the proposed estimator (if applicable)\n- The experimental setting seems to be such that IPW has high variance, which is fine since it is a known characteristic of IPW. However, there are also settings where IPW performs well (see [2] for an in-depth review of the performance of state-of-the-art OPE estimators). It could have been useful to see a comparison between the baseline estimators and MR in such settings, even only in the appendix.\n\n\n[1] A. Swaminathan and T. Joachims, The self-normalized estimator for counterfactual learning, NeurIPS 2015.\n\n[2] Saito et al., Evaluating the Robustness of Off-Policy Evaluation, RecSys 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating the novel point of view presented in our paper and our careful experimental analysis. Below we address some of the questions raised: \n\n> Since the problem with IPW and DR is high variance, a really simple way to reduce the variance for such estimators is the Self-Normalization trick [1]. I would have preferred to see the empirical comparison also against the Self-Normalized variant of IPW and DR.\n\nThis is an interesting suggestion and indeed, self-normatlization trick is applicable to the MR estimator. Specifically, the self-normalized MR estimator (denoted as $\\hat{\\theta}_{\\text{SNMR}}$) can be written as follows:\n\n$$\n \\hat{\\theta}\\_{SNMR} \\coloneqq\\sum_{i=1}^n \\frac{\\hat{w}(Y_i)}{\\sum_{j=1}^n \\hat{w}(Y_j)}Y_i.\n$$\n\nWe conducted experiments to investigate the effect of self-normalisation on the performance of the IPW, DR and MR estimators as suggested. Figure 2 in our rebuttal show results for three different choices of parameter configurations. Overall, we observe that in all settings, the MR and self-normalised MR (SNMR) estimator outperform all other baselines including the self-normalised IPW and DR estimators (denoted as SNIPW and SNDR respectively).\nMoreover, in some settings, where the importance ratios $\\rho(a, x)$ achieve very high values, self-normalisation can reduce the variance and MSE of the corresponding estimator (for example, Figure 2b). However, we also observe cases in which self-normalization does not significantly change the results (Figure 2a), or may even slightly worsen the MSE of the estimators (Figure 2c). We thank the reviewer again for their insightful suggestion and will add these experiments in the updated version of our paper. It is worth noting that our investigation of self-normalization only considers the synthetic data setup, and we will include a more thorough investigation in the updated version of the paper.\n\n> The experimental setting seems to be such that IPW has high variance, which is fine since it is a known characteristic of IPW. However, there are also settings where IPW performs well (see [2] for an in-depth review of the performance of state-of-the-art OPE estimators). It could have been useful to see a comparison between the baseline estimators and MR in such settings, even only in the appendix.\n\nWe thank the reviewer for their suggestion and in the updated version we will include the settings considered in [Saito et al., 2021] where IPW performs relatively well. It is worth mentioning here that we conducted additional experiments with **known** policy ratios $\\rho(a, x)$ and $p(e\\mid x, a)$ (for IPW and MIPS estimators), and **estimated** marginal ratios $\\hat{w}(y)$ (for MR estimator). \nFor a fair comparison, we use a fixed budget of datapoints (denoted by $N$) for each baseline and in the case of MR we allocate $m=2000$ of the available datapoints to estimate $\\hat{w}(y)$ and the rest of data to evaluate the MR estimator (i.e. $n = N-2000$ for MR). In contrast, for IPW and MIPS since the importance ratios are already known, we use all of the $N$ datapoints for evaluation of the off-policy value (i.e. $n=N$ for IPW and MIPS).   \n\nThe results in Table 1 of our rebuttal document show that MR achieves the smallest MSE among the baselines for $N\\leq 6400$. However, we observe that the MSE of IPW, DR and MIPS (with true importance weights) falls below that of MR (with estimated weights $\\hat{w}$) when the data size $N$ is large enough (i.e., $N\\geq 10000$). This is to be expected since in this setting, the IPW, DR and MIPS estimators are unbiased (i.e., use ground truth importance ratios) whereas MR uses estimated weights (and hence may not be unbiased). MR, despite being at a disadvantage in this setting, still performs the best when $N\\leq 6400$.\n\nWe hope the above addressed the questions raised sufficiently."}, {"review_id": "zfNsMIT9M0", "reviewer": "Reviewer_LKfa", "summary": "This paper introduced a new off-policy evaluation (OPE) method called Marginal Ratio (MR) to contextual bandits problems. Conventional methods such as Inverse Probability Weighting (IPW) and Doubly Robust (DR) rely on estimating policy ratios, which could incur large variances when there is low overlap between the target policy and the behavior policy. \n\nVariance and bias analyses of the MR estimator are provided. Specifically, the MR estimator achieves smaller variance compared to both IPW and DR when the important weights $\\rho$ and marginal ratio $\\omega$ are known. When both of them are unknown, the authors proposed a method that can approximate the weights and showed that with high probability the MR estimator is more favorable than the IPW counterpart. \n\nExtensive numerical experiments are provided, where the MR estimator outperformed baselines in terms of the mean square losses while maintained small variances.", "questions": "1. Can the order in the term $\\mathcal{O}(m^{-1/3})$ be improved, e.g.,  $\\mathcal{O}(m^{-1/2})$? \n\n2. I am a bit confused by the connections to MIPS and the application to causal inference. As you mentioned, MIPS is trying to solve large action space problem while in causal inference the action space is usually binary. What is the benefits of using the MR estimator? Is it more applicable to large action space problems or small action space problems? Also, the connection and comparison to MIPS is not very straightforward to me as the final analyses do not explicitly depend on the action space size. Can you provide more explanations here?", "limitations": "N.A.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is overall well-written and easy to follow. The theorems, propositions and comparisons are well-organized. The authors provided a good motivation to study the MR estimator, which addressed the high variance issues when using IPW and DR in practice. The idea is novel and interesting and the simulation results showed great potentials of the proposed estimator.", "weaknesses": "The assumptions of proposition 3.3 and 3.4 are quite strong. Though the MR estimator achieves lower variances comparing to IPW and DR estimators, I think the assumption of knowing the ratio $\\omega$ is much stronger than knowing the importance weight $\\rho$. Thus, it is not a very fair comparison. \n\nWhen the behavior policy and the marginal ratio $\\omega$ are known, the authors showed that as the training data size $m$ increases, the biases of MR and IPW estimators are almost the same with high probability and the variance of MR estimator could be smaller than the IPW counterpart. I don't think this is a strong result as the MR estimator depends on the unknown importance ratio $\\rho$ and the order term $\\mathcal{O}(m^{-1/3})$ in the variance analysis could be very large in practice. \n\nOverall, I think the theoretical results the MR estimator are sound yet incremental compared to existing methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the novelty of our idea and that our ``simulation results show great potential'' of our method.  Below we clarify some of the questions raised by the reviewer:\n\n> [...] I think the assumption of knowing the ratio $w(y)$ is much stronger than knowing the importance weight $\\rho(a, x)$. Thus, it is not a very fair comparison.\n\nWhile our results in Propostions 3.3 and 3.4 assume that the importance weights $w(y)$ are known, we also include theoretical results in the case where the weights $w(y)$ are not known in Section 3.1.2 and Appendix C. Specifically, Proposition 3.7 intuitively shows that if our weights $\\hat{w}$ are estimated `well enough', then the variance of the MR estimator will be smaller than that of the IPW estimator while incurring little extra bias. We note that this result also holds for the case where $\\rho(a, x)$ is known exactly, whereas $w(y)$ is not. Our empirical results also corroborate this finding, as in all of our experiments the marginal ratio $\\hat{w}(y)$ are estimated (and not exact), and we observe that overall MR consistently achieves a lower variance than most of the baselines considered. \n\nFurthermore, we conducted additional experiments with **known** policy ratios $\\rho(a, x)$ and $p(e\\mid x, a)$ (for IPW and MIPS estimators), and **estimated** marginal ratios $\\hat{w}(y)$ (for MR estimator). For a fair comparison, we use a fixed budget of datapoints (denoted by $N$) for each baseline and in the case of MR we allocate $m=2000$ of the available datapoints to estimate $\\hat{w}(y)$ and the rest of data to evaluate the MR estimator (i.e. $n = N-2000$ for MR). In contrast, for IPW and MIPS since the importance ratios are already known, we use all of the $N$ datapoints for evaluation of the off-policy value (i.e. $n=N$ for IPW and MIPS).   \n\nThe results in Table 1 of our rebuttal document show that MR achieves the smallest MSE among the baselines for $N\\leq 6400$. However, we observe that the MSE of IPW, DR and MIPS (with true importance weights) falls below that of MR (with estimated weights $\\hat{w}$) when the data size $N$ is large enough (i.e., $N\\geq 10000$). This is to be expected since in this setting, the IPW, DR and MIPS estimators are unbiased (i.e., use ground truth importance ratios) whereas MR uses estimated weights (and hence may not be unbiased). MR still performs the best when $N\\leq 6400$.\n\n> When the behavior policy and the marginal ratio are known, [...] as the training data size increases, the biases of MR and IPW estimators are almost the same [...]. I don't think this is a strong result [...].\n\nWe thank the reviewer for their comment. We assume that the reviewer meant ``When the behavior policy and the marginal ratio \n are **unknown**, [...]''. \n We would like to note that the main purpose of Theorem C.1 (which provides the $\\mathcal{O}(m^{-1/3})$ order term), is to show that when estimating the marginal ratios $\\hat{w}(y)$, there exists one model class (i.e. neural networks) for which at least asymptotically in the number of training data, the MR estimator will achieve a lower variance than that of the IPW estimator, while incurring no additional bias. As we mention in Section 3.1.2, this theorem uses recent results regarding the generalization of 2-layer wide neural networks [Lai et al., 2023], and therefore may not capture the convergence rate for general models. \n In contrast, Proposition 3.7 which expresses the variance of MR in terms of the weight approximation error $\\epsilon$ is a general result which is not specific to a model class, and does not rely on any regularity assumptions (unlike Theorem C.1). \n We will further clarify this in the updated version of the paper.\n\n> Can the order in the term $\\mathcal{O}(m^{-1/3})$ be improved, e.g., $\\mathcal{O}(m^{-1/2})$?\n\nOur result uses Theorem 4.1 from [Lai et al., 2023] which theoretically investigates the generalization of wide neural networks with 1d inputs and outputs, as we mention in Section 3.1.2. This theorem provides the convergence rate of $\\mathcal{O}(m^{-1/3})$.\n Improving this convergence rate is not straightforward as it will require improving upon the results provided in [Lai et al., 2023]. This is nonetheless an interesting direction to explore for future work.\n\n> [...] What is the benefits of using the MR estimator? Is it more applicable to large action space problems or small action space problems? Also, [...] the final analyses do not explicitly depend on the action space size. Can you provide more explanations here?\n\nWe would like to emphasise that the performance of MR is not directly dependent on the size of the action space, but rather on the variance of policy ratios $\\rho(a, x)$.\n Specifically, our theoretical results in Propositions 3.3, 3.4 and 3.7 show that MR performs especially well relative to the IPW and DR estimators in cases where the policy ratios $\\rho(a, x)$ have high variance. This high variance in $\\rho(a, x)$ can stem from many reasons such as large action spaces [Saito \\& Joachims, 2022] as well as large policy shifts (which can often occur in the setting of causal inference when estimating ATE). This is also supported by our experimental results where we observe that when either the policy shift increases (e.g. Figure 2b) or the size of action space increases (e.g. Figure 8), the variance and MSE of all baselines increases but this increase is smallest for the MR estimator (among all the baselines considered).\n\nWe hope that we were able to address all the reviewer's questions in the above and hope that the reviewer would consider increasing their score.\n\n Jianfa Lai, Manyun Xu, Rui Chen, and Qian Lin. Generalization ability of wide neural networks on $\\mathbb{R}$, 2023. URL https://arxiv.org/abs/2302.05933.\n\n Yuta Saito and Thorsten Joachims. Off-policy evaluation for large action spaces via embeddings. In Proceedings of the 39th International Conference on Machine Learning, pages 19089–19122. PMLR, 2022."}, {"review_id": "gDdIEI1PFY", "reviewer": "Reviewer_KWAf", "summary": "This paper proposes a marginal OPE estimator that directly corrects the distribution shift wrt rewards instead of correcting the distribution shift wrt actions. The proposed estimator generalizes the idea of marginal IPS (MIPS) and achieves the minimum variance among them when the marginal importance weight is accurately estimated. Experiments on both synthetic and classification-to-bandit datasets demonstrate that the proposed estimator enables more accurate OPE than baseline estimators.", "questions": "The following questions are related to my concern discussed in Weakness 1.\n\n1. In which condition the marginal importance weight is accurately estimated?\n\n2. Could you provide an error analysis on the estimation of the marginal importance weight (especially when the reward is continuous)?\n\n3. How the marginal importance weight is parametrized and estimated? Are there any assumptions or regularization? How difficult the hyperparameter tuning is?", "limitations": "As discussed in Weakness 1, it would be valuable to discuss the potential difficulty in estimating the marginal importance weight when the reward is continuous.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of applying importance sampling on rewards rather than actions is reasonable from the context. The paper is easy to follow, and the contribution is clearly stated.\n\n2. The paper also provides a data-driven method to estimate the marginal importance weight, which is a simple and straightforward approach.\n\n3. The connection between MIPS, which estimate the expected outcome by applying marginal importance weight wrt action embeddings is discussed, and the theory suggests that the proposed MR achieve minimum variance among marginal estimators.", "weaknesses": "1. As author(s) discuss in the limitation, one potential concern lies in the accuracy of the marginal importance weight. In my understanding, the proposed method seems to work well when the reward is discrete, as an adequate amount of data is observed for each reward. However, when the reward is continuous, I guess sometimes only one importance weight is observed for each reward, which leads to the variance in the estimation of the marginal importance weight. This may lead to a similar difficulty as [Kallus&Zhou, 18]. I have several questions regarding this concern, which I list in the following “Questions” section.\n\nNathan Kallus and Angela Zhou. “Policy Evaluation and Optimization with Continuous Treatments” (AISTATS’18)\n\n2. The marginal importance weight on reward has not been discussed in the standard OPE in contextual bandits, but some similar ideas have been discussed in OPE of RL [Rowland et al., 20] and another OPE framework which estimates the cumulative distribution function of a policy [Xu et al., 22] [Wu et al., 23]. As discussions with these methods are important to clarify the novelty of this work, I suggest discussing connections between these papers in related work.\n\nMark Rowland, Anna Harutyunyan, Hado van Hasselt, Diana Borsa, Tom Schaul, Rémi Munos, and Will Dabney. “Conditional Importance Sampling for Off-Policy Learning” (ICML’20)\n\nYang Xu, Chengchun Shi, Shikai Luo, Lan Wang, and Rui Song. “Quantile Off-Policy Evaluation via Deep Conditional Generative Learning” (‘22)\n\nRunzhe Wu, Masatoshi Uehara, and Wen Sun. “Distributional Offline Policy Evaluation with Predictive Error Guarantee” (ICML’23)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time reviewing our paper and for appreciating the clarity of our paper. Below we clarify some of the questions raised:\n\n> As author(s) discuss in the limitation, one potential concern lies in the accuracy of the marginal importance weight. In my understanding, the proposed method seems to work well when the reward is discrete, as an adequate amount of data is observed for each reward. However, when the reward is continuous, I guess sometimes only one importance weight is observed for each reward, which leads to the variance in the estimation of the marginal importance weight. This may lead to a similar difficulty as [Kallus\\&Zhou, 18]... \nIn which condition the marginal importance weight is accurately estimated?\n\nWhen the reward $Y$ is continuous, the estimation of our marginal ratios $\\hat{w}(y)$ can be obtained by solving a simple 1d regression problem given a policy ratio estimate $\\hat{\\rho}$, as shown in the Eqn 3 in our paper:\n$$\nw = \\arg\\min_f E_{\\pi^b}[(\\hat{\\rho}(A, X) - f(Y))^2].\n$$\nThis is a straightforward regression problem since both, the inputs and outputs of $f$ are scalars. Moreover, this also means that given enough training data and an expressive enough model class (e.g. neural networks), we will be able to accurately estimate the marginal ratios $\\hat{w}$.\nIn our experiments (Section 5 in main paper and Appendix F), we consider two synthetic data setups where the reward is continuous. Figures 10 and 21 in the Appendix show that in both setups, even a moderately sized training dataset (roughly 1000 datapoints) is enough to estimate the marginal ratios well enough that the MR estimator outperforms all the baselines considered. \n\nIn comparison, the setup considered in [Kallus \\& Zhou, 18] is different as they consider continuous valued actions (in their setting, the reward may or may not be continuous). Consequently, the challenges posed in their setting are orthogonal to our setting because their methodology involves estimating the density of the continuous valued actions under observational data, which may be a difficult task in general.\nOur methodology in contrast only involves a simple scalar to scalar regression which is much easier to estimate in general. \n\n> Could you provide an error analysis on the estimation of the marginal importance weight (especially when the reward is continuous)?\n\nSection 3.1.2 and Appendix C in our paper provides theoretical analyses regarding weight estimation errors which hold when the reward is continuous. Specifically, let $\\epsilon \\coloneqq \\hat{w}(Y) - E_{\\pi^b}[\\hat{\\rho}(A, X)\\mid Y]$ denote the approximation error of weights $\\hat{w}(y)$.  In Appendix C, we use some recent results regarding the generalization of 2-layer wide neural networks [Lai et al., 2023] to show that, when using wide neural networks to estimate the weights $\\hat{w}(y)$, we have that\n$$E_{\\pi^b}[\\epsilon^2] \\leq \\mathcal{O}(m^{-2/3})$$\nholds with a high probability, where $m$ is the number of training data. \n\nIn addition, Proposition 3.7 shows how the variance and bias of the MR estimator depends on the weight approximation error $\\epsilon$. Specifically, the result shows that as $E_{\\pi^b}[\\epsilon^2]\\rightarrow 0$, MR achieves a lower variance than IPW, while incurring little extra bias. This result is general as it is not specific to any model class and does not require any strong assumptions.\n\n> How the marginal importance weight is parametrized and estimated? Are there any assumptions or regularization? How difficult the hyperparameter tuning is?\n\nThroughout our experiments, whenever the outcome $Y$ was continuous, we used a fully connected neural network with three hidden layers with 512, 256 and 32 nodes respectively (and ReLU activation function) to estimate the weights $\\hat{w}(y)$. Moreover, we used sklearn library with the default learning rate of 0.001, and $l_2$ regularization coefficient of 0.01 to perform the regression. We found that the same architecture and hyperparameters worked well across all of our experimental settings with continuous rewards, and therefore did not need to perform hyperparameter tuning separately for each setting or dataset. While we mention the model architecture details in Appendix F, we will also add the hyperparameter details in the updated version of our paper.\n\n> The marginal importance weight on reward has not been discussed in the standard OPE in contextual bandits, but some similar ideas have been discussed in OPE of RL [Rowland et al., 20] and another OPE framework which estimates the cumulative distribution function of a policy [Xu et al., 22] [Wu et al., 23]. As discussions with these methods are important to clarify the novelty of this work, I suggest discussing connections between these papers in related work.\n\nWe thank the reviewer for the relevant literature. We will add a detailed discussion regarding these works in the updated version of our paper.\n\nWe hope the above addressed the concerns raised by the reviewer and that the reviewer will consider increasing their score. \n\nJianfa Lai, Manyun Xu, Rui Chen, and Qian Lin. Generalization ability of wide neural networks on $\\mathbb{R}$, 2023. URL https://arxiv.org/abs/2302.05933."}, {"review_id": "KKRJLFrezr", "reviewer": "Reviewer_jTTf", "summary": "This paper addresses Off-Policy Evaluation (OPE) in contextual bandits, a significant problem in fields such as healthcare and personalized recommendation systems. OPE involves evaluating the performance of new policies using only existing data generated by a current policy, which can pose a challenge due to high variance in estimators, particularly in situations with large action or context spaces. The study proposes a new OPE estimator called the Marginal Ratio (MR) estimator, which mitigates this issue by considering the shift in the marginal distribution of the outcome resulting from the policy shift, rather than the policy shift itself. This approach makes the MR estimator more robust to increasing sizes of action and context spaces than existing methods like Inverse Probability Weighting (IPW) or Doubly Robust (DR).", "questions": "- Could you provide some additional experiment results comparing MR with an estimated weight $\\hat{w}(y)$ and IPS, DR, and MIPS with the true respective weights?\n\n\n- How does MR perform when $\\rho(x,a)$ takes some extreme values, making solving Eq.(3) hard? Specifically, I would love to see experiment results, for example with $n =5,000, n_a = 5000, \\alpha = 1.0$, which should produce the situation I am interested in. Figure 8 in the appendix is relevant, but $n$ is too small to observe large empirical $\\rho(x,a)$.\n\n\n- What would be the theoretical comparison between DM and MR? Why does MR perform much better than DM, even though both perform OPE by estimating some aspect of the reward distribution?\n\n\n- Can we extend MR to a doubly-robust version? How would it look, and how would its variance compare to that of DR with $\\rho(x,a)$?\n", "limitations": "The limitations of the work are stated in the final section and they look reasonable. \n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles the practically relevant problem of off-policy evaluation\n\n\n- The paper is well-written and it was easy to follow the main arguments. The intuition statements right after each theorem were also useful \n\n\n- The paper proposes a new estimator called the MR estimator, which defines the weight in terms of the shift in the conditional reward distribution between the logging and target policies and is expected to reduce the variance compared to IPW, DR, and MIPS. In an ideal case, MR can also be unbiased.\n\n\n- Advantages of the proposed method over a range of conventional methods such as IPW, DR, and MIPS are demonstrated in a comprehensive way\n", "weaknesses": "- Eq.(3) may be unstable in cases where the target and behavior policies greatly differ, and there are many actions where the variance of $\\rho(x,a)$ becomes very large.\n\n\n- It is intuitive that MR reduces the variance from IPS and MIPS, but it needs to estimate the weight in terms of the marginal reward distributions, which are always unknown. If we can estimate the reward distributions well, then we could simply rely on DM. Therefore, it would be necessary to formalize the advantages of MR over DM in the case of an unknown $w(y)$.\n\n\n- The experiment design could be improved. As I understand it, in the experiments, both $\\rho(x,a)$, $p(e|x,a)$, and $w(y)$ are estimated from the logged data. While it's true that $w(y)$ is always unknown, in many industry applications where we can control $\\pi_b$, $\\rho(x,a)$ of IPW&DR and $p(e|x,a)$ of MIPS are known. Therefore, it would be necessary to compare MR with estimated $\\rho(x,a)$ and IPS, DR, and MIPS with their respective true weights.\n\n\n- This is not critical, but there are a few concurrent work that aims to further improve MIPS so would be great to discuss these papers in a revised version:\n\n    - Jie Peng, Hao Zou, Jiashuo Liu, Shaoming Li, Yibao Jiang, Jian Pei, Peng Cui. Offline Policy Evaluation in Large Action Spaces via Outcome-Oriented Action Grouping. WWW2023. https://dl.acm.org/doi/abs/10.1145/3543507.3583448 \n\n    - Yuta Saito, Qingyang Ren, Thorsten Joachims. Off-Policy Evaluation for Large Action Spaces via Conjunct Effect Modeling. https://arxiv.org/abs/2305.08062 ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their useful comments and acknowledging the practical relevance of our proposed method. \nWe clarify some of the misunderstandings below. \n\n> Eq.(3) may be unstable in cases where the target and behavior policies greatly differ, and there are many actions where the variance of $\\rho(a, x)$ becomes very large.\n\nWe agree with the reviewer that the variance of $\\rho(A, X)$ might increase for large policy shifts. However, since Eq. (3) is a simple scalar to scalar regression it is easy to optimise in practice. Our experimental results which include settings where variance of $\\rho$ is large, show that even with 1000 to 2000 training datapoints we are able to estimate the weights $\\hat{w}$ well. For example, Figures 10 and 20 in the App.\n\nAdditionally, we emphasise that while an increase in the variance of $\\rho(a, x)$ could lead to an increase in the variance of MR, it can also lead to a **comparatively larger** increase in the variance of IPW and DR (see Prop. 3.3 and 3.4). Recall the Eq. 4 in Prop. 3.7 of our paper (after minor modifications): \n\n$$\nVar_{\\pi^b}[\\tilde\\theta_{IPW}] - Var_{\\pi^b}[\\tilde\\theta_{MR}]= \\frac{1}{n}E_{\\pi^b}[Var_{\\pi^b}[\\hat{\\rho}(A, X)\\mid Y]Y^2] + \\Delta.\n$$\n\nIntuitively $\\Delta$ is an error term arising from the approximation error of weights $\\hat{w}(Y)$ and $\\Delta \\rightarrow 0$ as $E_{\\pi^b}[(\\hat{w}(Y) - E_{\\pi^b}[\\hat{\\rho}(A, X)\\mid Y])^2] \\rightarrow 0$.\nHence as $Var_{\\pi^b}[\\hat{\\rho}(A, X)\\mid Y]$ increases, so will the difference $Var_{\\pi^b}[\\tilde\\theta_{IPW}] - Var_{\\pi^b}[\\tilde{\\theta}_{MR}]$. In other words, the variance of the IPW estimator is likely to increase relative to that of MR as the variance of $\\hat{\\rho}(A, X)$ increases. Our results confirm this claim. In all our experiments, using both synthetic and real-world data, we observed that the variance of the MR estimator increases with the policy shift. However, the increase is significantly smaller compared to other baselines, particularly IPW and DR methods. See Figures 2b, 6, and 18.\n\n> If we can estimate the reward distributions well, then we could simply rely on DM. [...] Why does MR perform much better than DM, even though both perform OPE by estimating some aspect of the reward distribution?\n\nWe emphasise that MR does not rely on estimating the target reward distribution directly (i.e. $p_{\\pi^\\ast}(y)$ or $p(y\\mid x, a)$).  Instead, the MR estimator involves estimating the ratio $w(y) = p_{\\pi^\\ast}(y)/p_{\\pi^b}(y)$ which, as shown in Lemma 3.1 can be estimated by solving a simple scalar to scalar regression:\n$$\nw = \\arg\\min_{f} E_{\\pi^b}[(\\hat{\\rho}(A, X) - f(Y))^2].\n$$\nIn contrast, DM involves estimating the expected conditional reward $\\mu(a, x)\\coloneqq E[Y\\mid X=x, A=a]$, which is obtained by regressing from $(X, A)$ to $Y$. Intuitively, this can be a significantly more challenging regression in practice when the $(X, A)$ space is high-dimensional, as compared to regressing from scalars to scalars (as in MR). Therefore DM might suffer from higher bias. \n\nWhen the action space is discrete, estimating $\\mu(a, x)$ requires using only datapoints with $A=a$. This is limiting, especially with many actions and few observed instances of some actions. Unlike this, MR doesn't need such data partitioning, making better use of training data. Section 5 and Appendix F show MR outperforms DM in MSE and bias.\n\n> The experiment design could be improved [...] Could you provide some additional experiment results comparing MR with an estimated weight $\\hat{w}(y)$ and IPS, DR, and MIPS with the true respective weights?\n\nOur setting of unknown importance policy ratios $\\rho(a, x)$ captures a wide variety of real-world applications, ranging form health-care to autonomous driving. \nIn addition, to demonstrate the utility of MR in settings with **known** $\\rho(a, x)$ and $p(e\\mid x, a)$ and **unknown** $w(y)$ (for our proposed method, MR), we have conducted additional experiments. Here, we use a fixed budget of datapoints (denoted by $N$) for each baseline and for MR we allocate $m=2000$ of the available datapoints to estimate $\\hat{w}(y)$ and used the remaining for evaluating the MR estimator (i.e., $n=N - 2000$ for MR). In contrast, for IPW and MIPS (since the importance ratios are already known), we use all of the $N$ datapoints to evaluate the off-policy value (i.e. $n=N$ for IPW and MIPS).\n\nThe results included in Table 1 in our rebuttal show that MR achieves the smallest MSE among the baselines for $N\\leq 6400$. However, we observe that the MSE of IPW, DR and MIPS (with true importance weights) falls below that of MR (with estimated weights $\\hat{w}$) when the data size $N$ is large enough (i.e., $N\\geq 10000$). This is to be expected since IPW, DR and MIPS are unbiased (i.e., use ground truth importance ratios $\\rho(a, x)$) whereas MR uses estimated weights $\\hat{w}(y)$ (and hence may be biased). MR still performs the best when $N\\leq 6400$.\n\n> I would love to see experiment results, for example with $n=5,000, n_a=5000, \\alpha=1.0$ [...]\n\nWe have included the requested results in Figure 1 of our rebuttal document. The figure shows that MR achieves the smallest MSE and bias among all the baselines considered even when $n_a = 5000$.\n\n> Can we extend MR to a doubly-robust version?\n\nWe thank the reviewer for this suggestion. We in fact discuss this in the Appendix D.2 (line 764), where we point out that the natural idea used to derive DR extensions of IPW does not work in the case of MR. However, this does not completely rule out the existence of **some** doubly-robust extension of MR. This is an interesting direction, which we leave for future work.\n\nWe hope that the above has addressed the questions raised by the reviewer adequately, and that the reviewer will consider raising their score."}]

Author rebuttal:
Firstly, we would like to thank the reviewers for taking the time to review our paper, appreciating the quality of our work and providing many insightful comments regarding it. To address some of the concerns raised, we have conducted additional experiments and included the results in the attached PDF file. We hope that our rebuttal along with the additional results answer the reviewers' questions, and that the reviewers will consider increasing their scores.

Meta-review: This paper explores the off-policy evaluation problem in bandits. In an effort to reduce the variance of existing off-policy evaluation (OPE) estimators, the authors propose a new estimator, the Marginal Ratio (MR) estimator, which focuses on shifts in the marginal distribution of outcomes. The authors offer robust analysis of the proposed method, demonstrating that the MR achieves lower variance than existing methodologies. 

The reviewers unanimously recommend acceptance of this paper, a stance with which I concur. However, this endorsement is conditional on the authors incorporating all the changes they've promised during the rebuttal phase into the final version of the paper. These include a comparison with overlooked related works, additional experiments on image datasets like CIFAR-100, among others.
Decision: Accept (poster)