SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Muhammad Faaiz Taufiq*
Rob Cornish
Jean-François Ton
Off-Policy Evaluation (OPE) in contextual bandits is crucial for assessing new policies using existing data without costly experimentation. However, current OPE methods, such as Inverse Probability Weighting (IPW) and Doubly Robust (DR) estimators, suffer from high variance, particularly in cases of low overlap between target and behavior policies or large action and context spaces. In this paper, we introduce a new OPE estimator for contextual bandits, the Marginal Ratio (MR) estimator, which focuses on the shift in the marginal distribution of outcomes  $Y$  instead of the policies themselves. Through rigorous theoretical analysis, we demonstrate the benefits of the MR estimator compared to conventional methods like IPW and DR in terms of variance reduction. Additionally, we establish a connection between the MR estimator and the state-of-the-art Marginalized Inverse Propensity Score (MIPS) estimator, proving that MR achieves lower variance among a generalized family of MIPS estimators. We further illustrate the utility of the MR estimator in causal inference settings, where it exhibits enhanced performance in estimating Average Treatment Effects (ATE). Our experiments on synthetic and real-world datasets corroborate our theoretical findings and highlight the practical advantages of the MR estimator in OPE for contextual bandits.
In contextual bandits, the objective is to select an action  $A$ , guided by contextual information  $X$ , to maximize the resulting outcome  $Y$ . This paradigm is prevalent in many real-world applications such as healthcare, personalized recommendation systems, or online advertising [1-3]. The objective is to perform actions, such as prescribing medication or recommending items, which lead to desired outcomes like improved patient health or higher click-through rates. Nonetheless, updating the policy presents challenges, as naively implementing a new, untested policy may raise ethical or financial concerns. For instance, prescribing a drug based on a new policy poses risks, as it may result in unexpected side effects. As a result, recent research [4-11] has concentrated on evaluating the performance of new policies (target policy) using only existing data that was generated using the current policy (behaviour policy). This problem is known as Off-Policy Evaluation (OPE).
Current OPE methods in contextual bandits, such as the Inverse Probability Weighting (IPW) [12] and Doubly Robust (DR) [13] estimators primarily account for the policy shift by re-weighting the
*Part of the work was done during an internship at ByteDance Research.
Correspondence to muhammad.taufiq@stats.ox.ac.uk and jeanfrancois@bytedance.com
data using the ratio of the target and behaviour polices to estimate the target policy value. This can be problematic as it may lead to high variance in the estimators in cases of substantial policy shifts. The issue is further exacerbated in situations with large action or context spaces [14], since in these cases the estimation of policy ratios is even more difficult leading to extreme bias and variance.
In this work we show that this problem of high variance in OPE can be alleviated by using methods which directly consider the shift in the marginal distribution of the outcome  $Y$  resulting from the policy shift, instead of considering the policy shift itself (as in IPW and DR). To this end, we propose a new OPE estimator for contextual bandits called the Marginal Ratio (MR) estimator, which weights the data directly based on the shift in the marginal distribution of outcomes  $Y$  and consequently is much more robust to increasing sizes of action and context spaces than existing methods like IPW or DR. Our extensive theoretical analyses show that MR enjoys better variance properties than the existing methods making it highly attractive for a variety of applications in addition to OPE. One such application is the estimation of Average Treatment Effect (ATE) in causal inference, for which we show that MR provides greater sample efficiency than the most commonly used methods.
Our contributions in this paper are as follows:
- Firstly, we introduce MR, an OPE estimator for contextual bandits, that focuses on the shift in the marginal distribution of  $Y$  rather than the joint distribution of  $(X, A, Y)$ . We show that MR has favourable theoretical properties compared to existing methods like IPW and DR. Our analysis also encompasses theory on the approximation errors of our estimator.
- Secondly, we explicitly lay out the connection between MR and Marginalized Inverse Propensity Score (MIPS) [14], a recent state-of-the-art contextual bandits OPE method, and prove that MR attains lowest variance among a generalized family of MIPS estimators.
- Thirdly, we show that the MR estimator can be applied in the setting of causal inference to estimate average treatment effects (ATE), and theoretically prove that the resulting estimator is more data-efficient with higher accuracy and lower variance than commonly used methods.
- Finally, we verify all our theoretical analyses through a variety of experiments on synthetic and real-world datasets and empirically demonstrate that the MR estimator achieves better overall performance compared to current state-of-the-art methods.
2.1 Setup and Notation
We consider the standard contextual bandit setting. Let  $X \in \mathcal{X}$  be a context vector (e.g., user features),  $A \in \mathcal{A}$  denote an action (e.g., recommended website to the user), and  $Y \in \mathcal{Y}$  denote a scalar reward or outcome (e.g., whether the user clicks on the website). The outcome and context are sampled from unknown probability distributions  $p(y \mid x, a)$  and  $p(x)$  respectively. Let  $\mathcal{D} := \{(x_i, a_i, y_i)\}_{i=1}^n$  be a historically logged dataset with  $n$  observations, generated by a (possibly unknown) behaviour policy  $\pi^b(a \mid x)$ . Specifically,  $\mathcal{D}$  consists of i.i.d. samples from the joint density under behaviour policy,
$$
p _ {\pi^ {b}} (x, a, y) := p (y \mid x, a) \pi^ {b} (a \mid x) p (x). \tag {1}
$$
We denote the joint density of  $(X, A, Y)$  under the target policy as
$$
p _ {\pi^ {*}} (x, a, y) := p (y \mid x, a) \pi^ {*} (a \mid x) p (x). \tag {2}
$$
Moreover, we use  $p_{\pi^b}(y)$  to denote the marginal density of  $Y$  under the behaviour policy,
$$
p _ {\pi^ {b}} (y) = \int_ {\mathcal {A} \times \mathcal {X}} p _ {\pi^ {b}} (x, a, y) d a d x,
$$
and likewise for the target policy  $\pi^{*}$ . Similarly, we use  $\mathbb{E}_{\pi^b}$  and  $\mathbb{E}_{\pi^*}$  to denote the expectations under the joint densities  $p_{\pi^b}(x,a,y)$  and  $p_{\pi^*}(x,a,y)$  respectively.
Off-policy evaluation (OPE) The main objective of OPE is to estimate the expectation of the outcome  $Y$  under a given target policy  $\pi^{*}$ , i.e.,  $\mathbb{E}_{\pi^{*}}[Y]$ , using only the logged data  $\mathcal{D}$ .
Throughout this work, we assume that the support of the target policy  $\pi^{*}$  is included in the support of the behaviour policy  $\pi^b$ . This is to ensure that importance sampling yields unbiased off-policy estimators, and is satisfied for exploratory behaviour policies such as the  $\epsilon$ -greedy policies.
Assumption 2.1 (Support). For any  $x \in \mathcal{X}, a \in \mathcal{A}, \pi^{*}(a \mid x) > 0 \implies \pi^{b}(a \mid x) > 0$ .
2.2 Existing off-policy evaluation methodologies
Next, we will present some of the most commonly used OPE estimators before outlining the limitations of these methodologies. This motivates our proposal of an alternative OPE estimator.
The value of the target policy can be expressed as the expectation of outcome  $Y$  under the target data distribution  $p_{\pi^*}(x, a, y)$ . However in most cases, we do not have access to samples from this target distribution and hence we have to resort to importance sampling methods.
Inverse Probability Weighting (IPW) estimator One way to compute the target policy value,  $\mathbb{E}_{\pi^*}[Y]$ , when only given data generated from  $p_{\pi^b}(x,a,y)$  is to rewrite the policy value as follows:
$$
\mathbb {E} _ {\pi^ {*}} [ Y ] = \int y p _ {\pi^ {*}} (x, a, y)   \mathrm {d} y   \mathrm {d} a   \mathrm {d} x = \int y \underbrace {\frac {p _ {\pi^ {*}} (x , a , y)}{p _ {\pi^ {b}} (x , a , y)}} _ {\rho (a, x)} p _ {\pi^ {b}} (x, a, y)   \mathrm {d} y   \mathrm {d} a   \mathrm {d} x = \mathbb {E} _ {\pi^ {b}}   [ Y   \rho (A, X) ],
$$
where  $\rho(a, x) := \frac{p_{\pi^*}(x, a, y)}{p_{\pi^b}(x, a, y)} = \frac{\pi^*(a|x)}{\pi^b(a|x)}$ , given the factorizations in Eqns. (1) and (2). This leads to the commonly used Inverse Probability Weighting (IPW) [12] estimator:
$$
\hat {\theta} _ {\mathrm {I P W}} := \frac {1}{n} \sum_ {i = 1} ^ {n} \rho \left(a _ {i}, x _ {i}\right) y _ {i}.
$$
When the behaviour policy is known, IPW is an unbiased and consistent estimator. However, it can suffer from high variance, especially as the overlap between the behaviour and target policies decreases.
Doubly Robust (DR) estimator To alleviate the high variance of IPW, [13] proposed a Doubly Robust (DR) estimator for OPE. DR uses an estimate of the conditional mean  $\hat{\mu}(a,x)\approx \mathbb{E}[Y\mid X = x,A = a]$  (outcome model), as a control variate to decrease the variance of IPW. It is also doubly robust in that it yields accurate value estimates if either the importance weights  $\rho(a,x)$  or the outcome model  $\hat{\mu}(a,x)$  is well estimated [13, 15]. The DR estimator for  $\mathbb{E}_{\pi^{*}}[Y]$  can be written as follows:
$$
\hat {\theta} _ {\mathrm {D R}} = \frac {1}{n} \sum_ {i = 1} ^ {n} \rho \left(a _ {i}, x _ {i}\right) \left(y _ {i} - \hat {\mu} \left(a _ {i}, x _ {i}\right)\right) + \hat {\eta} \left(\pi^ {*}\right),
$$
where  $\hat{\eta} (\pi^{*}) = \frac{1}{n}\sum_{i = 1}^{n}\sum_{a^{\prime}\in \mathcal{A}}\hat{\mu} (a^{\prime},x_{i})\pi^{*}(a^{\prime}\mid x_{i})\approx \mathbb{E}_{\pi^{*}}[\hat{\mu} (A,X)]$  . Here,  $\hat{\eta} (\pi^{*})$  is referred to as the Direct Method (DM) as it uses  $\hat{\mu} (a,x)$  directly to estimate target policy value.
2.3 Limitation of existing methodologies
To estimate the value of the target policy  $\pi^{*}$ , the existing methodologies consider the shift in the joint distribution of  $(X, A, Y)$  as a result of the policy shift (by weighting samples by policy ratios). As we show in Section 3.1, considering the joint shift can lead to inefficient policy evaluation and high variance especially as the policy shift increases [16]. Since our goal is to estimate  $\mathbb{E}_{\pi^{*}}[Y]$ , we will show in the next section that considering only the shift in the marginal distribution of the outcomes  $Y$  from  $p_{\pi^b}(Y)$  to  $p_{\pi^*}(Y)$ , leads to a more efficient OPE methodology compared to existing approaches.
To better comprehend why only considering the shift in the marginal distribution is advantageous, let us examine an extreme example where we assume that  $Y \perp A \mid X$ , i.e., the outcome  $Y$  for a user  $X$  is independent of the action  $A$  taken. In this specific instance,  $\mathbb{E}_{\pi^*}[Y] = \mathbb{E}_{\pi^b}[Y] \approx 1 / n \sum_{i=1}^{n} y_i$ , indicating that an unweighted empirical mean serves as a suitable unbiased estimator of  $\mathbb{E}_{\pi^*}[Y]$ . However, IPW and DR estimators use policy ratios  $\rho(a, x) = \frac{\pi^*(a|x)}{\pi^b(a|x)}$  as importance weights. In case of large policy shifts, these ratios may vary significantly, leading to high variance in IPW and DR.
In this particular example, the shift in policies is inconsequential as it does not impact the distribution of outcomes  $Y$ . Hence, IPW and DR estimators introduce additional variance due to the policy ratios when they are not actually required. This limitation is not exclusive to this special case; in general, methodologies like IPW and DR exhibit high variance when there is low overlap between target and behavior policies [16] even if the resulting shift in marginals of the outcome  $Y$  is not significant.
Therefore, we propose the Marginal Ratio (MR) OPE estimator for contextual bandits in the subsequent section, which circumvents these issues by focusing on the shift in the marginal distribution of the outcomes  $Y$ . Additionally, we provide extensive theoretical insights on the comparison of MR to existing state-of-the-art methods, such as IPW and DR.
3 Marginal Ratio (MR) estimator
Our method's key insight involves weighting outcomes by the marginal density ratio of outcome  $Y$ :
$$
\mathbb {E} _ {\pi^ {*}} [ Y ] = \int_ {\mathcal {Y}} y p _ {\pi^ {*}} (y) \mathrm {d} y = \int_ {\mathcal {Y}} y \frac {p _ {\pi^ {*}} (y)}{p _ {\pi^ {b}} (y)} p _ {\pi^ {b}} (y) \mathrm {d} y = \mathbb {E} _ {\pi^ {b}} \left[ Y w (Y) \right],
$$
where  $w(y) \coloneqq \frac{p_{\pi^*}(y)}{p_{\pi^b}(y)}$ . This leads to the Marginal Ratio OPE estimator:
$$
\hat {\theta} _ {\mathrm {M R}} := \frac {1}{n} \sum_ {i = 1} ^ {n} w \left(y _ {i}\right) y _ {i}.
$$
In Section 3.1 we prove that by only considering the shift in the marginal distribution of outcomes, the MR estimator achieves a lower variance than the standard OPE methods. In fact, this estimator does not depend on the shift between target and behaviour policies directly. Instead, it depends on the shift between the marginals  $p_{\pi^b}(y)$  and  $p_{\pi^*}(y)$ .
Estimation of  $w(y)$  When the weights  $w(y)$  are known exactly, the MR estimator is unbiased and consistent. However, in practice the weights  $w(y)$  are often not known and must be estimated using the logged data  $\mathcal{D}$ . Here, we outline an efficient way to estimate  $w(y)$  by first representing it as a conditional expectation, which can subsequently be expressed as the solution to a regression problem.
Lemma 3.1. Let  $w(y) = \frac{p_{\pi^*}(y)}{p_{\pi^b}(y)}$  and  $\rho(a, x) = \frac{\pi^*(a|x)}{\pi^b(a|x)}$ , then  $w(y) = \mathbb{E}_{\pi^b}[\rho(A, X) \mid Y = y]$ , and
$$
w = \arg \min  _ {f} \mathbb {E} _ {\pi^ {b}} \left[ \left(\rho (A, X) - f (Y)\right) ^ {2} \right]. \tag {3}
$$
Lemma 3.1 allows us to approximate  $w(y)$  using a parametric family  $\{f_{\phi} : \mathbb{R} \to \mathbb{R} \mid \phi \in \Phi\}$  (e.g. neural networks) and defining  $\hat{w}(y) \coloneqq f_{\phi^*}(y)$ , where  $\phi^*$  solves the regression problem in Eq. (3).
Note that MR can also be estimated alternatively by directly estimating  $h(y) \coloneqq w(y)y$  using a similar regression technique as above and computing  $\hat{\theta}_{\mathrm{MR}} = 1 / n\sum_{i = 1}^{n}h(y_{i})$ . We include additional details along with empirical comparisons in Appendix F.1.1.
3.1 Theoretical analysis
Recall that the traditional OPE estimators like IPW and DR use importance weights which account for the shift in the joint distributions of  $(X, A, Y)$ . In this section, we prove that by considering only the shift in the marginal distribution of  $Y$  instead, MR achieves better variance properties than these estimators. Our analysis in this subsection assumes that the ratios  $\rho(a, x)$  and  $w(y)$  are known exactly. Since the OPE estimators considered are unbiased in this case, our analysis of variance is analogous to that of the mean squared error (MSE) here. We address the case where the weights are not known exactly in Section 3.1.2. First, we make precise our intuition that the shift in the joint distribution of  $(X, A, Y)$  is 'greater' than the shift in the marginal distribution of outcomes  $Y$ . We formalise this using the notion of  $f$ -divergences.
Proposition 3.2. Let  $f:[0,\infty)\to \mathbb{R}$  be a convex function with  $f(1) = 0$ , and  $\mathrm{D}_f(P||Q)$  denotes the  $f$ -divergence between distributions  $P$  and  $Q$ . Then,
$$
\mathrm {D} _ {f} \left(p _ {\pi^ {*}} (x, a, y) | | p _ {\pi^ {b}} (x, a, y)\right) \geq \mathrm {D} _ {f} \left(p _ {\pi^ {*}} (y) | | p _ {\pi^ {b}} (y)\right).
$$
**Intuition** Proposition 3.2 shows that the shift in the joint distributions is at least as 'large' as the shift in the margins of the outcome  $Y$ . Traditional OPE estimators, therefore take into consideration more of a distribution shift than needed, and consequently lead to inefficient estimators. In contrast, the MR estimator mitigates this problem by only considering the shift in the marginal distributions of outcomes resulting from the policy shift. This provides further intuition on why the MR estimator has lower variance compared to existing methods.
Proposition 3.3 (Variance comparison with IPW estimator). When the weights  $\rho(a, x)$  and  $w(y)$  are known exactly, we have that  $\mathrm{Var}_{\pi^b}[\hat{\theta}_{\mathrm{MR}}] \leq \mathrm{Var}_{\pi^b}[\hat{\theta}_{\mathrm{IPW}}]$ . In particular,
$$
\operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {I P W}} \right] - \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {M R}} \right] = \frac {1}{n} \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} \left[ \rho (A, X) \mid Y \right] Y ^ {2} \right] \geq 0.
$$
**Intuition** Proposition 3.3 shows that the variance of MR estimator is smaller than that of the IPW estimator when the weights are known exactly. Moreover, the proposition also shows that the difference between the two variances will increase as the variance  $\mathrm{Var}_{\pi^b}[\rho (A,X)\mid Y]$  increases. This variance is likely to be large when the policy shift between  $\pi^b$  and  $\pi^{*}$  is large, or when the dimensions of contexts  $X$  and/or the actions  $A$  is large, and therefore in these cases the MR estimator will perform increasingly better than the IPW estimator. A similar phenomenon occurs for DR as we show next, even though in this case the variance of MR is not in general smaller than that of DR.
Proposition 3.4 (Variance comparison with DR estimator). When the weights  $\rho(a, x)$  and  $w(y)$  are known exactly and  $\mu(A, X) \coloneqq \mathbb{E}[Y \mid X, A]$ , we have that,
$$
\operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {D R}} \right] - \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {M R}} \right] \geq \frac {1}{n} \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} \left[ \rho (A, X) Y \mid Y \right] - \operatorname {V a r} _ {\pi^ {b}} \left[ \rho (A, X) \mu (A, X) \mid X \right] \right].
$$
Intuition Proposition 3.4 shows that if  $\operatorname{Var}_{\pi^b}[\rho(A, X)Y|Y]$  is greater than  $\operatorname{Var}_{\pi^b}[\rho(A, X)\mu(A, X)|X]$  on average, the variance of the MR estimator will be less than that of the DR estimator. Intuitively, this will occur when the dimension of context space  $\mathcal{X}$  is high because in this case the conditional variance over  $X$  and  $A$ ,  $\operatorname{Var}_{\pi^b}[\rho(A, X)Y|Y]$  is likely to be greater than the conditional variance over  $A$ ,  $\operatorname{Var}_{\pi^b}[\rho(A, X)\mu(A, X)|X]$ . Our empirical results in Appendix F.2 are consistent with this intuition. Additionally, we also provide theoretical comparisons with other extensions of DR, such as Switch-DR [5] and DR with Optimistic Shrinkage (DRos) [17] in Appendix B, and show that a similar intuition applies for these results. We emphasise that the well known results in [5] which show that IPW and DR estimators achieve the optimal worst case variance (where the worst case is taken over a class of possible outcome distributions  $Y|X,A$ ) are not at odds with our results presented here (as the distribution of  $Y|X,A$  is fixed in our setting).
3.1.1 Comparison with Marginalised Inverse Propensity Score (MIPS) [14]
In this section, we compare MR against the recently proposed Marginalised Inverse Propensity Score (MIPS) estimator [14], which uses a marginalisation technique to reduce variance and provides a robust OPE estimate specifically in contextual bandits with large action spaces. We prove that the MR estimator achieves lower variance than the MIPS estimator and doesn't require new assumptions.
MIPS estimator As we mentioned earlier, the variance of the IPW estimator may be high when the action  $A$  is high dimensional. To mitigate this, the MIPS estimator assumes the existence of a (potentially lower dimensional) action embedding  $E$ , which summarises all 'relevant' information about the action  $A$ . Formally, this assumption can be written as follows:
Assumption 3.5. The action  $A$  has no direct effect on the outcome  $Y$ , i.e.,  $Y \perp A \mid X, E$ .
For example, in the setting of a recommendation system where  $A$  corresponds to the items recommended,  $E$  may correspond to the item categories. Assumption 3.5 then intuitively means that item category  $E$  encodes all relevant information about the item  $A$  which determines the outcome  $Y$ . Assuming that such action embedding  $E$  exists, [14] prove that the MIPS estimator  $\hat{\theta}_{\mathrm{MIPS}}$ , defined as
$$
\hat {\theta} _ {\mathrm {M I P S}} := \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {p _ {\pi^ {*}} (e _ {i} , x _ {i})}{p _ {\pi^ {b}} (e _ {i} , x _ {i})} y _ {i} = \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {p _ {\pi^ {*}} (e _ {i} \mid x _ {i})}{p _ {\pi^ {b}} (e _ {i} \mid x _ {i})} y _ {i},
$$
provides an unbiased estimator of target policy value  $\mathbb{E}_{\pi^*}[Y]$ . Moreover,  $\mathrm{Var}_{\pi^b}[\hat{\theta}_{\mathrm{MIPS}}] \leq \mathrm{Var}_{\pi^b}[\hat{\theta}_{\mathrm{IPW}}]$ .
Figure 1: Bayesian network corresponding to Assumption 3.5.
Intuition The context-embedding pair  $(X,E)$  can be seen as a representation of the context-action pair  $(X,A)$  which contains less 'redundant information' regarding the outcome  $Y$ . Intuitively, the MIPS estimator, which only considers the shift in the distribution of  $(X,E)$  is therefore more efficient than the IPW estimator (which considers the shift in the distribution of  $(X,A)$  instead).
MR achieves lower variance than MIPS Given the intuition above, we should achieve greater variance reduction as the amount of redundant information in the representation  $(X,E)$  decreases. We formalise this in Appendix D and show that the variance of MIPS estimator decreases as the representation gets closer to  $Y$  in terms of information content. As a result, we achieve the greatest variance reduction by considering the marginal shift in the outcome  $Y$  itself (as in MR) rather than the shift in the representation  $(X,E)$  (as in MIPS). The following result formalizes this finding.
Theorem 3.6. When the weights  $w(y)$ ,  $\frac{p_{\pi^*}(e,x)}{p_{\pi^b}(e,x)}$  and  $\rho(a, x)$  are known exactly, then under Assumption 3.5,  $\mathbb{E}_{\pi^b}[\hat{\theta}_{\mathrm{MR}}] = \mathbb{E}_{\pi^b}[\hat{\theta}_{\mathrm{MIPS}}] = \mathbb{E}_{\pi^*}[Y]$ , and  $\operatorname{Var}_{\pi^b}[\hat{\theta}_{\mathrm{MR}}] \leq \operatorname{Var}_{\pi^b}[\hat{\theta}_{\mathrm{MIPS}}] \leq \operatorname{Var}_{\pi^b}[\hat{\theta}_{\mathrm{IPW}}]$ .
This analysis provides a link between the MR and MIPS estimators in the framework of contextual bandits, and shows that the MR estimator achieves lower variance than MIPS estimator while not requiring any additional assumptions (e.g. Assumption 3.5 as in MIPS). We also verify this empirically in Section 5.1 by reproducing the experimental setup in [14] along with the MR baseline.
3.1.2 Weight estimation error
Our analysis so far assumes prior knowledge of the behavior policy  $\pi^b$  and the marginal ratios  $w(y)$ . However, in practice, both quantities are often unknown and must be estimated from data. To this end, we assume access to an additional training dataset  $\mathcal{D}_{\mathrm{tr}} = \{(x_i^{\mathrm{tr}},a_i^{\mathrm{tr}},y_i^{\mathrm{tr}})\}_{i = 1}^m$  (for weight estimation), in addition to the evaluation dataset  $\mathcal{D} = \{(x_i,a_i,y_i)\}_{i = 1}^n$  (for computing the OPE estimate). The estimation of  $\hat{w} (y)$  involves a two-step process that exclusively utilizes data from  $\mathcal{D}_{\mathrm{tr}}$ :