SlowGuess/NeurIPS_2023_OCR huggingface dataset:
There is no analogous DR extension of the MR estimator A consequence of considering the embedding  $R = Y$  (as in MR) is that in this case we do not have an analogous doubly robust extension as above. To see why this is the case, note that when  $R = Y$ , we get that  $\tilde{\mu}(r) = \mathbb{E}[Y \mid R = r] = \mathbb{E}[Y \mid Y = y] = y$ . If we substitute this  $\tilde{\mu}(r)$  in (8), we are simply left with  $\tilde{\eta}(\pi^{*})$  on the right hand side (as the first term cancels out). This means that the resulting estimator does not retain the doubly robust nature as we no longer obtain an accurate estimate if either the outcome model or the importance ratios are well estimated.
2the  $\sum_{r^{\prime}\in \mathcal{R}}$  can be replaced with  $\int_{r^{\prime}\in \mathcal{R}}\mathrm{d}r^{\prime}$  when  $\mathcal{R}$  is continuous
E Application to causal inference
In this section, we investigate the application of the MR estimator for the estimation of average treatment effect (ATE). In this setting, we suppose that  $\mathcal{A} = \{0,1\}$ , and the goal is to estimate ATE defined as follows:
$$
\operatorname {A T E} := \mathbb {E} [ Y (1) - Y (0) ]
$$
Here, we use the potential outcomes notation [44] to denote the outcome under a deterministic policy  $\pi^{*}(a^{\prime}\mid x) = \mathbb{1}(a^{\prime} = a)$  as  $Y(a)$ .
Specifically, the IPW estimator applied to ATE estimation yields:
$$
\widehat {\mathrm {A T E}} _ {\mathrm {I P W}} = \frac {1}{n} \sum_ {i = 1} ^ {n} \rho_ {\mathrm {A T E}} \left(a _ {i}, x _ {i}\right) \times y _ {i},
$$
$$
\rho_ {\text {A T E}} (a, x) := \frac {\mathbb {1} (a = 1) - \mathbb {1} (a = 0)}{\pi^ {b} (a | x)}.
$$
Similarly, the MR estimator can be written as
$$
\widehat {\mathrm {A T E}} _ {\mathrm {M R}} = \frac {1}{n} \sum_ {i = 1} ^ {n} w _ {\mathrm {A T E}} \left(y _ {i}\right) \times y _ {i},
$$
$$
w _ {\text {A T E}} (y) = \frac {p _ {\pi^ {(1)}} (y) - p _ {\pi^ {(0)}} (y)}{p _ {\pi^ {b}} (y)},
$$
and  $\pi^{(a)}(a' \mid x) \coloneqq \mathbb{1}(a' = a)$  for  $a \in \{0, 1\}$ .
Again, using the fact that  $w_{\mathrm{ATE}}(Y) \stackrel{\mathrm{a.s.}}{=} \mathbb{E}[\rho_{\mathrm{ATE}}(A, X) \mid Y]$ , we can obtain  $w_{\mathrm{ATE}}$  by minimising a simple mean-squared loss:
$$
w _ {\text {A T E}} = \arg \min  _ {f} \mathbb {E} _ {\pi^ {b}} \left[ \frac {\mathbb {1} (A = 1) - \mathbb {1} (A = 0)}{\pi^ {b} (A | X)} - f (Y) \right] ^ {2}.
$$
Proposition E.1 (Variance comparison with IPW ATE estimator). When the weights  $\rho_{\mathrm{ATE}}(a,x)$  and  $w_{\mathrm{ATE}}(y)$  are known exactly, we have that  $\operatorname{Var}[\widehat{\mathrm{ATE}}_{\mathrm{MR}}] \leq \operatorname{Var}[\widehat{\mathrm{ATE}}_{\mathrm{IPW}}]$ . Specifically,
$$
\operatorname {V a r} [ \widehat {\mathrm {A T E}} _ {\mathrm {I P W}} ] - \operatorname {V a r} [ \widehat {\mathrm {A T E}} _ {\mathrm {M R}} ] = \frac {1}{n} \mathbb {E} \left[ \operatorname {V a r} \left[ \rho_ {\mathrm {A T E}} (A, X) | Y \right] Y ^ {2} \right] \geq 0.
$$
Proof of Proposition E.1. We have
$$
\operatorname {V a r} \left[ \widehat {\mathrm {A T E}} _ {\mathrm {I P W}} \right] - \operatorname {V a r} \left[ \widehat {\mathrm {A T E}} _ {\mathrm {M R}} \right] = \frac {1}{n} \left(\operatorname {V a r} \left[ \rho_ {\text {A T E}} (A, X) Y \right] - \operatorname {V a r} \left[ w _ {\text {A T E}} (Y) Y \right]\right). \tag {9}
$$
Using the tower law of variance, we get that
$$
\begin{array}{l} \operatorname {V a r} \left[ \rho_ {\mathrm {A T E}} (A, X) Y \right] = \operatorname {V a r} \left[ \mathbb {E} \left[ \rho_ {\mathrm {A T E}} (A, X) Y \mid Y \right] \right] + \mathbb {E} \left[ \operatorname {V a r} \left[ \rho_ {\mathrm {A T E}} (A, X) Y \mid Y \right] \right] \\ = \operatorname {V a r} \left[ \mathbb {E} \left[ \rho_ {\mathrm {A T E}} (A, X) \mid Y \right] Y \right] + \mathbb {E} \left[ \operatorname {V a r} \left[ \rho_ {\mathrm {A T E}} (A, X) \mid Y \right] Y ^ {2} \right] \\ = \operatorname {V a r} [ w _ {\text {A T E}} (Y) Y ] + \mathbb {E} [ \operatorname {V a r} [ \rho_ {\text {A T E}} (A, X) \mid Y ] Y ^ {2} ]. \\ \end{array}
$$
Putting this together with (9) we obtain,
$$
\mathrm {V a r} [ \widehat {\mathrm {A T E}} _ {\mathrm {I P W}} ] - \mathrm {V a r} [ \widehat {\mathrm {A T E}} _ {\mathrm {M R}} ] = \frac {1}{n} \mathbb {E} [ \mathrm {V a r} [ \rho_ {\mathrm {A T E}} (A, X) \mid Y ] Y ^ {2} ],
$$
which straightforwardly leads to the result.
Given the above definitions, the IPW estimator for  $\mathbb{E}[Y(a)]$  would only consider datapoints with  $A = a$ , as it weights the samples using the policy ratios  $\mathbb{1}(A = a) / \pi^b (A|X)$  which are only nonzero when  $A = a$ . This is however not the case with the MR estimator, as it uses the weights  $p_{\pi^{*}}(Y) / p_{\pi^{b}}(Y)$  which are not necessarily zero for  $A\neq a$ . Therefore, MR uses all evaluation
datapoints  $\mathcal{D}$  when estimating  $\mathbb{E}[Y(a)]$ . The MR estimator therefore leads to a more efficient use of evaluation data in this example.
Likewise, the doubly robust (DR) estimator applied to ATE estimation yields,
$$
\widehat {\mathrm {A T E}} _ {\mathrm {D R}} := \frac {1}{n} \sum_ {i = 1} ^ {n} \rho_ {\mathrm {A T E}} \left(a _ {i}, x _ {i}\right) \left(y _ {i} - \hat {\mu} \left(a _ {i}, x _ {i}\right)\right) + \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\hat {\mu} \left(1, x _ {i}\right) - \hat {\mu} \left(0, x _ {i}\right)\right),
$$
where  $\hat{\mu}(a, x) \approx \mathbb{E}[Y \mid X = x, A = a]$ . Like in classical off-policy evaluation, DR yields an accurate estimator of ATE when either the weights  $\rho_{\mathrm{ATE}}(a, x)$  or the outcome model i.e.,  $\hat{\mu}(a, x) = \mathbb{E}[Y \mid X = x, A = a]$ , are well estimated. However, despite this doubly robust nature of the estimator, we can show that the variance of the DR estimator may be higher than that of the MR estimator in many cases. The following result formalises this variance comparison between the DR and MR estimators, and is analogous to the result in Proposition 3.4 derived for classical off-policy evaluation.
Proposition E.2 (Variance comparison with DR ATE estimator). When the weights  $\rho_{\mathrm{ATE}}(a,x)$  and  $w_{\mathrm{ATE}}(y)$  are known exactly,
$$
\operatorname {V a r} \left[ \widehat {\mathrm {A T E}} _ {\mathrm {D R}} \right] - \operatorname {V a r} \left[ \widehat {\mathrm {A T E}} _ {\mathrm {M R}} \right] \geq \frac {1}{n} \mathbb {E} \left[ \operatorname {V a r} \left[ \rho_ {\text {A T E}} (A, X) Y \mid Y \right] - \operatorname {V a r} \left[ \rho_ {\text {A T E}} (A, X) \mu (A, X) \mid X \right] \right],
$$
where  $\mu (A,X)\coloneqq \mathbb{E}[Y\mid X,A]$
Proof of Proposition E.2. Using the law of total variance, we get that
$$
\begin{array}{l} n \operatorname {V a r} [ \widehat {\mathrm {A T E}} _ {\mathrm {D R}} ] = \operatorname {V a r} \left[ \rho_ {\mathrm {A T E}} (A, X) (Y - \hat {\mu} (A, X)) + (\hat {\mu} (1, X) - \hat {\mu} (0, X)) \right] \\ = \operatorname {V a r} [ \mathbb {E} [ \rho_ {\mathrm {A T E}} (A, X) (Y - \hat {\mu} (A, X)) + (\hat {\mu} (1, X) - \hat {\mu} (0, X)) \mid X, A) ] \\ + \mathbb {E} [ \operatorname {V a r} [ \rho_ {\mathrm {A T E}} (A, X) (Y - \hat {\mu} (A, X)) + (\hat {\mu} (1, X) - \hat {\mu} (0, X)) | X, A ] ] \\ = \operatorname {V a r} \left[ \rho_ {\mathrm {A T E}} (A, X) (\mu (A, X) - \hat {\mu} (A, X)) + (\hat {\mu} (1, X) - \hat {\mu} (0, X)) \right] \\ + \mathbb {E} \left[ \rho_ {\text {A T E}} ^ {2} (A, X) \operatorname {V a r} [ Y \mid X, A ] \right]. \\ \end{array}
$$
Again, using the law of total variance we can rewrite the first term on the RHS above as,
$$
\begin{array}{l} \operatorname {V a r} \left[ \rho_ {\mathrm {A T E}} (A, X) (\mu (A, X) - \hat {\mu} (A, X)) + (\hat {\mu} (1, X) - \hat {\mu} (0, X)) \right] \\ = \operatorname {V a r} [ \mathbb {E} [ \rho_ {\mathrm {A T E}} (A, X) (\mu (A, X) - \hat {\mu} (A, X)) + (\hat {\mu} (1, X) - \hat {\mu} (0, X)) \mid X ] ] \\ + \mathbb {E} [ \operatorname {V a r} [ \rho_ {\mathrm {A T E}} (A, X) (\mu (A, X) - \hat {\mu} (A, X)) + (\hat {\mu} (1, X) - \hat {\mu} (0, X)) | X) ] \\ \geq \operatorname {V a r} [ \mathbb {E} [ \rho_ {\mathrm {A T E}} (A, X) (\mu (A, X) - \hat {\mu} (A, X)) + (\hat {\mu} (1, X) - \hat {\mu} (0, X)) \mid X ] ] \\ = \operatorname {V a r} \left[ \mathbb {E} \left[ \rho_ {\text {A T E}} (A, X) (\mu (A, X) - \hat {\mu} (A, X)) + \rho_ {\text {A T E}} (A, X) \hat {\mu} (A, X) \mid X \right] \right. \\ = \operatorname {V a r} [ \mathbb {E} [ \rho_ {\text {A T E}} (A, X) \mu (A, X) \mid X ] ], \\ \end{array}
$$
where, in the second last step above we use the fact that
$$
\mathbb {E} \left[ \rho_ {\mathrm {A T E}} (A, X) \hat {\mu} (A, X) \mid X \right] = \hat {\mu} (1, X) - \hat {\mu} (0, X).
$$
$$
n \operatorname {V a r} [ \widehat {\mathrm {A T E}} _ {\mathrm {D R}} ] \geq \operatorname {V a r} [ \mathbb {E} [ \rho_ {\mathrm {A T E}} (A, X) \mu (A, X) \mid X ] ] + \mathbb {E} [ \rho_ {\mathrm {A T E}} ^ {2} (A, X) \operatorname {V a r} [ Y \mid X, A ] ].
$$
$$
\begin{array}{l} n \left(\operatorname {V a r} \left[ \widehat {\mathrm {A T E}} _ {\mathrm {D R}} \right] - \operatorname {V a r} \left[ \widehat {\mathrm {A T E}} _ {\mathrm {M R}} \right]\right) \\ \geq \operatorname {V a r} [ \mathbb {E} [ \rho_ {\mathrm {A T E}} (A, X) \mu (A, X) | X ] ] + \mathbb {E} [ \rho_ {\mathrm {A T E}} ^ {2} (A, X) \operatorname {V a r} [ Y | X, A ] ] - \operatorname {V a r} [ w _ {\mathrm {A T E}} (Y) Y ] \\ = \operatorname {V a r} [ \mathbb {E} [ \rho_ {\mathrm {A T E}} (A, X) \mu (A, X) \mid X ] ] + \mathbb {E} [ \operatorname {V a r} [ \rho_ {\mathrm {A T E}} (A, X) Y \mid X, A ] ] - \operatorname {V a r} [ w _ {\mathrm {A T E}} (Y) Y ] \\ = \operatorname {V a r} [ \mathbb {E} [ \rho_ {\mathrm {A T E}} (A, X) \mu (A, X) \mid X ] ] + \operatorname {V a r} [ \rho_ {\mathrm {A T E}} (A, X) Y ] - \operatorname {V a r} [ \mathbb {E} [ \rho_ {\mathrm {A T E}} (A, X) Y \mid X, A ] ] \\ - \operatorname {V a r} [ w _ {\text {A T E}} (Y) Y ] \\ = \operatorname {V a r} [ \mathbb {E} [ \rho_ {\mathrm {A T E}} (A, X) \mu (A, X) \mid X ] ] + \operatorname {V a r} [ \mathbb {E} [ \rho_ {\mathrm {A T E}} (A, X) \mid Y ] Y ] + \mathbb {E} [ \operatorname {V a r} [ \rho_ {\mathrm {A T E}} (A, X) \mid Y ] Y ^ {2} ] \\ - \operatorname {V a r} [ \mathbb {E} [ \rho_ {\text {A T E}} (A, X) Y \mid X, A ] ] - \operatorname {V a r} [ w _ {\text {A T E}} (Y) Y ] \\ = \operatorname {V a r} [ \mathbb {E} [ \rho_ {\mathrm {A T E}} (A, X) \mu (A, X) \mid X ] ] + \operatorname {V a r} [ w _ {\mathrm {A T E}} (Y) Y ] + \mathbb {E} [ \operatorname {V a r} [ \rho_ {\mathrm {A T E}} (A, X) \mid Y ] Y ^ {2} ] \\ - \operatorname {V a r} [ \mathbb {E} [ \rho_ {\text {A T E}} (A, X) Y \mid X, A ] ] - \operatorname {V a r} [ w _ {\text {A T E}} (Y) Y ] \\ = \operatorname {V a r} [ \mathbb {E} [ \rho_ {\mathrm {A T E}} (A, X) \mu (A, X) \mid X ] ] - \operatorname {V a r} [ \mathbb {E} [ \rho_ {\mathrm {A T E}} (A, X) Y \mid X, A ] ] + \mathbb {E} [ \operatorname {V a r} [ \rho_ {\mathrm {A T E}} (A, X) \mid Y ] Y ^ {2} ] \\ = \operatorname {V a r} \left[ \rho_ {\mathrm {A T E}} (A, X) \mu (A, X) \right] - \mathbb {E} \left[ \operatorname {V a r} \left[ \rho_ {\mathrm {A T E}} (A, X) \mu (A, X) \mid X \right] \right] - \operatorname {V a r} \left[ \rho_ {\mathrm {A T E}} (A, X) \mu (A, X) \right] \\ + \mathbb {E} [ \operatorname {V a r} [ \rho_ {\mathrm {A T E}} (A, X) \mid Y ] Y ^ {2} ] \\ = \mathbb {E} \left[ \operatorname {V a r} \left[ \rho_ {\mathrm {A T E}} (A, X) \mid Y \right] Y ^ {2} - \operatorname {V a r} \left[ \rho_ {\mathrm {A T E}} (A, X) \mu (A, X) \mid X \right] \right]. \\ \end{array}
$$
Proposition E.2 shows that if  $\operatorname{Var}[Y\rho_{\mathrm{ATE}}(A,X)\mid Y]$  is greater than  $\operatorname{Var}[\rho_{\mathrm{ATE}}(A,X)\mu (A,X)\mid X]$  on average, the variance of the MR estimator will be less than that of the DR estimator. Intuitively, this is likely to happen when the dimension of context space  $\mathcal{X}$  is high because in this case, the conditional variance over  $X$  and  $A$ ,  $\operatorname{Var}[Y\rho_{\mathrm{ATE}}(A,X)\mid Y]$  is likely to be greater than the conditional variance over  $A$ ,  $\operatorname{Var}[\rho_{\mathrm{ATE}}(A,X)\mu (A,X)\mid X]$ .
In this section, we provide additional experimental details for the results presented in the main text. We also include extensive experimental results to provide further empirical evidence in favour of the MR estimator.
Computational details We ran our experiments on Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz with 8GB RAM per core. We were able to use 150 CPUs in parallel to iterate over different configurations and seeds. However, we would like to note that for each run our algorithms only requires 1 CPU and at most 30 minutes to run as our neural networks are relatively small. Throughout our experiments, whenever the outcome  $Y$  was continuous, we used a fully connected neural network with three hidden layers with 512, 256 and 32 nodes respectively (and ReLU activation function) to estimate the weights  $\hat{w}(y)$ . On the other hand, when the outcome is discrete we can directly estimate  $\hat{w}(y) \approx \mathbb{E}[\hat{\rho}(A, X) \mid Y = y]$  by calculating the sample mean of  $\hat{\rho}(A, X)$  on samples with  $Y = y$ . Additionally, for each configuration of parameters in our experiments, we ran experiments for 10 different seeds (in  $\{0, 1, \dots, 9\}$ ).
F.1 Alternative methodology of estimating MR
In addition to the OPE baselines like IPW, DM and DR estimators considered in the main text, we also include empirically investigate an alternative methodology of estimating MR. Below we describe this methodology, denoted as 'MR (alt)', in greater detail:
F.1.1 MR (alt)
Recall our definition of MR estimator:
In the main text, we propose estimating the weights  $w(y)$  first and using this to estimate  $\hat{\theta}_{\mathrm{MR}}$  using the above expression. Alternatively, we can estimate  $h(y) \coloneqq yw(y)$  using
$$
h = \arg \min _ {f} \mathbb {E} _ {\pi^ {b}} \left[ \left(Y \frac {\pi^ {*} (A | X)}{\pi^ {b} (A | X)} - f (Y)\right) ^ {2} \right].
$$
Subsequently, the MR estimator can be written as:
$$
\hat {\theta} _ {\mathrm {M R}} = \frac {1}{n} \sum_ {i = 1} ^ {n} h (y _ {i}).
$$
We refer to this alternative methodology as 'MR-alt' and compare it empirically against the original methodology (which we simply refer to as 'MR'). In general, it is difficult to say which of the two methods will perform better. Intuitively speaking, in cases where the behaviour of the quantity  $Y \frac{\pi^{*}(A|X)}{\pi^{b}(A|X)}$  with varying  $Y$  is 'smoother' than that of  $\frac{\pi^{*}(A|X)}{\pi^{b}(A|X)}$ , the alternative method is expected to perform better. Our empirical results in the next sections show that the relative performance of the two methods varies for different data generating mechanisms.
F.2 Synthetic data experiments
Here, we include additional experimental details for the synthetic data experiments presented in Section 5.1 for completeness. For this experiment, we use the same setup as the synthetic data experiment in [14], reproduced by reusing their code with minor modifications.
Setup Here, we sample the  $d$ -dimensional context vectors  $x$  from a standard normal distribution. The setup used also includes 3-dimensional categorical action embeddings  $E \in \mathcal{E}$ , which are sampled from the following conditional distribution given action  $A = a$ ,
$$
p (e \mid a) = \prod_ {k = 1} ^ {3} \frac {\exp \left(\alpha_ {a , e _ {k}}\right)}{\sum_ {e ^ {\prime} \in \mathcal {E} _ {k}} \exp \left(\alpha_ {a , e ^ {\prime}}\right)},
$$
which is independent of the context  $X$ .  $\{\alpha_{a,e_k}\}$  is a set of parameters sampled independently from the standard normal distribution. Each dimension of  $\mathcal{E}$  has a cardinality of 10, i.e.,  $\mathcal{E}_k = \{1,2,\dots ,10\}$ .
Reward function The expected reward is then defined as:
$$
q (x, e) = \sum_ {k = 1} ^ {3} \eta_ {k} \cdot \left(x ^ {T} M x _ {e _ {k}} + \theta_ {x} ^ {T} x + \theta_ {e} ^ {T} x _ {e _ {k}}\right),
$$
where  $M$ ,  $\theta_{x}$  and  $\theta_{e}$  are parameter matrices or vectors sampled from a uniform distribution with range  $[-1, 1]$ .  $x_{e_k}$  is a context vector corresponding to the  $k$ -th dimension of the action embedding, which is unobserved to the estimators.  $\eta_{k}$  specifies the importance of the  $k$ -th dimension of the action embedding, sampled from Dirichlet distribution so that  $\sum_{k=1}^{3} \eta_{k} = 1$ .
Behaviour and target policies The behaviour policy  $\pi^b$  is defined by applying the softmax function to  $q(x,a) = \mathbb{E}[q(X,E)\mid A = a,X = x]$  as
$$
\pi^ {b} (a \mid x) = \frac {\exp \left(- q (x , a)\right)}{\sum_ {a ^ {\prime} \in \mathcal {A}} \exp \left(- q (x , a ^ {\prime})\right)}.
$$
For the target policy, we define the class of parametric policies,
$$
\pi^ {\alpha^ {*}} (a | x) = \alpha^ {*} \mathbb {1} (a = \arg \max  _ {a ^ {\prime} \in \mathcal {A}} q (x, a ^ {\prime})) + \frac {1 - \alpha^ {*}}{| \mathcal {A} |},
$$
where  $\alpha^{*} \in [0,1]$  controls the shift between the behaviour and target policies. As shown in the main text, as  $\alpha^{*} \to 1$ , the shift between behaviour and target policies increases.
Baselines In the main text, we compare MR with DM, IPW, DR and MIPS estimators. In addition to these baselines, here we also consider Switch-DR [5] and DR with Optimistic Shrinkage (DRos) [17]. Following [14], we use the random forest [45] along with 2-fold cross-fitting [46] to obtain  $\hat{q}(x,e)$  for DR and DM methods. To estimate  $p_{\pi^b}(a \mid x,e)$  for MIPS estimator, we use logistic regression. We also include the results for MR estimated using the alternative methodology described in Section F.1.1. We refer to this as 'MR (alt)'
Estimation of behaviour policy  $\widehat{\pi}^b$  and marginal ratio  $\hat{w}(y)$  We do not assume that the true behaviour policy  $\pi^b$  is known, and therefore estimate  $\widehat{\pi}^b$  using the available training data. For the MR estimator, we estimate the behaviour policy using a random forest classifier trained on  $50\%$  of the training data and use the rest of the training data to estimate the marginal ratios  $\hat{w}(y)$  using multi-layer perceptrons (MLP). Moreover, for a fair comparison we use a different behaviour policy estimate  $\widehat{\pi}^b$  for all other baselines which is trained on the entire training data.
(a)  $d = 1000, n_{a} = 100, \alpha^{*} = 0.8$
(b)  $d = 5000, n_{a} = 250, \alpha^{*} = 0.8$
(c)  $d = 5000$ ,  $n_a = 250$ ,  $\alpha^* = 1.0$
Figure 5: MSE with varying size of evaluation dataset  $n$  for different choices of parameters.
F.2.1 Results
For this experiment, the results are computed over 10 different sets of logged data replicated with different seeds, and in Figures 5 - 8 we use a total of  $m = 5000$  training data.
Varying size of evaluation data  $n$  Figure 5 shows that MR outperforms the other baselines, in terms of MSE and squared bias, when the number of evaluation data  $n \leq 1000$ . Additionally, we observe that in this experiment, MR estimated using our original methods ('MR'), yields better results than the alternative method of estimating MR ('MR (alt)'). Moreover, while the variance of DM is lower than that of MR, the DM method has a high bias and consequently a high MSE. We note that while the difference between MSE and variance of MIPS and MR estimators decreases with increasing evaluation data size, MR still outperforms MIPS in terms of both MSE and variance.
(a)  $d = 100$ ,  $n_a = 100$ ,  $n = 100$
(b)  $d = 100$ ,  $n_a = 250$ ,  $n = 100$
(c)  $d = 1000, n_a = 250, n = 100$
Figure 6: MSE with varying  $\alpha^{*}$  for different choices of parameters.
Varying  $\alpha^{*}$  Figure 6 shows the results with increasing policy shift. It can be seen that overall MR methods achieve the smallest MSE with increasing policy shift. Moreover, the difference between MSE and variance of MR and IPW/DR methods increases with increasing policy shift, showing that MR performs especially better than these baselines when the difference between behaviour and target policies is large. Similarly, we observe in Figure 6 that as the shift between the behaviour and target policy increases with increasing  $\alpha^{*}$ , so does the difference between the MSE and variance of MR and the MIPS estimators. This shows that generally MR outperforms MIPS estimator in terms of variance and MSE, and that MR performs especially better than MIPS as the difference between behaviour and target policies increases.
Varying  $d$  and  $n_a$  Figures 7 and 8 show that MR outperforms the other baselines as the context dimensions and/or number of actions increase. In fact, these figures show that MR is significantly robust to increasing dimensions of action and context spaces, whereas baselines like IPW and DR perform poorly in large action spaces.
Varying  $m$  Figure 10 shows the results with increasing number of training data  $m$ . We again observe that the MR methods 'MR' and 'MR (alt)' outperforms the other baselines in terms of the MSE and squared bias even when the number of training data is low. Moreover, the variance of both the MR estimators continues to improve with increasing number of training data.
In this experiment, we observe that overall 'MR (alt)' performs worse than the original MR estimator ('MR' in the figures). However, as we observe in Appendix F.5, this does not happen consistently across all experiments, which suggests that the comparative performance of the two MR methods depends on the data generating mechanism.
(a)  $n_a = 20, n = 200, \alpha^* = 0.8$
(b)  $n_a = 100$ ,  $n = 200$ ,  $\alpha^* = 0.8$
(c)  $n_a = 250, n = 200, \alpha^* = 0.8$
Figure 7: MSE with varying context dimensions  $d$  for different choices of parameters.
Table 3: Mean-squared error results with 2 standard errors for synthetic data setup considered in Section 5.1 with  $d = 5000$ ,  $n_a = 50$ ,  $\alpha^* = 0.8$ . We use a fixed budget of datapoints (denoted by  $N$ ) for each baseline and in the case of MR we use  $m = 2000$  of the available datapoints to estimate  $\hat{w}(y)$  and the rest of data to evaluate the MR estimator (i.e.  $n = N - 2000$  for MR). In contrast, for IPW and MIPS since the importance ratios are already known, we use all of the  $N$  datapoints for evaluation of the off-policy value (i.e.  $n = N$  for IPW and MIPS).
N	2800	3200	6400	10000	12000
GT weights ρ(a, x) and estimated reward model μ(a, x) (m = 2000 used for training μ(a, x) and n = N - 2000 used for evaluation)	DM	0.137±0.028	0.099±0.012	0.103±0.012	0.093±0.010	0.089±0.010
DR	0.227±0.065	0.068±0.035	0.068±0.022	0.024±0.011	0.045±0.015
DRos	0.128±0.027	0.072±0.011	0.049±0.014	0.063±0.014	0.051±0.016
SwitchDR	0.128±0.027	0.059±0.014	0.052±0.013	0.061±0.015	0.056±0.016
GT weights (all of N datapoints are used for evaluation)	IPW	0.237±0.062	0.066±0.036	0.067±0.021	0.025±0.011	0.044±0.014
MIPS	0.236±0.062	0.065±0.035	0.067±0.021	0.025±0.011	0.044±0.014
Estimated weights w(y) (m = 2000 used for training and n = N - 2000 used for evaluation)	MR (Ours)	0.045±0.015	0.042±0.014	0.048±0.020	0.049±0.020	0.047±0.016
F.2.2 Known policy ratios  $\rho (a,x)$
Our previous setting of unknown importance policy ratios  $\rho(a, x)$  captures a wide variety of real-world applications, ranging from health care to autonomous driving. In addition, to demonstrate the utility of MR in settings with known  $\rho(a, x), p(e \mid a, x)$  and unknown  $w(y)$  (for our proposed method, MR), we have conducted additional experiments. Here, we use a fixed budget of datapoints (denoted by  $N$ ) for each baseline and for MR we allocate  $m = 2000$  of the available datapoints to estimate  $\hat{w}(y)$  and use the remaining for evaluating the MR estimator (i.e.,  $n = N - 2000$  for MR). In contrast, for IPW and MIPS (since the importance ratios are already known), we use all of the  $N$  datapoints to evaluate the off-policy value (i.e.  $n = N$  for IPW and MIPS).
(a)  $d = 1000$ ,  $n = 100$ ,  $\alpha^{*} = 0.4$
(b)  $d = 1000$ ,  $n = 100$ ,  $\alpha^{*} = 0.8$
(c)  $d = 1000, n = 100, \alpha^{*} = 1.0$
Figure 8: MSE with varying number of actions  $n_a$  for different choices of parameters.