SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The results included in Table 3 show that MR achieves the smallest MSE among the baselines for  $N \leq 6400$ . However, we observe that the MSE of IPW, DR and MIPS (with true importance weights) falls below that of MR (with estimated weights  $\hat{w}$ ) when the data size  $N$  is large enough (i.e.,  $N \geq 10,000$ ). This is to be expected since IPW, DR and MIPS are unbiased (i.e., use ground truth importance ratios  $\rho(a,x)$ ) whereas MR uses estimated weights  $\hat{w}(y)$  (and hence may be biased). MR still performs the best when  $N \leq 6400$ .
F.3 Experiments on classification datasets
Here, we conduct experiments on four classification datasets, OptDigits, PenDigits, SatImage and Letter datasets from the UCI repository [37], the Digits dataset from scikit-learn library, as well as the Mnist [38] and CIFAR-100 datasets [39].
Setup Following previous works [13, 22, 36, 5], the classification datasets are transformed to contextual bandit feedback data. The classification dataset comprises  $\{x_{i},a_{i}^{\mathrm{gt}}\}_{i = 1}^{n_{0}}$ , where  $x_{i}\in \mathcal{X}$  are feature vectors and  $a_i^{\mathrm{gt}}\in \mathcal{A}$  are the ground-truth labels. In the contextual bandits setup, the feature vectors  $x_{i}$  are considered to be the contexts, whereas the actions correspond to the possible class of labels. We split the dataset into training and testing datasets of sizes  $m$  and  $n$  respectively. We present the results for a range of different values of  $m$  and  $n$ .
Reward function Let  $X$  be a context with ground truth label  $A^{\mathrm{gt}}$ , we define the reward for action  $A$  as:
$$
Y := \mathbb {1} (A = A ^ {\mathrm {g t}}).
$$
Behaviour and target policies Using the  $m$  training datapoints, we first train a classifier  $f:\mathcal{X}\to \mathbb{R}^{|\mathcal{A}|}$  which takes as input the feature vectors  $x_{i}$  and outputs a vector of softmax probabilities over
labels, i.e. the  $a$ -th component of the vector  $f(x)$ , denoted as  $(f(x))_a$  corresponds to the estimated probability  $\mathbb{P}(A^{\mathrm{gt}} = a \mid X = x)$ .
Next, we use  $f$  to define the ground truth behaviour policy,
$$
\pi^ {b} (a \mid x) = (f (x)) _ {a}.
$$
For the target policies, we use  $f$  to define a parametric class of target policies using a trained classifier  $f: \mathcal{X} \to \mathbb{R}^{|A|}$ .
$$
\pi^ {\alpha^ {*}} (a \mid x) = \alpha^ {*} \cdot \mathbb {1} (a = \arg \max _ {a ^ {\prime} \in \mathcal {A}} (f (x)) _ {a ^ {\prime}}) + \frac {1 - \alpha^ {*}}{| \mathcal {A} |},
$$
where  $\alpha^{*} \in [0,1]$ . A value of  $\alpha^{*}$  close to 1 leads to a near-deterministic and well-performing policy. As  $\alpha^{*}$  decreases, the policy gets increasingly worse and 'noisy'. In this experiment, we consider target policies  $\pi^{*} = \pi^{\alpha^{*}}$  for  $\alpha^{*} \in \{0.0,0.2,0.4,\dots,1.0\}$ .
Using the behaviour policy defined above, we generate the contextual bandits data described with training and evaluation datasets of sizes  $m$  and  $n$  respectively.
Estimation of behaviour policy  $\widehat{\pi}^b$  and marginal ratio  $\hat{w}(y)$  We do not assume that the behaviour policy  $\pi^b$  is known, and therefore estimate it using training data. To estimate the behaviour policy  $\widehat{\pi}^b$ , we train a random forest classifier using the training data. This estimate of behaviour policy is used for all the baselines in our experiment. Since the reward is binary, we can estimate the marginal ratios  $\hat{w}(y) = \mathbb{E}_{\pi^b}[\hat{\rho}(A,X) \mid Y = y]$  by directly estimating the sample mean of  $\hat{\rho}(A,X)$  for datapoints with  $Y = y$ . We re-use the  $m$  training datapoints to estimate this sample mean.
Baselines We compare our estimator with Direct Method (DM), IPW and DR estimators. In addition, we also consider Switch-DR [5] and DR with Optimistic Shrinkage (DRos) [17]. To estimate  $\hat{q}(x,a)$  for DM and DR estimators, we use random forest classifiers (since reward  $Y$  is binary). Moreover, because of the binary nature of  $Y$ , the alternative method of estimating MR yields the same estimator as the original method, therefore we do not consider the two separately here. Additionally, in this experiment, we do not include MIPS (or G-MIPS) baseline, as there is no natural informative embedding  $E$  of the action  $A$ .
F.3.1 Results
For this experiment, we compute the results over 10 different sets of logged data replicated with different seeds. Figures 11 - 17 show the results corresponding to each baseline for the different datasets. It can be seen that across all datasets, the MR achieves the smallest MSE with increasing evaluation data size  $n$ . Moreover, across all datasets, MR attains the minimum MSE with relatively small number of evaluation data ( $n \leq 100$ ).
Unlike the experiments in Section 5.1, we observe that the KL-divergence between target and behaviour policy decreases as  $\alpha^{*}$  increases (see Figure 9). Therefore, as  $\alpha^{*}$  increases the shift between target and behaviour policies decreases. Figures 11 - 16 show that as  $\alpha^{*}$  increases, the difference between the MSE, squared bias and variance of MR and the other baselines decreases. This confirms our findings from earlier experiments that MR performs especially better than the other baselines when the difference between behaviour and target policies is large.
Moreover, the figures also include results with increasing number of training data  $m$ . It can be seen that MR out-performs the baselines even when the number of training data  $m$  is small ( $m = 100$ ). Moreover, the relative advantage of MR improves with increasing  $m$ .
F.4 Application to Average Treatment Effect (ATE) estimation
In this subsection, we provide additional details for our experiment applying MR to the problem of ATE estimation presented in the main text. We begin by describing the dataset being used in this experiment.
Twins dataset We use the Twins dataset as studied by [40], which comprises data from twin births in the USA between 1989-1991. The treatment  $a = 1$  corresponds to being born the heavier twin and the outcome  $Y$  corresponds to the mortality of each of the twins in their first year of life. Since
Figure 9: KL divergence  $D_{\mathrm{KL}}(\pi^b || \pi^*)$  with increasing  $\alpha^*$  for the classification data experiments. Here, we only include the results for a specific choice of parameters for the Letter dataset. We observe similar results for other datasets and parameter choices.
(a)  $d = 1000, n = 10, \alpha^{*} = 0.8$
(b)  $d = 1000$ ,  $n = 200$ ,  $\alpha^{*} = 0.8$
(c)  $d = 1000$ ,  $n = 800$ ,  $\alpha^{*} = 0.8$
Figure 10: MSE with varying number of training data  $m$  for different choices of parameters.
the data includes records for both twins, their outcomes would be considered as the two potential outcomes. Specifically,  $Y(1)$  corresponds to the mortality of the heavier twin (and likewise for  $Y(0)$ ). Closely following the methodology of [40], we only chose twins which are the same sex and weigh less than 2kgs. This provides us with a dataset of 11984 pairs of twins.
The mortality rate for the lighter twin is  $18.9\%$  and for the heavier twin is  $16.4\%$ , leading to the ATE value being  $\theta_{\mathrm{ATE}} = -2.5\%$ . For each twin-pair we obtained 46 covariates relating to the parents, the pregnancy and birth.
Treatment assignment To simulate an observational study, we selectively hide one of the two twins by defining the treatment variable  $A$  which depends on the feature GESTAT10. This feature, which takes integer values from 0 to 9, is obtained by grouping the number of gestation weeks prior to birth into 10 groups. Then we sample actions  $A$  as follows,
$$
A \mid X \sim \operatorname {B e r n} (Z / 1 0),
$$
(a) Results with varying  $n$  for  $\alpha^{*} = 0.2$  and  $m = 1000$
(b) Results with varying  $\alpha^{*}$  for  $m = n = 1000$
(c) Results with varying  $m$  for  $n = 1000$  and  $\alpha^{*} = 0.6$
Figure 11: Results for OptDigits dataset
where  $Z$  is GESTAT10, and  $X$  are all the 46 features corresponding to a twin pair (including GESTAT10).
Using the treatment assignments defined above, we generate the observational data by selectively hiding one of the two twins from each pair. Next, we randomly split this dataset into training and evaluation datasets of sizes  $m$  and  $n$  respectively. In this experiment, we consider  $m = 5000$  training datapoints.
Baselines Recall that ATE estimation can be formulated as the difference between off-policy values of deterministic policies  $\pi^{(1)}\coloneqq \mathbb{1}(A = 1)$  and  $\pi^{(0)}\coloneqq \mathbb{1}(A = 0)$ . Therefore, any OPE estimator can be applied to ATE estimation. In this experiment, we compare our estimator against the baselines considered in our OPE experiments in Section F.3. This includes the Direct Method (DM), IPW and DR estimators as well as Switch-DR [5] and DR with Optimistic Shrinkage (DRos) [17]. To estimate  $\hat{q} (x,a)$  for DM and DR estimators, we use multi-layer perceptrons (MLP) trained on the  $m$  training datapoints. Additionally, we estimate the behaviour policy  $\widehat{\pi}^b$  using random forest classifier trained on the full training dataset.
Since the outcome in this experiment is binary, we estimate the weights  $w(y) = \mathbb{E}_{\pi^b}[\hat{\rho}(A,X) \mid Y = y]$  directly by estimating the sample mean of  $\hat{\rho}(A,X)$  for datapoints with  $Y = y$ . This means that the alternative method of estimating MR yields the same value as the default method. We therefore do not consider these estimators separately. Additionally, since there is no natural embedding  $R$  of the covariate-action space which satisfies the conditional dependence Assumption D.1, we do not consider the G-MIPS (or MIPS) estimator either.
Performance metric For our evaluation, we consider the absolute error in ATE estimation,  $\epsilon_{\mathrm{ATE}}$  defined as:
$$
\epsilon_ {\mathrm {A T E}} := | \hat {\theta} _ {\mathrm {A T E}} ^ {(n)} - \theta_ {\mathrm {A T E}} |.
$$
(b) Results with varying  $\alpha^{*}$  for  $m = n = 1000$
(c) Results with varying  $m$  for  $\alpha^{*} = 0.6$  and  $n = 1000$
Figure 12: Results for PenDigits dataset
Here,  $\hat{\theta}_{\mathrm{ATE}}^{(n)}$  denotes the value of the ATE estimated using  $n$  evaluation datapoints. For example, for the IPW estimator, the  $\hat{\theta}_{\mathrm{ATE}}^{(n)}$  can be written as:
$$
\hat {\theta} _ {\mathrm {A T E}} ^ {(n)} = \widehat {\mathrm {A T E}} _ {\mathrm {I P W}} = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\frac {\mathbb {1} (a _ {i} = 1) - \mathbb {1} (a _ {i} = 0)}{\hat {\pi} ^ {b} (a _ {i} \mid x _ {i})}\right) y _ {i}.
$$
All results for this experiment are provided in the main text.
F.5 Additional synthetic data experiments
In addition to the synthetic data experiments provided in Section 5.1, we also consider an additional synthetic data setup to obtain further empirical evidence in favour of the MR estimator, and also compare it against the generalised version of the MIPS estimator (described as G-MIPS in Appendix D). Here, we use a similar setup to [14] (albeit without action embeddings  $E$ ) where the  $d$ -dimensional context vectors  $x$  are sampled from a standard normal distribution. Likewise, the action space is finite and comprises of  $n_a$  actions, i.e.  $\mathcal{A} = \{0, \dots, n_a - 1\}$ , with  $n_a$  taking a range of different values. The reward function is defined as follows:
Reward function The expected reward  $q(x, a) \coloneqq \mathbb{E}[Y \mid x, a]$  for these experiments is defined as follows:
$$
q (x, a) = \sin \left(a \cdot | | x | | _ {2}\right).
$$
The reward  $Y$  is obtained by adding a normal noise random variable to  $q(x, a)$
$$
Y = q (X, A) + \epsilon ,
$$
where  $\epsilon \sim \mathcal{N}(0,0.01)$ . Here, it can be seen that conditional on  $R = (||X||_2,A)$ , the reward  $Y$  does not depend on  $(X,A)$ , i.e., the embedding  $R$  satisfies the conditional independence assumption  $Y\perp (X,A)\mid R$ .
(b) Results with varying  $\alpha^{*}$  for  $n = 1000$
(c) Results with varying  $m$  for  $\alpha^{*} = 0.6$  and  $n = 1000$
Figure 13: Results for SatImage dataset
Behaviour and target policies We first define a behaviour policy by applying softmax function to  $q(x, a)$  as
$$
\pi^ {b} (a \mid x) = \frac {\exp (q (x , a))}{\sum_ {a ^ {\prime} \in \mathcal {A}} \exp (q (x , a ^ {\prime}))}.
$$
Just like in Section 5.1, to investigate the effect of increasing policy shift, we define a class of policies,
where  $\alpha^{*} \in [0,1]$  allows us to control the shift between  $\pi^b$  and  $\pi^{*}$ . Again, the shift between  $\pi^b$  and  $\pi^{*}$  increases as  $\alpha^{*} \to 1$ . Using the ground truth behaviour policy  $\pi^b$ , we generate a dataset which is split into training and evaluation datasets of sizes  $m$  and  $n$  respectively.
In Figures 18 - 21, we present the results for this experimental setup for different choices of paramater configurations.
Estimation of behaviour policy  $\hat{\pi}^b$  and marginal ratio  $\hat{w}(y)$  For the MR estimator, we estimate the behaviour policy using a random forest classifier trained on  $50\%$  of the training data and use the rest of the training data to estimate the marginal ratios  $\hat{w}(y)$  using multi-layer perceptrons (MLP). Moreover, for a fair comparison we use a different behaviour policy estimate  $\hat{\pi}^b$  for all other baselines which is trained on the entire training data.
Additional Baselines In addition to the baselines considered in the main text (Section 5.1), we also consider Switch-DR [5] and DR with Optimistic Shrinkage (DRos) [17]. In addition, we also include the results for MR estimated using the alternative method ('MR (alt)') outlined in Section F.1.1. For
(b) Results with varying  $\alpha^{*}$  for  $m = n = 1000$
(c) Results with varying  $m$  for  $\alpha^{*} = 0.6$  and  $n = 1000$
Figure 14: Results for Letter dataset
the G-MIPS estimator (defined in Appendix D) considered here, we use  $R = (a, ||x||_2)^3$ . To estimate  $\hat{q}(x, a)$  for DM and DR estimators, we use multi-layer perceptrons (MLPs).
F.5.1 Results
For this experiment, the results are computed over 10 different sets of logged data replicated with different seeds, and in Figures 18 - 21 we use a total of  $m = 5000$  training data.
Varying  $n$  Figure 18 shows that MR outperforms the other baselines, in terms of MSE and squared bias, when the number of evaluation data  $n \leq 1000$ . Additionally, we observe that in this experiment, MR estimated using alternative methods, MR (alt), yields better results than the original method of estimating MR. Moreover, while the variance of DM is lower than that of MR, the DM method has a high bias and consequently a high MSE.
Varying  $\alpha^{*}$  Figure 19 shows the results with increasing policy shift. It can be seen that overall MR methods achieve the smallest MSE with increasing policy shift. Moreover, the difference between MSE and variance of MR and IPW/DR methods increases with increasing policy shift, showing that MR performs especially better than these baselines when the difference between behaviour and target policies is large.
Varying  $d$  and  $n_a$  Figures 20 and 21 show that MR outperforms the other baselines as the context dimensions and/or number of actions increase. In fact, Figure 21 shows that MR is significantly robust to increasing action space, whereas baselines like IPW and DR perform poorly in large action spaces.
<sup>3</sup>It is easy to see that in our setup, the embedding  $R = (a, ||x||_2)$  satisfies the conditional independence assumption  $Y \perp (X, A) \mid R$  needed for G-MIPS estimator to be unbiased
(b) Results with varying  $\alpha^{*}$  for  $m = n = 1000$
(c) Results with varying  $m$  for  $\alpha^{*} = 0.6$  and  $n = 1000$
Figure 15: Results for Mnist dataset
Varying  $m$  Figure 22 shows the results with increasing number of training data  $m$ . We again observe that the MR methods 'MR' and 'MR (alt)' outperforms the other baselines in terms of the MSE and squared bias even when the number of training data is low. Moreover, the variance of both the MR estimators continues to improve with increasing number of training data.
Unlike our experimental results in Section F.2, 'MR (alt)' performs better than the original MR estimator overall. This shows that one of these two methods is not better than the other consistently in all cases, and their relative performance depends on the dataset under consideration.
F.6 Self-normalised MR estimator
Self-normalization trick has been used in practice to reduce the variance in off-policy estimators [30]. This technique is also applicable to the MR estimator, and leads to the self-normalized MR estimator (denoted as  $\theta_{\mathrm{SNMR}}$ ) defined as follows:
$$
\theta_ {\text {S N M R}} := \sum_ {i = 1} ^ {n} \frac {w \left(Y _ {i}\right)}{\sum_ {j = 1} ^ {n} w \left(Y _ {j}\right)} Y _ {i}.
$$
We conducted experiments to investigate the effect of self-normalisation on the performance of the IPW, DR and MR estimators. Figure 23 shows results for three different choices of parameter configurations. Overall, we observe that in all settings, the MR and self-normalised MR (SNMR) estimator outperform all other baselines including the self-normalised IPW and DR estimators (denoted as SNIPW and SNDR respectively). Moreover, in some settings, where the importance ratios achieve very high values, self-normalisation can reduce the variance and MSE of the corresponding estimator (for example, Figure 23b). However, we also observe cases in which self-normalization does not significantly change the results (Figure 23a), or may even slightly worsen the MSE of the estimators (Figure 23c).
(a) Results with varying  $n$  for  $\alpha^{*} = 0.2$  and  $m = 500$
(b) Results with varying  $\alpha^{*}$  for  $n = 500$  and  $m = 1000$
(c) Results with varying  $m$  for  $\alpha^{*} = 0.6$  and  $n = 500$
Figure 16: Results for Digits dataset. Note that compared to other datasets we consider smaller maximum dataset sizes  $m, n$  here as the total number of available datapoints was 1797.
(a) Results with varying  $n$  for  $\alpha^{*} = 0.4$  and  $m = 2000$
(b) Results with varying  $\alpha^{*}$  for  $n = 100$  and  $m = 2000$
(c) Results with varying  $m$  for  $\alpha^{*} = 0.4$  and  $n = 100$
Figure 17: Results for CIFAR-100 dataset.
(a)  $d = 1000$ ,  $n_a = 100$ ,  $\alpha^* = 0.4$ .
(b)  $d = 10000$ ,  $n_a = 100$ ,  $\alpha^* = 0.4$ .
Figure 18: Results with varying size of evaluation dataset  $n$ .
(a)  $d = 1000$ ,  $n_a = 100$ ,  $n = 100$ .
(b)  $d = 10000, n_{a} = 100, n = 100$ .
Figure 19: Results with varying  $\alpha^{*}$
(a)  $n_a = 100$ ,  $n = 100$ ,  $\alpha^* = 0.4$ .
(b)  $n_a = 500$ ,  $n = 100$ ,  $\alpha^* = 0.4$ .
Figure 20: Results with varying context dimensions  $d$ .
(a)  $d = 100$ ,  $n = 100$ ,  $\alpha^{*} = 0.2$
(b)  $d = 100$ ,  $n = 100$ ,  $\alpha^{*} = 0.4$ .
Figure 21: Results with varying number of actions  $n_a$ .
(a)  $d = 5000$ ,  $n = 100$ ,  $n_a = 10$ ,  $\alpha^* = 0.2$ .
(b)  $d = 5000$ ,  $n = 100$ ,  $n_a = 10$ ,  $\alpha^* = 0.4$ .
Figure 22: Results with varying number of training data  $m$ .
(a)  $d = 10000$ ,  $n = 200$ ,  $n_a = 20$ ,  $m = 5000$ .
(b)  $d = 5000$ ,  $n = 200$ ,  $n_a = 20$ ,  $m = 1000$ .
(c)  $d = 10000$ ,  $n = 200$ ,  $n_a = 20$ ,  $m = 5000$ .
Figure 23: Results for self-normalised estimators with varying target policy shift  $\alpha^{*}$  for synthetic data setup considered in Section 5.1. Here, "SN" denotes self-normalised estimators.