SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof of Proposition 3.4. Let  $\hat{\mu}(a, x) \approx \mathbb{E}[Y \mid X = x, A = a]$  denote the outcome model in DR estimator. Then, using multiple applications of the law of total variance we get that
$$
\begin{array}{l} n \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {D R}} \right] = \operatorname {V a r} _ {\pi^ {b}} \left[ \rho (A, X) \left(Y - \hat {\mu} (A, X)\right) + \sum_ {a ^ {\prime} \in A} \hat {\mu} \left(a ^ {\prime}, X\right) \pi^ {*} \left(a ^ {\prime} \mid X\right) \right] \\ = \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) (Y - \hat {\mu} (A, X)) + \mathbb {E} _ {\pi^ {*}} [ \hat {\mu} (A, X) \mid X ] ] \\ = \mathbb {E} _ {\pi^ {b}} [ \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) (Y - \hat {\mu} (A, X)) + \mathbb {E} _ {\pi^ {*}} [ \hat {\mu} (A, X) | X ] | X, A) ] \\ + \operatorname {V a r} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) (Y - \hat {\mu} (A, X)) + \mathbb {E} _ {\pi^ {*}} [ \hat {\mu} (A, X) | X ] | X, A ] \right] \\ = \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) ^ {2} \operatorname {V a r} [ Y \mid X, A ] ] \\ + \operatorname {V a r} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) (Y - \hat {\mu} (A, X)) + \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) \hat {\mu} (A, X) | X ] | X, A ] \right] \\ = \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) ^ {2} \operatorname {V a r} [ Y \mid X, A ] ] \\ + \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) (\mu (A, X) - \hat {\mu} (A, X)) + \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) \hat {\mu} (A, X) \mid X) ] \\ = \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) ^ {2} \operatorname {V a r} [ Y \mid X, A ] ] \\ + \operatorname {V a r} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) (\mu (A, X) - \hat {\mu} (A, X)) + \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) \hat {\mu} (A, X) | X ] | X ] \right] \\ + \mathbb {E} _ {\pi^ {b}} [ \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) (\mu (A, X) - \hat {\mu} (A, X)) + \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) \hat {\mu} (A, X) | X ] | X ] ] \\ = \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) ^ {2} \operatorname {V a r} [ Y \mid X, A ] ] + \operatorname {V a r} _ {\pi^ {b}} [ \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) \mu (A, X) \mid X) ] \\ + \mathbb {E} _ {\pi^ {b}} [ \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) (\mu (A, X) - \hat {\mu} (A, X)) \mid X) ] \\ \geq \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) ^ {2} \operatorname {V a r} [ Y \mid X, A ] ] + \operatorname {V a r} _ {\pi^ {b}} [ \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) \mu (A, X) \mid X ] ]. \\ \end{array}
$$
Using this, we get that
$$
\begin{array}{l} n \left(\operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {D R}} \right] - \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {M R}} \right]\right) \\ \geq \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) ^ {2} \operatorname {V a r} [ Y \mid X, A ] ] + \operatorname {V a r} _ {\pi^ {b}} [ \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) \mu (A, X) \mid X ] ] - \operatorname {V a r} _ {\pi^ {b}} [ w (Y) Y ]. \\ \end{array}
$$
Again, using the law of total variance,
$$
\begin{array}{l} \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) Y ] = \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) Y \mid X, A ] \right] + \operatorname {V a r} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) Y \mid X, A ] \right] \\ = \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) ^ {2} \operatorname {V a r} [ Y \mid X, A ] ] + \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) \mu (A, X) ] \\ = \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) ^ {2} \operatorname {V a r} [ Y \mid X, A ] ] + \operatorname {V a r} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) \mu (A, X) \mid X ] \right] \\ + \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) \mu (A, X) \mid X ] \right]. \\ \end{array}
$$
Rearranging and substituting back into the expression earlier, we get that
$$
\begin{array}{l} n \left(\operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {D R}} \right] - \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {M R}} \right]\right) \\ \geq \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) Y ] - \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) \mu (A, X) \mid X ] \right] - \operatorname {V a r} _ {\pi^ {b}} [ w (Y) Y ]. \\ \end{array}
$$
Now, from Proposition 3.3 we know that
$$
n \big (\mathrm {V a r} _ {\pi^ {b}} [ \hat {\theta} _ {\mathrm {I P W}} ] - \mathrm {V a r} _ {\pi^ {b}} [ \hat {\theta} _ {\mathrm {M R}} ] \big) = \mathrm {V a r} _ {\pi^ {b}} [ \rho (A, X) Y ] - \mathrm {V a r} _ {\pi^ {b}} [ w (Y) Y ] = \mathbb {E} _ {\pi^ {b}} \left[ \mathrm {V a r} _ {\pi^ {b}} [ \rho (A, X) | Y ] Y ^ {2} \right].
$$
$$
\begin{array}{l} n \left(\operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {D R}} \right] - \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {M R}} \right]\right) \\ \geq \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) \mid Y ] Y ^ {2} \right] - \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) \mu (A, X) \mid X ] \right] \\ = \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) Y \mid Y ] - \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) \mu (A, X) \mid X ] \right]. \\ \end{array}
$$
Proof of Theorem 3.6. This result follows straightforwardly from Proposition D.4 in Appendix D.
Proof of Proposition 3.7.
$$
\begin{array}{l} \operatorname {B i a s} \left(\hat {\theta} _ {\mathrm {I P W}}\right) = \mathbb {E} _ {\pi^ {b}} [ \hat {\rho} (A, X) Y ] - \mathbb {E} _ {\pi^ {*}} [ Y ] \\ = \mathbb {E} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} [ \hat {\rho} (A, X) \mid Y ] Y \right] - \mathbb {E} _ {\pi^ {*}} [ Y ] \\ = \mathbb {E} _ {\pi^ {b}} [ \hat {w} (Y) Y ] - \mathbb {E} _ {\pi^ {b}} [ \epsilon Y ] - \mathbb {E} _ {\pi^ {*}} [ Y ] \\ = \operatorname {B i a s} \left(\hat {\theta} _ {\mathrm {M R}}\right) - \mathbb {E} _ {\pi^ {b}} [ \epsilon Y ]. \\ \end{array}
$$
Next, to prove the variance result, we first use the law of total variance to obtain
$$
\begin{array}{l} \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {I P W}} \right] = \frac {1}{n} \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\rho} (A, X) Y \right] \\ = \frac {1}{n} \left(\operatorname {V a r} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} [ \hat {\rho} (A, X) Y \mid Y ] \right] + \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} [ \hat {\rho} (A, X) Y \mid Y ] \right]\right) \\ = \frac {1}{n} \left(\operatorname {V a r} _ {\pi^ {b}} [ \tilde {w} (Y) Y ] + \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} [ \hat {\rho} (A, X) Y \mid Y ] \right]\right). \\ \end{array}
$$
Moreover, using the fact that  $\hat{w}(Y) = \tilde{w}(Y) + \epsilon$  we get that,
$$
\begin{array}{l} \operatorname {V a r} _ {\pi^ {b}} [ \hat {\theta} _ {\mathrm {M R}} ] = \frac {1}{n} \operatorname {V a r} _ {\pi^ {b}} [ \hat {w} (Y) Y ] \\ = \frac {1}{n} \operatorname {V a r} _ {\pi^ {b}} [ (\tilde {w} (Y) + \epsilon) Y ] \\ = \frac {1}{n} \left(\operatorname {V a r} _ {\pi^ {b}} [ \tilde {w} (Y) Y ] + \operatorname {V a r} _ {\pi^ {b}} [ \epsilon Y ] + 2 \operatorname {C o v} (\tilde {w} (Y) Y, \epsilon Y)\right). \\ \end{array}
$$
Putting together the two variance expressions derived above, we get that
$$
\begin{array}{l} \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {I P W}} \right] - \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {M R}} \right] \\ = \frac {1}{n} \left(\mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} [ \hat {\rho} (A, X) \mid Y ] Y ^ {2} \right] - \operatorname {V a r} _ {\pi^ {b}} [ \epsilon Y ] - 2 \operatorname {C o v} (\tilde {w} (Y) Y, \epsilon Y)\right). \\ \end{array}
$$
B Comparison with extensions of the doubly robust estimator
In this section, we theoretically investigate the variance of MR against the commonly used extensions of the DR estimator, namely Switch-DR [5] and DR with Optimistic Shrinkage (DRos) [17]. At a high level, these estimators seek to reduce the variance of the vanilla DR estimator by considering modified importance weights, thereby trading off the variance for additional bias. Below, we provide the explicit definitions of these estimators for completeness.
Switch-DR estimator The original DR estimator can still have a high variance when the importance weights are large due to a large policy shift. Switch-DR [5] aims to circumvent this problem by switching to DM when the importance weights are large:
$$
\hat {\theta} _ {\text {S w i t c h D R}} := \frac {1}{n} \sum_ {i = 1} ^ {n} \rho \left(a _ {i}, x _ {i}\right) \left(y _ {i} - \hat {\mu} \left(a _ {i}, x _ {i}\right)\right) \mathbb {1} \left(\rho \left(a _ {i}, x _ {i}\right) \leq \tau\right) + \hat {\eta} \left(\pi^ {*}\right),
$$
where  $\tau \geq 0$  is a hyperparameter,  $\hat{\mu}(a,x) \approx \mathbb{E}[Y \mid X = x, A = a]$  is the outcome model, and
$$
\hat {\eta} \left(\pi^ {*}\right) = \frac {1}{n} \sum_ {i = 1} ^ {n} \sum_ {a ^ {\prime} \in \mathcal {A}} \hat {\mu} \left(a ^ {\prime}, x _ {i}\right) \pi^ {*} \left(a ^ {\prime} \mid x _ {i}\right) \approx \mathbb {E} _ {\pi^ {*}} [ \hat {\mu} (A, X) ]
$$
where  $a_i^* \sim \pi^*(\cdot \mid x_i)$ .
Doubly Robust with Optimal Shrinkage (DRos) DRos proposed by [17] uses new weights  $\hat{\rho}_{\lambda}(a_i, x_i)$  which directly minimises sharp bounds on the MSE of the resulting estimator,
$$
\hat {\theta} _ {\mathrm {D R o s}} := \frac {1}{n} \sum_ {i = 1} ^ {n} \hat {\rho} _ {\lambda} \left(a _ {i}, x _ {i}\right) \left(y _ {i} - \hat {\mu} \left(a _ {i}, x _ {i}\right)\right) + \hat {\eta} \left(\pi^ {*}\right),
$$
where  $\lambda \geq 0$  is a pre-defined hyperparameter and  $\hat{\rho}_{\lambda}$  is defined as
$$
\hat {\rho} _ {\lambda} (a, x) := \frac {\lambda}{\rho^ {2} (a , x) + \lambda} \rho (a, x).
$$
When  $\lambda = 0$ ,  $\hat{\rho}_{\lambda}(a,x) = 0$  leads to DM, whereas as  $\lambda \to \infty$ ,  $\hat{\rho}_{\lambda}(a,x) \to \rho(a,x)$  leading to DR.
More generally, both of these estimators can be written as follows:
$$
\hat {\theta} _ {\mathrm {D R}} ^ {\tilde {\rho}} := \frac {1}{n} \sum_ {i = 1} ^ {n} \tilde {\rho} \left(a _ {i}, x _ {i}\right) \left(y _ {i} - \hat {\mu} \left(a _ {i}, x _ {i}\right)\right) + \hat {\eta} \left(\pi^ {*}\right).
$$
Here, when  $\tilde{\rho}(a,x) = \rho(a,x)\mathbb{1}(\rho(a_i,x_i) \leq \tau)$ , we recover the Switch-DR estimator and likewise when  $\tilde{\rho}(a,x) = \hat{\rho}_{\lambda}(a,x)$ , we recover DRos.
B.1 Variance comparison with the DR extensions
Next, we provide a theoretical result comparing the variance of the MR estimator with these DR extension methods.
Proposition B.1. When the weights  $w(y)$  are known exactly and the outcome model is exact, i.e.,  $\hat{\mu}(a,x) = \mu(a,x) = \mathbb{E}[Y \mid X = x, A = a]$  in the DR estimator  $\hat{\theta}_{\mathrm{DR}}^{\tilde{\rho}}$  defined above,
$$
\operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {D R}} ^ {\tilde {\rho}} \right] - \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {M R}} \right] \geq \frac {1}{n} \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} \left[ \rho (A, X) \mid Y \right] Y ^ {2} - \operatorname {V a r} _ {\pi^ {b}} \left[ \rho (A, X) \mu (A, X) \mid X \right] \right] - \Delta ,
$$
$$
w h e r e \Delta := \frac {1}{n} \mathbb {E} _ {\pi^ {b}} \left[ \left(\rho^ {2} (A, X) - \tilde {\rho} ^ {2} (A, X)\right) \operatorname {V a r} [ Y \mid X, A ] \right].
$$
Proof of Proposition B.1. Using the fact that  $\hat{\mu}(a, x) = \mu(a, x)$  and the law of total variance, we get that
$$
\begin{array}{l} n \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {D R}} ^ {\tilde {\rho}} \right] = \operatorname {V a r} _ {\pi^ {b}} \left[ \tilde {\rho} (A, X) \left(Y - \hat {\mu} (A, X)\right) + \sum_ {a ^ {\prime} \in \mathcal {A}} \hat {\mu} \left(a ^ {\prime}, X\right) \pi^ {*} \left(a ^ {\prime} \mid X\right) \right] \\ = \operatorname {V a r} _ {\pi^ {b}} [ \tilde {\rho} (A, X) (Y - \hat {\mu} (A, X)) + \mathbb {E} _ {\pi^ {*}} [ \hat {\mu} (A, X) \mid X) ] \\ = \operatorname {V a r} _ {\pi^ {b}} [ \tilde {\rho} (A, X) (Y - \mu (A, X)) + \mathbb {E} _ {\pi^ {*}} [ \mu (A, X) \mid X ] ] \\ = \operatorname {V a r} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} [ \tilde {\rho} (A, X) (Y - \mu (A, X)) + \mathbb {E} _ {\pi^ {*}} [ \mu (A, X) | X ] | X, A ] \right] \\ + \mathbb {E} _ {\pi^ {b}} [ \operatorname {V a r} _ {\pi^ {b}} [ \tilde {\rho} (A, X) (Y - \mu (A, X)) + \mathbb {E} _ {\pi^ {*}} [ \mu (A, X) | X ] | X, A) ] \\ = \operatorname {V a r} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {*}} [ \mu (A, X) \mid X ] \right] + \mathbb {E} _ {\pi^ {b}} \left[ \tilde {\rho} ^ {2} (A, X) \operatorname {V a r} [ Y \mid X, A ] \right] \\ = \operatorname {V a r} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {*}} [ \mu (A, X) \mid X ] \right] + \mathbb {E} _ {\pi^ {b}} \left[ \rho^ {2} (A, X) \operatorname {V a r} [ Y \mid X, A ] \right] \\ + \underbrace {\mathbb {E} _ {\pi^ {b}} [ (\tilde {\rho} ^ {2} (A , X) - \rho^ {2} (A , X)) \operatorname {V a r} [ Y \mid X , A ] ]} _ {- n \Delta} \\ = \operatorname {V a r} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) \mu (A, X) \mid X ] \right] + \mathbb {E} _ {\pi^ {b}} [ \rho^ {2} (A, X) \operatorname {V a r} [ Y \mid X, A ] ] - n \Delta . \\ \end{array}
$$
Again, using the law of total variance we can rewrite the second term on the RHS above as,
$$
\begin{array}{l} \mathbb {E} _ {\pi^ {\flat}} \left[ \rho^ {2} (A, X) \operatorname {V a r} [ Y \mid X, A ] \right] \\ = \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) Y ] - \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) \mu (A, X) ] \\ = \operatorname {V a r} _ {\pi^ {b}} [ \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) \mid Y ] Y ] + \mathbb {E} _ {\pi^ {b}} [ \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) \mid Y ] Y ^ {2} ] \\ - \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) \mu (A, X) ] \\ = \operatorname {V a r} _ {\pi^ {b}} [ w (Y) Y ] + \mathbb {E} _ {\pi^ {b}} [ \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) \mid Y ] Y ^ {2} ] - \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) \mu (A, X) ] \\ = n \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {M R}} \right] + \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) \mid Y ] Y ^ {2} \right] - \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) \mu (A, X) ]. \\ \end{array}
$$
Putting this together, we get that
$$
\begin{array}{l} n \operatorname {V a r} _ {\pi^ {b}} [ \hat {\theta} _ {\mathrm {D R}} ^ {\tilde {\rho}} ] \\ = n \operatorname {V a r} _ {\pi^ {b}} [ \hat {\theta} _ {\mathrm {M R}} ] + \mathbb {E} _ {\pi^ {b}} [ \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) \mid Y ] Y ^ {2} ] - \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) \mu (A, X) ] \\ + \operatorname {V a r} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) \mu (A, X) \mid X ] \right] - n \Delta \\ = n \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {M R}} \right] + \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) \mid Y ] Y ^ {2} \right] - \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) \mu (A, X) \mid X ] \right] - n \Delta , \\ \end{array}
$$
where in the last step above, we again use the law of total variance. Rearranging the above leads us to the result.
Intuition Note that for both of the DR extensions under consideration, the modified ratios  $\tilde{\rho}(a,x)$  satisfy  $0 \leq \tilde{\rho}(a,x) \leq \rho(a,x)$  and hence  $\Delta \geq 0$  (using the definition of  $\Delta$  in Proposition B.1). When the modified ratios  $\tilde{\rho}(a,x)$  are 'close' to the true policy ratios  $\rho(a,x)$ , then using the definition of  $\Delta$ , we have that  $\Delta \approx 0$ . In this case, the result above provides a similar intuition to Proposition 3.4 in the main text. Specifically, in this case we have that if  $\mathrm{Var}_{\pi^b}[\rho(A,X)Y|Y]$  is greater than  $\mathrm{Var}_{\pi^b}[\rho(A,X)\mu(A,X)|X]$  on average, the variance of the MR estimator will be less than that of the DR extension under consideration. Intuitively, this will occur when the dimension of context space  $\mathcal{X}$  is high because in this case the conditional variance over  $X$  and  $A$ ,  $\mathrm{Var}_{\pi^b}[\rho(A,X)Y|Y]$  is likely to be greater than the conditional variance over  $A$ ,  $\mathrm{Var}_{\pi^b}[\rho(A,X)\mu(A,X)|X]$ .
In contrast if the modified ratios  $\tilde{\rho}(a, x)$  differ substantially from  $\rho(a, x)$ , then  $\Delta$  will be large and the variance of MR may be higher than that of the resulting DR extension. However, this comes at the cost of significantly higher bias in the DR extension and consequently MSE of the DR extension will be high in this case.
C Weight estimation error
In this section, we theoretically investigate the effects of using the estimated importance weights  $\hat{w}(y)$  rather than  $\hat{\rho}(a,x)$  on the bias and variance of the resulting OPE estimator. Further to our discussion in Section 3.1.2, we focus in this section on the approximation error when using a wide neural network to estimate the weights  $\hat{w}(y)$ . To this end, we use recent results regarding the generalization of wide neural networks [19] to show that the estimation error of the approximation step (ii) in the Section 3.1.2 declines with increasing number of training data when  $\hat{w}(y)$  is estimated using wide neural networks. Before providing the main result, we explicitly lay out the assumptions needed.
C.1 Using wide neural networks to approximate the weights  $\hat{w}(y)$
Assumption C.1. Let  $\tilde{w}(y) \coloneqq \mathbb{E}_{\pi^b}[\hat{\rho}(A, X) \mid Y = y]$ . Suppose  $\tilde{w} \in \mathcal{H}_1$  and  $||\tilde{w}||_{\mathcal{H}_1} \leq R$  for some constant  $R$ , where  $\mathcal{H}_1$  is the reproducing kernel Hilbert space (RKHS) associated with the Neural Tangent Kernel  $K_1$  associated with 2 layer neural network defined on  $\mathbb{R}$ .
Assumption C.2. There exists an  $M \in [0, \infty)$  such that  $\mathbb{P}_{\pi^b}(|Y| \leq M) = 1$ .
Assumption C.3.  $\hat{\rho} (a_i,x_i)$  satisfies
$$
\hat {\rho} \left(a _ {i}, x _ {i}\right) = \tilde {w} \left(y _ {i}\right) + \eta_ {i},
$$
where  $\eta_i\stackrel {\mathrm{iid}}{\sim}\mathcal{N}(0,\sigma^2)$  for some  $\sigma >0$
Theorem C.4. Suppose that the IPW and MR estimators are defined as,
$$
\tilde {\theta} _ {\mathrm {I P W}} := \frac {1}{n} \sum_ {i = 1} ^ {n} \hat {\rho} \left(a _ {i}, x _ {i}\right) y _ {i}, \quad a n d \quad \tilde {\theta} _ {\mathrm {M R}} := \frac {1}{n} \sum_ {i = 1} ^ {n} \hat {w} _ {m} \left(y _ {i}\right) y _ {i},
$$
where  $\hat{w}_m(y)$  is obtained by regressing to the estimated policy ratios  $\hat{\rho}(a, x)$  using m i.i.d. training samples  $\mathcal{D}_{\mathrm{tr}} := \{(x_i^{\mathrm{tr}}, a_i^{\mathrm{tr}}, y_i^{\mathrm{tr}})\}_{i=1}^{m}$ , i.e., by minimising the loss
$$
\mathcal {L} (\phi) = \mathbb {E} _ {(X, A, Y) \sim \mathcal {D} _ {\mathrm {t r}}} \left[ \left(\hat {\rho} (A, X) - f _ {\phi} (Y)\right) ^ {2} \right].
$$
Suppose Assumptions C.1-C.3 hold, then for any given  $\delta \in (0,1)$ , if  $f_{\phi}$  is a two-layer neural network with width  $k$  that is sufficiently large and stops the gradient flow at time  $t_* \propto m^{2/3}$ , then for sufficiently large  $m$ , there exists a constant  $C_1$  independent of  $\delta$  and  $m$ , such that
$$
\left| \operatorname {B i a s} \left(\tilde {\theta} _ {\mathrm {M R}}\right) - \operatorname {B i a s} \left(\tilde {\theta} _ {\mathrm {I P W}}\right) \right| \leq C _ {1} m ^ {- 1 / 3} \log \frac {6}{\delta}
$$
holds with probability at least  $(1 - \delta)(1 - o_k(1))$ . Moreover, there exist constants  $C_2, C_3$  independent of  $\delta$  and  $m$  such that
$$
n\bigl(\operatorname{Var}_{\pi^{b}}\bigl[\tilde{\theta}_{\mathrm{IPW}}\bigr ] - \operatorname{Var}_{\pi^{b}}\bigl[\tilde{\theta}_{\mathrm{MR}}\bigr ]\bigr)\geq \underbrace{\mathbb{E}_{\pi^{b}}\bigl[ \operatorname{Var}_{\pi^{b}}\bigl[ \hat{\rho} (A,X)  Y  |  Y\bigr]\bigr]}_{\geq 0} - C_{2}  m^{-2 / 3}  \log^{2}\frac{6}{\delta} -C_{3}  m^{-1 / 3}  \log \frac{6}{\delta}
$$
holds with probability at least  $(1 - \delta)(1 - o_k(1))$ . Here, the randomness comes from the joint distribution of training samples and random initialization of parameters in the neural network  $f_{\phi}$ .
Proof of Theorem C.4. The proof of this theorem relies on [19, Theorem 4.1]. Recall the definition  $\tilde{w}(Y) \coloneqq \mathbb{E}_{\pi^b}[\hat{\rho}(A,X) \mid Y]$ . We can rewrite our setup in the setting of [19, Theorem 4.1], by relabelling  $\hat{\rho}(a,x)$  in our setup as  $y$  in their setup and relabelling  $y$  in our setup as  $x$  in their setup. Then, given  $\delta \in (0,1)$ , from [19, Theorem 4.1], it follows that under Assumptions C.1-C.3 that there exists a constant  $C$  independent of  $\delta$  and  $m$ , such that
$$
\mathbb {E} _ {\pi^ {b}} \left[ \epsilon^ {2} \right] \leq C m ^ {- 2 / 3} \log^ {2} \frac {6}{\delta} \tag {5}
$$
holds with probability at least  $(1 - \delta)(1 - o_k(1))$ , where  $\epsilon \coloneqq \hat{w}_m(Y) - \tilde{w}(Y)$ . Recall from Proposition 3.7 that
$$
\left| \operatorname {B i a s} \left(\tilde {\theta} _ {\mathrm {M R}}\right) - \operatorname {B i a s} \left(\tilde {\theta} _ {\mathrm {I P W}}\right) \right| = \left| \mathbb {E} _ {\pi^ {b}} [ \epsilon Y ] \right|.
$$
From this it follows using Cauchy-Schwarz inequality that,
$$
\left| \operatorname {B i a s} \left(\tilde {\theta} _ {\mathrm {M R}}\right) - \operatorname {B i a s} \left(\tilde {\theta} _ {\mathrm {I P W}}\right) \right| = \left| \mathbb {E} _ {\pi^ {b}} [ \epsilon Y ] \right| \leq \left(\mathbb {E} _ {\pi^ {b}} [ \epsilon^ {2} ] \mathbb {E} _ {\pi^ {b}} [ Y ^ {2} ]\right) ^ {1 / 2}.
$$
Combining the above with Eqn. (5), it follows that,
$$
| \mathrm {B i a s} (\tilde {\theta} _ {\mathrm {M R}}) - \mathrm {B i a s} (\tilde {\theta} _ {\mathrm {I P W}}) | \leq C ^ {1 / 2} m ^ {- 1 / 3} \log \frac {6}{\delta} (\mathbb {E} _ {\pi^ {b}} [ Y ^ {2} ]) ^ {1 / 2} = C _ {1} m ^ {- 1 / 3} \log \frac {6}{\delta}
$$
holds with probability at least  $(1 - \delta)(1 - o_k(1))$ , where  $C_1 = C^{1/2}(\mathbb{E}_{\pi^b}[Y^2])^{1/2}$ .
Next, to prove the variance result, we recall from Proposition 3.7 that
$$
n \left(\operatorname {V a r} _ {\pi^ {b}} \left[ \tilde {\theta} _ {\mathrm {I P W}} \right] - \operatorname {V a r} _ {\pi^ {b}} \left[ \tilde {\theta} _ {\mathrm {M R}} \right]\right) = \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} [ \hat {\rho} (A, X) \mid Y ] Y ^ {2} \right] - \operatorname {V a r} _ {\pi^ {b}} [ \epsilon Y ] - 2 \operatorname {C o v} (\epsilon Y, \tilde {w} (Y) Y)
$$
Now note that, under Assumption C.2,
$$
\operatorname {V a r} _ {\pi^ {b}} [ \epsilon Y ] \leq \mathbb {E} _ {\pi^ {b}} [ (\epsilon Y) ^ {2} ] \leq M ^ {2} \mathbb {E} _ {\pi^ {b}} [ \epsilon^ {2} ] \leq C M ^ {2} m ^ {- 2 / 3} \log^ {2} \frac {6}{\delta} = C _ {2} m ^ {- 2 / 3} \log^ {2} \frac {6}{\delta},
$$
holds with probability at least  $(1 - \delta)(1 - o_k(1))$ , where  $C_2 = CM^2$ . Similarly, we have that with probability at least  $(1 - \delta)(1 - o_k(1))$