SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} | \operatorname {C o v} (\epsilon Y, \tilde {w} (Y) Y) | = | \mathbb {E} _ {\pi^ {b}} [ \epsilon \tilde {w} (Y) Y ^ {2} ] - \mathbb {E} _ {\pi^ {b}} [ \epsilon Y ] \mathbb {E} _ {\pi^ {b}} [ \tilde {w} (Y) Y ] | \\ \leq \left| \mathbb {E} _ {\pi^ {b}} [ \epsilon \tilde {w} (Y) Y ^ {2} ] \right| + \left| \mathbb {E} _ {\pi^ {b}} [ \epsilon Y ] \mathbb {E} _ {\pi^ {b}} [ \tilde {w} (Y) Y ] \right| \\ \leq \left(\mathbb {E} _ {\pi^ {b}} [ \epsilon^ {2} ] \mathbb {E} _ {\pi^ {b}} [ \tilde {w} (Y) ^ {2} Y ^ {4} ]\right) ^ {1 / 2} + (\mathbb {E} _ {\pi^ {b}} [ \epsilon^ {2} ] \mathbb {E} _ {\pi^ {b}} [ Y ^ {2} ]) ^ {1 / 2} | \mathbb {E} _ {\pi^ {b}} [ \tilde {w} (Y) Y ] | \\ = \left(\mathbb {E} _ {\pi^ {b}} [ \epsilon^ {2} ]\right) ^ {1 / 2} \left(\left(\mathbb {E} _ {\pi^ {b}} [ \tilde {w} (Y) ^ {2} Y ^ {4} ]\right) ^ {1 / 2} + \left(\mathbb {E} _ {\pi^ {b}} [ Y ^ {2} ]\right) ^ {1 / 2} | \mathbb {E} _ {\pi^ {b}} [ \tilde {w} (Y) Y ] |\right) \\ \leq C _ {3} m ^ {- 1 / 3} \log \frac {6}{\delta}, \\ \end{array}
$$
where  $C_3 = C(\mathbb{E}_{\pi^b}[\tilde{w}(Y)^2 Y^4])^{1/2} + (\mathbb{E}_{\pi^b}[Y^2])^{1/2}|\mathbb{E}_{\pi^b}[\tilde{w}(Y)Y]|$ , and we use Cauchy-Schwarz inequality in the third step above. Putting this together, we obtain the required result.  $\square$
Intuition This theorem shows that as the number of training samples  $m$  increases, the biases of MR and IPW estimators become roughly equal, whereas the variance of MR estimator falls below that of the IPW estimator. The empirical results shown in Appendix F.2 are consistent with this result. Moreover, in Theorem C.4, the estimated policy ratio  $\hat{\rho}(a, x)$  is fixed for increasing  $m$ , i.e., we do not update  $\hat{\rho}(a, x)$  as more training data becomes available. While this may seem as a disadvantage for the IPW estimator, we point out that the result also holds when the policy ratio is exact (i.e.,  $\hat{\rho}(a, x) = \rho(a, x)$ ) and hence the IPW estimator is unbiased.
Relaxing Assumption C.3 [19][Theorem 4.1] suppose that the data has the relationship shown in Assumption C.3. However, the theorem relies on Corollary 4.4 in [43], which requires a strictly weaker assumption (Assumption 1 in [43]). Therefore, we can relax Assumption C.3 to the following assumption.
Assumption C.5. There exists positive constants  $Q$  and  $M$  such that for all  $l \geq 2$  with  $l \in \mathbb{N}$
$$
\mathbb {E} _ {\pi^ {b}} [ \hat {\rho} (A, X) ^ {l} \mid Y ] \leq \frac {1}{2} l! M ^ {l - 2} Q ^ {2}
$$
$p_{\pi^b}$  -almost surely.
It is easy to check that Assumption C.5 is strictly weaker than Assumption C.3, and is also satisfied if the policy ratio  $\hat{\rho}(A,X)$  is almost surely bounded. For simplicity, we use the stronger assumption in our Proposition C.4.
D Generalised formulation of the MIPS estimator [14]
As described in Section 3.1.1, the MIPS estimator proposed by [14] assumes the existence of action embeddings  $E$  which summarise all relevant information about the action  $A$ , and achieves a lower variance than the IPW estimator. To achieve this, the MIPS estimator only considers the shift in the distribution of  $(X, E)$  as a result of policy shift, instead of considering the shift in  $(X, A)$  (as in IPW estimator). In this section, we show that this idea can be generalised to instead consider general representations  $R$  of the context-action pair  $(X, A)$ , which encapsulate all relevant information about the outcome  $Y$ . The MIPS estimator is a special case of this generalised setting where the representation  $R$  is of the form  $(X, E)$ .
Generalised MIPS (G-MIPS) estimator Suppose that there exists an embedding  $R$  of the context-action pair  $(X, A)$ , with the Bayesian network shown in Figure 3. Here,  $R$  may be a lower-dimensional representation of the  $(X, A)$  pair which contains all the information necessary to predict the outcome  $Y$ . This corresponds to the following conditional independence assumption:
Assumption D.1. The context-action pair  $(X,A)$  has no direct effect on the outcome  $Y$  given  $R$ , i.e.,  $Y \perp (X,A) \mid R$ .
Figure 3: Bayesian network corresponding to Assumption D.1.
As illustrated in Figure 3, Assumption D.1 means that the embedding  $R$  fully mediates every possible effect of  $(X,A)$  on  $Y$ . The generalised MIPS estimator  $\hat{\theta}_{\mathrm{G - MIPS}}$  of target policy value,  $\mathbb{E}_{\pi^{*}}[Y]$ , is defined as
$$
\hat {\theta} _ {\mathrm {G - M I P S}} := \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {p _ {\pi^ {*}} (r _ {i})}{p _ {\pi^ {b}} (r _ {i})} y _ {i},
$$
where  $p_{\pi^b}(r)$  denote the density of  $R$  under the behaviour policy (likewise for  $p_{\pi^*}(r)$ ). Under assumption D.1,
$\hat{\theta}_{\mathrm{G - MIPS}}$  provides an unbiased estimator of target policy value. Similar to Lemma 3.1, the density ratio  $\frac{p_{\pi^*}(r)}{p_{\pi^b}(r)}$  can be estimated by solving the regression problem
$$
\arg \min  _ {f} \mathbb {E} _ {\pi^ {b}} \left(\frac {\pi^ {*} (A \mid X)}{\pi^ {b} (A \mid X)} - f (R)\right) ^ {2}. \tag {6}
$$
D.1 Variance reduction of G-MIPS estimator
By only considering the shift in the embedding  $R$ , the G-MIPS estimator achieves a lower variance relative to the vanilla IPW estimator. The following result, which is a straightforward extension of [14, Theorem 3.6], formalises this.
Proposition D.2 (Variance reduction of G-MIPS). When the ratios  $\rho(a, x)$  and  $\frac{p_{\pi^*}(r)}{p_{\pi^b}(r)}$  are known exactly then under Assumption D.1, we have that  $\mathbb{E}_{\pi^b}[\hat{\theta}_{\mathrm{IPW}}] = \mathbb{E}_{\pi^b}[\hat{\theta}_{\mathrm{G-MIPS}}] = \mathbb{E}_{\pi^*}[Y]$ . Moreover,
$$
\operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {I P W}} \right] - \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {G - M I P S}} \right] \geq \frac {1}{n} \mathbb {E} _ {\pi^ {b}} \left[ \mathbb {E} \left[ Y ^ {2} \mid R \right] \operatorname {V a r} _ {\pi^ {b}} \left[ \rho (A, X) \mid R \right] \right] \geq 0.
$$
Proof of Proposition D.2. The following proof, which is included for completeness, is a straightforward extension of [14, Theorem 3.6].
$$
\begin{array}{l} n \left(\operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {I P W}} \right] - \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {M I P S}} \right]\right) \\ = \operatorname {V a r} _ {\pi^ {b}} \left[ \frac {\pi^ {*} (A | X)}{\pi^ {b} (A | X)} Y \right] - \operatorname {V a r} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R)}{p _ {\pi^ {b}} (R)} Y \right] \\ = \operatorname {V a r} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} \left[ \frac {\pi^ {*} (A | X)}{\pi^ {b} (A | X)} Y \mid R \right] \right] + \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} \left[ \frac {\pi^ {*} (A | X)}{\pi^ {b} (A | X)} Y \mid R \right] \right] - \operatorname {V a r} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R)}{p _ {\pi^ {b}} (R)} Y \mid R \right] \right] \\ - \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R)}{p _ {\pi^ {b}} (R)} Y \mid R \right] \right] \\ \end{array}
$$
Now using the conditional independence Assumption D.1, the first term on the RHS above becomes,
$$
\begin{array}{l} \operatorname {V a r} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} \left[ \frac {\pi^ {*} (A | X)}{\pi^ {b} (A | X)} Y \mid R \right] \right] = \operatorname {V a r} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} \left[ \frac {\pi^ {*} (A | X)}{\pi^ {b} (A | X)} \mid R \right] \mathbb {E} _ {\pi^ {b}} [ Y | R ] \right] \\ = \operatorname {V a r} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R)}{p _ {\pi^ {b}} (R)} \mathbb {E} _ {\pi^ {b}} [ Y | R ] \right], \\ \end{array}
$$
where in the last step above we use the fact that
$$
\mathbb {E} _ {\pi^ {b}} \left[ \frac {\pi^ {*} (A | X)}{\pi^ {b} (A | X)} \Bigg | R \right] = \frac {p _ {\pi^ {*}} (R)}{p _ {\pi^ {b}} (R)}.
$$
$$
\begin{array}{l} n \left(\operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {I P W}} \right] - \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {M I P S}} \right]\right) \\ = \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} \left[ \frac {\pi^ {*} (A | X)}{\pi^ {b} (A | X)} Y \mid R \right] \right] - \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R)}{p _ {\pi^ {b}} (R)} Y \mid R \right] \right]. \tag {7} \\ \end{array}
$$
Since we have that
$$
\mathbb {E} _ {\pi^ {b}} \left[ \frac {\pi^ {*} (A | X)}{\pi^ {b} (A | X)} Y \Bigg | R \right] = \mathbb {E} _ {\pi^ {b}} \left[ \frac {\pi^ {*} (A | X)}{\pi^ {b} (A | X)} \Bigg | R \right] \mathbb {E} _ {\pi^ {b}} [ Y | R ] = \frac {p _ {\pi^ {*}} (R)}{p _ {\pi^ {b}} (R)} \mathbb {E} _ {\pi^ {b}} [ Y | R ],
$$
Eq. (7) becomes,
$$
\begin{array}{l} \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} \left[ \frac {\pi^ {*} (A | X)}{\pi^ {b} (A | X)} Y \mid R \right] \right] - \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R)}{p _ {\pi^ {b}} (R)} Y \mid R \right] \right] \\ = \mathbb {E} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} \left[ \left(\frac {\pi^ {*} (A | X)}{\pi^ {b} (A | X)} Y\right) ^ {2} \mid R \right] - \mathbb {E} _ {\pi^ {b}} \left[ \left(\frac {p _ {\pi^ {*}} (R)}{p _ {\pi^ {b}} (R)} Y\right) ^ {2} \mid R \right] \right] \\ = \mathbb {E} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} \left[ \left(\frac {\pi^ {*} (A | X)}{\pi^ {b} (A | X)}\right) ^ {2} \Bigg | R \right] \mathbb {E} _ {\pi^ {b}} \left[ Y ^ {2} | R \right] - \left(\frac {p _ {\pi^ {*}} (R)}{p _ {\pi^ {b}} (R)}\right) ^ {2} \mathbb {E} _ {\pi^ {b}} \left[ Y ^ {2} | R \right] \right] \\ = \mathbb {E} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} [ Y ^ {2} | R ] \left(\mathbb {E} _ {\pi^ {b}} \left[ \left(\frac {\pi^ {*} (A | X)}{\pi^ {b} (A | X)}\right) ^ {2} \mid R \right] - \left(\mathbb {E} _ {\pi^ {b}} \left[ \frac {\pi^ {*} (A | X)}{\pi^ {b} (A | X)} \mid R \right]\right) ^ {2}\right) \right] \\ = \mathbb {E} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} \left[ Y ^ {2} | R \right] \operatorname {V a r} _ {\pi^ {b}} \left[ \frac {\pi^ {*} (A | X)}{\pi^ {b} (A | X)} \mid R \right] \right]. \\ \end{array}
$$
Intuition Here,  $R$  contains all relevant information regarding the outcome  $Y$ . Moreover, intuitively  $R$  can be thought of as the state obtained by 'filtering out' relevant information about  $Y$  from  $(X, A)$ . Therefore,  $R$  contains less 'redundant' information regarding the outcome  $Y$  as compared to the covariate-action pair  $(X, A)$ . As a result, the G-MIPS estimator which only considers the shift in the marginal distribution of  $R$  due to the policy shift is more efficient than the IPW estimator, which considers the shift in the joint distribution of  $(X, A)$  instead. In fact, as the amount of 'redundant' information regarding  $Y$  decreases in the embedding  $R$ , the G-MIPS estimator becomes increasingly efficient with decreasing variance. We formalise this as follows:
Assumption D.3. Assume there exist embeddings  $R^{(1)}, R^{(2)}$  of the covariate-action pair  $(X, A)$ , with Bayesian network shown in Figure 4. This corresponds to the following conditional independence assumptions:
$$
R ^ {(2)} \perp (X, A) \mid R ^ {(1)}, \qquad \text {a n d} \qquad Y \perp (R ^ {(1)}, X, A) \mid R ^ {(2)}.
$$
Figure 4: Bayesian network corresponding to Assumption D.3.
We can define G-MIPS estimators for these embeddings to obtain unbiased OPE estimators under Assumption D.3 as follows:
$$
\hat {\theta} _ {\mathrm {G - M I P S}} ^ {(j)} := \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {p _ {\pi^ {*}} (r _ {i} ^ {(j)})}{p _ {\pi^ {b}} (r _ {i} ^ {(j)})} y _ {i},
$$
for  $j \in \{1, 2\}$ . Here,  $\frac{p_{\pi^*}(r^{(j)})}{p_{\pi^b}(r^{(j)})}$  is the ratio of marginal densities of  $R^{(j)}$  under target and behaviour policies. We next show that the variance of  $\hat{\theta}_{\mathrm{G - MIPS}}^{(j)}$  decreases with increasing  $j$ .
Proposition D.4. When the ratios  $\rho(a, x)$ ,  $w(y)$  and  $\frac{p_{\pi^*}(r^{(j)})}{p_{\pi^b}(r^{(j)})}$  are known exactly for  $j \in \{1, 2\}$ , then under Assumption D.3 we get that
$$
\mathbb {E} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {I P W}} \right] = \mathbb {E} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {G - M I P S}} ^ {(1)} \right] = \mathbb {E} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {G - M I P S}} ^ {(2)} \right] = \mathbb {E} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {M R}} \right] = \mathbb {E} _ {\pi^ {*}} \left[ Y \right].
$$
$$
\operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {I P W}} \right] \geq \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {G - M I P S}} ^ {(1)} \right] \geq \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {G - M I P S}} ^ {(2)} \right] \geq \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {M R}} \right].
$$
Proof of Proposition D.4. First, we prove that the G-MIPS estimators are unbiased using induction on  $j$ . We define  $R^{(0)} \coloneqq (X, A)$  and  $\hat{\theta}_{\mathrm{G-MIPS}}^{(0)}$  defined as
$$
\hat {\theta} _ {\mathrm {G - M I P S}} ^ {(0)} := \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {p _ {\pi^ {*}} \left(r _ {i} ^ {(0)}\right)}{p _ {\pi^ {b}} \left(r _ {i} ^ {(0)}\right)} y _ {i},
$$
recovers the IPW estimator  $\hat{\theta}_{\mathrm{IPW}}$ . When  $j = 0$ , we know that  $\hat{\theta}_{\mathrm{G-MIPS}}^{(0)} = \hat{\theta}_{\mathrm{IPW}}$  is unbiased. Now, assume that  $\mathbb{E}_{\pi^b}[\hat{\theta}_{\mathrm{G-MIPS}}^{(j)}] = \mathbb{E}_{\pi^*}[Y]$ .
Conditional on  $R^{(j)}$ ,  $R^{(j + 1)}$  does not depend on the policy. Therefore,
$$
\frac {p _ {\pi^ {*}} (r ^ {(j)})}{p _ {\pi^ {b}} (r ^ {(j)})} = \frac {p _ {\pi^ {*}} (r ^ {(j)}) p (r ^ {(j + 1)} \mid r ^ {(j)})}{p _ {\pi^ {b}} (r ^ {(j)}) p (r ^ {(j + 1)} \mid r ^ {(j)})} = \frac {p _ {\pi^ {*}} (r ^ {(j)} , r ^ {(j + 1)})}{p _ {\pi^ {b}} (r ^ {(j)} , r ^ {(j + 1)})}.
$$
And therefore,
$$
\begin{array}{l} \frac {p _ {\pi^ {*}} \left(r ^ {(j + 1)}\right)}{p _ {\pi^ {b}} \left(r ^ {(j + 1)}\right)} = \int_ {r ^ {(j)}} \frac {p _ {\pi^ {*}} \left(r ^ {(j)} , r ^ {(j + 1)}\right)}{p _ {\pi^ {b}} \left(r ^ {(j)} , r ^ {(j + 1)}\right)} p _ {\pi^ {b}} \left(r ^ {(j)} \mid r ^ {(j + 1)}\right) d r ^ {(j)} \\ = \int_ {r ^ {(j)}} \frac {p _ {\pi^ {*}} (r ^ {(j)})}{p _ {\pi^ {b}} (r ^ {(j)})} p _ {\pi^ {b}} (r ^ {(j)} \mid r ^ {(j + 1)}) d r ^ {(j)} \\ = \mathbb {E} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R ^ {(j)})}{p _ {\pi^ {b}} (R ^ {(j)})} \Bigg | R ^ {(j + 1)} = r ^ {(j + 1)} \right]. \\ \end{array}
$$
Using this and the fact that  $R^{(j)} \perp Y \mid R^{(j+1)}$ , we get that
$$
\begin{array}{l} \mathbb {E} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {G - M I P S}} ^ {(j + 1)} \right] = \mathbb {E} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R ^ {(j + 1)})}{p _ {\pi^ {b}} (R ^ {(j + 1)})} Y \right] \\ = \mathbb {E} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R ^ {(j + 1)})}{p _ {\pi^ {b}} (R ^ {(j + 1)})} \mathbb {E} _ {\pi^ {b}} [ Y | R ^ {(j + 1)} ] \right] \\ = \mathbb {E} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R ^ {(j)})}{p _ {\pi^ {b}} (R ^ {(j)})} \mid R ^ {(j + 1)} \right] \mathbb {E} _ {\pi^ {b}} [ Y | R ^ {(j + 1)} ] \right] \\ = \mathbb {E} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R ^ {(j)})}{p _ {\pi^ {b}} (R ^ {(j)})} Y \Bigg | R ^ {(j + 1)} \right] \right] \\ = \mathbb {E} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} \left(R ^ {(j)}\right)}{p _ {\pi^ {b}} \left(R ^ {(j)}\right)} Y \right] \\ = \mathbb {E} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {G - M I P S}} ^ {(j)} \right] = \mathbb {E} _ {\pi^ {*}} [ Y ]. \\ \end{array}
$$
Next, to prove the variance result we consider the difference
$$
\begin{array}{l} \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathbf {G} - \text {M I P S}} ^ {(j)} \right] - \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathbf {G} - \text {M I P S}} ^ {(j + 1)} \right] \\ = \frac {1}{n} \left(\operatorname {V a r} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R ^ {(j)})}{p _ {\pi^ {b}} (R ^ {(j)})} Y \right] - \operatorname {V a r} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R ^ {(j + 1)})}{p _ {\pi^ {b}} (R ^ {(j + 1)})} Y \right]\right) \\ = \frac {1}{n} \left(\operatorname {V a r} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R ^ {(j)})}{p _ {\pi^ {b}} (R ^ {(j)})} Y \mid R ^ {(j + 1)} \right] \right] + \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R ^ {(j)})}{p _ {\pi^ {b}} (R ^ {(j)})} Y \mid R ^ {(j + 1)} \right] \right] \right. \\ \left. - \operatorname {V a r} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R ^ {(j + 1)})}{p _ {\pi^ {b}} (R ^ {(j + 1)})} \mathbb {E} _ {\pi^ {b}} [ Y \mid R ^ {(j + 1)} ] \right] - \mathbb {E} _ {\pi^ {b}} \left[ \left(\frac {p _ {\pi^ {*}} (R ^ {(j + 1)})}{p _ {\pi^ {b}} (R ^ {(j + 1)})}\right) ^ {2} \operatorname {V a r} _ {\pi^ {b}} [ Y \mid R ^ {(j + 1)} ] \right]\right) \\ \end{array}
$$
where in the last step we use the law of total variance. Now, using the fact that  $R^{(j)} \perp Y \mid R^{(j + 1)}$ , we can rewrite the expression above as
$$
\begin{array}{l} = \frac {1}{n} \left(\operatorname {V a r} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R ^ {(j)})}{p _ {\pi^ {b}} (R ^ {(j)})} \right\rvert   R ^ {(j + 1)} \right] \mathbb {E} _ {\pi^ {b}} [ Y | R ^ {(j + 1)} ] \right] + \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R ^ {(j)})}{p _ {\pi^ {b}} (R ^ {(j)})} Y \right\rvert   R ^ {(j + 1)} \right] \Bigg ] \\ \left. - \operatorname {V a r} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R ^ {(j + 1)})}{p _ {\pi^ {b}} (R ^ {(j + 1)})} \mathbb {E} _ {\pi^ {b}} [ Y \mid R ^ {(j + 1)} ] \right] - \mathbb {E} _ {\pi^ {b}} \left[ \left(\frac {p _ {\pi^ {*}} (R ^ {(j + 1)})}{p _ {\pi^ {b}} (R ^ {(j + 1)})}\right) ^ {2} \operatorname {V a r} _ {\pi^ {b}} [ Y \mid R ^ {(j + 1)} ] \right]\right) \\ = \frac {1}{n} \left(\mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R ^ {(j)})}{p _ {\pi^ {b}} (R ^ {(j)})} Y \mid R ^ {(j + 1)} \right] \right] - \mathbb {E} _ {\pi^ {b}} \left[ \left(\frac {p _ {\pi^ {*}} (R ^ {(j + 1)})}{p _ {\pi^ {b}} (R ^ {(j + 1)})}\right) ^ {2} \operatorname {V a r} _ {\pi^ {b}} [ Y \mid R ^ {(j + 1)} ] \right]\right). \\ \end{array}
$$
Moreover, again using the conditional independence  $R^{(j)} \perp Y \mid R^{(j + 1)}$ , we can expand the first term in the expression above as follows:
$$
\begin{array}{l} \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R ^ {(j)})}{p _ {\pi^ {b}} (R ^ {(j)})} Y \mid R ^ {(j + 1)} \right] \right] = \mathbb {E} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} ^ {2} (R ^ {(j)})}{p _ {\pi^ {b}} ^ {2} (R ^ {(j)})} \mid R ^ {(j + 1)} \right] \mathbb {E} _ {\pi^ {b}} [ Y ^ {2} | R ^ {(j + 1)} ] \right. \\ \left. - \left(\mathbb {E} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R ^ {(j)})}{p _ {\pi^ {b}} (R ^ {(j)})} \mid R ^ {(j + 1)} \right] \mathbb {E} _ {\pi^ {b}} [ Y | R ^ {(j + 1)} ]\right) ^ {2} \right] \\ \geq \mathbb {E} _ {\pi^ {b}} \left[ \left(\mathbb {E} _ {\pi^ {b}} \left[ \frac {p _ {\pi^ {*}} (R ^ {(j)})}{p _ {\pi^ {b}} (R ^ {(j)})} \right\rvert R ^ {(j + 1)} \right]\right) ^ {2} \mathbb {E} _ {\pi^ {b}} [ Y ^ {2} | R ^ {(j + 1)} ] \\ \left. - \left(\frac {p _ {\pi^ {*}} \left(R ^ {(j + 1)}\right)}{p _ {\pi^ {b}} \left(R ^ {(j + 1)}\right)} \mathbb {E} _ {\pi^ {b}} [ Y | R ^ {(j + 1)} ]\right) ^ {2} \right] \\ = \mathbb {E} _ {\pi^ {b}} \left[ \left(\frac {p _ {\pi^ {*}} (R ^ {(j + 1)})}{p _ {\pi^ {b}} (R ^ {(j + 1)})}\right) ^ {2} \operatorname {V a r} _ {\pi^ {b}} [ Y \mid R ^ {(j + 1)} ] \right]. \\ \end{array}
$$
Here, to get the inequality above, we use the fact that  $\mathbb{E}[X^2] \geq (\mathbb{E}[X])^2$ . Putting this together, we get that  $\mathrm{Var}_{\pi^b}[\hat{\theta}_{\mathrm{G-MIPS}}^{(j)}] - \mathrm{Var}_{\pi^b}[\hat{\theta}_{\mathrm{G-MIPS}}^{(j+1)}] \geq 0$ .
Moreover, the result  $\mathrm{Var}_{\pi^b}[\hat{\theta}_{\mathrm{G - MIPS}}^{(2)}]\geq \mathrm{Var}_{\pi^b}[\hat{\theta}_{\mathrm{MR}}]$  follows straightforwardly from above by defining  $R^{(3)}\coloneqq Y$  . Then, the embeddings satisfy the causal structure
$$
R ^ {(0)} \rightarrow R ^ {(1)} \rightarrow R ^ {(2)} \rightarrow R ^ {(3)} \rightarrow Y.
$$
Using the result above, we know that  $\mathrm{Var}_{\pi^b}[\hat{\theta}_{\mathrm{G - MIPS}}^{(2)}]\geq \mathrm{Var}_{\pi^b}[\hat{\theta}_{\mathrm{G - MIPS}}^{(3)}]$ . But now it is straightforward to see that  $\hat{\theta}_{\mathrm{G - MIPS}}^{(3)} = \hat{\theta}_{\mathrm{MR}}$ , and the result follows.
Intuition Here,  $R^{(j + 1)}$  can be thought of as the embedding obtained by 'filtering out' relevant information about  $Y$  from  $R^{(j)}$ . As such, the amount of 'redundant' information regarding the outcome  $Y$  decreases successively along the sequence  $R^{(0)} \coloneqq (X,A)$ ,  $R^{(1)}, R^{(2)}$ . As a result, the G-MIPS estimators which only consider the shift in the marginal distributions of  $R^{(j)}$  due to policy shift become increasingly efficient with decreasing variance as  $j$  increases. Define the representation  $R^{(3)} \coloneqq Y$ , then the corresponding G-MIPS estimator reduces to the MR estimator, i.e.,  $\hat{\theta}_{\mathrm{G - MIPS}}^{(3)} = \hat{\theta}_{\mathrm{MR}}$ . Moreover, this estimator has minimum variance among all the G-MIPS estimators  $\{\hat{\theta}_{\mathrm{G - MIPS}}^{(j)}\}_{0 \leq j \leq k}$ , as the representation  $R^{(3)}$  contains precisely the least amount of information necessary to obtain the outcome  $Y$ . In other words,  $Y$  itself serves as the 'best embedding' of covariate-action pair  $R^{(0)}$  which contains all relevant information regarding  $Y$ . We verify this empirically in Appendix F.2 by reproducing the experimental setup in [14] along with the MR baseline. Additionally, the MR estimator does not rely on assumptions like D.1 for unbiasedness.
In addition to this, solving the regression problem in Eq. (6) will typically be more difficult when  $R$  is higher dimensional (as is likely to be the case for many choices of embeddings  $R$ ), leading to high bias. In contrast, for MR the embedding  $R = Y$  is one dimensional and therefore the regression problem is significantly easier to solve and yields lower bias. Our empirical results in Appendix F confirm this.
D.2 Doubly robust G-MIPS estimators
Consider the setup for the G-MIPS estimator shown in Figure 3. In this case, we can derive a doubly robust extension of the G-MIPS estimator, denoted as GM-DR, which uses an estimate of the conditional mean  $\tilde{\mu}(r) \approx \mathbb{E}[Y \mid R = r]$  as a control variate to decrease the variance of G-MIPS estimator. This can be explicitly written as follows:
$$
\tilde {\theta} _ {\mathrm {D M - D R}} := \frac {1}{n} \sum_ {i = 1} ^ {n} \frac {p _ {\pi^ {*}} \left(r _ {i}\right)}{p _ {\pi^ {b}} \left(r _ {i}\right)} \left(y _ {i} - \tilde {\mu} \left(r _ {i}\right)\right) + \tilde {\eta} \left(\pi^ {*}\right). \tag {8}
$$
where  $\tilde{\eta} (\pi^{*}) = \frac{1}{n}\sum_{i = 1}^{n}\sum_{r^{\prime}\in \mathcal{R}}\tilde{\mu} (r^{\prime})p_{\pi^{*}}(r^{\prime}\mid x_{i})$  is the analogue of the direct method. Here,  $\mathcal{R}$  denotes the space of the possible of the representations  $R^2$ . Moreover, given the density  $p(r\mid x,a)$ , we can compute  $p_{\pi^*}(r\mid x)$  using
$$
p _ {\pi^ {*}} (r \mid x) = \sum_ {a ^ {\prime} \in \mathcal {A}} p (r \mid x, a ^ {\prime}) \pi^ {*} \left(a ^ {\prime} \mid x\right).
$$
It is straightforward to extend ideas from [13] to show that estimator  $\tilde{\theta}_{\mathrm{DM - DR}}$  is doubly robust in that it will yield accurate value estimates if either the importance weights  $\frac{p_{\pi^*}(r)}{p_{\pi^b}(r)}$  or the outcome model  $\tilde{\mu} (r)$  is well estimated.