SlowGuess/NeurIPS_2023_OCR huggingface dataset:
(i) First, we estimate the policy ratio  $\hat{\rho}(a,x)\approx \frac{\pi^{*}(a|x)}{\pi^{b}(a|x)}$ . This can be achieved by estimating the behaviour policy  $\widehat{\pi}^b$ , and defining  $\hat{\rho}(a,x)\coloneqq \frac{\pi^{*}(a|x)}{\widehat{\pi}^{b}(a|x)}$ . Alternatively,  $\hat{\rho}(a,x)$  can also be estimated directly by using density ratio estimation techniques as in [18].
(ii) Secondly, we estimate the weights  $\hat{w} (y)$  using Eq. (3) with  $\hat{\rho}$  instead of  $\rho$
In practice, one may consider splitting  $\mathcal{D}_{\mathrm{tr}}$  for each estimation step outlined above. Moreover, each approximation step may introduce bias and therefore, the MR estimator may have two sources of bias. While classical OPE methods like IPW and DR also suffer from bias because of  $\hat{\rho}$  estimation, the estimation of  $\hat{w}(y)$  is specific to MR. However, we show below that given any policy ratio estimate  $\hat{\rho}$ , if  $\hat{w}(y)$  approximates  $\mathbb{E}_{\pi^b}[\hat{\rho}(A,X) \mid Y = y]$  'well enough' (i.e., the estimation step (ii) shown above is 'accurate enough'), then MR achieves a lower variance than IPW and incurs little extra bias.
Proposition 3.7. Suppose that the IPW and MR estimators are defined as,
$$
\tilde {\theta} _ {\mathrm {I P W}} := \frac {1}{n} \sum_ {i = 1} ^ {n} \hat {\rho} (a _ {i}, x _ {i}) y _ {i}, \quad \text {a n d} \quad \tilde {\theta} _ {\mathrm {M R}} := \frac {1}{n} \sum_ {i = 1} ^ {n} \hat {w} (y _ {i}) y _ {i},
$$
and define the approximation error as  $\epsilon \coloneqq \hat{w}(Y) - \tilde{w}(Y)$ , where  $\tilde{w}(Y) \coloneqq \mathbb{E}_{\pi^b}[\hat{\rho}(A, X) \mid Y]$ . Then we have that,  $\mathrm{Bias}(\tilde{\theta}_{\mathrm{MR}}) - \mathrm{Bias}(\tilde{\theta}_{\mathrm{IPW}}) = \mathbb{E}_{\pi^b}[\epsilon Y]$ . Moreover,
$$
\operatorname {V a r} _ {\pi^ {b}} \left[ \tilde {\theta} _ {\mathrm {I P W}} \right] - \operatorname {V a r} _ {\pi^ {b}} \left[ \tilde {\theta} _ {\mathrm {M R}} \right] = \frac {1}{n} \left(\underbrace {\mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} [ \hat {\rho} (A , X) Y \mid Y ] \right]} _ {\geq 0} - \operatorname {V a r} _ {\pi^ {b}} [ \epsilon Y ] - 2 \operatorname {C o v} (\tilde {w} (Y) Y, \epsilon Y)\right). \tag {4}
$$
Intuition The  $\epsilon$  term defined in Proposition 3.7 denotes the error of the second approximation step outlined above. As a direct consequence of this result, we show in Appendix C that as the error  $\epsilon$  becomes small (specifically as  $\mathbb{E}_{\pi^b}[\epsilon^2] \to 0$ ), the difference between biases of MR and IPW estimator becomes negligible. Likewise, the terms  $\mathrm{Var}_{\pi^b}[\epsilon Y]$  and  $\mathrm{Cov}(\tilde{w}(Y), \epsilon Y)$  in Eq. (4) will also be small and as a result the variance of MR will be lower than that of IPW (as the first term is positive).
In fact, using recent results regarding the generalisation error of neural networks [19], we show that when using 2-layer wide neural networks to approximate the weights  $\hat{w}(y)$ , the estimation error  $\epsilon$  declines with increasing training data size  $m$ . Specifically, under certain regularity assumptions we obtain  $\mathbb{E}_{\pi^b}[\epsilon^2] = O(m^{-2/3})$ . Using this we show that as the training data size  $m$  increases, the biases of MR and IPW estimators become roughly equal with a high probability, and
$$
\operatorname {V a r} _ {\pi^ {b}} \left[ \tilde {\theta} _ {\mathrm {I P W}} \right] - \operatorname {V a r} _ {\pi^ {b}} \left[ \tilde {\theta} _ {\mathrm {M R}} \right] = \frac {1}{n} \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} [ \hat {\rho} (A, X) Y \mid Y ] \right] + O \left(m ^ {- 1 / 3}\right).
$$
Therefore the variance of MR estimator falls below that of IPW for large enough  $m$ . The empirical results shown in Appendix F.2 are consistent with this result. Due to space constraints, the main technical result has been included in Appendix C.
3.2 Application to causal inference
Beyond contextual bandits, the variance reduction properties of the MR estimator make it highly useful in a wide variety of other applications. Here, we show one such application in the field of causal inference, where MR can be used for the estimation of average treatment effect (ATE) [20] and leads to some desirable properties in comparison to the conventional ATE estimation approaches. Specifically, we illustrate that the MR estimator for ATE utilizes the evaluation data  $\mathcal{D}$  more efficiently and achieves lower variance than state-of-the-art ATE estimators and consequently provides more accurate ATE estimates. To be concrete, the goal in this setting is to estimate ATE, defined as follows:
$$
\operatorname {A T E} := \mathbb {E} [ Y (1) - Y (0) ].
$$
Here  $Y(a)$  corresponds to the outcome under a deterministic policy  $\pi_{a}(a' \mid x) \coloneqq \mathbb{1}(a' = a)$ . Hence any OPE estimator can be used to estimate  $\mathbb{E}[Y(a)]$  (and therefore ATE) by considering target policy  $\pi^{*} = \pi_{a}$ . An important distinction between MR and existing approaches (like IPW or DR) is that, when estimating  $\mathbb{E}[Y(a)]$ , the existing approaches only use datapoints in  $\mathcal{D}$  with  $A = a$ . To see why this is the case, we note that the policy ratios  $\frac{\pi^{*}(A|X)}{\pi^{b}(A|X)} = \frac{\mathbb{1}(A = a)}{\pi^{b}(A|X)}$  are zero when  $A \neq a$ . In contrast, the MR weights  $\frac{p_{\pi^{*}}(Y)}{p_{\pi^{b}}(Y)}$  are not necessarily zero for datapoints with  $A \neq a$ , and therefore the MR estimator uses all evaluation datapoints when estimating  $\mathbb{E}[Y(a)]$ .
As such we show that MR applied to ATE estimation leads to a smaller variance than the existing approaches. Moreover, because MR is able to use all datapoints when estimating  $\mathbb{E}[Y(a)]$ , MR will generally be more accurate than the existing methods especially in the setting where the data is imbalanced, i.e., the number of datapoints with  $A = a$  is small for a specific action  $a$ . In Appendix E, we formalise this variance reduction of the MR ATE estimator compared to IPW and DR estimators, by deriving analogous results to Propositions 3.3 and 3.4. In addition, we also show empirically in Section 5.3 that the MR ATE estimator outperforms the most commonly used ATE estimators.
Off-Policy evaluation is a central problem both in contextual bandits [13, 5, 21, 6, 7, 17, 22, 8, 23] and in RL [24-27]. Existing OPE methodologies can be broadly categorised into Direct Method (DM), Inverse Probability Weighting (IPW), and Doubly Robust (DR). While DM typically has a low variance, it suffers from high bias when the reward model is misspecified [28]. On the other hand, IPW [12] and DR [13, 5, 17] use policy ratios as importance weights when estimating policy value and suffer from high variance as overlap between behaviour and target policies increases or as the action/context space gets larger [29, 14]. To circumvent this problem, techniques like weight clipping or normalisation [4, 30, 31] are often employed, however, these can often increase bias.
In contrast to these approaches, [14] propose MIPS, which considers the marginal shift in the distribution of a lower dimensional embedding of the action space. While this approach reduces the variance associated with IPW, we show in Section 3.1.1 that the MR estimator achieves a lower variance than MIPS while not requiring any additional assumptions (like Assumption 3.5).
In the context of Reinforcement Learning (RL), various marginalisation techniques of importance weights have been used to propose OPE methodologies. [21, 25, 26] use methods which consider the shift in the marginal distribution of the states, and applies importance weighting with respect to this marginal shift rather than the trajectory distribution. Similarly, [32] use marginalisation for OPE in deep RL, where the goal is to consider the shift in marginal distributions of state and action. Although marginalization is a key trick of these estimators, these techniques do not consider the marginal shift in reward as in MR and are aimed at resolving the curse of horizon, a problem specific to RL. Apart from this, [33] propose a general framework of OPE based on conditional expectations of importance ratios for variance reduction. While their proposed framework includes reward conditioned importance ratios, this is not the main focus and there is little theoretical and empirical comparison of their proposed methodology with existing state-of-the-art methods like DR.
(a) Results with varying size of evaluation dataset  $n$ .
(b) Results with varying  $\alpha^{*}$ .
Figure 2: Results for synthetic data experiment. In 2a we have  $\alpha^{*} = 0.8$  and in 2b we have  $n = 800$ .
Finally we note that the idea of approximating the ratio of intractable marginal densities via leveraging the fact that this ratio can be reformulated as the conditional expectation of a ratio of tractable densities is a standard idea in computational statistics [34] and has been exploited more recently to perform likelihood-free inference [35]. In particular, while [34] typically approximates this expectation through Markov chain Monte Carlo, [35] uses regression instead, however without any theory.
In this section, we provide empirical evidence to support our theoretical results by investigating the performance of our MR estimator against the current state-of-the-art OPE methods. The code to reproduce our experiments has been made available at: github.com/faaizT/MR-OPE.
5.1 Experiments on synthetic data
For our synthetic data experiment, we reproduce the experimental setup for the synthetic data experiment in [14] by reusing their code with minor modifications. Specifically,  $\mathcal{X} \subseteq \mathbb{R}^d$ , for various values of  $d$  as described below. Likewise, the action space  $\mathcal{A} = \{0, \dots, n_a - 1\}$ , with  $n_a$  taking a range of different values. Additional details regarding the reward function, behaviour policy  $\pi^b$ , and the estimation of weights  $\hat{w}(y)$  have been included in Appendix F.2 for completeness.
Target policies To investigate the effect of increasing policy shift, we define a class of policies,
$$
\pi^ {\alpha^ {*}} (a | x) = \alpha^ {*} \mathbb {1} (a = \arg \max  _ {a ^ {\prime} \in \mathcal {A}} q (x, a ^ {\prime})) + \frac {1 - \alpha^ {*}}{| \mathcal {A} |} \quad \text {w h e r e} \quad q (x, a) := \mathbb {E} [ Y \mid X = x, A = a ],
$$
where  $\alpha^{*} \in [0,1]$  allows us to control the shift between  $\pi^b$  and  $\pi^{*}$ . In particular, as we show later, the shift between  $\pi^b$  and  $\pi^{*}$  increases as  $\alpha^{*} \to 1$ . Using the ground truth behaviour policy  $\pi^b$ , we generate a dataset which is split into training and evaluation datasets of sizes  $m$  and  $n$  respectively.
Baselines We compare our estimator with DM, IPW, DR and MIPS estimators. Our setup includes action embeddings  $E$  satisfying Assumption 3.5, and so MIPS is unbiased. Additional baselines have been considered in Appendix F.2. For MR, we split the training data to estimate  $\widehat{\pi}^b$  and  $\hat{w}(y)$ , whereas for all other baselines we use the entire training data to estimate  $\widehat{\pi}^b$  for a fair comparison.
Results We compute the target policy value using the  $n$  evaluation datapoints. Here, the MSE of the estimators is computed over 10 different sets of logged data replicated with different seeds. The results presented have context dimension  $d = 1000$ , number of actions  $n_a = 100$  and training data size  $m = 5000$ . More experiments for a variety of parameter values are included in Appendix F.2.
Varying number of evaluation data  $n$  In Figure 2a we plot the results with increasing size of evaluation data  $n$  increases. MR achieves the smallest MSE among all the baselines considered when  $n$  is small, with the MSE of MR being at least an order of magnitude smaller than every baseline for  $n \leq 500$ . This shows that MR is significantly more accurate than the baselines when the size of the evaluation data is small. As  $n \to \infty$ , the difference between the results for MR and MIPS decreases. However, MR attains smaller variance and MSE than MIPS generally, verifying our analysis in Section 3.1.1. Moreover, Figure 2a shows that while the variance of MR is greater than that of DM, it still achieves the lowest MSE overall, owing to the high bias of DM.
Varying  $\alpha^{*}$  As  $\alpha^{*}$  parameter of the target policy increases, so does the shift between the policies  $\pi^b$  and  $\pi^{\alpha^{*}}$  as illustrated by the figure on the right, which plots the KL-divergence  $D_{\mathrm{KL}}(\pi^b ||\pi^{\alpha^*})$  as a function of  $\alpha$ . Figure 2b plots the results for increasing policy shift. Overall, the MSE of MR estimator is lowest among all the baselines. Moreover, while the MSE and variance of all estimators increase with increasing  $\alpha^{*}$  the increase in these quantities is lower for the MR estimator than for the other baselines. Therefore, the relative performance of MR estimator improves with increasing policy shift and MR remains robust to increase in policy shift.
Additional ablation studies In Appendix F.2, we investigate the effect of varying context dimensions  $d$ , number of actions  $n_a$  and number of training data  $m$ . In every case, we observe that the MR estimator has a smaller MSE than all other baselines considered. In particular, MR remains robust to increasing  $n_a$  whereas the MSE and variance of IPW and DR estimators degrade substantially when  $n_a \geq 2000$ . Likewise, MR outperforms the baselines even when the training data size  $m$  is small.
Table 1: Mean squared error of target policy value with standard errors over 10 different seeds for different classification datasets. Here, number of evaluation data  $n = 1000$ , and  $\alpha^{*} = 0.6$ .
Dataset	Digits	Letter	OptDigits	PenDigits	SatImage	Mnist	CIFAR-100
DM	0.1508±0.0015	0.0886±0.0026	0.0485±0.0016	0.0520±0.0016	0.0208±0.0009	0.1109±0.0014	0.0020±0.0001
DR	0.1334±0.0400	35.085±17.768	0.0464±0.0061	0.2343±0.1404	0.0560±0.0395	0.2617±0.0139	3823.9±2023.2
DRos	0.0847±0.0025	0.2363±0.0586	0.0384±0.0025	0.0138±0.0029	0.0078±0.0008	0.2151±0.0061	0.2628±0.1087
IPW	0.1632±0.0462	45.253±22.057	0.0844±0.0056	0.1342±0.0531	0.0900±0.0676	0.3359±0.0118	4116.9±2097.9
SwitchDR	0.0982±0.0032	0.2387±0.0507	0.0557±0.0047	0.0342±0.0090	0.0136±0.0012	0.2750±0.0102	1.1644±0.8227
MR (Ours)	0.0034±0.0001	0.0018±0.0004	0.0006±0.0002	0.0008±0.0002	0.0016±0.0003	0.0121±0.0009	0.0007±0.0002
5.2 Experiments on classification datasets
Following previous works on OPE in contextual bandits [13, 22, 36, 5], we transform classification datasets into contextual bandit feedback data in this experiment. We consider five UCI classification datasets [37] as well as Mnist [38] and CIFAR-100 [39] datasets, each of which comprises  $\{(x_i, a_i^{\mathrm{gt}})\}_i$ , where  $x_i \in \mathcal{X}$  are feature vectors and  $a_i^{\mathrm{gt}} \in \mathcal{A}$  are the ground-truth labels. In the contextual bandits setup, the feature vectors  $x_i$  are considered to be the contexts, whereas the actions correspond to the possible class of labels. For the context vector  $x_i$  and the action  $a_i$ , the reward  $y_i$  is defined as  $y_i := \mathbb{1}(a_i = a_i^{\mathrm{gt}})$ , i.e., the reward is 1 when the action is the same as the ground truth label and 0 otherwise. Here, the baselines considered include the DM, IPW and DR estimators as well as Switch-DR [5] and DR with Optimistic Shrinkage (DRos) [17]. We do not consider a MIPS baseline here as there is no natural embedding  $E$  of  $A$ . Additional details are provided in Appendix F.3.
In Table 1, we present the results with number of evaluation data  $n = 1000$  and number of training data  $m = 500$ . The table shows that across all datasets, MR achieves the lowest MSE among all methods. Moreover, for the Letter and CIFAR-100 datasets the IPW and DR yield large bias and variance arising from poor policy estimates  $\widehat{\pi}^b$ . Despite this, the MR estimator which utilizes the same  $\widehat{\pi}^b$  for the estimation of  $\hat{w}(y)$  leads to much more accurate results. We also verify that MR outperforms the baselines for increasing policy shift and evaluation data  $n$  in Appendix F.3.
5.3 Application to ATE estimation
In this experiment, we investigate the empirical performance of the MR estimator for ATE estimation.
Twins dataset We use the Twins dataset studied in [40], which comprises data from twin births in the USA between 1989-1991. The treatment  $a = 1$  corresponds to being born the heavier twin and the outcome  $Y$  corresponds to the mortality of each of the twins in their first year of life. Specifically,  $Y(1)$  corresponds to the mortality of the heavier twin (and likewise for  $Y(0)$ ). To simulate the observational study, we follow a similar strategy as in [40] to selectively hide one of the two twins as explained in Appendix F.4. We obtain a total of 11,984 datapoints, of which 5000 datapoints are used to train the behaviour policy  $\hat{\pi}^b$  and outcome model  $\hat{q}(x, a)$ .
Table 2: Mean absolute ATE estimation error  ${\epsilon }_{\text{ATE }}$  with standard errors over 10 different seeds,for increasing number of evaluation data  $n$  .
n	50	200	1600	3200
DM	0.092±0.003	0.092±0.003	0.092±0.004	0.092±0.004
DR	0.101±0.024	0.065±0.009	0.071±0.005	0.069±0.004
DRos	0.100±0.017	0.089±0.006	0.093±0.004	0.087±0.004
IPW	0.092±0.024	0.088±0.014	0.067±0.007	0.067±0.007
SwitchDR	0.101±0.024	0.065±0.009	0.071±0.005	0.069±0.004
MR (Ours)	0.062±0.007	0.065±0.007	0.061±0.005	0.061±0.006
Here, we consider the same baselines as the classification data experiments in previous section. For our evaluation, we consider the absolute error in ATE estimation,  $\epsilon_{\mathrm{ATE}}$ , defined as:  $\epsilon_{\mathrm{ATE}} \coloneqq |\hat{\theta}_{\mathrm{ATE}}^{(n)} - \theta_{\mathrm{ATE}}|$ . Here,  $\hat{\theta}_{\mathrm{ATE}}^{(n)}$  denotes the value of the ATE estimated using  $n$  evaluation datapoints. We compute the ATE value using the  $n$  evaluation datapoints, over 10 different sets of observational data (using different seeds). Table 2 shows that MR achieves the lowest estimation error  $\epsilon_{\mathrm{ATE}}$  for all values of  $n$  considered here. While the performance of other baselines improves with increasing  $n$ , MR outperforms them all.
In this paper, we proposed an OPE method for contextual bandits called marginal ratio (MR) estimator, which considers only the shift in the marginal distribution of the outcomes resulting from the policy shift. Our theoretical and empirical analysis showed that MR achieves better variance and MSE compared to the current state-of-the-art methods and is more data efficient overall. Additionally, we demonstrated that MR applied to ATE estimation provides more accurate results than most commonly used methods. Next, we discuss limitations of our methodology and possible avenues for future work.
Limitations The MR estimator requires the additional step of estimating  $\hat{w}(y)$  which may introduce an additional source of bias in the value estimation. However,  $\hat{w}(y)$  can be estimated by solving a simple 1d regression problem, and as we show empirically in Appendix F, MR achieves the smallest bias among all baselines considered in most cases. Most notably, our ablation study in Appendix F.2 shows that even when the training data is reasonably small, MR outperforms the baselines considered.
Future work The MR estimator can also be applied to policy optimisation problems, where the data collected using an 'old' policy is used to learn a new policy. This approach has been used in Proximal Policy Optimisation (PPO) [41] for example, which has gained immense popularity and has been applied to reinforcement learning with human feedback (RLHF) [42]. We believe that the MR estimator applied to these methodologies could lead to improvements in the stability and convergence of these optimisation schemes, given its favourable variance properties.
We would like to thank Jake Fawkes, Siu Lun Chau, Shahine Bouabid and Robert Hu for their useful feedback. We also appreciate the insights and constructive criticisms provided by the anonymous reviewers. MFT acknowledges his PhD funding from Google DeepMind.
[1] Lihong Li, Wei Chu, John Langford, and Robert E. Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web, WWW '10, page 661-670, New York, NY, USA, 2010. Association for Computing Machinery. ISBN 9781605587998. doi: 10.1145/1772690.1772758. URL https://doi.org/10.1145/1772690.1772758.
[2] Hamsa Bastani and Mohsen Bayati. Online decision making with high-dimensional covariates. Operations Research, 68, 11 2019. doi: 10.1287/opre.2019.1902.
Proof of Lemma 3.1. First, we express the weights  $w(y)$  as the conditional expectation as follows:
$$
\begin{array}{l} w (y) = \frac {p _ {\pi^ {*}} (y)}{p _ {\pi^ {b}} (y)} \\ = \int_ {\mathcal {X}, \mathcal {A}} \frac {p _ {\pi^ {*}} (x , a , y)}{p _ {\pi^ {b}} (y)} d a d x \\ = \int_ {\mathcal {X}, \mathcal {A}} \frac {p _ {\pi^ {*}} (x , a , y)}{p _ {\pi^ {b}} (y)} \frac {p _ {\pi^ {b}} (x , a \mid y)}{p _ {\pi^ {b}} (x , a \mid y)} d a d x \\ = \int_ {\mathcal {X}, \mathcal {A}} \frac {p _ {\pi^ {*}} (x , a , y)}{p _ {\pi^ {b}} (x , a , y)} p _ {\pi^ {b}} (x, a \mid y) d a d x \\ = \int_ {\mathcal {X}, \mathcal {A}} \rho (a, x) p _ {\pi^ {b}} (x, a \mid y) d a d x \\ = \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) \mid Y = y ], \\ \end{array}
$$
where  $\rho(a, x) = \frac{p_{\pi^*}(x, a, y)}{p_{\pi^b}(x, a, y)} = \frac{\pi^*(a|x)}{\pi^b(a|x)}$ . Since conditional expectations can be defined as the solution of regression problem, the result follows.
Proof of Proposition 3.2. We have
$$
\begin{array}{l} \mathrm {D} _ {f} \left(p _ {\pi^ {*}} (x, a, y) | | p _ {\pi^ {b}} (x, a, y)\right) = \mathbb {E} _ {\pi^ {b}} \left[ f \left(\frac {p _ {\pi^ {*}} (X , A , Y)}{p _ {\pi^ {b}} (X , A , Y)}\right) \right] \\ = \mathbb {E} _ {\pi^ {b}} \left[ f \left(\frac {\pi^ {*} (A \mid X)}{\pi^ {b} (A \mid X)}\right) \right] \\ = \mathbb {E} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} \left[ f \left(\frac {\pi^ {*} (A \mid X)}{\pi^ {b} (A \mid X)}\right) \mid Y \right] \right] \\ \geq \mathbb {E} _ {\pi^ {b}} \left[ f \left(\mathbb {E} _ {\pi^ {b}} \left[ \frac {\pi^ {*} (A \mid X)}{\pi^ {b} (A \mid X)} \mid Y \right]\right) \right] \quad (\text {J e s s e n ' s i n e q u a l i t y}) \\ = \mathbb {E} _ {\pi^ {b}} \left[ f \left(\frac {p _ {\pi^ {*}} (Y)}{p _ {\pi^ {b}} (Y)}\right) \right] \\ = \mathrm {D} _ {f} \left(p _ {\pi^ {*}} (y) | | p _ {\pi^ {b}} (y)\right). \\ \end{array}
$$
Proof of Proposition 3.3. Since  $\mathbb{E}_{\pi^b}[\hat{\theta}_{\mathrm{IPW}}] = \mathbb{E}_{\pi^b}[\hat{\theta}_{\mathrm{MR}}] = \mathbb{E}_{\pi^*}[Y]$ , we have that,
$$
\begin{array}{l} \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {I P W}} \right] - \operatorname {V a r} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {M R}} \right] = \mathbb {E} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {I P W}} \right] ^ {2} - \mathbb {E} _ {\pi^ {b}} \left[ \hat {\theta} _ {\mathrm {M R}} \right] ^ {2} \\ = \frac {1}{n} \left(\mathbb {E} _ {\pi^ {b}} [ \rho (A, X) ^ {2} Y ^ {2} ] - \mathbb {E} _ {\pi^ {b}} [ w (Y) ^ {2} Y ^ {2} ]\right) \\ = \frac {1}{n} \left(\mathbb {E} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) ^ {2} \mid Y ] Y ^ {2} \right] - \mathbb {E} _ {\pi^ {b}} \left[ w (Y) ^ {2} Y ^ {2} \right]\right) \\ = \frac {1}{n} \left(\mathbb {E} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) ^ {2} \mid Y ] Y ^ {2} \right] - \mathbb {E} _ {\pi^ {b}} \left[ \mathbb {E} _ {\pi^ {b}} [ \rho (A, X) \mid Y ] ^ {2} Y ^ {2} \right]\right) \\ = \frac {1}{n} \mathbb {E} _ {\pi^ {b}} \left[ \operatorname {V a r} _ {\pi^ {b}} [ \rho (A, X) \mid Y ] Y ^ {2} \right]. \\ \end{array}
$$
In the second last step above, we use the fact that  $w(y) = \mathbb{E}_{\pi^b}[\rho (A,X)\mid Y = y]$ .