SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} _ {a \sim \pi_ {b} (s)} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \rho^ {+} (\tilde {a} | s) \bar {W} (\tilde {a} | s, a) \bar {R} (s, \tilde {a}) \right] \\ = \frac {\pi_ {e} (\nearrow)}{\pi_ {b ^ {+}} (\nearrow)} \bar {W} (\nearrow | \nearrow) r _ {0} + \frac {\pi_ {e} (\searrow)}{\pi_ {b ^ {+}} (\searrow)} \bar {W} (\searrow | \nearrow) r _ {1} \\ = \frac {\alpha}{w} \times w \times r _ {0} + \frac {1 - \alpha}{1 - w} \times (1 - w) \times r _ {1} \\ = \alpha r _ {0} + (1 - \alpha) r _ {1} \\ \end{array}
$$
Then, the second variance term becomes
$$
\begin{array}{l} \beta \left(\frac {\alpha}{\beta w _ {0} + (1 - \beta) (1 - w _ {1})} \times w _ {0} \times r _ {0} + \frac {1 - \alpha}{\beta (1 - w _ {0}) + (1 - \beta) w _ {1}} \times (1 - w _ {0}) \times r _ {1} - \left(\alpha r _ {0} + (1 - \alpha) r _ {1}\right)\right) ^ {2} \\ + (1 - \beta) \left(\frac {\alpha}{\beta w _ {0} + (1 - \beta) (1 - w _ {1})} \times (1 - w _ {1}) \times r _ {0} + \frac {1 - \alpha}{\beta (1 - w _ {0}) + (1 - \beta) w _ {1}} \times w _ {1} \times r _ {1} - (\alpha r _ {0} + (1 - \alpha) r _ {1})\right) ^ {2} \\ = \beta \left[ \alpha r _ {0} \left(\frac {w _ {0}}{\beta w _ {0} + (1 - \beta) (1 - w _ {1})} - 1\right) + (1 - \alpha) r _ {1} \left(\frac {(1 - w _ {0})}{\beta (1 - w _ {0}) + (1 - \beta) w _ {1}} - 1\right) \right] ^ {2} \\ + (1 - \beta) \left[ \alpha r _ {0} \left(\frac {1 - w _ {1}}{\beta w _ {0} + (1 - \beta) (1 - w _ {1})} - 1\right) + (1 - \alpha) r _ {1} \left(\frac {w _ {1}}{\beta (1 - w _ {0}) + (1 - \beta) w _ {1}} - 1\right) \right] ^ {2} \\ = \beta \left[ \alpha^ {2} r _ {0} ^ {2} \left(\frac {w _ {0}}{\beta w _ {0} + (1 - \beta) (1 - w _ {1})} - 1\right) ^ {2} + (1 - \alpha) ^ {2} r _ {1} ^ {2} \left(\frac {(1 - w _ {0})}{\beta (1 - w _ {0}) + (1 - \beta) w _ {1}} - 1\right) ^ {2} \right. \\ \left. + 2 \alpha (1 - \alpha) r _ {0} r _ {1} \left(\frac {w _ {0}}{\beta w _ {0} + (1 - \beta) (1 - w _ {1})} - 1\right) \left(\frac {(1 - w _ {0})}{\beta (1 - w _ {0}) + (1 - \beta) w _ {1}} - 1\right) \right] \\ + (1 - \beta) \left[ \alpha^ {2} r _ {0} ^ {2} \left(\frac {1 - w _ {1}}{\beta w _ {0} + (1 - \beta) (1 - w _ {1})} - 1\right) ^ {2} + (1 - \alpha) ^ {2} r _ {1} ^ {2} \left(\frac {w _ {1}}{\beta (1 - w _ {0}) + (1 - \beta) w _ {1}} - 1\right) ^ {2} \right. \\ \left. + 2 \alpha (1 - \alpha) r _ {0} r _ {1} \left(\frac {1 - w _ {1}}{\beta w _ {0} + (1 - \beta) (1 - w _ {1})} - 1\right) \left(\frac {w _ {1}}{\beta (1 - w _ {0}) + (1 - \beta) w _ {1}} - 1\right) \right] \\ = \alpha^ {2} r _ {0} ^ {2} \left[ \beta \left(\frac {w _ {0}}{\beta w _ {0} + (1 - \beta) (1 - w _ {1})} - 1\right) ^ {2} + (1 - \beta) \left(\frac {1 - w _ {1}}{\beta w _ {0} + (1 - \beta) (1 - w _ {1})} - 1\right) ^ {2} \right] \\ + (1 - \alpha) ^ {2} r _ {1} ^ {2} \left[ \beta \left(\frac {1 - w _ {0}}{\beta (1 - w _ {0}) + (1 - \beta) w _ {1}} - 1\right) ^ {2} + (1 - \beta) \left(\frac {w _ {1}}{\beta (1 - w _ {0}) + (1 - \beta) w _ {1}} - 1\right) ^ {2} \right] \\ + 2 \alpha (1 - \alpha) r _ {0} r _ {1} \left[ \left(\frac {w _ {0}}{\beta w _ {0} + (1 - \beta) (1 - w _ {1})} - 1\right) \left(\frac {1 - w _ {0}}{\beta (1 - w _ {0}) + (1 - \beta) w _ {1}} - 1\right) \right. \\ \left. + \left(\frac {1 - w _ {1}}{\beta w _ {0} + (1 - \beta) (1 - w _ {1})} - 1\right) \left(\frac {w _ {1}}{\beta (1 - w _ {0}) + (1 - \beta) w _ {1}} - 1\right) \right] \\ \end{array}
$$
As shown above, the variance expression is a rather complicated function of  $(w_0, w_1)$ . To solve for its minimum, one may take the derivatives with respect to each of  $w_0$  and  $w_1$  and solve for the zeros. The final solution will depend on the problem parameters, including  $r_0, r_1, \sigma_0^2, \sigma_1^2, \alpha, \beta$ . Here, we do not solve for the final solution, and note that this approach may not be applicable to real-world problems due to its explicitly dependence on problem parameters that may be unknown. We encourage future work to explore different methods to find the variance-minimizing weighting schemes.
E Extended Experiments
E.1 Synthetic Domains - Bandits
E.1.1 Two-State Bandits
We expand on the experiments shown in Section 5.1, where we consider a class of bandit problems with two states  $\{s_1,s_2\}$  (drawn with equal probability), two actions  $\mathcal{A} = \{\nearrow ,\searrow \}$  (recall Figure 3), and corresponding reward distributions  $R(s_{i},a)\sim \mathcal{N}(\bar{R}_{(s_{i},a)},\sigma^{2})$  . Without loss of generality, we assume  $\nearrow$  is always taken from  $s_2$  by both  $\pi_b$  and  $\pi_e$  . For  $s_1$  , we consider deterministic policies (in which one action is always taken) as well as stochastic policies (which take the two actions with probabilities that sum to 1); see column/row header in Table 3. Given  $(\pi_b,\pi_e)$  , we draw 1,000 samples following  $\pi_b$  and then evaluate  $\pi_e$  using various estimators, including standard IS, the naive baseline of adding counterfactual annotations as new samples (Section 3.1), and C\*-IS. We assume that counterfactual annotations are only available for  $s_1$  , and all annotations are drawn from the true reward function. We measure the bias, standard deviation (square root of variance), and root mean-squared error (RMSE) of the estimators with respect to  $v(\pi_e)$  . In Table 3 we consider three settings of the rewards: (i)  $\bar{R}_{(s_1,\nearrow)} = 1,\bar{R}_{(s_1,\searrow)} = 2$  , i.e., both actions lead to a positive reward. (ii)  $\bar{R}_{(s_1,\nearrow)} = -1,\bar{R}_{(s_1,\searrow)} = 1$  , i.e., one action leads to a positive reward and the other leads to a negative reward, (iii)  $\bar{R}_{(s_1,\nearrow)} = -1,\bar{R}_{(s_1,\searrow)} = -2$  , i.e., both actions lead to a negative reward. In all cases, the reward for state  $s_2$  is set to 0 for both actions.
Naive baseline fails due to bias. The naive baseline often has a nonzero bias and worse RMSE than IS regardless of whether the offline data has support. This is consistent with our analyses and example provided in Appendix B.
Bias reduction in support-deficient settings. In the first two rows of each sub-table in Table 3,  $\pi_b$  is deterministic and the untaken action has poor support in the offline data. IS is often biased for these cases; note that in cases where  $\pi_e$  assigns a small probability to the unsupported action (e.g., row 2, column 4,  $\pi_e$  takes  $\nearrow$  with probability 0.1), the bias is small and shows up as 0 after rounding. In particular, when rewards as all positive (the first table of Table 3), the bias is negative, and when rewards are all negative (the third table of Table 3), the bias is positive. In the second table of Table 3, the direction of bias depends on the reward of the unsupported action. In the last three rows, IS is unbiased when the offline data has full support. In contrast, C*-IS is unbiased in all cases (but for rounding errors), and can reduce bias compared to IS in support-deficient settings by making use of counterfactual annotations. RMSE of C*-IS is often (but not always) reduced compared to IS even though variance can sometimes increase. This is consistent with our analyses in Appendix C.2.
Variance reduction in well-supported settings. In the last three rows of each sub-table in Table 3, the offline data has full support because  $\pi_b$  is stochastic, and IS is unbiased in these cases. C*-IS is also unbiased in these cases and achieves lower variance, leading to lower RMSE than IS. This is consistent with our analyses in Appendix C.3.
E.1.2 One-State Bandits
To simplify further experiments, we modify the two-state bandits above so that  $s_2$  is drawn with probability 0 — equivalently, we have a set of one-state bandits (with only  $s_1$ ) with two actions  $\mathcal{A} = \{0,1\}$  where 0 corresponds to  $\nearrow$  and 1 corresponds to  $\searrow$ , and corresponding reward distributions  $R_0 \sim \mathcal{N}(\bar{R}_0, \sigma_0^2)$  and  $R_1 \sim \mathcal{N}(\bar{R}_1, \sigma_1^2)$ . For  $\pi_b$  and  $\pi_e$ , we consider the same set of policies as before, and the same evaluation setup. We omit the naive approach in this setting because it is equivalent to our proposed approach when there is only one state (the state distribution is not affected by adding counterfactual annotations directly).
Results for this setting are summarized in Table 4 and show similar trends as above. Next, we use the one-state bandit to study the effect of weights  $w$  on OPE performance.
Table 3: Summary of performance on the two-state bandit problem for various  $\pi_b$  (rows) and  $\pi_e$  (columns), where each policy is denoted by its probabilities assigned to the two actions from  $s_1$ . Each cell of the table corresponds to a  $(\pi_b, \pi_e)$  combination, for which we report (bias, std, RMSE) for three estimators: IS in the top row, naive in the middle row, and C*-IS in the bottom row.
$\bar{R}_{(s_1,\nearrow)} = 1,\bar{R}_{(s_1,\searrow)} = 2$
πeπb	[1,0]	[0,1]	[0.5,0.5]	[0.1,0.9]	[0.8,0.2]
[1.0,0.0]	0	0.7	0.7	-1	0.4	1	-0.5	0.5	0.7	-0.9	0.4	1	-0.2	0.6	0.6
0.2	1.1	1.2	0.3	2	2	0.3	0.9	1	0.3	1.8	1.8	0.2	0.9	0.9
0	0.7	0.7	0	1.1	1.1	0	0.9	0.9	0	1.1	1.1	0	0.7	0.7
[0.0,1.0]	-0.5	0.4	0.6	0	1.1	1.1	-0.2	0.6	0.7	0	1	1	-0.4	0.4	0.6
0.2	1.1	1.1	0.3	2	2	0.3	0.9	1	0.3	1.8	1.8	0.2	0.8	0.9
0	0.7	0.7	0	1.1	1.1	0	0.9	0.9	0	1.1	1.1	0	0.7	0.7
[0.5,0.5]	0	1	1	0	1.8	1.8	0	1	1	0	1.6	1.6	0	0.8	0.8
0.2	1.1	1.1	0.3	2	2	0.3	0.9	1	0.3	1.8	1.8	0.2	0.8	0.9
0	0.7	0.7	0	1.1	1.1	0	0.9	0.9	0	1.1	1. 1	0	0.7	0.7
[0.1,0.9]	0.1	2.6	2.6	0	1.2	1.2	0	1.3	1.3	0	1.1	1.1	0.1	2	2
0.2	1.1	1.1	0.3	2	2	0.3	0.9	1	0.3	1.8	1.8	0.2	0.8	0.9
0	0.7	0.7	0	1.1	1.1	0	0.9	0.9	0	1.1	1. I	0	0.7	0.7
[0.8,0.2]	0	0.8	0.8	0.1	3.2	3.2	0	1.6	1.6	0	2.9	2.9	0	0.8	0.8
0.2	1.1	1.2	0.3	2	2	0.3	0.9	1	0.3	1.7	1.8	0.2	0.8	0.9
0	0.7	0.7	0	1.1	1.1	0	0.9	0.9	0	1.1	1.1	0	0.7	0.7
$\bar{R}_{(s_1,\nearrow)} = -1, \bar{R}_{(s_1,\searrow)} = 1$
πeπb	[1,0]	[0,1]	[0.5,0.5]	[0.1,0.9]	[0.8,0.2]
[1.0,0.0]	0	0.7	0.7	-0.5	0.4	0.6	-0.2	0.5	0.5	-0.4	0.4	0.6	-0.1	0.6	0.6
-0.1	1.1	1.1	0.2	1.1	1.1	0	0.9	0.9	0.2	1.1	1.1	-0.1	1	1
0	0.7	0.7	0	0.7	0.7	0	0.4	0.4	0	0.6	0.6	0	0.6	0.6
[0.0,1.0]	0.5	0.4	0.6	0	0.7	0.7	0.3	0.5	0.5	0.1	0.6	0.7	0.4	0.4	0.6
-0.1	1.1	1.1	0.2	1.1	1.2	0	0.9	0.9	0.2	1.1	1.1	-0.1	1	1
0	0.7	0.7	0	0.7	0.7	0	0.4	0.4	0	0.6	0.6	0	0.6	0.6
[0.5,0.5]	0	1.1	1.1	0	1.1	1.1	0	0.9	0.9	0	1	1	0	0.9	0.9
-0.1	1.1	1.1	0.2	1.2	1.2	0	1	1	0.2	1.1	1.1	-0.1	1	1
0	0.7	0.7	0	0.7	0.7	0	0.4	0.4	0	0.6	0.6	0	0.6	0.6
[0.1,0.9]	0	2.4	2.4	0	0.7	0.7	0	1.4	1.4	0	0.8	0.8	0	2	2
-0.1	1.1	1.1	0.2	1.1	1.2	0	0.9	0.9	0.2	1.1	1.1	-0.1	1	1
0	0.7	0.7	0	0.7	0.7	0	0.4	0.4	0	0.6	-0.1	0.6	0.6	0.6
[0.8,0.2]	0	0.8	0.8	0.1	1.8	1.8	0	1.1	1.1	0.1	1.7	1.7	0	0.8	0.8
-0.1	1.1	1.1	0.2	1.1	1.2	0	0.9	0.9	0.1	1.1	1.1	-0.1	1	1
0.1	0.7	0.7	0	0.7	0.7	0	0.4	0.4	0	0.6	0.6	0	0.5	0.5
$\bar{R}_{(s_1,\nearrow)} = -1,\bar{R}_{(s_1,\searrow)} = -2$
πeπb	[1,0]	[0,1]	[0.5,0.5]	[0.1,0.9]	[0.8,0.2]
[1.0,0.0]	0	0.7	0.7	1	0.4	1.1	0.5	0.5	0.7	0.9	0.4	1	0.2	0.6	0.7
-0.1	1.1	1.1	-0.3	2	2	-0.2	1	1	-0.3	1.7	1.8	-0.2	0.9	0.9
0	0.7	0.7	0.1	1.1	1.1	0.1	0.9	0.9	0.1	1.1	1.1	0	0.8	0.8
[0.0,1.0]	0.5	0.4	0.6	0.1	1.1	1.1	0.3	0.7	0.7	0.1	1	1	0.4	0.4	0.6
-0.1	1.1	1.1	-0.3	2	2	-0.2	1	1	-0.3	1.7	1.8	-0.2	0.9	0.9
0	0.7	0.7	0.1	1.1	1.1	0.1	0.9	0.9	0,1	1.1	1.1	0	0.8	0.8
[0.5,0.5]	0	1.1	1.1	0.1	1.8	1.8	0.1	1	1	0.1	1.6	1.6	0	0.9	0.9
-0.1	1.1	1.1	-0.3	2	2	-0.2	1	1	-0.3	1.7	1.8	-0.2	0.8	0.9
0	0.7	0.7	0.1	1.1	1.1	0.1	0.9	0.9	0.1	1.1	1.1	0	0.8	0.8
[0.1,0.9]	0	2.4	2.4	0.1	1.2	1.2	0	1.3	1.3	0.1	1.1	1.1	0	1.9	1.9
-0.1	1.1	1.1	-0.3	2	2	-0.2	1	1	-0.3	1.7	1.8	-0.2	0.9	0.9
0	0.7	0.7	0.1	1.1	1.1	0.1	0.9	0.9	0..1	1.1	1.1	0	0.8	0.8
[0.8,0.2]	0	0.8	0.8	0.1	3.1	3.1	0	1.5	1.5	0.1	2.8	2.8	0	0.8	0.8
-0.1	1.1	1.1	-0.3	2	2	-0.2	1	1	-0.3	1.8	1.8	-0.2	0.8	0.9
0.1	0.7	0.7	0	1.1	1.1	0.1	0.9	0.9	0	1.1	1.1	0.1	0.8	0.8
Table 4: Summary of performance on the one-state bandit problem for various  $\pi_b$  (rows) and  $\pi_e$  (columns), where each policy is denoted by its probabilities assigned to the two actions from  $s_1$ . Each cell of the table corresponds to a  $(\pi_b, \pi_e)$  combination, for which we report (bias, std, RMSE) for three estimators: IS in the top row, naive in the middle row, and C*-IS in the bottom row.
$\bar{R}_0 = 1,\bar{R}_1 = 2$
πe
πb	[1,0]	[0,1]	[0.5,0.5]	[0.1,0.9]	[0.8,0.2]
[1.0,0.0]	0.03	0.5	0.5	-2	0	2	-0.98	0.25	1.01	-1.8	0.05	1.8	-0.37	0.4	0.55
0.03	0.5	0.5	0.02	0.47	0.47	0.03	0.34	0.35	0.02	0.43	0.43	0.03	0.41	0.41
[0.0,1.0]	-1	0	1	0.03	0.5	0.5	-0.48	0.25	0.54	-0.07	0.45	0.45	-0.79	0.1	0.8
0.02	0.47	0.47	0.03	0.5	0.5	0.03	0.34	0.35	0.03	0.45	0.45	0.02	0.39	0.39
[0.5,0.5]	0.01	1.23	1.23	0.09	2.17	2.17	0.05	0.71	0.71	0.08	1.86	1.86	0.02	0.69	0.69
0.03	0.47	0.47	0.02	0.5	0.5	0.03	0.34	0.35	0.02	0.46	0.46	0.03	0.39	0.39
[0.1,0.9]	0.08	3.46	3.46	0.02	0.88	0.88	0.05	1.45	1.45	0.03	0.59	0.59	0.07	2.64	2.64
0.01	0.47	0.47	0.04	0.5	0.5	0.03	0.34	0.35	0.03	0.45	0.45	0.02	0.39	0.39
[0.8,0.2]	0.03	0.75	0.75	0.05	4.28	4.28	0.04	1.92	1.92	0.05	3.8	3.81	0.03	0.65	0.65
0.04	0.48	0.48	0.01	0.49	0.49	0.03	0.34	0.35	0.02	0.44	0.44	0.03	0.4	0.4
$\bar{R}_0 = -1, \bar{R}_1 = 1$
πe
πb	[1,0]	[0,1]	[0.5,0.5]	[0.1,0.9]	[0.8,0.2]
[1.0,0.0]	0.03	0.5	0.5	-1	0	1	-0.48	0.25	0.54	-0.9	0.05	0.9	-0.17	0.4	0.43
0.03	0.5	0.5	0.02	0.47	0.47	0.03	0.34	0.35	0.02	0.43	0.43	0.03	0.41	0.41
[0.0,1.0]	1	0	1	0.03	0.5	0.5	0.52	0.25	0.57	0.13	0.45	0.47	0.81	0.1	0.81
0.02	0.47	0.47	0.03	0.5	0.5	0.03	0.34	0.35	0.03	0.45	0.45	0.02	0.39	0.39
[0.5,0.5]	0.06	1.17	1.17	0.06	1.28	1.28	0.06	1.12	1.12	0.06	1.23	1.23	0.06	1.13	1.13
0.03	0.47	0.47	0.02	0.5	0.5	0.03	0.34	0.35	0.02	0.46	0.46	0.03	0.39	0.39
[0.1,0.9]	-0.04	3.37	3.37	0.03	0.64	0.64	-0.01	1.86	1.86	0.02	0.8	0.8	-0.03	2.76	2.76
0.01	0.47	0.47	0.04	0.5	0.5	0.03	0.34	0.35	0.03	0.45	0.45	0.02	0.39	0.39
[0.8,0.2]	0.02	0.74	0.74	0.06	2.42	2.42	0.04	1.46	1.46	0.06	2.22	2.23	0.03	0.95	0.96
0.04	0.48	0.48	0.01	0.49	0.49	0.03	0.34	0.35	0.02	0.44	0.44	0.03	0.4	0.4
$\bar{R}_0 = -1,\bar{R}_1 = -2$
πe
πb	[1,0]	[0,1]	[0.5,0.5]	[0.1,0.9]	[0.8,0.2]
[1.0,0.0]	0.03	0.5	0.5	2	0	2	1.02	0.25	1.05	1.8	0.05	1.8	0.43	0.4	0.58
0.03	0.5	0.5	0.02	0.47	0.47	0.03	0.34	0.35	0.02	0.43	0.43	0.03	0.41	0.41
[0.0,1.0]	1	0	1	0.03	0.5	0.5	0.52	0.25	0.57	0.13	0.45	0.47	0.81	0.1	0.81
0.02	0.47	0.47	0.03	0.5	0.5	0.03	0.34	0.35	0.03	0.45	0.45	0.02	0.39	0.39
[0.5,0.5]	0.06	1.17	1.17	-0.01	2.1	2.1	0.02	0.7	0.71	0	1.8	1.8	0.04	0.67	0.67
0.03	0.47	0.47	0.02	0.5	0.5	0.03	0.34	0.35	0.02	0.46	0.46	0.03	0.39	0.39
[0.1,0.9]	-0.04	3.37	3.37	0.05	0.86	0.86	0	1.41	1.41	0.04	0.58	0.58	-0.02	2.57	2.57
0.01	0.47	0.47	0.04	0.5	0.5	0.03	0.34	0.35	0.03	0.45	0.45	0.02	0.39	0.39
[0.8,0.2]	0.02	0.74	0.74	0.09	4.02	4.02	0.06	1.8	1.8	0.08	3.57	3.57	0.04	0.63	0.63
0.04	0.48	0.48	0.01	0.49	0.49	0.03	0.34	0.35	0.02	0.44	0.44	0.03	0.4	0.4
Using equal weights (in C*-IS) is a reasonable heuristic though not always variance-minimizing. In the results above, C*-IS assumed the weights are split equally for factual and counterfactual data. Next, we explore the effect of different weighting schemes on the performance of C-IS, applied to the same class of bandit problems described above. Note that, consistent with Theorem 1, C-IS remains unbiased regardless of weights, except potentially at extreme values of weights (0 or 1) that ignore either the factual data or counterfactual annotations. Across different settings (Figure 7a-d), we found that the ideal weighting scheme is problem-specific, and certain weights may lead to higher variance compared to standard IS (e.g., lower left region of Figure 7b). Encouragingly, these results also demonstrate that C*-IS, though not always variance-minimizing, consistently achieves lower variance than standard IS (when the data has full support and the estimator is unbiased). Additionally, variance in the weight distributions directly contributes to variance in the resulting estimator (Figure 7e), corroborating our analysis in Theorem 13. Overall, our results suggest that C*-IS, which uses constant weights split equally among actions, is a promising heuristic for using C-IS in practice.
(b)
Figure 7: Heatmaps of log std for C-IS using different weights applied to various problems (with different bandit parameters,  $\pi_b$ ,  $\pi_e$ , and annotation quality; exact parameter settings are specified below).  $W(\tilde{a} | s, a)$  is the weight  $w^{\tilde{a}}$  assigned to action  $\tilde{a}$  given the factual sample  $(s, a)$ . Asterisks * indicate the weights under which the variance is minimized for each problem. Gray crosses × indicate where the estimator is biased. Red circles  $\bigcirc$  correspond to C*-IS. Yellow dots  $\square$  in the upper right corners correspond to standard IS (where applicable). (a-b) Constant weights where we sweep the factual weight within the range [0, 1] for both actions. (c-d) Similar to (a), where the annotations have a larger (c) or smaller (d) variance than the reward function. (e) Weights are drawn from uniform distributions centered at 0.5. We sweep the range of the uniform distributions within [0, 1].
Imputing missing annotations can reduce variance. Finally, we explore the impact of missing annotations on the performance of C-IS. As shown in Figure 8a, obtaining more counterfactual annotations generally helps to reduce the variance. However, as noted in our variance analyses, if annotations for the same factual  $(s,a)$  are sometimes missing, we cannot directly apply C*-IS with equal weights; this may lead to increased variance due to variance in the weights. Here, we use a simple strategy to impute the missing annotations with the average of other annotations (for the same counterfactual  $\tilde{a}$  of the same factual  $(s,a)$ ). As shown in Figure 8b, this reduces the variance further especially when not all annotations are available.
(a) Without imputation (b) With imputation
Figure 8: Heatmaps of log std for C-IS where some counterfactual annotations may be missing. We vary the fraction of acquired annotations for the two actions independently within [0, 1]. (a) Equal weights (as in  $\mathbf{C}^*$ -IS) are used when annotations are available; otherwise, the factual weight is set to 1 when annotations are missing. (b) The missing annotations are imputed using other similar annotations. Imputing missing annotations allows for  $\mathbf{C}^*$ -IS to be applied directly and it achieves lower variance. Parameter settings are the same as Figure 7a.
E.2 Healthcare Domain - Sepsis Simulator
E.2.1 Experimental Setup
**Simulator Description.** The patient state is characterized by five variables: a binary indicator for diabetes status, and four ordinal-valued vital signs (heart rate, blood pressure, oxygen concentration, glucose). Following prior work [22], we used a discrete state space with  $|S| = 1440$ . The action space corresponds to the administration of vasopressors and takes on a binary value (on/off), which may increase or decrease the values of certain vital signs (with pre-specified probabilities) at the next time step. The action space in the original simulator formulation involves combinations of 3 treatments: antibiotics, vasopressors, and mechanical ventilation; we focus on only vasopressors for the purpose of illustration. The episode ends when the patient is discharged or died; discharge only occurs when all vitals are normal and all treatments are turned off, whereas death occurs if three or more vitals are abnormal. Rewards are sparse and only assigned at the end of each episode, with  $+1$  for survival and  $-1$  for death. Episodes that reach the maximum length of 20 are truncated with zero terminal reward.
Evaluation Setup. Following prior work [22], we collected 50 offline datasets from the sepsis simulator (using different random seeds) each with 1000 episodes by following an  $\epsilon$ -greedy behavior policy with respect to the optimal policy where  $\epsilon = 0.1$ . For the evaluation policies, we created a set of deterministic policies by perturbing the optimal policy such that each policy takes the non-optimal actions in a randomly selected subset of states. We varied the number of "action-flipped" states in  $\{50, 100, 200, 300, 400\}$  and generated 5 different evaluation policies for each; we also included the optimal policy in the candidate set, resulting in a total of 26 candidate evaluation policies. These represent policies that may be derived from offline data by typical RL approaches that aim to learn deterministic policies, and are of diverse quality where approximately half are superior to the behavior policy while the other half are inferior (Figure 9). As the baseline estimator that only makes use of offline data, we applied standard PDIS which does not rely on counterfactual annotations. To apply C*-PDIS, we assume all counterfactual annotations are collected in our main experiments in which they may be drawn from different annotation functions; we explore the impact of missing annotations in subsequent sensitivity analyses. For C*-PDIS, we considered three different annotation functions: (i)  $G = Q^{\pi_e}$ , the Q-function of the evaluation policy  $\pi_e$ ; (ii)  $G = Q^{\pi_b}$ , the Q-function of the behavior policy, and (iii)  $G = Q^{\pi_b} \mapsto Q^{\pi_e}$ , where we apply the bias correction procedure discussed in Appendix D.1. We also compare to two naive baselines (given perfect annotations): "naive unweighted" simply adds counterfactual annotations as new trajectories and has the same issue discussed in Section 3.1, whereas "naive weighted" reweights the annotations at the trajectory level instead of per-decision. More formally, assuming a binary action space  $\mathcal{A} = \{0, 1\}$  (without loss of generality), given a trajectory of length  $T$  with counterfactual annotations at each step,  $\tau = [s_t, a_t, r_t]_{t=1}^T$ ,  $\pmb{g} = \{g_t^{1 - a_t}\}_{t=1}^T$  where  $1 - a_t$  is the counterfactual action for  $a_t$ , the naive weighted estimator is defined as
$$
\left. \left(1 - \sum_ {t = 1} ^ {T} w _ {t}\right) \rho_ {1: T} \left(\sum_ {t = 1} ^ {T} r _ {t}\right) + \sum_ {t = 1} ^ {T} \left(w _ {t} \rho_ {1: t - 1} \rho_ {t} ^ {1 - a _ {t}} \left(\sum_ {t ^ {\prime} = 1} ^ {t - 1} r _ {t ^ {\prime}} + g _ {t} ^ {1 - a _ {t}}\right)\right) \right.
$$
Intuitively, this first converts each annotation into a sub-trajectory that terminates at the step of annotation with the counterfactual action  $[s_1, a_1, r_1, \dots, s_t, 1 - a_t, g_t]$ , and then performs IS on each sub-trajectory (including the original trajectory), and finally computes a weighted sum of these  $T + 1$  estimates (1 factual estimate,  $T$  counterfactual estimates) using weights  $(1 - \sum_{t=1}^{T} w_t)$ ,  $w_1, \dots, w_T$ . The reason why "naive weighted" does not work is more subtle: while reweighting the (partial) trajectories constructed from the counterfactual annotations correctly maintains the initial state distribution, it does not correctly maintain the intermediate state distributions.
Figure 9: The true value  $v(\pi_e)$  of each of the 26 evaluation policies. The two dashed line are, respectively, the value of the optimal policy and the value of the behavior policy  $v(\pi_b)$  (which is  $\epsilon$ -greedy with respect to the optimal policy where  $\epsilon = 0.1$ ). On the right, we additionally plot error bars representing  $\pm$  standard deviation of the values at initial states for each of the policies. The average std of initial state values over all 26 policies is 0.312.
E.2.2 Results
C*-PDIS outperforms all baselines in all metrics for the ideal setting. As shown in Table 5, when all counterfactuals are available and annotated with the evaluation policy's Q-function  $(G = Q^{\pi_e})$ , C*-PDIS outperforms baseline PDIS (without annotations) in all metrics, suggesting that it provides more accurate OPE estimates. In contrast, the two naive approaches fail to provide accurate estimates and often underperform standard PDIS.