SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$\mathbf{C}^*$ -PDIS is robust to biased annotations. Under the more realistic scenario where  $G = Q^{\pi_b}$ , i.e., annotations summarize the future returns under  $\pi_{b}$  rather than  $\pi_{e}$ , we observe a degradation in all metrics compared to the ideal case, though  $\mathbf{C}^*$ -PDIS is still superior to PDIS (Table 5). Applying the bias correction procedure  $G = Q^{\pi_b} \mapsto \hat{Q}^{\pi_e}$  (see Appendix D.1) helps recover performance to be closer to the ideal case, and is especially helpful for  $\pi_{e}$  that are far away from  $\pi_{b}$  (Figure 10).
Table 5: Comparison of baseline and proposed estimators in terms of OPE performance (RMSE, ESS), ranking performance (Spearman's rank correlation) and binary classification performance (accuracy, FPR, FNR) on the sepsis simulator, reported as mean ± std from 50 repeated runs. Bolded results are the best for each metric, whereas highlighted results outperform all baselines. The upper table shows the overall results; the lower table shows the breakdown by ordinary IS and weighted IS (OIS vs WIS) applied to each approach.
Estimator	↓ RMSE	↑ ESS	↑ Spearman	↑ %Accuracy	↓ %FPR	↓ %FNR
Baseline	PDIS (w/o annot.)	0.113 ±0.038	76.8 ±44.0	0.596 ±0.110	76.5 ±3.5	33.7 ±8.7	15.9 ±4.6
Naive unweighted (G = Qπc)	0.128 ±0.006	207.2 ±91.5	0.089 ±0.089	50.0 ±6.0	11.6 ±8.3	78.1 ±13.6
Naive weighted (G = Qπc)	0.097 ±0.006	300.8 ±117.6	0.420 ±0.097	64.3 ±4.7	24.0 ±12.7	44.3 ±11.4
Proposed	C*-PDIS (G = Qπc)	0.013 ±0.005	994.0 ±10.1	0.995 ±0.003	95.7 ±3.1	4.5 ±6.9	4.2 ±5.3
C*-PDIS (G = Qπb)	0.070 ±0.003	994.0 ±10.1	0.961 ±0.011	86.8 ±8.2	22.0 ±20.1	8.2 ±11.3
C*-PDIS (G = Qπb → Qπc)	0.028 ±0.007	994.0 ±10.1	0.979 ±0.010	90.1 ±5.4	4.2 ±6.6	14.1 ±9.7
Estimator	↓ RMSE	↑ ESS	↑ Spearman	↑ %Accuracy	↓ %FPR	↓ %FNR
Breakdown OIS vs WIS	PDOIS (w/o annot.)	0.079 ±0.054	76.8 ±44.0	0.868 ±0.087	79.8 ±5.3	6.4 ±7.3	30.3 ±8.8
PDWIS (w/o annot.)	0.136 ±0.033	76.8 ±44.0	0.523 ±0.178	73.2 ±5.1	61.1 ±13.9	1.6 ±3.4
C*-PDOIS (G = Qπc)	0.013 ±0.005	994.0 ±10.1	0.995 ±0.003	95.6 ±3.2	4.5 ±6.9	4.3 ±5.5
C*-PDWIS (G = Qπc)	0.013 ±0.005	994.0 ±10.1	0.995 ±0.003	95.7 ±3.0	4.5 ±6.9	4.1 ±5.1
C*-PDOIS (G = Qπb)	0.070 ±0.003	994.0 ±10.1	0.962 ±0.012	86.7 ±8.3	20.2 ±20.1	8.3 ±11.4
C*-PDWIS (G = Qπb)	0.070 ±0.003	994.0 ±10.1	0.961 ±0.012	86.9 ±8.1	19.8 ±20.1	8.1 ±11.2
C*-PDOIS (G = Qπb → Qπc)	0.028 ±0.007	994.0 ±10.1	0.979 ±0.010	90.1 ±5.4	4.2 ±6.6	14.1 ±9.7
C*-PDWIS (G = Qπb → Qπc)	0.028 ±0.007	994.0 ±10.1	0.979 ±0.010	90.1 ±5.4	4.2 ±6.6	14.1 ±9.7
Figure 10: RMSE of C\*-PDIS vs. distance to  $\pi_b$  (in terms of KL divergence) for each  $\pi_e$ , plotted with linear trend lines. OPE error increases as  $\pi_e$  becomes more different from behavior.
Variance reduction of C*-PDIS outweighs the effect of noisy annotations. To understand the robustness of our estimator to annotation noise, we perturbed the annotations with varying amounts of noise. Specifically, each annotation is added with a noise value drawn from zero-mean Gaussian distributions with a pre-specified standard deviation which we vary. As the level of annotation noise increases (Figure 11-left), performance degradation is minimal even at the highest level of noise tested (with a std of 1, which is large relative to the reward range  $[-1, 1]$ , and most notably larger than 0.31, the std of initial state values of this domain). Our estimator remains competitive relative to the baseline PDIS, suggesting that the benefit of variance reduction from additional data (through counterfactual annotations) outweighs the variance increase from annotation noise, even when annotations are much noisier than factual data. The same trend holds when only  $10\%$  of the counterfactual annotations are collected (Figure 11-right).
Figure 11: Performance of our proposed C*-PDIS estimator is generally robust to noisy annotations. Trend lines show average of 50 runs  $\pm$  one std. When all annotations are available (left), the performance degradation of C*-PDIS is minimal even at the highest level of noise tested. When only  $10\%$  of the counterfactuals are annotated (right), performance degradation is more noticeable and eventually becomes worse than baseline PDIS without annotations.
Collecting more annotations and imputing missing annotations improves performance. As the amount of available annotations increases (Figure 12), our approach interpolates between baseline PDIS and the ideal case of C*-PDIS with an monotonic improvement in performance. Furthermore, imputing annotations (as described in Appendix D.2) achieves better performance, suggesting it is a promising strategy to handle missing annotations when not all annotations can be obtained in practice. The same trend holds under different amounts of annotation noise (Figure 12 left vs right).
Figure 12: Performance of our proposed C*-PDIS estimator is robust to missing annotations, especially when the missing annotations are imputed. Trend lines show average of 50 runs  $\pm$  one std. As the fraction of annotated samples increases, performance interpolates between baseline PDIS and the ideal case where all counterfactual annotations are available. The imputed version outperforms the unimputed version and maintains a competitive performance (relative to the ideal case) even in the presence of high degrees of missingness. The same general trend holds across the two settings with different amounts of annotation noise (left: std of 0.2, right: std of 1.0), though the imputed annotations have a larger bias when annotations are noisier, leading to slightly worse performance even when all annotations are available.