SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Intuitively, when Assumption 3 does not hold, the C-IS estimator can make use of information from the counterfactual annotations for unsupported actions, thereby reducing bias compared to IS (Section 4.1). For cases when IS is already unbiased, counterfactual annotations play the role of additional data and should help further reduce variance (Section 4.2).
4.1 Bias Analyses for C-IS
To formalize the effect of counterfactual annotations on support, we state the following assumption.
Assumption 4 (Common support with annotations).  $\pi_e(a|s) > 0\to \pi_{b^+}(a|s) > 0,\forall s\in S,a\in \mathcal{A}.$
Assumption 4 is a weaker version of Assumption 3, because  $\pi_{b^{+}}(a|s) > 0$  requires either  $\bar{W}(a|s,a)\pi_b(a|s) > 0$  (same as Assumption 3, assuming  $\bar{W}(a|s,a) \neq 0$ ) or  $\bar{W}(a|s,\check{a})\pi_b(\check{a}|s) > 0$  for at least some  $\check{a} \in \mathcal{A} \setminus \{a\}$ . In other words, information about action  $a$  can be from either a factual sample or counterfactual annotations (recall Definition 1). Next, we state the main results for the bias of C-IS (unless specified otherwise, expectations are taken with respect to  $\mathbb{E}_{\tau^{+},w}$ ). These results hold for any nonzero  $w$  and directly generalize to the special case of C*-IS where the weights are  $1 / |\mathcal{A}|$ .
Theorem 1 (Unbiasedness of C-IS). In the bandit setting, when both Assumptions 1 and 4 hold, the C-IS estimator is unbiased,  $\mathbb{E}[\hat{v}^{\mathrm{C - IS}}] = v(\pi_e)$
Proposition 2 (Bias of C-IS due to support). When Assumption 1 holds but Assumption 4 is violated,  $\mathrm{Bias}[\hat{v}^{\mathrm{C - IS}}] = \mathbb{E}_{s\sim d_1}\left[-\sum_{a\in \mathcal{U}(s,\pi_{b^+})}\pi_e(a|s)\bar{R} (s,a)\right]$  where  $\mathcal{U}(s,\pi_{b^{+}}) = \{a:\pi_{b^{+}}(a|s) = 0\}$  are unsupported actions in the counterfactual-augmented dataset.
Proposition 3 (Bias of C-IS due to imperfect annotations). When Assumption 4 holds but Assumption 1 is violated,  $\mathrm{Bias}[\hat{v}^{\mathrm{C - IS}}] = \mathbb{E}_{s\sim d_1}\mathbb{E}_{a\sim \pi_e(s)}\big[\delta_W(s,a)\epsilon_G(s,a)\big]$ , where we measure violation of Assumption 1 as  $\epsilon_{G}(s,a) = \mathbb{E}_{g\sim G(s,a)}[g] - \bar{R} (s,a)$ , and  $\delta_W(s,a) = \big(1 - \frac{\bar{W}(a|s,a)\pi_b(a|s)}{\pi_{b^+}(a|s)}\big)$ .
Proposition 4 (A sufficient condition for bias reduction). If Assumption 1 holds (but Assumption 4 is violated),  $\overline{R}(s, a) \geq 0$  for all  $s \in S, a \in \mathcal{A}$ , and there exists  $(s, a)$  such that  $\pi_b(a|s) = 0$ ,  $\pi_{b^+}(a|s) > 0$ ,  $\pi_e(a|s) > 0$ ,  $\overline{R}(s, a) > 0$ , then  $|\mathrm{Bias}[\hat{v}^{\mathrm{C - IS}}]| < |\mathrm{Bias}[\hat{v}^{\mathrm{IS}}]|$ .
There are two sources of bias for C-IS: missing annotations contribute to the bias as the rewards of unsupported actions (Proposition 2), whereas imperfect annotations contribute to the bias as the annotation error over supported actions (Proposition 3). If both assumptions are violated, the resulting bias is the combination of the two (see Appendix C). If both assumptions hold, C-IS is unbiased (Theorem 1). Even when not all counterfactual annotations are collected (Assumption 4 is violated), C-IS can evaluate more policies without bias (assuming perfect annotations), because there is a larger space of policies "supported" by the counterfactual-augmented dataset. In particular, if there is at least one counterfactual annotation for an action with no support in the factual data, C-IS has less bias than IS (under mild conditions, Proposition 4). Lastly, we note the a useful corollary of Theorem 1.
Corollary 5 (Expectation of augmented importance ratios). Let  $\rho_W^+ = w^a\rho^a +\sum_{\tilde{a}\in \mathcal{A}\backslash \{a\}}w^{\tilde{a}}\rho^{\tilde{a}}$  given  $\tau$  and  $\pmb{w}$ . Under Assumption 4,  $\mathbb{E}[\rho_W^+ ] = 1$ .
Remark. Corollary 5 suggests that for each sample,  $\rho_W^+$  plays a similar role as the standard importance ratio  $\rho$  in IS, which may be used for calculating the effective sample size (ESS) [31]. Naturally, we can also create a weighted version of our proposed estimators (e.g., C-WIS), with the normalization factor defined using  $\rho_W^+$ .
4.2 Variance Analyses for C-IS
Compared to the bias analyses above, the variance of C-IS has a more involved dependence on weights  $\boldsymbol{w}$  as well as the variance of the annotation function,  $\sigma_G(s,a)^2 = \mathbb{V}_{g\sim G(s,a)}[g]$ . For clarity, we defer the full derivations to Appendix C.3; here, we present results for  $\mathbf{C}^*$ -IS where the weights are all set to  $1 / |\mathcal{A}|$  and the annotation function has the same variance as the reward function.
Theorem 6 (Variance of  $\mathbf{C}^*$ -IS). Assuming  $\sigma_G(s,a)^2 = \sigma_R(s,a)^2$ , under Assumptions 1 and 4,
$$
\mathbb {V} \left[ \hat {v} ^ {\mathrm {C} ^ {*} - \mathrm {I S}} \right] = \mathbb {V} _ {s \sim d _ {1}} \left[ V ^ {\pi_ {e}} (s) \right] + \mathbb {E} _ {s \sim d _ {1}} \mathbb {E} _ {a \sim \pi_ {b} (s)} \left[ \pi_ {b} (a | s) \rho (a | s) ^ {2} \sigma_ {R} (s, a) ^ {2} \right] \tag {2}
$$
where  $\rho (a|s) = \frac{\pi_e(a|s)}{\pi_b(a|s)}$  is the importance ratio under the original behavior policy.
Proposition 7 (Variance Reduction of  $\mathbf{C}^*$ -IS). Under the premise of Theorem 6,  $\mathbb{V}[\hat{v}^{\mathbf{C}^* -\mathrm{IS}}]\leq \mathbb{V}[\hat{v}^{\mathrm{IS}}]$
Comparing Eqn. (2) with the three terms of the variance decomposition of IS in Eqn. (1), we note that the first term  $\mathbb{V}_{s\sim d_1}[V^{\pi_e}(s)]$  is identical, the dropped second term (of Eqn. (1)) is a non-negative variance term, and the third term is scaled by a factor  $\pi_b(a|s)\leq 1$  (for each instantiation of the expression inside the expectation), leading to a guaranteed variance reduction (Proposition 7). We derive the full variance decomposition for C-IS in Theorem 13. Unlike C\*-IS, variance reduction is not guaranteed for C-IS. This is due to additional non-negative terms that depend on the variance/covariance of weights  $\pmb{w}$  and terms that depend on the difference in variance between annotations and rewards  $\sigma_G(s,a)^2 -\sigma_R(s,a)^2$  (could be positive or negative); these terms all vanish to zero in the case of C\*-IS where weights are constant  $(1 / |\mathcal{A}|)$  and  $\sigma_G(s,a)^2 = \sigma_R(s,a)^2$
4.3 Extensions to C-PDIS
We note that the corresponding results in the bandit setting can be derived for the MDP setting using an induction-style proof, with similar interpretations of the factors that contribute to reduced bias and reduced variance when compared to standard PDIS. Below, we briefly demonstrate how the unbiasedness result (Theorem 1) extends to the MDP setting. We further explore the sequential RL setting in the empirical experiments.
Theorem 8 (Unbiasedness of C-PDIS). In the MDP setting, when both Assumptions 2 and 4 hold, the C-PDIS estimator is unbiased,  $\mathbb{E}[\hat{v}^{\mathrm{C - PDIS}}] = v(\pi_e)$
Proof Sketch. Here, we explain the intuition behind the proof for  $\mathbf{C}^*$ -PDIS, which proceeds via a backward induction (Figure 4). In the recursive definition (Definition 4), we aim to show that every  $v_{T - t + 1}$  is an unbiased estimator of the horizon- $t$  value function. At each horizon  $t$ , we can view the problem as a one-step bandit problem (reducing the estimator to  $\mathbf{C}^*$ -IS), where the factual action leads to a factual trajectory and the counterfactual action(s) leads to the counterfactual annotation(s), both of which are used to construct unbiased estimates of horizon- $t$  Q-values (for  $a_t$  and  $\tilde{a}$ , respectively). In the end, the estimates from the two branches are combined according to  $\pi_e$ , resulting in the correct expectation of the horizon- $t$  value of state  $s$ . See full proof for C-PDIS in Appendix C.4.
Figure 4: Illustration of proof idea. (i) The factual estimate  $v_{T - t}$  provides an unbiased estimate of the horizon-  $(t + 1)$  value of state  $s_{t + 1}$ . (ii) When combined with the factual reward  $r_t$ , we obtain an unbiased estimate of the horizon-  $t$  Q-value of  $(s_t, a_t)$ . (iii) By assumption, the counterfactual annotations provide unbiased estimates of the horizon-  $t$  Q-value of  $(s_t, \tilde{a})$ . (iv) The factual and counterfactual estimates are combined using  $\pi_e$ . For clarity, we omit details about state distributions here (see appendix).
4.4 Practical Implications
So far, we have been focusing on the theoretical framework for incorporating counterfactual annotations into OPE. However, the actual implementation of this approach poses several practical challenges. We believe this underscores the fact that this is a rich area for research with many potential directions. In this section, we address several real-world scenarios that do not adhere to the theoretical assumptions – specifically, when annotations are biased, noisy, or missing.
**Correcting annotation bias.** Comparing Assumptions 1 and 2, we note an important distinction between the bandit setting and the sequential RL setting. For bandits, we simply want  $G$  to mimic the reward model  $R$ . In contrast, for the RL setting,  $G_{t}$  should ideally mimic the Q-function of the evaluation policy  $Q_{t:T}^{\pi_e}$ . Such annotations can be difficult if not impossible to obtain in practice (e.g., for healthcare, it would require asking clinicians to reason about a sequence of counterfactual actions and predict the outcome). Thus, we additionally consider a relaxation of Assumption 2 where instead  $\mathbb{E}_{g\sim G_t(s,a)}[g] = Q_{t:T}^{\pi_b}$  reflecting expected return under the behavior policy (which is more likely in practice). This annotation bias  $\epsilon_G = Q_{t:T}^{\pi_b} - Q_{t:T}^{\pi_e}$  results in a biased estimator (Proposition 3). To correct this bias, we suggest first estimating the annotation bias function  $\hat{\epsilon}_G(s,a)$  using an approximate MDP built from offline data, and then mapping each annotation as  $\hat{g}^{\tilde{a}} = g^{\tilde{a}} - \hat{\epsilon}_G(s,\tilde{a})$  (see Appendix D.1 for details). In Section 5.2, we empirically measure the impact of this alternative assumption and the bias-correction procedure (which we denote as  $Q^{\pi_b}\mapsto \hat{Q}^{\pi_e}$ ) on OPE performance.
Reweighting noisy annotations. So long as Assumption 1 or 2 is satisfied, the noise (i.e., variance) of the annotations does not affect unbiasedness; however, as shown in Theorem 13, the variance of our proposed estimators directly depends on how noisy the annotations are. Intuitively, if the annotation variance is smaller than the reward variance, we want the final estimate to "listen" more to the annotations and less to the factual data (and vice versa). We empirically explore the impact of annotation noise in both Section 5 and Appendix E, noting that while using equal weights (as in C*-IS) outperforms standard IS, adjusting weights based on the relative magnitudes of  $\sigma_R^2$  and  $\sigma_G^2$  can further improve OPE performance.
**Imputing missing annotations.** For many real-world domains, it is unlikely that we can obtain an annotation for every counterfactual action at every time step (the total number of annotations needed is  $(|\mathcal{A}| - 1)NT$ ). While desirable to use equal weights as in C $^*$ -IS (and C $^*$ -PDIS) due to its variance reduction guarantee, this is not possible if some annotations are missing (in which case the factual data must have  $w^a = 1$ ), and this can actually lead to higher variance (see Appendix D.2 for an example). To alleviate this variance increase, we suggest estimating an annotation model  $\hat{G}$  from available annotations and using it to impute the missing annotations (see Appendix D.2). Although  $\hat{G}$  may be a biased estimator of  $G$  (due to annotation noise) and introduce additional bias to the final estimate, we empirically observe a favorable bias-variance trade-off (Section 5.2).
In general, one would expect more counterfactual annotations to reduce both bias and variance; at the same time, these annotations may be imperfect (biased or noisy) and directly increase bias or variance. Our analyses in Section 4 and Appendix C show that both bias and variance depend on weights  $w$ , suggesting the weighting scheme as a key mechanism to achieve optimal bias and variance. In Appendix D.3 we explore an analytical approach for solving the variance-minimizing weighting scheme and note the solution is highly non-trivial. We empirically explore the impact of different weights in Appendix E.1 and note that using equal weights (as in C*-IS) is a promising heuristic, since it achieves good performance in most settings. We believe that optimizing the weights can further improve OPE performance and is an interesting direction for future work.
First, through a suite of simple bandit problems, we verify the theoretical properties of C-IS. Then, we apply our approach to a healthcare-inspired RL simulation domain, where we compare the performance of our proposed approach, C-PDIS, to several baselines in terms of their OPE accuracy and ability to rank policies, and explore robustness to bias, noise, and missingness in the annotations.
5.1 Synthetic Domains - Bandits
We consider a class of bandit problems with two states  $\{s_1, s_2\}$  (drawn with equal probability), two actions  $\mathcal{A} = \{\nearrow, \searrow\}$  (recall Figure 3), and corresponding reward distributions  $R(s_i, a) \sim \mathcal{N}(\bar{R}_{(s_i, a)}, \sigma^2)$ . Without loss of generality, we assume  $\nearrow$  is always taken from  $s_2$  by both  $\pi_b$  and  $\pi_e$ . For  $s_1$ , we consider deterministic policies in which one action is always taken, and a stochastic policy that takes the two actions with equal probability (see column/row header in Table 1). Given  $(\pi_b, \pi_e)$ , we draw 1,000 samples following  $\pi_b$  and then evaluate  $\pi_e$  using various estimators, including standard IS, the naive baseline of adding counterfactual annotations as new samples (Section 3.1), and C*-IS. We assume that counterfactual annotations are only available for  $s_1$ , and all annotations are drawn from the true reward function. We measure the bias, standard deviation (std, the square root of variance), and root mean-squared error (RMSE) of the estimators with respect to  $v(\pi_e)$ .
Naive baseline fails due to bias, whereas C*-IS can reduce bias and/or variance compared to IS. In Table 1, we display the results for  $\bar{R}_{(s_1,\nearrow)} = 1$ ,  $\bar{R}_{(s_1,\searrow)} = 2$ ,  $\bar{R}_{(s_2,\cdot)} = 1$ ,  $\sigma = 0.5$  (other settings in Appendix E.1). The naive baseline fails to improve upon IS (and often underperforms IS in terms of RMSE) and can have a nonzero bias even when IS is unbiased (Table 1, row 2, column 2). C*-IS achieves a lower RMSE than IS across all settings considered. The benefits of C*-IS align with our theoretical analyses. (i) Bias Reduction for Support-Deficient Data. When  $\pi_b$  is deterministic (first row), the unselected action has poor support and IS has a nonzero bias whereas C*-IS is unbiased. Though C*-IS sometimes has a larger variance than IS, the bias reduction outweighs the variance increase and leads to overall lower RMSE. (ii) Variance Reduction for Well-Supported Data. In the second row, data generated by  $\pi_b$  has full support. Both IS and C*-IS are unbiased, but C*-IS leverages counterfactual annotations to achieve lower variance and lower RMSE.
Table 1: Summary of performance on the bandit problem for various  $\pi_b$  (rows) and  $\pi_e$  (columns), where each policy is denoted by its probabilities assigned to the two actions from  $s_1$ . Each cell of the table corresponds to a  $(\pi_b, \pi_e)$  combination, for which we report (bias, std, RMSE) for three estimators: IS in the top row, naive in the middle row, and C*-IS in the bottom row. Settings with  $\pi_b(s_1) = [0,1]$  are omitted due to symmetry.
πeπb	[1,0]	[0,1]	[0.5,0.5]
[1,0]	-	-1 0.6 1.2	-0.5 0.5 0.7
0.2 1.8 1.8	0.1 0.7 0.7
0 0.7 0.7	0 0.5 0.5
[0.5,0.5]	0 0.9 0.9	0 1.6 1.6
0 1 1	0.2 1.8 1.8	-
0 0.5 0.5	0 0.7 0.7
We vary the assumptions (e.g., weights in C-IS, noisy/missing annotations) and present further experiments in Appendix E.1. These results suggest that (i) Equal weights (in  $\mathbf{C}^*$ -IS) is a good heuristic though not always "optimal" and (ii) Imputing missing annotations can reduce variance.
5.2 Healthcare Domain - Sepsis Simulator
Next, we apply our approach to evaluate policies in a simulated RL domain modeled after the physiology of sepsis patients [32]. Following prior work [22], we collected 50 offline datasets from the sepsis simulator (using different random seeds) each with 1000 episodes by following an  $\epsilon$ -greedy behavior policy with respect to the optimal policy where  $\epsilon = 0.1$ . We considered a set of deterministic policies (including the optimal policy) as evaluation policies, which have different performance and
Table 2: Comparison of baseline and proposed estimators in terms of OPE performance (RMSE, ESS), ranking performance (Spearman's rank correlation) and binary classification performance (accuracy, FPR, FNR) on the sepsis simulator, reported as mean ± std from 50 repeated runs. Bolded results are the best for each metric, whereas highlighted results outperform all baselines.
Estimator	↓ RMSE	↑ ESS	↑ Spearman	↑%Accuracy	↓ %FPR	↓ %FNR
Baseline	PDIS (w/o annot.)	0.113 ±0.038	76.8 ±44.0	0.596 ±0.110	76.5 ±3.5	33.7 ±8.7	15.9 ±4.6
Naive unweighted (G = Qπc)	0.128 ±0.006	207.2 ±91.5	0.089 ±0.089	50.0 ±6.0	11.6 ±8.3	78.1 ±13.6
Naive weighted (G = Qπc)	0.097 ±0.006	300.8 ±117.6	0.420 ±0.097	64.3 ±4.7	24.0 ±12.7	44.3 ±11.4
Proposed	C*-PDIS (G = Qπc)	0.013 ±0.005	994.0 ±10.1	0.995 ±0.003	95.7 ±3.1	4.5 ±6.9	4.2 ±5.3
C*-PDIS (G = Qπb)	0.070 ±0.003	994.0 ±10.1	0.961 ±0.011	86.8 ±8.2	22.0 ±20.1	8.2 ±11.3
C*-PDIS (G = Qπb →ˆQπc)	0.028 ±0.007	994.0 ±10.1	0.979 ±0.010	90.1 ±5.4	4.2 ±6.6	14.1 ±9.7
Figure 5: (Left) RMSE of C*-PDIS vs. distance to  $\pi_b$  (in terms of KL divergence) for each  $\pi_e$ , plotted with linear trend lines. OPE error increases as  $\pi_e$  becomes more different from behavior. (Center&Right) Performance of our proposed approach under noisy and missing annotations. Trend lines show average of 50 runs  $\pm$  one std. C*-PDIS is generally robust to noise, and imputing the missing annotations can help maintain competitive performance (relative to the ideal setting) even in the presence of high degrees of missingness.
varying degrees of similarity vs behavior. We compared our proposed estimator (with different annotation functions) with a set of baselines, including standard PDIS (without annotations) and two naive baselines (with perfect annotations): "naive unweighted" simply adds counterfactual annotations as new trajectories and has the same issue discussed in Section 3.1, whereas "naive weighted" reweights the annotations at the trajectory level instead of per-decision. See Appendix E.2 for detailed experimental setup. As the main OPE metric, we report RMSE of value estimates vs true values as well as the effective sample size (ESS). Additionally, we report metrics for two downstream uses of OPE for model selection. (i) When used to rank policies. We report the Spearman's rank correlation between  $\hat{v} (\pi_e)$  and  $v(\pi_{e})$  (computed over all  $\pi_e$ 's) [22, 26]. (ii) When used to determine whether  $\pi_{e}$  is better or worse than  $\pi_{b}$ . We formulate a binary classification problem of  $v(\pi_e)\geq v(\pi_b)$  vs  $v(\pi_e) < v(\pi_b)$ , and report the accuracy, false positive rate (FPR) and false negative rate (FNR).
C*-PDIS outperforms all baselines across all metrics in the ideal setting. As shown in Table 2, when all counterfactuals are available and annotated with the evaluation policy's Q-function  $(G = Q^{\pi_e})$ , C*-PDIS outperforms baseline PDIS (without annotations) in all metrics, demonstrating that it provides more accurate OPE estimates. In contrast, the two naive approaches fail to provide accurate estimates and often underperform standard PDIS.
$\mathbf{C}^*$ -PDIS is robust to biased annotations. Under the more realistic scenario where  $G = Q^{\pi_b}$ , i.e., annotations summarize the future returns under  $\pi_{b}$  rather than  $\pi_{e}$ , we observe a degradation in all metrics compared to the ideal case, though  $\mathbf{C}^*$ -PDIS is still superior to PDIS (Table 2). Applying the bias correction procedure ( $G = Q^{\pi_b} \mapsto \hat{Q}^{\pi_e}$ , see Appendix D.1) helps in recovering performance closer to the ideal case, especially when  $\pi_{e}$  is far from  $\pi_{b}$  (Figure 5-left).
Variance reduction of C*-PDIS outweighs the effect of noisy annotations. When annotations are perturbed with increasing amounts of noise (Figure 5-center), performance degradation is minimal even at the highest level of noise tested (with a std of 1, which is large relative to the reward range  $[-1,1]$ , and larger than 0.31 the std of initial state values). Our estimator remains competitive relative to baselines, suggesting that the benefit of variance reduction from additional data (through counterfactual annotations) outweighs the variance increase from annotation noise, even when annotations are much noisier than factual data. See Appendix E.2 for variations of this experiment.
Collecting more annotations and imputing missing annotations improves performance. As the amount of available annotations increases (Figure 5-right), our approach interpolates between baseline PDIS and the ideal case of  $\mathrm{C}^*$ -PDIS with a monotonic improvement in performance. Furthermore, imputing annotations achieves better performance, suggesting it is a promising strategy when not all annotations can be collected in practice. See Appendix E.2 for variations of this experiment where the imputed annotations have varying degrees of bias due to annotation noise.
There is a rich literature on statistical methods for offline policy evaluation (OPE), including direct methods (DM), importance sampling (IS), and doubly-robust (DR) approaches [24, 26]. DM directly uses offline data to learn the value function of the evaluation policy (e.g., using model-based or model-free approaches) [33, 34]. We do not consider DM in our work since it often involves function approximators whose bias and variance may be difficult to analyze [35]. On the other hand, IS uses a weighted average of trajectory returns to correct the distributional mismatch between the evaluation and behaviour policies [28, 27]. Our proposed estimator directly builds on IS and uses counterfactual annotations to reduce its variance (and bias), while carefully addressing the nuances involved with reweighting the counterfactual annotations to maintain the unbiasedness property. Finally, DR approaches combine IS and DM and reduce the variance of IS by using the estimates from DM as a control variate [27, 36–38]. Our approach is a complementary source of variance reduction and may be combined with DR approaches by modifying our current definitions to include a control variate. We note that other approaches exist for managing the variance of IS in long-horizon settings by considering the stationary or marginalized state distributions [39–41]. Incorporating counterfactual annotations into these estimators is an interesting direction of future research.