SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proposition 3 (Bias of C-IS due to imperfect annotations). When Assumption 4 holds but Assumption 1 is violated,  $\mathrm{Bias}[\hat{v}^{\mathrm{C - IS}}] = \mathbb{E}_{s\sim d_1}\mathbb{E}_{a\sim \pi_e(s)}\big[\delta_W(s,a)\epsilon_G(s,a)\big]$ , where we measure violation of Assumption 1 as  $\epsilon_G(s,a) = \mathbb{E}_{g\sim G(s,a)}[g] - \bar{R} (s,a)$ , and  $\delta_W(s,a) = \big(1 - \frac{\bar{W}(a|s,a)\pi_b(a|s)}{\pi_{b^+}(a|s)}\big)$ .
Proposition 12 (Bias of C-IS, combined). When both Assumptions 1 and 4 are violated,  $\mathbb{B}\mathrm{is}[\hat{v}^{\mathrm{C - IS}}] = \mathbb{E}_{s\sim d_1}\left[-\sum_{a\in \mathcal{U}(s,\pi_{b^+})}\pi_e(a|s)\bar{R} (s,a)\right] + \mathbb{E}_{s\sim d_1}\left[\sum_{a\in \mathcal{A}\backslash \mathcal{U}(s,\pi_b)}\delta_W(s,a)\pi_e(a|s)\epsilon_G(s,a) + \sum_{a\in \mathcal{U}(s,\pi_b)\backslash \mathcal{U}(s,\pi_{b^+})}\pi_e(a|s)\epsilon_G(s,a)\right]$ , where  $\mathcal{U}(s,\pi_{b^{+}}) = \{a:\pi_{b^{+}}(a|s) = 0\}$  are unsupported actions in the counterfactual-augmented dataset,  $\epsilon_G(s,a) = \mathbb{E}_{g\sim G(s,a)}[g] - \bar{R} (s,a)$ , and  $\delta_W(s,a) = \bigl (1 - \frac{\bar{W}(a|s,a)\pi_b(a|s)}{\pi_{b^+}(a|s)}\bigr)$ .
Remark. There are two main sources of bias for C-IS, resulting from each of the two assumptions being violated. In Proposition 12, missing annotations (i.e., violation of Assumption 4) contribute to the bias through the first term as the rewards of unsupported actions, whereas imperfect annotations (i.e., violation of Assumption 1) contribute to the bias through the second term as the annotation error over supported actions. When both assumptions hold, C-IS only requires a weaker version of the common support assumption to remain unbiased (Theorem 1); consequently, compared to IS, there is a larger space of policies that C-IS can evaluate without bias. Note that the unbiasedness property is not affected by the user-defined weighting scheme (except that the weights must not be to 0 or 1) and directly applies to  $\mathbf{C}^*$ -IS. When only Assumption 1 holds (Proposition 2), the bias is related to the negative rewards over unsupported actions. On the other hand, when only Assumption 4 holds (Proposition 3), the annotation error  $\epsilon_G$  contributes to the resulting bias and this contribution is scaled by a factor of  $\delta_W(s,a)\leq 1$  when the estimator can make use of (unbiased) factual samples of the state-action pair  $(s,a)$  (i.e., when  $\pi_b(a|s) > 0$  and  $\bar{W} (a|s,a) > 0$ ).
We show Proposition 12 first and then discuss Propositions 2 and 3 as two special cases.
Proof of Proposition 12.
$$
\begin{array}{l} \operatorname {B i a s} [ \hat {v} ^ {\mathrm {C - I S}} ] = \mathbb {E} [ \hat {v} ^ {\mathrm {C - I S}} ] - v (\pi_ {e}) \\ = \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {a \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b})} \pi_ {b} (a | s) \left(\mathbb {E} _ {w ^ {a} \sim W (a | s, a)} [ w ^ {a} ] \frac {\pi_ {e} (a | s)}{\pi_ {b ^ {+}} (a | s)} \mathbb {E} _ {r \sim R (s, a)} [ r ] \right. \right. \\ \left. \left. + \sum_ {\tilde {a} \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b ^ {+}}) \backslash \{a \}} \mathbb {E} _ {w ^ {\tilde {a}} \sim W (\tilde {a} | s, a)} \left[ w ^ {\tilde {a}} \right] \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} \mathbb {E} _ {g ^ {\tilde {a}} \sim G (s, \tilde {a})} \left[ g ^ {\tilde {a}} \right]\right) \right] - v \left(\pi_ {e}\right) \\ \end{array}
$$
$$
\begin{array}{l} = \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {a \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b})} \pi_ {b} (a | s) \left(\sum_ {\tilde {a} \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b ^ {+}})} \bar {W} (\tilde {a} | s, a) \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} \left(\bar {R} (s, \tilde {a}) + \epsilon_ {G} (s, \tilde {a})\right) \right. \right. \\ \left. \left. - \bar {W} (a | s, a) \frac {\pi_ {e} (a | s)}{\pi_ {b ^ {+}} (a | s)} \epsilon_ {G} (s, a)\right) \right] - v \left(\pi_ {e}\right) \\ \end{array}
$$
$$
\begin{array}{l} = \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {a \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b})} \pi_ {b} (a | s) \sum_ {\tilde {a} \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b ^ {+}})} \bar {W} (\tilde {a} | s, a) \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} \bar {R} (s, \tilde {a}) \right] - v \left(\pi_ {e}\right) \\ + \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {a \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b})} \pi_ {b} (a | s) \sum_ {\tilde {a} \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b +})} \bar {W} (\tilde {a} | s, a) \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} \epsilon_ {G} (s, \tilde {a}) \right] \\ - \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {a \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b})} \pi_ {b} (a | s) \bar {W} (a | s, a) \frac {\pi_ {e} (a | s)}{\pi_ {b ^ {+}} (a | s)} \epsilon_ {G} (s, a) \right] \\ = \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {\tilde {a} \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b ^ {+}})} \left(\sum_ {a \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b})} \pi_ {b} (a | s) \bar {W} (\tilde {a} | s, a)\right) \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} \bar {R} (s, \tilde {a}) \right] - v \left(\pi_ {e}\right) \\ \end{array}
$$
$$
\begin{array}{l} + \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {\tilde {a} \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b ^ {+}})} \left(\sum_ {a \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b})} \pi_ {b} (a | s) \bar {W} (\tilde {a} | s, a)\right) \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} \epsilon_ {G} (s, \tilde {a}) \right] \\ - \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {a \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b})} \pi_ {b} (a | s) \bar {W} (a | s, a) \frac {\pi_ {e} (a | s)}{\pi_ {b ^ {+}} (a | s)} \epsilon_ {G} (s, a) \right] \\ = \mathbb {E} _ {s \sim d _ {1}} \left[ - \sum_ {a \in \mathcal {U} (s, \pi_ {b +})} \pi_ {e} (a | s) \bar {R} (s, a) \right] + \mathbb {E} _ {s \sim d _ {1}} \left[ \sum_ {a \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b +})} \pi_ {e} (a | s) \epsilon_ {G} (s, a) \right] \\ - \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {a \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b})} \pi_ {e} (a | s) \bar {W} (a | s, a) \frac {\pi_ {b} (a | s)}{\pi_ {b ^ {+}} (a | s)} \epsilon_ {G} (s, a) \right] \\ = \mathbb {E} _ {s \sim d _ {1}} \left[ - \sum_ {a \in \mathcal {U} (s, \pi_ {b +})} \pi_ {e} (a | s) \bar {R} (s, a) \right] \\ + \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {a \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b})} \delta_ {W} (s, a) \pi_ {e} (a | s) \epsilon_ {G} (s, a) + \sum_ {a \in \mathcal {U} (s, \pi_ {b}) \backslash \mathcal {U} (s, \pi_ {b ^ {+}})} \pi_ {e} (a | s) \epsilon_ {G} (s, a) \right] \\ \end{array}
$$
where  $\delta_W(s,a) = \left(1 - \frac{\bar{W}(a|s,a)\pi_b(a|s)}{\pi_{b^+}(a|s)}\right)$ .
Note that  $\mathcal{U}(s, \pi_{b^+}) \subseteq \mathcal{U}(s, \pi_b)$ , because  $\pi_{b^+}(a|s) = 0$  implies  $\pi_b(a|s) = 0$  (factual data does not contain action  $a$ ) and  $\tilde{W}(a|s, \check{a}) = 0, \forall \check{a} \in \mathcal{A}$  (counterfactual annotations for other actions  $\check{a}$  also do not contain action  $a$ ).
Proof for Proposition 2. Given Assumption 1 but not Assumption 4,
$$
\begin{array}{l} \operatorname {B i a s} [ \hat {v} ^ {\mathrm {C - I S}} ] = \mathbb {E} [ \hat {v} ^ {\mathrm {C - I S}} ] - v (\pi_ {e}) \\ = \mathbb {E} _ {s \sim d _ {1}} \left[ \sum_ {a \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b})} \pi_ {b} (a | s) \left(\sum_ {\tilde {a} \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b ^ {+}})} \bar {W} (\tilde {a} | s, a) \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} \bar {R} (s, \tilde {a})\right) \right] - v \left(\pi_ {e}\right) \\ = \mathbb {E} _ {s \sim d _ {1}} \left[ \sum_ {\tilde {a} \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b ^ {+}})} \left(\left(\sum_ {a \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b})} \pi_ {b} (a | s) \bar {W} (\tilde {a} | s, a)\right) \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} \bar {R} (s, \tilde {a})\right) \right] - \mathbb {E} _ {s \sim d _ {1}} \left[ \sum_ {a \in \mathcal {A}} \pi_ {e} (a | s) \bar {R} (s, a) \right] \\ = \mathbb {E} _ {s \sim d _ {1}} \left[ \sum_ {a \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b ^ {+}})} \pi_ {e} (a | s) \bar {R} (s, a) - \sum_ {a \in \mathcal {A}} \pi_ {e} (a | s) \bar {R} (s, a) \right] \\ = \underset {s \sim d _ {1}} {\mathbb {E}} \left[ - \sum_ {a \in \mathcal {U} (s, \pi_ {b ^ {+}})} \pi_ {e} (a | s) \bar {R} (s, a) \right] \\ \end{array}
$$
Proof for Proposition 3. Given Assumptions 3 and 4 but not Assumption 1,
$$
\begin{array}{l} \operatorname {B i a s} [ \hat {v} ^ {\mathrm {C - I S}} ] = \mathbb {E} [ \hat {v} ^ {\mathrm {C - I S}} ] - v (\pi_ {e}) \\ = \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {a \in \mathcal {A}} \pi_ {b} (a | s) \left(\bar {W} (a | s, a) \frac {\pi_ {e} (a | s)}{\pi_ {b ^ {+}} (a | s)} \bar {R} (s, \tilde {a}) + \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} \bar {W} (\tilde {a} | s, a) \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} \left(\bar {R} (s, \tilde {a}) + \epsilon_ {G} (s, \tilde {a})\right)\right) \right] - v (\pi_ {e}) \\ = \mathbb {E} _ {s \sim d _ {1}} \left[ \sum_ {a \in \mathcal {A}} \pi_ {b} (a | s) \left(\sum_ {\tilde {a} \in \mathcal {A}} \bar {W} (\tilde {a} | s, a) \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} \bar {R} (s, \tilde {a}) + \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} \bar {W} (\tilde {a} | s, a) \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} \epsilon_ {G} (s, \tilde {a})\right) \right] - v \left(\pi_ {e}\right) \\ = \mathbb {E} _ {s \sim d _ {1}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \left(\left(\sum_ {a \in \mathcal {A}} \pi_ {b} (a | s) \bar {W} (\tilde {a} | s, a)\right) \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} \bar {R} (s, \tilde {a})\right) \right] - v \left(\pi_ {e}\right) \\ \end{array}
$$
$$
\begin{array}{l} + \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \left(\left(\sum_ {a \in \mathcal {A}} \pi_ {b} (a | s) \bar {W} (\tilde {a} | s, a)\right) \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} \epsilon_ {G} (s, \tilde {a})\right) \right] \\ - \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \left(\sum_ {a \in \mathcal {A}} \pi_ {b} (a | s) \bar {W} (a | s, a) \frac {\pi_ {e} (a | s)}{\pi_ {b ^ {+}} (a | s)} \epsilon_ {G} (s, a)\right) \right] \\ = \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \pi_ {e} (\tilde {a} | s) \epsilon_ {G} (s, \tilde {a}) \right] - \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {a \in \mathcal {A}} \pi_ {e} (a | s) \bar {W} (a | s, a) \frac {\pi_ {b} (a | s)}{\pi_ {b ^ {+}} (a | s)} \epsilon_ {G} (s, a) \right] \\ = \underset {s \sim d _ {1}} {\mathbb {E}} \underset {a \sim \pi_ {e} (s)} {\mathbb {E}} \left[ \left(1 - \frac {\bar {W} (a | s , a) \pi_ {b} (a | s)}{\pi_ {b ^ {+}} (a | s)}\right) \epsilon_ {G} (s, a) \right] \\ \end{array}
$$
While it is generally true that the space of policies that C-IS can evaluate without bias is larger than that of IS, since the lack of support leads to C-IS being biased towards 0 (similar to the case of standard IS), the magnitude of bias of C-IS is not guaranteed to be less than standard IS without additional assumptions about the rewards. However, if all rewards are non-negative, then under mild assumptions, we can prove the following bias reduction result.
Proposition 4 (A sufficient condition for bias reduction). If Assumption 1 holds (but Assumption 4 is violated),  $\bar{R}(s, a) \geq 0$  for all  $s \in S, a \in \mathcal{A}$ , and there exists  $(s, a)$  such that  $\pi_b(a|s) = 0$ ,  $\pi_{b^+}(a|s) > 0$ ,  $\pi_e(a|s) > 0$ ,  $\bar{R}(s, a) > 0$ , then  $|\mathrm{Bias}[\hat{v}^{\mathrm{C - IS}}]| < |\mathrm{Bias}[\hat{v}^{\mathrm{IS}}]|$ .
$$
\begin{array}{l} \text {P r o o f .} | \text {B i a s} [ \hat {v} ^ {\mathrm {I S}} ] | - | \text {B i a s} [ \hat {v} ^ {\mathrm {C - I S}} ] | \\ = \left| \mathbb {E} _ {s \sim d _ {1}} \left[ - \sum_ {a \in \mathcal {U} (s, \pi_ {b})} \pi_ {e} (a | s) \bar {R} (s, a) \right] \right| - \left| \mathbb {E} _ {s \sim d _ {1}} \left[ - \sum_ {a \in \mathcal {U} (s, \pi_ {b +})} \pi_ {e} (a | s) \bar {R} (s, a) \right] \right| \\ = \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {a \in \mathcal {U} (s, \pi_ {b})} \pi_ {e} (a | s) \bar {R} (s, a) \right] - \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {a \in \mathcal {U} (s, \pi_ {b ^ {+}})} \pi_ {e} (a | s) \bar {R} (s, a) \right] \\ = \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {a \in \mathcal {U} (s, \pi_ {b}) \backslash \mathcal {U} (s, \pi_ {b ^ {+}})} \pi_ {e} (a | s) \bar {R} (s, a) \right] > 0 \\ \end{array}
$$
Lastly, we note a useful corollary of Theorem 1.
Corollary 5 (Expectation of augmented importance ratios). Let  $\rho_W^+ = w^a\rho^a +\sum_{\tilde{a}\in \mathcal{A}\backslash \{a\}}w^{\tilde{a}}\rho^{\tilde{a}}$  given  $\tau$  and  $w$ . Under Assumption 4,  $\mathbb{E}[\rho_W^+ ] = 1$ .
Proof of Corollary 5. Starting with Theorem 1 and substituting  $R(s, a) = G(s, a) = 1$  as the constant reward and annotation for all  $s \in S, a \in \mathcal{A}$ , we have:  $\mathbb{E}_{\tau^{+}}[w^{a}\rho^{a} + \sum_{\tilde{a} \in \mathcal{A} \setminus \{a\}} w^{\tilde{a}}\rho^{\tilde{a}}] = \mathbb{E}_{s \sim d_{1}}\mathbb{E}_{\tilde{a} \sim \pi_{e}(s)}[1] = 1$ .
C.3 C-IS: Variance Analyses
For variance analyses, we focus on the scenario where Assumptions 1 and 4 hold and bias is zero. Below, we state the variance decomposition results under a few assumptions about the weighting scheme and the annotation variance.
Theorem 13 (Variance of C-IS). Let  $\rho^{+}(a|s) = \rho^{a} = \frac{\pi_{e}(a|s)}{\pi_{b^{+}}(a|s)}$  be the importance ratio defined using the augmented behavior policy, and assume variance of the annotation function and the variance of the reward function are related by  $\sigma_R(s,a)^2 = \sigma_R(s,a)^2 +\Delta_\sigma (s,a)$  where  $\Delta_{\sigma}(s,a)\in \mathbb{R}$ , then under Assumptions 1 and 4,
$$
\begin{array}{l} \mathbb {V} [ \hat {v} ^ {\mathrm {C - I S}} ] = \mathbb {V} _ {s \sim d _ {1}} [ V ^ {\pi_ {e}} (s) ] + \mathbb {E} _ {s \sim d _ {1}} \left[ \mathbb {V} _ {a \sim \pi_ {b} (s)} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \rho^ {+} (\tilde {a} | s) \bar {W} (\tilde {a} | s, a) \bar {R} (s, \tilde {a}) \right] \right] \\ + \mathbb {E} _ {s \sim d _ {1}} \mathbb {E} _ {a \sim \pi_ {b} (s)} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \rho^ {+} (\tilde {a} | s) ^ {2} \bar {W} (\tilde {a} | s, a) ^ {2} \sigma_ {R} (s, \tilde {a}) ^ {2} \right] \\ + \mathbb {E} _ {s \sim d _ {1}} \mathbb {E} _ {a \sim \pi_ {b} (s)} \left[ \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} \rho^ {+} (\tilde {a} | s) ^ {2} \bar {W} (\tilde {a} | s, a) ^ {2} \Delta_ {\sigma} (s, \tilde {a}) \right] \\ + \mathbb {E} _ {s \sim d _ {1}} \mathbb {E} _ {a \sim \pi_ {b} (s)} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \rho^ {+} (\tilde {a} | s) ^ {2} \bar {R} (s, \tilde {a}) ^ {2} \sigma_ {W} (\tilde {a} | s, a) ^ {2} \right] \tag {3} \\ + \mathbb {E} _ {s \sim d _ {1}} \mathbb {E} _ {a \sim \pi_ {b} (s)} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \rho^ {+} (\tilde {a} | s) ^ {2} \sigma_ {R} (s, \tilde {a}) ^ {2} \sigma_ {W} (\tilde {a} | s, a) ^ {2} \right] \\ + \mathbb {E} _ {s \sim d _ {1}} \mathbb {E} _ {a \sim \pi_ {b} (s)} \Big [ C (s, a) \Big ] \\ + \mathbb {E} _ {s \sim d _ {1}} \mathbb {E} _ {a \sim \pi_ {b} (s)} \Big [ \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} \rho^ {+} (\tilde {a} | s) ^ {2} \Delta_ {\sigma} (s, \tilde {a}) \sigma_ {W} (\tilde {a} | s, a) ^ {2} \Big ] \\ \end{array}
$$
where  $C(s,a) = 2\sum_{a_i,a_j\in \mathcal{A}}^{a_i\neq a_j}\rho^+ (a_i|s)\rho^+ (a_j|s)\bar{R} (s,a_i)\bar{R} (s,a_j)\operatorname {Cov}\bigl {(}W(a_i|s,a),W(a_j|s,a)\bigr).$
Corollary 14 (Variance of  $\mathbf{C}^*$ -IS). Assuming  $\sigma_G(s, a)^2 = \sigma_R(s, a)^2 + \Delta_\sigma(s, a)$  where  $\Delta_\sigma(s, a) \in \mathbb{R}$ , then under Assumptions 1 and 4,
$$
\begin{array}{l} \mathbb {V} \left[ \hat {v} ^ {\mathrm {C} ^ {*} - \mathrm {I S}} \right] = \mathbb {V} _ {s \sim d _ {1}} \left[ V ^ {\pi_ {e}} (s) \right] + \mathbb {E} _ {s \sim d _ {1}} \mathbb {E} _ {a \sim \pi_ {b} (s)} \left[ \pi_ {b} (a | s) \rho (a | s) ^ {2} \sigma_ {R} (s, a) ^ {2} \right] \\ + \mathbb {E} _ {s \sim d _ {1}} \left[ \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} \pi_ {e} (a | s) ^ {2} \Delta_ {\sigma} (s, a) \right] \\ \end{array}
$$
where  $\rho(a|s) = \frac{\pi_e(a|s)}{\pi_b(a|s)}$  is the importance ratio under the original behavior policy.
Remark. The variance decomposition in Theorem 13 contains eight terms. The first three terms correspond to the three terms in the IS variance decomposition in Eqn. (1), the fourth and eighth terms are related to the variance difference between annotations and rewards, whereas the remaining three terms are all related to the variance (and covariance) of the weight distribution  $W(\cdot |s,a)$ . As we will demonstrate empirically (Appendix E.1), depending on the weight distributions, variance of C-IS may actually be larger than that of IS. While it is difficult to guarantee a general variance reduction, we provide intuition for a few special cases. (i) If the weights are constant (i.e.,  $W(\cdot |s,a)$  is the same value for all instantiations of  $(s,a)$  in the data), then the last four terms will all vanish to zero. (ii) When the annotation function and reward function have the same variance,  $\Delta_{\sigma}(s,a) = 0$ , the fourth and eight terms vanish to zero; with this assumption, we can straightforwardly obtain Theorem 6 from Corollary 14. (iii) Suppose for each state-action pair  $(s,a)$ , we set factual weights  $W(a|s,a) = 1$  and counterfactual weights  $W(\tilde{a} |s,a) = 0$ . Then, we effectively ignore all counterfactual annotations, and the variance of C-IS becomes identical to that of IS.
We prove the most general case for Theorem 13 first, and then derive Corollary 14 as a special case.
Proof of Theorem 13. We apply the law of total variance:
$$
\begin{array}{l} \mathbb{V}[\hat{v}^{\text{C - IS}}] = \mathbb{V}_{\substack{s\sim d_{1},a\sim \pi_{b}(s),r\sim R(s,a)\\ \boldsymbol {g}\sim G(s,\cdot),\boldsymbol {w}\sim W(s,a)}}\left[w^{a}\rho^{a}r + \sum_{\tilde{a}\in \mathcal{A}\setminus \{a\}}w^{\tilde{a}}\rho^{\tilde{a}}g^{\tilde{a}}\right] \\ = \underbrace {\mathbb {V} _ {s \sim d _ {1}} \left[ \mathbb {E} _ {\substack {a \sim \pi_ {b} (s) , r \sim R (s , a) \\ \boldsymbol {g} \sim G (s , \cdot) , \boldsymbol {w} \sim W (s , a)}} \left[ w ^ {a} \rho^ {a} r + \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} w ^ {\tilde {a}} \rho^ {\tilde {a}} g ^ {\tilde {a}} \right] \right]} _ {(1)} + \underbrace {\mathbb {E} _ {s \sim d _ {1}} \left[ \mathbb {V} _ {\substack {a \sim \pi_ {b} (s) , r \sim R (s , a) \\ \boldsymbol {g} \sim G (s , \cdot) , \boldsymbol {w} \sim W (s , a)}} \left[ w ^ {a} \rho^ {a} r + \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \} } w ^ {\tilde {a}} \rho^ {\tilde {a}} g ^ {\tilde {a}} \right] \right]} _ {(1 ^ {\prime})} \\ \end{array}
$$
As shown in the bias analyses, we have  $(1) = \mathbb{V}_{s\sim d_1}[V^{\pi_e}(s)]$ . We further apply the law of total variance on  $(1')$ :
$$
\begin{array}{l} (1 ^ {\prime}) = \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \underset {a \sim \pi_ {b} (s)} {\mathbb {V}} \left[ \underset {\boldsymbol {w} \sim W (s, a)} {\mathbb {E}} \left[ w ^ {a} \rho^ {a} r + \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} w ^ {\tilde {a}} \rho^ {\tilde {a}} g ^ {\tilde {a}} \right] \right] \right] \dots (2) \\ + \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \underset {a \sim \pi_ {b} (s)} {\mathbb {E}} \left[ \underset {\boldsymbol {w} \sim W (s, a)} {\mathbb {V}} _ {r \sim R (s, a), \boldsymbol {g} \sim G (s, \cdot)} \left[ w ^ {a} \rho^ {a} r + \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} w ^ {\tilde {a}} \rho^ {\tilde {a}} g ^ {\tilde {a}} \right] \right] \right] \dots (2 ^ {\prime}) \\ \end{array}
$$
Since  $(r, g)$  and  $\boldsymbol{w}$  are conditionally independent given  $(s, a)$ ,
$$
\begin{array}{l} (2) = \mathbb{E}_{s\sim d_{1}}\left[\mathbb{V}_{a\sim \pi_{b}(s)}\left[\rho^{a}\mathbb{E}_{\substack{w^{a}\sim W(a|s,a)\\ r\sim R(s,a)}}[w^{a}r] + \sum_{\tilde{a}\in \mathcal{A}\setminus \{a\}}\rho^{\tilde{a}}\mathbb{E}_{\substack{w^{\tilde{a}}\sim W(\tilde{a} |s,a)\\ g^{\tilde{a}}\sim G(s,\tilde{a})}}[w^{\tilde{a}}g^{\tilde{a}}]\right]\right] \\ = \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \underset {a \sim \pi_ {b} (s)} {\mathbb {V}} \left[ \rho^ {a} \mathbb {E} _ {w ^ {a} \sim W (a | s, a)} [ w ^ {a} ] \mathbb {E} _ {r \sim R (s, a)} [ r ] + \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} \rho^ {\tilde {a}} \mathbb {E} _ {w ^ {\tilde {a}} \sim W (\tilde {a} | s, a)} [ w ^ {\tilde {a}} ] \mathbb {E} _ {g ^ {\tilde {a}} \sim G (s, \tilde {a})} [ g ^ {\tilde {a}} ] \right] \right] \\ = \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \underset {a \sim \pi_ {b} (s)} {\mathbb {V}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \rho^ {+} (a | s) \bar {W} (\tilde {a} | s, a) \bar {R} (s, \tilde {a}) \right] \right] \\ \end{array}
$$
where in the last step, we apply Assumption 1 to combine the expressions involving  $\mathbb{E}_{r\sim R(s,a)}[r]$  and  $\mathbb{E}_{g^{\tilde{a}}\sim G(s,\tilde{a})}[g^{\tilde{a}}]$  (both are equal to  $\bar{R} (s,\tilde{a}))$  into a single summation over  $\tilde{a}\in \mathcal{A}$ .
We further apply the law of total variance on  $(2^{\prime})$ :
$$
\begin{array}{l} (2 ^ {\prime}) = \underset {s \sim d _ {1}} {\mathbb {E}} \underset {a \sim \pi_ {b} (s)} {\mathbb {E}} \left[ \underset {\boldsymbol {g} \sim G (s, \cdot)} {\mathbb {V}} _ {r \sim R (s, a)} \underset {w \sim W (s, a)} {\mathbb {E}} _ {\boldsymbol {w} \sim W (s, a)} \left[ w ^ {a} \rho^ {a} r + \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} w ^ {\tilde {a}} \tilde {\rho} ^ {\tilde {a}} g ^ {\tilde {a}} \right] \right] \dots (3) \\ + \underset {s \sim d _ {1}} {\mathbb {E}} \underset {a \sim \pi_ {b} (s)} {\mathbb {E}} \left[ \underset {\boldsymbol {g} \sim G (s, \cdot)} {\mathbb {E}} _ {r \sim R (s, a)} \mathbb {V} _ {\boldsymbol {w} \sim W (s, a)} \left[ w ^ {a} \rho^ {a} r + \sum_ {\tilde {a} \in \mathcal {A} \setminus \{a \}} w ^ {\tilde {a}} \rho^ {\tilde {a}} g ^ {\tilde {a}} \right] \right] \dots (3 ^ {\prime}) \\ \end{array}
$$
$$
\begin{array}{l} (3) = \underset {s \sim d _ {1}} {\mathbb {E}} \underset {a \sim \pi_ {b} (s)} {\mathbb {E}} \left[ \mathbb {V} _ {r \sim R (s, a)} \left[ \rho^ {a} \mathbb {E} _ {w ^ {a} \sim W (a | s, a)} [ w ^ {a} ] r \right] + \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} \mathbb {V} _ {\boldsymbol {g} \sim G (s, \cdot)} \left[ \rho^ {\tilde {a}} \mathbb {E} _ {w ^ {\tilde {a}} \sim W (\tilde {a} | s, a)} [ w ^ {\tilde {a}} ] g ^ {\tilde {a}} \right] \right] \\ = \underset {s \sim d _ {1}} {\mathbb {E}} \underset {a \sim \pi_ {b} (s)} {\mathbb {E}} \left[ \rho^ {+} (a | s) ^ {2} \bar {W} (a | s, a) ^ {2} \mathbb {V} _ {r \sim R (s, a)} [ r ] + \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} \rho^ {+} (\tilde {a} | s) ^ {2} \bar {W} (\tilde {a} | s, a) ^ {2} \mathbb {V} _ {g ^ {\tilde {a}} \sim G (s, \tilde {a})} [ g ^ {\tilde {a}} ] \right] \\ = \underset {s \sim d _ {1}} {\mathbb {E}} \underset {a \sim \pi_ {b} (s)} {\mathbb {E}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \rho^ {+} (\tilde {a} | s) ^ {2} \bar {W} (\tilde {a} | s, a) ^ {2} \sigma_ {R} (s, \tilde {a}) ^ {2} \right] + \underset {s \sim d _ {1}} {\mathbb {E}} \underset {a \sim \pi_ {b} (s)} {\mathbb {E}} \left[ \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} \rho^ {+} (\tilde {a} | s) ^ {2} \bar {W} (\tilde {a} | s, a) ^ {2} \Delta_ {\sigma} (s, \tilde {a}) \right] \\ \end{array}
$$