SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where in the last step we substitute  $\sigma_G(s,\tilde{a})^2 = \sigma_R(s,\tilde{a})^2 +\Delta_\sigma (s,\tilde{a}).$
Letting  $g^{a} = r$  for clarity, we have
$$
(3 ^ {\prime}) = \underset {s \sim d _ {1}} {\mathbb {E}} \underset {a \sim \pi_ {b} (s)} {\mathbb {E}} \underset {\boldsymbol {g} \sim G (s, \cdot)} {\mathbb {E}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \mathbb {V} _ {w ^ {\tilde {a}} \sim W (\tilde {a} | s, a)} [ \rho^ {\tilde {a}} g ^ {\tilde {a}} w ^ {\tilde {a}} ] + 2 \sum_ {a _ {i}, a _ {j} \in \mathcal {A}} ^ {a _ {i} \neq a _ {j}} \operatorname {C o v} _ {\boldsymbol {w} \sim W (s, a)} \left(\rho^ {a _ {i}} g ^ {a _ {i}} w ^ {a _ {i}}, \rho^ {a _ {j}} g ^ {a _ {j}} w ^ {a _ {j}}\right) \right]
$$
$$
\begin{array}{l} = \underset {s \sim d _ {1}} {\mathbb {E}} \underset {a \sim \pi_ {b} (s)} {\mathbb {E}} \underset {\boldsymbol {g} \sim G (s, \cdot)} {\mathbb {E}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} (\rho^ {\tilde {a}}) ^ {2} (g ^ {\tilde {a}}) ^ {2} \mathbb {V} _ {w ^ {\tilde {a}} \sim W (\tilde {a} | s, a)} [ w ^ {\tilde {a}} ] \right. \\ \left. + 2 \sum_ {a _ {i}, a _ {j} \in \mathcal {A}} ^ {a _ {i} \neq a _ {j}} \left(\rho^ {a _ {i}}\right) \left(\rho^ {a _ {j}}\right) \left(g ^ {a _ {i}}\right) \left(g ^ {a _ {j}}\right) \times \operatorname {C o v} \left(W (a _ {i} | s, a), W (a _ {j} | s, a)\right) \right] \\ = \underset {s \sim d _ {1}} {\mathbb {E}} \underset {a \sim \pi_ {b} (s)} {\mathbb {E}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \rho^ {+} (\tilde {a} | s) ^ {2} \Big (\bar {R} (s, \tilde {a}) ^ {2} + \sigma_ {R} (s, \tilde {a}) ^ {2} \Big) \sigma_ {W} (\tilde {a} | s, a) ^ {2} + \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} \rho^ {+} (\tilde {a} | s) ^ {2} \Delta_ {\sigma} (s, \tilde {a}) \sigma_ {W} (\tilde {a} | s, a) ^ {2} \right. \\ \left. + 2 \sum_ {a _ {i}, a _ {j} \in \mathcal {A}} ^ {a _ {i} \neq a _ {j}} \rho^ {+} (a _ {i} | s) \rho^ {+} (a _ {j} | s) \bar {R} (s, a _ {i}) \bar {R} (s, a _ {j}) \times \operatorname {C o v} \left(W (a _ {i} | s, a), W (a _ {j} | s, a)\right) \right] \\ = \underbrace {\underset {s \sim d _ {1}} {\mathbb {E}} \underset {a \sim \pi_ {b} (s)} {\mathbb {E}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \rho^ {+} (\tilde {a} | s) ^ {2} \bar {R} (s , \tilde {a}) ^ {2} \sigma_ {W} (\tilde {a} | s , a) ^ {2} \right]} _ {(4)} + \underbrace {\underset {s \sim d _ {1}} {\mathbb {E}} \underset {a \sim \pi_ {b} (s)} {\mathbb {E}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \rho^ {+} (\tilde {a} | s) ^ {2} \sigma_ {R} (s , \tilde {a}) ^ {2} \sigma_ {W} (\tilde {a} | s , a) ^ {2} \right]} _ {(5)} \\ + \underbrace {\mathbb {E} _ {s \sim d _ {1}} \mathbb {E} _ {a \sim \pi_ {b} (s)} \left[ \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} \rho^ {+} (\tilde {a} | s) ^ {2} \Delta_ {\sigma} (s , \tilde {a}) \sigma_ {W} (\tilde {a} | s , a) ^ {2} \right]} _ {(6)} \\ + \underbrace {\underset {s \sim d _ {1} a \sim \pi_ {b} (s)} {\mathbb {E}} \underset {a \in \mathcal {A}} {\mathbb {E}} \left[ 2 \sum_ {a _ {i} , a _ {j} \in \mathcal {A}} ^ {a _ {i} \neq a _ {j}} \rho^ {+} \left(a _ {i} | s\right) \rho^ {+} \left(a _ {j} | s\right) \bar {R} \left(s , a _ {i}\right) \bar {R} \left(s , a _ {j}\right) \times \operatorname {C o v} \left(W \left(a _ {i} | s, a\right) , W \left(a _ {j} | s, a\right)\right) \right]} _ {(7)} \\ \end{array}
$$
Putting together expressions (1) through (7), we have the desired decomposition for  $\mathbb{V}[\hat{v}^{\mathrm{C - IS}}]$ .
Proof of Corollary 14. Here we derive the variance of  $\mathbf{C}^*$ -IS, which is C-IS with  $W(\tilde{a}|s,a) = |\mathcal{A}|^{-1}$ . Since the weights are constant, expressions (4)(5)(6)(7) all vanish to zero because the variance and covariance associated with  $W(\cdot |s,a)$  are both zero. We focus on simplifying the second, third, and fourth terms in the variance decomposition (which correspond to expressions (2) and (3) in the proof above; we denote the two parts of (3) as (3.1) and (3.2)). First, note that
$$
\begin{array}{l} \pi_ {b ^ {+}} (a | s) = \sum_ {\tilde {a} \in \mathcal {A}} \tilde {W} (a | s, \check {a}) \pi_ {b} (\check {a} | s) \\ = \sum_ {\check {a} \in \mathcal {A}} | \mathcal {A} | ^ {- 1} \pi_ {b} (\check {a} | s) \\ = | \mathcal {A} | ^ {- 1} \left(\sum_ {\check {a} \in \mathcal {A}} \pi_ {b} (\check {a} | s)\right) \\ = | \mathcal {A} | ^ {- 1} \\ \end{array}
$$
$$
\begin{array}{l} (2) = \mathbb {E} _ {s \sim d _ {1}} \left[ \mathbb {V} _ {a \sim \pi_ {b} (s)} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \rho^ {+} (\tilde {a} | s) \bar {W} (\tilde {a} | s, a) \bar {R} (s, \tilde {a}) \right] \right] \\ = \mathbb {E} _ {s \sim d _ {1}} \left[ \mathbb {V} _ {a \sim \pi_ {b} (s)} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \frac {\pi_ {e} (\tilde {a} | s)}{1 / | \mathcal {A} |} (1 / | \mathcal {A} |) \bar {R} (s, \tilde {a}) \right] \right] \\ = \mathbb {E} _ {s \sim d _ {1}} \left[ \mathbb {V} _ {a \sim \pi_ {b} (s)} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \pi_ {e} (\tilde {a} | s) \bar {R} (s, \tilde {a}) \right] \right] \\ = \mathbb {E} _ {s \sim d _ {1}} \left[ \mathbb {V} _ {a \sim \pi_ {b} (s)} \left[ V ^ {\pi_ {e}} (s) \right] \right] \\ = 0 \\ \end{array}
$$
$$
\begin{array}{l} (3. 1) = \mathbb {E} _ {s \sim d _ {1}} \mathbb {E} _ {a \sim \pi_ {b} (s)} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \rho^ {+} (\tilde {a} | s) ^ {2} \bar {W} (\tilde {a} | s, a) ^ {2} \sigma_ {R} (s, \tilde {a}) ^ {2} \right] \\ = \mathbb {E} _ {s \sim d _ {1}} \mathbb {E} _ {a \sim \pi_ {b} (s)} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \frac {\pi_ {e} (\tilde {a} | s) ^ {2}}{(1 / | \mathcal {A} |) ^ {2}} (1 / | \mathcal {A} |) ^ {2} \sigma_ {R} (s, \tilde {a}) ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} = \mathbb {E} _ {s \sim d _ {1}} \left[ \sum_ {a \in \mathcal {A}} \pi_ {b} (a | s) \sum_ {\tilde {a} \in \mathcal {A}} \pi_ {e} (\tilde {a} | s) ^ {2} \sigma_ {R} (s, \tilde {a}) ^ {2} \right] \\ = \mathbb {E} _ {s \sim d _ {1}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \left(\sum_ {a \in \mathcal {A}} \pi_ {b} (a | s)\right) \pi_ {e} (\tilde {a} | s) ^ {2} \sigma_ {R} (s, \tilde {a}) ^ {2} \right] \\ = \mathbb {E} _ {s \sim d _ {1}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \pi_ {e} (\tilde {a} | s) ^ {2} \sigma_ {R} (s, \tilde {a}) ^ {2} \right] \\ = \mathbb {E} _ {s \sim d _ {1}} \left[ \sum_ {a \in \mathcal {A}} \pi_ {b} (a | s) ^ {2} \rho (a | s) ^ {2} \sigma_ {R} (s, a) ^ {2} \right] \\ = \mathbb {E} _ {s \sim d _ {1}} \mathbb {E} _ {a \sim \pi_ {b} (s)} \left[ \pi_ {b} (a | s) \rho (a | s) ^ {2} \sigma_ {R} (s, a) ^ {2} \right] \\ \end{array}
$$
$$
\begin{array}{l} (3. 2) = \mathbb {E} _ {s \sim d _ {1}} \mathbb {E} _ {a \sim \pi_ {b} (s)} \Big [ \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} \rho^ {+} (\tilde {a} | s) ^ {2} \bar {W} (\tilde {a} | s, a) ^ {2} \Delta_ {\sigma} (s, \tilde {a}) \Big ] \\ = \mathbb {E} _ {s \sim d _ {1}} \mathbb {E} _ {a \sim \pi_ {b} (s)} \left[ \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} \frac {\pi_ {e} (\tilde {a} | s) ^ {2}}{(1 / | \mathcal {A} |) ^ {2}} (1 / | \mathcal {A} |) ^ {2} \Delta_ {\sigma} (s, \tilde {a}) \right] \\ = \mathbb {E} _ {s \sim d _ {1}} \left[ \sum_ {a \in \mathcal {A}} \pi_ {b} (a | s) \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} \pi_ {e} (\tilde {a} | s) ^ {2} \Delta_ {\sigma} (s, \tilde {a}) \right] \\ = \mathbb {E} _ {s \sim d _ {1}} \left[ \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} \left(\sum_ {a \in \mathcal {A}} \pi_ {b} (a | s)\right) \pi_ {e} (\tilde {a} | s) ^ {2} \Delta_ {\sigma} (s, \tilde {a}) \right] \\ = \mathbb {E} _ {s \sim d _ {1}} \left[ \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} \pi_ {e} (\tilde {a} | s) ^ {2} \Delta_ {\sigma} (s, \tilde {a}) \right] \\ \end{array}
$$
C.4 C-PDIS: Bias Analyses
First, we introduce a few definitions useful for the proof. Let the  $t$ -step state distribution be denoted by  $d_t^\pi(s) = \operatorname*{Pr}(s_t = s \mid s_1 \sim d_1, a_{t'} \sim \pi(s_{t'}))$ . Similarly, the  $t$ -step state-action distribution is  $d_t^\pi(s, a) = \operatorname*{Pr}(s_t = s, a_t = a \mid s_1 \sim d_1, a_{t'} \sim \pi(s_{t'}))$ . Note that  $d_{t+1}^\pi(s') = \mathbb{E}_{s, a \sim d_t^\pi}[p(s'|s, a)]$ . Let  $d_t = d_t^{\pi_b}$  denote the distribution under the behavior policy. Recall the horizon- $t$  value functions:  $V_{t:T}^{\pi_e}(s) = \mathbb{E}_{\pi_e}[\sum_{t'=t}^{T} \gamma^{t'-1} r_{t'}|s_t = s]$ ,  $Q_{t:T}^{\pi_e}(s, a) = \mathbb{E}_{\pi_e}[\sum_{t'=t}^{T} \gamma^{t'-1} r_{t'}|s_t = s, a_t = a]$ . Based on the Bellman equation,  $Q_{t:T}^{\pi_e}(s, a) = r(s, a) + \gamma \mathbb{E}_{s' \sim p(s, a)}[V_{(t+1):T}^{\pi_e}(s')]$ . Note that  $v(\pi_e) = \mathbb{E}_{s \sim d_1}[V_{1:T}^{\pi_e}(s)]$ . Also recall Assumption 2, the perfect annotation function in the MDP setting satisfies  $\mathbb{E}_{g \sim G_t(s, a)}[g] = Q_{t:T}^{\pi_e}(s, a)$  for  $(s, a)$  at step  $t$  of the trajectory.
Recall the recursive definition of C-PDIS (Definition 4):  $\hat{v}^{\mathrm{C - PDIS}} = v_{T}$ , with  $v_{0} = 0$ ,  $v_{T - t + 1} = w_{t}^{a_{t}}\rho_{t}^{a_{t}}(r_{t} + \gamma v_{T - t}) + \sum_{\tilde{a}\in \mathcal{A}\backslash \{a_{t}\}}w_{t}^{\tilde{a}}\rho_{t}^{\tilde{a}}q_{t}^{\tilde{a}}$  for  $t = T\dots 1$ , where  $\rho_t^{\tilde{a}} = \frac{\pi_e(\tilde{a}|s_t)}{\pi_{b^+}(\tilde{a}|s_t)}$ .
Proof. We show this via backward induction on a sequence of horizon- $t$  value functions of  $\pi_e$  denoted by  $V_{t:T}^{\pi_e}(s)$ . The goal is to show that  $\mathbb{E}_{\tau^+, \boldsymbol{w}}[v_{T-t}] = \mathbb{E}_{s \sim d_{t+1}}[V_{(t+1):T}^{\pi_e}(s)]$  for all  $t = T\dots 0$  (for clarity, the subscript of expectation of the estimator may be omitted and assumed to be  $\mathbb{E}_{\tau^+, \boldsymbol{w}}$  with  $\tau$  generated by the behavior policy  $\pi_b$  unless otherwise specified).
Base case. It is trivially true that  $\mathbb{E}[v_0] = \mathbb{E}_{s\sim d_{T + 1}}[V_{(T + 1):T}^{\pi_e}(s)] = 0$  since there are no more steps after  $t = T$  and  $s\sim d_{T + 1}$  can be seen as a dummy absorbing state.
Inductive step. Suppose  $\mathbb{E}[v_{T - t}] = \mathbb{E}_{s\sim d_{t + 1}}[V_{(t + 1):T}^{\pi_e}(s)]$  holds. For factual state-action pairs  $(s,a)\sim d_t$  occurring at step  $t$ , we have
$$
\begin{array}{l} \underset {s, a \sim d _ {t}} {\mathbb {E}} \left[ r _ {t} + \gamma v _ {T - t} \right] = \underset {s, a \sim d _ {t}} {\mathbb {E}} \left[ r _ {t} + \gamma \underset {s ^ {\prime} \sim d _ {t + 1}} {\mathbb {E}} \left[ V _ {(t + 1): T} ^ {\pi_ {e}} (s ^ {\prime}) \right] \right] \\ = \underset {s, a \sim d _ {t}} {\mathbb {E}} \left[ r _ {t} + \gamma \underset {s ^ {\prime} \sim p (\cdot | s, a)} {\mathbb {E}} \left[ V _ {(t + 1): T} ^ {\pi_ {e}} (s ^ {\prime}) \right] \right] \\ = \underset {s, a \sim d _ {t}} {\mathbb {E}} \left[ Q _ {t: T} ^ {\pi_ {e}} (s, a) \right] \\ \end{array}
$$
To show the case for  $t - 1$
$$
\begin{array}{l} \mathbb {E} [ v _ {T - t + 1} ] = \mathbb {E} \left[ w _ {t} ^ {a _ {t}} \rho_ {t} ^ {a _ {t}} (r _ {t} + \gamma v _ {T - t}) + \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a _ {t} \}} w _ {t} ^ {\tilde {a}} \rho_ {t} ^ {\tilde {a}} q _ {t} ^ {\tilde {a}} \right] \\ = \underset {s \sim d _ {t}} {\mathbb {E}} \underset {a \sim \pi_ {b} (s)} {\mathbb {E}} \left[ \mathbb {E} _ {w _ {t} ^ {a} \sim W (a | s, a)} \left[ w _ {t} ^ {a} \right] \frac {\pi_ {e} (a | s)}{\pi_ {b ^ {+}} (a | s)} \mathbb {E} _ {s, a \sim d _ {t}} \left[ r _ {t} + \gamma v _ {T - t} \right] \right. \\ \left. + \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} \mathbb {E} _ {w _ {t} ^ {\tilde {a}} \sim W (\tilde {a} | s, a)} \left[ w _ {t} ^ {\tilde {a}} \right] \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} \mathbb {E} _ {g _ {t} ^ {\tilde {a}} \sim G _ {t} (s, \tilde {a})} \left[ g _ {t} ^ {\tilde {a}} \right] \right] \\ \stackrel {(1)} {=} \underset {s \sim d _ {t}} {\mathbb {E}} \left[ \sum_ {a \in \mathcal {A}} \pi_ {b} (a | s) \left(\sum_ {\tilde {a} \in \mathcal {A}} \bar {W} (\tilde {a} | s, a) \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} Q _ {t: T} ^ {\pi_ {e}} (s, \tilde {a})\right) \right] \\ \stackrel {(2)} {=} \underset {s \sim d _ {t}} {\mathbb {E}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \left(\sum_ {a \in \mathcal {A}} \pi_ {b} (a | s) \bar {W} (\tilde {a} | s, a) \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} Q _ {t: T} ^ {\pi_ {e}} (s, \tilde {a})\right) \right] \\ \stackrel {(3)} {=} \underset {s \sim d _ {t}} {\mathbb {E}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \left(\left(\sum_ {a \in \mathcal {A}} \pi_ {b} (a | s) \bar {W} (\tilde {a} | s, a)\right) \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} Q _ {t: T} ^ {\pi_ {e}} (s, \tilde {a})\right) \right] \\ \stackrel {(4)} {=} \mathbb {E} _ {s \sim d _ {t}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \pi_ {b ^ {+}} (\tilde {a} | s) \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} Q _ {t: T} ^ {\pi_ {e}} (s, \tilde {a}) \right] \\ = \underset {s \sim d _ {t}} {\mathbb {E}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \pi_ {e} (\tilde {a} | s) Q _ {t: T} ^ {\pi_ {e}} (s, \tilde {a}) \right] \\ = \underset {s \sim d _ {t}} {\mathbb {E}} \underset {\tilde {a} \sim \pi_ {e}} {\mathbb {E}} \left[ Q _ {t: T} ^ {\pi_ {e}} (s, \tilde {a}) \right] = \underset {s \sim d _ {t}} {\mathbb {E}} \left[ V _ {t: T} ^ {\pi_ {e}} (s) \right] \\ \end{array}
$$
where in (1) we replace  $\mathbb{E}_{g_t^{\tilde{a}}\sim G_t(s,\tilde{a})}[g_t^{\tilde{a}}]$  with  $Q_{t:T}^{\pi_e}(s,\tilde{a})$  following Assumption 2 and combine it with  $\mathbb{E}_{s,a\sim d_t}[r_t + \gamma v_{T - t}] = Q_{t:T}^{\pi_e}(s,\tilde{a})$  (as shown above) in a single summation over  $\tilde{a}\in \mathcal{A}$ , in (2) we swap the order of summations, in (3) we take out common factors that do not depend on the inner summation  $a$ , and in (4) we use Definition 1 for  $\pi_{b^{+}}(\tilde{a}|s)$ , which cancels out the denominator in the importance ratio on the next line. The proof techniques are similar to that in Appendix C.2.
Since  $\mathbb{E}[v_{T - t}] = \mathbb{E}_{s\sim d_{h + 1}}[V_{(t + 1):T}^{\pi_e}(s)]$  implies  $\mathbb{E}[v_{T - t + 1}] = \mathbb{E}_{s\sim d_t}[V_{t:T}^{\pi_e}(s)]$ , and the base case  $\mathbb{E}[v_0] = \mathbb{E}_{s\sim d_{T + 1}}[V_{(T + 1):T}^{\pi_e}(s)]$  is true, by mathematical induction, we have the desired property that  $\mathbb{E}[\hat{v}^{\mathrm{C - PDIS}}] = \mathbb{E}[v_T] = \mathbb{E}_{s\sim d_1}[V_{1:T}^{\pi_e}(s)] = v(\pi_e)$ .
D Extended Discussions on Practical Implications
D.1 Using an Approximate MDP Model to Correct Annotation Bias
In the sequential RL setting, if the annotation function reflects the expected returns under the behavior policy, i.e.  $G_{t} = Q_{t:T}^{\pi_{b}}$ , then annotations have a nonzero bias of  $\epsilon_{G_t} = Q_{t:T}^{\pi_b} - Q_{t:T}^{\pi_e}\neq 0$  (since  $Q_{t:T}^{\pi_e}\neq Q_{t:T}^{\pi_b}$  for meaningful OPE problems where  $\pi_e\neq \pi_b$ ), and consequently, C-PDIS using such annotations becomes biased. While such annotations may aid in model selection (as shown in experiments), here we further propose a procedure to convert the annotations so that they better reflect the evaluation policy.
From the offline data, we first learn an approximate model of the MDP  $\hat{\mathcal{M}}$ , which includes the transition model  $\hat{p}(s'|s,a)$  and reward model  $\hat{r}(s,a)$ .  $\hat{\mathcal{M}}$  is then used to evaluate both  $\pi_b$  and  $\pi_e$  using a model-based approach, leading to estimated horizon-specific Q-functions,  $\hat{Q}_{t:T}^{\pi_b}$  and  $\hat{Q}_{t:T}^{\pi_e}$ . We then approximate the annotation error for state  $s_t$  occurring at horizon  $t$  and counterfactual action  $\tilde{a}$  as  $\hat{\epsilon}_{G_t}(s_t,\tilde{a}) = \hat{Q}_{t:T}^{\pi_b}(s_t,\tilde{a}) - \hat{Q}_{t:T}^{\pi_e}(s_t,\tilde{a})$ . Given the factual state-action pair  $(s_t^{(i)},a_t^{(i)})$  and the counterfactual annotation  $g_t^{(i),\tilde{a}} \sim Q_{t:T}^{\pi_b}(s_t^{(i)},\tilde{a})$  for action  $\tilde{a}$ , we convert the annotation as
$$
\hat {g} _ {t} ^ {(i), \tilde {a}} = g _ {t} ^ {(i), \tilde {a}} - \hat {\epsilon} _ {G _ {t}} (s _ {t}, \tilde {a})
$$
One may verify that  $\hat{g}_t^{(i),\tilde{a}}\sim Q_{t:T}^{\pi_e}(s_t^{(i)},\tilde{a})$  in expectation. Note that this approach for annotation conversion is only possible if the counterfactual action has support in the offline data (i.e.,  $(s_t^{(i)},\tilde{a})$  is seen in the data); otherwise, we suggest using the annotation as collected.
D.2 Imputing Missing Annotations
Consider the following scenario: for two instances of the same factual  $(s, a)$  in the dataset, only one instance has a counterfactual annotation but not the other. This means that the weights for the factual and counterfactual are  $(0.5, 0.5)$  and  $(1, 0)$  (assuming a binary action space), and the weight distribution  $W(\cdot | s, a)$  has a nonzero variance and covariance. Specifically, in the example above with two samples,  $\sigma_W(a | s, a)^2 = \sigma_W(\tilde{a} | s, a)^2 = \mathrm{Cov}(W(a | s, a), W(\tilde{a} | s, a)) = 0.0625$ . These variances appear in the variance decomposition of C-IS as shown in Theorem 13 and may lead to an overall larger variance compared to IS.
To address this issue, we suggest a procedure to impute the missing annotations using other available annotations when possible. Given all annotations  $\pmb{g}_{\mathcal{D}} = \{g_{t}^{(i),\tilde{a}}:c_{t}^{(i),\tilde{a}} = 1\}$  associated with dataset  $\mathcal{D} = \{\tau^{(i)}\}_{i=1}^{N}$ , we first build an approximate annotation model  $\hat{G}$  by solving a regression problem on  $\{(s_{t}^{(i)},\tilde{a}),g_{t}^{(i),\tilde{a}}):c_{t}^{(i),\tilde{a}} = 1\}$  using all available annotations  $\pmb{g}_{\mathcal{D}}$ . For discrete state and action spaces, this is essentially to averaging the annotations for each state-action pair. Then,  $\hat{G}$  can be used to impute the missing annotations as  $\hat{\pmb{g}} = \{\hat{G}(s_{t}^{(i)},\tilde{a}):c_{t}^{(i),\tilde{a}} = 0,\tilde{a}\in \mathcal{A}\setminus \{a_{t}^{(i)}\}\}$  if there is "support" for annotations of the same state and action, i.e.,  $c_{t}^{(i),\tilde{a}} = 0$  (annotation is missing) and  $\sum_{i'=1}^{N}\sum_{t'=1}^{T}\mathbb{1}[s_{t'}^{(i')} = s_{t}^{(i)}]c_{t'}^{(i'),\tilde{a}} > 0$  (annotation has support and thus can be imputed). This approximate annotation model  $\hat{G}$  may be biased, but generally its bias can be outweighed by the benefit of variance reduction for having equal weights (we also observed this empirically in Appendices E.1 and E.2).
D.3 Optimizing Weighting Schemes to Minimize Variance
Using a worked example, we illustrate that optimizing weights for variance reduction is a highly non-trivial problem. Suppose we have a bandit problem with one state  $\{s\}$ , two actions  $\{\nearrow, \searrow\}$ , and the reward function is  $R(s, \nearrow) = \mathcal{N}(r_0, \sigma_0^2)$ ,  $R(s, \searrow) = \mathcal{N}(r_1, \sigma_1^2)$ . Furthermore, suppose all annotations are available, allowing us to use constant weights and eliminate the weight variances, and suppose the annotation function is identical to the reward function, i.e.,  $G = R$ . We can simplify the variance decomposition of C-IS to be
$$
\begin{array}{l} \mathbb {V} \left[ \hat {v} ^ {\mathrm {C - I S}} \right] = \mathbb {V} _ {s \sim d _ {1}} \left[ V ^ {\pi_ {e}} (s) \right] + \mathbb {E} _ {s \sim d _ {1}} \left[ \mathbb {V} _ {a \sim \pi_ {b} (s)} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \rho^ {+} (\tilde {a} | s) \bar {W} (\tilde {a} | s, a) \bar {R} (s, \tilde {a}) \right] \right] \\ + \mathbb {E} _ {s \sim d _ {1}} \mathbb {E} _ {a \sim \pi_ {b} (s)} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \rho^ {+} (\tilde {a} | s) ^ {2} \bar {W} (\tilde {a} | s, a) ^ {2} \sigma_ {R} (s, \tilde {a}) ^ {2} \right] \tag {4} \\ \end{array}
$$
We denote policies in terms of the probabilities assigned to the two actions.
Setting 1. Consider  $\pi_b(s) = [1,0]$ ,  $\pi_e(s) = [\alpha,1 - \alpha]$ . The offline dataset contains only  $(s,a = \nearrow)$  with counterfactual annotations  $(s,\tilde{a} = \searrow)$ , and we assume they are each assigned a weight of  $w$  and  $(1 - w)$ . The augmented behavior policy is then  $\pi_{b^{+}}(s) = [w,1 - w]$ . The first and second terms of Eqn. (4) are zero because no variance is associated with sampling state-action pairs (only one possibility under  $\pi_b$ ). The third term becomes (state  $s$  is omitted from the expressions)
$$
\begin{array}{l} \frac {\pi_ {e} (\nearrow)}{\pi_ {b ^ {+}} (\nearrow)} \bar {W} (\nearrow | \nearrow) ^ {2} \sigma_ {0} ^ {2} + \frac {\pi_ {e} (\searrow)}{\pi_ {b ^ {+}} (\searrow)} \bar {W} (\searrow | \nearrow) ^ {2} \sigma_ {1} ^ {2} \\ = \frac {\alpha}{w} \times w ^ {2} \times \sigma_ {0} ^ {2} + \frac {1 - \alpha}{1 - w} \times (1 - w) ^ {2} \times \sigma_ {1} ^ {2} \\ = w \alpha \sigma_ {0} ^ {2} + (1 - w) (1 - \alpha) \sigma_ {1} ^ {2} \\ = \left(\alpha \sigma_ {0} ^ {2} - (1 - \alpha) \sigma_ {1} ^ {2}\right) w + (1 - \alpha) \sigma_ {1} ^ {2} \\ \end{array}
$$
This is a linear function of  $w$ , and where the variance achieves the minimum value depends on the slope  $(\alpha \sigma_0^2 - (1 - \alpha) \sigma_1^2)$ .
- If  $\alpha \sigma_0^2 = (1 - \alpha) \sigma_1^2$ , the slope is zero, and  $w$  does not affect variance.
- If  $\alpha \sigma_0^2 < (1 - \alpha) \sigma_1^2$ , the slope is negative, and  $w^* \to 1$  achieves minimum variance of  $\alpha \sigma_0^2$ .
- If  $\alpha \sigma_0^2 > (1 - \alpha) \sigma_1^2$ , the slope is positive, and  $w^* \to 0$  achieves minimum variance of  $(1 - \alpha) \sigma_1^2$ .
Setting 2. Consider  $\pi_b(s) = [\beta, 1 - \beta]$ ,  $\pi_e(s) = [\alpha, 1 - \alpha]$ . In the offline dataset,  $(s, a = \nearrow)$  appears with probability  $\beta$  and  $(s, a = \searrow)$  appears with probability  $1 - \beta$ . When the factual data is  $(s, a = \nearrow)$ , the counterfactual annotation is for  $(s, \tilde{a} = \searrow)$ , and let the weights be  $w_0$  and  $(1 - w_0)$ ; when the factual data is  $(s, a = \searrow)$ , the counterfactual annotation is for  $(s, \tilde{a} = \nearrow)$ , and let the weights be  $w_1$  and  $(1 - w_1)$ . The augmented behavior policy is then
$$
\pi_ {b ^ {+}} (s) = \left\{ \begin{array}{l} \beta w _ {0} + (1 - \beta) (1 - w _ {1}) \\ \beta (1 - w _ {0}) + (1 - \beta) w _ {1} \end{array} \right..
$$
The first term of Eqn. (4) is zero because there is only one state. The third term is
$$
\begin{array}{l} \beta \left(\frac {\pi_ {e} (\nearrow)}{\pi_ {b ^ {+}} (\nearrow)} \bar {W} (\nearrow | \nearrow) ^ {2} \sigma_ {0} ^ {2} + \frac {\pi_ {e} (\searrow)}{\pi_ {b ^ {+}} (\searrow)} \bar {W} (\searrow | \nearrow) ^ {2} \sigma_ {1} ^ {2}\right) \\ + (1 - \beta) \left(\frac {\pi_ {e} (\searrow)}{\pi_ {b ^ {+}} (\searrow)} \bar {W} (\nearrow | \searrow) ^ {2} \sigma_ {0} ^ {2} + \frac {\pi_ {e} (\nearrow)}{\pi_ {b ^ {+}} (\nearrow)} \bar {W} (\nearrow | \searrow) ^ {2} \sigma_ {1} ^ {2}\right) \\ = \beta \left(\frac {\alpha}{\beta w _ {0} + (1 - \beta) (1 - w _ {1})} \times w _ {0} ^ {2} \times \sigma_ {0} ^ {2} + \frac {1 - \alpha}{\beta (1 - w _ {0}) + (1 - \beta) w _ {1}} \times (1 - w _ {0}) ^ {2} \times \sigma_ {1} ^ {2}\right) \\ + (1 - \beta) \left(\frac {\alpha}{\beta w _ {0} + (1 - \beta) (1 - w _ {1})} \times (1 - w _ {1}) ^ {2} \times \sigma_ {0} ^ {2} + \frac {1 - \alpha}{\beta (1 - w _ {0}) + (1 - \beta) w _ {1}} \times w _ {1} ^ {2} \times \sigma_ {1} ^ {2}\right) \\ = \frac {\beta w _ {0} ^ {2} + (1 - \beta) (1 - w _ {1}) ^ {2}}{\beta w _ {0} + (1 - \beta) (1 - w _ {1})} \times \alpha \sigma_ {0} ^ {2} + \frac {\beta (1 - w _ {0}) ^ {2} + (1 - \beta) w _ {1} ^ {2}}{\beta (1 - w _ {0}) + (1 - \beta) w _ {1}} \times (1 - \alpha) \sigma_ {1} ^ {2} \\ \end{array}
$$
Next, we will attempt to simplify the second variance term:
$$
\left. \mathbb {V} _ {a \sim \pi_ {b} (s)} \left[ \sum_ {\tilde {a} \in A} \rho^ {+} (\tilde {a} | s) \bar {W} (\tilde {a} | s, a) \bar {R} (s, \tilde {a}) \right]. \right.
$$