SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our work focuses on OPE rather than policy learning, but the broader theme of human input for RL has recently gained renewed attention, particularly in natural language processing tasks [42-44]. In many of these problems, human input is in the form of preferences (or rankings) over actions, states, or (sub-)trajectories [45]. Other related annotation approaches also exist in non-RL areas, where past work has proposed to ask annotators to alter text to match a counterfactual target label [46] and incorporating annotations of potential unmeasured confounders [47]. In contrast, our work investigates the role of a specific form of human input, counterfactual annotations in offline RL, in improving OPE performance. We focus on offline evaluation due to its practical importance in high-stakes RL domains such as healthcare, though our insights could also potentially benefit offline learning in these domains. While we did not discuss how counterfactual annotations are obtained, our theoretical analysis establishes a thorough understanding of their desirable characteristics. This can help motivate methods for converting different forms of human feedback into useful counterfactual annotations (e.g., learning reward/annotation models from human preferences [42]). Conversely, progress in the field of preference learning and learning-to-rank may benefit our approach by providing mechanisms to solicit high-quality annotations [48].
7 Discussion & Conclusion
In this paper, we propose a novel semi-offline policy evaluation framework that incorporates counterfactual annotations into traditional IS estimators. We emphasize that the naive approach of viewing annotations as additional data can lead to bias and propose a simple reweighting scheme to address this issue. We formally study the theoretical properties of our approach, identifying scenarios where bias and variance reduction is guaranteed. Driven by a deep understanding of these theoretical properties, we further propose practically motivated strategies to handle biased, noisy, or missing annotations. Through proof-of-concept experiments on bandits and a healthcare-inspired RL simulator, we demonstrate that our approach outperforms standard IS and is robust to imperfect annotations. Our semi-offline framework serves as an intermediate step between offline and online evaluations and has the potential to enable practical applications of RL in high-stakes domains. Though motivated by current limitations of offline evaluation, we caution that our approach is not meant to replace existing OPE methods, but rather to complement them. Collecting annotations from domain experts comes at a cost (of real human time and labor), and thus, our approach should only be applied after a policy has passed all checks on retrospective data. While not explored in this paper, future work should focus on assessing the quality of counterfactual annotations in the domains of interest through human experiments. See Appendix A for more detailed discussions on limitations, societal impacts, and future directions. Overall, we believe our contributions will inspire further investigations into the practical obstacles that emerge in semi-offline evaluation (e.g., devising human-centered strategies for soliciting counterfactual annotations that align with theoretical assumptions) and will bring RL closer to reality in healthcare and other high-stakes decision-making domains.
This work was supported by the National Library of Medicine of the National Institutes of Health (grant R01LM013325 to JW). The views and conclusions in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the National Institutes of Health. The authors would like to thank Michael Sjoding and members of the MLD3 group for helpful discussions regarding this work, as well as the anonymous reviewers for constructive feedback.
Data and Code Availability
The code for all experiments is available at https://github.com/MLD3/CounterfactualAnnot-SemiOPE.
[39] Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite-horizon off-policy estimation. In Advances in Neural Information Processing Systems, 2018.
[40] Tengyang Xie, Yifei Ma, and Yu-Xiang Wang. Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling. In Advances in Neural Information Processing Systems, 2019.
[41] Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. DualDICE: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems, 2019.
[42] Nisan Stiannon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
[43] Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=8aHzds2uUyB.
[44] Qisen Yang, Shenzhi Wang, Matthieu Gaetan Lin, Shiji Song, and Gao Huang. Boosting offline reinforcement learning with action preference query. In International Conference on Machine Learning, 2023.
[45] Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Furnkranz, et al. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research, 18(136):1-46, 2017.
[46] Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations, 2020. URL https://openreview.net/forum?id=SklgssONFvr.
[47] Megha Srivastava, Tatsunori Hashimoto, and Percy Liang. Robustness to spurious correlations via human annotations. In International Conference on Machine Learning, pages 9109-9119. PMLR, 2020.
[48] Banghua Zhu, Jiantao Jiao, and Michael Jordan. Principled reinforcement learning with human feedback from pairwise or  $k$ -wise comparisons. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models, 2023.
Limitations. This work proposes and theoretically studies a new framework of semi-offline evaluation of RL policies. While our core idea is to incorporate human annotations into the evaluation process, we did not make use of real human annotations in our experiments and relied on simulations instead. Given the costs associated with user studies, we opted not to conduct user studies before we have a thorough understanding of when and why our approach works (or does not work) and how it can be implemented in practice. Our present paper focuses on describing and analyzing a formal mathematical framework of counterfactual annotations, which provides valuable insights as to what types of annotations are useful, how they should be incorporated into OPE, and which factors impact performance (see method description in Section 3 and theoretical analyses in Section 4). Our experiments further demonstrate the robustness of our approach when we deviate from the ideal setting (see Section 5). Despite best efforts, our experiments do not capture all possible scenarios in the real world. We list several important practical implications in Section 4.4 and encourage future research into these directions. Additional areas of interest that we did not explore include: alternative forms of annotations (e.g., preference, immediate reward), whether these other annotation types could be converted to match our assumptions so that our framework still applies, how best to design the question phrasing for the annotators, evaluating annotation quality, targeted annotation solicitation and optimizing annotation solicitation under budget constraint.
Ethical Considerations and Societal Impact. Since our experiments only involved simulations and no real humans, we did not pose immediate ethical concerns or safety threats. Nonetheless, in high-stakes decision-making tasks such as healthcare, computationally-derived RL policies must be carefully validated before their final adoption. While we are motivated by the current limitations of standard offline evaluation methods, we caution that our approach is not meant to replace, but rather to complement, existing OPE approaches. Collecting these annotations from real human domain experts comes at a cost, and thus we recommend applying our approach only when a policy has passed all checks on retrospective data. We note that compared to traditional qualitative evaluations (e.g., typically done by asking domain experts whether RL recommendations make sense), our approach is less likely (though still possible) to affect the existing decision making or suffer from confirmation bias, since we do not need to reveal the evaluation policy at the annotation collection stage. Future work that adopts our framework should carefully design the annotation solicitation process (including when and how the question is posed to the annotators) so as to achieve safe, non-disruptive evaluations of offline RL policies before their prospective use. Additionally, more work is needed to understand the extent to which humans can provide accurate annotations of counterfactuals. These human experiments should be addressed in the context of specific application domains and the target groups of expert human annotators.
Future Directions on Counterfactual Annotations. Our present paper establishes important theoretical groundwork for using counterfactual annotations in semi-offline policy evaluation. However, we did not collect any real annotations with human annotators as those are outside of the scope of the main research question we seek to address in our current paper. To fully realize the impact of this work, real-world human experiments would be the necessary next step, with a focus on obtaining faithful annotations (with small bias and small noise). More specifically, it would be important to empirically measure if and how various factors influence annotation quality, including: horizon  $t$  (annotating beginning of an episode or the terminal step), the inherent stochasticity associated with the annotated state-action pair and how often they appear in data (under  $\pi_b$ ). Equipped with this knowledge, it is then important to select which annotations to prioritize given a limited annotation budget. Many of these questions delve into the realm of HCI and are outside the scope of this paper's methodological contributions for offline RL and OPE, and we encourage researchers and practitioners in different research areas (e.g., RL, HCI, healthcare, education) to build upon the ideas in our work.
B Toy Example for Intuition
(b) Factual dataset
Figure 6: For ease of reference, Figure 3 is reproduced here. (a) The state diagram of a bandit problem with two states and two actions. (b) A factual dataset containing two samples. (c) The factual samples augmented with counterfactual annotations. (d) The (unweighted) augmented dataset constructed from factual samples and counterfactual annotations. Compared to the factual dataset, the relative frequency of  $s_1$  vs  $s_2$  changed from  $1:1$  to  $2:1$ .
(c) Counterfactual Annotations
Recall the example discussed in Section 3.1 (figure reproduced here in Figure 6). The bandit problem has two states  $\{s_1, s_2\}$  (drawn with equal probability) and two actions, up  $(\nearrow)$  and down  $(\searrow)$ . The reward from  $s_1$  is  $+1$  and from  $s_2$  is  $0$  (i.e., rewards do not depend on the action), meaning all policies have an expected reward of  $0.5$ . Suppose the behavior policy always selects  $\nearrow$ , generating a dataset with poor support for policies that assign nonzero probabilities to  $\searrow$  (Figure 6b). Now suppose we also have access to human-provided annotations of counterfactual actions, but not all counterfactual annotations are available (either because they were never queried or the users declined to provide annotations). In our example (Figure 6c), one annotation is collected for the counterfactual action  $\searrow$  at state  $s_1$ , indicating that the human annotator believes the reward for taking action  $\searrow$  from state  $s_1$  is  $+1$ . To make use of this information, one might consider adding the counterfactual annotation as a new sample. The augmented dataset (Figure 6d) would allow us to evaluate policies (e.g., using IS) that assign non-zero probabilities to  $\searrow$  in state  $s_1$ . Unfortunately, this naive approach leads to biased results, which we walk through in detail below.
Consider an evaluation policy that takes  $\searrow$  in  $s_1$  and  $\nearrow$  in  $s_2$ , i.e.,  $\pi_e(s_1) = [0, 1]$  and  $\pi_e(s_2) = [1, 0]$  (denoted in terms of the probabilities assigned to the two actions,  $\nearrow$  and  $\searrow$ ).
- If we use the original  $\pi_b$  in the factual dataset (Figure 6b) where  $\pi_b(s_1) = [1,0]$ ,  $\pi_b(s_2) = [1,0]$ , the IS estimator is ill-defined because we will encounter a divide-by-zero error:
$$
\frac {1}{3} \left(\frac {0}{1} \times (+ 1) + \frac {1}{0} \times (+ 1) + \frac {1}{1} \times (0)\right) = \text {u n d e f i n e d}
$$
- Instead, one may consider the behavior policy in the augmented dataset (Figure 6d), which gives  $\pi_b(s_1) = [0.5, 0.5]$ ,  $\pi_b(s_2) = [1, 0]$ . The IS estimate is:
$$
\frac {1}{3} \left(\frac {0}{0 . 5} \times (+ 1) + \frac {1}{0 . 5} \times (+ 1) + \frac {1}{1} \times (0)\right) = \frac {2}{3}
$$
However, as stated above, all policies should have a value of 0.5, meaning that  $\frac{2}{3}$  is a biased estimate. The core of the issue is because directly adding counterfactual annotations inadvertently changes the state distribution and results in a dataset inconsistent with the original problem. In particular, comparing Figure 6b vs 6d, the relative frequency of  $s_1$  vs  $s_2$  has changed from  $1:1$  to  $2:1$ .
Our proposed estimator addresses this issue by reweighting the factual data and counterfactual annotations in order to maintain the state distribution. Suppose we assign a weight  $\alpha$  to  $(s_1, \nearrow)$  and  $1 - \alpha$  to  $(s_1, \searrow)$  for some  $\alpha \in (0, 1)$ , i.e., the two non-negative weights associated with  $s_1$  sum to 1. The sample  $(s_2, \nearrow)$  receives a weight of 1 by default since it does not have an associated counterfactual annotation. In this way, the state distribution remains equally split between  $s_1$  and  $s_2$ .
Using Definition 1, we can calculate the augmented behavior policy  $\pi_{b^{+}}(s_{1}) = [\alpha ,1 - \alpha ],\pi_{b^{+}}(s_{2}) = [1,0]$ . Applying C-IS as defined in Definition 3, we see our proposed approach produces the correct, unbiased estimate of 0.5:
$$
\mathrm {f o r} s _ {1}, \left(\alpha \times \frac {0}{\alpha} \times (+ 1) + (1 - \alpha) \times \frac {1}{1 - \alpha} \times (+ 1)\right) = 1
$$
$$
\mathrm {f o r} s _ {2}, \left(1 \times \frac {1}{1} \times (0)\right) = 0
$$
$$
\text {o v e r a l l :} \frac {1}{2} (1 + 0) = 0. 5
$$
B.1 Another Example Illustrating the Weighting Scheme
Suppose similar to Figure 6, we now have a bandit problem with only a single state  $\{s\}$  such that all policies have an expected reward of 1. Suppose the behavior policy always selects  $\nearrow$ , generating a dataset containing two samples: sample #1  $(s, \nearrow)$  with an annotation for  $\searrow$ , sample #2 is simply  $(s, \nearrow)$  with no annotation for  $\searrow$ . In applying our approach, for sample #1 we can assign weights  $[\alpha, 1 - \alpha]$  for some  $\alpha \in (0, 1)$ , i.e., the two non-negative weights associated with sample #1 sum to 1; for sample #2 the weights are simply  $[1, 0]$ . Then, the average weights for state  $s$  are  $\bar{W}(\nearrow |s, \nearrow) = \frac{1 + \alpha}{2}$  and  $\bar{W}(\searrow |s, \nearrow) = \frac{1 - \alpha}{2}$ . For example, if  $\alpha = 0.8$ , then the weights for sample #1 are  $[0.8, 0.2]$  and the average weights associated with state  $s$  are  $[0.9, 0.1]$ . Note that while the average weights  $\bar{W}$  (used for calculating  $\pi_{b^+}$  and C-IS) are state-specific, the user-assigned weights  $w^{\tilde{a}}$  are sample-specific; furthermore, the sample-specific weights should not be interpreted as "random", as a user may deliberately set weights to split equally into  $[0.5, 0.5]$ , or to  $[1, 0]$  which ignores the annotation if they believe it is of poor quality.
Using Definition 1, we can calculate the augmented behavior policy  $\pi_{b^{+}}(s) = [\frac{1 + \alpha}{2},\frac{1 - \alpha}{2} ]$  Applying C-IS as defined in Definition 2, we see our proposed approach produces an unbiased estimate of 1:
$$
\left(\frac {1 + \alpha}{2} \times \frac {0}{\frac {1 + \alpha}{2}} \times (+ 1) + \frac {1 - \alpha}{2} \times \frac {1}{\frac {1 - \alpha}{2}} \times (+ 1)\right) = 1
$$
C Extended Theoretical Analyses
Unless otherwise stated, the estimators are for  $\pi_e$ , i.e.,  $\hat{v} = \hat{v}(\pi_e)$ .
C.1 IS: Bias & Variance
We formally state and prove the bias and variance results for IS (informally described in Section 4). The proofs are adapted from existing literature [27, 30].
Theorem 9 (Unbiasedness of IS). In the bandit setting, when Assumption 3 holds,  $\mathbb{E}[v^{\mathrm{IS}}] = v(\pi_e)$
Proposition 10 (Bias of IS). In the bandit setting, when Assumption 3 is violated,  $\mathrm{Bias}[\hat{v}^{\mathrm{IS}}] = \mathbb{E}_{s\sim d_1}\left[-\sum_{a\in \mathcal{U}(s,\pi_b)}\pi_e(a|s)\bar{R} (s,a)\right]$  where  $\mathcal{U}(s,\pi_b) = \{a:\pi_b(a|s) = 0\}$  are unsupported actions in the dataset.
Proposition 11 (Variance of IS). In the bandit setting, when Assumption 3 holds, the variance of IS can be written as:
$$
\mathbb {V} [ \hat {v} ^ {\mathrm {I S}} ] = \mathbb {V} _ {s \sim d _ {1}} [ V ^ {\pi_ {e}} (s) ] + \mathbb {E} _ {s \sim d _ {1}} \left[ \mathbb {V} _ {a \sim \pi_ {b} (s)} [ \rho (a | s) \bar {R} (s, a) ] \right] + \mathbb {E} _ {s \sim d _ {1}} \left[ \mathbb {E} _ {a \sim \pi_ {b} (s)} [ \rho (a | s) ^ {2} \sigma_ {R} (s, a) ^ {2} ] \right]
$$
where  $\sigma_{R}(s,a)^{2} = \mathbb{V}_{r\sim R(s,a)}[r]$  is the variance associated with the reward function  $R(s,a)$ .
Derivation for Bias of IS (adapted from [30]).
$$
\begin{array}{l} \operatorname {B i a s} [ \hat {v} ^ {\mathrm {I S}} ] = \mathbb {E} [ \hat {v} ^ {\mathrm {I S}} ] - v (\pi_ {e}) \\ = \mathbb {E} _ {s \sim d _ {1}} \left[ \sum_ {a \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b})} \pi_ {b} (a | s) \frac {\pi_ {e} (a | s)}{\pi_ {b} (a | s)} \bar {R} (s, a) \right] - \mathbb {E} _ {s \sim d _ {1}} \left[ \sum_ {a \in \mathcal {A}} \pi_ {e} (a | s) \bar {R} (s, a) \right] \\ = \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {a \in \mathcal {A} \backslash \mathcal {U} (s, \pi_ {b})} \pi_ {e} (a | s) \bar {R} (s, a) - \sum_ {a \in \mathcal {A}} \pi_ {e} (a | s) \bar {R} (s, a) \right] \\ \end{array}
$$
$$
= \underset {s \sim d _ {1}} {\mathbb {E}} \left[ - \sum_ {a \in \mathcal {U} (s, \pi_ {b})} \pi_ {e} (a | s) \bar {R} (s, a) \right]
$$
If Assumption 3 holds,  $\pi_e(a|s) = 0$  for  $a\in \mathcal{U}(s,\pi_b)$  and thus  $\mathrm{Bias}[\hat{v}^{\mathrm{IS}}] = 0$
Derivation for Variance of IS (adapted from [27]). We apply the law of total variance:
$$
\begin{array}{l} \mathbb {V} [ \hat {v} ^ {\mathrm {I S}} ] = \mathbb {V} _ {s \sim d _ {1}, a \sim \pi_ {b} (s), r \sim R (s, a)} [ \rho r ] \\ = \mathbb {V} _ {s \sim d _ {1}} \left[ \mathbb {E} _ {a \sim \pi_ {b} (s), r \sim R (s, a)} [ \rho r ] \right] + \mathbb {E} _ {s \sim d _ {1}} \left[ \mathbb {V} _ {a \sim \pi_ {b} (s), r \sim R (s, a)} [ \rho r ] \right] \\ = \mathbb {V} _ {s \sim d _ {1}} \left[ \mathbb {E} _ {a \sim \pi_ {b} (s)} \left[ \frac {\pi_ {e} (a | s)}{\pi_ {b} (a | s)} \mathbb {E} _ {r \sim R (s, a)} [ r ] \right] \right] + \mathbb {E} _ {s \sim d _ {1}} \left[ \mathbb {V} _ {a \sim \pi_ {b} (s)} \mathbb {E} _ {r \sim R (s, a)} [ \rho r ] + \mathbb {E} _ {a \sim \pi_ {b} (s)} \mathbb {V} _ {r \sim R (s, a)} [ \rho r ] \right] \\ = \mathbb {V} _ {s \sim d _ {1}} \left[ \mathbb {E} _ {a \sim \pi_ {e} (s)} [ \bar {R} (s, a) ] \right] + \mathbb {E} _ {s \sim d _ {1}} \left[ \mathbb {V} _ {a \sim \pi_ {b} (s)} [ \rho \bar {R} (s, a) ] \right] + \mathbb {E} _ {s \sim d _ {1}} \mathbb {E} _ {a \sim \pi_ {b} (s)} \left[ \rho^ {2} \mathbb {V} _ {r \sim R (s, a)} [ r ] \right] \\ = \mathbb {V} _ {s \sim d _ {1}} \left[ V ^ {\pi_ {e}} (s) \right] + \mathbb {E} _ {s \sim d _ {1}} \left[ \mathbb {V} _ {a \sim \pi_ {b} (s)} [ \rho (a | s) \bar {R} (s, a) ] \right] + \mathbb {E} _ {s \sim d _ {1}} \left[ \mathbb {E} _ {a \sim \pi_ {b} (s)} [ \rho (a | s) ^ {2} \sigma_ {R} (s, a) ^ {2} ] \right] \\ \end{array}
$$
C.2 C-IS: Bias Analyses
We start by showing unbiasedness of C-IS in the ideal case (Theorem 1).
Proof of Theorem 1. Starting with Definition 2,
$$
\mathbb {E} \left[ \hat {v} ^ {\mathrm {C - I S}} \right] = \mathbb {E} \left[ w ^ {a} \rho^ {a} r + \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} w ^ {\tilde {a}} \rho^ {\tilde {a}} q ^ {\tilde {a}} \right]
$$
$$
\begin{array}{l} = \underset {s \sim d _ {1}} {\mathbb {E}} \underset {a \sim \pi_ {b} (s)} {\mathbb {E}} \left[ \mathbb {E} _ {w ^ {a} \sim W (a | s, a)} [ w ^ {a} ] \frac {\pi_ {e} (a | s)}{\pi_ {b ^ {+}} (a | s)} \mathbb {E} _ {r \sim R (s, a)} [ r ] \right. \\ \left. + \sum_ {\tilde {a} \in \mathcal {A} \backslash \{a \}} \mathbb {E} _ {w ^ {\tilde {a}} \sim W (\tilde {a} | s, a)} [ w ^ {\tilde {a}} ] \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} \mathbb {E} _ {g ^ {\tilde {a}} \sim G (s, \tilde {a})} [ g ^ {\tilde {a}} ] \right] \\ \end{array}
$$
$$
\begin{array}{l} \stackrel {(1)} {=} \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {a \in \mathcal {A}} \pi_ {b} (a | s) \left(\sum_ {\tilde {a} \in \mathcal {A}} \bar {W} (\tilde {a} | s, a) \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} \bar {R} (s, \tilde {a})\right) \right] \\ \stackrel {(2)} {=} \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \left(\sum_ {a \in \mathcal {A}} \pi_ {b} (a | s) \bar {W} (\tilde {a} | s, a) \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} \bar {R} (s, \tilde {a})\right) \right] \\ \stackrel {(3)} {=} \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \left(\left(\sum_ {a \in \mathcal {A}} \pi_ {b} (a | s) \bar {W} (\tilde {a} | s, a)\right) \frac {\pi_ {e} (\tilde {a} | s)}{\pi_ {b ^ {+}} (\tilde {a} | s)} \bar {R} (s, \tilde {a})\right) \right] \\ \stackrel {(4)} {=} \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \underline {{\pi_ {b ^ {+}} (\tilde {a} | s)}} \frac {\pi_ {e} (\tilde {a} | s)}{\underline {{\pi_ {b ^ {+}} (\tilde {a} | s)}}} \bar {R} (s, \tilde {a}) \right] \\ = \underset {s \sim d _ {1}} {\mathbb {E}} \left[ \sum_ {\tilde {a} \in \mathcal {A}} \pi_ {e} (\tilde {a} | s) \bar {R} (s, \tilde {a}) \right] \\ = \underset {s \sim d _ {1}} {\mathbb {E}} \underset {\tilde {a} \sim \pi_ {e}} {\mathbb {E}} [ \bar {R} (s, \tilde {a}) ] \\ = \underset {s \sim d _ {1}} {\mathbb {E}} \left[ V ^ {\pi_ {e}} (s) \right] \\ = v \left(\pi_ {e}\right) \\ \end{array}
$$
where in (1) we replace  $\mathbb{E}_{g\sim G(s,\tilde{a})}[g]$  with  $\bar{R} (s,\tilde{a})$  following Assumption 1 and combine it with  $\mathbb{E}_{r\sim R(s,a)}[r]$  in a single summation over  $\tilde{a}\in \mathcal{A}$ , in (2) we swap the order of summations, in (3) we take out common factors that do not depend on  $a$ , and in (4) we use Definition 1 for  $\pi_{b^{+}}(\tilde{a}|s)$ , which cancels out the denominator in the importance ratio on the next line.
Next, we look at two factors contributing to the bias of C-IS: lack of support and imperfect annotations.