SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Shengpu Tang, Jenna Wiens
University of Michigan, Ann Arbor, MI, USA
{tangsp,wiensj} @umich.edu
Reviewed on OpenReview: https://openreview.net/forum?id=dsH244r9fA
In applying reinforcement learning (RL) to high-stakes domains, quantitative and qualitative evaluation using observational data can help practitioners understand the generalization performance of new policies. However, this type of off-policy evaluation (OPE) is inherently limited since offline data may not reflect the distribution shifts resulting from the application of new policies. On the other hand, online evaluation by collecting rollouts according to the new policy is often infeasible, as deploying new policies in these domains can be unsafe. In this work, we propose a semi-offline evaluation framework as an intermediate step between offline and online evaluation, where human users provide annotations of unobserved counterfactual trajectories. While tempting to simply augment existing data with such annotations, we show that this naive approach can lead to biased results. Instead, we design a new family of OPE estimators based on importance sampling (IS) and a novel weighting scheme that incorporate counterfactual annotations without introducing additional bias. We analyze the theoretical properties of our approach, showing its potential to reduce both bias and variance compared to standard IS estimators. Our analyses reveal important practical considerations for handling biased, noisy, or missing annotations. In a series of proof-of-concept experiments involving bandits and a healthcare-inspired simulator, we demonstrate that our approach outperforms purely offline IS estimators and is robust to imperfect annotations. Our framework, combined with principled human-centered design of annotation solicitation, can enable the application of RL in high-stakes domains.
Figure 1: Left - The state transition diagram of a tree MDP with 3 steps and 2 actions. States are denoted by  $\bigcirc$ , actions are denoted by arrows  $\{\nearrow, \searrow\}$ , rewards are denoted in red and given only at terminal transitions. Center - The behavior policy takes a specific sequence of actions and leads to the factual trajectory, leaving the rest of the state-action space with poor support. Right - The counterfactual annotations provided by human annotators (indicated by  $\widehat{A}$ ) capture information (in this example, the terminal reward under any policy that takes  $(\nearrow, \nearrow)$  for the second and third steps) about support-deficient regions of the state-action space not visited by the behavior policy.
Reinforcement learning (RL) has gained popularity in recent years for its ability to solve sequential decision-making problems in various domains [1-7]. Despite these successes, it remains challenging to deploy and use RL in highly consequential or safety-critical domains, such as healthcare, education, and public policy [8-12]. One of the major roadblocks that distinguishes RL-based systems from their supervised learning counterparts is evaluation.
Evaluation of supervised learning models often involves calculating prediction accuracy against a labeled test set [13]. In contrast, evaluation of RL policies is less straightforward and often involves interacting with the environment [2, 3, 14-19]. For domains that lack accurate simulators, this means deploying new policies in the real environment. For instance, in healthcare, online evaluation would require clinicians to follow RL recommendations in selecting treatments for real patients. While mathematically sound, this presents clear safety issues and potential disruptions to workflows. Therefore, most work in these areas has relied exclusively on retrospective evaluations using observational data [20-22], focusing on both quantitative and qualitative aspects. Quantitative evaluations make use of statistical off-policy evaluation (OPE) methods to account for the distribution shift resulting from the application of new policies [23-25]. Despite their wide use, OPE is fundamentally limited by the available offline data. In particular, past work has noted that unexpected bias and large variance [20] among other reasons make these approaches unreliable [22, 26]. On the other hand, qualitative evaluations typically aim to verify with domain experts whether the RL recommendations are reasonable, but are difficult to standardize and may be susceptible to confirmation bias [20].
In this work, we consider an intermediate step before prospective deployment that improves upon offline evaluation of RL policies. Specifically, we assume human domain experts can provide annotations of unobserved counterfactual trajectories that are small deviations of the observed trajectory (Figure 1), where each annotation is some summary of the expected outcomes of counterfactual trajectories. For example, in healthcare domains, such annotations may be obtained by asking clinicians what they think would happen to the patient if a different treatment were to be used. Intuitively, these counterfactual annotations can make up for regions of the state-action space with poor support in the offline dataset. However, as we demonstrate, simply adding the annotations as new trajectories to the offline dataset will change the state distribution and lead to biased results. Thus, we design a new OPE estimator based on importance sampling (IS) that incorporates both the offline factual data and counterfactual annotations without introducing additional bias. We analyze the theoretical properties of our proposed estimator, noting its advantages over standard IS. Specifically, our estimator requires a weaker condition on support to achieve unbiasedness and has the potential to reduce variance. Through a series of proof-of-concept experiments using toy problems and a healthcare-inspired simulator, we show the benefits of our approach in making use of counterfactual annotations to enable better evaluations of RL policies, even when annotations are biased, noisy, or missing. Our semi-offline evaluation framework represents an important step that complements offline evaluations by providing additional confidence in RL policies.
We consider Markov decision processes (MDPs) defined by a tuple  $\mathcal{M} = (\mathcal{S},\mathcal{A},P,R,d_1,\gamma ,T)$ , where  $\mathcal{S}$  and  $\mathcal{A}$  are the state and action spaces,  $P:S\times \mathcal{A}\to \Delta (\mathcal{S})$  and  $R:S\times \mathcal{A}\rightarrow \Delta (\mathbb{R})$  are the transition and reward functions,  $d_{1}\in \Delta (\mathcal{S})$  is the initial state distribution,  $\gamma \in [0,1]$  is the discount factor,  $T\in \mathbb{Z}^{+}$  is the fixed horizon.  $p(s'|s,a)$  denotes the probability density function of  $P$ , and  $\tilde{R}(s,a)$  denotes the expected reward. A policy  $\pi : S\to \Delta (\mathcal{A})$  specifies a mapping from each state to a probability distribution over actions. A  $T$ -step trajectory following policy  $\pi$  is denoted by  $\tau = [(s_t,a_t,r_t)]_{t = 1}^T$  where  $s_1\sim d_1,a_t\sim \pi (s_t),r_t\sim R(s_t,a_t),s_{t + 1}\sim p(s_t,a_t)$ . Here,  $a\sim \pi (s)$  is short for  $a\sim \pi (\cdot |s)$  and  $s^\prime \sim p(s,a)$  for  $s^\prime \sim p(\cdot |s,a)$ . Let  $J = \sum_{t = 1}^{T}\gamma^{t - 1}r_{t}$  denote the return of the trajectory, which is the discounted sum of rewards. The value of a policy  $\pi$  is the expected return, defined as  $v(\pi) = \mathbb{E}_{\pi}[J]$ . The value function of policy  $\pi$ , denoted by  $V^{\pi}:S\to \mathbb{R}$ , maps each state to the expected return starting from that state following policy  $\pi$ . Similarly, the action-value function (i.e., the Q-function),  $Q^{\pi}:S\times \mathcal{A}\to \mathbb{R}$ , is defined by further restricting the action taken from the starting state. Formally,  $V^{\pi}(s) = \mathbb{E}_{\pi}[J|s_1 = s]$ , and  $Q^{\pi}(s,a) = \mathbb{E}_{\pi}[J|s_1 = s,a_1 = a]$ . We also consider value functions at specific horizons:  $V_{t:T}^{\pi}(s) = \mathbb{E}_{\pi}[\sum_{t'=t}^{T}\gamma^{t'-1}r_{t'}|s_t = s]$ , and  $Q_{t:T}^{\pi}(s,a) = \mathbb{E}_{\pi}[\sum_{t'=t}^{T}\gamma^{t'-1}r_{t'}|s_t = s,a_t = a]$ . Throughout the paper we also consider the non-sequential, bandit setting with horizon  $T = 1$ . In this case, a "trajectory" (or, a sample) is denoted by  $\tau = (s,a,r)$  where we omit the time step subscript.
Our goal is to estimate  $v(\pi_e)$ , the value of an evaluation policy  $\pi_e$ , given data that were previously collected by some behavior policy  $\pi_b$  in the same environment defined by  $\mathcal{M}$ . Let  $\mathcal{D} = \{\tau^{(i)}\}_{i=1}^N$  denote the dataset containing  $N$  independent trajectories drawn according to  $\pi_b$  and  $\mathcal{M}$ .
OPE. The typical approach to this problem relies on off-policy evaluation (OPE). Importance sampling (IS) is a common OPE approach that reweights samples based on how likely they are to occur under  $\pi_e$  relative to  $\pi_b$ . Given a trajectory  $\tau$ , the 1-step and cumulative IS ratios are defined as  $\rho_t = \frac{\pi_e(a_t|s_t)}{\pi_b(a_t|s_t)}$  and  $\rho_{1:t} = \prod_{t'=1}^{t} \rho_{t'}$ . The per-decision IS estimator,  $\hat{v}^{\mathrm{PDIS}} = \sum_{t=1}^{T} \rho_{1:t} \gamma^{t-1} r_t$ , is an unbiased estimator of  $v(\pi_e)$  [27, 28]. We also consider its recursive definition:  $\hat{v}^{\mathrm{PDIS}} = v_T$  where  $v_0 = 0$ ,  $v_{T-t+1} = \rho_t(r_t + \gamma v_{T-t})$ . In this paper, we discuss the properties of IS-based estimators over a single trajectory; our results naturally generalize to dataset  $\mathcal{D}$  containing  $N$  trajectories where the final estimator is the average over trajectories. For the bandit setting, we refer to PDIS simply as the IS estimator,  $\hat{v}^{\mathrm{IS}} = \rho r = \frac{\pi_e(a|s)}{\pi_b(a|s)} r$ .
Counterfactual Annotations. In addition to the offline dataset  $\mathcal{D}$ , our semi-offline framework assumes access to accompanying counterfactual annotations. To introduce the notation, we start with the non-sequential, bandit setting where  $T = 1$ , dropping the time step subscripts. Given a factual sample  $\tau = (s,a,r)$ , let  $c^{\tilde{a}} \in \{0,1\}$  be a binary indicator for whether the counterfactual action  $\tilde{a} \in \mathcal{A} \setminus \{a\}$  is associated with an annotation, and let the annotation be  $g^{\tilde{a}} \in \mathbb{R}$ . We use  $G: S \times \mathcal{A} \to \Delta(\mathbb{R})$  to denote the annotation function such that  $g^{\tilde{a}} \sim G(s,\tilde{a})$ . A counterfactual-augmented sample  $\tau^{+} = (\tau,g)$  consists of the factual sample  $\tau$  and counterfactual annotations  $g = \{g^{\tilde{a}}: c^{\tilde{a}} = 1\}$ , where each  $g^{\tilde{a}} \sim G(s,\tilde{a})$ . Intuitively, a "good" annotation should reflect the scenario where the counterfactual action  $\tilde{a}$  is taken and the reward  $g^{\tilde{a}} \sim R(s,\tilde{a})$  is observed.
Assumption 1 (Perfect annotation, bandit).  $\mathbb{E}_{g\sim G(s,a)}[g] = \bar{R} (s,a),\forall s\in S,a\in \mathcal{A}$
For the sequential setting, we define the corresponding notation with time step subscripts: for  $(s_t,a_t,r_t)$  occurring at step  $t$  of trajectory  $\tau$ , we define counterfactual indicators  $c_{t}^{\tilde{a}}$  for  $\tilde{a}\in \mathcal{A}\setminus \{a_t\}$  and annotations  $\pmb {g}_t = \{g_t^{\tilde{a}}:c_t^{\tilde{a}} = 1\}$ . Figure 2 provides an example trajectory with counterfactual annotations. Here, each  $g_{t}^{\tilde{a}}\sim G_{t}(s_{t},\tilde{a})$  is drawn from the horizon-  $t$  annotation function  $G_{t}$ . While the general notion of counterfactual annotations could be used to capture different information (e.g., the instantaneous reward of the counterfactual action,  $R(s,\tilde{a})$ ), in this work, we study a specific version that allows us to extend the theory of the bandit setting. Specifically, the annotation for counterfactual action  $\tilde{a}$  summarizes the annotator's belief of the expected future return (sum
Figure 2: A trajectory augmented with counterfactual annotations, where the action space is  $\mathcal{A} = \{\mathsf{x},\mathsf{y},\mathsf{z}\}$ . The factual trajectory  $\tau$  is shown in black. Solid blue arrows indicate the counterfactual annotations were queried and obtained; dashed gray arrows indicate the annotations are not available. Each transition arrow is labeled with (action, value), where value is either an observed immediate reward or a counterfactual annotation.
of rewards) in the remaining  $T - t + 1$  steps after taking the counterfactual action  $\tilde{a}$  from state  $s_t$ , and then following the evaluation policy  $\pi_e$ . In other words, the annotation plays the same role as the Q-function. This leads to a more refined assumption on the horizon-specific annotation function  $G_t$ .
Assumption 2 (Perfect annotation, MDP).  $\mathbb{E}_{g\sim G_t(s,a)}[g] = Q_{t:T}^{\pi_e}(s,a),\forall s\in S,a\in \mathcal{A}.$
Under Assumption 2, if we obtained infinitely many annotations for all initial states and all actions, then evaluation becomes trivial (we essentially recover the Q-function of all initial states). However, we consider the non-asymptotic regime where not every annotation is available, as certain annotations might be difficult to obtain. For example, annotating initial states requires reasoning about the full horizon  $T$ . Furthermore, since this is a rather strong assumption (we need different annotations for each  $\pi_e$ ), later we explore a relaxation where the annotations reflect the behavior policy  $\pi_b$  instead.
To motivate our approach, we begin with a didactic bandit example to illustrate how the naive incorporation of counterfactual annotations can yield biased estimates. In order to address this issue, we propose a modification of IS estimators that reweights the factual data and counterfactual annotations. We formally describe how this idea applies to IS (in the bandit setting) and PDIS (in the sequential RL setting), giving rise to a family of semi-offline counterfactual-augmented IS estimators. We study the impact of different assumptions regarding the annotations on the performance of our proposed estimators both theoretically (Section 4) and empirically (Section 5).
Consider a one-step bandit (Figure 3a) with two states  $\{s_1, s_2\}$  (drawn with equal probability) and two actions, up  $(\nearrow)$  and down  $(\searrow)$ . The reward from  $s_1$  is  $+1$  and from  $s_2$  is  $0$  (i.e., rewards do not depend on the action), meaning all policies have an expected reward of  $0.5$ . Suppose the behavior policy always selects  $\nearrow$ , generating a dataset with poor support for policies that assign nonzero probabilities to  $\searrow$  (Figure 3b). Now suppose we also have access to human-provided annotations of counterfactual actions, but not all counterfactual annotations are available (either because they were never queried or the users declined to provide annotations). In our example (Figure 3c), one annotation is collected for the counterfactual action  $\searrow$  at state  $s_1$ , indicating that the human annotator believes the reward for taking action  $\searrow$  from state  $s_1$  is  $+1$  (which is the true reward). To make use of this information, one might be tempted to add the counterfactual annotation as a new sample. The augmented dataset (Figure 3d) would allow us to evaluate policies (e.g., using IS) that assign non-zero probabilities to  $\searrow$  in state  $s_1$ . While seemingly plausible, this naive approach inadvertently changes the state distribution and results in a dataset inconsistent with the original problem (it looks like state  $s_1$  is seen more often than reality). A quick calculation reveals that applying IS to this unweighted augmented dataset gives a biased estimate of a new policy as  $2/3$  instead of  $0.5$  (see Appendix B). To address this issue, in Section 3.2 we propose a new reweighting procedure that maintains the state distribution of the original dataset while incorporating counterfactual annotations.
(a) State diagram
(c) Counterfactual Annotations
Figure 3: (a) The state diagram of a bandit problem with two states and two actions. (b) A factual dataset containing two samples. (c) The factual samples augmented with counterfactual annotations. (d) The (unweighted) augmented dataset constructed from factual samples and counterfactual annotations. Compared to the original factual dataset, the relative frequency of  $s_1$  vs  $s_2$  has changed from  $1:1$  to  $2:1$ .
(b) Factual dataset
(d) Unweighted augmented dataset
3.2 Augmenting IS Estimators with Counterfactual Annotations
To avoid the bias issue described in Section 3.1, informally, we want to split the contribution of each sample between the factual data and counterfactual annotations. Given a factual sample  $(s,a,r)$  and the associated counterfactual annotations  $g$ , let  $\boldsymbol{w} = \{w^{a}\} \cup \{w^{\tilde{a}}: \tilde{a} \in \mathcal{A} \setminus \{a\}\}$  be a set of user-defined non-negative weights that satisfy  $w^{a} + \sum_{\tilde{a} \in \mathcal{A} \setminus \{a\}} w^{\tilde{a}} = 1$ . These weights specify how much we want the estimator to "listen" to the counterfactual annotations  $(w^{\tilde{a}})$  relative to the factual data  $(w^{a})$ . We restrict  $w^{\tilde{a}} = 0$  when  $c^{\tilde{a}} = 0$ , i.e., non-zero weight is only allowed when the annotation is available. In general, one may assign different weights for each occurrence of  $(s,a)$  (e.g., the counterfactual annotation is obtained for one instance but missing for another); let  $\bar{W}(\tilde{a}|s,a) = \mathbb{E}[w^{\tilde{a}}]$  denote the average weight assigned to  $\tilde{a}$  when the factual data is  $(s,a)$  (see example in Appendix B.1). After reweighting, the state distribution is maintained (since the weights associated with each sample sum to 1) but the state-conditional action distributions have changed; this "weighted" augmented dataset can be seen as if it was generated using a different behavior policy.
Definition 1 (Augmented behavior policy).
$$
\pi_ {b ^ {+}} (a | s) = \bar {W} (a | s, a) \pi_ {b} (a | s) + \sum_ {\check {a} \in \mathcal {A} \backslash \{a \}} \bar {W} (a | s, \check {a}) \pi_ {b} (\check {a} | s).
$$
Here,  $\pi_{b^{+}}(a|s)$  represents the probability that information about action  $a$  is observed for state  $s$  (similar to how an "average policy" may be defined for multiple behavior policies [29]), either as a factual action in the dataset, or as an annotated counterfactual action when some other action  $\check{a}$  is the factual action. Next, we define our proposed estimators (for bandits) based on IS.
Definition 2 (Counterfactual-augmented IS). Given a counterfactual-augmented sample  $\tau^{+} = (\tau, g)$  and weights  $\boldsymbol{w} = \{w^{\tilde{a}} : \tilde{a} \in \mathcal{A}\}$ , where  $\tau = (s, a, r)$ ,  $\boldsymbol{g} = \{g^{\tilde{a}} : c^{\tilde{a}} = 1\}$ , the C-IS estimator is  $\hat{v}^{\mathrm{C - IS}} = w^{a}\rho^{a}r + \sum_{\tilde{a} \in \mathcal{A} \setminus \{a\}} w^{\tilde{a}}\rho^{\tilde{a}}g^{\tilde{a}}$ , where  $\rho^{\tilde{a}} = \frac{\pi_{e}(\tilde{a}|s)}{\pi_{b^{+}}(\tilde{a}|s)}$  for each  $\tilde{a} \in \mathcal{A}$ .
The C-IS estimator is a weighted convex combination of the factual IS estimate  $\rho^a r$  and the counterfactual IS estimates  $\rho^{\tilde{a}}g^{\tilde{a}}$  for all counterfactual actions  $\tilde{a}\in \mathcal{A}\setminus \{a\}$ . We also study a special case where all annotations are available and the weights are split equally among actions, such that  $w^{a} = w^{\tilde{a}} = 1 / |\mathcal{A}|$ . Then,  $\pi_{b^{+}}$  becomes the uniformly random policy, and after substituting into Definition 2, we obtain the following estimator.
Definition 3 (C-IS with equal weights). Given a counterfactual-augmented sample  $\tau^{+} = (\tau, g)$ , the C*-IS estimator is  $\hat{v}^{\mathrm{C}^{*}\mathrm{-IS}} = \pi_{e}(a|s)r + \sum_{\tilde{a} \in \mathcal{A} \setminus \{a\}} \pi_{e}(\tilde{a}|s)g^{\tilde{a}}$ .
Remark. Definition 3 provides an alternative interpretation of the estimator when using equal weights: if Assumption 1 holds (i.e., the annotation function  $G$  is the true reward function  $R$ ), we effectively observe both the factual and counterfactual rewards from  $R$ . Then, we can directly use the definition of the value function to calculate the expected reward under  $\pi_e$  using the action probabilities  $\pi_e(\cdot | s)$ .
For the sequential setting, given a trajectory with  $T$  steps, we define the collection of weights over all time steps,  $\boldsymbol{w} = \{w_{t}^{a_{t}} : t = 1\dots T\} \cup \{w_{t}^{\tilde{a}} : \tilde{a} \in \mathcal{A} \setminus \{a_{t}\}, t = 1\dots T\}$ . The augmented behavior policy  $\pi_{b^{+}}$  is similarly defined (see Definition 1). By extending the recursive definition of PDIS, we obtain the following two estimators (assuming either arbitrary weights or equal weights).
Definition 4 (Counterfactual-augmented PDIS). Given a counterfactual-augmented trajectory  $\tau^{+} = (\tau, g)$  and weights  $\boldsymbol{w}$  as defined above, where  $\tau = [(s_t, a_t, r_t)]_{t=1}^T$ ,  $\boldsymbol{g} = \{g_t^{\tilde{a}} : c_t^{\tilde{a}} = 1\}$ , the C-PDIS estimator is  $\hat{v}^{\mathrm{C - PDIS}} = v_T$ , with  $v_T$  defined recursively as  $v_0 = 0$ ,  $v_{T-t+1} = w_t^{a_t} \rho_t^{a_t}(r_t + \gamma v_{T-t}) + \sum_{\tilde{a} \in \mathcal{A} \setminus \{a_t\}} w_t^{\tilde{a}} \rho_t^{\tilde{a}} g_t^{\tilde{a}}$  for  $t = T\dots 1$ , where  $\rho_t^{\tilde{a}} = \frac{\pi_e(\tilde{a}|s_t)}{\pi_{b^+}(\tilde{a}|s_t)}$  for each  $\tilde{a} \in \mathcal{A}$ .
Definition 5 (C-PDIS with equal weights). Given a counterfactual-augmented trajectory  $\tau^{+} = (\tau, g)$ , the C $^{*}$ -PDIS estimator is  $\hat{v}^{\mathrm{C}^{*}\text{-PDIS}} = v_{T}$ , with  $v_{T}$  defined recursively as  $v_{0} = 0$ ,  $v_{T - t + 1} = \pi_{e}(a_{t}|s_{t})(r_{t} + \gamma v_{T - t}) + \sum_{\tilde{a}\in \mathcal{A}\backslash \{a_{t}\}}\pi_{e}(\tilde{a}|s_{t})g_{t}^{\tilde{a}}$  for  $t = T\dots 1$ .
Next, we study the theoretical properties of our proposed estimators, relating their OPE performance (in terms of bias and variance) to assumptions on counterfactual annotations and the offline dataset.
4 Theoretical Analyses
We first present results for the bandit setting, where we study and compare the properties of the C-IS estimator with standard IS in terms of bias, variance, and the assumptions required, highlighting scenarios where bias and variance reduction is guaranteed. We then show how these results generalize to C-PDIS in the sequential RL setting. Finally, we discuss practical implications of the theoretical results. Full derivations are in Appendix C.
To begin, we review existing results for IS. Recall the following assumption of common support.
Assumption 3 (Common support).  $\pi_{e}(a|s) > 0\to \pi_{b}(a|s) > 0,\forall s\in S,a\in \mathcal{A}.$
If Assumption 3 holds, IS is unbiased (i.e.,  $\mathbb{E}_{\tau}[\hat{v}^{\mathrm{IS}}] = v(\pi_e))$  , and its variance is [27]:
$$
\mathbb {V} [ \hat {v} ^ {\mathrm {I S}} ] = \mathbb {V} _ {s \sim d _ {1}} [ V ^ {\pi_ {e}} (s) ] + \mathbb {E} _ {s \sim d _ {1}} \left[ \mathbb {V} _ {a \sim \pi_ {b} (s)} [ \rho (a | s) \bar {R} (s, a) ] \right] + \mathbb {E} _ {s \sim d _ {1}} \left[ \mathbb {E} _ {a \sim \pi_ {b} (s)} [ \rho (a | s) ^ {2} \sigma_ {R} (s, a) ^ {2} ] \right] (1)
$$
where  $\sigma_R(s,a)^2 = \mathbb{V}_{r\sim R(s,a)}[r]$  is the variance associated with the reward function  $R(s,a)$ . The first term reflects the inherent randomness from the state distribution not related to importance sampling. The second term reflects the randomness in the behavior policy, whereas the third term reflects the randomness in rewards; these two terms are affected by the distribution of importance ratios  $\rho (a|s)$ . When Assumption 3 is not satisfied, the IS estimator is biased [30], where the bias is related to actions with no support:  $\mathrm{Bias}[\hat{v}^{\mathrm{IS}}] = \mathbb{E}[\hat{v}^{\mathrm{IS}}] - v(\pi_e) = \mathbb{E}_{s\sim d_1}\left[-\sum_{a\in \mathcal{U}(s,\pi_b)}\pi_e(a|s)\bar{R} (s,a)\right]$ , with  $\mathcal{U}(s,\pi_b) = \{a:\pi_b(a|s) = 0\}$  denoting the set of unsupported actions.