Title: Counterfactual-Augmented Importance Sampling for Semi-Offline Policy Evaluation
Venue: NeurIPS 2023
Accepted: yes
Keywords: healthcare

reinforcement learning

offline RL

off-policy evaluation

counterfactuals
Abstract: In applying reinforcement learning (RL) to high-stakes domains, quantitative and qualitative evaluation using observational data can help practitioners understand the generalization performance of new policies. However, this type of off-policy evaluation (OPE) is inherently limited since offline data may not reflect the distribution shifts resulting from the application of new policies. On the other hand, online evaluation by collecting rollouts according to the new policy is often infeasible, as deploying new policies in these domains can be unsafe. In this work, we propose a semi-offline evaluation framework as an intermediate step between offline and online evaluation, where human users provide annotations of unobserved counterfactual trajectories. While tempting to simply augment existing data with such annotations, we show that this naive approach can lead to biased results. Instead, we design a new family of OPE estimators based on importance sampling (IS) and a novel weighting scheme that incorporate counterfactual annotations without introducing additional bias. We analyze the theoretical properties of our approach, showing its potential to reduce both bias and variance compared to standard IS estimators. Our analyses reveal important practical considerations for handling biased, noisy, or missing annotations. In a series of proof-of-concept experiments involving bandits and a healthcare-inspired simulator, we demonstrate that our approach outperforms purely offline IS estimators and is robust to imperfect annotations. Our framework, combined with principled human-centered design of annotation solicitation, can enable the application of RL in high-stakes domains.

Counterfactual-Augmented Importance Sampling for Semi-Offline Policy Evaluation

Abstract

In applying reinforcement learning (RL) to high-stakes domains, quantitative and qualitative evaluation using observational data can help practitioners understand the generalization performance of new policies. However, this type of off-policy evaluation (OPE) is inherently limited since offline data may not reflect the distribution shifts resulting from the application of new policies. On the other hand, online evaluation by collecting rollouts according to the new policy is often infeasible, as deploying new policies in these domains can be unsafe. In this work, we propose a semi-offline evaluation framework as an intermediate step between offline and online evaluation, where human users provide annotations of unobserved counterfactual trajectories. While tempting to simply augment existing data with such annotations, we show that this naive approach can lead to biased results. Instead, we design a new family of OPE estimators based on importance sampling (IS) and a novel weighting scheme that incorporate counterfactual annotations without introducing additional bias. We analyze the theoretical properties of our approach, showing its potential to reduce both bias and variance compared to standard IS estimators. Our analyses reveal important practical considerations for handling biased, noisy, or missing annotations. In a series of proof-of-concept experiments involving bandits and a healthcare-inspired simulator, we demonstrate that our approach outperforms purely offline IS estimators and is robust to imperfect annotations. Our framework, combined with principled human-centered design of annotation solicitation, can enable the application of RL in high-stakes domains.

+1

+1

+1

0

+1

0

0

0

?

?

+1

?

?

?

?

?

+1

0

+1

+1

?

?

?

1 Introduction

Reinforcement learning (RL) has gained popularity in recent years for its ability to solve sequential decision-making problems in various domains [1-7]. Despite these successes, it remains challenging to deploy and use RL in highly consequential or safety-critical domains, such as healthcare, education, and public policy [8-12]. One of the major roadblocks that distinguishes RL-based systems from their supervised learning counterparts is evaluation.

Evaluation of supervised learning models often involves calculating prediction accuracy against a labeled test set [13]. In contrast, evaluation of RL policies is less straightforward and often involves interacting with the environment [2, 3, 14-19]. For domains that lack accurate simulators, this means deploying new policies in the real environment. For instance, in healthcare, online evaluation would require clinicians to follow RL recommendations in selecting treatments for real patients. While mathematically sound, this presents clear safety issues and potential disruptions to workflows. Therefore, most work in these areas has relied exclusively on retrospective evaluations using observational data [20-22], focusing on both quantitative and qualitative aspects. Quantitative evaluations make use of statistical off-policy evaluation (OPE) methods to account for the distribution shift resulting from the application of new policies [23-25]. Despite their wide use, OPE is fundamentally limited by the available offline data. In particular, past work has noted that unexpected bias and large variance [20] among other reasons make these approaches unreliable [22, 26]. On the other hand, qualitative evaluations typically aim to verify with domain experts whether the RL recommendations are reasonable, but are difficult to standardize and may be susceptible to confirmation bias [20].

In this work, we consider an intermediate step before prospective deployment that improves upon offline evaluation of RL policies. Specifically, we assume human domain experts can provide annotations of unobserved counterfactual trajectories that are small deviations of the observed trajectory (Figure 1), where each annotation is some summary of the expected outcomes of counterfactual trajectories. For example, in healthcare domains, such annotations may be obtained by asking clinicians what they think would happen to the patient if a different treatment were to be used. Intuitively, these counterfactual annotations can make up for regions of the state-action space with poor support in the offline dataset. However, as we demonstrate, simply adding the annotations as new trajectories to the offline dataset will change the state distribution and lead to biased results. Thus, we design a new OPE estimator based on importance sampling (IS) that incorporates both the offline factual data and counterfactual annotations without introducing additional bias. We analyze the theoretical properties of our proposed estimator, noting its advantages over standard IS. Specifically, our estimator requires a weaker condition on support to achieve unbiasedness and has the potential to reduce variance. Through a series of proof-of-concept experiments using toy problems and a healthcare-inspired simulator, we show the benefits of our approach in making use of counterfactual annotations to enable better evaluations of RL policies, even when annotations are biased, noisy, or missing. Our semi-offline evaluation framework represents an important step that complements offline evaluations by providing additional confidence in RL policies.

2 Problem Setup

We consider Markov decision processes (MDPs) defined by a tuple M = ( S , A , P, R, d 1 , γ, T ) , where S and A are the state and action spaces, P : S × A → ∆( S ) and R : S × A → ∆( R ) are the transition and reward functions, d 1 ∈ ∆( S ) is the initial state distribution, γ ∈ [0 , 1] is the discount factor, T ∈ Z + is the fixed horizon. p ( s ′ | s, a ) denotes the probability density function of P , and ¯ R ( s, a ) denotes the expected reward. A policy π : S → ∆( A ) specifies a mapping from each state to a probability distribution over actions. A T -step trajectory following policy π is denoted by τ = [( s t , a t , r t )] T t =1 where s 1 ∼ d 1 , a t ∼ π ( s t ) , r t ∼ R ( s t , a t ) , s t +1 ∼ p ( s t , a t ) . Here, a ∼ π ( s ) is short for a ∼ π ( ·| s ) and s ′ ∼ p ( s, a ) for s ′ ∼ p ( ·| s, a ) . Let J = ∑ T t =1 γ t -1 r t denote the return of the trajectory, which is the discounted sum of rewards. The value of a policy π is the expected return, defined as v ( π ) = E π [ J ] . The value function of policy π , denoted by V π : S → R , maps each state to the expected return starting from that state following policy π . Similarly, the action-value function (i.e., the Q-function), Q π : S × A → R , is defined by further restricting the action taken from the starting state. Formally, V π ( s ) = E π [ J | s 1 = s ] , and Q π ( s, a ) = E π [ J | s 1 = s, a 1 = a ] . We also consider value functions at specific horizons: V π t : T ( s ) = E π [ ∑ T t ′ = t γ t ′ -1 r t ′ | s t = s ] , and Q π t : T ( s, a ) = E π [ ∑ T t ′ = t γ t ′ -1 r t ′ | s t = s, a t = a ] . Throughout the paper we also consider the nonsequential, bandit setting with horizon T = 1 . In this case, a 'trajectory' (or, a sample) is denoted by τ = ( s, a, r ) where we omit the time step subscript.

Our goal is to estimate v ( π e ) , the value of an evaluation policy π e , given data that were previously collected by some behavior policy π b in the same environment defined by M . Let D = { τ ( i ) } N i =1 denote the dataset containing N independent trajectories drawn according to π b and M .

OPE. The typical approach to this problem relies on off-policy evaluation (OPE). Importance sampling (IS) is a common OPE approach that reweights samples based on how likely they are to occur under π e relative to π b . Given a trajectory τ , the 1-step and cumulative IS ratios are defined as ρ t = π e ( a t | s t ) π b ( a t | s t ) and ρ 1: t = ∏ t t ′ =1 ρ t ′ . The per-decision IS estimator, ˆ v PDIS = ∑ T t =1 ρ 1: t γ t -1 r t , is an unbiased estimator of v ( π e ) [27, 28]. We also consider its recursive definition: ˆ v PDIS = v T where v 0 = 0 , v T -t +1 = ρ t ( r t + γv T -t ) . In this paper, we discuss the properties of IS-based estimators over a single trajectory; our results naturally generalize to dataset D containing N trajectories where the final estimator is the average over trajectories. For the bandit setting, we refer to PDIS simply as the IS estimator, ˆ v IS = ρr = π e ( a | s ) π b ( a | s ) r .

Counterfactual Annotations. In addition to the offline dataset D , our semi-offline framework assumes access to accompanying counterfactual annotations. To introduce the notation, we start with the non-sequential, bandit setting where T = 1 , dropping the time step subscripts. Given a factual sample τ = ( s, a, r ) , let c ˜ a ∈ { 0 , 1 } be a binary indicator for whether the counterfactual action ˜ a ∈ A \ { a } is associated with an annotation, and let the annotation be g ˜ a ∈ R . We use G : S × A → ∆( R ) to denote the annotation function such that g ˜ a ∼ G ( s, ˜ a ) . A counterfactualaugmented sample τ + = ( τ, g ) consists of the factual sample τ and counterfactual annotations g = { g ˜ a : c ˜ a = 1 } , where each g ˜ a ∼ G ( s, ˜ a ) . Intuitively, a 'good' annotation should reflect the scenario where the counterfactual action ˜ a is taken and the reward g ˜ a ∼ R ( s, ˜ a ) is observed.

Assumption 1 (Perfect annotation, bandit) . E g ∼ G ( s,a ) [ g ] = ¯ R ( s, a ) , ∀ s ∈ S , a ∈ A .

For the sequential setting, we define the corresponding notation with time step subscripts: for ( s t , a t , r t ) occurring at step t of trajectory τ , we define counterfactual indicators c ˜ a t for ˜ a ∈ A \ { a t } and annotations g t = { g ˜ a t : c ˜ a t = 1 } . Figure 2 provides an example trajectory with counterfactual annotations. Here, each g ˜ a t ∼ G t ( s t , ˜ a ) is drawn from the horizont annotation function G t . While the general notion of counterfactual annotations could be used to capture different information (e.g., the instantaneous reward of the counterfactual action, R ( s, ˜ a ) ), in this work, we study a specific version that allows us to extend the theory of the bandit setting. Specifically, the annotation for counterfactual action ˜ a summarizes the annotator's belief of the expected future return (sum

Figure 2: A trajectory augmented with counterfactual annotations, where the action space is A = { x , y , z } . The factual trajectory τ is shown in black. Solid blue arrows indicate the counterfactual annotations were queried and obtained; dashed gray arrows indicate the annotations are not available. Each transition arrow is labeled with ( action, value ), where value is either an observed immediate reward or a counterfactual annotation.

of rewards) in the remaining T -t +1 steps after taking the counterfactual action ˜ a from state s t , and then following the evaluation policy π e . In other words, the annotation plays the same role as the Q-function. This leads to a more refined assumption on the horizon-specific annotation function G t .

Assumption 2 (Perfect annotation, MDP) . E g ∼ G t ( s,a ) [ g ] = Q π e t : T ( s, a ) , ∀ s ∈ S , a ∈ A .

Under Assumption 2, if we obtained infinitely many annotations for all initial states and all actions, then evaluation becomes trivial (we essentially recover the Q-function of all initial states). However, we consider the non-asymptotic regime where not every annotation is available, as certain annotations might be difficult to obtain. For example, annotating initial states requires reasoning about the full horizon T . Furthermore, since this is a rather strong assumption (we need different annotations for each π e ), later we explore a relaxation where the annotations reflect the behavior policy π b instead.

3 Methods

To motivate our approach, we begin with a didactic bandit example to illustrate how the naive incorporation of counterfactual annotations can yield biased estimates. In order to address this issue, we propose a modification of IS estimators that reweights the factual data and counterfactual annotations. We formally describe how this idea applies to IS (in the bandit setting) and PDIS (in the sequential RL setting), giving rise to a family of semi-offline counterfactual-augmented IS estimators. We study the impact of different assumptions regarding the annotations on the performance of our proposed estimators both theoretically (Section 4) and empirically (Section 5).

3.1 Intuition

Consider a one-step bandit (Figure 3a) with two states { s 1 , s 2 } (drawn with equal probability) and two actions, up ( ↗ ) and down ( ↘ ). The reward from s 1 is +1 and from s 2 is 0 (i.e., rewards do not depend on the action), meaning all policies have an expected reward of 0 . 5 . Suppose the behavior policy always selects ↗ , generating a dataset with poor support for policies that assign nonzero probabilities to ↘ (Figure 3b). Now suppose we also have access to human-provided annotations of counterfactual actions, but not all counterfactual annotations are available (either because they were never queried or the users declined to provide annotations). In our example (Figure 3c), one annotation is collected for the counterfactual action ↘ at state s 1 , indicating that the human annotator believes the reward for taking action ↘ from state s 1 is +1 (which is the true reward). To make use of this information, one might be tempted to add the counterfactual annotation as a new sample. The augmented dataset (Figure 3d) would allow us to evaluate policies (e.g., using IS) that assign non-zero probabilities to ↘ in state s 1 . While seemingly plausible, this naive approach inadvertently changes the state distribution and results in a dataset inconsistent with the original problem (it looks like state s 1 is seen more often than reality). A quick calculation reveals that applying IS to this unweighted augmented dataset gives a biased estimate of a new policy as 2 / 3 instead of 0 . 5 (see Appendix B). To address this issue, in Section 3.2 we propose a new reweighting procedure that maintains the state distribution of the original dataset while incorporating counterfactual annotations.

Figure 3: (a) The state diagram of a bandit problem with two states and two actions. (b) A factual dataset containing two samples. (c) The factual samples augmented with counterfactual annotations. (d) The (unweighted) augmented dataset constructed from factual samples and counterfactual annotations. Compared to the original factual dataset, the relative frequency of s 1 vs s 2 has changed from 1 : 1 to 2 : 1 .

3.2 Augmenting IS Estimators with Counterfactual Annotations

To avoid the bias issue described in Section 3.1, informally, we want to split the contribution of each sample between the factual data and counterfactual annotations. Given a factual sample ( s, a, r ) and the associated counterfactual annotations g , let w = { w a } ∪ { w ˜ a : ˜ a ∈ A \ { a }} be a set of user-defined non-negative weights that satisfy w a + ∑ ˜ a ∈A\{ a } w ˜ a = 1 . These weights specify how much we want the estimator to 'listen' to the counterfactual annotations ( w ˜ a ) relative to the factual data ( w a ). We restrict w ˜ a = 0 when c ˜ a = 0 , i.e., non-zero weight is only allowed when the annotation is available. In general, one may assign different weights for each occurrence of ( s, a ) (e.g., the counterfactual annotation is obtained for one instance but missing for another); let ¯ W (˜ a | s, a ) = E [ w ˜ a ] denote the average weight assigned to ˜ a when the factual data is ( s, a ) (see example in Appendix B.1). After reweighting, the state distribution is maintained (since the weights associated with each sample sum to 1 ) but the state-conditional action distributions have changed; this 'weighted' augmented dataset can be seen as if it was generated using a different behavior policy.

Definition 1 (Augmented behavior policy) .

Here, π b + ( a | s ) represents the probability that information about action a is observed for state s (similar to how an 'average policy' may be defined for multiple behavior policies [29]), either as a factual action in the dataset, or as an annotated counterfactual action when some other action ˇ a is the factual action. Next, we define our proposed estimators (for bandits) based on IS.

Definition 2 (Counterfactual-augmented IS) . Given a counterfactual-augmented sample τ + = ( τ, g ) and weights w = { w ˜ a : ˜ a ∈ A} , where τ = ( s, a, r ) , g = { g ˜ a : c ˜ a = 1 } , the C-IS estimator is ˆ v C-IS = w a ρ a r + ∑ ˜ a ∈A\{ a } w ˜ a ρ ˜ a g ˜ a , where ρ ˜ a = π e (˜ a | s ) π b + (˜ a | s ) for each ˜ a ∈ A .

The C-IS estimator is a weighted convex combination of the factual IS estimate ρ a r and the counterfactual IS estimates ρ ˜ a g ˜ a for all counterfactual actions ˜ a ∈ A \ { a } . We also study a special case where all annotations are available and the weights are split equally among actions, such that w a = w ˜ a = 1 / |A| . Then, π b + becomes the uniformly random policy, and after substituting into Definition 2, we obtain the following estimator.

Definition 3 (C-IS with equal weights) . Given a counterfactual-augmented sample τ + = ( τ, g ) , the C*-IS estimator is ˆ v C*-IS = π e ( a | s ) r + ∑ ˜ a ∈A\{ a } π e (˜ a | s ) g ˜ a .

Remark. Definition 3 provides an alternative interpretation of the estimator when using equal weights: if Assumption 1 holds (i.e., the annotation function G is the true reward function R ), we effectively observe both the factual and counterfactual rewards from R . Then, we can directly use the definition of the value function to calculate the expected reward under π e using the action probabilities π e ( ·| s ) .

For the sequential setting, given a trajectory with T steps, we define the collection of weights over all time steps, w = { w a t t : t = 1 ...T } ∪ { w ˜ a t : ˜ a ∈ A \ { a t } , t = 1 ...T } . The augmented behavior policy π b + is similarly defined (see Definition 1). By extending the recursive definition of PDIS, we obtain the following two estimators (assuming either arbitrary weights or equal weights).

Definition 4 (Counterfactual-augmented PDIS) . Given a counterfactual-augmented trajectory τ + = ( τ, g ) and weights w as defined above, where τ = [( s t , a t , r t )] T t =1 , g = { g ˜ a t : c ˜ a t = 1 } , the C-PDIS estimator is ˆ v C-PDIS = v T , with v T defined recursively as v 0 = 0 , v T -t +1 = w a t t ρ a t t ( r t + γv T -t ) + ∑ ˜ a ∈A\{ a t } w ˜ a t ρ ˜ a t g ˜ a t for t = T... 1 , where ρ ˜ a t = π e (˜ a | s t ) π b + (˜ a | s t ) for each ˜ a ∈ A .

Next, we study the theoretical properties of our proposed estimators, relating their OPE performance (in terms of bias and variance) to assumptions on counterfactual annotations and the offline dataset.

Definition 5 (C-PDIS with equal weights) . Given a counterfactual-augmented trajectory τ + = ( τ, g ) , the C*-PDIS estimator is ˆ v C*-PDIS = v T , with v T defined recursively as v 0 = 0 , v T -t +1 = π e ( a t | s t )( r t + γv T -t ) + ∑ ˜ a ∈A\{ a t } π e (˜ a | s t ) g ˜ a t for t = T... 1 .

4 Theoretical Analyses

We first present results for the bandit setting, where we study and compare the properties of the C-IS estimator with standard IS in terms of bias, variance, and the assumptions required, highlighting scenarios where bias and variance reduction is guaranteed. We then show how these results generalize to C-PDIS in the sequential RL setting. Finally, we discuss practical implications of the theoretical results. Full derivations are in Appendix C.

To begin, we review existing results for IS. Recall the following assumption of common support.

Assumption 3 (Common support) . π e ( a | s ) > 0 → π b ( a | s ) > 0 , ∀ s ∈ S , a ∈ A .

If Assumption 3 holds, IS is unbiased (i.e., E τ [ˆ v IS ] = v ( π e ) ), and its variance is [27]:

where σ R ( s, a ) 2 = V r ∼ R ( s,a ) [ r ] is the variance associated with the reward function R ( s, a ) . The first term reflects the inherent randomness from the state distribution not related to importance sampling. The second term reflects the randomness in the behavior policy, whereas the third term reflects the randomness in rewards; these two terms are affected by the distribution of importance ratios ρ ( a | s ) . When Assumption 3 is not satisfied, the IS estimator is biased [30], where the bias is related to actions with no support: Bias[ˆ v IS ] = E [ˆ v IS ] -v ( π e ) = E s ∼ d 1 [ -∑ a ∈U ( s,π b ) π e ( a | s ) ¯ R ( s, a ) ] , with U ( s, π b ) = { a : π b ( a | s ) = 0 } denoting the set of unsupported actions.

Intuitively, when Assumption 3 does not hold, the C-IS estimator can make use of information from the counterfactual annotations for unsupported actions, thereby reducing bias compared to IS (Section 4.1). For cases when IS is already unbiased, counterfactual annotations play the role of additional data and should help further reduce variance (Section 4.2).

4.1 Bias Analyses for C-IS

To formalize the effect of counterfactual annotations on support, we state the following assumption. Assumption 4 (Common support with annotations) . π e ( a | s ) > 0 → π b + ( a | s ) > 0 , ∀ s ∈ S , a ∈ A .

̸

Assumption 4 is a weaker version of Assumption 3, because π b + ( a | s ) > 0 requires either ¯ W ( a | s, a ) π b ( a | s ) > 0 (same as Assumption 3, assuming ¯ W ( a | s, a ) = 0 ) or ¯ W ( a | s, ˇ a ) π b (ˇ a | s ) > 0 for at least some ˇ a ∈ A\{ a } . In other words, information about action a can be from either a factual sample or counterfactual annotations (recall Definition 1). Next, we state the main results for the bias of C-IS (unless specified otherwise, expectations are taken with respect to E τ + , w ). These results hold for any nonzero w and directly generalize to the special case of C*-IS where the weights are 1 / |A| .

Theorem 1 (Unbiasedness of C-IS) . In the bandit setting, when both Assumptions 1 and 4 hold, the C-IS estimator is unbiased, E [ˆ v C-IS ] = v ( π e ) .

Proposition 2 (Bias of C-IS due to support) . When Assumption 1 holds but Assumption 4 is violated, Bias[ˆ v C-IS ] = E s ∼ d 1 [ -∑ a ∈U ( s,π b + ) π e ( a | s ) ¯ R ( s, a ) ] where U ( s, π b + ) = { a : π b + ( a | s ) = 0 } are unsupported actions in the counterfactual-augmented dataset.

Proposition 3 (Bias of C-IS due to imperfect annotations) . When Assumption 4 holds but Assumption 1 is violated, Bias[ˆ v C-IS ] = E s ∼ d 1 E a ∼ π e ( s ) [ δ W ( s, a ) ϵ G ( s, a ) ] , where we measure violation of Assumption 1 as ϵ G ( s, a ) = E g ∼ G ( s,a ) [ g ] -¯ R ( s, a ) , and δ W ( s, a ) = ( 1 -¯ W ( a | s,a ) π b ( a | s ) π b + ( a | s ) ) .

Proposition 4 (A sufficient condition for bias reduction) . If Assumption 1 holds (but Assumption 4 is violated), ¯ R ( s, a ) ≥ 0 for all s ∈ S , a ∈ A , and there exists ( s, a ) such that π b ( a | s ) = 0 , π b + ( a | s ) > 0 , π e ( a | s ) > 0 , ¯ R ( s, a ) > 0 , then | Bias[ˆ v C-IS ] | < | Bias[ˆ v IS ] | .

There are two sources of bias for C-IS: missing annotations contribute to the bias as the rewards of unsupported actions (Proposition 2), whereas imperfect annotations contribute to the bias as the annotation error over supported actions (Proposition 3). If both assumptions are violated, the resulting bias is the combination of the two (see Appendix C). If both assumptions hold, C-IS is unbiased (Theorem 1). Even when not all counterfactual annotations are collected (Assumption 4 is violated), C-IS can evaluate more policies without bias (assuming perfect annotations), because there is a larger space of policies 'supported' by the counterfactual-augmented dataset. In particular, if there is at least one counterfactual annotation for an action with no support in the factual data, C-IS has less bias than IS (under mild conditions, Proposition 4). Lastly, we note the a useful corollary of Theorem 1.

Corollary 5 (Expectation of augmented importance ratios) . Let ρ + W = w a ρ a + ∑ ˜ a ∈A\{ a } w ˜ a ρ ˜ a given τ and w . Under Assumption 4, E [ ρ + W ] = 1 .

Remark. Corollary 5 suggests that for each sample, ρ + W plays a similar role as the standard importance ratio ρ in IS, which may be used for calculating the effective sample size (ESS) [31]. Naturally, we can also create a weighted version of our proposed estimators (e.g., C-WIS), with the normalization factor defined using ρ + W .

4.2 Variance Analyses for C-IS

Compared to the bias analyses above, the variance of C-IS has a more involved dependence on weights w as well as the variance of the annotation function, σ G ( s, a ) 2 = V g ∼ G ( s,a ) [ g ] . For clarity, we defer the full derivations to Appendix C.3; here, we present results for C*-IS where the weights are all set to 1 / |A| and the annotation function has the same variance as the reward function.

Theorem 6 (Variance of C*-IS) . Assuming σ G ( s, a ) 2 = σ R ( s, a ) 2 , under Assumptions 1 and 4,

Proposition 7 (Variance Reduction of C*-IS) . Under the premise of Theorem 6, V [ˆ v C*-IS ] ≤ V [ˆ v IS ] .

where ρ ( a | s ) = π e ( a | s ) π b ( a | s ) is the importance ratio under the original behavior policy.

Comparing Eqn. (2) with the three terms of the variance decomposition of IS in Eqn. (1), we note that the first term V s ∼ d 1 [ V π e ( s )] is identical, the dropped second term (of Eqn. (1)) is a non-negative variance term, and the third term is scaled by a factor π b ( a | s ) ≤ 1 (for each instantiation of the expression inside the expectation), leading to a guaranteed variance reduction (Proposition 7). We derive the full variance decomposition for C-IS in Theorem 13. Unlike C*-IS, variance reduction is not guaranteed for C-IS. This is due to additional non-negative terms that depend on the variance/covariance of weights w and terms that depend on the difference in variance between annotations and rewards σ G ( s, a ) 2 -σ R ( s, a ) 2 (could be positive or negative); these terms all vanish to zero in the case of C*-IS where weights are constant ( 1 / |A| ) and σ G ( s, a ) 2 = σ R ( s, a ) 2 .

4.3 Extensions to C-PDIS

We note that the corresponding results in the bandit setting can be derived for the MDP setting using an induction-style proof, with similar interpretations of the factors that contribute to reduced bias and reduced variance when compared to standard PDIS. Below, we briefly demonstrate how the unbiasedness result (Theorem 1) extends to the MDP setting. We further explore the sequential RL setting in the empirical experiments.

Theorem 8 (Unbiasedness of C-PDIS) . In the MDP setting, when both Assumptions 2 and 4 hold, the C-PDIS estimator is unbiased, E [ˆ v C-PDIS ] = v ( π e ) .

Proof Sketch. Here, we explain the intuition behind the proof for C*-PDIS, which proceeds via a backward induction (Figure 4). In the recursive definition (Definition 4), we aim to show that every v T -t +1 is an unbiased estimator of the horizont value function. At each horizon t , we can view the problem as a one-step bandit problem (reducing the estimator to C*-IS), where the factual action leads to a factual trajectory and the counterfactual action(s) leads to the counterfactual annotation(s), both of which are used to construct unbiased estimates of horizont Q-values (for a t and ˜ a , respectively). In the end, the estimates from the two branches are combined according to π e , resulting in the correct expectation of the horizont value of state s . See full proof for C-PDIS in Appendix C.4.

ΓαΓLΓLΓβ

ΓαΓLΓβ

ΓαΓLΓLΓLΓβ

ΓαΓLΓYΓβ

4.4 Practical Implications

So far, we have been focusing on the theoretical framework for incorporating counterfactual annotations into OPE. However, the actual implementation of this approach poses several practical challenges. We believe this underscores the fact that this is a rich area for research with many potential directions. In this section, we address several real-world scenarios that do not adhere to the theoretical assumptions - specifically, when annotations are biased, noisy, or missing.

Correcting annotation bias. Comparing Assumptions 1 and 2, we note an important distinction between the bandit setting and the sequential RL setting. For bandits, we simply want G to mimic the reward model R . In contrast, for the RL setting, G t should ideally mimic the Q-function of the evaluation policy Q π e t : T . Such annotations can be difficult if not impossible to obtain in practice (e.g., for healthcare, it would require asking clinicians to reason about a sequence of counterfactual actions and predict the outcome). Thus, we additionally consider a relaxation of Assumption 2 where instead E g ∼ G t ( s,a ) [ g ] = Q π b t : T reflecting expected return under the behavior policy (which is more likely in practice). This annotation bias ϵ G = Q π b t : T -Q π e t : T results in a biased estimator (Proposition 3). To correct this bias, we suggest first estimating the annotation bias function ˆ ϵ G ( s, a ) using an approximate MDP built from offline data, and then mapping each annotation as ˆ g ˜ a = g ˜ a -ˆ ϵ G ( s, ˜ a ) (see Appendix D.1 for details). In Section 5.2, we empirically measure the impact of this alternative assumption and the bias-correction procedure (which we denote as Q π b ↦→ ˆ Q π e ) on OPE performance.

Reweighting noisy annotations. So long as Assumption 1 or 2 is satisfied, the noise (i.e., variance) of the annotations does not affect unbiasedness; however, as shown in Theorem 13, the variance of our proposed estimators directly depends on how noisy the annotations are. Intuitively, if the annotation variance is smaller than the reward variance, we want the final estimate to 'listen' more to the annotations and less to the factual data (and vice versa). We empirically explore the impact of annotation noise in both Section 5 and Appendix E, noting that while using equal weights (as in C*-IS) outperforms standard IS, adjusting weights based on the relative magnitudes of σ 2 R and σ 2 G can further improve OPE performance.

Imputing missing annotations. For many real-world domains, it is unlikely that we can obtain an annotation for every counterfactual action at every time step (the total number of annotations needed is ( |A| 1) NT ). While desirable to use equal weights as in C*-IS (and C*-PDIS) due to its variance reduction guarantee, this is not possible if some annotations are missing (in which case the factual data must have w a = 1 ), and this can actually lead to higher variance (see Appendix D.2 for an example). To alleviate this variance increase, we suggest estimating an annotation model ˆ G from available annotations and using it to impute the missing annotations (see Appendix D.2). Although ˆ G may be a biased estimator of G (due to annotation noise) and introduce additional bias to the final estimate, we empirically observe a favorable bias-variance trade-off (Section 5.2).

Figure 4: Illustration of proof idea. (i) The factual estimate v T -t provides an unbiased estimate of the horizon-( t +1) value of state s t +1 . (ii) When combined with the factual reward r t , we obtain an unbiased estimate of the horizont Q-value of ( s t , a t ) . (iii) By assumption, the counterfactual annotations provide unbiased estimates of the horizont Q-value of ( s t , ˜ a ) . (iv) The factual and counterfactual estimates are combined using π e . For clarity, we omit details about state distributions here (see appendix).

In general, one would expect more counterfactual annotations to reduce both bias and variance; at the same time, these annotations may be imperfect (biased or noisy) and directly increase bias or variance. Our analyses in Section 4 and Appendix C show that both bias and variance depend on weights w , suggesting the weighting scheme as a key mechanism to achieve optimal bias and variance. In Appendix D.3 we explore an analytical approach for solving the variance-minimizing weighting scheme and note the solution is highly non-trivial. We empirically explore the impact of different weights in Appendix E.1 and note that using equal weights (as in C*-IS) is a promising heuristic, since it achieves good performance in most settings. We believe that optimizing the weights can further improve OPE performance and is an interesting direction for future work.

5 Experiments

First, through a suite of simple bandit problems, we verify the theoretical properties of C-IS. Then, we apply our approach to a healthcare-inspired RL simulation domain, where we compare the performance of our proposed approach, C-PDIS, to several baselines in terms of their OPE accuracy and ability to rank policies, and explore robustness to bias, noise, and missingness in the annotations.

5.1 Synthetic Domains - Bandits

We consider a class of bandit problems with two states { s 1 , s 2 } (drawn with equal probability), two actions A = {↗ , ↘} (recall Figure 3), and corresponding reward distributions R ( s i , a ) ∼ N ( ¯ R ( s i ,a ) , σ 2 ) . Without loss of generality, we assume ↗ is always taken from s 2 by both π b and π e . For s 1 , we consider deterministic policies in which one action is always taken, and a stochastic policy that takes the two actions with equal probability (see column/row header in Table 1). Given ( π b , π e ) , we draw 1 , 000 samples following π b and then evaluate π e using various estimators, including standard IS, the naive baseline of adding counterfactual annotations as new samples (Section 3.1), and C*-IS. We assume that counterfactual annotations are only available for s 1 , and all annotations are drawn from the true reward function. We measure the bias, standard deviation (std, the square root of variance), and root mean-squared error (RMSE) of the estimators with respect to v ( π e ) .

Naive baseline fails due to bias, whereas C*-IS can reduce bias and/or variance compared to IS. In Table 1, we display the results for ¯ R ( s 1 , ↗ ) = 1 , ¯ R ( s 1 , ↘ ) = 2 , ¯ R ( s 2 , · ) = 1 , σ = 0 . 5 (other settings in Appendix E.1). The naive baseline fails to improve upon IS (and often underperforms IS in terms of RMSE) and can have a nonzero bias even when IS is unbiased (Table 1, row 2, column 2). C*-IS achieves a lower RMSE than IS across all settings considered. The benefits of C*-IS align with our theoretical analyses. (i) Bias Reduction for Support-Deficient Data. When π b is deterministic (first row), the unselected action has poor support and IS has a nonzero bias whereas C*-IS is unbiased. Though C*-IS sometimes has a larger variance than IS, the bias reduction outweighs the variance increase and leads to overall lower RMSE. (ii) Variance Reduction for WellSupported Data. In the second row, data generated by π b has full support. Both IS and C*-IS are unbiased, but C*-IS leverages counterfactual annotations to achieve lower variance and lower RMSE.

Table 1: Summary of performance on the bandit problem for various π b (rows) and π e (columns), where each policy is denoted by its probabilities assigned to the two actions from s 1 . Each cell of the table corresponds to a ( π b , π e ) combination, for which we report (bias, std, RMSE) for three estimators: IS in the top row, naive in the middle row, and C*-IS in the bottom row. Settings with π b ( s 1 ) = [0 , 1] are omitted due to symmetry.

π

b

π

e

[ 1

,

0 ]

[ 0

,

1 ]

[0

.

5

,

0

.

5]

[ 1

,

0 ]

-

-

1 0

.

6 1

.

2

0

.

2 1

.

8 1

.

8

0

0

.

7 0

.

7

-

0

.

5 0

.

5 0

.

7

0

.

1

0

.

7 0

.

7

0

0

.

5 0

.

5

[0

.

5

,

0

.

5]

0

0

.

9 0

.

9

0

1

1

0

0

.

5 0

.

5

0

1

.

6 1

.

6

0

.

2 1

.

8 1

.

8

0

0

.

7 0

.

7

-

We vary the assumptions (e.g., weights in C-IS, noisy/missing annotations) and present further experiments in Appendix E.1. These results suggest that (i) Equal weights (in C*-IS) is a good heuristic though not always 'optimal' and (ii) Imputing missing annotations can reduce variance .

5.2 Healthcare Domain - Sepsis Simulator

Next, we apply our approach to evaluate policies in a simulated RL domain modeled after the physiology of sepsis patients [32]. Following prior work [22], we collected 50 offline datasets from the sepsis simulator (using different random seeds) each with 1000 episodes by following an ϵ -greedy behavior policy with respect to the optimal policy where ϵ = 0 . 1 . We considered a set of deterministic policies (including the optimal policy) as evaluation policies, which have different performance and

Estimator

↓

RMSE

↑

ESS

↑

Spearman

↑

%Accuracy

↓

%FPR

↓

%FNR

Baseline

PDIS (w/o annot.)

0.113

±

0.038

76.8

±

44.0

0.596

±

0.110

76.5

±

3.5

33.7

±

8.7

15.9

±

4.6

Naive unweighted (

G

=

Q

π

e

)

0.128

±

0.006

207.2

±

91.5

0.089

±

0.089

50.0

±

6.0

11.6

±

8.3

78.1

±

13.6

Naive weighted

(

G

=

Q

π

e

)

0.097

±

0.006

300.8

±

117.6

0.420

±

0.097

64.3

±

4.7

24.0

±

12.7

44.3

±

11.4

Proposed

C*-PDIS (

G

=

Q

π

e

)

0.013

±

0.005

994.0

±

10.1

0.995

±

0.003

95.7

±

3.1

4.5

±

6.9

4.2

±

5.3

}

⋆

ideal case

C*-PDIS (

G

=

Q

π

b

)

0.070

±

0.003

994.0

±

10.1

0.961

±

0.011

86.8

±

8.2

22.0

±

20.1

8.2

±

11.3

}

relaxing

Assump. 2

C*-PDIS (

G

=

Q

π

b

ˆ

Q

π

e

)

0.028

±

0.007

994.0

±

10.1

0.979

±

0.010

90.1

±

5.4

4.2

±

6.6

14.1

±

9.7

0

.

0

0

.

5

1

.

0

0

.

0

0

.

1

0

.

2

RMSE(

v

(

π

e

)

,

ˆ

v

(

π

e

)

)

PDIS (baseline)

C*-PDIS (

G

=

Q

π

b

)

C*-PDIS (

G

=

Q

π

b

ˆ

Q

π

e

)

C*-PDIS (

G

=

Q

π

e

)

C*-PDIS, noisy annot.

PDIS (baseline)

ideal case

C-PDIS , w/o imputation

C*-PDIS, w/ imputation

0.0

0.1

RMSE

0.5

1.0

Spearman

Corr.

0.0

0.5

1.0

Std of annotation noise

0.8

1.0

Bin. Class.

0.0

0.1

RMSE

0.5

1.0

Spearman

Corr.

0.0

0.5

1.0

fraction of annotated samples

0.8

1.0

Bin. Class.

varying degrees of similarity vs behavior. We compared our proposed estimator (with different annotation functions) with a set of baselines, including standard PDIS (without annotations) and two naive baselines (with perfect annotations): 'naive unweighted' simply adds counterfactual annotations as new trajectories and has the same issue discussed in Section 3.1, whereas 'naive weighted' reweights the annotations at the trajectory level instead of per-decision. See Appendix E.2 for detailed experimental setup. As the main OPE metric, we report RMSE of value estimates vs true values as well as the effective sample size (ESS). Additionally, we report metrics for two downstream uses of OPE for model selection. (i) When used to rank policies. We report the Spearman's rank correlation between ˆ v ( π e ) and v ( π e ) (computed over all π e 's) [22, 26]. (ii) When used to determine whether π e is better or worse than π b . We formulate a binary classification problem of v ( π e ) ≥ v ( π b ) vs v ( π e ) < v ( π b ) , and report the accuracy, false positive rate (FPR) and false negative rate (FNR).

C*-PDIS outperforms all baselines across all metrics in the ideal setting. As shown in Table 2, when all counterfactuals are available and annotated with the evaluation policy's Q-function ( G = Q π e ), C*-PDIS outperforms baseline PDIS (without annotations) in all metrics, demonstrating that it provides more accurate OPE estimates. In contrast, the two naive approaches fail to provide accurate estimates and often underperform standard PDIS.

C*-PDIS is robust to biased annotations. Under the more realistic scenario where G = Q π b , i.e., annotations summarize the future returns under π b rather than π e , we observe a degradation in all metrics compared to the ideal case, though C*-PDIS is still superior to PDIS (Table 2). Applying the bias correction procedure ( G = Q π b ↦→ ˆ Q π e , see Appendix D.1) helps in recovering performance closer to the ideal case, especially when π e is far from π b (Figure 5-left).

Variance reduction of C*-PDIS outweighs the effect of noisy annotations. When annotations are perturbed with increasing amounts of noise (Figure 5-center), performance degradation is minimal even at the highest level of noise tested (with a std of 1, which is large relative to the reward range [ -1 , 1] , and larger than 0 . 31 the std of initial state values). Our estimator remains competitive relative to baselines, suggesting that the benefit of variance reduction from additional data (through counterfactual annotations) outweighs the variance increase from annotation noise, even when annotations are much noisier than factual data. See Appendix E.2 for variations of this experiment.

Collecting more annotations and imputing missing annotations improves performance. As the amount of available annotations increases (Figure 5-right), our approach interpolates between baseline PDIS and the ideal case of C*-PDIS with a monotonic improvement in performance. Furthermore, imputing annotations achieves better performance, suggesting it is a promising strategy when not all annotations can be collected in practice. See Appendix E.2 for variations of this experiment where the imputed annotations have varying degrees of bias due to annotation noise.

6 Related Work

There is a rich literature on statistical methods for offline policy evaluation (OPE), including direct methods (DM), importance sampling (IS), and doubly-robust (DR) approaches [24, 26]. DM directly uses offline data to learn the value function of the evaluation policy (e.g., using model-based or model-free approaches) [33, 34]. We do not consider DM in our work since it often involves function approximators whose bias and variance may be difficult to analyze [35]. On the other hand, IS uses a weighted average of trajectory returns to correct the distributional mismatch between the evaluation and behaviour policies [28, 27]. Our proposed estimator directly builds on IS and uses counterfactual annotations to reduce its variance (and bias), while carefully addressing the nuances involved with reweighting the counterfactual annotations to maintain the unbiasedness property. Finally, DR approaches combine IS and DM and reduce the variance of IS by using the estimates from DM as a control variate [27, 36-38]. Our approach is a complementary source of variance reduction and may be combined with DR approaches by modifying our current definitions to include a control variate. We note that other approaches exist for managing the variance of IS in long-horizon settings by considering the stationary or marginalized state distributions [39-41]. Incorporating counterfactual annotations into these estimators is an interesting direction of future research.

Our work focuses on OPE rather than policy learning, but the broader theme of human input for RL has recently gained renewed attention, particularly in natural language processing tasks [42-44]. In many of these problems, human input is in the form of preferences (or rankings) over actions, states, or (sub-)trajectories [45]. Other related annotation approaches also exist in non-RL areas, where past work has proposed to ask annotators to alter text to match a counterfactual target label [46] and incorporating annotations of potential unmeasured confounders [47]. In contrast, our work investigates the role of a specific form of human input, counterfactual annotations in offline RL, in improving OPE performance. We focus on offline evaluation due to its practical importance in high-stakes RL domains such as healthcare, though our insights could also potentially benefit offline learning in these domains. While we did not discuss how counterfactual annotations are obtained, our theoretical analysis establishes a thorough understanding of their desirable characteristics. This can help motivate methods for converting different forms of human feedback into useful counterfactual annotations (e.g., learning reward/annotation models from human preferences [42]). Conversely, progress in the field of preference learning and learning-to-rank may benefit our approach by providing mechanisms to solicit high-quality annotations [48].

7 Discussion & Conclusion

In this paper, we propose a novel semi-offline policy evaluation framework that incorporates counterfactual annotations into traditional IS estimators. We emphasize that the naive approach of viewing annotations as additional data can lead to bias and propose a simple reweighting scheme to address this issue. We formally study the theoretical properties of our approach, identifying scenarios where bias and variance reduction is guaranteed. Driven by a deep understanding of these theoretical properties, we further propose practically motivated strategies to handle biased, noisy, or missing annotations. Through proof-of-concept experiments on bandits and a healthcare-inspired RL simulator, we demonstrate that our approach outperforms standard IS and is robust to imperfect annotations. Our semi-offline framework serves as an intermediate step between offline and online evaluations and has the potential to enable practical applications of RL in high-stakes domains. Though motivated by current limitations of offline evaluation, we caution that our approach is not meant to replace existing OPE methods, but rather to complement them. Collecting annotations from domain experts comes at a cost (of real human time and labor), and thus, our approach should only be applied after a policy has passed all checks on retrospective data. While not explored in this paper, future work should focus on assessing the quality of counterfactual annotations in the domains of interest through human experiments. See Appendix A for more detailed discussions on limitations, societal impacts, and future directions. Overall, we believe our contributions will inspire further investigations into the practical obstacles that emerge in semi-offline evaluation (e.g., devising human-centered strategies for soliciting counterfactual annotations that align with theoretical assumptions) and will bring RL closer to reality in healthcare and other high-stakes decision-making domains.

Acknowledgments

This work was supported by the National Library of Medicine of the National Institutes of Health (grant R01LM013325 to JW). The views and conclusions in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the National Institutes of Health. The authors would like to thank Michael Sjoding and members of the MLD3 group for helpful discussions regarding this work, as well as the anonymous reviewers for constructive feedback.

Data and Code Availability

The code for all experiments is available at https://github.com/MLD3/CounterfactualAnnot-SemiOPE.

References

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing Atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature , 518(7540):529-533, 2015. URL https://doi.org/10.1038/nature14236.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature , 529(7587):484-489, 2016. URL https://doi.org/10.1038/nature16961.

Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving Rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113 , 2019.

Nevena Lazic, Craig Boutilier, Tyler Lu, Eehern Wong, Binz Roy, MK Ryu, and Greg Imwalle. Data center cooling using model-predictive control. Advances in Neural Information Processing Systems , 31, 2018.

Jeff Wu, Long Ouyang, Daniel M Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively summarizing books with human feedback. arXiv preprint arXiv:2109.10862 , 2021.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Gray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=TG8KACxEON.

Chao Yu, Jiming Liu, Shamim Nemati, and Guosheng Yin. Reinforcement learning in healthcare: A survey. ACM Computing Surveys (CSUR) , 55(1):1-36, 2021.

Katyaini Lakkaraju. Human-Centric Machine Learning: Enabling Machine Learning for High-Stakes Decision-Making . PhD thesis, Stanford University, 2018.

Shayan Doroudi, Vincent Aleven, and Emma Brunskill. Where's the reward? A review of reinforcement learning for instructional sequencing. International Journal of Artificial Intelligence in Education , 29: 568-620, 2019.

Shengpu Tang, Aditya Modi, Michael Sjoding, and Jenna Wiens. Clinician-in-the-loop decision making: Reinforcement learning with near-optimal set-valued policies. In International Conference on Machine Learning , pages 9387-9396, 2020. URL http://proceedings.mlr.press/v119/tang20c.html.

Shengpu Tang, Maggie Makar, Michael Sjoding, Finale Doshi-Velez, and Jenna Wiens. Leveraging factored action spaces for efficient offline reinforcement learning in healthcare. In Advances in Neural Information Processing Systems , pages 34272-34286, 2022. URL https://openreview.net/forum?id=Jd70afzIvJ4.

Christopher M Bishop and Nasser M Nasrabadi. Pattern recognition and machine learning , volume 4. Springer, 2006.

Lihong Li, Wei Chu, John Langford, and Robert E Schapire. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th International Conference on World Wide Web , pages 661-670, 2010.

Dmitry Kalashnikov, Alex Irpan, Peter Pastor, Julian Ibarz, Alexander Herzog, Eric Jang, Deirdre Quillen, Ethan Holly, Mrinal Kalakrishnan, Vincent Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning , pages 651-673, 2018.

BMAdams, HT Banks, Hee-Dae Kwon, and Hien T Tran. Dynamic multidrug therapies for HIV: Optimal and STI control approaches. Mathematical Biosciences & Engineering , 1(2):223-241, 2004.

Chiara Dalla Man, Francesco Micheletto, Dayu Lv, Marc Breton, Boris Kovatchev, and Claudio Cobelli. The UVA/PADOVA type 1 diabetes simulator: New features. Journal of diabetes science and technology , 8(1):26-34, 2014.

Jinyu Xie. Simglucose (v0.2.1), 2018. Available: https://github.com/jxx123/simglucose. Accessed on 2023-01-01.

Ian Fox, Joyce Lee, Rodica Pop-Busui, and Jenna Wiens. Deep reinforcement learning for closed-loop blood glucose control. In Machine Learning for Healthcare Conference , pages 508-536. PMLR, 2020.

Omer Gottesman, Fredrik Johansson, Joshua Meier, Jack Dent, Donghun Lee, Srivatsan Srinivasan, Linying Zhang, Yi Ding, David Wihl, Xuefeng Peng, et al. Evaluating reinforcement learning algorithms in observational health settings. arXiv preprint arXiv:1805.12298 , 2018.

Omer Gottesman, Fredrik Johansson, Matthieu Komorowski, Aldo Faisal, David Sontag, Finale DoshiVelez, and Leo Anthony Celi. Guidelines for reinforcement learning in healthcare. Nature medicine , 25(1): 16-18, 2019.

Shengpu Tang and Jenna Wiens. Model selection for offline reinforcement learning: Practical considerations for healthcare settings. In Machine Learning for Healthcare Conference , pages 2-35, 2021. URL https://proceedings.mlr.press/v149/tang21a.html.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Cameron Voloshin, Hoang Minh Le, Nan Jiang, and Yisong Yue. Empirical study of off-policy policy evaluation for reinforcement learning. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks , 2021. URL https://openreview.net/forum?id=IsK8iKbL-I.

Sonali Parbhoo, Shalmali Joshi, and Finale Doshi-Velez. Generalizing off-policy evaluation from a causal perspective for sequential decision-making. arXiv preprint arXiv:2201.08262 , 2022.

Tom Le Paine, Cosmin Paduraru, Andrea Michi, Caglar Gulcehre, Konrad Zolna, Alexander Novikov, Ziyu Wang, and Nando de Freitas. Hyperparameter selection for offline reinforcement learning. arXiv preprint arXiv:2007.09055 , 2020.

Miroslav Dudík, Dumitru Erhan, John Langford, and Lihong Li. Doubly robust policy evaluation and optimization. Statistical Science , pages 485-511, 2014.

Doina Precup, Richard S Sutton, and Satinder P Singh. Eligibility traces for off-policy policy evaluation. In Proceedings of the Seventeenth International Conference on Machine Learning , pages 759-766, 2000.

Aman Agarwal, Soumya Basu, Tobias Schnabel, and Thorsten Joachims. Effective evaluation using logged bandit feedback from multiple loggers. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , page 687-696, 2017.

Noveen Sachdeva, Yi Su, and Thorsten Joachims. Off-policy bandits with deficient support. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 965-975, 2020.

Víctor Elvira, Luca Martino, and Christian P Robert. Rethinking the effective sample size. International Statistical Review , 90(3):525-550, 2022.

Michael Oberst and David Sontag. Counterfactual off-policy evaluation with gumbel-max structural causal models. In International Conference on Machine Learning , pages 4881-4890. PMLR, 2019.

Cosmin Paduraru. Off-policy Evaluation in Markov Decision Processes . PhD thesis, McGill University, 2012.

Hoang Le, Cameron Voloshin, and Yisong Yue. Batch policy learning under constraints. In International Conference on Machine Learning , pages 3703-3712. PMLR, 2019.

Sonali Parbhoo, Omer Gottesman, and Finale Doshi-Velez. Shaping control variates for off-policy evaluation. In Offline Reinforcement Learning Workshop at Neural Information Processing Systems (NeurIPS) , 2020.

Philip Thomas and Emma Brunskill. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning , pages 2139-2148. PMLR, 2016.

Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning , pages 652-661. PMLR, 2016.

Mehrdad Farajtabar, Yinlam Chow, and Mohammad Ghavamzadeh. More robust doubly robust off-policy evaluation. In International Conference on Machine Learning , pages 1447-1456. PMLR, 2018.

Qiang Liu, Lihong Li, Ziyang Tang, and Dengyong Zhou. Breaking the curse of horizon: Infinite-horizon off-policy estimation. In Advances in Neural Information Processing Systems , 2018.

Tengyang Xie, Yifei Ma, and Yu-Xiang Wang. Towards optimal off-policy evaluation for reinforcement learning with marginalized importance sampling. In Advances in Neural Information Processing Systems , 2019.

Ofir Nachum, Yinlam Chow, Bo Dai, and Lihong Li. DualDICE: Behavior-agnostic estimation of discounted stationary distribution corrections. In Advances in Neural Information Processing Systems , 2019.

Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning to summarize with human feedback. Advances in Neural Information Processing Systems , 33:3008-3021, 2020.

Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id= 8aHzds2uUyB.

Qisen Yang, Shenzhi Wang, Matthieu Gaetan Lin, Shiji Song, and Gao Huang. Boosting offline reinforcement learning with action preference query. In International Conference on Machine Learning , 2023.

Christian Wirth, Riad Akrour, Gerhard Neumann, Johannes Fürnkranz, et al. A survey of preference-based reinforcement learning methods. Journal of Machine Learning Research , 18(136):1-46, 2017.

Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. Learning the difference that makes a difference with counterfactually-augmented data. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=Sklgs0NFvr.

Megha Srivastava, Tatsunori Hashimoto, and Percy Liang. Robustness to spurious correlations via human annotations. In International Conference on Machine Learning , pages 9109-9119. PMLR, 2020.

Banghua Zhu, Jiantao Jiao, and Michael Jordan. Principled reinforcement learning with human feedback from pairwise or k -wise comparisons. In ICLR 2023 Workshop on Mathematical and Empirical Understanding of Foundation Models , 2023.

A Further Discussions

Limitations. This work proposes and theoretically studies a new framework of semi-offline evaluation of RL policies. While our core idea is to incorporate human annotations into the evaluation process, we did not make use of real human annotations in our experiments and relied on simulations instead. Given the costs associated with user studies, we opted not to conduct user studies before we have a thorough understanding of when and why our approach works (or does not work) and how it can be implemented in practice. Our present paper focuses on describing and analyzing a formal mathematical framework of counterfactual annotations, which provides valuable insights as to what types of annotations are useful, how they should be incorporated into OPE, and which factors impact performance (see method description in Section 3 and theoretical analyses in Section 4). Our experiments further demonstrate the robustness of our approach when we deviate from the ideal setting (see Section 5). Despite best efforts, our experiments do not capture all possible scenarios in the real world. We list several important practical implications in Section 4.4 and encourage future research into these directions. Additional areas of interest that we did not explore include: alternative forms of annotations (e.g., preference, immediate reward), whether these other annotation types could be converted to match our assumptions so that our framework still applies, how best to design the question phrasing for the annotators, evaluating annotation quality, targeted annotation solicitation and optimizing annotation solicitation under budget constraint.

Ethical Considerations and Societal Impact. Since our experiments only involved simulations and no real humans, we did not pose immediate ethical concerns or safety threats. Nonetheless, in high-stakes decision-making tasks such as healthcare, computationally-derived RL policies must be carefully validated before their final adoption. While we are motivated by the current limitations of standard offline evaluation methods, we caution that our approach is not meant to replace, but rather to complement, existing OPE approaches. Collecting these annotations from real human domain experts comes at a cost, and thus we recommend applying our approach only when a policy has passed all checks on retrospective data. We note that compared to traditional qualitative evaluations (e.g., typically done by asking domain experts whether RL recommendations make sense), our approach is less likely (though still possible) to affect the existing decision making or suffer from confirmation bias, since we do not need to reveal the evaluation policy at the annotation collection stage. Future work that adopts our framework should carefully design the annotation solicitation process (including when and how the question is posed to the annotators) so as to achieve safe, non-disruptive evaluations of offline RL policies before their prospective use. Additionally, more work is needed to understand the extent to which humans can provide accurate annotations of counterfactuals. These human experiments should be addressed in the context of specific application domains and the target groups of expert human annotators.

Future Directions on Counterfactual Annotations. Our present paper establishes important theoretical groundwork for using counterfactual annotations in semi-offline policy evaluation. However, we did not collect any real annotations with human annotators as those are outside of the scope of the main research question we seek to address in our current paper. To fully realize the impact of this work, real-world human experiments would be the necessary next step, with a focus on obtaining faithful annotations (with small bias and small noise). More specifically, it would be important to empirically measure if and how various factors influence annotation quality, including: horizon t (annotating beginning of an episode or the terminal step), the inherent stochasticity associated with the annotated state-action pair and how often they appear in data (under π b ). Equipped with this knowledge, it is then important to select which annotations to prioritize given a limited annotation budget. Many of these questions delve into the realm of HCI and are outside the scope of this paper's methodological contributions for offline RL and OPE, and we encourage researchers and practitioners in different research areas (e.g., RL, HCI, healthcare, education) to build upon the ideas in our work.

B Toy Example for Intuition

(a) State diagram

Recall the example discussed in Section 3.1 (figure reproduced here in Figure 6). The bandit problem has two states { s 1 , s 2 } (drawn with equal probability) and two actions, up ( ↗ ) and down ( ↘ ). The reward from s 1 is +1 and from s 2 is 0 (i.e., rewards do not depend on the action), meaning all policies have an expected reward of 0 . 5 . Suppose the behavior policy always selects ↗ , generating a dataset with poor support for policies that assign nonzero probabilities to ↘ (Figure 6b). Now suppose we also have access to human-provided annotations of counterfactual actions, but not all counterfactual annotations are available (either because they were never queried or the users declined to provide annotations). In our example (Figure 6c), one annotation is collected for the counterfactual action ↘ at state s 1 , indicating that the human annotator believes the reward for taking action ↘ from state s 1 is +1 . To make use of this information, one might consider adding the counterfactual annotation as a new sample. The augmented dataset (Figure 6d) would allow us to evaluate policies (e.g., using IS) that assign non-zero probabilities to ↘ in state s 1 . Unfortunately, this naive approach leads to biased results, which we walk through in detail below.

Consider an evaluation policy that takes ↘ in s 1 and ↗ in s 2 , i.e., π e ( s 1 ) = [0 , 1] and π e ( s 2 ) = [1 , 0] (denoted in terms of the probabilities assigned to the two actions, ↗ and ↘ ).

If we use the original π b in the factual dataset (Figure 6b) where π b ( s 1 ) = [1 , 0] , π b ( s 2 ) = [1 , 0] , the IS estimator is ill-defined because we will encounter a divide-by-zero error:

Instead, one may consider the behavior policy in the augmented dataset (Figure 6d), which gives π b ( s 1 ) = [0 . 5 , 0 . 5] , π b ( s 2 ) = [1 , 0] . The IS estimate is:

However, as stated above, all policies should have a value of 0 . 5 , meaning that 2 3 is a biased estimate. The core of the issue is because directly adding counterfactual annotations inadvertently changes the state distribution and results in a dataset inconsistent with the original problem. In particular, comparing Figure 6b vs 6d, the relative frequency of s 1 vs s 2 has changed from 1 : 1 to 2 : 1 .

Our proposed estimator addresses this issue by reweighting the factual data and counterfactual annotations in order to maintain the state distribution. Suppose we assign a weight α to ( s 1 , ↗ ) and 1 -α to ( s 1 , ↘ ) for some α ∈ (0 , 1) , i.e., the two non-negative weights associated with s 1 sum to 1 . The sample ( s 2 , ↗ ) receives a weight of 1 by default since it does not have an associated counterfactual annotation. In this way, the state distribution remains equally split between s 1 and s 2 .

Using Definition 1, we can calculate the augmented behavior policy π b + ( s 1 ) = [ α, 1 -α ] , π b + ( s 2 ) = [1 , 0] . Applying C-IS as defined in Definition 3, we see our proposed approach produces the correct, unbiased estimate of 0 . 5 :

B.1 Another Example Illustrating the Weighting Scheme

Suppose similar to Figure 6, we now have a bandit problem with only a single state { s } such that all policies have an expected reward of 1 . Suppose the behavior policy always selects ↗ , generating a dataset containing two samples: sample #1 ( s, ↗ ) with an annotation for ↘ , sample #2 is simply ( s, ↗ ) with no annotation for ↘ . In applying our approach, for sample #1 we can assign weights [ α, 1 -α ] for some α ∈ (0 , 1) , i.e., the two non-negative weights associated with sample #1 sum to 1 ; for sample #2 the weights are simply [1 , 0] . Then, the average weights for state s are ¯ W ( ↗ | s, ↗ ) = 1+ α 2 and ¯ W ( ↘ | s, ↗ ) = 1 -α 2 . For example, if α = 0 . 8 , then the weights for sample #1 are [0 . 8 , 0 . 2] and the average weights associated with state s are [0 . 9 , 0 . 1] . Note that while the average weights ¯ W (used for calculating π b + and C-IS) are state-specific , the user-assigned weights w ˜ a are sample-specific ; furthermore, the sample-specific weights should not be interpreted as 'random', as a user may deliberately set weights to split equally into [0 . 5 , 0 . 5] , or to [1 , 0] which ignores the annotation if they believe it is of poor quality.

Using Definition 1, we can calculate the augmented behavior policy π b + ( s ) = [ 1+ α 2 , 1 -α 2 ] . Applying C-IS as defined in Definition 2, we see our proposed approach produces an unbiased estimate of 1 :

C Extended Theoretical Analyses

Unless otherwise stated, the estimators are for π e , i.e., ˆ v = ˆ v ( π e ) .

C.1 IS: Bias & Variance

We formally state and prove the bias and variance results for IS (informally described in Section 4). The proofs are adapted from existing literature [27, 30].

Theorem 9 (Unbiasedness of IS) . In the bandit setting, when Assumption 3 holds, E [ˆ v IS ] = v ( π e ) . Proposition 10 (Bias of IS) . In the bandit setting, when Assumption 3 is violated, Bias[ˆ v IS ] = E s ∼ d 1 [ -∑ a ∈U ( s,π b ) π e ( a | s ) ¯ R ( s, a ) ] where U ( s, π b ) = { a : π b ( a | s ) = 0 } are unsupported actions in the dataset.

Proposition 11 (Variance of IS) . In the bandit setting, when Assumption 3 holds, the variance of IS can be written as:

where σ R ( s, a ) 2 = V r ∼ R ( s,a ) [ r ] is the variance associated with the reward function R ( s, a ) .

Derivation for Bias of IS (adapted from [30]).

If Assumption 3 holds, π e ( a | s ) = 0 for a ∈ U ( s, π b ) and thus Bias[ˆ v ] = 0

Derivation for Variance of IS (adapted from [27]). We apply the law of total variance:

C.2 C-IS: Bias Analyses

We start by showing unbiasedness of C-IS in the ideal case (Theorem 1) .

Theorem 1 (Unbiasedness of C-IS) . In the bandit setting, when both Assumptions 1 and 4 hold, the C-IS estimator is unbiased, E [ˆ v C-IS ] = v ( π e ) .

Proof of Theorem 1. Starting with Definition 2,

where in (1) we replace E g ∼ G ( s, ˜ a ) [ g ] with ¯ R ( s, ˜ a ) following Assumption 1 and combine it with E r ∼ R ( s,a ) [ r ] in a single summation over ˜ a ∈ A , in (2) we swap the order of summations, in (3) we take out common factors that do not depend on a , and in (4) we use Definition 1 for π b + (˜ a | s ) , which cancels out the denominator in the importance ratio on the next line.

Next, we look at two factors contributing to the bias of C-IS: lack of support and imperfect annotations. Proposition 2 (Bias of C-IS due to support) . When Assumption 1 holds but Assumption 4 is violated, Bias[ˆ v C-IS ] = E s ∼ d 1 [ -∑ a ∈U ( s,π b + ) π e ( a | s ) ¯ R ( s, a ) ] where U ( s, π b + ) = { a : π b + ( a | s ) = 0 } are unsupported actions in the counterfactual-augmented dataset.

Proposition 3 (Bias of C-IS due to imperfect annotations) . When Assumption 4 holds but Assumption 1 is violated, Bias[ˆ v C-IS ] = E s ∼ d 1 E a ∼ π e ( s ) [ δ W ( s, a ) ϵ G ( s, a ) ] , where we measure violation of Assumption 1 as ϵ G ( s, a ) = E g ∼ G ( s,a ) [ g ] -¯ R ( s, a ) , and δ W ( s, a ) = ( 1 -¯ W ( a | s,a ) π b ( a | s ) π b + ( a | s ) ) .

Remark. There are two main sources of bias for C-IS, resulting from each of the two assumptions being violated. In Proposition 12, missing annotations (i.e., violation of Assumption 4) contribute to the bias through the first term as the rewards of unsupported actions, whereas imperfect annotations (i.e., violation of Assumption 1) contribute to the bias through the second term as the annotation error over supported actions. When both assumptions hold, C-IS only requires a weaker version of the common support assumption to remain unbiased (Theorem 1); consequently, compared to IS, there is a larger space of policies that C-IS can evaluate without bias. Note that the unbiasedness property is not affected by the user-defined weighting scheme (except that the weights must not be to 0 or 1) and directly applies to C*-IS. When only Assumption 1 holds (Proposition 2), the bias is related to the negative rewards over unsupported actions. On the other hand, when only Assumption 4 holds (Proposition 3), the annotation error ϵ G contributes to the resulting bias and this contribution is scaled by a factor of δ W ( s, a ) ≤ 1 when the estimator can make use of (unbiased) factual samples of the state-action pair ( s, a ) (i.e., when π b ( a | s ) > 0 and ¯ W ( a | s, a ) > 0 ).

Proposition 12 (Bias of C-IS, combined) . When both Assumptions 1 and 4 are violated, Bias[ˆ v C-IS ] = E s ∼ d 1 [ -∑ a ∈U ( s,π b + ) π e ( a | s ) ¯ R ( s, a ) ] + E s ∼ d 1 [∑ a ∈A\U ( s,π b ) δ W ( s, a ) π e ( a | s ) ϵ G ( s, a ) + ∑ a ∈U ( s,π b ) \U ( s,π b + ) π e ( a | s ) ϵ G ( s, a ) ] , where U ( s, π b + ) = { a : π b + ( a | s ) = 0 } are unsupported actions in the counterfactual-augmented dataset, ϵ G ( s, a ) = E g ∼ G ( s,a ) [ g ] -¯ R ( s, a ) , and δ W ( s, a ) = ( 1 -¯ W ( a | s,a ) π b ( a | s ) π b + ( a | s ) ) .

We show Proposition 12 first and then discuss Propositions 2 and 3 as two special cases.

Proof of Proposition 12.

where δ W ( s, a ) = ( 1 -¯ W ( a | s,a ) π b ( a | s ) π b + ( a | s ) ) .

Note that U ( s, π b + ) ⊆ U ( s, π b ) , because π b + ( a | s ) = 0 implies π b ( a | s ) = 0 (factual data does not contain action a ) and ¯ W ( a | s, ˇ a ) = 0 , ∀ ˇ a ∈ A (counterfactual annotations for other actions ˇ a also do not contain action a ).

Proof for Proposition 2. Given Assumption 1 but not Assumption 4,

Proof for Proposition 3. Given Assumptions 3 and 4 but not Assumption 1,

While it is generally true that the space of policies that C-IS can evaluate without bias is larger than that of IS, since the lack of support leads to C-IS being biased towards 0 (similar to the case of standard IS), the magnitude of bias of C-IS is not guaranteed to be less than standard IS without additional assumptions about the rewards. However, if all rewards are non-negative, then under mild assumptions, we can prove the following bias reduction result.

Proposition 4 (A sufficient condition for bias reduction) . If Assumption 1 holds (but Assumption 4 is violated), ¯ R ( s, a ) ≥ 0 for all s ∈ S , a ∈ A , and there exists ( s, a ) such that π b ( a | s ) = 0 , π b + ( a | s ) > 0 , π e ( a | s ) > 0 , ¯ R ( s, a ) > 0 , then | Bias[ˆ v C-IS ] | < | Bias[ˆ v IS ] | .

Lastly, we note a useful corollary of Theorem 1.

Corollary 5 (Expectation of augmented importance ratios) . Let ρ + W = w a ρ a + ∑ ˜ a ∈A\{ a } w ˜ a ρ ˜ a given τ and w . Under Assumption 4, E [ ρ + W ] = 1 .

Proof of Corollary 5. Starting with Theorem 1 and substituting R ( s, a ) = G ( s, a ) = 1 as the constant reward and annotation for all s ∈ S , a ∈ A , we have: E τ + [ w a ρ a + ∑ ˜ a ∈A\{ a } w ˜ a ρ ˜ a ] = E s ∼ d 1 E ˜ a ∼ π e ( s ) [1] = 1 .

C.3 C-IS: Variance Analyses

For variance analyses, we focus on the scenario where Assumptions 1 and 4 hold and bias is zero. Below, we state the variance decomposition results under a few assumptions about the weighting scheme and the annotation variance.

Theorem 13 (Variance of C-IS) . Let ρ + ( a | s ) = ρ a = π e ( a | s ) π b + ( a | s ) be the importance ratio defined using the augmented behavior policy, and assume variance of the annotation function and the variance of the reward function are related by σ R ( s, a ) 2 = σ R ( s, a ) 2 +∆ σ ( s, a ) where ∆ σ ( s, a ) ∈ R , then under Assumptions 1 and 4,

̸

Corollary 14 (Variance of C*-IS) . Assuming σ G ( s, a ) 2 = σ R ( s, a ) 2 +∆ σ ( s, a ) where ∆ σ ( s, a ) ∈ R , then under Assumptions 1 and 4,

Remark. The variance decomposition in Theorem 13 contains eight terms. The first three terms correspond to the three terms in the IS variance decomposition in Eqn. (1), the fourth and eighth terms are related to the variance difference between annotations and rewards, whereas the remaining three terms are all related to the variance (and covariance) of the weight distribution W ( ·| s, a ) . As we will demonstrate empirically (Appendix E.1), depending on the weight distributions, variance of C-IS may actually be larger than that of IS. While it is difficult to guarantee a general variance reduction, we provide intuition for a few special cases. (i) If the weights are constant (i.e., W ( ·| s, a ) is the same value for all instantiations of ( s, a ) in the data), then the last four terms will all vanish to zero. (ii) When the annotation function and reward function have the same variance, ∆ σ ( s, a ) = 0 , the fourth and eight terms vanish to zero; with this assumption, we can straightforwardly obtain Theorem 6 from Corollary 14. (iii) Suppose for each state-action pair ( s, a ) , we set factual weights W ( a | s, a ) = 1 and counterfactual weights W (˜ a | s, a ) = 0 . Then, we effectively ignore all counterfactual annotations, and the variance of C-IS becomes identical to that of IS.

where ρ ( a | s ) = π e ( a | s ) π b ( a | s ) is the importance ratio under the original behavior policy.

We prove the most general case for Theorem 13 first, and then derive Corollary 14 as a special case.

Proof of Theorem 13. We apply the law of total variance:

As shown in the bias analyses, we have (1) = V s ∼ d 1 [ V π e ( s )] . We further apply the law of total variance on (1 ′ ) :

Since ( r, g ) and w are conditionally independent given ( s, a ) ,

where in the last step, we apply Assumption 1 to combine the expressions involving E r ∼ R ( s,a ) [ r ] and E g ˜ a ∼ G ( s, ˜ a ) [ g ˜ a ] (both are equal to ¯ R ( s, ˜ a ) ) into a single summation over ˜ a ∈ A .

We further apply the law of total variance on (2 ′ ) :

Then we have

where in the last step we substitute σ G ( s, ˜ a ) 2 = σ R ( s, ˜ a ) 2 +∆ σ ( s, ˜ a ) . Letting g a = r for clarity, we have

̸

Then,

̸

̸

Putting together expressions (1) through (7) , we have the desired decomposition for V [ˆ v C-IS ] .

Proof of Corollary 14. Here we derive the variance of C*-IS, which is C-IS with W (˜ a | s, a ) = |A| -1 . Since the weights are constant, expressions (4)(5)(6)(7) all vanish to zero because the variance and covariance associated with W ( ·| s, a ) are both zero. We focus on simplifying the second, third, and fourth terms in the variance decomposition (which correspond to expressions (2) and (3) in the proof above; we denote the two parts of (3) as (3 . 1) and (3 . 2) ). First, note that

̸

C.4 C-PDIS: Bias Analyses

Theorem 8 (Unbiasedness of C-PDIS) . In the MDP setting, when both Assumptions 2 and 4 hold, the C-PDIS estimator is unbiased, E [ˆ v C-PDIS ] = v ( π e ) .

First, we introduce a few definitions useful for the proof. Let the t -step state distribution be denoted by d π t ( s ) = Pr( s t = s | s 1 ∼ d 1 , a t ′ ∼ π ( s t ′ )) . Similarly, the t -step state-action distribution is d π t ( s, a ) = Pr( s t = s, a t = a | s 1 ∼ d 1 , a t ′ ∼ π ( s t ′ )) . Note that d π t +1 ( s ′ ) = E s,a ∼ d π t [ p ( s ′ | s, a )] . Let d t = d π b t denote the distribution under the behavior policy. Recall the horizont value functions: V π e t : T ( s ) = E π e [ ∑ T t ′ = t γ t ′ -1 r t ′ | s t = s ] , Q π e t : T ( s, a ) = E π e [ ∑ T t ′ = t γ t ′ -1 r t ′ | s t = s, a t = a ] . Based on the Bellman equation, Q π e t : T ( s, a ) = r ( s, a ) + γ E s ′ ∼ p ( s,a ) [ V π e ( t +1): T ( s ′ )] . Note that v ( π e ) = E s ∼ d 1 [ V π e 1: T ( s )] . Also recall Assumption 2, the perfect annotation function in the MDP setting satisfies E g ∼ G t ( s,a ) [ g ] = Q π e t : T ( s, a ) for ( s, a ) at step t of the trajectory.

Recall the recursive definition of C-PDIS (Definition 4): ˆ v C-PDIS = v T , with v 0 = 0 , v T -t +1 = w a t t ρ a t t ( r t + γv T -t ) + ∑ ˜ a ∈A\{ a t } w ˜ a t ρ ˜ a t q ˜ a t for t = T... 1 , where ρ ˜ a t = π e (˜ a | s t ) π b + (˜ a | s t ) .

Proof. Weshow this via backward induction on a sequence of horizont value functions of π e denoted by V π e t : T ( s ) . The goal is to show that E τ + , w [ v T -t ] = E s ∼ d t +1 [ V π e ( t +1): T ( s )] for all t = T... 0 (for clarity, the subscript of expectation of the estimator may be omitted and assumed to be E τ + , w with τ generated by the behavior policy π b unless otherwise specified).

Base case. It is trivially true that E [ v 0 ] = E s ∼ d T +1 [ V π e ( T +1): T ( s )] = 0 since there are no more steps after t = T and s ∼ d T +1 can be seen as a dummy absorbing state.

Inductive step. Suppose E [ v T -t ] = E s ∼ d t +1 [ V π e ( t +1): T ( s )] holds. For factual state-action pair ( s, a ) ∼ d t occurring at step t , we have

To show the case for t -1 ,

where in (1) we replace E g ˜ a t ∼ G t ( s, ˜ a ) [ g ˜ a t ] with Q π e t : T ( s, ˜ a ) following Assumption 2 and combine it with E s,a ∼ d t [ r t + γv T -t ] = Q π e t : T ( s, ˜ a ) (as shown above) in a single summation over ˜ a ∈ A , in (2) we swap the order of summations, in (3) we take out common factors that do not depend on the inner summation a , and in (4) we use Definition 1 for π b + (˜ a | s ) , which cancels out the denominator in the importance ratio on the next line. The proof techniques are similar to that in Appendix C.2.

Since E [ v T -t ] = E s ∼ d h +1 [ V π e ( t +1): T ( s )] implies E [ v T -t +1 ] = E s ∼ d t [ V π e t : T ( s )] , and the base case E [ v 0 ] = E s ∼ d T +1 [ V π e ( T +1): T ( s )] is true, by mathematical induction, we have the desired property that E [ˆ v C-PDIS ] = E [ v T ] = E s ∼ d 1 [ V π e 1: T ( s )] = v ( π e ) .

D Extended Discussions on Practical Implications

D.1 Using an Approximate MDP Model to Correct Annotation Bias

̸

In the sequential RL setting, if the annotation function reflects the expected returns under the behavior policy, i.e. G t = Q π b t : T , then annotations have a nonzero bias of ϵ G t = Q π b t : T -Q π e t : T = 0 (since Q π e t : T = Q π b t : T for meaningful OPE problems where π e = π b ), and consequently, C-PDIS using such annotations becomes biased. While such annotations may aid in model selection (as shown in experiments), here we further propose a procedure to convert the annotations so that they better reflect the evaluation policy.

̸

̸

From the offline data, we first learn an approximate model of the MDP ˆ M , which includes the transition model ˆ p ( s ′ | s, a ) and reward model ˆ r ( s, a ) . ˆ M is then used to evaluate both π b and π e using a model-based approach, leading to estimated horizon-specific Q-functions, ˆ Q π b t : T and ˆ Q π e t : T . We then approximate the annotation error for state s t occurring at horizon t and counterfactual action ˜ a as ˆ ϵ G t ( s t , ˜ a ) = ˆ Q π b t : T ( s t , ˜ a ) -ˆ Q π e t : T ( s t , ˜ a ) . Given the factual state-action pair ( s ( i ) t , a ( i ) t ) and the counterfactual annotation g ( i ) , ˜ a t ∼ Q π b t : T ( s ( i ) t , ˜ a ) for action ˜ a , we convert the annotation as

One may verify that ˆ g ( i ) , ˜ a t ∼ Q π e t : T ( s ( i ) t , ˜ a ) in expectation. Note that this approach for annotation conversion is only possible if the counterfactual action has support in the offline data (i.e., ( s ( i ) t , ˜ a ) is seen in the data); otherwise, we suggest using the annotation as collected.

D.2 Imputing Missing Annotations

Consider the following scenario: for two instances of the same factual ( s, a ) in the dataset, only one instance has a counterfactual annotation but not the other. This means that the weights for the factual and counterfactual are (0 . 5 , 0 . 5) and (1 , 0) (assuming a binary action space), and the weight distribution W ( ·| s, a ) has a nonzero variance and covariance. Specifically, in the example above with two samples, σ W ( a | s, a ) 2 = σ W (˜ a | s, a ) 2 = Cov( W ( a | s, a ) , W (˜ a | s, a )) = 0 . 0625 . These variances appear in the variance decomposition of C-IS as shown in Theorem 13 and may lead to an overall larger variance compared to IS.

To address this issue, we suggest a procedure to impute the missing annotations using other available annotations when possible. Given all annotations g D = { g ( i ) , ˜ a t : c ( i ) , ˜ a t = 1 } associated with dataset D = { τ ( i ) } N i =1 , we first build an approximate annotation model ˆ G by solving a regression problem on { ( ( s ( i ) t , ˜ a ) , g ( i ) , ˜ a t ) : c ( i ) , ˜ a t = 1 } using all available annotations g D . For discrete state and action spaces, this is essentially to averaging the annotations for each state-action pair. Then, ˆ G can be used to impute the missing annotations as ˆ g = { ˆ G ( s ( i ) t , ˜ a ) : c ( i ) , ˜ a t = 0 , ˜ a ∈ A \ { a ( i ) t }} if there is 'support' for annotations of the same state and action, i.e., c ( i ) , ˜ a t = 0 (annotation is missing) and ∑ N i ′ =1 ∑ T t ′ =1 ✶ [ s ( i ′ ) t ′ = s ( i ) t ] c ( i ′ ) , ˜ a t ′ > 0 (annotation has support and thus can be imputed). This approximate annotation model ˆ G may be biased, but generally its bias can be outweighed by the benefit of variance reduction for having equal weights (we also observed this empirically in Appendices E.1 and E.2).

D.3 Optimizing Weighting Schemes to Minimize Variance

Using a worked example, we illustrate that optimizing weights for variance reduction is a highly non-trivial problem. Suppose we have a bandit problem with one state { s } , two actions {↗ , ↘} , and the reward function is R ( s, ↗ ) = N ( r 0 , σ 2 0 ) , R ( s, ↘ ) = N ( r 1 , σ 2 1 ) . Furthermore, suppose all annotations are available, allowing us to use constant weights and eliminate the weight variances, and suppose the annotation function is identical to the reward function, i.e., G = R . We can simplify the variance decomposition of C-IS to be

We denote policies in terms of the probabilities assigned to the two actions.

Setting 1. Consider π b ( s ) = [1 , 0] , π e ( s ) = [ α, 1 -α ] . The offline dataset contains only ( s, a = ↗ ) with counterfactual annotations ( s, ˜ a = ↘ ) , and we assume they are each assigned a weight of w and (1 -w ) . The augmented behavior policy is then π b + ( s ) = [ w, 1 -w ] . The first and second terms of Eqn. (4) are zero because no variance is associated with sampling state-action pairs (only one possibility under π b ). The third term becomes (state s is omitted from the expressions)

This is a linear function of w , and where the variance achieves the minimum value depends on the slope ( ασ 2 0 -(1 -α ) σ 2 1 ) .

If ασ 2 0 = (1 -α ) σ 2 1 , the slope is zero, and w does not affect variance.

If ασ 2 0 > (1 -α ) σ 2 1 , the slope is positive, and w ∗ → 0 achieves minimum variance of (1 -α ) σ 2 1 .

If ασ 2 0 < (1 -α ) σ 2 1 , the slope is negative, and w ∗ → 1 achieves minimum variance of ασ 2 0 .

Setting 2. Consider π b ( s ) = [ β, 1 -β ] , π e ( s ) = [ α, 1 -α ] . In the offline dataset, ( s, a = ↗ ) appears with probability β and ( s, a = ↘ ) appears with probability 1 -β . When the factual data is ( s, a = ↗ ) , the counterfactual annotation is for ( s, ˜ a = ↘ ) , and let the weights be w 0 and (1 -w 0 ) ; when the factual data is ( s, a = ↘ ) , the counterfactual annotation is for ( s, ˜ a = ↗ ) , and let the weights be w 1 and (1 -w 1 ) . The augmented behavior policy is then

The first term of Eqn. (4) is zero because there is only one state. The third term is

Next, we will attempt to simplify the second variance term:

First note that

Then, the second variance term becomes

As shown above, the variance expression is a rather complicated function of ( w 0 , w 1 ) . To solve for its minimum, one may take the derivatives with respect to each of w 0 and w 1 and solve for the zeros. The final solution will depend on the problem parameters, including r 0 , r 1 , σ 2 0 , σ 2 1 , α, β . Here, we do not solve for the final solution, and note that this approach may not be applicable to real-world problems due to its explicitly dependence on problem parameters that may be unknown. We encourage future work to explore different methods to find the variance-minimizing weighting schemes.

E Extended Experiments

E.1 Synthetic Domains - Bandits

E.1.1 Two-State Bandits

We expand on the experiments shown in Section 5.1, where we consider a class of bandit problems with two states { s 1 , s 2 } (drawn with equal probability), two actions A = {↗ , ↘} (recall Figure 3), and corresponding reward distributions R ( s i , a ) ∼ N ( ¯ R ( s i ,a ) , σ 2 ) . Without loss of generality, we assume ↗ is always taken from s 2 by both π b and π e . For s 1 , we consider deterministic policies (in which one action is always taken) as well as stochastic policies (which take the two actions with probabilities that sum to 1 ); see column/row header in Table 3. Given ( π b , π e ) , we draw 1 , 000 samples following π b and then evaluate π e using various estimators, including standard IS, the naive baseline of adding counterfactual annotations as new samples (Section 3.1), and C*-IS. We assume that counterfactual annotations are only available for s 1 , and all annotations are drawn from the true reward function. We measure the bias, standard deviation (square root of variance), and root mean-squared error (RMSE) of the estimators with respect to v ( π e ) . In Table 3 we consider three settings of the rewards: (i) ¯ R ( s 1 , ↗ ) = 1 , ¯ R ( s 1 , ↘ ) = 2 , i.e., both actions lead to a positive reward. (ii) ¯ R ( s 1 , ↗ ) = -1 , ¯ R ( s 1 , ↘ ) = 1 , i.e., one action leads to a positive reward and the other leads to a negative reward, (iii) ¯ R ( s 1 , ↗ ) = -1 , ¯ R ( s 1 , ↘ ) = -2 , i.e., both actions lead to a negative reward. In all cases, the reward for state s 2 is set to 0 for both actions.

Naive baseline fails due to bias. The naive baseline often has a nonzero bias and worse RMSE than IS regardless of whether the offline data has support. This is consistent with our analyses and example provided in Appendix B.

Bias reduction in support-deficient settings. In the first two rows of each sub-table in Table 3, π b is deterministic and the untaken action has poor support in the offline data. IS is often biased for these cases; note that in cases where π e assigns a small probability to the unsupported action (e.g., row 2, column 4, π e takes ↗ with probability 0 . 1 ), the bias is small and shows up as 0 after rounding. In particular, when rewards as all positive (the first table of Table 3), the bias is negative, and when rewards are all negative (the third table of Table 3), the bias is positive. In the second table of Table 3, the direction of bias depends on the reward of the unsupported action. In the last three rows, IS is unbiased when the offline data has full support. In contrast, C*-IS is unbiased in all cases (but for rounding errors), and can reduce bias compared to IS in support-deficient settings by making use of counterfactual annotations. RMSE of C*-IS is often (but not always) reduced compared to IS even though variance can sometimes increase. This is consistent with our analyses in Appendix C.2.

Variance reduction in well-supported settings. In the last three rows of each sub-table in Table 3, the offline data has full support because π b is stochastic, and IS is unbiased in these cases. C*-IS is also unbiased in these cases and achieves lower variance, leading to lower RMSE than IS. This is consistent with our analyses in Appendix C.3.

E.1.2 One-State Bandits

To simplify further experiments, we modify the two-state bandits above so that s 2 is drawn with probability 0 -equivalently, we have a set of one-state bandits (with only s 1 ) with two actions A = { 0 , 1 } where 0 corresponds to ↗ and 1 corresponds to ↘ , and corresponding reward distributions R 0 ∼ N ( ¯ R 0 , σ 2 0 ) and R 1 ∼ N ( ¯ R 1 , σ 2 1 ) . For π b and π e , we consider the same set of policies as before, and the same evaluation setup. We omit the naive approach in this setting because it is equivalent to our proposed approach when there is only one state (the state distribution is not affected by adding counterfactual annotations directly).

Results for this setting are summarized in Table 4 and show similar trends as above . Next, we use the one-state bandit to study the effect of weights w on OPE performance.

Using equal weights (in C*-IS) is a reasonable heuristic though not always variance-minimizing. In the results above, C*-IS assumed the weights are split equally for factual and counterfactual data. Next, we explore the effect of different weighting schemes on the performance of C-IS, applied to the same class of bandit problems described above. Note that, consistent with Theorem 1, C-IS remains unbiased regardless of weights, except potentially at extreme values of weights (0 or 1) that ignore either the factual data or counterfactual annotations. Across different settings (Figure 7a-d), we found that the ideal weighting scheme is problem-specific, and certain weights may lead to higher variance compared to standard IS (e.g., lower left region of Figure 7b). Encouragingly, these results also demonstrate that C*-IS, though not always variance-minimizing, consistently achieves lower variance than standard IS (when the data has full support and the estimator is unbiased). Additionally, variance in the weight distributions directly contributes to variance in the resulting estimator (Figure 7e), corroborating our analysis in Theorem 13. Overall, our results suggest that C*-IS, which uses constant weights split equally among actions, is a promising heuristic for using C-IS in practice.

(a)

(b)

(c)

(d)

(e)

0

1

W

(˜

a

= 0

s, a

= 0)

0

1

W

(˜

a

= 1

|

s, a

= 1)

∗

∗

∗

∗

∗

∗

∗

∗

∗

0

1

0

1

W

(˜

a

= 0

s, a

= 0)

0

1

W

(˜

a

= 1

|

s, a

= 1)

∗

0

1

0

1

W

(˜

a

= 0

s, a

= 0)

0

1

W

(˜

a

= 1

|

s, a

= 1)

∗

1

2

1

0

1

W

(˜

a

= 0

s, a

= 0)

0

1

W

(˜

a

= 1

|

s, a

= 1)

∗

0

1

0

1

range(

W

(

s, a

= 0))

0

1

range(

W

(

·|

s, a

= 1))

∗

0

1

4

|

|

|

|

·|

Parameter settings: (a) Base setting, R = [1 , 2] , σ R = [1 , 1] , π b = [0 . 1 , 0 . 9] , π e = [0 . 8 , 0 . 2] , σ G = σ R . (b) Annotations are not useful, R = [1 , 2] , σ R = [1 , 1] , π b = [0 . 9 , 0 . 1] , π e = [0 . 95 , 0 . 05] , σ G = σ R . (c) Annotations has larger variance than rewards, R = [1 , 2] , σ R = [1 , 1] , π b = [0 . 1 , 0 . 9] , π e = [0 . 8 , 0 . 2] , σ G = 2 σ R . (d) Annotations has smaller variance than rewards, R = [1 , 2] , σ R = [1 , 1] , π b = [0 . 1 , 0 . 9] , π e = [0 . 8 , 0 . 2] , σ G = 0 . 5 σ R . (e) R = [1 , 2] , σ R = [1 , 1] , π b = [0 . 1 , 0 . 9] , π e = [0 . 8 , 0 . 2] , σ G = σ R .

Imputing missing annotations can reduce variance. Finally, we explore the impact of missing annotations on the performance of C-IS. As shown in Figure 8a, obtaining more counterfactual annotations generally helps to reduce the variance. However, as noted in our variance analyses, if annotations for the same factual ( s, a ) are sometimes missing, we cannot directly apply C*-IS with equal weights; this may lead to increased variance due to variance in the weights. Here, we use a simple strategy to impute the missing annotations with the average of other annotations (for the same counterfactual ˜ a of the same factual ( s, a ) ). As shown in Figure 8b, this reduces the variance further especially when not all annotations are available.

(a) Without imputation (b) With imputation

0

1

Fraction annotated

(action 0)

0

1

Fraction annotated

(action 1)

∗

0

.

0

0

.

5

1

.

0

1

.

5

0

1

Fraction annotated

(action 0)

0

1

Fraction annotated

(action 1)

∗

0

.

0

0

.

5

1

.

0

1

.

5

Figure 8: Heatmaps of log std for C-IS where some counterfactual annotations may be missing. We vary the fraction of acquired annotations for the two actions independently within [0 , 1] . (a) Equal weights (as in C*-IS) are used when annotations are available; otherwise, the factual weight is set to 1 when annotations are missing. (b) The missing annotations are imputed using other similar annotations. Imputing missing annotations allows for C*-IS to be applied directly and it achieves lower variance. Parameter settings are the same as Figure 7a.

E.2 Healthcare Domain - Sepsis Simulator

E.2.1 Experimental Setup

Simulator Description. The patient state is characterized by five variables: a binary indicator for diabetes status, and four ordinal-valued vital signs (heart rate, blood pressure, oxygen concentration, glucose). Following prior work [22], we used a discrete state space with |S| = 1440 . The action space corresponds to the administration of vasopressors and takes on a binary value (on/off), which may increase or decrease the values of certain vital signs (with pre-specified probabilities) at the next time step. The action space in the original simulator formulation involves combinations of 3 treatments: antibiotics, vasopressors, and mechanical ventilation; we focus on only vasopressors for the purpose of illustration. The episode ends when the patient is discharged or died; discharge only occurs when all vitals are normal and all treatments are turned off, whereas death occurs if three or more vitals are abnormal. Rewards are sparse and only assigned at the end of each episode, with +1 for survival and -1 for death. Episodes that reach the maximum length of 20 are truncated with zero terminal reward.

Evaluation Setup. Following prior work [22], we collected 50 offline datasets from the sepsis simulator (using different random seeds) each with 1000 episodes by following an ϵ -greedy behavior policy with respect to the optimal policy where ϵ = 0 . 1 . For the evaluation policies, we created a set of deterministic policies by perturbing the optimal policy such that each policy takes the non-optimal actions in a randomly selected subset of states. We varied the number of 'action-flipped' states in { 50 , 100 , 200 , 300 , 400 } and generated 5 different evaluation policies for each; we also included the optimal policy in the candidate set, resulting in a total of 26 candidate evaluation policies. These represent policies that may be derived from offline data by typical RL approaches that aim to learn deterministic policies, and are of diverse quality where approximately half are superior to the behavior policy while the other half are inferior (Figure 9). As the baseline estimator that only makes use of offline data, we applied standard PDIS which does not rely on counterfactual annotations. To apply C*-PDIS, we assume all counterfactual annotations are collected in our main experiments in which they may be drawn from different annotation functions; we explore the impact of missing annotations in subsequent sensitivity analyses. For C*-PDIS, we considered three different annotation functions: (i) G = Q π e , the Q-function of the evaluation policy π e ; (ii) G = Q π b , the Q-function of the behavior policy, and (iii) G = Q π b ↦→ Q π e , where we apply the bias correction procedure discussed in Appendix D.1. We also compare to two naive baselines (given perfect annotations): 'naive unweighted' simply adds counterfactual annotations as new trajectories and has the same issue discussed in Section 3.1, whereas 'naive weighted' reweights the annotations at the trajectory level instead of per-decision. More formally, assuming a binary action space A = { 0 , 1 } (without loss of generality), given a trajectory of length T with counterfactual annotations at each step, τ = [ s t , a t , r t ] T t =1 , g = { g 1 -a t t } T t =1 where 1 -a t is the counterfactual action for a t , the naive weighted estimator is defined as

Intuitively, this first converts each annotation into a sub-trajectory that terminates at the step of annotation with the counterfactual action [ s 1 , a 1 , r 1 , · · · , s t , 1 -a t , g t ] , and then performs IS on each sub-trajectory (including the original trajectory), and finally computes a weighted sum of these T + 1 estimates ( 1 factual estimate, T counterfactual estimates) using weights (1 -∑ T t =1 w t ) , w 1 , · · · , w T . The reason why 'naive weighted' does not work is more subtle: while reweighting the (partial) trajectories constructed from the counterfactual annotations correctly maintains the initial state distribution, it does not correctly maintain the intermediate state distributions.

0

5

10

15

20

25

policy index

0.0

0.1

0.2

0.3

0.4

policy value

optimal

behavior

0

5

10

15

20

25

policy index

0.4

0.2

0.0

0.2

0.4

0.6

policy value

optimal

behavior

E.2.2 Results

C*-PDIS outperforms all baselines in all metrics for the ideal setting. As shown in Table 5, when all counterfactuals are available and annotated with the evaluation policy's Q-function ( G = Q π e ), C*-PDIS outperforms baseline PDIS (without annotations) in all metrics, suggesting that it provides more accurate OPE estimates. In contrast, the two naive approaches fail to provide accurate estimates and often underperform standard PDIS.

C*-PDIS is robust to biased annotations. Under the more realistic scenario where G = Q π b , i.e., annotations summarize the future returns under π b rather than π e , we observe a degradation in all metrics compared to the ideal case, though C*-PDIS is still superior to PDIS (Table 5). Applying the bias correction procedure G = Q π b ↦→ ˆ Q π e (see Appendix D.1) helps recover performance to be closer to the ideal case, and is especially helpful for π e that are far away from π b (Figure 10).

}

}

⋆

ideal case relaxing

Assump. 2

0

.

0

0

.

5

1

.

0

D

KL

(

π

b

π

e

)

0

.

0

0

.

1

0

.

2

RMSE(

v

(

π

e

)

,

ˆ

v

(

π

e

)

)

PDIS (baseline)

C*-PDIS (

G

=

Q

π

b

)

C*-PDIS (

G

=

Q

π

b

ˆ

Q

π

e

)

C*-PDIS (

G

=

Q

π

e

)

||

Variance reduction of C*-PDIS outweighs the effect of noisy annotations. To understand the robustness of our estimator to annotation noise, we perturbed the annotations with varying amounts of noise. Specifically, each annotation is added with a noise value drawn from zero-mean Gaussian distributions with a pre-specified standard deviation which we vary. As the level of annotation noise increases (Figure 11-left), performance degradation is minimal even at the highest level of noise tested (with a std of 1, which is large relative to the reward range [ -1 , 1] , and most notably larger than 0 . 31 , the std of initial state values of this domain). Our estimator remains competitive relative to the baseline PDIS, suggesting that the benefit of variance reduction from additional data (through counterfactual annotations) outweighs the variance increase from annotation noise, even when annotations are much noisier than factual data. The same trend holds when only 10% of the counterfactual annotations are collected (Figure 11-right).

PDIS (baseline)

ideal case

C*-PDIS, noisy annot.

0.0

0.1

RMSE

0.5

1.0

Spearman

Corr.

0.0

0.5

1.0

Std of annotation noise

0.8

1.0

Bin. Class.

0.0

0.1

RMSE

0.5

1.0

Spearman

Corr.

0.0

0.5

1.0

Std of annotation noise

0.8

1.0

Bin. Class.

baseline

ideal setting

proposed

Collecting more annotations and imputing missing annotations improves performance. As the amount of available annotations increases (Figure 12), our approach interpolates between baseline PDIS and the ideal case of C*-PDIS with an monotonic improvement in performance. Furthermore, imputing annotations (as described in Appendix D.2) achieves better performance, suggesting it is a promising strategy to handle missing annotations when not all annotations can be obtained in practice. The same trend holds under different amounts of annotation noise (Figure 12 left vs right).

PDIS (baseline)

ideal case

C-PDIS , w/o imputation

C*-PDIS, w/ imputation

0.0

0.1

RMSE

0.5

1.0

Spearman

Corr.

0.0

0.5

1.0

fraction of annotated samples

0.8

1.0

Bin. Class.

0.0

0.1

RMSE

0.5

1.0

Spearman

Corr.

0.0

0.5

1.0

fraction of annotated samples

0.8

1.0

Bin. Class.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed a novel method for off-policy evaluation with offline data that entails human input. Specifically, the paper assumes access to human annotations for counterfactual state-action pairs as counterfactual immediate reward (in bandits) or Q values (in RL) under the behaviour policy (as a relaxation to Q values under the evaluation policy). They used a bandit example to show that simply incorporating counterfactual annotations into the offline data can shift state distribution and therefore bias overall estimates. To keep the state distribution unchanged, they further assume access to a human-privided weight for each visited / annotated counterfactual state-action pair. Effectively this creates an augmented behaviour policy as if smoothed towards a uniform policy. They are then able to reweigh the importance sampling ratios and conduct OPE as a weighted sum of the conventional backwards bootstrapping and human annotation. They then move on to give theoretic analyses about the bias and variance of the proposed OPE estimator. Experiments conducted on a synthetic bandit problem and a healthcare problem with simulator demonstrated overall improved performance than vanilla IS and ablated versions of the proposed method itself\n\nSTRENGTHS:\n1. proposed a novel method for OPE with offline data, which seems increasingly timely as machine learning is marching into real life\n2. experiments have tested thorough aspects of the proposed idea\n\nWEAKNESSES:\n1. even if the paper goes into length to keep state distribution not affected by incorporating human annotations, it still does not address the distribution shift between the state distribution induced by the behaviour policy and by the evaluation policy\n2. expert annotations for counterfactual state-action pairs along with their weights are also expensive to get (though argubly safer / easier than rolling out the evaluation policy in real world) and inevitably subjective\n3. unclear how the promising results could extend to real-world scenarios\n\nQUESTIONS:\n1. how would human experts provide Q values under the behaviour policy in practice? It does not seem any easier than providing \nQ values under the evaluation policy both of which entail a sequence of counterfactual state-action pairs that are not their own choices\n2. Appendix D.2, to relax Q under the evaluation policy to Q under the behaviour policy in the annotation, why simply considering a bias term in the annotation is enough? It seems to me you would also need to reweigh the annotation based on how relatively likely the counterfactual action is selected. Yes you have indeed reweighed for this between the evaluation policy and the augmented behaviour policy, but now you have a third policy, i.e. the original behaviour policy before augmentation (under which you have the Q values). I would imagine in addition to the IS ratio as you have in place already, you would also need to reweigh the Q value with e.g. product of subsequent IS ratios between the evaluation policy and the original behaviour policy. However, I see in your experiments you showed that the performance is not much affected with biased annotations, so I suppose not reweighing the biased annotations is not a major issue, but it would still be good to see further justification for that.\n\nLIMITATIONS:\nplease see above", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- Propose a setting in OPE where experts can annotate the value of counterfactual actions.\n- A new importance sampling scheme for the setting is proposed.\n- Theoretical analysis and experiments confirm superiority over the unweighted case, etc.\n\nSTRENGTHS:\n1. The proposed setting of expert annotating the counterfactual values seems like a direction worth exploring\n1. Clearly written with some theoretical guarantees. In particular, the bias and variance are carefully discussed for cases where assumptions such as the coverage are not met, and benefit such as bias reduction compared to importance sampling (IS) is shown.\n1. A useful heuristic is proposed (equal weighting)\n\nWEAKNESSES:\n1. The method itself and the baselines are fairly naive.\n    1. Comparisons with standard OPE baselines, such as the direct and doubly robust methods as well as the IS method, may be of value to better estimate the benefit of annotating counterfactuals for readers.\n    1. Also have a question about the claimed problem with the naive IS method, which is pointed out in section 3.1. See Question 2 below.\n1. Other similar annotation augmentation approaches (e.g., [1,2], but not limited to) are not compared neigher theoretically nor experimentally. I believe that the proposal to add counterfactual annotations is a costly and significant change to the problem setting and should be widely compared to similar approaches widely.\n\n[1] Srivastava, Megha, Tatsunori Hashimoto, and Percy Liang. \"Robustness to spurious correlations via human annotations.\" International Conference on Machine Learning. PMLR, 2020.\n\n[2] Kaushik, Divyansh, Eduard Hovy, and Zachary Lipton. \"Learning The Difference That Makes A Difference With Counterfactually-Augmented Data.\" International Conference on Learning Representations. 2019.\n\nQUESTIONS:\n1. What is the exact definition of \"naive weighted\"? What is the theoretical advantage of the proposed method?\n1. The problem with the naive IS method is shown in section 3.1, but is this not simply because the state visitation probability $d_\\pi(s)$ is not corrected? The problem can be solved by simply applying any OPE method that takes $d_\\pi(s)$ into account, such as DualDICE [3], to the augmented dataset.\n\n[3] Nachum, Ofir, et al. \"Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections.\" Advances in neural information processing systems 32 (2019).\n\nLIMITATIONS:\n1. The proposed method requires costly accurate annotation, which is properly investigated theoretically and experimentally.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper offers a new OPE estimator leveraging additional annotations of counterfactual actions. The paper provided an extremely thorough and comprehensive discussion of the theoretical property of this estimator. This helps us develop an understanding of how it operates. This paper also discussed fully the limitations of the approach and how the estimator would behave if different assumptions have been violated.\n\nI generally find this to be a good paper, with limited experiments and not enough real-world motivation, but I can potentially see an application of this method, even though, for now, I don’t yet know what it could be.\n\nSTRENGTHS:\nThe theoretical discussion of the estimator is very clear and very thorough. The paper discussed how bias of the annotator functions would impact the bias of the estimator. The paper also discussed when and how annotations could help reduce bias and variance of the estimator. The paper is also very clearly written. The proofs in the appendix are easy to follow (although I have not checked all of them). The experiments in the appendix are well documented.\n\nWEAKNESSES:\nThere are a few weaknesses of this paper that prevents it from having a higher impact.\n1. No human experiment has been offered. There is no actual investigation of what human inter-annotator variance would look like. It is somewhat known in a few important domains, such as dialogue evaluation (perhaps not surprisingly, this is a huge application field where RL is applied to natural language processing), inter-annotator agreement on certain metrics can be huge [1] (Table 2). The authors didn’t collect real human annotations at all, and the paper lacks key data that can shed light on whether this approach can work with real human annotations. Also, collecting $\\omega$ would seem to be difficult as well. It is unfortunate that the authors didn’t opt to do this, but I don’t think this eclipses other amazing parts of this paper.\n2. The experiments are too synthetic. This is a complaint similar to but slightly different from the previous point. Most RL/Bandit tasks have real-world motivations. By using a synthetic bandit experiment and Sepsis, the authors did not convince me that these two tasks would clearly benefit from human annotations. When are human annotations desirable in an RL/bandit task? How hard is it to obtain human annotations, especially for a sequential setting where some level of “imagine what the policy would do in the future” is involved? Can the authors shed some light on what kind of real-world task they hope to apply this OPE estimator to?\n\n[1] Lowe, Ryan, et al. \"Towards an automatic turing test: Learning to evaluate dialogue responses.\" arXiv preprint arXiv:1708.07149 (2017).\n\nQUESTIONS:\n1. Section 4.2 discusses the variance of the annotation function that would impact the variance of C*-IS. It assumes that the annotation function has the same variance as the reward function. Can the authors provide some insights on what happens when the annotation function has a lower variance than the reward function (this is possible because human annotations are malleable) or higher variance?\n\nLIMITATIONS:\nThe authors addressed the limitations in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper considers OPE with annotation to improve the performance of OPE. The paper explains why naively incorporating such information may lead to a biased estimate. And the paper provides a theoretical analysis of the bias and variance. Finally, the paper demonstrates the performance of the proposed method using synthetic bandit and healthcare experients.\n\nSTRENGTHS:\nOPE is well-known to suffer from high variance. And incorporating other information is important and relevant. The paper studies an interesting and important question and proposes useful solutions with theoretical guarantees.\n\nWEAKNESSES:\nThe clarity of the paper needs to improve. The main concept of this paper, annotation, is first introduced in Section 2. And In the Introduction, the authors only mention this notion in Figure 1. I think the authors should introduce and discuss \"annotation\" in the introduction to facilitate the reader's understanding. There are some other confusing points that may hinder comprehension, which I list in the Question section.\n\nAssumptions 1 and 2 seem very strong. Could the authors discuss the impact of the bias of the annotation?\n\nIt seems in the current formulation, a behavior trajectory with annotations is very similar to many behavior trajectories. At least in the bandit case, I think they are equivalent. If it is correct, it will affect the paper's novelty and contribution.\n\nRelated to the clarity, \"distribution shifts\" are mentioned in the abstract, but never show up again in the paper. I think it is better to discuss the issues of distribution shifts induced by different policies in RL.\n\nQUESTIONS:\n1. IN Section 3, are weights state-dependent? And it seems the weights are random. How should I understand it? \n\n2. Why the variance does not depend on the variance of the annotation var(g)?\n\nLIMITATIONS:\nThe authors discussed some limitations in Section 4.4.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies offline policy evaluation in reinforcement learning, where the behavior policy, some interaction trajectories, and some human annotated trajectories (modified from interaction trajectories) are available. The paper points out that, since the number of human annotated trajectories are not the same across interaction trajectories, naively treating these human annotated trajectories the same as interaction trajectories can introduce bias (for example, in bandits, the context distribution changes), even if human annotations are perfect. So the paper proposes to weight the trajectories such that for each interaction trajectory, the weights of this trajectory and the trajectories  annotated from this trajectory sum to one. This ensures estimating the value of a policy from the correct state distribution. \n\nThe paper analyzed the bias and variance of the proposed estimator in the bandit setting and the bias of the estimator in the RL setting, under some specific settings (sometimes when the weights are the same and every trajectory is annotated, sometimes human annotations are perfect) and conducted empirical evaluation to show that the proposed estimator can reduce estimation error.\n\nSTRENGTHS:\nThe paper is well-written and easy to follow. \n\nThe paper studies an interesting problem of off-policy evaluation in reinforcement learning with the help of both interaction data and human annotations. This setting makes sense when data is sparse and human annotations are accurate and affordable. \n\nThe paper both theoretically and empirically analyze the bias and variance of the proposed estimator.\n\nWEAKNESSES:\nMy major concern is about the novelty of the paper. The paper did not answer the question of how to set the weights of interaction data and human annotations. To me, it seems that weighting is an obvious approach to try when correcting for the state distribution. I believe there are other works using similar idea. For example [1]. The interesting question to me is how to set the weights for optimal estimation error reduction. Unfortunately, this is not discussed in the paper. \n\nIt would be nice to discuss the relationship between this work and policy evaluation from multiple loggers (e.g. [2]). To me, they are similar problems since human annotations are like collected from another logging policy (except we need to correct for the state distribution). \n\nI am a bit confused about the definition of C^\\star-IS. The authors mentioned that C^\\star-IS corresponds to w = 1/|A|. The authors also mentioned that the weight should be zero when there are no annotations. This means only when we have full information data can we use C^\\star-IS. First, this seems a less interesting setting to analyze since the fundamental problem of missing data in RL is gone. Second, in the experiments, some actions were not annotated, but C^\\star-IS is still used. There seems to be a contradiction? \n\nMinor:\n\nIn line 50, the paper mentioned that the estimator requires a weaker condition. To me, this is an overclaim, since in reality, only the support condition is weaker, but an additional strong assumption that human annotations are perfect is also required. \n\n[1] Learning to Rank with Selection Bias in Personal Search Xuanhui Wang, Michael Bendersky, Donald Metzler, Marc Najork SIGIR 16\n\n[2] Effective Evaluation using Logged Bandit Feedback from Multiple Loggers A. Agarwal and S. Basu and T. Schnabel and T. Joachims KDD 17\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nThe authors adequately addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-28"}