Title: Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes
Venue: NeurIPS 2024
Accepted: yes
Keywords: Sensitivity Analysis

Robust MDPs

Semiparametric Efficiency
Abstract: We study the evaluation of a policy under best- and worst-case perturbations to a Markov decision process (MDP), using transition observations from the original MDP, whether they are generated under the same or a different policy. This is an important problem when there is the possibility of a shift between historical and future environments, \emph{e.g.} due to unmeasured confounding, distributional shift, or an adversarial environment. We propose a perturbation model that allows changes in the transition kernel densities up to a given multiplicative factor or its reciprocal, extending the classic marginal sensitivity model (MSM) for single time-step decision-making to infinite-horizon RL. We characterize the sharp bounds on policy value under this model -- \emph{i.e.}, the tightest possible bounds based on transition observations from the original MDP -- and we study the estimation of these bounds from such transition observations. We develop an estimator with several important guarantees: it is semiparametrically efficient, and remains so even when certain necessary nuisance functions, such as worst-case Q-functions, are estimated at slow, nonparametric rates. Our estimator is also asymptotically normal, enabling straightforward statistical inference using Wald confidence intervals. Moreover, when certain nuisances are estimated inconsistently, the estimator still provides valid, albeit possibly not sharp, bounds on the policy value. We validate these properties in numerical simulations. The combination of accounting for environment shifts from train to test (robustness), being insensitive to nuisance-function estimation (orthogonality), and addressing the challenge of learning from finite samples (inference) together leads to credible and reliable policy evaluation.

Efficient and Sharp Off-Policy Evaluation in Robust Markov Decision Processes

Abstract

We study the evaluation of a policy under best- and worst-case perturbations to a Markov decision process (MDP), using transition observations from the original MDP, whether they are generated under the same or a different policy. This is an important problem when there is the possibility of a shift between historical and future environments, e.g. due to unmeasured confounding, distributional shift, or an adversarial environment. We propose a perturbation model that allows changes in the transition kernel densities up to a given multiplicative factor or its reciprocal, extending the classic marginal sensitivity model (MSM) for single timestep decision-making to infinite-horizon RL. We characterize the sharp bounds on policy value under this model i.e. , the tightest possible bounds based on transition observations from the original MDP - and we study the estimation of these bounds from such transition observations. We develop an estimator with several important guarantees: it is semiparametrically efficient, and remains so even when certain necessary nuisance functions, such as worst-case Q-functions, are estimated at slow, nonparametric rates. Our estimator is also asymptotically normal, enabling straightforward statistical inference using Wald confidence intervals. Moreover, when certain nuisances are estimated inconsistently, the estimator still provides valid, albeit possibly not sharp, bounds on the policy value. We validate these properties in numerical simulations. The combination of accounting for environment shifts from train to test (robustness), being insensitive to nuisancefunction estimation (orthogonality), and addressing the challenge of learning from finite samples (inference) together leads to credible and reliable policy evaluation.

1 Introduction

Offline policy evaluation (OPE) from historical data is crucial in domains where active, on-policy experimentation is costly, risky, unethical, or otherwise operationally infeasible. Relevant domains range from medicine to finance and recommendation systems. However, whenever historical data is used to study future behavior, there is a concern of non-stationarity - shift between the environment generating the data (training environment) and the environment in which a policy will be deployed (test environment). This may occur, e.g. , due to general distributional shifts in the environment over time, unobserved confounding in the observed historical data, or adversarial elements of the environment (such as other agents) that may react when the agent is deployed. While standard OPE in offline reinforcement learning (ORL) accounts for the change between the logging and evaluation policies, it may overlook the fact that the Markov decision process (MDP) too has changed. While

this issue is particularly critical in high-stakes domains, it is broadly appealing to understand how value shifts across different environments in any application domain.

Robust MDPs [34, 56] model unknown environments by allowing an adversary to choose from any one environment in a set. Therefore, they offer a natural model for unknown environment shifts by simply considering all environments to which we could possibly shift. A variety of work addresses questions such as planning in a known robust MDP [30, 51, 80] as well as online learning [6, 79]. Here we focus on a purely statistical estimation question: given observations of transitions from some unknown transition kernel, we wish to estimate the worst-case (or best-case) value of a given evaluation policy in a robust MDP, defined by a set of MDPs whose transition functions are centered around the observed transition kernel.

This setting captures the previously studied unconfounded robust OPE problem [73], where the observed transition kernel corresponds to an MDP, and the observed transitions are the result of applying some logging policy within this MDP. In such cases, the goal is to estimate policy values that are robust to future changes in the MDP dynamics. However, our setting is more general in that it also captures problems where the observed transitions are confounded by some unobserved variables, in which case they do not correspond to observations from the transition kernel of an MDP. In this case, the robust MDP and the robust policy value estimates are designed to account for worst-case (or best-case) impact of this confounding bias. In either case, as in ORL, we emphasize that we do not know the observational MDP, and can only access it via a sample of transitions. Furthermore, even in the simple case with no unmeasured confounding, in a notable departure from standard ORL, the problem can be difficult even if the logging and evaluation policies are the same (the usually easy on-policy setting), since the policy can induce very different visitation distributions in the original and perturbed MDPs.

Such robust offline evaluation from transition data was considered in recent work [12, 59]. We build on this recent work by focusing the question of statistically efficient and robust estimation of the sharp bounds ( i.e. , the tightest possible given the data). Previous work focused on evaluation using only the Q-function under the worst-case environment (in some cases under a relaxation of the adversary, leading to loose bounds). Thus, any error in its estimation translates directly to error in evaluation. In other words, flexible nonparametric modeling of this function can mean slow rates for estimated bounds and a lack semiparametric efficiency. Moreover, without a clear understanding of the noise in the estimates, we cannot add confidence bands to the bounds, leading to bounds that are too tight.

We address these challenges by developing an orthogonalized estimation method that combines several nuisance functions: the worst-case Q -function, the state-visitation frequency in the worst-case environment, and a threshold function characterizing the worst-case transition kernel. Our first key result is that, to first order, our estimator behaves as a sample average using the true values of these functions without having to estimate them at all, provided we just estimate them at certain slow nonparametric rates. This ensures we not only have a √ n -rate of estimation even when nuisances are estimated more slowly, but also that our estimator is asymptotically normal. This allows for the construction of confidence bands on the bounds, providing assurance that the true bound is captured. We further show that our asymptotic variance is in fact the minimum variance among all regular and asymptotically linear (RAL) estimators, ensuring semiparametric efficiency. Our second key result is that even if we do not estimate some of the nuisance functions correctly, we are still consistent to sharp or valid bounds. That is, even when we are biased due to misestimation of nuisances, our bias (if any) only enlarges our bounds, so they remain valid. We illustrate these guarantees numerically. Collectively, these guarantees lend substantial credibility to the bounds generated by our method.

Our contributions are summarized as follows:

We provide novel algorithms and analysis for learning robust Q -functions (Section 3) and robust visitation density ratios (Section 4) under the function approximation setting.

We derive the sharp and efficient estimator for the robust policy value, which is optimal in the local-minimax sense and is the gold standard in semiparametric estimation (Section 5).

We empirically validate the efficiency and sharpness of our approach (Section 6).

1.1 Related Works

Unobserved Confounding in Sequential Decision-Making. OPE in robust MDPs is related to OPE bounds in confounded MDPs, where the behavior policy and the transition kernel are influenced by

unobserved confounders. The constraint Eq. (1) that defines our target robust MDP aligns with the Marginal Sensitivity Model (MSM) [66] employed in sensitivity analysis for causal inference. Yet, unlike the MSM, which limits the ratio of policy densities, our approach directly constrains the ratio of the transition kernels. Our formulation can be viewed as a generalization of the MSM from traditional two-action no-horizon causal effects (where the constrains coincide) to multi-action infinite-horizon discounted MDPs, where the next state is the 'potential outcome'. In that sense, our model essentially serves as an outcome-based sensitivity model [10]. This distinction is crucial as it enables our model to subsume the policy-based MSM in cases where the policy is confounded. Nonetheless, the reverse does not hold, and the policy-based MSM does not imply a transition kernel-based MSM for A > 2 . This point is further corroborated by [12], who explore the policy-based MSM within confounded MDPs and obtain non-sharp identification bounds when A > 2 . In contrast, our approach yields sharp identification in general, regardless of the number of actions and without placing assumptions on the behavior policy, which may or may not be confounded.

[13] also considered an MSM-like model in the transition kernel but their formulation assumes A = 2 . [40] operates under the setting of [12] and required tabular states. We note that all these works including ours considers i.i.d. confounders at each step, which translates to a robust MDP with ( s, a ) -rectangularity and ensures that the worst-case problem is still an MDP rather than a POMDP. The importance of this assumption was verified by [55], who showed that without it, the non-memoryless confounder can create exponential-in-horizon changes in value.

Neyman Orthogonality and Semiparametric Efficient Estimation. We leverage a body of research focusing on learning with nuisances functions (e.g., Q-functions) that we need to estimate from data but are not the primary target (e.g., policy value). Much of this research [7, 16, 17, 29, 64, 70, among others] aims to identify Neyman-orthogonal estimators, which are first order orthogonal (insensitive) to nuisance errors. This literature is tightly linked to the semiparametric efficient estimation literature since Neyman-orthogonal scores can arise naturally from efficient influence functions [33, 62]. Going beyond the no-horizon causal inference setting, some explore such estimators in off-policy sequential-decisions contexts [19, 38, 42, 48, 50]. Notably, [39] derive efficient influence functions and orthogonal estimation in standard, non-robust OPE in infinite-horizon RL, which coincides with our unconfounded no-uncertainty case ( Λ = 1 ).

Moving beyond point-identified settings, some works explore orthogonality and efficiency for partial identification and sensitivity analysis. In the causal inference literature, efficient/orthogonal estimation in the no-horizon setting has been studied extensively under several sensitivity models [10, 18, 24, 58]. Closest to our work is [24] who provide an orthogonal estimator and convergence rates under the MSM[66], which coincides with our setting under γ = 1 . In the sequential setting, [55] considers confounding at a single time step under the MSM, but their estimator is not orthogonal when the quantile function is unknown. [12] provide a fitted-Q-iteration learner with an orthogonalized loss function, but not orthogonal/efficient estimates of worst-case policy value.

2 Preliminaries

We consider an MDP with state space S , action space A , transition kernel P ( s ′ | s, a ) , reward function r ( s, a ) ∈ [0 , 1] and initial state distribution d 1 ∈ ∆( S ) . We do not require S or A to be finite. We assume r and d 1 are known for simplicity, and it is standard to extend our analysis to when they are unknown. We are given a dataset D of n i.i.d. tuples ( s i , a i , r i , s ′ i ) such that ( s i , a i ) ∼ ν , s ′ i ∼ P ( · | s, a ) and r i = r ( s i , a i ) , where ν is an arbitrary data-generating distribution. For discount factor γ ∈ [0 , 1) , let the Q function be the discounted cumulative rewards under a policy π : S → A , Q π,P ( s, a ) = E π,P [ ∑ ∞ t =0 γ t r t ( s t , a t ) | s 1 = s, a 1 = a ] . Similarly, define the value function as V π,P ( s ) = Q π,P ( s, π ) , where we use the notation f ( s, π ) := E a ∼ π ( s ) [ f ( s, a )] for any function f : S × A → R .

We are interested in estimating the value of a fixed target policy π t (a.k.a. evaluation policy) in an unobserved MDP with a feasible perturbed transition kernel U . We say U is a feasible perturbation of the observed, nominal kernel P if for all s, a, s ′ : we have

where Λ( s, a ) ∈ [1 , ∞ ) is a sensitivity parameter chosen by the practitioner. On the extremes, Λ = 1 corresponds to no-confounding ( i.e. , classic OPE setting) and Λ = ∞ corresponds to maximalconfounding ( i.e. , worst or best outcome). We denote the set of all feasible perturbations of P by

U ( P ) , which is an s, a -rectangular set [51]. We define the best- and worst-case Q functions of π t as

Thus, the goal of this paper is to estimate the best- and worst-case value of π t at the initial state,

where V ± ( s ) = E a ∼ π t ( s ) [ Q ± ( s, a )] and the ± symbol signals that an equation should be read twice, once with ± = + and once with ± = -. For clarity, we focus the discussion in the main text on estimating the worst-case policy value, V -d 1 . We provide a similar analysis for policy values under best-case perturbations ( V + d 1 ) in Appendix B.

Compared to standard OPE, robust OPE is more challenging since the best- and worst-case transition kernels U ± are unobserved as our dataset D is generated under P . For example, standard OPE is easy in the on-policy case i.e. , if D were generated by π t , but our problem is still 'off-data' and non-trivial.

Discounted Visitation Distributions. For any transition kernel U , define the discounted visitation distribution of π t under U as: d π t , ∞ d 1 ,U ( s ) := (1 -γ ) ∑ ∞ h =1 γ h -1 d π t ,h d 1 ,U ( s ) , where d π t ,h d 1 ,U ( s ) is the probability of reaching state s in the Markov chain induced by U and policy π t starting from d 1 ( · ) . We use d -, ∞ as shorthand for d π t , ∞ d 1 ,U -, where U -denotes the worst-case kernel in U ( P ) .

Bellman-type Operators. For any function f : S × A → R and transition kernel U , recall the Bellman operator is defined as T U f ( s, a ) := r ( s, a ) + γ E U [ f ( s ′ , π t ) | s, a ] . For robust OPE, we define the following robust analog T + rob f ( s, a ) := r ( s, a ) + γ sup U ∈U ( P ) E U [ f ( s ′ , π t ) | s, a ] and T -rob f ( s, a ) := r ( s, a ) + γ inf U ∈U ( P ) E U [ f ( s ′ , π t ) | s, a ] . Moreover, we define J U f ( s, a ) := γ E U [ f ( s ′ , π t ) | s, a ] -f ( s, a ) . For any linear operator T , also let T ′ denote its adjoint: that is, for all f, g ∈ L 2 ( ν ) , ⟨ f, T g ⟩ = ⟨T ′ f, g ⟩ , where ⟨· , ·⟩ is the inner product in L 2 ( ν ) .

Conditional Value-at Risk (CVaR). For a random variable X , its upper/lower CVaRs at level τ ∈ [0 , 1] is defined as the average outcome of the upper/lower τ -fraction of cases, and are formally defined as follows [61]:

where y + := max(0 , y ) and y -:= min(0 , y ) for y ∈ R . The optima are attained at the upper/lower τ -th quantile of X which we denote as β + τ ( X ) /β -τ ( X ) , i.e. ,

CVaR

- (

s

,

a

)

CVaR

+ (

s

,

a

)



[

v

(

s

0

) ∣

s

,

a

]

β

- (

s

,

a

)

β

+ (

s

,

a

)

v

(

s

0

) ∣

s

,

a

If X has a cumulative distribution function (CDF) which is differentiable at β ± τ ( X ) , its CVaRs simplify to CVaR + τ ( X ) = E [ X | X ≥ β + τ ( X )] and CVaR -τ ( X ) = E [ X | X ≤ β -τ ( X )] . In the paper, τ will often be set to (Λ + 1) -1 ∈ [0 , 0 . 5] .

Notations. We use x ≲ y to mean that x ≤ Cy holds for some universal constant C . The indicator function I [ p ] takes value 1 if p is true and 0 otherwise. For a measure µ , we let ∥ f ∥ µ := ( E µ | f ( X ) | 2 ) 1 / 2 denote the L 2 norm of f , provided it exists. When µ is clear from context, we also use ∥ f ∥ p := ( E | f ( X ) | p ) 1 /p to denote the L p norm of f and ∥ f ∥ p,n := ( E n | f ( X ) | p ) 1 /p to denote the empirical analog. For a data sample of size n , we define the empirical mean as E n [ f ( X )] = 1 n ∑ n i =1 f ( x i ) . For a nuisance function f , we reserve f ∗ as its true value and ̂ f as the learned value from data. Moreover, we employ + and -to denote functions corresponding to bestand worst-case bounds, respectively. See Appendix A for a comprehensive notation table.

2.1 Background: Non-robust OPE

We provide a quick primer on the double RL (DRL) estimator for classic OPE in non-robust MDPs [38], which combines estimates of the Q -function and density ratio w to achieve orthogonality, double robustness and semiparametric efficiency. This sets the stage for our orthogonal estimator

in Section 5, which generalizes DRL to robust MDPs by incorporating the robust Q -function and density ratio in the worst-case MDP, as described in Section 3 and Section 4 respectively.

The DRL estimator involves two nuisances: (1) q , for which the oracle (true value) is the Q -function of the target policy Q π t , and (2) w , for which the oracle is the density ratio of the target policy's visitation distribution and the data distribution w π t = d d π t , ∞ d 1 ,P / d ν . In this section, let η = ( w,q ) denote the DRL nuisances (outside this section, we use η to denote our robust estimator's nuisances) and let η ⋆ = ( w π t , Q π t ) denote their true values, then the recentered efficient influence function (EIF) of V π t d 1 in non-robust MDPs is given by:

The DRL estimator uses cross-fitting to learn nuisances ̂ η [ k ] on all data excluding the k -th fold D k , for k = 1 , 2 , . . . , K and estimates the OPE value via:

As we will see, this paves the way for the EIF of the robust value (Theorem 5.1) and our orthogonal estimator (Algorithm 3). There are two main guarantees for DRL: double robustness and semiparametric efficiency. Let r w n and r q n be rate functions depending on n = |D| such that ∥ ̂ q [ k ] -Q π t ∥ 2 ≤ r q n and ∥ ̂ w [ k ] -w π t ∥ 2 ≤ r w n . Then, DRL enjoys | ̂ V DRL d 1 -V π t d 1 | ≤ O p ( n -1 / 2 + r w n r q n ) , which confers the algorithm double robustness properties. Moreover, if Σ ope is the efficiency bound ( i.e. , minimum achievable asymptotic variance among RAL estimators in nonparametric models for ( s, a, s ′ ) ), then √ n ( ̂ V DRL d 1 -V π t d 1 ) d →N (0 , Σ ope ) . We seek similar guarantees for our orthogonal robust estimator.

3 Robust Q -Function Estimation with FittedQ Evaluation

In this section, we identify the robust Q -function using the robust Bellman equation and then derive convergence rates for iteratively minimizing the robust Bellman error.

3.1 Identification of the worst-case Q -function

The robust worst-case Q -function of π t , denoted as Q -, satisfies the robust Bellman equation Q -( s, a ) = T -rob Q -( s, a ) , ∀ s, a since the uncertainty set U ( P ) factorizes over s, a [34]. While these equations may seem intractable due to the inf in the definition of T -rob , [12] showed that T -rob has a closed form solution in terms of the CVaR under the observed kernel P .

-

Lemma 3.1 implies that Q is identified via the following equation of observable distributions: Q -( s, a ) = r ( s, a )+ γ Λ -1 ( s, a ) E [ Q -( s ′ , π t ) | s, a ]+ γ (1 -Λ -1 ( s, a )) CVaR -τ ( s,a ) [ Q -( s ′ , π t ) | s, a ] . Under no confounding ( Λ( s, a ) = 1 ), this recovers the classic Bellman equation.

3.2 Estimating the Robust Q -Function with Robust FQE

In this section, we estimate Q -via an iterative fitting algorithm based on fitted Q-evaluation (FQE) [54]. Our algorithm RobustFQE (Algorithm 1) proceeds for M iterations with two main steps in each iteration i . First, in Line 5, we estimate the lower-quantile of ̂ v i -1 ( s ′ ) | s, a . Here, we assume access to an oracle QR for quantile regression, which is a well-established problem, allowing for the use of various existing algorithms. Second, in Line 6, we solve the tractable robust Bellman equation in Lemma 3.1 with the CVaR term estimated by its orthogonal estimating equation with the learned quantiles [57]. By orthogonally estimating CVaR , we achieve second-order dependence on the quantile estimation errors from the first step. Next, we minimize the mean squared error using a general function class, Q ⊂ S × A ↦→ [0 , (1 -γ ) -1 ] .

To enable convergence guarantees, we make two assumptions. First, we assume that the quantile regression oracle has a specific convergence rate, which can be guaranteed under certain smoothness conditions [9, 14, 27, 28, 52, 60, 65]. Distributional RL may also be modified to learn quantiles of the next state value and have shown benefits in practice [21, 22] and in theory [5, 75, 76, 78].

Algorithm 1 RobustFQE : Iterative fitting for estimating Q -and β -τ .

1: Input: Number of iterations M , Dataset D of size n , Q -function class Q .

̂ 3: for i = 1 , 2 , . . . , M do

2: Initialize v -0 ( s ′ ) = 0 .

4: Set D i = D [ ni/M : n ( i +1) /M ] .

5: On the first half of D i , estimate the τ ( s, a ) lower quantile of ̂ v -i -1 ( s ′ ) , s ′ ∼ P ( · | s, a ) . Let ̂ β -i ( s, a ) denote the learned lower quantiles from the quantile regression oracle QR .

6: Using the second half of D i , solve the empirical robust Bellman equation by minimizing squared prediction error for the pseudo-outcome:

7: Output: ̂ q -M , ̂ β -M .

Assumption 3.2 (QR Oracle) . For any v : S ↦→ [0 , (1 -γ ) -1 ] , let the true τ ( s, a ) -quantile of v ( s ′ ) , s ′ ∼ P ( s, a ) be denoted by β v τ ( s, a ) . Given a dataset D QR , we assume QR outputs estimates ̂ β v with bounded ℓ ∞ error: for any δ , w.p. 1 -δ , ∥ ̂ β q -β q τ ∥ ∞ < err QR ( |D QR | , δ ) .

The second assumption is completeness under the robust Bellman T -rob . Completeness is a standard assumption in algorithms based on temporal-difference learning and without it, fitted-Q can diverge or converge to suboptimal fixed points [45, 68].

Assumption 3.3 (Completeness) . For all q ∈ Q , we have T -rob q ∈ Q .

We note that the current proofs of [12, 59] require a stronger completeness: T β q ∈ Q for all q ∈ Q and feasible β . We circumvent the need for the stronger 'allβ ' completeness by bounding model misspecification of least squares regression with second order error in the quantile regression.

Finally, we express our bounds with the critical radius ε Q n , a standard tool for deriving fast rates in statistics; see Appendix D.2 for a summary. Also, we denote the standard concentrability coefficient with C -d 1 := ∥ ∥ d d -, ∞ µ / d d 1 ∥ ∥ ∞ , a standard and necessary quantity for OPE.

Theorem 3.4. Let ε Q n denote the critical radius of Q . Under Assumptions 3.2 and 3.3, RobustFQE ensures that for any δ ∈ (0 , 1) , w.p. 1 -δ ,

For parametric classes ( e.g. , finite or linear), the critical radius converges at the standard ˜ O ( n -1 / 2 ) rate. Due to the orthogonal estimation of CVaR, we benefit from a favorable second-order dependence on err QR which allows for quantile regression to converge at slower ˜ O ( n -1 / 4 ) rates. The main disadvantage of this direct approach is that it converges at a slow sub-√ n rate if ε Q n converges at a sub-√ n , e.g. , ε Q n converges at a ˜ O ( n -1 / 4 ) rate if Q is nonparametric with metric entropy at most 1 /t 2 [71]. In Section 5, we present an orthogonal estimator that is both robust to slower rates of Q and achieves semiparametric efficiency.

4 Robust w -Function Estimation with Minimax Learning

Before we present our orthogonal estimator, we study another essential nuisance function: the robust visitation density ratio, i.e. , the robust w -function [2, 39]. In this section, we first identify the worstcase transition kernel U -in our uncertainty set U ( P ) . Then, we propose a minimax estimator [69] for the robust w -function, an important nuisance function for our orthogonal estimator in Section 5.

Identification of U -. The robust transition kernel U -is defined as the feasible perturbed kernel that achieves the inf in the robust Bellman equation Q -( s, a ) = T -rob Q -( s, a ) . Let F -( y | s, a ) =

Algorithm 2 RobustMIL : Minimax Estimation of w ± with a Stabilizer

1: Input: Dataset D , prior stage estimate ˜ ζ , function classes W , F , stabilizer weight λ > 0 .

2: Define weights ξ -( s, a, s ′ ) := Λ -1 ( s, a ) + (1 -Λ -1 ( s, a )) τ -1 ( s, a ) I [ ˜ ζ ( s, a, s ′ ) ≤ 0] .

3: Output:

P ( V -( s ′ ) ≤ y | s, a ) be the next-state pushforward measure of the robust value function V -. Then, U -is a convex combination of the nominal kernel P and a reweighting of P by an indicator function.

Lemma 4.1. Suppose F -( β -τ ( s, a ) | s, a ) = τ , where β -τ ( s, a ) is the lower τ -th quantile of F -( · | s, a ) . Then,

The proof strategy decomposes U -into its nominal and perturbed components, leveraging the primal solution of CVaR τ [3]; we formalize this in Appendix E.2.

Identification of w -. Using the identification of U -in Lemma 4.1, we can now identify the robust w -function based on the Bellman flow equations in the worst-case MDP. The Bellman flow in the robust MDP is given by d -, ∞ ( s ) = (1 -γ ) d 1 ( s ) + γ E ˜ s ∼ d -, ∞ , ˜ a ∼ π t ( ˜ s ) U -( s | ˜ s, ˜ a ) . where d -, ∞ ( s ) was defined in Section 2. Thus, the robust visitation density, defined as w -( s ) := d d -, ∞ ( s ) / d ν ( s ) , satisfies the following moment condition for all f : S ↦→ R :

where we relaxed notation and defined w -( s, a ) := w ( s ) · π t ( a | s ) /ν ( a | s ) . As before, in the unconfounded base ( Λ = 1 ), this result recovers the classic Bellman flow.

4.1 Estimating w -with Robust Minimax Indirect Learning

We now propose a penalized minimax estimator for w -that generalizes the Minimax Indirect Learning (MIL) of [69] to our robust MDP setting. Our estimator, RobustMIL (Algorithm 2), leverages a general function class W ⊂ S × A ↦→ R + to approximately solve the moment equation in Eq. (5). It does so by minimizing the difference between the left- and right-hand sides of the equation across a sufficiently large set of adversaries f in a discriminator class F ⊂ S × A ↦→ R . Since U -is unknown, we approximate it via Eq. (4) by plugging in a threshold ˜ ζ ( s, a, s ′ ) in the indicator function to approximate the true threshold ζ -( s, a, s ′ ) := V -( s ′ ) -β -τ ( s,a ) ( s, a ) . This yields the minimax objective in Eq. (6), where we also allow for an optional regularization of the adversary's norm which can be useful for obtaining fast convergence rates.

We make the following assumptions for MIL [69]. The first is a regularity condition that (i) our function class has bounded outputs and (ii) ζ is continuously distributed around the threshold.

Assumption 4.2 (Regularity) . (i) sup w ∈W∪{ w -} ∥ w ∥ ∞ < ∞ ; (ii) the marginal CDF of V -( s ′ ) -β -( s, a ) , i.e. , F ( y ) = P ( V -( s ′ ) -β -τ ( s,a ) ( s, a ) ≤ y ) , is boundedly differentiable around 0 .

If next-value distribution is discrete, we can use the discrete form of CVaR and (ii) can be removed.

The second is that the adversary class is rich enough to capture all projected errors under the adjoint of the operator J U -f ( s, a ) := γ E U -[ f ( s ′ , π t ) | s, a ] -f ( s, a ) .

Assumption 4.3 ( w --realizability and completeness) . w -∈ W and J ′ U -( Ww -) ⊂ F .

We note that Assumption 4.3 is monotone in the function class size and can be satisfied by making the function class more expressive, e.g. , increasing size of the neural net. Our algorithms are also robust to violations in Assumption 4.3, which we show in Appendix G.

We are now ready to state the main estimation result for w -in terms of the critical radius (Appendix D.2) of the function class.

Algorithm 3 Orthogonal Estimator for V

1: Input: Dataset D , number of splits K . 2: for k = 1 , 2 , . . . , K do 3: Use data D\D k to learn ( q -, [ k ] , β -, [ k ] ) with Algorithm 1 and w -, [ k ] with Algorithm 2 4: for i = ⌊ ( k -1) n/K ⌋ , . . . , ⌊ kn/K ⌋ -1 do ψ -i = ψ ( s i , a i , s ′ i , ̂ η -) 5: Output: ̂ V -d 1 = 1 n ∑ n i =1 ψ -i .

Theorem 4.4. Let ε W n denote the maximum critical radii of the following classes:

Under Assumptions 4.2 and 4.3, RobustMIL ensures that for any δ , w.p. 1 -δ ,

As before, the critical radius ε W n converges at an ˜ O ( n -1 / 2 ) rate for parametric classes. Notably, our bounds degrade linearly w.r.t. the ℓ ∞ error in ˜ ζ -for estimating ζ -. For example, if ˜ ζ ( s, a, s ′ ) = ̂ v ( s ′ ) -̂ β ( s, a ) where ̂ v, ̂ β are estimated with RobustFQE , then the ζ -error can be bounded by O ( ∥ ̂ v -v -∥ ∞ + ∥ ̂ β -β -∥ ∞ ) . Wepresent the full proof in Appendix G, where we also present a more general result that is robust to misspecifications to realizability and completeness (Assumption 4.3).

5 Orthogonal and Efficient Estimator for Robust Policy Value

In this section, we propose an orthogonal estimator that is robust against errors in the nuisances (exhibiting only second-order sensitivity), achieves semiparametric efficiency, and enables inference. Our estimator is based on the efficient influence function (EIF) of V -d 1 , which is the canonical gradient of a statistical estimand [67]. The adoption of EIFs for developing efficient estimators is a broadly employed technique in causal inference [16, 43] and reinforcement learning [35, 39].

We define the collection of nuisance parameters by η -= ( w -, q -, β -) . The notation ̂ η indicates that these functions are estimated from data, while the notation η denotes their true values.

Theorem 5.1 ((Recentered) Efficient Influence Function) . The (R)EIF of V -d 1 is given by:

Remark 5.2 . When Λ = 1 , there is no shift in the target environment, and the weight on the CVaR term is zero. The (R)EIF then reduces to the (R)EIF in [39] for regular OPE with an infinite horizon. As Λ →∞ , the CVaR term becomes predominant, with the quantile β -( s, a ) taking extreme values. This yields the (novel) (R)EIF for the problem in [25], where the expected value term is replaced solely by a CVaR component in the Bellman equation.

The (R)EIF forms the basis of our orthogonal estimator. First, we note that E [ ψ ( s, a, s ′ ; η -)] is an unbiased estimator of V -d 1 . Furthermore, the expression for ψ ( s, a, s ′ ; η -) depends only on quantities w -, q -, β -which can be estimated from data. Thus, we can cast the expression E [ ψ ( s, a, s ′ ; η -)] as a statistical estimand to be learned from the observed sample. This suggests a natural two-stage estimator that we summarize in Algorithm 3. In the first stage, we estimate the nuisance parameters ̂ η from the data with K -fold cross-fitting; in the second stage, these estimates are incorporated into the (R)EIF expression and we calculate the empirical average using the observed data. We summarize our procedure in Algorithm 3.

The nuisance estimation is detailed in Sections 3.2 and 4.1. The reliance on the EIF confers our estimator desirable statistical properties including a second order bias due to the nuisances, meaning the bias has a product structure with respect to the nuisance errors. Thus, this special structure orthogonalizes away the dependency on ̂ Q -errors which now only appear in second order. Furthermore, our estimator is semiparametrically efficient in the sense that under mild consistency assumptions, it achieves minimum variance among all regular and asymptotically linear (RAL) estimators. We provide theoretical justifications for these properties in the next section.

5.1 Theoretical Guarantees of the Orthogonal Estimator

We now characterize the theoretical properties of our orthogonal estimator. We consider the K -fold cross-fitted estimator in Algorithm 3 given by

where nuisances ̂ η [ k ] , k ∈ [ K ] are trained on all data excluding the k th fold D k . The following theorem outlines the theoretical guarantees of this estimator:

Theorem 5.3 (Efficiency of ̂ V -d 1 ) . Let r w n,p , r q n,p , r β n,p be functions of the same size n = |D| such that ∥J ′ U -( ̂ w -, [ k ] -w ) ∥ p ≤ r w n,p , ∥ ̂ q -, [ k ] -q ∥ p ≤ r q n,p , and ∥ β -, [ k ] -β ∥ p ≤ r β n,p for any k ∈ [ K ] . Furthermore, assume that the regularity conditions in Assumption 4.2 hold. Then:

Furthermore, if r w n, 2 ∨ r q n, 2 = o p (1) , r w n, 2 r q n, 2 = o p ( n -1 / 2 ) , r q n, ∞ = o p ( n -1 / 4 ) , and r β n, ∞ = o p ( n -1 / 4 ) , then ̂ V -d 1 satisfies:

Moreover, Σ is the minimum achievable asymptotic variance among RAL estimators in the nonparametric model for ( s, a, s ′ ) (the efficiency bound).

We provide the intuition along with a detailed proof in Appendix H. The first part of Theorem 5.3 implies that as long as we estimate the nuisances at rates faster that n -1 / 4 , then we can learn ̂ V -d 1 at parametric rates. The second part of Theorem 5.3 states that under mild consistency assumptions, our estimator attains the efficiency bound and is asymptotically normal. That means, for example, we can construct asymptotically valid lower 95%-confidence bound on ̂ V -d 1 by simply subtracting 1.64 times ˆ se = 1 n ( ∑ K k =1 ∑ ( s,a,s ′ ) ∈D k ( ψ ( s, a, s ′ ; ̂ η [ k ] ) -̂ V -d 1 ) 2 ) 1 / 2 . Then, we can be sure to have a bound on the worst-case RL policy value, accounting both for potential environment shift and finite data. Finally, in Appendix J, we describe two settings when our orthogonal estimator remains valid even if some nuisances are inconsistent , which is a desirable guarantee for sensitivity analysis [23].

Bringing it all together. We can instantiate Theorem 5.3 with the nuisance estimators from the previous sections. First, use RobustFQE to estimate ̂ q -and ̂ β -, ensuring ∥ ̂ q --Q -∥ 2 ≤ O ( ε Q n + err 2 QR ) . Under smoothness conditions (Lemma D.2), the L 2 guarantee for ̂ q -implies an L ∞ guarantee for ̂ q -, which also ensures an L ∞ guarantee for ̂ β -. This ensures max( ∥ ̂ q --Q -∥ ∞ , ∥ ̂ β --β -∥ ∞ ) is well-controlled. Then, we can set ˜ ζ -( s, a, s ′ ) = ̂ q -( s ′ , π t ) -̂ β -( s, a ) and run RobustMIL for estimating ̂ w -. By Theorem 4.4, its projectedL 2 error is O ( ε W n + ∥ ̂ q --Q -∥ ∞ + ∥ ̂ β --β -∥ ∞ ) . Therefore, the final rate via Theorem 5.3 is O (( ε Q n + err 2 QR ) · ε W n + ∥ ̂ q --Q -∥ 2 ∞ + ∥ ̂ β --β -∥ 2 ∞ ) .

6 Empirical Evaluation

We now provide a proof-of-concept empirical investigation to validate our theoretical findings. We experiment with our proposed methodology in a simple synthetic environment. First, we discuss our environment, followed by our approach for solving for the nuisances functions η -. Then, we provide empirical results for our orthogonal estimator, and compare its performance to weighted or direct estimators using the Q -or w -nuisances only. The code for our experiments is open-sourced and available at https://github.com/CausalML/adversarial-ope/ .

Experimental Setup We consider a synthetic MDP with a one-dimensional state and two actions, modeled after a simple control problem with non-deterministic dynamics. The task is to estimate the worst-case policy value V -d 1 of a fixed candidate policy π t , across four different constant values of the sensitivity parameter: Λ( s, a ) ∈ { 1 , 2 , 4 , 8 } .

1

2

4

8

(

s

,

a

)

0.2

0.4

0.6

0.8

Estimated

Vd

1

estimator

Q

W

Orth

Λ

Mean squared error (MSE) to true worst-case policy value

Q

W

Orth

1

.

000240

±

.

000170

.

002722

±

.

002266

.

000005

±

.

000006

2

.

004053

±

.

001329

.

003584

±

.

002311

.

003244

±

.

000600

4

.

009799

±

.

005172

.

013862

±

.

009228

.

008721

±

.

002543

8

.

006247

±

.

003980

.

052643

±

.

050839

.

005713

±

.

002730

We considered three methods for estimating the robust value V -d 1 :

Q ( RobustFQE ): Direct method using the estimated robust quality function ̂ Q -only.

̂ 3. Orth : Our orthogonal estimator which combines the former two, as described in Algorithm 3.

W ( RobustMIL ): Importance-sampling method using the estimated robust density ratio w -only.

We performed 10 replications of our experimental procedure, where for each replication we: (1) sampled a dataset of 20,000 tuples using a different fixed logging policy π b ; (2) fit the nuisance functions Q -, β -, and w -following the method outlined in Algorithms 1 and 2 for each Λ ; and (3) estimated the corresponding robust policy value V -d 1 for all estimators using the fitted nuisances.

Results We summarize our results in Fig. 2. We note that all of our estimators are consistently valid for all values of Λ in our experiment. Notably, Orth consistently has the lowest mean squared error for the true worst-case policy value. In particular, incorporating the robust importance-sampling weights improves the RobustFQE estimator Q , even though these importance-sampling weights by themselves (as in W ) are much noisier estimators. This is consistent with our theory that the orthogonal estimator is semiparametrically efficient and insensitive to errors in the nuisance functions.

Full experimental details, including our MDP, target/logging policies, methodology for computing the true robust policy values V -d 1 , and nuisance estimation, are provided in Appendix K. Finally, we also performed an empirical evaluation in the real-world medical problem of sepsis management using the MIMIC-III dataset [36]. We detail these results in Appendix L.

7 Conclusion

We consider the problem of infinite-horizon OPE in RL settings when there can be unknown, but bounded, shifts in the transition distribution compared to the transition distribution generating the data. This can arise due to unobserved confounding, where observed transitions do not reflect the true causal ones, non-stationarity in the environment, or adversarial environments. We propose a sensitivity model for such transition kernel shifts analogous to the classic MSM for static decision making, and provide theoretical guarantees for identifying and estimating the sharp ( i.e. , tightest possible) bounds on the best/worst-case policy value, as well as the corresponding robust Q -function and state density ratio functions. Our estimator for the best/worst-case policy value is orthogonal (insensitive to how the nuisance functions are estimated) and achieves semiparametric efficiency (attaining the best possible asymptotic variance). Finally, our estimator also supports inference, ensuring we can derive reliable bounds for the robust policy value even with finite data.

References

Alekh Agarwal, Yuda Song, Wen Sun, Kaiwen Wang, Mengdi Wang, and Xuezhou Zhang. Provable benefits of representational transfer in reinforcement learning. In The Thirty Sixth Annual Conference on Learning Theory , pages 2114-2187. PMLR, 2023.

Philip Amortila, Dylan J Foster, Nan Jiang, Ayush Sekhari, and Tengyang Xie. Harnessing density ratios for online reinforcement learning. arXiv preprint arXiv:2401.09681 , 2024.

Marcus Ang, Jie Sun, and Qiang Yao. On the dual representation of coherent risk measures. Annals of Operations Research , 262:29-46, 2018.

Jean-Yves Audibert and Alexandre B Tsybakov. Fast learning rates for plug-in classifiers under the margin condition. arXiv preprint math/0507180 , 2005.

Alex Ayoub, Kaiwen Wang, Vincent Liu, Samuel Robertson, James McInerney, Dawen Liang, Nathan Kallus, and Csaba Szepesvári. Switching the loss reduces the cost in batch reinforcement learning. International Conference of Machine Learning , 2024.

Kishan Panaganti Badrinath and Dileep Kalathil. Robust reinforcement learning using least squares policy iteration with provable performance guarantees. In International Conference on Machine Learning , pages 511-520. PMLR, 2021.

Alexandre Belloni, Victor Chernozhukov, Ivan Fernandez-Val, and Christian Hansen. Program evaluation and causal inference with high-dimensional data. Econometrica , 85(1):233-298, 2017.

Andrew Bennett, Nathan Kallus, and Miruna Oprescu. Low-rank mdps with continuous action spaces. arXiv preprint arXiv:2311.03564 , 2023.

Pallab K Bhattacharya and Ashis K Gangopadhyay. Kernel and nearest-neighbor estimation of a conditional quantile. The Annals of Statistics , pages 1400-1415, 1990.

Matteo Bonvini, Edward Kennedy, Valerie Ventura, and Larry Wasserman. Sensitivity analysis for marginal structural models. arXiv preprint arXiv:2210.04681 , 2022.

Haïm Brezis and Petru Mironescu. Where sobolev interacts with gagliardo-nirenberg. Journal of functional analysis , 277(8):2839-2864, 2019.

David Bruns-Smith and Angela Zhou. Robust fitted-q-evaluation and iteration under sequentially exogenous unobserved confounders. arXiv preprint arXiv:2302.00662 , 2023.

David A Bruns-Smith. Model-free and model-based policy evaluation when causality is uncertain. In International Conference on Machine Learning , pages 1116-1126. PMLR, 2021.

Domagoj ´ Cevid, Loris Michel, Jeffrey Näf, Nicolai Meinshausen, and Peter Bühlmann. Distributional random forests: Heterogeneity adjustment and multivariate distributional regression. arXiv preprint arXiv:2005.14458 , 2020.

Jonathan Chang, Kaiwen Wang, Nathan Kallus, and Wen Sun. Learning bellman complete representations for offline policy evaluation. In International Conference on Machine Learning , pages 2938-2971. PMLR, 2022.

Victor Chernozhukov, Denis Chetverikov, Mert Demirer, Esther Duflo, Christian Hansen, Whitney Newey, and James Robins. Double/debiased machine learning for treatment and structural parameters. The Econometrics Journal , 21(1):C1-C68, 2018. doi: 10.1111/ectj.12097.

Victor Chernozhukov, Mert Demirer, Esther Duflo, and Ivan Fernandez-Val. Generic machine learning inference on heterogeneous treatment effects in randomized experiments, with an application to immunization in india. Technical report, National Bureau of Economic Research, 2018.

Victor Chernozhukov, Carlos Cinelli, Whitney Newey, Amit Sharma, and Vasilis Syrgkanis. Long story short: Omitted variable bias in causal machine learning. Technical report, National Bureau of Economic Research, 2022.

Victor Chernozhukov, Whitney Newey, Rahul Singh, and Vasilis Syrgkanis. Automatic debiased machine learning for dynamic treatment effects and general nested functionals. arXiv preprint arXiv:2203.13887 , 2022.

Yinlam Chow, Aviv Tamar, Shie Mannor, and Marco Pavone. Risk-sensitive and robust decisionmaking: a cvar optimization approach. Advances in neural information processing systems , 28, 2015.

Will Dabney, Georg Ostrovski, David Silver, and Rémi Munos. Implicit quantile networks for distributional reinforcement learning. In International conference on machine learning , pages 1096-1105. PMLR, 2018.

Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Jacob Dorn and Kevin Guo. Sharp sensitivity analysis for inverse propensity weighting via quantile balancing. Journal of the American Statistical Association , 118(544):2645-2657, 2023.

Jacob Dorn, Kevin Guo, and Nathan Kallus. Doubly-valid/doubly-sharp sensitivity analysis for causal inference with unmeasured confounding. arXiv preprint arXiv:2112.11449 , 2021.

Yihan Du, Siwei Wang, and Longbo Huang. Provably efficient risk-sensitive reinforcement learning: Iterated cvar and worst path. In The Eleventh International Conference on Learning Representations , 2022.

Yaqi Duan, Chi Jin, and Zhiyuan Li. Risk bounds and rademacher complexity in batch reinforcement learning. In International Conference on Machine Learning , pages 2892-2902. PMLR, 2021.

Anouar El Ghouch and Marc G Genton. Local polynomial quantile regression with parametric features. Journal of the American Statistical Association , 104(488):1416-1429, 2009.

Kevin Elie-Dit-Cosaque and Véronique Maume-Deschamps. Random forest estimation of conditional distribution functions and conditional quantiles. Electronic Journal of Statistics , 16 (2):6553-6583, 2022.

Dylan J Foster and Vasilis Syrgkanis. Orthogonal statistical learning. The Annals of Statistics , 51(3):879-908, 2023.

Vineet Goyal and Julien Grand-Clement. Robust markov decision processes: Beyond rectangularity. Mathematics of Operations Research , 48(1):203-226, 2023.

Oliver Hines, Oliver Dukes, Karla Diaz-Ordaz, and Stijn Vansteelandt. Demystifying statistical learning based on efficient influence functions. The American Statistician , 76(3):292-304, 2022.

Ronald A Howard and James E Matheson. Risk-sensitive markov decision processes. Management science , 18(7):356-369, 1972.

Hidehiko Ichimura and Whitney K Newey. The influence function of semiparametric estimators. Quantitative Economics , 13(1):29-61, 2022.

Garud N Iyengar. Robust dynamic programming. Mathematics of Operations Research , 30(2): 257-280, 2005.

Nan Jiang and Lihong Li. Doubly robust off-policy value evaluation for reinforcement learning. In International Conference on Machine Learning , pages 652-661. PMLR, 2016.

Alistair EW Johnson, Tom J Pollard, Lu Shen, Li-wei H Lehman, Mengling Feng, Mohammad Ghassemi, Benjamin Moody, Peter Szolovits, Leo Anthony Celi, and Roger G Mark. Mimic-iii, a freely accessible critical care database. Scientific data , 3(1):1-9, 2016.

Nathan Kallus. What's the harm? sharp bounds on the fraction negatively affected by treatment. Advances in Neural Information Processing Systems , 35:15996-16009, 2022.

Nathan Kallus and Masatoshi Uehara. Double reinforcement learning for efficient off-policy evaluation in markov decision processes. The Journal of Machine Learning Research , 21(1): 6742-6804, 2020.

Nathan Kallus and Masatoshi Uehara. Efficiently breaking the curse of horizon in off-policy evaluation with double reinforcement learning. Operations Research , 70(6):3282-3302, 2022.

Nathan Kallus and Angela Zhou. Confounding-robust policy evaluation in infinite-horizon reinforcement learning. Advances in neural information processing systems , 33:22293-22304, 2020.

Nathan Kallus, Xiaojie Mao, Kaiwen Wang, and Zhengyuan Zhou. Doubly robust distributionally robust off-policy evaluation and learning. In International Conference on Machine Learning , pages 10598-10632. PMLR, 2022.

Edward H Kennedy. Nonparametric causal effects based on incremental propensity score interventions. Journal of the American Statistical Association , 114(526):645-656, 2019.

Edward H Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. arXiv preprint arXiv:2004.14497 , 2020.

Amirhossein Kiani, Chris Wang, and Angela Xu. Sepsis world model: A mimic-based openai gym" world model" simulator for sepsis treatment. arXiv preprint arXiv:1912.07127 , 2019.

J Kolter. The fixed points of off-policy td. Advances in Neural Information Processing Systems , 24, 2011.

Navdeep Kumar, Kfir Levy, Kaixin Wang, and Shie Mannor. Efficient policy iteration for robust markov decision processes via regularization. arXiv preprint arXiv:2205.14327 , 2022.

Navdeep Kumar, Esther Derman, Matthieu Geist, Kfir Yehuda Levy, and Shie Mannor. Policy gradient for rectangular robust markov decision processes. In Thirty-seventh Conference on Neural Information Processing Systems , 2023. URL https://openreview.net/forum?id= NLpXRrjpa6 .

Mark J Laan and James M Robins. Unified methods for censored longitudinal data and causality . Springer, 2003.

Giovanni Leoni. A first course in Sobolev spaces . American Mathematical Soc., 2017.

Greg Lewis and Vasilis Syrgkanis. Double/debiased machine learning for dynamic treatment effects. In NeurIPS , pages 22695-22707, 2021.

Shie Mannor, Ofir Mebel, and Huan Xu. Robust mdps with k-rectangular uncertainty. Mathematics of Operations Research , 41(4):1484-1509, 2016.

Nicolai Meinshausen and Greg Ridgeway. Quantile regression forests. Journal of machine learning research , 7(6), 2006.

Volodymyr Mnih. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602 , 2013.

Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research , 9(5), 2008.

Hongseok Namkoong, Ramtin Keramati, Steve Yadlowsky, and Emma Brunskill. Off-policy policy evaluation for sequential decisions under unobserved confounding. Advances in Neural Information Processing Systems , 33:18819-18831, 2020.

Arnab Nilim and Laurent El Ghaoui. Robust control of markov decision processes with uncertain transition matrices. Operations Research , 53(5):780-798, 2005.

Tomasz Olma. Nonparametric estimation of truncated conditional expectation functions. arXiv preprint arXiv:2109.06150 , 2021.

Miruna Oprescu, Jacob Dorn, Marah Ghoummaid, Andrew Jesson, Nathan Kallus, and Uri Shalit. B-learner: Quasi-oracle bounds on heterogeneous causal effects under hidden confounding. In International Conference on Machine Learning , pages 26599-26618. PMLR, 2023.

Kishan Panaganti, Zaiyan Xu, Dileep Kalathil, and Mohammad Ghavamzadeh. Robust reinforcement learning using offline data. Advances in neural information processing systems , 35: 32211-32224, 2022.

Jeffrey S Racine and Kevin Li. Nonparametric conditional quantile estimation: A locally weighted quantile kernel approach. Journal of Econometrics , 201(1):72-94, 2017.

R Tyrrell Rockafellar and Stanislav Uryasev. Conditional value-at-risk for general loss distributions. Journal of banking & finance , 26(7):1443-1471, 2002.

Anton Schick. On asymptotically efficient estimation in semiparametric models. The Annals of Statistics , pages 1139-1151, 1986.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Vira Semenova and Victor Chernozhukov. Debiased machine learning of conditional average treatment effects and other causal functions. The Econometrics Journal , 24(2):264-289, 2021.

Ichiro Takeuchi, Quoc V Le, Timothy D Sears, Alexander J Smola, and Chris Williams. Nonparametric quantile estimation. Journal of machine learning research , 7, 2006.

Zhiqiang Tan. A distributional approach for causal inference using propensity scores. Journal of the American Statistical Association , 101(476):1619-1637, 2006.

Anastasios A Tsiatis. Semiparametric theory and missing data , volume 4. Springer, 2006.

John Tsitsiklis and Benjamin Van Roy. Analysis of temporal-diffference learning with function approximation. Advances in neural information processing systems , 9, 1996.

Masatoshi Uehara, Masaaki Imaizumi, Nan Jiang, Nathan Kallus, Wen Sun, and Tengyang Xie. Finite sample analysis of minimax offline reinforcement learning: Completeness, fast rates and first-order efficiency. arXiv preprint arXiv:2102.02981 , 2021.

Mark J van der Laan, Sherri Rose, Wenjing Zheng, and Mark J van der Laan. Cross-validated targeted minimum-loss-based estimation. Targeted learning: causal inference for observational and experimental data , pages 459-474, 2011.

Aad W Van der Vaart. Asymptotic statistics , volume 3. Cambridge university press, 2000.

Martin J Wainwright. High-dimensional statistics: A non-asymptotic viewpoint , volume 48. Cambridge university press, 2019.

Jie Wang, Rui Gao, and Hongyuan Zha. Reliable off-policy evaluation for reinforcement learning. Operations Research , 72(2):699-716, 2024.

Kaiwen Wang, Nathan Kallus, and Wen Sun. Near-minimax-optimal risk-sensitive reinforcement learning with cvar. In International Conference on Machine Learning , pages 35864-35907. PMLR, 2023.

Kaiwen Wang, Kevin Zhou, Runzhe Wu, Nathan Kallus, and Wen Sun. The benefits of being distributional: Small-loss bounds for reinforcement learning. Advances in Neural Information Processing Systems , 36, 2023.

Kaiwen Wang, Nathan Kallus, and Wen Sun. The central role of the loss function in reinforcement learning. arXiv preprint arXiv:2409.12799 , 2024.

Kaiwen Wang, Dawen Liang, Nathan Kallus, and Wen Sun. Risk-sensitive rl with optimized certainty equivalents via reduction to standard rl. arXiv preprint arXiv:2403.06323 , 2024.

Kaiwen Wang, Owen Oertell, Alekh Agarwal, Nathan Kallus, and Wen Sun. More benefits of being distributional: Second-order bounds for reinforcement learning. International Conference of Machine Learning , 2024.

Yue Wang and Shaofeng Zou. Online robust reinforcement learning with model uncertainty. Advances in Neural Information Processing Systems , 34:7193-7206, 2021.

Wolfram Wiesemann, Daniel Kuhn, and Berç Rustem. Robust markov decision processes. Mathematics of Operations Research , 38(1):153-183, 2013.

Wenhao Xu, Xuefeng Gao, and Xuedong He. Regret bounds for Markov decision processes with recursive optimized certainty equivalents. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett, editors, Proceedings of the 40th International Conference on Machine Learning , volume 202 of Proceedings of Machine Learning Research , pages 38400-38427. PMLR, 23-29 Jul 2023. URL https://proceedings.mlr.press/v202/xu23d.html .

A Notations

Appendices

S , A

State and action spaces.

∆(

S

)

The set of distributions supported by set S .

d

1

The initial state distribution.

Λ(

s, a

)

Tolerance parameter for kernel shift at ( s, a ) . Takes values [1 , ∞ ] .

V

±

, Q

±

Robust value and quality functions of the target policy π t .

U

±

(

s

′

|

s, a

)

Robust transition kernel which attains the best- or worst-case value.

T

U

,

T

±

Bellman operator under U and the robust Bellman operators.

J

U

β

±

τ

(

s, a

)

The upper τ -th quantile of V + ( s ′ ) and lower τ -th quantile of V - ( s ′ ) , s ′ ∼ P ( s, a ) .

d

π

t

,

∞

d

1

,U

The γ -discounted average visitation of π t under MDP with transition U starting from d 1 . d ± , ∞ = d π t , ∞ d 1 ,U ± .

d

±

,

∞

ν

(

s

)

, ν

(

s, a

)

Data generating distribution. ν ( s ) marginalizes over actions.

w

±

w ± = d d ± , ∞ / d ν . This is valid both as a function of s or ( s, a ) .

ω

(

s, a

)

ω

(

s, a

) =

π

t

(

a

|

s

)

ν

(

a

|

s

)

.

x

+

, x

-

max(0 , x ) , min(0 , x ) respectively, for x ∈ R .

x

≲

y

x ≤ Cy for some constant C .

E

n

Empirical average over n samples.

∥

f

∥

p

L

p

norm,

(

E

|

f

(

X

)

|

p

)

1

/p

.

f

⋆

True (oracle) value of a parameter or function f .

f,

¯

f

Putative value of a parameter or function f .

̂

f

Estimated value of a parameter or function f .

B Results for Policy Evaluation Under Best-Case Perturbations

In this section, we present analogous results for the best-case perturbation under the uncertainty set, corresponding to the supremum case of Eq. (2). We derive a similar orthogonal estimator with the properties outlined in Theorem 5.3, following the same reasoning presented in the main text.

Q + Identification and Estimation. We present the results of Lemma 3.1 for T + rob :

Next, applying Assumption 3.2 and Assumption 3.3 to T + rob , we derive from Theorem 3.4 for Q -that:

w + Identification and Estimation. We first state the identification result for U -as in Lemma 4.1:

Algorithm 4 Orthogonal Estimator for V + d 1

1: Input: Dataset D , number of splits K .

2: for k = 1 , 2 , . . . , K do

3: Use data D\D k to learn ( q + , [ k ] , β + , [ k ] ) with Algorithm 1 and w + , [ k ] with Algorithm 2

4: for i = ⌊ ( k -1) n/K ⌋ , . . . , ⌊ kn/K ⌋ -1 do ψ + i = ψ ( s i , a i , s ′ i , ̂ η + )

5: Output: ̂ V + d 1 = 1 n ∑ n i =1 ψ + i .

Then, under Assumption 4.2 and Assumption 4.3 formulated for U + , the minimax rates from Theorem 4.4 are given by:

Orthogonal and Efficient Estimator for V + d 1 . Let the set of nuisance parameters be denoted by η + = ( w + , q + , β + ) . Then, the (recentered) efficient influence function (R)EIF (see Theorem 5.1) for in V + d 1 is formulated as:

Using this (R)EIF, the orthogonal estimator for V + d 1 is presented in Algorithm 4. We now restate Theorem 5.3 for ̂ V + d 1 :

Theorem B.1 (Efficiency of ̂ V + d 1 ) . Let r w n,p , r q n,p , r β n,p be functions of n = |D| such that ∥J ′ U + ( ̂ w + , [ k ] -w ∗ ) ∥ p ≤ r w n,p , ∥ ̂ q + , [ k ] -q ∗ ∥ p ≤ r q n,p , and ∥ β + , [ k ] -β ∗ ∥ p ≤ r β n,p for any k ∈ [ K ] . Furthermore, assume that the regularity conditions in Assumption 4.2 hold. Then:

Furthermore, if r w n, 2 ∨ r q n, 2 = o p (1) , r w n, 2 r q n, 2 = o p ( n -1 / 2 ) , r q n, ∞ = o p ( n -1 / 4 ) , and r β n, ∞ = o p ( n -1 / 4 ) , then ̂ V + d 1 satisfies:

Moreover, Σ is the minimum achievable asymptotic variance among RAL estimators in the nonparametric model for ( s, a, s ′ ) (the efficiency bound).

C Additional Related Works

Robust MDPs. There is a rich literature on Robust MDPs [30, 34, 51, 80] with s, a -rectangular uncertainty sets, but these foundational works assumed knowledge of the transition kernel. Recently, learning-based robust MDP algorithms have been proposed for uncertainty sets under the total variation [47, 59] and more generally L p balls [46]. These L p uncertainty sets are additive in nature, i.e. , the adversary adds or subtracts a vector in the ℓ p ball to P ( · | s, a ) , whereas our uncertainty set is multiplicative in nature, i.e. , the adversary can multiply or divide a bounded factor and is more commonly used in causal inference to model unobserved confounding. In the contextual bandit setting, [41] also derived efficiency bounds for robust OPE where both state distribution and reward distributions may shift - their work is however restricted to the one-step bandit setting while our full RL setting is more challenging.

Risk-Sensitive RL. Risk-sensitive RL is the problem of optimizing the risk measure of cumulative rewards [32] and is tightly related to robust MDPs [20]. For example, as we proved in Lemma 3.1, the MSM uncertainty set is indeed equivalent to risk-sensitive RL with the dynamic risk measure Λ E +(1 -Λ)CVaR τ . We note that efficient online RL algorithms have been proposed for similar measures [25, 81]. Static risk-sensitive RL also modifies the Bellman equations in an augmented MDP [74, 77]. Our focus is on deriving the optimal off-policy evaluation estimators for the problem, which involves a different set of challenges such as deriving the efficiency bound and ensuring sharpness guarantees even when nuisances are estimated slowly.

D Additional Technical Details

D.1 Higher Order Norms via Smoothness

For any x ∈ R + , define ⌊ x ⌋ as the greatest integer that is strictly less than x , and let x and { x } = x -⌊ x ⌋ represent the fractional part. Thus, we obtain the distinct decomposition x = ⌊ x ⌋ + { x } , where ⌊ x ⌋ ∈ N and { x } ∈ (0 , 1] .

Definition D.1 ( α -smooth functions) . Given α ∈ (0 , ∞ ) and X ⊆ R m , f : X → R is an α -smooth function if (1) the mixed derivatives up to ⌊ α ⌋ -order exist and are bounded; and (2) all ⌊ α ⌋ -order derivatives are { α } -Hölder continuous [49].

Lemma D.2 ( L ∞ Bound for α -Smooth Functions) . Let f : X → R , X ⊆ R m be an α -smooth function as in Definition D.1. Then, if X is R m , a half-space or a bounded Lipschitz domain in R m , there exists a constant C such the following inequality holds:

Proof. This lemma is a direct application of the fractional Gagliardo-Nirenberg interpolation inequality (Theorem 1 in [11]) from the functional analysis literature. For a more comprehensive exposition on this result, see Appendix A.1 in [8].

D.2 Localized Rademacher Complexity and Critical Radius

Here, we recap the localized Rademacher complexity and critical radius which is a standard complexity measure for obtaining fast rates for squared loss [72]. Let G be a class of functions g : Z → R . Given n datapoints z 1 , z 2 , . . . , z n , the empirical localized Rademacher complexity is:

where E σ is expectation over n independent Rademacher random variables σ 1 , σ 2 , . . . , σ n , i.e. , E σ [ · ] = 1 2 n ∑ σ ∈{-1 , 1 } n [ · ] . Note that when ε = ∞ , there is no localization and R n ( ∞ , G ) reduces to the vanilla Rademacher complexity. Let C := sup g ∈G ∥ g ∥ ∞ be the envelope of G . Then, the critical radius of G with n , called ε n , is the smallest ε that satisfies R n ( ε, G ) ≤ ε 2 / C .

Unless otherwise stated, we will posit that G is star-shaped: there exists g 0 ∈ G such that for all g ∈ G and α ∈ [0 , 1] , we have αg 0 +(1 -α ) g ∈ G . If not, we can replace G by its star-hull, i.e. , the smallest star-shaped set containing G . We will also posit that G is symmetric for simplicity.

The critical radius is a well-studied quantity in statistics [72] and also recently in RL [26, 69]. For example if G has d VC-subgraph dimension, then w.p. 1 -δ , ε n ≤ O ( √ d log n/n ) . For nonparametric models with metric entropy at most 1 /t β , the critical radius can also be bounded by O ( n -1 / (max(2+ β, 2 β )) ) [69], e.g. , is O ( n -1 / 4 ) if β = 2 .

E Proofs for Identification Results

E.1 Identification of robust Q

Lemma 3.1. Set τ ( s, a ) = (Λ( s, a ) + 1) -1 . Then, for any q : S × A → R ,

where v ( s ′ ) = E a ′ ∼ π t ( s ′ ) [ q ( s ′ , a ′ )] , and E , CVaR τ are under the observed kernel P ( · | s, a ) .

Proof. Consider the uncertainty set in T rob where the constraint on U (Eq. (1)) can be rewritten as:

Therefore, we can write U ( s ′ | s, a ) = Λ -1 ( s, a ) P ( s ′ | s, a ) + (1 -Λ -1 ) G ( s ′ | s, a ) where we define G ( s ′ | s, a ) := U ( s ′ | s,a ) -Λ -1 ( s,a ) P ( s ′ | s,a ) 1 -Λ -1 ( s,a ) . Thus, the constraints on G are that G ( · | s, a ) ≪ P ( · | s, a ) and ∥ d G ( s ′ | s,a ) d P ( s ′ | s,a ) ∥ ≤ Λ( s, a ) + 1 . Setting τ ( s, a ) = 1 Λ( s,a )+1 , we can apply the primal form of CVaR [3, 24] to obtain

Therefore, the supremum in T rob can be expressed as Λ -1 ( s, a ) times the expectation under nominal P and (1 -Λ -1 ( s, a )) times the above CVaR expression, which finishes the proof of the -case.

For the + case, we can simply use sup instead of inf and upper CVaR instead of lower CVaR.

E.2 Identification of robust kernel and visitation

Lemma 4.1. Suppose F -( β -τ ( s, a ) | s, a ) = τ , where β -τ ( s, a ) is the lower τ -th quantile of F -( · | s, a ) . Then,

Lemma E.1. Fix any v : S → R and define the pushforward F v ( y | s, a ) = P ( v ( s ′ ) ≤ y | s, a ) . Suppose F v ( β ± τ,F v ( ·| s,a ) ( s, a ) | s, a ) = 1 2 ± ( 1 2 -τ ) , where β ± τ,F v is the upper/lower τ -quantile of F v . Then, sup U ∈U ( P ) E U [ v ( s ′ ) | s, a ] = E s ′ ∼ U + v ( s,a ) [ v ( s ′ )] and inf U ∈U ( P ) E U [ v ( s ′ ) | s, a ] = E s ′ ∼ U -v ( s,a ) [ v ( s ′ )] , where

Proof. We start with some intuitions. First, if the CDF of v ( s ′ ) is differentiable β + τ ( s, a ) , then CVaR + τ ( v ( s ′ ) | s, a ) = E [ v ( s ′ ) | f ( s ′ ) ≥ β + τ ( s, a ) , s, a ] and the result follows immediately from Lemma 3.1 by noticing that the form of U + exactly recovers the convex combination of expectation and CVaR. Alternatively, one can use the closed form solution of the primal CVaR as derived in [3] to obtain the result.

We now provide a formal proof. Fix any s, a and let τ = τ ( s, a ) . Fix any function v ( s ′ ) ∈ R . We want to show that the worst-case U + = argmax U ∈U ( P ) E U [ v ( s ′ ) | s, a ] has a closed form expression as shown in line 725. By the proof of Lemma 3.1 above, we can rewrite U + ( s ′ | s, a ) = Λ -1 ( s, a ) P ( s ′ | s, a ) + (1 -Λ -1 ( s, a )) G + ( s ′ | s, a ) , where G + = arg max G ≪ P : | dG ( ·| s,a ) /dP ( ·| s,a ) | ∞ ≤ τ -1 ( s,a ) E G [ v ( s ′ )] . Thus, it suffices to simplify G + . To do so, we invoke the premise that the CDF of v ( s ′ ) is differentiable at β + τ , i.e. F v ( β + τ,F v ( s, a ) | s, a ) = 1 -τ . This implies that the CVaR is exactly the conditional expectation of the 1 -τ ( s, a ) -fraction of best outcomes, i.e. CVaR + τ ( v ( s ′ ) | s, a ) = E [ v ( s ′ ) | v ( s ′ ) ≥ β + τ ( s, a ) , s, a ] , which in turn is equal to τ -1 E [ v ( s ′ ) I [ v ( s ′ ) ≥ β + τ ( s, a )] | s, a ] . Thus, G + ( s ′ | s, a ) = τ -1 P ( s ′ | s, a ) I [ v ( s ′ ) ≥ β + τ ( s, a )] . This concludes the proof for the + case. The proof for the -case follows identical steps.

F Proofs for Robust FQE

We prove a more general result with approximate completeness, which shows that Theorem 3.4 is robust to approximate completeness.

Assumption F.1 (Approximate Completeness) . max q ∈Q min g ∈Q ∥ g -T ± CVaR q ∥ ν ≤ ε QComp .

Theorem F.2. Assume Assumption F.1. Under the same setup as Theorem 3.4, we have

and

Proof. Let U ± denote the worst-case kernel that satisfies V ± d 1 = (1 -γ ) E d 1 V π t U ± ( s 1 ) . Then,

Consider any k = 1 , 2 , . . . . By definition of U ± , we have

where β ⋆ k ( s, a ) is the true quantile of ̂ v k -1 ( s ′ ) . Denote q ⋆ k := T ± rob ̂ q k -1 and let β ⋆ k be the true upper/lower quantile of ̂ q k -1 . Recall the population loss function is

The empirical loss ̂ L k ( q, β ) is if E is replaced by E n . Note that ̂ q k = arg min q ∈Q ̂ L k ( q, ̂ β k ) .

Nonparametric Least Squares with Model Misspecification. We will directly invoke [72, Theorem 13.13], which gives a fast rate for misspecified least squares with general nonparametric classes. We now bound the misspecification. Recall that at the k -th iteration, our regression Bayes-optimal is E [ y ̂ β k k ( s, a, s ′ ) | s, a ] = T ̂ β k ̂ q k -1 ( s, a ) . By Lemma H.3, we know this is close to T β ⋆ k ̂ q k -1 ( s, a ) with second order errors in β : for any µ , we have

Finally, by approximate completeness (Assumption F.1), there exists g ∈ Q such that ∥T β ⋆ k ̂ q k -1 ( s, a ) -g ∥ ≤ ε QComp . Putting this together: for any k , there exists a g ∈ Q such that

Therefore, [72, Theorem 13.13] (and concentration of least squares) certifies that:

Therefore, we have proven:

This concludes the proof.

Lemma F.3 (Performance Difference) . For any π , transition kernel P , and function f : S × A → R , we have

Lemma F.4 (Unrolling) . For any π , transition kernel P , and functions f 0 , f 1 , . . . , f K : S × A → R satisfying f 0 ( s, a ) = 0 , we have ∥ f K -T π P f K ∥ d π, ∞ P ≤ 4 1 -γ max k =1 , 2 ,... ∥ f k -T π P f k -1 ∥ d π, ∞ P + γ K/ 2 .

G Proofs for Robust Minimax Algorithm

Assumption G.1 (Approximate W -realizability and completeness) . Assume the following hold for W and F :

(A) Approximate realizability: min w ∈W ∥J U ± ( w ± -w ) ∥ 2 ≤ ε WReal ;

(B) Approximate completeness: max w ∈W min f ∈F ∥ ∥ f -J ′ U ± ( w -w ± ) ∥ ∥ 2 ≤ ε WComp .

We prove a more general result with approximate realizability and completeness, which implies Theorem 4.4 that is robust to misspecification in its assumptions.

Theorem G.2. Under Assumption G.1 and the same setup as Theorem 4.4, we have

Proof. For this proof, we focus on the worst-case kernel P ⋆ of the form P ⋆ ( s ′ | s,a ) P ( s ′ | s,a ) = τ -1 ( s, a ) I [ ζ ⋆ ( s, a, s ′ ) ≤ 0] where ζ ⋆ ( s, a, s ′ ) = V -( s ′ ) -β -( s, a ) . This corresponds to the pure CVaR case of T -rob ; the E part is identical to standard non-robust RL so we omit it. The best-case kernel U + can be handled similarly. Let ̂ P ( s ′ | s, a ) denote our estimated robust kernel, which satisfies ̂ P ( s ′ | s,a ) P ( s ′ | s,a ) = τ -1 ( s, a ) I [ ̂ ζ ( s, a, s ′ ) ≤ 0] , where ̂ ζ ( s, a, s ′ ) is the given prior stage estimate of ζ ⋆ ( s, a, s ′ ) = V -( s ′ ) -β -( s, a ) .

The key and only difference between our Algorithm 2 and the MIL algorithm ( ̂ w mil) of [69] is that our next-state samples are importance weighted with ξ ± ( s, a, s ′ ) , which is the density ratio of the estimated robust kernel ̂ P ( s ′ | s, a ) and the nominal kernel P ( s ′ | s, a ) . Note also that ξ ± ( s, a, s ′ ) ≤ τ -1 ( s, a ) < ∞ , and hence | E n [ ζ ( s, a, s ′ ) f ( s ′ )] -E s,a ∼ ν,s ′ ∼ ̂ P ( s,a ) [ f ( s ′ )] | ≲ √ log(1 /δ ) /n w.p. 1 -δ . Therefore, up to O ( √ log(1 /δ ) /n ) errors, our Algorithm 2 can be viewed as MIL applied to the MDP with kernel ̂ P .

To invoke the result of [69, Theorem 6.1] (in MDP with kernel ̂ P ), we need to show that its assumptions are met by bounding the model misspecification, i.e. , Eq. (6) and Appendix C of [69]. Note that these misspecifications are w.r.t. the MDP with kernel ̂ P , since this is the MDP in which we're applying Theorem 6.1 of [69]. Specifically, the two errors we need to bound are, (A) approximate realizability: ε A = min w ∈W ∥J ′ ̂ P ( w ̂ P -w ) ∥ 2 ; and (B) approximate completeness: ε B = max w ∈W min f ∈F ∥ f -J ′ ̂ P ( w -w ̂ P ) ∥ 2 where recall that J P is the linear operator defined as J P f ( s, a ) := γ E P [ f ( s ′ , π t ) | s, a ] -f ( s, a ) and J ′ P is the adjoint.

Bounding misspecifications by ∥ ̂ ζ -ζ ⋆ ∥ ∞ . Since ζ ⋆ ( s, a, s ′ ) has a marginal CDF that's boundedly differentiable around 0 ( i.e. , (ii) of Assumption 4.2), [37, Lemma 3] implies that ζ ⋆ ( s, a, s ′ ) satisfies a 1 -margin (Definition H.2). Hence, Lemma H.3 and the continuity of ζ ⋆ ( s, a, s ′ ) implies that

̸

Thus, for any v : S → R ,

or equivalently

̸

̸

Equipped with Eq. (7), we can now bound the following two types of errors: (i) ⟨ f, ( T P ⋆ -T ̂ P ) g ⟩ , and (ii) ⟨ w ˆ P -w P ⋆ , h ⟩ , where f, g : S × A → R and h : S → R , and T P and w P are the Bellman operator and visitation density of target policy π t in the MDP with kernel P .

For (i):

For (ii):

where C = ∥ d d P ⋆ / d ν ∥ ∞ < ∞ .

For approximate realizability ( ε A ): for any w ∈ W , we have

where g 1 = (( J P ⋆ - J ̂ P ) ′ ( w ̂ P -w )) / ∥ ( J P ⋆ - J ̂ P ) ′ ( w ̂ P -w ) ∥ 2 , g 2 = ( J ′ P ⋆ ( w ̂ P -w P ⋆ )) / ∥J ′ P ⋆ ( w ̂ P -w P ⋆ ) ∥ 2 . The last inequality uses (i) and (ii) with the fact that ∥ g 1 ∥ ∞ < ∞ and ∥ g 2 ∥ ∞ < ∞ as the w terms are bounded by our premise. Therefore, taking min over w and using Assumption G.1, we have ε A ≲ ∥ ̂ ζ -ζ ⋆ ∥ ∞ + ε WReal .

For approximate completeness ( ε B ): for any w ∈ W and f ∈ F , we have

̸

̸

for the same reason as ε A as the error terms are the same. Thus, ε B ≲ ∥ ̂ ζ -ζ ⋆ ∥ ∞ + ε WComp .

In sum, we have shown that the misspecification is at most O ( ∥ ̂ ζ -ζ ⋆ ∥ ∞ + ε WReal + ε WComp ) . Therefore, [69, Theorem 6.1 and Appendix C] ensures that w.p. 1 -δ , our learned ̂ w satisfies,

Concluding the proof. The final step is to translate the above guarantee to ∥J ′ P ⋆ ( ̂ w -w P ⋆ ) ∥ 2 . The following shows that the switching cost is O ( ∥ ̂ ζ -ζ ⋆ ∥ ∞ ) as before:

This concludes the proof.

Lemma G.3 (Visitation performance-difference) . Let P, U : S → R + be non-negative measures, which should be thought of as transitions in a discounted Markov chain. Assume U satisfies ∑ s ′ U ( s ′ | s ) ≤ 1 . Define d U = (1 -γ ) ∑ ∞ h =1 γ h -1 d h U , where d h U = ∫ s 1 ,s 2 ,...,s h -1 d 1 ( s 1 ) U ( s 2 | s 1 ) . . . U ( s | s h -1 )d s 1: h -1 . Assume the same for P .

Let F ⊂ S → R be a function class that satisfies f ∈ F = ⇒ g ( s ) = E s ′ ∼ P ( s ) [ f ( s ′ )] ∈ F , i.e., closed under projection with P . Then, define the integral (probability) metric ∥ P -U ∥ F := sup f ∈F | ( E P -E U )[ f ( s )] | . Then we have,

Proof. Recall Bellman's flow, which is d P ( s ) = (1 -γ ) d 1 ( s ) + γ E ˜ s ∼ d P P ( s | ˜ s ) . Fix any f ∈ F . The initial state distributions cancel, so we have,

Thus, taking supremum over F , we have

Rearranging terms finishes the proof.

If F is the class of functions with ∥ f ∥ ∞ ≤ 1 , then this recovers the TV distance, which gives,

This generalizes Lemma E.3 of [1] to infinite horizon.

H Proofs and Additional Details for the Orthogonal Estimator

H.1 Intuition for Theorem 5.3

We provide some intuition for the results in Theorem 5.3. Consider the V -bound and let us decouple the indicator I [ v ( s ′ ) -β ( s, a ) ≤ 0] that appears implicitly in the ( v -( s ′ ) -β -( s, a )) -notation of Theorem 5.1. We augment the set of nuisances with ζ ( s, a, s ′ ) = v -( s ′ ) -β -( s, a ) such that ( v -( s ′ ) -β -( s, a )) -= ( v -( s ′ ) -β -( s, a )) I [ ζ ( s, a, s ′ ) ≤ 0] . We state the following lemma (which we elaborate upon in Lemmas H.4 and H.5 in the Appendix):

Lemma H.1 (Double sharpness with correct ζ ⋆ ) . Let E [ ψ ( s, a, s ′ ; q, w, β, ζ ⋆ )] be the expectation of the (R)EIF with an arbitrary nuisance set η = ( w,q, β ) , but where the indicator I [ v -( s ′ ) ≤ β -( s, a )] has been replaced with the correct indicator I [ ζ ⋆ ( s, a, s ′ ) ≤ 0] . Then:

This lemma implies that if β -= ( β ∗ ) -and ζ = ζ ∗ , then the estimator ̂ V -d 1 has a property known as 'double-robustness" [43] or 'double-sharpness" [24] in q and w , meaning the bias vanishes when either q or w is consistent. Moreover, the convergence rate would be O p ( r w n, 2 r q n, 2 ) . This condition holds provided that β and ζ are correctly specified. However, estimation errors in β introduce an additional O p ( ( r β n, ∞ ) 2 ) term, reflecting that β is first-order optimal for the CVaR component. Additionally, discrepancies between ζ and ζ ∗ contribute an extra O p ( ( r q n, ∞ ) 2 ) to the error. While this discussion gives some insight into how we achieve the results in Theorem 5.3, we provide a a rigorous analysis in the next section.

H.2 Preliminaries

For this proof, our focus will be on ̂ V -d 1 . The argument for ̂ V + d 1 is analogous, following a symmetric approach. To improve the clarity of our exposition, we will omit the -and τ indices, assuming their presence is clear from the context.

For simplicity, we assume that n is a multiple of K such that n = Kn K , where n K is the size of a fold. We let E n , E k denote the empirical averages over the entire sample and the k th fold, respectively. Recall that we use ̂ η = ( ̂ w, ̂ q, ̂ β ) and η ∗ = ( w ∗ , q ∗ , β ∗ ) to denote the estimated and oracle nuisances, respectively.

We further suppress the dependency on s, a in Λ and τ and we write the ρ term in Theorem 5.1 as

We justify this by noting that the analysis holds regardless of whether λ and τ depend on s, a . Sometimes, it will be useful to decouple the indicator I [ v ( s ′ ) -β ( s, a ) ≤ 0] implicit in the definition of ρ . In this case, we augment the set of nuisances with ζ ( s, a, s ′ ) = v ( s ′ ) -β ( s, a ) and write ρ as

Similarly define ψ ( · ; w,q, β, ζ ) with the ρ ( · ; v, β, ζ ) .

H.3 Auxiliary Lemmas

Definition H.2 (Margin Condition) . A function f : X → R of some random variable X is said to satisfy the margin condition with sharpness α ∈ [0 , ∞ ] (or more succinctly, an α -margin) if there exist a fixed constant c > 0 such that

If f ( X ) is either zero or bounded away from zero almost surely, then f satisfies an infinite margin, i.e. , α = ∞ [37, Lemma 2]. If f ( X ) is continuously distributed in a neighborhood around 0 , i.e. ,

its CDF is boundedly differentiable on ( -ε, 0) ∪ (0 , ε ) for some ε > 0 , then f has a 1 -margin [37, Lemma 3].

Lemma H.3 (Margin Guarantees) . For any f : X → R satisfying α -margin (Definition H.2), p ∈ [1 , ∞ ] , and any g : X → R , the following statements hold for some constant C > 0 :

̸

̸

where ∥·∥ p is the L p norm and we set ∞ t/ ∞ = t in the exponents.

The proof of Eq. (12) for any p ∈ [1 , ∞ ] and of Eq. (13) for p = ∞ is given in [4, Lemmas 5.1 and 5.2]. The proof of Eq. (13) for p < ∞ is given in [37, Lemma 5].

Lemma H.4 (Sharpness with correct q ⋆ and β ⋆ ) . 1 n ∑ ( s,a,s ′ ) ∼D ψ ( s, a, s ′ ; w,q, β ) is an unbiased estimator of V ⋆ d 1 when q = q ⋆ , β = β ⋆ , i.e.,

Proof. Since q ⋆ and β ⋆ are correct, the robust Bellman equation holds, and so for every s, a ,

Thus, multiplying by any w does not change the fact that the debiasing term in ψ has expectation zero. Since we have v ⋆ , the first term in ψ is exactly the estimand, which concludes the proof.

Lemma H.5 (Sharpness with correct w ∗ and ζ ∗ ) . 1 n ∑ ( s,a,s ′ ) ∼D ψ ( s, a, s ′ ; w,q, β, ζ ) is an unbiased estimator of V ⋆ d 1 when w = w ⋆ , ζ = ζ ⋆ , i.e.,

Proof. Let P ⋆ denote the robust transition kernel and let d ⋆ denote the robust visitation measure under π , which satisfies: for all functions f ,

Since ζ ⋆ is correct, for any v, s, a , we have

where in ⋆ we used E s ′ ∼ P ( s,a ) [ β ( s, a ) ( 1 -τ -1 I [ ζ ⋆ ( s, a, s ′ ) ≤ 0] )] = β ( s, a ) ( 1 -τ -1 τ ) = 0 . That is, for all function f , we have

This concludes the proof.

H.4 Proof of Rates

The estimation error is given by:

We wish need to bound ∣ ∣ E k [ ψ ( s, a, s ′ ; ̂ η [ k ] )] -V ∗ d 1 ∣ ∣ . We have that:

The first term is O p ( n -1 / 2 ) by the CLT. We are now interested in bounding the second term:

where we dropped the [ k ] indicator without loss of generality. We further decompose ε ( ̂ η ) into two error terms, ε A and ε B , as follows:

Bounding ε A : Error from the incorrect indicator ζ .

We break these terms down as follows:

We first bound ε A 1 . Assumption 4.2 implies

where c ′ < 1 is the min of 1 and the given neighborhood of zero and c ′′ ≥ 1 is the max of 1 and the bound on the density in that neighborhood. This implies a margin condition with α = 1 and c = c ′′ /c ′ .

Wecan instantiate the first part of Lemma H.3 with f ( X ) = v ⋆ ( s ′ ) -β ⋆ ( s, a ) , g ( X ) = ̂ v ( s ′ ) -̂ β ( s, a ) and obtain

To bound ε A 2 , first write

̸

̸

We can bound P ( I [ ̂ v ( s ′ ) -̂ β ( s, a ) ≤ 0 ] = I [ v ⋆ ( s ′ ) -β ⋆ ( s, a ) ≤ 0] ) using the second part of Lemma H.3 such that

Putting the ε A 1 and ε A 2 together, we have

Bounding ε B : Error with correct indicator but wrong nuisances. Now we focus on bounding ε B .

In the Lemma H.4 step, we used

Finally, note that

Due to continuity of the CDF of v ⋆ ( s ′ ) at β ⋆ ( s, a ) for all s, a , we have Pr( ζ ⋆ ( s ′ , s, a ) ≤ 0 | s, a ) = τ and so the last term vanishes. Thus, we're left with a quantity that is at most ≲ ( ̂ v ( s ′ ) -v ⋆ ( s ′ )) . Therefore,

Putting everything together, we obtain the desired rates:

H.5 Proof of Normality & Efficiency

In this part of the theorem, we let:

Then, we can write the following equality:

The second term converges in distribution to N (0 , Σ) from the CLT and the fact that ψ is the efficient influence function. Thus, it remains to show that the first term is o p (1) . We decompose the first term as follows:

In Eq. (15), we have that | E [ ψ ( s, a, s ′ ; ̂ η [ k ] )] -E [ ψ ( s, a, s ′ ; η ∗ )] | is bounded as in Eq. (Rates). Given the theorem's assumption about the nuisance rates, this term is o p ( n -1 / 2 ) and Eq. (15) is o p (1) . We now seek to control the ε k term in Eq. (16). Letting D k represent the samples in the k th fold, we leverage sample splitting to show that the mean of ε k | D k is 0 :

where we consider ̂ η [ k ] fixed with respect to the second expectation. The result follows from the fact that ̂ η [ k ] does not depend on D k . Then, we can invoke Chebyshev's inequality to obtain the following bound:

Thus, we have shown that ε k | D k = O p (Var[ ε k | D k ] 1 / 2 ) = O p ( n -1 / 2 E [( ψ ( s, a, s ′ ; ̂ η [ k ] ) -ψ ( s, a, s ′ ; η ∗ )) 2 | D k ] 1 / 2 ) . Here, we used the fact that n K = n/K (the size of D k ) and that K is a fixed integer that doesn't grow with n . Moreover, ε k has 0 conditional mean.

For the remainder of the analysis, we leave the conditioning on D k implicit for simplicity. To bound E [( ψ ( s, a, s ′ ; ̂ η [ k ] ) -ψ ( s, a, s ′ ; η ∗ )) 2 | D k ] 1 / 2 = ∥ ψ ( s, a, s ′ ; ̂ η [ k ] ) -ψ ( s, a, s ′ ; η ∗ ) ∥ 2 , we use similar notation and techniques as in Appendix H.4:

where we invoked Cauchy-Schwarz for the L 2 norm. We bound σ 2 as follows:

Given our rate assumptions, our boundedness assumptions for ̂ w , the implicit boundedness of q ∗ , v ∗ , w ∗ , β ∗ , as well as the ordering of the L 2 and L ∞ norms, σ 2 is o p (1) . We now bound the σ 1 term:

There are two cases in which the difference of indicators is non-zero:

In the first case, ̂ v ( s ′ ) -̂ β ( s, a ) ≤ 0 , β ∗ ( s, a ) -v ∗ ( s ′ ) < 0 and thus

In the second case, ̂ v ( s ′ ) -̂ β ( s, a ) > 0 , β ∗ ( s, a ) -v ∗ ( s ′ ) ≤ 0 and

Going back to σ 1 , we have:

By out theorem's assumptions, this term is also o p (1) . Putting σ 1 and σ 2 together, we have that ∥ ψ ( s, a, s ′ ; ̂ η [ k ] ) -ψ ( s, a, s ′ ; η ∗ ) ∥ 2 is o p (1) and ε k | D k is o p ( n -1 / 2 ) . By the bounded convergence theorem, this implies that ε k is also o p ( n -1 / 2 ) . Then, the term in 16 is o p (1) , which further means that √ n ( ̂ V d 1 -˜ V d 1 ) = o p (1) . Our proof is now complete.

I Derivation of the Efficient Influence Function

We use the ε -contamination approach of [31] to derive an influence function (IF) for our estimand V -d 1 . The proof for V + d 1 follows symmetrically. We note that since our tangent space is the whole space as it factorizes in the trivial way (as in [39, Page 54]), the IF we derive is actually the efficient influence function (EIF).

Let P ( s, a, s ′ ) denote the data distribution. Consider the ε -contamination P ε ( s, a, s ′ ) = (1 -ε ) P ( s, a, s ′ ) + εδ (¯ s, ¯ a, ¯ s ′ ) , where δ (¯ z ) is the dirac delta at ¯ z , i.e. , δ (¯ z ) has infinite mass at ¯ z and 0 mass elsewhere. Let V -ε denote the robust value function under the transition kernel P ε ( s ′ | s, a ) . Omitting the ε subscript means ε = 0 . The IF of V -d 1 is then given by

We dedicate the rest of this section towards this goal, which will be obtained in Theorem I.5. Lemma I.1.

Proof. Use the fact P ε ( s ′ | s, a ) = P ε ( s,a,s ′ ) P ε ( s,a ) = (1 -ε ) P ( s,a,s ′ )+ εδ (¯ s, ¯ a, ¯ s ′ ) (1 -ε ) P ( s,a )+ εδ (¯ s, ¯ a ) and take derivative.

Lemma I.2 (IF of conditional expectation) . For any s, a and f ε ,

where f = f 0 .

Proof.

Lemma I.3 (IF of conditional CVaR) . For any τ, s, a and f ε ,

where f = f 0 and β τ ( s, a ) be the (1 -τ ) -th quantile of f ( s ′ ) , s ′ ∼ P ( s, a ) .

Proof.

where the last equality is due to Danskin's theorem and the fact that β τ ( s, a ) is the maximizer of the CVaR dual form at ε = 0 . Continuing, let g ε ( s ′ ; s, a ) := β τ ( s, a ) + τ -1 ( f ε ( s ′ ) -β τ ( s, a )) -, so

This concludes the proof.

We now prove the key 'one-step forward' lemma.

Lemma I.4 (One-Step Forward) . For any state distribution ν ( s ) , we have

Proof. For any s 1 , we have

Taking expectation over s 1 ∼ ν , we have

Finally recall that V -satisfies the Bellman equation, so

This concludes the proof.

Equipped with our main one-step lemma, we can now unroll it an infinite number of steps to derive the IF of our estimand.

Theorem I.5 (IF of Estimand) . Let us denote

Proof. Let d h denote the h -th step visitation in the robust MDP, with transition P rob satisfying P rob ( s ′ | s,a ) P ( s ′ | s,a ) = (1 -λ ) + λτ -1 I [ V -( s ′ ) ≤ β τ ( s, a )] . Then notice that the final term of Lemma I.4 is exactly E s ∼ ν [ E π,P rob [ d d ε V -ε ( s ′ ) | ε =0 | s ]] . Therefore,

Iterating the process, we have

as desired.

Finally, we can conclude that the IF in Theorem I.5 is in fact the efficient IF (EIF) because it is in the tangent space, as the tangent space is contains all functions [39].

J Additional Validity Guarantees for Orthogonal Estimator

Our orthogonal estimator has additional desirable properties such as validity when some nuisances are misspecified. Specifically, the bounds returned by our orthogonal estimator will be asymptotically valid, though possibly loose, when some nuisances are inconsistent, i.e. , do not converge to the their true values. Below, we detail conditions under which we achieve validity. To be concise, we focus on the -case as the + case is symmetric.

Validity with correct Q ± . If ̂ Q = Q ± , we obtain valid bounds even if w,β are inconsistent. Lemma J.1. For any w,β , we have E [ ψ ( s, a, s ′ ; Q -, β, w )] ≤ V -with equality when β = β - d 1 τ .

Validity with Q = T ± β Q . Even if ̂ Q is misspecified, we still have a valid bound if it solves a Bellman-type equation of the dual CVaR form. For a β : S × A → R , define:

Remark J.3 . Lemmas J.1 and J.2 are dual to each other: in Lemma J.1, the plug-in is consistent while the debiasing correction errs in the valid direction ( i.e. , ≥ 0 for + and ≤ 0 for -). In Lemma J.2, the plug-in is valid while the debiasing correction has expectation zero.

J.1 Proofs for validity

Lemma J.1. For any w,β , we have E [ ψ ( s, a, s ′ ; Q -, β, w )] ≤ V -d 1 with equality when β = β -τ .

Proof.

where the inequality comes from the fact that β is sub-optimal for E [ β ( s, a ) + τ -1 ( V -( s ′ ) -β ( s, a )) -] . The same proof applies for Q + .

We now prove Lemma J.2. First, we show that the T β perspective gives rise to a dual definition of Q ± (dual to Eq. (2)).

Lemma J.4.

Proof. Unroll Q -( s, a ) = r ( s, a ) + γ inf U ∈U ( P ) E U [ r ( s ′ , a ′ ) + γ inf U ∈U ( P ) E U [ . . . ]] , replacing each inf U ∈U ( P ) with the convex combination of E and CVaR from Lemma 3.1. Then, write each CVaR using the dual form, i.e. , max β { β ( s, a ) + τ -1 ( s, a ) E [( · · · -β ( s, a )) + ] } . By s, a -rectangularity, the scalar max β separates per s, a , so we can pull all the maxes out front as a max over β ( s, a ) functions. Note that not all β ( s, a ) functions have a well-defined infinite sum in this manner, as T β is not always a contraction. The condition Q β = T -β Q β exactly characterizes when this unrolling is well-defined. Thus, Q -is exactly the minimum Q β whenever this procedure of unrolling with β is well-defined. This concludes the proof.

Lemma J.2. Fix any w,β . If Q ± β = T ± β Q ± β , then E [ ψ ( s, a, s ′ ; Q -β , β, w )] ≤ V -d 1 .

Proof.

The first equality is because the correction term is T -β Q -β -Q -β , which is zero since Q -β is a fixed point. The inequality is due to Lemma J.4.

K Additional Details for Main Experiment

K.1 Environment

We consider a simple MDP with a one-dimensional state space S = [0 , 5] , a binary action space A = { 0 , 1 } , reward function

which we note takes values in the range [0 , 1] , and with transitions given by

where UnifClip [ a, b ] denotes a uniform distribution between max( a, 0) and min( b, 5) . In addition, the environment always starts in initial state s 0 = 2 . Essentially, this is a simple control environment, where high rewards are obtained by maintaining state as close to zero as possible, the action a = 1 is a control action that (in expectation) moves the state closer to zero, and which occurs a small reward cost, and the action a = 0 is a passive action that allows the state to freely drift (with an overall drift away from zero).

K.2 Target Policy

We focus on estimating the worst-case policy value V -d 1 for the simple threshold-based target policy π t which takes action a = 1 when s ≥ 2 , and a = 0 whenever s < 2 .

K.3 Logging Policy and Data Sampling Procedure

We sample data using an evaluation policy π b which is an ϵ -smoothed threshold policy similar to π t . Specifically, π b takes action a = 1 when s ≥ 1 . 5 with probability 0.95, and takes action a = 0 when s < 1 . 5 with probability 0.95. We obtain a dataset { s i , a i , s ′ i , r i } by first rolling out with π b for 1000 burn-in time steps, and then sampling the tuple ( s, a, s ′ , r ) every 10 time steps. For each replication of our experiment, we sample 10,000 tuples in total.

K.4 Calculation of True Worst-Case Policy Values

Amajor challenge in studying robust policy value estimation is that, even with ground truth knowledge of the MDP and/or access to a simulator, it may be intractable to estimate the robust policy values V ± d 1 . Fortunately, the above environment has the desirable property that we can analytically compute the best/worst-case transition distributions allowed by our sensitivity model, since no matter what policy π t the agent is acting with, it always strictly prefers transitions to smaller states. In detail, suppose that for some state, action pair ( s, a ) we have P ( · | s, a ) = Unif [ x, y ] , for some 0 ≤ x ≤ y ≤ 5] . Then, letting α = 1 / (1 + Λ( s, a )) , it is easy to verify that the worst case transition kernel is given by

That is, the worst case transition kernel is given by a mixture of two uniform distributions. Therefore, we can easily simulate rollouts with the best/worst case transition kernels, and accurately estimate the robust policy values. This allows us to validate our methodology in this synthetic environment. Specifically, for each Λ( s, a ) we experiment with, we can compute the corresponding ground truth V -d 1 up to arbitrary precision via Monte Carlo sampling, by rolling out trajectories with π t in the adversarial MDP according to the above worst-case transition kernel.

Note as well that if one wanted to estimate the best-case policy value, analogous reasoning would give us

However, in our experiments we only concern ourselves with worst-case policy value estimation.

K.5 Nuisance Estimation

We instantiate slight variations of Algorithms 1 and 2 using neural nets for the classes Q , B , and W used for fitting Q -, β -, and w -respectively, and linear sieves for the corresponding critic class Q that we perform maximization over for the minimax estimation of w -. Specifically, we grow the linear sieve for the critic class in a data-driven way, as follows: at each step k of the respective algorithm, we compute the best response q k ∈ Q to the previous iterate solution w k ∈ W by optimizing over a neural net class, and then we append this best-response function to the set of functions in our linear sieve for the corresponding critic class. Full exact nuisance estimation details necessary for reproducibility will be available in our code release.

K.6 Estimators

We estimate the worst-case policy value using three different estimators:

Q : Direct estimator given by:

where

Orth : Our orthogonal estimator using EIF, given by

Note as well that we used a simpler data splitting procedure rather than the cross-fitting procedure described in Algorithm 3. Specifically, we used the first 10,000 tuples for estimating nuisances, and the second 10,000 tuples for the final estimators. This was done for the sake of computational ease in running experiments with many replications, and was performed in the same way for all methods.

In addition, for extra robustness, in each experiment replication we ran the nuisance estimation pipeline 5 times (on the same fixed sampled dataset), and took the 80th percentile policy value estimates, since the estimators tend to under-estimate the true policy value by design, with greater under-estimation when the nuisance estimates are less well optimized.

where s 1 is the deterministic initial state.

W : Importance sampling-style estimator using ˆ w -, which is given by:

L Empirical Investigation on Medical Application

Here, we describe an additional empirical investigation of our methodology on medical data. Specifically, we consider the problem of sepsis management using RL. For all parts of the investigation described below, fully complete details can be obtained from our code release.

L.1 Motivation of Investigation

Training RL models in simulated environments derived from real-world data is an exciting avenue for leveraging AI towards critical medical use cases. However, doing this obviously has the downside that, unless one undergoes the very risky process of training an RL agent online via real medical interventions, one has to resort to training within simulators, and then has to account for the inevitable 'sim-to-real' gap. Therefore, our robust OPE methodology provides an interesting approach for estimating worst-case performance of RL models under potential changes in dynamics when moving to real application.

L.2 RL Environment

Our RL environment is based on the OpenAI Gym sepsis simulator environment of [44]. This RL environment allows for simulation of dynamic sepsis management, which was created by training a blackbox ML model to mimic observed transition dynamics from the real-world electronic health record-based MIMIC-III dataset [36]. This existing sepsis simulator is an episodic environment that continues until the agent either recovers or dies. It has a 46-dimensional state space containing various vital measurements, a discrete action space containing 24 possible actions (where an action is essentially the Cartesian product of some independent base actions). The reward function in this original simulator gives zero reward whenever an episode has not terminated, a +15 reward at termination when if the patient survives, or a -15 reward at termination if the patient dies. Please see [44] and the code release linked therein for additional details.

We built an RL environment for our investigation by creating a simple wrapper around this existing sepsis simulator, in order to make it fit our setup. In particular, we made the following key changes:

We made the environment infinite-horizon, by automatically looping to a new random starting state for a fresh patient whenever the episode in the base simulator terminates

We normalized the reward function so that it lies in range [0 , 1] , where:

(a) r ( s, a ) = 0 if patient dies

(b) r ( s, a ) = 1 if patient recovers and is discharged

(c) r ( s, a ) = 0 . 5 if treatment has not terminated for current patient

In addition, for this environment, we perform all experiments with γ = 0 . 95 .

L.3 Policies for Investigation

We constructed RL policies for our empirical investigation by training some deep RL models using the sepsis simulator environment.

In the case of the behavioral policy π b used to generate the observational offline data, we trained this policy by running Proximal Policy Optimization (PPO: [63]) over a relatively large (16,000) number timesteps, in order to emulate a reasonably good 'current best practices' model for creating observational data.

In the case of the target policy π t to be evaluated, we trained this policy using Deep Q Learning (DQL: [53]), over a relatively small (1,600) number of timesteps, in order to emulate a potentially risky new candidate model.

L.4 Creating an Offline Dataset

Using our behavioral policy π b which we created as above, we generated a fixed offline dataset consisting of 20,000 observed tuples of state, action, reward, and next state. Unlike with our main empirical investigation in the main paper, we did not perform any 'thinning' on these sampled tuples to make them more independent, so that the observed transitions are sequentially correlated as with real-world medical data.

L.5 Nuisance Estimation

We perform nuisance estimation almost identically as in our main empirical investigation, with the only change being a slight change to our neural network architectures to better handle the large discrete action space. Specifically, instead of training neural networks that take state as input and produce |A| outputs (one per action), we train neural networks that take both state and action as inputs, using a learnt low-dimensional encoding of the actions, and produce a single output. Please see our code release for details.

L.6 Estimators

We consider the same three estimators ( Q , W , and Orth ) as in our main empirical investigation. As in that investigation, we use these to estimate the worst-case policy value for the given Λ( s, a ) . In addition, as in the main experiments, we consider these estimators for various fixed Λ( s, a ) that do not depend on s or a . In this case, we consider Λ ∈ { 1 , 2 , 4 } , as these reflect a reasonable range of possible confounding strength for real application.

L.7 Results

Below, in Table 2 we show the estimated policy value for all three estimators for each fixed Λ ∈ { 1 , 2 , 4 } . Here, we present the median policy value estimate over 5 runs of our estimators from random starting seeds after removing outliers. 2 In addition, we present a ± spread given by half the difference between the 80th and 20th percentiles.

Although for this investigation we cannot analytically compute the ground truth 'true' adversarial policy values to evaluate against when Λ > 1 , we can still analyze the trends of these estimators and compare them to those observed in our main synthetic experiment, and we can also compare their accuracy when Λ = 1 .

First, in the case of Λ = 1 , we computed the true policy value of π t to be within the range 0 . 532 ± 0 . 002 with 95% confidence. This is almost exactly equal to the median Orth estimator, but far outside the spread of outputs of the Q estimator. That is, although the Q estimator has somewhat lower variance in outputs over multiple runs for Λ = 1 compared with Orth , it appears to be far more biased.

Next, looking more broadly across all values of Λ , as in our main experiment, the Q and Orth estimators generally result in similar estimates to each other, and the W estimators are very variable.

This may reflect the relative difficulty of estimating the w -nuisance function compared with Q -and β -; although both Orth and W are affected by this difficulty, the Orth estimator has a theoretical robustness to the errors of these nuisance functions that the W estimator does not, as outlined in our theory.

We also observe that when Λ = 1 the Q estimator is significantly more stable than Orth , but when Λ > 1 the stability of Orth is either comparable to or superior to Q . In order to understand this, we first note that unlike in our main experiments, here the repetitions are re-runs of the estimators with the same offline sepsis dataset, so these ± spreads reflect potential computational errors rather than statistical errors. Given this, this pattern of errors could be explained by the fact that when Λ = 1 the Q estimation is extremely simple, reducing to standard FQI, whereas when Λ > 1 it requires a more complex robust FQI estimation with simultaneous estimation of β -. That is, the difference in computational difficulty of estimating Orth versus Q may be smaller for Λ > 1 .

Overall, although it is hard to definitively compare the accuracy of these estimators for Λ > 1 given a fundamental lack of ground truth, given both a similar pattern of results as in our synthetic experiments, as well as the far greater accuracy of Orth when Λ = 1 , it seems reasonably to believe based on these results that our proposed Orth estimator may be more reliable than the existing robust FQI approach of the Q estimator.

Finally, we consider the implication of our results for the problem of learning sepsis management policies from simulators. Our Orth estimator suggests that there is relatively little sensitivity of this environment to deviations allowed by Λ = 2 , but very significant deviation allowed by Λ = 4 . Indeed, given the reward structure described above, the worst-case results under Λ = 4 imply an extremely high mortality rate. Whether worst-case deviations of this magnitude are reasonable or not is unclear, and this is something that requires further investigation for future work on RL for sepsis management.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] .

Justification: Yes, we provide complete proofs for our theorems and describe detailed empirical validation for our proposed algorithms.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes] .

Justification: Yes, we discussed where our assumptions may fail and settings not captured by the current paper, which we believe are directions for future research.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Yes, we provide full assumptions in the main paper and the complete proofs are written in the Appendix.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes] .

Justification: Yes, our experimental section includes all details needed to reproduce the main experimental results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes] .

Justification: Yes, our code is open-sourced at https://github.com/CausalML/ adversarial-ope/ .

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Yes, please see our experimental section and appendices for all training and evaluation details.

Guidelines:

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes] .

Justification: Yes, our experiments are replicated over multiple seeds and we report the confidence intervals.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes] .

Justification: Yes, this paper is mostly focused on theory and our experiment is a proof of concept and can be run on a standard GPU.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes] .

Justification: Yes, we have reviewed the code of ethics and believe our research conforms to it.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA] .

Justification: This paper is about foundational research not tied to particular applications so we do not feel the need to highlight any societal impacts.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: This paper is about foundational research not tied to particular applications so we do not feel the need to highlight any risks for misuse here.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA] .

Justification: The paper does not use any existing assets.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA] .

Justification: The paper does not release new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: The paper does not involve crowdsourcing nor research with human subjects.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a statistical theory for off-policy evaluation given observational data from an unknown transition kernel of an original MDP, possibly under policies the same as or different from the target policy. This paper focuses on the situation of environment shift, i.e., the transition kernel of the MDPs at test time can shift from the original MDP to a certain extent. These transition kernels are assumed to be center around the observational transition kernel. This paper proposes new algorithms and sharp estimators for robust estimation of policy value. The proposed estimation method has several desired properties.. First, it is semiparametrically efficient, with a $\\sqrt{n}$-rate of estimation even when certain nuisance functions are estimated at slow parametric rates. The estimator is also asymptotically normal which enables easy statistical inference. Even when the nuisance functions are estimated with bias, the estimator is still constant. Numerical simulations verify the efficiency and sharpness of the proposed estimator.\n\nSTRENGTHS:\n**Significance**: \n\n**1.** This paper focuses on an important and challenging theoretical problem of policy evaluation under transition kernel shift. This is a very realistic and probably more common scenario compared to the classical OPE setting where only the policies shift.\n\n**2** This paper presents a credible statistical theory for policy evaluation under transition kernel shift and the existence of nuisance functions. The statistical analysis shows that strong statistical guarantee can be achieved for the proposed estimator.\n\n**Quality**: this paper has a good quality. \n\n**1.** The paper has a solid theoretical analysis, with well specified definitions and theorems.\n\nWEAKNESSES:\n**1.** This paper is purely theoretical and thus it is very hard to evaluate the practicability of the proposed estimation framework. The numerical simulation also only covers a very simplified synthetic environment. Thus it is unclear how useful or computationally efficient the proposed estimation framework really is.\n\n**2.** I feel this paper is missing a discussion of the line of work on OPE under POMDP. It seems that the OPE on POMDP is naturally connected with the topic of this paper, though I am not very familiar with that line of work. \n\n**Minor**:\n \n**1.** The notation of this paper is a bit heavy. For example, the plus-minus sign confused me for a while when I read about its meaning (“read twice”) at the bottom of page 3. But this is not a big issue and is sort of understandable given the complex framework of this paper. \n\n**2.** This paper is a bit hard to read for readers unfamiliar with this topic. No example or demonstration of any kind is given.\n\nQUESTIONS:\nPlease see weakness.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the problem of evaluating a policy under best- and worst-case perturbations to a Markov decision process (MDP), given transition observations from the original MDP. The first contribution is the robust Q learing algorithm for CVaR problem, the second contribution is the estimation of robust visitation distribution in the function approximation setting and their complexity bounds.\n\nSTRENGTHS:\nThe problem studies in this paper is of significant practical importance, which is to evaluate the worst/best performance over an uncertainty set. The authors designed algorithms and derived their complexity bounds, which are shown to be optimal.\n\nWEAKNESSES:\n- The authors seem to be using offline and off-policy interchangably, which is misleading and confusing.\n- One important challenge in offline reinforcement learning is due to partial coverage and distribution shift. However,  this paper does not explicitly define the single-policy concentrability coefficient that measures the gap between the sampling distribution $\\nu$ and the target policy $\\pi$, and therefore did not consider this key challenge in the offline setting. \n- The notations are unnecessarily complicated, and many are used without being defined. For example, in algorithm 1 line 6, the hat on top of the expectation, and the subscript of the expectation is not clear.\n- There are many technical assumptions that remain unjustified. Please justify each of the assumptions, and provide examples when those assumptions can be satisfied, and how to verify those assumptions in practice.\n- The perturbation model in eq. (1) is unjustified. It could happen easily that $\\Lambda(s,a)$ is zero or infinite for some $(s,a)$. The authors shall explain why this perturbation model is of interest, and why this model is superior over existing uncertainty set models, e.g., those defined by KL divergence, TV.\n\nQUESTIONS:\n- There are several other important papers on robust offline reinforcement learning that are missing in this paper, e.g., \nDouble Pessimism is Provably Efficient for Distributionally Robust Offline Reinforcement Learning: Generic Algorithm and Robust Partial Coverage\nJose Blanchet, Miao Lu, Tong Zhang, Han Zhong\nDistributionally Robust Model-Based Offline Reinforcement Learning with Near-Optimal Sample Complexity\nLaixi Shi, Yuejie Chi\nComparison with existing robust offline RL is not clear in this paper. \n- The writing of this paper needs to be improved. Is Section 3 and Algorithm 1 only about CVaR?\n\nLIMITATIONS:\nna", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies off-policy evaluation problems in robust Markov decision processes, where the environment dynamics may change over time. The authors consider a perturbation model that modifies the transition probability up to a given multiplicative factor. They first propose two new algorithms for learning the Q-value function $Q$ and the density ratio function $w$. Then, they design a new policy value estimator based on the estimated nuisance parameters. The authors provide a theoretical analysis of the new algorithms and estimators, along with numerical experimental results to verify their effectiveness.\n\nSTRENGTHS:\na. The studied problem of off-policy evaluation in robust MDPs is interesting and meaningful.\n\nb. The theoretical analysis is solid. The proposed $Q$-function estimation algorithm achieves a $\\widetilde{\\mathcal{O}}(n^{-1/2})$ rate for parametric classes. More importantly, the proposed orthogonal estimator in Section 5 has a second-order dependence on the estimation error of nuisance parameters. As a result, it only requires the rate of nuisance parameter estimation to be faster than $n^{-1/4}$ to achieve an $n^{-1/2}$ rate.\n\nc. The numerical results align with the theoretical analysis; the orthogonal estimator consistently achieves the best performance compared to the $Q$ estimator and $w$ estimator.\n\nWEAKNESSES:\na. This paper considers a perturbation model where the ratio between the modified transition kernel and the nominal kernel is bounded by a sensitivity parameter. In Algorithms 1 and 2, the learner needs access to the parameter $\\Lambda(s,a)$. However, it is unclear how to estimate $\\Lambda(s,a)$ in real-world scenarios. Additionally, the authors claim that the perturbation model is more general and captures confounded settings. More detailed analysis and explanations are needed to support this claim.\n\nb. Algorithm 2 requires solving a minimax optimization problem, which is usually very time-consuming in practice. In the experiments, the authors use an iterative method as an approximation. Is there any theoretical analysis for that implementation?\n\nc. Additional experiments using real-world datasets would help to better demonstrate the practical applicability of the proposed algorithms.\n\nMinor:\nRedundant content in line 194\n\nQUESTIONS:\nSee weakness part.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies statistically-efficient robust/optimistic (e.g., worst- or best-case within the uncertainty set $\\mathcal{U}$) off-policy evaluation with bounded Radon-Nikodym uncertainty sets centered around known nominal models. It proposes novel algorithms for computing robust/optimistic Q functions, and provides theoretical analysis for them. It further claims that the sharp and efficient estimators are optimal in the local-minimax sense, and provides preliminary experimental results to support the claim.\n\nSTRENGTHS:\n* The paper seems to include a wide collection of new theoretical results, with assumptions clearly stated and proofs attached.\n* The problem itself is simple, fundamental, and stated in an comprehensible way.\n\nWEAKNESSES:\n* The writing of this paper needs to be improved, at least for non-experts in this specific area.\n    * Frankly speaking, I'm having a lot of trouble following the paper, or even understand the problems and the results. Though I have to admit that I major in CS and don't have a strong statistics background.\n    * First and foremost, it doesn't seem like the $\\pm$ superscript is doing any good here, except confusing the reader for the first few moments. If I were to write this paper, I would just sell the story of \"robust OPE\", and say a sentence or two on how to re-interpret it as a optimistic evaluation scheme (I do think negating rewards is easier to understand).\n    * Some sentences are broken and not fully comprehensible (e.g., lines 193~195), some notations are left undefined before their first use (e.g., the $\\frac{dU}{dP}$ notation, which I regard as Radon-Nikodym derivative), and some paragraphs are duplicated with subtle differences (e.g., Lemma 4.1 and G.1).\n    * The literature review part (Section 1.1 and Appendix B) seems to be very incomplete, and some paragraphs seem irrelevant (e.g., lines 84~104). Since this is a paper on robust OPE, a more comprehensive review on existing works regarding robust MDPs and OPE methods is definitely needed, especially a comparison on the settings and the bounds.\n    * As far as I'm concerned, the flow of the whole paper needs major revision. For example, it's unclear why we need to estimate $w^{\\pm}$ in Section 4, until in the estimator for $V^{\\pm}\\_{d\\_1}$ is proposed in Section 5. Another issue is that I don't fully get the point why we need a few more complex procedures to estimate $V^{\\pm}\\_{d\\_1}$ after we've got the estimates for $Q^{\\pm}$. Besides, the key results are not explained in plain English (e.g., given the sentences between lines 211 and 217, I still don't see clearly what Theorem 3.4 means. What is the typical order of $\\varepsilon^{\\mathcal{Q}}_n$? What about the $\\mathrm{err}$ term?)\n    * Simply put, Sections 3~5 are not well-motivated, the results are not intuitively interpreted, and their connections need to be further specified.\n    * Personally, I expect to first see a full overview of the proposed algorithm and the \"ultimate\" end-to-end sample complexity result (which is now hidden in the paragraph \"Putting everything together\" on lines 316~322) to get the big picture, before diving into all the technical details introducing the subroutines.\n* Since I'm not an expert in this area, I don't fully understand the technical details. (***To AC:*** Please do take other reviewers' opinions for technical soundness. I've tried hard but failed to fully check the proofs.)\n    * Specifically, I don't follow the proofs in Sections E.5 (for the second part of Theorem 5.3), G.1 (for Lemma 3.1) and G.2 (for Lemma 4.1). The latter two don't seem like legit proofs in my opinion.\n\nQUESTIONS:\n* Why do you use bounded Radon-Nikodym derivative for defining ambiguity sets, as I haven't seen people doing this? Does it have any connections with other \"distances\", e.g. total variation, KL-divergence, or Wasserstein distance?\n\nLIMITATIONS:\nThere are a few sentences discussing limitations of the setting, but a summary is missing.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-22"}