Title: Worst-Case Offline Reinforcement Learning with Arbitrary Data Support
Venue: NeurIPS 2024
Accepted: yes
Keywords: offline reinforcement learning

worst-case learning

support free

sample complexity analysis
Abstract: We propose a method of offline reinforcement learning (RL) featuring the performance guarantee without any assumptions on the data support. Under such conditions, estimating or optimizing the conventional performance metric is generally infeasible due to the distributional discrepancy between data and target policy distributions. To address this issue, we employ a worst-case policy value as a new metric and constructively show that the sample complexity bound of $O(\epsilon^{−2})$ is attainable without any data-support conditions, where $\epsilon>0$ is the policy suboptimality in the new metric. Moreover, as the new metric generalizes the conventional one, the algorithm can address standard offline RL tasks without modification. In this context, our sample complexity bound can be seen as a strict improvement on the previous bounds under the single-policy concentrability and the single-policy realizability.

Worst-Case Offline Reinforcement Learning with Arbitrary Data Support

Abstract

We propose a method of offline reinforcement learning (RL) featuring the performance guarantee without any assumptions on the data support. Under such conditions, estimating or optimizing the conventional performance metric is generally infeasible due to the distributional discrepancy between data and target policy distributions. To address this issue, we employ a worst-case policy value as a new metric and constructively show that the sample complexity bound of O ( ϵ -2 ) is attainable without any data-support conditions, where ϵ > 0 is the policy suboptimality in the new metric. Moreover, as the new metric generalizes the conventional one, the algorithm can address standard offline RL tasks without modification. In this context, our sample complexity bound can be seen as a strict improvement on the previous bounds under the single-policy concentrability and the single-policy realizability.

1 Introduction

Offline reinforcement learning (RL) (Levine et al., 2020; Prudencio et al., 2023) is a framework for learning decision-making policies while constrained to a fixed batch of data, preventing the learner from acquiring new information about the environment during training.

The primary challenges of offline RL are thus originated from the discrepancy between the stateaction distribution of the batch data µ ( s ) β ( a | s ) and the visitation distribution of the trained policy d π ( s ) π ( a | s ) . Most of the previous studies have avoided directly dealing with this discrepancy by posing the assumption known as concentrability (Munos and Szepesvári, 2008; Antos et al., 2008; Chen and Jiang, 2019; Xie et al., 2022). Roughly speaking, the condition asserts that the ratio between these two distributions d π π/µβ is well-defined and uniformly bounded over the entire state-action space. This, in turn, constrains the trained policy π to strictly stay inside the state space covered by the data support.

However, concetrability may be impractical in real-world applications for several reasons. First, one often ends up with a poor coverage of the state-action space when exhaustive data collection is expensive or practically infeasible as in the domains of autonomous driving (Fang et al., 2022), healthcare (Yu et al., 2021) and public policy-making (Abe et al., 2010). Moreover, the precise shape of the partial coverage is unknown if the considerations making it partial are not well-documented or disclosed. On the other hand, it is generally difficult to accurately predict if a policy will visit a given state or not based only on the knowledge of the policy and the batch dataset. As a result, the set of concentrable policies in a hypothesis space may be too small to achieve reasonable performance or

even empty. 2 Therefore, for applying offline RL in such domains, we need a method that works well without concentrability or any coverage-related conditions.

To tackle with this issue, we study offline RL with arbitrary data support. We present two major results in this paper.

We develop worst-case offline RL (Problem 4.1), a new offline RL framework for handling poor state-action coverage, which can be seen as a natural generalization of conventional offline RL (Corollary 4.2).

ii) We develop worst-case minimax RL (WMRL, Section 6.3), a model-free algorithm addressing worst-case offline RL (Corollary 6.3). The resulting sample complexity bound improves the previous state of the art in terms of both the weakness of the assumptions and the strength of the bound (Table 1).

The rest of the paper is organized as follows. In Section 2, we review the previous work in the literature of offline RL, centered around theoretical studies on the role of concentrability. In Section 3, we introduce some preliminaries around Markov decision process (MDP), offline RL and concentrability. Then, in Section 4, with the observation that offline RL is ill-posed without concentrability, we introduce worst-case offline RL as a natural generalization and discuss some of its properties useful in our subsequent analysis. In Section 5, we establish the connection between worst-case offline RL and the Lagrangians derived from the saddle-point formulation of offline RL. In Section 6, exploiting the connection established earlier, we construct a method for solving worst-case offline RL with polynomial sample complexity. Finally, we discuss the limitation and the future work in Section 7.

2 Related Work

The notion of concentrability is introduced by Munos (2003); Munos and Szepesvári (2008); Antos et al. (2008) to analyze the value/policy iteration algorithms, not necessarily in the context of offline RL. Recently, it has been increasingly gaining traction as one of the key characteristics of the difficulty of offline RL (Chen and Jiang, 2019) due to the distribution mismatch. In its original definition, concentrability requires the norm of the density ratio ∥ d π π/µβ ∥ ∞ to be bounded uniformly for all the policies π . Liu et al. (2020) showed that this uniform boundedness can be relaxed to the single-policy boundedness with the principle of pessimism in the face of uncertainty (PFU) . Considering the case where the single-policy concentrability is even slightly violated, Xie et al. (2021) further analyzed the

performance degradation caused by the lack of concentrability. Finally, we completely remove the concentrability assumption by incorporating it into a new performance metric.

The removal of concentrability is useful not only for widening the applicability of offline RL, but also strengthening the sample complexity bound by streamlining the analysis. Previously, Zhan et al. (2022) established polynomial sample complexity bounds under the weakest known model-free assumptions, yet being unable to achieving the statistically reasonable rate O ( ϵ -2 ) . Also, Chen and Jiang (2022); Ozdaglar et al. (2023); Uehara et al. (2023) gave improved rates additionally assuming that the minimum action value gap is bounded away from zero. On the other hand, under a set of assumptions as weak as Zhan et al. (2022), our sample complexity bound achieves the rate of O ( ϵ -2 ) . See Appendix A for more detailed discussions.

Algorithmically, the PFU principle is often materialized as the pessimistic or behavioral regularization (Kumar et al., 2020; Fujimoto and Gu, 2021; Yu et al., 2020). Previous analyses are often sensitive to the hyperparameters controlling the degree of such regularization, such as the truncation threshold b in Liu et al. (2020), the Bellman consistency threshold ε in Xie et al. (2021); Chen and Jiang (2022) and the regularization weight in Zhan et al. (2022); Uehara et al. (2023). On the other hand, our method has no hyperparameter other than the choice of the function approximators. One may see the root cause of this difference in that the PFU principle is built into our single new performance metric, whereas the previous studies adopt it as an additional objective, resulting in bi-objective optimizations.

3 Preliminaries

We denote the set of nonnegative real numbers by R + = [0 , ∞ ) and the uniform norm of a function g over its domain by ∥ g ∥ ∞ := sup z ∈ dom( g ) | g ( z ) | . We also denote by ∆( X ) the set of (generalized) probability density functions on X relative to a suitable base measure, 3 such as the counting measure and the Lebesgue measure.

Markov decision process (MDP) and RL. Let M = ( S , A , R, T ) be an MDP consisting of the state space S , the action space A , the reward function R : S × A → ∆([0 , 1]) and the transition probability T : S × A → ∆( S ) . We assume both S and A are finite sets for simplicity. The goal of RL in general is to optimizing policy π : S → ∆( A ) in terms of the policy value ,

with a discount factor γ ∈ (0 , 1) . Here, the expectation E π is taken with respect to the Markov chain generated with a t ∼ π ( ·| s t ) , r t ∼ R ( s t , a t ) and s t +1 ∼ T ( s t , a t ) , t ≥ 0 , starting from a known initial-state distribution s 0 ∼ p 0 ( s ) .

Offline constraint. In maximizing J ( π ) , the offline constraint prohibits us to access the environment M except through the offline dataset D := { ( s i , a i , r i , s ′ i ) } n i =1 . We assume the dataset is sampled from a fixed distribution p M data ∈ ∆( S × A × [0 , 1] ×S ) n such that

where µ ∈ ∆( S ) and β : S → ∆( A ) are the behavior state distribution and the behavior policy, respectively. Typically, p M data ( D ) represents the distribution of the past observational data. The problem of offline RL is now formally given as follows.

Problem 3.1 (Offline RL) . Given the offline dataset D and a small number ϵ > 0 , find a policy π achieving J ∗ -J ( π ) ≤ ϵ , where J ∗ := max π : S→ ∆( A ) J ( π ) .

Value, visitation and weight functions. Let r ( s, a ) := E y ∼ R ( s,a ) [ y ] be the expected reward function and r π ( s ) := ∑ a r ( s, a ) π ( a | s ) be its marginalization with respect to policy π . Let T , T π and

T π ∗ be the raw transition operator, the policy transition operator and its adjoint given by T v ( s, a ) = ∑ s ′ v ( s ′ ) T ( s ′ | s, a ) , T π v ( s ) = ∑ a T v ( s, a ) π ( a | s ) and T π ∗ d ( s ) = ∑ s ′ ,a ′ d ( s ′ ) π ( a ′ | s ′ ) T ( s | s ′ , a ′ ) , respectively. Then, the state value function v π : S → R , the action value function q π : S × A → R and the state visitation distribution d π ∈ ∆( S ) are given by

as well as the state weight function w π : supp( µ ) → R and the action weight function f π : supp( µβ ) → R by

̸

where supp( g ) := { x ∈ dom( g ) | g ( x ) = 0 } denotes the support of function g and ρ π ( s, a ) := π ( a | s ) β ( a | s ) is the density ratio of π to β . We also define the optimal value functions by v ∗ ( s ) := max π v π ( s ) and q ∗ ( s, a ) = max π q π ( s, a ) as well as the set of the optimal policies Π ∗ := { π : S → ∆( A ) : v π = v ∗ } , which by definition all attain J ∗ . See Table 2 for the summary of the notation introduced above.

Concentrability. A policy π is said to be concentrable (or satisfying concentrability) if its stateaction visitation is contained in the data support, supp( d π π ) ⊂ supp( µβ ) . We denote the set of all the concentrable policies by Π CC := { π : S → ∆( A ) : supp( d π π ) ⊂ supp( µβ ) } .

4 Worst-Case Offline Reinforcement Learning

In offline RL (Problem 3.1), the information on M is restricted by the data support supp( µβ ) . In such situations, one cannot know about the transition probability T ( s, a ) and the reward probability R ( s, a ) for ( s, a ) ̸∈ supp( µβ ) . Consequently, the accurate estimation of J ( π ) is infeasible (even with n = ∞ ) for unconcentrable policies, and thus previous analyses on Problem 3.1 often require that there exists at least one concentrable optimal policy, i.e., Π CC ∩ argmax π J ( π ) = ∅ .

̸

To remove such dependency on concentrability, we introduce a performance metric alternative to J ( π ) . Let U := {M ′ : p M ′ data = p M data } be the set of the environments indistinguishable from the true environment M with respect to the resulting data distribution p M data . Noting that U is the information-theoretic limit of the uncertainty on M under the offline constraint, we follow the pessimism-in-the-face-of-uncertainty principle and consider the worst case within U ,

which we refer to as worst-case policy value . Replacing J ( π ) with ˜ J ( π ) in Problem 3.1, we arrive at the following problem.

Problem 4.1 (Worst-case offline RL) . Given the offline dataset D and a small number ϵ > 0 , find a policy π achieving ˜ J ∗ -˜ J ( π ) ≤ ϵ , where ˜ J ∗ := max π : S→ ∆( A ) ˜ J ( π ) .

To facilitate the subsequent analysis on Problem 4.1, we next introduce the notion of truncated environment , which is similar to, yet different from those previously considered by Liu et al. (2020); Yin and Wang (2021) as it is based on the true and unknown data support rather than the empirical one. The truncation is useful for characterizing the worst-case policy value ˜ J ( π ) .

Definition 4.1 (Truncated environment) . The truncation of M with respect to µ and β is given by ˜ M = ( ˜ S , A , ˜ T, ˜ R ) , where ˜ S = S ∪ {⊥} with ⊥ being an absorbing state with reward zero and

for s ∈ S and a ∈ A , where χ µ,β ( s, a ) := I { µ ( s ) > 0 } I { β ( a | s ) > 0 } is the indicator function of the support of µ ( s ) β ( a | s ) and δ x ∈ ∆( X ) is the Dirac's delta function located at x ∈ X .

Theorem 4.1. We have ˜ J ( π ) = J ( π | ˜ M ) for all π : S → ∆( A ) .

Proof (sketch). It suffices to show J ( π | ˜ M ) ≤ ˜ J ( π ) ≤ J ( π | ˜ M ) , where the first inequalilty follows from J ( π | ˜ M ) ≤ J ( π |M ′ ) for all M ′ ∈ U and the second inequality follows from ˜ M∈ U . See Appendix D.1 for the complete proof.

In other words, worst-case offline RL is nothing but offline RL with the truncated environment ˜ M . Thus, in principle, one can exploit conventional offline RL methods to solve Problem 4.1. We hereafter refer to the truncated counterparts (those defined by replacing M with ˜ M ) of v π , v ∗ , q π , q ∗ , Π ∗ , d π , w π and f π as ˜ v π , ˜ v ∗ , ˜ q π , ˜ q ∗ , ˜ Π ∗ , ˜ d π , ˜ w π and ˜ f π , respectively. Likewise, let ˜ r , ˜ r π , ˜ T , ˜ T π and ˜ T π ∗ be the truncated counterparts of r , r π , T , T π and T π ∗ , respectively.

Let us remark several key implications of Theorem 4.1. First, since the unknown parameters ˜ R and ˜ T of the truncated environment ˜ M are only nontrivial on the data support, it is intuitively obvious that ˜ J ( π ) can be accurately estimated even without the concetrability, given sufficiently large n . Thus, it is reasonable to expect that Problem 4.1 does not require any concetrabilities to be well-posed, unlike Problem 3.1. Second, the constructive existence of ˜ M makes the relationship between J ( π ) and ˜ J ( π ) clearer, as stated in the following corollary.

Corollary 4.1. We have ˜ J ( π ) = J ( π ) if π ∈ Π CC and ˜ J ( π ) ≤ J ( π ) otherwise.

Proof.

According to Corollary 4.1, the pessimism introduced by the truncation is mild in the sense that it conserves the values of concentrable policies. Finally, it also clarifies the relationship between the suboptimality metrics of the conventional and the worst-case problems.

Corollary 4.2. For all π : S → ∆( A ) , we have

if Π CC ∩ argmax π J ( π ) = ∅ . Moreover, the equality is attained if in addition π ∈ Π CC.

̸

Proof. Trivial from Corollary 4.1.

In other words, solutions of the worst-case problem are also valid as solutions of the conventional problem under the standard assumption, while the two solution concepts are identical if only concentrable policies are concerned. In this sense, worst-case offline RL is a natural generalization of the conventional offline RL for handling arbitrary data distributions.

Finally, we conclude this section by showing a useful property of the worst-case optimal policies. Let Π β := { π : S → ∆( A ) | supp( µπ ) ⊂ supp( µβ ) } be the set of the on-support policies, i.e., the policies with the support covered by the behavior policy. The following lemma allows us to limit the scope of policy optimization to Π β without sacrificing the optimality in terms of ˜ M . The proof is relegated to Appendix D.3.

̸

Lemma 4.1. There is at least one worst-case optimal policy that is on-support, i.e., ˜ Π ∗ ∩ Π β = ∅ .

5 Lagrangians for Worst-Case Offline Reinforcement Learning

In this section, we set up theoretical foundation of worst-case offline RL. Specifically, in Section 5.1, we show a connection between ˜ J ( π ) and the Lagrangian of RL (Puterman, 2014). However, since the Lagrangian in its original form is unstable to the function approximation error (Section 5.2), we further introduce a regularized variant of it (Section 5.3).

5.1 Unregularized Lagrangian

Consider the following functional of v : S → R + and f : supp( µβ ) → R + ,

where E p 0 and E µ,β are the expectation operators with respect to s ∼ p 0 ( s ) and ( s, a ) ∼ µ ( s ) β ( a | s ) , respectively, and δ TD : R S → R S×A is the time-difference error operator given by δ TD v ( s, a ) = r ( s, a ) + γ T v ( s, a ) -v ( s ) .

We refer to L ( v, f ) as the (unregularized) Lagrangian since it has been known as the Lagrangian of the linear-programming-based formulations of RL (Puterman, 2014; Chen and Wang, 2016; Nachum et al., 2019; Zhang et al., 2021; Zhan et al., 2022). The following theorem reveals that, perhaps surprisingly, it is also connected with worst-case offline RL.

Theorem 5.1. For all π ∈ Π β ∩ ˜ Π ∗ , (˜ v ∗ , ˜ f π ) is a saddle point of L ( v, f ) in R S + × R S×A + .

Proof (sketch). The key of the proof is the following identity of Lagrangian.

For all π ∈ Π ∩ ˜ Π ∗ , we have

Now, observe that the third term D π V ( v ) and the fourth term D ∗ F ( f ) in Eq. (6) are nonnegative since all of v, f, ˜ d π and ˜ v ∗ -˜ q ∗ are nonnegative. Therefore, Lemma 5.1 implies that Lagrangian is bounded from below with L ( v, f ) ≥ ˜ J ∗ taking f = ˜ f π , while bounded from above with L ( v, f ) ≤ ˜ J ∗ taking v = ˜ v ∗ . Combining these two inequalities, a class of the saddle points of Lagrangian is identified as desired. The full proof is found in Appendix E.2.

Note that the previous studies on the LP-based formulation of RL typically consider the saddle points in a different domain, R S × R S×A + , where the domain of the primal variable v is unconstrained, unlike our setting with the constraint v ≥ 0 . This constraint is the key to establish the connection with the worst-case environment ˜ M .

Theorem 5.1 superficially suggests that finding the saddle points of L ( v, f ) is a reasonable way of finding the optimal policies with respect to ˜ M . However, in the next section, we show that it is unstable and easily breaks down by the function approximation error.

5.2 Instability of Unregularized Lagrangian

When the state space S is large, it is practically infeasible to find a saddle point of L ( v, f ) naïvely searching over the whole space R S + × R S×A + . Therefore, one may introduce compact function classes V ⊂ R S + and F ⊂ R S×A + and limit the scope of the search to these classes. Since we do not know the saddle points (which motivates us to find one), such function classes likely incur the function approximation error. Thus, it is likely that the saddle points (˜ v ∗ , ˜ f π ) may sit near the search space V × F , but not exactly included in the space, (˜ v ∗ , ˜ f π ) ̸∈ V × F .

In this context, we show even a tiny function approximation error can completely disrupt the connection established in Theorem 5.1. Consider the function classes V ϵ = { v ∈ R S + | v ≥ ˜ v ∗ + ϵ } and F ϵ = { f ∈ R S×A + | ∃ π ∈ Π β ∩ ˜ Π ∗ s.t. f ≤ ˜ f π -ϵ } with a small constant ϵ > 0 . Then, even though the function approximation error is small ( ϵ for both V and F in terms of the L ∞ -norm), the saddle point is collapsed to zero under the approximations with V ϵ and F ϵ . The proof is relegated to Appendix E.3.

Corollary 5.1. Suppose V ϵ and F ϵ are nonempty. Then, the saddle points of L ( v, f ) in V ϵ × R S×A + must satisfy f = 0 . Moreover, the saddle points of L ( v, f ) in R S + ×F ϵ must satisfy v = 0 .

5.3 Regularized Lagrangian

Corollary 5.1 shows that the saddle points of Lagrangian cannot be used as a reliable way to find the optimal policies with the function approximation. As a workaround, we introduce a regularized Lagrangian ,

where ¯ µ := µ + γ T β ∗ µ and ∥ v ∥ p, ¯ µ := { ∑ s ¯ µ ( s ) v p ( s ) } 1 /p denotes the L p (¯ µ ) -norm of the functions over S . Then, it is shown that the regularized Lagrangian is also connected with worst-case offline RL through its saddle points. To see this, let us define the regularized counterparts of ˜ d π , ˜ w π and ˜ f π with ˘ d π := (1 -γ )( I -γ ˜ T π ∗ ) -1 ˘ p π , ˘ w π := ˘ d π /µ , ˘ f π := ˘ w π ρ π , obtained by substituting the initial state distribution p 0 with ˘ p π := µ +(1 -γ )˜ v π ¯ µ .

Theorem 5.2. For all π ∈ Π β ∩ ˜ Π ∗ , (˜ v ∗ , ˘ f π ) is a saddle point of K ( v, f ) in R S + × R S×A + . Moreover, the primal solution ˜ v ∗ is unique on supp(¯ µ ) .

Proof (sketch). Similarly as the proof of Theorem 5.1, the key is the following identity.

Lemma 5.2. There exist U ∗ ∈ R such that, for all π ∈ Π β ∩ ˜ Π ∗ ,

The first claim of Theorem 5.2 then follows from the fact ˘ D π V ( v ) is nonnegative, and the second claim follows from the strong convexity of K ( v, f ) with respect to v on supp(¯ µ ) . The full proof is in Appendix E.5.

Comparing Theorems 5.1 and 5.2, it turns out that the regularization does not alter the primal part of the saddle points ˜ v ∗ . Moreover, since K ( v, f ) is strongly convex in terms of v , the regularized solution is more stable against the function approximation error as opposed to the unregularized solution. To see this, denote the regularized primal solution under the function approximation by

where V ⊂ R S + and F ⊂ R S×A + are compact function classes. Also denote the individual function approximation errors of V and F by

where ¯ B V := max { 1 + γB V , B V } and B V := max v ∈V ∥ v ∥ ∞ are scale factors of V . Then, the following lemma shows the stability of the approximate solution ˜ v ∗ ♯ in terms of the aggregated function approximation error

where B F := max f ∈F ∥ f ∥ ∞ is the scale factor of F . The proof is relegated to Appendix E.6. Lemma 5.3 (Stability of the regularized primal solution) . We have ∥ ˜ v ∗ ♯ -˜ v ∗ ∥ 2 , ¯ µ ≤ ε app , V ( V , F ) .

Note that the approximation error of F is trivially bounded by a simpler error term

i.e., the L 1 -error with respect to the function ˘ f π of the optimal on-support policy π ∈ Π β ∩ ˜ Π ∗ . Our definition of the error is weaker than that, measuring the error with respect to that of the possibly suboptimal on-support policy π ∈ Π β in exchange for the additional suboptimality cost 2(1 -γ ) ∥ ˜ v π -˜ v ∗ ∥ 1 , ¯ µ . This is beneficial if the optimal policies are difficult to approximate, like deterministic policies in a continuous action space, yet some near-optimal policies such as the soft-optimal policies π ( a | s ) ∝ exp {-η ˜ q ∗ ( s, a ) } are easy to approximate.

We also note that the idea of stabilizing the saddle-point-based policy optimization via a strongly convex regularizer is not new (Nachum et al., 2019; Lee et al., 2021; Zhan et al., 2022; Uehara et al., 2023). The major difference here (other than the truncation) is that we regularize the value function v (like Uehara et al. (2023)) but extract the information of the optimal policy from f (like Nachum et al. (2019); Lee et al. (2021); Zhan et al. (2022)), which, combined with our worst-case framework, results in a striking improvement in the sample complexity.

6 Worst-Case Minimax Reinforcement Learning

Now, we present a method to solve worst-case offline RL with the saddle points of K ( v, f ) . We first introduce a method of extracting policy from the dual variable f (Section 6.1), then show the suboptimality bound of the extracted policy (Section 6.2), which is our main result, and finally show the sample complexity bound taking into account the finite sample approximation (Section 6.3).

6.1 Policy Extraction

Motivated by Theorem 5.2, we propose a method of extracting the worst-case optimal policy π ∗ from the saddle point of K ( v, f ) . Specifically, we consider minimizing the loss function given by

where w : S → R is an auxiliary weight function, w ( s ) := max { 1 -γ, w ( s ) } is its lower clipping and Ξ ⊂ R S×A is a class of discriminator functions. Note that D Ξ ( f ; w,π ) is the integral probability metric (IPM) (Sriperumbudur et al., 2009) between f ( s, a ) µ ( s ) β ( a | s ) and w ( s ) µ ( s ) π ( a | s ) with respect to the discriminators Ξ . With a sufficiently rich Ξ , this implies D Ξ ( f ; w,π ) attains its minimum value (i.e., zero) only if π = π f , 4 thereby informally justifies the minimization of Eq. (11) as a way of policy extraction.

This approach introduces additional (functional) variables to be optimized, w : S → R and π : S → ∆( A ) . To simplify the notation, consider a parameter space Θ and suppose f , w and π share the same parameter space Θ , i.e., there exists a mapping Θ ∋ θ ↦→ ( f θ , w θ , π θ ) , and redefine the dual space with F = F (Θ) := { f θ | θ ∈ Θ } . 5 We define the associated function approximation error with

where ¯ B V , Ξ := max { ¯ B V , B Ξ , ∥ ˜ ξ ∗ ∥ ∞ } , B Ξ := max ξ ∈ Ξ ∥ ξ ∥ ∞ and

denotes the π -specific function approximation error of Θ . Here, B W := max θ ∈ Θ ∥ w θ ∥ ∞ is the boundedness of w θ ( s ) and ∥ · ∥ TV ,µ is the mean total variation (TV) distance with respect to µ , given by ∥ π -π ′ ∥ TV ,µ := E µ ∑ a | π ( a | s ) -π ′ ( a | s ) | .

Finally, we conclude this section by introducing key quantities of the policy extraction for the subsequent analysis. Let B Π := max θ ∈ Θ ∥ π θ /π 0 ∥ ∞ denote the size of the policy class with respect to some fixed base policy π 0 . Also let ϵ Ξ := min ξ ∈ Ξ ∥ ξ -˜ ξ ∗ ∥ 1 ,µ ( β + π 0 ) be the function approximation error of Ξ , where ˜ ξ ∗ ( s, a ) := ˜ q ∗ ( s, a ) -˜ v ∗ ( s ) is the optimal advantage function.

6.2 The Suboptimality Bound

Unifying the saddle-point problem and the policy extraction problem, we arrive at the aggregated loss function

where L SP ( θ ) := -min v ∈V K ( v, f θ ) is the loss of f θ as a dual solution (cf. Eq. (7)) and L X ( θ ) := D Ξ ( f θ ; w θ , π θ ) is the loss of the policy extraction from f θ to π θ (cf. Eq. (11)). Let us denote the corresponding estimation error by

We also define the aggregated function approximation error with

The following theorem establishes an upper bound on the policy suboptimality in terms of ϵ est ( θ ) and ε app , Π ( V , Θ , Ξ) .

Theorem 6.1. For all θ ∈ Θ , we have

Proof (sketch). At the heart of the proof is the following inequalities:

where Γ( π θ ) := E µ,π θ [˜ v ∗ ( s ) -˜ q ∗ ( s, a )] denotes the average action value gap with policy π θ and B ′ := B F +(1 -γ ) B Π . Here, the lower clipping of w (Eq. (11)) and the stability of the primal solution (Lemma 5.3) are instrumental in deriving the first and the second inequality, respectively. Then, the proof is completed by bounding ˜ J ∗ -˜ J ( π θ ) in terms of Γ( π θ ) invoking the performance difference lemma in the worst-case environment. See Appendix F.1 for the full proof.

Theorem 6.1 suggests that one can minimize the policy suboptimality up to the function approximation error on two conditions, i.e., the weight factor ∥ ˜ w π θ ∥ ∞ is appropriately bounded and the loss function L ( θ ) is minimized. In the following, we first discuss how to satisfy the first condition.

A trivial way of bounding ∥ ˜ w π θ ∥ ∞ is uniformly bounding it with respect to all θ ∈ Θ . Define ˜ C ∞ := max θ ∈ Θ ∥ ˜ w π θ ∥ ∞ , which we refer to as the uniform truncated concentrability (UTC) coefficient . Since ∥ ˜ w π θ ∥ ∞ ≤ ˜ C ∞ , we get the following simple suboptimality bound.

Corollary 6.1. For all θ ∈ Θ , we have

Note that ˜ C ∞ is always finite because of the compactness of the whole policy space ∆( A ) S and the continuity and well-definedness of π ↦→∥ ˜ w π ∥ ∞ . Thus, Eq. (18) is non-vacuous for arbitrary data distributions as opposed to the conventional concentrability-based results. Moreover, if the conventional bounds are non-vacuous, ˜ C ∞ recovers the conventional concentrability coefficient as ˜ w π = w π .

Eq. (18) can be further refined using the localized variants of the uniform coefficient ˜ C ∞ . The results are presented as Corollaries F.2 and F.3 in Appendix F.3 due to space limitation.

6.3 Sample Complexity Analysis

Now that given Corollary 6.1 (and Corollaries F.2 and F.3 as well) bounds the weight factor ∥ ˜ w π θ ∥ ∞ with some milder variants of the concentrability coefficient, the remaining task, minimizing L ( θ ) , is handled within the framework of the statistical learning, leading to a sample complexity bound. Let

be the empirical loss function where

is the one-sample loss function, z ≡ ( s, a, r, s ′ ) ∈ S × A × [0 , 1] ×S denotes a transition record. Note that Eq. (20) is an unbiased estimator of the objective function L ( θ ) . 6 Therefore, it is expected that the oracle loss L ( θ ) can be approximated with the empirical loss ˆ L ( θ ) and hence the oracle estimation error ϵ est ( θ ) can be approximated with the empirical estimation error

Formalizing such an intuition, the following corollary shows the empirical counterpart of Corollary 6.1. See Appendix F.5 for the proof.

Corollary 6.2. Let H ≡ H ( V , Θ , Ξ) := { z ↦→ ˆ L z ( θ ; v, ξ ) | θ ∈ Θ , v ∈ V , ξ ∈ Ξ } be the class of the one-sample loss functions and R n ( H ) be its Rademacher complexity (Definition B.1). Then, for all θ ∈ Θ and δ ∈ (0 , 1) , with probability 1 -δ , we have

where B all := (1 -γ ) B V + B F ¯ B V +(1 -γ ) 2 B 2 V +( B F + B W ) B Ξ is the aggregated scale factor.

The corollary above implies that minimizing Eq. (19), which is possible with the minimax optimizers such as the one developed by Thekumparampil et al. (2019), gives a near-optimal policy in terms of the worst-case environment ˜ M , up to the error proportional to the sum of the optimization error ˆ ϵ est ( θ ) , the approximation error ε app , Π ( V , Θ , Ξ) and the statistical error O ( R n ( H ) + n -1 / 2 ) . We refer to this method as worst-case minimax reinforcement learning (WMRL) .

The next corollary gives the sample complexity of WMRL in the simplified case where the function approximators V , Θ and Ξ are all finite sets.

Corollary 6.3. Suppose V , F and Ξ are all finite sets and ˆ ϵ est ( θ ) = ε app , Π ( V , Θ , Ξ) = 0 . Take any ϵ > 0 and 0 < δ < 1 . Then, we have ˜ J ∗ -˜ J ( π θ ) ≤ ϵ with probability 1 -δ if

where N := |V| | Θ | | Ξ | denote the product of the cardinalities of the function approximators.

Proof. It follows from Corollary 6.2 with Massart's lemma (Lemma B.2).

A few remarks follow. First, we can replace the UTC coefficient ˜ C ∞ with the localized variants (Definitions F.1 and F.2) to obtain tighter bounds, by making the same argument starting from Corollaries F.2 and F.3 instead of Corollary 6.1, respectively. Second, there are implicit dependencies B V ≥ ∥ ˜ v ∗ ∥ ∞ and B Ξ ≥ ∥ ˜ ξ ∗ ∥ ∞ due to the realizability ˜ v ∗ ∈ V and ˜ ξ ∗ ∈ Ξ that follows from ε app , Π ( V , Θ , Ξ) = 0 . Hence, Eq. (22) has an implicit γ -dependency through the scale factor B all , which brings an extra Θ((1 -γ ) -2 ) factor in the worst case. Table 1 adopts this form for the fairness of comparison.

7 Conclusion

To develop an offline RL method for challenging data distributions, we have introduced and studied a generalization of the conventional framework called worst-case offline RL . As a result, we have shown it is possible to learn a worst-case optimal policy without any data-support conditions. Moreover, the presented sample complexity bound strictly improves the previous state of the art under the single-policy realizability and the single-policy concentrability, suggesting the utility of the proposed method even with non-challenging data distributions.

We anticipate the presented results are readily extendable to continuous state-action spaces, except that the truncated concentrability coefficients are not unconditionally finite anymore. The results in Appendix F.3 are particularly useful in this context, yet the complete picture on the conditions of their boundedness largely remains to be studied in future work.

Acknowledgments and Disclosure of Funding

The author is grateful to LY Corporation for providing travel funding.

References

Abe, N., Melville, P., Pendus, C., Reddy, C. K., Jensen, D. L., Thomas, V. P., Bennett, J. J., Anderson, G. F., Cooley, B. R., Kowalczyk, M., Domick, M., and Gardinier, T. (2010). Optimizing debt

collections using constrained reinforcement learning. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , KDD '10, page 75-84, New York, NY, USA. Association for Computing Machinery.

Antos, A., Szepesvári, C., and Munos, R. (2008). Learning near-optimal policies with bellmanresidual minimization based fitted policy iteration and a single sample path. Machine Learning , 71:89-129.

Chen, J. and Jiang, N. (2019). Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning , pages 1042-1051. PMLR.

Chen, J. and Jiang, N. (2022). Offline reinforcement learning under value and density-ratio realizability: the power of gaps. In Uncertainty in Artificial Intelligence , pages 378-388. PMLR.

Chen, Y. and Wang, M. (2016). Stochastic primal-dual methods and sample complexity of reinforcement learning. arXiv preprint arXiv:1612.02516 .

Fang, X., Zhang, Q., Gao, Y., and Zhao, D. (2022). Offline reinforcement learning for autonomous driving with real world driving data. In 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC) , pages 3417-3422.

Fujimoto, S. and Gu, S. S. (2021). A minimalist approach to offline reinforcement learning. Advances in neural information processing systems , 34:20132-20145.

Jiang, N. and Huang, J. (2020). Minimax value interval for off-policy evaluation and policy optimization. In Proceedings of the 34th International Conference on Neural Information Processing Systems , NIPS '20, Red Hook, NY, USA. Curran Associates Inc.

Kumar, A., Zhou, A., Tucker, G., and Levine, S. (2020). Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:1179-1191.

Lee, J., Jeon, W., Lee, B., Pineau, J., and Kim, K.-E. (2021). Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning , pages 6120-6130. PMLR.

Levine, S., Kumar, A., Tucker, G., and Fu, J. (2020). Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 .

Liu, Y., Swaminathan, A., Agarwal, A., and Brunskill, E. (2020). Provably good batch off-policy reinforcement learning without great exploration. Advances in neural information processing systems , 33:1264-1274.

Munos, R. (2003). Error bounds for approximate policy iteration. In Proceedings of the Twentieth International Conference on International Conference on Machine Learning , ICML'03, page 560-567. AAAI Press.

Munos, R. and Szepesvári, C. (2008). Finite-time bounds for fitted value iteration. Journal of Machine Learning Research , 9(5).

Nachum, O., Dai, B., Kostrikov, I., Chow, Y., Li, L., and Schuurmans, D. (2019). Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02074 .

Ozdaglar, A. E., Pattathil, S., Zhang, J., and Zhang, K. (2023). Revisiting the linear-programming framework for offline RL with general function approximation. In International Conference on Machine Learning , pages 26769-26791. PMLR.

Prudencio, R. F., Maximo, M. R., and Colombini, E. L. (2023). A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems .

Puterman, M. L. (2014). Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons.

Rashidinejad, P., Zhu, H., Yang, K., Russell, S., and Jiao, J. (2023). Optimal conservative offline RL with general function approximation via augmented lagrangian. In The Eleventh International Conference on Learning Representations .

Shalev-Shwartz, S. and Ben-David, S. (2014). Understanding machine learning: From theory to algorithms . Cambridge university press.

Sriperumbudur, B. K., Fukumizu, K., Gretton, A., Schölkopf, B., and Lanckriet, G. R. (2009). On integral probability metrics, \ phi-divergences and binary classification. arXiv preprint arXiv:0901.2698 .

Thekumparampil, K. K., Jain, P., Netrapalli, P., and Oh, S. (2019). Efficient algorithms for smooth minimax optimization. In Wallach, H., Larochelle, H., Beygelzimer, A., d'Alché-Buc, F., Fox, E., and Garnett, R., editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc.

Uehara, M., Kallus, N., Lee, J. D., and Sun, W. (2023). Offline minimax soft-q-learning under realizability and partial coverage. In Oh, A., Naumann, T., Globerson, A., Saenko, K., Hardt, M., and Levine, S., editors, Advances in Neural Information Processing Systems , volume 36, pages 12797-12809. Curran Associates, Inc.

Uehara, M. and Sun, W. (2022). Pessimistic model-based offline reinforcement learning under partial coverage. In International Conference on Learning Representations .

Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. (2021). Bellman-consistent pessimism for offline reinforcement learning. In Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems .

Xie, T., Foster, D. J., Bai, Y ., Jiang, N., and Kakade, S. M. (2022). The role of coverage in online reinforcement learning. arXiv preprint arXiv:2210.04157 .

Yin, M. and Wang, Y.-X. (2021). Towards instance-optimal offline reinforcement learning with pessimism. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W., editors, Advances in Neural Information Processing Systems , volume 34, pages 4065-4078. Curran Associates, Inc.

Yu, C., Liu, J., Nemati, S., and Yin, G. (2021). Reinforcement learning in healthcare: A survey. ACM Comput. Surv. , 55(1).

Yu, T., Thomas, G., Yu, L., Ermon, S., Zou, J. Y., Levine, S., Finn, C., and Ma, T. (2020). MOPO: Model-based offline policy optimization. Advances in Neural Information Processing Systems , 33:14129-14142.

Zanette, A. (2023). When is realizability sufficient for off-policy reinforcement learning? In International Conference on Machine Learning , pages 40637-40668. PMLR.

Zhan, W., Huang, B., Huang, A., Jiang, N., and Lee, J. (2022). Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory , pages 2730-2775. PMLR.

Zhang, J., Hong, M., Wang, M., and Zhang, S. (2021). Generalization bounds for stochastic saddle point problems. In International Conference on Artificial Intelligence and Statistics , pages 568-576. PMLR.

A Additional Discussion on Table 1

In Table 1, we compare our result with the previous results on the sample complexity of the conventional offline RL under the weakest known assumptions. The weakest known means that it relies only on the assumptions of the single-policy realizability and the single-policy concentrability. Specifically, they do not assume the Bellman completeness (Munos and Szepesvári, 2008; Xie et al., 2021; Chen and Jiang, 2022) or its variants (Zanette, 2023; Rashidinejad et al., 2023), the model-based realizability (Uehara and Sun, 2022) or the all-policy realizability (Jiang and Huang, 2020), which

are strictly stronger than the single-policy realizability and deemed to be rather stringent (Chen and Jiang, 2019). Below, we discuss each of the methods listed in the table.

Leveraging the Lagrangian-based formulation of RL, Zhan et al. (2022) showed the polynomial sample complexity only with the single-policy realizability and the single-policy concentrability for the first time. However, the order of the sample complexity bound O ( ϵ -6 ) is significantly looser than the standard statistical rate O ( ϵ -2 ) . Moreover, their realizability assumption requires the function approximators to include the value/weight functions associated with a policy π n depending on the sample size n . In particular, the resulting realizability condition is difficult to interpret since there is no explicit characterization of π n .

̸

Chen and Jiang (2022) took a different approach, the pessimistic value learning, achieving the statistically reasonable rate O ( ϵ -2 ) . One of the main drawbacks of their result is, however, that the sample complexity bound blows up if the action value gap C gap is near zero. Here, the action value gap is defined as the minimum gap in the values of the best action and the second best action, C gap = min s ∈S max a ∈A { q ∗ ( s, a ) -max a ′ = a q ∗ ( s, a ′ ) } , which becomes (near) zero if there exist two actions that are (near) optimal for even one state. In addition, it only competes with the optimal policy π ∗ and requires the data to cover the corresponding visitation distribution d π ∗ . Finally, the time-horizon dependency O ( H 5 ) is a bit worse than the other results in the table.

Ozdaglar et al. (2023) showed two distinct results: one requiring a completeness-type assumption and the other requiring realizability and action-value-gap assumptions, in addition to concentrability. We included the latter to the table. Roughly speaking, their result is similar to that of Chen and Jiang (2022) except with the difference in the infinite/finite time horizons. Consequently, it also requires the action gap to be bounded away from zero. We note that their algorithm relies on a constrained LP, where the number of the constraints is equal to the size of S , making it possibly difficult to scale to practical problems.

Uehara et al. (2023) also proposed two distinct methods: one establishes slower O ( ϵ -8 ) rate with the entropy regularization method, and the other establishes a sample complexity depending on so-called the soft action value gap . We only shows the latter in Table 1. For the latter result, the new condition on the soft action value gap relaxes those imposed on the ordinary action value gap by Chen and Jiang (2022); Ozdaglar et al. (2023). The order of the resulting sample complexity bound is O ( ϵ -2+4 /β gap ) , where β gap > 0 corresponds to the lower-tail exponent of the distribution of the state-wise action

value gaps. We also note that these bounds are explicitly depending on the size of the action space A , which could be a potential drawback of the entropy-based method.

Compared to these results, our sample complexity bound has the following advantages. First of all, it gives the performance guarantees even if there is no concentrable policies. Second, it only requires the model-free realizability with respect to a (fixed) worst-case optimal policy ˜ π ∗ . Third, it achieves the statistically reasonable rate O ( ϵ -2 ) without any dependencies on the action value gap. Finally, it has no explicit dependency in the algorithm on the size of S and A , even in the policy extraction process.

B Rademacher Complexity and Uniform Convergence

In this section, we introduce the Rademacher complexity and its properties as well as the celebrated uniform convergence theorem. Below, let Z be a sample space, p ∈ ∆( Z ) be a probability distribution on it, and G ⊂ R Z be a set of functions from Z to R .

Definition B.1 (Rademacher complexity) . The Rademacher complexity of G with the sample size n ≥ 1 is given by

where E σ n ,z n denotes the expectation with respect to samples σ n = { σ i } n i =1 and z n = { z i } n i =1 drawn from Uniform n ( {-1 , +1 } ) and p n , respectively.

Lemma B.1 (Uniform convergence theorem) . Suppose ∥ g ∥ ∞ ≤ c for all g ∈ G . Then, for all δ ∈ (0 , 1) , we have

with probability 1 -δ on the draw of z n ∼ p n .

Proof. Refer to Claim 1, Theorem 26.5, Shalev-Shwartz and Ben-David (2014) for oneside high-probability bound and apply it to both of sup g ∈G { 1 n ∑ n i =1 [ g ( z i )] -E [ g ( z 1 )] } and sup g ∈G ( -1) { 1 n ∑ n i =1 [ g ( z i )] -E [ g ( z 1 )] } setting the confidence parameter to δ/ 2 . The proof is completed by taking the union of the events that these high-probability bounds do not hold.

The following is another well-known result of the Rademacher complexity.

Lemma B.2 (Massart's lemma) . For a finite set G , we have

Refer to Shalev-Shwartz and Ben-David (2014), Lemma 26.8.

where

Proof.

C Basic Properties of Regularized Lagrangian

In this section, we show basic property of regularized Lagrangian (Eq. (7)).

Lemma C.1 (Primal Lipschitz continuity) . For all v, v ′ : S × A → [0 , (1 -γ ) -1 ] and f ∈ F , we have

Proof. Observe that by Eq. (7)

The second term on the RHS is further bounded as

where (a) follows from Hölder's inequality and (b) follows from the triangle inequality. We obtain the desired result by summing up both sides of the two inequalities since µ ≤ ¯ µ .

Lemma C.2 (Stability of minimax value against dual error) . For all v ≥ 0 and π ∈ Π β , we have

where ¯ B V := max { 1 + γB V , B V } .

Proof. Observe that

where (a) follows from the nonegativity of D π V ( v ) , (b) from Lemma E.3, (c) from D π F ( ˘ f π ) = 0 , (d) from ˜ q π ( s, a ) -γ ˜ T v ( s, a ) = r ( s, a ) and (e) from Hölder's inequality with | r + γ ˜ T v -v | ≤ ¯ B V . The claim is then proved by the continuity of U ( π ) ,

D Proofs of Section 4

D.1 Proof of Theorem 4.1

Proof. The claim ˜ M ∈ U is trivial from Definition 4.1. The other claim, J ( π | ˜ M ) ≤ J ( π |M ′ ) , follows from that i) ˜ r ( s, a ) ≤ r ( s, a ) with ˜ r ( s, a ) = E y ∼ R ( y | s,a ) [ y ] for all s ∈ S and a ∈ A and ii) the truncated transition probability ˜ T ( ·| s, a ) , when in conflict with T ( ·| s, a ) , leads to the absorbing state ⊥ , which has the lowest possible cumulative discounted value, zero.

D.2 Proof of Corollary 4.1

Proof. Since the inequality is trivial from Eq. (1), we show the equality. Observe that the state-action pair ( s t , a t ) at time t ≥ 0 stays inside supp( µβ ) almost surely for all t ≥ 0 if supp( d π π ) ⊂ supp( µβ ) . Hence, the law of the reward sequence { r t } t ≥ 0 generated with π is the same under M and ˜ M , leading to J ( π | ˜ M ) = J ( π |M ) .

D.3 Proof of Lemma 4.1

Proof. We show denying the conclusion results in contradiction. Take a policy π ∈ ˜ Π ∗ . Let χ β ( s, a ) := I { ( s, a ) ∈ supp( β ) } be the indicator function of supp( β ) . Let z π ( s ) := ∑ a χ β ( s, a ) π ( a | s ) be the mass of π ( ·| s ) inside the support of β , which satisfies 0 ≤ z π ( s ) ≤ 1 for all s ∈ S .

Let π ′ : S → ∆( A ) be a policy proportional to π with the support restricted to supp( β ) , i.e., z π ( s ) π ′ ( a | s ) = χ β ( s, a ) π ( a | s ) for all s ∈ S and a ∈ A .

By the definitions of ˜ r π and ˜ T π , we now have

and

for all v : S → [0 , ∞ ) and s ∈ S . Therefore, we have for all s ∈ S

which leads to the strong optimality ˜ v π ′ = ˜ v ∗ . However, we also have π ′ ∈ Π β by the definition, contradicting with the assumption ˜ Π ∗ ∩ Π β = ∅ .

E Proofs of Section 5

E.1 Proof of Lemma 5.1

The proof relies on two lemmas (Lemmas E.1 and E.2), where the second one is built on top of the first one. Then, Lemma 5.1 is immediately proved as a special case of Lemma E.2 with π being restricted to ˜ Π ∗ .

Lemma E.1 gives a saddle-point decomposition of Lagrangian ignoring the offline constraint.

Lemma E.1 (Incomplete saddle-point decomposition of Lagrangian) . For all π : S → ∆( A ) ,

Proof. Comparing the LHS and the RHS, it suffices to show

This is seen by simplifying the RHS of the above equation as follows,

Here, (a) follows from ∑ a π ( a | s ) δ TD v ( s, a ) = ( r π + γ T π v -v )( s ) and ˜ r π = ( I -γ T π ) v π , (b) from ( I -γ T π ∗ ) being the adjoint operator of ( I -γ T π ) , (c) from (1 -γ ) p 0 = ( I -γ T π ∗ ) d π , and (d) from the definition of J ( π ) .

Note that the second term of Eq. (24) may have no root with respect to f if the data support does not cover the visitation distribution d π , hence incomplete . Lemma E.2 gives a modification of Eq. (24) to fix this problem.

Lemma E.2 (Generalized saddle-point decomposition of Lagrangian) . For all π ∈ Π β , we have

Proof. Since Lagrangian is defined with r ( s, a ) and T ( ·| s, a ) only on the support of the offline data distribution supp( µβ ) , Lemma E.1 together with the indistinguishability of ˜ M and M gives

where ˜ δ TD v := ˜ r + γ ˜ T v -v . The second term of the RHS of the above equation further evaluated by separating the summation to the on-support and off-support terms

where (a) follows from that ˜ f π ( s, a ) µ ( s ) β ( a | s ) = ˜ d π ( s ) π ( a | s ) if ( s, a ) ∈ supp( µβ ) and δ TD v ( s, a ) = -v ( s ) if ( s, a ) ̸∈ supp( µβ ) , (b) from transforming ˜ r in the first term, within ˜ δ TD , with ˜ r = ( I -γ ˜ T )˜ v π +(˜ q π -˜ v π ) and simplify the resulting (˜ q π -˜ v π ) with E a ∼ β ( a | s ) [ ˜ f π ( s, a )(˜ q π ( s, a ) -˜ v π ( s ))] = 0 , and (c) from evaluating the last summation as zero with the fact supp( µπ ) ⊂ supp( µβ ) .

Finally, Lemma 5.1 is shown by taking π such that π ∈ Π β ∩ ˜ Π ∗ .

E.2 Proof of Theorem 5.1

Proof. First, see Appendix E.1 for the proof of Lemma 5.1. Then, recall that (˜ v ∗ , ˜ f π ) is a saddle point of L ( v, f ) if L (˜ v ∗ , f ) ≤ L (˜ v ∗ , ˜ f π ) ≤ L ( v, ˜ f π ) for all v, f ≥ 0 . By Eq. (6) with the nonnegativity of D π V ( v ) and D ∗ F ( f ) , it suffices to show D π V (˜ v ∗ ) = 0 and D ∗ F ( ˜ f π ) = 0 . The former follows from ˜ v ∗ ( s ) = 0 for all s ̸∈ supp( µ ) and the latter follows from E a ∼ β ( a | s ) [ ˜ f π ( s, a ) { ˜ v ∗ ( s ) -˜ q ∗ ( s, a ) } ] = ˜ w π ( s ) { ˜ v ∗ ( s ) -E a ∼ π ( a | s ) [˜ q ∗ ( s, a )] } = 0 for all π ∈ Π β ∩ ˜ Π ∗ .

E.3 Proof of Corollary 5.1

Proof. Note that the existence of the saddle points is always ensured since the (restricted) domains of v and f are all convex. Therefore, the first claim is reduced to 0 = argmax f ≥ 0 min v ∈V ϵ L ( v, f ) , which is shown by Lemma 5.1 and the fact ( I -γ ˜ T )( v -˜ v ) ≥ (1 -γ ) ϵ > 0 for all v ∈ V ϵ . Besides, the second claim is reduced to 0 = argmin v ≥ 0 max f ∈F ϵ L ( v, f ) , which is also shown by Lemma 5.1 and the fact E µ,β [( f -˜ f π )( s, a ) ( I -γ ˜ T )( v -˜ v ∗ )( s, a )] ≥ (1 -γ ) ϵ E ˜ T π ∗ µ [( v -˜ v ∗ )( s )] , which attains the minimum uniquely with v = 0 .

E.4 Proof of Lemma 5.2

Define

Then, Lemma 5.2 is proved as a special case of the following lemma with the restriction π ∈ Π β ∩ ˜ Π ∗ and U ∗ := (1 -γ ) E µ [˜ v ∗ ( s )] + (1 -γ 2 ) 2 ∥ ˜ v ∗ ∥ 2 2 , ¯ µ .

Lemma E.3 (Generalized saddle-point decomposition of regularized Lagrangian) . For all π ∈ Π β ,

Proof. Observe that

Thus, multiplying both sides with (1 -γ ) 2 / 2 and plugging to Eq. (7), we get

Now, applying Lemma E.2 on the last two terms of the RHS with the formal substitution p 0 ← ˘ p π , we have

where ˜ J ( π ) , ˜ d π ( s ) and ˜ f π ( s, a ) in Lemma 5.1 are replaced with ∑ s ˘ d π ( s ) r π ( s ) , ˘ d π ( s ) and ˘ f π ( s, a ) , respectively, due to the substitution. Finally, the proof is concluded by simplifying the first term on

the RHS

where (a) follows from the fact ( I -γ ˜ T π ∗ ) -1 is the adjoint operator of ( I -γ ˜ T π ) -1 and (b) is owing to ˜ v π = ( I -γ ˜ T π )˜ r π .

E.5 Proof of Theorem 5.2

Proof. First, see Appendix E.4 for the proof of Lemma 5.2. Then, note that the third term ˘ D π V ( v ) is nonegative for all v ≥ 0 since ˘ d π ≥ 0 . Thus, we have K ( v, f ) ≥ U ∗ taking f = ˘ f π and K ( v, f ) ≤ U ∗ taking v = ˜ v ∗ . The first claim is then proved by combining these two inequalities, in the same manner as the proof of Theorem 5.1. The second claim, the uniqueness of ˜ v ∗ , follows from the strong convexity of K ( · , f ) .

E.6 Proof of Lemma 5.3

Proof. Denote the primal excess risk function relative to F by

and its minimizer by v ∗ := argmin v ≥ 0 max f ∈F K ( v, f ) . Now, the strong convexity of K ( · , f ) implies the strong convexity of ε PER and thus, for all v : S → R ,

We utilize this bound via the triangle inequality

Then, each term on the RHS is bounded by

completing the proof. The proofs of Eqs. (27) and (28) are separately given below.

Proof of Eq. (27) . Recall that by definition

Then, on the RHS, the first term is upper bounded with U ∗ by Theorem 5.2 and the second term (without the negative sign) is lower bounded by Lemma C.2, leading to the inequality

for all π ∈ Π β . The proof is completed by taking the minimum with respect to π .

Proof of Eq. (28) . It is immediately seen from

where the last inequality follows from Lemma C.1.

F Proofs of Section 6

F.1 Proof of Theorem 6.1

Recall that the average action value gap is given by

The following lemma establishes the connection of Γ( π θ ) and ϵ est ( θ ) .

Lemma F.1. For all θ ∈ Θ , we have

Then, Theorem 6.1 is proved by combining Lemma F.1 with the following lemma.

Lemma F.2. For all π : S → ∆( A ) , we have

Proof. It follows directly from Hölder's inequality with the performance difference lemma for the truncated environment (Lemma F.3).

F.2 Proof of Lemma F.1

The proof relies on the following variant of the performance difference lemma adopted for the worst-case environment ˜ M .

Lemma F.3 (Worst-case performance difference lemma) . For all π : S → ∆( A ) , we have

Consequently, for all π ∈ Π β ,

Proof. Observe that

where the second equality follows from ( I -γ ˜ T π ∗ ) ˜ d π = p 0 and the third equality follows from ˜ v ∗ ( s ) = 0 and ˜ T π v ( s ) = 0 for all s ̸∈ supp( µ ) and π : S → ∆( A ) . Thus,

where the second equality follows from ˜ r π + γ ˜ T π ˜ v ∗ = P π ˜ q ∗ . This proves the first claim.

The second claim follows from the definition of ˜ f π .

Let ˘ p ∗ := µ +(1 -γ )˜ v ∗ ¯ µ be an alternative (unnormalized) initial state distribution and

be the corresponding action visitation weight function. Applying Lemma F.3 to this setting, we obtain the following corollary.

Corollary F.1. Define

and let ˜ U ∗ := (1 -γ ) ∑ s ˘ p ∗ ( s ) ˜ v ∗ ( s ) be its maximum. Then, we have

Now we are prepared to prove Lemma F.1.

Proof of Lemma F.1. Let ξ ∗ ♯ ∈ argmin ξ ∈ Ξ ∥ ξ -˜ ξ ∗ ∥ 1 ,µβ 0 . Now, observe that

where (a) follows from ∥ ξ ∗ ♯ -˜ ξ ∗ ∥ 1 ,µπ θ ≤ B Π ϵ Ξ , (b) from Hölder's inequality with -˜ ξ ∗ ( s, a ) ≥ 0 and w θ ≥ 1 -γ , and (c) from Eq. (11), (d) from ∥ ξ ∗ ♯ -˜ ξ ∗ ∥ 1 ,µβ ≤ B Π ϵ Ξ , and (e) from the definition of D ∗ F ( f ) (Lemma 5.1) and L X ( θ ) . In the rest of the proof, we bound D ∗ F ( f θ ) + L X ( θ ) with ϵ est ( θ ) .

Fix any θ ′ ∈ Θ and π ∈ Π β . Let ˜ v ∗ ♯ ∈ argmin v ∈V max f ∈F K ( v, f ) and ˘ f ∗ ♯ ∈ argmax f ∈F K (˜ v ∗ ♯ , f ) . Also let ¯ K ∗ := K (˜ v ∗ ♯ , ˘ f ∗ ♯ ) = min v ∈V max f ∈F K ( v, f ) be the corresponding minimax value. Then, ¯ K ∗ + L SP ( θ ) is lower-bounded with D ∗ F ( f θ ) -D ∗ F ( f θ ′ ) up to an error term,

where (a) follows from compromising the maximum with v = ˜ v ∗ ♯ ∈ V , (b) from the definition of ˘ f ∗ ♯ above, (c) from Lemma 5.2, (d) from ∥ f θ ∥ ∞ , ∥ f θ ′ ∥ ∞ ≤ B F and (e) from Lemma 5.3. Moreover, let

be the intrinsic error of θ ′ and fix arbitrary π ∗ ∈ Π β ∩ ˜ Π ∗ . Then, ¯ K ∗ + L SP ( θ ′ ) is upper-bounded with Φ( θ ′ ) up to another error term,

where (a) follows from Lemma 5.2, (b) from Theorem 5.2, (c) from Lemma C.1 and Lemma C.2 and (d) from Lemma 5.3.

Subtracting both sides of Eq. (31) from those of Eq. (32), we get

which, summed with L X ( θ ) ≤ L X ( θ ) + ϵ est ( θ ′ ) = ϵ est ( θ ) -L SP ( θ ) + L ( θ ′ ) on both sides, yields

The remaining task is to choose θ ′ ∈ Θ such that D ∗ F ( f θ ′ ) + L X ( θ ′ ) + Φ( θ ′ ) is nicely bounded. Now, observe that

where (a) follows from ˘ f π, ∗ ≥ ˘ f π (see Eq. (29) for the definition of ˘ f π, ∗ ), (b) from Lemma F.3, (c) from Hölder's inequality and (d) from ˘ p ∗ ≤ 2¯ µ and ∥ ˜ ξ ∗ ∥ ∞ ≤ ¯ B V , Ξ . Moreover, for all π ∈ Π β , we have

where (a) follows from the telescoping and (b) from ∥ ξ ∥ ∞ ≤ B Ξ for all ξ ∈ Ξ and ∥ ˘ w π -w θ ∥ 1 ,µ ≤ ∥ ˘ w π -w θ ∥ 1 ,µ since ˘ w π ≥ 1 -γ . Adding both sides of Eqs. (34) and (35) and taking π and θ ′ as the minimizers of Eqs. (12) and (13), respectively, we arrive at

where the last inequality follows from Φ( θ ′ ) ≤ ϵ Θ (Eq. (12)). The proof is concluded by adding both sides of Eqs. (30), (33) and (36) and simplifying the error terms with Eq. (16).

F.3 Extensions of Corollary 6.1

The suboptimality bound established by Corollary 6.1 requires the uniform boundedness of ∥ ˜ w π θ ∥ ∞ . With more careful analysis, however, we can obtain policy suboptimality bounds with milder conditions. We present two of such upper bounds below.

To this end, we introduce two types of local truncated concentrability. Let Π ≡ Π(Θ) := { π θ | θ ∈ Θ } be the set of all the policy candidates.

Definition F.1. Let ˜ C ϵ := max {∥ ˜ w π ∥ ∞ | ˜ J ∗ -˜ J ( π ) ≤ ϵ, π ∈ Π } be the ϵ -weakly local truncated concentrability ( ϵ -WLTC) coefficient for ϵ > 0 . We also define the limit WLTC coefficient as ˜ C 0 := lim ϵ → 0+ ˜ C ϵ .

Definition F.2. Let ˜ c ϵ := max {∥ ˜ w π ∥ ∞ | max ( s,a ) ∈ supp( µπ ) { ˜ v ∗ ( s ) -˜ q ∗ ( s, a ) } ≤ ϵ, π : S → ∆( A ) } be the ϵ -strongly local truncated concentrability ( ϵ -SLTC) coefficient for ϵ > 0 . We also define the limit SLTC coefficient as ˜ c 0 := lim ϵ → 0+ ˜ c ϵ .

Intuitively, both the WLTC and SLTC coefficients bound the norm of ˜ w π locally for near-optimal policies π . The difference is the ways they measure the locality: WLTC uses the policy suboptimality and SLTC uses the maximum action value gap. Note that WLTC dominates SLTC, ˜ c ϵ ≤ ˜ C ϵ ≤ ˜ C ∞ , if Π(Θ) covers the entire policy space ∆( A ) S . In general, there is no particular order between WLTC and SLTC and we just have ˜ C ϵ ≤ ˜ C ∞ .

The following lemma is the foundation of our local concentrability results.

Lemma F.4. For any π ∈ Π , we have

where ϵ 0 ( π ) := √ ˜ c 0 Γ( π ) / (1 -γ ) .

Proof.

Combining it with Lemma F.1 and discarding the non-asymptotic term for the simplicity, we get the first bound as the following corollary.

where a ≲ b means lim sup b → 0+ a/b ≤ 1 .

In words, Eq. (37) allows us to replace the uniform concentrability coefficient ˜ C ∞ with the limit SLTC coefficient ˜ c 0 at the cost of the slower convergence rate due to the square root.

Moreover, a faster bound can be obtained exploiting the limit WLTC coefficient ˜ C 0 instead of ˜ c 0 , leading to our second bound.

Proof. Observe that Corollary F.2 implies ∥ ˜ w π θ ∥ ∞ is bounded with ˜ C ϵ , where ϵ is taken as the RHS of Eq. (37). The claim thus follows from Theorem 6.1 with ∥ ˜ w π θ ∥ ∞ ≤ ˜ C ϵ → ˜ C 0 as ϵ est ( θ ) + ε app , Π ( V , Θ , Ξ) → 0 .

Similarly as Corollary F.2, in comparison to Theorem 6.1, the coefficient of the upper bound is improved from ˜ C ∞ to ˜ C 0 , meaning that asymptotically we only need the weakly local, not uniform, concentrability.

F.4 Proof of Lemma F.4

Let us denote the set of the ϵ -strongly near-optimal policies by

where ˜ A ∗ ϵ ( s ) := { a ∈ A| ˜ v ∗ ( s ) -˜ q ∗ ( s, a ) ≤ ϵ } is the ϵ -optimal action subset for s ∈ S . 7 Note that by definition ˜ Π ∗ 0 = ˜ Π ∗ and ˜ Π ∗ ϵ is monotone nondecreasing with respect to ϵ , reaching the set of the all policies with ϵ ≥ 1 / (1 -γ ) . Then, the SLTC coefficient can be written in terms of ˜ Π ∗ ϵ ,

Now, to prove Lemma F.4, we show that there exists a strongly optimal policy ¯ π ∈ ˜ Π ∗ ϵ that approximates the target policy π if Γ( π ) is small.

Lemma F.5. For all π : S → ∆( A ) and ϵ > 0 , we have

Proof. Take π ′ ∈ ˜ Π ∗ ϵ as a projection of π onto ˜ Π ∗ ϵ , i.e.,

with arbitrary π 0 ∈ ˜ Π ∗ ϵ and c ( s ) := 1 -∑ a ∈ ˜ A ∗ ϵ ( s ) π ( a | s ) . Observe that, by the triangle inequality,

Then, we have

Now, plugging

on the RHS, we arrive at the desired inequality

As a corollary, we can also bound the difference of their policy values based on the SLTC coefficient.

Corollary F.4. For all π : S → ∆( A ) and ϵ > 0 , we have

Proof. Take arbitrary ¯ π ∈ ˜ Π ∗ ϵ and observe that

Thus, we have

where (a) follows from the fact ˜ T π ∗ and ˜ T ∗ can be identified as non-expansive mappings of types L 1 (supp( µ )) → L 1 (supp( µ )) and L 1 (supp( µβ )) → L 1 (supp( µ )) , respectively, and (b) follows from ¯ π ∈ ˜ Π ∗ ϵ . Finally, observe that

where (a) follows from the triangle inequality, (b) from the fact | ˜ r ( s, a ) | ≤ 1 and (c) from Eq. (39). The proof is concluded by applying Lemma F.5 on the RHS.

We now arrive at the following corollary, which immediately implies Lemma F.4 as a special case with the substitution ϵ = ϵ 0 ( π ) .

Corollary F.5. For all ϵ > 0 , we have

for all ¯ π ∈ ˜ Π ∗ ϵ . Thus, decomposing the suboptimality as

and applying Corollary F.4 results in the desired result.

F.5 Proof of Corollary 6.2

Proof. The celebrated uniform convergence theorem (Lemma B.1) with the boundedness ∣ ∣ ∣ ˆ L z ( θ ; v, ξ ) ∣ ∣ ∣ ≤ B all implies that, for all δ ∈ (0 , 1) ,

with probability 1 -δ . Now, by definition, ϵ est ( θ ) is uniformly approximated with ˆ ϵ est ( θ ) up to as twice as the statistical error given by Eq. (40). Plugging this into Eq. (18), we obtain the desired bound.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The worst-case policy value is introduced in Section 4 and the sample complexity bound is given by Corollary 6.3, wherein the realizability condition ( ε app , Π ( V , Θ , Ξ) = 0 ) is the only major assumption. The improvements over the previous sample complexity bounds are summarized in Table 1 and discussed in Section 2 and Appendix A.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations are discussed in Section 7.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All the assumptions are explicit in the main text. The complete proof is presented in Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: No experimental result is presented.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [NA]

Justification: No experimental result is presented.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: No experimental result is presented.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: No experimental result is presented.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: No experimental result is presented.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: This paper is purely theoretical.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no immediate societal impact to be considered of the present work since the results are purely theoretical.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: No dataset or model is disclosed.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: The paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this submission, the authors propose some new bounds for offline reinforcement learning. Their contribution is twofold. First, they remove the classical data support assumption by solving a relaxed problem, in which any non observed transition is replaced by a transition to an absorbing state. This new MDP yields a lower reward than the original one, hence the worst-case name, but can be learned with the given data. Second, they use a regularized version of the Linear Programming formulation of the optimal policy problem to estimate the good policy. Their improved bounds come from the direct use of the dual variable to obtain the policy, rather than using a policy-improvement step on the primal variable (the value function).\n\nSTRENGTHS:\nThe approach is interesting with simultaneously a new “worst-case” setting and new way to obtain result with the regularized linear programming approach.\n\nThe paper is well written, the proofs seem correct (I did not check them carefully).\n\nWEAKNESSES:\nThe paper is very technical and is written in a linear way. I know that the space constraint is strong, but I would have liked a better summary of the main idea at the beginning of the paper so that the position with respect to the relative work would be easier to understand.\n\nThe notations often differ only by a slight detail. Using larger difference may be beneficial for the readers.\n\nQUESTIONS:\nThe bounds obtained are upper bound. Are the author aware of corresponding lower bounds?\n\nTypos and misc.:\n- 167: formulation\n- 179: Why is $v \\geq 0$ so important\n- 209: $\\bar v$ vs $\\tilde v$ : hard to distinguish\n- 259: $\\epsilon_\\theta$ is often called an excess risk, or something similar.\n- 306: $n \\geq$ rather than $n =$\n\nLIMITATIONS:\nThe only limitation is that there is no numerical experiment to support the claim, but this should not be expected from such a theoretical paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-10"}

comments: SUMMARY:
This paper studies offline reinforcement learning with arbitrary data support. More specifically, they truncate the original environment so that the new environment (called truncated environment) is always covered by the offline data. They further show that the optimal policy of the truncated environment can be found by solving a regularized Lagrangian problem and analyze the corresponding sample complexity. As a byproduct, they attain a $O(1/\epsilon^2)$ sample complexity bound for learning an $\epsilon$-optimal policy against any comparator policy with bounded concentrability and realizability.

STRENGTHS:
1. The connection between the truncated environment and the regularized Lagrangian problem is very novel and useful in my opinion. 
2. The authors improve the sample complexity of learning with single-policy concentrability and realizability to $1/\epsilon^2$, which is a significant contribution.

WEAKNESSES:
1. In the sample complexity of Corollary 6.16.2/6.3, the rates scale with $\tilde{C}_{\infty}$, which is indeed the concentrability bound for all policies within the policy class. This seems stronger than the naive single-policy concentrability.
2. The algorithm requires the knowledge of the behavior policy, which might be hard to estimate in practice.

QUESTIONS:
Similar to the weaknesses:
1. can we get rid of $\tilde{C}_{\infty}$?
2. can the algorithm work without the knowledge of the behavior policy?

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-16

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper develops a framework for evaluating offline reinforcement learning methods without any data-support conditions. Traditional techniques rely on the concentrability coefficient to ground the difference between the offline data distribution and the induced policy distribution. However, the concentrability condition is often unrealistic in practice, especially when the state-action space is very large, as is frequently the case. This paper proves by construction a sample complexity upper bound of $O(\\epsilon^{-2}(1 - \\gamma)^{-4}\\ln(1/\\delta))$ is achievable without relying on the concentrability condition. This improves upon several bounds from previous work, involving similar assumptions.\n\nSTRENGTHS:\n- The paper shows some originality in creating a new framework that does not rely on the frequent concentrability condition found in offline RL literature. The associated restriction in this paper is even less-so than the single-policy concentrability condition found in Zhan, Huang, Huang, et al.\n- The paper simplifies the traditional offline RL evaluation framework by removing the need to directly incorporate pessimism or behavioral cloning via hyperparameters into the analysis.\n- The paper provides a strong theoretical proof to establish the stated upper bound.\n\nWEAKNESSES:\n- I feel the proposed truncated environment is overly restrictive and ignores information between similar state-action pairs, even if they're not represented in the support of the offline data distribution. Although the benefit of doing this is that it removes the need to control for pessimism and behavioral cloning, there could be some useful information discarded, which could help improve the bound derived in the paper.\n\nUPDATE: this has been addressed by the authors' followup.\n\nQUESTIONS:\nNo questions.\n\nLIMITATIONS:\nThe authors have mentioned that although their results extend to continuous state-action spaces, the concentrability coefficients are not unconditionally finite anymore.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies offline RL with no data support assumptions. To address the data coverage problem, the paper studies a new setting with worst-case policy value. It formulates the RL problem with Lagrangian and shows the instability compared with the regularized Lagrangian. Improved sample complexity is proved in the new framework.\n\nSTRENGTHS:\n1. The proposed new framework is novel and interesting in the study of offline RL.\n2. The paper provides analysis of the stability of lagrangian and regularized lagrangian.\n3. The paper proved an improved sample complexity.\n\nWEAKNESSES:\n1. The paper studies everything in a new framework, where the transition is constrained in the range of the data. It seems unfair to compare the results in the new framework with previous works, as they can also work for out-of-distribution data to some extent. As shown in Theorem 4.1, the loss will become smaller in the new framework.\n\n2. The behavior of the truncated transition kernel (2) is strange. As the indicator function does not contain any information about the next state s’, $\\sum_{s’} \\tilde T(s’|s,a) = \\chi_{\\mu, \\beta}(s,a)$, which takes values in 1 or 0 only. It does not match the leaky description in the paper, i.e., it may sum to less than unity.\n\n3. The paper misses comparison with some relevant papers. For example, [1] [2]\n\n[1] Rashidinejad et al 2022. Optimal conservative offline rl with general function approximation via augmented lagrangian. In ICLR.\n\n[2]  Ozdaglar et al. 2023. Revisiting the linear-programming framework for offline rl with general function approximation. In ICML.\n\n4. The superscripts used are too complex and lack explanations, making the writing hard to follow.\n\nQUESTIONS:\n1. Why do you think the realizability assumption in [ZHH+22] is $\\pi_n$-realizable? Do you think there is a huge influence on the result?\n\n2. What are the $\\bar w$ and $\\|\\cdot\\|_{1,\\bar w}$ in equation (10)?\n\n3. In equation (17), the empirical loss function is defined from the one-sample loss function. Is it possible to generalize the method using the batch loss function?\n\nLIMITATIONS:\nThe authors have addressed the limitations in section 7.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-27"}