Title: Bi-Level Offline Policy Optimization with Limited Exploration
Venue: NeurIPS 2023
Accepted: yes
Keywords: Offline Reinforcement Learning

Sample Efficiency

Regret Bound

Data Coverage
Abstract: We study offline reinforcement learning (RL) which seeks to learn a good policy based on a fixed, pre-collected dataset. A fundamental challenge behind this task is the distributional shift due to the dataset lacking sufficient exploration, especially under function approximation. To tackle this issue, we propose a bi-level structured policy optimization algorithm that models a hierarchical interaction between the policy (upper-level) and the value function (lower-level). The lower level focuses on constructing a confidence set of value estimates that maintain sufficiently small weighted average Bellman errors, while controlling uncertainty arising from distribution mismatch. Subsequently, at the upper level, the policy aims to maximize a conservative value estimate from the confidence set formed at the lower level. This novel formulation preserves the maximum flexibility of the implicitly induced exploratory data distribution, enabling the power of model extrapolation. In practice, it can be solved through a computationally efficient, penalized adversarial estimation procedure. Our theoretical regret guarantees do not rely on any data-coverage and completeness-type assumptions, only requiring realizability. These guarantees also demonstrate that the learned policy represents the ``best effort'' among all policies, as no other policies can outperform it. We evaluate our model using a blend of synthetic, benchmark, and real-world datasets for offline RL, showing that it performs competitively with state-of-the-art methods.

Bi-Level Offline Policy Optimization with Limited Exploration

Abstract

We study offline reinforcement learning (RL) which seeks to learn a good policy based on a fixed, pre-collected dataset. A fundamental challenge behind this task is the distributional shift due to the dataset lacking sufficient exploration, especially under function approximation. To tackle this issue, we propose a bi-level structured policy optimization algorithm that models a hierarchical interaction between the policy (upper-level) and the value function (lower-level). The lower level focuses on constructing a confidence set of value estimates that maintain sufficiently small weighted average Bellman errors, while controlling uncertainty arising from distribution mismatch. Subsequently, at the upper level, the policy aims to maximize a conservative value estimate from the confidence set formed at the lower level. This novel formulation preserves the maximum flexibility of the implicitly induced exploratory data distribution, enabling the power of model extrapolation. In practice, it can be solved through a computationally efficient, penalized adversarial estimation procedure. Our theoretical regret guarantees do not rely on any data-coverage and completeness-type assumptions, only requiring realizability. These guarantees also demonstrate that the learned policy represents the 'best effort' among all policies, as no other policies can outperform it. We evaluate our model using a blend of synthetic, benchmark, and real-world datasets for offline RL, showing that it performs competitively with state-of-the-art methods.

1 Introduction

Offline reinforcement learning (RL) is a task to learn a good policy using only a pre-collected, fixed dataset, without further exploration with the environment. This distinctive characteristic positions offline RL as a promising approach for solving real-world sequential decision-making problems in healthcare [35, 61], financial marketing [46], robotics [47] and education [32], as acquiring diverse or expert-quality data in these fields can be costly or practically unattainable.

Arguably, two of the biggest challenges in offline RL are the distributional shift between the datagenerating distribution and those induced by candidate policies, and the stringent requirements on the properties of function approximation [28]. It has been observed that, in practice, the distributional mismatch often results in unsatisfactory performance of many existing algorithms, and even amplifying with function approximation [18, 27]. Many prior works [39, 13, 2, 14] crucially rely on a global data-coverage assumption and completeness-type function approximation condition in a technical sense. The former necessitates that the dataset to contain any state-action pair with a lower bounded probability so that the distributional shift can be well calibrated. The latter requires the function class to be closed under Bellman updates. Both assumptions are particularly strong and are likely to be violated in practice [58]. Consequently, algorithms that depend on these assumptions may experience performance degradation and instability [52]. Therefore, it is crucial to develop novel algorithms that relax these assumptions, offering robust and widely applicable solutions for real-world scenarios.

To address the aforementioned challenges in offline settings, one fundamental principle is the concept of pessimism, which aims to maximize rewards in the worst possible MDP consistent with the offline dataset [18, 53]. In practice, these methods have generally been shown to be more robust when coverage assumptions are violated [27].

Although many such pessimistic algorithms have been developed, very few works can tackle datacoverage and function approximation issues simultaneously, while establishing strong regret guarantees. For instance, deep offline RL algorithms [18, 25, 23] exhibit impressive empirical performance, but their theoretical consistency guarantees are limited to tabular Markov decision processes (MDPs). The works of [30, 41, 22, 53, 56, 58, 10] relax the global coverage to a partial coverage condition, wherein the offline data only covers a single comparator policy. However, all of these methods require Bellman completeness for the function class. The most recent works [9, 59] take a significant step towards relaxing Bellman completeness to realizability, that the function class can capture the target ground-truth function. Nonetheless, these algorithms are unable to provide a meaningful regret guarantee without any data-coverage assumption (when both global and partial coverage fails), and also empirical evaluations are absent. Even without additional conditions, the learned policies of these algorithms can only compete with the (Bellman flow) optimal policy, resulting in a lack of robustness when the optimal policy is not covered by data, a situation that frequently occurs. Due to page limit, we have only discussed the closest related work here, and the rest is deferred to Appendix.

Our contribution. In this paper, we develop a provably sample-efficient offline RL framework. Our information-theoretic algorithm is designed based on the concept of bi-level (upper and lower level) structured optimization, which leads to a hierarchical interpretation and naturally enjoys learning stability and algorithmic convergence from a game-theoretic perspective. In particular, at the lower level, one component is to construct a confidence set with consistent value estimates regarding the appropriately small weighted average Bellman error, effectively preventing overly pessimistic evaluation. Meanwhile, the second component, which deals with uncertainty control, implicitly enhances the power of model extrapolation. In addition to the information-theoretic algorithm, we also develop a computationally efficient counterpart that is solved by a penalized adversarial estimation algorithm with proximal-mapping updating, allowing both non-linear and linear function approximation. From a theoretical standpoint, we establish a strong regret guarantee for both information-theoretical and practical algorithms under only realizability without requiring any data-coverage (neither global nor partial coverage) and completeness-type assumptions. As a special case study, we further refine our developed mixture density ratio-based concentrability coefficient to a relative condition number in linear MDP settings. The sample complexity of our regret bound improves or at least matches the prior results in the fully exploratory or partial coverage settings where the Bellman-completeness holds. Notably, compared with existing works, either focusing on theoretical or empirical development, we provide a comprehensive theoretical analysis of the proposed framework and also conduct synthetic, benchmark, and real data experiments for empirical evaluation.

2 Preliminaries and Notations

Markov decision process. Weconsider an infinite-horizon discounted MDP M = {S , A , P , γ, r, s 0 } [40], where S is the state space, A is the action space, P : S × A → ∆( S ) is the Markov transition kernel for some probabilistic simplex ∆ , r : S × A → [0 , ¯ R ] is the reward function for ¯ R ≥ 0 , γ ∈ [0 , 1) is the discounted factor and s 0 is the initial state. A policy π : S → ∆( A ) induces a distribution of the trajectory s 0 , a 0 , r 0 , s 1 , . . . , where a t ∼ π ( ·| s t ) , r t = r ( s t , a t ) , s t +1 ∼ P ( ·| s t , a t ) for any t ≥ 0 . The expected discounted return of a policy is defined as J ( π ) = E [ ∑ ∞ t =0 γ t r t | π ] . The discounted return when the trajectory starts with ( s, a ) and all remaining actions are taken according to π is called q -function q π : S×A → [0 , ¯ V ] . The q π is the unique fixed point of the Bellman operator B π , satisfying the Bellman equation [45]: B π q ( s, a ) := r ( s, a )+ γ E s ′ ∼ P ( ·| s,a ) [ q ( s ′ , π )] . Here q ( s ′ , π ) is denoted as shorthand for E a ′ ∼ π ( ·| s ′ ) [ q ( s ′ , a ′ )] , and we define P π q ( s, a ) := E s ′ ∼ P ( ·| s,a ) [ q ( s ′ , π )] . Additionally, it is helpful to remember that J ( π ) = q π ( s 0 , π ) . Another important notion is the normalized discounted visitation of π , defined as d π ( s, a ) := (1 -γ ) ∑ ∞ t =0 γ t d π,t ( s, a ) , where d π,t is the marginal state-action distribution at the time-step t .

Offline RL under function approximation. In the offline RL setting, there exists an unknown offline data-generating distribution µ induced by behavior policies. Despite the unknowns of µ , we can observe a set of transition pairs, as offline dataset D 1: n := { s i , a i , r i , s ′ i } n i =1 sampling from µ . For a given policy π , the density-ratio (importance-weight), τ d π /µ ( s, a ) = d π ( s, a ) /µ ( s, a ) , measures how

effectively µ covers the visitation induced by π . The primary objective of offline policy optimization is to learn an optimal policy that maximizes the return, J ( π ) , using the offline dataset. Under the function approximation setting, we assume access to two function classes Q : S × A → R and Ω : S × A → R , which are utilized to capture q π and τ d π /µ , respectively.

Exploration and coverage. In general, when saying an offline dataset is well-explored, it means that a well-designed behavior policy has been executed, allowing for comprehensive exploration of the MDP environment. As a result, the dataset is likely to contain possibly all state-action pairs. This implicitly requires µ has the global coverage [18, 49]. In this context, the global coverage means that the density ratio-based concentrability coefficient, sup s,a { d π ( s, a ) /µ ( s, a ) } , is upper-bounded by a constant c ∈ R + for all policies π ∈ Π , where Π is some policy class. This condition is frequently employed in offline RL [2, 8, 12]. However, in practice, this assumption may not hold true, as devising an exploratory policy is a challenging task for large-scale RL problems. Instead, our goal is to learn a good policy with strong theoretical guarantees that can compete against any arbitrarily covered comparator policy under much weaker conditions than the global coverage.

3 Bi-Level Offline Policy Optimization Algorithm

In this section, we introduce our bi-level offline policy optimization framework. The development of the framework consists of three major steps.

Step 1: robust interval learning. In this step, we aim to provide a robust off-policy interval evaluation. The major advantage of this interval formulation is its robustness to the model-misspecification of the importance-weight class Ω , and the encoding of distributional-shift information in the policy evaluation process. First, we define a detection function D ( · ) , which is used to measure the degree of the distributional-shift in terms of density ratio.

Definition 3.1. For x, c 1 , c 2 , C ∈ R + and C ≥ 1 , the detection function D ( · ) satisfies the following conditions: (1) 1-minimum: D (1) = 0 . (2) Non-negativity: D ( x ) ≥ 0 . (3) Boundedness on first-order derivative: | D ′ ( x ) | ≤ c 2 if x ∈ [0 , C ] . (4) Boundedness on value: | D ( x ) | ≤ c 1 for x ∈ [0 , C ] . (5) Strong convexity: D ( x ) is M -strongly convex with respect to x .

The family of Rényi entropy [42], Bhattacharyya distance [11], and simple quadratic form functions [60]all satisfy the conditions outlined in Definition 3.1. Under this definition, it can easily observe that D has a convex conjugate function [6], D ∗ with D ∗ ( x ∗ ) = sup x { x · x ∗ -D ( x ) } , that satisfies D ∗ (0) = 0 . It follows from Bellman equation B π q π ( s, a ) = q π ( s, a ) for any s, a , then J ( π ) = q π ( s 0 , π ) + E µ [ λ D ∗ (( B π q π ( s, a ) -q π ( s, a ) /λ )) / (1 -γ )] for λ ≥ 0 . Applying Fenchel-Legendre transformation [36, 21], and model x in a restricted importance weight class Ω for any s, a , we obtain

Suppose q π is well-sepcified, i.e., q π ∈ Q , we can find a lower bound of (2), which is valid for any τ ∈ Ω , via replacing q π with inf q ∈Q as follows:

After following a similar derivation, we can establish an upper bound for J ( π ) as well, and thus construct a value interval for J ( π ) . This interval holds for any τ and is therefore robust against model-misspecification of Ω . In order to obtain a tighter interval, we can shrink the interval width by maximizing the lower bound and minimizing the upper bound, both with respect to τ . This procedure can be interpreted as searching for some good τ ∈ Ω to minimize the function approximation error.

While the interval offers a robust method for dealing with the bias introduced by function approximation when estimating J ( π ) , it lacks a crucial and non-trivial step for handling statistical uncertainty.

Step 2: uncertainty quantification. In this step, we quantify the uncertainty of the interval (3), and establish a non-asymptotic confidence interval (CI) for J ( π ) which integrates bias and uncertainty quantifications in a single interval inspired by [60]. Given offline data D 1: n , our formal result for quantifying sampling uncertainty in order to establish the CI for J ( π ) .

Theorem 3.1 (Non-asymptotic confidence interval) . For a target policy π , the return J ( π ) is within a CI for any τ ∈ Ω with probability at least 1 -δ , i.e., J ( π ) ∈ [ ̂ J -n ( π ; τ ) , ̂ J + n ( π ; τ )] for

if the uncertainty deviation σ n satisfies

Similar to the value interval, the CI [ ̂ J -n ( π ; τ ) , ̂ J + n ( π ; τ )] also holds for any τ ∈ Ω . Therefore, we can optimize the confidence lower and upper bounds in (4) over τ ∈ Ω to tighten the CI, and obtain:

Step 3: bridge policy evaluation to policy optimization. In this step, we aim to formulate a policy optimization based on the derived high-confidence policy evaluation from the previous steps. Given the consistent CI estimation of J ( π ) , we can naturally incorporate the pessimism principle, i.e., using the CI lower bounds of J ( π ) as the value estimate of the policy evaluation of π [22]. With such a procedure, our objective is to maximize these lower bounds over some family Π of policies:

Although (5) is algorithmically feasible for obtaining a policy solver ̂ π , it lacks direct interpretation without taking advantage of the bi-level optimization structure in hindsight. Therefore, we propose to reformulate (5) via a dual-to-prime conversion (shown in Theorem 3.2), which naturally lends itself to lower-upper optimization with guaranteed convergence. Specifically, we formulate (5) as a bi-level framework problem:

At the upper level, the learned policy ̂ π attempts to maximize the value estimate of q π over some policy class Π , while at the lower level, q π is to seek the q -function with the pessimistic policy evaluation value from the confidence set Q ε n with consistency guarantee and uncertainty control. For consistency , whenever q π or its good approximator is included in Q (realizability for Q class is satisfied), the set Q ε n ensures the estimation consistency of q π in terms of 'sufficently small' weighted average Bellman error. For uncertainty control , the constrained set ˜ Ω ˜ σ n attempts to control the uncertainty arising from distributional shift via a user-specific thresholding hyperparameter ˜ σ n . The feasible (uncertainty controllable) candidates τ ∈ ˜ σ n are used as weights for the average Bellman error, helping to construct the consistent set Q ε n . Risk-averse users can specify a lower value for the thresholding hyperparameter or consider a higher ˜ σ n to tolerate a larger distribution shift. In other words, the chosen value of ˜ σ n depends on the degree of pessimism users want to incorporate in the policy optimization.

Interestingly, the new form in (6) characterizes our policy optimization framework as a two-player general-sum game [16], which is a sequential game involving two players. Each player aims to maximize their own payoffs while considering the decisions of other players. Our bi-level optimization framework has been demonstrated to improve learning stability and ensure algorithmic convergence, benefiting from the existence of a local equilibrium [51].

To close this section, we remark that the establishment of consistency with respect to the weighted average Bellman error is the key point for us to relax the completeness-type assumptions. In the famous API/AVI-type algorithms [13, 14, 8], they target to minimize a squared or minimax Bellman error for finding q ∈ Q so that ∥ q -B π q ∥ 2 L 2 ( µ ) ≈ 0 to obtain q ≈ q π . Unfortunately, even with the infinite amount of data, the empirical estimate of ∥ q -B π q ∥ 2 L 2 ( µ ) , i.e., squared empirical Bellman error) is biased due to the appearance of unwanted conditional variance, i.e., the double sampling issue [3]. The API/AVI-type algorithms need a separate helper function class ˜ Q for modeling B π q , and [8] has shown that when the class ˜ Q realizes the Bayes optimal regressor B q (Bellman-completeness condition), the estimation is consistent and unbiased. In contrast, thanks to not using the squared loss, our weighted average Bellman error can be estimated from an unbiased estimate without concern about the double sampling issue, and thus no need for any completeness-type conditions.

4 Information-Theoretic Results

In this section, we provide theoretical analyses of our algorithm, which reveals the advantages of the proposed policy optimization method from a technical standpoint.

Notably, to the best of our knowledge, Theorem 4.1 is the first result of regret guarantee under only realizability without requiring any data coverage or completeness-type assumptions. Additionally, in contrast to most existing works that assume finite function classes, we carefully quantify the space complexities for infinite function classes (e.g., a class of real-valued functions generated by neural networks) using Pollard's pseudo-dimension [38]. The formal definition is provided in Appendix. It notices that the pseudo-dimension is a generalization of the well-known VC dimension [50]. In the following, we first introduce the necessary assumptions before presenting the guarantees for our algorithm.

Assumption 1 (Realizability for q -function class) . For any policy π ∈ Π , we have q π ∈ Q . When this assumption holds approximately, we measure violation by inf q ∈Q sup ρ E ρ [( q ( s, a ) -B π q ( s, a )) 2 ] ≤ ε Q , where ε Q ≥ 0 and ρ is some data distribution such that ρ ∈ { d ˜ π : ˜ π ∈ Π } .

We would like to emphasize that we do not require Bellman-completeness condition [53, 58], which is much stronger than the realizability condition. In addition, we do not impose realizability on the importance-weight class Ω , thereby allowing model misspecification on Ω . Having stated the major assumptions, we now turn to the routine ones on boundedness before presenting the main results.

Assumption 2 (Boundedness on Q ) . There exists a non-negative constant ¯ V < ∞ , the function q ( s, a ) ∈ [0 , ¯ V ] , ∀ q ∈ Q , s ∈ S , a ∈ A .

Assumption 3 (Boundedness on Ω ) . There exists a non-negative constant 1 ≤ U τ ∞ < ∞ , the function τ ( s, a ) ∈ [0 , U τ ∞ ] , ∀ τ ∈ Ω , s ∈ S , a ∈ A .

Theorem 4.1. Under Assumptions 1-3 and denote supremum of µ -weighted L 2 norm of Ω , i.e., sup τ ∈ Ω ∥ τ ( s, a ) ∥ L 2 ( µ ) , as U τ 2 . Let ̂ π be the output of solving (6) when we set ε n = ˜ O ( n -1 / 2 U τ 2 ( √ ln { Vol(Θ) /δ } + U τ ∞ √ ε Q ) and ˜ σ n = ˜ O ( n -1 / 2 U τ 2 L √ ln { Vol(Θ) /δ } + M ( U τ 2 -1) 2 ) , then for any policy π ∈ Π and some constant U ⋆ 2 ∈ [1 , U τ 2 ) , w.p. ≥ 1 -δ ,

Here ∆ q π -q π ( s, a ) = q π ( s, a ) -q π ( s, a ) for q π := arg max q ∈Q εn q ( s 0 , π ) and q π := arg min q ∈Q εn q ( s 0 , π ) . For Pollard's pseudo-dimensions D Ω , D Q , D Π , Vol(Θ) = ( e D max { D Ω , D Q , D Π } + 1) 3 ( { 1 ∨ L }U τ 2 ) 2 D with the effective pseudo dimension D = D Ω + D Q + D Π , where L is Lipschitz constant of M -strongly convex function D ( · ) . Moreover, C x and ˜ O denote constant terms depending on x , and big-Oh notation ignoring high-order terms, respectively.

In the upper bound of Theorem 4.1, we split the regret into four different parts: the on-support intrinsic uncertainty ϵ σ , the on-support bias ϵ b , the violation of realizability ϵ mis, and the off-support extrapolation error ϵ off. Recall that we require q π ∈ Q as in Assumption 1, in fact, we can further relax the condition to requiring q π to be in the linear hull of Q [48], which is more robust to the realizability error ϵ mis. In the following, we focus on investigating the roles of the on-support and off-support error terms in the regret bound.

On-support errors: bias and uncertainty tradeoff. The on-support error consists of two terms: ϵ b and ϵ σ . The on-support uncertainty deviation, ϵ σ , is scaled by a weighted L 2 -based concentrability coefficient U ⋆ 2 := ∥ ρ/µ ∥ L 2 ( µ ) , which measures the distribution mismatch between the implicit exploratory data distribution and the baseline data distribution µ . Meanwhile, ϵ b depends on the probability mass of ( d π -ρ ) + 1 µ> 0 , and represents the bias weighted by the probability mass difference between d π and ρ in the support region of µ . In general, a small value of U ⋆ 2 necessitates choice of the distribution ρ to be closer to µ which reduces ϵ σ , reducing ϵ σ but potentially increasing the on-support bias ϵ b due to the possible mismatch between d π and ρ . Consequently, within the on-support region, there is a trade-off between ϵ σ and ϵ b , which is adjusted through U ⋆ 2 .

Off-support error: enhanced model extrapolation. One of our main algorithmic contributions is that the off-support extrapolation error ϵ off can be minimized by selecting the best possible ρ without worrying about balancing the error trade-off, unlike the on-support scenario. This desirable property is essential for allowing the model to harness its extrapolation capabilities to minimize ϵ off, while simultaneously achieving a good on-support estimation error. As a result, the model attains a small regret. Recall the bi-level formulation; at the lower level, (7) addresses uncertainty arising from the distributional shift using L 2 ( µ ) control rather than L ∞ control. This plays an important role in enhancing the power of the model extrapolation. In particular, Specifically, there exists an implicit exploratory data distribution ρ with on-support behavior ( ρ 1 µ> 0 ) close to µ , such that ∥ ρ/µ ∥ L 2 ( µ ) is small. On the other hand, its off-support behavior ( ρ 1 µ =0 ) can be arbitrarily flexible, ensuring that d π 1 µ =0 is close to ρ 1 µ =0 . Consequently, ( d π -ρ ) + 1 µ =0 is small, as is ϵ off .

When a dataset with partial coverage, as indicated in [49], it is necessary to provide a guarantee: learn the policy with 'best efforts' which is competitive to any policy as long as it is covered. Before we state the near-optimal regret guarantee of our algorithm, we formally define a notion of covered policies according to a newly-defined concentrability coefficient.

Definition 4.1 ( U τ 2 -covered policy class) . Let Π( U τ 2 ) denote the U τ 2 -covered policy class of µ for U τ 2 ≥ 1 , defined as

Note that this mixture density ratio concentrability coefficient is always bounded by the L ∞ -based concentrability coefficient. Thus such single-policy concentrability assumption in terms of the mixture density ratio is weaker than the standard L ∞ density ratio-based assumption.

Corollary 4.1 (Near-optimal regret) . Under Assumptions 1-3 with ε Q ∈ [0 , 1) , and we set ε n , ˜ σ n as in Theorem 4.1, then for any good comparator policy π ⋄ ∈ Π( U τ 2 ) (not necessary the optimal policy π ∗ ), w.p. ≥ 1 -δ , the output policy ̂ π of (6) satisfies

A close prior result to Corollary 4.1 is that of [9], which develops a pessimistic algorithm based on a nontrivial performance gap condition. Their regret guarantees only hold if the data covers the optimal policy π ∗ , in particular, requiring a bounded L ∞ single-policy concentrability with respect to π ∗ . In comparison, our guarantee can still provide a meaningful guarantee even when π ∗ is not covered by data. In the following, we include the sample complexity of our algorithm when ε Q = 0 .

Corollary 4.2 (Polynomial sample complexity) . Under the conditions in Corollary 4.1, the output policy ̂ π of solving (6) satisfies J ( π ⋄ ) -J ( ̂ π ) ≤ ε w.p. ≥ 1 -δ , if

The sample complexity consists of three terms corresponding to the slow rate O ( n -1 / 2 ) and the two faster rate O ( n -1 ) and O ( n -3 / 4 ) terms in Corollary 4.1. When U τ 2 and U τ ∞ are not too much larger than U ⋆ 2 , the fast rate terms are dominated, and the sample complexity is of order O (1 /ε 2 ) , which is much faster than O (1 /ε 6 ) in the close work of [59]. It is worth noting that even in exploratory settings where the global coverage assumption holds, our sample complexity rate matches the fast rate in popular offline RL frameworks with general function approximation [8, 54, 12].

In addition to the near-optimal regret guarantee, in safety-critical applications, an offline RL algorithm should consistently improve upon the baseline (behavior) policies that collected the data [19, 26]. Our algorithm also achieves this improvement guarantee with respect to the baseline policy.

Theorem 4.2 (Baseline policy improvement) . Under Assumptions 1-3 with ε Q = 0 and set ε n , ˜ σ n as in Theorem 4.1. Suppose 1 ∈ Ω and the baseline policy π b ∈ Π such that d π b = µ , then the regret (1 -γ )( J ( π b ) -J ( ̂ π )) for the output policy ̂ π of solving (6) , w.p. ≥ 1 -δ , is upper bounded by

The aforementioned information-theoretic results enhance the understanding of the developed algorithm, in terms of the function approximation and coverage conditions, sample complexity, horizon dependency, and bound tightness. In practice, although the information-theoretic algorithm offers a feasible solution to the problem, it is not yet tractable and computationally efficient due to the need to solve constrained optimization. In the following section, we develop a practical algorithm as a computationally efficient counterpart for the information-theoretic algorithm.

5 Penalized Adversarial Estimation Algorithm

Although the information-theoretic algorithm offers a feasible solution to the problem, it is not yet tractable and computationally efficient due to the need to solve constrained optimization. In this section, we develop an adversarial estimation proximal-mapping algorithm that still adheres to the pessimism principle, but through penalization. Specifically, the adversarial estimation loss is constructed as follows: max τ min q L ( q, τ, π, c ∗ , λ ) for solving

We observe that the inner minimization for solving q is relatively straightforward, as we can obtain a closed-form global solver using the maximum mean discrepancy principle [20, 44]. In contrast, optimizing τ ψ is more involved, often requiring a sufficiently expressive non-linear function approximation class, e.g., neural networks. However, concavity typically does not hold for such a class of functions [21]. From this perspective, our problem can be viewed as solving a non-concave maximization problem, conditional on the solved global optimizer ¯ q := arg min q L ( q, τ, π, c ∗ , λ ) . At each iteration, we propose to update τ by solving the proximal mapping [37] using the Euclidean distance to reduce the computational burden. As a result, the pre-iteration computation is quite low.

Algorithm 1 Adversarial proximal-mapping algorithm

1: Input observed data D 1: n = { ( s i , a i , r i , s ′ i ) } n i =1 and parameters q 0 , τ 0 , π 0 , c ∗ , λ and ζ .

2: For k = 1 to ¯ K :

3: Update τ k and q k by solving max τ min q L ( q, τ, π k -1 , c ∗ , λ )

4: Update π k by solving π k ( ·| s ) = argmax π ∈ Π ζ 〈 q k ( · , s ) , π ( ·| s ) 〉 -D NegEntropy ( π ( ·| s ) , π k ( ·| s ) ) .

5: Return the policy ̂ π , which randomly selects a policy from the set { π k } ¯ K k =1 .

Once q and τ are solved, we apply mirror descent in terms of the negative entropy D NegEntropy [5]. That is, given a stochastic gradient direction of π we solve the prox-mapping in each iteration as outlined in step 4 of Algorithm 1. A detailed version of Algorithm 1 with extended discussions on convergence and complexity is provided in Appendix. In the following, we establish the regret guarantee for the policy output by Algorithm 1.

Theorem 5.1. Under Assumptions 1-3 with ε Q = 0 , we properly choose λ = λ ( U τ 2 ) , i.e., λ well depends on U τ 2 , and c ∗ = ˜ O (√ n ¯ V / ( λL U τ 2 ln { Vol(Θ † ) /δ } ) ) . After running ¯ K ≥ log |A| rounds of Algorithm 1 with the stepsize ζ = √ log |A| / (2 ¯ V ¯ K ) , for any policy π ∈ Π , the output policy ̂ π of the algorithm, w.p ≥ 1 -δ , satisfies,

where ∆ U ⋆ 2 := { ρ k : ∥ ρ k µ ∥ L 2 ( µ ) < U ⋆ 2 } , C 1 ¯ V ,λ,L , C 2 ¯ V ,λ,L are some constant terms, and the function class complexity Vol(Θ † ) = ( e D max { D Ω , D Q , D Π } +1) 3 ( { 1 ∨ L }U τ 2 ) 2 D for D = D Ω + D Q + D Π .

Trajectory-adaptive exploratory data distribution. Similar to Theorem 4.1, the penalized algorithm also exhibits a desirable extrapolation property for minimizing extrapolation error while simultaneously preserving small on-support estimation errors. This is achieved through adaptations of the implicit exploratory data distributions, ρ k for k ∈ [ ¯ K ] . In contrast to the information-theoretic algorithm, the automatic splitting by ρ k now depends on the optimization trajectory. At each iteration k , the penalized algorithm allows each implicit exploratory data distribution ρ k to adapt to the comparator policy π . This results in a more flexible adaptation than the one in the information-theoretic algorithm, either for balancing the trade-off between on-support bias and uncertainty incurred by the distributional mismatch between d π and ρ k , or for selecting the best implicit exploratory to minimize model extrapolation error.

Opimization error. Blessed by the reparametrization in the proximal-mapping policy update, which projects the mixture policies into the parametric space Π ω , the complexity of the restricted policy class is independent of the class of Q and the horizon optimization trajectory ¯ K . As a result, the optimization error O ( √ ¯ V log |A| / ¯ K ) can be reduced arbitrarily by increasing the maximum number of iterations, ¯ K , without sacrificing overall regret to balance statistical error and optimization error. This allows for the construction of tight regret bounds. This distinguishes our algorithm from API-style algorithms, which do not possess a policy class that is independent of Q [2, 43, 53].

5.1 An Application to Linear MDPs with Refined Concentrability Coefficient

In this section, we conduct a case study in linear MDPs with insufficient data coverage. The concept of the linear MDP is initially developed in the fully exploratory setting [55]. Let ϕ : S×A → R d be a d -dimensional feature mapping. We assume throughout that these feature mappings are normalized, such that ∥ ϕ ( s, a ) ∥ L 2 ≤ 1 uniformly for all ( s, a ) ∈ S × A . We focus on action-value functions that are linear in ϕ and consider families of the following form: Q θ := { ( s, a ) ↦→⟨ ϕ ( s, a ) , θ ⟩ | ∥ θ ∥ L 2 ≤ c θ } , where c θ ∈ [0 , ¯ V ] . For stochastic policies, we consider the soft-max policy class Π ω := { π ω ( a | s ) ∝ e ⟨ ϕ ( s,a ) ,ω ⟩ | ∥ ω ∥ L 2 ≤ c ω } , where c ω ∈ (0 , ∞ ) . Note that the softmax policy class is consistent with the implicit policy class produced by the mirror descent updates with negative entropy in Algorithm 1, where the exponentiated gradient update rule is applied in each iteration. For the importance-weight class, we also consider the following form: Ω ψ := { ( s, a ) ↦→⟨ ϕ ( s, a ) , ψ ⟩ | ∥ ψ ∥ L 2 ≤ c ψ } where c ψ ∈ (0 , ∞ ) . To simplify the analysis, we assume the realizability condition for Q θ is exactly met. In this linear MDP setting, we further refine the density ratio to a relative condition number to characterize partial coverage. This concept is recently introduced in the policy gradient literature [1] and is consistently upper-bounded by the L ∞ -based density ratio concentrability coefficient.

Definition 5.1 (Relative condition number) . For any policy π ∈ Π ω and behavior policy π b such that d π b = µ , the relative condition number is defined as ι ( d π , µ ) = sup x ∈ R d x T E dπ [ ϕ ( s,a ) ϕ ( s,a ) ⊤ ] x ⊤ E ⊤ .

Assumption 4 (Bounded relative condition number) . For any π ∈ Π ω , ι ( d π , µ ) < ∞

x µ [ ϕ ( s,a ) ϕ ( s,a ) ] x .

Intuitively, this implies that as long as a high-quality comparator policy exists, which only visits the subspace defined by the feature mapping ϕ and is covered by the offline data, our algorithm can effectively compete against it [49]. This partial coverage assumption, in terms of the relative condition number, is considerably weaker than density ratio-based assumptions. In the following, we present our main near-optimal guarantee in linear MDPs. In addition, we design and conduct numerical experiments to empirically validate Theorem 5.2 in terms of the regret rate of convergence.

Theorem 5.2. Under Assumption 4, if we set propertly choose λ = λ ( c ψ ) and c ∗ = ˜ O ( 4 √ n/d ln { (1 + e √ n (1 ∨ L ) ¯ V c ψ c ω ) /δ } ) , and suppose ̂ π lr is returned by Algorithm 1 with linear function approxiamiton after running ¯ K ≫ log |A| rounds, then for any policy in π ∈ Π ω ( U lr 2 ) for U lr 2 ≥ 1 , w.p. ≥ 1 -δ , J ( π ) -J ( ̂ π lr ) is bounded by

To the best of our knowledge, this is the first result PAC guarantees for an offline model-free RL algorithm in linear MDPs, requiring only realizability and single-policy concentrability. The regret bound we obtain is at least linear and, at best, sub-linear with respect to the feature dimension d . Our approach demonstrates a sample complexity improvement in terms of feature dimension compared to prior work by [22], with a complexity of O ( d 1 / 2 ) versus O ( d ) . It is worth noting that [22] only establishes results that compete with the optimal policy, and when specialized to linear MDPs, assumes the offline data has global coverage. Another previous study by [53] achieves a similar sub-linear rate in d as our approach; however, their algorithm is computationally intractable, relying on a much stronger Bellman-completeness assumption and requiring a small action space.

6 Experiments

In this section, we evaluate the performance of our practical algorithm by comparing to the model-free offline RL baselines including CQL [25], BEAR [24], BCQ [18], OptiDICE [27], ATAC [10], IQL [23], and TD3+BC [17]. We also compete with a popular model-based approach COMBO [57].

Synthetic data. We consider two synthetic environments: a synthetic CartPole environment from the OpenAI Gym [7] and a simulated environment. Detailed discussions on the experimental designs are deferred to the Appendix. In both settings, following [48], we first learn a sub-optimal policy using DQN [34] and then apply softmax to its q -function, divided by a temperature parameter α to set the action probabilities to define a behavior policy π b .

A smaller α implies π b is less explored, and thus the support of µ = d π b is relatively small. We vary different values of α for evaluating the algorithm performance in 'low', 'medium' and 'relatively high' offline data exploration scenarios. We use γ = 0 . 95 with the sample-size n = 1500 in all experiments. Tuning parameter selection is an open problem in offline policy optimization. Fortunately, Theorem 5.2 suggests an offline selection rule for hyperparameters λ and c ∗ . In the following

Simulated

CartPole

6

Methods

4

60

Proposed

COMBO

OptiDICE

ATAC

2

IQL

40

CQL

BCQ

BEAR

TD3+BC

20

Low

Medium

Relatively High

LOw

Medium

Relatively High

ExplorationDegree

ExplorationDegree

experiments, we set the hyper-parameters satisfying the condition O ( n 1 / 4 d log( ¯ V √ n ) ) . Figure 1 shows that our algorithm almost consistently outperforms competing methods in different settings. This performance mainly benefits from the advantages exposed in our theoretical analysis, such as model extrapolation enhancement, relaxation of completeness-type assumptions on function approximation, etc. The only exception is the slightly poorer performance compared to COMBO in a high exploration setting, where COMBO may learn a good dynamic model with relatively sufficient exploration. We provide the experiment details in Appendix due to page limit.

Benchmark data. We evaluate our proposed approach on the D4RL benchmark of OpenAI Gym locomotion (walker2d, hopper, halfcheetah) and Maze2D tasks [15], which encompasses a variety

of dataset settings and domains and positions our algorithm within the existing baselines. We take the results of COMBO, OptiDICE and ATAC from their original papers for Gym locomotion, and run COMBO and ATAC using author-provided implementations for Maze2D. The results of BCQ, BEAR methods from the D4RL original paper. In addition, CQL, IQL and TD3+BC are re-run to ensure a fair evaluation process for all tasks. As shown in Table 1, the proposed algorithm achieves the best performance in 7 tasks and is comparable to the baselines in the remaining tasks. In addition to the evaluation of the policy performance, we also conduct sensitivity analyses on the hyperparameter-tuning and study the regret rate of convergence.

Real-world application. The Ohio Type 1 Diabetes (OhioT1DM) dataset [33] comprises a cohort of patients with Type-1 diabetes, where each patient exhibits different dynamics and 8 weeks of life-event data, including health status measurements and insulin injection dosages. Clinicians aim to adjust insulin injection dose levels [33, 4] based on a patient's health status in order to maintain glucose levels within a specific range for safe dose recommendations. The state variables consist of health status measurements, and the action space is a bounded insulin dose range. The glycemic index serves as a reward function to assess the quality of dose suggestions. Since the data-generating process is unknown, we follow [31, 29] to utilize the Monte Carlo approximation of the estimated value function on the initial state of each trajectory to evaluate the performance of each method. The mean and standard deviation of the improvements on the Monto Carlo discounted returns are presented in Table 2. As a result, our algorithm achieves the best performance for almost all patients, except for Patient 552 . The main reason for the desired performance in real data is from the enhanced model extrapolation and relaxed function approximation requirements and outperforms the competing methods. This finding is consistent with the results in the synthetic and benchmark datasets, demonstrating the potential applicability of the proposed algorithm in real-world environments.

7 Conclusion

Westudy offline RL with limited exploration in function approximation settings. We propose a bi-level policy optimization framework, which can be further solved by a computationally practical penalized adversarial estimation algorithm, offering strong theoretical and empirical guarantees. Regarding limitations and future work, while the penalized adversarial estimation is more computationally efficient than the previously constrained problem, it may still be more challenging to solve than singlestage optimization problems. Another future direction is to explore environments with unobservable confounders. It will be interesting to address these limitations in future works.

8 Acknowledments

The author is grateful to the five anonymous reviewers and the area chair for their valuable comments and suggestions.

References

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. Optimality and approximation with policy gradient methods in markov decision processes. In Conference on Learning Theory , pages 64-66. PMLR, 2020.

András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellman-residual minimization based fitted policy iteration and a single sample path. Machine Learning , 71(1):89-129, 2008.

Leemon Baird. Residual algorithms: Reinforcement learning with function approximation. In Machine Learning Proceedings 1995 , pages 30-37. Elsevier, 1995.

Jiansong Bao, Heather R Gilbertson, Robyn Gray, Diane Munns, Gabrielle Howard, Peter Petocz, Stephen Colagiuri, and Jennie C Brand-Miller. Improving the estimation of mealtime insulin dose in adults with type 1 diabetes: the normal insulin demand for dose adjustment (nidda) study. Diabetes Care , 34(10):2146-2151, 2011.

Amir Beck. First-order methods in optimization . SIAM, 2017.

Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex Optimization . Cambridge University Press, 2004.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning , pages 1042-1051. PMLR, 2019.

Jinglin Chen and Nan Jiang. Offline reinforcement learning under value and density-ratio realizability: the power of gaps. In Uncertainty in Artificial Intelligence , pages 378-388. PMLR, 2022.

Ching-An Cheng, Tengyang Xie, Nan Jiang, and Alekh Agarwal. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning , pages 3852-3878. PMLR, 2022.

Euisun Choi and Chulhee Lee. Feature extraction based on bhattacharyya distance. Pattern Recognition , 36(8):1703-1709, 2003.

Yaqi Duan, Zeyu Jia, and Mengdi Wang. Minimax-optimal off-policy evaluation with linear function approximation. In International Conference on Machine Learning , pages 2701-2709. PMLR, 2020.

Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research , 6:503-556, 2005.

Amir-massoud Farahmand, Mohammad Ghavamzadeh, Csaba Szepesvári, and Shie Mannor. Regularized policy iteration with nonparametric function spaces. The Journal of Machine Learning Research , 17(1):4809-4874, 2016.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Drew Fudenberg and Jean Tirole. Game theory . MIT press, 1991.

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems , 34:20132-20145, 2021.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference on Machine Learning , pages 2052-2062. PMLR, 2019.

Mohammad Ghavamzadeh, Marek Petrik, and Yinlam Chow. Safe policy improvement by minimizing robust baseline regret. Advances in Neural Information Processing Systems , 29, 2016.

Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research , 13(1):723-773, 2012.

Nan Jiang and Jiawei Huang. Minimax value interval for off-policy evaluation and policy optimization. Advances in Neural Information Processing Systems , 33:2747-2758, 2020.

Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? In International Conference on Machine Learning , pages 5084-5096. PMLR, 2021.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169 , 2021.

Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing offpolicy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems , 32, 2019.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:11791191, 2020.

Romain Laroche, Paul Trichelair, and Remi Tachet Des Combes. Safe policy improvement with baseline bootstrapping. In International conference on machine learning , pages 3652-3661. PMLR, 2019.

Jongmin Lee, Wonseok Jeon, Byungjun Lee, Joelle Pineau, and Kee-Eung Kim. Optidice: Offline policy optimization via stationary distribution correction estimation. In International Conference on Machine Learning , pages 6120-6130. PMLR, 2021.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Yuhan Li, Wenzhuo Zhou, and Ruoqing Zhu. Quasi-optimal reinforcement learning with continuous actions. In The Eleventh International Conference on Learning Representations , 2023.

Yao Liu, Adith Swaminathan, Alekh Agarwal, and Emma Brunskill. Provably good batch off-policy reinforcement learning without great exploration. Advances in neural information processing systems , 33:1264-1274, 2020.

Daniel J Luckett, Eric B Laber, Anna R Kahkoska, David M Maahs, Elizabeth Mayer-Davis, and Michael R Kosorok. Estimating dynamic treatment regimes in mobile health using v-learning. Journal of the American Statistical Association , 115(530):692-706, 2020.

Travis Mandel, Yun-En Liu, Sergey Levine, Emma Brunskill, and Zoran Popovic. Offline policy evaluation across representations with applications to educational games. In AAMAS , volume 1077, 2014.

Cindy Marling and Razvan Bunescu. The ohiot1dm dataset for blood glucose level prediction: Update 2020. KHD@IJCAI , 2020.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Susan A Murphy, Mark J van der Laan, James M Robins, and Conduct Problems Prevention Research Group. Marginal mean models for dynamic regimes. Journal of the American Statistical Association , 96(456):1410-1423, 2001.

Ofir Nachum and Bo Dai. Reinforcement learning via fenchel-rockafellar duality. arXiv preprint arXiv:2001.01866 , 2021.

Neal Parikh, Stephen Boyd, et al. Proximal algorithms. Foundations and trends® in Optimization , 1(3):127-239, 2014.

David Pollard. Empirical processes: theory and applications. Ims, 1990.

Doina Precup. Eligibility traces for off-policy policy evaluation. Computer Science Department Faculty Publication Series , page 80, 2000.

Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley & Sons, 2014.

Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems , 34:11702-11716, 2021.

Alfréd Rényi. On measures of entropy and information. In Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability, Volume 1: Contributions to the Theory of Statistics , volume 4, pages 547-562. University of California Press, 1961.

Bruno Scherrer and Boris Lesner. On the use of non-stationary policies for stationary infinitehorizon markov decision processes. Advances in Neural Information Processing Systems , 25, 2012.

Chengchun Shi, Masatoshi Uehara, Jiawei Huang, and Nan Jiang. A minimax learning approach to off-policy evaluation in confounded partially observable markov decision processes. In International Conference on Machine Learning , pages 20057-20094. PMLR, 2022.

Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction . MIT press, 2018.

Georgios Theocharous, Yash Chandak, Philip S Thomas, and Frits de Nijs. Reinforcement learning for strategic recommendations. arXiv preprint arXiv:2009.07346 , 2020.

Philip Thomas, Georgios Theocharous, and Mohammad Ghavamzadeh. High-confidence offpolicy evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 29, 2015.

Masatoshi Uehara, Jiawei Huang, and Nan Jiang. Minimax weight and q-function learning for off-policy evaluation. In International Conference on Machine Learning , pages 9659-9668. PMLR, 2020.

Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. In International Conference on Learning Representations , 2022.

Vladimir N Vapnik and A Ya Chervonenkis. On the uniform convergence of relative frequencies of events to their probabilities. Measures of Complexity: Festschrift for Alexey Chervonenkis , pages 11-30, 2015.

Heinrich Von Stackelberg. Market structure and equilibrium . Springer Science & Business Media, 2010.

Ruosong Wang, Yifan Wu, Ruslan Salakhutdinov, and Sham Kakade. Instabilities of offline rl with pre-trained neural representation. In International Conference on Machine Learning , pages 10948-10960. PMLR, 2021.

Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellmanconsistent pessimism for offline reinforcement learning. Advances in neural information processing systems , 34, 2021.

Tengyang Xie and Nan Jiang. Q* approximation schemes for batch reinforcement learning: A theoretical comparison. In Conference on Uncertainty in Artificial Intelligence , pages 550-559. PMLR, 2020.

Lin Yang and Mengdi Wang. Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound. In International Conference on Machine Learning , pages 10746-10756. PMLR, 2020.

Ming Yin, Yu Bai, and Yu-Xiang Wang. Near-optimal offline reinforcement learning via double variance reduction. Advances in neural information processing systems , 34:7677-7688, 2021.

Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems , 34:28954-28967, 2021.

Andrea Zanette, Martin J Wainwright, and Emma Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems , 34:13626-13640, 2021.

Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory , pages 2730-2775. PMLR, 2022.

Wenzhuo Zhou, Yuhan Li, Ruoqing Zhu, and Annie Qu. Distributional shift-aware off-policy interval estimation: A unified error quantification framework. arXiv preprint arXiv:2309.13278 , 2023.

Wenzhuo Zhou, Ruoqing Zhu, and Annie Qu. Estimating optimal infinite horizon dynamic treatment regimes via pt-learning. Journal of the American Statistical Association , pages 1-14, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript proposes a bi-level structure for solving offline RL problem. The authors design the lower level constructing a set of value estimates, while the upper level maximizing the conservative value estimate. Specifically, the author provides a penalized adversarial estimation algorithm. The superiority of the proposed method is shown from both real world and synthetic data.\n\nSTRENGTHS:\n1. The hierarchical structure for solving offline RL problem is very interesting.\n2. The proposed method considers a confidence set of value estimates as well as the uncertainty, and it effectively prevents from being too pessimistic.\n3. The manuscript provides solid theoretical results, and it shows the proposed method could work without requiring data-coverage, which sounds quite useful in offline RL scenario.\n\nWEAKNESSES:\n1.  It is suggested to add some important steps in Algorithm 1 from Line 270 ~ Line 281, otherwise the Algorithm 1 is too simple while there are many details in 270 ~ Line 281, which is somehow difficult to read.\n\n2. The authors considers the synthetic data (CarPole) and real-world data (Type I Diabetes), as a common benchmark in offline RL research, the D4RL tasks are widely studied in offline RL experiments. It is suggested to consider some D4RL task in the experiment.\n\nQUESTIONS:\n1. There are some confusion in understanding the definition of Consistency  above line 154, should the $\\mathcal{Q}_{\\epsilon_n}$ be related with policy $\\pi$?\n2. In line 3 of Algorithm 1, it is required to solve a maxmin problem with iterative method, so how is the time consumption of the proposed method? Could the author provides some experiment study or conceptual analysis on this issue?\n3. Any practical suggestions on selecting $\\mathbb{G}(x)$ and the initial function of $\\tau(s,a)$? The author also mentions the proposed method does not require data-coverage, could this issue be demonstrated from experiments? Say using a biased policy as behavior policy to generate data to learn offline RL policies.\n\nLIMITATIONS:\nsee questions and limitations", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOffline Reinforcement Learning suffers from distribution shift, arising as a consequence of limited exploration and function approximation. The paper aims to address these two challenges by proposing a bi-level hierarchical framework for policy optimization. The upper-level optimizes a policy improvement objective using a pessimistic value function. The lower level learns a conservative estimate of the value function by maintaining a confidence set for the upper level. Development of the framework is carried out by first constructing robust intervals for addressing approximation bias and then using these intervals in uncertainty quantification. Motivated by its theoretical results, the work further proposes an empirical penalized adversarial estimation algorithm which incorporates the bi-level objective using an adversarial estimation loss. While the lower level is optimized efficiently using maximum mean discrepancy principle, the upper level utilizes function approximation for learning the proximal mapping.\n\nSTRENGTHS:\n* The paper is well written and organized. Specifically, the paper builds the framework in a structured way.\n* While the paper borrows from past results in tackling distribution shift and constrained optimization, the bi-level approach and its abstraction between policy and value objectives is novel.\n\nWEAKNESSES:\n* **Empirical Evaluation:** My main concerns are centered at the empirical evaluation and experiments carried out to assess the bi-level framework. Experiments do not validate the theoretical claims of the work and only evaluate the performance of the framework. Ideally the empirical evaluation should validate whether regret bounds of Theorem 4.1, Theorem 5.1 or Theorem 5.2 would hold or not. Authors can compare empirically whether the bound of Theorem 5.2 is tight and that if $\\lambda$ and $c^{*}$ are chosen as per the set criteria, regret is close to the bound of Theorem 5.2. Similarly, experiments do not demonstrate the benefits of bi-level approach in addressing distribution shift. It would be worthwhile to compare the performance with and without the upper/lower levels and assess if regret bounds still hold.\n* **Significance of Information-Theoretic Results:** Section 4 appears to be disconnected from the rest of the paper. While the Information Theoretic algorithm does not provide an empirically feasible method, it produces a baseline policy improvement scheme using bias and uncertainty errors. However, the significance of these results is unclear. To the best of my knowledge, the information theoretic results are not used to generate the penalized adversarial estimation algorithm. Neither they are used in the empirical evaluation or in producing a tighter regret bound. It would be helpful if the authors could explain the significance and utility of these results and how they should be interpreted. It would also be helpful if the paper could throw light on the benefit of near-optimal regret in Corollary 4.1.\n* **Utility of $\\tilde{\\sigma_{n}}$:** From my understanding, $\\tilde{\\sigma_{n}}$ trades off pessimism for policy improvement. This indicates that the utility of $\\tilde{\\sigma_{n}}$ is not well understood. Setting $\\tilde{\\sigma_{n}}$ to a lower value can hinder policy performance as the lower level will yield an overly conservative value function. On the other hand, setting $\\tilde{\\sigma_{n}}$ to a higher value leads to distribution shift. It would be helpful to have an optimal value of $\\tilde{\\sigma_{n}}$ characterized or a criterion for selecting $\\tilde{\\sigma_{n}}$ based on policy performance or pessimism.\n\n## Minors\n\n* line 29: amplifying -> amplifies\n* line 32: to contain -> contains\n* line 79: discounted -> discount\n* line 125: sepcified -> specified\n* line 137: established the CI for $J(\\pi)$ is presented in Theorem 3.1.\n* line 234: concentrability -> concentratibility\n* line 275: conditional -> conditioned\n* line 329: set propertly choose -> properly choose\n\nQUESTIONS:\n* Can the empirical evaluation demonstrate the bounds of Theorem 4.1 and Theorem 5.2? Can experiments show the benefit of bi-level approach in tackling distribution shift (i.e- comparing with and without upper/lower levels)?\n* Can you please explain how are the Information Theoretic results useful? Are these results used to generate the penalized adversarial estimation algorithm? If not, then how should one interpret Theorem 4.1 and Corollary 4.1?\n* What is the optimal $\\tilde{\\sigma_{n}}$? Is there a criterion for setting $\\tilde{\\sigma_{n}}$?\n\nLIMITATIONS:\nAuthors have discussed limitations and future work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper presents a bi-level structured policy optimization algorithm for offline reinforcement learning (RL). The algorithm aims to address the distributional shift and function approximation challenges in offline RL by modeling a hierarchical interaction between the policy and the value function. The lower level focuses on constructing a confidence set of value estimates to control uncertainty arising from distribution mismatch, while the upper level aims to maximize a conservative value estimate. The paper provides theoretical guarantees for the algorithm and evaluates its performance on both synthetic and real-world datasets.

STRENGTHS:
- The paper proposes a novel bi-level structured policy optimization algorithm that addresses the challenges of distributional shift and function approximation in offline RL. This algorithm offers a comprehensive solution for real-world scenarios.
- The algorithm preserves the flexibility of the exploratory data distribution and enables model extrapolation, enhancing its power.
- The paper provides theoretical regret guarantees for the proposed algorithm without relying on data-coverage and completeness-type assumptions, increasing its applicability.

WEAKNESSES:
- There is only a weak connection between several parts of the paper. It is hard to find a central claim that supports the logical flow.
- There remain some issues unsolved in the paper. See the questions for details.

QUESTIONS:
- At the lower level, how can the confidence set, which constraints model learning, effectively prevent overly pessimistic evaluation?
- How can the proposed algorithm build on weaker conditions than the global data coverage? Does it benefit from the occupancy distribution estimation?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a pessimistic offline RL method. Different from the previous method in the function approximation setting, the method proposed in this paper does not require coverage and completeness assumption. Moreover, in terms of algorithm design, this paper derives a lower confidence bound for RL objective function and explicitly optimizes it to obtain the policy, which is the first attempt in this direction. Both theoretical and empirical results are presented to support the effectiveness of this algorithm design.\n\nSTRENGTHS:\n(1) This paper proposes a novel algorithm design to address the offline policy optimization problem by explicitly optimizing a lower bound of RL objective function. This design is first proposed to my best knowledge, which is valuable.\n(2) The algorithm design proposed in this paper has a very global optimality guarantee under weaker assumptions compared with previous works.\n(3) The author also derives a practical version of the algorithm which also has a theoretical guarantee. Moreover, to demonstrate the effectiveness of this practical version, several experiment results are presented.\n\nWEAKNESSES:\nThe experiment is not very strong:\n(1) Most of the standard simulation experiments are not considered thus the practical effectiveness of the algorithm is not convincing. The author might need to include more experiments are also included in OptiDICE paper (hopper, walker, halfcheetah, maze etc)\n(2) I was also wondering how the algorithm proposed in this paper compared most recent baseline such as TD3 + BC?\n\nQUESTIONS:\n(1) If $\\mathcal{U}^\\tau_{\\infty}=+\\infty$, then how would the constant $\\mathcal{U}^\\tau_2$ and $\\mathcal{U}^*_2$ look like, will them become unbounded as well? If that is the case, then the proposed algorithm still needs completeness assumption and coverage assumption to derive the bound in Theorem 5.1.? I don't think requiring coverage assumption for this paper should be a reason for rejection since this paper already made a lot of contributions, but the author needs to provide a clear discussion for this part otherwise it will be somehow misleading since it is not obvious how to achieve global optimal policy without any coverage requirements.\n\n(2) In linear MDP the algorithm complexity is $n^{-1/4}$? If so then why the algorithm has worse dependence on $n$? Can the author also provide some high-level explanation?\n\n(3) OptDICE has been proven to outperform CQL and other baselines (see the original paper of OptiDICE, why OptiDICE performs much worse than CQL?\n\nI will raise my score if the author can answer the above questions appropriately.\n\nLIMITATIONS:\nNo limitation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of offline reinforcement learning (RL). Offline RL involves computing a policy based on a static/pre-collected dataset, with no further interaction with the environment permitted. A significant hurdle in offline RL is the distributional shift resulting from insufficient exploration within the dataset. Moreover, previous work may necessitate the assumption of Bellman completeness for the utilized function class.  To overcome these challenges, the authors propose a bi-level structured policy optimization algorithm. This algorithm models a hierarchical interaction between the policy and the value function. At the lower level, the focus is on constructing a set of confident value estimates. The upper level, in contrast, aims to maximize a conservative estimate derived from this confidence set. The algorithm can be solved efficiently through a penalized adversarial estimation procedure. The authors provide theoretical regret guarantees for their algorithm and assess its performance using both synthetic and real-world datasets. Performance is compared against several well-known offline RL algorithms, such as CQL, IQL, and COMBO.\n\nSTRENGTHS:\nStrengths:\n\n1. The concept of optimizing the confidence lower and upper bounds in equation (4) over τ within Ω, with the aim of tightening the Confidence Interval (CI), is intriguing and holds potential for generating a more precise confidence interval.\n\n2. The paper, overall, is well-written and well-structured, demonstrating a clear and logical progression of ideas.\n\n3. The authors have given a comprehensive view of their algorithm by providing both theoretical analysis and empirical versions. Additionally, the experimental results demonstrate the effectiveness of the proposed algorithm, as shown on various synthetic datasets as well as a real-world dataset.\n\nWEAKNESSES:\nWeaknesses:\n\n1. Complexity: The algorithm, in comparison to alternatives such as IQL and COMBO, is notably more complex. It employs a game-theoretical formulation and requires the resolution of two optimization problems at each iteration. This results in an increase in both the number of hyperparameters that require tuning and the overall computational cost, and potentially instability in the optimization (i.e., convergence).\n\n2. Ambiguities and Clarifications in Section 3 (Lines 119 to 126): \n   - Shouldn't $G (x_∗) = \\sup_x \\{x \\cdot x_∗ − G(x)\\}$ be revised to $G_* (x_∗) = \\sup_x \\{x \\cdot x_∗ − G(x)\\}$?\n   - For $J(\\pi) = q_\\pi(s_0, a) + E_\\mu[\\lambda G_*((B^\\pi q^\\pi (s, a) − q^\\pi (s, a))/(1 − \\gamma))]$, where $\\lambda \\geq 0$, the initial term appears as if it should be $q_{\\pi}(s_0, \\pi)$. Additionally, could you explain how $\\lambda$ is typically chosen? Does the choice of $\\lambda$ significantly impact the algorithm's performance?\n   - The derivation from $J(\\pi) = q_\\pi(s_0, a) + E_\\mu[\\lambda G_*((B^\\pi q^\\pi (s, a) − q^\\pi (s, a))/(1 − \\gamma))]$ to equation (1) is unclear. Why is $\\lambda$ merely a coefficient of $G(x)$? Shouldn't it also be a coefficient of  $x · (B^\\pi q^\\pi (s, a) − q^\\pi(s, a))$?\n   - The terms L(τ,q,π) and ξ(G,τ) require explicit definitions.\n\n3. Justification for Assumption 2: The justifiability of Assumption 2 is uncertain. It appears to be more of a requirement for feasible derivation rather than a naturally occurring condition. Can it be guaranteed that $\\tau(s,a)$ won't reach infinity in real-world problems? More justification on this point would be appreciated.\n\n4. Need for Intuition: More insight into why your framework doesn't require the Bellman completeness assumption would be beneficial.\n\n5. Additional Evaluations: It would be helpful if you could evaluate your proposed algorithm using more standard benchmarks, such as D4RL. This would significantly aid in positioning your work within the existing literature.\n\nQUESTIONS:\nSee Weakness.\n\nLIMITATIONS:\nSee Weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-31"}