Title: Offline Minimax Soft-Q-learning Under Realizability and Partial Coverage
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement learning theory

PAC RL

Offline Reinforcement learning
Abstract: We consider offline reinforcement learning (RL) where we only have only access to offline data. In contrast to numerous offline RL algorithms that necessitate the uniform coverage of the offline data over state and action space, we propose value-based algorithms with PAC guarantees under partial coverage, specifically, coverage of offline data against a single policy, and realizability of soft Q-function (a.k.a., entropy-regularized Q-function) and another function, which is defined as a solution to a saddle point of certain minimax optimization problem). Furthermore, we show the analogous result for Q-functions instead of soft Q-functions. To attain these guarantees, we use novel algorithms with minimax loss functions to accurately estimate soft Q-functions and Q-functions with 
-convergence guarantees measured on the offline data. We introduce these loss functions by casting the estimation problems into nonlinear convex optimization problems and taking the Lagrange functions.

Offline Minimax Soft-Q-learning Under Realizability and Partial Coverage

Abstract

In offline RL, we have no opportunity to explore so we must make assumptions that the data is sufficient to guide picking a good policy, and we want to make these assumptions as harmless as possible. In this work, we propose value-based algorithms for offline RL with PAC guarantees under just partial coverage, specifically, coverage of just a single comparator policy, and realizability of the soft (entropy-regularized) Q-function of the single policy and a related function defined as a saddle point of certain minimax optimization problem. This offers refined and generally more lax conditions for offline RL. We further show an analogous result for vanilla Q-functions under a soft margin condition. To attain these guarantees, we leverage novel minimax learning algorithms and analyses to accurately estimate either soft or vanilla Q-functions with strong L 2 -convergence guarantees. Our algorithms' loss functions arise from casting the estimation problems as nonlinear convex optimization problems and Lagrangifying. Surprisingly we handle partial coverage even without explicitly enforcing pessimism.

1 Introduction

In offline Reinforcement Learning (RL), we must learn exclusively from offline data and are unable to actively interact with the environment (Levine et al., 2020). Offline RL has garnered considerable interest in a range of applications where experimentation may be prohibitively costly or risky.

Offline RL is generally based on two types of assumptions: sufficient coverage in the offline data and sufficient function approximation. For instance, classical Fitted-Q-iteration (Antos et al., 2008; Chen and Jiang, 2019) requires (a) full coverage in the offline data, max ( s,a ) d π,µ 0 ( s, a ) /P π b ( s, a ) < ∞ for any policy π where P π b ( s, a ) is the offline data's distribution on the states and actions and d π,µ 0 ( s, a ) is the state-action occupancy distribution under a policy π and initial-state distribution µ 0 ( s ) ; (b) realizability of the Q ∗ -function in a hypothesis class; and (c) Bellman completeness, i.e. , the Bellman operator applied to any function in the hypothesis class remains in the class. Full coverage (a) and Bellman completeness (c) can be particularly stringent because offline data is often insufficiently exploratory and Bellman completeness significantly restricts transition dynamics.

To overcome these challenges, we here propose algorithms with guarantees under realizability of single functions and refined partial coverage of single policies, and without Bellman completeness. Wetackle this by introducing two novel value-based algorithms. The first algorithm, MSQP (mimimax soft-Q-learning with penalization), comprises of two steps: learning soft Q-functions (a.k.a., entropy-

Nathan Kallus

regularized Q-functions, as defined in Fox et al., 2015; Schulman et al., 2017) from offline data, and using the softmax policies of the learned soft Q-functions. The second algorithm, MQP (mimimax Q-learning with penalization), consists of two steps: learning standard Q-functions from offline data and employing the greedy policy of the learned Q function on the offline data.

Using the above-mentioned two algorithms, we attain PAC guarantees under partial coverage and realizability, yet without Bellman completeness. In particular, in MSQP using soft Q-functions, we ensure strong performance under the realizability of q ⋆ α , l ⋆ α and the (density-ratio-based) partial coverage max ( s,a ) d π ⋆ α ,µ 0 ( s, a ) /P b ( s, a ) < ∞ . Here q ⋆ α is a soft Q-function, l ⋆ α is a function that possesses a certain dual relation to q ⋆ α , π ⋆ α is the soft-max optimal policy, and α is the temperature parameter for the entropy-regularization. Notably, max ( s,a ) d π ⋆ α ,µ 0 ( s, a ) /P b ( s, a ) < ∞ is significantly less stringent than the uniform coverage in that the coverage is only imposed against a policy π ⋆ α . In MQP using Q-functions, we similarly ensure strong performance under a soft margin, the realizability of q ⋆ , l ⋆ , and the partial coverage max ( s,a ) d π ⋆ ,µ 0 ( s, a ) /P b ( s, a ) < ∞ . Here q ⋆ is the vanilla Q-function and l ⋆ is a function that possesses a certain dual relation to q ⋆ , and π ⋆ is the usual optimal policy. Note the soft margin is introduced to allow realizability on standard Qfunctions rather than soft Q-functions. However, the conditions max ( s,a ) d π ⋆ α ,µ 0 ( s, a ) /P b ( s, a ) < ∞ or max ( s,a ) d π ⋆ ,µ 0 ( s, a ) /P b ( s, a ) < ∞ may still be strong as these marginal density ratios may not exist in large-scale MDPs. For example, this condition is easily violated when the initial distribution µ 0 is not covered by P b ( i.e. , max s µ 0 ( s ) /P b ( s ) = ∞ where P b ( s ) := ∑ a P b ( s, a ) ). Therefore, as an additional innovation, in our algorithms we can further relax these density-ratio-based partial coverage conditions. Specifically, we can demonstrate results under a refined partial coverage , which is adaptive to Q-function classes, even when the initial distribution µ 0 is not covered by P b . 2

The primary challenge lies in the design of loss functions for effectively learning soft Q-functions and vanilla Q-functions from offline data without Bellman completness . To tackle this, we devise new minimax loss functions with certain regularization terms to achieve favorable L 2 -convergence rates on the offline data ( i.e. , in terms of E ( s,a ) ∼ P b [ { ˆ q α -q } 2 ( s, a )] given an estimator ˆ q ). This result serves as the key building block for obtaining refined partial coverage under realizability and is of independent interest in its own right. Existing results are often constrained to specific models, such as linear models (Shi et al., 2022a), or they require Bellman completeness (Antos et al., 2008; Chen and Qi, 2022; Chen and Jiang, 2019). In contrast, our guarantee is applicable to any function

approximation method, without the need for Bellman completeness. To the best of our knowledge, this is the first guarantee of its kind.

Our work exhibits marked improvements over two closely related studies (Zhan et al., 2022; Chen and Jiang, 2022). Similar to our work, they propose algorithms that operate under the realizability of specific functions and partial coverage, yet without Bellman completeness. Zhan et al. (2022) ensures a PAC guarantee under (a') partial coverage in the offline data max ( s,a ) d ˜ π ⋆ α ,µ 0 ( s, a ) /P b ( s, a ) < ∞ where ˜ π ⋆ α is a specific near-optimal policy under the regularization, which differs from the soft optimal policy, and (b') realizability of d ˜ π ⋆ α ,µ 0 /P b and the regularized value function. However, unlike MSQP, it is unclear how to refine the abovementioned coverage, i.e. , the guarantee could be vacuous when the initial distribution is not covered by offline data. A similar guarantee, but without regularization, is obtained under the additional hard margin (a.k.a., gap) condition in Chen and Jiang (2022). Our soft margin is a strict relaxation of the hard margin, which is important because, unlike the soft margin, the hard margin generally does not hold in continuous state spaces and involves very large constants in discrete state spaces. Lastly, although Chen and Jiang (2022); Zhan et al. (2022) use completely different algorithms and attain guarantees for regularized value-functions and non-regularized value functions, respectively, our guarantee can afford guarantees for regularized and non-regularized value-functions in a unified manner since MQP can be seen as a limit of MSQP when α goes to 0 .

Our contributions are summarized below and in Table 1.

We establish that the optimal policy can be learned under partial coverage and realizability of the optimal soft Q-function and its dual. Notably, we abstain from the use of possibly stronger conditions in offline RL, such as full coverage, Bellman completeness, and uniform realizability over the policy class (such as q π ∈ Q for any π as in Jiang and Huang, 2020). In particular, while a similar guarantee is provided in Zhan et al. (2022), our partial coverage guarantee has an advantage in that we are able to potentially accommodate scenarios where the initial distribution is not covered by P b . This is feasible because our algorithm is valuebased in nature, which allows us to leverage the structure of the Q-function classes and refine the coverage condition.

We demonstrate that the optimal policy can be learned under partial coverage, realizability of the Q-function and its dual, and a soft margin. While a similar guarantee is obtained in Chen and Jiang (2022), our guarantee has the advantage that the soft margin is significantly less stringent than the hard margin required therein.

1.1 Related Works

We summarize related works as follows. Further related works is discussed in Section A.

Offline RL under partial coverage. There is a growing number of results under partial coverage following the principle of pessimism in offline RL (Yu et al., 2020; Kidambi et al., 2020). In comparison to works that focus on tabular (Rashidinejad et al., 2021; Li et al., 2022; Shi et al., 2022b; Yin and Wang, 2021) or linear models (Jin et al., 2020; Chang et al., 2021; Zhang et al., 2022; Nguyen-Tang et al., 2022; Bai et al., 2022), our emphasis is on general function approximation (Jiang and Huang, 2020; Uehara and Sun, 2021; Xie et al., 2021; Zhan et al., 2022; Zhu et al., 2023; Rashidinejad et al., 2022; Zanette and Wainwright, 2022; Ozdaglar et al., 2023). Among them, we specifically focus on model-free methods. The representative work is summarized in Table 1.

Soft (entropy-regularized) Q-functions. Soft Q-functions are utilized in various contexts in RL (Geist et al., 2019; Neu et al., 2017). They have been shown to improve performance in online RL settings, as demonstrated in Soft Q-Learning (Fox et al., 2015; Schulman et al., 2017) and Soft Actor Critic (Haarnoja et al., 2018). In the field of imitation learning, they play a crucial role in Maximum Entropy IRL (Ziebart et al., 2008, 2010). Furthermore, within the realm of offline RL, these soft Q-functions are utilized to make the learned policy and behavior policy sufficiently similar (Wu et al., 2019; Fakoor et al., 2021). However, to the best of the authors' knowledge, none of these proposals in the context of offline RL have provided sample complexity results under partial coverage.

Lagrangian view of offline RL. In the realm of offline policy evaluation (OPE), Nachum and Dai (2020); Yang et al. (2020); Huang and Jiang (2022) have formulated the problem as a constrained

linear optimization problem. Notably, within the context of policy optimization, Zhan et al. (2022) have proposed estimators for regularized density ratios with L 2 -convergence guarantees, which is a crucial step in obtaining a near-optimal policy. Our work is similarly motivated, but with a key distinction: our target functions are the soft Q-function and Q-function, rather than the regularized density ratio, which presents additional analytical challenges due to the nonlinear constraint.

2 Preliminaries

We consider an infinite-horizon discounted MDP M = ⟨S , A , P, r, γ, µ 0 ⟩ where S is the state space, A is the finite action space, γ ∈ [0 , 1) is the discount factor, reward r is a random variable following P r ( · | s, a ) on [ R min , R max ] ( R min ≥ 0 ), µ 0 is the initial distribution. A policy π : S → ∆( A ) is a map from the state to the distribution over actions. We denote the discounted state-action occupancy distribution under a policy π starting from an initial distribution µ 0 by d π,µ 0 ( s, a ) . With slight abuse of notation, we denote d π,µ 0 ( s ) = ∑ a d π,µ 0 ( s, a ) . We define the value under π as J ( π ) := E π [ ∑ ∞ t =0 γ t ˜ r ( s t , a t )] where the expectation is taken under π . We denote the optimal policy arg max π J ( π ) by π ⋆ , and its Q-function E π ⋆ [ ∑ t γ t ˜ r ( s t , a t ) | s 0 = s, a 0 = a ] by q ⋆ ( s, a ) .

In offline RL, using offline data D = { ( s i , a i , r i , s ′ i ) : i = 1 , . . . , n } , we search for the policy π ⋆ that maximizes the policy value. We suppose each ( s i , a i , r i , s ′ i ) is sampled i.i.d. from s i ∼ P b , a i ∼ π b ( · | s ) , r i ∼ P r ( · | s i , a i ) , s ′ i ∼ P ( · | s i , a i ) . We denote the sample average of f by E n [ f ( s, a, r, s ′ )] = 1 n ∑ n i =1 f ( s i , a i , r i , s ′ i ) , and the expectation of f with respect to the offline data distribution by E [ f ( s, a, r, s ′ )] (without any scripts). The policy π b used to collect data is typically referred to as a behavior policy. With slight abuse of notation, we denote P b ( s, a ) = P b ( s ) π b ( a | s ) .

̸

Notation. Wedenote the support of P b ( · ) by ( S×A ) b , and the L ∞ -norm on ( S×A ) b by ∥·∥ ∞ ,b . The L ∞ -norm on ( S × A ) is denoted by ∥ · ∥ ∞ . We define w π ( s, a ) = d π,µ 0 ( s, a ) /P b ( s, a ) (if it exists). We define sofmax( h ) = exp( h ( s,a )) ∑ a exp( h ( s,a )) and ∥ h ∥ 2 = E ( s,a ) ∼ P b [ h 2 ( s, a )] 1 / 2 for h : S × A → R . We denote universal constants by c 1 , c 2 , . . . . We use the convention a/ 0 = ∞ when a = 0 and 0 / 0 = 0 .

3 Algorithms

In this section, we present two algorithms. The first algorithm aims to estimate the soft optimal policy by first estimating a soft Q-function. The second algorithm estimates the optimal policy after estimating the Q-function.

3.1 Minimax Soft-Q-learning with Penalization

Our ultimate aim is to mimic the optimal policy π ⋆ . As a first step, we begin by finding a policy that maximizes the following regularized objective: arg max π J α ( π ) where for α > 0 we define

This objective function is used in a variety of contexts in RL as mentioned in Section 1.1. The optimal policy that maximizes J α ( π ) with respect to π is

where q ⋆ α : S × A → R is the soft Q-function uniquely characterized by the soft Bellman equation:

where Ω α,π b : [ S × A → R ] → [ S → R ] has Ω α,π b ( q )( s ) = α log ∑ a { exp( q ( s, a ′ ) /α ) π b ( a ′ | s ) } . As opposed to the standard objective function with α = 0 , the KL penalty term serves as a regularization term that renders π ⋆ α sufficiently proximate to π b . As α approaches ∞ , the optimal policy π ⋆ α approaches π b . On the other hand, when α = 0 , π ⋆ α is π ⋆ . Thus, in order to compete with π ⋆ , it is necessary to keep α sufficiently small. We elaborate on this selection procedure in Section 5.

The natural method for offline RL using this formulation involves learning q ⋆ α from the offline data and plugging it into (1). The question that remains is how to accurately learn q ⋆ α from the offline data. We consider the following optimization problem:

Algorithm 1 MSQP (Minimax Soft-Q-learning with Penalization)

1: Require : Parameter α ∈ R + , Models Q , L ⊂ [ S × A → R + ] .

2: Estimate q ⋆ α as follows:

3: Estimate the soft optimal policy: ˆ π α = softmax(ˆ q α /α +log π b ) .

Algorithm 2 MQP (Minimax Q ⋆ -learning with Penalization)

1: Require : Models Q , L ⊂ [ S × A → R + ] .

2: Estimate q ⋆ as follows:

3: Estimate the optimal policy: ˆ π 0 ( a | s ) = arg max a : π b ( a | s ) > 0 ˆ q 0 ( s, a ) .

where Q ′ consists of all functions q : S × A → R satisfying

Here, because of the constraint (3), the solution is q ⋆ α . Furthermore, we use q 2 ( s, a ) in (2) because this choice relaxes the equality in (3) to an inequality ≤ 0 as we will demonstrate in Section B. Consequently, the entire optimization problem outlined in (2) and (3) transforms into a convex optimization problem.

Then, using the method of Lagrange multipliers, (2) is transformed into

Being motivated by the above formulation, our MSQP algorithm, specified in Algorithm 1, approximates this formulation by replacing expectations with sample averages and restricting optimization to function classes with bounded complexity.

Remark 1 (Computation) . Although minimax optimization is generally difficult to solve, it is computationally feasible when we choose RKHS or linear function classes for L . In this case, we can solve the inner maximization problem analytically in closed form, as the objective function is linear in l . As a result, the minimax optimization problem reduces to empirical risk minimization.

3.2 Minimax Q ⋆ -learning with Penalization

Next, we examine a policy learning algorithm utilizing Q ⋆ -functions. To learn Q ⋆ , our objective function is derived from the constrained optimization problem:

where Q ⋆ ′ consists of all functions q : S × A → R satisfying

Next, again using the method of Lagrange multipliers, (6) is transformed into min

q

l

L

0

(

q, l

)

, L

0

(

q, l

) :=

E

Note L 0 is the limit of L α as α → 0 .

Our MQP algorithm, specified in Algorithm 2, similarly approximates this formulation by replacing expectations with sample averages and restricting optimization to function classes with bounded complexity. Our final policy is greedy with respect to the learned Q-function but restricting to the support of the offline data in order to avoid exploiting regions not covered by the offline data.

Remark 2 (Prominent differences) . There exist several other minimax estimators for Q ⋆ including BRM (Antos et al., 2008) and MABO (Xie and Jiang, 2020). Although these ensure convergence guarantees in terms of Bellman residual errors, they do not ensure the guarantee in terms of L 2 -errors, which is our focus. Our minimax objective function differs significantly from that of the aforementioned approaches, and its unique design plays a pivotal role in enabling L 2 -rates.

[

q

2

(

s, a

)

/

2 +

{

γ

a

′

q

(

s

′

′

, a

) +

r

-

q

(

s, a

)

}

l

(

s, a

)]

.

(7)

4 L 2 -convergence Rates for Soft Q -functions and Q ⋆ -functions

To analyze our Q-estimators we first establish conditions that ensure q ⋆ α = arg min q ∈Q max l ∈L L α ( q, l ) on the support ( S × A ) b . Building on this, we prove L 2 -convergence rates for ˆ q α and ˆ q 0 . These L 2 -convergence guarantees are subsequently translated into performance guarantees of the policies we output in Section 5.

4.1 Identification of Soft Q-functions

Consider an L 2 -space H where the inner product is define as ⟨ h 1 , h 2 ⟩ = E ( s,a ) ∼ P b [ h 1 ( s, a ) h 2 ( s, a )] . Then we define two operators and a key function: 3

̸

These satisfy a key adjoint property, which we leverage to show ( q ⋆ α , l ⋆ α ) is a saddle point of L α ( q, l ) .

Our first assumption ensures that l ⋆ α exists.

Assumption 1. Suppose ∥ d π ⋆ α ,P b /P b ∥ ∞ < ∞ . Note the infinity norm ∥ · ∥ ∞ is over S × A .

Proposition 1. Under Assumption 1, we have ∥ l ⋆ α ∥ ∞ < ∞ .

Proposition 1 is immediate noting that ( I -γ { P ⋆ α } ⊤ ) -1 ( P b ( · ) q ⋆ α ( · )) = ∑ ∞ t =0 γ t ( { P ⋆ α } ⊤ ) t ( P b q ⋆ α ) and recalling the discounted occupancy measure under π ⋆ α with initial distribution µ 0 is written as d π ⋆ α ,µ 0 = (1 -γ )( I -γ { P ⋆ α } ⊤ ) -1 ( µ 0 ) . Hence, ∥ l ⋆ α ∥ ∞ ≤ (1 -γ ) -1 R max ∥ d π ⋆ α,P b /P b ∥ ∞ . Note that ∥ d π ⋆ α,P b /P b ∥ ∞ crucially differs with the standard density-ratio-based concentrability coefficient ∥ d π ∗ α ,µ 0 /P b ∥ ∞ in offline RL. Unlike ∥ d π ∗ α ,P b /P b ∥ ∞ , the value of ∥ d π ∗ α ,µ 0 /P b ∥ ∞ can be infinite when the initial distribution µ 0 is not covered by offline data P b as the practical motivating example is explained in the footnote in Section 1 and Example 2.

Our next assumption ensures q ⋆ α ≥ 0 , which also guarantees that l ⋆ α ≥ 0 .

Assumption 2 can be satisfied by rescaling reward ( i.e. , rescaling R min ) as long as ∥ π ∗ α /π b ∥ ∞ is finite. Hence, it is very mild. Putting Lemma 1 together with our assumptions we have the following.

Lemma 2. Suppose Assumptions 1 and 2 hold. Then, ( q ⋆ α , l ⋆ α ) is a saddle point of L α ( q, l ) over q ∈ H , l ∈ H , i.e. , L α ( q, l ⋆ α ) ≥ L α ( q ⋆ α , l ⋆ α ) ≥ L α ( q ⋆ α , l ) ∀ q ∈ H , ∀ l ∈ H .

Recall that a point (˜ q, ˜ l ) is a saddle point if and only if the strong duality holds, and ˜ q ∈ arg min q ∈H sup l ∈H L α ( q, l ) , ˜ l ∈ arg max l ∈H inf q ∈H L α ( q, l ) using the general characterization (Bertsekas, 2009). Hence, Lemma 2 ensures q ⋆ α ∈ arg min q ∈H max l ∈H L α ( q, l ) .

Next, we consider the constrained optimization problem when we use function classes Q ⊂ H , L ⊂ H . As long as the saddle point is included in ( Q , L ) , we can prove that q ⋆ α is a unique minimaxer.

Lemma 3. Suppose Assumptions 1 and 2 hold, q ⋆ α ∈ Q , and l ⋆ α ∈ L . Then, we have that q ⋆ α = arg min q ∈Q sup l ∈L L α ( q, l ) on the support ( S × A ) b .

This establishes that realizability ( q ⋆ α ∈ Q , l ⋆ α ∈ L ) is sufficient to identify q ⋆ α on the offline data distribution. At a high level, q ⋆ α ∈ arg min q ∈Q sup l ∈L L α ( q, l ) is established through the invariance of saddle points, i.e. , saddle points over original sets remain saddle points over restricted sets. Its uniqueness is verified by the strong convexity in q of L α ( q, l ) induced by E ( s,a ) ∼ P b [ q 2 ( s, a )] .

4.2 L 2 -convergence Rate for Soft Q-estimators

Based on the population-level results in Section 4.1, we give a finite-sample error analysis of ˆ q α Assumption 3 (Realizability of soft Q-function) . Suppose q ⋆ α ∈ Q and ∥ q ∥ ∞ ≤ B Q ∀ q ∈ Q . Assumption 4 (Realizability of Lagrange multiplier) . Suppose l ⋆ α ∈ L and ∥ l ∥ ∞ ≤ B L ∀ l ∈ L .

It is natural to set B Q = (1 -γ ) -1 R max and B L = (1 -γ ) -1 R max ∥ d π ∗ α ,P b /P b ∥ ∞ , but letting these be arbitrary offers further flexibility to our results.

Theorem 1 ( L 2 -convergence of soft Q-estimators) . Suppose Assumptions 1, 2, 3, and 4 hold. Then, with probability 1 -δ , the L 2 -error ∥ ˆ q α -q ⋆ α ∥ 2 is upper-bounded by

Our result is significant as it relies on realizability-type conditions rather than Bellman closedness. Since the majority of existing works focus on non-regularized Q-functions, we postpone the comparison to these existing works to the next section. Note when Q and L are infinite, we can easily replace |Q| , |L| with their L ∞ -covering numbers following Uehara et al. (2021). Details are given in the appendix.

4.3 L 2 -convergence Rate for Q ⋆ -functions

Next, we give analogous finite-sample error analysis of ˆ q 0 leveraging the same reasoning. Assumption 5 (Realizability of Q ⋆ -functions) . Suppose q ⋆ ∈ Q and ∥ q ∥ ∞ ≤ B Q ∀ q ∈ Q

Next, we define the Lagrange multiplier:

While l ⋆ involves the density ratio, this is always well-defined as long as ∥ d π ⋆ ,P b /P b ∥ ∞ < ∞ . Then, it can be similarly established that ( q ⋆ , l ⋆ ) is a saddle point of L 0 ( q, l ) over q ∈ H , l ∈ H as we show in Lemma 2. We lastly require its realizability.

Assumption 6 (Realizability of Lagrange multiplier) . Suppose ∥ d π ⋆ ,P b /P b ∥ ∞ < ∞ and l ⋆ ∈ L . Further suppose ∥ l ∥ ∞ ≤ B L ∀ l ∈ L .

Theorem 2 ( L 2 -convergence of Q-estimators) . Suppose Assumptions 5 and 6 hold. Then, with probability 1 -δ , the L 2 -error ∥ ˆ q 0 -q ⋆ ∥ 2 is upper-bounded by

To the best of our knowledge, this is the first guarantee on L 2 errors for learning q ∗ using general function approximation without relying on Bellman completeness . This is highly nontrivial, and we have carefully crafted our algorithm to obtain this guarantee. Existing results are often specific to particular models, such as linear models (Shi et al., 2022a), or they require Bellman completeness (Chen and Jiang, 2019; Chen and Qi, 2022), or they are limited to offline policy evaluation scenarios (Huang and Jiang, 2022) ( i.e. , cases involving linear Bellman operators, but nonlinear Bellman operators). Actually, it seems that even under the assumption of Bellman completeness, obtaining an L2 guarantee without strong coverage assumptions remains unclear. A detailed comparison among these different approaches is presented in Section A.

5 Finite Sample Guarantee of MSQP

In this section, we present our primary sample complexity guarantee for our MSQP algorithm under the assumptions of realizability of q ⋆ α and l ⋆ α and partial coverage. We first show the learned policy ˆ π α can compete with π ⋆ α . Finally we show ˆ π α can compete with π ⋆ by selecting α properly.

We first introduce the flattened behavior policy π ⋄ b , which is uniform on the support of π b . We use it as a technical device to define a model-free concentrability coefficient following Xie et al. (2021).

.

Definition 1 (Model-free concentrability coefficient) . Define

where π ⋄ b ( · | s ) = { 0 π b ( · | s ) = 0 1 / |{ a ∈ A | π b ( a | s ) > 0 }| π b ( · | s ) > 0 is the flattened behavior policy.

Clearly, C Q ,d π ⋆ α ,µ 0 is smaller than density-ratio-based concentrability coefficient, in other words,

Here, we always have ∥ π ⋄ b /π b ∥ < ∞ even if π b ( a | s ) is 0 for some ( s, a ) . In the special case where π b ( a | s ) ≥ 1 /C ′ for any ( s, a ) , we have C Q ,d π ⋆ α ,µ 0 ≤ C ′ ∥ d π ⋆ α ,µ 0 /P b ∥ ∞ . The coefficient C Q ,d π ⋆ α ,µ 0 is is a refined concentrability coefficient, which adapts to a function class Q . For example, in linear MDPs, it reduces to a relative condition number as follows. Similar properties are obtained in related works (Xie et al., 2021; Uehara and Sun, 2021).

Example 1 (Linear MDPs) . A linear MDP is one such that, for a known feature vector ϕ : S × A → R d , the true density satisfies P ( s ′ | s, a ) = ⟨ µ ⋆ ( s ′ ) , ϕ ( s, a ) ⟩ for some µ ⋆ : S → R d and the reward function satisfies E [ r | s, a ] = ⟨ θ r , ϕ ( s, a ) ⟩ for some θ r ∈ R d .

In linear MDPs, q ⋆ α is clearly linear in ϕ ( s, a ) . Hence, the natural function class is Q = {⟨ θ, ϕ ( s, a ) ⟩ | ∥ θ ∥ ≤ B } for a certain B ∈ R + . Then, we have

̸

We are now prepared to present our main result, which states that given the realizability of the soft Q-function q ⋆ α and Lagrange multiplier l ⋆ α , it is possible to compete with π ⋆ α under the coverage condition C Q ,d π ⋆ α ,µ 0 < ∞ , ∥ d π ⋆ α ,P b /P b ∥ ∞ < ∞ .

Theorem 3 ( ˆ π α can compete with π ⋆ α ) . Fix α > 0 . Suppose Assumptions 1, 2, 3, and 4 hold. With probability 1 -δ , the regret J ( π ⋆ α ) -J (ˆ π α ) is upper-bounded by

The proof mainly consists of two steps: (1) obtaining L 2 -errors of ˆ q α as previously demonstrated in Theorem 1, (2) translating this error into the error of ˆ π α . In the second step, the Lipshitz continuity of the softmax function plays a crucial role. If there is no regularization ( α = 0 ) and the greedy policy of q ⋆ 0 is utilized, the second step does not proceed (without any further additional assumptions).

Our ultimate goal is to compete with π ⋆ . Theorem 3 serves as the primary foundation for this goal. The remaining task is to analyze the approximation error J ( π ⋆ ) -J ( π ⋆ α ) . Fortunately, this term can be controlled through α and the density ratio between π ⋆ and π b . Then, by properly controlling α , we can obtain the following sample complexity result.

Theorem 4 (PAC guarantee of ˆ π α ) . Fix any ϵ > 0 . Suppose Assumptions 1, 2, 3, and 4 hold for α = c/n 1 / 8 and ∥ π ⋆ 0 /π b ∥ ∞ ≤ C 0 , C Q ,d π ⋆ α ,µ 0 < ∞ . Then, if n is at least

with probability at least 1 -δ , we can ensure J ( π ⋆ ) -J (ˆ π α ) ≤ ϵ .

In summary, the realiazability of q ⋆ α , l ⋆ α , per-step coverage ∥ π ⋆ 0 /π b ∥ ∞ < ∞ and partial coverage C Q ,d π ⋆ α ,µ 0 < ∞ , ∥ d π ⋆ α ,P b /P b ∥ ∞ < ∞ are sufficient to compete with π ⋆ . This is a novel and attractive result. Firstly, if we solely use the naïve FQI or Bellman residual minimization, existing PAC results require the global coverage ∥ d π,µ 0 /P b ∥ ∞ < ∞ for any possible policy π (Munos and Szepesvári, 2008; Antos et al., 2008). Our result only requires coverage under a single policy π ⋆ α (near-optimal policy). Secondly, we only require the realizability of two functions, and we do not necessitate

realizability-type conditions for all policies in the policy class or Bellman completeness, unlike existing works with partial coverage (Xie et al., 2021; Jiang and Huang, 2020).

The most similar result is Zhan et al. (2022). However, our guarantee possesses a certain advantage over their guarantee as follows. They demonstrate the realizability of certain functions ˜ w ⋆ α , v ⋆ α and partial coverage ∥ d ˜ π α ,µ 0 /P b ∥ < ∞ are sufficient conditions in offline RL, where ˜ w ⋆ α = d ˜ π α ,µ 0 /P b ( ˜ π α is a certain regularized optimal policy, but fundamentally distinct from π ⋆ α ) and v ⋆ α is a nearoptimal regularized value function parameterized by α . Here, we have ˜ w ⋆ 0 = w ⋆ , v ⋆ 0 = v ⋆ . Our guarantee has a similar flavor in the sense that it roughly illustrates realizability and partial coverage are sufficient conditions. However, the meanings of realizability and partial coverage are significantly different. In particular, by employing our algorithm, we can ensure PAC guarantees under the boundedness of the refined concentrability coefficient C Q ,d π ⋆ α ,µ 0 < ∞ (and ∥ d π ⋆ α ,P b /P b ∥ ∞ through B L ). As a result, the L ∞ -norm of the density-ratio-based concentrability coefficient ∥ d π ⋆ α ,µ 0 /P b ∥ ∞ can even be infinite. More specifically, we can permit situations where max s µ 0 ( s ) /P b ( s ) = ∞ as we will see the practical example soon. Conversely, Zhan et al. (2022) excludes this possibility since the algorithm explicitly estimates the density ratio ˜ w ⋆ α .

Example 2 (Contextual bandit under external validity) . We consider the contextual bandit setting where we want to optimize J ( π ) = E s ∼ µ 0 ,a ∼ π ( s ) ,r ∼ P r ( s,a ) [ r ] using offline data s ∼ P b , a ∼ π b ( s ) , r ∼ P ( s, a ) . This is the simplest RL setting with γ = 0 . Here, note µ 0 could be different from P b . This case often happens in practice as discussed in the literature on causal inference related to external validity (Pearl and Bareinboim, 2014; Dahabreh et al., 2019; Uehara et al., 2020), which refers to the shift between the target population and the offline data. Here, our PAC guarantee does not require that µ 0 ( s ) is covered by P b ( s ) in terms of the density ratio as long as the relative condition number is upper-bounded when we use linear models. On the other hand, Zhan et al. (2022) excludes this possibility.

Despite the aforementioned advantage of our approach, unfortunately, our sample complexity of O (1 /ϵ 8 ) is slower compared to that of O (1 /ϵ 6 ) in Zhan et al. (2022). In the following, we demonstrate that MQP, which is a special version of MSQP when α → 0 , can achieve a faster rate of O (1 /ϵ 2 ) .

6 Finite Sample Guarantee of MQP

In this section, building upon the convergence result of ˆ q 0 , we demonstrate the finite sample guarantee of our MQP algorithm under partial coverage. We first introduce the soft margin.

Assumption 7 (Soft margin) . For any a ′ ∈ A , there exists t 0 ∈ R + , β ∈ (0 , ∞ ] such that

In the extreme case, if there exists a gap in q ⋆ (also known as a hard margin) so that the best action is always better than the second-best by some lower bounded amount, then the soft margin is satisfied with β = ∞ . Thus, the soft margin is more general than the gap condition used in Simchowitz and Jamieson (2019); Wu et al. (2022). Crucially, a gap generally does not exist in continuous state spaces unless Q-functions are discontinuous or one action is always option, or a gap involves a large t 0 constant in discrete state spaces with bad dependence on the number of states. In contrast, a soft margin with some β > 0 generally holds (see, e.g. , lemma 4 in Hu et al., 2021). The soft margin is widely used in the literature on classification, decision making, and RL (Audibert and Tsybakov, 2007; Perchet and Rigollet, 2013; Luedtke and Chambaz, 2020; Hu et al., 2021, 2022).

Theorem 5 (PAC guarantee of ˆ π 0 ) . Suppose Assumptions 5, 6, and 7 hold and ∥ π ⋆ /π b ∥ ∞ ≤ C 0 . Fix any ϵ > 0 . Then, if n is at least

{ with probability at least 1 -δ , we can ensure J ( π ⋆ ) -J (ˆ π 0 ) ≤ ϵ .

The proof mainly consists of two steps: (1) obtaining L 2 -errors of ˆ q 0 as demonstrated in Theorem 2, (2) translating this error into the error of ˆ π 0 . In the second step, the soft margin plays a crucial role.

These theorems indicate that the realizability of the Q -function q ⋆ and Lagrange multiplier l ⋆ , and the soft margin are sufficient for the PAC guarantee under partial coverage C Q ,d π ⋆ ,µ 0 < ∞ ,

∥ d π ⋆ ,P b /P b ∥ ∞ < ∞ . Our algorithm is agnostic to β and operates under any value of β . In particular, when there is a gap ( β = ∞ ), we can achieve sample complexity of O (1 /ϵ 2 ) 4 . In comparison to Theorem 3, although we additionally use the soft margin, the realizability in Theorem 5 is more appealing since it is imposed on the standard Q-function q ⋆ . The closest guarantee to our work can be found in Chen and Jiang (2022), which demonstrates that the existence of the gap in q ⋆ , the realizability of q ⋆ , w ⋆ (:= d π ⋆ ,µ 0 /P π b ) , and partial coverage ∥ w ⋆ ∥ ∞ < ∞ are sufficient conditions. A similar comparison is made in Ozdaglar et al. (2023). In comparison to their work, we use the soft margin, which is significantly less stringent.

7 Conclusions

We propose two value-based algorithms, MSQP and MQP, that operate under realizability of certain functions and partial coverage ( i.e. , single-policy-coverage). Notably, our guarantee does not require Bellman completeness and uniform-type realizability over the policy class. While guarantees with similar flavors are obtained in Zhan et al. (2022); Chen and Jiang (2022), MSQP can potentially relax the density-ratio-based partial coverage regarding the initial distribution as opposed to Zhan et al. (2022), and MQP can operate under the soft-margin, which is less stringent than the hard margin imposed in Chen and Jiang (2022). Moreover, both algorithms work on Q-functions, which are more commonly used in practice.

References

András Antos, Csaba Szepesvári, and Rémi Munos. Learning near-optimal policies with bellmanresidual minimization based fitted policy iteration and a single sample path. Machine Learning , 71 (1):89-129, 2008.

Jean-Yves Audibert and Alexandre B Tsybakov. Fast learning rates for plug-in classifiers. The Annals of statistics , 35(2):608-633, 2007.

Chenjia Bai, Lingxiao Wang, Zhuoran Yang, Zhihong Deng, Animesh Garg, Peng Liu, and Zhaoran Wang. Pessimistic bootstrapping for uncertainty-driven offline reinforcement learning. arXiv preprint arXiv:2202.11566 , 2022.

Dimitri Bertsekas. Convex optimization theory , volume 1. Athena Scientific, 2009.

Jonathan Chang, Masatoshi Uehara, Dhruv Sreenivas, Rahul Kidambi, and Wen Sun. Mitigating covariate shift in imitation learning via offline data with partial coverage. Advances in Neural Information Processing Systems , 34:965-979, 2021.

Jinglin Chen and Nan Jiang. Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning , pages 1042-1051. PMLR, 2019.

Jinglin Chen and Nan Jiang. Offline reinforcement learning under value and density-ratio realizability: the power of gaps. arXiv preprint arXiv:2203.13935 , 2022.

Xiaohong Chen and Zhengling Qi. On well-posedness and minimax optimal rates of nonparametric q-function estimation in off-policy evaluation. arXiv preprint arXiv:2201.06169 , 2022.

Stephen R Cole and Elizabeth A Stuart. Generalizing evidence from randomized clinical trials to target populations: the actg 320 trial. American journal of epidemiology , 172(1):107-115, 2010.

Issa J Dahabreh, Sarah E Robertson, Eric J Tchetgen, Elizabeth A Stuart, and Miguel A Hernán. Generalizing causal inferences from individuals in randomized trials to all trial-eligible individuals. Biometrics , 75(2):685-694, 2019.

Rasool Fakoor, Jonas W Mueller, Kavosh Asadi, Pratik Chaudhari, and Alexander J Smola. Continuous doubly constrained batch reinforcement learning. Advances in Neural Information Processing Systems , 34:11260-11273, 2021.

Roy Fox, Ari Pakman, and Naftali Tishby. Taming the noise in reinforcement learning via soft updates. arXiv preprint arXiv:1512.08562 , 2015.

Bolin Gao and Lacra Pavel. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint arXiv:1704.00805 , 2017.

Matthieu Geist, Bruno Scherrer, and Olivier Pietquin. A theory of regularized markov decision processes. In International Conference on Machine Learning , pages 2160-2169. PMLR, 2019.

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning , pages 1861-1870. PMLR, 2018.

Yichun Hu, Nathan Kallus, and Masatoshi Uehara. Fast rates for the regret of offline reinforcement learning. arXiv preprint arXiv:2102.00479 , 2021.

Yichun Hu, Nathan Kallus, and Xiaojie Mao. Fast rates for contextual linear optimization. Management Science , 68(6):4236-4245, 2022.

Audrey Huang and Nan Jiang. Beyond the return: Off-policy function estimation under user-specified error-measuring distributions. In Neurips , 2022.

Nan Jiang and Jiawei Huang. Minimax value interval for off-policy evaluation and policy optimization. Advances in Neural Information Processing Systems , 33:2747-2758, 2020.

Ying Jin, Zhuoran Yang, and Zhaoran Wang. Is pessimism provably efficient for offline rl? arXiv preprint arXiv:2012.15085 , 2020.

Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Modelbased offline reinforcement learning. In Advances in Neural Information Processing Systems , volume 33, pages 21810-21823. Curran Associates, Inc., 2020.

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Gene Li, Cong Ma, and Nathan Srebro. Pessimism for offline linear contextual bandits using lp confidence sets. arXiv preprint arXiv:2205.10671 , 2022.

Alex Luedtke and Antoine Chambaz. Performance guarantees for policy learning. In Annales de l'IHP Probabilites et statistiques , volume 56, page 2162. NIH Public Access, 2020.

Rémi Munos and Csaba Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research , 9(5), 2008.

Ofir Nachum and Bo Dai. Reinforcement learning via fenchel-rockafellar duality. arXiv preprint arXiv:2001.01866 , 2020.

Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of entropy-regularized markov decision processes. arXiv preprint arXiv:1705.07798 , 2017.

Thanh Nguyen-Tang, Ming Yin, Sunil Gupta, Svetha Venkatesh, and Raman Arora. On instancedependent bounds for offline reinforcement learning with linear function approximation. arXiv preprint arXiv:2211.13208 , 2022.

Asuman E Ozdaglar, Sarath Pattathil, Jiawei Zhang, and Kaiqing Zhang. Revisiting the linearprogramming framework for offline rl with general function approximation. In International Conference on Machine Learning , pages 26769-26791. PMLR, 2023.

Judea Pearl and Elias Bareinboim. External validity: From do-calculus to transportability across populations. In Statistical Science , pages 451-482. 2014.

Vianney Perchet and Philippe Rigollet. The multi-armed bandit problem with covariates. The Annals of Statistics , 41(2):693-721, 2013.

Paria Rashidinejad, Banghua Zhu, Cong Ma, Jiantao Jiao, and Stuart Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. arXiv preprint arXiv:2103.12021 , 2021.

Paria Rashidinejad, Hanlin Zhu, Kunhe Yang, Stuart Russell, and Jiantao Jiao. Optimal conservative offline rl with general function approximation via augmented lagrangian. arXiv preprint arXiv:2211.00716 , 2022.

John Schulman, Xi Chen, and Pieter Abbeel. Equivalence between policy gradients and soft q-learning. arXiv preprint arXiv:1704.06440 , 2017.

Chengchun Shi, Sheng Zhang, Wenbin Lu, and Rui Song. Statistical inference of the value function for reinforcement learning in infinite-horizon settings. Journal of the Royal Statistical Society Series B: Statistical Methodology , 84(3):765-793, 2022a.

Laixi Shi, Gen Li, Yuting Wei, Yuxin Chen, and Yuejie Chi. Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. arXiv preprint arXiv:2202.13890 , 2022b.

Max Simchowitz and Kevin G Jamieson. Non-asymptotic gap-dependent regret bounds for tabular mdps. Advances in Neural Information Processing Systems , 32, 2019.

Masatoshi Uehara and Wen Sun. Pessimistic model-based offline reinforcement learning under partial coverage. ICLR 2022 , 2021.

Masatoshi Uehara, Masahiro Kato, and Shota Yasui. Off-policy evaluation and learning for external validity under a covariate shift. Advances in Neural Information Processing Systems , 33:49-61, 2020.

Masatoshi Uehara, Masaaki Imaizumi, Nan Jiang, Nathan Kallus, Wen Sun, and Tengyang Xie. Finite sample analysis of minimax offline reinforcement learning: Completeness, fast rates and first-order efficiency. arXiv preprint arXiv:2102.02981 , 2021.

Xinqi Wang, Qiwen Cui, and Simon S Du. On gap-dependent bounds for offline reinforcement learning. Advances in Neural Information Processing Systems , 35:14865-14877, 2022.

Jingfeng Wu, Vladimir Braverman, and Lin Yang. Gap-dependent unsupervised exploration for reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 4109-4131. PMLR, 2022.

Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 , 2019.

Tengyang Xie and Nan Jiang. Q* approximation schemes for batch reinforcement learning: A theoretical comparison. In Conference on Uncertainty in Artificial Intelligence , pages 550-559. PMLR, 2020.

Tengyang Xie, Ching-An Cheng, Nan Jiang, Paul Mineiro, and Alekh Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems , 34:6683-6694, 2021.

Mengjiao Yang, Ofir Nachum, Bo Dai, Lihong Li, and Dale Schuurmans. Off-policy evaluation via the regularized lagrangian. Advances in Neural Information Processing Systems , 33:6551-6561, 2020.

Ming Yin and Yu-Xiang Wang. Towards instance-optimal offline reinforcement learning with pessimism. Advances in neural information processing systems , 34:4065-4078, 2021.

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In Advances in Neural Information Processing Systems , volume 33, pages 14129-14142, 2020.

Andrea Zanette and Martin J Wainwright. Bellman residual orthogonalization for offline reinforcement learning. arXiv preprint arXiv:2203.12786 , 2022.

Wenhao Zhan, Baihe Huang, Audrey Huang, Nan Jiang, and Jason Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory , pages 2730-2775. PMLR, 2022.

Xuezhou Zhang, Yiding Chen, Xiaojin Zhu, and Wen Sun. Corruption-robust offline reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 5757-5773. PMLR, 2022.

Hanlin Zhu, Paria Rashidinejad, and Jiantao Jiao. Importance weighted actor-critic for optimal conservative offline reinforcement learning. arXiv preprint arXiv:2301.12714 , 2023.

Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai , volume 8, pages 1433-1438. Chicago, IL, USA, 2008.

Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. In ICML , 2010.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents algorithms based on soft Q-learning for offline RL with finite-sample guarantees. The guarantees are given for general function approximation hold under weaker requirements such as a (partially) weaker variant of single-policy concentrability and avoids the Bellman-completeness assumption.\n\nSTRENGTHS:\n- The paper proposes algorithms based on soft Q-learning that only requires single-policy concentrability and realizability, without Bellman completeness. While a couple of recent papers such as Zhan et al. 2022 have also accomplished this, the guarantees presented in this work hold under a weaker single-policy coverage requirement and the MQP algorithm avoids the realizability of regularized optimal solutions, unlike the PRO-RL algorithm of Zhan et al. 2022.\n- The proposed algorithms appear to be implementable compared to prior works such as Chen and Jiang.\n\nWEAKNESSES:\n- The dual variables $l(s,a)$ here seems to be closely connected to importance weights $w(s,a)$. The Bellman consistency term in (4) seems to be similar to the importance weighted regularization considered in Zhu et al. 2023, in which there is a max_w. The main difference is using $\\Omega$ here. In that paper, the variable $w(s,a)$ are importance weights, which ensure that the average Bellman error is small under relevant distributions (including the data and optimal distributions). Would you please provide a comparison between the two objectives and comment on the connections between the variable $l(s,a)$ in your paper vs. $w(s,a)$ in their paper? Does the function $\\Omega$ contribute to avoiding the Bellman completeness assumption? What is the key factor in your proposed objective or analysis that removes the need for Bellman completeness compared to Zhu et al?\n- Convergence rates are slower than 1/sqrt{N} (or involve the gap), which are achieved by some of the recent works such as Zhu et al. 2023.\n\n\n**References**\n\nZhu, Hanlin, et al. \"Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning.\" arXiv preprint arXiv:2301.12714 (2023)\n\nQUESTIONS:\nPlease refer to the questions in the above segment.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-27"}

comments: SUMMARY:
This paper proposes a q-learning based algorithm for solving offline minimax problem in reinforcement learning. Under certain partial coverage and realizability assumption for the regularized problem, they prove $O(1/\epsilon^4)$ sample complexity.

STRENGTHS:
This is the first paper for minimax problem in offline RL that only requires realizability assumption and partial data coverage (without completeness and all-policy concentrability).

WEAKNESSES:
Weakness and comments:
1. The partial data coverage and realizability assumptions are made for the regularized problem instead of the original problem. It is unclear what the relationship  between these assumptions with the standard partial coverage and realizability assumptions for the original problem as in ``Revisiting the linear-programming framework for offline rl with general function approximation''.
2. The technique seems to  be similar to paper ``Refined value-based offline rl under realizability and partial coverage
''. The authors should explain more about the novelty.
3. References missing: ``Revisiting the linear-programming framework for offline rl with general function approximation'' also proposes an algorithm which requires $1/\epsilon^2$ sample complexity without completeness assumption for offline RL.

QUESTIONS:
None

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This paper proposed value-based algorithms for offline RL without Bellman completeness and full coverage of data support. By proposing the optimization objective of selecting the soft-Q from the set, where element in set satisfies the empirical soft-Bellman operator and the selected Q should minimize the square value under the expectation of behavior data distribution, and transforming it into a minimax problem, MSQP and MSQ could provide the property of convergence and sample complexity guarantee only with partial coverage and realizability.

STRENGTHS:
1. This paper propose novel offline algorithms without the assumption of full coverage and Bellman completeness.
2. The proposed algorithms are in a value-based manner, and avoid the assumption of initial distribution of ground truth MDP.
3. The proposed algorithms could have the property of convergence and sample complexity guarantee under more related assumptions.
4. This paper is well written and easy to follow.

WEAKNESSES:
1. There lack some intuitions for the proposed learning objective in Equation 2. and Equation 3., since the square value of Q is not an explicit constraint for offline learning problem.

QUESTIONS:
1. Could the author explain more about the intuitions of the proposed learning objective in Equation 2, though the convergence and sample complexity guarantee is provided?

LIMITATIONS:
It could be better if empirical results beyond tabular are provided.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission studies offline RL with function approximation. Under the relatively mild assumption of partial coverage and realizability of the function approximations, the paper proposes the algorithm M(S)QP that approximates the (soft-)Q-function by solving a minimax optimization problem, and establishes guarantees of $L^2$-convergence of the Q-function estimator and the sub-optimality gap of the learned policy.\n\nSTRENGTHS:\nA novelty of this work is that it studies offline RL based on soft-Q-function, while previous works mostly focus on the standard Q-function or the LP formulation. There are also several technical improvements compared to prior works, including $L^2$-convergence of the estimators and slightly milder technical assumptions.\n\nWEAKNESSES:\n(1) The proposed schemes are based on minimax optimization and hence are mainly of theoretical interest, and the dual function $l$ is hard to interpret. \n\n(2) The bounds on PAC sample complexity are either worse than the results in related literatures, or require an extra margin condition.\n\nQUESTIONS:\n(1) In Remark 1, the authors remark that the minimax optimization is computationally feasible when $\\mathcal{L}$ is chosen to be RKHS or linear function classes. I would like to know if there is a concrete class of offline RL problems where such a choice of $\\mathcal{L}$ satisfies realizability, or otherwise this remark is vacuous.\n\n(2) The bounds presented in this paper seem to be far from optimal (e.g. $O(n^{-1/4})$ convergence or $O(\\epsilon^{-8})$ rate), and they are also worse than the results in previous works (e.g. the case $\\alpha=0$ requires an extra assumption). I wonder if this is an artifact of the analysis or a price of milder assumptions.\n\n(3) Line 303: \"In the following, we demonstrate that MQP, which is a special version of MSQP when $\\alpha \\rightarrow 0$, can achieve a faster rate of $O\\left(1 / \\epsilon^2\\right)$.\"\n\nIn my opinion, this remark is slightly misleading, because the faster $O\\left(1 / \\epsilon^2\\right)$ rate is achieved under a stronger assumption of \"soft margin\". For example, in tabular offline RL with gap $\\Delta>0$ and single policy concentrability $C^*$, the minimax optimal sample complexity is of order $\\frac{SC^*}{\\Delta\\epsilon}$ (Wang et al., ignoring the horizon factor), which is better than the gap-independent sample complexity $\\frac{SC^*}{\\epsilon^2}$ as $\\epsilon\\to 0$. \n\nXinqi Wang, Qiwen Cui, and Simon S. Du. \"On Gap-dependent Bounds for Offline Reinforcement Learning.\"\n\n(4) Writing:\n\n(a) The notation $\\left\\|d\\_{\\pi\\_\\alpha^{\\star}, P\\_b} / P\\_b\\right\\|\\_{\\infty}$ is ambiguous, because the measure $d\\_{\\pi\\_\\alpha^{\\star}, P\\_b}$ is defined on both the space $\\mathcal{S}$ and the space $\\mathcal{S}\\times\\mathcal{A}$, and so does the measure $P\\_b$.\n\n(b) Line 187 & 188 (typos): $d\\_{\\pi\\_{\\alpha^*, P\\_b}}$ -> $d\\_{\\pi\\_\\alpha^*, P\\_b} $\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies offline reinforcement learning and proposes a soft Q-learning algorithm based on reformulating the solution to the Bellman equation as a saddle point of a certain minimax optimization problem. The authors then provide PAC guarantees for estimating the entropy-regularized Q-function, which in turn gives the sample complexity bounds. A key feature of this work is that it requires weaker assumptions (only partial coverage assumption is required) compared with existing literature.\n\nSTRENGTHS:\nTheoretical analysis of Q-learning with function approximation is one of the key topics in reinforcement learning (RL). This paper provides a sample complexity bound of soft Q-learning under only a partial coverage assumption, which might be of interest to the broad community of RL.\n\nWEAKNESSES:\nI have two major concerns about this work.\n\n(1) I am confused about the reformulation of solving the Bellman equation as a minimax optimization problem. It is well-known that both the vanilla Bellman operator and the entropy regularized Bellman operator are contraction mappings, which imply the uniqueness of both $q^*$ and $q_\\alpha^*$. In that case, it is unclear what Eq. (2) means since there is really no optimization problem here. Am I missing anything?\n\n(2) I read the proof of Theorem 5 and have some questions. In the second inequality in line 637, the event $q^*(s,a')-q^*(s,\\pi^*(s))\\leq 0$ happens surely because the optimal policy is greedy with respect to the optimal Q-function. In that case, why do we need to add this event to the indicator? In addition, since $q^*(s,a')-q^*(s,\\pi^*(s))\\leq 0$ happens surely, the first term on the right-hand side of the first inequality in line 638 is exactly equal to $1$. As a result, the bound seems to be meaningless and Assumption 7 is redundant. What am I missing here?\n\nQUESTIONS:\nSee the weakness section.\n\nLIMITATIONS:\nNo negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-28"}