Title: Offline Primal-Dual Reinforcement Learning for Linear MDPs
Venue: NeurIPS 2023
Accepted: no
Keywords: Reinforcement Learning

Offline Reinforcement Learning

Linear MDPs
Abstract: Offline Reinforcement Learning (RL) aims to learn a near-optimal
    policy from a fixed dataset of transitions collected by another policy.
    This problem has attracted a lot of attention recently, but most existing
    methods with strong theoretical guarantees are restricted to finite-horizon
    or tabular settings. In constrast, few algorithms for
    infinite-horizon settings with function approximation and minimal assumptions
    on the dataset are both sample and computationally efficient.
    Another gap in the current literature is the lack of theoretical analysis for
    the average-reward setting, which is more challenging than the discounted setting.
    In this paper, we address both of these issues by proposing a primal-dual
    optimization method based on the linear programming formulation of RL.
Our key contribution is a new reparametrization that allows us to derive low-variance gradient estimators that can be used in a stochastic optimization scheme using only samples from the behavior policy.
    Our method finds an $\varepsilon$-optimal policy with
     $O(\varepsilon^{-4})$ samples, improving on the previous $O(\varepsilon^{-5})$,
     while being computationally efficient for
    infinite-horizon discounted and average-reward MDPs with realizable linear
    function approximation and partial coverage. Moreover, to the best of our
    knowledge, this is the first theoretical result for average-reward offline RL.

Abstract

Offline Reinforcement Learning (RL) aims to learn a near-optimal policy from a fixed dataset of transitions collected by another policy. This problem has attracted a lot of attention recently, but most existing methods with strong theoretical guarantees are restricted to finite-horizon or tabular settings. In constrast, few algorithms for infinite-horizon settings with function approximation and minimal assumptions on the dataset are both sample and computationally efficient. Another gap in the current literature is the lack of theoretical analysis for the average-reward setting, which is more challenging than the discounted setting. In this paper, we address both of these issues by proposing a primal-dual optimization method based on the linear programming formulation of RL. Our key contribution is a new reparametrization that allows us to derive low-variance gradient estimators that can be used in a stochastic optimization scheme using only samples from the behavior policy. Our method finds an ε -optimal policy with O ( ε -4 ) samples, improving on the previous O ( ε -5 ) , while being computationally efficient for infinite-horizon discounted and average-reward MDPs with realizable linear function approximation and partial coverage. Moreover, to the best of our knowledge, this is the first theoretical result for average-reward offline RL.

1 Introduction

We study the setting of Offline Reinforcement Learning (RL), where the goal is to learn an ε -optimal policy without being able to interact with the environment, but only using a fixed dataset of transitions collected by a behavior policy . Learning from offline data proves to be useful especially when interacting with the environment can be costly or dangerous [16].

In this setting, the quality of the best policy learnable by any algorithm is constrained by the quality of the data, implying that finding an optimal policy without further assumptions on the data is not feasible. Therefore, many methods [23, 33] make a uniform coverage assumption, requiring that the behavior policy explores sufficiently well the whole state-action space. However, recent work [17, 31] demonstrated that partial coverage of the state-action space is sufficient. In particular, this means that the behavior policy needs only to sufficiently explore the state-actions visited by the optimal policy.

Moreover, like its online counterpart, modern offline RL faces the problem of learning efficiently in environments with very large state spaces, where function approximation is necessary to compactly represent policies and value functions. Although function approximation, especially with neural networks, is widely used in practice, its theoretical understanding in the context of decision-making is still rather limited, even when considering linear function approximation.

In fact, most existing sample complexity results for offline RL algorithms are limited either to the tabular and finite horizon setting, by the uniform coverage assumption, or by lack of computational

efficiency - see the top section of Table 1 for a summary. Notable exceptions are the recent works of

Xie et al. [36] and Cheng et al. [9] who provide computationally efficient methods for infinite-horizon 37 discounted MDPs under realizable linear function approximation and partial coverage. Despite 38 being some of the first implementable algorithms, their methods work only with discounted rewards, 39 have superlinear computational complexity and find an ε -optimal policy with O ( ε -5 ) samples - see 40 the bottom section of Table 1 for more details. Therefore, this work is motivated by the following 41 research question: 42

46

47

48

49

50

51

52

53

We answer this question positively by designing a method based on the linear-programming (LP) formulation of sequential decision making [20]. Albeit less known than the dynamic-programming formulation [3] that is ubiquitous in RL, it allows us to tackle this problem with the powerful tools of convex optimization. We turn in particular to a relaxed version of the LP formulation [21, 2] that considers action-value functions that are linear in known state-action features. This allows to reduce the dimensionality of the problem from the cardinality of the state space to the number of features. This relaxation still allows to recover optimal policies in linear MDPs [37, 13], a structural assumption that is widely employed in the theoretical study of RL with linear function approximation.

54

55

56

57

58

59

60

61

62

63

Our algorithm for learning near-optimal policies from offline data is based on primal-dual optimization of the Lagrangian of the relaxed LP. The use of saddle-point optimization in MDPs was first proposed by Wang & Chen [34] for planning in small state spaces, and was extended to linear function approximation by Chen et al. [8], Bas-Serrano & Neu [1], and Neu & Okolo [26]. We largely take inspiration from this latter work, which was the first to apply saddle-point optimization to the relaxed LP. However, primal-dual planning algorithms assume oracle access to a transition model, whose samples are used to estimate gradients. In our offline setting, we only assume access to i.i.d. samples generated by a possibly unknown behavior policy. To adapt the primal-dual optimization strategy to this setting we employ a change of variable, inspired by Nachum & Dai [24], which allows easy computation of unbiased gradient estimates.

2 Preliminaries 75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

We study discounted Markov decision processes [MDP, 29] denoted as ( X , A , p, r, γ ) , with discount factor γ ∈ [0 , 1] and finite, but potentially very large, state space X and action space A . For every state-action pair ( x, a ) , we denote as p ( · | x, a ) ∈ ∆ X the next-state distribution, and as r ( x, a ) ∈ [0 , 1] the reward, which is assumed to be deterministic and bounded for simplicity. The transition function p is also denoted as the matrix P ∈ R |X×A|×|X| and the reward as the vector r ∈ R |X×A| . The objective is to find an optimal policy π ∗ : X → ∆ A . That is, a stationary policy that maximizes the normalized expected return ρ ( π ∗ ) . = (1 -γ ) E π ∗ [ ∑ ∞ t =0 r ( X t , A t )] , where the initial state X 0 is sampled from the initial state distribution ν 0 , the other states according to X t +1 ∼ p ( ·| X t , A t ) and where the notation E π [ · ] is used to denote that the actions are sampled from policy π as A t ∼ π ( ·| X t ) . Moreover, we define the following quantities for each policy π : its state-action value function q π ( x, a ) . = E π [ ∑ ∞ t =0 γ t r ( X t , A t ) | X 0 = x, A 0 = a ] , its value function v π ( x ) . = E π [ q π ( x, A 0 )] , its state occupancy measure ν π ( x ) . = (1 -γ ) E π [ ∑ ∞ t =0 ✶ { X t = x } ] , and its state-action occupancy measure µ π ( x, a ) . = π ( a | x ) ν π ( x ) . These quantities are known to satisify the following useful relations, more commonly known respectively as Bellman's equation and flow constraint for policy π [4]:

Given this notation, we can also rewrite the normalized expected return in vector form as ρ ( π ) = (1 -γ ) 〈 ν 0 , v π 〉 or equivalently as ρ ( π ) = 〈 r , µ π 〉 .

Our work is based on the linear programming formulation due to Manne [19] (see also 29) which transforms the reinforcement learning problem into the search for an optimal state-action occupancy measure, obtained by solving the following Linear Program (LP):

where E ∈ R |X×A|×|X| denotes the matrix with components E ( x,a ) ,x ′ . = ✶ { x = x ′ } . The constraints of this LP are known to characterize the set of valid state-action occupancy measures. Therefore, an optimal solution µ ∗ of the LP corresponds to the state-action occupancy measure associated to a policy π ∗ maximizing the expected return, and which is therefore optimal in the MDP. This policy can be extracted as π ∗ ( a | x ) . = µ ∗ ( x, a ) / ∑ ¯ a ∈A µ ∗ ( x, ¯ a ) . However, this linear program cannot be directly solved in an efficient way in large MDPs due to the number of constraints and dimensions of the variables scaling with the size of the state space X . Therefore, taking inspiration from the previous works of Bas-Serrano et al. [2], Neu & Okolo [26] we assume the knowledge of a feature map ϕ , which we then use to reduce the dimension of the problem. More specifically we consider the setting of Linear MDPs [13, 37].

Definition 2.1 (Linear MDP) . An MDP is called linear if both the transition and reward functions can be expressed as a linear function of a given feature map ϕ : X × A → R d . That is, there exist ψ : X → R d and ω ∈ R d such that, for every x, x ′ ∈ X and a ∈ A :

We assume that for all x, a , the norms of all relevant vectors are bounded by known constants as ‖ ϕ ( x, a ) ‖ 2 ≤ D ϕ , ‖ ∑ x ′ ψ ( x ′ ) ‖ 2 ≤ D ψ , and ‖ ω ‖ 2 ≤ D ω . Moreover, we represent the feature map with the matrix Φ ∈ R |X×A|× d with rows given by ϕ ( x, a ) T , and similarly we define Ψ ∈ R d ×|X| as the matrix with columns given by ψ ( x ) .

With this notation we can rewrite the transition matrix as P = ΦΨ . Furthermore, it is convenient to assume that the dimension d of the feature map cannot be trivially reduced, and therefore that the matrix Φ is full-rank. An easily verifiable consequence of the Linear MDP assumption is that state-action value functions can be represented as a linear combinations of ϕ . That is, there exist θ π ∈ R d such that:

It can be shown that for all policies π , the norm of θ π is at most D θ = D ω + D ψ 1 -γ (cf. Lemma B.1 in 13). We then translate the linear program (2) to our setting, with the addition of the new variable λ ∈ R d , resulting in the following new LP and its corresponding dual:

It can be immediately noticed how the introduction of λ did not change neither the set of admissible 121 µ s nor the objective, and therefore did not alter the optimal solution. The Lagrangian associated to 122 this set of linear programs is the function: 123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

It is known that finding optimal solutions ( λ glyph[star] , µ glyph[star] ) and ( v glyph[star] , θ glyph[star] ) for the primal and dual LPs is equivalent to finding a saddle point ( v glyph[star] , θ glyph[star] , λ glyph[star] , µ glyph[star] ) of the Lagrangian function [5]. In the next section, we will develop primal-dual methods that aim to find approximate solutions to the above saddle-point problem, and convert these solutions to policies with near-optimality guarantees.

3 Algorithm and Main Results

This section introduces the concrete setting we study in this paper, and presents our main contributions.

We consider the offline-learning scenario where the agent has access to a dataset D = ( W t ) n t =1 , collected by a behavior policy π B , and composed of n random observations of the form W t = ( X 0 t , X t , A t , R t , X ′ t ) . The random variables X 0 t , ( X t , A t ) and X ′ t are sampled, respectively, from the initial-state distribution ν 0 , the discounted occupancy measure of the behavior policy, denoted as µ B , and from p ( ·| X t , A t ) . Finally, R t denotes the reward r ( X t , A t ) . We assume that all observations W t are generated independently of each other, and will often use the notation ϕ t = ϕ ( X t , A t ) .

Our strategy consists in finding approximately good solutions for the LPs (4) and (5) using stochastic optimization methods, which require access to unbiased gradient estimates of the Lagrangian (Equation 6). The main challenge we need to overcome is constructing suitable estimators based only on observations drawn from the behavior policy. We address this challenge by introducing the matrix Λ = E X,A ∼ µ B [ ϕ ( X,A ) ϕ ( X,A ) T ] (supposed to be invertible for the sake of argument for now), and rewriting the gradient with respect to λ as

This suggests that the vector within the expectation can be used to build an unbiased estimator of the desired gradient. A downside of using this estimator is that it requires knowledge of Λ . However, this can be sidestepped by a reparametrization trick inspired by Nachum & Dai [24]: introducing the parametrization β = Λ -1 λ , the objective can be rewritten as

This can be indeed seen to generalize the tabular reparametrization of Nachum & Dai [24] to the case of linear function approximation. Notably, our linear reparametrization does not change the structure of the saddle-point problem, but allows building an unbiased estimator of ∇ β L ( β , µ ; v , θ ) without knowledge of Λ as

In what follows, we will use the more general parametrization β = Λ -c λ , with c ∈ { 1 / 2 , 1 } , and construct a primal-dual stochastic optimization method that can be implemented efficiently in the offline setting based on the observations above. Using c = 1 allows to run our algorithm without knowledge of Λ , that is, without knowing the behavior policy that generated the dataset, while using c = 1 / 2 results in a tighter bound, at the price of having to assume knowledge of Λ .

Our algorithm (presented as Algorithm 1) is inspired by the method of Neu & Okolo [26], originally designed for planning with a generative model. The algorithm has a double-loop structure, where

157

158

159

160

161

162

163

164

165

167

168

169

170

171

172

173

174

175

176

177

178

179

180

Algorithm 1 Offline Primal-Dual RL

Input: Learning rates α, ζ, η , initial points θ 0 ∈ B ( D θ ) , β 1 ∈ B ( D β ) , π 1 , and data D = ( W t ) n t =1 for t = 1 to T do

Initialize

θ

t,

1

=

θ

for to

k

= 1

K

-

Obtain sample

t

-

1

1

do

W

µ

t,k

t

=

π

◦

[

(1

˜

g

θ

,t,i

=

Φ

T

µ

t,k

-

t,k

= (

γ

-

θ

t,k

+1

= Π

B

(

D

θ

X

)

0

t,k

X

e

c

Λ

)

at each iteration t we run one step of stochastic gradient ascent for β , and also an inner loop which runs K iterations of stochastic gradient descent on θ making sure that 〈 ϕ ( x, a ) , θ t 〉 is a good approximation of the true action-value function of π t . Iterations of the inner loop are indexed by k . The main idea of the algorithm is to compute the unbiased estimators ˜ g θ ,t,k and ˜ g β ,t of the gradients ∇ θ L ( β t , µ t ; · , θ t,k ) and ∇ β L ( β t , · ; v t , θ t ) , and use them to update the respective variables iteratively. We then define a softmax policy π t at each iteration t using the θ parameters as π t ( a | x ) = σ ( α ∑ t -1 i =1 〈 ϕ ( x, a ) , θ i 〉 ) . The other higher-dimensional variables ( µ t , v t ) are defined symbolically in terms of β t , θ t and π t , and used only as auxiliary variables for computing the estimates ˜ g θ ,t,k and ˜ g β ,t . Specifically, we set these variables as

Finally, the gradient estimates can be defined as 166

These gradient estimates are then used in a projected gradient ascent/descent scheme, with the glyph[lscript] 2 projection operator denoted by Π . The feasible sets of the two parameter vectors are chosen as glyph[lscript] 2 balls of radii D θ and D β , denoted respectively as B ( D θ ) and B ( D β ) . Notably, the algorithm does not need to compute v t ( x ) , µ t,k ( x, a ) , or π t ( a | x ) for all states x , but only for the states that are accessed during the execution of the method. In particular, π t does not need to be computed explicitly, and it can be efficiently represented by the single d -dimensional parameter vector ∑ t i =1 θ i .

Due to the double-loop structure, each iteration t uses K samples from the dataset D , adding up to a total of n = KT samples over the course of T iterations. Each gradient update calculated by the method uses a constant number of elementary vector operations, resulting in a total computational complexity of O ( |A| dn ) elementary operations. At the end, our algorithm outputs a policy selected uniformly at random from the T iterations.

3.1 Main result

We are now almost ready to state our main result. Before doing so, we first need to discuss the quantities appearing in the guarantee, and provide an intuitive explanation for them.

(

θ

t,k

0

t,k

, X

+

ϕ

1

-

-

t,k

γ

(

, A

〈

ϕ

X

η

t,k

X

(

t,k

˜

θ

, X

t,k

, A

,t,i

)

′

t,k

, A

t,k

)

〈

)

t,k

)

,

Λ

t

β

〉

′

t,k

X

e

ϕ

t,k t,k

t

(

X

, A

)

,

β

〉

// Stochastic gradient descent

g

c

-

1

]

Similarly to previous work, we capture the partial coverage assumption by expressing the rate of 181 convergence to the optimal policy in terms of a coverage ratio that measures the mismatch between 182 the behavior and the optimal policy. Several definitions of coverage ratio are surveyed by Uehara & 183 Sun [32]. In this work, we employ a notion of feature coverage ratio for linear MDPs that defines 184 coverage in feature space rather than in state-action space, similarly to Jin et al. [14], but with a 185 smaller ratio. 186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

Definition 3.1. Let c ∈ { 1 / 2 , 1 } . We define the generalized coverage ratio as

We defer a detailed discussion of this ratio to Section 6, where we compare it with similar notions in the literature. We are now ready to state our main result.

Theorem 3.2. Given a linear MDP (Definition 2.1) such that θ π ∈ B ( D θ ) for any policy π . Assume that the coverage ratio is bounded C ϕ,c ( π ∗ ; π B ) ≤ D β . Then, for any comparator policy π ∗ , the policy output by an appropriately tuned instance of Algorithm 1 satisfies E [ 〈 µ π ∗ -µ π out , r 〉 ] ≤ ε with a number of samples n glyph[epsilon1] that is O ( ε -4 D 4 θ D 8 c ϕ D 4 β d 2 -2 c log |A| ) .

The concrete parameter choices are detailed in the full version of the theorem in Appendix A. The main theorem can be simplified by making some standard assumptions, formalized by the following corollary.

Corollary 3.3. Assume that the bound of the feature vectors D ϕ is of order O (1) , that D ω = D ψ = √ d and that D β = c · C ϕ,c ( π ∗ ; π B ) for some positive universal constant c . Then, under the same assumptions of Theorem 3.2, n ε is of order O ( d 4 C ϕ,c ( π ∗ ; π B ) 2 log |A| d 2 c (1 -γ ) 4 ε 4 ) .

4 Analysis

This section explains the rationale behind some of the technical choices of our algorithm, and sketches the proof of our main result.

First, we explicitly rewrite the expression of the Lagrangian (6), after performing the change of variable λ = Λ c β :

We aim to find an approximate saddle-point of the above convex-concave objective function. One 205 challenge that we need to face is that the variables v and µ have dimension proportional to the size of 206 the state space |X| , so making explicit updates to these parameters would be prohibitively expensive 207 in MDPs with large state spaces. To address this challenge, we choose to parametrize µ in terms of a 208 policy π and β through the symbolic assignment µ = µ β ,π , where 209

This choice can be seen to satisfy the first constraint of the primal LP (4), and thus the gradient of the Lagrangian (12) evaluated at µ β ,π with respect to v can be verified to be 0 . This parametrization makes it possible to express the Lagrangian as a function of only θ , β and π as

210

211

212

For convenience, we also define the quantities ν β = E T µ β ,π and v θ ,π ( s ) . = ∑ a π ( a | s ) 〈 θ , ϕ ( x, a ) 〉 , 213 which enables us to rewrite f as 214

The above choices allow us to perform stochastic gradient / ascent over the low-dimensional parame215 ters θ and β and the policy π . In order to calculate an unbiased estimator of the gradients, we first 216

observe that the choice of µ t,k in Algorithm 1 is an unbiased estimator of µ β t ,π t : 217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

where we used the fact that p ( x | ¯ x, ¯ a ) = 〈 ψ ( x ) , ϕ (¯ x, ¯ a ) 〉 , and the definition of Λ . This in turn facilitates proving that the gradient estimate ˜ g θ ,t,k , defined in Equation 10, is indeed unbiased:

E t,k [˜ g θ ,t,k ] = Φ T E t,k [ µ t,k ] -Λ c -1 E t,k [ ϕ t,k ϕ T t,k ] β t = Φ T µ β t ,π t -Λ c β t = ∇ θ L ( β t , µ t ; v t , · ) . A similar proof is used for ˜ g β ,t and is detailed in Appendix B.3.

Our analysis is based on arguments by Neu & Okolo [26], carefully adapted to the reparametrized version of the Lagrangian presented above. The proof studies the following central quantity that we refer to as dynamic duality gap :

Here, ( θ t , β t , π t ) are the iterates of the algorithm, θ ∗ 1: T = ( θ ∗ t ) T t =1 a sequence of comparators for θ , and finally β ∗ and π ∗ are fixed comparators for β and π , respectively. Our first key lemma relates the suboptimality of the output policy to G T for a specific choice of comparators.

Lemma 4.1. Let θ ∗ t . = θ π t , π ∗ be any policy, and β ∗ = Λ -c Φ glyph[latticetop] µ π ∗ . Then, E [ 〈 µ π ∗ -µ π out , r 〉 ] = G T ( β ∗ , π ∗ ; θ ∗ 1: T ) .

The proof is relegated to Appendix B.1. Our second key lemma rewrites the gap G T for any choice of comparators as the sum of three regret terms:

Lemma 4.2. With the choice of comparators of Lemma 4.1

where g θ ,t = Φ glyph[latticetop] µ β t ,π t -Λ c β t and g β ,t = Λ c ( ω + γ Ψ v θ t ,π t -θ t ) .

The proof is presented in Appendix B.2. To conclude the proof we bound the three terms appearing in Lemma 4.2. The first two of those are bounded using standard gradient descent/ascent analysis (Lemmas B.1 and B.2), while for the latter we use mirror descent analysis (Lemma B.3). The details of these steps are reported in Appendix B.3.

5 Extension to Average-Reward MDPs

In this section, we briefly explain how to extend our approach to offline learning in average reward MDPs , establishing the first sample complexity result for this setting. After introducing the setup, we outline a remarkably simple adaptation of our algorithm along with its performance guarantees for this setting. The reader is referred to Appendix C for the full details, and to Chapter 8 of Puterman [29] for a more thorough discussion of average-reward MDPs.

In the average reward setting we aim to optimize the objective ρ π ( x ) = lim inf T →∞ 1 T E π [∑ T t =1 r ( x t , a t ) ∣ ∣ x 1 = x ] , representing the long-term average reward of policy π when started from state x ∈ X . Unlike the discounted setting, the average reward criterion prioritizes long-term frequency over proximity of good rewards due to the absence of discounting which expresses a preference for earlier rewards. As is standard in the related literature, we will assume that ρ π is well-defined for any policy and is independent of the start state, and thus will

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

use the same notation to represent the scalar average reward of policy π . Due to the boundedness of the rewards, we clearly have ρ π ∈ [0 , 1] . Similarly to the discounted setting, it is possible to define quantities analogous to the value and action value functions as the solutions to the Bellman equations q π = r -ρ π 1 + Pv π , where v π is related to the action-value function as v π ( x ) = ∑ a π ( a | x ) q π ( x, a ) . We will make the following standard assumption about the MDP (see, e.g., Section 17.4 of Meyn & Tweedie [22]):

Assumption 5.1. For all stationary policies π , the Bellman equations have a solution q π satisfying sup x,a q π ( x, a ) -inf x,a q π ( x, a ) < D q .

Furthermore, we will continue to work with the linear MDP assumption of Definition 2.1, and will additionally make the following minor assumption:

Assumption 5.2. The all ones vector 1 is contained in the column span of the feature matrix Φ . Furthermore, let glyph[rho1] ∈ R d such that for all ( x, a ) ∈ Z , 〈 ϕ ( x, a ) , glyph[rho1] 〉 = 1 .

Using these insights, it is straightforward to derive a linear program akin to (2) that characterize the optimal occupancy measure and thus an optimal policy in average-reward MDPs. Starting from this formulation and proceeding as in Sections 2 and 4, we equivalently restate this optimization problem as finding the saddle-point of the reparametrized Lagrangian defined as follows:

As previously, the saddle point can be shown to be equivalent to an optimal occupancy measure under the assumption that the MDP is linear in the sense of Definition 2.1. Notice that the above Lagrangian slightly differs from that of the discounted setting in Equation (11) due to the additional optimization parameter ρ , but otherwise our main algorithm can be directly generalized to this objective. We present details of the derivations and the resulting algorithm in Appendix C. The following theorem states the performance guarantees for this method.

Theorem 5.3. Given a linear MDP (Definition 2.1) satisfying Assumption 5.2 and such that θ π ∈ B ( D θ ) for any policy π . Assume that the coverage ratio is bounded C ϕ,c ( π ∗ ; π B ) ≤ D β . Then, for any comparator policy π ∗ , the policy output by an appropriately tuned instance of Algorithm 2 satisfies E [ 〈 µ π ∗ -µ π out , r 〉 ] ≤ ε with a number of samples n glyph[epsilon1] that is O ( ε -4 D 4 θ D 12 c -2 ϕ D 4 β d 2 -2 c log |A| ) .

As compared to the discounted case, this additional dependence of the sample complexity on D ϕ is due to the extra optimization variable ρ . We provide the full proof of this theorem along with further discussion in Appendix C.

6 Discussion and Final Remarks

In this section, we compare our results with the most relevant ones from the literature. Our Table 1 can be used as a reference. As a complement to this section, we refer the interested reader to the recent work by Uehara & Sun [32], which provides a survey of offline RL methods with their coverage and structural assumptions. Detailed computations can be found in Appendix E.

An important property of our method is that it only requires partial coverage. This sets it apart from classic batch RL methods like FQI [11, 23], which require a stronger uniform-coverage assumption. Algorithms working under partial coverage are mostly based on the principle of pessimism. However, our algorithm does not implement any form of explicit pessimism. We recall that, as shown by Xiao et al. [35], pessimism is just one of many ways to achieve minimax-optimal sample efficiency.

Let us now compare our notion of coverage ratio to the existing notions previsouly used in the literature. Jin et al. [14] (Theorem 4.4) rely on a feature coverage ratio which can be written as

By Jensen's inequality, our C ϕ, 1 / 2 (Definition 3.1) is never larger than C glyph[diamondmath] . Indeed, notice how the random features in Equation (17) are coupled, introducing an extra variance term w.r.t. C ϕ, 1 / 2 . Specifically, we can show that C ϕ, 1 / 2 ( π ∗ ; π B ) = C glyph[diamondmath] ( π ∗ ; π B ) -V X,A ∼ µ ∗ [ Λ -1 / 2 ϕ ( X,A ) ] , where V [ Z ] = E [ ‖ Z -E [ Z ] ‖ 2 ] for a random vector Z . So, besides fine comparisons with existing notions of coverage ratios, we can regard C ϕ, 1 / 2 as a low-variance version of the standard feature coverage ratio. However, our sample complexity bounds do not fully take advantage of this low-variance

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

property, since they scale quadratically with the ratio itself, rather than linearly, as is more common in previous work.

To scale with C ϕ, 1 / 2 , our algorithm requires knowledge of Λ , hence of the behavior policy. However, so does the algorithm from Jin et al. [14]. Zanette et al. [38] remove this requirement at the price of a computationally heavier algorithm. However, both are limited to the finite-horizon setting.

Uehara & Sun [32] and Zhang et al. [39] use a coverage ratio that is conceptually similar to Equation (17),

Some linear algebra shows that C † ≤ C glyph[diamondmath] ≤ dC † . Therefore, chaining the previous inequalities we know that C ϕ, 1 / 2 ≤ C glyph[diamondmath] ≤ dC † . It should be noted that the algorithm from Uehara & Sun [32] also works in the representation-learning setting, that is, with unknown features. However, it is far from being efficiently implementable. The algorithm from Zhang et al. [39] instead is limited to the finite-horizon setting.

In the special case of tabular MDPs, it is hard to compare our ratio with existing ones, because in this setting, error bounds are commonly stated in terms of sup x,a µ ∗ ( x,a ) / µ B ( x,a ) , often introducing an explicit dependency on the number of states [e.g., 17], which is something we carefully avoided. However, looking at how the coverage ratio specializes to the tabular setting can still provide some insight. With known behavior policy, C ϕ, 1 / 2 ( π ∗ ; π B ) = ∑ x,a µ ∗ ( x,a ) 2 / µ B ( x,a ) is smaller than the more standard C glyph[diamondmath] ( π ∗ ; π B ) = ∑ x,a µ ∗ ( x,a ) / µ B ( x,a ) . With unknown behavior, C ϕ, 1 ( π ∗ ; π B ) = ∑ x,a ( µ ∗ ( x,a ) / µ B ( x,a ) ) 2 is non-comparable with C glyph[diamondmath] in general, but larger than C ϕ, 1 / 2 . Interestingly, C ϕ, 1 ( π ∗ ; π B ) is also equal to 1+ X 2 ( µ ∗ ‖ µ B ) , where X 2 denotes the chi-square divergence, a crucial quantity in off-distribution learning based on importance sampling [10]. Moreover, a similar quantity to C ϕ, 1 was used by Lykouris et al. [18] in the context of (online) RL with adversarial corruptions.

We now turn to the works of Xie et al. [36] and Cheng et al. [9], which are the only practical methods to consider function approximation in the infinite horizon setting, with minimal assumption on the dataset, and thus the only directly comparable to our work. They both use the coverage ratio C F ( π ∗ ; π B ) = max f ∈F ‖ f -T f ‖ 2 µ ∗ / ‖ f -T f ‖ 2 µ B , where F is a function class and T is Bellman's operator. This can be shown to reduce to Equation (18) for linear MDPs. However, the specialized bound of Xie et al. [36] (Theorem 3.2) scales with the potentially larger ratio from Equation (17). Both their algorithms have superlinear computational complexity and a sample complexity of O ( ε -5 ) . Hence, in the linear MDP setting, our algorithm is a strict improvement both for its O ( ε -4 ) sample complexity and its O ( n ) computational complexity. However, It is very important to notice that no practical algorithm for this setting so far, including ours, can match the minimax optimal sample complexity rate of O ( ε 2 ) [35, 31]. This leaves space for future work in this area. In particular, by inspecting our proofs, it should be clear the the extra O ( ε -2 ) factor is due to the nested-loop structure of the algorithm. Therefore, we find it likely that our result can be improved using optimistic descent methods [6] or a two-timescale approach [15, 30].

As a final remark, we remind that when Λ is unknown, our error bounds scales with C ϕ, 1 , instead of the smaller C ϕ, 1 / 2 . However, we find it plausible that one can replace the Λ with an estimate that is built using some fraction of the overall sample budget. In particular, in the tabular case, we could simply use all data to estimate the visitation probabilities of each-state action pairs and use them to build an estimator of Λ . Details of a similar approach have been worked out by Gabbianelli et al. [12]. Nonetheless, we designed our algorithm to be flexible and work in both cases.

To summarize, our method is one of the few not to assume the state space to be finite, or the dataset to have global coverage, while also being computationally feasible. Moreover, it offers a significant advantage, both in terms of sample and computational complexity, over the two existing polynomialtime algorithms for discounted linear MDPs with partial coverage [36, 9]; it extends to the challenging average-reward setting with minor modifications; and has error bounds that scale with a low-variance version of the typical coverage ratio. These results were made possible by employing algorithmic principles, based on the linear programming formulation of sequential decision making, that are new in offline RL. Finally, the main direction for future work is to develop a single-loop algorithm to achieve the optimal rate of ε -2 , which should also improve the dependence on the coverage ratio from C ϕ,c ( π ∗ ; π B ) 2 to C ϕ,c ( π ∗ ; π B ) .

348

349

350

351

352

353

354

355

References

Bas-Serrano, J. and Neu, G. Faster saddle-point optimization for solving large-scale markov decision processes. In L4DC , volume 120 of Proceedings of Machine Learning Research , pp. 413-423. PMLR, 2020.

Bas-Serrano, J., Curi, S., Krause, A., and Neu, G. Logistic q-learning. In Banerjee, A. and Fukumizu, K. (eds.), Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , volume 130 of Proceedings of Machine Learning Research , pp. 3610-3618. PMLR, 13-15 Apr 2021. URL https://proceedings.mlr.press/v130/bas-serrano21a.html .

Bellman, R. Dynamic programming. Technical report, RAND CORP SANTA MONICA CA, 356 1956. 357

Bellman, R. Dynamic programming. Science , 153(3731):34-37, 1966. 358

Bertsekas, D. P. Constrained Optimization and Lagrange Multiplier Methods . Academic Press, 359 1982. ISBN 978-0-12-093480-5. 360

Borkar, V. S. Stochastic approximation with two time scales. Systems & Control Letters , 29(5): 361 291-294, 1997. 362

Cesa-Bianchi, N. and Lugosi, G. Prediction, Learning, and Games . Cambridge University 363 Press, New York, NY, USA, 2006. 364

Chen, Y., Li, L., and Wang, M. Scalable bilinear learning using state and action features. In 365 ICML , volume 80 of Proceedings of Machine Learning Research , pp. 833-842. PMLR, 2018. 366

367

368

369

370

371

Cheng, C.-A., Xie, T., Jiang, N., and Agarwal, A. Adversarially trained actor critic for offline reinforcement learning. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.), Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pp. 3852-3878. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/cheng22b.html .

Cortes, C., Mansour, Y., and Mohri, M. Learning bounds for importance weighting. In NeurIPS , 372 pp. 442-450. Curran Associates, Inc., 2010. 373

Ernst, D., Geurts, P., and Wehenkel, L. Tree-based batch mode reinforcement learning. J. Mach. 374 Learn. Res. , 6:503-556, 2005. 375

376

377

378

379

Gabbianelli, G., Neu, G., and Papini, M. Online learning with off-policy feedback. In Agrawal, S. and Orabona, F. (eds.), ALT , volume 201 of Proceedings of Machine Learning Research , pp. 620-641. PMLR, 20 Feb-23 Feb 2023. URL https://proceedings.mlr.press/v201/ gabbianelli23a.html .

Jin, C., Yang, Z., Wang, Z., and Jordan, M. I. Provably efficient reinforcement learning with 380 linear function approximation. In COLT , volume 125 of Proceedings of Machine Learning 381 Research , pp. 2137-2143. PMLR, 2020. 382

Jin, Y., Yang, Z., and Wang, Z. Is pessimism provably efficient for offline rl? In International 383 Conference on Machine Learning , pp. 5084-5096. PMLR, 2021. 384

Korpelevich, G. The extragradient method for finding saddle points and other problems. 385 Matecon , 12:747-756, 1976. 386

Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline reinforcement learning: Tutorial, review, 387 and perspectives on open problems. 2020. 388

Liu, Y., Swaminathan, A., Agarwal, A., and Brunskill, E. Provably good batch off-policy 389 reinforcement learning without great exploration. In NeurIPS , 2020. 390

Lykouris, T., Simchowitz, M., Slivkins, A., and Sun, W. Corruption-robust exploration in 391 episodic reinforcement learning. In COLT , volume 134 of Proceedings of Machine Learning 392 Research , pp. 3242-3245. PMLR, 2021. 393

Manne, A. S. Linear programming and sequential decisions. Manage. Sci. , 6(3):259-267, 394 apr 1960. ISSN 0025-1909. doi: 10.1287/mnsc.6.3.259. URL https://doi.org/10.1287/ 395 mnsc.6.3.259 . 396

Manne, A. S. Linear programming and sequential decisions. Management Science , 6(3): 397 259-267, 1960. 398

Mehta, P. G. and Meyn, S. P. Q-learning and pontryagin's minimum principle. In CDC , pp. 399 3598-3605. IEEE, 2009. 400

Meyn, S. and Tweedie, R. Markov Chains and Stochastic Stability . Springer-Verlag, 1996. 401

Munos, R. and Szepesvári, C. Finite-time bounds for fitted value iteration. J. Mach. Learn. 402 Res. , 9:815-857, 2008. 403

Nachum, O. and Dai, B. Reinforcement learning via fenchel-rockafellar duality. 2020. 404

Nemirovski, A. and Yudin, D. Problem Complexity and Method Efficiency in Optimization . 405 Wiley Interscience, 1983. 406

Neu, G. and Okolo, N. Efficient global planning in large mdps via stochastic primal-dual 407 optimization. In ALT , volume 201 of Proceedings of Machine Learning Research , pp. 1101408 1123. PMLR, 2023. 409

410

411

Neu, G., Jonsson, A., and Gómez, V. A unified view of entropy-regularized Markov decision processes. arXiv preprint arXiv:1705.07798 , 2017.

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

Orabona, F. A modern introduction to online learning. arXiv preprint arXiv:1912.13213 , 2019.

Puterman, M. L. Markov Decision Processes: Discrete Stochastic Dynamic Programming . John Wiley & Sons, Inc., USA, 1994. ISBN 0471619779.

Rakhlin, A. and Sridharan, K. Optimization, learning, and games with predictable sequences. In Advances in Neural Information Processing Systems , pp. 3066-3074, 2013.

Rashidinejad, P., Zhu, B., Ma, C., Jiao, J., and Russell, S. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. IEEE Trans. Inf. Theory , 68(12):8156-8196, 2022.

Uehara, M. and Sun, W. Pessimistic model-based offline reinforcement learning under partial coverage. In ICLR . OpenReview.net, 2022.

Uehara, M., Huang, J., and Jiang, N. Minimax weight and q-function learning for off-policy evaluation. In III, H. D. and Singh, A. (eds.), Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pp. 96599668. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/uehara20a. html .

Wang, M. and Chen, Y. An online primal-dual method for discounted markov decision processes. In CDC , pp. 4516-4521. IEEE, 2016.

Xiao, C., Wu, Y., Mei, J., Dai, B., Lattimore, T., Li, L., Szepesvári, C., and Schuurmans, D. On the optimality of batch policy optimization algorithms. In ICML , volume 139 of Proceedings of Machine Learning Research , pp. 11362-11371. PMLR, 2021.

Xie, T., Cheng, C.-A., Jiang, N., Mineiro, P., and Agarwal, A. Bellman-consistent pessimism for offline reinforcement learning. In Ranzato, M., Beygelzimer, A., Dauphin, Y., Liang, P., and Vaughan, J. W. (eds.), Advances in Neural Information Processing Systems , volume 34, pp. 6683-6694. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/ paper_files/paper/2021/file/34f98c7c5d7063181da890ea8d25265a-Paper.pdf .

Yang, L. and Wang, M. Sample-optimal parametric q-learning using linearly additive features. In ICML , volume 97 of Proceedings of Machine Learning Research , pp. 6995-7004. PMLR, 2019.

Zanette, A., Wainwright, M. J., and Brunskill, E. Provable benefits of actor-critic methods for offline reinforcement learning. In NeurIPS , pp. 13626-13640, 2021.

Zhang, X., Chen, Y., Zhu, X., and Sun, W. Corruption-robust offline reinforcement learning. In AISTATS , volume 151 of Proceedings of Machine Learning Research , pp. 5757-5773. PMLR, 2022.

Zinkevich, M. Online convex programming and generalized infinitesimal gradient ascent. In 444 Proceedings of the Twentieth International Conference on Machine Learning (ICML) , 2003. 445

Peer reviews:
comments: SUMMARY:
This paper studied offline RL in linear MDP setting, where the transition and reward have low-rank structures and the feature $\phi$ is known. The authors formulated the problem in a primal-dual way and proposed a gradient-based algorithm. They provided convergence guarantees, which only requires coverage over optimal policy.

STRENGTHS:
The paper writing is clear and easy to follow.

The discussion and comparison with previous works is very detailed.

The algorithm is computationally efficient. The algorithm design has some interesting points, especially the reparameterization design to avoid knowlegde of $\Lambda^{-1}$ and updates for variables $v$ and $u$.

The coverage assumption seems weaker than previous literatures.

WEAKNESSES:
I didn't see too much technical novelty in the method and proof.

The setting is linear MDP, which is kind of restrictive.

Convergance rate is kind of far away from optimal.

QUESTIONS:
Given the knowledge of $\Lambda$, why not consider the reparameterization $\beta = \lambda$, i.e., set $c=0$, but consider $c=1/2$? It seems to me the RHS of Eq. in Line 141 can be estimated unbiasedly?

Line 154 claimed that $c = 1/2$ results in a tighter bound, however, according to Theorem 3.2, and Corollary 3.3, the larger $c$ is, the lower dependence on $d$ the sample complexity will have. So how the "tighter bound" is reflected? It seems contradictory.

LIMITATIONS:
N.A.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed an primal-dual framework for offline reinforcement learning in linear MDP Contrary to the more common case of finite horizon, they considered the case of infinite horizon with discounted reward. They reduced the problem of offline reinforcement learning to a problem about solving the saddle-point of a Lagrange form. They designed an algorithm which uses stochastic gradient-based optimization to solve the saddle point. They provide a sample complexity of O(\\eps^-4) for both cases of discounted MDP and averaged-reward MDP, and their algorithm is also computational efficient. \n\nTo summarize, the formulation of offline RL into a linear programming problem is very interesting. The proof seems very solid, and I like the comparison for the concentrability constant in the last discussion section. The comparison for the constant C is thorough and very good. \n\nHowever, I still have some questions about some details in the main text.\n\nSTRENGTHS:\n1. The formulation of offline reinforcement learning to a linear programming problem is very good.\n2. The algorithm is clearly motivated by solving the saddle points of a Lagrange form. The algorithm itself is simple and computationally efficient, with a guaranteed sample complexity for both discounted MDP and averaged-reward MDP.\n3. They proposed a new concentrability constant C and compare it to other constants appearing in other literatures about offline RL. I think the understanding of the relationship of these concentrability constant is basically correct and very clearly expressed.\n4. The proof seems very solid and the result in averaged-reward case is new.\n\nWEAKNESSES:\n1. I have some question about your comparison to previous results. Your main references are Cheng et al and Xie et al. \n\n1.1 For Xie et al, the Theorem 3.2 in https://proceedings.neurips.cc/paper_files/paper/2021/file/34f98c7c5d7063181da890ea8d25265a-Paper.pdf implies that their sample complexity is O(1/\\eps^2) when applied in linear function approximation. This result is based on assumption3 in their paper. This assumption naturally holds in your paper since you consider linear MDP and they consider the case of 'linear function approximation' (for their difference, see point 2). So it is natural for you to compare your sample complexity to this result, not the O(1/\\eps^5) one. [notice that, their algorithm in section 3 is computationally inefficient]\n\n1.2 In Theorem 4.1 in Xie's paper, their sample complexity is O(1/\\eps^5) when applied on general function approximation, and O(1/\\eps^3) when reduced to linear function approximation case (see paragraph 'Dependence oon T'). Again, their assumption for linear function approximation holds in your case. **This algorithm, however, is computationally efficient.** So you should also compare with this alg with  O(1/\\eps^3) sample complexity.\n\n1.3 In Cheng's paper, in theorem 5, their sample complexity seems to be O(1/\\eps^3), not O(1/\\eps^5). I wonder how you derive their sample complexity in Table one.\n\n1.4 I am not sure how you get the O(n^{7/5}) computational complexity for Xie's paper. Could you derive it in more detail?\n\n2. I think in many places in your paper, you confuse the two terms: linear MDP and linear function approximation. Your case is called linear MDP instead of linear function approximation, so I suggest you changing the wrong terms. For reference, both Xie's paper and Cheng's paper consider the 'linear function approximation' case.\n\nQUESTIONS:\n1. In Line 82 and 87, there are typos in the definition of rho and state occupancy measure? There should be a power of gamma in the sum?\n\n2. In line 174-176, you calculate the computational complexity of your algorithm. Can you show this in more detail? I am not sure why each gradient update needs a constant number of elementary vector operation. What is your definition for elementary vector operation?\n\n3. In section 3 and 4, you use c \\in {1/2, 1} to state all your results. I wonder whether c can only take these two values? If so, then why cannot c take other values (for example, 2 or 3 or 1/4)? If not, then how will the algorithm behave and how will the sample complexity be when we take c to be other values? If c can take multiple values besides 1/2 and 1, how can you tune c to get the best performance?\n\n4. In theorem 3.2, the sample complexity depend on constant c by D_\\phi^{8c} * d^{-2c}, right? Does the constant hidden by the big O depend on c or nor? If not, then suppose D_\\phi^{8} / d^2 < 1, if we take c to be large or even converge to the infinify, then the term inside the big O will approach zero. How can that happen? (Let's suppose D)\\beta is a constant and does not depend on c). For example, in the most common setting, c = 1 and d > 1, then when we take c \\to \\infty, the term in the bracket wiil be zero. If the term hidden by the big O depends on constant c, then you should point it out in the paper.\n\n5. In Line 198, you say D_\\beta = c * C for some positive universal constant c. Is this c multiplied before the concentrabilty coefficient the same as that c in definition 3.1? If not, then you should use another notation. \n\n6. In line 315, you claim C_{\\phi,1} = 1 + chi-square divergence. However, C_{\\phi,1} is \\sum (\\mu^*)^2 / (\\mu_B)^2 and the chi-square divergence should be \\sum (\\mu^*)^2 / (\\mu_B), right? So they should not be equal. For the definition of chi-square divergence, see https://en.wikipedia.org/wiki/F-divergence. \n\n7. In line 77, you claim the state space is finite but potentially very large; but in Line 338, you claim that 'not to assume the state space to be finite'. From the proof, I think the latter one is correct, and the first one is just for deriving some equations like equation 1 and 2. So I think you should write somewhere in section 3 that you relax this setting of finite state space and the derivation there also holds for infinite state space but finite action space.\n\nLIMITATIONS:\n/", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies offline reinforcement learning (RL) with linear function approximation and partial data coverage. The authors propose a primal-dual optimization method based on the linear programming (LP) formulation of RL. They prove a $O(\\epsilon^{-4})$ sample complexity in both discounted setting and average-reward setting.\n\nSTRENGTHS:\n1.\tThe algorithm proposed in this paper only requires near-minimal dataset coverage assumption, which is important in offline RL.\n2.\tThe paper also considers average-reward offline RL, which is often neglected by literature.\n3.\tI like the table for comparison to previous work, which makes the presentation more clear (although I think there is some missing important literature, which I will mention in the weakness section).\n4.\tThe proposed algorithm is both computationally and sample efficient.\n\nWEAKNESSES:\n1.\tThe first concern is the ‘linear function approximation’ setting, which is restricted. Actually, the main motivation that this paper studies function approximation beyond tabular settings is large state (or action) spaces in practice. However, in real settings, the linear function approximation assumption hardly ever holds. Even in Table 1, many algorithms in previous work apply to general function approximation, which further makes the setting studied in this paper restricted.\n2.\tAlgorithm 1 in this paper achieves a $O(\\epsilon^{-4})$ sample complexity. This is in terms of expectation (as shown in Theorem 3.2) instead of high probability. The previous results that the authors are comparing to are high probability bound (e.g., [1,2]), so it would be more comparable if the authors could also show a $O(\\epsilon^{-4})$ sample complexity bound under high probability. Also, since the previous work studies general function approximation while this paper studies only linear function approximation, it is hard to say that a $O(\\epsilon^{-4})$ sample complexity bound in linear function approximation setting is better than a $O(\\epsilon^{-5})$ bound in general function approximation setting. Moreover, [3] achieves the near-optimal sample complexity $O(\\epsilon^{-2})$ with near-identical settings of [1,2]. (So I disagree with the statement that ‘It is very important to notice that no practical algorithm for this setting so far, including ours, can match the minimax optimal sample complexity rate of $O(\\epsilon^{-2})$’. Therefore, a $O(\\epsilon^{-4})$ in linear function approximation is not that attractive compared to previous work.\n3.\tThe authors use an LP formulation of offline RL. I think it would be better to compare to other work using LP formulation, e.g. [4,5], where [4] is computational and sample efficient under partial data coverage and general function approximation, and [5] achieves near-optimal sample complexity under similar settings.\n4.\tThe authors compare the computational complexity. However, it is not that direct to compare an $O(n)$ complexity in linear settings to a $O(n^{7/5})$ complexity in general settings. If the authors really want to demonstrate that their algorithm has better computational complexity, it would be better to do some simulations in the same environment (even in some toy examples).\n5.\tAnother advantage that the authors claim is that their algorithm could be adapted to average-reward setting. However, neither did the authors emphasize and explain the importance and challenges of average-reward settings, nor discuss why (or whether) previous work could not be adapted to average-reward settings. I suggest the authors discuss this a bit more.\n\n**References**\n\n[1] Xie, T., Cheng, C. A., Jiang, N., Mineiro, P., & Agarwal, A. (2021). Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems, 34, 6683-6694.\n\n[2] Cheng, C. A., Xie, T., Jiang, N., & Agarwal, A. (2022, June). Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning (pp. 3852-3878). PMLR.\n\n[3] Zhu, H., Rashidinejad, P., & Jiao, J. (2023). Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning. arXiv preprint arXiv:2301.12714.\n\n[4] Zhan, W., Huang, B., Huang, A., Jiang, N., & Lee, J. (2022, June). Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory (pp. 2730-2775). PMLR.\n\n[5] Rashidinejad, P., Zhu, H., Yang, K., Russell, S., & Jiao, J. (2022). Optimal conservative offline rl with general function approximation via augmented lagrangian. arXiv preprint arXiv:2211.00716.\n\nQUESTIONS:\n1.\tWould it be easy to convert the result from the expectation form to the high-probability version?\n2.\tIs it easy to modify previous algorithms for the average-reward settings? If not, what is the difficulty?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}

comments: SUMMARY:
This paper studies offline reinforcement learning with linear function approximation. They propose a primal-dual algorithm, formulating linear RL into a minimax problem and solving it with gradient descent-ascent. Sample complexity analysis is provided for infinite-horizon discounted and average-reward MDPs, where the rate is $O(\frac{1}{\epsilon^4})$ for both settings.

STRENGTHS:
1. The algorithm is primal-dual and thus easy to implement in practice.
2. The paper provides rigid theoretical analysis.

WEAKNESSES:
1.  The newly defined coverage ratio $C_{\phi,c}$ is a little strange when $c\neq \frac{1}{2}$. For example, when we choose $c=1$ and thus we don't need the knowledge of $\Lambda$, the coverage ratio $C_{\phi,1}=\sum_{x,a}(\frac{\mu^*(x,a)}{\mu_B(x,a)})^2$. Then when $\mu^*=\mu_B$, $C_{\phi,1}$ will become $|X||A|$. However, in the literature, when the behavior policy is the same as the optimal policy, the coverage is typically 1. The authors claim that we can estimate the $\Lambda$ via the offline dataset so that we can choose $c=\frac{1}{2}$, but do not provide any theoretical analysis about this point. I will be more convinced if the authors can give more rigid proofs for this method.
2. The sample complexity is worse than the typical rate $\frac{1}{\epsilon^2}$.

QUESTIONS:
None.

LIMITATIONS:
None.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-27

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the problem of offline reinforcement learning (RL) for linear Markov Decision Processes (MDPs) under the infinite-horizon discounted and average-reward settings. The authors propose a primal-dual optimization method based on the linear programming formulation of RL, which allows for efficient learning of near-optimal policies from a fixed dataset of transitions under partial coverage. The proposed algorithms improve the sample complexity compared to previous methods from $O(\\epsilon^{-5})$ to $O(\\epsilon^{-4})$ under the discounted setting and provide the first line of result in the average-reward setting with realizable linear function approximation and partial coverage.\n\nSTRENGTHS:\n1 The proposed algorithm improves existing algorithms in both statistical efficiency and computational efficiency under the discounted reward setting with the linear function approximation (we note the baseline may handle problems beyond the linear MDPs).\n\n2 The algorithms presented in this paper do not explicitly leverage the principle of pessimism, but focus on the linear programming formulation of MDP, and rely on a new reparametrization trick extended from the tabular case. The technique itself seems to be novel to me.\n\n3 The algorithms present the first line of work for the offline average-reward MDP.  \n\n4 The paper is easy to follow, with a thorough comparison with existing work that clearly positions the results in the literature.\n\nWEAKNESSES:\n1 I am confused about the requirement of $\\Lambda$ to be invertible (line 140) as this seems to be very closely related to the uniform coverage condition where we assume that the smallest eigenvalue of $\\Lambda$ is lower bounded from zero. I am wondering what is the key difference between them. Can you elaborate on this with some intuitions or examples?\n\n2 The authors discuss the relationship between the coverage condition considered in this paper and that of [1] and show that the coverage condition is a low-variance version of the standard feature coverage ratio if $c=1/2$. However, in this case, the algorithm explicitly uses $\\Lambda$, while the PEVI proposed in [1] does not. In contrast, $c=1$ leads to a worse bound but we do not need the knowledge of $\\Lambda$. Could you provide a more detailed characterization or example to illustrate the difference between these two cases?\n\n\ntypo: line 328, $\\epsilon^2 \\to \\epsilon^{-2}$\n\n[1] is pessimism provably efficient for offline rl\n\nQUESTIONS:\nsee weakness\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-27"}

comments: SUMMARY:
The authors investigate offline RL in linear MDPs and introduce a novel LP-based method. They assert that their proposed approach achieves the lowest sample complexity of $O(1/\epsilon^4)$ among computationally efficient algorithms. In comparison, existing computationally efficient algorithms can achieve $O(1/\epsilon^5)$. Additionally, the author's theory can be extended to the average reward setting.

STRENGTHS:
* To the best of the author’s knowledge, in offline linear MDPs, the result in the average-reward setting is novel. 
* The LP formulation in linear MDPs is worthwhile to investigate

WEAKNESSES:
* I am uncertain about whether it is appropriate to claim that existing offline RL algorithms in linear MDPs achieve $O(1/\epsilon^5)$. It appears that [38] may have better sample complexity. In Table 1 of the manuscript, the author mentions that [38] cannot handle the discounted setting. However, extending from the finite-horizon to the discounted infinite-horizon setting is relatively straightforward. Hence, this comparison may not be entirely fair. If [38] indeed has better sample complexity, it significantly impacts the author's contribution. Thus, I currently rate the paper with a score of 4.

* I am not entirely certain about the significance of the extension to the average reward case.

* Presently, I cannot determine whether the reason [9] and [36] cannot handle the average reward case is due to the algorithms or their analysis. If this limitation arises from their analysis, their algorithm has the potential to be superior as it can handle more general MDPs.

QUESTIONS:
Could you provide further elaboration on my concerns regarding (1) the comparison between the author's work and [38], and (2) the third point in the weakness section?

LIMITATIONS:
They discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-08-02