Title: Offline RL via Feature-Occupancy Gradient Ascent
Venue: NeurIPS 2024
Accepted: no
Keywords: Offline Reinforcement Learning

Linear MDPs

Provably efficient RL.
Abstract: We study offline Reinforcement Learning  in large infinite-horizon discounted Markov Decision Processes (MDPs) when the reward and transition models are linearly realizable under a known feature map. Starting from the classic linear-program formulation of the optimal control problem in MDPs, we develop a new algorithm that performs a form of gradient ascent in the space of feature occupancies, defined as the expected feature vectors that can potentially be generated by executing policies in the environment.	We show that the resulting simple algorithm satisfies strong computational and sample complexity guarantees, achieved under the least restrictive data coverage assumptions known in the literature. In particular, we show that the sample complexity of our method scales optimally with the desired accuracy level and depends on a weak notion of coverage that only requires the empirical feature covariance matrix to cover a single direction in the feature space (as opposed to covering a full subspace). Additionally, our method is easy to implement and requires no prior knowledge of the coverage ratio (or even an upper bound on it), which altogether make it the strongest known algorithm for this setting to date.

Abstract

We study offline Reinforcement Learning in large infinite-horizon discounted Markov Decision Processes (MDPs) when the reward and transition models are linearly realizable under a known feature map. Starting from the classic linearprogram formulation of the optimal control problem in MDPs, we develop a new algorithm that performs a form of gradient ascent in the space of feature occupancies, defined as the expected feature vectors that can potentially be generated by executing policies in the environment. We show that the resulting simple algorithm satisfies strong computational and sample complexity guarantees, achieved under the least restrictive data coverage assumptions known in the literature. In particular, we show that the sample complexity of our method scales optimally with the desired accuracy level and depends on a weak notion of coverage that only requires the empirical feature covariance matrix to cover a single direction in the feature space (as opposed to covering a full subspace). Additionally, our method is easy to implement and requires no prior knowledge of the coverage ratio (or even an upper bound on it), which altogether make it the strongest known algorithm for this setting to date.

1 Introduction

We study Offline Reinforcement Learning (ORL) in sequential decision making problems whereby a learner aims to find a near-optimal policy with sole access to a static dataset of interactions with the underlying environment [Levine et al., 2020]. This line of work is naturally relevant to realworld tasks for which learning an accurate simulator of the environment is potentially intractable or impossible, trial-and-error learning could have grave consequences, yet logged interaction data is readily available. For example, in a high-stake application such as autonomous driving, building a sufficiently accurate simulator for the vehicle and its environment would require modelling very complex systems, which can be intractable both statistically and computationally. At the same time, running experiments in the real world could endanger the lives of other road users or result in damages to the vehicle. Yet, with the advent of tools for efficient sensory-data collection and processing, large volumes of logged data from human drivers are readily available.

An efficient ORL method is one which finds a near-optimal policy after a tractable number of elementary computations and samples from the dataset. It is well-known in this setting that the quality

of the solution has to heavily depend on the quality of the data, and in particular one cannot hope 31

to find a near-optimal policy if the data covers the space of states and actions poorly. To formalize 32

this intuition, many notions of data coverage have been proposed in the offline RL literature, ranging 33

from a very restrictive uniform coverage assumption that requires the data-generating policy to cover 34

the entire state-action space [Munos and Szepesvári, 2008] to a variety of partial coverage conditions 35

whereby this exploratory condition is only required for state-action pairs that are of interest to the 36

optimal policy [Liu et al., 2020, Rashidinejad et al., 2021, Uehara and Sun, 2021, Zhan et al., 2022, 37

Rashidinejad et al., 2022, Li et al., 2024]. In the present work, we study the setting of linear Markov 38

Decision Processes (MDPs) [Jin et al., 2020, Yang and Wang, 2019] where the reward and transition 39 matrix admit a low rank structure in terms of a known feature map, and data-coverage assumptions 40 can be defined in the space of features. As shown by [Zanette et al., 2021], in this setting it is possible 41 to obtain strong guarantees if the offline data is well-aligned with the expectation of the feature vector 42 generated by the optimal policy (as opposed to requiring alignment with the entire distribution of 43 features as required by other common offline RL methods [Jin et al., 2021, Xie et al., 2021, Uehara 44 and Sun, 2021, Zhang et al., 2022]). In the present paper, we propose a simple and efficient algorithm 45 that yields the best known sample complexity guarantees for this problem setting, all while only 46 requiring the weakest known data-coverage assumptions of Zanette et al. [2021]. 47

48

49

50

51

52

53

54

55

56

57

Our approach is based on the LP formulation of optimal control in infinite-horizon discounted MDPs due to Manne [1960], and more specifically on its low-dimensional saddle-point reparametrization for linear MDPs proposed by Gabbianelli et al. [2024] (which itself builds on earlier work by Neu and Okolo, 2023 and Bas-Serrano et al., 2021). Primal variables of this saddle-point objective correspond to expectations of feature vectors under the state-action distribution of each policy (called feature occupancies ), and dual variables correspond to parameters of linear approximations of actionvalue functions. We design an algorithm based on the idea of optimizing the unconstrained primal function that is derived from the saddle-point objective by eliminating the dual variables via a classic dualization trick. More precisely, we design a sample-based estimator of the primal function and optimize it via a variant of gradient ascent in the space of feature occupancies.

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

This approach is to be contrasted with the method of Gabbianelli et al. [2024], which instead optimized the original saddle-point objective via stochastic primal-dual methods. Their algorithm interleaved a sequence of 'policy improvement' steps with an inner loop performing 'policy evaluation', which resulted in a suboptimal use of sample transitions due to the costly inner loop. This issue was addressed in the very recent work of Hong and Tewari [2024] who, instead of relying on stochastic optimization, built an estimator of the saddle-point objective and optimized it via a deterministic primal-dual method. Our approach is directly inspired by their idea of estimating the saddle-point objective, but our algorithm design is significantly simpler: instead of directly optimizing the primal function in terms of feature occupancies, Hong and Tewari [2024] relied on a sophisticated reparametrization of the primal variables, and used a computationally involved procedure to update the dual variables. Both of these steps required prior knowledge of a tight bound on the featurecoverage ratio of the optimal policy, which is typically not available in problems of practical interest. Such knowledge is not required by our algorithm, thanks to the incorporation of a recently proposed stabilization trick that we make use of in our algorithm [Jacobsen and Cutkosky, 2023, Neu and Okolo, 2024]. We provide a more detailed discussion of these closely related works in Section 5.

Notation. We use boldface lowercase letters m to denote vectors and and bold uppercase M for matrices. We define the Euclidean ball in R d of radius D by B d ( D ) = { x ∈ R d | ∥ x ∥ 2 ≤ D } and the A -simplex over a finite set A of cardinality A as ∆ A = { p ∈ R A + | ∥ p ∥ 1 = 1 } .

2 Preliminaries

We consider infinite-horizon Discounted Markov Decision Processes (DMDPs) [Puterman, 1994] of the form ( X , A , r , P , γ ) where X denotes a finite (yet large) set of X states and A is a finite action space of cardinality A = |A| . We refer to r ∈ [0 , 1] XA as the reward vector, P ∈ R XA × X + the transition matrix and γ ∈ (0 , 1) the discount factor. For a state-action pair ( x, a ) ∈ X × A we also use the notation r ( x, a ) = r [( x, a )] to denote the reward of taking action a in state x and p ( x ′ | x, a ) = P [( x, a ) , x ′ ] as the probability of ending up in state x ′ afterwards.

The MDP models a sequential decision making process where an agent interacts with its environment as follows. For each step k = 0 , 1 , 2 , · · · , , the agent observes the current state X k of the environment and then goes on to select its action A k . Based on this action in the current state, it receives a reward r ( X k , A k ) , transits to a new state X k +1 ∼ p ( ·| X k , A k ) and the process continues. The objective of the agent is to find a decision-making rule that maximizes its total discounted reward when the initial state X 0 is sampled according to a fixed initial-state distribution distribution ν 0 ∈ ∆ X . Without loss of generality, we assume that the initial state is fixed almost surely as X 0 = x 0 , and use ν 0 to refer to the corresponding delta distribution. It is known that this objective can be achieved by executing a stationary stochastic policy π : X → ∆ A , with π ( a | x ) denoting the probability of the agent selecting action A k = a in state X k = x for all k . We will use Π to denote the set of all such behavior rules

and will often simply call them policies . We define the normalized discounted return of each policy π 93 as 94

95

96

97

where the role of the discount factor γ ∈ (0 , 1) is to emphasize the importance of earlier rewards, and the notation E ν 0 ,π [ · ] highlights that the initial state is sampled from ν 0 and all actions are sampled according to the policy π . We will use π ∗ to denote any policy that maximizes the return.

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

We will consider the offline RL setting where we are given access to a data set of n sample transitions D n = { ( X i , A i , R i , X ′ i ) } n i =1 , where X ′ i ∼ p ( ·| X i , A i ) is sampled independently for each i and R i = r ( X i , A i ) . Otherwise, no assumption is made about the state-action pairs ( X i , A i ) , and in particular we do not require these to be generated by a fixed behavior policy or to be independent of each other.

For describing the approach we take towards solving this problem, we need to introduce some further standard notations. The value function and action-value function associated with policy π are respectively defined as

and the state-occupancy and state-action-occupancy measures under π as

The value functions and occupancy measures adhere to the following recursive equations, respectively termed the Bellman equation and Bellman flow condition [Bellman, 1966]:

Here, the composition operation ◦ is defined so that for any policy π and state distribution ν ∈ R X , we have ( π ◦ ν ) ( x, a ) = π ( a | x ) ν ( x ) . Notice that we can express the return of π in terms of value functions and occupancy measures as ρ ( π ) = (1 -γ ) ⟨ ν 0 , v π ⟩ = ⟨ µ π , r ⟩ . On this note, for a given target accuracy ε > 0 , we say policy π is ε -optimal if it satisfies 〈 µ π ∗ -µ π , r 〉 ≤ ε .

In the present work, we well make use of the linear MDP assumption due to Jin et al. [2020], Yang and Wang [2019], which is defined formally as follows:

Definition 2.1 (Linear MDP) . An MDP is called linear if both the transition and reward functions can be expressed as a linear function of a given feature map φ : X × A → R d . That is, there exist ψ : X → R d and ω ∈ R d such that, for every x, x ′ ∈ X and a ∈ A :

We denote by Φ ∈ R |X×A|× d the feature matrix with rows given by φ ( x, a ) T and Ψ ∈ R d ×|X| as the weight matrix with columns ψ ( x ) . Further, we will assume that ∥ ω ∥ 2 ≤ √ d , that ∥ Ψ v ∥ 2 ≤ B √ d holds for all v ∈ [ -B,B ] , and that all feature vectors satisfy ∥ φ ( x, a ) ∥ 2 ≤ R for some R ≥ 1 .

An immediate consequence of this assumption is that the action-value function of any policy π can be written as a linear function of the features as q π = Φ θ π , with θ π = ω + γ Ψ v π ∈ R d . For the rest of the paper we explicitly assume that the feature matrix Φ is full rank - which is enough to ensure uniqueness of θ π . It is common to assume that the feature dimension d ≪ X such that the transition operator is low-rank. As common in this setting, we will suppose throughout the paper that the feature map Φ is known.

Our algorithm design will be based on the linear programming formulation of MDPs, first proposed in a number of papers in the 1960's [Manne, 1960, de Ghellinck, 1960, d'Epenoux, 1963, Denardo, 1970]. This formulation frames the problem of finding an optimal control policy as the following pair of primal and dual linear programs:

Here, the operator E ∈ R XA × X is defined such that for each x, a and vectors µ ∈ R XA , v ∈ R X , 131

It is known that the occupancy measure of an optimal policy µ π ∗ is an optimal solution of the primal LP (1). In fact, the feasible set of the primal is precisely the space of valid state-action occupancy measures that can be induced by stationary policies. Therefore, given any feasible solution µ , we can extract the inducing policy as π µ ( a | x ) = µ ( x, a ) / ∑ a ′ µ ( x, a ′ ) when ∑ a µ ( x, a ) = 0 . Likewise, the state value function of the optimal policy π ∗ is an optimal solution to the dual LP. That said, since the LP features XA variables and constraints, it cannot be solved directly in large MDPs.

̸

In view of the above limitations, we consider the following reduced version of the above intractable LPs due to Gabbianelli et al. [2024] (see also Neu and Okolo, 2023, Bas-Serrano et al., 2021):

132

133

134

135

136

137

138

139

140

In view of the second constraint of the primal LP (3), λ should be thought of as expectations of feature 141 vectors under occupancy measures, and we thus refer to them as feature occupancy vectors. Similarly, 142 the second constraint of the dual LP (4) suggests that θ should be thought of as parameters of the 143 approximate action-value function q θ = Φ θ = Φ ( ω + γ Ψ v ) = r + γ Pv . Weuse λ π ∗ = Φ T µ π ∗ to 144 denote the feature occupancy associated with the optimal policy π ∗ and θ π ∗ to denote the parameter145 vector of the optimal action-value function q π ∗ . The Lagrangian corresponding to the LPs is given as 146

It is easy to verify that by the linear MDP property, the feasible sets of the above LPs coincide 147 with those of the original LPs in an appropriate sense, and their optimal solutions correspond to 148 the optimal state-action occupancy measure and state-value function respectively (see Appendix A). 149

In order to further reduce the complexity of the LPs above, we introduce a policy π and parametrize 150 the remaining high-dimensional variables v and µ as 151

152

153

154

155

156

157

158

Plugging this choice back into the Lagrangian, we obtain the objective

̸

It is easy to see that for any π and λ π = Φ T µ π , we have f ( λ π , π ; θ ) = ⟨ µ π , r ⟩ for all θ ∈ R d . Furthermore, whenever λ = λ π then the θ -player has a winning strategy that can force min θ f ( λ , π ; θ ) = -∞ . This (informally) suggests that an optimal policy can be found by solving the unconstrained saddle-point optimization problem max λ ∈ R d ,π ∈ Π min θ ∈ R d f ( λ , π ; θ ) . Furthermore, since the optimal policy can be written as π ∗ ( a | x ) = I { a = argmax b ⟨ θ π ∗ ,φ ( x,b ) ⟩} , it is sufficient to consider softmax policies of the form

which can approximate π ∗ to good precision when the diameter D π is set to be large enough. This 159 parametrization effectively reduces the high-dimensional LP into a low-dimensional saddle-point 160 optimization problem. 161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

3 Feature-occupancy gradient ascent for offline RL in linear MDPs

A natural idea for developing RL methods is to build an empirical approximation of the function f defined in the previous section, and use primal-dual methods to find a saddle-point of the resulting approximation. For offline RL, this approach has been explored by Gabbianelli et al. [2024] and Hong and Tewari [2024]. In this work, we develop an alternative approach that seeks to directly optimize the return by approximately maximizing the unconstrained primal function f ∗ : R d × Π , defined for each feature-occupancy vector λ and policy π as

for an appropriately chosen feasible set B d ( D θ ) . Given the discussion in the previous section, maximizing this function with respect to λ and π is rightly expected to result in an optimal policy (which intuition will be made formal in our analysis). Notably, the so-called objective f in Equation (7) depends on the transition weight matrix Ψ which is unknown in general. As we soon show, this matrix dominates the loss of the θ -player and λ -player. Based on these observations, our approach consists of building a well-chosen estimator ̂ f of f , and then maximizing the associated primal function ̂ f ∗ defined as

The objective ̂ f is built via a least-squares estimator inspired by the classic LSTD model estimate of Bradtke and Barto [1996], Parr et al. [2008], which has been successfully used for analyzing finite-horizon linear MDPs in a variety of recent works (e.g., Jin et al., 2020, Neu and Pike-Burke, 2020). In particular, we fit an estimator ̂ Ψ of the true matrix Ψ using samples from the dataset D n = { ( X i , A i , R i , X ′ i ) } n i =1 as follows. Let φ i = φ ( X i , A i ) denote the feature vector of ( X i , A i ) and Λ n = β I n + 1 n ∑ n i =1 φ i φ T i the empirical feature covariance matrix. We define the regularized least squares estimate of Ψ at x ∈ X as

so that the estimate can be written as

With this matrix at hand, we define ̂ f as

where ̂ µ λ ,π ( x, a ) = π ( a | x ) [ (1 -γ ) ν 0 ( x ) + γ ⟨ ̂ ψ ( x ) , λ ⟩ ] is a sample-based approximation of µ λ ,π .

For the purpose of optimization, we will employ appropriately chosen versions of mirror ascent [Nemirovski and Yudin, 1983, Beck and Teboulle, 2003] to iteratively optimize the primal variables. Denoting the iterates for each t = 1 , 2 , . . . , T by λ t and π t , and defining θ t = arg min θ ∈ B d ( D θ ) ̂ f ( λ t , π t ; θ ) , the updates are defined as follows. Using g λ ( t ) = ∇ λ t ̂ f ∗ ( λ t , π t ) to denote the gradient of ̂ f ∗ with respect to the feature occupancies, the first set of variables is updated as

where the first regularization term acts as proximal regularization (necessary for mirror-ascent-style methods), and the second one has a stabilization effect whose role will be made clear later in the analysis. The resulting update can be written in closed form, and is equivalent to a preconditioned gradient-ascent step on ̂ f ∗ . The policies are updated in each state-action pair x, a as

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

Algorithm 1 Feature-Occupancy Gradient Ascent ( FOGAS )

corresponding to performing an entropy-regularized mirror ascent step in each state x (cf. Neu et al., 2017). We use the shorthand notation π t +1 = σ ( α Φ ∑ t k =1 θ k ) to denote the resulting softmax policy, and note that it is fully specified by a d -dimensional vector that can be stored compactly. After the final iterate is computed, the algorithm picks the index J uniformly at random and outputs the policy π J . We refer to the resulting algorithm as Feature-Occupancy Gradient AScent ( FOGAS ), and present its detailed pseudocode featuring the explicit expressions of λ t and θ t as Algorithm 1.

The following theorem states our main result regarding the performance of FOGAS .

Theorem 3.1. Let π 1 be the uniform policy and λ 1 = 0 . Also set D θ = √ d/ (1 -γ ) , D π = αTD θ and δ > 0 . Suppose that we run FOGAS for T ≥ 2 R 2 n log A log(1 /δ ) rounds with parameters β = R 2 /dT as well as

Then, with probability at least 1 -δ , the following bound is satisfied for any comparator policy π ∗ and the associated feature-occupancy vector λ π ∗ = Φ T µ π ∗ :

with the expectation taken with respect to the random index J .

The most important factor in the bound of Theorem 3.1 is ∥ λ ∗ ∥ 2 Λ -1 n , which measures the extent to which the data D n covers the comparator policy π ∗ in feature space. We accordingly refer to this quantity as the feature coverage ratio between the policy π ∗ and the data set D n , and we discuss its relationship with other notions of data coverage in Section 5. Notably, the bound holds simultaneously for all comparator policies π ∗ , and thus it can be restated in an oracle-inequality form. On the same note, FOGAS does not need any prior upper bounds on the comparator norm ∥ λ ∗ ∥ 2 Λ -1 n , and in particular it does not project the iterates λ t to a bounded set. These nontrivial properties are enabled by a recently proposed stabilization trick due to Jacobsen and Cutkosky [2023] and Neu and Okolo [2024], which amounts to augmenting the standard mirror-ascent update of Equation (9) with the regularization term ϱ 2 ∥ λ ∥ 2 Λ -1 n . Without this additional regularization, the bounds would feature an additional factor of the order 1 T ∑ T t =1 ∥ λ t ∥ 2 Λ -1 n , which cannot be controlled without projecting the iterates and in any case make it impossible to prove a comparator-adaptive bound. We defer further discussion of the result to Section 5.

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

4 Analysis

This section is dedicated to proving our main result, Theorem 3.1. While we have defined FOGAS as a 'primal-only' algorithm above, its analysis will be most convenient if we regard it as a primal-dual algorithm with implicitly defined dual updates. In particular, we will view the updates of FOGAS as a sequence of steps in a zero sum game between two teams of players: the max players that control λ t and π t , and the min player that picks θ t . The min player uses the simple best-response strategy of picking θ t = arg min θ ∈ B d ( D θ ) ̂ f ( λ , π t ) , and the other two players perform their updates via appropriate versions mirror ascent on their respective objectives. Importantly, the updates of the λ -player are based on the gradients of ̂ f ∗ , which satisfy

where the last equality follows from an application of Danskin's theorem. This property enables a major conceptual simplification that allows the interpretation of the updates as optimizing the unconstrained primal ̂ f ∗ directly. We refer the interested reader to Chapter 6 of Bertsekas [1997] for more context on such use of primal-dual analysis.

More concretely, we make use of an analysis technique first developed by Neu and Okolo [2023], and further refined by Gabbianelli et al. [2024] and Hong and Tewari [2024]. The core idea is to introduce the dynamic duality gap defined on a sequence of iterates { ( λ t , π t , θ t ) } T t =1 produced by some iterative method, and a set of well-chosen comparators ( λ ∗ , π ∗ ; { θ ∗ t } T t =1 ) as

Similar to Lemma 4.1 of Gabbianelli et al. [2024], we show in Lemma 4.1 below that with an appropriate choice of the comparator points, we can relate the gap to the expected suboptimality of policy π J where J ∼ U (1 , · · · , T ) . We leave the proof in Appendix B.1.1.

Lemma 4.1. Suppose that D θ = √ d/ (1 -γ ) . Choose ( λ ∗ , π ∗ , θ ∗ t ) = ( Φ T µ π ∗ , π ∗ , θ π t ) ∈ R d × Π( D π ) × B d ( D θ ) for t = 1 , · · · , T where µ π ∗ is a valid occupancy measure induced by π ∗ . Then,

We will show below that the dynamic duality gap can be written in terms of the regrets of each player and an additional term related to the estimation error of ̂ f , and then proceed to provide bounds on all of these quantities. Specifically, the regrets of each player with respect to each of their respective comparators are defined as

where ν ∗ = (1 -γ ) ν 0 ( x ) + γ ⟨ ψ ( x ) , λ ∗ ⟩ . Furthermore, we define the gap-estimation error as

The following lemma rewrites the duality gap using the above terms.

Lemma 4.2. The dynamic duality gap satisfies 250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

The proof directly follows from a straightforward calculation similar to the proof of Lemma 4.2 of Gabbianelli et al. [2024] and Section E.1 of Hong and Tewari [2024] which is reproduced in Appendix B.1.2 for completeness. It remains to bound the regret of the players, as well as the gap-estimation error. An obstacle we need to face in the analysis is that our bound of the latter error term scale with 1 T ∑ T t =1 ∥ λ t ∥ 2 Λ -1 n , which is undesirable given our aspiration to achieve bounds that scale only with the comparator norm ∥ λ ∗ ∥ 2 Λ -1 n without requiring prior upper bounds on this quantity (that would enable us to project the iterates to a bounded domain). This challenge is addressed by making use of the stabilization technique of Jacobsen and Cutkosky [2023] and Neu and Okolo [2024] in the updates for the λ -player, which effectively eliminates these problematic terms. We briefly outline the remaining parts of the analysis below.

4.1 Regret analysis

The regrets of each player are respectively controlled by the following three lemmas.

Lemma 4.3. Suppose that ν ∗ ∈ ∆ X . Let π 1 be the uniform policy which selects all actions with equal probability in each state. Under the conditions on the feature map in Definition 2.1, the regret of the π -player against π ∗ satisfies 1 T R T ( π ∗ ) ≤ log A αT + αR 2 D 2 θ 2 .

The proof is a standard application of the analysis of exponential-weight updates, stated as Lemma E.1. Lemma 4.4. Let λ 1 = 0 and C = 6 β ( d + D 2 θ ) +3 d (1 + RD θ ) 2 +3 γ 2 dR 2 D 2 θ . Then, the regret of the λ -player against any comparator λ ∗ ∈ R d satisfies

The proof (provided in Appendix B.2.2) follows from applying the standard analysis of compositeobjective mirror descent due to Duchi et al. [2010] (stated as Lemma C.1 in the Appendix) and the bound ∥ Λ n g λ ( t ) ∥ 2 Λ -1 n ≤ C on the weighted norm of the gradients for all t provided in Lemma C.2. √

As we show in Appendix B.2.3, the above statement holds trivially thanks to the 'best-response' definition of θ t . This concludes our regret analysis.

4.2 Bounding the gap-estimation error

The following statement (proved in Appendix B.3) provides a bound on err ̂ Ψ :

Lemma 4.6. Suppose that ∥ φ ( x, a ) ∥ 2 ≤ R for all ( x, a ) ∈ X × A , D θ = √ d/ (1 -γ ) and α = √ 2 (1 -γ ) 2 log A/R 2 dT to optimize R T ( π ∗ ) . Then, for any T ≥ 2 R 2 n log A log(1 /δ ) and and ξ ≥ 0 , the following holds with probability at least 1 -δ :

4.3 The proof of Theorem 3.1

The proof follows from applying Lemmas 4.1 and Lemma 4.2 when ( λ ∗ , π ∗ , θ ∗ t ) = ( Φ T µ π ∗ , π ∗ , θ π t ) ∈ R d × Π( D π ) × B d ( D θ ) for t = 1 , · · · , T . Then, adding up the bounds stated in Lemmas 4.3-4.6 under the respective conditions, yields

Then, setting ρ = γ ξT simplifies the second term and eliminates the third term. The claim then follows 284 after optimizing the hyperparameters, with the full details provided in Appendix B.4. 285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

5 Discussion

We discuss various aspects of our results below.

Relation with previous work. As discussed in the introduction, our work draws heavily on previous contributions of Gabbianelli et al. [2024] and Hong and Tewari [2024]. In particular, our idea of building a least-squares estimator of the transition function is directly borrowed from the latter of these works, and our implicit update rule for θ t is also inspired by their work to a good extent. Their approach, however, failed to reach the same degree of efficiency due to a number of suboptimal design choices. First, they used an alternative parametrization of the feature occupancies which only allowed them to work under a more restrictive coverage condition, so that their bounds depend on ∥ λ ∗ ∥ Λ -2 n which can be much larger than the feature coverage ratio appearing in our bounds. Second, their algorithm required a prior upper bound on this coverage parameter, with the guarantees scaling with the bound rather than the actual coverage. Such bounds are typically difficult to obtain in practice. Third, the implementation of their algorithm required intricate computational steps necessitated by their feature-occupancy parametrization. Our work has successfully removed these limitations and reduced the complexity of their method, thanks to a new primal-only analysis style that we hope will find further uses in reinforcement learning.

Computational and statistical efficiency. As can be inferred from our main result, the sample complexity of finding an ε -optimal policy using our algorithm is of the order d 2 ∥ λ ∗ ∥ 2 Λ -1 n /ε 2 (1 -γ ) 2 , which is optimal in terms of scaling with ε . The rate can be improved to scale linearly with the feature coverage ratio ∥ λ ∗ ∥ Λ -1 n , if a tight upper bound is known on it which can be used for hyperparameter tuning. We find this scenario to be unlikely, and are curious to see if future work can attain this improved scaling without such prior knowledge. As for computational complexity, we point out that the cost of each iteration of our method scales linearly with the sample size n , due to having to compute the matrix-vector products ̂ Ψ v θ t ,π t . Indeed, the matrix ̂ Ψ is sparse with n non-zero rows, and as such computing this product takes linear time in n . Since the iteration complexity of FOGAS scales linearly with the sample size n , this makes for an overall runtime complexity of order n 2 . This limitation is of course shared with all methods using the same least-squares transition estimator for the transition model, including all work that builds on Jin et al. [2020], but we nevertheless wonder if a substantial improvement is possible on this front.

Data coverage assumptions. The only works we are aware of that scale with the feature-coverage ratio ∥ λ ∗ ∥ Λ -1 n are due to Zanette et al. [2021] and Gabbianelli et al. [2024]. The latter work only achieves this bound under the assumption that the data is drawn i.i.d. from a fixed behavior policy with known feature covariance matrix, which is a much more restricted setting that we consider here. Such assumptions are not needed by Zanette et al. [2021], however their results are restricted to the simpler finite-horizon MDP setting, and their algorithm is arguably more complex than ours. Using our notation, their approach can be interpreted as solving a 'pessimistic' version of the the relaxed dual LP (4) that features some additional quadratic constraints. This approach is not computationally viable for the infinite-horizon discounted case we consider, as it requires solving a fixed-point equation with respect to the estimated transition operator (cf. Wei et al., 2021).

Possible extensions. Our approach can be extended and generalized in a variety of ways. First, following Gabbianelli et al. [2024], we believe that it is straightforward to extend our analysis to undiscounted infinite-horizon MDPs. Second, we similarly believe that an extension to constrained MDPs is possible without major challenges, following Hong and Tewari [2024]. We did not pursue these extensions because we believe that they add little additional insight. There are other potential directions that we did not explore because we found them to be too ambitious for the moment. These include extending our results beyond linear MDPs to other MDP models with linear function approximation, including MDPs with low inherent Bellman rank (which may be within reach of the current theory, c.f. Zanette et al., 2020), linearly Q π -realizable MDPs (which are known to be challenging, c.f. Weisz et al., 2022, 2024). Even more ambitiously, one can ask if it is possible to extend our methods to work under more general notions of function approximation. This looks very challenging given the central role of feature occupancies in our formalism, which are strictly tied to linear function approximation. We are nevertheless optimistic that the ideas presented in this work will find use in other contexts, possibly including nonlinear function approximation in the future.

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

References

Abbasi-Yadkori, D. Pál, and C. Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Bas-Serrano, S. Curi, A. Krause, and G. Neu. Logistic q-learning. In International Conference on Artificial Intelligence and Statistics , pages 3610-3618. PMLR, 2021.

Beck and M. Teboulle. Mirror descent and nonlinear projected subgradient methods for convex optimization. Operations Research Letters , 31(3):167-175, 2003.

Bellman. Dynamic programming. science , 153(3731):34-37, 1966.

P. Bertsekas. Nonlinear programming. Journal of the Operational Research Society , 48(3): 334-334, 1997.

J. Bradtke and A. G. Barto. Linear least-squares algorithms for temporal difference learning. Machine Learning , 22:33-57, 1996.

Cesa-Bianchi and G. Lugosi. Prediction, learning, and games . Cambridge university press, 2006.

de Ghellinck. Les problèmes de décisions séquentielles. Cahiers du Centre d'Études de Recherche Opérationnelle , 2:161-179, 1960.

V. Denardo. On linear programming in a Markov decision problem. Management Science , 16(5): 281-288, 1970.

d'Epenoux. A probabilistic production and inventory problem. Management Science , 10(1): 98-108, 1963.

C. Duchi, S. Shalev-Shwartz, Y. Singer, and A. Tewari. Composite objective mirror descent. In COLT , volume 10, pages 14-26. Citeseer, 2010.

Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences , 55:119-139, 1997.

Gabbianelli, G. Neu, M. Papini, and N. M. Okolo. Offline primal-dual reinforcement learning for linear mdps. In International Conference on Artificial Intelligence and Statistics , pages 3169-3177. PMLR, 2024.

Hong and A. Tewari. A primal-dual algorithm for offline constrained reinforcement learning with low-rank mdps. arXiv preprint arXiv:2402.04493 , 2024.

Jacobsen and A. Cutkosky. Unconstrained online learning with unbounded losses. In International Conference on Machine Learning , pages 14590-14630. PMLR, 2023.

Jin, Z. Yang, Z. Wang, and M. I. Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on learning theory , pages 2137-2143. PMLR, 2020.

Jin, Z. Yang, and Z. Wang. Is pessimism provably efficient for offline RL? In International Conference on Machine Learning , pages 5084-5096. PMLR, 2021.

Lattimore and C. Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Li, L. Shi, Y. Chen, Y. Chi, and Y. Wei. Settling the sample complexity of model-based offline reinforcement learning. The Annals of Statistics , 52(1):233-260, 2024.

N. Littlestone and M. Warmuth. The weighted majority algorithm.

108:212-261, 1994.

Liu, A. Swaminathan, A. Agarwal, and E. Brunskill. Provably good batch off-policy reinforcement learning without great exploration. Advances in neural information processing systems , 33: 1264-1274, 2020.

Information and Computation

,

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

S. Manne. Linear programming and sequential decisions. Management Science , 6(3):259-267, 1960.

Munos and C. Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research , 9(5), 2008.

Nemirovski and D. Yudin. Problem Complexity and Method Efficiency in Optimization . Wiley Interscience, 1983.

Neu and N. Okolo. Efficient global planning in large MDPs via stochastic primal-dual optimization. In International Conference on Algorithmic Learning Theory , pages 1101-1123. PMLR, 2023.

Neu and N. Okolo. Dealing with unbounded gradients in stochastic saddle-point optimization. arXiv preprint arXiv:2402.13903 , 2024.

Neu and C. Pike-Burke. A unifying view of optimism in episodic reinforcement learning. In Advances in Neural Information Processing Systems , pages 1392-1403, 2020.

Neu, A. Jonsson, and V. Gómez. A unified view of entropy-regularized Markov decision processes. arXiv preprint arXiv:1705.07798 , 2017.

Parr, L. Li, G. Taylor, C. Painter-Wakefield, and M. L. Littman. An analysis of linear models, linear value-function approximation, and feature selection for reinforcement learning. In Proceedings of the 25th international conference on Machine learning , pages 752-759, 2008.

L. Puterman. Markov decision processes: discrete stochastic dynamic programming . John Wiley &Sons, 1994.

Rashidinejad, B. Zhu, C. Ma, J. Jiao, and S. Russell. Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems , 34: 11702-11716, 2021.

Rashidinejad, H. Zhu, K. Yang, S. Russell, and J. Jiao. Optimal conservative offline rl with general function approximation via augmented lagrangian. arXiv preprint arXiv:2211.00716 , 2022.

Shalev-Shwartz and S. Ben-David. Understanding machine learning: From theory to algorithms . Cambridge university press, 2014.

Uehara and W. Sun. Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06226 , 2021.

Vovk. Aggregating strategies. In Proceedings of the third annual workshop on Computational learning theory (COLT) , pages 371-386, 1990.

C.-Y. Wei, M. J. Jahromi, H. Luo, and R. Jain. Learning infinite-horizon average-reward mdps with linear function approximation. In International Conference on Artificial Intelligence and Statistics , pages 3007-3015. PMLR, 2021.

Weisz, A. György, T. Kozuno, and C. Szepesvári. Confident approximate policy iteration for efficient local planning in q π -realizable mdps. Advances in Neural Information Processing Systems , 35:25547-25559, 2022.

Weisz, A. György, and C. Szepesvári. Online rl in linearly q π -realizable mdps is as easy as in linear mdps if you learn what to ignore. Advances in Neural Information Processing Systems , 36, 2024.

Xie, C.-A. Cheng, N. Jiang, P. Mineiro, and A. Agarwal. Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems , 34:6683-6694, 2021.

Yang and M. Wang. Sample-optimal parametric Q-learning using linearly additive features. In International conference on machine learning , pages 6995-7004. PMLR, 2019.

Zanette, A. Lazaric, M. Kochenderfer, and E. Brunskill. Learning near optimal policies with low inherent bellman error. In International Conference on Machine Learning , pages 10978-10989. PMLR, 2020.

429

430

431

432

433

434

435

436

Zanette, M. J. Wainwright, and E. Brunskill. Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems , 34:13626-13640, 2021.

Zhan, B. Huang, A. Huang, N. Jiang, and J. Lee. Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory , pages 2730-2775. PMLR, 2022.

Zhang, Y. Chen, X. Zhu, and W. Sun. Corruption-robust offline reinforcement learning. In International Conference on Artificial Intelligence and Statistics , pages 5757-5773. PMLR, 2022.

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

Appendix

A Missing proofs of Section 2

A.1 Properties of the relaxed LP

In this section we prove a basic result about the feasible sets of the relaxed linear programs defined in Equations (3) and (4). We remark that similar results have been previously shown in Proposition 4 of Bas-Serrano et al. [2021] and Appendix A.1 of Neu and Okolo [2023].

LemmaA.1. Suppose that the MDP satisfies the linear MDP assumption in the sense of Definition 2.1, consider the relaxed linear programs 3 and 4 and their respective feasible sets:

Then, the following statements hold:

The set M = { µ : ( λ , µ ) ∈ M P Φ } coincides with the feasible set of the primal LP (1) . Furthermore, for all ( λ ∗ , µ ∗ ) ∈ arg max ( λ , µ ) ∈M P Φ ⟨ λ , ω ⟩ , we have that µ ∗ is the occupancy measure of an optimal policy.

The set V = { v : ( v , θ ) ∈ M D Φ } coincides with the feasible set of the dual LP (2) . Furthermore, the optimal value function v π ∗ and the parameter vector θ π ∗ satisfying q π ∗ = Φ θ π ∗ satisfy ( v π ∗ , θ π ∗ ) ∈ arg min ( v , θ ) ∈M D Φ (1 -γ ) ⟨ ν 0 , v ⟩ .

Proof. We first show that for any feasible point µ of the LP (1), the tuple ( λ , µ ) is feasible for the relaxed LP with λ = Φ T µ . This choice of λ satisfies the second primal constraint by definition, so it remains to verify that the first constraint is also satisfied. Indeed, this follows from

where we have used the linear MDP property to write Ψ T Φ T = P T in the first step and that µ is a valid occupancy measure in the last one. Conversely, supposing that ( λ , µ ) ∈ M P Φ are feasible for the relaxed LP, we have that

thus verifying that µ is indeed a valid occupancy measure. Optimality of ( λ ∗ , µ ∗ ) follows from the fact that for any ( λ , µ ) ∈ M P Φ , we can write the LP objective as ⟨ λ , ω ⟩ = ⟨ µ , r ⟩ by the linear MDP assumption, and the standard fact that any solution µ ∗ to the primal LP 1 is the occupancy measure of an optimal policy (cf. Theorem 6.9.4 in Puterman, 1994). This concludes the first part of the proof.

For the second part of the proof, let us first consider a feasible solution v for the original dual LP (2). Then, the choice θ = ω + γ Ψ v satisfies the second dual constraint by definition. The first constraint can be verified by writing

where we used the choice of θ in the first step and the feasibility of v for the original LP in the second step. Conversely, supposing that ( θ , v ) ∈ M D Φ , we note that

which implies the feasibility of v in the LP 2. Optimality of v ∗ for both LPs follows from the fact that their objectives are identical, and the standard fact that v ∗ is an optimal solution of the dual LP (2) (cf. Theorem 6.2.2 in Puterman, 1994).

470

471

472

473

474

475

476

477

478

479

480

B Missing proofs of Section 4

In this section, we provide performance guarantees for Algorithm 1 in terms of the expected suboptimality of the output policy π J , and in particular prove the lemmas provided in Section 4 in the main text. Auxiliary lemmas and technical results for proving some of these are included in Appendix E.

B.1 Properties of the Dynamic Duality Gap

We first prove our claims regarding the dynamic duality gap introduced in Section 4 of the main text. First, we relate the gap to the expected suboptimality (in terms of return) of π J against a comparator policy π ∗ in Appendix B.1.1. Next, we relate the dynamic duality gap to the average regret of each player in Appendix B.1.2.

B.1.1 Proof of Lemma 4.1

By definition of the the dynamic duality gap, we have that

Considering the first term, we see that 481

where we have used ( a ) the linear MDP property (definition 2.1) and ( b ) the following relation: 482

Now for the second term, we have 483

484

485

486

487

where we have used the Bellman equations q π t = Φ θ π t = r + γ Pv π t = Φ ( ω + γ Ψ v π t ) , which together with the fact that Φ is full rank implies that θ π t = ω + γ Ψ v π t . Substituting the above expressions for f ( Φ T µ π ∗ , π ∗ ; θ t ) and f ( λ t , π t ; θ π t ) in the dynamic duality gap and noting that π J is such that 1 T ∑ T t =1 ⟨ µ π t , r ⟩ = E J [ ⟨ µ π J , r ⟩ ] we get

This completes the proof. 488

B.1.2 Proof of Lemma 4.2 489

Recall that for any comparator points ( λ ∗ , π ∗ ; { θ ∗ t } T t =1 ) , the dynamic duality gap is defined as 490

Then, by adding and subtracting some terms we express the dynamic duality gap in terms of the 491 average loss of each player with respect to the objective f ( λ , π ; θ ) . This gives 492

Consider the first set of terms from the above expression. By definition of f in Equation (7), we 493 immediately obtain the instantaneous regret of the π -player as 494

where ν ∗ ( x ) = (1 -γ ) ν 0 ( x ) + γ ⟨ ψ ( x ) , λ ∗ ⟩ . For the regret of the λ and θ -players, notice that we 495 can express the estimator ̂ f in terms of the objective f as follows: 496

Taking advantage of this relation, we now consider the last two set of terms in Equation (11). Indeed, 497 for the second set of terms in the equation, we write 498

Notice that the last equality follows directly from definition of ̂ f . Along these lines, we can also 499 express the last set of terms in Equation (11) as follows: 500

Plugging the above derivations in the dynamic duality gap, we have that 501

This matches the claim of the lemma, thus completing the proof. 502

B.2 Bounding the Regret Terms 503

504

505

In this section we provide the proofs of the our claims made in the main text about the regret of each player-precisely, Lemmas 4.3-4.5.

506

B.2.1 Proof of Lemma 4.3

Consider the regret of the π -player introduced in the main text as, 507

We have used ( a ) the standard Mirror descent analysis of softmax policy iterates recalled in Lemma E.1 for completeness, and ( b ) the fact that π 1 is a uniform policy and ν ∗ ∈ ∆ X . Dividing the above expression by T completes the proof.

B.2.2 Proof of Lemma 4.4

Recall the total regret of the λ -player against any fixed comparator λ ∗ ∈ R d is given as

Since the feature-occupancy updates of Algorithm 1 simply implements a version of the composite513 objective mirror descent scheme due to Duchi et al. [2010] we apply the standard analysis of this 514 method (recalled as Lemma C.1 in Appendix C) to bound the instantaneous regret as 515

Then, taking the sum for t = 1 , · · · , T , evaluating the telescoping sums and upper-bounding some 516 negative terms by zero yields the expression 517

In the equality, we have used that λ 1 = 0 . Dividing the resulting term by T gives the following 518 bound on the average regret: 519

The proof is completed by applying Lemma C.2 to bound the norm of the gradients and plugging the 520 result into the bound above. 521

508

509

510

511

512

B.2.3 Proof of Lemma 4.5 522

For the regret of the θ -player, first note that for any policy π with corresponding state-action value 523 function weights θ π = ω + γ Ψ v π , we have 524

525

526

527

528

529

530

531

532

where we have used the triangle inequality in the second line. The last inequality uses Definition 2.1 and the fact that ∥ v π ∥ ∞ ≤ 1 (1 -γ ) since the rewards are bounded in [0 , 1] . Thanks to this bound, we can ensure that θ ∗ t = θ π t ∈ B d ( D θ ) holds with the choice D θ = √ d/ (1 -γ ) as required by the lemma.Therefore, by construction of value-parameter updates in Algorithm 1, we have

This concludes the proof.

B.3 Bounding the gap-estimation error

In this section, we provide the proof of Lemma 4.6 which bounds the gap-estimation error defined for an arbitrary comparator sequence ( λ ∗ , π t , θ ∗ t ) ∈ R d × Π( D π ) × B d ( D θ ) for t = 1 , . . . , T as,

We control the above term with the now-classic techniques developed by Jin et al. [2020] for bounding 533 model-estimation errors for linear MDPs. These results also make heavy use of self-normalized tail 534 inequalities as popularized by Abbasi-Yadkori et al. [2011] (see also Lattimore and Szepesvári, 2020). 535 R d R X

To make this clear, we first note that, for any λ ∈ , v ∈ , and ξ > 0 , 536

Here, we have first used ( a ) the Cauchy-Schwarz inequality, and ( b ) the inequality of arithmetic and 537 geometric means. Using this expression, we can upper-bound the gap estimation error as 538

To control the last two terms in the bound, we employ two main lemmas stated below. 539

Lemma B.1. Let v ∈ [ -B,B ] X . With probability at least 1 -δ , we have that: 540

Lemma B.2. Consider the function class, 541

Let D π = αTD θ so that v θ t ,π t ∈ V . For any ϵ ∈ (0 , 1) , with probability at least 1 -δ , 542

The rather tedious but otherwise standard proofs of the above lemmas are given in Appendix D. Now, 543 taking into account the fact that for D θ large enough v π t ∈ V yields Corollary B.3 below. 544

Corollary B.3. In the linear MDP setting described in Definition 2.1, notice that v π t = v θ πt ,π t with 545 θ π t = ω + γ Ψ v θ πt ,π t . Furthermore, with RD θ = R √ d/ (1 -γ ) ≥ ∥ v π t ∥ ∞ and D π = αTD θ we 546 have that v π t ∈ V . Therefore, for all ϵ > 0 with probability at least 1 -δ , the following holds: 547

548

549

550

551

552

553

554

555

556

557

558

559

560

In the following, we apply these results to bound the last two terms in the right-hand side of Equation (12). Precisely, using D θ = √ d/ (1 -γ ) , α = √ 2 log A/R 2 D 2 θ T = √ 2 (1 -γ ) 2 log A/R 2 dT (which follows from optimizing the regret of the π -player in Lemma 4.3), as well as ϵ = 4 αR 2 d/ (1 -γ ) 2 = √ 32 R 2 d log A (1 -γ ) √ T and β = R 2 /dT we have that in any round t , with probability at least 1 -δ ,

Then plugging in the above bounds in Equation (12) with T ≥ 2 R 2 n log A log(1 /δ ) , it follows that for D ̂ Ψ = √ 320 d 2 log(2 T/δ ) n (1 -γ ) 2 ,

with probability at least 1 -δ , thus proving the claim.

B.4 Full proof of Theorem 3.1

To control the expected suboptimality of the output policy π J of Algorithm 1, we study the repective regret and gap-estimation error at the selected comparator points. Precisely, combining Lemma 4.1 and 4.2 when ( λ ∗ , π ∗ , θ ∗ 1: T ) = ( Φ T µ π ∗ , π ∗ , θ π t ) ∈ R d × Π( D π ) × B d ( D θ ) , we have that,

where, 561

Notice that for this choice of λ ∗ , by Definition 2.1 ν ∗ ( x ) = (1 -γ ) ν 0 ( x ) + γ ⟨ ψ ( x ) , µ ∗ ⟩ = ν π ∗ ( x ) 562 is a valid state occupancy measure. Next, introducing the bounds stated in Lemmas 4.3-4.6 under 563 the required conditions D θ = √ d/ (1 -γ ) , α = √ 2 (1 -γ ) 2 log A/R 2 dT , D π = αTD θ = 564 √ 2 T log A/R 2 and T ≥ 2 R 2 n log A log(1 /δ ) , as well as ξ ≥ 0 and D ̂ Ψ = √ 320 d 2 log(2 T/δ ) n (1 -γ ) 2 yields, 565

with probability at least 1 -δ , where C = 6 β ( d + D 2 θ ) +3 d (1 + RD θ ) 2 +3 γ 2 dR 2 D 2 θ . Rearranging 566 the bound and selecting ϱ = γ/ξT to eliminate the (potentially large) norm of the iterates, we obtain 567

Furthermore, choosing ξ = 1 /TD ̂ Ψ i.e ϱ = γD ̂ Ψ , we further simplify the above bound on the regret 568 in terms of the optimization error arising from the policy and feature occupancy updates as, 569

Moving our attention to our earlier bound on the norm of g λ ( t ) , 570

The inequality follows from our earlier choice of β = R 2 /dT and that T ≥ 1 /d 2 . Plugging the values 571 of C and D ̂ Ψ in the bound, then choosing η = √ (1 -γ ) 2 27 R 2 d 2 T and using the condition T ≥ 2 R 2 n log A log(1 /δ ) , 572 we have that with probability at least 1 -δ , 573

This completes the proof. 574

C Missing proofs of Section B.2 575

Lemma C.1. (cf. Lemma 1 of Duchi et al. [2010]) Let g λ ( t ) = ω + γ ̂ Ψ v θ t ,π t -θ t Given λ 1 = 0 576 and ϱ, η > 0 and the sequence of iterates { λ t } T t =2 defined for t = 1 , · · · , T as: 577

Then, for any λ ∗ ∈ R d , 578

Proof. The proof of Lemma C.1 follows directly from the referenced Lemma from Duchi et al. 579 [2010]. Consider, 580

We have used 581

582

( a )

583

The first order optimality condition on Equation (14): For any λ ∈ R d ,

( b ) The relation: 584

585

586

( c ) By definition of the squared L 2 -norm for vectors a = Λ -1 / 2 n ( λ t +1 -λ ∗ ) and b = Λ -1 / 2 n ( λ t -λ t +1 ) :

Note that a and b are well defined since Λ n is both symmetric and positive definite. 587

588

Rearranging the terms and plugging in g λ ( t ) = ω + γ ̂ Ψ v θ t ,π t -θ t completes the proof. 589

Finally, we will use the following result that bounds the gradient norms appearing in the bound above. 590

Lemma C.2. Under the conditions of the linear MDP setting we have that, 591

Proof. Recall that for t = 1 , · · · , T g λ ( t ) = ω + γ ̂ Ψ v θ t ,π t -θ t . Then, 592

Now to bound each of the three terms, we use that 593

where the first inequality uses the assumption that ∥ ω ∥ 2 ≤ d (cf. Definition 2.1) and θ t ∈ B d ( D θ ) . 594 Next, we have that 595

The last step follows from the fact that the rewards are bounded in [0 , 1] , ∥ φ i ∥ ≤ R , θ t ∈ B d ( D θ ) 596 and Equation (17). The last remaining term is bounded as 597

Therefore, we obtain 598

and this completes the proof. 599

D Missing proofs of Section B.3 600

In this section, we prove the lemmas stated in Section B.3. 601

D.1 Proof of Lemma B.1 602

By definition of Λ n Section 3 and ̂ Ψ in Equation (8), we can write: 603

In the last equality we used definition 2.1 to write φ T i Ψ = p ( ·| x i , a i ) T . Let ξ i = v ( x ′ i ) -604 ⟨ p ( ·| x i , a i ) , v ⟩ . Then, 605

We easily control the second term with the relation: 606

The last inequality follows from the fact that ∥ ∥ ∥ Λ -1 / 2 n ∥ ∥ ∥ 2 ≤ 1 / √ β and by definition 2.1 ∥ Ψ v ∥ 2 ≤ B √ d for v ∈ [ -B,B ] X .

607

608

Now, to handle the first term, let D 0 = ∅ . We construct a filtration F i -1 = D i -1 ∪ ( x 0 i , x i , a i , r i ) 609 for i = 1 , 2 , · · · , n . Notice that by construction of the dataset ξ i is a martingale difference sequence 610 (i.e E [ ξ i |F i -1 ] = 0 ) taking values in the range [ -2 B, 2 B ] . Then, we can directly apply Lemma E.3 611 to obtain a bound on the first term as: 612

with probability 1 -δ . In the last inequality we have used the AM-GM inequality and bound on the 613 feature vectors: 614

Putting everything together, we have that w.p 1 -δ , 615

616

This completes the proof.

D.2 Proof of Lemma B.2 617

Unlike Lemma B.1, we now aim to control the error term ∥ ∥ ∥ Λ n ( ̂ Ψ -Ψ ) v ∥ ∥ ∥ Λ -1 n when v is random. 618

Also, notice that with π 1 ( a | x ) = e ⟨ φ ( x,a ) , 0 ⟩ ∑ a ′ ∈A e ⟨ φ ( x,a ′ ) , 0 ⟩ as the uniform policy, for t = 1 , · · · , T we have 619 that, 620

where θ 0 = 0 . Furthermore, since { θ t } T t =1 ⊂ B d ( D θ ) , for any t ∥ ∥ ∥ α ∑ t k =0 θ k ∥ ∥ ∥ 2 ≤ αTD θ . Hence, 621 with D π = αTD θ , π t ∈ Π( D π ) and v θ t ,π t ∈ V . 622

Therefore, as we have seen in previous works [Jin et al., 2020, Hong and Tewari, 2024], the quantity 623 ∥ ∥ ∥ Λ n ( ̂ Ψ -Ψ ) v θ t ,π t ∥ ∥ ∥ Λ -1 n can be controlled without any dependence on the size of the state space 624 with a uniform covering argument over V . Let C v be an ϵ -cover of V . That is, for v π t , θ t ∈ V , there 625 exists v ′ ∈ C v such that ∥ v π, θ t -v ′ ∥ ∞ ≤ ϵ . Then, we can write: 626

Consider the first term in the bound. Note that v ′ is still random with respect to uncertainty in the 627 learning process. However, due to the structure of V we know that C v exists and has cardinality 628 log | C v | = O ( d log ( 1 + 4 RD π RD θ ϵ )) (see Lemma E.6). Inspired by Lemma B.1, consider the event: 629

Since C v ⊆ V , we know from Lemma B.1 that P ( E v ) ≤ δ ′ . Now, taking the union bound over the 630 cover C v we have that, 631

Therefore for any v ′ ∈ C v with probability at least 1 -δ , 632

Now, for the second term in Equation (16) we write, 633

We have used ( a ) Jensen's inequality and ( b ) since Λ n ≻ 0 , the relation, 634

For the last term, notice that: 635

This follows from ( a ) Equation (15) since v = v ′ -v θ t ,π t ∈ [ -ϵ, ϵ ] X and ( b ) monotonicity of the 636 square root function as well as Jensen's inequality and ( c ) Equation (17). 637

Finally, plugging the above results back into Equation (16), we have that with probability at least 638 1 -δ , 639

The proof of Lemma B.2 is complete. 640

641

642

643

644

E Auxiliary Lemmas

Lemma E.1. Let q 1 , · · · , q t be a sequence of iterates satisfying ∥ q t ∥ ∞ ≤ RD θ by virtue of definition 2.1 and θ t ∈ B d ( D θ ) . Given an initial policy π 1 and learning rate α > 0 , and sequence of policies { π t } T t =2 defined as:

Then, for any comparator policy π ∗ and ν ∗ some state distribution, 645

The proof of the lemma follows from bounding the regret of the π -player in each state x as 646

647

648

649

650

651

holds for any θ , θ ′ ∈ R d . 652

Proof. Recall that, 653

For π θ , π θ ′ ∈ Π( D π ) using Pinsker's inequality we have that, 654

Furthermore, taking into account the specific structure of the policies, we can write: 655

using that ( a ) the relation, 656

and ( b ) Hoeffding's lemma (cf. Lemma A.1 of Cesa-Bianchi and Lugosi [2006]). The final statement 657 follows from substituting this result in Equation (18). 658

via the application of the standard analysis of the exponentially weighted forecaster of Vovk [1990], Littlestone and Warmuth [1994], Freund and Schapire [1997] (see, e.g., Theorem 2.2 in Cesa-Bianchi and Lugosi, 2006), and noting that ∥ q t ∥ ∞ ≤ RD θ for all t .

Lemma E.2. Suppose that ∥ φ ( x, a ) ∥ 2 ≤ R for all ( x, a ) ∈ X × A . Let π θ , π θ ′ be softmax policies. Then, for all states x ∈ X we have that:

Lemma E.3. (Self-Normalized Bound for Vector-Valued Martingales - Theorem 1 of Abbasi-Yadkori 659 et al. [2011]) Let {F i -1 } ∞ i =1 be a filtration and { ξ i } ∞ i =1 a real-valued stochastic process such that ξ i 660 for i = 1 , · · · is zero-mean (i.e E [ ξ i |F i -1 ] = 0 ) and conditionally s -subgaussian for s ≥ 0 . That is, 661 for all b ∈ R , 662

Also, let { φ i } ∞ i =1 be F i -1 -measurable. Then, 663

Lemma E.4. (e.g. see Chapter 27 of Shalev-Shwartz and Ben-David [2014]) For all ϵ > 0 , 664

Corollary E.5. Under the conditions of Lemma E.2, for all ϵ > 0 , 665

Lemma E.6. Consider the function class, 666

667

we have that:

and, 668

Proof. Let C π denote the ϵ π -cover of Π( D π ) with respect to the norm ∥·∥ ∞ , 1 and C θ the ϵ θ -cover 669 of B d ( D θ ) under the L 2 -norm. For ( π, θ ) ∈ Π( D π ) × B d ( D θ ) and ( π ′ , θ ′ ) ∈ C π × C θ , it follows 670 that for any state x ∈ X , 671

Let C v = { v π, θ : X → [ -RD θ , RD θ ] ∣ ∣ ∣ π ∈ C π , θ ∈ C θ } . Then, C v is an ϵ -cover of V with respect 672 to the L ∞ -norm when ϵ π = ϵ/ 2 RD θ and ϵ θ = ϵ/ 2 R . Therefore, we can derive a bound on the 673 covering number of C v as: 674

Hence, 675

This completes the proof. 676

677

678

679

680

681

682

683

684

685

686

687

688

689

690

691

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

728 729 730 731 732 733 734 735 736 737 738 739 740 741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781

Justification:

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This is a theory paper, there are no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

782

783

784

785

786

787

788

789

790

791

792

793

794

795

796

797

798

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

Answer: [NA]

Justification: This is a theory paper, there is no data or code.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This is a theory paper, there are no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This is a theory paper, there are no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This is a theory paper, there are no experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This is a theory paper, with no specific societal impacts foreseen in the near future.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This is a theory paper, this question is not relevant.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This is a theory paper, there are no assets used.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: This is a theory paper, there are no new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This is a theory paper, we haven't used crowdsourced data or worked with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This is a theory paper, we haven't worked with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper studies the offline policy optimization problem, i.e. to find a policy whose value function is close to the optimal value function using offline samples.  Under the assumption of linear MDP, they proposed a gradient ascent algorithm. 

The sample complexity of the algorithm only depends on the feature coverage of the best policy and does not require coverage over any other policies.

STRENGTHS:
This paper is well written. The algorithm, theorems and lemmas presented in this paper are all very clear.

The problem studied in this paper is offline policy optimization, which has significant value to the community, both empirically and theoretically. 

The algorithm in this paper is simple and computationally tractable.

In contrast to other offline RL paper, this paper does not require that the offline data is sampled i.i.d. or to be admissible, and they can handle arbitrary offline data as long as the data has sufficient coverage to the best policy.

WEAKNESSES:
Compared to Zanette (2021), the algorithm idea is somehow similar. Specifically, both algorithms use the actor-critic update, and the optimization in the algorithm of this paper is similar to the pessimism estimation in Zanette (2021).

The assumption made in this paper is the linear MDP assumption, which is stronger than the assumption in Zanette (2021).

QUESTIONS:
Can your results extend to the case under linear Q-function and linear Bellman completeness assumptions?

LIMITATIONS:
Yes. The authors addressed all the limitations listed in the guidelines.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper propose a new algorithm for offline reinforcement learning in linear infinite-horizon discounted MDP, which achieves a strong sample complexity under the weakest data coverage assumption. Moreover, their algorithm is easy to implement and computationally efficient. Their algorithm design is based on a reduced version of linear programming formulation of MDP, which approximately transforms the original problem into a unconstrained saddle-point optimization problem.\n\nSTRENGTHS:\n1. The algorithm proposed in this paper has many nice properties: it is computationally efficient, easy to implement, and works under the weakest data coverage assumption.\n2. The developed techniques and the observations on transforming the optimization problem is insightful.\n3. The paper is also well-written, with clear proof sketch and is well-positioned among related work.\n\nWEAKNESSES:\nThe paper is pretty notation heavy and a bit hard to follow. I would suggest including a table of notations with descriptions. The choices of notation can also be optimized. For example, I found the mixed use of $D_{\\pi}$ and $D_{\\theta}$ confusing, and they looks like they are dependent on the value of $\\pi$ and $\\theta$.\n\nQUESTIONS:\n1. is there a way to output a single policy (not a uniformly sampled policy) that achieves the same sample complexity?\n2. I am wondering whether your algorithm can be extended to the stochastic shortest path setting [1, 2], which include finite-horizon MDP and discounted infinite-horizon MDP as special cases.\n\n[1] Yin, M., Chen, W., Wang, M., and Wang, Y.-X. Offline stochastic shortest path: Learning, evaluation and towards optimality\n[2] Chen, L., Jain, R., and Luo, H. Improved no-regret algorithms for stochastic shortest path with linear mdp\n\nLIMITATIONS:\nNothing necessary stands out.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

comments: SUMMARY:
This paper provides an approach for solving Offline RL problems in domains where the underlying MDP problem has reward and transition models that are linearly realisable under a known feature map. The paper is well presented.

STRENGTHS:
1. Some of the tricks employed in the approach are quite interesting. 
2. The analytical results are good.

WEAKNESSES:
1. This work is only for MDPs where rewards and transitions are linear. Many of the moderately interesting problems are not linearly realisable, so it is quite important that authors provided a detailed discussion on how it can be addressed. 
2. I am not entirely confident about this, so will wait for inputs from authors. The approach seems to be built based on works by Hong and Tiwari, and stabilisation trick [Neu and okolo, Jacobson et al.]. I was not sure on the key significant contributions of this paper on top of those works. 
3. For me the biggest concern is that there are no experimental results. How would such an approach work for an MDP where the transition and reward models are not linear? Also, the approach is still approximate (given the bound), so would have been important to show the real results.

QUESTIONS:
Apart from the questions mentioned above in weaknesses, can the authors also provide answers to the following questions:
1. Theorem 3.1 is mentioned as the most important results. However, I am not clear on why it is a significant result? Why is this bound good?
2. Also the feature coverage ratio provided by this algorithm, why is it significant? And do others also achieve this ratio for Linear MDPs only or in the more general case?

LIMITATIONS:
There is no limitations mentioned.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-21