Title: Pessimistic Nonlinear Least-Squares Value Iteration for Offline Reinforcement Learning
Venue: NeurIPS 2023
Accepted: no
Keywords: Offline Reinforcement Learning

Markov Decision Process

Nonlinear Function Approximation

Generalized Eluder Dimension
Abstract: Offline reinforcement learning, where the agent aims to learn the optimal policy based on the data collected by a behavior policy, has attracted increasing attention in recent years. While offline RL with linear function approximation has been extensively studied with optimal results being achieved under various assumptions, the theoretical understanding of offline RL with non-linear function approximation is still limited. Specifically, most existing works on offline RL with non-linear function approximation either have a poor dependency on the function class complexity or require an inefficient planning phase. 
    In this paper, we propose an oracle-efficient algorithm VAPVI for offline RL with non-linear function approximation. Our algorithm enjoys a regret bound that has a tight dependence on the function class complexity and achieves minimax optimal instance-dependent regret when specialized to linear function approximation. In our theoretical analysis, we introduce a new coverage assumption for general function approximation, bridging the minimum-eigenvalue assumption and the uncertainty measure widely used in online nonlinear RL. Our algorithmic design includes 1) a variance-based weighted regression scheme for general function classes; 2) a variance estimation subroutine and 3) a pessimistic value iteration planning phase. To the best of our knowledge, this is the first statistically optimal algorithm for nonlinear offline RL. 

Abstract

Offline reinforcement learning, where the agent aims to learn the optimal policy based on the data collected by a behavior policy, has attracted increasing attention in recent years. While offline RL with linear function approximation has been extensively studied with optimal results achieved under certain assumptions, the theoretical understanding of offline RL with non-linear function approximation is still limited. Specifically, most existing works on offline RL with non-linear function approximation either have a poor dependency on the function class complexity or require an inefficient planning phase. In this paper, we propose an oracle-efficient algorithm PNLSVI for offline RL with non-linear function approximation. Our algorithmic design comprises three innovative components: (1) a variance-based weighted regression scheme that can be applied to a wide range of function classes, (2) a subroutine for variance estimation, and (3) a planning phase that utilizes a pessimistic value iteration approach. Our algorithm enjoys a regret bound that has a tight dependency on the function class complexity and achieves minimax optimal problem-dependent regret when specialized to linear function approximation. Our theoretical analysis introduces a new coverage assumption for nonlinear Q function, bridging the minimum-eigenvalue assumption and the uncertainty measure widely used in online nonlinear RL. To the best of our knowledge, this is the first statistically optimal algorithm for nonlinear offline RL.

1 Introduction

Offline reinforcement learning (RL), also known as batch RL, is a learning paradigm where an agent learns to make decisions based on a set of pre-collected data, instead of interacting with the environment in real-time like online RL. The goal of offline RL is to learn a policy that performs well in a given task, based on historical data that was collected from an unknown environment. Recent years have witnessed significant progress in developing offline RL algorithms that can leverage large amounts of data to learn effective policies. These algorithms often incorporate powerful function approximation techniques, such as deep neural networks, to generalize across large state-action spaces. They have achieved excellent performances in a wide range of domains, including the games of Go and chess (Silver et al., 2017; Schrittwieser et al., 2020), robotics (Gu et al., 2017; Levine et al., 2018), and control systems (Degrave et al., 2022).

Several studies have studied the theoretical guarantees of tabular offline RL and proved near-optimal 31 sample complexities in this setting (Xie et al., 2021b; Shi et al., 2022; Li et al., 2022). However, these 32 algorithms cannot handle numerous real-world applications with large state spaces. Consequently, 33 a significant body of research has shifted its focus to offline RL with function approximation. For 34 example, several works have analyzed the sample efficiency of offline RL with linear function 35 approximation under different MDP models, including linear MDPs and their variants (Jin et al., 36 2021b; Zanette et al., 2021; Min et al., 2021; Yin et al., 2022a). To handle nonlinear function class, a 37 recent line of research considered offline RL with general function approximation (Chen and Jiang, 38

Submitted to 37th Conference on Neural Information Processing Systems (NeurIPS 2023). Do not distribute.

2019; Xie et al., 2021a; Zhan et al., 2022). While these algorithms have sample efficiency guarantees, 39 they often require an inefficient planning phase or have a poor dependency on the function class 40 complexity. For example, Xie et al. (2021a) proposed an information-theoretic algorithm that requires 41 solving an optimization problem over all potential policy and corresponding version space, which 42 includes all functions with lower Bellman error. To overcome this limitation, Xie et al. (2021a) 43 proposed a practical implementation, as a cost, the algorithm have a poor dependency on the function 44 class complexity. Recently, (Yin et al., 2022b) studied the general differentiable function class 45 and propose a computation efficient algorithm (PFQL). However, their result also have an addition 46 dependence on the dimension d of the parameter. 47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

Therefore, a natural question arises:

Can we design a computationally efficient algorithm that achieves the minimax optimality with respect to the complexity of nonlinear function class?

We give an affirmative answer to the above question in this work. Our contributions are listed as follows:

We propose a pessimism-based algorithm PNLSVI designed for nonlinear function approximation, which strictly generalizes the existing pessimism-based algorithms for both linear and differentiable function approximation (Xiong et al., 2022; Yin et al., 2022b). Our algorithm is oracle-efficient, i.e., it is computationally efficient when there exists an efficient regression oracle and bonus oracle for the function class (e.g., generalized linear function class).

We prove a data-dependent regret bound with the widely used D 2 -divergence in online nonlinear RL regime, which is optimal with respect to the function class complexity. Our analysis closes the gap to optimality for differentiable function approximation, which was previously an open problem (Yin et al., 2022b).

We introduce a novel uniform coverage assumption for general function approximation that is generalized over the assumption in Yin et al. (2022b). Our assumption bridges between the minimum-eigenvalue assumption used in linear models and the generalized dimension for nonlinear function class, offering new insights into the function approximation problem in RL.

Notation: In this work, we use lowercase letters to denote scalars and use lower and uppercase boldface letters to denote vectors and matrices respectively. For a vector x ∈ R d and matrix Σ ∈ R d × d , we denote by ∥ x ∥ 2 the Euclidean norm and ∥ x ∥ Σ = √ x ⊤ Σx . For two sequences { a n } and { b n } , we write a n = O ( b n ) if there exists an absolute constant C such that a n ≤ Cb n , and we write a n = Ω( b n ) if there exists an absolute constant C such that a n ≥ Cb n . We use ˜ O ( · ) and ˜ Ω( · ) to further hide the logarithmic factors. For any a ≤ b ∈ R , x ∈ R , let [ x ] [ a,b ] denote the truncate function a · 1 ( x ≤ a ) + x · 1 ( a ≤ x ≤ b ) + b · 1 ( b ≤ x ) , where 1 ( · ) is the indicator function. For a positive integer n , we use [ n ] = { 1 , 2 , .., n } to denote the set of integers from 1 to n .

2 Related Work

RL with function approximation. As one of the simplest function approximation classes, linear representation in RL has been extensively studied in recent years (Jiang et al., 2017; Dann et al., 2018; Yang and Wang, 2019; Jin et al., 2020; Wang et al., 2020c; Du et al., 2019; Sun et al., 2019; Zanette et al., 2020a,b; Weisz et al., 2021; Yang and Wang, 2020; Modi et al., 2020; Ayoub et al., 2020; Zhou et al., 2021; He et al., 2021). Several assumptions on the linear structure of the underlying MDPs have been made in these works, ranging from the linear MDP assumption (Yang and Wang, 2019; Jin et al., 2020; Hu et al., 2022; He et al., 2022; Agarwal et al., 2022) to the low Bellman-rank assumption (Jiang et al., 2017) and the low inherent Bellman error assumption (Zanette et al., 2020b). Extending the previous theoretical guarantees to more general problem classes, RL with nonlinear function classes has garnered increased attention in recent years (Wang et al., 2020b; Jin et al., 2021a; Foster et al., 2021; Du et al., 2021; Agarwal and Zhang, 2022; Agarwal et al., 2022). Various complexity measures of function classes have been studied including Bellman rank (Jiang et al., 2017), Bellman-Eluder dimension (Jin et al., 2021a), Decision-Estimation Coefficient (Foster et al., 2021) and generalized Eluder dimension (Agarwal et al., 2022). Among these works, the setting in our paper is most related to Agarwal et al. (2022) where D 2 -divergence (Gentile et al., 2022) was introduced in RL to indicate the uncertainty of a sample with respect to a particular sample batch.

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

Offline tabular RL. There is a line of works integrating the principle of pessimism to develop statistically efficient algorithms for offline tabular RL setting (Rashidinejad et al., 2021; Yin and Wang, 2021; Xie et al., 2021b; Shi et al., 2022; Li et al., 2022). More specifically, Xie et al. (2021b) utilized the variance of transition noise and proposed a nearly optimal algorithm based on pessimism and Bernstein-type bonus. Subsequently, Li et al. (2022) proposed a model-based approach that achieves minimax-optimal sample complexity without burn-in cost for tabular MDPs. Shi et al. (2022) also contributed by proposing the first nearly minimax-optimal model-free offline RL algorithm.

Offline RL with linear function approximation. Jin et al. (2021b) presented the initial theoretical results on offline linear MDPs. They introduced a pessimism-principled algorithmic framework for offline RL and proposed an algorithm based on LSVI (Jin et al., 2020). Min et al. (2021) subsequently considered offline policy evaluation (OPE) in linear MDPs, assuming independence between data samples across time steps to obtain tighter confidence sets and proposed an algorithm with optimal d dependence. Yin et al. (2022a) took one step further and considered the policy optimization in linear MDPs, which implicitly requires the same independence assumption. Zanette et al. (2021) proposed an actor-critic-based algorithm that establishes pessimism principle by directly perturbing the parameter vectors in a linear function approximation framework. Recently, Xiong et al. (2022) proposed a novel uncertainty decomposition technique via a reference function, which leads to a minimax-optimal sample complexity bound for offline linear MDPs without additional assumptions.

Offline RL with general function approximation. Chen and Jiang (2019) critically examined the assumptions underlying value-function approximation methods and established an informationtheoretic lower bound. Xie et al. (2021a) introduced the concept of Bellman-consistent pessimism, which enables sample-efficient guarantees by relying solely on the Bellman-completeness assumption. Uehara and Sun (2021) focused on model-based offline RL with function approximation under partial coverage, demonstrating that realizability in the function class and partial coverage are sufficient for policy learning. Zhan et al. (2022) proposed an algorithm that achieves polynomial sample complexity under the realizability and single-policy concentrability assumptions. Nguyen-Tang and Arora (2023) proposed a method of random perturbations and pessimism for neural function approximation. For differentiable function classes, Yin et al. (2022b) made advancements by improving the sample complexity with respect to the stage H . However, their result had an additional dependence on the dimension d of the parameter space, whereas in linear function approximation, the dependence is typically on √ d .

3 Preliminaries

In our work, we consider the inhomogeneous episodic Markov Decision Processes (MDP), which can be denoted by a tuple of M ( S , A , H, { r h } H h =1 , { P h } H h =1 ) . In specific, S is the state space, A is the finite action space, H is the length of each episode. For each stage h ∈ [ H ] , r h : S×A → [0 , 1] is the reward function 1 and P h ( s ′ | s, a ) is the transition probability function, which denotes the probability for state s to transfer to next state s ′ with current action a . A policy π := { π h } H h =1 is a collection of mappings π h from a state s ∈ S to the simplex of action space A . For simplicity, we denote the state-action pair as z := ( s, a ) . For any policy π and stage h ∈ [ H ] , we define the value function V π h ( s ) and the action-value function Q π h ( s, a ) as the expected cumulative rewards starting at stage h , which can be denoted as follows:

where s h ′ +1 ∼ P h ( ·| s h ′ , a h ′ ) denotes the observed state at stage h ′ +1 . By this definition, the value function V π h ( s ) and action-value function Q π h ( s, a ) are bounded in [0 , H ] . In addition, we define the optimal value function V ∗ h and the optimal action-value function Q ∗ h as V ∗ h ( s ) = max π V π h ( s ) and Q ∗ h ( s, a ) = max π Q π h ( s, a ) . We denote the corresponding optimal policy by π ∗ . For any function V : S → R , we denote [ P h V ]( s, a ) = E s ′ ∼ P h ( ·| s,a ) V ( s ′ ) and [ Var h V ]( s, a ) = [ P h V 2 ]( s, a ) -( [ P h V ]( s, a ) ) 2 for simplicity. For any function f : S → R , we define the Bellman operator T h as T h f ( s h , a h ) = E s h +1 ∼ P h ( ·| s h ,a h ) [ r h ( s h , a h ) + f ( s h +1 )] , where we use the shorthand f ( s ) = max a ∈A f ( s, a ) for simplicity. Based on this definition, for every stage h ∈ [ H ] and policy π , we

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

have the following Bellman equation for value functions Q π h ( s, a ) and V π h ( s ) , as well as the Bellman optimality equation for optimal value functions:

where V π H +1 ( s ) = V ∗ H +1 ( s ) = 0 . We also define the Bellman operator for second moment as T 2 ,h f ( s h , a h ) = E s h +1 ∼ P h ( ·| s h ,a h ) [ ( r h ( s h , a h ) + f ( s h +1 ) ) 2 ] . For simplicity, we omit the subscripts h in the Bellman operator without causing confusion.

Offline Reinforcement Learning: In offline RL, the agent only have access to a batch-dataset D = { s k h , a k h , r k h : h ∈ [ H ] , k ∈ [ K ] } , which is collected by a behavior policy µ , and the agent cannot interact with the environment. Given the batch dataset, the goal of offline RL is finding a near-optimal policy π that minimize the sub-optimality V ∗ 1 ( s ) -V π 1 ( s ) . In addition, for each stage h and behavior policy µ , we denote the induced distribution of the state-action pair as d µ h .

General Function Approximation: In this work, we focus on a special class of episodic MDPs, where the value function satisfies the following completeness assumption.

Assumption 3.1 ( ϵ -completeness under general function approximation, Agarwal et al. 2022) . Given a general function class {F h } h ∈ [ H ] , where each function class F h is composed of functions f h : S × A → [0 , L ] . We assume for each stage h ∈ [ H ] , and any function V : S → [0 , H ] , there exists functions f h , f 2 ,h ∈ F h such that

In addition, for each stage h ∈ [ H ] , we assume there exist a function f ∗ h ∈ F h closed to the optimal value function such that ∥ f ∗ h -Q ∗ h ∥ ∞ ≤ ϵ . For simplicity, we assume L = O ( H ) throughout the paper and denote N = max h ∈ [ H ] |F h | .

To deal with general function class F , Agarwal et al. (2022) introduce the following measure to capture the function class complexity for online learning.

Definition 3.2 (Generalized Eluder dimension, Agarwal et al. 2022) . Given λ > 0 , a sequence of state-action pairs Z = { z i } i ∈ [ K ] and a sequence of non-negative weights σ = { σ i } i ∈ [ K ] . Let F be a function class consisting of functions f : S × A → [0 , L ] . The generalized Eluder dimension of F is given by dim α,K ( F ) := sup Z, σ : | Z | = K, σ ≥ α dim ( F , Z, σ ) , where

Here, the inequality σ ≥ α represents that σ i ≥ α holds for all i ∈ [ K ] and we use the notation z [ i -1] , σ [ i -1] to represent the sequences { z s } i -1 s =1 , { σ s } i -1 s =1 .

However, in offline RL, the proposed Generalized Eluder dimension fails to capture the relationship between function class F and the pre-collected dataset D . To generalize this definition to offline environment, for a batch dataset D = { ( s k h , a k h , r k h ) } H,K h,k =1 and a function class F h consisting of functions f : S × A → R . We denote D h = { ( s k h , a k h , r k h ) } k ∈ [ K ] as the subset of the dataset D that corresponds to the observations collected up to stage h in the MDP. Then for any weight function σ h ( · , · ) : S × A → R , we introduce the following D 2 -divergence:

Data Coverage Assumption: In offline RL, there exist a discrepancy between the state-action 173 distribution generated by the behavior policy and the distribution from the learned policy. Under this 174 situation, the distribution shift problem can cause the learned policy to perform poorly or even fail in 175 offline RL. Therefore, we propose the following data coverage assumption to control the distribution 176 shift. 177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

Algorithm 1 Pessimistic Nonlinear Least-Squares Value Iteration (PNLSVI)

Require: Input confidence parameters β ′ 1 ,h , β ′ 2 ,h , β h and ϵ > 0 .

1: Initialize : Split the input dataset into D = { s k h , a k h , r k h } K,H k,h =1 , D ′ = { ¯ s k h , ¯ a k h , ¯ r k h } K,H k,h =1

2: for stage h = H,.. . , 1 do

; Set the value function ̂ f H +1 ( · ) = ̂ f ′ H +1 ( · ) = 0 .

3: ˜ f ′ h = argmin f h ∈F h ∑ k ∈ [ K ] ( f h (¯ s k h , ¯ a k h ) -¯ r k h -̂ f ′ h +1 (¯ s k h +1 ) ) 2 .

4: ˜ g ′ h = argmin g h ∈F h ∑ k ∈ [ K ] ( g h (¯ s k h , ¯ a k h ) -( ¯ r k h + ̂ f ′ h +1 (¯ s k h +1 ) ) 2 ) 2 .

5: Use the bonus oracle (Definition 4.1) to calculate the bonus function b ′ h = B (1 , D ′ h , F h , ˜ f ′ h , β ′ 1 ,h + β ′ 2 ,h , λ, ϵ ) ,

6: ̂ f ′ h ←{ ˜ f ′ h -b ′ h -ϵ } [0 ,H -h +1] ;

7: Construct the variance estimator

σ

2

{

h

̂

(

s, a

1

,

′

h

˜

g

(

8: end for

9: for stage h = H,.. . , 1 do

10: ˜ f h = argmin f h ∈F h ∑ k ∈ [ K ] 1 ̂ σ 2 h ( s k h ,a k h ) ( f h ( s k h , a k h ) -r k h -̂ f h +1 ( s k h +1 ) ) 2

11: Use the bonus oracle (Definition 4.1) to calculate the bonus function b h = B ( ̂ σ h , D h , F h , ˜ f h , β h , λ, ϵ ) ;

12: ̂ f h ←{ ˜ f h -b h -ϵ } [0 ,H -h +1] ;

13: ̂ π h ( ·| s ) = argmax a ̂ f h ( s, a ) .

14: end for

15: Output: ̂ π = { ̂ π h } H h =1 .

Assumption 3.3 (Uniform Data Coverage) . There exists a constant κ > 0 , such that for any stage h and functions f 1 , f 2 ∈ F h , the following inequality holds,

where the state-action pair (at stage h ) ( s h , a h ) is stochastic generated from behavior policy µ .

Remark 3.4. Data coverage assumption is widely used in offline RL to guarantee that the collected dataset contains enough information of the state-action space to learn an effective policy. In Yin et al. (2022b), they studied the general differentiable function, where the function class is defined as

Under this definition, Yin et al. (2022b) introduce the following coverage assumption (Assumption 2.3) such that for all stage h ∈ [ H ] , there exists a constant κ ,

We can prove that our assumption is weaker than the first assumption (*). For the second assumption (**), there is no direct counterpart in the general setting.

In addition, for the linear function class, the coverage assumption in Yin et al. (2022b) will reduce to the following linear function coverage assumption (Wang et al., 2020a; Min et al., 2021; Yin et al., 2022a; Xiong et al., 2022).

Therefore, our assumption is also weaker than the linear function coverage assumption when dealing with the linear function class. Due to space limitations, we provide the detailed proof in the appendix.

4 Algorithm

In this section, we provide a comprehensive and detailed description of our algorithm (PNLSVI), as displayed in Algorithm 1. In the sequel, we introduce the key ideas of the proposed algorithm.

s, a

)

-

(

f

′

h

˜

(

s, a

2

))

-

O

(

√

log

NN

b

Kκ

√

H

3

)}

.

4.1 Pessimistic Value Iteration Based Planning 196

197

198

199

Our algorithm operates in two distinct phases, Variance Estimate Phase and Pessimistic Planning Phase. At the beginning of the algorithm, the data-set is divided into two disjoint subsets D , D ′ , and each assigned to a specific phase.

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

The basic framework of our algorithm follows the pessimistic value iteration, which was initially introduced by Jin et al. (2021b). In details, for each stage h ∈ [ H ] , we construct the estimator value function ˜ f h by solving the following variance-weighted ridge regression (Line 11):

where ̂ σ 2 h is the estimated variance and will be discussed in section 4.2. In Line 12, we subtract the confidence bonus function b h from the estimator value function ˜ f h to construct the pessimistic value function ̂ f h . With the help of the confidence bonus function b h , the pessimistic value function ̂ f h is almost a lower bound for the optimal value function f ∗ h . The details of the bonus function and bonus oracle will be discussed in section 4.3.

Based on the pessimistic value function ̂ f h for stage h , we recursively perform the value iteration for the stage h -1 . Finally, we use the pessimistic value function ̂ f h to do planning and output the greedy policy with respect to the pessimistic value function ̂ f h (Line 13 - Line 15).

4.2 Variance Estimate Phase

In this phase, we provide a estimator for the variance ̂ σ h in the weighted ridge regression. According to the definition of Bellman operators T and T 2 , the variance of the function ̂ f ′ h +1 for each state-action pair ( s, a ) can be denoted by

Therefore, we need the evaluate the first-order and second-order moments for ̂ f ′ h . We perform nonlinear least-squares regression separately for each of these moments. Specifically, in Line 3, we conduct regression to estimate the first-order moment.

In Line 4, we perform regression for the second-order moment. 218

In this phase, we set the variance function to 1 for each state-action pair ( s, a ) and derive an estimator with confidence radius β ′ 1 ,h , β ′ 2 ,h . Combing these two regression results and subtracting a confidence bonus function b ′ h , we create a pessimistic estimator for the variance function (Lines 6 to 7).

4.3 Nonlinear Bonus Oracle

As we discussed in sections 4.1 and 4.2, we introduce a uncertainty bonus function to construct a pessimistic estimate of the value function. Unfortunately, for a general class, the uncertainty bonus may varies greatly across different state-action pair. Under this situation, the addition uncertainty bonus function will highly increase the complexity of the pessimistic function class, which make it difficult to construct a accurate estimation and may significant deteriorate the final performance. To address this issue, we assume there exist a function class W with cardinally |W| = N b and can approximate the bonus function well. In addition, we assume there exist a nonlinear bonus oracle (Agarwal and Zhang, 2022), which can output the approximate bonus function in the class W for each dataset D h .

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

Definition 4.1 (Oracle for bonus function) . For an offline dataset D = { s k h , a k h , r k h } H,K h,k =1 , given index h ∈ [ H ] , let D h = { ( s k h , a k h , r k h ) } k ∈ [ K ] denote the subset of the dataset D that corresponds to the observations collected up to stage h in the MDP. ̂ σ h ( · , · ) : S × A → R is a variance function. F h is a function class such that ̂ f h ∈ F h . The parameters β h , λ ≥ 0 , error parameter ϵ ≥ 0 . The bonus oracle B ( ̂ σ, D h , F h , ̂ f h , β h , λ, ϵ ) outputs a bonus function b h ( · ) such that

b h : S × A → R ≥ 0 belongs to function class W .

Remark 4.2. To address the concern of function class complexity, some previous studies (Xie et al., 2021a) have approached the problem differently. Instead of introducing a pointwise bonus in the estimated value function, they solve a complicated optimization problem to guarantee the optimism solely in the intial state. This method can prevent the complexity from bonus function, as a cost, they requires solving an optimization problem over all potential policy and corresponding version space, which includes all functions with lower Bellman error.

5 Main Results

In this section we prove an problem-dependent regret bound of Algorithm 1.

Theorem 5.1. Under Assumption 3.3, for K ≥ ˜ Ω ( log( NN b ) H 6 κ 2 ) , if we set the parameters β ′ 1 ,h , β ′ 2 ,h = ˜ O ( √ log NN b H 2 ) and β h = ˜ O ( √ log N ) in Algorithm 1, then with the probability of at least 1 -δ , for any state s ∈ S , we have

where [ V h V ∗ h +1 ]( s, a ) = max { 1 , [ Var h V ∗ h +1 ]( s, a ) } is the truncated conditional variance.

Remark 5.2. When reduce to the linear MDP environment, the following function classes

satisfy the completeness assumption (Assumption 3.1) (Jin et al., 2020). Let F lin h ( ϵ ) be a ϵ -net of the linear function class F lin h . In this case, the covering number satisfies log |F lin h ( ϵ ) | = ˜ O ( d ) and the dependency of function class will reduce to ˜ O ( √ log N ) = ˜ O ( √ d ) . For linear function class, Xiong et al. (2022) proposed the following regret guarantee,

where Σ ∗ h = ∑ k ∈ [ K ] ϕ ( s k h , a k h ) ϕ ( s k h , a k h ) ⊤ / [ V h V ∗ h +1 ]( s k h , a k h ) + λ I . In comparison, we can prove the following inequality:

This shows that Theorem 5.1 matches the optimal result in Xiong et al. (2022) for linear function class.

6 Key Techniques

In this section, we provide an overview of the key techniques in our algorithm design and analysis.

6.1 Variance Estimator with Nonlinear Function Class

The technique of variance-weighted ridge regression, first introduced in Zhou et al. (2021), has demonstrated its effectiveness in the online RL setting with linear function approximation. For offline

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

setting, Xiong et al. (2022) modified the variance-weighted ridge regression technique, and showed that using an accurate and independent variance estimator can improves the performance of the pessimistic value iteration (PEVI) algorithm (Jin et al., 2021b).

In our work, we extend this technique to general nonlinear function class F , and use the following nonlinear least-squares regression to estimate the underlying value function:

For this regression, it is crucial to obtain a reliable evaluation for the variance of the estimated cumulative reward r k h + ̂ f h +1 ( s k h +1 ) . According to the definition of Bellman operators T and T 2 , the variance of the function ̂ f ′ h +1 for each state-action pair ( s, a ) can be denoted by

To evaluate the first and second moment for the Bellman operator, we perform nonlinear least-squares regression on a separate dataset D ′ with uniform weight ( ̂ σ h ( s, a ) = 1 for all state-action pair ( s, a ) ).

For simplicity, we denote the empirical variance as B h ( s, a ) = ˜ g ′ h ( s, a ) -( ˜ f ′ h ( s, a ) ) 2 , and the difference between empirical variance B h ( s, a ) with actually variance [ Var h ̂ f ′ h +1 ]( s, a ) is upper bound by

For these nonlinear function estimator, the following Lemmas provide coarse concentration properties for the first and second order Bellman operators.

Lemma6.1. Given a stage h ∈ [ H ] , let ̂ f ′ h +1 ( · , · ) ≤ H be the estimated value function constructed in Algorithm 1 Line 6. By utilizing Assumption 3.1, there exists a function ¯ f ′ h ∈ F h , such that | ¯ f ′ h ( z h ) -T h ̂ f ′ h +1 ( z h ) | ≤ ϵ holds for all state-action pair z h = ( s h , a h ) . Then with the probability of at least 1 -δ/ 4 H , it holds that ∑ k ∈ [ K ] ( ¯ f ′ h (¯ z k h ) -˜ f ′ h (¯ z k h ) ) 2 ≤ ( β ′ 1 ,h ) 2 , where β ′ 1 ,h = ˜ O ( √ log NN b H 2 ) , and ˜ f ′ h is the estimated function for first-moment Bellman operator (Line 3 in Algorithm 1).

Lemma6.2. Given a stage h ∈ [ H ] , let ̂ f ′ h +1 ( · , · ) ≤ H be the estimated value function constructed in Algorithm 1 Line 6. By utilizing Assumption 3.1, there exists a function ¯ g ′ h ∈ F h , such that | ¯ g ′ h ( z h ) -T 2 ,h ̂ f ′ h +1 ( z h ) | ≤ ϵ holds for all state-action pair z h = ( s h , a h ) . Then with the probability of at least 1 -δ/ 4 H , it holds that ∑ k ∈ [ K ] ( ¯ g ′ h (¯ z k h ) -˜ g ′ h (¯ z k h ) ) 2 ≤ ( β ′ 2 ,h ) 2 , where β ′ 2 ,h = ˜ O ( √ log NN b H 2 ) , and ˜ g ′ h is the estimated function for second-moment Bellman operator (Line 4 in Algorithm 1).

Notice that all of the previous analysis focuses on the estimated function ̂ f ′ h +1 . By leveraging an induction procedure similar to existing works in the linear case (Jin et al., 2021b; Xiong et al., 2022), we can control the distance between the estimated function ̂ f ′ h +1 and the optimal value function f ∗ h . In details, with high probability, for all stage h ∈ [ H ] , the distance is upper bounded by O ( √ log NN b H 3 / √ Kκ ) . This result allows us to further bound [ Var h ̂ f ′ h +1 ]( s, a ) and [ Var h f ∗ h +1 ]( s, a ) .

Therefore, the concentration properties in Lemmas 6.1 and 6.2 enable us to construct the pessimistic variance estimator, which satisfies the following property:

where [ V h V ∗ h +1 ]( s, a ) = max { 1 , [ Var h V ∗ h +1 ]( s, a ) } is the truncated conditional variance. Compared 299 with the results in the linear function class, we utilize the logarithm of the covering number of the 300 function class as a substitute for the linear dimension d , which is a common technique in nonlinear 301 function approximation. 302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

6.2 Reference-Advantage Decomposition

The reference-advantage decomposition is a powerful technique to tackle the challenge of additional error from uniform concentration over whole function class F h . Such an analysis approach has been first studied in the online RL setting Azar et al. (2017); Zhang et al. (2021); Hu et al. (2022); He et al. (2022); Agarwal et al. (2022) and later in the offline environment by Xiong et al. (2022).

For offline RL, in the context of nonlinear function classes, without a explicit linear expression, the increased complexity of the function class structure poses a significant obstacle to effectively utilizing this technique. Previous works, such as Yin et al. (2022b), have struggled to adapt the reference-advantage decomposition to their nonlinear function class, resulting in a parameter space dependence that scales with d , instead of the optimal √ d . We provide detailed insights into this approach as follows:

We decompose the Bellman error into two parts: the Reference uncertainty and the Advantage uncertainty. For the first term, the optimal value function f ∗ h +1 is fixed and not related to the precollected dataset, which circumvents additional uniform concentration over the whole function class and avoid any dependence on the function class size. For the second term, it is worth to notice that the distance between the estimated function ̂ f ′ h +1 and the optimal value function f ∗ h is decreased as O (1 / √ Kκ ) . Though, we still need to maintain the uniform convergence guarantee, the Advantage uncertainty is dominated by the Reference uncertainty when the number of episode K is large enough. By integrating these results, we can prove a variance-weighted concentration inequality for Bellman operators.

Lemma 6.3. For each stage h ∈ [ H ] , assuming the variance estimator ̂ σ h satisfies (6.1), let ̂ f h +1 ( · , · ) ≤ H be the estimated value function constructed in Algorithm 1 Line 12. ing Assumption 3.1, there exist a function ¯ f h ∈ F h , such that | ¯ f h ( z h ) - T h ̂ f h +1 ( z h ) | ≤ ϵ holds for all state-action pair z = ( s , a ) . Then with the probability of at least 1 -δ/ 4 H , it holds that

∑ k ∈ [ K ] 1 ( ̂ σ h ( z k h )) 2 ¯ f h ( z k h ) -˜ f h ( z k h ) ≤ ( β h ) 2 , where β h = ˜ O ( log N ) and ˜ f h is the estimated function from the weighted ridge regression (Line 10 in Algorithm 1).

By utilizh h h ( ) 2 √

After controlling the Bellman error, with a similar argument to Jin et al. (2021b); Xiong et al. (2022), we obtain the following lemma, which provide an upper bound for the regret.

Lemma 6.4 (Regret Decomposition Property) . If |T h ̂ f h +1 ( z ) -˜ f h ( z ) | ≤ b h ( z ) holds for all stage h ∈ [ H ] and state-action pair z = ( s, a ) ∈ S × A , then the regret of Algorithm 1 can be bounded as

Here, the expectation E π ∗ is with respect to the trajectory induced by π ∗ in the underlying MDP.

Combing the results in Lemmas 6.3 and 6.4, we have proved Theorem 5.1.

7 Conclusion and Future Work

In this paper, we present PNLSVI, an oracle-efficient algorithm for offline RL with non-linear function approximation. It achieves minimax optimal problem-dependent regret when specialized to linear function approximation.

Regarding future work, we observe that instead of using the uniform coverage assumption, a series of works, such as (Liu et al., 2020; Xie et al., 2021a; Uehara and Sun, 2021; Zhan et al., 2022), only relies on partial coverage assumption. In these works, the offline data distribution only encompasses the state-action distribution of a select high-quality comparator policy π ∗ . It would be of significant interest to investigate whether it's possible to design practical algorithms for nonlinear function classes under this weaker partial coverage assumption, while still preserving the inherent efficiency found in linear function approximation.

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

References

AGARWAL, A., JIN, Y. and ZHANG, T. (2022). Vo q l: Towards optimal regret in model-free rl with nonlinear function approximation. arXiv preprint arXiv:2212.06069 .

AGARWAL, A. and ZHANG, T. (2022). Model-based rl with optimistic posterior sampling: Structural conditions and sample complexity. arXiv preprint arXiv:2206.07659 .

AYOUB, A., JIA, Z., SZEPESVARI, C., WANG, M. and YANG, L. (2020). Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning . PMLR.

AZAR, M. G., OSBAND, I. and MUNOS, R. (2017). Minimax regret bounds for reinforcement learning. In International Conference on Machine Learning . PMLR.

CHEN, J. and JIANG, N. (2019). Information-theoretic considerations in batch reinforcement learning. In International Conference on Machine Learning . PMLR.

DANN, C., JIANG, N., KRISHNAMURTHY, A., AGARWAL, A., LANGFORD, J. and SCHAPIRE, R. E. (2018). On oracle-efficient pac rl with rich observations. Advances in neural information processing systems 31 .

DEGRAVE, J., FELICI, F., BUCHLI, J., NEUNERT, M., TRACEY, B., CARPANESE, F., EWALDS, T., HAFNER, R., ABDOLMALEKI, A., DE LAS CASAS, D. ET AL. (2022). Magnetic control of tokamak plasmas through deep reinforcement learning. Nature 602 414-419.

DU, S., KAKADE, S., LEE, J., LOVETT, S., MAHAJAN, G., SUN, W. and WANG, R. (2021). Bilinear classes: A structural framework for provable generalization in rl. In International Conference on Machine Learning . PMLR.

DU, S. S., KAKADE, S. M., WANG, R. and YANG, L. F. (2019). Is a good representation sufficient for sample efficient reinforcement learning? arXiv preprint arXiv:1910.03016 .

FOSTER, D. J., KAKADE, S. M., QIAN, J. and RAKHLIN, A. (2021). The statistical complexity of interactive decision making. arXiv preprint arXiv:2112.13487 .

GENTILE, C., WANG, Z. and ZHANG, T. (2022). Achieving minimax rates in pool-based batch active learning. In International Conference on Machine Learning . PMLR.

GU, S., HOLLY, E., LILLICRAP, T. and LEVINE, S. (2017). Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates. In 2017 IEEE international conference on robotics and automation (ICRA) . IEEE.

HE, J., ZHAO, H., ZHOU, D. and GU, Q. (2022). Nearly minimax optimal reinforcement learning for linear markov decision processes. arXiv preprint arXiv:2212.06132 .

HE, J., ZHOU, D. and GU, Q. (2021). Logarithmic regret for reinforcement learning with linear function approximation. In International Conference on Machine Learning . PMLR.

HU, P., CHEN, Y. and HUANG, L. (2022). Nearly minimax optimal reinforcement learning with linear function approximation. In International Conference on Machine Learning . PMLR.

JIANG, N., KRISHNAMURTHY, A., AGARWAL, A., LANGFORD, J. and SCHAPIRE, R. E. (2017). Contextual decision processes with low bellman rank are pac-learnable. In International Conference on Machine Learning . PMLR.

JIN, C., LIU, Q. and MIRYOOSEFI, S. (2021a). Bellman eluder dimension: New rich classes of rl problems, and sample-efficient algorithms. Advances in neural information processing systems 34 13406-13418.

JIN, C., YANG, Z., WANG, Z. and JORDAN, M. I. (2020). Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory . PMLR.

JIN, Y., YANG, Z. and WANG, Z. (2021b). Is pessimism provably efficient for offline rl? In International Conference on Machine Learning . PMLR.

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

LEVINE, S., PASTOR, P., KRIZHEVSKY, A., IBARZ, J. and QUILLEN, D. (2018). Learning handeye coordination for robotic grasping with deep learning and large-scale data collection. The International journal of robotics research 37 421-436.

LI, G., SHI, L., CHEN, Y., CHI, Y. and WEI, Y. (2022). Settling the sample complexity of model-based offline reinforcement learning. arXiv preprint arXiv:2204.05275 .

LIU, Y., SWAMINATHAN, A., AGARWAL, A. and BRUNSKILL, E. (2020). Provably good batch off-policy reinforcement learning without great exploration. Advances in neural information processing systems 33 1264-1274.

MIN, Y., WANG, T., ZHOU, D. and GU, Q. (2021). Variance-aware off-policy evaluation with linear function approximation. Advances in neural information processing systems 34 7598-7610.

MODI, A., JIANG, N., TEWARI, A. and SINGH, S. (2020). Sample complexity of reinforcement learning using linearly combined model ensembles. In International Conference on Artificial Intelligence and Statistics . PMLR.

NGUYEN-TANG, T. and ARORA, R. (2023). Viper: Provably efficient algorithm for offline rl with neural function approximation. In The Eleventh International Conference on Learning Representations .

RASHIDINEJAD, P., ZHU, B., MA, C., JIAO, J. and RUSSELL, S. (2021). Bridging offline reinforcement learning and imitation learning: A tale of pessimism. Advances in Neural Information Processing Systems 34 11702-11716.

SCHRITTWIESER, J., ANTONOGLOU, I., HUBERT, T., SIMONYAN, K., SIFRE, L., SCHMITT, S., GUEZ, A., LOCKHART, E., HASSABIS, D., GRAEPEL, T. ET AL. (2020). Mastering atari, go, chess and shogi by planning with a learned model. Nature 588 604-609.

SHI, L., LI, G., WEI, Y., CHEN, Y. and CHI, Y. (2022). Pessimistic q-learning for offline reinforcement learning: Towards optimal sample complexity. In International Conference on Machine Learning . PMLR.

SILVER, D., SCHRITTWIESER, J., SIMONYAN, K., ANTONOGLOU, I., HUANG, A., GUEZ, A., HUBERT, T., BAKER, L., LAI, M., BOLTON, A. ET AL. (2017). Mastering the game of go without human knowledge. nature 550 354-359.

SUN, W., JIANG, N., KRISHNAMURTHY, A., AGARWAL, A. and LANGFORD, J. (2019). Modelbased rl in contextual decision processes: Pac bounds and exponential improvements over modelfree approaches. In Conference on learning theory . PMLR.

UEHARA, M. and SUN, W. (2021). Pessimistic model-based offline reinforcement learning under partial coverage. arXiv preprint arXiv:2107.06226 .

WANG, R., FOSTER, D. P. and KAKADE, S. M. (2020a). What are the statistical limits of offline rl with linear function approximation? arXiv preprint arXiv:2010.11895 .

WANG, R., SALAKHUTDINOV, R. R. and YANG, L. (2020b). Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension. Advances in Neural Information Processing Systems 33 6123-6135.

WANG, Y., WANG, R., DU, S. S. and KRISHNAMURTHY, A. (2020c). Optimism in reinforcement learning with generalized linear function approximation. In International Conference on Learning Representations .

WEISZ, G., AMORTILA, P. and SZEPESVÁRI, C. (2021). Exponential lower bounds for planning in mdps with linearly-realizable optimal action-value functions. In Algorithmic Learning Theory . PMLR.

XIE, T., CHENG, C.-A., JIANG, N., MINEIRO, P. and AGARWAL, A. (2021a). Bellman-consistent pessimism for offline reinforcement learning. Advances in neural information processing systems 34 6683-6694.

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

XIE, T., JIANG, N., WANG, H., XIONG, C. and BAI, Y. (2021b). Policy finetuning: Bridging sampleefficient offline and online reinforcement learning. Advances in neural information processing systems 34 27395-27407.

XIONG, W., ZHONG, H., SHI, C., SHEN, C., WANG, L. and ZHANG, T. (2022). Nearly minimax optimal offline reinforcement learning with linear function approximation: Single-agent mdp and markov game. arXiv preprint arXiv:2205.15512 .

YANG, L. and WANG, M. (2019). Sample-optimal parametric q-learning using linearly additive features. In International Conference on Machine Learning .

YANG, L. and WANG, M. (2020). Reinforcement learning in feature space: Matrix bandit, kernels, and regret bound. In International Conference on Machine Learning . PMLR.

YIN, M., DUAN, Y., WANG, M. and WANG, Y.-X. (2022a). Near-optimal offline reinforcement learning with linear representation: Leveraging variance information with pessimism. arXiv preprint arXiv:2203.05804 .

YIN, M., WANG, M. and WANG, Y.-X. (2022b). Offline reinforcement learning with differentiable function approximation is provably efficient. arXiv preprint arXiv:2210.00750 .

YIN, M. and WANG, Y.-X. (2021). Towards instance-optimal offline reinforcement learning with pessimism. Advances in neural information processing systems 34 4065-4078.

ZANETTE, A., BRANDFONBRENER, D., BRUNSKILL, E., PIROTTA, M. and LAZARIC, A. (2020a). Frequentist regret bounds for randomized least-squares value iteration. In International Conference on Artificial Intelligence and Statistics . PMLR.

ZANETTE, A., LAZARIC, A., KOCHENDERFER, M. and BRUNSKILL, E. (2020b). Learning near optimal policies with low inherent bellman error. In International Conference on Machine Learning . PMLR.

ZANETTE, A., WAINWRIGHT, M. J. and BRUNSKILL, E. (2021). Provable benefits of actor-critic methods for offline reinforcement learning. Advances in neural information processing systems 34 13626-13640.

ZHAN, W., HUANG, B., HUANG, A., JIANG, N. and LEE, J. (2022). Offline reinforcement learning with realizability and single-policy concentrability. In Conference on Learning Theory . PMLR.

ZHANG, Z., JI, X. and DU, S. (2021). Is reinforcement learning more difficult than bandits? a near-optimal algorithm escaping the curse of horizon. In Conference on Learning Theory . PMLR.

ZHOU, D., GU, Q. and SZEPESVARI, C. (2021). Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory . PMLR.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies offline RL with non-linear function approximation. The paper is mainly motivated as existing sample complexity guarantees on offline RL algorithms with general function approximation yield suboptimal dependency on the function class complexity, e.g. when the bounds are translated to the linear case. The paper proposes an oracle-efficient algorithm that achieves minimax optimal problem-dependent regret when the bounds are specialized to the linear case. The paper also introduces a new coverage definition.\n\nSTRENGTHS:\n- The paper appears to be technically sound with some new ideas in the algorithm design and formulation of dataset coverage.\n- The approach achieve minimax optimal rate in non-linear function approximation, when bounds are converted to linear.\n\nWEAKNESSES:\n- The main weakness is that the proposed approach either requires uniform coverage or non-linear bonus oracle. The non-linear bonus oracle is a strong requirement and in effect, simply removes the difficulties related to pessimism in offline RL. On the other hand, the uniform coverage assumption is too strong and thus, it is unfair to compare its efficiency to pessimistic offline RL algorithms.\n- A clear comparison to prior work is not presented. In particular, there are multiple axes of comparison, such as dependency on $\\epsilon$, dependency on function classes, data coverage requirement, type of oracle, computational efficiency/tractability, realizability assumptions, etc. It is difficult to clearly evaluate the results in this paper without such comparisons. For instance, it will be helpful to have a table as well as translating the bounds of the other algorithms into linear case to see in detail. Additionally, there are several pessimistic offline RL algorithms with general function approximation that only require optimization oracles instead of the more difficult bonus oracle, and no comparison with those papers are presented:\n\nCheng et al. Adversarially trained actor critic for offline reinforcement learning. In International Conference on Machine Learning (pp. 3852-3878). PMLR\n\nRashidinejad et al. \"Optimal conservative offline rl with general function approximation via augmented lagrangian.\" arXiv preprint arXiv:2211.00716 (2022).\n\nOzdaglar et al. Revisiting the Linear-Programming Framework for Offline RL with General Function Approximation. arXiv preprint arXiv:2212.13861\n\nZhu et al. Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning. arXiv preprint arXiv:2301.12714.\n\nQUESTIONS:\n- Is the suboptimality of dependency on the function-class complexity in the bounds of prior algorithms with general function approximation inherent to the algorithm design or a byproduct of analysis?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers variance-weighted least-squared regression for offline RL with general function approximation. Under a uniform data coverage assumption, they show that the proposed algorithm obtains a sub-optimality bound that scales with the $D^2$-divergence of the offline data set, the positive lower-bounded constant of the uniform data coverage, and the complexity of the function class. Their bound obtains the right order when realized in the linear case.\n\nSTRENGTHS:\n- clear presentation (though some parts can be improved further -- see Weaknesses)\n- the obtained result is new and relevant to the offline RL community\n\nWEAKNESSES:\n- The main weakness is that the uniform data coverage assumption is very strong. In the linear case, this assumption is equivalent to that the behavior policy is exploratory overall dimensions of the linear feature. A question for the authors is that in such a case, why would we even need pessimism?  Pessimism is used when the data coverage is partial thus we become pessimistic about uncertain actions. But when the coverage is uniform, it can eliminate the need for pessimism and we can simply use greedy algorithms. I understand that without such a uniform data coverage assumption, it seems difficult to get a reliable estimation of the variance of the transition kernel and it would be interesting to get rid of this assumption. But if we could not get rid of it yet, the very least expectation is that we need to explain this assumption further, especially regarding where pessimism is really needed with this assumption. \n\n\n- Writing can be improved further. For example, the $D^2$-divergence and the definition of the bonus function (Def 4) can be explained and motivated further. The current presentation of these concepts are not very helpful \n\n- Some claims might be potentially misleading. It's not comfortable to view the proposed algorithm as computationally efficient even in the oracle sense. Specifically, the construction of the bonus function in Definition 4.1 is far from being computationally efficient since it is essentially a constrained optimization over the version space. That said, it is nowhere more computationally efficient than version-space-based algorithms such as the \"Bellman-consistent pessimism\" of Xie et al. \n\n- Though the main result is new, it appears expected given the already-developed machinery in Argawal et al. 2022 and Xiong et al. 2022. What are the technical challenges in the current problem that the existing techniques cannot resolve? \n\n- Some minor: PNLSVI is never introduced before used\n\nQUESTIONS:\nPlease see the Weaknesses section\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a pessimistic nonlinear least-squares value iteration algorithm to tackle the offline reinforcement learning problem. The main motivation of the paper is to propose an algorithm that are both computationally efficient and minimax optimal w.r.t. the complexity of nonlinear function class. The proposed pessimism-based algorithm strictly generalizes the existing pessimism-based algorithms for both linear and differentiable function approximation and is oracle efficient. Also, the proposed algorithm is proven to be optimal w.r.t. the function class complexity, closing the gap originated from the previous work on differentiable function approximation.\n\nSTRENGTHS:\n1) The proposed algorithm is proven to be optimal w.r.t. the complexity of nonlinear function class, closing the gap from the previous work on the differentiable function class and generalizes it to the wider nonlinear function class.\n 2) The proposed algorithm is computationally efficient if there exist the efficient oracles for both regression minimization and bonus function optimization/searching.\n\nWEAKNESSES:\n1) The paper's presentation needs some work. For example, the terminology definition is not consistent. The D^2 divergence definition in Definition 3.2 is not consistent with the later terminology of D_F in line 239. The language itself needs some work too. For example, lots of places where it needs 'an', but 'a' is used and vice versa. Please define RL before using it in the abstract. There are also some ambiguities in the definitions that needs clarification in the Question section. \n2) The paper's claimed contribution is a bit exaggerated. Although the proposed algorithm does not need the computationally heavy optimization as previous works in planning phase, it transfers the main computation burden to the Oracle to find the satisfied bonus function, which seems to be a very time-consuming task. It also applies to the claim of being the first statistically optimal algorithm for nonlinear offline RL. Being able to get optimal result in the reduced linear function class does not necessarily mean it's optimal in the broader nonlinear class.\n3) Although the considered class is the nonlinear one and general than the previously considered linear or differentiable class, the techniques used in the analysis are nothing new in my opinion, except re-defining the metrics in the nonlinear function class and connect the results together along with additional assumptions.\nOverall, I think the paper is well motivated, but given the presentation and the insignificant contribution, it's not ready to be published.\n\nQUESTIONS:\n1) In line 137, the paper defines the Bellman operator for the function f: S->R. Then why in the operator, it takes both state and action as input?\n2) In line 141, should we replace the V with Q?\n3) In line 172, the generalized definition to offline setting, If D_h corresponds to the observations collected up to stage h in the MDP, what is z_h^k? Is it only one (state,action) pair or the observations collected up to stage h?\n4) Line 214, should the function be all \\hat{f^{\\prime}}?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-06"}