Title: Beyond Stationarity: Convergence Analysis of Stochastic Softmax Policy Gradient Methods
Venue: NeurIPS 2023
Accepted: no
Keywords: reinforcement learning

policy gradient

stochastic approximation

finite-time MDP
Abstract: Markov Decision Processes (MDPs) deliver a formal framework for modeling and solving sequential decision-making problems. In this paper, we make several contributions towards the theoretical understanding of (stochastic) policy gradient methods for MDPs. The focus lies on proving convergence (rates) of softmax policy gradient towards global optima in undiscounted finite-time horizon problems, i.e. $\gamma=1$, without regularization. Such problems are relevant for instance for optimal stopping or specific supply chain problems. Our estimates must differ significantly from several recent articles that involve powers of $(1-\gamma)^{-1}$.

The main contributions are the following. For undiscounted finite-time MDPs we prove asymptotic convergence of policy gradient to a global optimum and derive a convergence rate using a weak Polyak-\L ojasiewicz (PL) inequality. In each decision epoch, the derived error bound depends linearly on the remaining duration of the MDP. In the second part of the analysis, we quantify the convergence behavior for the stochastic version of policy gradient. The analysis yields complexity bounds for an approximation arbitrarily close to the global optimum with high probability. 

As a by-product, our stochastic gradient arguments prove that the plain vanilla REINFORCE algorithm for softmax policies indeed approximates global optima for sufficiently large batch sizes.

Abstract

Markov Decision Processes (MDPs) deliver a formal framework for modeling and solving sequential decision-making problems. In this paper, we make several contributions towards the theoretical understanding of (stochastic) policy gradient methods for MDPs. The focus lies on proving convergence (rates) of softmax policy gradient towards global optima in undiscounted finite-time horizon problems, i.e. γ = 1 , without regularization. Such problems are relevant for instance for optimal stopping or specific supply chain problems. Our estimates must differ significantly from several recent articles that involve powers of (1 -γ ) -1 .

The main contributions are the following. For undiscounted finite-time MDPs we prove asymptotic convergence of policy gradient to a global optimum and derive a convergence rate using a weak Polyak-Łojasiewicz (PL) inequality. In each decision epoch, the derived error bound depends linearly on the remaining duration of the MDP. In the second part of the analysis, we quantify the convergence behavior for the stochastic version of policy gradient. The analysis yields complexity bounds for an approximation arbitrarily close to the global optimum with high probability.

As a by-product, our stochastic gradient arguments prove that the plain vanilla REINFORCE algorithm for softmax policies indeed approximates global optima for sufficiently large batch sizes.

1 Introduction

Policy gradient methods continue to enjoy great popularity in practice due to their model-free nature and high flexibility. Despite their far-reaching history (Williams, 1992; Sutton et al., 1999; Konda and

Tsitsiklis, 1999; Kakade, 2001), there were no proofs for the global convergence of these algorithms

for a long time. Nevertheless, they have been very successful in many applications, which is why

numerous variants have been developed in the last few decades, whose convergence analysis, if

available, is mostly limited to convergence to stationary points (Pirotta et al., 2013; Schulman et al., 25

26

2015; Papini et al., 2018; Clavera et al., 2018; Shen et al., 2019; Xu et al., 2020b; Huang et al., 2020;

Xu et al., 2020a; Huang et al., 2022). 27

In recent years, notable advancements have been achieved in the convergence analysis towards 28

global optima (Fazel et al., 2018; Agarwal et al., 2021; Mei et al., 2020; Bhandari and Russo, 2021, 29

2022; Cen et al., 2022; Xiao, 2022; Alfano and Rebeschini, 2023). These achievements are partially 30

attributed to the utilization of (weak) gradient domination or Polyak-Łojasiewicz (PL) inequalities 31

(Polyak, 1963). As examined in Karimi et al. (2016) a PL-inequality and smoothness implies a 32

linear convergence rate for gradient descent methods. In certain cases, only a weaker form of the 33

PL inequality can be derived, which states that it is only possible to limit the norm of the gradient 34

instead of the squared norm of the gradient by the distance to the optimum. Despite this limitation, 35

O (1 /n ) -convergence can still be achieved in some instances. 36

The research community has predominantly focused on discounted Markov decision processes 37 (MDPs) with infinite time horizon: ( S , A , p, r, γ ) is an MDP, where S is a finite state space, A 38 is a finite action space, p is a transition function such that p ( s ′ | s, a ) denotes the probability of 39 transitioning from state s to state s ′ under action a . The reward function is given by r : S × A → R , 40 where R ⊆ R is usually assumed to be bounded and positive, and γ ∈ (0 , 1) is a discount factor. The 41 value function under consideration takes the form 42

for all s ∈ S . Investigating a stationary policy applied in every time point suffices for discounted 43 MDPs (Puterman, 2005, Theorem 6.1.1). Yet, in this paper, we focus on MDPs with finite-time 44 horizons and without a discount factor, i.e., γ = 1 . There is a prevailing argument that finite-time 45 MDPs do not require additional scrutiny as they can be transformed into infinite horizon MDPs. 46 However, specific challenges arise in certain scenarios, such as optimal stopping (Li et al., 2009) or 47 finite-time inventory control problems (Bhandari and Russo, 2022), where a non-stationary policy 48 becomes necessary. Unlike in infinite time horizon MDPs, reducing the problem to stationary policies 49 is inadequate for finite-time MDPs, and a new policy must be trained recursively at each time step 50 (Puterman, 2005). Our convergence analysis comprises two steps: firstly, we investigate convergence 51 at each time step and secondly, we examine the error accumulation through backward induction. A 52 detailed discussion of finite-time MDPs is presented in Section 2. There are some recent articles also 53 studying policy gradient of finite-time horizon MDPs considering fictitious discount algorithms (Guo 54 et al., 2022) or finite-time linear quadratic control problems (Hambly et al., 2021, 2022; Zhang et al., 55 2021). 56

57

58

59

60

61

62

63

We begin with a discussion of relevant results for discounted MDPs that encourage our contributions. In Agarwal et al. (2021), the global asymptotic convergence of policy gradient is demonstrated under tabular softmax parametrization, and convergence rates are derived using log-barrier regularization and natural policy gradient. Building upon this work, Mei et al. (2020) showed the first convergence rates for policy gradient using non-uniform PL-inequalities (Mei et al., 2021), specifically for tabular softmax parametrization. However, this convergence rate is fundamentally dependent on the discount factor, (1 -γ ) -6 , and cannot be readily extrapolated to undiscounted MDPs with finite-time horizons.

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

To bridge this gap, we consider policy gradient under tabular softmax parametrization, but in undiscounted MDPs with finite-time horizons and non-stationary policies. In Section 3, we show asymptotic convergence to a global optimum and subsequently derive a global convergence rate using a weaker form of the PL-inequality. The convergence rate at a fixed time point is linearly depending on the remaining duration of the MDP, which is a better property compared to (1 -γ ) -6 . The issue of dependency on γ when it approaches 1 is a significant subject in the context of discounting, and various efforts have been made to mitigate this dependency. For instance, employing entropy regularization as demonstrated in Mei et al. (2020) or applying mirror descent as described in Xiao (2022) can enhance the rate of convergence.

In the second part of the paper, we abandon the assumption that the exact gradient is known and focus on the model free stochastic policy gradient method. For this type of algorithm, very little is known even in the discounted case. Agarwal et al. (2021) discussed the approximate natural policy gradient for log-linear policies, and Ding et al. (2022) derived complexity bounds for entropy-regularized stochastic policy gradient. They use a well-chosen stopping time which measures the distance to the set of optimal parameters, and simultaneously guarantees convergence to the regularized optimum prior to the occurrence of the stopping time by using a small enough step size and large enough batch size. Similar to this idea, we construct a different stopping time in this work, which allows us to analyze convergence of the stochastic policy gradient method in the finite, non-stationary case and also in the infinite discounted case without regularization. The stopping time we propose measures the distance between the policy gradient and stochastic policy gradient trajectories and stops when the stochastic gradient differs too far from the exact gradient updates. This allows us to derive complexity bounds for an approximation arbitrarily close to the global optimum that does not require a set of optimal parameters, which is relevant when considering softmax parametrization.

To the best of our knowledge, the results presented in this paper provide the first convergence analysis 87 of softmax policy gradient in the undiscounted finite-time MDP setting without regularization. We 88 note that discussions in Bhandari and Russo (2022) do not apply to softmax parametrization, as they 89 assume the existence of optimal parameters in the parameter space. 90

The remainder of this manuscript is structured as follows: In Section 2, we discuss finite-time 91 MDPs and explain how to solve them using backward induction. In Section 3, we show asymptotic 92 convergence to a global optimum and derive the corresponding convergence rate. Moreover, in 93 Section 4, we present the results pertaining to finite-time stochastic policy gradient and in Section 5 94 we analyze the error accumulation using backward induction for exact and stochastic gradients. In 95 Section 6, we provide our findings regarding infinite discounted MDPs, where we derive complexity 96

bounds for the REINFORCE algorithm. 97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

2 Finite-time horizon MDPs

A finite-time MDP is defined by a tuple ( H , S , A , r, p ) with H = { 0 , . . . , H -1 } decision epochs, finite state space S = S 0 ∪ · · · ∪ S H -1 , finite action space A = ⋃ s ∈S A s , a reward function r : S × A → R and transition function p : S × A → ∆( S ) with p ( S h +1 | s, a ) = 1 for every h < H -1 , s ∈ S h and a ∈ A s . Let ∆( D ) denote the set of all probability measures over a finite set D . Due to finite decision epochs, the choice of the action is time dependent, i.e. non-stationary policies π = ( π h ) H -1 h =0 must be considered, where π h : S h → ∆( A ) for every h ∈ H is such that π h ( A s | s ) = 1 for every s ∈ S h . Denote by π ( h ) = ( π k ) H -1 k = h the sub-policy of π form h to H -1 , and define the h -state value function under policy π for every s ∈ S h by

where E π ( h ) s is the expectation under the measure such that S h = s , A k ∼ π k ( ·| S k ) and S k +1 ∼ p ( ·| S k , A k ) for h ≤ k < H -1 . The h -state-action value function for every tuple ( s, a ) ∈ S h ×A s is defined by

Note that Q h is independent of policy π h and for H -1 , Q H -1 ( s, a ) := r ( s, a ) independently of any policy. Furthermore, define the h -state-action advantage function

In the following, we will suppress the dependence of π ( h ) and write π in the superscripts of V h , Q h and A h , when the policy is clear out of context. We denote by

the value function for an initial state distribution µ h on S h in epoch h ∈ H . The performance difference lemma (Kakade and Langford, 2002) is a useful identity to compare policies. It turns out to be very useful to prove convergence of policy gradient methods (Agarwal et al., 2021). For finite-time MDPs the following version is proved in the supplementary material:

Lemma 2.1 (Performance difference lemma) . For any h ∈ H and for any pair of policies π and π ′ the following holds true for every s ∈ S h :

In order to address finite-time MDPs it becomes necessary to consider non-stationary policies because 120 the optimal decision at each time point depends on the time horizon until the end of the problem. 121 Thus, to solve finite-time MDPs with policy gradient a time-dependent parametrization of the policy 122 is required. Consider a parameter space denoted by Θ = Θ 0 ×···× Θ H -1 , where a policy parameter 123 θ = ( θ 0 , . . . , θ H -1 ) ∈ Θ includes H different parameters. A parametric policy π θ = ( π θ h ) H -1 h =0 is 124 defined such that the policy in epoch h depends only on the parameter θ h . It is worth noting that finite125 time MDPs are typically solved using backward induction as known from dynamic programming 126 theory (Puterman, 2005). In order to obtain the optimal solution for a finite-time MDP through 127 backward induction the parametrization must have the capability to approximate any deterministic 128 policy. This is because deterministic optimal policies exist for finite-time MDPs similar to discounted 129

MDPs. These conditions have made the tabular softmax policy a subject of extensive research in 130 the context of discounted MDPs, owing to its ability to meet these requirements (Mei et al., 2020; 131 Agarwal et al., 2021; Ding et al., 2022). Let Θ h = R d h for all h ∈ H , where d h = ∑ s ∈S h |A s | the 132 number of state-action pairs in epoch h . Then the tabular softmax parametrization is defined to be 133

134

135

136

137

138

139

In the forthcoming chapters, we will center our convergence analysis on this parametrization. Nevertheless, we emphasize that the results presented in this section are also valid for any other parametrization.

To solve a finite-time MDP the problem is partitioned into h sub-problems, with each epoch being considered separately. Given any fixed policy ˜ π , the objective function in epoch h is defined to be the h -state value function in state s ∈ S h under the policy ( π θ h , ˜ π ( h +1) ) := ( π θ h , ˜ π h +1 , . . . , ˜ π H -1 ) ,

An optimal parameter θ ∗ h is then sought such that J h,s ( θ ∗ h ) = sup θ ∈ Θ h J h,s ( θ ) , for all s ∈ S h . 140 In order to attain an optimal policy at each time point, this problem is approached via backward 141 induction, and the parametrization ˜ π in equation (6) is selected to be the pre-optimized one. Assuming 142 that the parametrization is able to approximate an optimal policy (e.g. the softmax parametrization), 143 then the backward induction yields optimal parameters θ ∗ h , . . . , θ ∗ H -1 in the sense that, see Puterman 144 (2005, Sec. 4.5), 145

for all s ∈ S h . To employ the policy gradient method, it is essential to compute the gradient of J h,s ( θ ) with respect to θ for a given policy ˜ π . Notably, the forthcoming policy ˜ π can be any policy, independent of the current parameter θ , which is trained during epoch h . This approach significantly deviates from the one used in discounted MDPs, such as in Sutton et al. (1999), where a stationary policy is parametrized and utilized at every time step. Despite the differences, a policy gradient theorem can still be attained, allowing the gradient of the objective function to be written as an expectation.

Theorem 2.2. For a fixed policy ˜ π and h ∈ H the gradient of J h,s ( θ ) defined in (6) is given by

As for the value function, we denote by J h ( θ ) := E s ∼ µ h [ J h,s ( θ )] the objective function under some initial state distribution µ h on S h . Algorithm 1 summarizes policy gradient in finite-time MDPs.

Algorithm 1: Policy Gradient for finite-time MDPs and non-stationary policies

Result: Approximate policy ˆ π ∗ ≈ π ∗ Initialize θ (0) = ( θ (0) 0 , . . . , θ (0) H -1 ) ∈ Θ for h = H -1 , . . . , 0 do Choose fixed step size η h and number of training steps N h for n = 0 , . . . , N h -1 do Calculate ∇ J h ( θ ( n ) h ) with fixed policy ˆ π ∗ after h θ ( n +1) h = θ ( n ) h + η h ∇ J h ( θ ( n ) h ) end Set ˆ π ∗ h = π θ ( Nh ) h end

Training each time point separately and having a fixed policy ˜ π after h , we state a version of the performance difference lemma given this specific setting.

Corollary 2.3. For any h ∈ H and two policies π and π ′ : If π ( h +1) = π ′ ( h +1) , it holds that

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

3 Convergence Analysis of Softmax Policy Gradient

Before we combine all decision epochs as stated in Algorithm 1, we provide convergence results for each h ∈ H given that the policy after h is fixed and denoted by ˜ π . The error analysis over time is then employed in Section 5.

Assumption 3.1 . Throughout the remaining manuscript we assume that the rewards are bounded in [0 , R ∗ ] , for some R ∗ > 0 .

3.1 Asymptotic convergence

The choice of tabular softmax parametrization is particularly convenient as derivatives are simple. Lemma 3.2. Let h ∈ H , then the partial derivatives of J h with respect to θ take the following form

Furthermore, J h is a smooth function with respect to θ . The proof is based on a more general result which proves smoothness for all parametrizations with bounded gradient and Hessian of the log-policy.

Proposition 3.3. Let h ∈ H and consider the objective function J h ( θ ) . If there exists G,M > 0 such that

for all s ∈ S h , a ∈ A s , then for any initial state distribution µ h of S h the function J h ( θ ) is β h -smooth in θ with β h = ( H -h ) R ∗ ( G 2 + M ) .

Smoothness under these assumptions in the discounted finite-time setting with stationary policy was shown for example in Xu et al. (2020b) and Xu et al. (2020a). We obtain the following smoothness parameter:

Lemma 3.4. Let h ∈ H , then the h -state value function under softmax parametrization, θ ↦→ J h ( θ ) , is β h -smooth with β h = 2( H -h ) R ∗ |A| .

We point out that the smoothness parameter is independent of the choice of ˜ π . A consequence of the smoothness is the asymptotic convergence of the objective function towards a global maximum. As each epoch is considered separately we just write θ n instead of θ ( n ) h until Section 5.

Theorem 3.5. Let h ∈ H and consider the gradient ascent updates

for arbitrary θ 0 ∈ R d h . We assume that µ h ( s ) > 0 for all s ∈ S h and 0 < η h ≤ 1 β h . Then, for all s ∈ S h , J h,s ( θ n ) converges to J ∗ h,s for n →∞ , where J ∗ h,s = sup θ J h,s ( θ ) < ∞ .

The difficulties that arise from softmax parametrization are the same as discussed in Agarwal et al. (2021) for the infinite time setting: The softmax policy approximates an optimal deterministic policy. Therefore, parameters converge to -∞ for suboptimal actions and to ∞ for optimal actions. The idea of the proof follows the outline of the discounted MDP setting except for one main distinction: the action-value function Q h is independent of the policy gradient updates such that no limiting process has to be constructed. A detailed proof is provided in B.1.

Note that the assumption µ h ( s ) > 0 for all s ∈ S h is necessary for sufficient exploration. The same assumption is needed for the initial distribution of a discounted MDP in Agarwal et al. (2021, Thm. 10). Furthermore, Mei et al. (2020, Prop. 3) have demonstrated the necessity of this assumption.

3.2 Convergence rate

In order to derive a convergence rate for tabular softmax parametrized finite-time MDPs we will establish a weaker form of the PL-inequality. Therefore, consider for h ∈ H a deterministic optimal policy π ∗ h , given that the policy after h is fixed by ˜ π , i.e. for all s ∈ S h ,

Please note here that the optimal policy and also J ∗ h,s depend on the choice of ˜ π . 199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

Lemma 3.6 (weak PL-inequality) . For the objective J h it holds that

where a ∗ h ( s ) = argmax a ∈A s π ∗ h ( a | s ) and J ∗ h = sup θ J h ( θ ) .

The term min s ∈S π θ ( a ∗ h ( s ) | s ) also appears in similar form in the discounted setting in Mei et al. (2020). The main challenge is to bound this term from below uniformly in θ appearing in the gradient ascent updates. Due to asymptotic convergence this can be achieved, where it is necessary to assume µ h ( s ) > 0 for all s ∈ S h .

Lemma 3.7. Let h ∈ H , µ h ( s ) > 0 for all s ∈ S h and consider the sequence ( θ n ) n ∈ N 0 generated by (7) for arbitrarily initialized θ 0 ∈ R d h . Then it holds that c h := inf n ≥ 0 min s ∈S h π θ n ( a ∗ h ( s ) | s ) > 0 .

We emphasize that the constant c h is influenced by the initial parameter θ 0 thereby making it a parameter dependent on the model, as it is also for discounted MDPs in Mei et al. (2020).

Theorem 3.8. Let h ∈ H , µ h ( s ) > 0 for all s ∈ S h and consider the sequence ( θ n ) n ∈ N 0 generated by (7) for arbitrarily initialized θ 0 ∈ R d h . Define c h := inf n ≥ 0 min s ∈S h π θ n ( a ∗ h ( s ) | s ) > 0 by Lemma 3.7 and choose step size η h = 1 β h with β h = 2( H -h ) R ∗ |A| . Then it holds that

where J ∗ h = sup θ J h ( θ ) .

The error bound depends on the time horizon up to the last time point, meaning intuitively that an optimal policy for earlier time points in the MDP (smaller h ) is harder to achieve and requires a longer learning period then later time points ( h near to H ). Comparing this result to the convergence rate for discounted MPDs we note that the linear dependency on the time horizon is less aggressive than the factor (1 -γ ) -1 . In addition, the magnitude of the state space S h does not have a direct impact on the rate. However, the constant c h indirectly introduces a dependency.

4 Convergence Analysis of Stochastic Softmax Policy Gradient

For the rest of this paper we drop the assumption of knowing ∇ J h ( θ ) . In this model-free setting it is only assumed that trajectories of the finite-time MDP can be simulated. Stochastic policy gradient is used to train the parameters, where in each iteration the gradient of the objective is approximated using Monte Carlo estimates. Consider K h trajectories ( s i k , a i k ) H -1 k = h , for i = 1 , . . . , K h , generated by s i h ∼ µ h , a i h ∼ π θ h and a i k ∼ ˜ π k for h < k < H . The estimator is defined by

where ˆ Q h ( s i h , a i h ) = ∑ H -1 k = h r ( s i k , a i k ) is an unbiased estimator of the h -state-action value function in ( s i h , a i h ) under policy ˜ π . Then the stochastic policy gradient updates for training the parameter θ are given by

To train an optimal policy with backward induction, ˜ π is chosen to be the already trained policies. As in Section 3 we first restrict our convergence analysis to one time point h given a fixed policy ˜ π after h . The entire stochastic policy gradient algorithm, often called REINFORCE, is summarized in Algorithm 2.

Under the softmax parametrization it holds true that ̂ ∇ J K h h ( θ ) is an unbiased estimator with uniformly bounded variance due to the bounded reward assumption (see Lemma C.1).

4.1 Asymptotic convergence to stationary point

Using stochastic policy gradient, we obtain almost sure convergence of the value function to a stationary point for decreasing step sizes. Note that, except for this theorem we assume a constant step size.

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

Algorithm 2: REINFORCE with Backward Iteration

Result:

Approximate policy

Initialize

θ

= (

θ

(0)

(0)

0

, . . . , θ

for h = H -1 , . . . , 0 do

Choose step size η , number of training steps N and batch size K

Theorem 4.1. For any h ∈ H consider the stochastic process ( θ n ) n ≥ 0 generated by 239

for arbitrary batch size K h ≥ 1 and initial θ 0 such that E [ J h ( θ 0 )] < ∞ . Furthermore, suppose that η ( n ) h is decreasing, such that ∑ n ≥ 0 η ( n ) h = ∞ and ∑ n ≥ 0 ( η ( n ) h ) 2 < ∞ . Then ∇ J h ( θ n ) → 0 almost surely for n →∞ .

With Lemma C.1 and the boundedness of the h -state value functions, this follows directly from the stochastic approximation theorem stated in Bertsekas and Tsitsiklis (2000) (see Proposition C.2 in the supplementary material).

4.2 Complexity bounds to approximate to global optimum with high probability

In the following denote by ( ¯ θ n ) n ≥ 1 the deterministic sequence generated by policy gradient with exact gradients,

Let ( θ n ) n ≥ 0 be the stochastic process from (9) such that the initial parameter agree, θ 0 = ¯ θ 0 , and the step size η h is the same for both processes. The natural filtration of ( θ n ) n ≥ 0 is denoted by ( F n ) n ≥ 0 . Recall that c h = min n ≥ 0 min s ∈S π ¯ θ n ( a ∗ ( s ) | s ) is bounded away from 0 by Lemma 3.7. The idea of the convergence analysis for stochastic softmax policy gradient is to define the following stopping time

This means, τ is the first time when the stochastic process ( θ n ) n ≥ 0 is too far away from the policy gradient trajectory ( ¯ θ n ) n ≥ 0 . Hence, all challenges encountered in the deterministic case transfer to the stochastic context, indicating that the model dependent constant c h naturally appears in the error bounds of the stochastic case. We emphasize that τ is a stopping time with respect to the filtration ( F n ) n ≥ 0 by construction.

First, consider the event { n ≤ τ } , i.e. ∥ θ n -¯ θ n ∥ 2 ≤ c h 4 . It follows by the √ 2 -Lipschitz continuity of θ ↦→ π θ ( a ∗ ( s ) | s ) (Lemma C.3) that min 0 ≤ k ≤ τ min s ∈S π θ k ( a ∗ ( s ) | s ) ≥ c h 2 > 0 (Lemma C.4). This allows us to use the weak PL-inequality of Lemma 3.6 to derive a convergence rate on the event { n ≤ τ } in the following sense:

Lemma 4.2. Suppose µ h ( s ) > 0 for all s ∈ S h , the batch size K ( n ) h ≥ 9 c 2 h C h 32 β 2 h N 3 2 h (1 -1 2 √ N h ) n 2 is increasing for some N h ≥ 1 and the step size η h = 1 √ , for fixed h ∈ H . Then,

β h N h

ˆ

π

∗

(0)

H

-

≈

1

)

π

∈

∗

Θ

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

Secondly, consider the complementary event { τ ≤ n } . We can bound the probability of this event by δ for a large enough batch size K h . The proof is based on a similar result obtained by Ding et al. (2022, Lem. 6.3) for discounted MDPs.

Lemma 4.3. Suppose µ h ( s ) > 0 for all s ∈ S h . Then, for any δ > 0 , we have P ( τ ≤ n ) < δ if K h ≥ 16 n 3 C h β 2 c 2 h δ 2 and η h = 1 √ nβ h .

We are now ready to formulate the main result of this section.

Theorem 4.4. Suppose the stochastic policy gradient updates are generated by (9) for arbitrary initialization θ 0 ∈ R d h . Suppose that µ h ( s ) > 0 for all s ∈ S h and choose for any δ, ϵ > 0 ,

(i) the number of training steps N h ≥ ( 64 β h 3 δc 2 h ϵ ) 2 ,

(ii) the step size η h = 1 β h √ N h and the batch size K h = 64 N 3 h C h β 2 c 2 h δ 2 .

It should be noted that the choice of step size η h and batch size K h are closely connected and both strongly depend on the number of training steps N h . Specifically, as N h increases, the batch size increases, while the step size tends to decrease to prevent exceeding the stopping time with high probability. However, it is possible to increase the batch size even further and simultaneously benefit from choosing a larger step size, or vice versa.

5 Error Analysis over Time

In this section, we will first examine the accumulation of error over time for the policy gradient Algorithm 1, and secondly, for the stochastic policy gradient Algorithm 2. In both cases the error accumulates linearly such that an ϵ H -error in each time point h results in an overall error of ϵ . This is due to the additive structure of the rewards and comes naturally from the backward induction of dynamic programming for finite-time MDPs.

Theorem 5.1. Assume that µ h ( s ) > 0 for all h ∈ H , s ∈ S h . Let ϵ > 0 , the step size η h = 1 β h and the batch size N h = 4( H -h ) HR ∗ |A| c 2 h ϵ ∥ ∥ 1 µ h ∥ ∥ ∞ . Denote by ˆ π ∗ = ( π θ N 0 0 , . . . , π θ NH -1 H -1 ) the final policy from Algorithm 1, then for all s ∈ S 0 ,

For the stochastic policy gradient algorithm, we obtain the following main result:

Theorem 5.2. Assume that µ h ( s ) > 0 for all h ∈ H , s ∈ S h . Let δ, ϵ > 0 , the step size η h = 1 β h N h , number of training steps N h = ( 64 β h H 2 ∥ ∥ 1 µ h ∥ ∥ ∞ 3 δc 2 h ϵ ) 2 and the batch size K h = 64 N 2 h H 2 C h β h c 2 h δ 2 . Denote by ˆ π ∗ = ( π θ N 0 0 , . . . , π θ NH -1 H -1 ) the final policy from Algorithm 2, then

In both results we observe that the number of training steps in each epoch depends on the constant ∥ ∥ 1 µ h ∥ ∥ ∞ = max s ∈S 1 µ h ( s ) . The proofs of Section D reveal that this constant occurs to ensure that the objective J h,s ( θ ( N h ) h ) is close to J ∗ h,s for every s ∈ S h .

6 Convergence Analysis of Stochastic Policy Gradient in Infinite Horizons

In this final section, we show how to combine the results of Mei et al. (2020) with our stochastic gradient arguments to show that the plain vanilla REINFORCE algorithm without regularization can approximate global maxima if the batch sizes are chosen properly. Our theoretically derived batch sizes are clearly not of practical use but give a first insight why REINFORCE requires large

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

batch sizes to give reasonable approximations. In the following, we consider the discounted MDP setting from Equation (1) with rewards taking values in [0 , 1] , i.e. R ∗ = 1 , and tabular softmax parametrization π θ from (5) with θ ∈ Θ = R |S||A| . The objective function J ( θ ) := E S 0 ∼ µ [ V π θ ( S 0 )] is defined for an initial state distribution µ . It is important to highlight that π θ is now a stationary policy used in every epoch. Our arguments rely on the weak PL-inequality for the exact value function. Mei et al. (2020) proved that

where a ∗ ( s ) = argmax π ∗ ( ·| s ) the optimal action in state s and ∥ ∥ ∥ d π ∗ ρ d π θ µ ∥ ∥ ∥ ∞ is the distribution mismatch coefficient introduced in Agarwal et al. (2021). We present an alternative version in Lemma E.2 without the constant |S| -1 / 2 . The typical approach to prove convergence of stochastic gradient schemes is to iteratively compare the stochastic gradient update to the deterministic one and then control the error. This is not always possible, but for stochastic softmax policy gradient we show that the error can be controlled for large enough batch sizes. We proceed in a manner similar to Section 4.2. Thus, to state the theorem let us denote by

the policy gradient and stochastic policy gradient schemes. Also denote by c := min n ≥ 0 min s ∈S π ¯ θ n ( a ∗ ( s ) | s ) the model dependent constant from the weak PL-inequality of (Mei et al., 2020, Lem. 8). For the algorithm we use the unbiased gradient estimator proposed by Zhang et al. (2020) which the authors used to prove convergence to a stationary point. Our main contribution is the following convergence result towards the global optimum:

Theorem 6.1. Let ( ¯ θ n ) n ≥ 0 and ( θ n ) n ≥ 0 be the (stochastic) policy gradient updates from (11) for arbitrary initial ¯ θ 0 = θ 0 ∈ Θ . Suppose µ ( s ) > 0 for all s ∈ S and choose for any δ, ϵ > 0 ,

We present more details on the algorithm and the proof in Section E of the supplementary material. We emphasize that the dependency on the distribution mismatch coefficient and the model dependent constant c are unavoidable since the stochastic gradient ascent is derived from the deterministic gradient ascent. To the best of our knowledge, this is the first convergence analysis for stochastic policy gradient with softmax parametrization without regularization. So far, Ding et al. (2022) derived complexity bounds for convergence of softmax policy gradient to the entropy-regularized optimum.

7 Conclusion and Future Work

In this paper, we have presented a convergence analysis of policy gradient methods for undiscounted MDPs with finite-time horizon in the tabular setting. Assuming exact gradients we have obtained an O (1 /n ) -convergence rate which is linearly dependent on the time horizon. In the model-free setting we have derived complexity bounds to approximate the error to global optima with high probability. Moreover, we were able to extend this result to discounted MDPs without regularization.

In the finite-time case, it would be intriguing to explore policy parametrizations with a smaller parameter space as for example log-linear policies. Additionally, investigating modern policy gradient algorithms such as TRPO and natural policy gradient within the context of finite-time MDPs could further enhance the convergence rate. In the stochastic setting, it is desirable to eliminate the model-dependent parameter from the complexity bounds to construct a practicable algorithm. This would require an improved convergence analysis of policy gradient with exact gradients.

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

References

Alekh Agarwal, Sham M. Kakade, Jason D. Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research , 22(98):1-76, 2021. URL http://jmlr.org/papers/v22/19-736.html .

Carlo Alfano and Patrick Rebeschini. Linear convergence for natural policy gradient with log-linear policy parametrization, 2023. URL https://arxiv.org/abs/2209.15382 .

Amir Beck. First-Order Methods in Optimization . Society for Industrial and Applied Mathematics, Philadelphia, PA, 2017. doi: 10.1137/1.9781611974997.

Dimitri P. Bertsekas and John N. Tsitsiklis. Gradient convergence in gradient methods with errors. SIAM Journal on Optimization , 10(3):627-642, 2000. doi: 10.1137/S1052623497331063.

Jalaj Bhandari and Daniel Russo. On the linear convergence of policy gradient methods for finite MDPs. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , volume 130 of Proceedings of Machine Learning Research , pages 2386-2394. PMLR, 13-15 Apr 2021. URL https://proceedings.mlr.press/v130/bhandari21a.html .

Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods, 2022. URL https://arxiv.org/abs/1906.01786 .

Shicong Cen, Chen Cheng, Yuxin Chen, Yuting Wei, and Yuejie Chi. Fast global convergence of natural policy gradient methods with entropy regularization. Operations Research , 70(4): 2563-2578, 2022. doi: 10.1287/opre.2021.2151.

Ignasi Clavera, Jonas Rothfuss, John Schulman, Yasuhiro Fujita, Tamim Asfour, and Pieter Abbeel. Model-based reinforcement learning via meta-policy optimization. In Proceedings of The 2nd Conference on Robot Learning , volume 87 of Proceedings of Machine Learning Research , pages 617629. PMLR, 29-31 Oct 2018. URL https://proceedings.mlr.press/v87/clavera18a. html .

Yuhao Ding, Junzi Zhang, and Javad Lavaei. Beyond exact gradients: Convergence of stochastic soft-max policy gradient methods with entropy regularization, 2022. URL https://arxiv.org/ abs/2110.10117 .

Maryam Fazel, Rong Ge, Sham Kakade, and Mehran Mesbahi. Global convergence of policy gradient methods for the linear quadratic regulator. In Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1467-1476. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/v80/fazel18a.html .

Xin Guo, Anran Hu, and Junzi Zhang. Theoretical guarantees of fictitious discount algorithms for episodic reinforcement learning and global convergence of policy gradient methods. Proceedings of the AAAI Conference on Artificial Intelligence , 36(6):6774-6782, Jun. 2022. doi: 10.1609/aaai. v36i6.20633.

Ben Hambly, Renyuan Xu, and Huining Yang. Policy gradient methods for the noisy linear quadratic regulator over a finite horizon. SIAM Journal on Control and Optimization , 59(5):3359-3391, 2021. doi: 10.1137/20M1382386.

Ben Hambly, Renyuan Xu, and Huining Yang. Policy gradient methods find the nash equilibrium in nplayer general-sum linear-quadratic games, 2022. URL https://arxiv.org/abs/2107.13090 .

Feihu Huang, Shangqian Gao, Jian Pei, and Heng Huang. Momentum-based policy gradient methods. In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 4422-4433. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/huang20a.html .

Feihu Huang, Shangqian Gao, and Heng Huang. Bregman gradient policy optimization. In International Conference on Learning Representations , 2022. URL https://openreview.net/ forum?id=ZU-zFnTum1N .

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In Proceedings of the Nineteenth International Conference on Machine Learning , page 267-274. Morgan Kaufmann Publishers Inc., 2002. doi: 10.5555/645531.656005.

Sham M Kakade. A natural policy gradient. In Advances in Neural Information Processing Systems , volume 14. MIT Press, 2001. URL https://proceedings.neurips.cc/paper_files/ paper/2001/file/4b86abe48d358ecf194c56c69108433e-Paper.pdf .

Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximalgradient methods under the Polyak-Łojasiewicz condition. In Machine Learning and Knowledge Discovery in Databases , pages 795-811, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46128-1.

Vijay Konda and John Tsitsiklis. Actor-critic algorithms. In Advances in Neural Information Processing Systems , volume 12. MIT Press, 1999. URL https://proceedings.neurips.cc/ paper_files/paper/1999/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf .

Yuxi Li, Csaba Szepesvari, and Dale Schuurmans. Learning exercise policies for american options. In Proceedings of the Twelth International Conference on Artificial Intelligence and Statistics , volume 5 of Proceedings of Machine Learning Research , pages 352-359, Hilton Clearwater Beach Resort, Clearwater Beach, Florida USA, 16-18 Apr 2009. PMLR. URL https://proceedings. mlr.press/v5/li09d.html .

Jincheng Mei, Chenjun Xiao, Csaba Szepesvari, and Dale Schuurmans. On the global convergence rates of softmax policy gradient methods. In Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 6820-6829. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/mei20b.html .

Jincheng Mei, Yue Gao, Bo Dai, Csaba Szepesvari, and Dale Schuurmans. Leveraging non-uniformity in first-order non-convex optimization. In Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 7555-7564. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/mei21a.html .

Yurii Nesterov. Introductory Lectures on Convex Optimization . Springer New York, NY, 2013. doi: 10.1007/978-1-4419-8853-9.

Matteo Papini, Damiano Binaghi, Giuseppe Canonaco, Matteo Pirotta, and Marcello Restelli. Stochastic variance-reduced policy gradient. In Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 4026-4035. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/v80/papini18a.html .

Matteo Pirotta, Marcello Restelli, and Luca Bascetta. Adaptive step-size for policy gradient methods. In Advances in Neural Information Processing Systems , volume 26. Curran Associates, Inc., 2013. URL https://proceedings.neurips.cc/paper_files/paper/2013/file/ f64eac11f2cd8f0efa196f8ad173178e-Paper.pdf .

B.T. Polyak. Gradient methods for the minimisation of functionals. USSR Computational Mathematics and Mathematical Physics , 3(4):864-878, 1963. doi: 10.1016/0041-5553(63)90382-3.

M.L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming . John Wiley & Sons, 2005. doi: 10.1002/9780470316887.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 1889-1897, Lille, France, 07-09 Jul 2015. PMLR. URL https://proceedings.mlr.press/v37/schulman15.html .

Zebang Shen, Alejandro Ribeiro, Hamed Hassani, Hui Qian, and Chao Mi. Hessian aided policy gradient. In Proceedings of the 36th International Conference on Machine Learning , volume 97 of Proceedings of Machine Learning Research , pages 5729-5738. PMLR, 09-15 Jun 2019. URL https://proceedings.mlr.press/v97/shen19d.html .

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

Richard S Sutton, David McAllester, Satinder Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems , volume 12. MIT Press, 1999. URL https://proceedings.neurips.cc/ paper_files/paper/1999/file/464d828b85b0bed98e80ade0a5c43b0f-Paper.pdf .

Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning , 8(3):229-256, 1992. doi: 10.1007/BF00992696.

Lin Xiao. On the convergence rates of policy gradient methods. Journal of Machine Learning Research , 23(282):1-36, 2022. URL http://jmlr.org/papers/v23/22-0056.html .

Pan Xu, Felicia Gao, and Quanquan Gu. Sample efficient policy gradient methods with recursive variance reduction. In International Conference on Learning Representations , 2020a. URL https://openreview.net/forum?id=HJlxIJBFDr .

Pan Xu, Felicia Gao, and Quanquan Gu. An improved convergence analysis of stochastic variancereduced policy gradient. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference , volume 115 of Proceedings of Machine Learning Research , pages 541-551. PMLR, 22-25 Jul 2020b. URL https://proceedings.mlr.press/v115/xu20a.html .

Kaiqing Zhang, Alec Koppel, Hao Zhu, and Tamer Ba¸ sar. Global convergence of policy gradient methods to (almost) locally optimal policies. SIAM Journal on Control and Optimization , 58(6): 3586-3612, 2020. doi: 10.1137/19M1288012.

Kaiqing Zhang, Xiangyuan Zhang, Bin Hu, and Tamer Basar. Derivative-free policy optimization for linear risk-sensitive and robust control design: Implicit regularization and sample complexity. In Advances in Neural Information Processing Systems , volume 34, pages 2949-2964. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/ file/1714726c817af50457d810aae9d27a2e-Paper.pdf .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a convergence (rate) analysis for a variant of policy-gradient learning of tabular-softmax policy models under finite-horizon MDP setting with discounting factor $\\gamma=1$. In this variant, the policy parameters are updated in an epoch-by-epoch manner, from the last decision epoch at the horizon back to the first decision epoch. For each epoch, the learning is a vanilla (stochastic) gradient ascent process, with the objective function being the value obtained assuming fixed return in future epochs. It seems to me that most results in the paper are obtained following similar ideas with previous theoretical studies of policy gradient under discounted MDP settings (e.g. Agarwal et al., 2021, Mei et al., 2020).\n\nSTRENGTHS:\nI appreciate the general motivation of this work which aims at analyzing RL algorithms under undiscounted setting. In my opinion (and as the authors also pointed out in the paper), many performance bounds of RL algorithms derived under the discounted MDP setting crucially depend on the factor  $\\frac{1}{1-\\gamma}$, thus fail to accurately characterize the performance of RL algorithm under undiscounted setting. On the other hand, many real-world problems in practice are indeed undiscounted MDP problems. Therefore, directly analyzing RL algorithms under undiscounted MDP settings, or deriving results that do not degenerate when $\\gamma$ approaches 1, is relevant and important research direction.\n\nWEAKNESSES:\nHowever, several major concerns make me hard to appreciate this particular work. Specifically,\n\n1. The so-called policy gradient algorithm as analyzed in this paper is not really the standard policy gradient algorithm as generally perceived and applied -- the latter updates the parameters of states for all decision epochs simultaneously in each gradient step while the algorithm variant targeted in this paper will update parameters only for one epochs each time. \nI think the standard co-updating strategy is much more practically relevant. For example, in RL training of language models, the problem is indeed an undiscounted MDP problem, and standard policy gradient methods (with PPO upgradation) are indeed widely used; see the InstructGPT paper (https://arxiv.org/abs/2203.02155) as an example. I thus have concern on the relevance and significance of the algorithm (not the problem setting) studied in this paper.\n\n2. The epoch-wise updating strategy as assumed in this paper makes most of the analysis a straightforward adaptation of the standard analysis method used in the literature. Moreover, it seems to me that the epoch-wise learning problem analyzed in this paper is just a contextual bandit problem -- the paper assumes the policy $\\tilde{\\pi}$ for all future epochs is fixed, as well as assuming that the distribution $\\mu_h$ for states encountered at the epoch is also fixed, but in this case we are essentially just maximizing an \"augmented immediate reward (with future returns counted, which is fixed)\" over actions conditioned on a context distribution. Since contextual bandit problem is a special case of discounted MDP (with arbitrary $\\gamma$), I wonder if most results presented in this paper trivially hold given known results about discounted MDP.  \n\n3. I also have concerns on the authors' interpretation of their mathematical results. For example, the paper claims (as a main contribution) that Theorem 3.8 derives a linear performance bound to the horizon H. However, the bound contains the factor c_h, and it's not clear if c_h itself can be a function of H (or of H-h). c_h can be considered a \"constant\" only with respect to n, not to H. Moreover, Theorem 3.8 only gives a per-epoch performance while previous results gives the overall performance. My understanding is that when we consider the convergence of epoch 0 (which is what we eventually care), we would need to count the time spent on epoch 1 ~ H-1 too, which would bring in another factor of H to the bound?\n\n4. Similarly, throughout the paper it's assumed that the state distribution $\\mu_h$ is given for all epoch h, which is not a reasonable assumption in my opinion because except for $\\mu_0$, the state distributions for all other epochs actually depend on the algorithm. This is in sharp contrast to the previous papers which only assume that the *initial* state distribution $\\mu_0$ is given (which is indeed given). Again, this $\\mu_h$ is considered a \"constant\" when interpreting the bound derived in Theorem 5.2, although $\\mu_h$ (for h>0) may heavily depend on structures both in the problem and in the algorithm, in a highly non-trivial manner.\n\n5. The majority of the paper assumes that the reward is nonnegative, which is a bit of a limitation, given that the sign of the rewards can greatly affects the performance of policy gradient.\n\nQUESTIONS:\n(A) You argued that \"*reducing the problem to stationary policies is inadequate for finite-time MDPs, and a new policy must be trained recursively at each time step*\" (Line 50). I am not sure that this assertion is true. The so-called non-stationary policy $\\pi$ (defined at Line 104) is exactly a stationary policy over the entire state space (note that you have defined the state space $S$ as the union of the disjoint epoch-wise state spaces; see Line 100). Since the policy $\\pi$ at Line 104 is just an ordinary stationary policy over $S$, the standard policy gradient algorithm can be applied, and we don't have to train it recursively. Since this disagreement directly leads to my major concern 1 as stated above, I am open to hear your rebuttal if I am misunderstanding something here.\n\n(B) Line 154: It seems to me that $J_h$ depends not only on $\\theta$ but also on $\\mu_h$ and $\\tilde{\\pi}$. However, the notation here is hiding this fact. This in turn hide some subtle issues in the analysis later. For example, explicitly writing $J_h(\\theta, \\mu_h, \\tilde{\\pi})$ would make it clear that in Algorithm 1 the $\\mu_h$ is left undetermined.\n\n(C) Line 263: What is $C_h$? I originally thought this is a typo for $c_h$, until I found that in Theorem 5.2 both $C_h$ and $c_h$ show up in the definition of $K_h$.\n\nLIMITATIONS:\nThe paper well discussed the limitations of this work in Section 7.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-29"}

comments: SUMMARY:
This paper proves asymptotic convergence and convergence rate of (stochastic) policy gradient descent to global optimum for un-discounted finite time Markov decision process (MDP). For the deterministic version, at each decision time, they show the error bound depends linearly on the remaining time step. For the stochastic version, they derived probability bounds. They extend their analysis to REINFORCE on discounted MDP.

STRENGTHS:
The presented results are sound and complete. Both convergence and convergence rate of policy gradient methods with softmax policy are established for the finite-time MDPs.

WEAKNESSES:
The main argument seems to be based on Mei et al (2020) with some modifications for un-discounted finite time MDPs case. I did not go over the details of proofs in appendix. It is hard  to evaluate the significance and the novelty of the contributions.

QUESTIONS:
See wekness

LIMITATIONS:
See weakness
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the convergence properties of stochastic policy gradient methods for finite-state MDPs for finite horizon problems with un-discounted optimality criteria. The convergence relies on the development of a weak PL condition. In the second part of the paper, the authors then extend their convergence analysis to the setting where the policy gradient is not available exactly and only a stochastic version of it can be used.\n\nSTRENGTHS:\nThe paper is well written and easy to follow. In particular, I find Section 4, where the convergence analysis is generalized to the stochastic setting strong and practical useful. Proofs are derived very clearly and detailed with I appreciate.\n\nWEAKNESSES:\n1) Numerical example missing: In my opinion, it would be interesting to include a numerical example to the paper. In particular it would be interesting to see how tight the complexity bounds of Theorem 4.4 are on a concrete example. \n\n2) Finite state/action spaces often are restrictive: What can you say about the results being generalised to continuous state and action spaces? Maybe for a linear system to start with.\n\n3) How relevant are un-discounted problems? Could you add a detailed motivation, why these problems are practically relevant?\n\nQUESTIONS:\n1) Could you clarify better which results are new and which are existing results. For example, I don’t understand if Theorem 2.2 is an existing result or if it is new.\n\n2) In Equation (5), I don’t understand the dimension of $\\theta$. I think there is something wrong here.\n\n3) In Lemma 2.1, why is the superscript of the expectation operator $\\pi_{(h)}$ and not $\\pi$ ?\n\nLIMITATIONS:\nWithout Section 4, the work would have been limited as the gradients typically are not known exactly. But thanks to the results in Section 4, I think the paper is rather complete and I do not see major limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-08"}

comments: SUMMARY:
The paper analyzes the convergence of non-stationary softmax policy gradient methods where the policy is parameterized by different parameters at each decision epoch. The convergence results on REINFORCE algorithm are provided under undiscounted finite-time and infinote-horizon cases.

STRENGTHS:
- The paper obtains new global convergence results for softmax policy under undiscounted finite-time and infinite-horizon MDPs. The assumptions used are standard in policy gradient literature.

- The analyses in the paper also cover policy parameterized by deep neural network in deep reinforcement learning.

- The paper is well-written and easy to follow.

WEAKNESSES:
- My major concern is about the factor $c_h$ in Theorem 3.8. The authors use it as a constant but I argue it is not. Since $c_h$ is the infimum over all iteration of the minimum output of softmax policy, it is not known or can be computed before-hand. In fact, it depends on the choice of number of iteration $n$ and the current parameter $\theta_n$. More discussion is needed in this case.

- The results in Theorem 3.8 and Theorem 4.4 are valid for the exact policy gradient which is rather restrictive and impractical. I believe having results in the stochastic case using stochastic policy gradient will strengthen the contribution of the paper.

QUESTIONS:
1. The factor $c_h$ can be very small (if the policy is well-trained). Can we impose certain condition to overcome this?

2. I wonder if the results can be extended to other class of policy gradient method, such as ones with different policy gradient estimator.

LIMITATIONS:
I do not find clear discussion on the limitation of the work but I find the use of exact gradient is the main limitation for the key results in the paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-10