Title: Optimistic Natural Policy Gradient: a Simple Efficient Policy Optimization Framework  for Online RL
Venue: NeurIPS 2023
Accepted: yes
Keywords: Theory of reinforcement learning

policy optimization
Abstract: While policy optimization algorithms have played an important role in recent empirical success of Reinforcement Learning (RL), the existing theoretical understanding of policy optimization remains rather limited---they are either restricted to tabular MDPs or suffer from highly suboptimal sample complexity, especial in online RL where exploration is necessary. This paper proposes a simple efficient policy optimization framework---Optimistic NPG for online RL. Optimistic NPG can be viewed as simply combining of the classic natural policy gradient (NPG) algorithm [Kakade, 2001]  with optimistic policy evaluation subroutines to encourage exploration. For $d$-dimensional linear MDPs, Optimistic NPG is computationally efficient, and learns an $\epsilon$-optimal policy within  $\tilde{\mathcal{O}}(d^2/\epsilon^3)$ samples, which is the first computationally efficient algorithm whose sample complexity has the optimal dimension dependence $\tilde{\Theta}(d^2)$. It also improves over state-of-the-art results of policy optimization algorithms [Zanette et al., 2021] by a factor of $d$. For general function approximation that subsumes linear MDPs, Optimistic NPG, to our best knowledge, is also the first policy optimization algorithm that achieves the polynomial sample complexity for learning near-optimal policies.

Optimistic Natural Policy Gradient: a Simple Efficient Policy Optimization Framework for Online RL

Abstract

While policy optimization algorithms have played an important role in recent empirical success of Reinforcement Learning (RL), the existing theoretical understanding of policy optimization remains rather limited-they are either restricted to tabular MDPs or suffer from highly suboptimal sample complexity, especially in online RL where exploration is necessary. This paper proposes a simple efficient policy optimization framework-OPTIMISTIC NPG for online RL. OPTIMISTIC NPG can be viewed as a simple combination of the classic natural policy gradient (NPG) algorithm [Kakade, 2001] and an optimistic policy evaluation subroutine to encourage exploration. For d -dimensional linear MDPs, OPTIMISTIC NPG is computationally efficient, and learns an ϵ -optimal policy within ˜ O ( d 2 /ϵ 3 ) samples, which is the first computationally efficient algorithm whose sample complexity has the optimal dimension dependence ˜ Θ( d 2 ) . It also improves over state-of-the-art results of policy optimization algorithms [Zanette et al., 2021] by a factor of d . For general function approximation that subsumes linear MDPs, OPTIMISTIC NPG, to our best knowledge, is also the first policy optimization algorithm that achieves the polynomial sample complexity for learning near-optimal policies.

1 Introduction

Policy optimization algorithms [Schulman et al., 2017, 2015] with neural network function approximation have played an important role in recent empirical success of reinforcement learning (RL), such as robotics [Finn et al., 2016], games [Berner et al., 2019] and large language models [OpenAI, 2022]. Motivated by the empirical success, the theory community made a large effort to design provably efficient policy optimization algorithms that work in the presence of linear function approximation [Agarwal et al., 2021, Bhandari and Russo, 2019, Liu et al., 2019, Neu et al., 2017, Abbasi-Yadkori et al., 2019, Agarwal et al., 2020, Zanette et al., 2021, Shani et al., 2020, Cai et al., 2020]. Early works focused on proving that policy optimization algorithms are capable to learn near-optimal policies using a polynomial number of samples under certain reachability (coverage) assumptions. [e.g., Agarwal et al., 2021, Bhandari and Russo, 2019, Liu et al., 2019, Neu et al., 2017, Abbasi-Yadkori et al., 2019]. While this was good for laying down the foundations for future work, the reachability assumptions basically imply that the state space is already well-explored or rather easy to explore, which avoids the challenge of performing strategic exploration - one of the central problems in both empirical and theoretical RL.

To address this limitation, later works [Agarwal et al., 2020, Zanette et al., 2021] proposed policy optimization algorithms that enjoy polynomial sample-complexity guarantee without making any reachability assumption, but at the cost of either complicating the algorithm design (and analysis) with various tricks or getting highly suboptimal sample-complexity guarantees. For example, the PC-PG algorithm [Agarwal et al., 2020], which, to our knowledge was the first policy optimization algorithm for learning linear MDPs without reachability assumptions, requires ˜ O (poly( d ) /ϵ 11 ) samples to learn an ϵ -optimal policy for d -dimensional linear MDPs. That ˜ O (1 /ϵ 11 ) samples were necessary for this task is highly unlikely. Indeed, Zanette et al. [2021] greatly improved this sample complexity to ˜ O ( d 3 /ϵ 3 ) at the cost of considerably complicating the algorithm design and the analysis. Moreover, we are not aware of efficient guarantees of policy optimization for generic function approximation, which is rich enough to subsume linear MDPs. This motivates us to ask the following question:

Can we design a simple, general policy optimization algorithm with sharp sample-complexity guarantees in the exploration setting 2 ?

In particular, we aim to achieve sharper sample complexity than Zanette et al. [2021] in the linear MDP setting, and achieve low-degree polynomial sample complexity in the general function approximation setting. This paper answers the highlighted question affirmatively by making the following contributions:

Sharper rate. We propose a computationally efficient policy optimization algorithmOPTIMISTIC NPG with sample complexity

for learning an ϵ -optimal policy in an online fashion while interacting with a d -dimensional linear MDP. This result improves over the best previous one [Zanette et al., 2021] in policy optimization by a factor of d . Moreover, to our knowledge, this is the first computationally efficient algorithm to achieve the optimal quadratic dimension dependence. Before moving on, we remark that previous FQI-style algorithms [Zanette et al., 2020, Jin et al., 2021] achieve the optimal sample complexity ˜ Θ( d 2 /ϵ 2 ) [Zanette et al., 2020] but they are computationally inefficient due to the mechanism of global optimism in their algorithm design. Several very recent works [He et al., 2022, Agarwal et al., 2022, Wagenmaker et al., 2022] achieve the sample complexity ˜ O ( d 2 /ϵ 2 + d ≥ 4 /ϵ ) using computationally efficient algorithms. Compared to our work, the aforementioned rate has worse dependence on d and better dependence on ϵ . Nonetheless, our result is better in the practically important regime where the feature dimension d is typically very large and the target accuracy ϵ is not too small. To summarize, the sample complexity of OPTIMISTIC NPG strictly improves over the best existing policy optimization algorithms and is not dominated by that of any existing computationally efficient algorithm.

To our best knowledge, this paper also achieves the first polynomial sample-complexity guarantee for policy optimization under general function approximation.

Simple on-policy algorithm. At a high level, OPTIMISTIC NPG is almost an on-policy version of natural policy gradient (NPG) [Kakade, 2001] with Boltzmann policies that use linear function approximation for optimistic action-value estimates. By 'on-policy', we mean the transition-reward data used to improve the current policy are obtained exactly by executing the current policy. However, the algorithm has a tuneable parameter (denoted by m in Algorithm 1) that allows it to reuse data sampled from earlier policies. By using such data, the algorithm becomes off-policy . The optimal choice of the tuneable parameter is dictated by theoretical arguments. Interestingly, our analysis shows that even the purely on-policy version of the algorithm (i.e., set m = 1 in Algorithm 1) enjoys a well-controlled sample complexity of ˜ O ( d 2 /ϵ 4 ) . To the best of our knowledge, this is the first time an on-policy method is shown to have polynomial sample complexity in the exploration setting, given that all previous policy optimization or Q-learning style or model-based algorithms [e.g., Jin et al., 2020, Zanette et al., 2020, 2021, Agarwal et al., 2020, Cai et al., 2020, Shani et al., 2020, etc] are off-policy.

New proof techniques. To achieve the improved rate, we introduce several new ideas in the proof, including but not limited to (a) exploiting the softmax parameterization of the policies to reuse data and improve sample efficiency (Lemma 3), (b) controlling on-policy uncertainty (Lemma 4) instead of cumulative uncertainty as in previous off-policy works, and (c) using a bonus term that is smaller than those in previous works by a factor of √ d (see Lemma 2 and the discussions that follow).

1.1 Related works

Since this paper studies policy optimization algorithms in the setting of linear MDPs, below we restrict our focus to previous theoretical works on either policy optimization or linear MDPs.

Policy optimization. This work is inspired by and builds upon two recent works [Shani et al., 2020, Cai et al., 2020] that combine NPG with bonus-based optimism to handle exploration. In terms of algorithmic design, this paper (OPTIMISTIC NPG) utilizes on-policy fresh data for value estimation while Shani et al. [2020], Cai et al. [2020] are off-policy and reuse all the historical data. In terms of theoretical guarantees, Shani et al. [2020] and Cai et al. [2020] only study tabular MDPs and linear mixture MDPs [Zhou et al., 2021] respectively, while this paper considers the more challenging setting of linear MDPs [Jin et al., 2020] (in our view, linear MDPs are more challenging as there the number of model parameters scale with the number of states). We remark that due to some subtle technical challenge, so far it still remains unknown whether it is possible to generalize the analysis of Cai et al. [2020] to handle linear MDPs. Agarwal et al. [2020], Zanette et al. [2021] derive the first line of policy optimization results for RL in linear MDPs without any reachability- (or coverage-) style assumption. Compared to Agarwal et al. [2020], Zanette et al. [2021], our work considers the same setting but designs a simpler algorithm with cleaner analysis and sharper sample-complexity guarantee. Nevertheless, we remark that in certain model-misspecification settings, the algorithms of Agarwal et al. [2020], Zanette et al. [2021] can potentially achieve stronger guarantees. Since we focus on the well-specified setting (that is, the environment is perfectly modeled by a linear MDP), we refer interested readers to Agarwal et al. [2020] for more details. Finally, there have been a long line of works [e.g., Agarwal et al., 2021, Bhandari and Russo, 2019, Liu et al., 2019, Neu et al., 2017, Abbasi-Yadkori et al., 2019, etc] that study policy optimization under reachability (or coverage) assumptions, which eliminates the need for performing strategic exploration. Throughout this paper we do not make any such assumption and directly tackle the exploration challenge.

Linear MDPs. For linear MDPs, Jin et al. [2020] proposed a computationally efficient algorithm (LSVI-UCB) with ˜ O ( d 3 /ϵ 2 ) sample complexity. Later on, Zanette et al. [2020] utilized the idea of global optimism to obtain the optimal sample complexity ˜ Θ( d 2 /ϵ 2 ) at the cost of sacrificing computational efficiency. Recently, He et al. [2022], Agarwal et al. [2022], Wagenmaker et al. [2022] designed new computationally efficient algorithms that can achieve ˜ O ( d 2 /ϵ 2 + d ≥ 4 /ϵ ) sample complexity. Compared to the above works, the sample complexity of OPTIMISTIC NPG is not strictly worse than that of any known computationally efficient algorithm. In fact, it is the only computationally efficient method to achieve the optimal dimension dependence. Nonetheless, for

learning a near-optimal policy with a vanishing suboptimality ϵ , the current sample complexity of OPTIMISTIC NPG is loose by a factor of ϵ -1 .

2 Preliminaries

MDP. We consider the model of episodic Markov Decision Processes (MDPs). Formally, an MDP is defined by a tuple ( S , A , P , R, H ) where S denotes the set of states, A denotes the set of actions, both of which are assumed to be finite, P = { P h } h ∈ [ H ] denotes the set of transition probability functions so that P h ( s ′ | s, a ) is equal to the probability of transitioning to state s ′ given that action a is taken at state s and step h , R = { R h } h ∈ [ H ] denotes the collection of expected reward functions so that R h ( s, a ) is equal to the expected reward to be received if action a is taken at state s and step h , and H denotes the length of each episode. An agent interacts an MDP in the form of episodes. Formally, we assume without loss of generality that each episode always starts from a fixed initial state s 1 . At the h th step of this episode, the agent first observes the current state s h , then takes action a h and receives reward r h ( s h , a h ) satisfying

After that, the environment transitions to

The current episode terminates immediately once r H is received. Throughout this paper, we assume P and R are unknown to the learner.

Linear MDP. A d -dimensional linear MDP [Jin et al., 2020] is defined by two sets of feature mappings, { ϕ h } h ∈ [ H ] ⊆ ( S × A → R d ) and { ψ h } h ∈ [ H ] ⊆ ( S → R d ) , and a set of vectors { w ⋆ h } h ∈ [ H ] ⊆ R d so that the transition probability functions can be represented as bilinear functions of feature mappings { ϕ h } h ∈ [ H ] and { ψ h } h ∈ [ H ] , and the reward functions can be represented as linear functions of { ϕ h } h ∈ [ H ] . Formally, we have that for all ( s, a, s ′ ) ∈ S × A × S :

For the purpose the regularity, linear MDPs also require that

and for any function V h +1 : S → [0 , 1] ,

Throughout this paper, we assume only { ϕ h } h ∈ [ H ] is available to the learner while { ψ h } h ∈ [ H ] and { w ⋆ h } h ∈ [ H ] are not.

Policy and value. A (Markov) policy is a set of conditional probability functions π = { π h } h ∈ [ H ] so that π h ( · | s ) ∈ ∆ A gives the distribution over the action set conditioned on the current state s at step h . We define the V-value functions of policy π by { V π h } h ∈ [ H ] ⊆ ( S → R ) so that V π h ( s ) is equal to the expected cumulative reward an agent will receive if she follows policy π starting from state s and step h . Formally,

where the expectation is with respect to the randomness of the transition, the reward and the policy. Similarly, we can define the Q-value functions of policy π by { Q π h } h ∈ [ H ] ⊆ ( S × A → R ) so that Q π h ( s, a ) is equal to the expected cumulative reward an agent will receive if she follows policy π starting from taking action a at state s and step h . Formally,

We denote by π ⋆ = { π ⋆ h } h ∈ [ H ] the optimal policy such that π ⋆ ∈ argmax π V π h ( s ) for all ( s, h ) ∈ S × [ H ] . By backward induction, one can easily prove that there always exists an optimal Markov policy. For simplicity of notations, we denote by V ⋆ h = V π ⋆ h and Q ⋆ h = Q π ⋆ h the optimal value functions. Note that given an MDP, the optimal value functions are unique despite that there may exist multiple optimal policies.

Learning objective. The objective of this paper is to design an efficient policy optimization algorithm to learn an ϵ -optimal policy π such that V π 1 ( s 1 ) ≥ V ⋆ 1 ( s 1 ) -ϵ . Here the optimality is only measured by the value at the initial state s 1 , because (a) each episode always starts from s 1 , and (b) this paper studies the online exploration setting without access to a simulator, which means some states might be unreachable and learning optimal policies starting from those states is in general impossible.

3 Optimistic Natural Policy Gradient

In this section, we present the algorithm OPTIMISTIC NPG (Optimistic Natural Policy Gradient) and its theoretical guarantees.

3.1 Algorithm

The pseudocode of OPTIMISTIC NPG is provided in Algorithm 1. At a high level, the algorithm consists of the following three key modules.

Periodic on-policy data collection (Lines 4-6): Similarly to the empirical PPO algorithm [Schulman et al., 2017] , OPTIMISTIC NPG discards all the old data, and executes the current policy π k to collect a batch of fresh data D k after every m steps of policy update. These data will later be used to evaluate and improve the policies in the next m steps. Noticeably, this on-policy data mechanism is very different from most existing works in theoretical RL, where they either need to keep the historical data or have to rerun historical policies to refresh the dataset for the technical purpose of elliptical potential arguments in proofs. In comparison, OPTIMISTIC NPG only uses fresh data collected by the current (or very recent) policy, which resembles practical policy optimization algorithms such as PPO [Schulman et al., 2017] and TRPO [Schulman et al., 2015].

Optimistic policy evaluation (Line 8): Given the above collected dataset D k , we estimate the value functions of the current policy π k by invoking Subroutine OPE (optimistic policy evaluation). In Section 4, we show how to implement Subroutine OPE for tabular MDPs, linear MDPs and RL problems of low eluder dimension.

Policy update (Line 9): Given the optimistic value estimates { Q k h } h ∈ [ H ] of π k , OPTIMISTIC NPG performs one-step mirror ascent from π k h ( · | s ) with gradient Q k h ( s, · ) to obtain π k +1 h ( · | s ) at each ( h, s ) . Importantly, this step can be implemented in a computationally efficient way, because by the update rule of mirror ascent π k h ( · | s ) ∝ exp( η ∑ k -1 t =1 Q t h ( s, · )) , we only need to store {{ Q t h } h ∈ [ H ] } t ∈ [ K ] , from which any π k h ( · | s ) can be computed on the fly.

3.2 Theoretical guarantee

OPTIMISTIC NPG is a generic policy optimization meta-algorithm, and we can show it provably learns near-optimal policies as long as subroutine OPE satisfies the following condition.

Condition 1 (Requirements for OPE) . Suppose parameter m and η in OPTIMISTIC NPG satisfy m ≤ ( ηH 2 ) -1 . Then with probability at least 1 -δ , OPE ( π k , D k ) returns { Q k h } h ∈ [ H ] ⊆ ( S ×A → [0 , H ]) that satisfy the following properties for all k ∈ [ K ] .

Algorithm 1 OPTIMISTIC NPG

1: input : number of iterations K , period of collecting fresh data m , batch size N , learning rate η

2: initialize: for all ( h, s ) ∈ [ H ] ×S set π 1 h ( · | s ) = Uniform( A ) 3: for k = 1 , . . . , K do 4: if k ≡ 1( mod m ) then 5: D k ←{ N fresh trajectories i.i.d. ∼ π k } 6: else 7: D k ←D k -1 8: { Q k h } h ∈ [ H ] ← OPE ( π k , D k ) 9: for all ( h, s ) ∈ [ H ] ×S update π k +1 h ( · | s ) ∝ π k h ( · | s ) · exp( η · Q k h ( s, · )) 10: output : π out that is sampled uniformly at random from { π k } k ∈ [ K ]

(1B) Consistency: There exists an absolute complexity measure L ∈ R + such that

Condition (1A) requires that the Q-value estimate returned by OPE satisfies the Bellman equation under policy π k optimistically. Requiring optimism is very common in analyzing RL algorithms, as most algorithms rely on the principle of optimism in face of uncertainty to handle exploration. Condition (1B) requires that the degree of over-optimism at initial state s 1 is roughly of order √ 1 /N with respect to the number of samples N . This is intuitive since as more data are collected from policy π k or some policy similar to π k (as is enforced by the precondition m ≤ ( ηH 2 ) -1 ), the value estimate at the initial state should be more accurate. In Section 4, we will provide concrete instantiations of Subroutine OPE for tabular MDPs, linear MDPs and RL problems of low eluder dimension, all of which satisfy Condition 1 with mild complexity measure L .

Under Condition 1, we have the following theoretical guarantees for OPTIMISTIC NPG, which we prove in Appendix A.

Theorem 1. Suppose Condition 1 holds. In Algorithm 1, if we choose

then with probability at least 1 / 2 , π out is O ( ϵ ) -optimal.

Below we emphasize two special choices of m (the period of sampling fresh data) in Theorem 1:

When choosing m = 1 , OPTIMISTIC NPG is purely on-policy as it only uses data sampled from the current policy to perform policy optimization. In this case, the total sample complexity is

which, to our knowledge, is the first polynomial sample complexity guarantee for purely on-policy algorithms.

When choosing m = [ H/ϵ ] , we obtain sample complexity

which improves over the above purely on-policy version by a factor of H/ϵ . In Section 4.2, we will specialize this result to linear MDPs to derive a sample complexity that improves the best existing one in policy optimization [Zanette et al., 2021] by a factor of d .

Finally we remark that for cleaner presentation, we state Theorem 1 for a fixed, constant failure probability. However, the result is easy to extend to the case when the failure probability is some arbitrary value of δ ∈ (0 , 1) at the expense of increasing the sample complexity by a factor of log(1 /δ ) : one simply need to run Algorithm 1 for log(1 /δ ) times, estimate the values of every output policy to accuracy ϵ , and then pick the one with the highest estimate.

4 Examples

In this section, we implement subroutine OPE for tabular MDPs, linear MDPs and RL problems of low eluder dimension, and derive the respective sample complexity guarantees of OPTIMISTIC NPG.

4.1 Tabular MDPs

The implementation of tabular OPE (Subroutine 1) is rather standard: first estimate the transition and reward from dataset D , then plug the estimated transition-reward into the Bellman equation under policy π to compute the Q-value estimate backwards. And to guarantee optimism, we additionally add standard counter-based UCB bonus to compensate the error in model estimation. Formally, we have the following guarantee for tabular OPE.

Proposition 1 (tabular MDPs) . Suppose we choose α = Θ ( H √ log( KH |S||A| ) ) in Subroutine 1, then Condition 1 holds with L = SAH 3 .

By combining Proposition 1 with Theorem 1, we prove that OPTIMISTIC NPG with Subroutine 1 learns an ϵ -optimal policy within ( KN/m ) = ˜ O ( H 6 |S||A| /ϵ 3 ) episodes for any tabular MDP. This rate is strictly worse than the best existing result ˜ O ( H 3 |S||A| /ϵ 2 ) in tabular policy optimization [Wu et al., 2022]. Nonetheless, the proof techniques in [Wu et al., 2022] are specially tailored to tabular MDPs and it is highly unclear how to generalize them to linear MDPs due to certain technical difficulty that arises from covering a prohibitively large policy space. In comparison, our policy optimization framework easily extends to linear MDPs and provides improved sample complexity over the best existing one, as is shown in the following section.

4.2 Linear MDPs

We provide the instantiation of OPE for linear MDPs in Subroutine 2. At a high level, linear OPE computes an upper bound Q for Q π by using the Bellman equation under policy π backwards from step H to step 1 while adding a bonus to compensate the uncertainty of parameter estimation. Specifically, the step of ridge regression (Line 5) utilizes the linear completeness property of linear MDPs in computing Q h from V h +1 , which states that for any function V h +1 : S → R , there exists θ h ∈ R d so that ⟨ ϕ h ( s, a ) , θ h ⟩ = R h ( s, a ) + E s ′ ∼ P h ( ·| s,a ) [ V h +1 ( s ′ )] for all ( s, a ) ∈ S × A . And in Line 6, we add an elliptical bonus b h ( s, a ) to compensate the error between our estimate ˆ θ h and the groundtruth θ h so that we can guarantee Q h ( s, a ) ≥ ( T π h Q h +1 )( s, a ) for all ( s, a, h ) with high probability .

Subroutine 2 Linear OPE ( π, D )

Subroutine 3 General OPE ( π, D )

1: required parameter : β

2: split D evenly into H disjoint subsets D 1 , . . . , D H

3: set V H +1 ( s ) ← 0 for all s ∈ S

4: for h = H : 1 do

5: for ( s, a ) ∈ S × A do

6: construct confidence set

with loss function defined as

7: compute

8: output { Q h } H h =1

Proposition 2 (linear MDPs) . Suppose we choose λ = 1 and α = Θ ( H √ d log( KN ) ) in Subroutine 2, then Condition 1 holds with L = d 2 H 3 .

By combining Proposition 2 with Theorem 1, we obtain that ( KN/m ) = ˜ O ( d 2 H 6 /ϵ 3 ) episodes are sufficient for OPTIMISTIC NPG to learn an ϵ -optimal policy, improving upon the state-of-the-art policy optimization results [Zanette et al., 2021] by a factor of d . Notably, this is also the first computationally efficient algorithm to achieve optimal quadratic dimension dependence for learning linear MDPs. The key factor behind this improvement is OPTIMISTIC NPG's periodic collection of fresh on-policy data, which eliminates the undesired correlation and avoids the union bound over certain nonlinear function class that are commonly observed in previous works. Consequently, OPTIMISTIC NPG uses a bonus function that is √ d times smaller than in previous works [e.g., Jin et al., 2020]. For further details on how we achieve this improvement, we refer interested readers to Appendix A, specifically Lemma 2.

4.3 General function approximation

Now we instantiate OPE for RL with general function approximation. In this setting, the learner is provided with a function class F = F 1 ×··· × F H for approximating the Q values of polices, where F h ⊆ ( S × A → [0 , H ]) .

General OPE. The pseudocode is provided in Subroutine 3. At each step h ∈ [ H ] , general OPE first constructs a confidence set B h which contains candidate estimates for T π h Q h +1 (Line 6). Specifically, B h consists of all the value candidates f h ∈ F h whose square temporal difference (TD) error on dataset D h is no larger than the smallest one by an additive factor β . Such construction can be viewed as a relaxation of the classic fitted Q-iteration (FQI) algorithm which only keeps the value candidate with the smallest TD error. In particular, if we pick β = 0 , B h collapses to the solution of FQI. Equipped with confidence set B h , general OPE then perform optimistic planning at every ( s, a ) ∈ S × A by computing Q h ( s, a ) to be the largest possible value that can be achieved by any candidate in confidence set B h (Line 7). We remark that general OPE shares a similar algorithmic spirit to the GOLF algorithm [Jin et al., 2021]. The key difference is that GOLF is a purely value-based algorithm and only performs optimistic planning at the initial state while general OPE performs optimistic planning at every state and is part of a policy gradient algorithm.

Theoretical guarantee. To state the main theorem, we need to first introduce two standard concepts: value closeness and eluder dimension, which have been widely used in previous works that study RL with general function approximation.

Assumption 1 (value closeness [Wang et al., 2020]) . For all h ∈ [ H ] and V h +1 : S → [0 , H ] , T h V h +1 ∈ F h where [ T h V h +1 ]( s, a ) = R h ( s, a ) + E [ V h +1 ( s ′ ) | s ′ ∼ P h ( · | s, a )] .

Intuitively, Assumption 1 can be understood as requiring that the application of the Bellman operator T h to any V-value function V h +1 results in a function that belongs to the value function class F h . This assumption holds in various settings, including tabular MDPs and linear MDPs. Furthermore, it is reasonable to assume that value closeness holds whenever the function class F has sufficient expressive power, such as the class of neural networks.

Definition 1 (eluder dimension [Russo and Van Roy, 2013]) . Let G be a function class from X to R and ϵ > 0 . We define the ϵ -eluder dimension of G , denoted as d E ( G , ϵ ) , to be the largest L ∈ N + such that there exists x 1 , . . . , x L ∈ X and g 1 , g ′ 1 , . . . , g L , g ′ L ∈ G satisfying: for all l ∈ [ L ] , ∑ i<l ( g l ( x i ) -g ′ l ( x i )) 2 ≤ ϵ but g l ( x l ) -g ′ l ( x l ) ≥ ϵ .

At a high level, eluder dimension d E ( G , ϵ ) measures how many mistakes we have to make in order to identify an unknown function from function class G to accuracy ϵ , in the worst case. It has been widely used as a sufficient condition for proving sample-efficiency guarantee for optimistic algorithms in RL with general function approximation, e.g., Wang et al. [2020], Jin et al. [2021].

Now we state the theoretical guarantee for general OPE.

Proposition 3 (general function approximation) . Suppose Assumption 1 holds and we choose β = Θ ( H 2 log( |F| NK/δ ) ) in Subroutine 3, then Condition 1 holds with L = H 3 log( |F| ) max h d E ( F h , 1 /N ) .

Plugging Proposition 3 back into Theorem 1, we obtain that ( KN/m ) = ˜ O ( d E log( |F| ) H 6 /ϵ 3 ) episodes are sufficient for OPTIMISTIC NPG to learn an ϵ -optimal policy, which to our knowledge is the first polynomial sample complexity guarantee for policy optimization with general function approximation.

5 Conclusions

We proposed a model-free, policy optimization algorithm-OPTIMISTIC NPG for online RL and analyzed its behavior in the episodic setting. In terms of algorithmic design, it is not only considerably simpler but also more closely resembles the empirical policy optimization algorithms (e.g., PPO, TRPO) than the previous theoretical algorithms. In terms of sample efficiency, for d -dimensional linear MDPs, it improves over state-of-the-art policy optimization algorithm by a factor of d , and is the first computationally efficient algorithm to achieve the optimal dimension dependence. To our best knowledge, OPTIMISTIC NPG is also the first sample-efficient policy optimization algorithm under general function approximation.

For future research, we believe that it is an important direction to investigate the optimal complexity for using policy optimization algorithms to solve linear MDPs. Despite the optimal dependence on dimension d , the current sample complexity of OPTIMISTIC NPG is worse than the optimal rate by a factor of 1 /ϵ . It remains unclear to us how to shave off this 1 /ϵ factor to achieve the optimal rate.

References

Yasin Abbasi-Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24, 2011.

Yasin Abbasi-Yadkori, Peter Bartlett, Kush Bhatia, Nevena Lazic, Csaba Szepesvari, and Gellért Weisz. Politex: Regret bounds for policy iteration using expert prediction. In International Conference on Machine Learning , pages 3692-3702. PMLR, 2019.

Alekh Agarwal, Mikael Henaff, Sham Kakade, and Wen Sun. Pc-pg: Policy cover directed exploration for provable policy gradient learning. Advances in neural information processing systems , 33: 13399-13412, 2020.

Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. J. Mach. Learn. Res. , 22(98): 1-76, 2021.

Alekh Agarwal, Yujia Jin, and Tong Zhang. VO q L: Towards optimal regret in model-free RL with nonlinear function approximation. arXiv preprint arXiv:2212.06069 , 2022.

Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Józefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680 , 2019.

Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. arXiv preprint arXiv:1906.01786 , 2019.

Qi Cai, Zhuoran Yang, Chi Jin, and Zhaoran Wang. Provably efficient exploration in policy optimization. In International Conference on Machine Learning , pages 1283-1294. PMLR, 2020.

Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning , pages 49-58. PMLR, 2016.

Jiafan He, Heyang Zhao, Dongruo Zhou, and Quanquan Gu. Nearly minimax optimal reinforcement learning for linear Markov decision processes. arXiv preprint arXiv:2212.06132 , 2022.

Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory , pages 2137-2143. PMLR, 2020.

Chi Jin, Qinghua Liu, and Sobhan Miryoosefi. Bellman eluder dimension: New rich classes of RL problems, and sample-efficient algorithms. Advances in neural information processing systems , 34:13406-13418, 2021.

Sham M Kakade. A natural policy gradient. Advances in neural information processing systems , 14, 2001.

Boyi Liu, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural proximal/trust region policy optimization attains globally optimal policy. arXiv preprint arXiv:1906.10306 , 2019.

Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of entropy-regularized Markov decision processes. arXiv preprint arXiv:1705.07798 , 2017.

OpenAI. ChatGPT: Optimizing language models for dialogue, 2022. URL https://openai.com/ blog/chatgpt/ .

Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems , 26, 2013.

John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning , pages 1889-1897. PMLR, 2015.

John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 , 2017.

Lior Shani, Yonathan Efroni, Aviv Rosenberg, and Shie Mannor. Optimistic policy optimization with bandit feedback. In International Conference on Machine Learning , pages 8604-8613. PMLR, 2020.

Andrew J Wagenmaker, Yifang Chen, Max Simchowitz, Simon Du, and Kevin Jamieson. Reward-free RL is no harder than reward-aware RL in linear Markov decision processes. In International Conference on Machine Learning , pages 22430-22456. PMLR, 2022.

Ruosong Wang, Russ R Salakhutdinov, and Lin Yang. Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension. Advances in Neural Information Processing Systems , 33:6123-6135, 2020.

Tianhao Wu, Yunchang Yang, Han Zhong, Liwei Wang, Simon Du, and Jiantao Jiao. Nearly optimal policy optimization with stable at any time guarantee. In International Conference on Machine Learning , pages 24243-24265. PMLR, 2022.

Andrea Zanette, Alessandro Lazaric, Mykel Kochenderfer, and Emma Brunskill. Learning near optimal policies with low inherent Bellman error. In International Conference on Machine Learning , pages 10978-10989. PMLR, 2020.

Andrea Zanette, Ching-An Cheng, and Alekh Agarwal. Cautiously optimistic policy optimization and exploration with linear function approximation. In Conference on Learning Theory , pages 4473-4525. PMLR, 2021.

Dongruo Zhou, Quanquan Gu, and Csaba Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture Markov decision processes. In Conference on Learning Theory , pages 4532-4576. PMLR, 2021.

A Proof of Theorem 1

Recall in Algorithm 1, π out is sampled uniformly at random from { π k } k ∈ [ K ] , so we have

To control Term (I), we import the following generalized policy difference lemma from [Shani et al., 2020, Cai et al., 2020].

Lemma 1 (generalized policy-difference lemma) . For any policy π and k ∈ [ K ] ,

By invoking Lemma 1 with π = π ⋆ , we have

where the second inequality follows from the standard regret bound of online mirror ascent. By Condition (1A) , the second term on the RHS of Equation (2) is upper bounded by zero. So we have

As for Term (II), we can directly upper bound it by Condition (1B) which gives:

We complete the proof by plugging in the choice of K , N , η .

B Proofs for Examples

In this section, we prove the three instantiations of OPE in Section 4 satisfy Condition 1 with mild complexity measure L . The proofs of the lemmas used in this section can be found in Appendix C.

B.1 Proof of Proposition 2 (linear OPE)

We start with the following lemma showing that Q k h is an optimistic estimate of T π k h Q k h +1 and the degree of over-optimism can be bounded by the bonus function.

Lemma 2 (optimism of value estimates) . Under the same choice of λ and α as in Proposition 2, with probability at least 1 -δ : for all ( k, h, s, a ) ∈ [ K ] × [ H ] ×S × A :

where { b k h } h ∈ [ H ] are the bonus functions defined in Line 6, Subroutine 2 linear OPE ( π k , D k ) .

Importantly, Lemma 2 holds with a bonus function that is smaller by a factor of √ d than the typical one used in linear function approximation [e.g., Jin et al., 2020], which is key to obtaining the optimal dimension dependence. The main reason that this smaller bonus function suffices is that we recollect fresh samples every m iterations and split the data (Line 2 in Subroutine 2) to eliminate the correlation between different steps h ∈ [ H ] .

Lemma 2 immediately implies that Condition (1A) holds. Below we prove Condition (1B).

By the second inequality in Lemma 2, the definition of V k and the Bellman equation under policy π k , one can show

Denote by t k the index of the last iteration of collecting fresh data before the k th iteration. By exploiting the softmax parameterization structure of { π k } k ∈ [ K ] , one can show that the visitation measure over state-action pairs induced by executing policy π k is actually very close to the one induced by π t k in the following sense.

Lemma 3 (policy Lipschitz) . Suppose we choose η and m such that ηm ≤ 1 /H 2 , then for any k ∈ N + and any function f : S × A → R + :

By combining Lemma 3 with Equation (3) and noticing that b k h = b t k h , we have

Since the bonus function b t k h is defined by using the dataset D t k collected from executing policy π t k , one could expect its average under π t k should be rather small when the batch size N is large enough, as formalized in the following lemma.

Lemma 4 (on-policy uncertainty) . Let π be an arbitrary policy and λ ≥ 1 . Suppose we sample { ( s n h , a n h ) } N n =1 i.i.d. from π . Denote Σ h = ∑ N i =1 ϕ h ( s i h , a i h ) ϕ T h ( s i h , a i h ) . Then with probability at least 1 -δ :

As a result, by Lemma 4 and the definition of b k h , we have that with probability at least 1 -δ : for all k ∈ [ K ] ,

where in applying Lemma 4 we use the fact |D k h | = N/H due to data splitting. Putting all pieces together, we get

As a result, Condition (1B) holds with L = d 2 H 3 .

B.2 Proof of Proposition 1 (tabular OPE)

The proof follows basically the same as that of Proposition 2. The only modification needed is to replace Lemma 2 and 4 by their tabular counterparts, which we state below.

Lemma 5 (optimism of value estimates: tabular) . Under the same choice of α as in Proposition 1, with probability at least 1 -δ : for all ( k, h, s, a ) ∈ [ K ] × [ H ] ×S × A :

where { b k h } h ∈ [ H ] are the bonus functions defined in Line 6, Subroutine 1 tabular OPE ( π k , D k ) .

Lemma 6 (on-policy uncertainty: tabular) . Let π be an arbitrary policy. Suppose we sample { ( s n h , a n h ) } N n =1 i.i.d. from π . Denote J h ( s, a ) = ∑ N i =1 1 (( s i h , a i h ) = ( s, a )) . Then with probability at least 1 -δ :

B.3 Proof of Proposition 3 (general OPE)

The proof follows basically the same as that of Proposition 2. The only modification needed is to replace Lemma 2 and 4 by their general counterparts, which we state below.

Lemma 7 (optimism of value estimates: general) . Suppose Assumption 1 holds. There exists an absolute constant c such that with probability at least 1 -δ : for all ( k, h, s, a ) ∈ [ K ] × [ H ] ×S×A :

Lemma 8 (on-policy uncertainty: general) . Let π be an arbitrary policy. Suppose we sample D h = { ( s n h , a n h ) } N n =1 i.i.d. from π . Then with probability at least 1 -δ :

C Proofs of Lemmas

C.1 Proof of Lemma 1

Lemma 1 is an immediate consequence of Lemma 4.2 in [Cai et al., 2020] or Lemma 1 in [Shani et al., 2020].

C.2 Proof of Lemma 2

Let us consider a fixed pair ( k, h ) ∈ [ K ] × [ H ] . Recall Q k h is defined as

where

By the standard linear completeness property of linear MDPs Jin et al. [2020], there exists θ k h such that ∥ θ k h ∥ 2 ≤ H √ d and

Therefore, to prove Lemma 2, it suffices to show that for all ( s, a )

To condense notations, denote by { ( s i h , a i h , r i h , s i h +1 ) } i M =1 the tuples in D k h . By the definition of ridge regression,

where the inequality follows from ∥ λθ k h ∥ (Σ k h + λ I d × d ) -1 ≤ √ λ ∥ θ k h ∥ 2 ≤ H √ d. It remains to bound the first term in the bracket. Now here comes the key observation that helps shave off the √ d factor from the bonus.

Observation 1. D k h and V k h +1 are independent conditioning on π t k where t k denotes the index of the last iteration of collecting fresh data before the k th iteration.

To see why, recall that after obtaining D t k Algorithm 2 splits it evenly into H disjoint subsets {D t k h } H h =1 and then only uses {D t k h ′ } H h ′ = h +1 for evaluating and improving { π l h ′ } H h ′ = h +1 for l ∈ [ t k , t k + m -1] . As a result, { V l h +1 } t k + m -1 l = t k (including V k h +1 ) is independent of D t k h = D k h conditioning on π t k . By the concentration of self-normalized processes [Abbasi-Yadkori et al., 2011], we conclude that with probability at least 1 -δ : for all ( k, h ) ∈ [ K ] × [ H ]

C.3 Proof of Lemma 3

We first introduce the following two auxiliary lemmas about the lipschitzness continuity of softmax paraterization.

Lemma9. There exists an absolute constant c > 0 such that for any policy π, ˆ π satisfying ˆ π h ( a | s ) ∝ π h ( a | s ) × exp( L h ( s, a )) where { L h ( s, a ) } h ∈ [ H ] is set of functions from S × A to [ -1 /H, 1 /H ] , we have that for any τ H := ( s 1 , a 1 , . . . , s H , a H ) ∈ ( S × A ) H :

Proof of Lemma 9. By using the relation between π and ˆ π and the normalization property of policy π , we have that for any ( h, s, a ) :

Therefore, for any τ H := ( s 1 , a 1 , . . . , s H , a H )

Lemma 10. Let µ, ν be two probability densities defined over X such that ∥ µ/ν ∥ ∞ ≤ α , then we have that for any function f : X → R + , E µ [ f ( x )] ≤ α E ν [ f ( x )] .

By the update rule of NPG, we have

Since we choose η and m such that ηm ≤ 1 /H 2 ,

Therefore, by invoking Lemma 9 with ˆ π = π k and π = π t k , we have that for any τ H ∈ ( S × A ) H :

By further invoking Lemma 10 with X = ( S × A ) H , µ = P π k and ν = P π t k , we obtain that for any function f : S × A → R + :

Similarly, we can show E π t k [ f ( s h , a h )] = O ( E π k [ f ( s h , a h )]) , which completes the proof of Lemma 3.

C.4 Proof of Lemma 4

To simplify notations, denote ϕ i h := ϕ h ( s i h , a i h ) . For the technical purpose of applying martingale concentration, we additionally define the intermediate empirical covariance matrices: Σ n h = ∑ n -1 i =1 ϕ i h ( ϕ i h ) T , for n ∈ [ N ] . We have that with probability at least 1 -δ ,

where the first inequality uses Σ n h ⪯ Σ h , the second one uses Azuma-Hoeffding inequality with λ = 1 , and the last one uses Cauchy-Schwarz inequality and the the standard elliptical potential argument.

C.5 Proof of Lemma 5

Let us consider a fixed pair ( k, h ) ∈ [ K ] × [ H ] . Recall Q k h is defined as

where ˆ P k h and ˆ R k h are the empirical estimates of transition and reward at step h from D k h , and b k h ( s, a ) = α ( J k h ( s, a ) + 1) -1 / 2 with J k h ( s, a ) = ∑ ( s h ,a h ) ∈D k h 1 (( s h , a h ) = ( s, a )) . Therefore, to prove Lemma 5, it suffices to show that for all ( s, a )

By observation 1, we know { V l h +1 } t k + m -1 l = t k is independent of D t k h = D k h conditioning on π t k , which implies V k h +1 is independent of ˆ P k h . Therefore, by Azuma-Hoeffding inequality and standard union bound, we conclude that with probability at least 1 -δ : for all ( k, h, s, a ) ∈ [ K ] × [ H ] ×S × A ,

C.6 Proof of Lemma 6

For the technical purpose of performing martingale concentration, we introduce the notion of intermediate counters: J n h ( s, a ) = ∑ n -1 i =1 1 (( s i h , a i h ) = ( s, a )) , for n ∈ [ N ] . We have that with probability at least 1 -δ ,

where the first inequality uses J n h ( s, a ) ≤ J h ( s, a ) for all ( s, a ) ∈ S × A , the second one uses Azuma-Hoeffding inequality, and the last one follows the standard pigeon-hole argument.

C.7 Proof of Lemma 7

By observation 1, we know D k h is independent of V k h +1 conditioning on π t k where t k is the index of the last iteration when OPTIMISTIC NPG collects fresh data before iteration k +1 . With this independence relation in mind, we can easily prove that confidence set B k h satisfy the following properties by standard concentration inequality and union bounds.

Lemma 11. Suppose Assumption 1 holds. There exists an absolute constant c such that with probability at least 1 -δ , for all k ∈ [ K ] and h ∈ [ H ] ,

The proof of Lemma 11 follows trivially from modifying the proofs of Lemma 39 and 40 in Jin et al. [2021], which we omit here.

Define

By using the first relation in Lemma 11 and the definition of Q k h , we immediately obtain that

and

where the last inequality uses ¯ B k h ⊂ B k h by Lemma 11.

C.8 Proof of Lemma 8

Recall we define

which implies that b h ( s, a, D h ) ≤ b h ( s, a, D h ) for any D h ⊆ D h . Let D ( n ) h = { ( s i h , a i h ) } n i =1 . We have that with probability at least 1 -δ ,

where the second inequality uses Azuma-Hoeffding inequality and the last one uses the standard regret guarantee for eluder dimension (e.g., Lemma 2 in Russo and Van Roy [2013]).

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a simple and efficient policy optimization method with rigorous theoretical guarantee. The algorithm combines the natural gradient ascent and the optimistic off policy evaluation and is computationally efficient. They provide the condition under which the algorithm is guaranteed to be efficient and can provably learn the optimal policy. The authors also gave three examples to show the efficiency of the algorithm: tabular case, linear function approximation and general function approximation. They show that their algorithm achieves the optimal dependence on d for linear case which improves the algorithm proposed by Zanette et al, 2021 by a factor of d. They also show that their algorithm is the first one to achieve polynomial sample complexity in general function approximation case. \n\nIn general, the proof is sound and also the writing is very good and clear. I like the simplicity of their algorithm.\n\nSTRENGTHS:\n1. The algorithm is simple, efficient and easy to understand.\n2. They provide a sound argument of theory and the proof is sound to my knowledge.\n3. In linear case, their algorithm achieves the optimal dependence w.r.t. the dimension d with computational efficiency.\n4. Besides 3, they also provide the sample complexity in tabular and general function approximation case.\n\nWEAKNESSES:\nI do not think this paper has any apparent weakness. Though, I still have some minor questions about it.\n\n1. In Theorem 1, you claim the result holds with probability at least 1/2, and in Line 207-211,  you claim that you can achieve the probability 1-\\delta for any \\delta with a sample complexity multiplies by a log factor. I understand that you only need to run the algorithm one for log(1/\\delta) times, but how do you 'estimate the values of every output policy to accuracy \\eps'? Can you show exactly how you do to achieve this? In this estimation stage, do you need to sample additional online data using the output policy or not?\n\n2. In line 223-226, you claim the best sample complexity for policy optimization in the tabular case is H^4 S A / \\eps^2. I am not sure whether this is true. Indeed, the Wu et al. you cited expressed their result in the form of regret, and you can of course uniformly sample one policy from the policies output in Wu et al and get the sample complexity. However, in this paper (https://arxiv.org/pdf/2007.13442.pdf), they propose the algorithm BPI-UCBVI for best policy identification (another name of 'policy optimization'?), and they showed the sample complexity to be H^3 S A / \\eps^2 log(1/\\delta). (This is the second algorithm in their paper and the first one is reward-free version. Even the traditional UCBVI algorithm should be able to achieve the H^3 S A / \\eps^2 * polylog(1/\\delta). I wonder whether they considered a different setting as you did and whether result you claimed is the best one.\n\n3. In all three subroutines about OPE, you split the data into H even disjoint subset. Do you think this splitting is inefficient? I remember in some papers, such as this one (https://arxiv.org/pdf/2106.04895.pdf), they split the data into three instead of H subsets to reduce the sample complexity. Do you think this will help you further reduce the sample complexity in your algorithm?\n\nQUESTIONS:\n/\n\nLIMITATIONS:\na typo:\nLine 284: in the worse case --> in the worst case.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents $\\texttt{OPTIMISTIC NPG}$, a new algorithm that combines the natural policy gradient with optimistic policy evaluation to encourage exploration in online RL. The algorithm demonstrates computational efficiency and achieves optimal sample complexity in linear MDPs, outperforming existing state-of-the-art approaches. Furthermore, it extends to general function approximation, making it the first algorithm to achieve polynomial sample complexity for learning near-optimal policies in this broader setting.\n\nSTRENGTHS:\n- The authors propose a computationally efficient policy optimization algorithm with improved sample complexity in $d$-dimensional linear MDPs.\n- The authors Introduce the first on-policy method in the exploration setting that achieves polynomial sample complexity, contrasting with previous off-policy algorithms.\n- The authors extend the linearMDP algorithm to a general function approximation algorithm, exhibiting polynomial sample complexity measured by the eluder dimension of a function class.\n\nWEAKNESSES:\n- The technical novelty of the algorithm for a general function class appears unclear, and further explanation is needed on how to construct confidence sets when working with function classes beyond linear functions.\n- Excluding the issue of computational intractability, Subroutine 3 appears to be applicable to general function classes with bounded eluder dimension. For example, Dong et al. (2021) proved that \"one-layer neural nets do not have polynomially-bounded eluder dimension,\" presenting an exponential eluder dimension. It is challenging to claim that the algorithm proposed in this paper is sample efficient for function classes with such characteristics.\n\n\\--\n\nDong, K., Yang, J., & Ma, T. (2021). Provable model-based nonlinear bandit and reinforcement learning: Shelve optimism, embrace virtual curvature. Advances in Neural Information Processing Systems, 34, 26168-26182.\n\nQUESTIONS:\n- On line 138, there appears to be a typo in the equation where $\\psi$, a $d$-dimensional vector, is integrated with respect to s, resulting in a real number bound.\n- In comparison to traditional off-policy policy gradient algorithms, what practical advantages does the on-policy algorithm offer?\n\nLIMITATIONS:\nSince this is a theoretical paper, its direct societal impact seems to be neutral.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
This work proposes a simple-efficient efficient policy optimization algorithm for online RL. When specialized to linear MDPs, the algorithm improves the best known result by a factor of $d$. Moreover, this is the first efficient policy optimization algorithm under general function approximation.

STRENGTHS:
1 This work is well-written and well-organized.

2 This work showcase a generic policy optimization meta-algorithm can be sample efficient. 

3 The theoretical guarantee outperforms the exisiting result in linear MDPs, and is the first sample efficient policy optimiztion algorithm under general function approximation.

WEAKNESSES:
1 When specialized to tabular scenario, the result is worse than the best existing result.

2 Under linear MDPs, the dependency on $\epsilon$ is not optimal.

QUESTIONS:
1 When specialized to linear MDPs, the algorithm is computationally efficient. When specialized to general function approximation, the algorithm is not computationally efficient due to the hardness of selecting optimitic Q-function in the confidence set. From what I know, [Ruosong Wang et al.] claims they developed a provably efficient (both computationally and statiscally) Q-learning algorithm. My question is that for the policy optimization under general function approximation, is it possible to devise a computationally efficient algorithm under the proposed optimistic NPG framework?

2 Can the authors comment on the subtle technical challenge in [Cai et al.] (linear mixture MDPs) in dealing with linear MDPs? Is the difficulty caused by covering number argument?

[Ruosong Wang et al.] Reinforcement Learning with General Value Function Approximation: Provably Efficient Approach via Bounded Eluder Dimension

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07

comments: SUMMARY:
This paper proposes optimistic NPG for online RL, which is computationally efficient and enjoys the polynomial sample complexity while learning near-optimal policies. The paper is well-written and organization is clear.

STRENGTHS:
There are several strengths:
1. Sample complexity benefits: compared with other computationally efficient policy optimization algorithms, the proposed algorithm in this work has better sample complexity guarantees; (I think this is the major strength)
2. The first polynomially sampled policy optimization algorithm under general function approximation.
3. Literature comparison and position is clear. The paper is well-written!

WEAKNESSES:
I think this paper is clear and I do not have concern for now.

QUESTIONS:
I do not have question on this paper for now.

LIMITATIONS:
As the authors mentioned in the Conclusions Section, I am also curious how to avoid the factor of 1/epsilon compared with the optimal rate.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-23

comments: SUMMARY:
This paper presents a model-free policy optimization algorithm Optimistic Natural Policy Gradient for online and episodic MDPs. The authors present sample complexity results which are better than existing results for linear MDPs. This makes it computationally efficient and optimal dimension dependence, first of its kind. The algorithm is also similar to popular RL algorithms like PPO which makes it even more interesting.

STRENGTHS:
Technically solid paper with state of the art sample complexity results. The contributions are very well highlighted. The details of the algorithm are well presented by highlighting the key three modules in the algorithm. The paper also studies the online exploration scenario without access to a simulator.

WEAKNESSES:
Agarwal et. al [2020] did have some experimental results on some environments to verify indeed the proposed algorithm enjoys what is proved theoretically. In this paper, any experimental proof of "computationally efficient" is missing.

QUESTIONS:
1. Zanette et al [2020] were able to improve the sample complexity of FQI-style algorithm by paying the price in computation. What is that the proposed algorithm compensates on, if anything, to achieve optimal rates?

LIMITATIONS:
-
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-08-01