Title: A Theoretical Analysis of Optimistic Proximal Policy Optimization in Linear Markov Decision Processes
Venue: NeurIPS 2023
Accepted: yes
Keywords: policy optimization

adversarial lienar MDPs

RL theory
Abstract: The proximal policy optimization (PPO) algorithm stands as one of the most prosperous methods in the field of reinforcement learning (RL). Despite its success, the theoretical understanding of PPO remains deficient. Specifically, it is unclear whether PPO or its optimistic variants can effectively solve linear Markov decision processes (MDPs), which are arguably the simplest models in RL with function approximation. 
    To bridge this gap, we propose an optimistic variant of PPO for episodic adversarial linear MDPs with full-information feedback, and establish a $\tilde{\mathcal{O}}(d^{3/4}H^2K^{3/4})$ regret for it. Here $d$ is the ambient dimension of linear MDPs, $H$ is the length of each episode, and $K$ is the number of episodes. Compared with existing policy-based algorithms, we achieve the state-of-the-art regret bound in both stochastic linear MDPs and adversarial linear MDPs with full information. Additionally, our algorithm design features a novel multi-batched updating mechanism and the theoretical analysis utilizes a new covering number argument of value and policy classes, which might be of independent interest.

A Theoretical Analysis of Optimistic Proximal Policy Optimization in Linear Markov Decision Processes

Abstract

The proximal policy optimization (PPO) algorithm stands as one of the most prosperous methods in the field of reinforcement learning (RL). Despite its success, the theoretical understanding of PPO remains deficient. Specifically, it is unclear whether PPO or its optimistic variants can effectively solve linear Markov decision processes (MDPs), which are arguably the simplest models in RL with function approximation. To bridge this gap, we propose an optimistic variant of PPO for episodic adversarial linear MDPs with full-information feedback, and establish a ˜ O ( d 3 / 4 H 2 K 3 / 4 ) regret for it. Here d is the ambient dimension of linear MDPs, H is the length of each episode, and K is the number of episodes. Compared with existing policy-based algorithms, we achieve the state-of-the-art regret bound in both stochastic linear MDPs and adversarial linear MDPs with full information. Additionally, our algorithm design features a novel multi-batched updating mechanism and the theoretical analysis utilizes a new covering number argument of value and policy classes, which might be of independent interest.

1 Introduction

Reinforcement learning (RL) [34] is a prominent approach to solving sequential decision making problems. Its tremendous successes [19, 32, 33, 5] can be attributed, in large part, to the advent of deep learning [22] and the development of powerful deep RL algorithms [24, 28, 29, 11]. Among these algorithms, the proximal policy optimization (PPO) [29] stands out as a particularly significant approach. Indeed, it continues to play a pivotal role in recent advancements in large language models [27].

Motivated by the remarkable empirical success of PPO, numerous studies seek to provide theoretical justification for its effectiveness. In particular, Cai et al. [6] develop an optimistic variant of the PPO algorithm in adversarial linear mixture MDPs with full-information feedback [4, 25], where the transition kernel is a linear combination of several base models. Theoretically, they show that the optimistic variant of PPO is capable of tackling problems with large state spaces by establishing a sublinear regret that is independent of the size of the state space. Building upon this work, He et al. [14] study the same setting and refine the regret bound derived by Cai et al. [6] using the weighted regression technique [44]. However, the algorithms in Cai et al. [6], He et al. [14] and other algorithms for linear mixture MDPs [4, 44] are implemented in a model-based manner and require an integration of the individual base model, which can be computationally expensive or even intractable in general. Another arguably simplest RL model involving function approximation is linear MDP [38, 17], which assumes that the reward functions and transition kernel enjoy a low-rank representation. For this model, several works [17, 15, 3, 13] propose value-based algorithms that directly approximate the value function and provide regret guarantees. To demonstrate the efficiency of PPO in linear MDPs from a theoretical perspective, one potential approach is to extend the results

of Cai et al. [6], He et al. [14] to linear MDPs. However, this extension poses significant challenges due to certain technical issues that are unique to linear MDPs. See §1.1 for a detailed description.

In this paper, we address this technical challenge and prove that the optimistic variant of PPO is provably efficient for stochastic linear MDPs and even adversarial linear MDPs with full-information feedback. Our contributions are summarized below.

In terms of algorithm design, we propose a new algorithm OPPO+ (Algorithm 1), an optimistic variant of PPO, for adversarial linear MDPs with full-information feedback. Our algorithm features two novel algorithm designs including a multi-batched updating mechanism and a policy evaluation step via average rewards .

Theoretically, we establish a ˜ O ( d 3 / 4 H 2 K 3 / 4 ) regret for OPPO+ , where d is the ambient dimension of linear MDPs, H is the horizon, and K is the number of episodes. To achieve this result, we employ two new techniques. Firstly, we adopt a novel covering number argument for the value and policy classes, as explicated in §C. Secondly, in Lemma 4.3, we meticulously analyze the drift between adjacent policies to control the error arising from the policy evaluation step using average rewards.

Compared with existing policy optimization algorithms, our algorithm achieves a better regret guarantee for both stochastic linear MDPs and adversarial linear MDPs with full-information feedback (to our best knowledge). See Table 1 for a detailed comparison.

In summary, our work provides a new theoretical justification for PPO in linear MDPs. To illustrate our theory more, we highlight the challenges and our novelties in §1.1.

1.1 Challenges and Our Novelties

Challenge 1: Covering Number of Value Function Class. In the analysis of linear MDPs (see Lemma B.3 or Lemma B.3 in Jin et al. [17]), we need to calculate the covering number of V k h , which is the function class of the estimated value function at the h -th step of the k -th episode and takes the following form:

where Q k h and π k h are estimated Q-function and policy at the h -th step of the k -th episode, respectively. For value-based algorithms (e.g., LSVI-UCB in Jin et al. [17]), π k h is the greedy policy with respect to Q k h . Then we have

Since max a is a contraction map, it suffices to calculate N ( Q k h ) , where Q k h is the function class of Q k h and N ( · ) denotes the covering number. By a standard covering number argument (Lemma D.4 or Lemma D.6 in Jin et al. [17]), we can show that log N ( Q k h ) ≤ ˜ O ( d 2 ) . However, for policybased algorithms such as PPO, π k h is a stochastic policy, which makes the log-covering number log N ( V k h ) may have a polynomial dependency on the size of action space A (log-covering number

k

k

i

k

i

Q k

1

k

i

π k

1

Q ki

-1

π k

i

-1

Q ki

π k

i

Q ki

+1

π k

i

+1

Rewards

Q kL

π k

L

K

k

L

Q k

2

π k

2

Rewards

Rewards

1

+1

-1

of |A| -dimensional probability distributions is at the order of |A| ). We also remark that linear mixture MDPs are more amenable to theoretical analysis compared to linear MDPs, as they do not necessitate the calculation of the covering number of V k h [4, 6]. As a result, the proof presented by Cai et al. [6] is valid for linear mixture MDPs, but cannot be extended to linear MDPs. See §A for more elaboration of technical challenges.

Novelty 1: Multi-batched Updating and A New Covering Number Argument. Our key observation is that if we improve the policy like PPO (see (3.2) or Schulman et al. [29], Cai et al. [6]), π k h admits a softmax form, i.e.,

Here { k i } l i =1 is a sequence of episodes, where k i ≤ k for all i ∈ [ l ] , denoting the episodes in which our algorithm performs policy optimization prior to the k -th episode. By a technical lemma (Lemma C.3), we can show that

If we perform the policy optimization in each episode like Cai et al. [6], Shani et al. [30], l may linear in K and the final regret bound becomes vacuous. Motivated by this, we use a multi-batched updating scheme. In specific, OPPO+ divides the whole learning process into several batches and only updates policies at the beginning of each batch. See Figure 1 for visualization. For example, if the number of batches is K 1 / 2 (i.e., each batch consists of consecutive K 1 / 2 episodes), we have l ≤ K 1 / 2 and the final regret is at the order of ˜ O ( K 3 / 4 ) . Here we assume K 1 / 2 is a positive integer for simplicity. See §C for details.

Challenge 2: Adversarial Rewards. Compared with previous value-based algorithms [e.g., 17], one superiority of optimistic PPO is that they can learn adversarial MDPs with full-information feedback [6, 30]. In Cai et al. [6], Shani et al. [30], the policy evaluation step is that

where r k h is the adversarial reward function and ̂ P k h V k h +1 is the estimator of the expected next step value of V k h +1 . This policy evaluation step is invalid if we use the multi-batched updating. Consider the following case, if the number of batches is K 1 / 2 and

.

Then the algorithm only uses zero rewards to find the optimal policy in hindsight with respect to arbitrary adversarial rewards, which is obviously impossible.

Novelty 2: Policy Evaluation via Average Reward and Smoothness Analysis. To tackle the above challenge, we adopt the following policy evaluation step at the beginning of each batch

where ¯ r k h the average reward of the last batch:

Let π k i denote the policy executed in the i -th batch. Intuitively, π k i +1 is the desired policy within i -th batch since its calculation only uses the rewards in the first ( i -1) batches (cf. Figure 1). Hence, compared with Cai et al. [6], Shani et al. [30], we need to handle the gap between the performance of π k i +1 and π k i in the i -th batch. Fortunately, this error can be controlled due to the 'smoothness' of policies in adjacent batches. See Lemma 4.3 for details.

1.2 Related Works

Policy Optimization Algorithms. The seminal work of Schulman et al. [29] proposes the PPO algorithm, and a line of following works seeks to provide theoretical guarantees for it. In particular, Cai et al. [6] proposes the optimistic PPO (OPPO) algorithm for adversarial linear mixture MDPs and establishes regret ˜ O ( d √ H 4 K ) for it. Then, He et al. [14] improve the regret to ˜ O ( d √ H 3 K ) by the weighted regression technique [44]. Besides their works on linear mixture MDPs, Shani et al. [30], Wu et al. [37] provide fine-grained analysis of optimistic variants of PPO in the tabular case. The works of Fei et al. [9], Zhong et al. [42] show that optimistic variants of PPO can solve nonstationary MDPs. However, none of these works show that the optimistic variant of PPO is provably efficient for linear MDPs.

There is another line of works [2, 10, 40] proposes optimistic policy optimization algorithms based on the natural policy gradient (NPG) algorithm [18] and the policy-cover technique. But their works are limited to the stochastic linear MDPs, while our work can tackle adversarial rewards. Compared with their results for stochastic linear MDPs, our work can achieve a better regret and compatible sample complexity. See Table 1 and Remark 3.2 for a detailed comparison. Several recent works [26, 23, 20, 8, 31] study the more challenging problem of learning adversarial linear MDPs with only bandit feedback, which is beyond the scope of our work. Without access to exploratory policies or even known transitions, their regret is at least ˜ O ( K 6 / 7 ) [31], while our work achieves a better ˜ O ( K 3 / 4 ) regret with the full-information feedback assumption.

RL with Linear Function Approximation. Our work is related to previous works proposing value-based algorithms for linear MDPs [38, 17]. The work of Yang and Wang [38] develops the first sample efficient algorithm for linear MDPs with a generative model. Then Jin et al. [17] proposes the first provably efficient algorithms for linear MDPs in the online setting. The results of Jin et al. [17] are later improved by [35, 15, 13, 3]. In particular, Agarwal et al. [3], He et al. [13] show that the nearly minimax optimal regret ˜ O ( d √ H 3 K ) is achievable in stochastic linear MDPs. Compared with these value-based algorithms, our work can tackle the more challenging adversarial linear MDPs with full-information feedback.

There is another line of works [4, 25, 6, 41, 14, 45, 44, 43] studying linear mixture MDPs, which is another model of RL with linear function approximation. It can be shown that linear MDPs and linear mixture MDPs are incompatible in the sense that neither model is a special case of the other. Among these works, Zhou et al. [44], Zhou and Gu [43] establishes nearly minimax regret ˜ O ( d √ H 3 K ) for stochastic linear mixture MDPs. Our work is more related to Cai et al. [6], He et al. [14] on adversarial linear mixture MDPs with full-information feedback. We have remarked that it is nontrivial to extend their results to linear MDPs.

2 Preliminaries

Notations. We use N + to denote the set of positive integers. For any H ∈ N + , we denote [ H ] = { 1 , 2 , . . . , H } . For any H ∈ N + and x ∈ R , we use the notation min { x, H } + =

min { H, max { 0 , x }} . Besides, we denote by ∆( A ) the set of probability distributions on the set A . For any two distributions P and Q over A , we denote KL( P ∥ Q ) = E a ∼ P [log d P ( a ) / d Q ( a )] .

Episodic Adversarial MDPs. We consider an episodic MDP M , which is denoted by a tuple

where S is the state space, A is the action space, H is the length of each episode, K is the number of episodes, r k h : S × A ↦→ [0 , 1] is the deterministic 1 reward function at the h -th step of k -th episode, P h is the transition kernel with P h ( s ′ | s, a ) being the transition probability for state s to transfer to the next state s ′ given action a at the h -th step. We consider the adversarial MDPs with full-information feedback , which means that the reward { r k h } h ∈ [ H ] is adversarially chosen by the environment at the beginning of the k -th episode and revealed to the learner after the k -th episode.

A policy π = { π h } h ∈ [ H ] is a collection of H functions, where π h : S ↦→ ∆( A ) is a function that maps a state to a distribution over action space at step h . For any policy π and reward function { r k h } h ∈ [ H ] , we define the value function V π,k h : S ↦→ R and Q-function Q π,k h : S × A ↦→ R as

for any ( x, a, k, h ) ∈ S × A × [ K ] × [ H ] . Here the expectation E π [ · ] is taken with respect to the randomness of the trajectory induced by policy π and transition kernel P . It is well-known that the value function and Q-function satisfy the following Bellman equation for any ( x, a ) ∈ S × A ,

where ⟨· , ·⟩ A denotes the inner product over the action space A and we will omit the subscript when it is clear from the context. Here P h is the operator defined as

Interaction Process and Learning Objective. We consider the online setting, where the learner improves her performance by interacting with the environment repeatedly. The learning process consists of K episodes and each episode starts from a fixed initial state x 1 2 . At the beginning of the k -th episode, the environment adversarially chooses reward functions { r k h } h ∈ [ H ] , which can depend on previous ( k -1) trajectories. Then the agent determines a policy π k and receives the initial state x k 1 = x 1 . At each step h ∈ [ H ] , the agent receives the state x k h , chooses an action a k h ∼ π k h ( · | x k h ) , receives the reward function r k h , and transits to the next state x k h +1 . The k -th episode ends after H steps.

We evaluate the performance of an online algorithm by the notion of regret [7], which is defined as the value difference between the executed policies and the optimal policy in hindsight:

For simplicity, we denote the optimal policy in hindsight by π ∗ , i.e., π ∗ = argmax π ∑ K k =1 V π,k 1 ( x 1 ) .

Linear MDPs. We focus on the linear MDPs [38, 17], where the transition kernels are linear in a known feature map.

Definition 2.1 (Linear MDP) . We say an MDP ( S , A , H, K, { r k h } ( k,h ) ∈ [ K ] × [ H ] , P = {P h } h ∈ [ H ] ) is a linear MDP if there exists a known feature ϕ : S × A ↦→ R d such that for any ( x, a, k, h ) ∈ S × A × [ K ] × [ H ] , we have

where µ h = ( µ (1) h , . . . , µ ( d ) h ) are d unknown signed measures over S satisfying ∥ µ h ( S ) ∥ 2 ≤ √ d .

Algorithm 1 OPPO+

Since we have access to the full-information feedback, we do not assume the reward functions are linear in the feature map ϕ like Jin et al. [17]. We also remark that the adversarial linear MDP with full-information feedback is a more challenging problem than the stochastic linear MDP with bandit feedback studied in Jin et al. [17]. In fact, for stochastic linear MDPs, we can assume the reward functions are known without loss of generality since learning the linear transition kernel is more difficult than the linear reward.

3 Algorithm

In this section, we propose a new algorithm OPPO+ (Algorithm 1) to solve adversarial linear MDPs with full-information feedback. In what follows, we highlight the key steps of OPPO+ .

Multi-batched Updating. Due to the technical issue elaborated in §1.1, we adopt the multi-batched updating rule. In specific, OPPO+ divides the total K episodes into L = K/B batches and each batch consists of B consecutive episodes. Here we assume K/B is a positive integer without loss of generality 3 . For ease of presentation, we use k i = ( i -1) · B +1 to denote the first episode in the i -th batch. When the k -th episode is the beginning of a batch (i.e, k = k i for some i ∈ [ L ] ), OPPO+ performs the following policy improvement step and policy evaluation step .

Policy Improvement. In the policy improvement step of the k -th episode ( k = k i for some i ∈ [ L ] ), OPPO+ calculates π k based on the previous policy π k -1 using PPO [29]. In specific, OPPO+ updates

π k by solving the following proximal policy optimization problem:

where α > 0 is the stepsize that will be specified in Theorem 3.1, and L k -1 ( π ) takes form

which is proportional to the local linear function of V π,k -1 1 ( x k 1 ) at π k -1 and replaces the unknown Q-function Q π k -1 ,k -1 h by the estimated one Q k -1 h for any h ∈ [ H ] . It is not difficult to show that the updated policy π k obtained in (3.1) admits the following closed form:

Policy Evaluation. In the policy evaluation step of the k -th episode ( k = k i for some i ∈ [ K ] ), OPPO+ lets V k H +1 be the zero function and iteratively calculates the estimated Q-function { Q k h } h ∈ [ H ] in the order of h = H,H -1 , . . . , 1 . Now we present the policy evaluation at the h -th step given estimated value V k h +1 . By the definitions of linear MDP in Definition 2.1 and the operator P h in (2.2), we know P h V k h +1 is linear in the feature map ϕ . Inspired by this, we estimate its linear coefficient by solving the following ridge regression:

where λ > 0 is the regularization parameter and I d is the identify matrix. By solving (3.3), we have

Based on this linear coefficient, we construct the estimator ̂ P k h V k h +1 as

where Γ k h = β · ( ϕ ( · , · ) ⊤ (Λ k h ) -1 ϕ ( · , · )) 1 / 2 is the bonus function and β > 0 is a parameter that will be specified in Theorem 3.1. This form of bonus function also appears in the literature on linear bandits [21] and linear MDPs [17]. Finally, we update Q k h and V k h by

Here ¯ r k h is the average reward function in the last batch, that is

where k = k i = ( i -1) · B +1 and k i -1 = ( i -2) · B +1 .

Here we would like to make some comparisons between our algorithm OPPO+ and other related algorithms. Different from previous value-based algorithms that take simply take the greedy policy with respect to the estimated Q-functions [17], OPPO+ involves a policy improvement step like PPO [29]. This step is key to tackling adversarial rewards. The most related algorithm is OPPO proposed by Cai et al. [6], which performs the policy optimization in linear mixture MDPs. The main difference between OPPO+ and OPPO is that we introduce a multi-batched updating and an average reward policy evaluation, which are important for solving linear MDPs (cf. §1.1). Finally, we remark that the multi-batched updating scheme is adopted by previous work on bandits [12] and RL [36]. But their algorithms are value-based and cannot tackle adversarial rewards. √

Theorem 3.1 (Regret) . Fix δ ∈ (0 , 1] and K ≥ d 3 . Let B = d 3 K and α = √ 2 B log |A| / ( KH 2 ) , λ = 1 , β = O ( d 1 / 4 HK 1 / 4 ι 1 / 2 ) with ι = log( dHK |A| /δ ) , then with probability at least 1 -δ , the regret of Algorithm 1 satisfies

To illustrate our theory more, we make several remarks as follows.

Remark 3.2 (Sample Complexity) . Since learning adversarial linear MDPs with full-information feedback is more challenging than learning stochastic linear MDPs with bandit feedback, the result in Theorem 3.1 also holds for stochastic linear MDPs. By the standard online-to-batch argument [16], we have that Algorithm 1 can find an ϵ -optimal policy using at most

samples. Compared with the sample complexity ˜ O ( d 3 H 13 /ϵ 3 ) in Zanette et al. [40], we have a better dependency on H but a worse dependency on ϵ . Moreover, by the standard sample complexity to regret argument [16], their sample complexity only gives a ˜ O ( d 3 / 4 H 13 / 4 K 3 / 4 ) , which is worse than our regret in Theorem 3.1. More importantly, the algorithm in Zanette et al. [40] lacks the ability to handle adversarial rewards, whereas our proposed algorithm overcomes this limitation.

Remark 3.3 (Optimality of Results) . For stochastic linear MDPs, Agarwal et al. [3], He et al. [13] design value-based algorithms with ˜ O ( d √ H 3 K ) regret, which matches the lower bound Ω( d √ H 3 K ) [44] up to logarithmic factors. It remains unclear whether policy-based based algorithms can achieve the nearly minimax optimal regret and we leave this as future work. For the more challenging adversarial linear MDPs with full-information feedback, we achieve the state-of-the-art regret bound. In this setup, a direct lower bound is Ω( d √ H 3 K ) [44, 14], and we conjecture this lower bound is tight. It would be interesting to design algorithms with √ K regret or even optimal regret in this setting.

Remark 3.4 (Beyond the Linear Function Approximation) . The work of Agarwal et al. [2] extends their results to the kernel function approximation setting. We conjecture that our results can also be extended to RL with kernel and neural function approximation by the techniques in Yang et al. [39].

4 Proof of Theorem 3.1

Proof. Recall that B ∈ N + is the batch size, L = K/B is the number of batches, and k i = ( i -1) · B +1 for any i ∈ [ L ] . For any k ∈ [ K ] , we use t k to denote the k i satisfying k i ≤ k < k i +1 . Moreover, we define the Bellman error as

Here V k h +1 and Q k h are the estimated value function and Q-function defined in (3.4), and P h is the operator defined in (2.2). Intuitively, (4.1) quantifies the violation of the Bellman equation in (2.1). With these notations, we have the following regret decomposition lemma.

Lemma 4.1 (Regret Decomposition) . It holds that

statistical error

Proof. This lemma is similar to the regret decomposition lemma in previous works [6, 30] on policy optimization. See §B.1 for a detailed proof.

Lemma 4.1 shows that the total regret consists of the policy optimization error and the statistical error related to the Bellman error defined in (4.1). Notably, different from previous works [6, 30, 37, 14] that optimize policy in each episode, our algorithm performs policy optimization infrequently. Despite this, we can bound the policy optimization error in Lemma 4.1 by the following lemma.

Proof. See §B.2 for a detailed proof.

For the statistical error in Lemma 4.1, by the definitions of the policy evaluation step in (3.4) and Bellman error in (4.1), we have

The following lemma establishes the upper bound of the cumulative reward mismatch error

and thus relates the statistical error in Lemma 4.1 to the transition estimation error in (4.2).

Lemma 4.3. It holds with probability at least 1 -δ/ 2 that

Proof. By calculation, we can show that the cumulative reward mismatch error in (4.3) is bounded by

which represents the smoothness of adjacent policies. Our smoothness analysis leverages the value difference lemma (Lemma B.1 or §B.1 in Cai et al. [6]) and the closed form of the policy improvement in (3.2). See §B.3 for a detailed proof.

Then we introduce the following lemma, which shows that the transition estimation error can be controlled by the bonus function.

Lemma 4.4. It holds with probability at least 1 -δ/ 2 that

Proof. The proof involves the standard analysis of self-normalized process [1] and a uniform concentration of the function class of V k h +1 [17]. As elaborated in §1.1, calculating the covering number of this function class is challenging and requires some new techniques. See §B.4 for a detailed proof.

Combining Lemmas 4.3 and 4.4, we have

Hence, it remains to bound the term ∑ K k =1 ∑ H h =1 min { H, Γ t k h ( x k h , a k h ) } , which is the purpose of the following lemma.

Lemma 4.5. It holds that

Proof. For all k ∈ [ K ] , let Γ k h = β · ( ϕ ⊤ (Λ k h ) -1 ϕ ) 1 / 2 with Λ k h = ∑ k -1 τ =1 ϕ ( x τ h , a τ h ) ϕ ( x τ h , a τ h ) ⊤ + λ · I d . By a doubling trick, we can prove that

which can be further bounded by elliptical potential lemma (Lemma D.5 or Lemma 11 in AbbasiYadkori et al. [1]). See §B.5 for a detailed proof.

Finally, putting Lemmas 4.1-4.5 together, we conclude the proof of Theorem 3.1.

Note the choice of the number of batches L determines the batch size B = K/L . We make the following remark to illustrate how the choice of L affects the final regret, which indicates that our choices of parameters are optimal based on our current analysis.

Remark 4.6 (Choices of Parameters and Final Regret) . If OPPO+ performs the policy optimization for L times, then by Lemma C.3, we have the complexity (logarithmic covering number) of policy class is roughly ˜ O ( L ) (ignoring the dependency of d and H ). Furthermore, by the new self-normalized analysis in §C, we have β = ˜ O ( √ L ) , which further implies that the model estimation error is bounded by β · ∑ K k =1 ∑ H h =1 √ ϕ ( x k h , a k h )(Λ k h ) -1 ϕ ( x k h , a k h ) ≤ ˜ O ( √ LK ) (cf. Lemma 4.5). Meanwhile, by Lemma 4.2, we have the policy optimization error is bounded by ˜ O ( K/ √ L ) = ˜ O ( K 3 / 4 ) . By choosing L = Θ( √ K ) , we obtain a regret of order ˜ O ( K 3 / 4 ) .

5 Conclusion

In this paper, we advance the theoretical study of PPO in stochastic linear MDPs and even adversarial linear MDPs with full-information feedback. We propose a novel algorithm, namely OPPO+ , which exhibits a state-of-the-art regret bound as compared to the prior policy optimization algorithms. Our work paves the way for future research in multiple directions. For instance, a significant open research question is to investigate whether policy-based algorithms can achieve the minimax regret in stochastic linear MDPs like previous value-based algorithms [3, 14]. Additionally, an interesting direction is to derive √ K -regret bounds for adversarial linear MDPs with full-information or even bandit feedback.

References

Abbasi-Yadkori, Y., P´ al, D. and Szepesv´ ari, C. (2011). Improved algorithms for linear stochastic bandits. Advances in neural information processing systems , 24 .

Agarwal, A., Henaff, M., Kakade, S. and Sun, W. (2020). Pc-pg: Policy cover directed exploration for provable policy gradient learning. Advances in neural information processing systems , 33 13399-13412.

Agarwal, A., Jin, Y. and Zhang, T. (2022). Vo q l: Towards optimal regret in model-free rl with nonlinear function approximation. arXiv preprint arXiv:2212.06069 .

Ayoub, A., Jia, Z., Szepesvari, C., Wang, M. and Yang, L. (2020). Model-based reinforcement learning with value-targeted regression. In International Conference on Machine Learning . PMLR.

Brown, N. and Sandholm, T. (2019). Superhuman ai for multiplayer poker. Science , 365 885890.

Cai, Q., Yang, Z., Jin, C. and Wang, Z. (2020). Provably efficient exploration in policy optimization. In International Conference on Machine Learning . PMLR.

Cesa-Bianchi, N. and Lugosi, G. (2006). Prediction, learning, and games . Cambridge university press.

Dai, Y., Luo, H., Wei, C.-Y. and Zimmert, J. (2023). Refined regret for adversarial mdps with linear function approximation. arXiv preprint arXiv:2301.12942 .

Fei, Y., Yang, Z., Wang, Z. and Xie, Q. (2020). Dynamic regret of policy optimization in nonstationary environments. Advances in Neural Information Processing Systems , 33 6743-6754.

Feng, F., Yin, W., Agarwal, A. and Yang, L. (2021). Provably correct optimization and exploration with non-linear policies. In International Conference on Machine Learning . PMLR.

Haarnoja, T., Zhou, A., Hartikainen, K., Tucker, G., Ha, S., Tan, J., Kumar, V., Zhu, H., Gupta, A., Abbeel, P. et al. (2018). Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 .

Han, Y., Zhou, Z., Zhou, Z., Blanchet, J., Glynn, P. W. and Ye, Y. (2020). Sequential batch learning in finite-action linear contextual bandits. arXiv preprint arXiv:2004.06321 .

He, J., Zhao, H., Zhou, D. and Gu, Q. (2022). Nearly minimax optimal reinforcement learning for linear markov decision processes. arXiv preprint arXiv:2212.06132 .

He, J., Zhou, D. and Gu, Q. (2022). Near-optimal policy optimization algorithms for learning adversarial linear mixture mdps. In International Conference on Artificial Intelligence and Statistics . PMLR.

Hu, P., Chen, Y. and Huang, L. (2022). Nearly minimax optimal reinforcement learning with linear function approximation. In International Conference on Machine Learning . PMLR.

Jin, C., Allen-Zhu, Z., Bubeck, S. and Jordan, M. I. (2018). Is q-learning provably efficient? Advances in neural information processing systems , 31 .

Jin, C., Yang, Z., Wang, Z. and Jordan, M. I. (2020). Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory . PMLR.

Kakade, S. M. (2001). A natural policy gradient. Advances in neural information processing systems , 14 .

Kober, J., Bagnell, J. A. and Peters, J. (2013). Reinforcement learning in robotics: A survey. The International Journal of Robotics Research , 32 1238-1274.

Kong, F., Zhang, X., Wang, B. and Li, S. (2023). Improved regret bounds for linear adversarial mdps via linear optimization. arXiv preprint arXiv:2302.06834 .

Lattimore, T. and Szepesv´ ari, C. (2020). Bandit algorithms . Cambridge University Press.

LeCun, Y., Bengio, Y. and Hinton, G. (2015). Deep learning. nature , 521 436-444.

Luo, H., Wei, C.-Y. and Lee, C.-W. (2021). Policy optimization in adversarial mdps: Improved exploration via dilated bonuses. Advances in Neural Information Processing Systems , 34 22931-22942.

Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G. et al. (2015). Human-level control through deep reinforcement learning. nature , 518 529-533.

Modi, A., Jiang, N., Tewari, A. and Singh, S. (2020). Sample complexity of reinforcement learning using linearly combined model ensembles. In International Conference on Artificial Intelligence and Statistics . PMLR.

Neu, G. and Olkhovskaya, J. (2021). Online learning in mdps with linear function approximation and bandit feedback. Advances in Neural Information Processing Systems , 34 1040710417.

Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A. et al. (2022). Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems , 35 27730-27744.

Schulman, J., Levine, S., Abbeel, P., Jordan, M. and Moritz, P. (2015). Trust region policy optimization. In International conference on machine learning . PMLR.

Schulman, J., Wolski, F., Dhariwal, P., Radford, A. and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 .

Shani, L., Efroni, Y., Rosenberg, A. and Mannor, S. (2020). Optimistic policy optimization with bandit feedback. In International Conference on Machine Learning . PMLR.

Sherman, U., Koren, T. and Mansour, Y. (2023). Improved regret for efficient online reinforcement learning with linear function approximation. arXiv preprint arXiv:2301.13087 .

Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanctot, M. et al. (2016). Mastering the game of go with deep neural networks and tree search. nature , 529 484-489.

Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A. et al. (2017). Mastering the game of go without human knowledge. nature , 550 354-359.

Sutton, R. S. and Barto, A. G. (2018). Reinforcement learning: An introduction . MIT press.

Wagenmaker, A. J., Chen, Y., Simchowitz, M., Du, S. and Jamieson, K. (2022). First-order regret in reinforcement learning with linear function approximation: A robust estimation approach. In International Conference on Machine Learning . PMLR.

Wang, T., Zhou, D. and Gu, Q. (2021). Provably efficient reinforcement learning with linear function approximation under adaptivity constraints. Advances in Neural Information Processing Systems , 34 13524-13536.

Wu, T., Yang, Y., Zhong, H., Wang, L., Du, S. and Jiao, J. (2022). Nearly optimal policy optimization with stable at any time guarantee. In International Conference on Machine Learning . PMLR.

Yang, L. and Wang, M. (2019). Sample-optimal parametric q-learning using linearly additive features. In International Conference on Machine Learning . PMLR.

Yang, Z., Jin, C., Wang, Z., Wang, M. and Jordan, M. I. (2020). On function approximation in reinforcement learning: Optimism in the face of large state spaces. arXiv preprint arXiv:2011.04622 .

Zanette, A., Cheng, C.-A. and Agarwal, A. (2021). Cautiously optimistic policy optimization and exploration with linear function approximation. In Conference on Learning Theory . PMLR.

Zhang, Z., Yang, J., Ji, X. and Du, S. S. (2021). Improved variance-aware confidence sets for linear bandits and linear mixture mdp. Advances in Neural Information Processing Systems , 34 4342-4355.

Zhong, H., Yang, Z., Wang, Z. and Szepesv´ ari, C. (2021). Optimistic policy optimization is provably efficient in non-stationary mdps. arXiv preprint arXiv:2110.08984 .

Zhou, D. and Gu, Q. (2022). Computationally efficient horizon-free reinforcement learning for linear mixture mdps. arXiv preprint arXiv:2205.11507 .

Zhou, D., Gu, Q. and Szepesvari, C. (2021). Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory . PMLR.

Zhou, D., He, J. and Gu, Q. (2021). Provably efficient reinforcement learning for discounted mdps with feature mapping. In International Conference on Machine Learning . PMLR.

A Further Elaboration of Challenge 1 in Section 1.1

Regarding the technical challenges involved in extending OPPO [6] to linear MDPs, we provided more detailed explanations below.

Technically, for linear mixture MDPs (Equation (B.20) in OPPO paper [6]), they need to analyze

Since V τ h +1 is adapted to F k,h, 1 = { ( x τ i , a τ i ) } ( τ,i ) ∈ [ k -1] × [ H ] ∪ { r τ } τ ∈ [ k ] ∪ { ( x k i , a k i ) } i ∈ [ h ] , they can bound this term with classical self-normalized process analysis directly (see Lemma D.1 in Cai et al. [6]).

In contrast, for linear MDPs (see e.g., (B.26) in our paper or Lemma B.3 in Jin et al. [17]), we need to bound the term

Since V τ h +1 is NOT adapted to F k,h, 1 = { ( x τ i , a τ i ) } ( τ,i ) ∈ [ k -1] × [ H ] ∪{ r τ } τ ∈ [ k ] ∪{ ( x k i , a k i ) } i ∈ [ h ] . We need to perform the uniform concentration on the function class of V k h +1 . The challenge of calculating the covering number of this function class has been elaborated in Challenge 1 of Section 1.1.

B Missing Proofs of Main Theorem

B.1 Proof of Lemma 4.1

Proof. Our proof relies on the following value difference lemma in Cai et al. [6].

Lemma B.1 (Value Difference Lemma) . Let π = { π h } h ∈ [ H ] and π ′ = { π ′ h } h ∈ [ H ] be two policies and ¯ Q = { ¯ Q h : S ×A ↦→ R } h ∈ [ H ] be any Q-functions. Moreover, for any h ∈ [ H ] , we define value function ¯ V h : S ↦→ R by letting ¯ V h ( x ) = ⟨ ¯ Q h ( x, · ) , π h ( · | x ) ⟩ . Then for any k ∈ [ K ] we have

Proof. See §B.1 in Cai et al. [6] for a detailed proof.

Back to our proof, for any k ∈ [ K ] , we have

Applying Lemma B.1 with π = π k , π ′ = π ∗ , and ¯ Q = { Q k h } h ∈ [ H ] , we have

where δ k h = r k h + Q k h -P h V k h +1 is the Bellman error defined in (4.1). Similarly, applying Lemma B.1 with π = π ′ = π k and ¯ Q = { Q k h } h ∈ [ H ] , we obtain

Plugging (B.2) and (B.3) into (B.1) and then taking summation across k ∈ [ K ] , we have

which concludes the proof of Lemma 4.1.

B.2 Proof of Lemma 4.2

Proof. Recall that B ∈ N + is the batch size, k i = ( i -1) · B +1 and L = K/B . Fix h ∈ [ H ] . By the multi-batched updating rule, we have

To derive the upper bound of (B.4), we need the following lemma.

Lemma B.2. For any ( i, h, x h ) ∈ [ L ] × [ H ] ×S , it holds that

Proof. By the updating rule in (3.2), we have

For ease of presentation, we denote Υ = ∑ a ∈A π k i h ( a | x h ) · exp { αQ k i h ( x h , a ) } . Then we have

where the first equality uses (B.5), and the second equality follows from the fact that ∑ a ∈A π ∗ h ( a | x h ) -π k i +1 h ( a | x h ) = 0 . Rearranging (B.6) gives that

Furthermore, we have

where the last inequality uses (B.7), Cauchy-Schwarz inequality, and the fact that ∥ Q k i h ∥ ∞ ≤ H . By Pinsker's inequality, we have KL( π k i +1 h ( · | x h ) ∥ π k i h ( · | x h )) ≥ ∥ π k i +1 h ( · | x h ) -π k i h ( · | x h ) ∥ 2 1 / 2 . Together with (B.8), we obtain that

where the last inequality uses the fact that max y ∈ R {-y 2 / 2 + αH · y } = α 2 H 2 / 2 . Rearranging (B.9) concludes the proof of Lemma B.2.

where the equality uses the fact that L = K/B , and the last inequality uses the non-negativity of KL-divergence and the fact that π k 1 h = π 1 h is the uniform policy. Combining (B.4), (B.10), and α = √ 2 B log |A| / ( KH 2 ) , we obtain for all h ∈ [ H ] :

Telescoping (B.11) across h ∈ [ H ] concludes the proof of Lemma 4.2.

B.3 Proof of Lemma 4.3

Proof. Recall that B ∈ N + is the batch size, k i = ( i -1) · B +1 and L = K/B . Fix h ∈ [ H ] . We have

where the first equality uses the definition of t k and the updating rule, and the second equality uses the definition of Q k h = ¯ r k h + ̂ P k h V k h +1 in (3.4). Furthermore, we have

where the first equality uses the updating rule, the second equality follows from the definition of ¯ r k h in (3.5), and the last inequality is obtained by the fact that ∥ r k h ∥ ∞ ≤ 1 for any ( k, h ) ∈ [ K ] × [ H ] . Combining (B.12) and (B.13) and then taking summation across h ∈ [ H ] , we obtain that

On the other hand, similar to the derivation of (B.12), we have

For the first term of (B.15), by the updating rule and calculation, we have

where the last inequality uses the fact that -∑ k L +1 -1 k = k L E π k L [ r k h ( x h , a h )] ≤ 0 . Summing over h ∈ [ H ] in (B.16) gives that

where the last inequality uses the value difference lemma (Lemma B.1). By the policy updating rule in (3.2), we have

for any x h ∈ S , which implies that

where the first inequality follows the fact that 0 ≤ Q k i h ( · , · ) ≤ H , and the last inequality uses the basic inequality exp( y ) ≥ 1 + y for all y ∈ R . Together with

we further obtain

for any x h ∈ S . Plugging (B.18) into (B.17) gives that

where the last inequality follows from the fact that Q π k i ,k h ( · , · ) ≤ H . Plugging (B.17) and (B.19) into (B.15), we have

where the last inequality uses Azuma-Hoeffding inequality. Putting (B.14) and (B.20) together, we conclude the proof of Lemma 4.3.

B.4 Proof of Lemma 4.4

Proof. Recall that

where w t k h and Γ t k h take form

Here Λ t k h is the covariance matrix:

Back to our proof, for any ( x, a ) ∈ S × A , we have

where the first and the last equality follows from the definition of linear MDP (Definition 2.1), and the third equality uses (B.23). Putting (B.22) and (B.24) together, we have

For Term ( ⋆ ) in (B.25), by Cauchy-Schwarz inequality, we have

where the last inequality follows from the following lemma.

Lemma B.3. Fix δ ∈ (0 , 1] . It holds for all ( k, h ) ∈ [ K ] × [ H ] that

Proof. See §C for a detailed proof.

For Term ( ⋆⋆ ) in (B.25), we have

where the second inequality uses the fact that λ · I d ⪯ Λ t k h , and last inequality follows from ∥⟨ µ h , V t k h +1 ⟩ S ∥ 2 ≤ H √ d , which is implied by Definition 2.1. Plugging (B.26) and (B.27) into (B.25), together with λ = 1 , we obtain

for any ( x, a ) ∈ S × A with β = O ( d 1 / 4 HK 1 / 4 ι 1 / 2 ) . Together with the definition of ̂ P t k h V t k h +1 in (B.21), we have

where the last inequality uses (B.28) and the fact that ∥ V t k h +1 ∥ ∞ ≤ H -h . Moreover, by (B.28), we have

Combining (B.29), (B.30), and the fact that ( P h V t k h +1 -̂ P t k h V t k h +1 )( · , · ) ≥ -2 H , we obtain

which concludes the proof of Lemma 4.4.

B.5 Proof of Lemma 4.5

Proof. For ease of presentation, we define

for all ( k, h ) ∈ [ K ] × [ H ] . Then we have the following lemma, which uses the elliptical potential to bound ∑ K k =1 ∑ H h =1 Γ k h ( x k h , a k h ) .

Then we have

Proof. By the definition of Γ k h in (B.31), we have

Applying Cauchy-Schwarz inequality to (B.32), we have

where the last inequality uses the elliptical potential lemma (Lemma D.5). For any h ∈ [ H ] , we have

where the inequality uses the fact that ∥ ϕ ( · , · ) ∥ 2 ≤ 1 . Plugging (B.34) into (B.33), we have

Together with the facts that λ = 1 , ι = log( dHK |A| /δ ) with δ ∈ (0 , 1] , and β = O ( d 1 / 4 HK 1 / 4 ι 1 / 2 ) , we conclude the proof of Lemma B.4.

We also need the following lemma to connect the quantity ∑ K k =1 ∑ H h =1 Γ k h ( x k h , a k h ) in Lemma B.4 and our target ∑ K k =1 ∑ H h =1 min { H, Γ t k h ( x k h , a k h ) } .

Lemma B.5. For any h ∈ [ H ] , we define the set E h as

Proof. For k ∈ E h , there exists i ∈ [ L ] such that k i ≤ k < k i +1 . Then we know t k = k i , which further implies that

where the first inequality follows from the fact that Λ k i +1 h ⪰ Λ k h , the second inequality uses Lemma D.6, and the last equality is obtained by the definitions of Γ k h and Γ k i h in (B.31). Together with the definition of E h in (B.35), we further obtain that

Meanwhile, we have

where the last inequality follows from the facts that

Here the last inequality uses ∥ ϕ ( · , · ) ∥ 2 ≤ 1 . Combining (B.36) and (B.37), we have

Since each batch contains B episodes, we obtain that

where the last inequality uses λ = 1 , ι = log( dHK |A| /δ ) , B = d 3 K and (B.38). Therefore, we conclude the proof of Lemma B.5.

Back to our proof of Lemma 4.5, we have

For any h ∈ [ H ] and k / ∈ E h , by the definition of E h , we have

Combining (B.39), (B.40), and the fact that min { H, Γ t k h ( x k h , a k h ) } ≤ H , we have

where the last inequality uses Lemmas B.4 and B.5. Therefore, we conclude the proof of Lemma 4.5.

C Proof for Concentration of Self-Normalized Processes

Proof of Lemma B.3. By the previous concentration lemma of self-normalized process (Lemma D.3), for any δ ′ ∈ (0 , 1] , ε > 0 , and ( k, h ) ∈ [ K ] × [ H ] we have

with probability 1 -δ ′ . Here N ε ( V k h +1 ) is the ε -covering number of function class V k h +1 , which is defined by

where Q k h +1 and Π k h +1 are Q-function class and policy class, respectively. Specifically, Q k h +1 is a function class with the following parametric form

Here we uses ∥ w k h +1 ∥ 2 ≤ H √ dK/λ (Lemma D.1) and λ min (Λ k h +1 ) ≥ λ to ensure that Q k h +1 ∈ Q k h +1 . Meanwhile, the policy class Π k h +1 is defined as

Notable, here l ≤ L since our algorithm only has L batches. Also, π t k h +1 in (C.2) belongs to this policy class since it takes the following form

For policy class defined in (C.4), we define its covering number with respect to the following distance

The following lemma connects the covering number of value class in (C.2) to covering numbers of the Q-function class in (C.3) and the policy class in (C.4).

Proof. See §C.1 for a detailed proof.

Following the standard covering argument (Lemma D.4), we can derive an upper bound for N ε/ 2 ( Q k h +1 ) . However, N ε/ (2 H ) (Π k h +1 ) is relatively difficult to bound since the log-covering number of a |A| -dimensional probability distribution is ˜ O ( |A| ) . Fortunately, we have the following lemma that utilizes the structure of policy class (C.4) and converts the covering number of the policy class to the covering numbers of several Q-function classes.

Proof. See §C.2 for a detailed proof.

Note that λ = 1 , l ≤ L , L = K/B , B = √ d 3 K , α = √ 2 B log |A| / ( KH 2 ) , ι = log( dHK |A| /δ ) , and β = O ( d 1 / 4 HK 1 / 4 ι 1 / 2 ) . Meanwhile, let ε = 1 /K and δ ′ = δ/ 2 . Combining (C.1) with Lemmas C.1, C.2, and D.4, we have

which concludes the proof of Lemma B.3.

C.1 Proof of Lemma C.1

Proof. Suppose (i) Q k ε/ 2 ,h +1 is an ε/ 2 -net of Q k h +1 with |Q k ε/ 2 ,h +1 | = N ε/ 2 ( Q k h +1 ) ; and (ii) Π k ε/ (2 H ) ,h +1 is an ε/ (2 H ) -net of Π k h +1 with | Π k ε/ (2 H ) ,h +1 | = N ε/ (2 H ) (Π k h +1 ) . Then we show Q k ε/ 2 ,h +1 × Π k ε/ (2 H ) ,h +1 induces an ε -net of V k h +1 , and thus obtaining the desired result. Specifically, for any V = ⟨ Q ( · , · ) , π ( · | · ) ⟩ with ( Q,π ) ∈ Q k h +1 × Π k h +1 , we can find ( Q ′ , π ′ ) ∈ Q k ε/ 2 ,h +1 × Π k ε/ (2 H ) ,h +1 such that

Let V ′ = ⟨ Q ′ ( · , · ) , π ′ ( · | · ) ⟩ , we have

where the first inequality follows from the triangle inequality, the second inequality uses CauchySchwarz inequality and the fact that ∥ Q ′ ∥ ∞ ≤ H , and the last inequality follows from (C.5). Therefore, we conclude the proof of Lemma C.1.

C.2 Proof of Lemma C.2

Proof. Suppose Q k i ε 2 / (16 αlH 2 ) ,h +1 is the minimum ε 2 / (16 αlH 2 ) -net of Q k i h +1 for all i ∈ [ l ] . Then for any π ∝ exp( α ∑ l i =1 Q i ) with Q i ∈ Q k i h +1 for all i ∈ [ l ] , we can choose { Q ′ i ∈ Q k i ε 2 / (16 αlH 2 ) ,h +1 } such that

Hence, we have

where the first inequality uses the triangle inequality and the last inequality follows from (C.6). Then we use the following lemma to establish the upper bound for sup x ∈S ∥ π ( · | x ) -π ′ ( · | x ) ∥ 1 . Lemma C.3. For π, π ′ ∈ ∆( X ) and Q,Q ′ : X ↦→ R + , if π ( · ) ∝ exp( Q ( · )) and π ′ ( · ) ∝ exp( Q ′ ( · )) , we have

Proof. See §C.3 for a detailed proof.

where the last inequality follows from (C.7). Therefore, we have

which concludes the proof of Lemma C.2.

C.3 Proof of Lemma C.3

Proof. Since π ( · ) ∝ exp( Q ( · )) and π ′ ( · ) ∝ exp( Q ′ ( · )) , we have for any x ∈ X :

Note that for any x ∈ X we have

which implies that

Hence, we have

Finally, by Pinsker's inequality, we have

which concludes the proof of Lemma C.3.

D Auxiliary Lemmas

LemmaD.1. For any ( i, h ) ∈ [ L ] × [ H ] , the linear coefficient w k i h defined in Line 12 of Algrotihm 1 satisfies

Proof. Fix ( i, h ) ∈ [ L ] × [ H ] . Our proof follows the proof of Lemma B.2 in Jin et al. [17]. For any v ∈ R d with ∥ v ∥ 2 = 1 , by the definition of w k i h we have

Since ∥ V k i h +1 ∥ ∞ ≤ H , we further have

where the second inequality is obtained by Cauchy-Schwarz inequality, and the last inequality uses λI d ⪯ Λ k i h , k i ≤ K , ∥ v ∥ 2 = 1 and Lemma D.2. Hence, we have

which concludes the proof of Lemma D.1.

Lemma D.2. Let Λ t = λ · I d + ∑ t i =1 ϕ i ϕ ⊤ i with ϕ i ∈ R d and λ > 0 . Then we have

Proof. See Lemma D.1 in Jin et al. [17] for a detailed proof.

LemmaD.3. Let { x τ ∈ S} ∞ τ =1 and { ϕ τ ∈ R d } ∞ τ =1 with ∥ ϕ τ ∥ 2 ≤ 1 be stochastic processes adapted to the filtration {F τ } ∞ τ =1 . Let Λ k = ∑ k -1 τ =1 ϕ τ ϕ ⊤ τ + λ · I d . Then for any δ ∈ (0 , 1] , with probability at least 1 -δ , for all k ∈ N + , and function V ∈ V satisfying sup x ∈S | V ( s ) | ≤ H , we have

where N ε is the ε -covering number of the function class V with respect to the distance dist( V, V ′ ) = sup x ∈S | V ( x ) -V ′ ( x ) | .

Proof. See Lemma D.4 of Jin et al. [17] for a detailed proof.

Lemma D.4. For any h ∈ [ H ] , let Q h be a function class mapping from S × A to R with the form

where r : S × A ↦→ [0 , 1] , ∥ w ∥ 2 ≤ L , λ min ( λ ) ≥ λ . Assuming ∥ ϕ ( · , · ) ∥ 2 ≤ 1 and β > 0 , we have

where N ε ( Q h ) is the ε -covering number of the function class Q h with respect to the distance dist( Q,Q ′ ) = sup ( x,a ) ∈S×A | Q ( x, a ) -Q ′ ( x, a ) | .

Proof. See Lemma D.6 of Jin et al. [17] for a detailed proof.

Lemma D.5 (Elliptical Potential Lemma) . Let { ϕ t ∈ R d } ∞ t =1 satisfy ∥ ϕ t ∥ 2 ≤ 1 for all t ∈ N + . Moreover, let Λ 0 ∈ R d × d be a positive-definite matrix with λ min (Λ 0 ) and Λ t = Λ 0 + ∑ t -1 i =1 ϕ i ϕ ⊤ i for any t ∈ N + . Then for any t ∈ N + , we have

Proof. See Lemma 11 of Abbasi-Yadkori et al. [1] for a detailed proof.

Lemma D.6. Suppose Λ , Λ ′ ∈ R d × d are two positive definite matrices and satisfy Λ ⪯ Λ ′ , then for any x ∈ R d , we have

See Lemma 12 of Abbasi-Yadkori et al. [1] for a detailed proof.

Proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors extend the theory of proximal policy optimization-based methods in the linear mixture MDPs and propose an optimistic variant PPO algorithm (OPPO+) for stochastic linear MDPs and adversarial linear MDPs with full information. \nThe proposed algorithm adopts a multi-batched updating rule from bandit literature, where the policy is updated at regular intervals of batches, instead of updating it every episode. During the policy improvement step, the policy is updated using proximal policy optimization, while the policy evaluation step involves estimating the value using least square value iteration with the average reward from the previous batch. \nThe proposed algorithm provides a regret guarantee for $\\tilde{O}(d^{3/4} H^2 K^{3/4})$ and has a sample complexity of $\\tilde{O}(d^3H^8/\\epsilon^4 + d^5H^4/\\epsilon^2)$. This regret bound is tighter than the existing PPO-based algorithms proposed for stochastic linear MDPs.\n\nSTRENGTHS:\n- The proposed algorithm can be applied not only to stochastic linear MDPs but also to adversarial linear MDPs with full-information feedback.\nIn order to cover the case of adversarial linear MDPs, the average reward technique is used, and for the analysis, result for bound on the gap between the value of the policy from the previous batch and the value of the policy from the current batch is suggested.\n- When applying the existing covering number theory from linear MDPs, the regret may depend on the size of the action space. To address this, a method using novel covering theory is proposed to reduce it to a logarithmic scale.\n- Compared to existing policy optimization algorithms, it provides a tighter regret guarantee in both the stochastic linear MDP and adversarial linear MDP with full-information feedback settings.\n\nWEAKNESSES:\n- The computational cost of policy improvement is not discussed in detail.\n- It remains unclear what advantages the proposed policy optimization algorithms offer compared to value-based algorithms, apart from their theoretical purposes.\n\nQUESTIONS:\n1. What does it mean for the proposed algorithm to be \"optimistic\"? What I mean is that for example, in value-based algorithms, optimism refers to the estimated values of the algorithm being more optimistic than the true optimal value (Lemma B.5 in [17]).\n2. Can you explain the computational cost of policy improvement in OPPO+ compared to the policy improvement in value-based algorithms [17]?\n3. Besides the ability to learn stochastic policies, what other advantages does OPPO+ have compared to value-based algorithms?\n4. In line 203, the author argued that value-based algorithms cannot handle adversarial rewards. Could you provide more details about this?\n5. By performing policy optimization infrequently, the algorithm achieves a regret of order $O(K^{3/4})$. What are the drawbacks or trade-offs associated with this approach? In other words, in line 81, it was mentioned that if policy optimization is performed every episode, the regret would have a linear dependence on $K$. What benefits can be obtained from this approach?\n\nLIMITATIONS:\nAs this is a theoretical paper, it does not seem to have any negative societal impact. However, the authors have not mentioned the limitations of this algorithm. (If they have mentioned any, please let me know, and I will check.) Nonetheless, they have provided an introduction to future work in Remark 3.3.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a theoretical analysis of the PPO algorithm. Some novel techniques such as batch-wise update are proposed so the algorithm can work on the adversarial setting of linear MDPs. A regret bound is given, which is better then or comparable to  previous results.\n\nSTRENGTHS:\nThe paper is well written, with a clear presentation and is easy to follow. The topic studied in the paper is important. The proposed technique is novel and the theoretical result is solid.\n\nWEAKNESSES:\n1. See Question 1. One concern is the similarity between the proposed method and NPG. And additional comparison is needed so the readers could understand the novelty and contribution of the paper more easily. \n\n2. The algorithm design of OPPO+ is similar to OPPO. The difference is OPPO+ replaces the step-wise update with a multi-batched update, and considers the linear MDP setting so the state-action feature $\\phi$ can be directly obtained instead of doing integration. The authors claimed the weakness of OPPO, or linear mixture MDP, is that the integration is computationally expensive. But I would rather just regard it as a separate model setting, instead of a weakness. Therefore, when doing comparisons with previous work, besides the literature on linear MDP, it could be great if the authors can also give a more detailed explanation on the setting of linear mixture MDP. In particular, given that the algorithm formulations are so similar, it might be helpful if the author can explain why it's not straightforward to migrate the proof of OPPO to the linear MDP setting.  For example, it can be added to the challenge & novelty section, which will greatly strengthen their argument.\n\nQUESTIONS:\n1. The updating rule of PPO has a very similar formula to Natural Policy Gradient (NPG). There also have been literatures that apply NPG to linear MDPs, can the authors make a comparison between their approach and these literatures? \n\nFor example, in section 4.2 of [1], the authors applied NPG to linear MDPs, and obtained a convergence rate of order $d^2H^6/\\varepsilon^3$, which should imply a regret bound of order $d^{1/2}H^{3/2}K^{3/4}$. \n\n[1] Liu et al., Optimistic Natural Policy Gradient: a Simple Policy Optimization Algorithm for Online Learning in Linear MDPs, https://arxiv.org/pdf/2305.11032.pdf \n\n2. Is it possible to extend the analysis to the general function approximation setting? \n\n3. One novelty of the paper is its ability to handle adversarial rewards. Can the authors explain which part of their algorithm is crucial to achieving this goal? Still take [1] as an example, I think the algorithms have similar formulation, and the difference of [1] is it doesn't use a multi-batched update. Does that mean the multi-batched updating rule is the crucial part for the adversarial setting?\n\nLIMITATIONS:\nSince the paper is focused on a theoretical side, it's unlikely to have potential negative social impact. And some limitations and future research directions are mentioned in the Conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the theoretical performance of an optimistic variant of PPO in episodic adversarial linear MDPs with full-information feedback (i.e., without assuming the reward functions are linear in the feature map), and establishes a regret bound of O(d^3/4 H^2 K^3/4) that matches the optimal regret bound in both stochastic linear MDPs and adversarial linear MDPs. The authors also introduce a new multi-batched updating mechanism to enable a new covering number argument of value and policy classes in their theoretical analysis.\n\nSTRENGTHS:\nExisting theoretical studies of PPO mainly focus on linear mixture MDPs with full-information feedback, which are implemented in a model-based manner and require an integration of the individual base model. In comparison, this work studies another set of linear MDPs that have low-rank representations and proposes a new optimistic variant of PPO that is provably efficient in both stochastic linear MDPs and adversarial linear MDPs. \n\nIn particular, this work exhibits several promising results:\n1. From the algorithmic perspective, the proposed algorithm involves the novel design of a multi-batched updating mechanism and a policy evaluation step via average rewards.\n\n2. Regarding the performance guarantee, the authors establish the optimal regret bound of O(d^3/4 H^2 K^3/4) with two fundamental findings. Instead of using the existing covering argument in linear MDPs, this work presents a new covering number argument for the value and policy classes. In addition, to ensure the sublinear regret, careful analysis has been done to analyze the drift between adjacent policies to control the error arising from the policy evaluation step.\n\n3. Apart from the regret guarantee, it also provides a PAC guarantee as sample complexity, which allows fair comparison with existing works in this line.\n\nThis work is well-organized and clearly articulates each part with the corresponding motivation, challenges, as well as technical novelties in its solutions. Overall, this paper is technically sound and demonstrates considerable technical novelties. It provides a better understanding of PPO in a class of MDPs with function approximation, which potentially benefits policy optimization in practice.\n\nWEAKNESSES:\nWhile this paper provides insights into the PPO algorithm in linear MDPs, it does have several drawbacks:\n1. One of the claimed novelties is the multi-batched updating mechanism, which coincides with the similar idea of \"policy switch\" in literature. However, there is no discussion of the existing works that involve \"policy switch\", and thus it is not clear whether the computational efficiency is solely brought by the policy switch scheme. \n\n2. Right now, the regret bound requires the batch size to be O(\\sqrt{d^3 K}), with leads to the number of batches being O(\\sqrt{K}). But whether this choice of batch size and the number of batches is optimal remains to be unknown. It would be worthwhile to study and discuss the balance between the batch size and the number of batches, the optimal choice, how the balance will affect the regret.\n\n3. Right now, this is solely theory-based work. As PPO is applied widely in practice, it will be beneficial to include simple benchmark empirical studies to demonstrate its effectiveness.\n\nQUESTIONS:\n1. In the introduction, the authors mentioned both Cai et al. and He et al. that study linear mixture MDPs are implemented in a model-based manner, whereas existing theoretical studies in linear MDPs are typically value-based methods. However, from the algorithmic perspective, algorithms in Cai et al. and He et al. also perform regularized least-square approximation on value functions in the policy evaluation step. Do you treat all algorithmics that tries to explicitly learn /approximate the transition model (i.e., \\hat{p}) as model-based methods in MDPs with function approximation? Could you explain whether your approach falls into the model-based category? If not, which step makes the difference? Is the multi-batched update the main reason that makes the proposed algorithm more computationally efficient compared to the direct extension of the existing OPPO method to linear MDPs from the algorithmic perspective?\n\nLIMITATIONS:\nThis is a theory paper with no potential negative social impact under the discussed context.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This work resolves the known issue for generalizing the policy-based algorithm proposed in [Cai el al.] for linear mixture MDPs to linear MDPs, by multi-batch updating and a bew covering number argument. The proposed model-free policy optimization algorithm advances the theoretical study of PPO in adversarial linear MDPs with full-information feedback.

STRENGTHS:
1 This work resolves the issue in generalizing the policy-based algorithm in [Cai. el al.] for linear mixture MDPs to linear MDPs.

2 Besides stochastic linear MDPs, the proposed algorithm can handle adversarial rewards with full-information feedback.

WEAKNESSES:
1 Full-information feedback instead of bandit feedback is considered.

2 The proposed novel techniques developed in this work are insufficient for the policy-based algorithm to achieve the minimax optimal regret.

QUESTIONS:
1 in Remark 3.3, the authors claim that they achieve the state-of-the-art regret bound for adversarial linear MDPs with full-information feedback. does it mean their result matches or improves the best existing result in linear MDPs? It would be better to give the reference of the best known exisiting result here.

2 Equipped with the new proposed techniques, is it possible to improve the result by refined analysis?

3 It seems the proposed novel techniques are specialized for policy-based algorithm for linear MDPs. Can the authors comment on the impact of the new techniques? For example, by applying those techniques, there is any existing result that can be improved or any hard problem that now can be solved.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07