SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Han Zhong
Center for Data Science
hanzhong@stu.pku.edu.cn
Tong Zhang
HKUST
tongzhang@tongzhang-ml.org
The proximal policy optimization (PPO) algorithm stands as one of the most prosperous methods in the field of reinforcement learning (RL). Despite its success, the theoretical understanding of PPO remains deficient. Specifically, it is unclear whether PPO or its optimistic variants can effectively solve linear Markov decision processes (MDPs), which are arguably the simplest models in RL with function approximation. To bridge this gap, we propose an optimistic variant of PPO for episodic adversarial linear MDPs with full-information feedback, and establish a  $\tilde{\mathcal{O}}(d^{3/4}H^2K^{3/4})$  regret for it. Here  $d$  is the ambient dimension of linear MDPs,  $H$  is the length of each episode, and  $K$  is the number of episodes. Compared with existing policy-based algorithms, we achieve the state-of-the-art regret bound in both stochastic linear MDPs and adversarial linear MDPs with full information. Additionally, our algorithm design features a novel multi-batched updating mechanism and the theoretical analysis utilizes a new covering number argument of value and policy classes, which might be of independent interest.
Reinforcement learning (RL) [34] is a prominent approach to solving sequential decision making problems. Its tremendous successes [19, 32, 33, 5] can be attributed, in large part, to the advent of deep learning [22] and the development of powerful deep RL algorithms [24, 28, 29, 11]. Among these algorithms, the proximal policy optimization (PPO) [29] stands out as a particularly significant approach. Indeed, it continues to play a pivotal role in recent advancements in large language models [27].
Motivated by the remarkable empirical success of PPO, numerous studies seek to provide theoretical justification for its effectiveness. In particular, Cai et al. [6] develop an optimistic variant of the PPO algorithm in adversarial linear mixture MDPs with full-information feedback [4, 25], where the transition kernel is a linear combination of several base models. Theoretically, they show that the optimistic variant of PPO is capable of tackling problems with large state spaces by establishing a sublinear regret that is independent of the size of the state space. Building upon this work, He et al. [14] study the same setting and refine the regret bound derived by Cai et al. [6] using the weighted regression technique [44]. However, the algorithms in Cai et al. [6], He et al. [14] and other algorithms for linear mixture MDPs [4, 44] are implemented in a model-based manner and require an integration of the individual base model, which can be computationally expensive or even intractable in general. Another arguably simplest RL model involving function approximation is linear MDP [38, 17], which assumes that the reward functions and transition kernel enjoy a low-rank representation. For this model, several works [17, 15, 3, 13] propose value-based algorithms that directly approximate the value function and provide regret guarantees. To demonstrate the efficiency of PPO in linear MDPs from a theoretical perspective, one potential approach is to extend the results
Table 1: A comparison with closely related works on policy optimization for linear MDPs. Here "Sto." and "Adv." represent stochastic rewards and adversarial rewards, respectively. Additionally, "Bandit" and "Full-infor." signify bandit feedback and full-information feedback. We remark that Zanette et al. [40] do not consider the regret minimization problem and the regret reported in the table is implied by their sample complexity. We will compare the sample complexity provided in Zanette et al. [40] and the complexity implied by our regret in Remark 3.2.
Sto. + Bandit	Adv. + Full-infor.	Adv. + Bandit	Regret
[40]	✓	✗	✗	O(d3/4H13/4K3/4)
[8]	✓	✓	✓	O(d2/3A1/9H20/9K8/9)
[31]	✓	✓	✓	O(dH2K6/7)
Our Work	✓	✓	✗	O(d3/4H2K3/4)
of Cai et al. [6], He et al. [14] to linear MDPs. However, this extension poses significant challenges due to certain technical issues that are unique to linear MDPs. See §1.1 for a detailed description.
In this paper, we address this technical challenge and prove that the optimistic variant of PPO is provably efficient for stochastic linear MDPs and even adversarial linear MDPs with full-information feedback. Our contributions are summarized below.
- In terms of algorithm design, we propose a new algorithm OPPO+ (Algorithm 1), an optimistic variant of PPO, for adversarial linear MDPs with full-information feedback. Our algorithm features two novel algorithm designs including a multi-batched updating mechanism and a policy evaluation step via average rewards.
- Theoretically, we establish a  $\widetilde{\mathcal{O}}(d^{3/4}H^2K^{3/4})$  regret for 0PP0+, where  $d$  is the ambient dimension of linear MDPs,  $H$  is the horizon, and  $K$  is the number of episodes. To achieve this result, we employ two new techniques. Firstly, we adopt a novel covering number argument for the value and policy classes, as explicated in §C. Secondly, in Lemma 4.3, we meticulously analyze the drift between adjacent policies to control the error arising from the policy evaluation step using average rewards.
- Compared with existing policy optimization algorithms, our algorithm achieves a better regret guarantee for both stochastic linear MDPs and adversarial linear MDPs with full-information feedback (to our best knowledge). See Table 1 for a detailed comparison.
In summary, our work provides a new theoretical justification for PPO in linear MDPs. To illustrate our theory more, we highlight the challenges and our novelties in §1.1.
1.1 Challenges and Our Novelties
Challenge 1: Covering Number of Value Function Class. In the analysis of linear MDPs (see Lemma B.3 or Lemma B.3 in Jin et al. [17]), we need to calculate the covering number of  $\mathcal{V}_h^k$ , which is the function class of the estimated value function at the  $h$ -th step of the  $k$ -th episode and takes the following form:
$$
\mathcal {V} _ {h} ^ {k} = \{V (\cdot) = \langle Q _ {h} ^ {k} (\cdot , \cdot), \pi_ {h} ^ {k} (\cdot | \cdot) \rangle_ {\mathcal {A}} \},
$$
where  $Q_h^k$  and  $\pi_h^k$  are estimated Q-function and policy at the  $h$ -th step of the  $k$ -th episode, respectively. For value-based algorithms (e.g., LSVI-UCB in Jin et al. [17]),  $\pi_h^k$  is the greedy policy with respect to  $Q_h^k$ . Then we have
$$
\mathcal {V} _ {h} ^ {k} = \{V (\cdot) = \max  _ {a} Q _ {h} ^ {k} (\cdot , a) \}.
$$
Since  $\max_{a}$  is a contraction map, it suffices to calculate  $\mathcal{N}(\mathcal{Q}_h^k)$ , where  $\mathcal{Q}_h^k$  is the function class of  $Q_h^k$  and  $\mathcal{N}(\cdot)$  denotes the covering number. By a standard covering number argument (Lemma D.4 or Lemma D.6 in Jin et al. [17]), we can show that  $\log \mathcal{N}(\mathcal{Q}_h^k) \leq \widetilde{\mathcal{O}}(d^2)$ . However, for policy-based algorithms such as PPO,  $\pi_h^k$  is a stochastic policy, which makes the log-covering number  $\log \mathcal{N}(\mathcal{V}_h^k)$  may have a polynomial dependency on the size of action space  $\mathcal{A}$  (log-covering number
Figure 1: Algorithm diagram. The entire  $K$  episodes are partitioned into  $L = K / B$  batches, with each batch containing  $B$  consecutive episodes. The policy/value updating only occurs at the beginning of each batch. The policy evaluation uses the reward function in the last batch.
of  $|\mathcal{A}|$ -dimensional probability distributions is at the order of  $|\mathcal{A}|$ . We also remark that linear mixture MDPs are more amenable to theoretical analysis compared to linear MDPs, as they do not necessitate the calculation of the covering number of  $\mathcal{V}_h^k$  [4, 6]. As a result, the proof presented by Cai et al. [6] is valid for linear mixture MDPs, but cannot be extended to linear MDPs. See §A for more elaboration of technical challenges.
Novelty 1: Multi-batched Updating and A New Covering Number Argument. Our key observation is that if we improve the policy like PPO (see (3.2) or Schulman et al. [29], Cai et al. [6]),  $\pi_h^k$  admits a softmax form, i.e.,
$$
\pi_ {h} ^ {k} (\cdot \mid \cdot) \propto \exp \left(\sum_ {i = 1} ^ {l} Q _ {h} ^ {k _ {i}} (\cdot , \cdot)\right).
$$
Here  $\{k_i\}_{i = 1}^l$  is a sequence of episodes, where  $k_{i}\leq k$  for all  $i\in [l]$ , denoting the episodes in which our algorithm performs policy optimization prior to the  $k$ -th episode. By a technical lemma (Lemma C.3), we can show that
$$
\log \mathcal {N} (\text {c l a s s o f} \pi_ {h} ^ {k}) \lesssim \sum_ {i = 1} ^ {l} \log \mathcal {N} \left(\mathcal {Q} _ {h} ^ {k _ {i}}\right) \lesssim \widetilde {\mathcal {O}} (l \cdot d ^ {2}).
$$
If we perform the policy optimization in each episode like Cai et al. [6], Shani et al. [30],  $l$  may linear in  $K$  and the final regret bound becomes vacuous. Motivated by this, we use a multi-batched updating scheme. In specific,  $\mathsf{OPPO}+$  divides the whole learning process into several batches and only updates policies at the beginning of each batch. See Figure 1 for visualization. For example, if the number of batches is  $K^{1/2}$  (i.e., each batch consists of consecutive  $K^{1/2}$  episodes), we have  $l \leq K^{1/2}$  and the final regret is at the order of  $\widetilde{\mathcal{O}}(K^{3/4})$ . Here we assume  $K^{1/2}$  is a positive integer for simplicity. See §C for details.
Challenge 2: Adversarial Rewards. Compared with previous value-based algorithms [e.g., 17], one superiority of optimistic PPO is that they can learn adversarial MDPs with full-information feedback [6, 30]. In Cai et al. [6], Shani et al. [30], the policy evaluation step is that
$$
Q _ {h} ^ {k} = r _ {h} ^ {k} + \widehat {\mathbb {P}} _ {h} ^ {k} V _ {h + 1} ^ {k} + \text {b o n u s f u n c t i o n}, \quad \forall h \in [ H ], \quad V _ {h + 1} ^ {k} = 0,
$$
where  $r_h^k$  is the adversarial reward function and  $\widehat{\mathbb{P}}_h^k V_{h+1}^k$  is the estimator of the expected next step value of  $V_{h+1}^k$ . This policy evaluation step is invalid if we use the multi-batched updating. Consider the following case, if the number of batches is  $K^{1/2}$  and
$$
r_{h}^{k}(\cdot ,\cdot) = \left\{ \begin{array}{cc}0 & k\in \{iK^{1 / 2} + 1\}_{i = 0}^{K^{1 / 2}}\\ \text{arbitrary} & \text{otherwise} \end{array} \right..
$$
Then the algorithm only uses zero rewards to find the optimal policy in hindsight with respect to arbitrary adversarial rewards, which is obviously impossible.
Novelty 2: Policy Evaluation via Average Reward and Smoothness Analysis. To tackle the above challenge, we adopt the following policy evaluation step at the beginning of each batch
$$
Q _ {h} ^ {k} = \bar {r} _ {h} ^ {k} + \widehat {\mathbb {P}} _ {h} ^ {k} V _ {h + 1} ^ {k} + \text {b o n u s f u n c t i o n}, \quad \forall h \in [ H ], \quad V _ {h + 1} ^ {k} = 0,
$$
where  $\bar{r}_h^k$  the average reward of the last batch:
$$
\bar {r} _ {h} ^ {k} = \frac {\sum (\text {r e w a r d f u n c t i o n s o f t h e l a s t b a t c h})}{\text {b a t c h s i z e}}.
$$
Let  $\pi^{k_i}$  denote the policy executed in the  $i$ -th batch. Intuitively,  $\pi_{k_{i+1}}$  is the desired policy within  $i$ -th batch since its calculation only uses the rewards in the first  $(i-1)$  batches (cf. Figure 1). Hence, compared with Cai et al. [6], Shani et al. [30], we need to handle the gap between the performance of  $\pi^{k_{i+1}}$  and  $\pi^{k_i}$  in the  $i$ -th batch. Fortunately, this error can be controlled due to the "smoothness" of policies in adjacent batches. See Lemma 4.3 for details.
Policy Optimization Algorithms. The seminal work of Schulman et al. [29] proposes the PPO algorithm, and a line of following works seeks to provide theoretical guarantees for it. In particular, Cai et al. [6] proposes the optimistic PPO (OPPO) algorithm for adversarial linear mixture MDPs and establishes regret  $\widetilde{\mathcal{O}}(d\sqrt{H^4K})$  for it. Then, He et al. [14] improve the regret to  $\widetilde{\mathcal{O}}(d\sqrt{H^3K})$  by the weighted regression technique [44]. Besides their works on linear mixture MDPs, Shani et al. [30], Wu et al. [37] provide fine-grained analysis of optimistic variants of PPO in the tabular case. The works of Fei et al. [9], Zhong et al. [42] show that optimistic variants of PPO can solve nonstationary MDPs. However, none of these works show that the optimistic variant of PPO is provably efficient for linear MDPs.
There is another line of works [2, 10, 40] proposes optimistic policy optimization algorithms based on the natural policy gradient (NPG) algorithm [18] and the policy-cover technique. But their works are limited to the stochastic linear MDPs, while our work can tackle adversarial rewards. Compared with their results for stochastic linear MDPs, our work can achieve a better regret and compatible sample complexity. See Table 1 and Remark 3.2 for a detailed comparison. Several recent works [26, 23, 20, 8, 31] study the more challenging problem of learning adversarial linear MDPs with only bandit feedback, which is beyond the scope of our work. Without access to exploratory policies or even known transitions, their regret is at least  $\tilde{\mathcal{O}}(K^{6/7})$  [31], while our work achieves a better  $\tilde{\mathcal{O}}(K^{3/4})$  regret with the full-information feedback assumption.
RL with Linear Function Approximation. Our work is related to previous works proposing value-based algorithms for linear MDPs [38, 17]. The work of Yang and Wang [38] develops the first sample efficient algorithm for linear MDPs with a generative model. Then Jin et al. [17] proposes the first provably efficient algorithms for linear MDPs in the online setting. The results of Jin et al. [17] are later improved by [35, 15, 13, 3]. In particular, Agarwal et al. [3], He et al. [13] show that the nearly minimax optimal regret  $\widetilde{\mathcal{O}}(d\sqrt{H^3K})$  is achievable in stochastic linear MDPs. Compared with these value-based algorithms, our work can tackle the more challenging adversarial linear MDPs with full-information feedback.
There is another line of works [4, 25, 6, 41, 14, 45, 44, 43] studying linear mixture MDPs, which is another model of RL with linear function approximation. It can be shown that linear MDPs and linear mixture MDPs are incompatible in the sense that neither model is a special case of the other. Among these works, Zhou et al. [44], Zhou and Gu [43] establishes nearly minimax regret  $\tilde{\mathcal{O}}(d\sqrt{H^3K})$  for stochastic linear mixture MDPs. Our work is more related to Cai et al. [6], He et al. [14] on adversarial linear mixture MDPs with full-information feedback. We have remarked that it is nontrivial to extend their results to linear MDPs.
Notations. We use  $\mathbb{N}^+$  to denote the set of positive integers. For any  $H \in \mathbb{N}^+$ , we denote  $[H] = \{1, 2, \ldots, H\}$ . For any  $H \in \mathbb{N}^+$  and  $x \in \mathbb{R}$ , we use the notation  $\min \{x, H\}^+ =$
$\min \{H, \max \{0, x\}\}$ . Besides, we denote by  $\Delta(\mathcal{A})$  the set of probability distributions on the set  $\mathcal{A}$ . For any two distributions  $P$  and  $Q$  over  $\mathcal{A}$ , we denote  $\mathrm{KL}(P \| Q) = \mathbb{E}_{a \sim P}[\log \mathrm{d}P(a) / \mathrm{d}Q(a)]$ .
Episodic Adversarial MDPs. We consider an episodic MDP  $\mathcal{M}$ , which is denoted by a tuple
$$
\left(\mathcal {S}, \mathcal {A}, H, K, \left\{r _ {h} ^ {k} \right\} _ {(k, h) \in [ K ] \times [ H ]}, \mathcal {P} = \left\{\mathcal {P} _ {h} \right\} _ {h \in [ H ]}\right),
$$
where  $S$  is the state space,  $\mathcal{A}$  is the action space,  $H$  is the length of each episode,  $K$  is the number of episodes,  $r_h^k: S \times \mathcal{A} \mapsto [0,1]$  is the deterministic reward function at the  $h$ -th step of  $k$ -th episode,  $\mathcal{P}_h$  is the transition kernel with  $\mathcal{P}_h(s' \mid s, a)$  being the transition probability for state  $s$  to transfer to the next state  $s'$  given action  $a$  at the  $h$ -th step. We consider the adversarial MDPs with full-information feedback, which means that the reward  $\{r_h^k\}_{h \in [H]}$  is adversariably chosen by the environment at the beginning of the  $k$ -th episode and revealed to the learner after the  $k$ -th episode.
A policy  $\pi = \{\pi_h\}_{h\in [H]}$  is a collection of  $H$  functions, where  $\pi_h:\mathcal{S}\mapsto \Delta (\mathcal{A})$  is a function that maps a state to a distribution over action space at step  $h$ . For any policy  $\pi$  and reward function  $\{r_h^k\}_{h\in [H]}$ , we define the value function  $V_{h}^{\pi ,k}:\mathcal{S}\mapsto \mathbb{R}$  and Q-function  $Q_{h}^{\pi ,k}:\mathcal{S}\times \mathcal{A}\mapsto \mathbb{R}$  as
$$
V _ {h} ^ {\pi , k} (x) = \mathbb {E} _ {\pi} \bigg [ \sum_ {h ^ {\prime} = h} ^ {H} r _ {h ^ {\prime}} ^ {k} (x _ {h ^ {\prime}}, a _ {h ^ {\prime}}) \Big | x _ {h} = x \bigg ],
$$
$$
Q _ {h} ^ {\pi , k} (x, a) = \mathbb {E} _ {\pi} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {h ^ {\prime}} ^ {k} \left(x _ {h ^ {\prime}}, a _ {h ^ {\prime}}\right) \Bigg | x _ {h} = x, a _ {h} = a \right],
$$
for any  $(x, a, k, h) \in S \times \mathcal{A} \times [K] \times [H]$ . Here the expectation  $\mathbb{E}_{\pi}[\cdot]$  is taken with respect to the randomness of the trajectory induced by policy  $\pi$  and transition kernel  $\mathcal{P}$ . It is well-known that the value function and Q-function satisfy the following Bellman equation for any  $(x, a) \in S \times \mathcal{A}$ ,
$$
V _ {h} ^ {\pi , k} (x) = \left\langle Q _ {h} ^ {\pi , k} (x, \cdot), \pi_ {h} (\cdot \mid x) \right\rangle_ {\mathcal {A}}, \quad Q _ {h} ^ {\pi , k} (x, a) = r _ {h} ^ {k} (x, a) + \left(\mathbb {P} _ {h} V _ {h + 1} ^ {\pi , k}\right) (x, a), \tag {2.1}
$$
where  $\langle \cdot, \cdot \rangle_{\mathcal{A}}$  denotes the inner product over the action space  $\mathcal{A}$  and we will omit the subscript when it is clear from the context. Here  $\mathbb{P}_h$  is the operator defined as
$$
\left(\mathbb {P} _ {h} V\right) (x, a) = \mathbb {E} _ {x ^ {\prime} \sim \mathcal {P} _ {h} (\cdot | x, a)} \left[ V \left(x ^ {\prime}\right) \right], \quad \forall V: \mathcal {S} \mapsto \mathbb {R}. \tag {2.2}
$$
Interaction Process and Learning Objective. We consider the online setting, where the learner improves her performance by interacting with the environment repeatedly. The learning process consists of  $K$  episodes and each episode starts from a fixed initial state  $x_{1}^{2}$ . At the beginning of the  $k$ -th episode, the environment adversarially chooses reward functions  $\{r_h^k\}_{h \in [H]}$ , which can depend on previous  $(k-1)$  trajectories. Then the agent determines a policy  $\pi^k$  and receives the initial state  $x_{1}^{k} = x_{1}$ . At each step  $h \in [H]$ , the agent receives the state  $x_{h}^{k}$ , chooses an action  $a_{h}^{k} \sim \pi_{h}^{k}(\cdot \mid x_{h}^{k})$ , receives the reward function  $r_{h}^{k}$ , and transits to the next state  $x_{h+1}^{k}$ . The  $k$ -th episode ends after  $H$  steps.
We evaluate the performance of an online algorithm by the notion of regret [7], which is defined as the value difference between the executed policies and the optimal policy in hindsight:
$$
\operatorname {R e g r e t} (K) = \max  _ {\pi} \sum_ {k = 1} ^ {K} \left(V _ {1} ^ {\pi , k} \left(x _ {1}\right) - V _ {1} ^ {\pi^ {k}, k} \left(x _ {1}\right)\right).
$$
For simplicity, we denote the optimal policy in hindsight by  $\pi^{*}$ , i.e.,  $\pi^{*} = \operatorname{argmax}_{\pi} \sum_{k=1}^{K} V_{1}^{\pi, k}(x_{1})$ .
Linear MDPs. We focus on the linear MDPs [38, 17], where the transition kernels are linear in a known feature map.
Definition 2.1 (Linear MDP). We say an MDP  $(S, \mathcal{A}, H, K, \{r_h^k\}_{(k,h) \in [K] \times [H]}, \mathcal{P} = \{\mathcal{P}_h\}_{h \in [H]})$  is a linear MDP if there exists a known feature  $\phi: S \times \mathcal{A} \mapsto \mathbb{R}^d$  such that for any  $(x,a,k,h) \in S \times \mathcal{A} \times [K] \times [H]$ , we have
$$
\mathcal {P} _ {h} \left(x ^ {\prime} \mid x, a\right) = \phi (x, a) ^ {\top} \mu_ {h} \left(x ^ {\prime}\right),
$$
where  $\mu_h = (\mu_h^{(1)},\dots,\mu_h^{(d)})$  are  $d$  unknown signed measures over  $\mathcal{S}$  satisfying  $\| \mu_h(\mathcal{S})\| _2\leq \sqrt{d}$ .
<sup>1</sup>This assumption is without loss of generality since our subsequent results are ready to be extended to the stochastic reward case.
2Our subsequent analysis can be generalized to the case where the initial state is chosen from a fixed distribution across all episodes.
Algorithm 1 OPPO+
Require: Batch size  $B\in \mathbb{N}^{+}$  , regularization parameter  $\lambda >0$  , and confidence radius  $\beta >0$  1: Initialize  $\{Q_h^0\} ,\{r_h^k\}_{-B\leq k\leq 0}$  as zero functions and  $\{\pi_h^0\}$  as uniform distributions on  $\mathcal{A},\forall h\in$ $[H]$    
2: Let  $L = K / B,i = 1$  , and  $k_{i} = (i - 1)\cdot B + 1$  for  $1\le i\in [L]$    
3: for episode  $k = 1,2,\ldots ,K$  do   
4:Receive the initial state  $x_{1}^{k}$    
5: if  $k = k_{i}$  then   
6:  $V_{H + 1}^{k}(\cdot)\gets 0.$    
7: for step  $h = 1,2,\dots ,H$  do   
8: Update the policy by  $\pi_h^k (\cdot |\cdot)\propto \pi_h^{k - 1}(\cdot |\cdot)\cdot \exp \{\alpha \cdot Q_h^{k - 1}(\cdot ,\cdot)\}$    
9: end for   
10: for step  $h = H,H - 1,\dots ,1$  do   
11:  $\Lambda_h^k\gets \sum_{\tau = 1}^{k - 1}\phi (x_h^\tau ,a_h^\tau)\phi (x_h^\tau ,a_h^\tau)^\top +\lambda \cdot I_d.$    
12:  $w_{h}^{k}\gets (\Lambda_{h}^{k})^{-1}\sum_{\tau = 1}^{k - 1}\phi (x_{h}^{\tau},a_{h}^{\tau})\cdot V_{h + 1}^{k}(x_{h + 1}^{\tau}).$    
13:  $\bar{r}_h^k (\cdot ,\cdot)\gets (\sum_{j = k_i - 1}^{k_i - 1}r_h^j (\cdot ,\cdot)) / B.$    
14:  $\Gamma_h^k (\cdot ,\cdot)\gets \beta \cdot [\phi (\cdot ,\cdot)^\top (\Lambda_h^k)^{-1}\phi (\cdot ,\cdot)]^{1 / 2}.$    
15:  $\widehat{\mathbb{P}}_h^k V_{h + 1}^k (\cdot ,\cdot)\gets \min \{\phi (\cdot ,\cdot)^\top w_h^k +\Gamma_h^k (\cdot ,\cdot),H - h\} ^+$    
16:  $Q_h^k (\cdot ,\cdot)\gets \bar{r}_h^k (\cdot ,\cdot) + \widehat{\mathbb{P}}_h^k V_{h + 1}^k (\cdot ,\cdot).$    
17:  $V_{h}^{k}(\cdot)\leftarrow \langle Q_{h}^{k}(\cdot ,\cdot),\pi_{h}^{k}(\cdot |\cdot)\rangle_{A}$    
18: end for   
19:  $i\gets i + 1$    
20: else   
21:  $Q_h^k\gets Q_h^{k - 1},V_h^k\gets V_h^{k - 1},\pi_h^k\gets \pi_h^{k - 1},\bar{r}_h^k\gets \bar{r}_h^{k - 1}\forall h\in [H].$    
22: end if   
23: for  $h = 1,2,\dots ,H$  do   
24: Take the action following  $a_h^k\sim \pi_h^k (\cdot |x_h^k)$    
25: Observe the reward function  $r_h^k (\cdot ,\cdot)$  and receive the next state  $x_{h + 1}^{k}$    
26: end for   
27: end for
Since we have access to the full-information feedback, we do not assume the reward functions are linear in the feature map  $\phi$  like Jin et al. [17]. We also remark that the adversarial linear MDP with full-information feedback is a more challenging problem than the stochastic linear MDP with bandit feedback studied in Jin et al. [17]. In fact, for stochastic linear MDPs, we can assume the reward functions are known without loss of generality since learning the linear transition kernel is more difficult than the linear reward.
In this section, we propose a new algorithm  $\mathsf{OPPO} +$  (Algorithm 1) to solve adversarial linear MDPs with full-information feedback. In what follows, we highlight the key steps of  $\mathsf{OPPO} +$ .
Multi-batched Updating. Due to the technical issue elaborated in §1.1, we adopt the multi-batched updating rule. In specific,  $\mathsf{OPPO}+$  divides the total  $K$  episodes into  $L = K / B$  batches and each batch consists of  $B$  consecutive episodes. Here we assume  $K / B$  is a positive integer without loss of generality<sup>3</sup>. For ease of presentation, we use  $k_{i} = (i - 1) \cdot B + 1$  to denote the first episode in the  $i$ -th batch. When the  $k$ -th episode is the beginning of a batch (i.e.,  $k = k_{i}$  for some  $i \in [L]$ ),  $\mathsf{OPPO}+$  performs the following policy improvement step and policy evaluation step.