SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Policy Improvement. In the policy improvement step of the  $k$ -th episode ( $k = k_{i}$  for some  $i \in [L]$ ),  $\mathsf{OPPO+}$  calculates  $\pi^k$  based on the previous policy  $\pi^{k-1}$  using PPO [29]. In specific,  $\mathsf{OPPO+}$  updates
<sup>3</sup>We can only consider the first  $B \cdot \lfloor K / B \rfloor$  episodes since the remaining episodes will lead at most  $BH$  regret, which is a non-dominant term in final regret bound
$\pi^k$  by solving the following proximal policy optimization problem:
$$
\pi^ {k} \leftarrow \operatorname * {a r g m a x} _ {\pi} \left\{L _ {k - 1} (\pi) - \alpha^ {- 1} \cdot \mathbb {E} _ {\pi^ {k - 1}} \left[ \sum_ {h = 1} ^ {H} \mathrm {K L} \left(\pi_ {h} (\cdot | x _ {h}) \| \pi_ {h} ^ {k - 1} (\cdot | x _ {h})\right) \right] \right\}, \tag {3.1}
$$
where  $\alpha > 0$  is the stepsize that will be specified in Theorem 3.1, and  $L_{k-1}(\pi)$  takes form
$$
L _ {k - 1} (\pi) = V _ {1} ^ {\pi^ {k - 1}, k - 1} (x _ {1} ^ {k}) + \mathbb {E} _ {\pi^ {k - 1}} \left[ \sum_ {h = 1} ^ {H} \langle Q _ {h} ^ {k - 1} (x _ {h}, \cdot), \pi_ {h} (\cdot \mid x _ {h}) - \pi_ {h} ^ {k - 1} (\cdot \mid x _ {h}) \rangle \right],
$$
which is proportional to the local linear function of  $V_{1}^{\pi, k-1}(x_{1}^{k})$  at  $\pi^{k-1}$  and replaces the unknown Q-function  $Q_{h}^{\pi^{k-1}, k-1}$  by the estimated one  $Q_{h}^{k-1}$  for any  $h \in [H]$ . It is not difficult to show that the updated policy  $\pi^{k}$  obtained in (3.1) admits the following closed form:
$$
\pi_ {h} ^ {k} (\cdot \mid x) \propto \pi_ {h} ^ {k - 1} (\cdot \mid x) \cdot \exp \left(\alpha \cdot Q _ {h} ^ {k - 1} (x, \cdot)\right), \quad \forall (x, h) \in \mathcal {S} \times [ H ]. \tag {3.2}
$$
Policy Evaluation. In the policy evaluation step of the  $k$ -th episode ( $k = k_{i}$  for some  $i \in [K]$ ),  $\mathsf{OPPO+}$  lets  $V_{H+1}^{k}$  be the zero function and iteratively calculates the estimated Q-function  $\{Q_h^k\}_{h \in [H]}$  in the order of  $h = H, H-1, \ldots, 1$ . Now we present the policy evaluation at the  $h$ -th step given estimated value  $V_{h+1}^{k}$ . By the definitions of linear MDP in Definition 2.1 and the operator  $\mathbb{P}_h$  in (2.2), we know  $\mathbb{P}_h V_{h+1}^k$  is linear in the feature map  $\phi$ . Inspired by this, we estimate its linear coefficient by solving the following ridge regression:
$$
w _ {h} ^ {k} = \underset {w \in \mathbb {R} ^ {d}} {\operatorname {a r g m i n}} \sum_ {\tau = 1} ^ {k - 1} \left(\phi \left(x _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) ^ {\top} w - V _ {h + 1} ^ {k} \left(x _ {h + 1} ^ {\tau}\right)\right) ^ {2} + \lambda \cdot I _ {d}, \tag {3.3}
$$
where  $\lambda > 0$  is the regularization parameter and  $I_d$  is the identify matrix. By solving (3.3), we have
$$
w _ {h} ^ {k} = \left(\Lambda_ {h} ^ {k}\right) ^ {- 1} \Big (\sum_ {\tau = 1} ^ {k - 1} \phi \left(x _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \cdot V _ {h + 1} ^ {k} \left(x _ {h + 1} ^ {\tau}\right) \Big), \quad \text {w h e r e} \Lambda_ {h} ^ {k} = \sum_ {\tau = 1} ^ {k - 1} \phi \left(x _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \phi \left(x _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) ^ {\top} + \lambda \cdot I _ {d}.
$$
Based on this linear coefficient, we construct the estimator  $\widehat{\mathbb{P}}_h^k V_{h + 1}^k$  as
$$
\left(\widehat {\mathbb {P}} _ {h} ^ {k} V _ {h + 1} ^ {k}\right) (\cdot , \cdot) = \min  \left\{\phi (\cdot , \cdot) ^ {\top} w _ {h} ^ {k} + \Gamma_ {h} ^ {k} (\cdot , \cdot), H - h \right\} ^ {+},
$$
where  $\Gamma_h^k = \beta \cdot (\phi(\cdot, \cdot)^\top (\Lambda_h^k)^{-1} \phi(\cdot, \cdot))^{1/2}$  is the bonus function and  $\beta > 0$  is a parameter that will be specified in Theorem 3.1. This form of bonus function also appears in the literature on linear bandits [21] and linear MDPs [17]. Finally, we update  $Q_h^k$  and  $V_h^k$  by
$$
Q _ {h} ^ {k} (\cdot , \cdot) = \bar {r} _ {h} ^ {k} (\cdot , \cdot) + \left(\widehat {\mathbb {P}} _ {h} ^ {k} V _ {h + 1} ^ {k}\right) (\cdot , \cdot), \quad V _ {h} ^ {k} (\cdot) = \langle Q _ {h} ^ {k} (\cdot , \cdot), \pi_ {h} ^ {k} (\cdot | \cdot) \rangle_ {\mathcal {A}}. \tag {3.4}
$$
Here  $\bar{r}_h^k$  is the average reward function in the last batch, that is
$$
\bar {r} _ {h} ^ {k} (\cdot , \cdot) = \frac {\sum_ {j = k _ {i - 1}} ^ {k _ {i} - 1} r _ {h} ^ {j} (\cdot , \cdot)}{B}, \tag {3.5}
$$
where  $k = k_{i} = (i - 1)\cdot B + 1$  and  $k_{i - 1} = (i - 2)\cdot B + 1$
Here we would like to make some comparisons between our algorithm  $\mathsf{OPPO}^+$  and other related algorithms. Different from previous value-based algorithms that take simply take the greedy policy with respect to the estimated Q-functions [17],  $\mathsf{OPPO}^+$  involves a policy improvement step like PPO [29]. This step is key to tackling adversarial rewards. The most related algorithm is OPPO proposed by Cai et al. [6], which performs the policy optimization in linear mixture MDPs. The main difference between  $\mathsf{OPPO}^+$  and OPPO is that we introduce a multi-batched updating and an average reward policy evaluation, which are important for solving linear MDPs (cf. §1.1). Finally, we remark that the multi-batched updating scheme is adopted by previous work on bandits [12] and RL [36]. But their algorithms are value-based and cannot tackle adversarial rewards.
Theorem 3.1 (Regret). Fix  $\delta \in (0,1]$  and  $K \geq d^3$ . Let  $B = \sqrt{d^3K}$  and  $\alpha = \sqrt{2B\log|\mathcal{A}| / (KH^2)}$ ,  $\lambda = 1$ ,  $\beta = \mathcal{O}(d^{1/4}HK^{1/4}\iota^{1/2})$  with  $\iota = \log(dHK|\mathcal{A}|/\delta)$ , then with probability at least  $1 - \delta$ , the regret of Algorithm 1 satisfies
$$
\operatorname {R e g r e t} (K) \leq \mathcal {O} \left(d ^ {3 / 4} H ^ {2} K ^ {3 / 4} \log | \mathcal {A} | \cdot \iota\right) + \mathcal {O} \left(d ^ {5 / 2} H ^ {2} K ^ {1 / 2} \cdot \iota\right).
$$
To illustrate our theory more, we make several remarks as follows.
Remark 3.2 (Sample Complexity). Since learning adversarial linear MDPs with full-information feedback is more challenging than learning stochastic linear MDPs with bandit feedback, the result in Theorem 3.1 also holds for stochastic linear MDPs. By the standard online-to-batch argument [16], we have that Algorithm 1 can find an  $\epsilon$ -optimal policy using at most
$$
\widetilde {\mathcal {O}} \Big (\frac {d ^ {3} H ^ {8}}{\epsilon^ {4}} + \frac {d ^ {5} H ^ {4}}{\epsilon^ {2}} \Big)
$$
samples. Compared with the sample complexity  $\widetilde{\mathcal{O}}(d^3 H^{13} / \epsilon^3)$  in Zanette et al. [40], we have a better dependency on  $H$  but a worse dependency on  $\epsilon$ . Moreover, by the standard sample complexity to regret argument [16], their sample complexity only gives a  $\widetilde{\mathcal{O}}(d^{3/4} H^{13/4} K^{3/4})$ , which is worse than our regret in Theorem 3.1. More importantly, the algorithm in Zanette et al. [40] lacks the ability to handle adversarial rewards, whereas our proposed algorithm overcomes this limitation.
Remark 3.3 (Optimality of Results). For stochastic linear MDPs, Agarwal et al. [3], He et al. [13] design value-based algorithms with  $\widetilde{\mathcal{O}}(d\sqrt{H^3K})$  regret, which matches the lower bound  $\Omega(d\sqrt{H^3K})$  [44] up to logarithmic factors. It remains unclear whether policy-based based algorithms can achieve the nearly minimax optimal regret and we leave this as future work. For the more challenging adversarial linear MDPs with full-information feedback, we achieve the state-of-the-art regret bound. In this setup, a direct lower bound is  $\Omega(d\sqrt{H^3K})$  [44, 14], and we conjecture this lower bound is tight. It would be interesting to design algorithms with  $\sqrt{K}$  regret or even optimal regret in this setting.
Remark 3.4 (Beyond the Linear Function Approximation). The work of Agarwal et al. [2] extends their results to the kernel function approximation setting. We conjecture that our results can also be extended to RL with kernel and neural function approximation by the techniques in Yang et al. [39].
4 Proof of Theorem 3.1
Proof. Recall that  $B \in \mathbb{N}^+$  is the batch size,  $L = K / B$  is the number of batches, and  $k_i = (i - 1) \cdot B + 1$  for any  $i \in [L]$ . For any  $k \in [K]$ , we use  $t_k$  to denote the  $k_i$  satisfying  $k_i \leq k < k_{i + 1}$ . Moreover, we define the Bellman error as
$$
\delta_ {h} ^ {k} = r _ {h} ^ {k} + \mathbb {P} _ {h} V _ {h + 1} ^ {k} - Q _ {h} ^ {k}, \quad \forall (k, h) \in [ K ] \times [ H ]. \tag {4.1}
$$
Here  $V_{h + 1}^{k}$  and  $Q_{h}^{k}$  are the estimated value function and Q-function defined in (3.4), and  $\mathbb{P}_h$  is the operator defined in (2.2). Intuitively, (4.1) quantifies the violation of the Bellman equation in (2.1). With these notations, we have the following regret decomposition lemma.
Lemma 4.1 (Regret Decomposition). It holds that
$$
\begin{array}{l} \operatorname {R e g r e t} (K) = \sum_ {k = 1} ^ {K} \left(V _ {1} ^ {\pi^ {*}, k} \left(x _ {1} ^ {k}\right) - V _ {1} ^ {\pi^ {k}, k} \left(x _ {1} ^ {k}\right)\right) \\ = \underbrace {\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {*}} \left[ \left\langle Q _ {h} ^ {k} \left(x _ {h} , \cdot\right) , \pi_ {h} ^ {*} (\cdot \mid x _ {h}) - \pi_ {h} ^ {k} (\cdot \mid x _ {h}) \right\rangle \right]} _ {\text {p o l i c y o p t i m i z a t i o n e r r o r}} \\ + \underbrace {\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left(\mathbb {E} _ {\pi^ {*}} [ \delta_ {h} ^ {k} (x _ {h} , a _ {h}) ] - \mathbb {E} _ {\pi^ {k}} [ \delta_ {h} ^ {k} (x _ {h} , a _ {h}) ]\right)} _ {\text {}}. \\ \end{array}
$$
statistical error
Proof. This lemma is similar to the regret decomposition lemma in previous works [6, 30] on policy optimization. See §B.1 for a detailed proof.  $\square$
Proof. By See §4 for a detailed proof.
Lemma 4.1 shows that the total regret consists of the policy optimization error and the statistical error related to the Bellman error defined in (4.1). Notably, different from previous works [6, 30, 37, 14] that optimize policy in each episode, our algorithm performs policy optimization infrequently. Despite this, we can bound the policy optimization error in Lemma 4.1 by the following lemma.
Lemma 4.2. It holds that
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {*}} \left[ \left\langle Q _ {h} ^ {k} \left(x _ {h}, \cdot\right), \pi_ {h} ^ {*} (\cdot | x _ {h}) - \pi_ {h} ^ {k} (\cdot | x _ {h}) \right\rangle \right] \leq \sqrt {2 B H ^ {4} K \cdot \log | \mathcal {A} |}.
$$
Proof. See §B.2 for a detailed proof.
For the statistical error in Lemma 4.1, by the definitions of the policy evaluation step in (3.4) and Bellman error in (4.1), we have
$$
\delta_ {h} ^ {k} = \underbrace {r _ {h} ^ {k} - \bar {r} _ {h} ^ {k}} + \underbrace {\mathbb {P} _ {h} V _ {h + 1} ^ {k} - \widehat {\mathbb {P}} _ {h} ^ {k} V _ {h + 1} ^ {k}}. \tag {4.2}
$$
reward mismatch error transition estimation error
The following lemma establishes the upper bound of the cumulative reward mismatch error
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left(\mathbb {E} _ {\pi^ {*}} \left[ \left(r _ {h} ^ {k} - \bar {r} _ {h} ^ {k}\right) \left(x _ {h}, a _ {h}\right) \right] - \mathbb {E} _ {\pi^ {k}} \left[ \left(r _ {h} ^ {k} - \bar {r} _ {h} ^ {k}\right) \left(x _ {h}, a _ {h}\right) \right]\right), \tag {4.3}
$$
and thus relates the statistical error in Lemma 4.1 to the transition estimation error in (4.2).
Lemma 4.3. It holds with probability at least  $1 - \delta /2$  that
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left(\mathbb {E} _ {\pi^ {*}} \left[ \delta_ {h} ^ {k} \left(x _ {h}, a _ {h}\right) \right] - \mathbb {E} _ {\pi^ {k}} \left[ \delta_ {h} ^ {k} \left(x _ {h}, a _ {h}\right) \right]\right) \\ \leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {*}} [ (\mathbb {P} _ {h} V _ {h + 1} ^ {t _ {k}} - \widehat {\mathbb {P}} _ {h} ^ {t _ {k}} V _ {h + 1} ^ {t _ {k}}) (x _ {h}, a _ {h}) ] + \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} (\widehat {\mathbb {P}} _ {h} ^ {t _ {k}} V _ {h + 1} ^ {t _ {k}} - \mathbb {P} _ {h} V _ {h + 1} ^ {t _ {k}}) (x _ {h} ^ {k}, a _ {h} ^ {k}) \\ + B H + \alpha H ^ {3} K + \sqrt {H ^ {3} K t}. \\ \end{array}
$$
Proof. By calculation, we can show that the cumulative reward mismatch error in (4.3) is bounded by
$$
\sum_ {i = 1} ^ {L - 1} \sum_ {k = k _ {i}} ^ {k _ {i + 1} - 1} \left(V _ {1} ^ {\pi^ {k _ {i + 1}, k}} \left(x _ {1}\right) - V _ {1} ^ {\pi^ {k _ {i}}, k} \left(x _ {1}\right)\right) + \text {e r r o r t e r m s},
$$
which represents the smoothness of adjacent policies. Our smoothness analysis leverages the value difference lemma (Lemma B.1 or §B.1 in Cai et al. [6]) and the closed form of the policy improvement in (3.2). See §B.3 for a detailed proof.
Then we introduce the following lemma, which shows that the transition estimation error can be controlled by the bonus function.
Lemma 4.4. It holds with probability at least  $1 - \delta /2$  that
$$
- 2 \min  \left\{H, \Gamma_ {h} ^ {t _ {k}} (x, a) \right\} \leq \left(\mathbb {P} _ {h} V _ {h + 1} ^ {t _ {k}} - \widehat {\mathbb {P}} _ {h} ^ {t _ {k}} V _ {h + 1} ^ {t _ {k}}\right) (x, a) \leq 0
$$
for all  $(k,h,x,a)\in [K]\times [H]\times S\times \mathcal{A}$
Proof. The proof involves the standard analysis of self-normalized process [1] and a uniform concentration of the function class of  $V_{h + 1}^{k}$  [17]. As elaborated in §1.1, calculating the covering number of this function class is challenging and requires some new techniques. See §B.4 for a detailed proof.
Combining Lemmas 4.3 and 4.4, we have
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left(\mathbb {E} _ {\pi^ {*}} \left[ \delta_ {h} ^ {k} \left(x _ {h}, a _ {h}\right) \right] - \mathbb {E} _ {\pi^ {k}} \left[ \delta_ {h} ^ {k} \left(x _ {h}, a _ {h}\right) \right]\right) \\ \leq 2 \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \min  \left\{H, \Gamma_ {h} ^ {t _ {k}} \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) \right\} + B H + \alpha H ^ {3} K + \sqrt {H ^ {3} K \iota}. \\ \end{array}
$$
Hence, it remains to bound the term  $\sum_{k=1}^{K} \sum_{h=1}^{H} \min \{H, \Gamma_{h}^{t_k}(x_h^k, a_h^k)\}$ , which is the purpose of the following lemma.
Lemma 4.5. It holds that
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \min  \left\{H, \Gamma_ {h} ^ {t _ {k}} \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) \right\} \leq \mathcal {O} \left(d ^ {3 / 4} H ^ {2} K ^ {3 / 4} \cdot \iota\right) + \mathcal {O} \left(d ^ {5 / 2} H ^ {2} K ^ {1 / 2} \cdot \iota\right).
$$
Proof. For all  $k \in [K]$ , let  $\Gamma_h^k = \beta \cdot (\phi^\top(\Lambda_h^k)^{-1}\phi)^{1/2}$  with  $\Lambda_h^k = \sum_{\tau=1}^{k-1} \phi(x_h^\tau, a_h^\tau)\phi(x_h^\tau, a_h^\tau)^\top + \lambda \cdot I_d$ . By a doubling trick, we can prove that
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \min  \left\{H, \Gamma_ {h} ^ {t _ {k}} \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) \right\} \lesssim \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \min  \left\{H, \Gamma_ {h} ^ {k} \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) \right\} + \text {e r r o r t e r m s},
$$
which can be further bounded by elliptical potential lemma (Lemma D.5 or Lemma 11 in Abbasi-Yadkori et al. [1]). See §B.5 for a detailed proof.
Finally, putting Lemmas 4.1-4.5 together, we conclude the proof of Theorem 3.1.
Note the choice of the number of batches  $L$  determines the batch size  $B = K / L$ . We make the following remark to illustrate how the choice of  $L$  affects the final regret, which indicates that our choices of parameters are optimal based on our current analysis.
Remark 4.6 (Choices of Parameters and Final Regret). If  $\mathrm{OPPO+}$  performs the policy optimization for  $L$  times, then by Lemma C.3, we have the complexity (logarithmic covering number) of policy class is roughly  $\widetilde{\mathcal{O}}(L)$  (ignoring the dependency of  $d$  and  $H$ ). Furthermore, by the new self-normalized analysis in §C, we have  $\beta = \widetilde{\mathcal{O}}(\sqrt{L})$ , which further implies that the model estimation error is bounded by  $\beta \cdot \sum_{k=1}^{K} \sum_{h=1}^{H} \sqrt{\phi(x_h^k, a_h^k)(\Lambda_h^k)^{-1} \phi(x_h^k, a_h^k)} \leq \widetilde{\mathcal{O}}(\sqrt{LK})$  (cf. Lemma 4.5). Meanwhile, by Lemma 4.2, we have the policy optimization error is bounded by  $\widetilde{\mathcal{O}}(K/\sqrt{L}) = \widetilde{\mathcal{O}}(K^{3/4})$ . By choosing  $L = \Theta(\sqrt{K})$ , we obtain a regret of order  $\widetilde{\mathcal{O}}(K^{3/4})$ .
In this paper, we advance the theoretical study of PPO in stochastic linear MDPs and even adversarial linear MDPs with full-information feedback. We propose a novel algorithm, namely  $0\mathsf{PP0}+$ , which exhibits a state-of-the-art regret bound as compared to the prior policy optimization algorithms. Our work paves the way for future research in multiple directions. For instance, a significant open research question is to investigate whether policy-based algorithms can achieve the minimax regret in stochastic linear MDPs like previous value-based algorithms [3, 14]. Additionally, an interesting direction is to derive  $\sqrt{K}$ -regret bounds for adversarial linear MDPs with full-information or even bandit feedback.
This work was done while Han Zhong visiting HKUST. The authors would like to thank Miao Lu, Haipeng Luo, Tianhao Wu, and Wei Xiong for helpful discussions and feedback.
[40] Zanette, A., Cheng, C.-A. and Agarwal, A. (2021). Cautiously optimistic policy optimization and exploration with linear function approximation. In Conference on Learning Theory. PMLR.
[41] Zhang, Z., Yang, J., Ji, X. and Du, S. S. (2021). Improved variance-aware confidence sets for linear bandits and linear mixture mdp. Advances in Neural Information Processing Systems, 34 4342-4355.
[42] Zhong, H., Yang, Z., Wang, Z. and Szepesvári, C. (2021). Optimistic policy optimization is provably efficient in non-stationary mdps. arXiv preprint arXiv:2110.08984.
[43] Zhou, D. and Gu, Q. (2022). Computationally efficient horizon-free reinforcement learning for linear mixture mdps. arXiv preprint arXiv:2205.11507.
[44] Zhou, D., Gu, Q. and Szepesvari, C. (2021). Nearly minimax optimal reinforcement learning for linear mixture markov decision processes. In Conference on Learning Theory. PMLR.
[45] Zhou, D., He, J. and Gu, Q. (2021). Provably efficient reinforcement learning for discounted mdps with feature mapping. In International Conference on Machine Learning. PMLR.
A Further Elaboration of Challenge 1 in Section 1.1
Regarding the technical challenges involved in extending OPPO [6] to linear MDPs, we provided more detailed explanations below.
Technically, for linear mixture MDPs (Equation (B.20) in OPPO paper [6]), they need to analyze
$$
\left\| \sum_ {\tau = 1} ^ {k - 1} \phi_ {h} ^ {\tau} (x _ {h} ^ {\tau}, a _ {h} ^ {\tau}) \cdot \left(V _ {h + 1} ^ {\tau} (x _ {h + 1} ^ {\tau}) - (\mathbb {P} _ {h} V _ {h + 1} ^ {\tau}) (x _ {h} ^ {\tau}, a _ {h} ^ {\tau})\right) \right\| _ {(\Lambda_ {h} ^ {k}) ^ {- 1}}.
$$
Since  $V_{h + 1}^{\tau}$  is adapted to  $\mathcal{F}_{k,h,1} = \{(x_i^\tau ,a_i^\tau)\}_{(\tau ,i)\in [k - 1]\times [H]}\cup \{r^\tau \}_{\tau \in [k]}\cup \{(x_i^k,a_i^k)\}_{i\in [h]}$ , they can bound this term with classical self-normalized process analysis directly (see Lemma D.1 in Cai et al. [6]).
In contrast, for linear MDPs (see e.g., (B.26) in our paper or Lemma B.3 in Jin et al. [17]), we need to bound the term
$$
\left\| \sum_ {\tau = 1} ^ {k - 1} \phi \left(x _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \cdot \left(V _ {h + 1} ^ {k} \left(x _ {h + 1} ^ {\tau}\right) - \left(\mathbb {P} _ {h} V _ {h + 1} ^ {k}\right) \left(x _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right\| _ {\left(\Lambda_ {h} ^ {k}\right) ^ {- 1}}.
$$
Since  $V_{h + 1}^{\tau}$  is NOT adapted to  $\mathcal{F}_{k,h,1} = \{(x_i^\tau ,a_i^\tau)\}_{(\tau ,i)\in [k - 1]\times [H]}\cup \{r^\tau \}_{\tau \in [k]}\cup \{(x_i^k,a_i^k)\}_{i\in [h]}$ . We need to perform the uniform concentration on the function class of  $V_{h + 1}^{k}$ . The challenge of calculating the covering number of this function class has been elaborated in Challenge 1 of Section 1.1.
B Missing Proofs of Main Theorem
B.1 Proof of Lemma 4.1
Proof. Our proof relies on the following value difference lemma in Cai et al. [6].
Lemma B.1 (Value Difference Lemma). Let  $\pi = \{\pi_h\}_{h\in [H]}$  and  $\pi^{\prime} = \{\pi_h^{\prime}\}_{h\in [H]}$  be two policies and  $\bar{Q} = \{\bar{Q}_h:\mathcal{S}\times \mathcal{A}\mapsto \mathbb{R}\}_{h\in [H]}$  be any Q-functions. Moreover, for any  $h\in [H]$ , we define value function  $\bar{V}_h:S\mapsto \mathbb{R}$  by letting  $\bar{V}_h(x) = \langle \bar{Q}_h(x,\cdot),\pi_h(\cdot \mid x)\rangle$ . Then for any  $k\in [K]$  we have
$$
\begin{array}{l} \bar {V} _ {1} (x _ {1}) - V _ {1} ^ {\pi^ {\prime}, k} (x _ {1}) = \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {\prime}} [ \langle \bar {Q} _ {h} (x _ {h}, \cdot), \pi_ {h} (\cdot | x _ {h}) - \pi_ {h} ^ {\prime} (\cdot | x _ {h}) \rangle ] \\ + \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {\prime}} \left[ \bar {Q} _ {h} \left(x _ {h}, a _ {h}\right) - \left(r _ {h} ^ {k} + \mathbb {P} _ {h} \bar {V} _ {h + 1}\right) \left(x _ {h}, a _ {h}\right) \right]. \\ \end{array}
$$
Proof. See §B.1 in Cai et al. [6] for a detailed proof.
Back to our proof, for any  $k \in [K]$ , we have
$$
V _ {1} ^ {\pi^ {*}, k} \left(x _ {1}\right) - V _ {1} ^ {\pi^ {k}, k} \left(x _ {1}\right) = \underbrace {V _ {1} ^ {\pi^ {*} , k} \left(x _ {1}\right) - V _ {1} ^ {k} \left(x _ {1}\right)} _ {\text {(i)}} + \underbrace {V _ {1} ^ {k} \left(x _ {1}\right) - V _ {1} ^ {\pi^ {k} , k} \left(x _ {1}\right)} _ {\text {(i i)}}. \tag {B.1}
$$
Applying Lemma B.1 with  $\pi = \pi^k$ ,  $\pi' = \pi^*$ , and  $\bar{Q} = \{Q_h^k\}_{h \in [H]}$ , we have
$$
\left. \right.\left(\mathrm {i}\right) = \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {*}} \left[\left\langle Q _ {h} ^ {k} \left(x _ {h}, \cdot\right), \pi_ {h} ^ {*} (\cdot \mid x _ {h}) - \pi_ {h} ^ {k} (\cdot \mid x _ {h}) \right\rangle\right] + \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {*}} \left[ \delta_ {h} ^ {k} \left(x _ {h}, a _ {h}\right)\right], \quad \tag {B.2}
$$
where  $\delta_h^k = r_h^k + Q_h^k - \mathbb{P}_h V_{h+1}^k$  is the Bellman error defined in (4.1). Similarly, applying Lemma B.1 with  $\pi = \pi' = \pi^k$  and  $\bar{Q} = \{Q_h^k\}_{h \in [H]}$ , we obtain
$$
\left(\mathrm {i i}\right) = - \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {k}} \left[ \delta_ {h} ^ {k} \left(x _ {h}, a _ {h}\right) \right]. \tag {B.3}
$$
Plugging (B.2) and (B.3) into (B.1) and then taking summation across  $k \in [K]$ , we have
$$
\begin{array}{l} \operatorname {R e g r e t} (K) = \sum_ {k = 1} ^ {K} \left(V _ {1} ^ {\pi^ {*}, k} \left(x _ {1} ^ {k}\right) - V _ {1} ^ {\pi^ {k}, k} \left(x _ {1} ^ {k}\right)\right) \\ = \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {*}} \left[ \left\langle Q _ {h} ^ {k} \left(x _ {h}, \cdot\right), \pi_ {h} ^ {*} (\cdot | x _ {h}) - \pi_ {h} ^ {k} (\cdot | x _ {h}) \right\rangle \right] \\ + \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left(\mathbb {E} _ {\pi^ {*}} \left[ \delta_ {h} ^ {k} \left(x _ {h}, a _ {h}\right) \right] - \mathbb {E} _ {\pi^ {k}} \left[ \delta_ {h} ^ {k} \left(x _ {h}, a _ {h}\right) \right]\right), \\ \end{array}
$$
which concludes the proof of Lemma 4.1.
B.2 Proof of Lemma 4.2
Proof. Recall that  $B \in \mathbb{N}^{+}$  is the batch size,  $k_{i} = (i - 1) \cdot B + 1$  and  $L = K / B$ . Fix  $h \in [H]$ . By the multi-batched updating rule, we have
$$
\sum_ {k = 1} ^ {K} \mathbb {E} _ {\pi^ {*}} \left[ \left\langle Q _ {h} ^ {k} \left(x _ {h}, \cdot\right), \pi_ {h} ^ {*} (\cdot \mid x _ {h}) - \pi_ {h} ^ {k} (\cdot \mid x _ {h}) \right\rangle \right] = B \sum_ {i = 1} ^ {L} \mathbb {E} _ {\pi^ {*}} \left[ \left\langle Q _ {h} ^ {k _ {i}} \left(x _ {h}, \cdot\right), \pi_ {h} ^ {*} (\cdot \mid x _ {h}) - \pi_ {h} ^ {k _ {i}} (\cdot \mid x _ {h}) \right\rangle \right]. \tag {B.4}
$$
To derive the upper bound of (B.4), we need the following lemma.
Lemma B.2. For any  $(i,h,x_h)\in [L]\times [H]\times S$ , it holds that