SlowGuess/NeurIPS_2023_OCR huggingface dataset:
which improves over the above purely on-policy version by a factor of  $H / \epsilon$ . In Section 4.2, we will specialize this result to linear MDPs to derive a sample complexity that improves the best existing one in policy optimization [Zanette et al., 2021] by a factor of  $d$ .
Finally we remark that for cleaner presentation, we state Theorem 1 for a fixed, constant failure probability. However, the result is easy to extend to the case when the failure probability is some arbitrary value of  $\delta \in (0,1)$  at the expense of increasing the sample complexity by a factor of  $\log(1/\delta)$ : one simply need to run Algorithm 1 for  $\log(1/\delta)$  times, estimate the values of every output policy to accuracy  $\epsilon$ , and then pick the one with the highest estimate.
Subroutine 1 Tabular OPE  $(\pi, \mathcal{D})$
1: required parameter:  $\alpha$   
2: split  $\mathcal{D}$  evenly into  $H$  disjoint subsets  $\mathcal{D}_1, \ldots, \mathcal{D}_H$   
3: compute empirical estimate  $(\hat{\mathbb{P}}_h, \hat{R}_h)$  of the transition and reward at step  $h$  by using  $\mathcal{D}_h$   
4: set  $\overline{V}_{H+1}(s) \gets 0$  for all  $s \in S$   
5: for  $h = H:1$  do  
6: for all  $(s,a) \in S \times \mathcal{A}$ , compute  $J_h(s,a) \gets \sum_{(s_h,a_h) \in \mathcal{D}_h} \mathbf{1}((s_h,a_h) = (s,a))$  and define  $\left\{ \begin{array}{l} \overline{Q}_h(s,a) = \min \left\{ H - h + 1, \mathbb{E}_{s' \sim \hat{\mathbb{P}}_h(s,a)}[\overline{V}_{h+1}(s')] + \hat{R}_h(s,a) + b_h(s,a) \right\} \\ \overline{V}_h(s) = \mathbb{E}_{a \sim \pi_h(\cdot|s)}[\overline{Q}_h(s,a)] \end{array} \right.$  with bonus function  $b_h(s,a) = \alpha(J_h(s,a) + 1)^{-1/2}$   
7: output  $\{\overline{Q}_h\}_{h=1}^H$
In this section, we implement subroutine OPE for tabular MDPs, linear MDPs and RL problems of low eluder dimension, and derive the respective sample complexity guarantees of OPTIMISTIC NPG.
4.1 Tabular MDPs
The implementation of tabular OPE (Subroutine 1) is rather standard: first estimate the transition and reward from dataset  $\mathcal{D}$ , then plug the estimated transition-reward into the Bellman equation under policy  $\pi$  to compute the Q-value estimate backwards. And to guarantee optimism, we additionally add standard counter-based UCB bonus to compensate the error in model estimation. Formally, we have the following guarantee for tabular OPE.
Proposition 1 (tabular MDPs). Suppose we choose  $\alpha = \Theta\left(H\sqrt{\log(KH|\mathcal{S}||\mathcal{A}|)}\right)$  in Subroutine 1, then Condition 1 holds with  $L = SAH^3$ .
By combining Proposition 1 with Theorem 1, we prove that OPTIMISTIC NPG with Subroutine 1 learns an  $\epsilon$ -optimal policy within  $(KN / m) = \tilde{\mathcal{O}}(H^6 |\mathcal{S}||\mathcal{A}| / \epsilon^3)$  episodes for any tabular MDP. This rate is strictly worse than the best existing result  $\tilde{\mathcal{O}}(H^3 |\mathcal{S}||\mathcal{A}| / \epsilon^2)$  in tabular policy optimization [Wu et al., 2022]. Nonetheless, the proof techniques in [Wu et al., 2022] are specially tailored to tabular MDPs and it is highly unclear how to generalize them to linear MDPs due to certain technical difficulty that arises from covering a prohibitively large policy space. In comparison, our policy optimization framework easily extends to linear MDPs and provides improved sample complexity over the best existing one, as is shown in the following section.
4.2 Linear MDPs
We provide the instantiation of OPE for linear MDPs in Subroutine 2. At a high level, linear OPE computes an upper bound  $\overline{Q}$  for  $Q^{\pi}$  by using the Bellman equation under policy  $\pi$  backwards from step  $H$  to step 1 while adding a bonus to compensate the uncertainty of parameter estimation. Specifically, the step of ridge regression (Line 5) utilizes the linear completeness property of linear MDPs in computing  $\overline{Q}_h$  from  $\overline{V}_{h + 1}$ , which states that for any function  $\overline{V}_{h + 1}:S\to \mathbb{R}$ , there exists  $\overline{\theta}_h\in \mathbb{R}^d$  so that  $\langle \phi_h(s,a),\overline{\theta}_h\rangle = R_h(s,a) + \mathbb{E}_{s'\sim \mathbb{P}_h(\cdot |s,a)}[\overline{V}_{h + 1}(s')]$  for all  $(s,a)\in S\times \mathcal{A}$ . And in Line 6, we add an elliptical bonus  $b_{h}(s,a)$  to compensate the error between our estimate  $\hat{\theta}_h$  and the groundtruth  $\bar{\theta}_h$  so that we can guarantee  $\overline{Q}_h(s,a)\geq (\mathcal{T}_h^\pi \overline{Q}_{h + 1})(s,a)$  for all  $(s,a,h)$  with high probability.
$$
\hat {\theta} _ {h} = \operatorname {a r g m i n} _ {\theta} \sum_ {\left(s _ {h}, a _ {h}, r _ {h}, s _ {h + 1}\right) \in \mathcal {D} _ {h}} \left(\phi_ {h} \left(s _ {h}, a _ {h}\right) ^ {\mathrm {T}} \theta - r _ {h} - \bar {V} _ {h + 1} \left(s _ {h + 1}\right)\right) ^ {2} + \lambda \| \theta \| _ {2} ^ {2}. \tag {1}
$$
Subroutine 2 Linear OPE  $(\pi, \mathcal{D})$
1: required parameter:  $\alpha, \lambda$
2: split  $\mathcal{D}$  evenly into  $H$  disjoint subsets  $\mathcal{D}_1, \ldots, \mathcal{D}_H$
3: set  $\overline{V}_{H + 1}(s)\gets 0$  for all  $s\in S$
4: for  $h = H : 1$  do
5: perform ridge regression according to Equation (1)
6: compute covariance matrix
$$
\Sigma_ {h} = \sum_ {\left(s _ {h}, a _ {h}\right) \in \mathcal {D} _ {h}} \phi_ {h} \left(s _ {h}, a _ {h}\right) \phi_ {h} \left(s _ {h}, a _ {h}\right) ^ {\mathrm {T}}
$$
$$
\left\{ \begin{array}{l} \overline {{Q}} _ {h} (s, a) = \operatorname {T r u n c a t e} _ {[ 0, H ]} \big (\langle \hat {\theta} _ {h}, \phi_ {h} (s, a) \rangle + b _ {h} (s, a) \big) \\ \overline {{V}} _ {h} (s) = \mathbb {E} _ {a \sim \pi_ {h} (\cdot | s)} \big [ \overline {{Q}} _ {h} (s, a) \big ] \end{array} \right.
$$
with bonus function
$$
b _ {h} (s, a) := \alpha \times \| \phi_ {h} (s, a) \| _ {\left(\Sigma_ {h} + \lambda \mathbf {I} _ {d \times d}\right) ^ {- 1}}
$$
7: output  $\{\overline{Q}_h\}_{h = 1}^H$
Subroutine 3 General OPE  $(\pi ,\mathcal{D})$
1: required parameter:  $\beta$
2: split  $\mathcal{D}$  evenly into  $H$  disjoint subsets  $\mathcal{D}_1, \ldots, \mathcal{D}_H$
3: set  $\overline{V}_{H + 1}(s)\gets 0$  for all  $s\in S$
4: for  $h = H : 1$  do
5: for  $(s,a)\in S\times \mathcal{A}$  do
6: construct confidence set
$$
\mathcal {B} _ {h} = \left\{f _ {h} \in \mathcal {F} _ {h}: \mathcal {L} _ {h} \left(\mathcal {D} _ {h}, f _ {h}\right) \leq \min  _ {g _ {h} \in \mathcal {F} _ {h}} \mathcal {L} _ {h} \left(\mathcal {D} _ {h}, g _ {h}\right) + \beta \right\}
$$
with loss function defined as
$$
\mathcal{L}_{h}(\mathcal{D}_{h},\zeta) = \sum_{(s_{h},a_{h},r_{h},s_{h + 1})\in \mathcal{D}_{h}}\bigl(\zeta (s_{h},a_{h}) - r_{h} - \overline{V}_{h + 1}(s_{h + 1})\bigr)^{2}
$$
7: compute
$$
\left\{ \begin{array}{l} \overline {{Q}} _ {h} (s, a) = \sup  _ {f _ {h} \in \mathcal {B} _ {h}} f _ {h} (s, a) \\ \overline {{V}} _ {h} (s) = \mathbb {E} _ {a \sim \pi_ {h} (\cdot | s)} \left[ \overline {{Q}} _ {h} (s, a) \right] \end{array} \right.
$$
8: output  $\{\overline{Q}_h\}_{h=1}^H$
Proposition 2 (linear MDPs). Suppose we choose  $\lambda = 1$  and  $\alpha = \Theta\left(H\sqrt{d\log(KN)}\right)$  in Subroutine 2, then Condition 1 holds with  $L = d^{2}H^{3}$ .
By combining Proposition 2 with Theorem 1, we obtain that  $(KN / m) = \tilde{\mathcal{O}} (d^{2}H^{6} / \epsilon^{3})$  episodes are sufficient for OPTIMISTIC NPG to learn an  $\epsilon$ -optimal policy, improving upon the state-of-the-art policy optimization results [Zanette et al., 2021] by a factor of  $d$ . Notably, this is also the first computationally efficient algorithm to achieve optimal quadratic dimension dependence for learning linear MDPs. The key factor behind this improvement is OPTIMISTIC NPG's periodic collection of fresh on-policy data, which eliminates the undesired correlation and avoids the union bound over certain nonlinear function class that are commonly observed in previous works. Consequently, OPTIMISTIC NPG uses a bonus function that is  $\sqrt{d}$  times smaller than in previous works [e.g., Jin et al., 2020]. For further details on how we achieve this improvement, we refer interested readers to Appendix A, specifically Lemma 2.
4.3 General function approximation
Now we instantiate OPE for RL with general function approximation. In this setting, the learner is provided with a function class  $\mathcal{F} = \mathcal{F}_1 \times \dots \times \mathcal{F}_H$  for approximating the Q values of polices, where  $\mathcal{F}_h \subseteq (\mathcal{S} \times \mathcal{A} \to [0, H])$ .
General OPE. The pseudocode is provided in Subroutine 3. At each step  $h \in [H]$ , general OPE first constructs a confidence set  $\mathcal{B}_h$  which contains candidate estimates for  $\mathcal{T}_h^\pi \overline{Q}_{h+1}$  (Line 6). Specifically,  $\mathcal{B}_h$  consists of all the value candidates  $f_h \in \mathcal{F}_h$  whose square temporal difference (TD) error on dataset  $\mathcal{D}_h$  is no larger than the smallest one by an additive factor  $\beta$ . Such construction can be viewed as a relaxation of the classic fitted Q-iteration (FQI) algorithm which only keeps the value candidate with the smallest TD error. In particular, if we pick  $\beta = 0$ ,  $\mathcal{B}_h$  collapses to the solution of FQI. Equipped with confidence set  $\mathcal{B}_h$ , general OPE then perform optimistic planning at every  $(s, a) \in S \times \mathcal{A}$  by computing  $\overline{Q}_h(s, a)$  to be the largest possible value that can be achieved by any candidate in confidence set  $\mathcal{B}_h$  (Line 7). We remark that general OPE shares a similar algorithmic spirit to the GOLF algorithm [Jin et al., 2021]. The key difference is that GOLF is a purely value-based algorithm and only performs optimistic planning at the initial state while general OPE performs optimistic planning at every state and is part of a policy gradient algorithm.
Theoretical guarantee. To state the main theorem, we need to first introduce two standard concepts: value closeness and eluder dimension, which have been widely used in previous works that study RL with general function approximation.
Assumption 1 (value closeness [Wang et al., 2020]). For all  $h \in [H]$  and  $V_{h+1} : S \to [0, H]$ ,  $\mathcal{T}_h V_{h+1} \in \mathcal{F}_h$  where  $[\mathcal{T}_h V_{h+1}](s, a) = R_h(s, a) + \mathbb{E}[V_{h+1}(s') \mid s' \sim \mathbb{P}_h(\cdot \mid s, a)]$ .
Intuitively, Assumption 1 can be understood as requiring that the application of the Bellman operator  $\mathcal{T}_h$  to any V-value function  $V_{h + 1}$  results in a function that belongs to the value function class  $\mathcal{F}_h$ . This assumption holds in various settings, including tabular MDPs and linear MDPs. Furthermore, it is reasonable to assume that value closeness holds whenever the function class  $\mathcal{F}$  has sufficient expressive power, such as the class of neural networks.
Definition 1 (eluder dimension [Russo and Van Roy, 2013]). Let  $\mathcal{G}$  be a function class from  $\mathcal{X}$  to  $\mathbb{R}$  and  $\epsilon > 0$ . We define the  $\epsilon$ -eluder dimension of  $\mathcal{G}$ , denoted as  $d_E(\mathcal{G}, \epsilon)$ , to be the largest  $L \in \mathbb{N}^+$  such that there exists  $x_1, \ldots, x_L \in \mathcal{X}$  and  $g_1, g_1', \ldots, g_L, g_L' \in \mathcal{G}$  satisfying: for all  $l \in [L]$ ,  $\sum_{i < l} (g_l(x_i) - g_l'(x_i))^2 \leq \epsilon$  but  $g_l(x_l) - g_l'(x_l) \geq \epsilon$ .
At a high level, eluder dimension  $d_{E}(\mathcal{G},\epsilon)$  measures how many mistakes we have to make in order to identify an unknown function from function class  $\mathcal{G}$  to accuracy  $\epsilon$ , in the worst case. It has been widely used as a sufficient condition for proving sample-efficiency guarantee for optimistic algorithms in RL with general function approximation, e.g., Wang et al. [2020], Jin et al. [2021].
Now we state the theoretical guarantee for general OPE.
Proposition 3 (general function approximation). Suppose Assumption 1 holds and we choose  $\beta = \Theta \left(H^{2}\log (|\mathcal{F}|NK / \delta)\right)$  in Subroutine 3, then Condition 1 holds with  $L = H^3\log (|\mathcal{F}|)\max_h d_E(\mathcal{F}_h,1 / N)$ .
Plugging Proposition 3 back into Theorem 1, we obtain that  $(KN / m) = \tilde{\mathcal{O}} (d_{E}\log (|\mathcal{F}|)H^{6} / \epsilon^{3})$  episodes are sufficient for OPTIMISTIC NPG to learn an  $\epsilon$ -optimal policy, which to our knowledge is the first polynomial sample complexity guarantee for policy optimization with general function approximation.
We proposed a model-free, policy optimization algorithm—OPTIMISTIC NPG for online RL and analyzed its behavior in the episodic setting. In terms of algorithmic design, it is not only considerably simpler but also more closely resembles the empirical policy optimization algorithms (e.g., PPO, TRPO) than the previous theoretical algorithms. In terms of sample efficiency, for  $d$ -dimensional linear MDPs, it improves over state-of-the-art policy optimization algorithm by a factor of  $d$ , and is the first computationally efficient algorithm to achieve the optimal dimension dependence. To our best knowledge, OPTIMISTIC NPG is also the first sample-efficient policy optimization algorithm under general function approximation.
For future research, we believe that it is an important direction to investigate the optimal complexity for using policy optimization algorithms to solve linear MDPs. Despite the optimal dependence on dimension  $d$ , the current sample complexity of OPTIMISTIC NPG is worse than the optimal rate by a factor of  $1/\epsilon$ . It remains unclear to us how to shave off this  $1/\epsilon$  factor to achieve the optimal rate.
Qinghua Liu would like to thank Yu Bai for valuable discussions. This work was partially supported by National Science Foundation Grant NSF-IIS-2107304.
Yasin Abbasi-Yadkori, David Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances in neural information processing systems, 24, 2011.
Yasin Abbasi-Yadkori, Peter Bartlett, Kush Bhatia, Nevena Lazic, Csaba Szepesvari, and Gellert Weisz. Politex: Regret bounds for policy iteration using expert prediction. In International Conference on Machine Learning, pages 3692-3702. PMLR, 2019.
Alekh Agarwal, Mikael Henaff, Sham Kakade, and Wen Sun. Pc-pg: Policy cover directed exploration for provable policy gradient learning. Advances in neural information processing systems, 33: 13399-13412, 2020.
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. J. Mach. Learn. Res., 22(98): 1-76, 2021.
Alekh Agarwal, Yujia Jin, and Tong Zhang. VOqL: Towards optimal regret in model-free RL with nonlinear function approximation. arXiv preprint arXiv:2212.06069, 2022.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Christopher Hesse, Rafal Jozefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning. arXiv preprint arXiv:1912.06680, 2019.
Jalaj Bhandari and Daniel Russo. Global optimality guarantees for policy gradient methods. arXiv preprint arXiv:1906.01786, 2019.
Qi Cai, Zhuoran Yang, Chi Jin, and Zhaoran Wang. Provably efficient exploration in policy optimization. In International Conference on Machine Learning, pages 1283-1294. PMLR, 2020.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pages 49-58. PMLR, 2016.
Jiafan He, Heyang Zhao, Dongruo Zhou, and Quanquan Gu. Nearly minimax optimal reinforcement learning for linear Markov decision processes. arXiv preprint arXiv:2212.06132, 2022.
Chi Jin, Zhuoran Yang, Zhaoran Wang, and Michael I Jordan. Provably efficient reinforcement learning with linear function approximation. In Conference on Learning Theory, pages 2137-2143. PMLR, 2020.
Chi Jin, Qinghua Liu, and Sobhan Miryosefi. Bellman eluder dimension: New rich classes of RL problems, and sample-efficient algorithms. Advances in neural information processing systems, 34:13406-13418, 2021.
Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001.
Boyi Liu, Qi Cai, Zhuoran Yang, and Zhaoran Wang. Neural proximal/trust region policy optimization attains globally optimal policy. arXiv preprint arXiv:1906.10306, 2019.
Gergely Neu, Anders Jonsson, and Vicenç Gómez. A unified view of entropy-regularized Markov decision processes. arXiv preprint arXiv:1705.07798, 2017.
OpenAI. ChatGPT: Optimizing language models for dialogue, 2022. URL https://openai.com/blog/chatgpt/.
Daniel Russo and Benjamin Van Roy. Eluder dimension and the sample complexity of optimistic exploration. Advances in Neural Information Processing Systems, 26, 2013.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Lior Shani, Yonathan Efroni, Aviv Rosenberg, and Shie Mannor. Optimistic policy optimization with bandit feedback. In International Conference on Machine Learning, pages 8604-8613. PMLR, 2020.
Andrew J. Wagenmaker, Yifang Chen, Max Simchowitz, Simon Du, and Kevin Jamieson. Reward-free RL is no harder than reward-aware RL in linear Markov decision processes. In International Conference on Machine Learning, pages 22430-22456. PMLR, 2022.
Ruosong Wang, Russ R Salakhutdinov, and Lin Yang. Reinforcement learning with general value function approximation: Provably efficient approach via bounded eluder dimension. Advances in Neural Information Processing Systems, 33:6123-6135, 2020.
Tianhao Wu, Yunchang Yang, Han Zhong, Liwei Wang, Simon Du, and Jiantao Jiao. Nearly optimal policy optimization with stable at any time guarantee. In International Conference on Machine Learning, pages 24243-24265. PMLR, 2022.
Andrea Zanette, Alessandro Lazaric, Mykel Kochenderfer, and Emma Brunskill. Learning near optimal policies with low inherent Bellman error. In International Conference on Machine Learning, pages 10978-10989. PMLR, 2020.
Andrea Zanette, Ching-An Cheng, and Alekh Agarwal. Cautiously optimistic policy optimization and exploration with linear function approximation. In Conference on Learning Theory, pages 4473-4525. PMLR, 2021.
Dongruo Zhou, Quanquan Gu, and Csaba Szepesvari. Nearly minimax optimal reinforcement learning for linear mixture Markov decision processes. In Conference on Learning Theory, pages 4532-4576. PMLR, 2021.
Recall in Algorithm 1,  $\pi^{\mathrm{out}}$  is sampled uniformly at random from  $\{\pi^k\}_{k\in [K]}$ , so we have
$$
\begin{array}{l} \mathbb {E} \left[ V _ {1} ^ {\star} (s _ {1}) - V _ {1} ^ {\pi^ {\mathrm {o u t}}} (s _ {1}) \right] \\ = \frac {1}{K} \sum_ {k = 1} ^ {K} \left[ V _ {1} ^ {\star} \left(s _ {1}\right) - V _ {1} ^ {\pi^ {k}} \left(s _ {1}\right) \right] \\ = \underbrace {\frac {1}{K} \sum_ {k = 1} ^ {K} \left[ V _ {1} ^ {\star} (s _ {1}) - \overline {{V}} _ {1} ^ {k} (s _ {1}) \right]} _ {\text {T e r m (I)}} + \underbrace {\frac {1}{K} \sum_ {k = 1} ^ {K} \left[ \overline {{V}} _ {1} ^ {k} (s _ {1}) - V _ {1} ^ {\pi^ {k}} (s _ {1}) \right]} _ {\text {T e r m (I I)}}. \\ \end{array}
$$
To control Term (I), we import the following generalized policy difference lemma from [Shani et al., 2020, Cai et al., 2020].
Lemma 1 (generalized policy-difference lemma). For any policy  $\pi$  and  $k\in [K]$
$$
\begin{array}{l} V _ {1} ^ {\pi} (s _ {1}) - \overline {{V}} _ {1} ^ {k} (s _ {1}) \\ = \sum_ {h = 1} ^ {H} \mathbb {E} _ {s _ {h} \sim \pi} \left[ \left\langle \pi_ {h} (\cdot \mid s _ {h}) - \pi_ {h} ^ {k} (\cdot \mid s _ {h}), \bar {Q} _ {h} ^ {k} (s _ {h}, \cdot) \right\rangle \right] \\ - \sum_ {h = 1} ^ {H} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \pi} \left[ \overline {{Q}} _ {h} ^ {k} \left(s _ {h}, a _ {h}\right) - \left(\mathcal {T} _ {h} ^ {\pi^ {k}} \overline {{Q}} _ {h + 1} ^ {k}\right) \left(s _ {h}, a _ {h}\right) \right]. \\ \end{array}
$$
By invoking Lemma 1 with  $\pi = \pi^{\star}$ , we have
$$
\begin{array}{l} \operatorname {T e r m} (\mathrm {I}) = \frac {1}{K} \sum_ {k = 1} ^ {K} \left[ V _ {1} ^ {\star} (s _ {1}) - \overline {{V}} _ {1} ^ {k} (s _ {1}) \right] \\ \leq H \max  _ {h, s} \left[ \frac {1}{K} \sum_ {k = 1} ^ {K} \left\langle \pi_ {h} ^ {\star} (\cdot | s) - \pi_ {h} ^ {k} (\cdot | s), \bar {Q} _ {h} ^ {k} (s, \cdot) \right\rangle \right] + H \max  _ {h, s, a} \left[ \left(\mathcal {T} _ {h} ^ {\pi^ {k}} \bar {Q} _ {h + 1} ^ {k} - \bar {Q} _ {h} ^ {k}\right) (s, a) \right] \tag {2} \\ \leq \mathcal {O} \left(\frac {H \log | \mathcal {A} |}{\eta K} + \eta H ^ {3}\right) + H \max  _ {h, s, a} \left[ \left(\mathcal {T} _ {h} ^ {\pi^ {k}} \overline {{Q}} _ {h + 1} ^ {k} - \overline {{Q}} _ {h} ^ {k}\right) (s, a) \right], \\ \end{array}
$$
where the second inequality follows from the standard regret bound of online mirror ascent. By Condition (1A), the second term on the RHS of Equation (2) is upper bounded by zero. So we have
$$
\mathrm {T e r m} (\mathrm {I}) \leq \mathcal {O} \left(\frac {H \log | \mathcal {A} |}{\eta K} + \eta H ^ {3}\right).
$$
As for Term (II), we can directly upper bound it by Condition (1B) which gives:
$$
\operatorname {T e r m} (\Pi) \leq H \sqrt {(L / N) \times \log (N K L / \delta)}.
$$
We complete the proof by plugging in the choice of  $K, N, \eta$ .
B Proofs for Examples
In this section, we prove the three instantiations of OPE in Section 4 satisfy Condition 1 with mild complexity measure  $L$ . The proofs of the lemmas used in this section can be found in Appendix C.
B.1 Proof of Proposition 2 (linear OPE)
We start with the following lemma showing that  $\overline{Q}_h^k$  is an optimistic estimate of  $\mathcal{T}_h^{\pi^k}\overline{Q}_{h + 1}^k$  and the degree of over-optimism can be bounded by the bonus function.
Lemma 2 (optimism of value estimates). Under the same choice of  $\lambda$  and  $\alpha$  as in Proposition 2, with probability at least  $1 - \delta$ : for all  $(k,h,s,a) \in [K] \times [H] \times \mathcal{S} \times \mathcal{A}$ :
$$
0 \leq \left(\overline {{Q}} _ {h} ^ {k} - \mathcal {T} _ {h} ^ {\pi^ {k}} \overline {{Q}} _ {h + 1} ^ {k}\right) (s, a) \leq 2 b _ {h} ^ {k} (s, a),
$$
where  $\{b_h^k\}_{h\in [H]}$  are the bonus functions defined in Line 6, Subroutine 2 linear OPE  $(\pi^k,\mathcal{D}^k)$ .
Importantly, Lemma 2 holds with a bonus function that is smaller by a factor of  $\sqrt{d}$  than the typical one used in linear function approximation [e.g., Jin et al., 2020], which is key to obtaining the optimal dimension dependence. The main reason that this smaller bonus function suffices is that we recollect fresh samples every  $m$  iterations and split the data (Line 2 in Subroutine 2) to eliminate the correlation between different steps  $h \in [H]$ .
Lemma 2 immediately implies that Condition (1A) holds. Below we prove Condition (1B).
By the second inequality in Lemma 2, the definition of  $\overline{V}^k$  and the Bellman equation under policy  $\pi^k$ , one can show
$$
\overline {{V}} _ {1} ^ {k} \left(s _ {1}\right) - V _ {1} ^ {\pi^ {k}} \left(s _ {1}\right) = \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {k}} \left[ \left(\bar {Q} _ {h} ^ {k} - \mathcal {T} _ {h} ^ {\pi^ {k}} \bar {Q} _ {h + 1} ^ {k}\right) \left(s _ {h}, a _ {h}\right) \right] \leq 2 \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {k}} \left[ b _ {h} ^ {k} \left(s _ {h}, a _ {h}\right) \right]. \tag {3}
$$
Denote by  $t_k$  the index of the last iteration of collecting fresh data before the  $k^{\text{th}}$  iteration. By exploiting the softmax parameterization structure of  $\{\pi^k\}_{k \in [K]}$ , one can show that the visitation measure over state-action pairs induced by executing policy  $\pi^k$  is actually very close to the one induced by  $\pi^{t_k}$  in the following sense.