SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qinghua Liu*
gellert@deepmind.com
agyoryg@deepmind.com
While policy optimization algorithms have played an important role in recent empirical success of Reinforcement Learning (RL), the existing theoretical understanding of policy optimization remains rather limited—they are either restricted to tabular MDPs or suffer from highly suboptimal sample complexity, especially in online RL where exploration is necessary. This paper proposes a simple efficient policy optimization framework—OPTIMISTIC NPG for online RL. OPTIMISTIC NPG can be viewed as a simple combination of the classic natural policy gradient (NPG) algorithm [Kakade, 2001] and an optimistic policy evaluation subroutine to encourage exploration. For  $d$ -dimensional linear MDPs, OPTIMISTIC NPG is computationally efficient, and learns an  $\epsilon$ -optimal policy within  $\tilde{\mathcal{O}}(d^2/\epsilon^3)$  samples, which is the first computationally efficient algorithm whose sample complexity has the optimal dimension dependence  $\tilde{\Theta}(d^2)$ . It also improves over state-of-the-art results of policy optimization algorithms [Zanette et al., 2021] by a factor of  $d$ . For general function approximation that subsumes linear MDPs, OPTIMISTIC NPG, to our best knowledge, is also the first policy optimization algorithm that achieves the polynomial sample complexity for learning near-optimal policies.
Policy optimization algorithms [Schulman et al., 2017, 2015] with neural network function approximation have played an important role in recent empirical success of reinforcement learning (RL), such as robotics [Finn et al., 2016], games [Berner et al., 2019] and large language models [OpenAI, 2022]. Motivated by the empirical success, the theory community made a large effort to design provably efficient policy optimization algorithms that work in the presence of linear function approximation [Agarwal et al., 2021, Bhandari and Russo, 2019, Liu et al., 2019, Neu et al., 2017, Abbasi-Yadkori et al., 2019, Agarwal et al., 2020, Zanette et al., 2021, Shani et al., 2020, Cai et al., 2020]. Early works focused on proving that policy optimization algorithms are capable to learn near-optimal policies using a polynomial number of samples under certain reachability (coverage) assumptions. [e.g., Agarwal et al., 2021, Bhandari and Russo, 2019, Liu et al., 2019, Neu et al., 2017, Abbasi-Yadkori et al., 2019]. While this was good for laying down the foundations for future work, the reachability assumptions basically imply that the state space is already well-explored or rather easy to explore, which avoids the challenge of performing strategic exploration — one of the central problems in both empirical and theoretical RL.
*This work was done when QL interned at DeepMind.
Table 1: A comparison of sample-complexity results for linear MDPs.
Algorithms	Sample Complexity	Computationally Efficient	Policy Optimization
Zanette et al. [2020]	Θ(d2/ε2)	×	×
Jin et al. [2021]
He et al. [2022]	O(d2/ε2+d≥4/ε)	✓	×
Agarwal et al. [2022]
Wagenmaker et al. [2022]
Agarwal et al. [2020]	O(poly(d)/ε11)	✓	✓
Zanette et al. [2021]	O(d3/ε3)	✓	✓
Optimistic NPG (this work)	O(d2/ε3)	✓	✓
To address this limitation, later works [Agarwal et al., 2020, Zanette et al., 2021] proposed policy optimization algorithms that enjoy polynomial sample-complexity guarantee without making any reachability assumption, but at the cost of either complicating the algorithm design (and analysis) with various tricks or getting highly suboptimal sample-complexity guarantees. For example, the PC-PG algorithm [Agarwal et al., 2020], which, to our knowledge was the first policy optimization algorithm for learning linear MDPs without reachability assumptions, requires  $\tilde{\mathcal{O}} (\mathrm{poly}(d) / \epsilon^{11})$  samples to learn an  $\epsilon$ -optimal policy for  $d$ -dimensional linear MDPs. That  $\tilde{\mathcal{O}} (1 / \epsilon^{11})$  samples were necessary for this task is highly unlikely. Indeed, Zanette et al. [2021] greatly improved this sample complexity to  $\tilde{\mathcal{O}} (d^3 /\epsilon^3)$  at the cost of considerably complicating the algorithm design and the analysis. Moreover, we are not aware of efficient guarantees of policy optimization for generic function approximation, which is rich enough to subsume linear MDPs. This motivates us to ask the following question:
Can we design a simple, general policy optimization algorithm with sharp sample-complexity guarantees in the exploration setting?
In particular, we aim to achieve sharper sample complexity than Zanette et al. [2021] in the linear MDP setting, and achieve low-degree polynomial sample complexity in the general function approximation setting. This paper answers the highlighted question affirmatively by making the following contributions:
- Sharper rate. We propose a computationally efficient policy optimization algorithm—OPTIMISTIC NPG with sample complexity
$$
\tilde {\mathcal {O}} (d ^ {2} / \epsilon^ {3})
$$
for learning an  $\epsilon$ -optimal policy in an online fashion while interacting with a  $d$ -dimensional linear MDP. This result improves over the best previous one [Zanette et al., 2021] in policy optimization by a factor of  $d$ . Moreover, to our knowledge, this is the first computationally efficient algorithm to achieve the optimal quadratic dimension dependence. Before moving on, we remark that previous FQI-style algorithms [Zanette et al., 2020, Jin et al., 2021] achieve the optimal sample complexity  $\tilde{\Theta}(d^2/\epsilon^2)$  [Zanette et al., 2020] but they are computationally inefficient due to the mechanism of global optimism in their algorithm design. Several very recent works [He et al., 2022, Agarwal et al., 2022, Wagenmaker et al., 2022] achieve the sample complexity  $\tilde{\mathcal{O}}(d^2/\epsilon^2 + d^{\geq 4}/\epsilon)$  using computationally efficient algorithms. Compared to our work, the aforementioned rate has worse dependence on  $d$  and better dependence on  $\epsilon$ . Nonetheless, our result is better in the practically important regime where the feature dimension  $d$  is typically very large and the target accuracy  $\epsilon$  is not too small. To summarize, the sample complexity of OPTIMISTIC NPG strictly improves over the best existing policy optimization algorithms and is not dominated by that of any existing computationally efficient algorithm.
<sup>2</sup>By "exploration setting", we mean a setup where there is no simulator nor reachability assumption and a learner can only influence the evolution of environmental states by taking actions.
To our best knowledge, this paper also achieves the first polynomial sample-complexity guarantee for policy optimization under general function approximation.
- Simple on-policy algorithm. At a high level, OPTIMISTIC NPG is almost an on-policy version of natural policy gradient (NPG) [Kakade, 2001] with Boltzmann policies that use linear function approximation for optimistic action-value estimates. By "on-policy", we mean the transition-reward data used to improve the current policy are obtained exactly by executing the current policy. However, the algorithm has a tuneable parameter (denoted by  $m$  in Algorithm 1) that allows it to reuse data sampled from earlier policies. By using such data, the algorithm becomes off-policy. The optimal choice of the tuneable parameter is dictated by theoretical arguments. Interestingly, our analysis shows that even the purely on-policy version of the algorithm (i.e., set  $m = 1$  in Algorithm 1) enjoys a well-controlled sample complexity of  $\tilde{\mathcal{O}}(d^2/\epsilon^4)$ . To the best of our knowledge, this is the first time an on-policy method is shown to have polynomial sample complexity in the exploration setting, given that all previous policy optimization or Q-learning style or model-based algorithms [e.g., Jin et al., 2020, Zanette et al., 2020, 2021, Agarwal et al., 2020, Cai et al., 2020, Shani et al., 2020, etc] are off-policy.
- New proof techniques. To achieve the improved rate, we introduce several new ideas in the proof, including but not limited to (a) exploiting the softmax parameterization of the policies to reuse data and improve sample efficiency (Lemma 3), (b) controlling on-policy uncertainty (Lemma 4) instead of cumulative uncertainty as in previous off-policy works, and (c) using a bonus term that is smaller than those in previous works by a factor of  $\sqrt{d}$  (see Lemma 2 and the discussions that follow).
Since this paper studies policy optimization algorithms in the setting of linear MDPs, below we restrict our focus to previous theoretical works on either policy optimization or linear MDPs.
Policy optimization. This work is inspired by and builds upon two recent works [Shani et al., 2020, Cai et al., 2020] that combine NPG with bonus-based optimism to handle exploration. In terms of algorithmic design, this paper (OPTIMISTIC NPG) utilizes on-policy fresh data for value estimation while Shani et al. [2020], Cai et al. [2020] are off-policy and reuse all the historical data. In terms of theoretical guarantees, Shani et al. [2020] and Cai et al. [2020] only study tabular MDPs and linear mixture MDPs [Zhou et al., 2021] respectively, while this paper considers the more challenging setting of linear MDPs [Jin et al., 2020] (in our view, linear MDPs are more challenging as there the number of model parameters scale with the number of states). We remark that due to some subtle technical challenge, so far it still remains unknown whether it is possible to generalize the analysis of Cai et al. [2020] to handle linear MDPs. Agarwal et al. [2020], Zanette et al. [2021] derive the first line of policy optimization results for RL in linear MDPs without any reachability- (or coverage-) style assumption. Compared to Agarwal et al. [2020], Zanette et al. [2021], our work considers the same setting but designs a simpler algorithm with cleaner analysis and sharper sample-complexity guarantee. Nevertheless, we remark that in certain model-misspecification settings, the algorithms of Agarwal et al. [2020], Zanette et al. [2021] can potentially achieve stronger guarantees. Since we focus on the well-specified setting (that is, the environment is perfectly modeled by a linear MDP), we refer interested readers to Agarwal et al. [2020] for more details. Finally, there have been a long line of works [e.g., Agarwal et al., 2021, Bhandari and Russo, 2019, Liu et al., 2019, Neu et al., 2017, Abbasi-Yadkori et al., 2019, etc] that study policy optimization under reachability (or coverage) assumptions, which eliminates the need for performing strategic exploration. Throughout this paper we do not make any such assumption and directly tackle the exploration challenge.
Linear MDPs. For linear MDPs, Jin et al. [2020] proposed a computationally efficient algorithm (LSVI-UCB) with  $\tilde{\mathcal{O}}(d^3/\epsilon^2)$  sample complexity. Later on, Zanette et al. [2020] utilized the idea of global optimism to obtain the optimal sample complexity  $\tilde{\Theta}(d^2/\epsilon^2)$  at the cost of sacrificing computational efficiency. Recently, He et al. [2022], Agarwal et al. [2022], Wagenmaker et al. [2022] designed new computationally efficient algorithms that can achieve  $\tilde{\mathcal{O}}(d^2/\epsilon^2 + d^{\geq 4}/\epsilon)$  sample complexity. Compared to the above works, the sample complexity of OPTIMISTIC NPG is not strictly worse than that of any known computationally efficient algorithm. In fact, it is the only computationally efficient method to achieve the optimal dimension dependence. Nonetheless, for
learning a near-optimal policy with a vanishing suboptimality  $\epsilon$ , the current sample complexity of OPTIMISTIC NPG is loose by a factor of  $\epsilon^{-1}$ .
MDP. We consider the model of episodic Markov Decision Processes (MDPs). Formally, an MDP is defined by a tuple  $(S, \mathcal{A}, \mathbb{P}, R, H)$  where  $S$  denotes the set of states,  $\mathcal{A}$  denotes the set of actions, both of which are assumed to be finite,  $\mathbb{P} = \{\mathbb{P}_h\}_{h \in [H]}$  denotes the set of transition probability functions so that  $\mathbb{P}_h(s' \mid s, a)$  is equal to the probability of transitioning to state  $s'$  given that action  $a$  is taken at state  $s$  and step  $h$ ,  $R = \{R_h\}_{h \in [H]}$  denotes the collection of expected reward functions so that  $R_h(s, a)$  is equal to the expected reward to be received if action  $a$  is taken at state  $s$  and step  $h$ , and  $H$  denotes the length of each episode. An agent interacts an MDP in the form of episodes. Formally, we assume without loss of generality that each episode always starts from a fixed initial state  $s_1$ . At the  $h^{\text{th}}$  step of this episode, the agent first observes the current state  $s_h$ , then takes action  $a_h$  and receives reward  $r_h(s_h, a_h)$  satisfying
$$
\mathbb {E} \left[ r _ {h} \left(s _ {h}, a _ {h}\right) \mid s _ {h}, a _ {h} \right] = R _ {h} \left(s _ {h}, a _ {h}\right).
$$
After that, the environment transitions to
$$
s _ {h + 1} \sim \mathbb {P} _ {h} (\cdot \mid s _ {h}, a _ {h}).
$$
The current episode terminates immediately once  $r_H$  is received. Throughout this paper, we assume  $\mathbb{P}$  and  $R$  are unknown to the learner.
Linear MDP. A  $d$ -dimensional linear MDP [Jin et al., 2020] is defined by two sets of feature mappings,  $\{\phi_h\}_{h\in [H]} \subseteq (\mathcal{S} \times \mathcal{A} \to \mathbb{R}^d)$  and  $\{\psi_h\}_{h\in [H]} \subseteq (\mathcal{S} \to \mathbb{R}^d)$ , and a set of vectors  $\{w_h^\star\}_{h\in [H]} \subseteq \mathbb{R}^d$  so that the transition probability functions can be represented as bilinear functions of feature mappings  $\{\phi_h\}_{h\in [H]}$  and  $\{\psi_h\}_{h\in [H]}$ , and the reward functions can be represented as linear functions of  $\{\phi_h\}_{h\in [H]}$ . Formally, we have that for all  $(s,a,s') \in S \times \mathcal{A} \times S$ :
$$
\mathbb {P} _ {h} \left(s ^ {\prime} \mid s, a\right) = \left\langle \phi_ {h} (s, a), \psi_ {h} \left(s ^ {\prime}\right) \right\rangle ,
$$
$$
R _ {h} (s, a) = \langle \phi_ {h} (s, a), w _ {h} ^ {\star} \rangle .
$$
For the purpose the regularity, linear MDPs also require that
$$
\max  _ {h, s, a} \| \phi_ {h} (s, a) \| _ {2} \leq 1, \quad \max  _ {h} \| w _ {h} ^ {\star} \| _ {2} \leq \sqrt {d},
$$
and for any function  $V_{h + 1}:\mathcal{S}\to [0,1]$
$$
\left\| \int_ {s \in \mathcal {S}} V _ {h + 1} (s) \psi_ {h + 1} (s) d s \right\| _ {2} \leq \sqrt {d}.
$$
Throughout this paper, we assume only  $\{\phi_h\}_{h\in [H]}$  is available to the learner while  $\{\psi_h\}_{h\in [H]}$  and  $\{w_{h}^{\star}\}_{h\in [H]}$  are not.
Policy and value. A (Markov) policy is a set of conditional probability functions  $\pi = \{\pi_h\}_{h\in [H]}$  so that  $\pi_h(\cdot |s)\in \Delta_{\mathcal{A}}$  gives the distribution over the action set conditioned on the current state  $s$  at step  $h$ . We define the V-value functions of policy  $\pi$  by  $\{V_h^\pi\}_{h\in [H]}\subseteq (\mathcal{S}\to \mathbb{R})$  so that  $V_{h}^{\pi}(s)$  is equal to the expected cumulative reward an agent will receive if she follows policy  $\pi$  starting from state  $s$  and step  $h$ . Formally,
$$
V _ {h} ^ {\pi} (s) = \mathbb {E} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {h ^ {\prime}} \left(s _ {h ^ {\prime}}, a _ {h ^ {\prime}}\right) \mid s _ {h} = s, a _ {h ^ {\prime}} \sim \pi_ {h ^ {\prime}} \left(s _ {h ^ {\prime}}\right), s _ {h + 1} ^ {\prime} \sim \mathbb {P} _ {h} \left(\cdot \mid s _ {h} ^ {\prime}, a _ {h} ^ {\prime}\right) \right],
$$
where the expectation is with respect to the randomness of the transition, the reward and the policy. Similarly, we can define the Q-value functions of policy  $\pi$  by  $\{Q_h^\pi\}_{h \in [H]} \subseteq (\mathcal{S} \times \mathcal{A} \to \mathbb{R})$  so that  $Q_h^\pi(s, a)$  is equal to the expected cumulative reward an agent will receive if she follows policy  $\pi$  starting from taking action  $a$  at state  $s$  and step  $h$ . Formally,
$$
Q _ {h} ^ {\pi} (s, a) = \mathbb {E} \left[ \sum_ {h ^ {\prime} = h} ^ {H} r _ {h ^ {\prime}} \left(s _ {h ^ {\prime}}, a _ {h ^ {\prime}}\right) \mid \left(s _ {h}, a _ {h}\right) = (s, a), a _ {h ^ {\prime}} \sim \pi_ {h ^ {\prime}} \left(s _ {h ^ {\prime}}\right), s _ {h + 1} ^ {\prime} \sim \mathbb {P} _ {h} \left(\cdot \mid s _ {h} ^ {\prime}, a _ {h} ^ {\prime}\right) \right].
$$
We denote by  $\pi^{\star} = \{\pi_h^{\star}\}_{h\in [H]}$  the optimal policy such that  $\pi^{\star}\in \mathrm{argmax}_{\pi}V_{h}^{\pi}(s)$  for all  $(s,h)\in S\times [H]$ . By backward induction, one can easily prove that there always exists an optimal Markov policy. For simplicity of notations, we denote by  $V_{h}^{\star} = V_{h}^{\pi^{\star}}$  and  $Q_{h}^{\star} = Q_{h}^{\pi^{\star}}$  the optimal value functions. Note that given an MDP, the optimal value functions are unique despite that there may exist multiple optimal policies.
Learning objective. The objective of this paper is to design an efficient policy optimization algorithm to learn an  $\epsilon$ -optimal policy  $\pi$  such that  $V_{1}^{\pi}(s_{1}) \geq V_{1}^{\star}(s_{1}) - \epsilon$ . Here the optimality is only measured by the value at the initial state  $s_{1}$ , because (a) each episode always starts from  $s_{1}$ , and (b) this paper studies the online exploration setting without access to a simulator, which means some states might be unreachable and learning optimal policies starting from those states is in general impossible.
3 Optimistic Natural Policy Gradient
In this section, we present the algorithm OPTIMISTIC NPG (Optimistic Natural Policy Gradient) and its theoretical guarantees.
The pseudocode of OPTIMISTIC NPG is provided in Algorithm 1. At a high level, the algorithm consists of the following three key modules.
- Periodic on-policy data collection (Lines 4-6): Similarly to the empirical PPO algorithm [Schulman et al., 2017], OPTIMISTIC NPG discards all the old data, and executes the current policy  $\pi^k$  to collect a batch of fresh data  $\mathcal{D}^k$  after every  $m$  steps of policy update. These data will later be used to evaluate and improve the policies in the next  $m$  steps. Noticeably, this on-policy data mechanism is very different from most existing works in theoretical RL, where they either need to keep the historical data or have to rerun historical policies to refresh the dataset for the technical purpose of elliptical potential arguments in proofs. In comparison, OPTIMISTIC NPG only uses fresh data collected by the current (or very recent) policy, which resembles practical policy optimization algorithms such as PPO [Schulman et al., 2017] and TRPO [Schulman et al., 2015].
- Optimistic policy evaluation (Line 8): Given the above collected dataset  $\mathcal{D}^k$ , we estimate the value functions of the current policy  $\pi^k$  by invoking Subroutine OPE (optimistic policy evaluation). In Section 4, we show how to implement Subroutine OPE for tabular MDPs, linear MDPs and RL problems of low eluder dimension.
- Policy update (Line 9): Given the optimistic value estimates  $\{\overline{Q}_h^k\}_{h\in [H]}$  of  $\pi^k$ , OPTIMISTIC NPG performs one-step mirror ascent from  $\pi_h^k (\cdot \mid s)$  with gradient  $\overline{Q}_h^k (s,\cdot)$  to obtain  $\pi_h^{k + 1}(\cdot \mid s)$  at each  $(h,s)$ . Importantly, this step can be implemented in a computationally efficient way, because by the update rule of mirror ascent  $\pi_h^k (\cdot \mid s)\propto \exp (\eta \sum_{t = 1}^{k - 1}\overline{Q}_h^t (s,\cdot))$  we only need to store  $\{\{\overline{Q}_h^t\}_{h\in [H]}\}_{t\in [K]}$ , from which any  $\pi_h^k (\cdot \mid s)$  can be computed on the fly.
OPTIMISTIC NPG is a generic policy optimization meta-algorithm, and we can show it provably learns near-optimal policies as long as subroutine OPE satisfies the following condition.
Condition 1 (Requirements for OPE). Suppose parameter  $m$  and  $\eta$  in OPTIMISTIC NPG satisfy  $m \leq (\eta H^2)^{-1}$ . Then with probability at least  $1 - \delta$ ,  $\mathrm{OPE}(\pi^k, \mathcal{D}^k)$  returns  $\{\overline{Q}_h^k\}_{h \in [H]} \subseteq (\mathcal{S} \times \mathcal{A} \to [0, H])$  that satisfy the following properties for all  $k \in [K]$ .
(1A) Optimism: For all  $(s, a, h) \in S \times \mathcal{A} \times [H]$
$$
\overline {{Q}} _ {h} ^ {k} (s, a) \geq \left(\mathcal {T} _ {h} ^ {\pi^ {k}} \overline {{Q}} _ {h + 1} ^ {k}\right) (s, a)
$$
where  $(\mathcal{T}_h^\pi f)(s,a) = R_h(s,a) + \mathbb{E}\left[f(s',a')\mid s'\sim \mathbb{P}_h(\cdot \mid s,a),a'\sim \pi_{h + 1}(s')\right]$ .
Algorithm 1 OPTIMISTIC NPG
1: input: number of iterations  $K$ , period of collecting fresh data  $m$ , batch size  $N$ , learning rate  $\eta$
2: initialize: for all  $(h,s)\in [H]\times S$  set  $\pi_h^1 (\cdot |s) = \mathrm{Uniform}(\mathcal{A})$
3: for  $k = 1, \dots, K$  do
4: if  $k \equiv 1 (\mathrm{mod} m)$  then
5:  $\mathcal{D}^k\gets \{N$  fresh trajectories  $\stackrel{\mathrm{i.i.d.}}{\sim}\pi^k\}$
6: else
7:  $\mathcal{D}^k\gets \mathcal{D}^{k - 1}$
8:  $\{\overline{Q}_h^k\}_{h\in [H]}\gets \mathrm{OPE}(\pi^k,\mathcal{D}^k)$
9: for all  $(h,s)\in [H]\times \mathcal{S}$  update  $\pi_h^{k + 1}(\cdot |s)\propto \pi_h^k (\cdot |s)\cdot \exp (\eta \cdot \overline{Q}_h^k (s,\cdot))$
10: output:  $\pi^{\mathrm{out}}$  that is sampled uniformly at random from  $\{\pi^k\}_{k\in [K]}$
(1B) Consistency: There exists an absolute complexity measure  $L \in \mathbb{R}^{+}$  such that
$$
\overline {{V}} _ {1} ^ {k} (s _ {1}) - V _ {1} ^ {\pi^ {k}} (s _ {1}) \leq \sqrt {(L / N) \times \log^ {2} (N K L / \delta)},
$$
where  $\overline{V}_1^k (s_1) = \mathbb{E}\left[\overline{Q}_1^k (s_1,a_1)\mid a_1\sim \pi^k (s_1)\right]$ .
Condition (1A) requires that the Q-value estimate returned by OPE satisfies the Bellman equation under policy  $\pi^k$  optimistically. Requiring optimism is very common in analyzing RL algorithms, as most algorithms rely on the principle of optimism in face of uncertainty to handle exploration. Condition (1B) requires that the degree of over-optimism at initial state  $s_1$  is roughly of order  $\sqrt{1 / N}$  with respect to the number of samples  $N$ . This is intuitive since as more data are collected from policy  $\pi^k$  or some policy similar to  $\pi^k$  (as is enforced by the precondition  $m \leq (\eta H^2)^{-1}$ ), the value estimate at the initial state should be more accurate. In Section 4, we will provide concrete instantiations of Subroutine OPE for tabular MDPs, linear MDPs and RL problems of low eluder dimension, all of which satisfy Condition 1 with mild complexity measure  $\bar{L}$ .
Under Condition 1, we have the following theoretical guarantees for OPTIMISTIC NPG, which we prove in Appendix A.
Theorem 1. Suppose Condition 1 holds. In Algorithm 1, if we choose
$$
K = \Theta \left(\frac {H ^ {4} \log | \mathcal {A} |}{\epsilon^ {2}}\right), \quad N = \Theta \left(\frac {L \log^ {2} (L K / \delta)}{\epsilon^ {2}}\right), \quad \eta = \Theta \left(\frac {\epsilon}{H ^ {3}}\right), \qquad m \leq \Theta \left(\frac {H}{\epsilon}\right),
$$
then with probability at least  $1 / 2$ ,  $\pi^{\mathrm{out}}$  is  $\mathcal{O}(\epsilon)$ -optimal.
Below we emphasize two special choices of  $m$  (the period of sampling fresh data) in Theorem 1:
- When choosing  $m = 1$ , OPTIMISTIC NPG is purely on-policy as it only uses data sampled from the current policy to perform policy optimization. In this case, the total sample complexity is
$$
\frac {\text {i t e r a t i o n}}{\text {p e r i o d o f s a m p l i n g}} \times \text {b a t c h s i z e} = \frac {K}{m} \times N = \tilde {\Theta} \left(\frac {L H ^ {4}}{\epsilon^ {4}}\right),
$$
which, to our knowledge, is the first polynomial sample complexity guarantee for purely on-policy algorithms.
- When choosing  $m = \left\lbrack  {H/\epsilon }\right\rbrack$  ,we obtain sample complexity
$$
\frac {\text {i t e r a t i o n}}{\text {p e r i o d o f s a m p l i n g}} \times \text {b a t c h s i z e} = \frac {K}{m} \times N = \tilde {\Theta} \left(\frac {L H ^ {3}}{\epsilon^ {3}}\right),
$$