SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lemma 3 (policy Lipschitz). Suppose we choose  $\eta$  and  $m$  such that  $\eta m \leq 1 / H^2$ , then for any  $k \in \mathbb{N}^+$  and any function  $f: S \times \mathcal{A} \to \mathbb{R}^+$ :
$$
\mathbb {E} _ {\pi^ {k}} [ f (s _ {h}, a _ {h}) ] = \Theta (\mathbb {E} _ {\pi^ {t _ {k}}} [ f (s _ {h}, a _ {h}) ]).
$$
By combining Lemma 3 with Equation (3) and noticing that  $b_h^k = b_h^{t_k}$ , we have
$$
\overline {{V}} _ {1} ^ {k} (s _ {1}) - V _ {1} ^ {\pi^ {k}} (s _ {1}) \leq \mathcal {O} \Big (\max  _ {k \in [ K ]} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {t _ {k}}} [ b _ {h} ^ {k} (s _ {h}, a _ {h}) ] \Big) = \mathcal {O} \Big (\max  _ {k \in [ K ]} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {t _ {k}}} [ b _ {h} ^ {t _ {k}} (s _ {h}, a _ {h}) ] \Big).
$$
Since the bonus function  $b_h^{t_k}$  is defined by using the dataset  $\mathcal{D}^{t_k}$  collected from executing policy  $\pi^{t_k}$ , one could expect its average under  $\pi^{t_k}$  should be rather small when the batch size  $N$  is large enough, as formalized in the following lemma.
Lemma 4 (on-policy uncertainty). Let  $\pi$  be an arbitrary policy and  $\lambda \geq 1$ . Suppose we sample  $\{(s_h^n, a_h^n)\}_{n=1}^N$  i.i.d. from  $\pi$ . Denote  $\Sigma_h = \sum_{i=1}^N \phi_h(s_h^i, a_h^i) \phi_h^{\mathrm{T}}(s_h^i, a_h^i)$ . Then with probability at least  $1 - \delta$ :
$$
\mathbb {E} _ {(s _ {h}, a _ {h}) \sim \pi} \left[ \| \phi_ {h} (s _ {h}, a _ {h}) \| _ {(\Sigma_ {h} + \lambda \mathbf {I} _ {d \times d}) ^ {- 1}} \right] = \mathcal {O} \left(\sqrt {\frac {d \log (N / \delta)}{N}}\right).
$$
As a result, by Lemma 4 and the definition of  $b_h^k$ , we have that with probability at least  $1 - \delta$ : for all  $k \in [K]$
$$
\sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi^ {t _ {k}}} \left[ b _ {h} ^ {k} (s _ {h}, a _ {h}) \right] = \mathcal {O} \left(H d \sqrt {\frac {H \log^ {2} (N K H / \delta)}{N}}\right)
$$
where in applying Lemma 4 we use the fact  $|\mathcal{D}_h^k| = N / H$  due to data splitting. Putting all pieces together, we get
$$
\overline {{V}} _ {1} ^ {k} (s _ {1}) - V _ {1} ^ {\pi^ {k}} (s _ {1}) \leq \mathcal {O} \left(\sqrt {\frac {d ^ {2} H ^ {3} \log^ {2} (K N H / \delta)}{N}}\right).
$$
As a result, Condition (1B) holds with  $L = d^{2}H^{3}$ .
B.2 Proof of Proposition 1 (tabular OPE)
The proof follows basically the same as that of Proposition 2. The only modification needed is to replace Lemma 2 and 4 by their tabular counterparts, which we state below.
Lemma 5 (optimism of value estimates: tabular). Under the same choice of  $\alpha$  as in Proposition 1, with probability at least  $1 - \delta$ : for all  $(k,h,s,a) \in [K] \times [H] \times \mathcal{S} \times \mathcal{A}$ :
$$
0 \leq (\overline {{Q}} _ {h} ^ {k} - \mathcal {T} _ {h} ^ {\pi^ {k}} \overline {{Q}} _ {h + 1} ^ {k}) (s, a) \leq 2 b _ {h} ^ {k} (s, a),
$$
where  $\{b_h^k\}_{h\in [H]}$  are the bonus functions defined in Line 6, Subroutine 1 tabular OPE  $(\pi^k,\mathcal{D}^k)$ .
Lemma 6 (on-policy uncertainty: tabular). Let  $\pi$  be an arbitrary policy. Suppose we sample  $\{(s_h^n, a_h^n)\}_{n=1}^N$ . i.i.d. from  $\pi$ . Denote  $J_h(s, a) = \sum_{i=1}^N \mathbf{1}((s_h^i, a_h^i) = (s, a))$ . Then with probability at least  $1 - \delta$ :
$$
\mathbb {E} _ {(s _ {h}, a _ {h}) \sim \pi} \left[ \sqrt {\frac {1}{J _ {h} (s , a) + 1}} \right] = \mathcal {O} \left(\sqrt {\frac {S A \log (N / \delta)}{N}}\right).
$$
B.3 Proof of Proposition 3 (general OPE)
The proof follows basically the same as that of Proposition 2. The only modification needed is to replace Lemma 2 and 4 by their general counterparts, which we state below.
Lemma 7 (optimism of value estimates: general). Suppose Assumption 1 holds. There exists an absolute constant  $c$  such that with probability at least  $1 - \delta$ : for all  $(k, h, s, a) \in [K] \times [H] \times S \times \mathcal{A}$ :
$$
0 \leq (\overline {{Q}} _ {h} ^ {k} - \mathcal {T} _ {h} ^ {\pi^ {k}} \overline {{Q}} _ {h + 1} ^ {k}) (s, a) \leq b _ {h} (s, a, \mathcal {D} _ {h} ^ {k}) := \left( \begin{array}{c} \sup  _ {f _ {h}, f _ {h} ^ {\prime} \in \mathcal {F} _ {h}} f _ {h} (s, a) - f _ {h} ^ {\prime} (s, a) \\ s. t. \sum_ {(s _ {h}, a _ {h}) \in \mathcal {D} _ {h} ^ {k}} (f _ {h} (s _ {h}, a _ {h}) - f _ {h} ^ {\prime} (s _ {h}, a _ {h})) ^ {2} \leq c \beta \end{array} \right).
$$
Lemma 8 (on-policy uncertainty: general). Let  $\pi$  be an arbitrary policy. Suppose we sample  $\mathcal{D}_h = \{(s_h^n, a_h^n)\}_{n=1}^N$  i.i.d. from  $\pi$ . Then with probability at least  $1 - \delta$ :
$$
\mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \pi} \left[ b _ {h} (s, a, \mathcal {D} _ {h}) \right] = \mathcal {O} \left(H \sqrt {\frac {d _ {E} (1 / N , \mathcal {F} _ {h}) \beta}{N}} + H \sqrt {\frac {\log (1 / \delta)}{N}}\right).
$$
Lemma 1 is an immediate consequence of Lemma 4.2 in [Cai et al., 2020] or Lemma 1 in [Shani et al., 2020].
Let us consider a fixed pair  $(k,h)\in [K]\times [H]$ . Recall  $\overline{Q}_h^k$  is defined as
$$
\overline {{Q}} _ {h} ^ {k} (s, a) = \operatorname {T r u n c a t e} _ {[ 0, H - h + 1 ]} \big (\langle \hat {\theta} _ {h} ^ {k}, \phi_ {h} (s, a) \rangle + b _ {h} ^ {k} (s, a) \big),
$$
$$
\hat{\theta}_{h}^{k} = \operatorname *{argmin}_{\theta}\sum_{(s_{h},a_{h},r_{h},s_{h + 1})\in \mathcal{D}_{h}^{k}}\left(\phi_{h}(s_{h},a_{h})^{\mathrm{T}}\theta -r_{h} - \overline{V}_{h + 1}^{k}(s_{h + 1})\right)^{2} + \lambda \| \theta \|_{2}^{2},
$$
$$
b _ {h} ^ {k} (s, a) = \alpha \times \| \phi_ {h} (s, a) \| _ {\left(\Sigma_ {h} ^ {k} + \lambda \mathbf {I} _ {d \times d}\right) ^ {- 1}}, \quad \text {a n d} \quad \Sigma_ {h} ^ {k} = \sum_ {(s _ {h}, a _ {h}) \in \mathcal {D} _ {h} ^ {k}} \phi (s _ {h}, a _ {h}) (\phi (s _ {h}, a _ {h})) ^ {\mathrm {T}}.
$$
By the standard linear completeness property of linear MDPs Jin et al. [2020], there exists  $\theta_h^k$  such that  $\| \theta_h^k\| _2\leq H\sqrt{d}$  and
$$
\langle \phi_ {h} (s, a), \theta_ {h} ^ {k} \rangle = R _ {h} (s, a) + \mathbb {E} _ {s ^ {\prime} \sim \mathbb {P} (\cdot | s, a)} \left[ \overline {{V}} _ {h + 1} ^ {k} \left(s ^ {\prime}\right) \right] \quad \text {f o r a l l} (s, a) \in \mathcal {S} \times \mathcal {A}.
$$
Therefore, to prove Lemma 2, it suffices to show that for all  $(s,a)$
$$
\left| \left\langle \phi_ {h} (s, a), \theta_ {h} ^ {k} - \hat {\theta} _ {h} ^ {k} \right\rangle \right| \leq b _ {h} ^ {k} (s, a).
$$
To condense notations, denote by  $\{(s_h^i,a_h^i,r_h^i,s_{h + 1}^i)\}_{i = 1}^M$  the tuples in  $\mathcal{D}_h^k$ . By the definition of ridge regression,
$$
\begin{array}{l} \left| \left\langle \phi_ {h} (s, a), \hat {\theta} _ {h} ^ {k} - \theta_ {h} ^ {k} \right\rangle \right| \\ = \left| \left\langle \phi_ {h} (s, a), \left(\Sigma_ {h} ^ {k} + \lambda \mathbf {I} _ {d \times d}\right) ^ {- 1} \times \right. \right. \\ \left. \left(\sum_ {i = 1} ^ {M} \phi_ {h} \left(s _ {h} ^ {i}, a _ {h} ^ {i}\right) \left(r _ {h} ^ {i} + \bar {V} _ {h + 1} ^ {k} \left(s _ {h + 1} ^ {i}\right) - \phi_ {h} \left(s _ {h} ^ {i}, a _ {h} ^ {i}\right) ^ {\mathrm {T}} \theta_ {h} ^ {k}\right) - \lambda \theta_ {h} ^ {k}\right) \right\rangle \\ \leq \left\| \phi_ {h} (s, a) \right\| _ {\left(\Sigma_ {h} ^ {k} + \lambda \mathbf {I} _ {d \times d}\right) ^ {- 1}} \times \\ \left(\left\| \sum_ {i = 1} ^ {M} \phi_ {h} \left(s _ {h} ^ {i}, a _ {h} ^ {i}\right) \left(r _ {h} ^ {i} + \overline {{V}} _ {h + 1} ^ {k} \left(s _ {h + 1} ^ {i}\right) - \phi_ {h} \left(s _ {h} ^ {i}, a _ {h} ^ {i}\right) ^ {\mathrm {T}} \theta_ {h} ^ {k}\right) \right\| _ {\left(\Sigma_ {h} ^ {k} + \lambda \mathbf {I} _ {d \times d}\right) ^ {- 1}} + H \sqrt {d}\right), \\ \end{array}
$$
where the inequality follows from  $\| \lambda \theta_h^k\|_{(\Sigma_h^k +\lambda \mathbf{I}_{d\times d})^{-1}}\leq \sqrt{\lambda}\| \theta_h^k\|_2\leq H\sqrt{d}$ . It remains to bound the first term in the bracket. Now here comes the key observation that helps shave off the  $\sqrt{d}$  factor from the bonus.
Observation 1.  $\mathcal{D}_h^k$  and  $\overline{V}_{h + 1}^k$  are independent conditioning on  $\pi^{t_k}$  where  $t_k$  denotes the index of the last iteration of collecting fresh data before the  $k^{\text{th}}$  iteration.
To see why, recall that after obtaining  $\mathcal{D}^{t_k}$  Algorithm 2 splits it evenly into  $H$  disjoint subsets  $\{\mathcal{D}_h^{t_k}\}_{h = 1}^H$  and then only uses  $\{\mathcal{D}_{h'}^{t_k}\}_{h' = h + 1}^H$  for evaluating and improving  $\{\pi_{h'}^l\}_{h' = h + 1}^H$  for  $l\in [t_k,t_k + m - 1]$ . As a result,  $\{\overline{V}_{h + 1}^l\}_{l = t_k}^{t_k + m - 1}$  (including  $\overline{V}_{h + 1}^k$ ) is independent of  $\mathcal{D}_h^{t_k} = \mathcal{D}_h^k$  conditioning on  $\pi^{t_k}$ . By the concentration of self-normalized processes [Abbasi-Yadkori et al., 2011], we conclude that with probability at least  $1 - \delta$ : for all  $(k,h)\in [K]\times [H]$
$$
\left\| \sum_ {i = 1} ^ {M} \phi_ {h} (s _ {h} ^ {i}, a _ {h} ^ {i}) \left(r _ {h} ^ {i} + \overline {{V}} _ {h + 1} ^ {k} (s _ {h + 1} ^ {i}) - \phi_ {h} (s _ {h} ^ {i}, a _ {h} ^ {i}) ^ {\mathrm {T}} \theta_ {h} ^ {k}\right) \right\| _ {(\Sigma_ {h} ^ {k} + \lambda \mathbf {I} _ {d \times d}) ^ {- 1}} \leq \mathcal {O} \left(H \sqrt {d \log (M K H / \delta)}\right).
$$
We first introduce the following two auxiliary lemmas about the lipschitzness continuity of softmax paraterization.
Lemma 9. There exists an absolute constant  $c > 0$  such that for any policy  $\pi, \hat{\pi}$  satisfying  $\hat{\pi}_h(a \mid s) \propto \pi_h(a \mid s) \times \exp(L_h(s, a))$  where  $\{L_h(s, a)\}_{h \in [H]}$  is a set of functions from  $S \times \mathcal{A}$  to  $[-1/H, 1/H]$ , we have that for any  $\tau_H := (s_1, a_1, \ldots, s_H, a_H) \in (S \times \mathcal{A})^H$ :
$$
\mathbb {P} ^ {\hat {\pi}} (\tau_ {H}) \leq c \times \mathbb {P} ^ {\pi} (\tau_ {H}).
$$
Proof of Lemma 9. By using the relation between  $\pi$  and  $\hat{\pi}$  and the normalization property of policy  $\pi$ , we have that for any  $(h, s, a)$ :
$$
\begin{array}{l} \hat {\pi} _ {h} (a \mid s) = \frac {\pi_ {h} (a \mid s) \times \exp (L _ {h} (s , a))}{\sum_ {a ^ {\prime}} \pi_ {h} (a ^ {\prime} \mid s) \times \exp (L _ {h} (s , a ^ {\prime}))} \\ \leq \frac {\pi_ {h} (a \mid s) \times \exp (1 / H)}{\sum_ {a ^ {\prime}} \pi_ {h} (a ^ {\prime} \mid s) \exp (- 1 / H)} \\ = \pi_ {h} (a \mid s) \times \exp (2 / H) \leq \pi_ {h} (a \mid s) \times \left(1 + \frac {\mathcal {O} (1)}{H}\right). \\ \end{array}
$$
Therefore, for any  $\tau_H\coloneqq (s_1,a_1,\ldots ,s_H,a_H)$
$$
\begin{array}{l} \mathbb {P} ^ {\hat {\pi}} (\tau_ {H}) = \left(\prod_ {h = 1} ^ {H - 1} \mathbb {P} _ {h} (s _ {h + 1} \mid s _ {h}, a _ {h})\right) \times \left(\prod_ {h = 1} ^ {H} \hat {\pi} _ {h} (a _ {h} \mid s _ {h})\right) \\ \leq \left(\prod_ {h = 1} ^ {H - 1} \mathbb {P} _ {h} \left(s _ {h + 1} \mid s _ {h}, a _ {h}\right)\right) \times \left(\prod_ {h = 1} ^ {H} \pi_ {h} \left(a _ {h} \mid s _ {h}\right)\right) \times \left(1 + \frac {\mathcal {O} (1)}{H}\right) ^ {H} \\ = \mathcal {O} (1) \times \left(\prod_ {h = 1} ^ {H - 1} \mathbb {P} _ {h} \left(s _ {h + 1} \mid s _ {h}, a _ {h}\right)\right) \times \left(\prod_ {h = 1} ^ {H} \pi_ {h} \left(a _ {h} \mid s _ {h}\right)\right) = \mathcal {O} (1) \times \mathbb {P} ^ {\pi} (\tau_ {H}). \\ \end{array}
$$
Lemma 10. Let  $\mu, \nu$  be two probability densities defined over  $\mathcal{X}$  such that  $\| \mu / \nu \|_{\infty} \leq \alpha$ , then we have that for any function  $f: \mathcal{X} \to \mathbb{R}^{+}$ ,  $\mathbb{E}_{\mu}[f(x)] \leq \alpha \mathbb{E}_{\nu}[f(x)]$ .
Proof of Lemma 10. By definition,  $\mathbb{E}_{\mu}[f(x)] = \int_{x} f(x) \mu(x) dx \leq \int_{x} f(x) (\alpha \nu(x)) dx = \alpha \mathbb{E}_{\nu}[f(x)]$ .
By the update rule of NPG, we have
$$
\pi_ {h} ^ {k} (\cdot \mid s) \propto \pi_ {h} ^ {t _ {k}} (\cdot \mid s) \times \exp \left(\eta \sum_ {i = t _ {k}} ^ {k - 1} \overline {{Q}} _ {h} ^ {i} (s, \cdot)\right).
$$
Since we choose  $\eta$  and  $m$  such that  $\eta m \leq 1 / H^2$ ,
$$
\left| \eta \sum_ {i = t _ {k}} ^ {k - 1} \overline {{Q}} _ {h} ^ {i} (s, \cdot) \right| \leq \eta (k - t _ {k}) H \leq \eta m H \leq 1 / H.
$$
Therefore, by invoking Lemma 9 with  $\hat{\pi} = \pi^k$  and  $\pi = \pi^{t_k}$ , we have that for any  $\tau_H\in (\mathcal{S}\times \mathcal{A})^H$ :
$$
\mathbb {P} ^ {\pi^ {k}} \left(\tau_ {H}\right) \leq c \times \mathbb {P} ^ {\pi^ {t _ {k}}} \left(\tau_ {H}\right).
$$
By further invoking Lemma 10 with  $\mathcal{X} = (\mathcal{S}\times \mathcal{A})^H$ ,  $\mu = \mathbb{P}^{\pi^k}$  and  $\nu = \mathbb{P}^{\pi^{t_k}}$ , we obtain that for any function  $f:\mathcal{S}\times \mathcal{A}\to \mathbb{R}^{+}$ :
$$
\mathbb {E} _ {\pi^ {k}} [ f (s _ {h}, a _ {h}) ] = \mathcal {O} \left(\mathbb {E} _ {\pi^ {t _ {k}}} [ f (s _ {h}, a _ {h}) ]\right).
$$
Similarly, we can show  $\mathbb{E}_{\pi^{t_k}}[f(s_h,a_h)] = \mathcal{O}(\mathbb{E}_{\pi^k}[f(s_h,a_h)])$ , which completes the proof of Lemma 3.
To simplify notations, denote  $\phi_h^i \coloneqq \phi_h(s_h^i, a_h^i)$ . For the technical purpose of applying martingale concentration, we additionally define the intermediate empirical covariance matrices:  $\Sigma_h^n = \sum_{i=1}^{n-1} \phi_h^i (\phi_h^i)^{\mathrm{T}}$ , for  $n \in [N]$ . We have that with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \pi} \left[ \left\| \phi_ {h} \left(s _ {h}, a _ {h}\right) \right\| _ {\left(\Sigma_ {h} + \lambda \mathbf {I} _ {d \times d}\right) ^ {- 1}} \right] \\ \leq \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \pi} \left[ \| \phi_ {h} \left(s _ {h}, a _ {h}\right) \| _ {\left(\Sigma_ {h} ^ {n} + \lambda \mathbf {I} _ {d \times d}\right) ^ {- 1}} \right] \\ \leq \frac {1}{N} \sum_ {n = 1} ^ {N} \| \phi_ {h} ^ {n} \| _ {\left(\Sigma_ {h} ^ {n} + \lambda \mathbf {I} _ {d \times d}\right) ^ {- 1}} + \mathcal {O} \left(\sqrt {\frac {\log (1 / \delta)}{N}}\right) \tag {4} \\ \leq \mathcal {O} \left(\sqrt {\frac {d \log N}{N}}\right) + \mathcal {O} \left(\sqrt {\frac {\log (1 / \delta)}{N}}\right), \\ \end{array}
$$
where the first inequality uses  $\Sigma_h^n\preceq \Sigma_h$  , the second one uses Azuma-Hoeffding inequality with  $\lambda = 1$  , and the last one uses Cauchy-Schwarz inequality and the standard elliptical potential argument.
$$
\overline {{Q}} _ {h} ^ {k} (s, a) = \min  \left\{H - h + 1, \mathbb {E} _ {s ^ {\prime} \sim \hat {\mathbb {P}} _ {h} ^ {k} (s, a)} [ \overline {{V}} _ {h + 1} ^ {k} (s ^ {\prime}) ] + \hat {R} _ {h} ^ {k} (s, a) + b _ {h} ^ {k} (s, a) \right\}
$$
where  $\hat{\mathbb{P}}_h^k$  and  $\hat{\mathbb{R}}_h^k$  are the empirical estimates of transition and reward at step  $h$  from  $\mathcal{D}_h^k$ , and  $b_h^k(s, a) = \alpha(J_h^k(s, a) + 1)^{-1/2}$  with  $J_h^k(s, a) = \sum_{(s_h, a_h) \in \mathcal{D}_h^k} \mathbf{1}((s_h, a_h) = (s, a))$ . Therefore, to prove Lemma 5, it suffices to show that for all  $(s, a)$
$$
\left| \left(\mathbb {E} _ {s ^ {\prime} \sim \hat {\mathbb {P}} _ {h} ^ {k} (s, a)} [ \overline {{V}} _ {h + 1} ^ {k} (s ^ {\prime}) ] - \mathbb {E} _ {s ^ {\prime} \sim \mathbb {P} _ {h} (s, a)} [ \overline {{V}} _ {h + 1} ^ {k} (s ^ {\prime}) ]\right) + \left(\hat {R} _ {h} ^ {k} (s, a) - R _ {h} (s, a)\right) \right| \leq b _ {h} ^ {k} (s, a).
$$
By observation 1, we know  $\{\overline{V}_{h + 1}^l\}_{l = t_k}^{t_k + m - 1}$  is independent of  $\mathcal{D}_h^{t_k} = \mathcal{D}_h^k$  conditioning on  $\pi^{t_k}$ , which implies  $\overline{V}_{h + 1}^k$  is independent of  $\hat{\mathbb{P}}_h^k$ . Therefore, by Azuma-Hoeffding inequality and standard union bound, we conclude that with probability at least  $1 - \delta$ : for all  $(k,h,s,a)\in [K]\times [H]\times S\times \mathcal{A}$
$$
\begin{array}{l} \left| \left(\mathbb {E} _ {s ^ {\prime} \sim \hat {\mathbb {P}} _ {h} ^ {k} (s, a)} [ \overline {{V}} _ {h + 1} ^ {k} (s ^ {\prime}) ] - \mathbb {E} _ {s ^ {\prime} \sim \mathbb {P} _ {h} (s, a)} [ \overline {{V}} _ {h + 1} ^ {k} (s ^ {\prime}) ]\right) + \left(\hat {R} _ {h} ^ {k} (s, a) - R _ {h} (s, a)\right) \right| \\ \leq \mathcal {O} \left(H \sqrt {\frac {\log (K N H S A / \delta)}{J _ {h} ^ {k} (s , a) + 1}}\right). \\ \end{array}
$$
For the technical purpose of performing martingale concentration, we introduce the notion of intermediate counters:  $J_{h}^{n}(s,a) = \sum_{i = 1}^{n - 1}\mathbf{1}((s_{h}^{i},a_{h}^{i}) = (s,a))$ , for  $n\in [N]$ . We have that with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \mathbb {E} _ {(s _ {h}, a _ {h}) \sim \pi} \left[ \sqrt {\frac {1}{J _ {h} (s , a) + 1}} \right] \\ \leq \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {(s _ {h}, a _ {h}) \sim \pi} \left[ \sqrt {\frac {1}{J _ {h} ^ {n} (s , a) + 1}} \right] \tag {5} \\ \leq \frac {1}{N} \sum_ {n = 1} ^ {N} \sqrt {\frac {1}{J _ {h} ^ {n} \left(s _ {h} ^ {n} , a _ {h} ^ {n}\right) + 1}} + \mathcal {O} \left(\sqrt {\frac {\log (1 / \delta)}{N}}\right) \\ \leq \mathcal {O} \left(\sqrt {\frac {S A \log N}{N}}\right) + \mathcal {O} \left(\sqrt {\frac {\log (1 / \delta)}{N}}\right), \\ \end{array}
$$
where the first inequality uses  $J_h^n (s,a) \leq J_h(s,a)$  for all  $(s,a) \in S \times \mathcal{A}$ , the second one uses Azuma-Hoeffding inequality, and the last one follows the standard pigeon-hole argument.
By observation 1, we know  $\mathcal{D}_h^k$  is independent of  $\overline{V}_{h + 1}^k$  conditioning on  $\pi^{t_k}$  where  $t_k$  is the index of the last iteration when OPTIMISTIC NPG collects fresh data before iteration  $k + 1$ . With this independence relation in mind, we can easily prove that confidence set  $\mathcal{B}_h^k$  satisfy the following properties by standard concentration inequality and union bounds.
Lemma 11. Suppose Assumption 1 holds. There exists an absolute constant  $c$  such that with probability at least  $1 - \delta$ , for all  $k \in [K]$  and  $h \in [H]$ ,
$\mathcal{T}_h^\pi^k\overline{Q}_{h + 1}^k\in \mathcal{B}_h^k,$
for any  $f_{h}\in \mathcal{B}_{h}^{k},\sum_{(s_{h},a_{h})\in \mathcal{D}_{h}^{k}}(f_{h}(s_{h},a_{h}) - \mathcal{T}_{h}^{\pi^{k}}\overline{Q}_{h + 1}^{k}(s_{h},a_{h}))^{2}\leq c\beta .$
The proof of Lemma 11 follows trivially from modifying the proofs of Lemma 39 and 40 in Jin et al. [2021], which we omit here.
$$
\bar {\mathcal {B}} _ {h} ^ {k} = \left\{f _ {h} \in \mathcal {F} _ {h}: \sum_ {(s _ {h}, a _ {h}) \in \mathcal {D} _ {h} ^ {k}} \left(f _ {h} (s _ {h}, a _ {h}) - \left(\mathcal {T} _ {h} ^ {\pi^ {k}} \overline {{Q}} _ {h + 1} ^ {k}\right) (s _ {h}, a _ {h})\right) ^ {2} \leq c \beta \right\}.
$$
By using the first relation in Lemma 11 and the definition of  $\overline{Q}_h^k$ , we immediately obtain that
$$
\left(\mathcal {T} _ {h} ^ {\pi^ {k}} \overline {{Q}} _ {h + 1} ^ {k}\right) (s, a) \leq \sup  _ {f _ {h} \in \mathcal {B} _ {h} ^ {k}} f _ {h} (s, a) = \overline {{Q}} _ {h} ^ {k} (s, a)
$$
$$
\overline {{Q}} _ {h} ^ {k} (s, a) - \left(\mathcal {T} _ {h} ^ {\pi^ {k}} \overline {{Q}} _ {h + 1} ^ {k}\right) (s, a) \leq \sup  _ {f _ {h}, f _ {h} ^ {\prime} \in \mathcal {B} _ {h} ^ {k}} \left(f _ {h} (s, a) - f _ {h} ^ {\prime} (s, a)\right) \leq \sup  _ {f _ {h}, f _ {h} ^ {\prime} \in \bar {\mathcal {B}} _ {h} ^ {k}} \left(f _ {h} (s, a) - f _ {h} ^ {\prime} (s, a)\right),
$$
where the last inequality uses  $\vec{\mathcal{B}}_h^k\subset \mathcal{B}_h^k$  by Lemma 11.
C.8 Proof of Lemma 8
Recall we define
$$
b_{h}(s,a,\mathcal{D}_{h}):= \left( \begin{array}{c}\sup_{f_{h},f_{h}^{\prime}\in \mathcal{F}_{h}}f_{h}(s,a) - f_{h}^{\prime}(s,a)\\ \text{s.t.}\sum_{(s_{h},a_{h})\in \mathcal{D}_{h}}(f_{h}(s_{h},a_{h}) - f_{h}^{\prime}(s_{h},a_{h}))^{2}\leq c\beta \end{array} \right),
$$
which implies that  $b_h(s, a, \mathcal{D}_h) \leq b_h(s, a, \underline{\mathcal{D}}_h)$  for any  $\underline{\mathcal{D}}_h \subseteq \mathcal{D}_h$ . Let  $\mathcal{D}_h^{(n)} = \{(s_h^i, a_h^i)\}_{i=1}^n$ . We have that with probability at least  $1 - \delta$ ,
$$
\begin{array}{l} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \pi} \left[ b _ {h} (s, a, \mathcal {D} _ {h}) \right] \leq \frac {1}{N} \sum_ {n = 1} ^ {N} \mathbb {E} _ {\left(s _ {h}, a _ {h}\right) \sim \pi} \left[ b _ {h} (s, a, \mathcal {D} _ {h} ^ {(n - 1)}) \right] \\ \leq \frac {1}{N} \sum_ {n = 1} ^ {N} b _ {h} \left(s _ {h} ^ {n}, a _ {h} ^ {n}, \mathcal {D} _ {h} ^ {(n - 1)}\right) + \mathcal {O} \left(H \sqrt {\frac {\log (1 / \delta)}{N}}\right) \\ \leq \mathcal {O} \left(H \sqrt {\frac {d _ {E} (1 / N , \mathcal {F} _ {h}) \beta}{N}} + H \sqrt {\frac {\log (1 / \delta)}{N}}\right), \\ \end{array}
$$
where the second inequality uses Azuma-Hoeffding inequality and the last one uses the standard regret guarantee for eluder dimension (e.g., Lemma 2 in Russo and Van Roy [2013]).