SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Next, we show that the resulting policy  $\pi$  is near-optimal. Assume by induction on  $h = H,\ldots ,1$ , that for all  $t\in [h + 1:H]$ , all  $s\in S_t$  and  $a\in [\mathcal{A}]$
$$
v ^ {\pi} (s) \geq v ^ {\star} (s) - (H - t + 1) \left(\varepsilon / H + 2 H ^ {2} \eta_ {0}\right) \quad \text {a n d} \tag {56}
$$
$$
\left\langle \varphi (s, a), \bar {\theta} _ {t} \right\rangle \approx_ {(H - t + 1) H \eta_ {0}} q ^ {\pi} (s, a). \tag {57}
$$
To prove the above for  $t = h$  as well, take any  $s \in S_h, a \in [\mathcal{A}]$ . Introduce the random variable  $P$  that, for a trajectory following  $\mathcal{P}_{\pi^0,s,a}$ , takes as its value the index of the first Bernoulli draw of 1 (starting from index  $h + 1$ ), when the Bernoullis have means  $\tau_{\hat{G}\bar{\theta}}(S_j)$  for  $j \in [h + 1:H]$ , and takes the value  $H + 1$  if all of these Bernoullis have outcome 0. Write  $\mathbb{E}_{\pi^0,s,a,P}[\cdot ]$  for  $\mathbb{E}_{\pi^0,s,a}\mathbb{E}_P[\cdot |S_{h + 1},\dots ,R_H]$ .
$$
\begin{array}{l} \underset {\pi^ {0}, s, a} {\mathbb {E}} E _ {\hat {G} \bar {\theta}} ^ {\rightarrow} (S _ {h + 1}, \dots , R _ {H}) + \sum_ {u = h} ^ {H} R _ {u} = \underset {\pi^ {0}, s, a, P} {\mathbb {E}} D _ {\hat {G} \bar {\theta}} (S _ {P}, \dots , R _ {H}) + \sum_ {u = h} ^ {H} R _ {u} \\ = \underset {\pi^ {0}, s, a, P} {\mathbb {E}} \sum_ {u = h} ^ {P - 1} R _ {u} + \mathbb {1} \left\{P <   H + 1 \right\} C _ {\hat {G} \bar {\theta}} (S _ {P}) \\ \end{array}
$$
where we use Eq. (9). Combining with Eq. (54),
$$
\begin{array}{l} \left\langle \varphi (s, a), \bar {\theta} _ {h} \right\rangle \approx_ {H \eta_ {0}} \underset {\pi^ {0}, s, a, P} {\mathbb {E}} \sum_ {u = h} ^ {P - 1} R _ {u} + \mathbb {1} \left\{P <   H + 1 \right\} C _ {\hat {G} \bar {\theta}} (S _ {P}) \\ = \mathop{\mathbb{E}}_{\pi^{0},s,a,P}\sum_{u = h}^{P - 1}R_{u} + \mathbb{1}\left\{P <   H + 1\right\} \operatorname {clip}_{[0,H]}\left\langle \varphi (S_{P},\pi_{\hat{G}\bar{\theta}}^{+}(S_{P})),\bar{\theta}_{P}\right\rangle \\ \approx_ {(H - h) H \eta_ {0}} \underset {\pi^ {0}, s, a, P} {\mathbb {E}} \sum_ {u = h} ^ {P - 1} R _ {u} + \mathbb {1} \left\{P <   H + 1 \right\} q ^ {\pi} \big (S _ {P}, \pi_ {\hat {G} \bar {\theta}} ^ {+} (S _ {P}) \big), \\ \end{array}
$$
where we used the inductive assumption along with the fact that action-values are bounded in  $[0, H]$ . Observe also that
$$
q ^ {\pi} (s, a) = \underset {\pi^ {0}, s, a, P} {\mathbb {E}} \sum_ {u = h} ^ {P - 1} R _ {u} + \mathbb {1} \left\{P <   H + 1 \right\} q ^ {\pi} (S _ {P}, \pi_ {\hat {G} \bar {\theta}} ^ {+} (S _ {P})) ,
$$
$$
\left\langle \varphi (s, a), \bar {\theta} _ {h} \right\rangle \approx_ {(H - h + 1) H \eta_ {0}} q ^ {\pi} (s, a),
$$
proving Eq. (57) of the inductive assumption for  $t = h$ .
To show Eq. (56) for  $t = h$ , by Eq. (57) for  $t = h$  and the inductive assumption for  $t > h$ ,
$$
\left\langle \varphi (s, a), \bar {\theta} _ {h} \right\rangle \approx_ {H ^ {2} \eta_ {0}} q ^ {\pi} (s, a) \geq q ^ {\star} (s, a) - (H - h) (\varepsilon / H + 2 H ^ {2} \eta_ {0}).
$$
Either  $\pi$  chooses the action  $a^\prime$  maximizing the inner product above, for which
$$
q ^ {\pi} (s, a ^ {\prime}) \geq \max _ {a \in [ \mathcal {A} ]} q ^ {\star} (s, a) - (H - h) (\varepsilon / H + 2 H ^ {2} \eta_ {0}) - 2 H ^ {2} \eta_ {0} \geq v ^ {\star} (s) - (H - h + 1) (\varepsilon / H + 2 H ^ {2} \eta_ {0}),
$$
or it chooses action 1. This can only happen with non-zero probability if  $\tau_{\hat{G}\bar{\theta}}(s) < 1$ , in which case we have by definition that  $\mathrm{range}_Q^{\hat{G}}(s) = \mathrm{range}_Q(s) \leq \frac{\varepsilon}{\sqrt{2d}H}$ . Combining with Eq. (3) and Proposition 4.5,  $\mathrm{range}(s) \leq \frac{\varepsilon}{H}$ , and therefore, using Eq. (56) for  $t = h + 1$ , in this case
$$
\begin{array}{l} q ^ {\pi} (s, 1) \geq q ^ {\star} (s, 1) - (H - h) (\varepsilon / H + 2 H ^ {2} \eta_ {0}) \\ \geq v ^ {\star} (s) - \frac {\varepsilon}{H} - 2 \eta - (H - h) (\varepsilon / H + 2 H ^ {2} \eta_ {0}) \geq v ^ {\star} (s) - (H - h + 1) (\varepsilon / H + 2 H ^ {2} \eta_ {0}). \\ \end{array}
$$
Therefore for any choice of action  $a^\prime$  of policy  $\pi$  in state  $s$ ,  $q^{\pi}(s,a^{\prime})\geq v^{\star}(s) - (H - h + 1)(\varepsilon /H + 2H^{2}\eta_{0})$ . Therefore
$$
v ^ {\pi} (s) \geq v ^ {\star} (s) - (H - h + 1) \left(\varepsilon / H + 2 H ^ {2} \eta_ {0}\right),
$$
finishing the induction.
We thus conclude that
$$
v ^ {\pi} (s _ {1}) \geq v ^ {\star} (s _ {1}) - \varepsilon - 2 H ^ {3} \eta_ {0}.
$$
Combined with Eq. (57) of the inductive assumption, the value of Optimization Problem 4.10 can be bounded as
$$
C _ {\hat {G} \bar {\theta}} (s _ {1}) = \operatorname {c l i p} _ {[ 0, H ]} \left\langle \varphi (s _ {1}, \pi (s _ {1})), \bar {\theta} _ {1} \right\rangle \geq H ^ {2} \eta_ {0} + v ^ {\pi} (s _ {1}) \geq v ^ {\star} (s _ {1}) - 2 \varepsilon ,
$$
by assumption on  $\eta$  being relatively small (Eq. (21)).
L Deferred lemmas
Lemma L.1 (Elliptical potential, Lemma 19.4 from Lattimore and Szepesvári [2020]). Let  $V_0 \in \mathbb{R}^{d \times d}$  be positive definite and  $a_1, \ldots, a_n \in \mathbb{R}^d$  be a sequence of vectors with  $\| a_t \|_2 \leq L < \infty$  for all  $t \in [n]$ ,  $V_t = V_0 + \sum_{s \leq t} a_s a_s^\top$ . Then,
$$
\sum_ {t = 1} ^ {n} \min  \left\{1, \| a _ {t} \| _ {V _ {t - 1} ^ {- 1}} ^ {2} \right\} \leq 2 \log \left(\frac {\det  V _ {n}}{\det  V _ {0}}\right) \leq 2 d \log \left(\frac {\operatorname {T r} V _ {0} + n L ^ {2}}{d \det  (V _ {0}) ^ {1 / d}}\right).
$$
Lemma L.2. Let  $V \in \mathbb{R}^{d \times d}$  be a symmetric positive definite matrix and  $(a_{i})_{i \in [n]}$  be a sequence of  $n$ $d$ -dimensional real vectors. Let  $V_{i} = V + \sum_{j \in [i]} a_{j} a_{j}^{\top}$ . Then,
$$
\sum_ {i \in [ n ]} \| a _ {i} \| _ {V _ {i} ^ {- 1}} ^ {2} \geq \min  \left\{1, \frac {1}{2} \sum_ {i \in [ n ]} \| a _ {i} \| _ {V ^ {- 1}} ^ {2} \right\}
$$
Proof. If  $\sum_{i\in [n]}a_i a_i^\top \preceq V$ , then  $V_{i}\preceq 2V$ , and therefore
$$
\sum_ {i \in [ n ]} \| a _ {i} \| _ {V _ {i} ^ {- 1}} ^ {2} \geq \sum_ {i \in [ n ]} \| a _ {i} \| _ {2 V ^ {- 1}} ^ {2} = \frac {1}{2} \| a _ {i} \| _ {V ^ {- 1}}.
$$
Otherwise,  $\sum_{i\in [n]}a_i a_i^\top V^{-1}$  has an eigenvalue that is at least 1. As all the other eigenvalues are non-negative (as  $V$  is symmetric positive definite), we have that
$$
\sum_ {i \in [ n ]} \| a _ {i} \| _ {V ^ {- 1}} ^ {2} = \operatorname {T r} \left(\sum_ {i \in [ n ]} a _ {i} a _ {i} ^ {\top} V ^ {- 1}\right) \geq 1.
$$
M Estimation error blow-up guarantees
We borrow Assumption M.1 and Theorem M.2 from Lattimore and Szepesvári [2020] and refer the reader to the book for the corresponding proof.
Assumption M.1 (Prerequisites for Theorem M.2). Let  $\lambda > 0$ . For  $k \in \mathbb{N}^+$ , let  $A_k$  be random variables taking values in  $\mathbb{R}^d$ . For some  $\theta_\star \in \mathbb{R}^d$ , let  $X_k = \langle A_k, \theta_\star \rangle + \eta_k$  for all  $k \in \mathbb{N}^+$ . Here,  $\eta_k$  is a conditionally 1-subgaussian random variable ("noise"), i.e. it satisfies:
$$
f o r a l l \alpha \in \mathbb {R} a n d t \geq 1, \quad \mathbb {E} [ \exp (\alpha \eta_ {k}) | \mathcal {F} _ {k - 1} ] \leq \exp \left(\frac {\alpha^ {2}}{2}\right) \quad a. s.,
$$
where  $\mathcal{F}_{k - 1}$  is such that  $A_{1},X_{1},\ldots ,A_{k - 1},X_{k - 1},A_{k}$  are  $\mathcal{F}_{k - 1}$ -measurable.
Theorem M.2 (Lattimore and Szepesvári [2020], Theorem 20.5). Let  $\zeta \in (0,1)$ . Under Assumption M.1, with probability at least  $1 - \zeta$ , it holds that for all  $k \in \mathbb{N}$
$$
\left\| \hat {\theta} _ {k} - \theta_ {\star} \right\| _ {V _ {k} (\lambda)} <   \sqrt {\lambda} \| \theta_ {\star} \| _ {2} + \sqrt {2 \log \left(\frac {1}{\zeta}\right) + \log \left(\frac {\det V _ {k} (\lambda)}{\lambda^ {d}}\right)},
$$
where for  $k\in \mathbb{N}$
$$
V _ {k} (\lambda) = \lambda I + \sum_ {s = 1} ^ {k} A _ {s} A _ {s} ^ {\top}
$$
$$
\hat {\theta} _ {k} = V _ {k} (\lambda) ^ {- 1} \sum_ {s = 1} ^ {k} X _ {s} A _ {s}
$$
We generalize this theorem to handle non-zero-mean noise with parametrized subgaussianity. To handle non-zero-mean noise, we use [Zanette et al., 2020, Lemma 8]. We state the lemma here and refer the reader to Zanette et al. [2020] for the proof:
Lemma M.3 (Zanette et al. [2020], Lemma 8). For  $n \in N^{+}$ , let  $\{A_i\}_{i=1,\dots,n}$  be any sequence of vectors in  $\mathbb{R}^d$  and  $\{\Delta_i\}_{i=1,\dots,n}$  be any sequence of scalars such that  $|\Delta_i| \leq \xi \in \mathbb{R}$  with  $\xi \geq 0$ . For any  $\lambda \geq 0$  and  $V(\lambda) = \sum_{i=1}^{n} A_i A_i^\top + \lambda I$  we have:
$$
\left\| \sum_ {i = 1} ^ {n} A _ {i} \Delta_ {i} \right\| _ {V (\lambda) ^ {- 1}} ^ {2} \leq n \xi^ {2}
$$
Lemma M.4. Let  $\zeta \in (0,1)$ ,  $\lambda > 0$ ,  $\sigma > 0$ , and  $\xi \geq 0$ . For  $k \in \mathbb{N}^+$ , let  $A_k$  be random variables taking values in  $\mathbb{R}^d$ . For some  $\theta_\star \in \mathbb{R}^d$ , let  $\tilde{X}_k = \langle A_k, \theta_\star \rangle + \eta_k$  for all  $k \in \mathbb{N}^+$ . Here,  $\eta_k$  is a conditionally  $\sigma$ -subgaussian random variable, i.e. it satisfies:
$$
f o r a l l \alpha \in \mathbb {R} a n d t \geq 1, \quad \mathbb {E} [ \exp (\alpha \eta_ {k}) | \mathcal {F} _ {k - 1} ] \leq \exp \left(\frac {\alpha^ {2} \sigma^ {2}}{2}\right) \quad a. s.,
$$
where  $\mathcal{F}_{k-1}$  is such that  $A_1, \tilde{X}_1, \ldots, A_{k-1}, \tilde{X}_{k-1}, A_k$  are  $\mathcal{F}_{k-1}$ -measurable. With probability at least  $1 - \zeta$ , it holds that for any sequence  $\{\Delta_i\}_{i=1,\ldots}$  such that  $|\Delta_i| \leq \xi$ , for all  $k \in \mathbb{N}$ ,
$$
\left\| \hat {\theta} _ {k} - \theta_ {\star} \right\| _ {V _ {k} (\lambda)} <   \sqrt {\lambda} \| \theta_ {\star} \| _ {2} + \xi \sqrt {k} + \sigma \sqrt {2 \log \left(\frac {1}{\zeta}\right) + \log \left(\frac {\det V _ {k} (\lambda)}{\lambda^ {d}}\right)}.
$$
$$
X _ {k} = \tilde {X} _ {k} + \Delta_ {k}
$$
Proof. Let  $X_{k}^{\prime} = (X_{k} - \Delta_{k}) / \sigma_{k}$ ,  $A_{k}^{\prime} = A_{k} / \sigma_{k}$ ,  $\lambda^{\prime} = \lambda / \sigma_{k}^{2}$ , and  $\theta_{\star}^{\prime} = \theta_{\star}$ ,  $V_{k}^{\prime}(\lambda^{\prime}) = \lambda^{\prime}I + \sum_{s=1}^{k} A_{s}^{\prime}A_{s}^{\prime}^{\top}$  and  $\hat{\theta}_{k}^{\prime} = V_{k}^{\prime}(\lambda^{\prime})^{-1}\sum_{s=1}^{k} X_{s}^{\prime}A_{s}^{\prime}$ . By assumption,  $X_{k}^{\prime}, A_{k}^{\prime}, \lambda^{\prime}$  and  $\theta_{\star}^{\prime}$  then satisfy Assumption M.1. Therefore by applying Theorem M.2, with probability at least  $1 - \zeta$ , it holds that for all  $k \in \mathbb{N}$ ,
$$
\left\| \hat {\theta} _ {k} ^ {\prime} - \theta_ {\star} \right\| _ {V _ {k} ^ {\prime} (\mathcal {X})} <   \sqrt {\lambda^ {\prime}} \| \theta_ {\star} \| _ {2} + \sqrt {2 \log \left(\frac {1}{\zeta}\right) + \log \left(\frac {\det V _ {k} ^ {\prime} (\mathcal {X} ^ {\prime})}{\lambda^ {\prime d}}\right)}.
$$
Under this high-probability event, since  $V_{k}^{\prime}(\lambda^{\prime}) = V_{k}(\lambda) / \sigma^{2}$ , substituting into the previous display yields
$$
\left\| \hat {\theta} _ {k} ^ {\prime} - \theta_ {\star} \right\| _ {V _ {k} (\lambda)} <   \sqrt {\lambda} \| \theta_ {\star} \| _ {2} + \sigma \sqrt {2 \log \left(\frac {1}{\zeta}\right) + \log \left(\frac {\det V _ {k} (\lambda)}{\lambda^ {d}}\right)}. \tag {58}
$$
Take any sequence  $\{\Delta_i\}_{i = 1,\dots}$  such that  $|\Delta_i|\leq \xi$  and apply the triangle inequality:
$$
\left\| \hat {\theta} _ {k} - \theta_ {\star} \right\| _ {V _ {k} (\lambda)} \leq \left\| \hat {\theta} _ {k} ^ {\prime} - \theta_ {\star} \right\| _ {V _ {k} (\lambda)} + \left\| \hat {\theta} _ {k} ^ {\prime} - \hat {\theta} _ {k} \right\| _ {V _ {k} (\lambda)}, \tag {59}
$$
so it remains to bound  $\left\| \hat{\theta}_k' - \hat{\theta}_k\right\|_{V_k(\lambda)}$
$$
\begin{array}{l} \left\| \hat {\theta} _ {k} ^ {\prime} - \hat {\theta} _ {k} \right\| _ {V _ {k} (\lambda)} = \left\| V _ {k} ^ {\prime} \left(\lambda^ {\prime}\right) ^ {- 1} \sum_ {s = 1} ^ {k} X _ {s} ^ {\prime} A _ {s} ^ {\prime} - V _ {k} (\lambda) ^ {- 1} \sum_ {s = 1} ^ {k} X _ {s} A _ {s} \right\| _ {V _ {k} (\lambda)} \\ = \left\| V _ {k} (\lambda) ^ {- 1} \sum_ {s = 1} ^ {k} \left(X _ {s} - \Delta_ {s}\right) A _ {s} - V _ {k} (\lambda) ^ {- 1} \sum_ {s = 1} ^ {k} X _ {s} A _ {s} \right\| _ {V _ {k} (\lambda)} \tag {60} \\ = \left\| V _ {k} (\lambda) ^ {- 1} \sum_ {s = 1} ^ {k} \Delta_ {s} A _ {s} \right\| _ {V _ {k} (\lambda)} = \left\| \sum_ {s = 1} ^ {k} \Delta_ {s} A _ {s} \right\| _ {V _ {k} (\lambda) ^ {- 1}} \\ \leq \sqrt {k} \xi , \\ \end{array}
$$
where the final inequality uses Lemma M.3. The proof is finished by plugging in the bounds of Eqs. (58) and (60) into the triangle inequality of Eq. (59).