SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \Gamma_ {h} ^ {k} \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) \leq \beta \sum_ {h = 1} ^ {H} \left(K \cdot \sum_ {k = 1} ^ {K} \phi \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) ^ {\top} \left(\Lambda_ {h} ^ {k}\right) ^ {- 1} \phi \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right)\right) ^ {1 / 2} \\ \leq \beta \sum_ {h = 1} ^ {H} \left[ 2 K \cdot \log \left(\frac {\det \left(\Lambda_ {h} ^ {K + 1}\right)}{\det \left(\Lambda_ {h} ^ {1}\right)}\right) \right] ^ {1 / 2}, \tag {B.33} \\ \end{array}
$$
where the last inequality uses the elliptical potential lemma (Lemma D.5). For any  $h \in [H]$ , we have
$$
\Lambda_ {h} ^ {1} = \lambda \cdot I _ {d}, \quad \Lambda_ {h} ^ {K + 1} = \sum_ {k = 1} ^ {K} \phi \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) \phi \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) ^ {\top} + \lambda I _ {d} \preceq (K + \lambda) \cdot I _ {d}, \tag {B.34}
$$
where the inequality uses the fact that  $\| \phi(\cdot, \cdot) \|_2 \leq 1$ . Plugging (B.34) into (B.33), we have
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \Gamma_ {h} ^ {k} \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) \leq H \beta \cdot \left[ 2 d K \cdot \log \left(\frac {K + \lambda}{\lambda}\right) \right] ^ {1 / 2}.
$$
Together with the facts that  $\lambda = 1$ ,  $\iota = \log(dHK|\mathcal{A}| / \delta)$  with  $\delta \in (0,1]$ , and  $\beta = \mathcal{O}(d^{1/4}HK^{1/4}\iota^{1/2})$ , we conclude the proof of Lemma B.4.
We also need the following lemma to connect the quantity  $\sum_{k=1}^{K} \sum_{h=1}^{H} \Gamma_h^k(x_h^k, a_h^k)$  in Lemma B.4 and our target  $\sum_{k=1}^{K} \sum_{h=1}^{H} \min\{H, \Gamma_h^{t_k}(x_h^k, a_h^k)\}$ .
Lemma B.5. For any  $h \in [H]$ , we define the set  $\mathcal{E}_h$  as
$$
\mathcal {E} _ {h} = \left\{k: \Gamma_ {h} ^ {t _ {k}} \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) / \Gamma_ {h} ^ {k} \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) > 2 \right\}. \tag {B.35}
$$
$$
\left| \mathcal {E} _ {h} \right| \leq \mathcal {O} \left(d ^ {5 / 2} K ^ {1 / 2} \cdot \iota\right).
$$
Proof. For  $k \in \mathcal{E}_h$ , there exists  $i \in [L]$  such that  $k_i \leq k < k_{i+1}$ . Then we know  $t_k = k_i$ , which further implies that
$$
\begin{array}{l} \log \left(\frac {\det  \left(\Lambda_ {h} ^ {k _ {i + 1}}\right)}{\det  \left(\Lambda_ {h} ^ {k _ {i}}\right)}\right) \geq \log \left(\frac {\det  \left(\Lambda_ {h} ^ {k}\right)}{\det  \left(\Lambda_ {h} ^ {k _ {i}}\right)}\right) \geq \log \left(\frac {\phi \left(x _ {h} ^ {k} , a _ {h} ^ {k}\right) ^ {\top} \left(\Lambda_ {h} ^ {k _ {i}}\right) ^ {- 1} \phi \left(x _ {h} ^ {k} , a _ {h} ^ {k}\right)}{\phi \left(x _ {h} ^ {k} , a _ {h} ^ {k}\right) ^ {\top} \left(\Lambda_ {h} ^ {k}\right) ^ {- 1} \phi \left(x _ {h} ^ {k} , a _ {h} ^ {k}\right)}\right) \\ = 2 \log \left(\frac {\Gamma_ {h} ^ {k _ {i}} (x _ {h} ^ {k} , a _ {h} ^ {k})}{\Gamma_ {h} ^ {k} (x _ {h} ^ {k} , a _ {h} ^ {k})}\right), \\ \end{array}
$$
where the first inequality follows from the fact that  $\Lambda_h^{k_{i+1}} \succeq \Lambda_h^k$ , the second inequality uses Lemma D.6, and the last equality is obtained by the definitions of  $\Gamma_h^k$  and  $\Gamma_h^{k_i}$  in (B.31). Together with the definition of  $\mathcal{E}_h$  in (B.35), we further obtain that
$$
\log \left(\frac {\det \left(\Lambda_ {h} ^ {k _ {i + 1}}\right)}{\det \left(\Lambda_ {h} ^ {k _ {i}}\right)}\right) \geq 2 \log 2. \tag {B.36}
$$
Meanwhile, we have
$$
\sum_ {i = 1} ^ {L} \log \left(\frac {\det \left(\Lambda_ {h} ^ {k _ {i + 1}}\right)}{\det \left(\Lambda_ {h} ^ {k _ {i}}\right)}\right) = \log \left(\frac {\det \left(\Lambda_ {h} ^ {K + 1}\right)}{\det \left(\Lambda_ {h} ^ {1}\right)}\right) \leq d \cdot \log \left(\frac {K + \lambda}{\lambda}\right), \tag {B.37}
$$
where the last inequality follows from the facts that
$$
\Lambda_ {h} ^ {1} = \lambda \cdot I _ {d}, \qquad \Lambda_ {h} ^ {K + 1} = \sum_ {k = 1} ^ {K} \phi \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) \phi \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) ^ {\top} + \lambda I _ {d} \preceq (K + \lambda) \cdot I _ {d}.
$$
Here the last inequality uses  $\| \phi (\cdot ,\cdot)\| _2\leq 1$ . Combining (B.36) and (B.37), we have
$$
\left| \left\{i \in [ L ]: \log \left(\frac {\det \left(\Lambda_ {h} ^ {k _ {i + 1}}\right)}{\det \left(\Lambda_ {h} ^ {k _ {i}}\right)}\right) \right\} \right| \leq \frac {d \log ((K + \lambda) / \lambda)}{2 \log 2}. \tag {B.38}
$$
Since each batch contains  $B$  episodes, we obtain that
$$
\left| \mathcal {E} _ {h} \right| \leq B \cdot \left| \left\{i \in [ L ]: \log \left(\frac {\operatorname* {d e t} \left(\Lambda_ {h} ^ {k _ {i + 1}}\right)}{\operatorname* {d e t} \left(\Lambda_ {h} ^ {k _ {i}}\right)}\right) \right\} \right| \leq \mathcal {O} \left(d ^ {5 / 2} K ^ {1 / 2} \cdot \iota\right),
$$
where the last inequality uses  $\lambda = 1$ ,  $\iota = \log(dHK|\mathcal{A}| / \delta)$ ,  $B = \sqrt{d^3K}$  and (B.38). Therefore, we conclude the proof of Lemma B.5.
Back to our proof of Lemma 4.5, we have
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \min  \left\{H, \Gamma_ {h} ^ {t _ {k}} \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) \right\} = \sum_ {h = 1} ^ {H} \sum_ {k \in \mathcal {E} _ {h}} \min  \left\{H, \Gamma_ {h} ^ {t _ {k}} \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) \right\} + \sum_ {h = 1} ^ {H} \sum_ {k \notin \mathcal {E} _ {h}} \min  \left\{H, \Gamma_ {h} ^ {t _ {k}} \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) \right\}. \tag {B.39}
$$
For any  $h\in [H]$  and  $k\notin \mathcal{E}_h$  , by the definition of  $\mathcal{E}_h$  , we have
$$
\min  \left\{H, \Gamma_ {h} ^ {t _ {k}} \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) \right\} \leq \min  \left\{H, 2 \Gamma_ {h} ^ {k} \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) \right\} \leq 2 \Gamma_ {h} ^ {k} \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right). \tag {B.40}
$$
Combining (B.39), (B.40), and the fact that  $\min \{H, \Gamma_h^{t_k}(x_h^k, a_h^k)\} \leq H$ , we have
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \min  \left\{H, \Gamma_ {h} ^ {t _ {k}} \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) \right\} \leq H \sum_ {h = 1} ^ {H} | \mathcal {E} _ {h} | + \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \Gamma_ {h} ^ {k} \left(x _ {h} ^ {k}, a _ {h} ^ {k}\right) \\ \leq \mathcal {O} \left(d ^ {3 / 4} H ^ {2} K ^ {3 / 4} \cdot \iota\right) + \mathcal {O} \left(d ^ {5 / 2} H ^ {2} K ^ {1 / 2} \cdot \iota\right), \\ \end{array}
$$
where the last inequality uses Lemmas B.4 and B.5. Therefore, we conclude the proof of Lemma 4.5.
C Proof for Concentration of Self-Normalized Processes
Proof of Lemma B.3. By the previous concentration lemma of self-normalized process (Lemma D.3), for any  $\delta' \in (0,1], \varepsilon > 0$ , and  $(k,h) \in [K] \times [H]$  we have
$$
\begin{array}{l} \left\| \sum_ {\tau = 1} ^ {t _ {k} - 1} \phi \left(x _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \cdot \left(V _ {h + 1} ^ {t _ {k}} \left(x _ {h + 1} ^ {\tau}\right) - \left(\mathbb {P} _ {h} V _ {h + 1} ^ {t _ {k}}\right) \left(x _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right\| _ {\left(\Lambda_ {h} ^ {t _ {k}}\right) ^ {- 1}} ^ {2} \\ \leq 4 H ^ {2} \cdot \left[ \frac {d}{2} \log \left(\frac {k + \lambda}{\lambda}\right) + \log \left(\frac {\mathcal {N} _ {\varepsilon} \left(\mathcal {V} _ {h + 1} ^ {k}\right)}{\delta^ {\prime}}\right) \right] + \frac {8 k ^ {2} \varepsilon^ {2}}{\lambda} \tag {C.1} \\ \end{array}
$$
with probability  $1 - \delta'$ . Here  $\mathcal{N}_{\varepsilon}(\mathcal{V}_{h+1}^{k})$  is the  $\varepsilon$ -covering number of function class  $\mathcal{V}_{h+1}^{k}$ , which is defined by
$$
\mathcal {V} _ {h + 1} ^ {k} = \left\{V (\cdot) = \langle Q (\cdot , \cdot), \pi (\cdot | \cdot) \rangle : Q \in \mathcal {Q} _ {h + 1} ^ {k}, \pi \in \Pi_ {h + 1} ^ {k} \right\}, \tag {C.2}
$$
where  $\mathcal{Q}_{h + 1}^k$  and  $\Pi_{h + 1}^{k}$  are Q-function class and policy class, respectively. Specifically,  $\mathcal{Q}_{h + 1}^k$  is a function class with the following parametric form
$$
\mathcal {Q} _ {h + 1} ^ {k} = \left\{\bar {r} _ {h} ^ {t _ {k}} + \left\{\phi^ {\top} w + \beta \cdot \left(\phi^ {\top} \Lambda^ {- 1} \phi\right) ^ {1 / 2}, H - h \right\} ^ {+}: \| w \| _ {2} \leq H \sqrt {d K / \lambda}, \lambda_ {\min } (\Lambda) \geq \lambda \right\}. \tag {C.3}
$$
Here we use  $\| w_{h + 1}^k\| _2\leq H\sqrt{dK / \lambda}$  (Lemma D.1) and  $\lambda_{\mathrm{min}}(\Lambda_{h + 1}^k)\geq \lambda$  to ensure that  $Q_{h + 1}^{k}\in \mathcal{Q}_{h + 1}^{k}$ . Meanwhile, the policy class  $\Pi_{h + 1}^{k}$  is defined as
$$
\Pi_ {h + 1} ^ {k} = \left\{\pi (\cdot | \cdot) \propto \exp \left(\sum_ {i = 1} ^ {l} \alpha Q _ {i} (\cdot , \cdot)\right): Q _ {i} \in \mathcal {Q} _ {h + 1} ^ {k _ {i}}, \forall i \in [ l ] \right\}, \text {w h e r e} l = \max  \{i: k _ {i} <   t _ {k} \}. \tag {C.4}
$$
Notable, here  $l \leq L$  since our algorithm only has  $L$  batches. Also,  $\pi_{h+1}^{t_k}$  in (C.2) belongs to this policy class since it takes the following form
$$
\pi_ {h + 1} ^ {t _ {k}} (a \mid x) = \frac {\exp \left(\sum_ {i = 1} ^ {l} \alpha Q _ {h + 1} ^ {k _ {i}} (x , a)\right)}{\sum_ {a ^ {\prime} \in \mathcal {A}} \exp \left(\sum_ {i = 1} ^ {l} \alpha Q _ {h + 1} ^ {k _ {i}} (x , a ^ {\prime})\right)},
$$
For policy class defined in (C.4), we define its covering number with respect to the following distance
$$
\operatorname {d i s t} (\pi , \pi^ {\prime}) = \sup  _ {x \in \mathcal {S}} \| \pi (\cdot | x) - \pi^ {\prime} (\cdot | x) \| _ {1}.
$$
The following lemma connects the covering number of value class in (C.2) to covering numbers of the Q-function class in (C.3) and the policy class in (C.4).
Lemma C.1. It holds that
$$
\mathcal {N} _ {\varepsilon} (\mathcal {V} _ {h + 1} ^ {k}) \leq \mathcal {N} _ {\varepsilon / 2} (\mathcal {Q} _ {h + 1} ^ {k}) \cdot \mathcal {N} _ {\varepsilon / (2 H)} (\Pi_ {h + 1} ^ {k}).
$$
Proof. See §C.1 for a detailed proof.
Following the standard covering argument (Lemma D.4), we can derive an upper bound for  $\mathcal{N}_{\varepsilon /2}(\mathcal{Q}_{h + 1}^k)$ . However,  $\mathcal{N}_{\varepsilon /(2H)}(\Pi_{h + 1}^k)$  is relatively difficult to bound since the log-covering number of a  $|\mathcal{A}|$ -dimensional probability distribution is  $\widetilde{\mathcal{O}} (|\mathcal{A}|)$ . Fortunately, we have the following lemma that utilizes the structure of policy class (C.4) and converts the covering number of the policy class to the covering numbers of several Q-function classes.
Lemma C.2. It holds that
$$
\mathcal {N} _ {\varepsilon / (2 H)} (\Pi_ {h + 1} ^ {k}) \leq \prod_ {i = 1} ^ {l} \mathcal {N} _ {\varepsilon^ {2} / (1 6 \alpha l H ^ {2})} (\mathcal {Q} _ {h + 1} ^ {k _ {i}})
$$
Proof. See §C.2 for a detailed proof.
Note that  $\lambda = 1, l \leq L, L = K / B, B = \sqrt{d^3K}, \alpha = \sqrt{2B\log|\mathcal{A}| / (KH^2)}, \iota = \log(dHK|\mathcal{A}| / \delta)$ , and  $\beta = \mathcal{O}(d^{1/4}HK^{1/4}\iota^{1/2})$ . Meanwhile, let  $\varepsilon = 1 / K$  and  $\delta' = \delta / 2$ . Combining (C.1) with Lemmas C.1, C.2, and D.4, we have
$$
\begin{array}{l} \left\| \sum_ {\tau = 1} ^ {t _ {k} - 1} \phi \left(x _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \cdot \left(V _ {h + 1} ^ {t _ {k}} \left(x _ {h + 1} ^ {\tau}\right) - \left(\mathbb {P} _ {h} V _ {h + 1} ^ {t _ {k}}\right) \left(x _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right)\right) \right\| _ {\left(\Lambda_ {h} ^ {t _ {k}}\right) ^ {- 1}} \leq \mathcal {O} (\sqrt {d ^ {2} H ^ {2} L \cdot \iota}) \\ = \mathcal {O} (d ^ {1 / 4} H K ^ {1 / 4} \iota^ {1 / 2}), \\ \end{array}
$$
which concludes the proof of Lemma B.3.
C.1 Proof of Lemma C.1
Proof. Suppose (i)  $\mathcal{Q}_{\varepsilon /2,h + 1}^k$  is an  $\varepsilon /2$  -net of  $\mathcal{Q}_{h + 1}^k$  with  $|\mathcal{Q}_{\varepsilon /2,h + 1}^k | = \mathcal{N}_{\varepsilon /2}(\mathcal{Q}_{h + 1}^k)$ ; and (ii)  $\Pi_{\varepsilon /(2H),h + 1}^k$  is an  $\varepsilon /(2H)$  -net of  $\Pi_{h + 1}^k$  with  $|\Pi_{\varepsilon /(2H),h + 1}^k | = \mathcal{N}_{\varepsilon /(2H)}(\Pi_{h + 1}^k)$ . Then we show  $\mathcal{Q}_{\varepsilon /2,h + 1}^k\times \Pi_{\varepsilon /(2H),h + 1}^k$  induces an  $\varepsilon$  -net of  $\nu_{h + 1}^k$ , and thus obtaining the desired result. Specifically, for any  $V = \langle Q(\cdot ,\cdot),\pi (\cdot |\cdot)\rangle$  with  $(Q,\pi)\in \mathcal{Q}_{h + 1}^{k}\times \Pi_{h + 1}^{k}$ , we can find  $(Q^{\prime},\pi^{\prime})\in$ $\mathcal{Q}_{\varepsilon /2,h + 1}^k\times \Pi_{\varepsilon /(2H),h + 1}^k$  such that
$$
\sup  _ {(x, a) \in \mathcal {S} \times \mathcal {A}} | Q (x, a) - Q ^ {\prime} (x, a) | \leq \varepsilon / 2, \quad \sup  _ {x \in \mathcal {S}} \| \pi (\cdot | x) - \pi^ {\prime} (\cdot | x) \| _ {1} \leq \varepsilon / (2 H). \tag {C.5}
$$
Let  $V^{\prime} = \langle Q^{\prime}(\cdot ,\cdot),\pi^{\prime}(\cdot |\cdot)\rangle$  , we have
$$
\begin{array}{l} \sup  _ {x \in \mathcal {S}} | V (x) - V ^ {\prime} (x) | = \sup  _ {x \in \mathcal {S}} \left| \left\langle Q (x, \cdot), \pi (\cdot \mid x) \right\rangle - \left\langle Q ^ {\prime} (x, \cdot), \pi^ {\prime} (\cdot \mid x) \right\rangle \right| \\ \leq \sup  _ {x \in \mathcal {S}} \left| \left\langle Q (x, \cdot) - Q ^ {\prime} (x, \cdot), \pi (\cdot | x) \right\rangle \right| + \sup  _ {x \in \mathcal {S}} \left| \left\langle Q ^ {\prime} (x, \cdot), \pi (\cdot | x) - \pi^ {\prime} (\cdot | x) \right\rangle \right| \\ \leq \sup  _ {(x, a) \in \mathcal {S} \times \mathcal {A}} | Q (x, a) - Q ^ {\prime} (x, a) | + H \cdot \sup  _ {x \in \mathcal {S}} \| \pi (\cdot | x) - \pi^ {\prime} (\cdot | x) \| _ {1} \\ \leq \varepsilon / 2 + \varepsilon / 2 = \varepsilon , \\ \end{array}
$$
where the first inequality follows from the triangle inequality, the second inequality uses Cauchy-Schwarz inequality and the fact that  $\| Q^{\prime}\|_{\infty}\leq H$ , and the last inequality follows from (C.5). Therefore, we conclude the proof of Lemma C.1.
C.2 Proof of Lemma C.2
Proof. Suppose  $\mathcal{Q}_{\varepsilon^2/(16\alpha lH^2),h+1}^{k_i}$  is the minimum  $\varepsilon^2/(16\alpha lH^2)$ -net of  $\mathcal{Q}_{h+1}^{k_i}$  for all  $i \in [l]$ . Then for any  $\pi \propto \exp(\alpha \sum_{i=1}^{l} Q_i)$  with  $Q_i \in \mathcal{Q}_{h+1}^{k_i}$  for all  $i \in [l]$ , we can choose  $\{Q_i' \in \mathcal{Q}_{\varepsilon^2/(16\alpha lH^2),h+1}^{k_i}\}$  such that
$$
\sup  _ {(x, a) \in \mathcal {S} \times \mathcal {A}} | Q _ {i} (x, a) - Q _ {i} ^ {\prime} (x, a) | \leq \frac {\varepsilon^ {2}}{1 6 \alpha l H ^ {2}}, \quad \forall i \in [ l ]. \tag {C.6}
$$
$$
\sup  _ {(x, a) \in \mathcal {S} \times \mathcal {A}} \left| \alpha \sum_ {i = 1} ^ {l} Q _ {i} (x, a) - \alpha \sum_ {i = 1} ^ {l} Q _ {i} ^ {\prime} (x, a) \right| \leq \alpha \sum_ {i = 1} ^ {l} \sup  _ {(x, a) \in \mathcal {S} \times \mathcal {A}} | Q _ {i} (x, a) - Q _ {i} ^ {\prime} (x, a) | \leq \frac {\varepsilon^ {2}}{1 6 H ^ {2}}, \tag {C.7}
$$
where the first inequality uses the triangle inequality and the last inequality follows from (C.6). Then we use the following lemma to establish the upper bound for  $\sup_{x\in S}\| \pi (\cdot \mid x) - \pi^{\prime}(\cdot \mid x)\| _1$
Lemma C.3. For  $\pi, \pi' \in \Delta(\mathcal{X})$  and  $Q, Q': \mathcal{X} \mapsto \mathbb{R}^+$ , if  $\pi(\cdot) \propto \exp(Q(\cdot))$  and  $\pi'(\cdot) \propto \exp(Q'(\cdot))$ , we have
$$
\left\| \pi - \pi^ {\prime} \right\| _ {1} \leq 2 \sqrt {\left\| Q - Q ^ {\prime} \right\| _ {\infty}}.
$$
Proof. See §C.3 for a detailed proof.
By Lemma C.3, we have
$$
\sup  _ {x \in \mathcal {S}} \| \pi (\cdot | x) - \pi^ {\prime} (\cdot | x) \| _ {1} \leq 2 \sqrt {\sup  _ {(x , a) \in \mathcal {S} \times \mathcal {A}} \left| \alpha \sum_ {i = 1} ^ {l} Q _ {i} (x , a) - \alpha \sum_ {i = 1} ^ {l} Q _ {i} ^ {\prime} (x , a) \right|} \leq \frac {\varepsilon}{2 H},
$$
where the last inequality follows from (C.7). Therefore, we have
$$
\mathcal {N} _ {\varepsilon / (2 H)} (\Pi_ {h + 1} ^ {k}) \leq \prod_ {i = 1} ^ {l} \mathcal {N} _ {\varepsilon^ {2} / (1 6 \alpha l H ^ {2})} (\mathcal {Q} _ {h + 1} ^ {k _ {i}}),
$$
which concludes the proof of Lemma C.2.
C.3 Proof of Lemma C.3
Proof. Since  $\pi(\cdot) \propto \exp(Q(\cdot))$  and  $\pi'(\cdot) \propto \exp(Q'(\cdot))$ , we have for any  $x \in \mathcal{X}$ :
$$
\frac {\pi (x)}{\pi^ {\prime} (x)} = \frac {\exp (Q (x))}{\exp (Q ^ {\prime} (x))} \cdot \frac {\sum_ {x ^ {\prime} \in \mathcal {X}} \exp (Q ^ {\prime} (x ^ {\prime}))}{\sum_ {x ^ {\prime} \in \mathcal {X}} \exp (Q (x ^ {\prime}))}.
$$
Note that for any  $x\in \mathcal{X}$  we have
$$
\left\{ \begin{array}{l} \frac {\exp (Q (x))}{\exp (Q ^ {\prime} (x))} = \exp (Q (x) - Q ^ {\prime} (x)) \leq \exp (\| Q - Q ^ {\prime} \| _ {\infty}) \\ \frac {\exp (Q ^ {\prime} (x))}{\exp (Q (x))} = \exp (Q ^ {\prime} (x) - Q (x)) \leq \exp (\| Q - Q ^ {\prime} \| _ {\infty}) \end{array} \right.,
$$
$$
\frac {\pi (x)}{\pi^ {\prime} (x)} \leq \exp (\| Q - Q ^ {\prime} \| _ {\infty}) \cdot \frac {\exp (\| Q - Q ^ {\prime} \| _ {\infty}) \cdot \sum_ {x ^ {\prime} \in \mathcal {X}} \exp (Q (x ^ {\prime}))}{\sum_ {x ^ {\prime} \in \mathcal {X}} \exp (Q (x ^ {\prime}))} = \exp (2 \| Q - Q ^ {\prime} \| _ {\infty}).
$$
$$
\operatorname {K L} \left(\pi \| \pi^ {\prime}\right) = \sum_ {x \in \mathcal {X}} \pi (x) \log \frac {\pi (x)}{\pi^ {\prime} (x)} \leq 2 \| Q - Q ^ {\prime} \| _ {\infty} \cdot \sum_ {x \in \mathcal {X}} \pi (x) = 2 \| Q - Q ^ {\prime} \| _ {\infty}.
$$
Finally, by Pinsker's inequality, we have
$$
\| \pi - \pi^ {\prime} \| _ {1} \leq \sqrt {2 \cdot \mathrm {K L} (\pi \| \pi^ {\prime})} \leq 2 \sqrt {\| Q - Q ^ {\prime} \| _ {\infty}},
$$
which concludes the proof of Lemma C.3.
D Auxiliary Lemmas
Lemma D.1. For any  $(i,h)\in [L]\times [H]$ , the linear coefficient  $w_{h}^{k_{i}}$  defined in Line 12 of Algrotihm 1 satisfies
$$
\left\| w _ {h} ^ {k _ {i}} \right\| \leq H \sqrt {d K / \lambda}.
$$
Proof. Fix  $(i,h)\in [L]\times [H]$ . Our proof follows the proof of Lemma B.2 in Jin et al. [17]. For any  $v\in \mathbb{R}^d$  with  $\| v\| _2 = 1$ , by the definition of  $w_{h}^{k_{i}}$  we have
$$
\left| v ^ {\top} w _ {h} ^ {k _ {i}} \right| = \left| v ^ {\top} \left(\Lambda_ {h} ^ {k _ {i}}\right) ^ {- 1} \sum_ {\tau = 1} ^ {k _ {i} - 1} \phi \left(x _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \cdot V _ {h + 1} ^ {k _ {i}} \left(x _ {h + 1} ^ {\tau}\right) \right|.
$$
Since  $\| V_{h + 1}^{k_i}\|_{\infty}\leq H$  , we further have
$$
\begin{array}{l} \left| v ^ {\top} w _ {h} ^ {k _ {i}} \right| \leq H \sum_ {\tau = 1} ^ {k _ {i} - 1} \left| v ^ {\top} \left(\Lambda_ {h} ^ {k _ {i}}\right) ^ {- 1} \phi \left(x _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \right| \\ \leq H \cdot \left[ \sum_ {\tau = 1} ^ {k _ {i} - 1} v ^ {\top} \left(\Lambda_ {h} ^ {k _ {i}}\right) ^ {- 1} v \right] ^ {1 / 2} \cdot \left[ \sum_ {\tau = 1} ^ {k _ {i} - 1} \phi \left(x _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) ^ {\top} \left(\Lambda_ {h} ^ {k _ {i}}\right) ^ {- 1} \phi \left(x _ {h} ^ {\tau}, a _ {h} ^ {\tau}\right) \right] ^ {1 / 2} \\ \leq H \sqrt {d K / \lambda}, \\ \end{array}
$$
where the second inequality is obtained by Cauchy-Schwarz inequality, and the last inequality uses  $\lambda I_d \preceq \Lambda_h^{k_i}$ ,  $k_i \leq K$ ,  $\|v\|_2 = 1$  and Lemma D.2. Hence, we have
$$
\| w _ {h} ^ {k _ {i}} \| _ {2} = \sup  _ {\| v \| _ {2} = 1} | v ^ {\top} w _ {h} ^ {k _ {i}} | \leq H \sqrt {d K / \lambda},
$$
which concludes the proof of Lemma D.1.
Lemma D.2. Let  $\Lambda_t = \lambda \cdot I_d + \sum_{i=1}^t \phi_i \phi_i^\top$  with  $\phi_i \in \mathbb{R}^d$  and  $\lambda > 0$ . Then we have
$$
\sum_ {i = 1} ^ {t} \phi_ {i} (\Lambda_ {t}) ^ {- 1} \phi_ {i} \leq d.
$$
Proof. See Lemma D.1 in Jin et al. [17] for a detailed proof.
Lemma D.3. Let  $\{x_{\tau} \in S\}_{\tau=1}^{\infty}$  and  $\{\phi_{\tau} \in \mathbb{R}^{d}\}_{\tau=1}^{\infty}$  with  $\|\phi_{\tau}\|_{2} \leq 1$  be stochastic processes adapted to the filtration  $\{\mathcal{F}_{\tau}\}_{\tau=1}^{\infty}$ . Let  $\Lambda_{k} = \sum_{\tau=1}^{k-1} \phi_{\tau} \phi_{\tau}^{\top} + \lambda \cdot I_{d}$ . Then for any  $\delta \in (0,1]$ , with probability at least  $1 - \delta$ , for all  $k \in \mathbb{N}^{+}$ , and function  $V \in \mathcal{V}$  satisfying  $\sup_{x \in S} |V(s)| \leq H$ , we have
$$
\left\| \sum_ {\tau = 1} ^ {k - 1} \phi_ {\tau} \big (V (x _ {\tau + 1}) - \mathbb {E} [ V (x _ {\tau + 1}) \mid \mathcal {F} _ {\tau} ] \big) \right\| _ {\Lambda_ {k} ^ {- 1}} ^ {2} \leq 4 H ^ {2} \cdot \left[ \frac {d}{2} \log \left(\frac {k + \lambda}{\lambda}\right) + \log \left(\frac {\mathcal {N} _ {\varepsilon}}{\delta}\right) \right] + \frac {8 k ^ {2} \varepsilon^ {2}}{\lambda},
$$
where  $\mathcal{N}_{\varepsilon}$  is the  $\varepsilon$ -covering number of the function class  $\mathcal{V}$  with respect to the distance  $\mathrm{dist}(V, V') = \sup_{x \in S} |V(x) - V'(x)|$ .
Proof. See Lemma D.4 of Jin et al. [17] for a detailed proof.
Lemma D.4. For any  $h \in [H]$ , let  $\mathcal{Q}_h$  be a function class mapping from  $\mathcal{S} \times \mathcal{A}$  to  $\mathbb{R}$  with the form
$$
\mathcal {Q} _ {h} (\cdot , \cdot) = r (\cdot , \cdot) + \min  \left\{\phi (\cdot , \cdot) ^ {\top} w + \beta \sqrt {\phi (\cdot , \cdot) ^ {\top} \Lambda^ {- 1} \phi (\cdot , \cdot)}, H - h \right\} ^ {+},
$$
where  $r:\mathcal{S}\times \mathcal{A}\mapsto [0,1]$ $\| w\| _2\leq L$ $\lambda_{\mathrm{min}}(\lambda)\geq \lambda$  Assuming  $\| \phi (\cdot ,\cdot)\| _2\leq 1$  and  $\beta >0$  we have
$$
\log \mathcal {N} _ {\varepsilon} (\mathcal {Q} _ {h}) \leq d \log (1 + 4 L / \varepsilon) + d ^ {2} \log \left(1 + 8 d ^ {1 / 2} \beta^ {2} / (\lambda \varepsilon)\right),
$$
where  $\mathcal{N}_{\varepsilon}(\mathcal{Q}_h)$  is the  $\varepsilon$ -covering number of the function class  $\mathcal{Q}_h$  with respect to the distance  $\mathrm{dist}(Q, Q') = \sup_{(x,a) \in S \times A} |Q(x,a) - Q'(x,a)|$ .
Proof. See Lemma D.6 of Jin et al. [17] for a detailed proof.
Lemma D.5 (Elliptical Potential Lemma). Let  $\{\phi_t\in \mathbb{R}^d\}_{t = 1}^\infty$  satisfy  $\| \phi_t\| _2\leq 1$  for all  $t\in \mathbb{N}^{+}$  Moreover, let  $\Lambda_0\in \mathbb{R}^{d\times d}$  be a positive-definite matrix with  $\lambda_{\mathrm{min}}(\Lambda_0)$  and  $\Lambda_{t} = \Lambda_{0} + \sum_{i = 1}^{t - 1}\phi_{i}\phi_{i}^{\top}$  for any  $t\in \mathbb{N}^{+}$  . Then for any  $t\in \mathbb{N}^{+}$  , we have
$$
\log \left(\frac {\operatorname* {d e t} (\Lambda_ {t + 1})}{\operatorname* {d e t} (\Lambda_ {1})}\right) \leq \sum_ {i = 1} ^ {t} \phi_ {i} ^ {\top} \Lambda_ {i} ^ {- 1} \phi_ {i} \leq 2 \log \left(\frac {\operatorname* {d e t} (\Lambda_ {t + 1})}{\operatorname* {d e t} (\Lambda_ {1})}\right).
$$
Proof. See Lemma 11 of Abbasi-Yadkori et al. [1] for a detailed proof.
Lemma D.6. Suppose  $\Lambda, \Lambda' \in \mathbb{R}^{d \times d}$  are two positive definite matrices and satisfy  $\Lambda \preceq \Lambda'$ , then for any  $x \in \mathbb{R}^d$ , we have
$$
\frac {\operatorname* {d e t} \left(\Lambda^ {\prime}\right)}{\operatorname* {d e t} (\Lambda)} \geq \frac {x ^ {\top} \Lambda^ {\prime} x}{x ^ {\top} \Lambda x}.
$$
Proof. See Lemma 12 of Abbasi-Yadkori et al. [1] for a detailed proof.