SlowGuess/NeurIPS_2023_OCR huggingface dataset:
contradicting with  $p' \in \Delta(\mathcal{A})$ . A similar argument holds for the case  $\lambda \leq \min_{a \in \mathcal{A}}(-Q(s,i))$ . Thus, we have  $\lambda \in [\min_{a \in \mathcal{A}}(-Q(s,a)), \max_{a \in \mathcal{A}}(-Q(s,a))]$ . Then, we have  $|Q(s,a) + \lambda + \mu(a)| \leq \max_{a \in \mathcal{A}} Q(s,a) - \min_{a \in \mathcal{A}} Q(s,a) \leq H$ . By the condition on  $\eta H \leq 1$ , we have  $p'(a) \in [\exp(-1)p(a), \exp(1)p(a)] \in [1/(4p(a)), 4p(a)]$ .
Case 2:  $\min_{a\in \mathcal{A}}Q(s,a) = \max_{a\in \mathcal{A}}Q(s,a)$ . In this case, it is clear that  $\lambda = -Q(s,a)$  must hold for all  $a\in \mathcal{A}$  to make  $p$  and  $p^\prime$  both discussions. Thus  $p^{\prime}(a) = p(a)$  for all  $a\in \mathcal{A}$ .
Combining the above two cases finishes the proof.
To prove Lemma 4, we first introduce the following two lemmas. Denote by  $\mathbb{P}_h^{\pi_h}(s' \mid s) = \sum_{a \in \mathcal{A}} \mathbb{P}_h(s' \mid s, a) \pi_h(a \mid s)$  the transition kernel of the MDP in step  $h$  under policy  $\pi_h$  and recall that  $\| \pi - \pi' \|_{1,\infty} = \max_{s \in \mathcal{S}} \| \pi(\cdot \mid s) - \pi'(\cdot \mid s) \|$ . The first lemma shows the difference between the state distribution of two policies can be bounded by the path length of the policies.
Lemma 6 (Zhao et al. [34, Lemma 7]). For any state distribution  $d$ , policy pair  $\pi$  and  $\pi'$ , and transition kernel  $\mathbb{P}$ , we have
$$
\left\| d \mathbb {P} _ {h} ^ {\pi_ {h}} (\cdot , s) - d \mathbb {P} _ {h} ^ {\pi_ {h} ^ {\prime}} (\cdot , s) \right\| _ {1} \leq \| \pi_ {h} - \pi_ {h} ^ {\prime} \| _ {1, \infty}, \forall h \in [ H ].
$$
Proof. Consider the case when  $d$  is a delta function on  $s$ . The difference in the next distributions is
$$
\begin{array}{l} \left| \left| \mathbb {P} _ {h} ^ {\pi_ {h}} (\cdot , s) - \mathbb {P} _ {h} ^ {\pi_ {h}} (\cdot , s) \right| \right| _ {1} = \sum_ {s ^ {\prime} \in \mathcal {S}} \sum_ {a \in \mathcal {A}} | \mathbb {P} (s ^ {\prime} \mid s, a) (\pi_ {h} (a \mid s) - \pi_ {h} (a \mid s)) | \\ \leq \sum_ {s ^ {\prime} \in \mathcal {S}} \sum_ {a \in \mathcal {A}} \mathbb {P} (s ^ {\prime} \mid s, a) \| \pi_ {h} (a \mid s) - \pi_ {h} ^ {\prime} (a \mid s) \| _ {1} \\ \end{array}
$$
$$
\leq \sum_ {a \in \mathcal {A}} | \pi (a \mid s) - \pi^ {\prime} (a \mid s) | \leq \| \pi_ {h} - \pi_ {h} ^ {\prime} \| _ {1, \infty}.
$$
Linearity of expectation leads to the result for arbitrary distributions. This finishes the proof.
The second lemma shows the difference between the state distribution of the policy starting from the different initial distributions can be bounded by the difference between the initial distributions.
Lemma 7 (Zhao et al. [34, Lemma 8]). For any two initial distributions  $d$  and  $d'$ , transition kernel  $\mathbb{P}$  and policy  $\pi$ , we have
$$
\left\| d \mathbb {P} _ {h} ^ {\pi_ {h}} - d ^ {\prime} \mathbb {P} _ {h} ^ {\pi_ {h}} \right\| _ {1} \leq \| d - d ^ {\prime} \| _ {1}, \forall h \in [ H ].
$$
Proof. Note the relationship that  $d(s') = \sum_{s \in S} d(s) \mathbb{P}_{s,s'}^{\pi_h}$ , we have
$$
\begin{array}{l} \left| \left| d \mathbb {P} _ {h} ^ {\pi_ {h}} - d ^ {\prime} \mathbb {P} _ {h} ^ {\pi_ {h}} \right| \right| _ {1} = \sum_ {s ^ {\prime}} \left| \sum_ {s} d (s) \mathbb {P} _ {h} ^ {\pi_ {h}} (s ^ {\prime} \mid s) - d ^ {\prime} (s) \mathbb {P} _ {h} ^ {\pi_ {h}} (s ^ {\prime} \mid s) \right| \\ \leq \sum_ {s ^ {\prime}} \sum_ {s} \left| d (s) \mathbb {P} _ {h} ^ {\pi_ {h}} \left(s ^ {\prime} \mid s\right) - d ^ {\prime} (s) \mathbb {P} _ {h} ^ {\pi_ {h}} \left(s ^ {\prime} \mid s\right) \right| \\ = \sum_ {s ^ {\prime}} \sum_ {s} | d (s) - d ^ {\prime} (s) | \mathbb {P} _ {h} ^ {\pi_ {h}} \left(s ^ {\prime} \mid s\right) \\ = \sum_ {s} | d (s) - d ^ {\prime} (s) | \sum_ {s ^ {\prime}} \mathbb {P} _ {h} ^ {\pi_ {h}} \left(s ^ {\prime} \mid s\right) \\ = \sum_ {s} | d (s) - d ^ {\prime} (s) | = \| d - d ^ {\prime} \| _ {1}. \\ \end{array}
$$
Now, we are ready to prove Lemma 4.
Proof of Lemma 4. For any policy  $\pi$  and  $\pi'$  and any initial state  $s_1$ , denote by  $d_h$  the distribution of the MDP in step  $h$  under policy  $\pi_h$  and  $d_h'$  the distribution of the MDP in step  $h$  under policy  $\pi_h'$ , that is  $d_h(s_h) = \mathbb{E}_{\pi}[\mathbb{1}(s_h) \mid s_1]$  and  $d_h'(s_h) = \mathbb{E}_{\pi'}[\mathbb{1}(s_h) \mid s_1]$ . We have
$$
\begin{array}{l} \left\| d _ {h} - d _ {h} ^ {\prime} \right\| _ {1} = \left\| d _ {h - 1} \mathbb {P} _ {h} ^ {\pi_ {h}} - d _ {h - 1} ^ {\prime} \mathbb {P} _ {h} ^ {\pi_ {h} ^ {\prime}} \right\| _ {1} \\ \leq \left\| d _ {h - 1} \mathbb {P} _ {h} ^ {\pi_ {h}} - d _ {h - 1} ^ {\prime} \mathbb {P} _ {h} ^ {\pi_ {h}} \right\| _ {1} + \left\| d _ {h - 1} ^ {\prime} \mathbb {P} _ {h} ^ {\pi_ {h}} - d _ {h - 1} ^ {\prime} \mathbb {P} _ {h} ^ {\pi_ {h} ^ {\prime}} \right\| _ {1} \\ \leq \left\| d _ {h - 1} - d _ {h - 1} ^ {\prime} \right\| _ {1} + \left\| \pi_ {h} - \pi_ {h} ^ {\prime} \right\| _ {1, \infty} \\ \leq \sum_ {i \in [ h ]} \left\| \pi_ {i} - \pi_ {i} ^ {\prime} \right\| _ {1, \infty}, \\ \end{array}
$$
where the second inequality holds by Lemmas 6 and 7 and the last inequality holds by a recursive calculation. Thus, we have
$$
\sum_ {k = 1} ^ {K} \left(\mathbb {E} _ {\pi_ {k} ^ {c}} - \mathbb {E} _ {\pi_ {k - 1} ^ {c}}\right) \left[ \sum_ {h = 1} ^ {H} \mathbb {1} (s _ {h}) \mid s _ {1} = s _ {k, 1} \right] \leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \sum_ {i = 1} ^ {h} \| \pi_ {k, i} ^ {c} - \pi_ {k - 1, i} ^ {c} \| _ {1, \infty} = H P _ {T}.
$$
C.5 Proof of Theorem 1
Proof. By Lemma 2, we have
$$
\begin{array}{l} \operatorname {D} - \operatorname {R e g r e t} (K) \leq \frac {\eta K H ^ {3}}{2} + \frac {1}{\eta} \left(P _ {T} \log \frac {4 A}{\gamma} + K H \log \frac {1}{1 - \gamma}\right) \tag {19} \\ + \frac {1}{\eta} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \sum_ {a \in \mathcal {A}} \left(\pi_ {k - 1, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k , h} ^ {\prime} (a \mid s _ {h})} - \pi_ {k, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k + 1 , h} ^ {\prime} (a \mid s _ {h})}\right) \right] \\ \end{array}
$$
Then, the last term can be bounded as follows:
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \sum_ {a \in \mathcal {A}} \left(\pi_ {k - 1, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k , h} ^ {\prime} (a \mid s _ {h})} - \pi_ {k, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k + 1 , h} ^ {\prime} (a \mid s _ {h})}\right) \right] \\ \leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left(\mathbb {E} _ {\pi_ {k - 1} ^ {c}} \left[ \sum_ {a \in \mathcal {A}} \pi_ {k - 1, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k , h} ^ {\prime} (a \mid s _ {h})} \right] - \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \sum_ {a \in \mathcal {A}} \pi_ {k, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k + 1 , h} ^ {\prime} (a \mid s _ {h})} \right]\right) \\ + \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c} - \pi_ {k - 1} ^ {c}} \left[ \sum_ {a \in \mathcal {A}} \pi_ {k - 1, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k , h} ^ {\prime} (a \mid s _ {h})} \right] \\ \leq \mathbb {E} _ {\pi_ {1} ^ {c}} \left[ \sum_ {a \in \mathcal {A}} \pi_ {1, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {1 , h} ^ {\prime} (a \mid s _ {h})} \right] + \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c} - \pi_ {k - 1} ^ {c}} \left[ \sum_ {a \in \mathcal {A}} \pi_ {k - 1, h} ^ {c} (a \mid s _ {h}) \log \frac {1}{\pi_ {k , h} ^ {\prime} (a \mid s _ {h})} \right] \\ \leq H \log A + \log \frac {4 A}{\gamma} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c} - \pi_ {k - 1} ^ {c}} [ \mathbb {1} (s _ {h}) \mid s _ {1} = s _ {k, 1} ] \\ \leq H \log A + \log \frac {4 A}{\gamma} H P _ {T}, \tag {20} \\ \end{array}
$$
where the second inequality holds by the telescoping sum and the third inequality holds because  $\pi_{1,h}^{\prime} = \pi_{1,h}$  is the uniform distribution and  $\pi_{k,h}^{\prime}(a|s)\geq \frac{\gamma}{4A}$  by the multiplicative stability lemma in Lemma 3 and the last inequality holds by Lemma 4.
Combining (19) and (20), we have
$$
\mathrm {D - R e g r e t} (K) \leq \frac {\eta K H ^ {3}}{2} + \frac {1}{\eta} \left(H \log A + (1 + H) \log \frac {4 A}{\gamma} P _ {T} + K H \log \frac {1}{1 - \gamma}\right),
$$
To prove Theorem 2, we first introduce the following key lemma omitted in the main paper due to the space limit, which shows the guarantee of the confidence set.
Lemma 8 (Zhou et al. [31, Lemma 5]). Let  $\widehat{C}_{k,h}$  be defined in (10) and set  $\widehat{\beta}_k$  as
$$
\widehat {\beta} _ {k} = 8 \sqrt {d \log (1 + k / \lambda) \log \left(4 k ^ {2} H / \delta\right)} + 4 \sqrt {d} \log \left(4 k ^ {2} H / \delta\right) + \sqrt {\lambda} B.
$$
Then, with probability at least  $1 - 3\delta$ , we have that simultaneously for all  $h \in [H]$  and  $k \in [K]$ ,
$$
\theta_ {h} ^ {*} \in \left\{\theta \big | \big \| \widehat {\Sigma} _ {k, h} ^ {1 / 2} (\theta - \widehat {\theta} _ {k, h}) \big \| _ {2} \leq \widehat {\beta} _ {k} \right\}, \Big | \left[ \bar {\mathbb {V}} _ {k, h} V _ {k, h + 1} \right] (s _ {k, h}, a _ {k, h}) - [ \mathbb {V} _ {h} V _ {k, h + 1} ] (s _ {k, h}, a _ {k, h}) \Big | \leq E _ {k, h},
$$
where  $E_{k,h}$  is defined as follows:
$$
\min  \left\{H ^ {2}, 2 H \tilde {\beta} _ {k} \left\| \widehat {\Sigma} _ {k, h} ^ {- 1 / 2} \phi_ {V _ {k, h + 1}} (s _ {k, h}, a _ {k, h}) \right\| _ {2} \right\} + \min  \left\{H ^ {2}, \widetilde {\beta} _ {k} \left\| \widetilde {\Sigma} _ {k, h} ^ {- 1 / 2} \phi_ {V _ {k, h + 1} ^ {2}} (s _ {k, h}, a _ {k, h}) \right\| _ {2} \right\}
$$
$$
\check {\beta} _ {k} = 8 d \sqrt {\log (1 + k / \lambda) \log \left(4 k ^ {2} H / \delta\right)} + 4 \sqrt {d} \log \left(4 k ^ {2} H / \delta\right) + \sqrt {\lambda} B,
$$
$$
\tilde {\beta} _ {k} = 8 \sqrt {d H ^ {4} \log (1 + k H ^ {4} / (d \lambda)) \log (4 k ^ {2} H / \delta)} + 4 H ^ {2} \log (4 k ^ {2} H / \delta) + \sqrt {\lambda} B. \tag {21}
$$
We denote by  $\mathcal{E}$  the event when the result of Lemma 8 holds, we have  $\operatorname*{Pr}(\mathcal{E})\geq 1 - 3\delta$
Then we present the following lemma that shows the value function difference can be decomposed into two parts, one is a martingale sequence and the other is the estimation error.
Lemma 9. For all  $k \in [K]$ ,  $h \in [H]$ , it holds that
$$
V _ {k, h} \left(s _ {k, h}\right) - V _ {k, h} ^ {\pi_ {k}} \left(s _ {k, h}\right) = \sum_ {h ^ {\prime} = h} ^ {H} \left(M _ {k, h ^ {\prime}, 1} + M _ {k, h ^ {\prime}, 2} - \iota_ {k, h ^ {\prime}} \left(s _ {k, h ^ {\prime}}, a _ {k, h ^ {\prime}}\right)\right),
$$
$$
M _ {k, h, 1} = \mathbb {E} _ {a \sim \pi_ {k} (\cdot | s _ {k, h})} [ Q _ {k, h} (s _ {k, h}, a) - Q _ {k, h} ^ {\pi_ {k}} (s _ {k, h}, a) ] - (Q _ {k, h} (s _ {k, h}, a _ {k, h}) - Q _ {k, h} ^ {\pi_ {k}} (s _ {k, h}, a _ {k, h})),
$$
$$
M _ {k, h, 2} = \left[ \mathbb {P} _ {h} \left(V _ {k, h + 1} - V _ {k, h + 1} ^ {\pi_ {k}}\right) \right] \left(s _ {k, h}, a _ {k, h}\right) - \left(V _ {k, h + 1} \left(s _ {k, h + 1}\right) - V _ {k, h + 1} ^ {\pi_ {k}} \left(s _ {k, h + 1}\right)\right).
$$
Proof. By the definition  $V_{k,h}(s_{k,h}) = \mathbb{E}_{a\sim \pi_k(\cdot \mid s_{k,h})}[Q_{k,h}(s_{k,h},a)]$ , we have
$$
\begin{array}{l} V _ {k, h} \left(s _ {k, h}\right) - V _ {k, h} ^ {\pi_ {k}} \left(s _ {k, h}\right) \\ = \mathbb {E} _ {a \sim \pi_ {k} (\cdot | s _ {k, h})} [ Q _ {k, h} (s _ {k, h}, a) - Q _ {k, h} ^ {\pi_ {k}} (s _ {k, h}, a) ] \\ = \mathbb {E} _ {a \sim \pi_ {k} (\cdot | s _ {k, h})} \left[ Q _ {k, h} (s _ {k, h}, a) - Q _ {k, h} ^ {\pi_ {k}} (s _ {k, h}, a) \right] - \left(Q _ {k, h} (s _ {k, h}, a _ {k, h}) - Q _ {k, h} ^ {\pi_ {k}} (s _ {k, h}, a _ {k, h})\right) \\ + \left(Q _ {k, h} \left(s _ {k, h}, a _ {k, h}\right) - Q _ {k, h} ^ {\pi_ {k}} \left(s _ {k, h}, a _ {k, h}\right)\right) \\ = \mathbb {E} _ {a \sim \pi_ {k} (\cdot | s _ {k, h})} \left[ Q _ {k, h} \left(s _ {k, h}, a\right) - Q _ {k, h} ^ {\pi_ {k}} \left(s _ {k, h}, a\right) \right] - \left(Q _ {k, h} \left(s _ {k, h}, a _ {k, h}\right) - Q _ {k, h} ^ {\pi_ {k}} \left(s _ {k, h}, a _ {k, h}\right)\right) \\ + \left[ \mathbb {P} _ {h} \left(V _ {k, h + 1} - V _ {k, h + 1} ^ {\pi_ {k}}\right) \right] \left(s _ {k, h}, a _ {k, h}\right) - \iota_ {k, h} \left(s _ {k, h}, a _ {k, h}\right) \\ = \underbrace {\mathbb {E} _ {a \sim \pi_ {k} (\cdot \mid s _ {k , h})} \left[ Q _ {k , h} \left(s _ {k , h} , a\right) - Q _ {k , h} ^ {\pi_ {k}} \left(s _ {k , h} , a\right) \right] - \left(Q _ {k , h} \left(s _ {k , h} , a _ {k , h}\right) - Q _ {k , h} ^ {\pi_ {k}} \left(s _ {k , h} , a _ {k , h}\right)\right)} _ {M _ {k, h, 1}} \\ + \underbrace {\left[ \mathbb {P} _ {h} \left(V _ {k , h + 1} - V _ {k , h + 1} ^ {\pi_ {k}}\right) \right] \left(s _ {k , h} , a _ {k , h}\right) - \left(V _ {k , h + 1} \left(s _ {k , h + 1}\right) - V _ {k , h + 1} ^ {\pi_ {k}} \left(s _ {k , h + 1}\right)\right)} _ {M _ {k, h, 2}} \\ + \left(V _ {k, h + 1} \left(s _ {k, h + 1}\right) - V _ {k, h + 1} ^ {\pi_ {k}} \left(s _ {k, h + 1}\right)\right) - \iota_ {k, h} \left(s _ {k, h}, a _ {k, h}\right) \\ \end{array}
$$
where the third equality holds by the fact  $Q_{k,h} = r_{k,h} + \mathbb{P}_h V_{k,h+1} - \iota_{k,h}$  and  $Q_{k,h}^{\pi_k} = r_{k,h} + \mathbb{P}_h V_{k,h+1}^{\pi_k}$ . Summing up the above equation from  $h$  to  $H$  recursively finishes the proof.
Note  $M_{k,h,1}$  is the noise from the stochastic policy and  $M_{k,h,2}$  is the noise from the state transition. Let  $M_{k,h} = M_{k,h,1} + M_{k,h,2}, \forall k \in [K], h \in [H]$ , we define two following high probability events:
$$
\mathcal {E} _ {1} = \left\{\forall h \in [ H ], \sum_ {k = 1} ^ {K} \sum_ {h ^ {\prime} = h} ^ {H} M _ {k, h ^ {\prime}} \leq 4 \sqrt {H ^ {3} K \log \frac {H}{\delta}} \right\}, \mathcal {E} _ {2} = \left\{\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} M _ {k, h, 2} \leq \sqrt {8 H ^ {3} K \log \frac {1}{\delta}} \right\}.
$$
According to the Azuma-Hoeffding inequality, we have  $\operatorname*{Pr}(\mathcal{E}_1) \geq 1 - \delta$  and  $\operatorname*{Pr}(\mathcal{E}_2) \geq 1 - \delta$ .
Then, we show the model prediction error  $\iota_{k,h}$  can be upper and lower bounded both.
Lemma 10. Define prediction error  $\iota_{k,h} = r_{k,h} + \mathbb{P}_hV_{k,h + 1} - Q_{k,h}$ , on the event  $\mathcal{E}$ , it holds that
$$
- 2 \widehat {\beta} _ {k} \left\| \widehat {\Sigma} _ {k, h} ^ {- 1 / 2} \phi_ {V _ {k, h + 1}} (\cdot , \cdot) \right\| _ {2} \leq \iota_ {k, h} (\cdot , \cdot) \leq 0, \forall k \in [ K ], h \in [ H ].
$$
Proof. First, we prove the left-hand side inequality. By the definition of  $\iota_{k,h}$ , we have
$$
\begin{array}{l} - \iota_ {k, h} (s, a) \\ = Q _ {k, h} (s, a) - \left(r _ {k, h} + \mathbb {P} _ {h} V _ {k, h + 1}\right) (s, a) \\ \leq r _ {k, h} (s, a) + \left\langle \widehat {\theta} _ {k, h}, \phi_ {V _ {k, h + 1}} (s, a) \right\rangle + \widehat {\beta} _ {k} \left\| \widehat {\Sigma} _ {k, h} ^ {- 1 / 2} \phi_ {V _ {k, h + 1}} (s, a) \right\| _ {2} - \left(r _ {k, h} + \mathbb {P} _ {h} V _ {k, h + 1}\right) (s, a) \\ = \left\langle \widehat {\theta} _ {k, h} - \theta_ {h} ^ {*}, \phi_ {V _ {k, h + 1}} (s, a) \right\rangle + \widehat {\beta} _ {k} \left\| \widehat {\Sigma} _ {k, h} ^ {- 1 / 2} \phi_ {V _ {k, h + 1}} (s, a) \right\| _ {2} \\ \leq 2 \widehat {\beta} _ {k} \left\| \widehat {\Sigma} _ {k, h} ^ {- 1 / 2} \phi_ {V _ {k, h + 1}} (s, a) \right\| _ {2}, \\ \end{array}
$$
where the first inequality holds by the configuration of  $Q_{k,h}$  in Algorithm 1, the second equality holds by the definition of linear mixture MDP such that  $[\mathbb{P}_hV_{k,h + 1}](s,a) = \langle \phi_{V_{k,h + 1}}(s,a),\theta_h^*\rangle$  and the last inequality holds by the configuration of confidence set in Lemma 8.
Next, we prove the right-hand side inequality. By the definition of  $\iota_{k,h}$ , we have
$$
\begin{array}{l} \iota_ {k, h} (s, a) \\ = \left(r _ {k, h} + \mathbb {P} _ {h} V _ {k, h + 1}\right) (s, a) - Q _ {k, h} (s, a) \\ \leq \left(r _ {k, h} + \mathbb {P} _ {h} V _ {k, h + 1}\right) (s, a) - \left[ r _ {k, h} (s, a) + \left\langle \widehat {\theta} _ {k, h}, \phi_ {V _ {k, h + 1}} (s, a) \right\rangle + \widehat {\beta} _ {k} \left\| \widehat {\Sigma} _ {k, h} ^ {- 1 / 2} \phi_ {V _ {k, h + 1}} (s, a) \right\| _ {2} \right] _ {[ 0, H - h + 1 ]} \\ \end{array}
$$
$$
\begin{array}{l} = \max  \Big \{\big \langle \widehat {\theta} _ {k, h} - \theta_ {h} ^ {*}, \phi_ {V _ {k, h + 1}} (s, a) \big \rangle - \widehat {\beta} _ {k} \big \| \widehat {\Sigma} _ {k, h} ^ {- 1 / 2} \phi_ {V _ {k, h + 1}} (s, a) \big \| _ {2}, (r _ {k, h} + \mathbb {P} _ {h} V _ {k, h + 1}) (s, a) - (H - h + 1) \Big \} \\ \leq 0, \end{array}
$$
Next, we show the prediction error can be bounded by the cumulative estimate variance.
Lemma 11. Define prediction error  $\iota_{k,h} = r_{k,h} + \mathbb{P}_h V_{k,h+1} - Q_{k,h}$ , on the event  $\mathcal{E}$ , it holds that
$$
- \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \iota_ {k, h} \left(s _ {k, h}, a _ {k, h}\right) \leq 2 \widehat {\beta} _ {K} \sqrt {\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \bar {\sigma} _ {k , h} ^ {2}} \sqrt {2 H d \log (1 + K / \lambda)}.
$$
Proof. By Lemma 10 and the definition of  $\iota_{k,h} = r_{k,h} + \mathbb{P}_hV_{k,h + 1} - Q_{k,h}$ , we have
$$
\begin{array}{l} - \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \iota_ {k, h} \left(s _ {k, h}, a _ {k, h}\right) \\ \leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} 2 \min  \left\{\widehat {\beta} _ {k} \left\| \widehat {\Sigma} _ {k, h} ^ {- 1 / 2} \phi_ {V _ {k, h + 1}} \left(s _ {k, h}, a _ {k, h}\right) \right\| _ {2}, H \right\} \\ \leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} 2 \widehat {\beta} _ {k} \bar {\sigma} _ {k, h} \min  \left\{\left\| \widehat {\Sigma} _ {k, h} ^ {- 1 / 2} \phi_ {V _ {k, h + 1}} \left(s _ {k, h}, a _ {k, h}\right) / \bar {\sigma} _ {k, h} \right\| _ {2}, 1 \right\} \\ \leq 2 \widehat {\beta} _ {K} \sqrt {\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \bar {\sigma} _ {k , h} ^ {2}} \sqrt {\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \min  \left\{\left\| \widehat {\Sigma} _ {k , h} ^ {- 1 / 2} \phi_ {V _ {k , h + 1}} \left(s _ {k, h}, a _ {k, h}\right) / \bar {\sigma} _ {k, h} \right\| _ {2} , 1 \right\}} \\ \leq 2 \widehat {\beta} _ {K} \sqrt {\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \bar {\sigma} _ {k , h} ^ {2}} \sqrt {2 H d \log (1 + K / \lambda)}, \\ \end{array}
$$
where the second inequality holds by  $2\widehat{\beta}_k\bar{\sigma}_{k,h}\geq \sqrt{d} H / \sqrt{d} = H$  , the third inequality is by CauchySchwarz inequality and the last inequality holds by the elliptical potential lemma in Lemma 16.
Define the event
$$
\mathcal {E} _ {3} = \left\{\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left[ \mathbb {V} _ {h} V _ {h + 1} ^ {\pi^ {k}} \right] \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) \leq 3 \left(H K + H ^ {3} \log (1 / \delta)\right) \right\},
$$
by the law of total variance in Lemma 15, we have  $\operatorname{Pr}(\mathcal{E}_3) \geq 1 - \delta$ . Then, we have the following lemma which bounds the estimated variance of the value function.
Lemma 12 (He et al. [33, Lemma 6.5]). On the events  $\mathcal{E} \cap \mathcal{E}_1 \cap \mathcal{E}_2 \cap \mathcal{E}_3$ , it holds that
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \bar {\sigma} _ {k, h} ^ {2} \leq 2 H ^ {3} K / d + 1 7 9 H ^ {2} K + 1 6 5 d ^ {3} H ^ {4} \log^ {2} \left(4 K ^ {2} H / \delta\right) \log^ {2} \left(1 + K H ^ {4} / \lambda\right) \\ + 2 0 6 2 d ^ {2} H ^ {5} \log^ {2} \left(4 K ^ {2} H / \delta\right) \log^ {2} (1 + K / \lambda). \\ \end{array}
$$
Now, we are ready to prove Theorem 2.
Proof of Theorem 2. On the events  $\mathcal{E} \cap \mathcal{E}_1 \cap \mathcal{E}_2 \cap \mathcal{E}_3$ , for any  $h \in [H]$ , it holds that
$$
\sum_ {k = 1} ^ {K} V _ {k, h} \left(s _ {k, h}\right) - \sum_ {k = 1} ^ {K} V _ {k, h} ^ {\pi_ {k}} \left(s _ {k, h}\right)
$$
$$
\begin{array}{l} \leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left(M _ {k, h, 1} + M _ {k, h, 2} - \iota_ {k, h} \left(s _ {k, h}, a _ {k, h}\right)\right) \\ \leq 4 \sqrt {H ^ {3} K \log (H / \delta)} + 2 \widehat {\beta_ {K}} \sqrt {\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \bar {\sigma} _ {k , h} ^ {2}} \sqrt {2 H d \log (1 + K / \lambda)} \\ \leq \widetilde {\mathcal {O}} \left(\sqrt {d H ^ {4} K + d ^ {2} H ^ {3} K}\right), \\ \end{array}
$$
where the first equality holds by Lemma 9, the second inequality holds by Lemma 11, and the last inequality holds by Lemma 12. This finishes the proof.
Proof. By Lemma 5, we can rewrite the dynamic regret as follows.
$$
\begin{array}{l} \mathrm {D} - \operatorname {R e g r e t} (K) = \sum_ {k = 1} ^ {K} V _ {k, 1} ^ {\pi_ {k} ^ {c}} \left(s _ {k, 1}\right) - \sum_ {k = 1} ^ {K} V _ {k, 1} ^ {\pi_ {k}} \left(s _ {k, 1}\right) \\ = \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} [ \langle Q _ {k, h} (s _ {h}, \cdot), \pi_ {k, h} ^ {c} (\cdot | s _ {h}) - \pi_ {k, h} (\cdot | s _ {h}) \rangle ] \\ + \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left(\mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \iota_ {k, h} \left(s _ {h}, a _ {h}\right) \right] - \iota_ {k, h} \left(s _ {k, h}, a _ {k, h}\right)\right) + \mathcal {M} _ {K, H}. \tag {22} \\ \end{array}
$$
By Lemma 10, we have  $\iota_{k,h}(s,a)\leq 0$  for any  $k\in [K],h\in [H],s\in \mathcal{S},a\in \mathcal{A}$ . Thus, we have
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \iota_ {k, h} \left(s _ {h}, a _ {h}\right) \right] \leq 0. \tag {23}
$$
By Theorem 2, we have
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left(- \iota_ {k, h} \left(s _ {k, h}, a _ {k, h}\right)\right) + \mathcal {M} _ {K, H} \leq \widetilde {\mathcal {O}} \left(\sqrt {d H ^ {4} K + d ^ {2} H ^ {3} K}\right). \tag {24}
$$
It remains to bound the first term. Note our algorithm is indeed updated based on the estimated action-value function  $Q_{k,h}$ . By Theorem 1, set  $\gamma = 1 / KH$  and note that  $\log (1 / (1 - \gamma)) \leq \gamma / (1 - \gamma)$  for all  $\gamma > 0$ . Then, we have
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} [ \langle Q _ {k, h} (s _ {h}, \cdot), \pi_ {k, h} ^ {c} (\cdot | s _ {h}) - \pi_ {k, h} (\cdot | s _ {h}) \rangle ] \\ \leq \frac {\eta K H ^ {3}}{2} + \frac {1}{\eta} \left(H \log A + (1 + H) \log \frac {A}{\gamma} P _ {T} + 2\right) \\ \end{array}
$$
It is clear that the optimal step size is  $\eta^{*} = \sqrt{(P_{T} + \log A) / (KH^{2})}$ . Note our step size is set as  $\eta = \min \{\sqrt{(P_T + \log A) / K}, 1\} / H$  to ensure  $\eta \leq 1 / H$ . We consider the following two cases:
Case 1:  $\eta^{*} \leq 1 / H$ . In this case, our step size is set as  $\eta = \sqrt{(P_T + \log A) / (KH^2)}$ . We have
$$
\frac {\eta K H ^ {3}}{2} + \frac {1}{\eta} \left(H \log A + (1 + H) \log \frac {A}{\gamma} P _ {T} + 2\right) \leq \widetilde {\mathcal {O}} (\sqrt {K H ^ {4} (1 + P _ {T})}).
$$
Case 2:  $\eta^{*} > 1 / H$ . In this case, our step size is set as  $\eta = 1 / H$ . Therefore, we have
$$
\frac {\eta K H ^ {3}}{2} + \frac {1}{\eta} \left(H \log A + (1 + H) \log \frac {A}{\gamma} P _ {T} + 2\right) \leq \widetilde {\mathcal {O}} \left(H ^ {2} P _ {T}\right),
$$
where the equality holds by  $P_T \geq K$  in this case. Combining these two cases, we have
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} [ \langle Q _ {k, h} (s _ {h}, \cdot), \pi_ {k, h} ^ {c} (\cdot | s _ {h}) - \pi_ {k, h} (\cdot | s _ {h}) \rangle ] \leq \widetilde {\mathcal {O}} (\sqrt {H ^ {4} (K + P _ {T}) (1 + P _ {T})}). \tag {25}
$$
Combining (22), (23), (24) and (25), we obtain
$$
\mathrm {D} - \operatorname {R e g r e t} (K) \leq \widetilde {\mathcal {O}} \left(\sqrt {d H ^ {4} K + d ^ {2} H ^ {3} K} + \sqrt {H ^ {4} (K + P _ {T}) (1 + P _ {T})}\right).
$$
C.8 Proof of Theorem 4