SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) ^ {2} \cdot \mathbb {E} _ {t} \left[ \left(Q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a; \widehat {\ell} _ {t}) - V ^ {\widehat {P} _ {t}, \pi_ {t}} (s; \widehat {\ell} _ {t})\right) ^ {2} \right] \\ \leq 2 q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) ^ {2} \left(1 - \pi_ {t} (a | s)\right) ^ {2} \left(\frac {L ^ {2}}{q ^ {\widehat {P} _ {t} , \pi_ {t}} (s , a)} + \frac {L C _ {t} ^ {\mathsf {P}}}{q ^ {\widehat {P} _ {t} , \pi_ {t}} (s , a)} \cdot \sum_ {k = k (s)} ^ {L - 1} \sum_ {x \in S _ {k}} \sum_ {y \in A} \left(\frac {q ^ {\widehat {P} _ {t} , \pi_ {t}} (x , y | s , a)}{u _ {t} (x , y)}\right)\right) \\ + 2 q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) ^ {2} \left(\frac {L ^ {2} \left(1 - \pi_ {t} (a | s)\right)}{q ^ {\widehat {P} _ {t} , \pi_ {t}} (s)} + \frac {L C _ {t} ^ {\mathsf {P}}}{q ^ {\widehat {P} _ {t} , \pi_ {t}} (s)} \sum_ {b \neq a} \pi_ {t} (b | s) \sum_ {k = k (s)} ^ {L - 1} \sum_ {x \in S _ {k}} \sum_ {y \in A} \left(\frac {q ^ {\widehat {P} _ {t} , \pi_ {t}} (x , y | s , b)}{u _ {t} (x , y)}\right)\right) \\ \leq \mathcal {O} \left(L ^ {2} q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) (1 - \pi_ {t} (a | s))\right) \\ + \mathcal {O} \left(L C _ {t} ^ {\mathsf {P}} \cdot q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) \sum_ {k = k (s)} ^ {L - 1} \sum_ {x \in S _ {k}} \sum_ {y \in A} \left(\frac {q ^ {\widehat {P} _ {t} , \pi_ {t}} (x , y | s , a)}{u _ {t} (x , y)}\right)\right) \\ + \mathcal {O} \left(L C _ {t} ^ {\mathbf {P}} \cdot q ^ {\widehat {P} _ {t}, \pi_ {t}} (s) \sum_ {b \neq a} \pi_ {t} (b | s) \sum_ {k = k (s)} ^ {L - 1} \sum_ {x \in S _ {k}} \sum_ {y \in A} \left(\frac {q ^ {\widehat {P} _ {t} , \pi_ {t}} (x , y | s , b)}{u _ {t} (x , y)}\right)\right) \\ \leq \mathcal {O} \left(L ^ {2} q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) (1 - \pi_ {t} (a | s))\right) \\ + \mathcal {O} \left(L C _ {t} ^ {\mathrm {P}} \cdot \sum_ {k = k (s)} ^ {L - 1} \sum_ {x \in S _ {k}} \sum_ {y \in A} \left(\frac {q ^ {\widehat {P} _ {t} , \pi_ {t}} (x , y | s , a) q ^ {\widehat {P} _ {t} , \pi_ {t}} (s , a)}{u _ {t} (x , y)}\right)\right) \\ + \mathcal {O} \left(L C _ {t} ^ {\mathrm {P}} \cdot \sum_ {k = k (s)} ^ {L - 1} \sum_ {x \in S _ {k}} \sum_ {y \in A} \left(\frac {\sum_ {b \neq a} q ^ {\widehat {P} _ {t} , \pi_ {t}} (s) \pi_ {t} (b | s) q ^ {\widehat {P} _ {t} , \pi_ {t}} (x , y | s , b)}{u _ {t} (x , y)}\right)\right) \\ \leq \mathcal {O} \left(L ^ {2} q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a) \left(1 - \pi_ {t} (a | s)\right) + L | S | C _ {t} ^ {\mathsf {P}}\right), \\ \end{array}
$$
where the third step follows from the facts that  $q^{\widehat{P}_t,\pi_t}(x,y|s,a)q^{\widehat{P}_t,\pi_t}(s,a)\leq q^{\widehat{P}_t,\pi_t}(x,y)\leq u_t(x,y)$  for any  $(s,a),(x,y)\in S\times A$ , and  $\sum_{b\neq a}q^{\widehat{P}_t,\pi_t}(s)\pi_t(b|s)q^{\widehat{P}_t,\pi_t}(x,y|s,b)\leq q^{\widehat{P}_t,\pi_t}(x,y)$  similarly.
The first part of Lemma C.7.2 ensures that  $\nu_{t}(s,a)\in [0,L^{2}]$ , which can be used to bound the growth of the learning rate.
Proposition C.7.3. The learning rate  $\gamma_{t}$  defined in Definition 5.2 satisfies for any state-action pair  $(s, a)$ :
$$
\gamma_ {t} (s, a) \leq \sqrt {D \sum_ {j = t _ {i (t)}} ^ {t - 1} \nu_ {j} (s , a)} + \gamma_ {t _ {i (t)}} (s, a)
$$
where  $i(t)$  is the epoch to which episode  $t$  belongs and  $t_i$  is the first episode of epoch  $i$ .
Proof. We prove this statement by induction on  $t$ . The equation trivially holds for  $t = t_{i(t)}$  which is the first round of epoch  $i(t)$ . For the induction step, we first note that  $D\nu_{t}(s,a)\in [0,L^{2}]$ . We introduce some notations to simplify the proof: we use  $c$  to denote  $D\nu_{t}(s,a)$ ,  $x$  to denote  $\gamma_{t}(s,a)$ , and the induction hypothesis is  $x\leq \sqrt{S} +\gamma$  where  $S = D\sum_{j = t_{i(t)}}^{t - 1}\nu_{j}(s,a)$  and  $\gamma = \gamma_{t_{i(t)}}(s,a)$ . Proving the induction step is the same as proving:
$$
x + \frac {c}{2 x} \leq \sqrt {S + c} + \gamma . \tag {41}
$$
First, we can verify that  $f(x) = x + \frac{c}{2x}$  is an increasing function of  $x$  for  $x \geq \sqrt{c / 2}$ . As  $c \in [0, L^2]$  and  $x \geq \gamma \geq L$ , we can use the induction hypothesis  $x \leq \sqrt{S} + \gamma$  to upper bound  $x$  in the left-hand side of Eq. (41), and we get:
$$
\begin{array}{l} x + \frac {c}{2 x} = \frac {x ^ {2} + c / 2}{x} \\ \leq \frac {S + 2 \gamma \sqrt {S} + \gamma^ {2} + c / 2}{\sqrt {S} + \gamma} \\ = \frac {S + \gamma \sqrt {S} + c / 2}{\sqrt {S} + \gamma} + \frac {\gamma \sqrt {S} + \gamma^ {2}}{\sqrt {S} + \gamma} \\ = \frac {\sqrt {S} + \gamma}{\sqrt {S} + \gamma} \left(\sqrt {S} + \frac {c}{2 \sqrt {S} + 2 \gamma}\right) + \frac {\gamma \sqrt {S} + \gamma^ {2}}{\sqrt {S} + \gamma} \\ = \left(\sqrt {S} + \frac {c}{2 \sqrt {S} + 2 \gamma}\right) + \gamma \\ \leq \left(\sqrt {S} + \frac {c}{2 \sqrt {S + c}}\right) + \gamma \\ \leq \sqrt {S + c} + \gamma , \\ \end{array}
$$
where the second to last step follows from  $c \leq L^2$  and  $\gamma \geq L$ , which ensures that  $2\sqrt{S} + 2\sqrt{\gamma} \geq 2\sqrt{S + \gamma^2} \geq 2\sqrt{S + c}$ . The last step follows from the concavity of the square-root function which ensures that  $\sqrt{S} \leq \sqrt{S + c} - \frac{c}{2\sqrt{S + c}}$ .
Then, we can use the second part of Lemma C.7.2 to continue the bound of Proposition C.7.3 and derive a bound that only depends on the suboptimal actions.
Proposition C.7.4. If  $\nu_{t}$  is defined as in Definition 5.2, we have for any deterministic policy  $\pi : S \to A$  and any state  $s \neq s_{L}$ :
$$
\begin{array}{l} \mathbb {E} _ {t _ {i}} \left[ \sum_ {a \in A} \sqrt {\sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \nu_ {t} (s , a)} \right] \leq \mathcal {O} \left(\sum_ {a \neq \pi (s)} \sqrt {\mathbb {E} _ {t _ {i}} \left[ L ^ {2} \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} q ^ {P _ {t} , \pi_ {t}} (s , a) \right]}\right) \\ + \mathcal {O} \left(\delta L ^ {2} | S | | A | (t _ {i + 1} - t _ {i}) + \sqrt {L | S | | A | ^ {2} \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} C _ {t} ^ {\mathcal {P}}}\right). \\ \end{array}
$$
Proof. For each epoch  $i$  and state-action pair  $(s, a)$ , we have:
$$
\begin{array}{l} \mathbb {E} _ {t _ {i}} \left[ \sqrt {\sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \nu_ {t} (s , a)} \right] \leq \sqrt {\mathbb {E} _ {t _ {i}} \left[ \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \nu_ {t} (s , a) \right]} \\ = \mathcal {O} \left(\sqrt {\mathbb {E} _ {t _ {i}} \left[ \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} L ^ {2} q ^ {\widehat {P} _ {t} , \pi_ {t}} (s , a) \left(1 - \pi_ {t} (a | s)\right) + L | S | C _ {t} ^ {\mathsf {P}} \right]}\right), \tag {42} \\ \end{array}
$$
where the first step follows from Jensen's inequality, and the second step uses Lemma C.7.2.
Note that, for  $a = \pi (s)$  , we have:
$$
q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, \pi (s)) (1 - \pi_ {t} (\pi (s) | s)) \leq q ^ {\widehat {P} _ {t}, \pi_ {t}} (s) \sum_ {b \neq \pi (s)} \pi_ {t} (b | s) \leq \sum_ {b \neq \pi (s)} q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, b).
$$
$$
\begin{array}{l} \mathbb {E} _ {t _ {i}} \left[ \sum_ {a \in A} \sqrt {\sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \nu_ {t} (s , a)} \right] \\ = \mathcal {O} \left(\sum_ {a \in A} \sqrt {\mathbb {E} _ {t _ {i}} \left[ \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} L ^ {2} q ^ {\widehat {P} _ {t} , \pi_ {t}} (s , a) (1 - \pi_ {t} (a | s)) + L | S | C _ {t} ^ {\mathsf {P}} \right]}\right) \\ = \mathcal {O} \left(\sum_ {a \in A} \sqrt {\mathbb {E} _ {t _ {i}} \left[ L ^ {2} \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} q ^ {\widehat {P} _ {t} , \pi_ {t}} (s , a) (1 - \pi_ {t} (a | s)) \right]} + \sqrt {L | S | \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} C _ {t} ^ {\mathbf {P}}} \right) \\ = \mathcal {O} \left(\sum_ {a \neq \pi (s)} \sqrt {\mathbb {E} _ {t _ {i}} \left[ L ^ {2} \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} q ^ {\widehat {P} _ {t} , \pi_ {t}} (s , a) \right]} + \sqrt {L | S | | A | ^ {2} \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} C _ {t} ^ {\mathsf {P}}} \right) \\ \leq \mathcal {O} \left(\delta L ^ {2} | S | | A | \left(t _ {i + 1} - 1 - t _ {i}\right) + \sum_ {a \neq \pi (s)} \sqrt {\mathbb {E} _ {t _ {i}} \left[ L ^ {2} \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} q ^ {P , \pi_ {t}} (s , a) \right]} + \sqrt {L | S | | A | ^ {2} \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} C _ {t} ^ {\mathsf {P}}}\right) \\ \leq \mathcal {O} \left(\delta L ^ {2} | S | | A | \left(t _ {i + 1} - 1 - t _ {i}\right) + \sum_ {a \neq \pi (s)} \sqrt {\mathbb {E} _ {t _ {i}} \left[ L ^ {2} \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} q ^ {P _ {t} , \pi_ {t}} (s , a) \right]} + \sqrt {L | S | | A | ^ {2} \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} C _ {t} ^ {\mathsf {P}}}\right), \\ \end{array}
$$
where the first step follows from Eq. (42); the second step uses the fact that  $\sqrt{x + y} \leq \sqrt{x} + \sqrt{y}$  for  $x, y \geq 0$ ; the fourth step follows from Lemma D.1.1 with the event  $\mathcal{E}_{\mathrm{CON}}$  and Corollary C.8.2; the last step applies Corollary D.3.6.
C.7.2 Bounding  $\mathrm{ESTREG}_i(\pi^{\star})$  for Varying Learning Rate
We now focus on bounding  $\mathrm{ESTREG}_i(\pi^\star)$  for an individual epoch  $i$ .
Notations for FTRL Analysis. For any fixed epoch  $i$  and any integer  $t \in [t_i, t_{i+1} - 1]$ , we introduce the following notations.
$$
F _ {t} (q) = \left\langle q, \sum_ {\tau = t _ {i}} ^ {t - 1} \left(\widehat {\ell} _ {\tau} - b _ {\tau}\right) \right\rangle + \phi_ {t} (q), \quad G _ {t} (q) = \left\langle q, \sum_ {\tau = t _ {i}} ^ {t} \left(\widehat {\ell} _ {\tau} - b _ {\tau}\right) \right\rangle + \phi_ {t} (q), \tag {43}
$$
$$
q _ {t} = \operatorname * {a r g m i n} _ {q \in \Omega (\tilde {P} _ {i})} F _ {t} (q)  , \quad \widetilde {q} _ {t} = \operatorname * {a r g m i n} _ {q \in \Omega (\tilde {P} _ {i})} G _ {t} (q).
$$
With these notations, we have  $q_{t} = \widehat{q}_{t} = q^{\widehat{P}_{t},\pi_{t}} = q^{\widetilde{P}_{i},\pi_{t}}$ . Also, according to the loss shifting technique (specifically Corollary D.4.2),  $q_{t}$  and  $\widetilde{q}_{t}$  can be equivalently written as
$$
q _ {t} = \underset {q \in \Omega (\tilde {P} _ {i})} {\operatorname {a r g m i n}} \left\langle q, \sum_ {\tau = t _ {i}} ^ {t - 1} \left(g _ {\tau} - b _ {\tau}\right) \right\rangle + \phi_ {t} (q), \quad \widetilde {q} _ {t} = \underset {q \in \Omega (\tilde {P} _ {i})} {\operatorname {a r g m i n}} \left\langle q, \sum_ {\tau = t _ {i}} ^ {t} \left(g _ {\tau} - b _ {\tau}\right) \right\rangle + \phi_ {t} (q), \tag {44}
$$
where  $g_t: S \times A \to \mathbb{R}$  is defined as
$$
g _ {t} (s, a) = Q ^ {\widehat {P} _ {t}, \pi_ {t}} (s, a; \widehat {\ell_ {t}}) - V ^ {\widehat {P} _ {t}, \pi_ {t}} (s; \widehat {\ell_ {t}}), \forall (s, a) \in S \times A. \tag {45}
$$
Finally, We also define  $u = q^{\widetilde{P}_i,\pi^*}$  and
$$
v = \left(1 - \frac {1}{T ^ {2}}\right) u + \frac {1}{T ^ {2} | A | | S |} \sum_ {s, a} q _ {s, a} ^ {\max }, \tag {46}
$$
where  $q_{s,a}^{\max}$  is the occupancy measure associated with a policy that maximizes the probability of visiting  $(s,a)$  given the optimistic transition  $\widetilde{P}_i$ . Then, we have  $v \in \Omega(\widetilde{P}_i)$  because  $u, q_{s,a}^{\max} \in \Omega(\widetilde{P}_i)$  for all  $(s,a)$ , and  $\Omega(\widetilde{P}_i)$  is convex [Jin and Luo, 2020, Lemma 10].
Therefore,  $\mathrm{ESTREG}_i(\pi^{\star})$  can be rewritten as:
$$
\begin{array}{l} \operatorname {E S T R E G} _ {i} \left(\pi^ {\star}\right) = \mathbb {E} _ {t _ {i}} \left[ \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \left\langle q _ {t} - u, \widehat {\ell} _ {t} - b _ {t} \right\rangle \right] \\ = \underbrace {\mathbb {E} _ {t _ {i}} \left[ \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \left\langle q _ {t} - v , \widehat {\ell} _ {t} - b _ {t} \right\rangle \right]} _ {=: (\mathrm {I})} + \underbrace {\mathbb {E} _ {t _ {i}} \left[ \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \left\langle v - u , \widehat {\ell} _ {t} - b _ {t} \right\rangle \right]} _ {=: (\mathrm {I I})}, \tag {47} \\ \end{array}
$$
where term (I) is equivalent to
$$
\mathbb {E} _ {t _ {i}} \left[ \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \left\langle q _ {t} - v, g _ {t} - b _ {t} \right\rangle \right], \tag {48}
$$
because  $\left\langle q_t - v, g_t - \widehat{\ell}_t \right\rangle = 0$  according to Lemma D.4.1.
We then present the following lemma to provide a bound for  $\mathrm{ESTREG}_i(\pi^{\star})$ .
Lemma C.7.5. For  $g_{t}$  define in Eq. (45),  $v$  defined in Eq. (46), and any non-decreasing learning rate such that  $\gamma_{1}(s,a)\geq 256L^{2}|S|$  for all  $(s,a)$ , Algorithm 4 ensures that  $\mathrm{ESTREG}_i(\pi^{\star})$  is bounded by
$$
\begin{array}{l} \mathcal {O} \left(\frac {L + L | S | C ^ {P}}{T} + \mathbb {E} _ {t _ {i}} \left[ \sum_ {s \neq s _ {L}} \sum_ {a \in A} \gamma_ {t _ {i + 1} - 1} (s, a) \log (\iota) \right]\right) \tag {49} \\ { + } { \mathcal { O } \left( \operatorname* { m i n } \left\{ \mathbb { E } _ { t _ { i } } \left[ \sum _ { t = t _ { i } } ^ { t _ { i + 1 } - 1 } \left\| \widehat { \ell _ { t } } - b _ { t } \right\| _ { \nabla ^ { - 2 } \phi _ { t } ( q _ { t } ) } ^ { 2 } \right] , \mathbb { E } _ { t _ { i } } \left[ \sum _ { t = t _ { i } } ^ { t _ { i + 1 } - 1 } \| g _ { t } - b _ { t } \| _ { \nabla ^ { - 2 } \phi _ { t } ( q _ { t } ) } ^ { 2 } \right] \right\} \right) , } \\ \end{array}
$$
Proof. We start from the decomposition in Eq. (47).
Bounding (I). By adding and subtracting  $F_{t}(q_{t}) - G_{t}(\tilde{q}_{t})$ , we decompose (I) into a stability term and a penalty term:
$$
\underbrace {\mathbb {E} _ {t _ {i}} \left[ \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \left(\left\langle q _ {t} , \widehat {\ell} _ {t} - b _ {t} \right\rangle + F _ {t} (q _ {t}) - G _ {t} (\tilde {q} _ {t})\right) \right]} _ {\text {S t a b i l i t y t e r m}} + \underbrace {\mathbb {E} _ {t _ {i}} \left[ \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \left(G _ {t} (\tilde {q} _ {t}) - F _ {t} (q _ {t}) - \left\langle v , \widehat {\ell} _ {t} - b _ {t} \right\rangle\right) \right]} _ {\text {P e n a l t y t e r m}}.
$$
As  $\gamma_1(s,a)\geq 256L^2 |S|$ , Lemma C.7.6 ensures  $\frac{1}{2} q_t(s,a)\leq \widetilde{q}_t(s,a)\leq 2q_t(s,a)$  for all  $(s,a)$ , which allows us to apply [Jin and Luo, 2020, Lemma 13] to bound the stability term by
$$
\text {S t a b i l i t y} = \mathcal {O} \left(\mathbb {E} _ {t _ {i}} \left[ \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \left\| \widehat {\ell} _ {t} - b _ {t} \right\| _ {\nabla^ {- 2} \phi_ {t} (q _ {t})} ^ {2} \right]\right). \tag {50}
$$
The penalty term without expectation is bounded as:
$$
\begin{array}{l} \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \left(G _ {t} (\tilde {q} _ {t}) - F _ {t} (q _ {t}) - \left\langle v, \widehat {\ell_ {t}} - b _ {t} \right\rangle\right) \\ = - F _ {t _ {i}} \left(q _ {t _ {i}}\right) + \sum_ {t = t _ {i} + 1} ^ {t _ {i + 1} - 1} \left(G _ {t - 1} \left(\tilde {q} _ {t - 1}\right) - F _ {t} \left(q _ {t}\right)\right) + G _ {t _ {i + 1} - 1} \left(\tilde {q} _ {t _ {i + 1} - 1}\right) - \left\langle v, \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \left(\widehat {\ell} _ {t} - b _ {t}\right) \right\rangle \\ \leq - F _ {t _ {i}} \left(q _ {t _ {i}}\right) + \sum_ {t = t _ {i} + 1} ^ {t _ {i + 1} - 1} \left(G _ {t - 1} \left(q _ {t}\right) - F _ {t} \left(q _ {t}\right)\right) + G _ {t _ {i + 1} - 1} (v) - \left\langle v, \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \left(\widehat {\ell} _ {t} - b _ {t}\right) \right\rangle \\ = - \phi_ {t _ {i}} \left(q _ {t _ {i}}\right) + \sum_ {t = t _ {i} + 1} ^ {t _ {i + 1} - 1} \left(\phi_ {t - 1} \left(q _ {t}\right) - \phi_ {t} \left(q _ {t}\right)\right) + \phi_ {t _ {i + 1} - 1} (v) \\ = - \phi_ {t _ {i}} \left(q _ {t _ {i}}\right) + \sum_ {t = t _ {i} + 1} ^ {t _ {i + 1} - 1} \sum_ {s \neq s _ {L}} \sum_ {a \in A} \left(\gamma_ {t - 1} (s, a) - \gamma_ {t} (s, a)\right) \log \left(\frac {1}{q _ {t} (s , a)}\right) + \phi_ {t _ {i + 1} - 1} (v) \\ = - \phi_ {t _ {i}} \left(q _ {t _ {i}}\right) + \phi_ {t _ {i}} (v) + \sum_ {t = t _ {i} + 1} ^ {t _ {i + 1} - 1} \sum_ {s \neq s _ {L}} \sum_ {a \in A} \left(\gamma_ {t} (s, a) - \gamma_ {t - 1} (s, a)\right) \log \left(\frac {q _ {t} (s , a)}{v (s , a)}\right) \\ = \sum_ {s \neq s _ {L}} \sum_ {a \in A} \gamma_ {t _ {i}} (s, a) \log \left(\frac {q _ {t _ {i}} (s , a)}{v (s , a)}\right) + \sum_ {t = t _ {i} + 1} ^ {t _ {i + 1} - 1} \sum_ {s \neq s _ {L}} \sum_ {a \in A} \left(\gamma_ {t} (s, a) - \gamma_ {t - 1} (s, a)\right) \log \left(\frac {q _ {t} (s , a)}{v (s , a)}\right) \\ \leq \sum_ {s \neq s _ {L}} \sum_ {a \in A} \gamma_ {t _ {i}} (s, a) \log \left(\frac {\iota^ {2} q _ {t _ {i}} (s , a)}{q _ {s , a} ^ {\max } (s , a)}\right) + \sum_ {t = t _ {i} + 1} ^ {t _ {i + 1} - 1} \sum_ {s \neq s _ {L}} \sum_ {a \in A} (\gamma_ {t} (s, a) - \gamma_ {t - 1} (s, a)) \log \left(\frac {\iota^ {2} q _ {t} (s , a)}{q _ {s , a} ^ {\max } (s , a)}\right) \\ \leq 2 \sum_ {s \neq s _ {L}} \sum_ {a \in A} \gamma_ {t _ {i}} (s, a) \log (\iota) + 2 \sum_ {t = t _ {i} + 1} ^ {t _ {i + 1} - 1} \sum_ {s \neq s _ {L}} \sum_ {a \in A} \left(\gamma_ {t} (s, a) - \gamma_ {t - 1} (s, a)\right) \log (\iota) \\ = 2 \sum_ {s \neq s _ {L}} \sum_ {a \in A} \gamma_ {t _ {i + 1} - 1} (s, a) \log (\iota), \tag {51} \\ \end{array}
$$
where the second step uses the optimality of  $\tilde{q}_{t-1}$  and  $\tilde{q}_{t_{i+1}-1}$  so that  $G_{t-1}(\tilde{q}_{t-1}) \leq G_{t-1}(q_t)$  and  $G_{t_{i+1}-1}(\tilde{q}_{t_{i+1}-1}) \leq G_{t_{i+1}-1}(v)$ ; the third step follows the definitions in Eq. (43); the seventh step lower-bounds  $v(s, a)$  by  $\frac{1}{\nu^2} q_{s, a}^{\max}(s, a)$  for each  $(s, a)$ ; the eighth step uses the definition of  $q_{s, a}^{\max}$ .
Now, by taking the equivalent perspective in Eq. (44) and Eq. (48) and repeating the exact same argument, the same bound holds with  $\widehat{\ell}_t$  replaced by  $g_{t}$ . Thus, we have shown
$$
\begin{array}{l} (I) = \mathcal {O} \left(\mathbb {E} _ {t _ {i}} \left[ \sum_ {s \neq s _ {L}} \sum_ {a \in A} \gamma_ {t _ {i + 1} - 1} (s, a) \log (\iota) \right]\right) \tag {52} \\ + \mathcal {O} \left(\min  \left\{\mathbb {E} _ {t _ {i}} \left[ \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \left\| \widehat {\ell_ {t}} - b _ {t} \right\| _ {\nabla^ {- 2} \phi_ {t} (q _ {t})} ^ {2} \right], \mathbb {E} _ {t _ {i}} \left[ \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \| g _ {t} - b _ {t} \| _ {\nabla^ {- 2} \phi_ {t} (q _ {t})} ^ {2} \right] \right\}\right). \\ \end{array}
$$
Bounding (II). By direct calculation, we have
$$
\begin{array}{l} \left(\mathrm {I I}\right) = \mathbb {E} _ {t _ {i}} \left[ \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \left\langle v - u, \widehat {\ell} _ {t} - b _ {t} \right\rangle \right] \\ = \frac {1}{T ^ {2}} \mathbb {E} _ {t _ {i}} \left[ \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \left\langle \frac {1}{| A | | S |} \sum_ {s, a} q _ {s, a} ^ {\max } - u, \widehat {\ell} _ {t} - b _ {t} \right\rangle \right] \\ \leq \frac {1}{T ^ {2}} \mathbb {E} _ {t _ {i}} \left[ \left\| \frac {1}{| A | | S |} \sum_ {s, a} q _ {s, a} ^ {\max} - u \right\| _ {1} \left\| \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} (\ell_ {t} - b _ {t}) \right\| _ {\infty} \right], \\ \end{array}
$$
where the first step uses the definition of  $v$  in Eq. (46) and the third step applies the Hölder's inequality. Then, we further show:
$$
\begin{array}{l} (\mathrm {I I}) \leq \frac {1}{T ^ {2}} \mathbb {E} _ {t _ {i}} \left[ \left\| \frac {1}{| A | | S |} \sum_ {s, a} q _ {s, a} ^ {\max} - u \right\| _ {1} \left\| \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} (\ell_ {t} - b _ {t}) \right\| _ {\infty} \right] \\ \leq \frac {1}{T ^ {2}} \mathbb {E} _ {t _ {i}} \left[ \left(\frac {1}{| A | | S |} \sum_ {s, a} \left\| q _ {s, a} ^ {\max } \right\| _ {1} + \| u \| _ {1}\right) \left\| \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \left(\ell_ {t} - b _ {t}\right) \right\| _ {\infty} \right] \\ \leq \frac {1}{T ^ {2}} \mathbb {E} _ {t _ {i}} \left[ 2 L \left(\left\| \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \ell_ {t} \right\| _ {\infty} + \left\| \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} b _ {t} \right\| _ {\infty}\right) \right] \\ \leq \mathbb {E} _ {t _ {i}} \left[ \frac {2 L (t _ {i + 1} - t _ {i}) + 4 L C ^ {\mathsf {P}} | S | T}{T ^ {2}} \right] \\ \leq \frac {2 L + 4 L C ^ {\mathrm {P}} | S |}{T}, \tag {53} \\ \end{array}
$$
where the second step repeatedly applies the triangle inequality, and those terms are bounded by  $2L$  in the next step; the fourth step bounds  $\ell_t(s,a)\leq 1$  for all  $(s,a)$  and bounds  $\sum_{t = t_i}^{t_{i + 1} - 1}b_t(s)\leq 2C^{\mathsf{P}}|S|T$  by using the fact  $u_{t}(s)\geq \frac{1}{|S|T}$  for all  $t,s$  (see Lemma D.2.8); the fifth step bounds  $t_{i + 1} - t_i$  by  $T$ .
Finally, we combine the bounds in Eq. (52) and Eq. (53) to complete the proof.
Lemma C.7.6 (Multiplicative Stability). For  $\widetilde{q}_t$  and  $q_{t}$  defined in Eq. (44), if the learning rate fulfills  $\gamma_t(s,a)\geq \gamma_1(s,a) = 256L^2 |S|$  for all  $t$  and  $(s,a)$ , then,  $\frac{1}{2} q_{t}(s,a)\leq \widetilde{q}_{t}(s,a)\leq 2q_{t}(s,a)$  holds for all  $(s,a)$ .
$$
\begin{array}{l} \left\| \widehat {\ell} _ {t} - b _ {t} \right\| _ {\nabla^ {- 2} \phi_ {t} (q _ {t})} ^ {2} = \sum_ {s, a} \left(\frac {\mathbb {I} _ {t} (s , a) \ell_ {t} (s , a)}{u _ {t} (s , a)} - b _ {t} (s)\right) ^ {2} \frac {q _ {t} (s , a) ^ {2}}{\gamma_ {t} (s , a)} \\ \leq \sum_ {s, a} \left(\frac {\mathbb {I} _ {t} (s , a) \ell_ {t} (s , a) ^ {2}}{u _ {t} (s , a) ^ {2}} + b _ {t} (s) ^ {2}\right) \frac {q _ {t} (s , a) ^ {2}}{\gamma_ {t} (s , a)} \\ \leq \sum_ {s, a} \left(\frac {\mathbb {I} _ {t} (s , a)}{u _ {t} (s , a) ^ {2}} + \frac {1 6 L ^ {2}}{u _ {t} (s) ^ {2}}\right) \frac {q _ {t} (s , a) ^ {2}}{\gamma_ {t} (s , a)} \\ \leq \sum_ {s, a} \frac {\mathbb {I} _ {t} (s , a)}{2 5 6 L ^ {2} | S |} + \sum_ {s, a} \frac {1 6 L ^ {2} \cdot q _ {t} (s , a) ^ {2}}{2 5 6 L ^ {2} | S | \cdot u _ {t} (s) ^ {2}} \\ \leq \frac {1}{2 5 6 L | S |} + \sum_ {s \neq s _ {L}} \frac {1}{1 6 | S |} \leq \frac {1}{8}, \tag {54} \\ \end{array}
$$
where the second step uses  $(x - y)^2 \leq x^2 + y^2$  for any  $x, y \geq 0$ ; the third step bounds  $\ell_t(s, a)^2 \leq 1$  and  $b_t(s)^2 \leq \frac{16L^2}{u_t(s)^2}$  according to the definition of Eq. (7); the fourth step holds as  $\gamma_t(s, a) \geq \gamma_1(s, a) = 256L^2|S|$  and  $q_t(s, a) \leq u_t(s, a)$  for all  $(s, a)$ ; the fifth steps use the fact that  $\sum_a q_t(s, a)^2 \leq (\sum_a q_t(s, a))^2 \leq u_t(s)^2$ .
Once Eq. (54) holds, one only needs to repeat the same argument of [Jin and Luo, 2020, Lemma 12] to obtain the claimed multiplicative stability.
Lemma C.7.7. Under the conditions of Lemma C.7.5, event  $\mathcal{E}_{\mathrm{CON}}$ , and the learning  $\gamma_{t}(s,a)$  defined in Definition 5.2, the following holds for any deterministic policy  $\pi^{\star}:S\to A$ :
$$
\mathbb {E} _ {t _ {i}} \left[ \sum_ {s \neq s _ {L}} \sum_ {a \in A} \gamma_ {t _ {i + 1} - 1} (s, a) \log (\iota) + \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} \| g _ {t} - b _ {t} \| _ {\nabla^ {- 2} \phi_ {t} (q _ {t})} ^ {2} \right]
$$
$$
\begin{array}{l} = \mathcal {O} \left(\mathbb {E} _ {t _ {i}} \left[ L \sqrt {\log (\iota)} \sum_ {s \neq s _ {L}} \sum_ {a \neq \pi^ {\star} (s)} \sqrt {\mathbb {E} \left[ \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} q ^ {P _ {t} , \pi_ {t}} (s , a) \right]} \right] + C ^ {P} \log (| S | T)\right) \\ + \mathcal {O} \left(\mathbb {E} _ {t _ {i}} \left[ \delta L ^ {2} | S | ^ {2} | A | \log (\iota) (t _ {i + 1} - t _ {i}) + \sqrt {L | S | ^ {3} | A | ^ {2} \log (\iota) \sum_ {t = t _ {i}} ^ {t _ {i + 1} - 1} C _ {t} ^ {\mathcal {P}}} \right]\right). \\ \end{array}
$$
Proof. We start by bounding the second part for each  $t$ :
$$
\begin{array}{l} \| g _ {t} - b _ {t} \| _ {\nabla^ {- 2} \phi_ {t} (q _ {t})} ^ {2} = \sum_ {s, a} \frac {1}{\gamma_ {t} (s , a)} q _ {t} (s, a) ^ {2} \left(g _ {t} (s, a) - b _ {t} (s)\right) ^ {2} \\ \leq 2 \sum_ {s, a} \frac {\nu_ {t} (s , a)}{\gamma_ {t} (s , a)} + 2 \sum_ {s, a} \frac {1}{\gamma_ {t} (s , a)} q _ {t} (s, a) ^ {2} b _ {t} (s) ^ {2}, \\ \end{array}
$$
where the second step uses the definition of  $\nu_{t}$  and  $(x - y)^{2} \leq 2(x^{2} + y^{2})$  for any  $x, y \in \mathbb{R}$ .
We first focus on bounding  $\sum_{t = t_i}^{t_{i + 1} - 1}\sum_{s,a}\frac{1}{\gamma_t(s,a)} q_t(s,a)^2 b_t(s)^2$