SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Proof. At a high level, we prove this lower bound by noting that optimizing the dynamic regret of linear mixture MDPs is harder than (i) optimizing the static regret of linear mixture MDPs with the unknown transition kernel, (ii) optimizing the dynamic regret of linear mixture MDPs with the known transition kernel, both. Thus, we can consider the lower bound of these two problems separately and combine them to obtain the lower bound of the dynamic regret of linear mixture MDPs with the unknown transition kernel. We present the details below.
First, we consider the lower bound of optimizing the static regret of adversarial linear mixture MDPs with the unknown transition kernel. From lower bound in He et al. [33, Theorem 5.3], we have the following lower bound in this case since the dynamic regret is no smaller than the static regret.
$$
\operatorname {D} - \operatorname {R e g r e t} (K) \geq \Omega \left(\sqrt {d ^ {2} H ^ {3} K}\right). \tag {26}
$$
Then, we consider the lower bound of optimizing the dynamic regret of adversarial linear mixture MDPs with the known transition kernel. We note that Zimin and Neu [12] show the lower bound of the static regret for adversarial episodic loop-free SSP with known transition kernel is  $\Omega (H\sqrt{K\log(SA)})$ , we utilize this lower bound to establish our lower bound as the episodic loop-free SSP is a special case of linear mixture MDPs with  $d = S^2 A$ . We consider two cases:
Case 1:  $\Gamma \leq 2H$ . In this case, we can directly utilize the established lower bound of static regret as a natural lower bound of dynamic regret,
$$
\operatorname {D} - \operatorname {R e g r e t} (K) \geq \Omega \left(H \sqrt {K \log (S A)}\right). \tag {27}
$$
Case 2:  $\Gamma > 2H$ . Without loss of generality, we assume  $L = \lceil \Gamma / 2H \rceil$  divides  $K$  and split the whole episodes into  $L$  pieces equally. Next, we construct a special policy sequence such that the policy sequence is fixed within each piece and only changes in the split point. Since the sequence changes at most  $L - 1 \leq \Gamma / 2H$  times and the path length of the policy sequence at each change point is at most  $2H$ , the total path length in  $K$  episodes does not exceed  $\Gamma$ . As a result, we have
$$
\operatorname {D} - \operatorname {R e g r e t} (K) \geq L H \sqrt {K / L \log (S A)} = H \sqrt {K L \log (S A)} \geq \Omega (\sqrt {K H \Gamma \log (S A)}). \tag {28}
$$
Combining (27) and (28), we have the following lower bound for the dynamic regret of adversarial linear mixture MDPs with the known transition kernel,
$$
\operatorname {D} - \operatorname {R e g r e t} (K) \geq \Omega \left(\max  \left\{H \sqrt {K \log (S A)}, \sqrt {K H \Gamma \log (S A)} \right\}\right) \geq \Omega \left(\sqrt {K H (H + \Gamma) \log (S A)}\right), \tag {29}
$$
where the last inequality holds by  $\max \{a,b\} \geq (a + b) / 2$
Combining two lower bounds (26) and (29), we have the lower bound of the dynamic regret of adversarial linear mixture MDPs with the unknown transition kernel,
$$
\begin{array}{l} \mathrm {D} - \operatorname {R e g r e t} (K) \geq \Omega \left(\max  \left\{\sqrt {d ^ {2} H ^ {3} K}, \sqrt {K H (H + \Gamma) \log (S A)} \right\}\right) \\ \geq \Omega \big (\sqrt {d ^ {2} H ^ {3} K} + \sqrt {K H (H + \Gamma) \log (S A)} \big). \\ \end{array}
$$
C.9 Proof of Theorem 5
$$
\begin{array}{l} \mathrm {D} - \operatorname {R e g r e t} (K) = \sum_ {k = 1} ^ {K} V _ {k, 1} ^ {\pi_ {k} ^ {c}} \left(s _ {k, 1}\right) - \sum_ {k = 1} ^ {K} V _ {k, 1} ^ {\pi_ {k}} \left(s _ {k, 1}\right) \\ = \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} [ \langle Q _ {k, h} (s _ {h}, \cdot), \pi_ {k, h} ^ {c} (\cdot | s _ {h}) - \pi_ {k, h} (\cdot | s _ {h}) \rangle ] \\ + \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left(\mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \iota_ {k, h} \left(s _ {h}, a _ {h}\right) \right] - \iota_ {k, h} \left(s _ {k, h}, a _ {k, h}\right)\right) + \mathcal {M} _ {K, H}. \\ \end{array}
$$
By Lemma 10, we have  $\iota_{k,h}(s,a)\leq 0$  for any  $k\in [K],h\in [H],s\in S,a\in \mathcal{A}$ . Thus, we have
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {C}} [ \iota_ {k, h} (s _ {h}, a _ {h}) ] \leq 0.
$$
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left(- \iota_ {k, h} \left(s _ {k, h}, a _ {k, h}\right)\right) + \mathcal {M} _ {K, H} \leq \widetilde {\mathcal {O}} \left(\sqrt {d H ^ {4} K + d ^ {2} H ^ {3} K}\right).
$$
It remains to bound the first term. We decompose this term as follows.
$$
\begin{array}{l} \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \langle Q _ {k, h} ^ {\pi_ {k}} (s _ {h}, \cdot), \pi_ {k, h} ^ {c} (\cdot | s _ {h}) - \pi_ {k, h} (\cdot | s _ {h}) \rangle \right] \\ = \underbrace {\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \left\langle Q _ {k , h} ^ {\pi_ {k}} (s _ {h} , \cdot) , \pi_ {k , h} ^ {c} (\cdot \mid s _ {h}) - \pi_ {k , h} ^ {i} (\cdot \mid s _ {h}) \right\rangle \right]} _ {\text {b a s e - r e g r e t}} \\ + \underbrace {\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \left\langle Q _ {k , h} ^ {\pi_ {k}} \left(s _ {h} , \cdot\right) , \pi_ {k , h} ^ {i} (\cdot \mid s _ {h}) - \pi_ {k , h} (\cdot \mid s _ {h}) \right\rangle \right]} _ {\text {m e t a - r e g r e t}}, \tag {30} \\ \end{array}
$$
where the decomposition holds for any base-learner  $i \in N$ . Next, we bound the two terms separately.
Upper bound of base regret. From Theorem 1, we have
$$
\text {b a s e - r e g r e t} \leq \frac {\eta_ {i} K H ^ {3}}{2} + \frac {1}{\eta_ {i}} \left(H \log A + (1 + H) \log \frac {A}{\gamma} P _ {T} + K H \log \frac {1}{1 - \gamma}\right)
$$
Set  $\gamma = 1 / KH$  and note that  $\log (1 / (1 - \gamma))\leq \gamma /(1 - \gamma)$  for all  $\gamma >0$ . Then, we have
$$
\begin{array}{l} \text {b a s e - r e g r e t} \leq \frac {\eta_ {i} K H ^ {3}}{2} + \frac {1}{\eta_ {i}} (H \log A + (1 + H) P _ {T} \log (K H A) + 2) \\ \leq \frac {\eta_ {i} K H ^ {3}}{2} + \frac {2 H}{\eta_ {i}} \left(\log A + P _ {T} \log (K H A)\right). \\ \end{array}
$$
It is clear that the optimal learning rate  $\eta^{*} = \sqrt{4(\log A + P_{T}\log(KHA)) / (KH^{2})}$ . By the definition  $P_{T} = \sum_{k = 1}^{K}\sum_{h = 1}^{H}\| \pi_{k,h}^{c} - \pi_{k - 1,h}^{c}\|_{1,\infty}$ , it holds that  $0\leq P_{T}\leq 2KH$ . Therefore, the range of the optimal learning rate is
$$
\eta_ {\min } = \sqrt {\frac {4 \log A}{K H ^ {2}}}, \text {a n d} \eta_ {\max } = \sqrt {\frac {4 \log A + 8 K H \log (K H A)}{K H ^ {2}}}.
$$
From the construction of the step size pool  $\mathcal{H} = \{\eta_i = (2^i / H)\sqrt{(\log A) / K} \mid i \in [N]\}$  with  $N = \left\lfloor \frac{1}{2}\log \left(\frac{K}{\log A}\right)\right\rfloor$ , we know that the step size therein is monotonically increasing, in particular
$$
\eta_ {1} = \sqrt {\frac {4 \log A}{K H ^ {2}}}, \text {a n d} \eta_ {N} = \frac {1}{H}.
$$
In the following, we consider two cases:
Case 1:  $\eta^{*} \in [\eta_{1}, \eta_{N}]$ . In this case, we ensure there exists  $i^{*} \in N$  such that  $\eta_{i^{*}} \leq \eta^{*} \leq 2\eta_{i^{*}}$ . Note the decomposition in (30) holds for any base-learner. Therefore, we choose the base-learner whose step size is  $\eta_{i^{*}}$  and have
$$
\begin{array}{l} \text {b a s e - r e g r e t} \leq \frac {\eta_ {i ^ {*}} K H ^ {3}}{2} + \frac {2 H}{\eta_ {i ^ {*}}} (\log A + P _ {T} \log (K H A)) \\ \leq \frac {\eta^ {*} K H ^ {3}}{2} + \frac {4 H}{\eta^ {*}} (\log A + P _ {T} \log (K H A)) \\ = 3 \sqrt {K H ^ {4} (\log A + P _ {T} \log (K H A))}, \\ \end{array}
$$
where the second inequality holds by the condition that  $\eta_{i^*} \leq \eta^* \leq 2\eta_{i^*}$  and the last equality holds by substituting  $\eta^* = \sqrt{4(\log A + P_T\log(KHA)) / (KH^2)}$ .
Case 2:  $\eta^{*} > \eta_{N}$ . In this case, we know that  $4(\log A + P_{T}\log (KHA)) > K$ . Therefore, we choose the base-learner whose step size is  $\eta_{N}$  and have
$$
\begin{array}{l} \text {b a s e - r e g r e t} \leq \frac {\eta_ {N} K H ^ {3}}{2} + \frac {2 H}{\eta_ {N}} \left(\log A + P _ {T} \log (K H A)\right) \\ = \frac {K H ^ {2}}{2} + 2 H ^ {2} (\log A + P _ {T} \log (K H A)) \\ \leq 4 H ^ {2} \left(\log A + P _ {T} \log (K H A)\right), \\ \end{array}
$$
where the last inequality holds by the condition that  $4(\log A + P_T\log (KHA)) > K$ .
Summing over the two upper bounds yields
$$
\begin{array}{l} \text {b a s e - r e g r e t} \leq 3 \sqrt {K H ^ {4} (\log A + P _ {T} \log (K H A))} + 4 H ^ {2} (\log A + P _ {T} \log (K H A)) \\ \leq 4 \sqrt {H ^ {4} (K + \log A + P _ {T} \log (K H A)) (\log A + P _ {T} \log (K H A))} \\ = \tilde {\mathcal {O}} \left(\sqrt {H ^ {4} \left(K + P _ {T}\right) \left(1 + P _ {T}\right)}\right), \tag {31} \\ \end{array}
$$
where the inequality holds by  $\sqrt{a} +\sqrt{b}\leq \sqrt{2(a + b)},\forall a,b\geq 0$
Upper bound of meta-regret. For meta-regret, we have
$$
\begin{array}{l} \text {m e t a - r e g r e t} = \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \left\langle Q _ {k, h} (s _ {h}, \cdot), \pi_ {k, h} ^ {i} (\cdot \mid s _ {h}) - \pi_ {k, h} (\cdot \mid s _ {h}) \right\rangle \right] \\ = \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \langle e ^ {i} (s _ {h}) - p _ {k, h} ^ {i} (s _ {h}), Q _ {k, h} (s _ {h}, \cdot) \cdot \pi_ {k, h} ^ {i} (\cdot | s _ {h}) \rangle \right] \\ \leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \left\langle e ^ {i _ {k, h} ^ {*} (s _ {h})} - p _ {k, h} ^ {i} (s _ {h}), Q _ {k, h} (s _ {h}, \cdot) \cdot \pi_ {k, h} ^ {i} (\cdot | s _ {h}) \right\rangle \right] \\ \leq \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \left\langle e ^ {i _ {k, h} ^ {*} (s _ {h})} - e ^ {i _ {k - 1, h} ^ {*}} (s _ {h}), Q _ {k, h} (s _ {h}, \cdot) \cdot \pi_ {k, h} ^ {i} (\cdot | s _ {h}) \right\rangle \right] \\ \leq H \sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \mathbb {E} _ {\pi_ {k} ^ {c}} \left[ \| e ^ {i _ {k, h} ^ {*}} (s _ {h}) - e ^ {i _ {k - 1, h} ^ {*}} (s _ {h}) \| _ {1} \right] \\ \leq 2 H S _ {T}, \tag {32} \\ \end{array}
$$
where the first inequality holds by the definition that  $i_{k,h}^{*} = \arg \max_{i\in [N]}\langle Q_{k,h}(s_{h},\cdot),\pi_{k,h}^{i}(\cdot \mid s_{h})\rangle$  the second inequality holds due to  $p_{k,h}^{i}(s_{h}) = e^{i_{k - 1,h}^{*}}(s_{h})$ , and the last equality holds by the definition  $S_{T} = \sum_{k = 1}^{K}\sum_{h = 1}^{H}\mathbb{E}_{\pi_{k}^{c}}\mathbb{1}[i_{k,h}^{*}(s_{h})\neq i_{k - 1,h}^{*}(s_{h})]$ .
Combining (31) and (32), by  $\sqrt{a} +\sqrt{b}\leq \sqrt{2(a + b)},\forall a,b\geq 0$  , we have
$$
\mathrm {D} - \operatorname {R e g r e t} (K) \leq \widetilde {\mathcal {O}} \Big (\sqrt {d H ^ {4} K + d ^ {2} H ^ {3} K} + \sqrt {H ^ {4} (K + P _ {T}) (1 + P _ {T}) + H ^ {2} S _ {T} ^ {2}} \Big).
$$
D Supporting Lemmas
In this section, we introduce the supporting lemmas used in the proofs.
First, we introduce the performance difference lemma which connects the difference between two policies to the difference between their expected total rewards through the Q-function.
Lemma 13 (Cai et al. [32, Lemma 3.2]). For any policies  $\pi, \pi' \in \Delta(\mathcal{A} \mid S, H)$ , it holds that
$$
V _ {k, 1} ^ {\pi^ {\prime}} (s _ {k, 1}) - V _ {k, 1} ^ {\pi} (s _ {k, 1}) = \mathbb {E} _ {\pi^ {\prime}} \left[ \sum_ {h = 1} ^ {H} \langle Q _ {k, h} ^ {\pi} (s _ {h}, \cdot), \pi_ {h} ^ {\prime} (\cdot | s _ {h}) - \pi_ {h} (\cdot | s _ {h}) \rangle \Big | s _ {1} = s _ {k, 1} \right].
$$
Then, we introduce the following lemmas which show the "one-step descent" guarantee.
Lemma 14 (Cai et al. [32, Lemma 3.3]). For any distributions  $p^*, p \in \Delta(\mathcal{A})$ , state  $s \in S$ , and function  $Q: \mathcal{S} \times \mathcal{A} \to [0, H]$ , it holds for  $p' \in \Delta(\mathcal{A})$  with  $p'(\cdot) \propto p(\cdot) \cdot \exp(\eta \cdot Q(s, \cdot))$  that
$$
\langle Q (s, \cdot), p ^ {*} (\cdot) - p (\cdot) \rangle \leq \eta H ^ {2} / 2 + \eta^ {- 1} \cdot \left(D _ {\mathrm {K L}} \left(p ^ {*} (\cdot) \| p (\cdot)\right) - D _ {\mathrm {K L}} \left(p ^ {*} (\cdot) \| p ^ {\prime} (\cdot)\right)\right).
$$
Next, we introduce the law of total variance, which bounds the variance of the value function.
Lemma 15 (Jin et al. [55, Lemma C.5]). With probability at least  $1 - \delta$ , it holds that
$$
\sum_ {k = 1} ^ {K} \sum_ {h = 1} ^ {H} \left[ \mathbb {V} _ {h} V _ {h + 1} ^ {\pi^ {k}} \right] \left(s _ {h} ^ {k}, a _ {h} ^ {k}\right) \leq 3 \left(H K + H ^ {3} \log \frac {1}{\delta}\right).
$$
Finally, we introduce the elliptical potential lemma, which is a key lemma in online linear regression.
Lemma 16 (Abbasi-Yadkori et al. [51, Lemma 11]). Let  $\{\mathbf{x}_t\}_{t=1}^{\infty}$  be a sequence in  $\mathbb{R}^d$  space,  $\mathbf{V}_0 = \lambda \mathbf{I}$  and define  $\mathbf{V}_t = \mathbf{V}_0 + \sum_{i=1}^{t} \mathbf{x}_i \mathbf{x}_i^\top$ . If  $\|\mathbf{x}_i\|_2 \leq L$ ,  $\forall i \in \mathbb{Z}_+$ , then for each  $t \in \mathbb{Z}_+$ ,
$$
\sum_ {i = 1} ^ {t} \min \left\{1, \| \mathbf {x} _ {i} \| _ {\mathbf {V} _ {i - 1} ^ {- 1}} \right\} \leq 2 d \log \left(\frac {d \lambda + t L ^ {2}}{d \lambda}\right).
$$