SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\sum_ {t \in I _ {i}} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t \in I _ {i}} \widetilde {L} _ {t} (\widehat {\theta} _ {t, j}) = \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} (\bar {\theta} _ {t}) - \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} (\bar {\theta} _ {t, j}) \leq C _ {T} ^ {(2)} (\delta) = \mathcal {O} (\beta \log (T / \delta) + \beta \ln K).
$$
In above, the first equality is due to Lemma 11 and the definition of the surrogate base-learner  $\mathcal{E}_i^\prime$  such that  $\bar{\theta}_{t,i} = \widehat{\theta}_{t,i}$  for any  $t\in I_i$  and  $\bar{\theta}_{t,i} = \widehat{\theta}_t$  for any  $t\in [T] / I_i$ . The first inequality is due to Lemma 12. In the last equality, we treat double logarithmic factors in  $T$  as a constant, following previous studies [73, 74]. We have completed the proof.
D.3.2 Useful Lemma
Proof of Lemma 11. We show that the predictions  $\widehat{\theta}_t$  and  $\bar{\theta}_t$  by the two algorithm are exactly the same for any  $t\in [T]$  by induction. At iteration  $t\in [T]$ , we have the following two induction hypotheses:
- IH1: the final predictions satisfy  $\widehat{\theta}_{\tau} = \bar{\theta}_{\tau}$  for all  $\tau \leq t$
- IH2: any base-algorithms  $\mathcal{E}_i \in \mathcal{A}_{\tau}$  satisfy  $p_{\tau, i} = \bar{p}_{\tau, i} / (\sum_{i \in \mathcal{A}_{\tau}} \bar{p}_{\tau, i})$  for all  $\tau \leq t$ .
Base case: For the base case of  $t = 1$ , there is only one active base-algorithm. Denote by  $i_1$  the index of the active base-algorithm, we have  $\widehat{\theta}_1 = \widehat{\theta}_{1,i_1}$ ,  $p_{1,i_1} = 1$  and  $\mathcal{A}_1 = \{i_1\}$ . Moreover, for the surrogate algorithm, we have  $\bar{p}_{1,i} = 1 / K$  for any  $i \in [K]$  and  $\bar{\theta}_1 = (\sum_{i \in [K] / \{i_1\}} \widehat{\theta}_1 + \widehat{\theta}_{1,i_1}) / |K| = \widehat{\theta}_1$ . We can check that  $\widehat{\theta}_1 = \bar{\theta}_1$  and  $p_{1,i} = \bar{p}_{1,i} / \sum_{i \in \mathcal{A}_1} \bar{p}_{1,i}$  for any  $i \in \mathcal{A}_1$ .
Induction step: For the induction step, we first show
$$
p _ {t + 1, i} = \frac {\bar {p} _ {t + 1 , i}}{\sum_ {i \in \mathcal {A} _ {t + 1}} \bar {p} _ {t + 1 , i}} \tag {46}
$$
for all  $i\in \mathcal{A}_{t + 1}$  . By the definition of  $\bar{p}_{t + 1,i}$  , it is sufficient to prove
$$
p _ {t + 1, i} = \frac {\bar {\varepsilon} _ {t , i} \bar {v} _ {t , i}}{\sum_ {i \in \mathcal {A} _ {t + 1}} \bar {\varepsilon} _ {t , i} \bar {v} _ {t , i}} \text {f o r a n y} i \in \mathcal {A} _ {t + 1}. \tag {47}
$$
For the base-algorithm  $\mathcal{E}_i$  active at iteration  $t + 1$  and his/her corresponding surrogate base-algorithm  $\mathcal{E}_i'$ , we can decompose the time horizon until  $t + 1$  into two intervals: asleep interval  $\mathcal{I}_i^{\mathrm{slp}} = [1, s_i - 1]$  and active part  $\mathcal{I}_i^{\mathrm{act}} = [s_i, t + 1]$ . For any time stamp belonging to the asleep interval  $\tau \in \mathcal{I}_i^{\mathrm{slp}}$ , we can check that the loss for the base-algorithm  $\mathcal{E}_i'$  satisfies
$$
\bar {m} _ {\tau , i} = \left\langle \widehat {L} _ {\tau} \left(\bar {\theta} _ {\tau}\right), \bar {\theta} _ {\tau} - \bar {\theta} _ {\tau , i} \right\rangle = \left\langle \widehat {L} _ {\tau} \left(\widehat {\theta} _ {\tau}\right), \widehat {\theta} _ {\tau} - \widehat {\theta} _ {\tau} \right\rangle = 0,
$$
where the first equality is by the definition of  $\bar{m}_{\tau,i}$ . The second equality is due to IH1 such that  $\bar{\theta}_{\tau} = \widehat{\theta}_{\tau}$  for any  $\tau \leq t$  and the definition of the surrogate base-learner  $\mathcal{E}_i'$  such that the prediction  $\bar{\theta}_{\tau,i} = \widehat{\theta}_{\tau}$  on the asleep interval. For any time stamp belonging to the active interval  $\tau \in \mathcal{T}_i^{\mathrm{act}}$ , similar argument shows that
$$
\bar {m} _ {\tau , i} = \left\langle \widehat {L} _ {\tau} (\bar {\theta} _ {\tau}), \bar {\theta} _ {\tau} - \bar {\theta} _ {\tau , i} \right\rangle = \left\langle \widehat {L} _ {\tau} (\widehat {\theta} _ {\tau}), \widehat {\theta} _ {\tau} - \widehat {\theta} _ {\tau , i} \right\rangle = m _ {\tau , i},
$$
Thus, by definition, we can draw the conclusion that  $\bar{v}_{t,i} = v_{t,i}$  and  $\bar{\varepsilon}_{t,i} = \varepsilon_{t,i}$  for any active base-algorithm with index  $i\in \mathcal{A}_{t + 1}$ , which would finally lead to (47).
Then, we show  $\widehat{\theta}_{t + 1} = \bar{\theta}_{t + 1}$  based on  $IH1$  and  $IH2$ . We have
$$
\begin{array}{l} \bar {\theta} _ {t + 1} = \sum_ {i \in \mathcal {A} _ {t + 1}} \bar {p} _ {t + 1, i} \widehat {\theta} _ {t + 1, i} + \sum_ {i \in [ K ] / \mathcal {A} _ {t + 1}} \bar {p} _ {t + 1, i} \widehat {\theta} _ {t + 1} \\ = \left(\sum_ {i \in \mathcal {A} _ {t + 1}} \bar {p} _ {t + 1, i}\right) \left(\sum_ {i \in \mathcal {A} _ {t + 1}} p _ {t + 1, i} \widehat {\theta} _ {t + 1, i}\right) + \left(1 - \sum_ {i \in \mathcal {A} _ {t + 1}} \bar {p} _ {t + 1, i}\right) \widehat {\theta} _ {t + 1} \\ = \left(\sum_ {i \in \mathcal {A} _ {t + 1}} \bar {p} _ {t + 1, i}\right) \widehat {\theta} _ {t} + \left(1 - \sum_ {i \in \mathcal {A} _ {t + 1}} \bar {p} _ {t + 1, i}\right) \widehat {\theta} _ {t + 1} = \widehat {\theta} _ {t}, \\ \end{array}
$$
where the first equality is due to (46) and the second inequality is due to the definition of  $\widehat{\theta}_{t + 1}$ . We have finished the induction steps and completed the proof.
Proof of Lemma 12. Since  $\widehat{\theta}_t$  and  $\widehat{\theta}_{t,i}$  are both independent of  $S_{t}$ , using the same arguments for obtaining (33), we can obtain that
$$
\sum_ {t = 1} ^ {T} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} (\widehat {\theta} _ {t, i}) \leq \underbrace {\sum_ {t = 1} ^ {T} \langle \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) , \widehat {\theta} _ {t} - \widehat {\theta} _ {t , i} \rangle} _ {\text {t e r m (a)}} + \underbrace {\sum_ {t = 1} ^ {T} \langle \nabla \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) , \widehat {\theta} _ {t} - \widehat {\theta} _ {t , i} \rangle} _ {\text {t e r m (b)}}
$$
$$
- \sum_ {t = 1} ^ {T} \frac {1}{2 (\beta + 1)} \mathbb {E} _ {t} \left[ \left(\nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right) ^ {\top} \left(\widehat {\theta} _ {t} - \widehat {\theta} _ {t, i}\right)\right) ^ {2} \right] \tag {48}
$$
For term (a), we have
$$
\begin{array}{l} \operatorname {t e r m} (\mathfrak {a}) \leq \frac {2 S R C _ {T} ^ {\prime}}{\sqrt {\ln K}} \sqrt {1 + \sum_ {t = 1} ^ {T} \left(\nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top} \left(\widehat {\theta} _ {t} - \widehat {\theta} _ {t , i}\right)\right) ^ {2}} + 2 C _ {T} ^ {\prime} \\ \leq \frac {2 4 (\beta + 1) S ^ {2} R ^ {2} \left(C _ {T} ^ {\prime}\right) ^ {2}}{\ln K} + \frac {1}{6 (\beta + 1)} + \frac {1}{6 (\beta + 1)} \sum_ {t = 1} ^ {T} \left(\nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right) ^ {\top} \left(\widehat {\theta} _ {t} - \widehat {\theta} _ {t, i}\right)\right) ^ {2} + 2 C _ {T} ^ {\prime}, \tag {49} \\ \end{array}
$$
for any  $i \in [K]$ , where  $C_T' = 3\ln K + \ln \left(1 + \frac{K}{2e} (1 + \ln (T + 1))\right) = \mathcal{O}(\ln K + \ln \ln T)$ . In above, the first inequality is a direct application of [43, Corollary 4] with the linearized loss  $M_t(\theta) = \langle \widehat{L}_t(\widehat{\theta}_t), \theta \rangle / (2SR) + 1/2$  and the second inequality is due to the AM-GM inequality.
For term (b), by Lemma 8 with  $\nu = 6(1 + \beta)$  and a similar argument in (36), we have
$$
\begin{array}{l} \text {t e r m} (\mathfrak {b}) \leq \frac {1}{1 2 (1 + \beta)} \sum_ {t = 1} ^ {T} \left(\left(\nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right) ^ {\top} \left(\widehat {\theta} _ {t} - \widehat {\theta} _ {t, i}\right)\right) ^ {2} + \mathbb {E} _ {t} \left[ \left(\nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right) ^ {\top} \left(\widehat {\theta} _ {t} - \widehat {\theta} _ {t, i}\right)\right) ^ {2} \right]\right) \\ + 2 4 (1 + \beta) \log \frac {\sqrt {2 T + 1}}{\delta} + 4 S R \sqrt {\log \frac {2 T + 1}{\delta^ {2}}} \tag {50} \\ \end{array}
$$
Combining the upper bound for term (a) and term (b) with (48) yields,
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} (\widehat {\theta} _ {t, i}) \\ \leq \underbrace {\frac {1}{4 (1 + \beta)} \sum_ {t = 1} ^ {T} \left(\nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top} (\widehat {\theta} _ {t} - \widehat {\theta} _ {t , i})\right) ^ {2} - \frac {5}{1 2 (1 + \beta)} \mathbb {E} _ {t} \left[ \left(\nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top} (\widehat {\theta} _ {t} - \widehat {\theta} _ {t , i})\right) ^ {2} \right]} _ {\text {t e r m (c)}} \\ + \frac {2 4 (1 + \beta) S ^ {2} R ^ {2} (C _ {T} ^ {\prime}) ^ {2}}{\ln K} + \frac {1}{6 (1 + \beta)} + 2 C _ {T} ^ {\prime} + 2 4 (1 + \beta) \log \frac {\sqrt {2 T + 1}}{\delta} + 4 S R \sqrt {\log \frac {2 T + 1}{\delta^ {2}}}. \\ \end{array}
$$
Then, according to Lemma 9, we can further bound term (c) by
$$
\begin{array}{l} \mathsf {t e r m} (\mathsf {c}) = \frac {1}{4 (1 + \beta)} \sum_ {t = 1} ^ {T} \left(\nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top} (\widehat {\theta} _ {t} - \widehat {\theta} _ {t, i})\right) ^ {2} - \frac {5}{1 2 (1 + \beta)} \mathbb {E} _ {t} \left[ \left(\nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top} (\widehat {\theta} _ {t} - \widehat {\theta} _ {t, i})\right) ^ {2} \right] \\ \leq \frac {1 1 S ^ {2} R ^ {2}}{1 + \beta} \log \frac {\sqrt {2 | I | + 1}}{\delta} + \frac {4 S ^ {2} R ^ {2}}{3 (1 + \beta)} \sqrt {\log \frac {2 T + 1}{\delta^ {2}}}, \\ \end{array}
$$
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} (\widehat {\theta} _ {t, i}) \\ \leq \frac {2 4 (1 + \beta) S ^ {2} R ^ {2} \left(C _ {T} ^ {\prime}\right) ^ {2}}{\ln K} + \frac {1}{6 (1 + \beta)} + 2 C _ {T} ^ {\prime} + C _ {1} ^ {\prime} \log \frac {\sqrt {2 T + 1}}{\delta} + C _ {2} ^ {\prime} \sqrt {\log \frac {2 T + 1}{\delta^ {2}}} \\ = \mathcal {O} (\beta \log (T / \delta) + \beta \ln K). \\ \end{array}
$$
In above,  $C_1' = 24(1 + \beta) + 11S^2 R^2 / (1 + \beta)$  and  $C_2' = 4SR + 4S^2 R^2 / (3(1 + \beta))$ . In the last equality, we treat double logarithmic factors in  $T$  as a constant, following previous studies [73, 74]. We have completed the proof.
D.4 Proof of Theorem 2
This part presents the proof for Theorem 2, which is based on the dynamic regret for the base-learner (Lemma 3) and that for the meta-learner (Lemma 10) presented in Appendix D.1.
D.4.1 Main Proof
Proof of Theorem 2. When choosing the Bregman divergence function as  $\psi (t) = \psi_{\mathrm{LR}}(t)\triangleq t\log t - (t + 1)\log (t + 1)$  and the hypothesis space  $\mathcal{H}_{\theta}^{\mathrm{LR}} = \{\mathbf{x}\mapsto \exp (-\theta^{\top}\phi (\mathbf{x}))\mid \| \theta \| _2\leq S\}$ . The expected loss function  $L_{t}(\theta)$  as defined by (4) is equal to
$$
L _ {t} (\theta) = \frac {1}{2} \Bigl (\mathbb {E} _ {\mathcal {D} _ {0}} [ \log (1 + e ^ {- \phi (\mathbf {x}) ^ {\top} \theta}) ] + \mathbb {E} _ {\mathcal {D} _ {t}} [ \log (1 + e ^ {\phi (\mathbf {x}) ^ {\top} \theta}) ] \Bigr).
$$
Let  $\theta_t^* \triangleq \arg \min_{\theta \in \mathbb{R}^d} L_t(\theta)$ . Since the ground-truth density ratio  $r_t^*(\mathbf{x}) = \mathcal{D}_t(\mathbf{x}) / \mathcal{D}_0(\mathbf{x})$  function is contained in the hypothesis space as  $\mathcal{D}_t(\mathbf{x}) / \mathcal{D}_0(\mathbf{x}) \in \mathcal{H}_\theta^{\mathrm{LR}}$ . One can show that  $\mathcal{D}_t(\mathbf{x}) / \mathcal{D}_0(\mathbf{x}) = \exp(-(\theta_t^*)^\top \phi(\mathbf{x}))$  and  $\theta_t^* \in \Theta = \{\theta \in \mathbb{R}^d \mid \| \theta \|_2 \leq S\}$ .
Then, for the base-learner  $\mathcal{E}_i$  running on any interval  $I_{i}$ , Lemma 3 in Appendix D.2 shows that its prediction  $\widehat{\theta}_{t,i}$  is comparable with the best model at every iteration  $\theta_t^*$  with an  $\mathcal{O}(|I_i|(V_{I_i}^\theta)^2)$  cost:
$$
\sum_ {t \in I _ {i}} \widetilde {L} _ {t} \left(\widehat {\theta} _ {t, i}\right) - \sum_ {t \in I _ {i}} \widetilde {L} _ {t} \left(\theta_ {t} ^ {*}\right) \leq \mathcal {O} \left(d \beta \log \left(| I _ {i} | / \delta\right) + | I _ {i} | \left(V _ {I _ {i}} ^ {\theta}\right) ^ {2} + \frac {| I _ {i} | \log (d T / \delta)}{N _ {0}}\right), \tag {51}
$$
where  $V_{I_i}^{\theta} = \sum_{t=s_i+1}^{e_i} \left\| \theta_t^* - \theta_{t-1}^* \right\|_2$  is the path length measuring the fluctuation of the minimizers.
Besides, Lemma 10 in Appendix D.3 ensures that the prediction  $\widehat{\theta}_t$  of the meta-learner is comparable with the prediction  $\widehat{\theta}_{t,i}$  of each base-learner  $\mathcal{E}_i$  on the associated interval  $I_{i}$  with an  $\mathcal{O}(\log T)$ .
$$
\sum_ {t \in I _ {i}} \tilde {L} _ {t} \left(\widehat {\theta} _ {t}\right) - \sum_ {t \in I _ {i}} \tilde {L} _ {t} \left(\widehat {\theta} _ {t, i}\right) \leq \mathcal {O} \left(\beta \log (T / \delta) + \beta \ln K\right). \tag {52}
$$
A direct combination of (51) and (52) shows that the meta-learner is able to track the best prediction  $\theta_t^*$  on any interval  $I_i\in \mathcal{C}$ . Actually, by exploiting the structure of the geometric covering (24), it can be shown that such a property can be extended to arbitrary interval  $I\in [T]$  as the following lemma, whose proof is deferred to Appendix D.4.2.
Lemma 13. With probability at least  $1 - 5\delta$  for any  $\delta \in (0,1)$ , for any interval  $[s,e] \in [T]$ , the meta-learner's prediction  $\widehat{\theta}_t$  returned by (9) ensures,
$$
\begin{array}{l} \sum_ {t \in I} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t \in I} \widetilde {L} _ {t} \left(\theta_ {t} ^ {*}\right) \\ \leq \frac {R ^ {2}}{4} | I | (V _ {I} ^ {\theta}) ^ {2} + 2 \log | I | \left(C _ {T} ^ {(1)} (\delta^ {\prime}) + C _ {T} ^ {2} (\delta^ {\prime})\right) + \frac {1 6 | I | \ln ((d + 1) \log | I | T / \delta)}{N _ {0}} + o \left(\frac {| I | \log (d T / \delta)}{N _ {0}}\right) \\ \end{array}
$$
where  $V_{I}^{\theta} = \sum_{t=s+1}^{e}\|\theta_{t-1}^{*} - \theta_{t}^{*}\|_{2}$  and  $\delta' = \delta/2\log |I|$ . The coefficient  $C_{T}^{(1)}(\delta) = \mathcal{O}(d\beta\log(T/\delta))$  and  $C_{T}^{(2)}(\delta) = \mathcal{O}\left(\beta\log(T/\delta)\right)$  is defined in Lemma 4 and Lemma 12, respectively.
Then, we can use the bin partition [19, Lemma 30] to convert the dynamic regret on each interval to a dynamic regret over  $T$  iterations. For self-containedness, we restate the lemma here.
Lemma 14 (Lemma 30 of [19]). There exists a partition  $\mathcal{P}$  of the time horizon  $[T]$  into  $M = \mathcal{O}(\max\{dT^{1/3}(U_T^\theta)^{2/3}, 1\})$  intervals, i.e.,  $\{I_i = [s_i, e_i]\}_{i=1}^M$  such that for any interval  $I_i \in \mathcal{P}$ ,  $U_{I_i}^\theta \leq D_{\max} / \sqrt{|I_i|}$ , where  $U_{I_i}^\theta = \sum_{t=s_i+1}^{e_i} \| \theta_t^* - \theta_{t-1}^* \|_1$  and  $D_{\max} = \max_{t \in [T]} \| \theta_t^* \|_\infty \leq \max_{t \in [T]} \| \theta_t^* \|_2 \leq S$ .
Then, we can decompose the overall dynamic regret of our method into those over the key partition  $\mathcal{P}$ . Specifically, with probability at least  $1 - 5\delta$ , we have
$$
\begin{array}{l} \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} \left(\theta_ {t} ^ {*}\right) \\ = \sum_ {I _ {i} \in \mathcal {P}} \left(\sum_ {t \in I _ {i}} \widetilde {L} _ {t} \left(\widehat {\theta} _ {t}\right) - \sum_ {I _ {i}} \widetilde {L} _ {t} \left(\theta_ {t} ^ {*}\right)\right) \\ \end{array}
$$
$$
\begin{array}{l} \leq \sum_ {I _ {i} \in \mathcal {P}} \frac {R ^ {2}}{4} | I _ {i} | \left(V _ {I _ {i}} ^ {\theta}\right) ^ {2} + \sum_ {I _ {i} \in \mathcal {P}} \frac {1 6 \left| I _ {i} \right| \ln \left(2 (d + 1) T \log \left| I _ {i} \right| M / \delta\right)}{N _ {0}} \\ + 2 \sum_ {I _ {i} \in \mathcal {P}} \log | I _ {i} | \left(C _ {T} ^ {(1)} \left(\delta / 2 M \log | I _ {i} |\right) + C _ {T} ^ {2} \left(\delta / 2 M \log | I _ {i} |\right)\right) \\ \end{array}
$$
$$
\leq \underbrace{\frac{MS^{2}R^{2}}{4}}_{\text{term (a)}} + \underbrace{2M\log T\left(C_{T}^{(1)}\Big(\delta / 2M\log T\Big) + C_{T}^{(2)}\Big(\delta / 2M\log T\Big)\right)}_{\text{term (b)}} + \underbrace{\frac{16T\ln\left(2(d + 1)T\log TM / \delta\right)}{N_{0}}}_{\text{term (c)}},
$$
where the second last inequality is due to Lemma 13 and a union bound taken over all intervals in the key partition. The last inequality is due to Lemma 14 such that  $V_{I_i}^\theta \leq U_{I_i}^\theta \leq S / \sqrt{I_i}$  for any interval  $I_i \in \mathcal{P}$ . Lemma 14 shows that we can bound the partition number by  $M = \mathcal{O}(\max\{dT^{1/3}(U_T^\theta)^{2/3}, 1\}) = \mathcal{O}(\max\{d^{4/3}T^{1/3}(V_T^\theta)^{2/3}, 1\})$ . Then, we can further show that
$$
\text {t e r m} (\mathsf {a}) \leq \mathcal {O} \left(d ^ {4 / 3} \max  \left\{T ^ {1 / 3} \left(V _ {T} ^ {\theta}\right) ^ {2 / 3}, 1 \right\}\right),
$$
$$
\mathbf {t e r m} (\mathfrak {b}) \leq \mathcal {O} \left(d ^ {4 / 3} \beta \log T \log (T / \delta) \max  \{T ^ {1 / 3} (V _ {T} ^ {\theta}) ^ {2 / 3}, 1 \}\right).
$$
$$
\text {t e r m} (\mathsf {c}) \leq \mathcal {O} \left(\frac {T \ln (d T / \delta)}{N _ {0}}\right),
$$
which implies that the overall dynamic regret is bounded by
$$
\sum_ {t = 1} ^ {T} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} (\theta_ {t} ^ {*}) \leq \mathcal {O} \left(d ^ {\frac {4}{3}} \beta \log T \log (T / \delta) \cdot \max  \{T ^ {\frac {1}{3}} (V _ {T} ^ {\theta}) ^ {\frac {2}{3}}, 1 \} + \frac {T \ln (d T / \delta)}{N _ {0}}\right).
$$
Hence, we complete the proof by showing  $\widetilde{L}_t(\theta_t^*) = \widetilde{L}_t(r_t^*)$  under the realizable assumption and converting the variation of the model parameter  $V_T^\theta = \sum_{t=2}^T \|\theta_t^* - \theta_{t-1}^*\|_2$  to the variation of the feature distribution  $V_T = \sum_{t=2}^T \|\mathcal{D}_t(\mathbf{x}) - \mathcal{D}_{t-1}(\mathbf{x})\|_1$  by Lemma 15.
D.4.2 Useful Lemmas
$$
\begin{array}{l} \sum_ {t \in I} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t \in I} \widetilde {L} _ {t} \left(\theta_ {t} ^ {*}\right) \\ \leq \frac {R ^ {2}}{4} | I | (V _ {I} ^ {\theta}) ^ {2} + 2 \log | I | \left(C _ {T} ^ {(1)} (\delta^ {\prime}) + C _ {T} ^ {2} (\delta^ {\prime})\right) + \frac {1 6 | I | \ln ((d + 1) \log | I | T / \delta)}{N _ {0}} + o \left(\frac {| I | \log (d T / \delta)}{N _ {0}}\right), \\ \end{array}
$$
where  $V_{I}^{\theta} = \sum_{t = s + 1}^{e}\| \theta_{t - 1}^{*} - \theta_{t}^{*}\|_{2}$  and  $\delta' = \delta /2\log |I|$ . The coefficient  $C_T^{(1)}(\delta) = \mathcal{O}\left(d\beta \log (T / \delta)\right)$  and  $C_T^{(2)}(\delta) = \mathcal{O}\Bigl (\beta \log (T / \delta)\Bigr)$  is defined in Lemma 4 and Lemma 12, respectively.
Proof of Lemma 13. A direct combination of Lemma 3 and 10 shows that, with probability at least  $1 - 5\delta$ , we have the following dynamic regret guarantee on any interval  $I_{i} = [s_{i}, e_{i}] \in \mathcal{C}$ .
$$
\begin{array}{l} \sum_ {t \in I _ {i}} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t \in I _ {i}} \widetilde {L} _ {t} \left(\theta_ {t} ^ {*}\right) \\ \leq C _ {I _ {i}} ^ {(1)} (\delta) + C _ {T} ^ {(2)} (\delta) + \frac {R ^ {2}}{4} | I _ {i} | \left(V _ {I _ {i}} ^ {\theta}\right) ^ {2} + \frac {1 6 | I _ {i} | \ln ((d + 1) | I _ {i} | / \delta)}{N _ {0}} + o \left(\frac {| I _ {i} | \log (d | I _ {i} | / \delta)}{N _ {0}}\right), \tag {53} \\ \end{array}
$$
where  $V_{I_i}^{\theta} = \sum_{t=s_i+1}^{e_i} \|\theta_{t-1}^* - \theta_t^*\|_2$ . The coefficient  $C_{I_i}^{(1)} = \mathcal{O}\left(d\beta \log(|I_i| / \delta)\right)$  is defined in Lemma 4 and  $C_T^{(2)}(\delta) = \mathcal{O}\left(\beta \log(T / \delta)\right)$  is defined in Lemma 12.
For any interval  $I = [s, e] \subseteq [T]$ , [75, Lemma 3] showed that it can be partitioned into two sequences of disjoint and consecutive intervals, denoted by  $I_{-p}, \ldots, I_0 \in \mathcal{C}$  and  $I_1, \ldots, I_q \in \mathcal{C}$ , such that
$$
\left| I _ {- i} \right| / \left| I _ {- i + 1} \right| \leq 1 / 2, \forall i \geq 1 \quad \text {a n d} \quad \left| I _ {i} \right| / \left| I _ {i - 1} \right| \leq 1 / 2, \forall i \geq 2. \tag {54}
$$
Then, with probability at least  $1 - 5\delta$ , we can decompose the regret over  $I = [s,e] \subseteq [T]$  as
$$
\begin{array}{l} \sum_ {t \in I} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t \in I} \widetilde {L} _ {t} (\theta_ {t} ^ {*}) = \sum_ {i = - p} ^ {q} \left(\sum_ {t \in I _ {i}} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t \in I _ {i}} \widetilde {L} _ {t} (\theta_ {t} ^ {*})\right) \\ \leq (p + q) \left(C _ {T} ^ {(1)} (\delta / p + q) + C _ {T} ^ {(2)} (\delta / p + q)\right) + \frac {R ^ {2}}{4} \sum_ {i = - p} ^ {q} \left| I _ {i} \right| \left(V _ {I _ {i}} ^ {\theta}\right) ^ {2} \\ + \sum_ {i = - p} ^ {q} \frac {1 6 | I _ {i} | \ln \left((d + 1) (p + q) T / \delta\right)}{N _ {0}} + \sum_ {i = - p} ^ {q} o \left(\frac {| I _ {i} | \log (d T (p + q) / \delta)}{N _ {0}}\right). \\ \end{array}
$$
where the last inequality is due to (53) and a union bound taken over all intervals. Then, due to the partition of the intervals (54), we can check that  $p + q \leq 2\log |I|$ . Then, we can further bound the above displayed inequality by
$$
\begin{array}{l} \sum_ {t \in I} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t \in I} \widetilde {L} _ {t} (\theta_ {t} ^ {*}) \leq 2 \log | I | \left(C _ {T} ^ {(1)} \left(\delta / (2 \log | I |\right)\right) + C _ {T} ^ {(2)} \left(\delta / (2 \log | I |\right)\left. \right) + \frac {R ^ {2}}{4} | I | \left(V _ {I} ^ {\theta}\right) ^ {2} \\ + \frac {1 6 | I | \ln (2 (d + 1) \log | I | T / \delta)}{N _ {0}} + o \left(\frac {| I | \log (d T \log | I | / \delta)}{N _ {0}}\right) \\ = \frac {R ^ {2}}{4} | I | (V _ {I} ^ {\theta}) ^ {2} + \mathcal {O} \left(d \beta \log | I | \log (^ {T} \log | I | / \delta) + \frac {| I | \log (^ {d T} \log | I | / \delta)}{N _ {0}}\right), \\ \end{array}
$$
where the first inequality is due to the fact that  $\sum_{i = -p}^{q}V_{I_i}^2\leq V_I^2$  . This completes the proof.
Lemma 15. Under the condition that  $\mathcal{D}_t(\mathbf{x}) / \mathcal{D}_0(\mathbf{x})\in \mathcal{H}_\theta^{\mathrm{LR}}\triangleq \{\exp (-\theta^\top \phi (\mathbf{x}))\mid \| \theta \| _2\leq S\}$  for any  $t\in [T]$  and  $\mathbb{E}_{\mathbf{x}\sim \mathcal{D}_0}[[\phi (\mathbf{x})^\top \mathbf{a}]]\geq \alpha \| \mathbf{a}\| _2$  for any  $\mathbf{a}\in \mathbb{R}^d$  with a certain  $\alpha >0$ . Then, we have  $\| \theta_{t - 1}^{*} - \theta_{t}^{*}\|_{2}\leq \beta \| \mathcal{D}_{t - 1}(\mathbf{x}) - \mathcal{D}_{t}(\mathbf{x})\|_{1} / \alpha$ .
Proof of Lemma 15. Under the realizable assumption such that  $\mathcal{D}_t(\mathbf{x}) / \mathcal{D}_0(\mathbf{x})\in \mathcal{H}_{\theta}^{\mathrm{LR}}$ . One can verify that  $\exp (-\phi (\mathbf{x})^\top \theta_t^*) = \mathcal{D}_t(\mathbf{x}) / \mathcal{D}_0(\mathbf{x})$ . Then, we can convert the variation of the model  $\| \theta_{t - 1}^{*} - \theta_{t}^{*}\|_{2}$  to the variation of the distribution  $\| \mathcal{D}_{t - 1}(\mathbf{x}) - \mathcal{D}_t(\mathbf{x})\| _1$ .