SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {x} \sim D _ {0}} \left[ f _ {r} (\mathbf {x}) \right] - \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ f _ {r} (\mathbf {x}) \right] \\ \leq \left(\frac {4 A _ {r} B _ {r}}{3} + \frac {1 6 A _ {r} ^ {2}}{\mu} + 3 \mu B _ {r} ^ {2}\right) \frac {\log (2 / \delta)}{N _ {0}} + \frac {\mu}{8} \mathbb {E} _ {S _ {0}} \left[ \left(r (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})\right) ^ {2} \right], \\ \end{array}
$$
with probability at least  $1 - \delta$
Then, based on the notion of covering number, we show a union bound of  $U_{t}$  over all  $r \in \mathcal{H}_r$ . Let  $\mathcal{N}(\mathcal{H}_r, \varepsilon, \| \cdot \|_{\infty})$  be the  $\varepsilon$ -net of  $\mathcal{H}_r$  such that for any  $r \in \mathcal{H}_r$  one can find a  $r' \in \mathcal{N}(\mathcal{H}_r, \varepsilon, \| \cdot \|_{\infty})$  satisfying  $\| r - r' \|_{\infty} \leq \varepsilon$ . The covering number  $N(\mathcal{H}_r, \varepsilon, \| \cdot \|_{\infty})$  is defined as the minimal cardinality of an  $\varepsilon$ -net of  $\mathcal{H}_r$ . By taking the union bound over all  $r' \in \mathcal{N}(\mathcal{H}_r, \varepsilon, \| \cdot \|_{\infty})$ , with probability at least  $1 - \delta$ , the following holds for any  $r' \in \mathcal{N}(\mathcal{H}_r, \varepsilon, \| \cdot \|_{\infty})$ ,
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {x} \sim D _ {0}} \left[ f _ {r ^ {\prime}} (\mathbf {x}) \right] - \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ f _ {r ^ {\prime}} (\mathbf {x}) \right] \\ \leq \frac {C _ {r} (\mu) \log \left(\frac {2 N \left(\mathcal {H} _ {r} , \varepsilon , \| \cdot \| _ {\infty}\right)}{\delta}\right)}{N _ {0}} + \frac {\mu}{8} \mathbb {E} _ {S _ {0}} \left[ \left(r ^ {\prime} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})\right) ^ {2} \right], \tag {23} \\ \end{array}
$$
where  $C_r(\mu) = 4A_rB_r / 3 + 16A_r^2 / \mu + 3\mu B_r^2 = \mathcal{O}(\max \{\mu, 1 / \mu\})$ . Then, for any model  $r \in \mathcal{H}_r$ , we can show
$$
\begin{array}{l} \mathbb {E} _ {\mathbf {x} \sim D _ {0}} [ f _ {r} (\mathbf {x}) ] - \mathbb {E} _ {\mathbf {x} \sim S _ {0}} [ f _ {r} (\mathbf {x}) ] \\ = \mathbb {E} _ {\mathbf {x} \sim D _ {0}} \left[ f _ {r ^ {\prime}} (\mathbf {x}) \right] - \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ f _ {r ^ {\prime}} (\mathbf {x}) \right] + \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0}} \left[ f _ {r} (\mathbf {x}) - f _ {r ^ {\prime}} (\mathbf {x}) \right] - \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ f _ {r} (\mathbf {x}) - f _ {r ^ {\prime}} (\mathbf {x}) \right] \\ \leq \mathbb {E} _ {\mathbf {x} \sim D _ {0}} \left[ f _ {r ^ {\prime}} (\mathbf {x}) \right] - \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ f _ {r ^ {\prime}} (\mathbf {x}) \right] + 2 \varepsilon A _ {r} \\ \stackrel {(2 3)} {\leq} \frac {C _ {r} (\mu) \log (2 N (\mathcal {H} _ {r} , \varepsilon , \| \cdot \| _ {\infty}) / \delta)}{N _ {0}} + \frac {\mu}{8} \mathbb {E} _ {S _ {0}} \left[ (r ^ {\prime} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})) ^ {2} \right] + 2 A _ {r} \varepsilon \\ \leq \frac {C _ {r} (\mu) \log \left(2 N \left(\mathcal {H} _ {r} , \varepsilon , \| \cdot \| _ {\infty}\right) / \delta\right)}{N _ {0}} + \frac {\mu}{4} \mathbb {E} _ {S _ {0}} \left[ (r (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})) ^ {2} \right] + 2 A _ {r} \varepsilon + \frac {\mu \varepsilon^ {2}}{4}, \\ \end{array}
$$
where the first inequality is due to the property of the  $\varepsilon$ -net such that
$$
\mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0}} \left[ f _ {r} (\mathbf {x}) - f _ {r ^ {\prime}} (\mathbf {x}) \right] \leq \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0}} \left[ | \partial^ {2} \psi \left(r _ {t} ^ {*} (\mathbf {x})\right) r ^ {\prime} (\mathbf {x}) | \cdot | r (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x}) | \right] \leq A _ {r} \| r - r ^ {\prime} \| _ {\infty} \leq A _ {r} \varepsilon .
$$
The last inequality is by
$$
\begin{array}{l} \frac {\mu}{8} \mathbb {E} _ {S _ {0}} \left[ (r ^ {\prime} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})) ^ {2} \right] \leq \frac {\mu}{4} \mathbb {E} _ {S _ {0}} \left[ (r (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})) ^ {2} \right] + \frac {\mu}{4} \mathbb {E} _ {S _ {0}} \left[ (r (\mathbf {x}) - r ^ {\prime} (\mathbf {x})) ^ {2} \right] \\ \leq \frac {\mu}{4} \mathbb {E} _ {S _ {0}} \left[ (r (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})) ^ {2} \right] + \frac {\mu}{4} \| r - r ^ {\prime} \| _ {\infty} ^ {2} \leq \frac {\mu}{4} \mathbb {E} _ {S _ {0}} \left[ (r (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x})) ^ {2} \right] + \frac {\mu \varepsilon^ {2}}{4}. \\ \end{array}
$$
Thus, we complete the proof.
Algorithm 1 Base-learner  $\varepsilon_{i}$
Input: Active interval  $I_{i} = [s_{i}, e_{i}]$ , regularization parameter  $\lambda$  and step size  $\eta$ , feasible domain  $\Theta$ .
1: Initialize the model  $\widehat{\theta}_{s_i,i}$  as any point in the domain  $\Theta$  and the matrix  $A_{s_i - 1} = \lambda I_d$ .
2: for  $t \in I_i = [s_i, e_i]$  do
3: Submit the model  $\widehat{\theta}_{t,i}$  to the meta-learner.
4: Update the matrix by  $A_{t,i} = A_{t - 1,i} + \nabla \widehat{L}_t(\widehat{\theta}_{t,i})\nabla \widehat{L}_t(\widehat{\theta}_{t,i})^\top$
5: Update the model for next iteration by
$$
\widehat {\theta} _ {t + 1, i} = \Pi_ {\Theta} ^ {A _ {t, i}} \left[ \widehat {\theta} _ {t, i} - \gamma A _ {t, i} ^ {- 1} \nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t, i}\right) \right],
$$
where  $\Pi_{\Theta}^{A_{t,i}}[\nu] = \arg \min_{\theta \in \Theta} \| \theta - \nu \|_{A_{t,i}}$  is the projection operator. 6: end for
D Omitted Details for Section 4
This section presents the omitted details for Section 4. We will first introduce the algorithm details for our online ensemble algorithm, characterized by its meta-base structure, in Appendix D.1. Then, we present the theoretical guarantees alongside their proofs for the base-learner (Lemma 3) and meta-learner (Lemma 10) in Appendix D.2 and Appendix D.3 respectively. These elements collectively contribute to the dynamic regret guarantee of the overall algorithm. The dynamic regret of the online ensemble algorithm (Theorem 2) is presented in Appendix D.4. We provide the proof for the average excess risk (Theorem 3) in Appendix D.5. Further discussions regarding the tightness of the bound and alternative choices for the Bregman divergence function are located in Appendices D.6 and D.7.
D.1 Omitted Algorithm Details
Here, we introduce the algorithmic details for the online ensemble method omitted in Section 4.1.
Interval Schedules. We run multiple based learners on the geometric covering:
$$
\mathcal {C} = \bigcup_ {k \in \mathbb {N} \cup \{0 \}} \mathcal {C} _ {k} \quad \text {a n d} \quad \mathcal {C} _ {k} = \left\{\left[ i \cdot 2 ^ {k}, (i + 1) \cdot 2 ^ {k} - 1 \right] \mid i \in \mathbb {N} \text {a n d} i \cdot 2 ^ {k} \leq T \right\} \tag {24}
$$
and then employ a meta-learner to combine them by weights. For any interval  $I_{i} = [s_{i},e_{i}] \in \mathcal{C}$  which starts at time  $t = s_i$  and end at  $t = e_i$ , we denote by  $\mathcal{E}_i$  the base-learner running over  $I_{i}$ .
Base-learner. The base-learner  $\mathcal{E}_i$  updates his/her model with the ONS algorithm [44] in the associated active period  $I_{i}$ . The algorithmic details of the base-learner are summarized in Algorithm 1. At every iteration  $t\in I_i$ , the base-learner  $\mathcal{E}_i$  will first submit his/her model  $\widehat{\theta}_{t,i}$  to the meta-learner and then update the model for the next iteration by Line 4 and Line 5 as shown in Algorithm 1.
Meta-learner. The meta-learner aggregates the predictions of each active base-learner by a weighted combination scheme. To obtain the weight  $p_{t,i}$  for each active base-learner  $\mathcal{E}_i$ , we maintain three terms  $m_{t,i}$ ,  $v_{t,i}$  and  $\varepsilon_{t,i}$  for each base-learner  $\mathcal{E}_i$  when he/she is active, i.e.,  $t \in I_i$ . Let  $g_t(\theta) = \langle \nabla \widehat{L}_t(\widehat{\theta}_t), \theta - \widehat{\theta}_t \rangle$  be a linearized loss for the original function  $\widehat{L}_t(\theta)$ . The first term is defined by
$$
m _ {t, i} \triangleq \frac {g _ {t} (\widehat {\theta} _ {t}) - g _ {t} (\widehat {\theta} _ {t , i})}{S R} = \frac {\langle \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) , \widehat {\theta} _ {t} - \widehat {\theta} _ {t , i} \rangle}{S R},
$$
which measures the performance gap between the meta-learner's prediction  $\widehat{\theta}_t$  and the  $i$ -th base-learner's prediction  $\widehat{\theta}_{t,i}$  over the linearized loss  $g_{t}(\theta)$ . The second term  $v_{t,i}$  is the "potential" of the base-learner  $\mathcal{E}_i$ , which measures the historical performance of  $\mathcal{E}_i$  over the interval. When invoking the base-learner  $\mathcal{E}_i$  at  $t = s_i$ , we initialize  $v_{s_i - 1,i} = 1 / K$  and the term is updated by
$$
v _ {t, i} = \left(v _ {t - 1, i} \cdot \left(1 + \varepsilon_ {t - 1, i} m _ {t, i}\right)\right) ^ {\frac {\varepsilon_ {t , j}}{\varepsilon_ {t - 1 , j}}}, \forall t \in I _ {i}. \tag {25}
$$
The last term  $\varepsilon_{t,i}$  is the learning rate, which is set as
$$
\varepsilon_ {t, i} = \min  \left\{\frac {1}{2}, \sqrt {\frac {\ln K}{1 + \sum_ {\tau = s _ {i}} ^ {t} m _ {\tau , i} ^ {2}}} \right\}. \tag {26}
$$
Algorithm 2 Meta-learner
Input: Interval set for base-learners  $\mathcal{C}$    
1: Initialize the set for active base-learner  $\mathcal{A}_0 = \emptyset$  and let  $K = |\mathcal{C}|$    
2: for  $t = 1,\dots ,T$  do   
3: for  $I_{i} = [s_{i},e_{i}]\in \mathcal{C}$  do   
4: if  $s_i$  is equal to  $t$  then   
5: Create a base-learner  $\mathcal{E}_i$  by Algorithm 1 and initialize the potential  $v_{s_i - 1,i} = 1 / K$  and step size  $\varepsilon_{s_i - 1,i} = \min \{1 / 2,\sqrt{\ln K}\}$    
6: Add the index to the active set  $\mathcal{A}_t = \mathcal{A}_{t - 1}\cup \{i\}$    
7: else if  $e_i + 1$  is equal to  $t$  then   
8: Remove the index from the active set  $\mathcal{A}_t = \mathcal{A}_{t - 1} / \{i\}$    
9: end if   
10: end for   
11: Receive the  $\widehat{\theta}_{i,t}$  from active base-learners  $\mathcal{E}_i\in \mathcal{A}_t$  and update their weights  $p_{t,i}$  by (27).   
12: Submit the model by  $\widehat{\theta}_t = \sum_{i\in \mathcal{A}_t}p_{t,i}\widehat{\theta}_{t,i}$    
13: for  $i\in \mathcal{A}_t$  do   
14: Update the step size  $\varepsilon_{t + 1,i}$  for all current active base-learner by (26).   
15: Update the potential  $v_{t + 1,i}$  for all current active base-learner by (25).   
16: end for   
17: end for
After obtaining the  $v_{t,i}$  and  $\varepsilon_{t,i}$  for each active base-learner, the weight for the next iteration  $t + 1$  is updated by
$$
p _ {t + 1, i} = \frac {\varepsilon_ {t , i} v _ {t , i}}{\sum_ {i \in \mathcal {A} _ {t + 1}} \varepsilon_ {t , i} v _ {t , i}}, \text {f o r a l l} i \in \mathcal {A} _ {t + 1}. \tag {27}
$$
The algorithm is summarized in Algorithm 2. At the beginning of each iteration  $t$ , the algorithm will update the index set for the active base-learner (Line 3-Line 10). Then, the meta-learner's prediction is generated by the weighted combination rule (Line 11). After that, the meta-learner will update the potential  $v_{t+1,i}$  and the step  $\varepsilon_{t+1,i}$  for the next iteration.
D.2 Regret Guarantee for Base-Algorithm (Lemma 3)
By suitable configurations of the parameters  $\gamma, \lambda > 0$ , we show that the ONS algorithm enjoys the following dynamic regret guarantees,
Lemma 3. Let  $\theta_t^* = \arg \min_{\theta \in \mathbb{R}^d} L_t(\theta)$  and all the minimizer  $\theta_t^*$ 's lie in the interior of  $\Theta$ . Set  $\gamma = 6(1 + \beta)$  and  $\lambda = 1$ . Then, with probability at least  $1 - \delta$ , the ONS running on the interval  $I_i = [s_i, e_i] \subseteq [T]$  ensures
$$
\sum_ {t \in I _ {i}} \widetilde {L} _ {t} (\widehat {\theta} _ {t, i}) - \sum_ {t \in I _ {i}} \widetilde {L} _ {t} (\theta_ {t} ^ {*}) \leq \mathcal {O} \left(d \beta \log \left(| I _ {i} | / \delta\right) + | I _ {i} | \left(V _ {I _ {i}} ^ {\theta}\right) ^ {2} + \frac {| I _ {i} | \log (d T / \delta)}{N _ {0}}\right)
$$
where  $V_{I_i}^{\theta} = \sum_{t=s_i+1}^{e_i} \|\theta_t^* - \theta_{t-1}^*\|_2$  is the path length measuring the fluctuation of the minimizers.
D.2.1 Main Proof
This section provides the proof of the ONS algorithm. Since the analysis of the single base-learner is independent from the online ensemble structure, for notation simplicity, we omit the subscribe of the base-learner in the analysis. Specifically, we abbreviate the prediction  $\widehat{\theta}_{t,i}$ , matrix  $A_{t,i}$  and the interval  $I_i$  as  $\widehat{\theta}_t$ , matrix  $A_{t}$  and the interval  $I$  respectively.
Proof of Lemma 3. We begin the proof of the dynamic regret with a lemma on the static regret of the ONS algorithm, whose proof is proved in Appendix D.2.2. For notation simplicity, we omit the subscribers of the base-learner, where the prediction  $\theta_{t,i}$ , matrix  $A_{t,i}$  and interval  $I_i$  is abbreviated as  $\theta_t$ ,  $A_t$  and  $I$ , respectively.
Lemma 4. Set  $\gamma = 6(1 + \beta)$  and  $\lambda = 1$ . With probability at least  $1 - 2\delta$ , for any  $\delta \in (0,1]$ , the ONS algorithm (8) running on the interval  $I = [s,e] \subseteq [T]$  ensures
$$
\sum_ {t \in I} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t \in I} \widetilde {L} _ {t} (\theta) \leq C _ {I} ^ {(1)} (\delta) = \mathcal {O} \left(d \beta \log (| I | / \delta)\right)
$$
for any  $\theta \in \Theta$ , where  $C_I^{(1)}(\delta) = 12d\beta' \log \left(1 + |I|R^2\right) + \frac{S^2}{3\beta'} + (48\beta' + \frac{6S^2R^2}{\beta'}) \log \frac{\sqrt{2|I| + 1}}{\delta} +$
$$
(4 S R + \frac {S ^ {2} R ^ {2}}{\beta^ {\prime}}) \sqrt {\log \frac {2 | I | + 1}{\delta^ {2}}} a n d \beta^ {\prime} = \beta + 1.
$$
Then, we can decompose the dynamic regret as follows,
$$
\begin{array}{l} \sum_ {t \in I} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t \in I} \widetilde {L} _ {t} \left(\theta_ {t} ^ {*}\right) = \sum_ {t \in I} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t \in I} \widetilde {L} _ {t} \left(\bar {\theta} _ {I} ^ {*}\right) + \sum_ {t \in I} \widetilde {L} _ {t} \left(\bar {\theta} _ {I} ^ {*}\right) - \sum_ {t \in I} \widetilde {L} _ {t} \left(\theta_ {t} ^ {*}\right) \\ \leq C _ {I} ^ {(1)} (\delta) + \sum_ {t \in I} \widetilde {L} _ {t} \left(\bar {\theta} _ {I} ^ {*}\right) - \sum_ {t \in I} \widetilde {L} _ {t} \left(\theta_ {t} ^ {*}\right), \tag {28} \\ \end{array}
$$
where  $\bar{\theta}_I^* = \frac{1}{|I|}\sum_{t\in I}\theta_t^*$  is an averaged prediction of the function minimizers over  $T$  rounds. Under the condition all function minimizers lie in the interior of  $\Theta$ , we have  $\bar{\theta}_I^*\in \Theta$ . As a consequence, the last inequality holds due to Lemma 4.
Then, we proceed to bound the last two terms of the R.H.S. of the above inequality. Since the first-order derivative of the sigmoid function  $\sigma'(z) = \exp(-z) / (1 + \exp(-z))^2 \leq 1/4$  for any  $z \in \mathbb{R}$ . Let  $I_d \in \mathbb{R}^{d \times d}$  be the identity matrix. We can check  $\widetilde{L}_t$  is a  $R^2/4$ -smooth function such that  $\nabla^2 \widetilde{L}_t(\theta) = \frac{1}{2} \mathbb{E}_{\mathbf{x} \sim S_0} [\sigma'(\mathbf{x}^\top \theta) \phi(\mathbf{x}) \phi(\mathbf{x})^\top] + \frac{1}{2} \mathbb{E}_{\mathbf{x} \sim \mathcal{D}_t} [\sigma'(\mathbf{x}^\top \theta) \phi(\mathbf{x}) \phi(\mathbf{x})^\top] \preceq \frac{R^2}{4} I_d$  for any  $\theta \in \Theta$ . Then, with probability at least  $1 - 2\delta$ , we have
$$
\begin{array}{l} \widetilde {L} _ {t} (\bar {\theta} _ {I} ^ {*}) - \widetilde {L} _ {t} (\theta_ {t} ^ {*}) \leq \langle \nabla \widetilde {L} _ {t} (\theta_ {t} ^ {*}), \bar {\theta} _ {I} ^ {*} - \theta_ {t} ^ {*} \rangle + \frac {R ^ {2}}{8} \| \bar {\theta} _ {I} ^ {*} - \theta_ {t} ^ {*} \| _ {2} ^ {2} \\ \leq \frac {8}{R ^ {2}} \| \nabla \widetilde {L} _ {t} (\theta_ {t} ^ {*}) \| _ {2} ^ {2} + \frac {R ^ {2}}{4} \| \bar {\theta} _ {I} ^ {*} - \theta_ {t} ^ {*} \| _ {2} ^ {2} \\ \leq \frac {1 6 \ln ((d + 1) / \delta)}{N _ {0}} + \frac {R ^ {2}}{4} \| \bar {\theta} _ {I} ^ {*} - \theta_ {t} ^ {*} \| _ {2} ^ {2} + o \left(\frac {\log (d / \delta)}{N _ {0}}\right) \\ \leq \frac {1 6 \ln ((d + 1) / \delta)}{N _ {0}} + \frac {R ^ {2}}{4} \left(V _ {I} ^ {\theta}\right) ^ {2} + o \left(\frac {\log (d / \delta)}{N _ {0}}\right) \tag {29} \\ \end{array}
$$
where  $V_{I}^{\theta} = \sum_{t=s+1}^{e}\|\theta_{t-1}^{*} - \theta_{t}^{*}\|_{2}$  with  $s$  being the start time of the interval  $I$  and  $e$  being the ending time. In above, the first inequality is due to the smoothness of the loss function and the second inequality comes from the AM-GM inequality. The third inequality is due to Lemma 5. The last inequality can be obtained by the definition of  $\bar{\theta}_{I}^{*}$  such that
$$
\| \bar {\theta} _ {I} ^ {*} - \theta_ {t} ^ {*} \| _ {2} ^ {2} = \left(\left\| \frac {1}{| I |} \sum_ {\tau \in I} \theta_ {\tau} ^ {*} - \theta_ {t} ^ {*} \right\| _ {2}\right) ^ {2} \leq \left(\frac {1}{| I |} \sum_ {\tau \in I} \| \theta_ {\tau} ^ {*} - \theta_ {t} ^ {*} \| _ {2}\right) ^ {2} \leq (V _ {I} ^ {\theta}) ^ {2}.
$$
Plugging (29) into (28) and taking the summation over  $T$  rounds, we obtain
$$
\begin{array}{l} \sum_ {t \in I} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \sum_ {t \in I} \widetilde {L} _ {t} (\theta) \\ \leq C _ {I} ^ {(1)} (\delta) + \frac {1 6 | I | \ln ((d + 1) | I | / \delta)}{N _ {0}} + \frac {R ^ {2}}{4} | I | (V _ {I} ^ {\theta}) ^ {2} + o \left(\frac {| I | \log (d | I | / \delta)}{N _ {0}}\right) \\ = \mathcal {O} \left(d \beta \log (| I | / \delta) + \frac {| I | \ln (d | I | / \delta)}{N _ {0}} + | I | \left(V _ {I} ^ {\theta}\right) ^ {2}\right) \\ \end{array}
$$
with probability at least  $1 - 3\delta$ . We have completed the proof.
D.2.2 Useful Lemmas
Lemma 4. Set  $\gamma = 6(1 + \beta)$  and  $\lambda = 1$ . With probability at least  $1 - 2\delta$ , for any  $\delta \in (0,1]$ , the ONS algorithm (8) running on the interval  $I \subseteq [T]$  ensures,
for any  $\theta \in \Theta$ , where  $C_I^{(1)}(\delta) = 12d\beta' \log \left(1 + |I|R^2\right) + \frac{S^2}{3\beta'} + (48\beta' + \frac{6S^2R^2}{\beta'}) \log \frac{\sqrt{2|I| + 1}}{\delta} + (4SR + \frac{S^2R^2}{\beta'}) \sqrt{\log \frac{2|I| + 1}{\delta^2}}$  and  $\beta' = \beta + 1$ .
Proof of Lemma 4. The main difference between the proof of Lemma 4 and the standard analysis for the ONS algorithm (e.g., proof of [13, Theorem 4.3]) is that the later one is for the full information online learning setting, in which the learner can exactly observe the loss function  $\widehat{L}_t$ . However, in the continuous covariate shift problem, we can only observe the empirical loss  $\widehat{L}_t$ , which poses the challenge to bound a generalization gap between the expected loss  $\widehat{L}_t$  and the empirical observation  $\widehat{L}_t$  (e.g. term (b) in (35)). We manage to show such a gap can be bounded by  $\mathcal{O}(\log T)$  with the self-normalized concentration inequality for the martingale different sequence.
We start with the analysis of the instantaneous regret. By the Taylor's theorem, we have
$$
\widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right) - \widehat {L} _ {t} (\theta) = \left\langle \nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right), \widehat {\theta} _ {t} - \theta \right\rangle - \frac {1}{2} \left(\widehat {\theta} _ {t} - \theta\right) ^ {\top} \nabla^ {2} \widehat {L} _ {t} \left(\boldsymbol {\xi} _ {t}\right) \left(\widehat {\theta} _ {t} - \theta\right), \tag {30}
$$
where  $\pmb{\xi}_t\in \mathbb{R}^d$  is a certain point on the line connecting  $\theta$  and  $\widehat{\theta}_t$ . In above, the second order derivative  $\nabla^2\widetilde{L}_t$  can be further lower bounded by
$$
\begin{array}{l} \nabla^ {2} \widehat {L} _ {t} (\pmb {\xi} _ {t}) = \frac {1}{2} \mathbb {E} _ {\mathbf {x} \sim S _ {0}} [ \sigma^ {\prime} (\mathbf {x} ^ {\top} \pmb {\xi} _ {t}) \phi (\mathbf {x}) \phi (\mathbf {x}) ^ {\top} ] + \frac {1}{2} \mathbb {E} _ {\mathbf {x} \sim S _ {t}} [ \sigma^ {\prime} (\mathbf {x} ^ {\top} \pmb {\xi} _ {t}) \phi (\mathbf {x}) \phi (\mathbf {x}) ^ {\top} ] \\ \succcurlyeq \frac {1}{4 (1 + e ^ {S R})} \left(\mathbb {E} _ {\mathbf {x} \sim S _ {0}} [ \phi (\mathbf {x}) \phi (\mathbf {x}) ^ {\top} ] + \mathbb {E} _ {\mathbf {x} \sim S _ {t}} [ \phi (\mathbf {x}) \phi (\mathbf {x}) ^ {\top} ]\right) \\ \succcurlyeq \frac {1}{2 (1 + \beta)} \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top}, \tag {31} \\ \end{array}
$$
where  $\sigma'(z) = \exp(-z) / (1 + \exp(-z))^2$  is the first-order derivative of the sigmoid function  $\sigma(z) = 1 / (1 + \exp(-z))$  and  $A \succcurlyeq B$  indicates  $A - B$  is a positive semi-definite matrix for any matrix  $A, B \in \mathbb{R}^{d \times d}$ . The first equality is due to the definition of  $\widetilde{L}_t$  and the second inequality is due to Lemma 6.
Plugging (31) into (30), we have
$$
\widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right) - \widehat {L} _ {t} (\theta) \leq \left\langle \nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right), \widehat {\theta} _ {t} - \theta \right\rangle - \frac {1}{4 (1 + \beta)} \left(\nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right) ^ {\top} \left(\widehat {\theta} _ {t} - \theta\right)\right) ^ {2}. \tag {32}
$$
Let  $\mathbb{E}_t[\cdot ] = \mathbb{E}_{S_t\sim \mathcal{D}_t}[\cdot |S_0,S_1,\ldots ,S_{t - 1}]$  be the expectation taken over the draw of  $S_{t}$  conditioned on the randomness until round  $t - 1$ . Since  $\widehat{\theta}_t$  is independent of  $S_{t}$ , we have  $\mathbb{E}_t[\widehat{L}_t(\widehat{\theta}_t)] = \widetilde{L}_t(\widehat{\theta}_t)$  and  $\mathbb{E}_t[\nabla \widehat{L}_t(\widehat{\theta}_t)] = \nabla \widetilde{L}_t(\widehat{\theta}_t)$ . Taking the expectation over both sides of (32) yields
$$
\begin{array}{l} \widetilde {L} _ {t} (\widehat {\theta} _ {t}) - \widetilde {L} _ {t} (\theta) \\ \leq \langle \nabla \widetilde {L} _ {t} (\widehat {\theta} _ {t}), \widehat {\theta} _ {t} - \theta \rangle - \frac {1}{4 (1 + \beta)} \mathbb {E} _ {t} \left[ \left(\nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}) ^ {\top} (\widehat {\theta} _ {t} - \theta)\right) ^ {2} \right] \\ = \left\langle \nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right), \widehat {\theta} _ {t} - \theta \right\rangle + \left\langle \nabla \widetilde {L} _ {t} \left(\widehat {\theta} _ {t}\right) - \nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right), \widehat {\theta} _ {t} - \theta \right\rangle - \frac {1}{4 (1 + \beta)} \mathbb {E} _ {t} \left[ \left(\nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right) ^ {\top} \left(\widehat {\theta} _ {t} - \theta\right)\right) ^ {2} \right] \tag {33} \\ \end{array}
$$
where the first term of the R.H.S. can be further bounded by
$$
\begin{array}{l} \langle \nabla \widehat {L} _ {t} (\widehat {\theta} _ {t}), \widehat {\theta} _ {t} - \theta \rangle \\ = \left\langle \nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right), \widehat {\theta} _ {t + 1} - \theta \right\rangle + \left\langle \nabla \widehat {L} _ {t} \left(\widehat {\theta} _ {t}\right), \widehat {\theta} _ {t} - \widehat {\theta} _ {t + 1} \right\rangle \\ \end{array}
$$