SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \left\| \theta_ {t - 1} ^ {*} - \theta_ {t} ^ {*} \right\| _ {2} \leq \frac {1}{\alpha} \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0}} \left[ | \phi (\mathbf {x}) ^ {\top} \left(\theta_ {t - 1} ^ {*} - \theta_ {t} ^ {*}\right) | \right] \\ \leq \frac {\beta}{\alpha} \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0}} \left[ | e ^ {\phi (\mathbf {x}) ^ {\top} \theta_ {t - 1} ^ {*}} - e ^ {\phi (\mathbf {x}) ^ {\top} \theta_ {t} ^ {*}} | \right] \leq \frac {\beta}{\alpha} \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0}} \left[ \left| \frac {\mathcal {D} _ {t} (\mathbf {x})}{\mathcal {D} _ {0} (\mathbf {x})} - \frac {\mathcal {D} _ {t - 1} (\mathbf {x})}{\mathcal {D} _ {0} (\mathbf {x})} \right| \right] = \frac {\beta}{\alpha} \| \mathcal {D} _ {t} (\mathbf {x}) - \mathcal {D} _ {t - 1} (\mathbf {x}) \| _ {1}, \\ \end{array}
$$
where the first inequality is by the condition  $\mathbb{E}_{\mathbf{x} \sim \mathcal{D}_0} [|\phi(\mathbf{x})^\top \mathbf{a}|] \geq \alpha \| \mathbf{a}\|_2$  and the second is by the mean value theorem and the boundedness of  $|\phi(\mathbf{x})^\top (\theta_{t-1}^* - \theta_t^*)| \leq 2SR$  for any  $\mathbf{x} \in \mathcal{X}$ . The last inequality is due to the realizable assumption.
D.5 Proof of Theorem 3
Proof of Theorem 3. Then, we can bound the overall excess risk over the model trained by IW-ERM (1) by a combination of Proposition 1 and Theorem 1 as
$$
\begin{array}{l} \frac {1}{T} \left(\sum_ {t = 1} ^ {T} R _ {t} \left(\widehat {\mathbf {w}} _ {t}\right) - \sum_ {t = 1} ^ {T} R _ {t} \left(\mathbf {w} _ {t} ^ {*}\right)\right) \leq \frac {B \cdot C _ {T} (\delta)}{\sqrt {N _ {0}}} + \frac {2 L}{T} \sum_ {t = 1} ^ {T} \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ | \widehat {r} _ {t} (\mathbf {x}) - r _ {t} ^ {*} (\mathbf {x}) | \right] \\ \leq \frac {B \cdot C _ {T} (\delta)}{\sqrt {N _ {0}}} + \frac {4 L \beta}{T} \sqrt {T \left[ \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} \left(\widehat {\theta} _ {t}\right) - \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} \left(\theta_ {t} ^ {*}\right) \right] _ {+}} + \mathcal {O} \left(\frac {\log (T / \delta) \sqrt {d}}{\sqrt {N _ {0}}}\right), \tag {55} \\ \end{array}
$$
where the first inequality is due to Proposition 1 and  $C_T(\delta) = 2GDR + 5L\sqrt{2\log(8T / \delta)} = \mathcal{O}(\log (T / \delta))$ . The second inequality is due to Theorem 1 with the choice of the logistic regression model  $\psi(t) = t\log t - (1 + t)\log (1 + t)$  and  $\widehat{r}_t(\mathbf{x}) = \exp(-\phi(\mathbf{x})^\top \widehat{\theta}_t)$  and the realizability assumption  $r_t^*(\mathbf{x}) = \exp(-\phi(\mathbf{x})^\top \theta_t^*)$ . Then, by Theorem 2, we have
$$
\sum_ {t = 1} ^ {T} \widetilde {L} _ {t} (\widehat {r} _ {t}) - \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} (r _ {t} ^ {*}) \leq \mathcal {O} \bigg (d ^ {\frac {4}{3}} \log T \log (T / \delta) \cdot \max \{T ^ {\frac {1}{3}} V _ {T} ^ {\frac {2}{3}}, 1 \} + \frac {T \ln (d T / \delta)}{N _ {0}} \bigg),
$$
$$
\sqrt {T \left[ \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} \left(\widehat {\theta} _ {t}\right) - \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} \left(\theta_ {t} ^ {*}\right) \right] _ {+}} \leq \mathcal {O} \left(d ^ {\frac {2}{3}} \log \left(T / \delta\right) \cdot \max  \left\{T ^ {\frac {2}{3}} V _ {T} ^ {\frac {1}{3}}, \sqrt {T} \right\} + T \sqrt {\frac {\ln (d T / \delta)}{N _ {0}}}\right). \tag {56}
$$
We complete the proof by substituting (56) it into (55).
D.6 Discussion on Tightness of the Bound
This part illustrates the tightness of our bound with the case where the labels of testing samples are available after the prediction.
Continuous Shift with Labeled Feedback. We consider a  $T$ -round online learning process. At iteration  $t$ , the learner submits her prediction  $\widehat{\mathbf{w}}_t \in \mathcal{W}$ . At the same time, the environments pick the data pair  $(\mathbf{x}_t, y_t) \sim \mathcal{D}_t$ . Then, the learner updates her model with the loss function  $\widehat{R}_t(\mathbf{w}) = \ell(\mathbf{w}^\top \mathbf{x}_t, y_t)$  and obtains the prediction  $\widehat{\mathbf{w}}_{t+1} \in \mathcal{W}$  for the next iteration. The goal of the learner is to minimize the cumulative excess risk against the sequence of optimal models  $\mathbf{w}_t^* \in \arg \min_{\mathbf{w} \in \mathcal{W}} R_t(\mathbf{w})$ , that is,
$$
\mathfrak {R} _ {T} \left(\left\{\widehat {\mathbf {w}} _ {t} \right\} _ {t = 1} ^ {T}\right) = \frac {1}{T} \sum_ {t = 1} ^ {T} R _ {t} \left(\widehat {\mathbf {w}} _ {t}\right) - \sum_ {t = 1} ^ {T} R _ {t} \left(\mathbf {w} _ {t} ^ {*}\right). \tag {57}
$$
The goal is the same as that of standard continuous covariate shift, see the problem setup in (1). The key difference is that now the label of testing data is available. Therefore, the noisy feedback  $\widehat{R}_t(\mathbf{w})$  observed by the learner is an unbiased estimator with respect to the true risk  $R_t(\mathbf{w})$ .
Tightness of Our Bound. The continuous shift with labeled feedback is essentially a non-stationary stochastic optimization problem [39]. For general convex functions, [39] showed that any gradient-based algorithm will suffer
$$
\mathbb {E} \left[ \Re_ {T} \left(\left\{\widehat {\mathbf {w}} _ {t} \right\} _ {t = 1} ^ {T}\right) \right] = \Omega \left(T ^ {- 1 / 3} \left(V _ {T} ^ {L}\right) ^ {1 / 3}\right) \tag {58}
$$
in the worst case. In above,  $V_{T}^{L} = \sum_{t=2}^{T} \max_{\mathbf{w} \in \mathcal{W}} |R_{t}(\mathbf{w}) - R_{t-1}(\mathbf{w})|$  measures the fluctuation of the risk function. For the same performance measure, our algorithm achieves
$$
\Re_ {T} \left(\left\{\widehat {\mathbf {w}} _ {t} \right\} _ {t = 1} ^ {T}\right) = \widetilde {\mathcal {O}} \left(\frac {1}{\sqrt {N _ {0}}} + \max  \left\{T ^ {- 1 / 3} V _ {T} ^ {1 / 3}, T ^ {- \frac {1}{2}} \right\}\right). \tag {59}
$$
In the non-stationary case, i.e.,  $V_{T} \geq \Theta (T^{-\frac{1}{2}})$ , our bound becomes  $\widetilde{\mathcal{O}}\left(1 / \sqrt{N_0} + T^{-1/3}V_T^{1/3}\right)$ . As discussed below Proposition 1, the first term  $\widetilde{\mathcal{O}}(1 / \sqrt{N_0})$  is hard to be improved. We thus focus on the tightness of the second term  $\widetilde{\mathcal{O}}(T^{-1/3}V_T^{1/3})$ .
Although the definition of  $V_{T}$  in our upper bound is different from the that of  $V_{T}^{L}$  in the lower bound (58), the two bounds share the same dependence on the time horizon  $T$ , which provides evidence that our bound is hard to improve. Indeed, consider the 1-dimensional case where the underlying distribution only shifts once from  $\mathcal{D}_1(x,y)$  to  $\mathcal{D}_2(x,y)$  at a certain time  $t\in [T]$ . In such a case,  $V_{T} = \mathcal{O}(1)$  and our bound implies an  $\mathcal{O}(T^{-1 / 3})$  rate for the second term of (59). On the other hand, the rate in the lower bound (58) becomes  $\Omega (T^{-1 / 3})$ . The same dependence on the time horizon  $T$  indicates our bound is hard to improve. We leave a precise lower bound argument as the future work.
D.7 On Other Choices of Bregman Divergence Function
In this section, we discuss how to apply the other two choice of the divergence function  $\psi_{\mathrm{LS}} = (t - 1)^2 /2$  and  $\psi_{\mathrm{KL}} = t\log t - t$  in our framework. Our analysis crucially relies on two conditions:
1. the Bregman divergence function  $\psi$  satisfies the conditions in Theorem 1.
2. the induced loss function  $\widehat{L}_t(\theta)$  is an exp-concave and smooth function with respect to  $\theta$ .
Choice of  $\psi_{\mathsf{LS}}(t) = (t - 1)^2 /2$ . Considering the Bregman divergence function  $\psi (t) = (t - 1)^2 /2$  and the hypothesis space  $H_{\theta} = \{\mathbf{x}\mapsto \phi (\mathbf{x})^{\top}\theta \mid \| \theta \| _2\leq S\}$ , where  $\phi :\mathcal{X}\to \mathbb{R}^{d}$  represents a specific basis function with bounded norm  $\| \phi (\mathbf{x})\| _2\leq R$ . Then, the loss  $\widehat{L}_t^\psi (\theta)$  as per (7) becomes
$$
\widehat {L} _ {t} (\theta) = \frac {1}{2} \mathbb {E} _ {S _ {0}} [ (\phi (\mathbf {x}) ^ {\top} \theta) ^ {2} ] - \mathbb {E} _ {S _ {t}} [ \phi (\mathbf {x}) ^ {\top} \theta ].
$$
The above configuration recovers the unconstrained least-squares importance fitting (uLSIF) method [28]. We can show such a choice of divergence function satisfy the condition required by Theorem 1 and enjoys favorable function properties.
- Conditions in Theorem 1: when Bregman divergence function is chosen as  $\psi_{\mathrm{LS}}$ , we have  $\partial^2\psi_{\mathrm{LS}}(t) = 1$  and  $\partial^3\psi_{\mathrm{LS}}(t) = 0$ . Thus,  $\psi_{\mathrm{LS}}$  is a 1-strongly convex function and  $t\partial^3\psi_{\mathrm{LS}}(t) = 0$  for  $t \in \mathbb{R}$ .
- Expconcavity and smoothness: We have  $\nabla^2\widehat{L}_t(\theta) = \mathbb{E}_{S_0}[\phi (\mathbf{x})\phi (\mathbf{x})^\top ]$ . Then, when the input is upper bounded by  $\| \phi (\mathbf{x})\| _2\leq R$  for all  $\phi (\mathbf{x})\in \mathcal{X}$ , we have  $\nabla^2\widehat{L}_t(\theta)\preccurlyeq R^2 I_d$ , which implies the smoothness. Furthermore, if the initial data ensure  $\mathbb{E}_{S_0}[\phi (\mathbf{x})\phi (\mathbf{x})^\top ]\succcurlyeq \alpha I_d$ , the  $\widehat{L}_t(\theta)$  is strongly convex and thus is exp-concave as shown by [60, Lemma 2]. We note that the regularity condition on the offline data  $\mathbb{E}_{S_0}[\phi (\mathbf{x})\phi (\mathbf{x})^\top ]\succcurlyeq \alpha I_d$  is used in the analysis of LSIF algorithm [28, Lemma 1].
**Choice of  $\psi_{\mathsf{KL}} = t\log t - t$ .** Considering the Bregman divergence function  $\psi (t) = t\log t - t$  and the hypothesis  $H_{\theta} = \{\mathbf{x}\mapsto \phi (\mathbf{x})^{\top}\theta \mid \| \theta \| _2\leq S\}$ , where  $\phi :\mathcal{X}\to \mathbb{R}_+^d$  represents a specific basis function with bounded norm  $\| \phi (\mathbf{x})\| _2\leq R$ . Then, the loss  $\widehat{L}_t^\psi (\theta)$  as per (7) becomes
$$
\widehat {L} _ {t} (\theta) = \mathbb {E} _ {S _ {0}} [ \phi (\mathbf {x}) ^ {\top} \theta) ] - \mathbb {E} _ {S _ {t}} [ \log (\phi (\mathbf {x}) ^ {\top} \theta) ].
$$
To ensure  $\widehat{L}_t(\theta)$  is well-defined, we further require the density function  $\widehat{r}(\mathbf{x}) = \phi(\mathbf{x})^\top \theta > 1/\beta$  for any  $\mathbf{x} \in \mathcal{X}$  and  $\theta \in \Theta \triangleq \{\theta \mid \| \theta \|_2 \leq S\}$  with a certain positive constant  $\beta > 0$ . The above configuration recovers the UKL method [37] with the generalized linear model. We can show such a choice of divergence function satisfy the condition required by Theorem 1 and enjoys favorable function properties.
- Conditions in Theorem 1: When Bregman divergence function is chosen as  $\psi_{\mathsf{KL}}$ , we have  $\partial^2\psi_{\mathsf{KL}}(t) = 1 / t$  and  $\partial^3\psi_{\mathsf{KL}}(t) = -1 / t^2$ . When the input of the loss function satisfies  $t \geq \beta$ ,  $\psi_{\mathsf{KL}}(t)$  is a  $1 / \beta$ -strongly convex function, and we can also check that  $t\partial^3\psi_{\mathsf{KL}}(t) \leq 0$  for all  $t \in \mathbb{R}_+$ .
- Expconcavity and smoothness: We have  $\nabla^2\widehat{L}_t(\theta) = \mathbb{E}_{S_t}[\phi (\mathbf{x})\phi (\mathbf{x})^\top /( \phi (\mathbf{x})^\top \theta)^2 ]$  . Then, since the output of the density ratio function  $\widehat{r} (\mathbf{x}) = \phi (\mathbf{x})^\top \theta \geq 1 / \beta$  for any  $\mathbf{x}\in \mathcal{X}$  and  $\theta \in \Theta$  and the boundedness of the input feature  $\| \phi (\mathbf{x})\| _2\le R$  , we have  $\nabla^2\widehat{L}_t(\theta)\preccurlyeq$ $\beta^2 R^2 I_d$  for any  $\theta \in \Theta$  . Then, the loss function is a smooth function. Besides, under the regularity condition  $\mathbb{E}_{S_0}[\mathbf{xx}^\top ]\succcurlyeq \alpha I_d$  of the initial data and again the boundedness of  $\Theta$  and  $\| \phi (\mathbf{x})\| _2\leq R$  , the loss function is also a strongly convex function by  $\nabla^2\widehat{L}_t(\theta)\succcurlyeq$ $\alpha /(SR)^2 I_d$  , which indicates the expconcavity of the loss function.
E Technical Lemmas
This section presents several useful technical lemmas used in the proof.
Lemma 16. For any  $\mathbf{a},\mathbf{b}\in \mathbb{R}^d$  , we have  $(\mathbf{a} + \mathbf{b})(\mathbf{a} + \mathbf{b})^\top \preccurlyeq 2(\mathbf{aa}^\top +\mathbf{bb}^\top)$  , where for any matrix  $A,B\in \mathbb{R}^{d\times d}$ $A\preccurlyeq B$  indicates  $B - A$  is a positive semi-definite matrix.
Proof. For any  $\mathbf{x} \in \mathbb{R}^d$ ,  $\mathbf{x}^\top \big(2(\mathbf{aa}^\top + \mathbf{bb}^\top) - (\mathbf{a} + \mathbf{b})(\mathbf{a} + \mathbf{b})^\top\big)\mathbf{x} = \mathbf{x}^\top \big(\mathbf{aa}^\top + \mathbf{bb}^\top - \mathbf{ab}^\top - \mathbf{ba}^\top\big)\mathbf{x}$ $= \mathbf{x}^\top \big((\mathbf{a} - \mathbf{b})(\mathbf{a} - \mathbf{b})^\top\big)\mathbf{x} \geq 0$ , which completes the proof.
Lemma 17 (Theorem 4 of [68]). Let  $\{Z_i : i \geq 1\}$  be a martingale difference with the filtration  $\mathfrak{F} = \{\mathcal{F}_n : n \geq 1\}$  and suppose  $|Z_i| \leq R$  for all  $i \geq 1$ . Then, for any  $\delta \in (0,1)$ ,  $r > 0$ , with probability at least  $1 - \delta$ ,
$$
\left| \sum_ {i = 1} ^ {t} Z _ {i} \right| \leq r \left(\sum_ {i = 1} ^ {t} Z _ {i} ^ {2} + \sum_ {i = 1} ^ {t} \mathbb {E} [ Z _ {i} ^ {2} \mid \mathcal {F} _ {i - 1} ]\right) + \frac {1}{r} \log \frac {\sqrt {2 t + 1}}{\delta} + R \sqrt {\log \frac {2 t + 1}{\delta^ {2}}}.
$$