SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yu-Jie Zhang $^{1}$ , Zhen-Yu Zhang $^{2}$ , Peng Zhao $^{3}$ , Masashi Sugiyama $^{2,1}$
<sup>1</sup> The University of Tokyo, Chiba, Japan
2 RIKEN AIP, Tokyo, Japan
$^{3}$  National Key Laboratory for Novel Software Technology, Nanjing University, Nanjing, China
Dealing with distribution shifts is one of the central challenges for modern machine learning. One fundamental situation is the covariate shift, where the input distributions of data change from the training to testing stages while the input-conditional output distribution remains unchanged. In this paper, we initiate the study of a more challenging scenario — continuous covariate shift — in which the test data appear sequentially, and their distributions can shift continuously. Our goal is to adaptively train the predictor such that its prediction risk accumulated over time can be minimized. Starting with the importance-weighted learning, we theoretically show the method works effectively if the time-varying density ratios of test and train inputs can be accurately estimated. However, existing density ratio estimation methods would fail due to data scarcity at each time step. To this end, we propose an online density ratio estimation method that can appropriately reuse historical information. Our method is proven to perform well by enjoying a dynamic regret bound, which finally leads to an excess risk guarantee for the predictor. Empirical results also validate the effectiveness.
How to deal with distribution shifts is one of the central challenges for modern machine learning [1, 2], which is also the key requirement of robust artificial intelligence in open and dynamic environments [3, 4]. Covariate shift is a representative problem setup and has attracted much attention [5, 6], where the input density  $\mathcal{D}(\mathbf{x})$  varies from training data to test data but their input-conditional output  $\mathcal{D}(y\mid \mathbf{x})$  remains unchanged. On the one hand, covariate shift serves as a foundational setup for theoretically understanding general distribution shifts [7], and on the other hand, it can encompass many real-world tasks such as brain-computer interface [8], speaker identification [9] and audio processing [10].
Existing works of coping with covariate shift mainly focused on the "one-step" adaptation, where the learner aims to train a model well-performed on a fixed testing distribution. To mitigate the distribution discrepancy, a common and classic solution is the importance-weighting framework [6], where one assigns an appropriate importance weight to each labeled training sample and then conducts weighted empirical risk minimization. The importance weight, also known as the density ratio of test and training inputs, usually needs to be estimated via a reasonable amount of unlabeled data sampled from the testing distribution [11, 12]. However, the one-step adaptation can be insufficient in many real-world covariate shift scenarios, especially when data are accumulated in an online fashion such that testing environments continuously shift and only a few unlabeled samples are observed at each time. For instance, in the speaker identification task, the speech features vary over time due to session
* Correspondence: Yu-Jie Zhang <yujie.zhang@ms.k.u-tokyo.ac.jp> and Peng Zhao <zhaop@lamda.nju.edu.cn>
dependent variation, the recording environment change, and physical conditions and emotions of the speakers [9]. Therefore, it is necessary to perform a prompt adaptation to the changes.
Motivated by such demands, we initiate the study of continuous covariate shift, where the learner's goal is to adaptively update the model to minimize the risk accumulated over time (see the formal definition in Section 3). To achieve this goal, we propose the ACCOUS approach (Adapt to Continuous Covariate shift with Unlabeled Stream) equipped with sound theoretical guarantees. Our approach is based on the classic importance-weighting framework yet requires innovations to make it applicable to the continuous shift scenario. Indeed, we theoretically identify that the importance-weighting works effectively if the cumulative estimation error of the time-varying density ratios of train and test inputs can be reduced. However, at each time step, a direct application of one-step density ratio estimation would lead to a high variance due to data scarcity; whereas reusing all previous data can be highly biased when testing distributions change dramatically. Thus, it is crucial to design an accurate estimation of time-varying test-train density ratios by appropriately reusing historical information.
To combat the difficulty, we propose a generic reduction of the time-varying density ratio estimation problem to the online convex optimization [13]: one can immediately obtain high-quality time-varying density ratios by a suitable online process to optimize its dynamic regret over a certain sequence of loss functions. Our reduction is based on the Bregman divergence density ratio matching framework [14] and applicable to various existing density ratio estimation models with specific configurations of the divergence functions. To minimize the dynamic regret, our key algorithmic ingredient of this online optimization is the online ensemble structure [15, 16], where a group of base-learners is maintained to perform density ratio estimation with different lengths of historical data, and a meta-algorithm is employed to combine the predictions. As such, we can properly reuse historical information without knowing the cumulative intensity of the covariate shift. We further instantiate the reduction framework with the logistic regression model [17]. Letting  $V_{T} = \sum_{t=2}^{T} \| \mathcal{D}_{t}(\mathbf{x}) - \mathcal{D}_{t-1}(\mathbf{x}) \|_{1}$ , we prove an  $\widetilde{\mathcal{O}}(T^{1/3}V_{T}^{2/3})$  dynamic regret bound for the density ratio estimator, which finally leads to an  $\widetilde{\mathcal{O}}(T^{-1/3}V_{T}^{1/3})$  averaged excess risk of the predictor trained by importance-weighted learning. The rate can hardly be improved even if one receives labels of the testing stream after prediction (see more elaborations below Theorem 3 and Appendix D.6). Finally, we conduct experiments to evaluate our approach, and the empirical results validate the theoretical findings.
Technical Contributions. We note that online ensemble was also employed by [18] to handle continuous label shift, another typical distribution shift assuming the change happens on the class prior  $\mathcal{D}_t(y)$ . However, their method crucially relies on the unbiasedness of weights in the risk estimator, whereas density ratio estimators for covariate shift adaptation cannot satisfy the condition in general. Instead of pursuing unbiasedness, we delicately design an online optimization process to learn the density ratio estimator, which is versatile to be implemented with various models. Our methodology is very general and might be of broader use, such as relaxing the unbiasedness requirement in the continuous label shift. Besides, the  $\widetilde{\mathcal{O}}(T^{1/3}V_T^{2/3})$  dynamic regret bound for the logistic regression density ratio estimator is obtained non-trivially. Our bound essentially holds for online learning with exp-concave functions under noisy feedback. The only one achieving this [19] holds in expectation and requires complicated analysis on the Karush-Kuhn-Tucker (KKT) condition [20] of comparators. By contrast, our bound holds in high probability, and the analysis is greatly simplified by virtue of exploiting the structure of comparators in our problem that they are essentially the minimizers of expected functions (hence without analyzing the KKT condition).
This section introduces the preliminaries and related work on the continuous covariate shift adaptation. We discuss the related work on non-stationary online learning in Appendix B
One-Step Covariate Shift Adaptation. Let  $\mathcal{D}_0(\mathbf{x},y)$  and  $\mathcal{D}_1(\mathbf{x},y)$  be the training and test distributions. The "one-step" adaptation problem studies how to minimize the testing risk  $R_{1}(\mathbf{w}) = \mathbb{E}_{(\mathbf{x},y)\sim \mathcal{D}_1(\mathbf{x},y)}[\ell (\mathbf{w}^\top \mathbf{x},y)]$  by the model trained with a labeled dataset  $S_0 = \{\mathbf{x}_i,y_i\}_{i = 1}^{N_0}$  sampled from  $\mathcal{D}_0(\mathbf{x},y)$  and unlabeled dataset  $S_{1} = \{\mathbf{x}_{i}\}_{i = 1}^{N_{1}}$  sampled from  $\mathcal{D}_1(\mathbf{x})$ . We call this one-step adaptation since the test distribution is fixed and a reasonable number of unlabeled data  $S_{1}$  is available.
Importance-Weighted ERM. A classic solution for the one-step covariate shift adaptation is the importance-weighted empirical risk minimization (IWERM), which mitigates the distribution shift
by minimizing the weighted empirical risk  $\widetilde{R}_1(\mathbf{w}) = \mathbb{E}_{(\mathbf{x},y)\sim S_0}[r_1^* (\mathbf{x})\ell (\mathbf{w}^\top \mathbf{x},y)]$ , where  $r_1^* (\mathbf{x}) = \mathcal{D}_1(\mathbf{x}) / \mathcal{D}_0(\mathbf{x})$  is the importance weight. The risk  $\widetilde{R}_1(\mathbf{w})$  is unbiased to  $R_{1}(\mathbf{w})$  and thus the learned model is consistent over the test distribution [5]. The importance weighted learning was studied from the lens of variance-bias trade off [5, 21], cross validation [22], model misspecification [23], and deep neural network implementation [24]. All those are conducted under the one-step adaptation scenario.
Density Ratio Estimation. The importance weighted estimation, also known as the density ratio estimation (DRE) [12], aims to estimate the density ratio  $r_1^*(\mathbf{x})$  using datasets input of  $S_0$  and  $S_1$ . Various methods were proposed with different statistical models [14, 17, 25, 26, 27, 28, 29]. All those methods focused on the one-step adaptation, and it is challenging to extend them to the continuous shift due to the limited unlabeled data at each time step. The work [30] studied how to update the density ratio estimator with streaming data, but the ground-truth density ratio is assumed to be fixed. Recently, the time-varying density ratio was investigated in [31]. Their problem setup fundamentally differs from ours as they assume sufficient online data at each iteration, while our challenge is dealing with limited data per iteration. The work [32] proposed to learn the time-varying density ratios using all past data at each time. Although learning theory insights are provided in the paper, it is still unclear how the learned density ratios balance sample complexity with environmental shift intensity. In contrast, our density ratio estimator updates in an online fashion with dynamic regret guarantees.
Continuous Distribution Shift. For broader continuous distribution shift problems, the study [33] focused on the label shift case and provided the first feasible solution. Then, the work [18] introduced modern non-stationary online learning techniques to the problem, developing the first method with dynamic regret guarantees. As shown in Remark 2, our solution refines the previous methods [33, 18] by decoupling density ratio estimation from predictor training, paving the way for a theoretically-grounded method for continuous covariate shift. Similarly, the subsequent work [34] developed a method for continuous label shift, which allows for training the predictor with various models by separately estimating the label probability through an online regression oracle. Another research [35] studied the change detection for continuous covariate shift. However, they only focused on identifying differences between the current and initial offline distributions, and it is still unclear how to update the model adaptively with the online data. Moreover, the method [35] can only detect at a given time granularity, while ours can perform the model update at each round.
3 Adapting to Continuous Covariate Shift
In this section, we formulate the problem setup of continuous covariate shift and then introduce our approach based on the IWERM framework and online density ratio estimation.
3.1 Problem Setup
There are two stages in continuous covariate shift. The first one is the offline initialization stage, where the learner can collect a reasonable number of label data  $S_0 = \{\mathbf{x}_n, y_n\}_{n=1}^{N_0}$  from the initial distribution  $\mathcal{D}_0$ . Then, we come to the online adaptation stage, where the unlabeled testing data arrive sequentially, and the underlying distributions can continuously shift. Consider a  $T$ -round online adaptation. At each round  $t \in [T] \triangleq \{1, \dots, T\}$ , the learner will receive an unlabeled dataset  $S_t = \{\mathbf{x}_n\}_{n=1}^{N_t}$  sampled from the underlying distribution  $\mathcal{D}_t$ . Without loss of generality, we consider  $N_t = 1$ . We have the following continuous covariate shift condition.
Assumption 1 (Continuous Covariate Shift). For all  $\mathbf{x} \in \mathcal{X}$  in the feature space,  $y \in \mathcal{Y}$  in the label space, and any  $t \in [T]$ , we have
$$
\mathcal {D} _ {t} (y \mid \mathbf {x}) = \mathcal {D} _ {0} (y \mid x) \text {a n d} r _ {t} ^ {*} (\mathbf {x}) = \mathcal {D} _ {t} (\mathbf {x}) / \mathcal {D} _ {0} (\mathbf {x}) \leq B <   \infty .
$$
We note that there are emerging discussions on the necessity of covariate shift adaptation [7, 23]. When there are infinite number of training samples, a well-specified large and over-parametrized model can be trained to approximate  $\mathcal{D}_0(y\mid \mathbf{x})$  and there is no need to perform covariate shift adaptation. However, in the finite-sample cases, one would prefer to train a model with constraint complexity to ensure its generalization ability, which leads to model misspecification and covariate shift adaptation is indeed necessary. In this paper, we study the continuous covariate shift under a misspecified hypothesis space  $\mathcal{W}$ , which generalizes the standard one-step covariate shift problem (see [6] and reference therein) to the continuous shift case. Our goal is to train a sequence of model
$\{\mathbf{w}_t\}_{t=1}^T$  that are comparable with the best model  $\mathbf{w}_t^* \in \arg \min_{\mathbf{w} \in \mathcal{W}} R_t(\mathbf{w})$  in the hypothesis space at each time. Consequently, we take the following average excess risk as the performance measure:
$$
\Re_ {T} \left(\left\{\widehat {\mathbf {w}} _ {t} \right\} _ {t = 1} ^ {T}\right) \triangleq \frac {1}{T} \left(\sum_ {t = 1} ^ {T} R _ {t} \left(\widehat {\mathbf {w}} _ {t}\right) - \sum_ {t = 1} ^ {T} R _ {t} \left(\mathbf {w} _ {t} ^ {*}\right)\right). \tag {1}
$$
We end this part by listing several common notations used throughout the paper. We denote by  $R = \max_{\mathbf{x} \in \mathcal{X}} \| \mathbf{x} \|_2$  the maximum norm of the input and by  $D = \max_{\mathbf{w}_1, \mathbf{w}_2 \in \mathcal{W}} \| \mathbf{w}_1 - \mathbf{w}_2 \|_2$  the diameter of the hypothesis space  $\mathcal{W} \subseteq \mathbb{R}^d$ . The constant  $G = \max_{\mathbf{x} \in \mathcal{X}, y \in \mathcal{Y}, \mathbf{w} \in \mathcal{W}} \| \nabla \ell(\mathbf{w}^\top \mathbf{x}, y) \|_2$  is the maximum gradient norm and  $L = \max_{\mathbf{x} \in \mathcal{X}, y \in \mathcal{Y}, \mathbf{w} \in \mathcal{W}} |\ell(\mathbf{w}^\top \mathbf{x}, y)|$  is the upper bound of the loss values. We use the  $\widetilde{\mathcal{O}}(\cdot)$ -notation to hide dependence on logarithmic factors of  $T$ .
3.2 Importance Weighted ERM for Continuous Shift
Our algorithm is based on the importance weighted learning. Consider the scenario where the learner has a predefined  $\widehat{r}_t:\mathcal{X}\to [0,B]$  as an estimation for the true density ratio  $r_t^* (\mathbf{x}) = \mathcal{D}_t(\mathbf{x}) / \mathcal{D}_0(\mathbf{x})$  for each time  $t\in [T]$ . Then, the predictor can be trained by the importance weighted ERM method:
$$
\widehat {\mathbf {w}} _ {t} = \arg \min  _ {\mathbf {w} \in \mathcal {W}} \mathbb {E} _ {\mathbf {x} \sim S _ {0}} \left[ \widehat {r} _ {t} (\mathbf {x}) \ell \left(\mathbf {w} ^ {\top} \mathbf {x}, y\right) \right]. \tag {2}
$$
The averaged excess risk of the predictor is closely related to the quality of the density ratio estimator. Proposition 1. For any  $\delta \in (0,1]$ , with probability at least  $1 - \delta$ , IwERM (2) with the estimator  $\widehat{r}_t(\mathbf{x})$  ensures  $\Re_T(\{\widehat{\mathbf{w}}_t\}_{t=1}^T) \leq 2\sum_{t=1}^T \mathbb{E}_{\mathbf{x} \sim S_0}\left[|\widehat{r}_t(\mathbf{x}) - r_t^*(\mathbf{x})| \right]/T + \mathcal{O}(\log(T/\delta)/\sqrt{N_0})$ .
In above, the  $\mathcal{O}(\log T / \sqrt{N_0})$  term measures the generalization gap of the predictor since it is trained over the empirical data  $S_{0}$  instead of  $\mathcal{D}_0$ . Such a rate is tight up to a logarithmic factor in  $T$ . Indeed, considering a stationary environment (i.e.,  $\mathcal{D}_t(\mathbf{x},y) = \mathcal{D}_1(\mathbf{x},y)$ ) and an exact density ratio estimation (i.e.,  $\widehat{r_t} (\mathbf{x}) = r_t^* (\mathbf{x}))$ , Proposition 1 indicates that the averaged model  $\overline{\mathbf{w}}_T = \sum_{t = 1}^{T}\widehat{\mathbf{w}}_t / T$  enjoys an excess risk bound of  $R_{1}(\overline{\mathbf{w}}_{T}) - R_{1}(\mathbf{w}_{1}^{*})\leq \mathcal{O}(\log T / \sqrt{N_{0}})$ , matching the lower bound for importance weighted learning [36, Proposition 2] up to an  $\mathcal{O}(\log T)$  factor.
Our main focus lies on the  $\sum_{t=1}^{T} \mathbb{E}_{\mathbf{x} \sim S_0} [|\widehat{r}_t(\mathbf{x}) - r_t^*(\mathbf{x})|] / T$  term, which is the averaged estimation error of the density ratio estimator  $\widehat{r}_t$  to the time-varying ground truth  $r_t^*$ . To minimize the estimation error, a fundamental challenge comes from the unknown non-stationarity exhibited in the online environments — how to select a right amount of historical data to reuse for each iteration? Intuitively, if the environments change slowly, it would be preferable to reuse all historical data to construct the density ratio estimator  $\widehat{r}_t$ . However, when distribution shifts occur frequently, earlier data could be useless even potentially harmful. In such scenarios, training the density ratio estimator with the most recent data would be a more rational strategy. Furthermore, even if we could capture a roughly stationary period, it remains unclear how to update the density ratio estimator in an online manner with guarantees. To our knowledge, online density ratio estimation is underexplored in the literature, even for the stationary testing streaming, not to mention the more challenging non-stationary setup.
3.3 Online Density Ratio Estimation
Here, we present a generic reduction of the online density ratio estimation problem to a dynamic regret minimization problem. This novel perspective paves our way for designing an algorithm in tackling the non-stationarity of environments, as mentioned in Section 3.2.
Bregman Divergence Density Ratio Matching. Our reduction is based on the Bregman divergence density ratio matching [14], a general framework that unifies various existing DRE methods. Specifically, in the framework, the discrepancy between the ground-truth density ratio function  $r_t^*$  and any density ratio function  $r$  is measured by the expected Bregman divergence over  $\mathcal{D}_0(\mathbf{x})$  defined by
$$
\operatorname {E B} _ {\psi} \left(r _ {t} ^ {*} \| r\right) = \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0} (\mathbf {x})} \left[ \mathcal {B} _ {\psi} \left(r _ {t} ^ {*} (\mathbf {x}) \| r (\mathbf {x})\right) \right], \tag {3}
$$
where  $\mathcal{B}_{\psi}(a\|b) \triangleq \psi(a) - \psi(b) - \partial \psi(b)(a - b)$  is the Bregman divergence and  $\psi: \operatorname{dom} \psi \to \mathbb{R}$  is the associated divergence function. The expected Bregman divergence can be equally rewritten as  $\mathrm{EB}_{\psi}(r_t^* \| r) = L_t^{\psi}(r) - L_t^{\psi}(r_t^*)$ , where  $L_t^{\psi}$  is the loss for the density ratio function defined by
$$
L _ {t} ^ {\psi} (r) = \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {0} (\mathbf {x})} \left[ \partial \psi (r (\mathbf {x})) r (\mathbf {x}) - \psi (r (\mathbf {x})) \right] - \mathbb {E} _ {\mathbf {x} \sim \mathcal {D} _ {t} (\mathbf {x})} \left[ \partial \psi (r (\mathbf {x})) \right]. \tag {4}
$$
As a consequence, one can train the density ratio estimator by minimizing the loss  $L_{t}^{\psi}(r)$ . We have  $r_{t}^{*} \in \arg \min_{r \in \mathcal{H}} L_{t}^{\psi}(r)$  when  $\mathcal{H}$  is the set of all measurable functions.
The Bregman divergence-based density ratio matching framework takes various DRE methods as special cases. For instance, choosing the divergence function as  $\psi_{\mathrm{LS}}(t) = (t - 1)^2 /2$  yields LSIF [28] and KMM [26]; choosing  $\psi_{\mathrm{LR}}(t) = t\log t - (t + 1)\log (t + 1)$  leads to the logistic regression method [17]; and one can also recover the UKL [37] and KLLEP [25] with  $\psi_{\mathrm{KL}}(t) = t\log t - t$
Online DRE via Dynamic Regret Minimization. For a single-round density ratio estimation, the Bregman divergence-based framework suggests to train the density ratio estimator  $\widehat{r}_t$  by minimizing the gap  $L_{t}^{\psi}(\widehat{r}_{t}) - L_{t}^{\psi}(r_{t}^{*})$ . Therefore, to derive an estimator sequence that performs well over time, we utilize the cumulative loss gap  $\sum_{t = 1}^{T}L_{t}^{\psi}(\widehat{r}_{t}) - \sum_{t = 1}^{T}L_{t}^{\psi}(r_{t}^{*})$  as a performance measure. Indeed, the cumulative loss gap serves as an upper bound of the estimation error of density ratio estimators.
Proposition 2. Let  $\psi$  be a  $\mu$ -strongly convex function. For any density ratio estimator sequence  $\{\widehat{r}_t\}_{t=1}^T$ , we have  $\frac{1}{T}\sum_{t=1}^{T}\mathbb{E}_{\mathbf{x}\sim \mathcal{D}_0}[\|\widehat{r}_t(\mathbf{x}) - r_t^*(\mathbf{x})\|] \leq \sqrt{2\left(\sum_{t=1}^{T}L_t^\psi(\widehat{r}_t) - \sum_{t=1}^{T}L_t^\psi(r_t^*)\right)/(\mu T)}$ .
Proposition 2 indicates that it suffices to optimize the cumulative loss gap to perform online density ratio estimation. Such a measure quantifies the performance difference between the online algorithm (that yields the estimated ratio sequence  $\{\widehat{r}_t\}_{t=1}^T$ ) and a sequence of time-varying comparators (the true ratio sequence  $\{r_t^*\}_{t=1}^T$ ). This is exactly the dynamic regret in online learning literature [38, 39]. As such, we have reduced the online density ratio estimation to a problem of dynamic regret minimization over loss functions  $\{L_t^\psi\}_{t=1}^T$  against comparators  $\{r_t^*\}_{t=1}^T$ .
Since the loss function involves the expectation over the underlying distribution  $\mathcal{D}_0(\mathbf{x})$  (see the definition in (4)), we need a counterpart result with respect to empirical data  $S_0$ .
Theorem 1. Let  $\psi$  be a  $\mu$ -strongly convex function satisfying  $t\partial^3\psi(t) \leq 0$  and  $\partial^3\psi(t) \leq 0$  for all  $t \in \operatorname{dom}\psi$ . Let  $\mathcal{H}_{\theta} = \{\mathbf{x} \mapsto h(\mathbf{x},\theta) \mid \theta \in \Theta\}$  be a hypothesis space of density ratio functions parameterized by a finite-dimensional bounded set  $\Theta \triangleq \{\theta \in \mathbb{R}^d \mid \| \theta \|_2 \leq S\}$  with a certain link function  $h: \mathcal{X} \times \Theta \mapsto \mathbb{R}$ . Denote by  $[z]_+ \triangleq \max\{z, 0\}$ . Then, for any density ratio estimator  $\widehat{r}_t \in \mathcal{H}_{\theta}$ , the empirical estimation error is bounded by
$$
\frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} _ {\mathbf {x} \sim S _ {0} (\mathbf {x})} \left[ | r _ {t} ^ {*} (\mathbf {x}) - \widehat {r} _ {t} (\mathbf {x}) | \right] \leq \sqrt {\frac {4}{\mu T} \left[ \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} ^ {\psi} (\widehat {r} _ {t}) - \sum_ {t = 1} ^ {T} \widetilde {L} _ {t} ^ {\psi} (r _ {t} ^ {*}) \right] _ {+}} + \mathcal {O} \left(\frac {\sqrt {d} \log (T / \delta)}{\mu \sqrt {N _ {0}}}\right),
$$
provided that  $h(\mathbf{x},\theta)$  is bounded for any  $\mathbf{x} \in \mathcal{X}$  and  $\theta \in \Theta$  and Lipschitz continuous. In the above,