Title: Adapting to Continuous Covariate Shift via Online Density Ratio Estimation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Covariate Shift

Density Ratio Estimation

Online Convex Optimization

Dynamic Regret

Logistic Regression
Abstract: Dealing with distribution shifts is one of the central challenges for modern machine learning. One fundamental situation is the covariate shift, where the input distributions of data change from the training to testing stages while the input-conditional output distribution remains unchanged. In this paper, we initiate the study of a more challenging scenario --- continuous covariate shift --- in which the test data appear sequentially, and their distributions can shift continuously. Our goal is to adaptively train the predictor such that its prediction risk accumulated over time can be minimized. Starting with the importance-weighted learning, we theoretically show the method works effectively if the time-varying density ratios of test and train inputs can be accurately estimated. However, existing density ratio estimation methods would fail due to data scarcity at each time step. To this end, we propose an online density ratio estimation method that can appropriately reuse historical information. Our method is proven to perform well by enjoying a dynamic regret bound, which finally leads to an excess risk guarantee for the predictor. Empirical results also validate the effectiveness.

Adapting to Continuous Covariate Shift via Online Density Ratio Estimation

Abstract

Dealing with distribution shifts is one of the central challenges for modern machine learning. One fundamental situation is the covariate shift , where the input distributions of data change from the training to testing stages while the input-conditional output distribution remains unchanged. In this paper, we initiate the study of a more challenging scenario continuous covariate shift - in which the test data appear sequentially, and their distributions can shift continuously. Our goal is to adaptively train the predictor such that its prediction risk accumulated over time can be minimized. Starting with the importance-weighted learning, we theoretically show the method works effectively if the time-varying density ratios of test and train inputs can be accurately estimated. However, existing density ratio estimation methods would fail due to data scarcity at each time step. To this end, we propose an online density ratio estimation method that can appropriately reuse historical information. Our method is proven to perform well by enjoying a dynamic regret bound, which finally leads to an excess risk guarantee for the predictor. Empirical results also validate the effectiveness.

1 Introduction

How to deal with distribution shifts is one of the central challenges for modern machine learning [1, 2], which is also the key requirement of robust artificial intelligence in open and dynamic environments [3, 4]. Covariate shift is a representative problem setup and has attracted much attention [5, 6], where the input density D ( x ) varies from training data to test data but their input-conditional output D ( y | x ) remains unchanged. On the one hand, covariate shift serves as a foundational setup for theoretically understanding general distribution shifts [7], and on the other hand, it can encompass many real-world tasks such as brain-computer interface [8], speaker identification [9] and audio processing [10].

Existing works of coping with covariate shift mainly focused on the 'one-step' adaptation, where the learner aims to train a model well-performed on a fixed testing distribution. To mitigate the distribution discrepancy, a common and classic solution is the importance-weighting framework [6], where one assigns an appropriate importance weight to each labeled training sample and then conducts weighted empirical risk minimization. The importance weight, also known as the density ratio of test and training inputs, usually needs to be estimated via a reasonable amount of unlabeled data sampled from the testing distribution [11, 12]. However, the one-step adaptation can be insufficient in many real-world covariate shift scenarios, especially when data are accumulated in an online fashion such that testing environments continuously shift and only a few unlabeled samples are observed at each time. For instance, in the speaker identification task, the speech features vary over time due to session

dependent variation, the recording environment change, and physical conditions and emotions of the speakers [9]. Therefore, it is necessary to perform a prompt adaptation to the changes.

Motivated by such demands, we initiate the study of continuous covariate shift , where the learner's goal is to adaptively update the model to minimize the risk accumulated over time (see the formal definition in Section 3). To achieve this goal, we propose the ACCOUS approach (Adapt to Continuous Covariate shift with Unlabeled Stream) equipped with sound theoretical guarantees. Our approach is based on the classic importance-weighting framework yet requires innovations to make it applicable to the continuous shift scenario. Indeed, we theoretically identify that the importance-weighting works effectively if the cumulative estimation error of the time-varying density ratios of train and test inputs can be reduced. However, at each time step, a direct application of one-step density ratio estimation would lead to a high variance due to data scarcity; whereas reusing all previous data can be highly biased when testing distributions change dramatically. Thus, it is crucial to design an accurate estimation of time-varying test-train density ratios by appropriately reusing historical information.

To combat the difficulty, we propose a generic reduction of the time-varying density ratio estimation problem to the online convex optimization [13]: one can immediately obtain high-quality time-varying density ratios by a suitable online process to optimize its dynamic regret over a certain sequence of loss functions. Our reduction is based on the Bregman divergence density ratio matching framework [14] and applicable to various existing density ratio estimation models with specific configurations of the divergence functions. To minimize the dynamic regret, our key algorithmic ingredient of this online optimization is the online ensemble structure [15, 16], where a group of base-learners is maintained to perform density ratio estimation with different lengths of historical data, and a meta-algorithm is employed to combine the predictions. As such, we can properly reuse historical information without knowing the cumulative intensity of the covariate shift. We further instantiate the reduction framework with the logistic regression model [17]. Letting V T = ∑ T t =2 ‖D t ( x ) -D t -1 ( x ) ‖ 1 , we prove an ˜ O ( T 1 / 3 V 2 / 3 T ) dynamic regret bound for the density ratio estimator, which finally leads to an ˜ O ( T -1 / 3 V 1 / 3 T ) averaged excess risk of the predictor trained by importance-weighted learning. The rate can hardly be improved even if one receives labels of the testing stream after prediction (see more elaborations below Theorem 3 and Appendix D.6). Finally, we conduct experiments to evaluate our approach, and the empirical results validate the theoretical findings.

Technical Contributions. We note that online ensemble was also employed by [18] to handle continuous label shift , another typical distribution shift assuming the change happens on the class prior D t ( y ) . However, their method crucially relies on the unbiasedness of weights in the risk estimator, whereas density ratio estimators for covariate shift adaptation cannot satisfy the condition in general. Instead of pursuing unbiasedness, we delicately design an online optimization process to learn the density ratio estimator, which is versatile to be implemented with various models. Our methodology is very general and might be of broader use, such as relaxing the unbiasedness requirement in the continuous label shift. Besides, the ˜ O ( T 1 / 3 V 2 / 3 T ) dynamic regret bound for the logistic regression density ratio estimator is obtained non-trivially. Our bound essentially holds for online learning with exp-concave functions under noisy feedback. The only one achieving this [19] holds in expectation and requires complicated analysis on the Karush-Kuhn-Tucker (KKT) condition [20] of comparators. By contrast, our bound holds in high probability, and the analysis is greatly simplified by virtue of exploiting the structure of comparators in our problem that they are essentially the minimizers of expected functions (hence without analyzing the KKT condition).

2 Preliminaries

This section introduces the preliminaries and related work on the continuous covariate shift adaptation. We discuss the related work on non-stationary online learning in Appendix B

One-Step Covariate Shift Adaptation. Let D 0 ( x , y ) and D 1 ( x , y ) be the training and test distributions. The 'one-step' adaptation problem studies how to minimize the testing risk R 1 ( w ) = E ( x ,y ) ∼D 1 ( x ,y ) [ /lscript ( w /latticetop x , y )] by the model trained with a labeled dataset S 0 = { x i , y i } N 0 i =1 sampled from D 0 ( x , y ) and unlabeled dataset S 1 = { x i } N 1 i =1 sampled from D 1 ( x ) . We call this one-step adaptation since the test distribution is fixed and a reasonable number of unlabeled data S 1 is available.

Importance-Weighted ERM. A classic solution for the one-step covariate shift adaptation is the importance-weighted empirical risk minimization (IWERM), which mitigates the distribution shift

by minimizing the weighted empirical risk ˜ R 1 ( w ) = E ( x ,y ) ∼ S 0 [ r ∗ 1 ( x ) /lscript ( w /latticetop x , y )] , where r ∗ 1 ( x ) = D 1 ( x ) / D 0 ( x ) is the importance weight. The risk ˜ R 1 ( w ) is unbiased to R 1 ( w ) and thus the learned model is consistent over the test distribution [5]. The importance weighted learning was studied from the lens of variance-bias trade off [5, 21], cross validation [22], model misspecification [23], and deep neural network implementation [24]. All those are conducted under the one-step adaptation scenario.

Density Ratio Estimation. The importance weighted estimation, also known as the density ratio estimation (DRE) [12], aims to estimate the density ratio r ∗ 1 ( x ) using datasets input of S 0 and S 1 . Various methods were proposed with different statistical models [14, 17, 25, 26, 27, 28, 29]. All those methods focused on the one-step adaptation, and it is challenging to extend them to the continuous shift due to the limited unlabeled data at each time step. The work [30] studied how to update the density ratio estimator with streaming data, but the ground-truth density ratio is assumed to be fixed. Recently, the time-varying density ratio was investigated in [31]. Their problem setup fundamentally differs from ours as they assume sufficient online data at each iteration, while our challenge is dealing with limited data per iteration. The work [32] proposed to learn the time-varying density ratios using all past data at each time. Although learning theory insights are provided in the paper, it is still unclear how the learned density ratios balance sample complexity with environmental shift intensity. In contrast, our density ratio estimator updates in an online fashion with dynamic regret guarantees.

Continuous Distribution Shift. For broader continuous distribution shift problems, the study [33] focused on the label shift case and provided the first feasible solution. Then, the work [18] introduced modern non-stationary online learning techniques to the problem, developing the first method with dynamic regret guarantees. As shown in Remark 2, our solution refines the previous methods [33, 18] by decoupling density ratio estimation from predictor training, paving the way for a theoreticallygrounded method for continuous covariate shift. Similarly, the subsequent work [34] developed a method for continuous label shift, which allows for training the predictor with various models by separately estimating the label probability through an online regression oracle. Another research [35] studied the change detection for continuous covariate shift. However, they only focused on identifying differences between the current and initial offline distributions, and it is still unclear how to update the model adaptively with the online data. Moreover, the method [35] can only detect at a given time granularity, while ours can perform the model update at each round.

3 Adapting to Continuous Covariate Shift

In this section, we formulate the problem setup of continuous covariate shift and then introduce our approach based on the IWERM framework and online density ratio estimation.

3.1 Problem Setup

There are two stages in continuous covariate shift. The first one is the offline initialization stage , where the learner can collect a reasonable number of label data S 0 = { x n , y n } N 0 n =1 from the initial distribution D 0 . Then, we come to the online adaptation stage , where the unlabeled testing data arrive sequentially, and the underlying distributions can continuously shift. Consider a T -round online adaptation. At each round t ∈ [ T ] ≜ { 1 , . . . , T } , the learner will receive an unlabeled dataset S t = { x n } N t n =1 sampled from the underlying distribution D t . Without loss of generality, we consider N t = 1 . We have the following continuous covariate shift condition.

Assumption 1 (Continuous Covariate Shift) . For all x ∈ X in the feature space, y ∈ Y in the label space, and any t ∈ [ T ] , we have

We note that there are emerging discussions on the necessity of covariate shift adaptation [7, 23]. When there are infinite number of training samples, a well-specified large and over-parametrized model can be trained to approximate D 0 ( y | x ) and there is no need to perform covariate shift adaptation. However, in the finite-sample cases, one would prefer to train a model with constraint complexity to ensure its generalization ability, which leads to model misspecification and covariate shift adaptation is indeed necessary. In this paper, we study the continuous covariate shift under a misspecified hypothesis space W , which generalizes the standard one-step covariate shift problem (see [6] and reference therein) to the continuous shift case. Our goal is to train a sequence of model

{ w t } T t =1 that are comparable with the best model w ∗ t ∈ arg min w ∈W R t ( w ) in the hypothesis space at each time. Consequently, we take the following average excess risk as the performance measure:

We end this part by listing several common notations used throughout the paper. We denote by R = max x ∈X ‖ x ‖ 2 the maximum norm of the input and by D = max w 1 , w 2 ∈W ‖ w 1 -w 2 ‖ 2 the diameter of the hypothesis space W ⊆ R d . The constant G = max x ∈X ,y ∈Y , w ∈W ‖∇ /lscript ( w /latticetop x , y ) ‖ 2 is the maximum gradient norm and L = max x ∈X ,y ∈Y , w ∈W | /lscript ( w /latticetop x , y ) | is the upper bound of the loss values. We use the ˜ O ( · ) -notation to hide dependence on logarithmic factors of T .

3.2 Importance Weighted ERM for Continuous Shift

Our algorithm is based on the importance weighted learning. Consider the scenario where the learner has a predefined ̂ r t : X → [0 , B ] as an estimation for the true density ratio r ∗ t ( x ) = D t ( x ) / D 0 ( x ) for each time t ∈ [ T ] . Then, the predictor can be trained by the importance weighted ERM method:

The averaged excess risk of the predictor is closely related to the quality of the density ratio estimator. Proposition 1. For any δ ∈ (0 , 1] , with probability at least 1 -δ , IWERM (2) with the estimator ̂ r t ( x ) ensures R T ( { ̂ w t } T t =1 ) ≤ 2 ∑ T t =1 E x ∼ S 0 [ | ̂ r t ( x ) -r ∗ t ( x ) | ] /T + O (log( T/δ ) / √ N 0 ) .

In above, the O (log T/ √ N 0 ) term measures the generalization gap of the predictor since it is trained over the empirical data S 0 instead of D 0 . Such a rate is tight up to a logarithmic factor in T . Indeed, considering a stationary environment (i.e., D t ( x , y ) = D 1 ( x , y ) ) and an exact density ratio estimation (i.e., ̂ r t ( x ) = r ∗ t ( x ) ), Proposition 1 indicates that the averaged model w T = ∑ T t =1 ̂ w t /T enjoys an excess risk bound of R 1 ( w T ) -R 1 ( w ∗ 1 ) ≤ O (log T/ √ N 0 ) , matching the lower bound for importance weighted learning [36, Proposition 2] up to an O (log T ) factor.

Our main focus lies on the ∑ T t =1 E x ∼ S 0 [ | ̂ r t ( x ) -r ∗ t ( x ) | ] /T term, which is the averaged estimation error of the density ratio estimator ̂ r t to the time-varying ground truth r ∗ t . To minimize the estimation error, a fundamental challenge comes from the unknown non-stationarity exhibited in the online environments - how to select a right amount of historical data to reuse for each iteration? Intuitively, if the environments change slowly, it would be preferable to reuse all historical data to construct the density ratio estimator ̂ r t . However, when distribution shifts occur frequently, earlier data could be useless even potentially harmful. In such scenarios, training the density ratio estimator with the most recent data would be a more rational strategy. Furthermore, even if we could capture a roughly stationary period, it remains unclear how to update the density ratio estimator in an online manner with guarantees. To our knowledge, online density ratio estimation is underexplored in the literature, even for the stationary testing streaming, not to mention the more challenging non-stationary setup.

3.3 Online Density Ratio Estimation

Here, we present a generic reduction of the online density ratio estimation problem to a dynamic regret minimization problem. This novel perspective paves our way for designing an algorithm in tackling the non-stationarity of environments, as mentioned in Section 3.2.

Bregman Divergence Density Ratio Matching. Our reduction is based on the Bregman divergence density ratio matching [14], a general framework that unifies various existing DRE methods. Specifically, in the framework, the discrepancy between the ground-truth density ratio function r ∗ t and any density ratio function r is measured by the expected Bregman divergence over D 0 ( x ) defined by

where B ψ ( a ‖ b ) ≜ ψ ( a ) -ψ ( b ) -∂ψ ( b )( a -b ) is the Bregman divergence and ψ : dom ψ → R is the associated divergence function. The expected Bregman divergence can be equally rewritten as EB ψ ( r ∗ t ‖ r ) = L ψ t ( r ) -L ψ t ( r ∗ t ) , where L ψ t is the loss for the density ratio function defined by

As a consequence, one can train the density ratio estimator by minimizing the loss L ψ t ( r ) . We have r ∗ t ∈ arg min r ∈H L ψ t ( r ) when H is the set of all measurable functions.

The Bregman divergence-based density ratio matching framework takes various DRE methods as special cases. For instance, choosing the divergence function as ψ LS ( t ) = ( t -1) 2 / 2 yields LSIF [28] and KMM [26]; choosing ψ LR ( t ) = t log t -( t + 1) log( t + 1) leads to the logistic regression method [17]; and one can also recover the UKL [37] and KLLEP [25] with ψ KL ( t ) = t log t -t .

Online DRE via Dynamic Regret Minimization. For a single-round density ratio estimation, the Bregman divergence-based framework suggests to train the density ratio estimator ̂ r t by minimizing the gap L ψ t ( ̂ r t ) -L ψ t ( r ∗ t ) . Therefore, to derive an estimator sequence that performs well over time, we utilize the cumulative loss gap ∑ T t =1 L ψ t ( ̂ r t ) -∑ T t =1 L ψ t ( r ∗ t ) as a performance measure. Indeed, the cumulative loss gap serves as an upper bound of the estimation error of density ratio estimators.

Proposition 2. Let ψ be a µ -strongly convex function. For any density ratio estimator sequence { ̂ r t } T t =1 , we have 1 T ∑ T t =1 E x ∼D 0 [ | ̂ r t ( x ) -r ∗ t ( x ) | ] ≤ √ 2 ( ∑ T t =1 L ψ t ( ̂ r t ) -∑ T t =1 L ψ t ( r ∗ t ) ) / ( µT ) .

Proposition 2 indicates that it suffices to optimize the cumulative loss gap to perform online density ratio estimation. Such a measure quantifies the performance difference between the online algorithm (that yields the estimated ratio sequence { ̂ r t } T t =1 ) and a sequence of time-varying comparators (the true ratio sequence { r ∗ t } T t =1 ). This is exactly the dynamic regret in online learning literature [38, 39]. As such, we have reduced the online density ratio estimation to a problem of dynamic regret minimization over loss functions { L ψ t } T t =1 against comparators { r ∗ t } T t =1 .

Since the loss function involves the expectation over the underlying distribution D 0 ( x ) (see the definition in (4)), we need a counterpart result with respect to empirical data S 0 .

Theorem 1. Let ψ be a µ -strongly convex function satisfying t∂ 3 ψ ( t ) ≤ 0 and ∂ 3 ψ ( t ) ≤ 0 for all t ∈ dom ψ . Let H θ = { x ↦→ h ( x , θ ) | θ ∈ Θ } be a hypothesis space of density ratio functions parameterized by a finite-dimensional bounded set Θ ≜ { θ ∈ R d | ‖ θ ‖ 2 ≤ S } with a certain link function h : X × Θ ↦→ R . Denote by [ z ] + ≜ max { z, 0 } . Then, for any density ratio estimator ̂ r t ∈ H θ , the empirical estimation error is bounded by

provided that h ( x , θ ) is bounded for any x ∈ X and θ ∈ Θ and Lipschitz continuous. In the above,

is the empirical approximation of the expected loss L ψ t using the data S 0 .

Remark 1 (assumptions on ψ ) . We imposed certain assumptions on the divergence function ψ in Theorem 1. One can check these conditions hold for commonly used ψ in density ratio estimation, including all the divergence functions mentioned earlier ( ψ LR and ψ KL are strongly convex when the inputs are upper bounded, which can be satisfied with suitable choices of H θ ). Section 4 will present an example with ψ LR to show how the conditions are satisfied. /triangleleft

Theorem 1 shows that we can immediately obtain a sequence of high-quality density ratio estimators { r ∗ t } T t =1 by minimizing the dynamic regret with respect to { ˜ L ψ t } T t =1 ,

One caveat is that the second term of ˜ L ψ t in the definition (5) requires the knowledge of underlying distribution D t , which is unavailable. Empirically, we can only observe ̂ L ψ t defined below, building upon the empirical observations S t ∼ D t ,

That said, we need to design an online optimization process to minimize the dynamic regret (6) defined over ˜ L ψ t based on the observed loss { ̂ L ψ t } T t =1 . Since the time-varying comparator r ∗ t in (6)

is not the minimizer of the observed loss ̂ L ψ t (but rather the minimizer of the expected loss L ψ t defined in (4)), directly minimizing the empirical loss ̂ L ψ t will lead to a high estimation error. In the next section, we introduce how to optimize the dynamic regret with the online ensemble framework developed in recent studies of non-stationary online convex optimization [15, 16, 19].

Remark 2 (comparison with previous work) . For the continuous label shift [18], the online ensemble framework was employed to train the predictor w t . However, the previous method crucially relies on the construction of an unbiased importance ratio estimator satisfying E D t [ ̂ r t ( x )] = r ∗ t ( x ) . Such a favorable property is hard to be satisfied in the covariate shift case. For example, in the Bregman divergence density ratio matching framework, one can only observe an unbiased loss ̂ L ψ t ( r ) , whose minimizer ̂ r t = arg min r ̂ L ψ t ( r ) is not unbiased to the true value r ∗ t in general. To this end, we disentangle the model training and importance weight estimation process in this paper. Our reduction holds for the general Bergman divergence matching framework and thus can be initiated with different models (not necessarily unbiased). It is possible to extend our framework to continuous label shift problem with other importance weight estimators besides the unbiased one [40] used in [18]. /triangleleft

4 Instantiation: Logistic Regression Model

When the loss function ̂ L ψ t is non-convex, it is generally intractable to conduct the online optimization, regardless of minimizing the standard regret or the strengthened dynamic regret. Fortunately, the attained loss functions are convex or enjoy even stronger curvature with the properly chosen hypothesis space and divergence function. In this section, we instantiate our framework with the logistic regression model. A corresponding online DRE method is presented with dynamic regret guarantees.

Example 1 (Logistic regression model) . Consider the function ψ = ψ LR ≜ t log t -( t +1)log( t +1) and the hypothesis space H LR θ = { x ↦→ exp ( -θ /latticetop φ ( x ) ) | ‖ θ ‖ 2 ≤ S } . Here, φ : X ↦→ R d represents a specific basis function with bounded norm ‖ φ ( x ) ‖ 2 ≤ R , which could be, for instance, the feature representation extractor from a deep neural network. Then, the loss function ̂ L ψ t ( θ ) as per (7) becomes

Let β = exp( SR ) . Then, the output of any density ratio function r ∈ H θ is bounded by [1 /β, β ] . It can be validated that ψ LR is 1 / ( β + β 2 ) -strongly convex and satisfies the condition ∂ 3 ψ LR ( z ) ≤ 0 required by Theorem 1. Besides, the logistic loss is a 1 / (1(1 + β )) -expconcave function and R 2 / 2 -smooth function , presenting a favorable function properties for online convex optimization [13].

We note that the other two divergence functions ψ LS and ψ KL also exhibit desirable function properties as ψ LR . When choosing the hypothesis space H θ = { x ↦→ θ /latticetop φ ( x ) | θ ∈ Θ } , the methods recover the the uLISF [28] and UKL [37] density ratio estimators equipped with the generalized linear model. Our analysis is also applicable in the two cases. More details are provided in Appendix D.7.

4.1 Density Ratio Estimation via Online Ensemble

In this part, we introduce our online ensemble method for online DRE with the logistic regression model. Since the logistic regression loss is exp-concave, [19] shows that one can employ the following-the-leading-history (FLH) method [41] to minimize the dynamic regret. However, a caveat is that the observed loss ̂ L t is only an empirical estimation of ˜ L t established on few online data S t . The result of [19] only implies an expected bound for our problem.

We twisted the FLH algorithm to achieve a high probability bound. As shown in Figure 1, our algorithm maintains multiple base-learners, each learning over different intervals of the time-horizon, and then employs a meta-learner to aggregate their predictions. We modify the meta-learner in FLH from Hedge [42] to Adapt-ML-Prod [43], which ensures that the meta-learner can track each baselearner on the associated time interval with high probability. This strategy allows us to selectively reuse the historical information to handle the non-stationary environments. We introduce ingredients of our algorithms as follows. A more detailed algorithm descriptions and comparison with the dynamic regret minimization literature in OCO can be found in Appendices D.1 and B respectively.

Base-learner. Our algorithm runs multiple base-learners that are active on different intervals of the time horizon. Let C = { I i = [ s i , e i ] ⊆ [ T ] } be a interval set. Each base-learner E i will only

t

=

1

t

=

2

t

=

3

t

=

4

t

=

5

t

=

6

t

=

7

t

=

8

t

=

9

. . .

. . .

. . .

. . .

:

̂

θ

t

=

∑

i

∈A

t

p

t,i

̂

θ

t,i

update

p

t,i

by (9)

meta-learner

update

̂

θ

t,i

by (8)

active base-learner

update and submit the model ̂ θ t,i to the meta-learner if t ∈ I i . We use the online Newton step (ONS) for the model update. For an exp-concave loss function, the method ensures the base-learner's model is comparable with the best fixed model over the interval up to a logarithmic factor in T [44]. Specifically, for each base-learner E i running on the interval [ s i , e i ] , ONS updates the model by

where the matrix A t,i = λI + ∑ t τ = s i ∇ ̂ L τ ( ̂ θ τ,i ) ∇ ̂ L s ( ̂ θ τ,i ) /latticetop . In the above, the projection function is defined as Π A t,i Θ [ θ 1 ] = arg min θ ∈ Θ ‖ θ -θ 1 ‖ A t,i and Θ = { θ ∈ R d | ‖ θ ‖ 2 ≤ S } is the parameter space. The constant λ > 0 and η > 0 are the regularizer parameter and step size to be specified latter.

Meta-learner. As shown by Figure 1, the meta-learner's role is to aggregate the models produced by the base-learners using a weighting scheme. At each iteration t , the meta-learner will maintain a weight p t,i for each 'active' base-learner E i , defined as the one whose associated interval contains t . Weupdate the weights for active base-learners based on the Adapt-ML-Prod method [43]. Specifically, for every active base-learner E i , our algorithm maintains a 'potential' v t,i ∈ R + at iteration t , which reflects the historical performance of the base-leaner until time t . Then, denoting by A t the index set of the active base-learners at time t , their weights and the output model are obtained by

In the above ε t,i > 0 is a step size that can be automatically tuned along the learning process. A noteworthy ingredient of our algorithm is the construction of the potential v t,i , using the linearlized loss 〈∇ ̂ L t ( ̂ θ t ) , ̂ θ t -̂ θ t,i 〉 / ( SR ) . Then, the generalization gap between ̂ L t and ˜ L t can be controlled with the negative term introduced by the exp-concave loss function. As a result, we can establish a high probability bound to ensure that ̂ θ t is competitive with any base-learner's model ̂ θ t,i on the corresponding interval I i . The detailed configurations of v t,i and ε t,i are deferred to Appendix D.1.

Schedule of Intervals. As shown in Figure 1, we specify the intervals with the geometric covering scheme [41], where the interval set is defined as C = ⋃ k ∈ N ∪{ 0 } C k with C k = { [ i · 2 k , ( i +1) · 2 k -1] | i ∈ N and i · 2 k ≤ T } . One can check that |C| is at most T , and the number of active base-learner is bounded as |A t | ≤ /ceilingleft log t /ceilingright . Thus, we only need to maintain at most O (log t ) base-learners at time t . The intervals specified by the geometric covering are informative to capture the non-stationarity of the environments, leading to the following dynamic regret guarantees.

4.2 Theoretical Guarantees

This part presents the theoretical guarantees of our method. The estimator ̂ r t ( x ) = exp( -φ ( x ) /latticetop ̂ θ t ) established on ̂ θ t returned by the online ensemble method (9) achieves the dynamic regret guarantee. Theorem 2. Assume the true density ratio r ∗ t ( x ) = D t ( x ) / D 0 ( x ) is contained in the hypothesis space as r ∗ t ∈ H LR θ ≜ { x ↦→ exp( -φ ( x ) /latticetop θ ) | θ ∈ Θ } for any t ∈ [ T ] . Then, with probability at least 1 -δ , the dynamic regret of the density ratio estimator ̂ r t ( x ) = exp( -φ ( x ) /latticetop ̂ θ t ) is bounded by

when the parameters are set as γ = 3(1 + β ) and λ = 1 . In the above, V T = ∑ T t =2 ‖D t ( x ) -D t -1 ( x ) ‖ 1 measures the variation of input densities.

Theorem 2 imposes a realizable assumption for the true density ratio function such that r ∗ t ∈ H LR θ . Such an assumption is required since we train the density ratio estimator on a given hypothesis space while the true density ratio function could be arbitrary. We note that the realizability assumption is frequently used in the analysis for the density ratio estimation problem [29, 45] and other related topics, including active learning [46] and contextual bandit [47], where the density ratio (or density) estimation is required. A possible direction to relax the assumption is to consider a richer function class, e.g., a neural network or a non-parametric model. We leave this as a future work.

The ˜ O ( T 1 3 V 2 3 T ) rate in our bound exhibits the same rate as the minimax optimal dynamic regret bound for the squared loss function with the noisy feedback [48]. Since the squared loss function enjoys even stronger curvature than the logistic loss, our result is hard to be improved. To achieve this fast-rate result, the key was to show a 'squared' formulation O ( max { 1 , | I | V 2 I } ) of the dynamic regret for the base-leaner on each interval I (Lemma 3 in Appendix D.2). [19] achieved this by a complicated analysis with the KKT condition to capture the structure of the comparators, while we greatly simplified the analysis by exploiting the structure that the comparator θ ∗ t is essentially the minimizers of expected functions L t (hence avoiding analyzing the KKT condition). Our analysis is applicable to the case where the minimiers lie in the interior of the decision set and only requires the smoothness of the loss function, which can be of independent interest in online convex optimization.

Averaged Excess Risk Bound for Continuous Covariate Shift. After obtaining the density ratio estimator, we can train the predictive model by IWERM (2), leading to the following guarantee.

Theorem 3. Under the same condition as Theorem 2 and letting V T = ∑ T t =2 ‖D t ( x ) -D t -1 ( x ) ‖ 1 , running IWERM (2) with the estimated density ratio function ̂ r t ( x ) = exp( -φ ( x ) /latticetop ̂ θ t ) yields

Theorem 3 shows that our learned predictor ̂ w t adapts to the environment with a converged average excess risk when compared with the per-round best predictor w ∗ t . By the discussion below Proposition 1, the ˜ O ( N -1 / 2 0 ) generalization error over the offline data S 0 is hard to improve. We focus on the error in the online learning part. When the environment is nearly-stationary, i.e., V T ≤ O ( T -1 / 2 ) , our bound implies an ˜ O ( N -1 / 2 0 + T -1 / 2 ) average excess risk, matching the same rate for the one-step adaptation [49], even if the unlabeled data appear sequentially and the comparator w ∗ t could change over time. When the environments shift quickly, the O ( T -1 / 3 V 1 / 3 T ) rate still exhibits a diminishing error for covariate shift adaptation once V T = o ( T ) . Indeed, our result has the same rate as that for continuous label shift [18] (with a slightly different definition of V T though). In Appendix D.6, we further provide evidence to show that the O (max { T -1 / 3 V 1 / 3 T , T -1 / 2 ) rate can hardly be improved, even if one can receive labels of the testing stream after prediction.

Discussion on Assumptions. We end this section by a discussion on possible future directions to relax the linear model assumption for DRE and covariate shift assumption. Since our analysis is based on the online convex optimization framework, we employed the (generalized) linear model for DRE to ensure the convexity. To go beyond the linear model while still having theoretical guarantees, one might extend the online ensemble framework to learn within the Reproducing Kernel Hilbert Space, leveraging advances in online kernel learning [50]. As for the step towards handling the general distribution shift, it is possible to consider the sparse joint shift model [51] where both covariate and label distribution could shift. Besides, studying how to handle the joint distribution shift with few online labeled data also presents an interesting future direction. Our research on the covariate shift might serve as a basic step towards addressing more complex real-world distribution shifts.

5 Experiments

Setups. We generate continuous covariate shift by combining two fixed distributions with a timevarying mixture proportion α t ∈ [0 , 1] . Specifically, given two fixed distributions D ′ ( x ) and D ′′ ( x ) , we generate samples from D t ( x ) = (1 -α t ) D ′ ( x )+ α t D ′′ ( x ) at each round. The mixture proportion α t shifts in four patterns: in Sin Shift and Squ Shift, α t changes periodically, following sine and square waves; in Lin Shift, the environment changes slowly from α 1 = 1 to α T = 0 linearly over T rounds, while in Ber Shift, the proportion α t flips quickly between 0 and 1 with a certain probability. For parameterization in the Accous implementation, we set R by directly calculating the data norm

Squ

Sin

Ber

25

30

35

40

Average Error (%)

0

1

2

3

4

5

6

7

8

9 10 11 12 13 14

Interval length (|

I

i

| = 2

i

)

16

32

64

128

256

512

Distribution Change Interval (

M

)

8.4

8.9

9.7 10.914.4 8.4

5.8

1.4

7.1

4.8

5.1

4.7

3.5

2.8

4.1

7.1

7.4

8.0

9.1 11.622.010.3 6.9

3.2

4.0

3.0

0.9

0.8

3.8

1.9

6.6

6.8

7.1

7.7

8.3 13.521.7 1.8

0.6

0.1

5.9

7.6

2.4

1.8

8.1

6.3

6.4

6.5

6.7

7.1

8.2 10.918.8 4.2

4.4

4.4

4.3

4.1

4.2

3.4

5.3

5.4

4.5

4.7

5.9

6.5

8.5 13.820.9 4.4

4.3

4.4

3.3

4.9

3.3

6.2

5.3

5.6

5.7

5.9

6.5

7.6

8.5 11.521.8 5.5

2.7

3.4

3.1

2.5

2.5

5.0

7.5

10.0

12.5

15.0

17.5

20.0

100

200

300

400

500

600

700

800

900

Time

0.2

0.25

0.3

0.35

0.4

0.45

0.5

0.55

0.6

Average Error (%)

0

0.2

0.4

0.6

0.8

1

Estimator Error

and set S = d/ 2 for all experiments. We repeat all experiments five times and evaluate the algorithms by the average classification error over the unlabeled test stream for 10,000 rounds.

Contenders. We compare our method with six algorithms, which can be divided into three groups. The first is a baseline approach that predicts directly using the model trained on initial offline data ( FIX ). The second group consists of the one-step covariate shift adaptation methods that do not reuse historical information: DANN [52] handles the shifts by learning an invariant feature space, IWKMM [26], IW-KEIEP [25] and IW-uLSIF [28] equip the one-step IWERM method (2) with different density ratio estimators. The third is OLRE , which serves as an ablation study for our Accous algorithm. The OLRE algorithm estimates the density ratio by running an ONS (8) with all historical data and then performs the IWERM at each round. All algorithm parameters are set by their default.

In the following, we focus on the results of empirical studies. Detailed configurations for the datasets, simulated shifts, and setups for the Accous algorithm and contenders can be found in Appendix A.

5.1 Illustrations on Synthetic Data

Average Classification Error Comparison. We summarize the comparison results on synthetic data with different types of covariate shifts with all contenders in Figure 2. The Accous algorithm outperforms almost all other methods in the four shift patterns. We observe that the offline model FIX performs poorly compared to the online methods. The best result of DANN, IW-KMM, IW-KLIEP and IW-uLSIF is called One-Step for comparison. By reusing historical data, the Accous algorithm achieves a lower average classification error compared to these One-Step competitors that use only one round of online data. The Accous also outperforms OLRE, especially when the environments change relatively quickly ( Squ , Sin , and Ber ). These results justify that performing density ratio estimation with either one round of online data or all historical data is inappropriate for non-stationary environments. Moreover, when the number of unlabeled data per round changes from N t = 5 to N t = 1 , the one-step methods suffer severe performance degradation. In contrast, our Accous algorithm still performs relatively well, highlighting the need for selective reuse of historical data.

Effectiveness of Online Density Ratio Estimation. We then take a closer look at the core component of our algorithm, the online density ratio estimator, in the Squ shift where the distribution of the online data shifts every M rounds. Figure 3 shows the weight assignment for base-learners with different interval lengths, averaged over their active period. The result shows that our meta-learner successfully assigns the largest weight to the base-learner whose interval length matches the switching period M , which ensures that the right amount of historical data is reused. Figure 4 also shows the average error of Accous and OLRE over 1 , 000 iterations. The results show that Accous (red line) can quickly adapt to new distributions after the covariate shift occurs. In contrast, OLRE (blue line) struggles because it uses all the historical data generated from different distributions. Furthermore, we denote the loss of the density ratio estimator ̂ L t ( ̂ θ t ) by the green line, which is quickly minimized after the covariate shift occurs. The similar tendency between the average error (red line) and the loss of the estimator (green line) confirms our theoretical finding in Theorem 1: we can minimize the excess risk of the predictor by minimizing the dynamic regret of the density ratio estimator.

5.2 Comparison on Real-world Data

Performance Comparison on Benchmarks. In Table 1 and Table 2, we present the results of the average classification error comparison with the contenders on four benchmark datasets. Our

algorithm Accous outperforms other contenders in the four types of shifts, whether in a slowly evolving environment ( Lin ) or in relatively non-stationary environments ( Squ , Sin , Ber ). The OLRE algorithm uses all historical data, but does not necessarily outperform contenders that use only current data, suggesting that we should selectively reuse the historical data.

Case Study on a Real-life Application. We conducted empirical studies on a real-world application using the yearbook dataset [53], which contains high school front-facing yearbook photos from 1930 to 2013. The photo distributions shift along years due to the changing social norms, fashion styles, and population demographics. Consequently, it is essential to adapt to these changes. We used photos before 1945 as the offline labeled dataset and generated the online unlabeled data stream using photos after 1945, arranged chronologically with 10 photos per round. Figure 5 shows the average error curves. The OLER algorithm performs well when the distributions do not change too much, but the method starts to suffer from large prediction errors after 400 iterations. In contrast, our method outperforms all competitors by selectively reusing historical data.

6 Conclusion

This paper initiated the study of the continuous covariate shift with the goal of minimizing the cumulative excess risk of the predictors. We showed that the importance-weighted ERM method works effectively given high-quality density ratio estimators (Proposition 1), whose estimation can be cast as a dynamic regret minimization problem (Theorem 1). Instantiating with the logistic regression density ratio estimation model, we proposed a novel online that can adaptively reuse the historical data and enjoy a tight dynamic regret bound (Theorem 2). The regret bound finally implies an ˜ O ( T -1 / 3 V 1 / 3 T ) averaged excess risk guarantee for the predictor. Experiments validate the effectiveness of the proposed method and demonstrate the need for selective reuse of historical data.

Acknowledgments

YJZ and MS were supported by the Institute for AI and Beyond, UTokyo. YJZ was also supported by Todai Fellowship. Peng Zhao was supported by NSFC (62206125) and JiangsuSF (BK20220776). The authors thank Yong Bai and Yu-Yang Qian for their assistance in the experiments.

100

200

300

400

500

600

700

800

900

Time

0.25

0.3

0.35

0.4

0.45

Average Error (%)

References

Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset Shift in Machine Learning . MIT Press, 2008.

Yoshua Bengio, Yann Lecun, and Geoffrey Hinton. Deep learning for AI. Communications of the ACM , 64(7):58-65, 2021.

Thomas G. Dietterich. Steps toward robust artificial intelligence. AI Magazine , 38(3):3-24, 2017.

Zhi-Hua Zhou. Open-environment machine learning. National Science Review , 9(8), 07 2022. nwac123.

Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference , 90(2):227-244, 2000.

Masashi Sugiyama and Motoaki Kawanabe. Machine Learning in Non-Stationary Environments - Introduction to Covariate Shift Adaptation . MIT Press, 2012.

Samory Kpotufe and Guillaume Martinet. Marginal singularity and the benefits of labels in covariate-shift. The Annals of Statistics , 49(6):3299-3323, 2021.

Yan Li, Hiroyuki Kambara, Yasuharu Koike, and Masashi Sugiyama. Application of covariate shift adaptation techniques in brain-computer interfaces. IEEE Transactions on Biomedical Engineering , 57(6):1318-1324, 2010.

Makoto Yamada, Masashi Sugiyama, and Tomoko Matsui. Semi-supervised speaker identification under covariate shift. Signal Processing , 90(8):2353-2361, 2010.

Ali Hassan, Robert I. Damper, and Mahesan Niranjan. On acoustic emotion recognition: Compensating for covariate shift. IEEE Transactions on Audio, Speech and Language Processing , 21(7):1458-1468, 2013.

Jiayuan Huang, Alexander J. Smola, Arthur Gretton, Karsten M. Borgwardt, and Bernhard Schölkopf. Correcting sample selection bias by unlabeled data. In Advances in Neural Information Processing Systems 19 (NIPS) , pages 601-608, 2006.

Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density Ratio Estimation in Machine Learning . Cambridge University Press, 2012.

Elad Hazan. Introduction to Online Convex Optimization. Foundations and Trends in Optimization , 2(3-4):157-325, 2016.

Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density-ratio matching under the Bregman divergence: a unified framework of density-ratio estimation. Annals of the Institute of Statistical Mathematics , 64(5):1009-1044, 2012.

Lijun Zhang, Shiyin Lu, and Zhi-Hua Zhou. Adaptive online learning in dynamic environments. In Advances in Neural Information Processing Systems 31 (NeurIPS) , pages 1330-1340, 2018.

Peng Zhao, Yu-Jie Zhang, Lijun Zhang, and Zhi-Hua Zhou. Dynamic regret of convex and smooth functions. In Advances in Neural Information Processing Systems 33 (NeurIPS) , pages 12510-12520, 2020.

Steffen Bickel, Michael Brückner, and Tobias Scheffer. Discriminative learning under covariate shift. Journal of Machine Learning Research , 10:2137-2155, 2009.

Yong Bai, Yu-Jie Zhang, Peng Zhao, Masashi Sugiyama, and Zhi-Hua Zhou. Adapting to online label shift with provable guarantees. In Advances in Neural Information Processing Systems 35 (NeurIPS) , pages 29960-29974, 2022.

Dheeraj Baby and Yu-Xiang Wang. Optimal dynamic regret in exp-concave online learning. In Proceedings of the 34th Conference on Learning Theory (COLT) , pages 359-409, 2021.

Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex Optimization . Cambridge University Press, 2004.

Makoto Yamada, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, and Masashi Sugiyama. Relative density-ratio estimation for robust distribution comparison. Neural Computation , 25(5):1324-1370, 2013.

Masashi Sugiyama, Matthias Krauledat, and Klaus-Robert Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research , 8:985-1005, 2007.

Junfeng Wen, Chun-Nam Yu, and Russell Greiner. Robust learning under uncertain test distributions: Relating covariate shift to model misspecification. In Proceedings of the 31th International Conference on Machine Learning (ICML) , pages 631-639, 2014.

Tongtong Fang, Nan Lu, Gang Niu, and Masashi Sugiyama. Rethinking importance weighting for deep learning under distribution shift. In Advances in Neural Information Processing Systems 33 (NeurIPS) , page 11996-12007, 2020.

Masashi Sugiyama, Taiji Suzuki, Shinichi Nakajima, Hisashi Kashima, Paul von Bünau, and Motoaki Kawanabe. Direct importance estimation for covariate shift adaptation. Annals of the Institute of Statistical Mathematics , 60(4):699-746, 2008.

Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bernhard Schölkopf. Covariate shift by kernel mean matching. Dataset Shift in Machine Learning , 3(4):5, 2009.

Masashi Sugiyama. Superfast-trainable multi-class probabilistic classifier by least-squares posterior fitting. IEICE Transactions on Information and Systems , 93-D(10):2690-2701, 2010.

Takafumi Kanamori, Shohei Hido, and Masashi Sugiyama. A least-squares approach to direct importance estimation. Journal of Machine Learning Research , 10:1391-1445, 2009.

Masahiro Kato and Takeshi Teshima. Non-negative Bregman divergence minimization for deep direct density ratio estimation. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning (ICML) , pages 5320-5333, 2021.

M. C. du Plessis, H. Shiino, and M. Sugiyama. Online direct density-ratio estimation applied to inlier-based outlier detection. Neural Computation , 27(9):1899-1914, 2015.

Yu Chen, Song Liu, Tom Diethe, and Peter A. Flach. Continual density ratio estimation in an online setting. ArXiv preprint , arXiv:2103.05276, 2021.

Rasool Fakoor, Jonas Mueller, Zachary C. Lipton, Pratik Chaudhari, and Alexander J. Smola. Time-varying propensity score to bridge the gap between the past and present. ArXiv preprint , arXiv: 2210.01422, 2022.

Ruihan Wu, Chuan Guo, Yi Su, and Kilian Q. Weinberger. Online adaptation to label distribution shift. In Advances in Neural Information Processing Systems 34 (NeurIPS) , pages 11340-11351, 2021.

Dheeraj Baby, Saurabh Garg, Tzu-Ching Yen, Sivaraman Balakrishnan, Zachary Chase Lipton, and Yu-Xiang Wang. Online label shift: Optimal dynamic regret meets practical algorithms. ArXiv preprint , arXiv:2103.05276, 2023.

Sooyong Jang, Sangdon Park, Insup Lee, and Osbert Bastani. Sequential covariate shift detection using classifier two-sample tests. In Proceedings of the 39th International Conference on Machine Learning (ICML) , pages 9845-9880, 2022.

Corinna Cortes, Yishay Mansour, and Mehryar Mohri. Learning bounds for importance weighting. In Advances in Neural Information Processing Systems 23 (NIPS) , pages 442-450, 2010.

XuanLong Nguyen, Martin J. Wainwright, and Michael I. Jordan. Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization. In Advances in Neural Information Processing Systems 20 (NeurIPS) , pages 1089-1096, 2007.

Martin Zinkevich. Online convex programming and generalized infinitesimal gradient ascent. In Proceedings of the 20th International Conference on Machine Learning (ICML) , pages 928-936, 2003.

Omar Besbes, Yonatan Gur, and Assaf J. Zeevi. Non-stationary stochastic optimization. Operations Research , 63(5):1227-1244, 2015.

Zachary C. Lipton, Yu-Xiang Wang, and Alexander J. Smola. Detecting and correcting for label shift with black box predictors. In Proceedings of the 35th International Conference on Machine Learning (ICML) , pages 3128-3136, 2018.

Elad Hazan and C. Seshadhri. Efficient learning algorithms for changing environments. In Proceedings of the 26th International Conference on Machine Learning (ICML) , pages 393-400, 2009.

Yoav Freund and Robert E. Schapire. A decision-theoretic generalization of on-line learning and an application to boosting. Journal of Computer and System Sciences , 55(1):119-139, 1997.

Pierre Gaillard, Gilles Stoltz, and Tim van Erven. A second-order bound with excess losses. In Proceedings of The 27th Conference on Learning Theory (COLT) , pages 176-196, 2014.

Elad Hazan, Amit Agarwal, and Satyen Kale. Logarithmic regret algorithms for online convex optimization. Machine Learning , 69(2-3):169-192, 2007.

Takafumi Kanamori, Taiji Suzuki, and Masashi Sugiyama. Statistical analysis of kernel-based least-squares density-ratio estimation. Machine Learning , 86(3):335-367, 2012.

Akshay Krishnamurthy, Alekh Agarwal, Tzu-Kuo Huang, Hal Daumé III, and John Langford. Active learning for cost-sensitive classification. In Proceedings of the 34th International Conference on Machine Learning (ICML) , pages 1915-1924, 2017.

Dylan J. Foster and Alexander Rakhlin. Beyond UCB: optimal and efficient contextual bandits with regression oracles. In Proceedings of the 37th International Conference on Machine Learning (ICML) , pages 3199-3210, 2020.

Dheeraj Baby and Yu-Xiang Wang. Online forecasting of total-variation-bounded sequences. In Advances in Neural Information Processing Systems 32 (NeurIPS) , pages 11071-11081, 2019.

Yaoliang Yu and Csaba Szepesvári. Analysis of kernel mean matching under covariate shift. In Proceedings of the 29th International Conference on Machine Learning (ICML) , pages 1147-1154, 2012.

Daniele Calandriello, Alessandro Lazaric, and Michal Valko. Efficient second-order online kernel learning with adaptive embedding. In Advances in Neural Information Processing Systems 30 (NeurIPS) , pages 6140-6150, 2017.

Lingjiao Chen, Matei Zaharia, and James Y. Zou. Estimating and explaining model performance when both covariates and labels shift. In Advances in Neural Information Processing Systems 35 (NeurIPS) , pages 11467-11479, 2022.

Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor S. Lempitsky. Domain-adversarial training of neural networks. Journal of Machine Learning Research , 17:59:1-59:35, 2016.

Huaxiu Yao, Caroline Choi, Bochuan Cao, Yoonho Lee, Pang Wei Koh, and Chelsea Finn. Wild-time: A benchmark of in-the-wild distribution shift over time. In Advances in Neural Information Processing Systems 35 (NeurIPS) Datasets and Benchmarks Track , pages 1030910324, 2022.

Shiry Ginosar, Kate Rakelly, Sarah M Sachs, Brian Yin, Crystal Lee, Philipp Krähenbühl, and Alexei A Efros. A century of portraits: A visual historical record of american high school yearbooks. IEEE Transactions on Computational Imaging , 3(3):421-431, 2017.

András György and Csaba Szepesvári. Shifting regret, mirror descent, and matrices. In Proceedings of the 33nd International Conference on Machine Learning (ICML) , pages 29432951, 2016.

Eric C. Hall and Rebecca Willett. Dynamical models and tracking regret in online convex programming. In Proceedings of the 30th International Conference on Machine Learning (ICML) , pages 579-587, 2013.

Ali Jadbabaie, Alexander Rakhlin, Shahin Shahrampour, and Karthik Sridharan. Online optimization : Competing with dynamic comparators. In Proceedings of the 18th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 398-406, 2015.

Aryan Mokhtari, Shahin Shahrampour, Ali Jadbabaie, and Alejandro Ribeiro. Online optimization in dynamic environments: Improved regret rates for strongly convex problems. In Proceedings of the 55th IEEE Conference on Decision and Control (CDC) , pages 7195-7201, 2016.

Tianbao Yang, Lijun Zhang, Rong Jin, and Jinfeng Yi. Tracking slowly moving clairvoyant: Optimal dynamic regret of online learning with true and noisy gradient. In Proceedings of the 33rd International Conference on Machine Learning (ICML) , pages 449-457, 2016.

Lijun Zhang, Tianbao Yang, Rong Jin, and Zhi-Hua Zhou. Dynamic regret of strongly adaptive methods. In Proceedings of the 35th International Conference on Machine Learning (ICML) , pages 5877-5886, 2018.

Yu-Jie Zhang, Peng Zhao, and Zhi-Hua Zhou. A simple online algorithm for competing with dynamic comparators. In Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI) , pages 390-399, 2020.

Dheeraj Baby and Yu-Xiang Wang. Adaptive online estimation of piecewise polynomial trends. In Advances in Neural Information Processing Systems 33 (NeurIPS) , pages 20462-20472, 2020.

Peng Zhao and Lijun Zhang. Improved analysis for dynamic regret of strongly convex and smooth functions. In Proceedings of the 3rd Conference on Learning for Dynamics and Control (L4DC) , pages 48-59, 2021.

Zhi-Hua Zhou. Ensemble Methods: Foundations and Algorithms . Chapman & Hall/CRC Press, 2012.

Shai Shalev-Shwartz and Shai Ben-David. Understanding Machine Learning: From Theory to Algorithms . Cambridge University Press, 2014.

Gilles Pisier. The Volume of Convex Bodies and Banach Space Geometry , volume 94. Cambridge University Press, 1999.

Peng Zhao, Yu-Jie Zhang, Lijun Zhang, and Zhi-Hua Zhou. Adaptivity and non-stationarity: Problem-dependent dynamic regret for online convex optimization. ArXiv preprint , arXiv: 2002.07729, 2021.

Mehrdad Mahdavi, Lijun Zhang, and Rong Jin. Lower and upper bounds on the generalization of stochastic exponentially concave optimization. In Proceedings of The 28th Conference on Learning Theory (COLT) , pages 1305-1320, 2015.

Joel A. Tropp. An introduction to matrix concentration inequalities. Foundations and Trends in Machine Learning , 8(1-2):1-230, 2015.

Kwang-Sung Jun, Francesco Orabona, Stephen Wright, and Rebecca Willett. Improved strongly adaptive online learning using coin betting. In Proceedings of the 20th International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 943-951, 2017.

Yoav Freund, Robert E. Schapire, Yoram Singer, and Manfred K. Warmuth. Using and combining predictors that specialize. In Proceedings of the 29th Annual ACM Symposium on the Theory of Computing (STOC) , pages 334-343, 1997.

Nicolò Cesa-Bianchi and Gábor Lugosi. Prediction, Learning, and Games . Cambridge University Press, 2006.

Haipeng Luo and Robert E. Schapire. Achieving all with no parameters: AdaNormalHedge. In Proceedings of the 28th Annual Conference Computational Learning Theory (COLT) , pages 1286-1304, 2015.

Dmitry Adamskiy, Wouter M. Koolen, Alexey V. Chernov, and Vladimir Vovk. A closer look at adaptive regret. Journal of Machine Learning Research , 17:23:1-23:21, 2016.

Lijun Zhang, Shiyin Lu, and Tianbao Yang. Minimizing dynamic regret and adaptive regret simultaneously. In Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS) , pages 309-319, 2020.

A Omitted Details for Experiments

In this section, we first provide additional details that were omitted in section 5. We then provide the descriptions of our experimental setups, which include details of the datasets used, simulations of continuous covariate shifts, descriptions of the competing algorithms, and the parameter configuration of our proposed Accous algorithm. After that, we supplement the detailed numerical results obtained from the synthetic experiments.

A.1 Experiment Setup

Continuous Covariate Shift Simulation. To simulate the continuous covariate shift, we generate offline and online samples from different marginal distributions and annotate them with the same labeling function. Given two different marginal distributions D ′ ( x ) and D ′′ ( x ) , the offline dataset S 0 is sampled from the distribution D 0 ( x ) = (1 -α 0 ) D ′ ( x ) + α 0 D ′′ ( x ) , where α 0 ∈ [0 , 1] is a mixture proportion coefficient. The online dataset S t received at each iteration t is sampled from the distribution D t ( x ) = (1 -α t ) D ′ ( x ) + α t D ′′ ( x ) , where α t ∈ [0 , 1] is the time-varying coefficient. By changing the coefficient α t for t ≥ 1 , we can simulate different types of continuous covariate shifts with different shift patterns.

Linear Shift (Lin) : We set α t = t/T to simulate a gradually changing environment.

Square Shift (Squ) : In this case, the parameter α t switches between 1 and 0 every M rounds, where 2 M is the periodic length. We set M = Θ( √ T ) to simulate a fastchanging environment with periodic patterns.

Sine Shift (Sin) : We set α t = sin iπ M , where i = t mod M and M is the periodic length. We set M = Θ( √ T ) to simulate a fast-changing environment with periodic patterns.

Bernoulli Shift (Ber) : In this case, we keep the α t = α t -1 with probability p ∈ [0 , 1] and otherwise set α t = 1 -α t -1 . We set p = Θ(1 / √ T ) to simulate a fastchanging environment but without periodic patterns.

Datasets. For the synthetic dataset, we generate D ′ ( x ) and D ′′ ( x ) using the multinomial Gaussian distribution, wherein each class is generated from a Gaussian distribution. For D ′ ( x ) , the positive class data is generated from N ( x ; µ p 1 , Σ) , and the negative class data is generated from N ( x ; µ n 1 , Σ) . Here, µ p 1 is set to -1 . 2 in R 12 , and µ n 1 is set to -0 . 8 in R 12 . As for D ′′ ( x ) , its positive class data is generated from N ( x ; µ p 2 , Σ) , and the negative class data is generated from N ( x ; µ n 2 , Σ) . Here, µ p 2 is specified as 1 . 2 in R 12 , and µ n 2 as 0 . 8 in R 12 . The covariance matrix is set as Σ = I d in R 12 × 12 , and the prior probability for the positive class data is fixed at 0 . 5 . Besides, we generated offline data with α 0 = 0 . 9 and introduced a time-varying coefficient, α t , to simulate continuous covariate shifts.

The benchmark datasets are generated from real-world datasets.

Diabetes (Dia) is a tabular UCI dataset. Each sample is an 8 -dimensional vector with a binary label. We split the dataset into two subsets and choose the split such that the classifier trained on one dataset generalizes poorly to the other. We choose one of them as the offline dataset and combine them with a time-varying coefficient to generate the online data.

Breast (Bre) is a tabular UCI dataset. Each sample is a 9 -dimensional vector with a binary label. This dataset is generated similarly to the Diabetes dataset.

MNIST-SVHN (M-S) contains the digital images collected from handwriting or natural scene images. A feature extractor pre-trained on this dataset produces a 512 -dimensional vector with a 10 -class label for each sample. We choose 10% MNIST data and 90% SVHN data as the offline dataset and generate the online data by mixing the MNIST and SVHN datasets with a time-varying proportion coefficient α t .

CIFAR10-CINIC10 (C-C) contains real-life subjective images collected from different sources. This dataset is generated similarly to the MNIST-SVHN dataset.

We also conduct experiments on a real-world dataset where the underlying data distributions exhibit a continuous shift.

Yearbook [53]: This dataset contains the 37,921 frontal-facing American high school yearbook photos from 1930 to 2013 [54]. Each photo is a 32 × 32 × 1 grayscale image associated with a binary label representing the student's gender. This dataset captures the real-life changing of social norms, fashion styles, and population demographics. The offline data is generated with photos before 1945, the online data is generated from 1946, and the data size is 10 per round.

Contenders. We compare our proposed approach with six contenders, including:

FIX is the baseline method that predicts with the classifier that are pre-trained on offline data without any updates in the online adaptation stage.

is a general domain adaptation algorithm that mitigates the discrepancy between the training and test distributions by learning an invariant feature representation. We adapt this method to the online setting by running DANN at each round t ∈ [ T ] based on the offline data S 0 and the online data S t available at that round.

IW-KMM is an importance weighting learning method that first estimates the density ratios by the Kernel Mean Matching (KMM) method [26] and then performs the IWERM to train the model. At each iteration t ∈ [ T ] , the density ratios are estimated with the offline dataset S 0 and the online dataset S t by the KMM. Then the model is trained by running the IWERM method on the offline data S 0 .

IW-KLIEP is also an importance weighting learning method, where the density ratios are estimated by the KLIEP method [25]. At each iteration t ∈ [ T ] , after the density ratios have been estimated by KLIEP, the learner performs the IWERM over the offline data S 0 .

IW-uLSIF is an importance weighting learning method where the density ratio estimated by the uLSIF method is used [28]. At each iteration t ∈ [ T ] , the learner first estimates the density ratios by the uLSIF and then performs the IWERM on the offline data S 0 .

LR serves as an ablation study for the proposed Accous algorithm. At each iteration t ∈ [ T ] , the learner first estimates the density ratios by an ONS with a logistic regression model and then performs the IWERM on the offline data S 0 .

OLRE also serves as an ablation study for the proposed Accous algorithm. It estimates the density ratios in an on-line manner by running an ONS with a logistic regression model with all the historical data and then performing the IWERM each round.

Accous Setup. We implement the Accous algorithm using a logistic regression model. Before running the Accous algorithm, two parameters must be defined. They are the maximum norm of the feature R and the maximum norm of the density functions S . We set R by directly calculating the data norm. The covariate shift level determines the parameter S , which is difficult to predict in advance. We found that setting S = d/ 2 is a good choice in the experiments. For stable performance, we set a threshold of 100 and truncate the large weights after the density ratio estimation in each round. We also ignore the base models with interval lengths 1 , 2 in the density ratio estimation.

For the synthetic, diabetes, and breast tabular datasets, the linear model is used. For the MNISTSVHN, CIFAR10-CINIC10, and Yearbook datasets, the deep model is used for density ratio estimation and predictive model training. The deep models consist of a backbone and a linear layer. In all experiments, only the linear layer of the neural network is updated, while the backbone parameters trained with offline data remain fixed. Backbone settings for different datasets are presented below.

MNIST-SVHN and CIFAR10-CINIC10 : The backbone is a pre-trained ResNet34 backbone from torchvision with its weights initialized by training on ImageNet. Both the weight estimator and the classifier share the same backbone and its corresponding parameters.

Yearbook : Our network adopts a similar structure and updating process as in the MNISTSVHN and CIFAR10-CINIC10 experiments, following the configuration described in [53]. The backbone of our network consists of a CNN model with the default settings.

Computational Resources. We run experiments with two Xeon Gold 6248R processors (24 cores, 3.0GHz base, 4.0GHz boost), eight Tesla V100S GPUs (32GB video memory each), 768GB RAM, all managed by the Ubuntu 20.04 operating system.

A.2 Supplementary Numerical Results for Section 5.1

In Section 5.1 we summarize the results of the synthetic experiments, and here in Table 3, we supplement their detailed numerical results. Overall, Accous outperforms almost all other methods in the four shift patterns. Compared to the competitors that use only one round of online data, our proposed method achieves a lower average error. The general domain adaptation method DANN is also inferior in the online setting due to the limited amount of data per round, which makes it difficult to learn a good representation. The proposed Accous algorithm also outperforms the OLRE algorithm, which reuses all historical data. The above evidence demonstrates the need for selective reuse of historical data and validates the effectiveness of our algorithm.

B More Related Work

B.1 Related Work on Non-stationary Online Learning

Online convex optimization (OCO) [13] is a powerful paradigm to handle sequential prediction problems. To adapt to the continuously shifting environments, we exploit the online ensemble technique [15, 16] developed in non-stationary online learning literature to hedge the uncertainty. In this part, we will introduce the related works and discuss the difference between our algorithm and the previous work.

Specifically, OCO paradigm considers a T -round game between a learner and the environment. At every round t , the learner makes a prediction θ t ∈ Θ , and, at the same time, the environments reveal the convex loss function L t : Θ → R . Then, the learner suffers from the loss L t ( θ t ) and observes certain information about L t to update the model for the next iteration. A classical performance measure for OCO framework in non-stationary environments is the dynamic regret,

which compares the learner's prediction with the function minimizer θ ∗ t at every iteration. Over the decades, a variety of online learning algorithms have been proposed to optimize this dynamic regret measure [38, 39, 55, 56, 57, 58, 59, 60, 61, 62, 63] in the online convex optimization problem.

Worst-case Dynamic Regret. Many works focus on the full information setting [57, 58, 59, 60, 63], where the learner can observe the loss functions entirely. We call the dynamic regret in such a case as 'worse-case' dynamic regret in the sense that the comparator θ ∗ t is exactly the minimizer of the observed function L t . According to different complexity measures for quantifying the environmental non-stationarity, there are two lines of results. The first is the path length of comparators V θ T = ∑ T t =2 ‖ θ ∗ t -θ ∗ t -1 ‖ 2 measuring the variation of the function minimizers; and the second one is the variation of the function values V L T = ∑ T t =2 max θ ∈ Θ | L t ( θ ) -L t -1 ( θ ) | . For the full information setting, the prior art [63] showed that a simple greedy strategy can already be already well-behaved and achieves an O (min { V θ T , V L T } ) dynamic regret bound that minimizes the two kinds of non-stationarity measure simultaneously.

Universal Dynamic Regret. Competing to the sequences of observed functions' minimizers can be sometimes too pessimistic for the online optimizing, which may lead to a severe overfitting. A more appropriate performance measure is the universal dynamic regret [38], which has drawn more and more attention in recent years. Universal dynamic regret compares the learner's prediction with an

arbitrary comparator sequence { ν t } T t =1 ,

Let V ν T = ∑ T t =1 ‖ ν t -ν t -1 ‖ 2 be the path length of the comparator sequence { ν t } T t =1 . [38] showed that the online gradient descent algorithm achieves an O ((1 + V ν T ) √ T ) universal dynamic regret bound for convex function, whereas there is still a gap to the Ω( √ T (1 + V ν T )) lower bound established by [15], who further proposed an algorithm attaining an O ( √ T (1 + V ν T )) dynamic regret bound and thus closed the gap. Their key algorithmic ingredient is the online ensemble structure [64], which hedges the non-stationarity of the environments by employing a meta-learner to ensemble the predictions from a group of base-learners. When, the loss function is convex and smooth, the O ( √ T (1 + V ν T )) can be improved to O ( √ (1 + min { F T , G T } + V ν T )(1 + V ν T )) [16], where F T = ∑ T t =1 L t ( ν t ) and G T = ∑ T t =2 max θ ∈ Θ ‖∇ L t ( θ ) -∇ L t -1 ( θ ) ‖ 2 2 are two problem-dependent quantities capturing different kinds of easiness of the environments. For the exp-concave and smooth functions, [19] showed an O ( T 1 / 3 ( V ν T ) 2 / 3 ) universal dynamic regret by an improper algorithm (i.e., the output decisions can be slightly out of the feasible domain, while comparators have to locate in the domain), which is proven to be minimax optimal for exp-concave functions. Note that the worst-case dynamic regret (10) is clearly a special case of the universal dynamic regret defined in (11), so a universal dynamic regret upper bound directly implies an upper bound for the worst-case dynamic regret, by substituting comparators { ν t } T t =1 as functions' minimizer { θ ∗ t } T t =1 . This method yields a path-length type bound. Furthermore, we mention that it is also possible to obtain a function-variation type worst-case dynamic regret bound from the universal dynamic regret guarantee, which is revealed in [61, Appendix A.2].

An Intermediate Dynamic Regret. There is also an intermediate case between the worst-case and universal dynamic regret. We refer to it the 'intermediate' dynamic regret in this paper. In this intermediate case, the online learner still aims to optimize the worst-case dynamic regret of the loss function L t ( θ ) , but she can only observe a noisy feedback ̂ L t ( θ ) at each iteration. Since the comparator θ ∗ t = arg min θ ∈ Θ L t ( θ ) is not the minimizer of the observed loss ̂ L t ( θ ) , such a kind of problem is more challenging than the worst-case dynamic regret minimization problem given the uncertainty of the comparator sequence. [39] showed a restarted online gradient descent algorithm can achieve an O ( T 1 / 3 ( V L T ) 2 / 3 ) dynamic regret bound for convex function and an O ( √ TV L T ) bound for strongly convex function under the noisy feedback. However, their algorithm requires the knowledge of the non-stationarity measure V L T = ∑ T t =2 max θ ∈ Θ | L t ( θ ) -L t -1 ( θ ) | defined over L t , which is generally unknown. [48] showed that an O ( T 1 / 3 ( V θ T ) 2 / 3 ) bound is achievable with the noisy feedback when L t is a one-dimensional squared loss. The proposed method is based on the de-noising technique and are free from the knowledge of V θ T . Evidently, this intermediate dynamic regret is still a feasible realization of the universal dynamic regret, so one can achieve the intermediate dynamic regret bound by the universal dynamic regret. However, this reduction will only provide an expected guarantee. More detailed discussions are provided below.

Comparison of Our Results and Earlier Works. Our Reg d T ( { ˜ L t , θ ∗ t } T t =1 ) ≤ O ( T 1 / 3 ( V θ T ) 2 / 3 ) high-probability dynamic regret bound for logistic regression (Theorem 2) falls into the intermediate case, where we aim to minimize the worst-case dynamic regret bound in terms of L t with noisy feedback. The closest related work is by [19], who also exploited an online ensemble structure and achieved a universal dynamic regret in the form of Reg d T ( { ˜ L t , ν t } T t =1 ) ≤ O ( T 1 / 3 ( V ν T ) 2 / 3 ) universal dynamic regret by taking ̂ L t as the input. Further choosing ν t = θ ∗ t and taking the expectation over S t for each round, their result implies an expectation bound E [ Reg d T ] ≤ O ( T 1 / 3 ( V θ T ) 2 / 3 ) . One main advantage of our result is that the bound holds with high probability. To achieve the high-probability bound, we twist the meta-learner from Hedge [42] to Adapt-ML-Prod [43], which helps control the generalization error between ̂ L t and ˜ L t with the negative term introduced by the exp-concavity of the loss function. Besides, similar to [19], our analysis crucially relies on providing a squared formulation of the dynamic regret O (max { 1 , | I | ( V θ T ) 2 } ) on each interval I . However, to obtain such a result, [19] crucially rely on a complicated analysis on the KKT condition to depict the structure of comparators. We greatly simplify the analysis by exploiting the fact that in our case the comparator θ ∗ t is actually the minimizer of expected functions L t , hence avoiding analyzing KKT condition. Our

analysis is applicable to the case when the minimizers lie in the interior of the feasible domain and require the smoothness of the online functions only (which does not rely on the particular properties of logistic loss). This result might be of independent interest in other online learning studies.

C Omitted Details for Section 3

This section provides the proofs for Section 3 in the main paper.

C.1 Proof of Proposition 1

Proof of Proposition 1. We first provide an excess risk bound for the model ̂ w t at each time t and then combine them for T rounds. For each time t ∈ [ T ] , let ˜ R t ( w ) = E ( x ,y ) ∼ S 0 [ r ∗ t ( x ) /lscript ( w /latticetop x , y ) ] be the risk built upon the true density ratio function r ∗ t and ̂ R t ( w ) = E ( x ,y ) ∼ S 0 [ ̂ r t ( x ) /lscript ( w /latticetop x , y ) ] be that built upon the empirical density ratio function ̂ r t . We can decompose the excess risk of the trained model ̂ w t as

For term (a), by a standard generalization error analysis as shown in Lemma 2, with probability at least 1 -δ/ (2 T ) , the gap between R t ( ̂ w t ) and ˜ R t ( ̂ w t ) is bounded by

Then, we proceed to bound term (b).

where the first inequality is due to the optimality of ̂ w t and the second inequality is by the condition max x ∈W ,y ∈Y , w ∈W | /lscript ( w /latticetop x , y ) | ≤ L .

Finally, as for term (c), since w ∗ t is independent of S 0 , a direct application of the Hoeffding's inequality [65, Lemma B.6] implies,

with probability at least 1 -δ/ (2 T ) . Combining the upper bounds for term (a), term (b) and term (c) and taking the summation over T rounds, we get

R T ( { ̂ w t } T t =1 ) ≤ 2 BGDRT √ N 0 +5 BLT √ 2 ln((8 T ) /δ ) N 0 +2 L T ∑ t =1 E x ∼ S 0 [ | r ∗ t ( x ) -̂ r t ( x ) | ] with probability at least 1 -δ , which completes the proof.

C.2 Proof of Proposition 2

Proof of Proposition 2. We begin with converting the estimation error to the squared formulation,

where the first inequality is by Jensen's inequality and the last inequality is due to the Cauchy-Schwarz inequality. Then, we can bound the squared density ratio estimation error as

where the first inequality is due to the strong convexity of the function ψ . The last equality is by the definition of the loss function L ψ t . Combining (12) and (13), we have

C.3 Proof of Theorem 1

Proof of Theorem 1. Following the same arguments in obtaining (12), the empirical cumulative estimation error of the density ratio estimators can be bounded as

Then, we proceed to bound the squared cumulative estimation error by the regret of the loss function ˜ L ψ t ( r ) defined in (5). For notation simplicity, we define

In such a case, the loss function ˜ L ψ t ( r ) = E x ∼ S 0 [ f 1 ( r ( x ))] -E x ∼D t ( x ) [ f 2 ( r ( x ))] and the regret over ˜ L ψ t can be written as

For the first term of R.H.S. of (15), by the Taylor's Theorem, for each x ∈ X , we have

In above ξ x ∈ dom ψ is a certain point on the line collecting ̂ r t ( x ) and r ∗ t ( x ) . The second inequality is due to the definition of f 1 such that,

where the inequality holds due to the µ -strong convexity of ψ ( z ) and the condition ξ x ∂ 3 ψ ( ξ x ) ≤ 0 for all ξ x ∈ dom ψ .

Then, we proceed to bound the second term of R.H.S. of (15),

In above ξ x ∈ dom ψ is a certain point on the line connecting ̂ r t ( x ) and r ∗ t ( x ) . The first inequality is due to the definition of f 2 and the condition on ψ , such that ∂ 2 f 2 ( ξ x ) = ∂ 3 ψ ( ξ x ) ≤ 0 . Plugging (16) and (17) into (15), and rearranging the terms, we arrive

where the term U t is defined as

It remains to analyze the generalization gap U t for every iteration. The main challenge is that the learned model ̂ r t depends on the initial dataset S 0 and the generalization error is related to the complexity of the hypothesis of the density ratio estimator. In the following lemma, we use the covering number [65, Chapter 27] to measure the complexity of the hypothesis space. It is possible to provide an upper bound of U t with other advanced complexity measure.

Lemma 1. Let H r ≜ { r : X → R | ‖ r ‖ ∞ ≤ B ′ r } be a hypothesis space of the density ratio function whose covering number is N ( H r , ε, ‖·‖ ∞ ) . Then, for all model r ∈ H r and δ ∈ (0 , 1) , we have the following upper bound

with probability at least 1 -δ for any ε > 0 , where C r ( µ ) = 4 A r B r / 3 + 16 A 2 r /µ + 3 µB 2 r = O (max { µ, 1 /µ } ) with constants A r = | max x ∈X ∂ 2 ψ ( r ∗ t ( x )) r ∗ t ( x ) | , and B r = max { B,B ′ r } .

Combining (18) and Lemma 1 with the hypothesis space H r = H θ ≜ { x ↦→ h ( x , θ ) | θ ∈ Θ } and rearranging the terms yields,

Further choosing ε = 1 /T and taking the summation of the T iterations, we have

where D r = 2 A r + µ/ (4 T ) = O (1) is a constant. Then, we proceed to bound the covering number of the hypothesis space H θ . Let θ, θ ′ ∈ Θ be the corresponding parameters for the two density ratio function r, r ′ ∈ H θ . Then, we can show that for any ‖ θ -θ ′ ‖ 2 ≤ ε , we have

where G h = max x ∈X ,θ ∈ Θ ‖∇ h ( x , θ ) ‖ 2 is the Lipschitz continuity constant. Then, we can check that the covering number of H θ in terms of ‖·‖ ∞ can be bounded by that of Θ in terms of ‖·‖ 2 as

In the above, the last inequality holds because the parameter space Θ is essentially a L 2 -ball with radius S , whose covering number is bounded by (3 S/ε ) d [66]. Then, plugging (20) into (19), we arrive

Then, we complete the proof by plugging this inequality into (14),

C.4 Useful Lemmas

Lemma 2. Let ˜ R t ( w ) = E ( x ,y ) ∼ S 0 [ r ∗ t ( x ) /lscript ( w /latticetop x , y ) ] . With probability at least 1 -δ over the drawn of S 0 , all predictions w ∈ W satisfy

Proof of Lemma 2. Since the risk function ˜ R t ( w ) is bounded by BL for all w ∈ W , the standard analysis on generalization bound with Rademacher complexity (e.g. [65, Theorem 26.5]) shows that

for any w ∈ W . In above, the function space L t is defined as L t = { ( x , y ) ↦→ r ∗ t ( x ) /lscript ( w /latticetop x , y ) | w ∈ W} and ̂ R S 0 ( L t ) = 1 N 0 E σ ∼{± 1 } N 0 [ sup w ∈W ∑ N 0 n =1 σ i r t ( x n ) /lscript ( w /latticetop x n , y n ) ] is the empirical Rademacher complexity of L t .

Note that the function φ n : R → R defined as φ n ( z ) = r ∗ t ( x n ) /lscript ( z, y n ) is BG -Lipschitz, such that | φ n ( w /latticetop 1 x n ) -φ n ( w /latticetop 2 x n ) | ≤ BG | ( w 1 -w 2 ) /latticetop x n | for any w 1 , w 2 ∈ W . Then, according to the Talagrand's lemma [65, Lemma 26.9], we have

where ˜ W = { x ↦→ w /latticetop x | w ∈ W} is a hypothesis space for the linear functions. By [65, Lemma 26.10], the Rademacher complexity of the linear function class is bounded by

We complete the proof by combining the above displayed inequalities.

Proof of Lemma 1. We begin with the analysis for the generalization gap for a fixed model r ∈ H r . For notation simplicity, we denote by

and let the random variable Z i = E x ∼D 0 [ f r ( x )] -f r ( x i ) for any x i ∈ S 0 . Since x i is i.i.d. sampled from D 0 and f r is independent of S 0 , we have E [ Z i ] = 0 and | Z i | ≤ 2 A r B r with the constants A r = max x ∈X | ∂ 2 ψ ( r ∗ t ( x )) r ∗ t ( x ) | and B r = max { B, max x ∈X ,r ∈H r | r ( x ) |} . Besides, the variance of Z i is bounded by

Then, by the Bernstein's inequality [65, Lemma B.9], with probability at least 1 -δ , we have

where the first inequality is due to the AM-GM inequality. The second inequality is due to the fact that | f r ( x ) | ≤ | ∂ 2 ψ ( r ∗ t ( x )) r ∗ t ( x ) | · | r ( x ) -r ∗ t ( x ) | ≤ A r | r ( x ) -r ∗ t ( x ) | for any x ∈ X .

Then, since r is a fixed model in H r , we can further bound E D 0 [ ( r ( x ) -r ∗ t ( x )) 2 ] by E S 0 [ ( r ( x ) -r ∗ t ( x )) 2 ] by applying the Bernstein's inequality to the random variable Y i = ( r ( x i ) -r ∗ t ( x i )) 2 / (4 B 2 r ) . Specifically, after checking E [ Y i ] = E D 0 [ ( r ( x ) -r ∗ t ( x )) 2 ] / (4 B 2 r ) and Y i ∈ [0 , 1] , a direct application of [65, Lemma B.10] implies

where the last inequality is due to the AM-GM inequality. Plugging (22) into (21), for a fixed r ∈ H r , we have

with probability at least 1 -δ .

Then, based on the notion of covering number, we show a union bound of U t over all r ∈ H r . Let N ( H r , ε, ‖·‖ ∞ ) be the ε -net of H r such that for any r ∈ H r one can find a r ′ ∈ N ( H r , ε, ‖·‖ ∞ ) satisfying ‖ r -r ′ ‖ ∞ ≤ ε . The covering number N ( H r , ε, ‖·‖ ∞ ) is defined as the minimal cardinality of an ε -net of H r . By taking the union bound over all r ′ ∈ N ( H r , ε, ‖·‖ ∞ ) , with probability at least 1 -δ , the following holds for any r ′ ∈ N ( H r , ε, ‖·‖ ∞ ) ,

where C r ( µ ) = 4 A r B r / 3 + 16 A 2 r /µ +3 µB 2 r = O (max { µ, 1 /µ } ) . Then, for any model r ∈ H r , we can show

where the first inequality is due to the property of the ε -net such that

The last inequality is by

Thus, we complete the proof.

Algorithm 1 Base-learner E i

Input: Active interval I i = [ s i , e i ] , regularization parameter λ and step size η , feasible domain Θ .

1: Initialize the model ̂ θ s i ,i as any point in the domain Θ and the matrix A s i -1 = λI d .

2: for t ∈ I i = [ s i , e i ] do

3: Submit the model ̂ θ t,i to the meta-learner.

4: Update the matrix by A t,i = A t -1 ,i + ∇ ̂ L t ( ̂ θ t,i ) ∇ ̂ L t ( ̂ θ t,i ) /latticetop .

5: Update the model for next iteration by

where Π A t,i Θ [ ν ] = arg min θ ∈ Θ ‖ θ -ν ‖ A t,i is the projection operator. 6: end for

D Omitted Details for Section 4

This section presents the omitted details for Section 4. We will first introduce the algorithm details for our online ensemble algorithm, characterized by its meta-base structure, in Appendix D.1. Then, we present the theoretical guarantees alongside their proofs for the base-learner (Lemma 3) and metalearner (Lemma 10) in Appendix D.2 and Appendix D.3 respectively. These elements collectively contribute to the dynamic regret guarantee of the overall algorithm. The dynamic regret of the online ensemble algorithm (Theorem 2) is presented in Appendix D.4. We provide the proof for the average excess risk (Theorem 3) in Appendix D.5. Further discussions regarding the tightness of the bound and alternative choices for the Bregman divergence function are located in Appendices D.6 and D.7.

D.1 Omitted Algorithm Details

Here, we introduce the algorithmic details for the online ensemble method omitted in Section 4.1.

Interval Schedules. We run multiple based learners on the geometric covering:

and then employ a meta-learner to combine them by weights. For any interval I i = [ s i , e i ] ∈ C , which starts at time t = s i and end at t = e i , we denote by E i the base-learner running over I i .

Base-learner. The base-learner E i updates his/her model with the ONS algorithm [44] in the associated active period I i . The algorithmic details of the base-learner are summarized in Algorithm 1. At every iteration t ∈ I i , the base-learner E i will first submit his/her model ̂ θ t,i to the meta-learner and then update the model for the next iteration by Line 4 and Line 5 as shown in Algorithm 1.

Meta-learner. The meta-learner aggregates the predictions of each active base-learner by a weighted combination scheme. To obtain the weight p t,i for each active base-learner E i , we maintain three terms m t,i , v t,i and ε t,i for each base-learner E i when he/she is active, i.e., t ∈ I i . Let g t ( θ ) = 〈∇ ̂ L t ( ̂ θ t ) , θ -̂ θ t 〉 be a linearized loss for the original function ̂ L t ( θ ) . The first term is defined by

which measures the performance gap between the meta-learner's prediction ̂ θ t and the i -th baselearner's prediction ̂ θ t,i over the linearized loss g t ( θ ) . The second term v t,i is the 'potential' of the base-learner E i , which measures the historical performance of E i over the interval. When invoking the base-learner E i at t = s i , we initialize v s i -1 ,i = 1 /K and the term is updated by

The last term ε t,i is the learning rate, which is set as

Algorithm 2 Meta-learner

After obtaining the v t,i and ε t,i for each active base-learner, the weight for the next iteration t +1 is updated by

The algorithm is summarized in Algorithm 2. At the beginning of each iteration t , the algorithm will update the index set for the active base-learner (Line 3-Line 10). Then, the meta-learner's prediction is generated by the weighted combination rule (Line 11). After that, the meta-learner will update the potential v t +1 ,i and the step ε t +1 ,i for the next iteration.

D.2 Regret Guarantee for Base-Algorithm (Lemma 3)

By suitable configurations of the parameters γ, λ > 0 , we show that the ONS algorithm enjoys the following dynamic regret guarantees,

Lemma 3. Let θ ∗ t = arg min θ ∈ R d L t ( θ ) and all the minimizer θ ∗ t 's lie in the interior of Θ . Set γ = 6(1 + β ) and λ = 1 . Then, with probability at least 1 -δ , the ONS running on the interval I i = [ s i , e i ] ⊆ [ T ] ensures

where V θ I i = ∑ e i t = s i +1 ‖ θ ∗ t -θ ∗ t -1 ‖ 2 is the path length measuring the fluctuation of the minimizers.

D.2.1 Main Proof

This section provides the proof of the ONS algorithm. Since the analysis of the single base-learner is independent from the online ensemble structure, for notation simplicity, we omit the subscribe of the base-learner in the analysis. Specifically, we abbreviate the prediction ̂ θ t,i , matrix A t,i and the interval I i as ̂ θ t , matrix A t and the interval I respectively.

Proof of Lemma 3. We begin the proof of the dynamic regret with a lemma on the static regret of the ONS algorithm, whose proof is proved in Appendix D.2.2. For notation simplicity, we omit the subscribes of the base-learner, where the prediction θ t,i , matrix A t,i and interval I i is abbreviated as θ t , A t and I , respectively.

Lemma 4. Set γ = 6(1 + β ) and λ = 1 . With probability at least 1 -2 δ , for any δ ∈ (0 , 1] , the ONS algorithm (8) running on the interval I = [ s, e ] ⊆ [ T ] ensures

for any θ ∈ Θ , where C (1) I ( δ ) = 12 dβ ′ log ( 1 + | I | R 2 ) + S 2 3 β ′ + (48 β ′ + 6 S 2 R 2 β ′ ) log √ 2 | I | +1 δ + (4 SR + S 2 R 2 β ′ ) √ log 2 | I | +1 δ 2 and β ′ = β +1 .

Then, we can decompose the dynamic regret as follows,

where ¯ θ ∗ I = 1 | I | ∑ t ∈ I θ ∗ t is an averaged prediction of the function minimizers over T rounds. Under the condition all function minimizers lie in the interior of Θ , we have ¯ θ ∗ I ∈ Θ . As a consequence, the last inequality holds due to Lemma 4.

Then, we proceed to bound the last two terms of the R.H.S. of the above inequality. Since the first-order derivative of the sigmoid function σ ′ ( z ) = exp( -z ) / (1 + exp( -z )) 2 ≤ 1 / 4 for any z ∈ R . Let I d ∈ R d × d be the identity matrix. we can check ˜ L t is a R 2 / 4 -smooth function such that ∇ 2 ˜ L t ( θ ) = 1 2 E x ∼ S 0 [ σ ′ ( x /latticetop θ ) φ ( x ) φ ( x ) /latticetop ] + 1 2 E x ∼D t [ σ ′ ( x /latticetop θ ) φ ( x ) φ ( x ) /latticetop ] ≼ R 2 4 I d for any θ ∈ Θ . Then, with probability at least 1 -2 δ , we have

where V θ I = ∑ e t = s +1 ‖ θ ∗ t -1 -θ ∗ t ‖ 2 with s being the start time of the interval I and e being the ending time. In above, the first inequality is due to the smoothness of the loss function and the second inequality comes from the AM-GM inequality. The third inequality is due to Lemma 5. The last inequality can be obtained by the definition of ¯ θ ∗ I such that

Plugging (29) into (28) and taking the summation over T rounds, we obtain

with probability at least 1 -3 δ . We have completed the proof.

D.2.2 Useful Lemmas

Lemma 4. Set γ = 6(1 + β ) and λ = 1 . With probability at least 1 -2 δ , for any δ ∈ (0 , 1] , the ONS algorithm (8) running on the interval I ⊆ [ T ] ensures,

for any θ ∈ Θ , where C (1) I ( δ ) = 12 dβ ′ log ( 1 + | I | R 2 ) + S 2 3 β ′ + (48 β ′ + 6 S 2 R 2 β ′ ) log √ 2 | I | +1 δ + (4 SR + S 2 R 2 β ′ ) √ log 2 | I | +1 δ 2 and β ′ = β +1 .

Proof of Lemma 4. The main difference between the proof of Lemma 4 and the standard analysis for the ONS algorithm (e.g., proof of [13, Theorem 4.3]) is that the later one is for the full information online learning setting, in which the learner can exactly observe the loss function ˜ L t . However, in the continuous covariate shift problem, we can only observe the empirical loss ̂ L t , which poses the challenge to bound a generalization gap between the expected loss ˜ L t and the empirical observation ̂ L t (e.g. term (b) in (35)). We manage to show such a gap can be bounded by O (log T ) with the self-normalized concentration inequality for the martingale different sequence.

We start with the analysis of the instantaneous regret. By the Taylor's theorem, we have

where ξ t ∈ R d is a certain point on the line connecting θ and ̂ θ t . In above, the second order derivative ∇ 2 ˜ L t can be further lower bounded by

where σ ′ ( z ) = exp( -z ) / (1 + exp( -z )) 2 is the first-order derivative of the sigmoid function σ ( z ) = 1 / (1 + exp( -z )) and A ≽ B indicates A -B is a positive semi-definite matrix for any matrix A,B ∈ R d × d . The first equality is due to the definition of ˜ L t and the second inequality is due to Lemma 6.

Plugging (31) into (30), we have

Let E t [ · ] = E S t ∼D t [ · | S 0 , S 1 , . . . , S t -1 ] be the expectation taken over the draw of S t conditioned on the randomness until round t -1 . Since ̂ θ t is independent of S t , we have E t [ ̂ L t ( ̂ θ t )] = ˜ L t ( ̂ θ t ) and E t [ ∇ ̂ L t ( ̂ θ t )] = ∇ ˜ L t ( ̂ θ t ) . Taking the expectation over both sides of (32) yields

where the first term of the R.H.S. can be further bounded by

In above, the first inequality is due to the update rule of the ONS algorithm and the Bregman proximal inequality [67, Lemma 5]. The second inequality is due to the AM-GM inequality. The last equality is due the definition of A t such that ‖ ̂ θ t -θ ‖ 2 A t -‖ ̂ θ t -θ ‖ 2 A t -1 = ( ∇ ̂ L t ( ̂ θ t ) /latticetop ( ̂ θ t -θ )) 2 .

Plugging (34) into (33) and taking the summation over the interval I = [ s, e ] , we have

Now, we proceed to bound term (a) and term (b), respectively. Firstly, according to Lemma 7, we can show term (a) is bounded by

Then, we handle term (b) with the Bernstein-type self-normalized concentration inequality [68, Theorem 4] . Specifically, let E t [ · ] = E S t ∼D t [ · | S 0 , S 1 , . . . , S t -1 ] be the expectation taken over the draw of S t conditioned on the randomness until round t -1 . Denoting by Z t = 〈∇ ˜ L t ( ̂ θ t ) -∇ ̂ L t ( ̂ θ t ) , ̂ θ t -θ 〉 , it is easy to check { Z t } e t = s is a martingale difference sequence such that E t [ Z t ] = 0 and | Z t | ≤ 4 SR . Lemma 8 with the choice ν = 1 / (48(1 + β )) indicates,

Combining the upper bounds for term (a) and term (b) with (35), we have

Then, setting γ = 6(1 + β ) , with probability at least 1 -δ , we can bound term (c) by

where the last inequality is due to Lemma 9. Combining the upper bound for term (c) and (37) and setting λ = 1 , we have

with probability at least 1 -2 δ , which completes the proof.

Lemma 5. When all the minimizers θ ∗ t = arg min θ ∈ R d L t ( θ ) lie in the interior of Θ , then we have

Proof of Lemma 5. Since all the minimizers θ ∗ t = arg min θ ∈ R d L t ( θ ) lie in the interior of Θ , then we have ∇ L t ( θ ∗ t ) = 0 and the term ‖∇ ˜ L t ( θ ∗ t ) ‖ 2 2 can be rewritten as

where { Z n } N 0 n =1 with Z n = ( σ ( φ ( x n ) /latticetop θ ∗ t ) -1) φ ( x n ) -E x ∼D 0 [( σ ( φ ( x ) /latticetop θ ∗ t ) -1) φ ( x )] are N 0 independent random vectors. We can check that E x n ∼D 0 [ Z n ] = 0 and ‖ Z n ‖ 2 ≤ 2 R for any x n ∈ X . Then by the Hoeffding's inequality for the random vector [69, Theorem 6.1.1], with probability at least 1 -δ , we have

Plugging (39) into (38) obtains ‖∇ ˜ L t ( θ ∗ t ) ‖ 2 2 = 2 R 2 ln(( d +1) /δ ) N 0 + o ( log( d/δ ) N 0 ) .

Proof of Lemma 6. For notation simplicity, we denote by a t = E x ∼ S 0 [( σ ( x /latticetop ̂ θ t ) -1) x ] and b t = E x ∼ S t [ σ ( x /latticetop ̂ θ t ) x ] . Then, according to the definition of ∇ ̂ L t ( ̂ θ t ) , we have

where the last inequality is due to Lemma 16. We can further bound the R.H.S. of the above inequality by

In above the first inequality is due to the fact that E [ a ] E [ a ] /latticetop ≼ E [ aa /latticetop ] for any random vector a ∈ R d . The second inequality comes from ( σ ( x /latticetop ̂ θ t ) -1) 2 ≤ 1 since the output value of the sigmoid function σ is bounded in (0 , 1) . A similar arguments shows b t b /latticetop t ≤ E x ∼ S t [ φ ( x ) φ ( x ) /latticetop ] . Combining it with above displayed inequalities completes the proof.

Lemma 7. The predictions returned by the ONS algorithm satisfies,

Proof of Lemma 7. The proof follows the standard arguments in the analysis of the online Newton step algorithm [13, Theorem 4.3]. We present the proof here for self-containedness.

where the last inequality is due to the fact that trace( A -1 ( A -B )) ≤ log( | A | / | B | ) for any positive definite matrix satisfying A ≽ B /follows 0 [13, Lemma 4.5].

By definition, we have A s = λI d ∈ R d × d and we have A e = λI d + ∑ t ∈ I ∇ ̂ L t ( ̂ θ t ) ∇ ̂ L t ( ̂ θ t ) /latticetop . Furthermore, since ‖∇ ̂ L t ( ̂ θ t ) ‖ 2 ≤ R , we have | A e | ≤ ( | I | R 2 + λ ) d . In such a case, we can bound the above inequality by

and finish the proof.

Lemma 8. Let Z t = 〈∇ ˜ L t ( ̂ θ t ) -∇ ̂ L t ( ̂ θ t ) , ̂ θ t -θ 〉 . Then, with probability at least 1 -δ , we have

for any δ ∈ (0 , 1) and ν > 0 .

Proof of Lemma 8. Wecan check that { Z t } e t = s is a martingale difference sequence such that E t [ Z t ] = 0 since ̂ θ t is independent of S t . Further noting that | Z t | ≤ ‖∇ ˜ L t ( ̂ θ t ) -∇ ̂ L t ( ̂ θ t ) ‖ 2 ‖ ̂ θ t -θ ‖ 2 ≤ 4 SR , a direct application of Lemma 17 shows that

with probability at least 1 -δ for any ν > 0 . Then, for any t ∈ [ T ] , we can further bound the term Z 2 t and E t [ Z 2 t ] by

and

where the first inequality is due to the fact ( a -b ) 2 ≤ 2 a 2 +2 b 2 for any a, b ∈ R . Then, combining the above two displayed inequalities and taking the summation over T iterations, we have

We complete the proof by plugging (42) into (40).

Lemma 9. With probability at least 1 -δ , we have

for any δ ∈ (0 , 1) , where Λ I ( δ ) = 32 S 2 R 2 log √ 2 | I | +1 δ +4 S 2 R 2 √ log 2 | I | +1 δ 2

Proof of Lemma 9. We can check { Y t } e t = s is a martingale difference sequence such that | Y t | ≤ 4 S 2 R 2 . As a consequence, a direct application of Lemma 17 implies,

where the first inequality follows the proof of Lemma 8 and the second inequaliuty is due to the parameter setting ν = 1 / (32 S 2 R 2 ) and the condition that ( ∇ ̂ L t ( ̂ θ t ) /latticetop ( ̂ θ t -θ )) 2 ≤ 4 S 2 R 2 . Rearranging the above inequlaity, we have

which completes the proof.

D.3 Regret Guarantee for Meta-Algorithm (Lemma 10)

We have the following guarantee on the meta-algorithm, which ensures that for any interval I i ∈ C , the final prediction is comparable with the prediction of the base-learner E i with an O (log T ) cost.

Lemma 10. For any interval I i ∈ C and δ ∈ (0 , 1) , with probability at least 1 -δ , the final prediction returned by our meta-learner satisfies

where K = |C| is the number of the intervals contained in the set C .

D.3.1 Main Proof

This part presents the proof of Lemma 10. Our analysis for the meta-learner is based on that of the strongly adaptive algorithm [70], where the problem of tracking base-learners on the corresponding intervals is converted to a sleeping expert problem [71]. The main challenge of our problem is that the feedback function is noisy. Thus, we require to provide tight analysis on the generalization gap between ̂ L t and ˜ L t without ruining the O (log T ) regret guarantee. By twisting the meta-algorithm from Hedge [42] used in the previous work to Adapt-ML-Prod [43], we show that the generalization gap can be cancelled by the negative term introduced by the exp-concavity of the loss functions (see how to bound term (b) in (48)).

Proof of Lemma 10. To prove Lemma 10, we show that the final prediction ̂ θ t generated by rule ̂ θ t = ∑ E i ∈A t p t,i ̂ θ t,i with the weight update procedure (25), (26) and (27) is identical to an algorithm learning in the standard prediction with expert advice (PEA) setting [72], where all base-learners submit their predictions at each time t ∈ [ T ] .

Lemma 11. For any base-learner E i , let E ′ i be a surrogate of E i , whose prediction ¯ θ t,i = ̂ θ t,i is the same as that of E i for any t ∈ I i and ¯ θ t,i = ̂ θ t for other iterations. Then, for any time t ∈ [ T ] , the surrogate meta-learner predicts as

where the potential ¯ v t,i and the learning rate ¯ ε t,i is defined as

and

with ¯ m t,i = 〈 ̂ L t ( ¯ θ t ) , ¯ θ t -¯ θ t,i 〉 , ¯ v 0 ,i = 1 /K and ¯ ε 0 ,i = min { 1 / 2 , ln K } for all i ∈ [ K ] . Then, we have ¯ θ t = ̂ θ t for any t ∈ [ T ] .

We have the following guarantee for the final prediction generated by the surrogate algorithm defined by (43).

Lemma 12. With probability at least 1 -2 δ , for any δ ∈ (0 , 1) , the prediction returned by the surrogate meta-algorithm defined as (43) ensures,

for any base-algorithm E ′ i , i ∈ [ K ] , where C (2) T ( δ ) = 16(1+ β ) S 2 R 2 ( C ′ T ) 2 ln K + 1 6(1+ β ) + 2 C ′ T + ( 24(1 + β ) + 11 S 2 R 2 1+ β ) log √ 2 T +1 δ + ( 4 SR + S 2 R 2 2(1+ β ) )√ log 2 T +1 δ 2 and C ′ T = 3ln K + ln ( 1 + K 2 e (1 + ln( T +1)) ) = O (ln K +lnln T ) .

Then, according to the relationship between E i and E ′ i , we can bound the regret of the final prediction ̂ θ t with respect to that of any base-learner E i over the interval I i by

In above, the first equality is due to Lemma 11 and the definition of the surrogate base-learner E ′ i such that ¯ θ t,i = ̂ θ t,i for any t ∈ I i and ¯ θ t,i = ̂ θ t for any t ∈ [ T ] /I i . The first inequality is due to Lemma 12. In the last equality, we treat double logarithmic factors in T as a constant, following previous studies [73, 74]. We have completed the proof.

D.3.2 Useful Lemma

Proof of Lemma 11. We show that the predictions ̂ θ t and ¯ θ t by the two algorithm are exactly the same for any t ∈ [ T ] by induction. At iteration t ∈ [ T ] , we have the following two induction hypotheses:

IH1: the final predictions satisfy ̂ θ τ = ¯ θ τ for all τ ≤ t .

IH2: any base-algorithms E i ∈ A τ satisfy p τ,i = ¯ p τ,i / ( ∑ i ∈A τ ¯ p τ,i ) for all τ ≤ t .

Base case: For the base case of t = 1 , there is only one active base-algorithm. Denote by i 1 the index of the active base-algorithm, we have ̂ θ 1 = ̂ θ 1 ,i 1 , p 1 ,i 1 = 1 and A 1 = { i 1 } . Moreover, for the surrogate algorithm, we have ¯ p 1 ,i = 1 /K for any i ∈ [ K ] and ¯ θ 1 = ( ∑ i ∈ [ K ] / { i 1 } ̂ θ 1 + ̂ θ 1 ,i 1 ) / | K | = ̂ θ 1 . We can check that ̂ θ 1 = ¯ θ 1 and p 1 ,i = ¯ p 1 ,i / ∑ i ∈A 1 ¯ p 1 ,i for any i ∈ A 1 .

Induction step: For the induction step, we first show

for all i ∈ A t +1 . By the definition of ¯ p t +1 ,i , it is sufficient to prove

For the base-algorithm E i active at iteration t +1 and his/her corresponding surrogate base-algorithm E ′ i , we can decompose the time horizon until t +1 into two intervals: asleep interval I slp i = [1 , s i -1] and active part I act i = [ s i , t +1] . For any time stamp belonging to the asleep interval τ ∈ I slp i , we can check that the loss for the base-algorithm E ′ i satisfies

where the first equality is by the definition of ¯ m τ,i . The second equality is due to IH1 such that ¯ θ τ = ̂ θ τ for any τ ≤ t and the definition of the surrogate base-learner E ′ i such that the prediction ¯ θ τ,i = ̂ θ τ on the asleep interval. For any time stamp belonging to the active interval τ ∈ I act i , similar argument shows that

Thus, by definition, we can draw the conclusion that ¯ v t,i = v t,i and ¯ ε t,i = ε t,i for any active base-algorithm with index i ∈ A t +1 , which would finally lead to (47).

Then, we show ̂ θ t +1 = ¯ θ t +1 based on IH1 and IH2 . We have

where the first equality is due to (46) and the second inequality is due to the definition of ̂ θ t +1 . We have finished the induction steps and completed the proof.

Proof of Lemma 12. Since ̂ θ t and ̂ θ t,i are both independent of S t , using the same arguments for obtaining (33), we can obtain that

For term (a), we have

for any i ∈ [ K ] , where C ′ T = 3ln K +ln ( 1 + K 2 e (1 + ln( T +1)) ) = O (ln K +lnln T ) . In above, the first inequality is a direct application of [43, Corollary 4] with the linearized loss M t ( θ ) = 〈 ̂ L t ( ̂ θ t ) , θ 〉 / (2 SR ) + 1 / 2 and the second inequality is due to the AM-GM inequality.

For term (b), by Lemma 8 with ν = 6(1 + β ) and a similar argument in (36), we have

Combining the upper bound for term (a) and term (b) with (48) yields,

Then, according to Lemma 9, we can further bound term (c) by

which implies

In above, C ′ 1 = 24(1 + β ) + 11 S 2 R 2 / (1 + β ) and C ′ 2 = 4 SR + 4 S 2 R 2 / (3(1 + β )) . In the last equality, we treat double logarithmic factors in T as a constant, following previous studies [73, 74]. We have completed the proof.

D.4 Proof of Theorem 2

This part presents the proof for Theorem 2, which is based on the dynamic regret for the base-learner (Lemma 3) and that for the meta-learner (Lemma 10) presented in Appendix D.1.

D.4.1 Main Proof

Proof of Theorem 2. When choosing the Bregman divergence function as ψ ( t ) = ψ LR ( t ) ≜ t log t -( t +1)log( t +1) and the hypothesis space H LR θ = { x ↦→ exp( -θ /latticetop φ ( x )) | ‖ θ ‖ 2 ≤ S } . The expected loss function L t ( θ ) as defined by (4) is equal to

Let θ ∗ t ≜ arg min θ ∈ R d L t ( θ ) . Since the ground-truth density ratio r ∗ t ( x ) = D t ( x ) / D 0 ( x ) function is contained in the hypothesis space as D t ( x ) / D 0 ( x ) ∈ H LR θ . One can show that D t ( x ) / D 0 ( x ) = exp( -( θ ∗ t ) /latticetop φ ( x )) and θ ∗ t ∈ Θ = { θ ∈ R d | ‖ θ ‖ 2 ≤ S } .

Then, for the base-learner E i running on any interval I i , Lemma 3 in Appendix D.2 shows that its prediction ̂ θ t,i is comparable with the best model at every iteration θ ∗ t with an O ( | I i | ( V θ I i ) 2 ) cost:

where V θ I i = ∑ e i t = s i +1 ‖ θ ∗ t -θ ∗ t -1 ‖ 2 is the path length measuring the fluctuation of the minimizers.

Besides, Lemma 10 in Appendix D.3 ensures that the prediction ̂ θ t of the meta-learner is comparable with the prediction ̂ θ t,i of each base-learner E i on the associated interval I i with an O (log T ) .

A direct combination of (51) and (52) shows that the meta-learner is able to track the best prediction θ ∗ t on any interval I i ∈ C . Actually, by exploiting the structure of the geometric covering (24), it can be shown that such a property can be extended to arbitrary interval I ∈ [ T ] as the following lemma, whose proof is deferred to Appendix D.4.2.

Lemma 13. With probability at least 1 -5 δ for any δ ∈ (0 , 1) , for any interval [ s, e ] ∈ [ T ] , the meta-learner's prediction ̂ θ t returned by (9) ensures,

where V θ I = ∑ e t = s +1 ‖ θ ∗ t -1 -θ ∗ t ‖ 2 and δ ′ = δ / 2 log | I | . The coefficient C (1) T ( δ ) = O ( dβ log( T/δ )) and C (2) T ( δ ) = O ( β log( T/δ ) ) is defined in Lemma 4 and Lemma 12, respectively.

Then, we can use the bin partition [19, Lemma 30] to convert the dynamic regret on each interval to a dynamic regret over T iterations. For self-containedness, we restate the lemma here.

Lemma 14 (Lemma 30 of [19]) . There exists a partition P of the time horizon [ T ] into M = O (max { dT 1 / 3 ( U θ T ) 2 / 3 , 1 } ) intervals ,i.e., { I i = [ s i , e i ] } i M =1 such that for any interval I i ∈ P , U θ I i ≤ D max / √ | I i | , where U θ I i = ∑ e i t = s i +1 ‖ θ ∗ t -θ ∗ t -1 ‖ 1 and D max = max t ∈ [ T ] ‖ θ ∗ t ‖ ∞ ≤ max t ∈ [ T ] ‖ θ ∗ t ‖ 2 ≤ S .

Then, we can decompose the overall dynamic regret of our method into those over the key partition P . Specifically, with probability at least 1 -5 δ , we have

where the second last inequality is due to Lemma 13 and a union bound taken over all intervals in the key partition. The last inequality is due to Lemma 14 such that V θ I i ≤ U θ I i ≤ S/ √ I i for any interval I i ∈ P . Lemma 14 shows that we can bound the partition number by M = O (max { dT 1 / 3 ( U θ T ) 2 / 3 ) , 1 } ) = O (max { d 4 / 3 T 1 / 3 ( V θ T ) 2 / 3 , 1 } ) . Then, we can further show that

which implies that the overall dynamic regret is bounded by

Hence, we complete the proof by showing ˜ L t ( θ ∗ t ) = ˜ L t ( r ∗ t ) under the realizable assumption and converting the variation of the model parameter V θ T = ∑ T t =2 ‖ θ ∗ t -θ ∗ t -1 ‖ 2 to the variation of the feature distribution V T = ∑ T t =2 ‖D t ( x ) -D t -1 ( x ) ‖ 1 by Lemma 15.

D.4.2 Useful Lemmas

Lemma 13. With probability at least 1 -5 δ for any δ ∈ (0 , 1) , for any interval [ s, e ] ∈ [ T ] , the meta-learner's prediction ̂ θ t returned by (9) ensures,

where V θ I = ∑ e t = s +1 ‖ θ ∗ t -1 -θ ∗ t ‖ 2 and δ ′ = δ / 2 log | I | . The coefficient C (1) T ( δ ) = O ( dβ log( T/δ )) and C (2) T ( δ ) = O ( β log( T/δ ) ) is defined in Lemma 4 and Lemma 12, respectively.

Proof of Lemma 13. A direct combination of Lemma 3 and 10 shows that, with probability at least 1 -5 δ , we have the following dynamic regret guarantee on any interval I i = [ s i , e i ] ∈ C .

where V θ I i = ∑ e i t = s i +1 ‖ θ ∗ t -1 -θ ∗ t ‖ 2 . The coefficient C (1) I i = O ( dβ log( | I i | /δ )) is defined in Lemma 4 and C (2) T ( δ ) = O ( β log( T/δ ) ) is defined in Lemma 12.

For any interval I = [ s, e ] ⊆ [ T ] , [75, Lemma 3] showed that it can be partitioned into two sequences of disjoint and consecutive intervals, denoted by I -p , . . . , I 0 ∈ C and I 1 , . . . , I q ∈ C , such that

Then, with probability at least 1 -5 δ , we can decompose the regret over I = [ s, e ] ⊆ [ T ] as

where the last inequality is due to (53) and a union bound taken over all intervals. Then, due to the partition of the intervals (54), we can check that p + q ≤ 2 log | I | . Then, we can further bound the above displayed inequality by

where the first inequality is due to the fact that ∑ q i = -p V 2 I i ≤ V 2 I . This completes the proof.

Lemma 15. Under the condition that D t ( x ) / D 0 ( x ) ∈ H LR θ ≜ { exp( -θ /latticetop φ ( x )) | ‖ θ ‖ 2 ≤ S } for any t ∈ [ T ] and E x ∼D 0 [ | φ ( x ) /latticetop a | ] ≥ α ‖ a ‖ 2 for any a ∈ R d with a certain α > 0 . Then, we have ‖ θ ∗ t -1 -θ ∗ t ‖ 2 ≤ β ‖D t -1 ( x ) -D t ( x ) ‖ 1 /α.

Proof of Lemma 15. Under the realizable assumption such that D t ( x ) / D 0 ( x ) ∈ H LR θ . One can verify that exp( -φ ( x ) /latticetop θ ∗ t ) = D t ( x ) / D 0 ( x ) . Then, we can convert the variation of the model ‖ θ ∗ t -1 -θ ∗ t ‖ 2 to the variation of the distribution ‖D t -1 ( x ) -D t ( x ) ‖ 1 .

where the first inequality is by the condition E x ∼D 0 [ | φ ( x ) /latticetop a | ] ≥ α ‖ a ‖ 2 and the second is by the mean value theorem and the boundedness of | φ ( x ) /latticetop ( θ ∗ t -1 -θ ∗ t ) | ≤ 2 SR for any x ∈ X . The last inequality is due to the realizable assumption.

D.5 Proof of Theorem 3

Proof of Theorem 3. Then, we can bound the overall excess risk over the model trained by IWERM (1) by a combination of Proposition 1 and Theorem 1 as

where the first inequality is due to Proposition 1 and C T ( δ ) = 2 GDR + 5 L √ 2 log(8 T/δ ) = O (log( T/δ )) . The second inequality is due to Theorem 1 with the choice of the logistic regression model ψ ( t ) = t log t -(1+ t ) log(1+ t ) and ̂ r t ( x ) = exp( -φ ( x ) /latticetop ̂ θ t ) and the realizability assumption r ∗ t ( x ) = exp( -φ ( x ) /latticetop θ ∗ t ) . Then, by Theorem 2, we have

t =1 t =1 , which implies √ √ √ √ T [ T ∑ t =1 ˜ L t ( ̂ θ t ) -T ∑ t =1 ˜ L t ( θ ∗ t ) ] + ≤ O ( d 2 3 log ( T/δ ) · max { T 2 3 V 1 3 T , √ T } + T √ ln( dT/δ ) N 0 ) . (56)

We complete the proof by substituting (56) it into (55).

D.6 Discussion on Tightness of the Bound

This part illustrates the tightness of our bound with the case where the labels of testing samples are available after the prediction.

Continuous Shift with Labeled Feedback. We consider a T -round online learning process. At iteration t , the learner submits her prediction ̂ w t ∈ W . At the same time, the environments pick the data pair ( x t , y t ) ∼ D t . Then, the learner updates her model with the loss function ̂ R t ( w ) = /lscript ( w /latticetop x t , y t ) and obtains the prediction ̂ w t +1 ∈ W for the next iteration. The goal of the learner is to minimize the cumulative excess risk against the sequence of optimal models w ∗ t ∈ arg min w ∈W R t ( w ) , that is,

The goal is the same as that of standard continuous covariate shift, see the problem setup in (1). The key difference is that now the label of testing data is available . Therefore, the noisy feedback ̂ R t ( w ) observed by the learner is an unbiased estimator with respect to the true risk R t ( w ) .

Tightness of Our Bound. The continuous shift with labeled feedback is essentially a non-stationary stochastic optimization problem [39]. For general convex functions, [39] showed that any gradientbased algorithm will suffer

in the worst case. In above, V L T = ∑ T t =2 max w ∈W | R t ( w ) -R t -1 ( w ) | measures the fluctuation of the risk function. For the same performance measure, our algorithm achieves

In the non-stationary case, i.e., V T ≥ Θ( T -1 2 ) , our bound becomes ˜ O ( 1 / √ N 0 + T -1 / 3 V 1 / 3 T ) . As discussed below Proposition 1, the first term ˜ O (1 / √ N 0 ) is hard to be improved. We thus focus on the tightness of the second term ˜ O ( T -1 / 3 V 1 / 3 T ) .

Although the definition of V T in our upper bound is different from the that of V L T in the lower bound (58), the two bounds share the same dependence on the time horizon T , which provides evidence that our bound is hard to improve. Indeed, consider the 1-dimensional case where the underlying distribution only shifts once from D 1 ( x, y ) to D 2 ( x, y ) at a certain time t ∈ [ T ] . In such a case, V T = O (1) and our bound implies an O ( T -1 / 3 ) rate for the second term of (59). On the other hand, the rate in the lower bound (58) becomes Ω( T -1 / 3 ) . The same dependence on the time horizon T indicates our bound is hard to improve. We leave a precise lower bound argument as the future work.

D.7 On Other Choices of Bregman Divergence Function

In this section, we discuss how to apply the other two choice of the divergence function ψ LS = ( t -1) 2 / 2 and ψ KL = t log t -t in our framework. Our analysis crucially relies on two conditions:

the Bregman divergence function ψ satisfies the conditions in Theorem 1.

the induced loss function ̂ L t ( θ ) is an exp-concave and smooth function with respect to θ .

Choice of ψ LS ( t ) = ( t -1) 2 / 2 . Considering the Bregman divergence function ψ ( t ) = ( t -1) 2 / 2 and the hypothesis space H θ = { x ↦→ φ ( x ) /latticetop θ | ‖ θ ‖ 2 ≤ S } , where φ : X → R d represents a specific basis function with bounded norm ‖ φ ( x ) ‖ 2 ≤ R . Then, the loss ̂ L ψ t ( θ ) as per (7) becomes

The above configuration recovers the unconstrained least-squares importance fitting (uLSIF) method [28]. We can show such a choice of divergence function satisfy the condition required by Theorem 1 and enjoys favorable function properties.

Conditions in Theorem 1: when Bregman divergence function is chosen as ψ LS , we have ∂ 2 ψ LS ( t ) = 1 and ∂ 3 ψ LS ( t ) = 0 . Thus, ψ LS is a 1 -strongly convex function and t∂ 3 ψ LS ( t ) = 0 for t ∈ R .

Expconcavity and smoothness: We have ∇ 2 ̂ L t ( θ ) = E S 0 [ φ ( x ) φ ( x ) /latticetop ] . Then, when the input is upper bounded by ‖ φ ( x ) ‖ 2 ≤ R for all φ ( x ) ∈ X , we have ∇ 2 ̂ L t ( θ ) ≼ R 2 I d , which implies the smoothness. Furthermore, if the initial data ensure E S 0 [ φ ( x ) φ ( x ) /latticetop ] ≽ αI d , the ̂ L t ( θ ) is strongly convex and thus is exp-concave as shown by [60, Lemma 2]. We note that the regularity condition on the offline data E S 0 [ φ ( x ) φ ( x ) /latticetop ] ≽ αI d is used in the analysis of LSIF algorithm [28, Lemma 1].

Choice of ψ KL = t log t -t . Considering the Bregman divergence function ψ ( t ) = t log t -t and the hypothesis H θ = { x ↦→ φ ( x ) /latticetop θ | ‖ θ ‖ 2 ≤ S } , where φ : X → R d + represents a specific basis function with bounded norm ‖ φ ( x ) ‖ 2 ≤ R . Then, the loss ̂ L ψ t ( θ ) as per (7) becomes

To ensure ̂ L t ( θ ) is well-defined, we further require the density function ̂ r ( x ) = φ ( x ) /latticetop θ > 1 /β for any x ∈ X and θ ∈ Θ ≜ { θ | ‖ θ ‖ 2 ≤ S } with a certain positive constant β > 0 . The above configuration recovers the UKL method [37] with the generalized linear model. We can show such a choice of divergence function satisfy the condition required by Theorem 1 and enjoys favorable function properties.

Conditions in Theorem 1: When Bregman divergence function is chosen as ψ KL , we have ∂ 2 ψ KL ( t ) = 1 /t and ∂ 3 ψ KL ( t ) = -1 /t 2 . When the input of the loss function satisfies t ≥ β , ψ KL ( t ) is a 1 /β -strongly convex function, and we can also check that t∂ 3 ψ KL ( t ) ≤ 0 for all t ∈ R + .

Expconcavity and smoothness: We have ∇ 2 ̂ L t ( θ ) = E S t [ φ ( x ) φ ( x ) /latticetop / ( φ ( x ) /latticetop θ ) 2 ] . Then, since the output of the density ratio function ̂ r ( x ) = φ ( x ) /latticetop θ ≥ 1 /β for any x ∈ X and θ ∈ Θ and the boundedness of the input feature ‖ φ ( x ) ‖ 2 ≤ R , we have ∇ 2 ̂ L t ( θ ) ≼ β 2 R 2 I d for any θ ∈ Θ . Then, the loss function is a smooth function. Besides, under the regularity condition E S 0 [ xx /latticetop ] ≽ αI d of the initial data and again the boundedness of Θ and ‖ φ ( x ) ‖ 2 ≤ R , the loss function is also a strongly convex function by ∇ 2 ̂ L t ( θ ) ≽ α/ ( SR ) 2 I d , which indicates the expconcavity of the loss function.

E Technical Lemmas

This section presents several useful technical lemmas used in the proof.

Lemma 16. For any a , b ∈ R d , we have ( a + b )( a + b ) /latticetop ≼ 2( aa /latticetop + bb /latticetop ) , where for any matrix A,B ∈ R d × d , A ≼ B indicates B -A is a positive semi-definite matrix.

Lemma 17 (Theorem 4 of [68]) . Let { Z i : i ≥ 1 } be a martingale difference with the filtration F = {F n : n ≥ 1 } and suppose | Z i | ≤ R for all i ≥ 1 . Then, for any δ ∈ (0 , 1) , r > 0 , with probability at least 1 -δ ,

Peer reviews:
comments: SUMMARY:
This work introduced an online density ratio estimation method that can adaptively update the model to minimize the risk accumulated over time in the continuous covariate shift scenario. This method is able to estimate density ratios between test and training samples when the test set is varying over time. Only a few unlabeled samples are required at each time. A theoretical analysis of the regret bound of the density ratio estimator is provided.

STRENGTHS:
The paper is well written with clear justification and useful theoretical analysis. The proposed method relaxes the requirement of unbiasedness of prior work and does not need a large amount of unlabeled data at each time step. The authors also proved the dynamic regret bound of the density ratio estimator. The experiments demonstrate the effectiveness of the proposed method.

WEAKNESSES:
I don't have many complaints about this paper. One weakness is that some important experimental results are in the appendix which is not reasonable as the appendix is optional to read by reviewers. I understand the page limitation but I would suggest moving some preliminary content to the appendix instead.

QUESTIONS:
I understand covariate shift is a well-established research area but the assumption of covariate shift (p(y|x) not change) is quite strong, so how can we verify or guarantee this assumption is valid in a real-world problem?

LIMITATIONS:
Since this method is for continuous covariate shift, the inherent limitation would be the assumption of covariate shift.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an online density ratio estimation method to adaptively train a predictor in the scenario of continuous covariate shift. The proposed method estimates the density ratio between the training and testing distributions using a small number of unlabelled samples and updates the predictor using a weighted empirical risk minimisation algorithm. The paper provides a clear and comprehensive explanation of the problem of continuous covariate shift and the challenges it poses. The paper also provides a thorough review of related work and explains how the proposed method builds on and improves existing methods. The authors also give a dynamic regret bound, which finally leads to an excess risk guarantee for the predictor. The regret bound is minimised by designing an online optimisation process to minimise the dynamic regret defined over the observed loss. The paper explains how to optimise the dynamic regret with the online ensemble framework developed in recent studies of non-stationary online convex optimisation. The proposed method is validated through empirical results on synthetic and real-world datasets, demonstrating its effectiveness compared to other algorithms.\n\nOverall the paper appears technically sound, is easy to follow, and gives theoretical guarantees. The paper is less strong from the empirical perspective. Further, the setting is a little esoteric (offline training followed by online adaptation for models where density ratios can be used), and may be of limited appeal to a wider audience.\n\n----\n\nPost rebuttal increasing my score from 6 to 7\n\nSTRENGTHS:\nFormulating the problem using the Bregman divergence was a useful tool to be able to generalise some existing methods, and also unlocked the analysis that follows. \n\nDisentangling the model training and importance weight estimation also allowed for adaptation to the continuous shift setting and abiility to change importance weight estimators.\n\nWEAKNESSES:\nIn terms of empirical study, this is limited to a study using synthetic data, and a small study on the yearbook dataset. For the synthetic scenario, the nature of the shift - sinusoidal and square waves altering a convex combination of distributions for the first two, then linear and “Ber” (which is not fully described in the main text, but is samples from a Bernoulli distribution) seem rather simplistic and artificial. For the single real world experiment, it’s not clear how the hyperparameter settings for the various methods were chosen, so it’s hard to know if this result is cherry-picked or robust. \n\nAnother thing that’s not clear is whether the intervals chosen for the ensemble members match up with the period of change in the synthetic experiments. If they do, it’s not a surprise that they do well, but would require knowledge ahead of time. \n\nFigures 2-5 are extremely small and hard to read when printed.\n\nQUESTIONS:\nHow sensitive is the method to the intervals in the ensemble members?\n\nHow crucial is the ensemble to the method overall?\n\nAm I right in thinking that the bound does not take account for the presence of the ensemble. Does this mean that the bound actually holds for the algorithm as employed in experiments?\n\nCould you plot the bound for one of the synthetic examples to show how tight it is?\n\nLIMITATIONS:\nThe authors did not address limitations or potential negative social impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the continuous covariate shift problem, where there exists an initial labelled dataset and in every subsequent round, a new unlabelled dataset is revealed. One needs to adapt the model for every round to achieve good performance. The paper uses importance-weighted ERM where the weights are estimated based on Bregman divergence minimization (Sec.3.3) and online ensemble (Sec.4.1). Theoretical analysis shows that the estimation has a linear dynamic regret (Theorem 2) and corresponding IWERM has an average excess risk that depends on problem instances (Theorem 3). Empirical studies on synthetic and real-world data show that the proposed method can outperform several alternatives.\n\nSTRENGTHS:\n- Clear writing and easy-to-follow presentation\n- Strong theoretical guarantees on both the ratio function (Theorem 2) and the learned model (Theorem 3)\n- Convincing empirical demonstrations on both synthetic and real-world datasets\n\nWEAKNESSES:\n- The algorithm and analysis rely on the assumption of using linear prediction models (see, e.g., Eq.(2) where the prediction is linear in w). Linear models are hardly sufficient in many cases, and more powerful models are needed.\n- The work of Baby et al (2023), albeit very recent, should be discussed as it achieves a similar bound to the current work in a different setting.\n- Baby, D., Garg, S., Yen, T.C., Balakrishnan, S., Lipton, Z.C. and Wang, Y.X., 2023. Online Label Shift: Optimal Dynamic Regret meets Practical Algorithms. *arXiv preprint arXiv:2305.19570*.\n- A somewhat minor point to mention is that in the experiment, the description of the Yearbook dataset is not very clear. The appendix only mentions 10 images per round, but it remains unclear how the images are sequentially sampled.\n\nSome minor comments\n- R_t should be defined explicitly after Eq.(1) to avoid confusion (since it is the population version, different from (2), the empirical version)\n- L173: Ref [37] doesn’t mention UKL and ref [25] is KLIEP instead of KLLEP\n- L262: latter -> later\n- L305: minimiers -> minimizers\n- L336: KEIEP -> KLIEP\n\nQUESTIONS:\nQ1: Are there any possible ways to extend the current work beyond linear models?\n\nQ2: How are the images being sampled for the Yearbook dataset?\n\nLIMITATIONS:\nThe assumption of using linear models should be explicitly mentioned.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on deriving theoretical bounds for online density ratio when there exists continuous covariate shift. The formulation is based on the importance-weighted empirical risk minimization, which is a conventional one for covariate shift adaptation. The paper chooses the Bregman Divergence Density Ratio Matching as the method and tries to bound the regret of the to a dynamically changing optimal density ratio. The results are first established for a general convex function class and then instantiated to a logistic regression model. The online ensemble method proposed mimics the previous continuous label shift work. Experiments are conducted on four different synthetic shift patterns and mint/cifar datasets.\n\nSTRENGTHS:\nPro:\n\nThe paper studies an important problem, is very clearly written, and has solid results.\n\nWEAKNESSES:\nCon:\nI am not sure how much the first part of the analysis adds to our knowledge about the continuous covariate shift. This is maybe my bias. To my understanding, if we are in this kind of online learning scenario (shift is changing continuously), the minimization of the cumulative dynamic regret is a very straightforward choice. The question is how to minimize it and whether the theory guides the algorithm design. So, bounding the empirical estimation error using the approximated cumulated regret (theorem 1) is not very informative to me. \n\nAlso, the paper shows that many density estimator function satisfies the assumption and can achieve the bounds. So the main novel part of the proposed methodology seems to be the twist to the FLH algorithm?\n\nTheoretically, even though it is nice to see a general and relatively standard algorithm can achieve the minimax optimal guarantee for the online density ratio estimation under continuous covariate shift, the theoretical contribution seems to be a bit limited: the high probability result (instead of the expected), and the greatly simplified analysis. \n\nThe experimental results can be presented in a better way.\n\nQUESTIONS:\nQuestions:\n\nIt would be nice to see different shift change patterns and how they affect the learning results. But it seems the errors are averaged before showing? Figure 2 only shows different methods but covers 4 kinds of shifts?\n\nLIMITATIONS:\nThere can be more discussions about limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}