Data Drift Correction via Time-varying Importance Weight Estimator
Authors: Rasool Fakoor
Jonas Mueller
Zachary Chase Lipton
Pratik Chaudhari
Alex Smola
Venue: Submitted to ICLR 2023 (2023)
Keywords: distribution shift
data drift over time
propensity scoring

Abstract: Real-world deployment of machine learning models is challenging when data evolves over time. And data does evolve over time. While no model can work when data evolves in an arbitrary fashion, if there is some pattern to these changes, we might be able to design methods to address it. This paper addresses situations when data evolves gradually. We introduce a novel time-varying importance weight estimator that can detect gradual shifts in the distribution of data. Such an importance weight estimator allows the training method to selectively sample past data---not just similar data from the past like a standard importance weight estimator would but also data that evolved in a similar fashion in the past. Our time-varying importance weight is quite general. We demonstrate different ways of implementing it that exploit some known structure in the evolution of data. We demonstrate and evaluate this approach on a variety of problems ranging from supervised learning tasks (multiple image classification datasets) where the data undergoes a sequence of gradual shifts of our design to reinforcement learning tasks (robotic manipulation and continuous control) where data undergoes a shift organically as the policy or the task changes.

Under review as a conference paper at ICLR 2023
DATA DRIFT CORRECTION VIA TIME-VARYING IMPOR- TANCE WEIGHT ESTIMATOR
Anonymous authors Paper under double-blind review
ABSTRACT
Real-world deployment of machine learning models challenging when data evolves over time. And data does evolve over time. While no model can work when data evolves in an arbitrary fashion, if there is some pattern to these changes, we might be able to design methods to address it. This paper addresses situations when data evolves gradually. We introduce novel time-varying importance weight estimator that can detect gradual shifts in the distribution of data. Such an importance weight estimator allows the training method to selectively sample past data-not just similar data from the past like a standard importance weight estimator would but also data that evolved in similar fashion in the past. Our time-varying importance weight is quite general. We demonstrate different ways of implementing it that exploit some known structure the evolution of data. We demonstrate and evaluate this approach on variety of problems ranging from supervised learning tasks (multiple image classification datasets) where the data undergoes a sequence of gradual shifts of our design to reinforcement learning tasks (robotic manipulation and continuous control) where data undergoes a shift organically the policy or the task changes.
INTRODUCTION
Real-world machine learning performance often drops during deployment when test data no longer stem from the same distribution from which previous training data were sampled. Thus many tools have been developed to address data distribution shift' (Heckman, 1979; Shimodaira, 2000; Huang et al., 2006; Bickel et al., 2007; Sugiyama et al., 2007b; 2008; Gretton et al., 2008), often using an estimate of the Radon-Nikodym (Rosenbaum & Rubin, 1983) derivative between the two distributions (also known as a propensity score or importance weight) to re-weight the training data so that its weighted distribution better matches the test data (Agarwal al., 2011; Wen et al., 2014; Reddi et al., 2015b; 5b; Chen et al., 2016; Fakoor et al., 2020b;a; Tibshirani et al., 2019). These methods mostly consider offline settings with one training and one test dataset.
In many applications, data for supervised learning is continuously collected from constantly evolv- ing distribution (c.g. due to pandemic (Callaway, 2020)) such that the single train/test paradigm no longer applies. Settings in which the data distribution drifts gradually over time (rather than experiencing abrupt changes) are particularly ubiquitous (Shalev-Shwartz, 2012). Assuming ob- servations statistically dependent over time (as in time-series), how best train models in such streaming/online learning settings remains key question (Lu et al., 2018). Some basic options include: fitting/updating the model to only the recent data (which is statistically suboptimal the distribution has not significantly shifted) or fitting the model to all previous observations (which leads to biased estimates if shift has occurred). Here consider less crude approach in which past data are weighted during training with continuous-valuee weights that vary over time. Our proposed estimator of these weights generalizes standard two-sample propensity scores, allowing the training process selectively emphasize past data collected at time t based the distributional similarity between the present and time t.
CCovariate, label, and concept shifts are all referred to as distribution shift in the paper to simplify things wherever it is clear from the context.

---

Under review as a conference paper at ICLR 2023
We evaluate proposed method in supervised and reinforcement learning settings involving sequence of gradually changing tasks with slow repeating patterns, In such settings, the model not only must continuously adapt changes the environment/task but also learn how select past data which may have become relevant for the current task. Comprehensive experiments demonstrate that our method can effectively detect shift in data and statistically account for such changes during learning and inference time.
2 RELATED WORK
Given its importance in real-world applications, the problem of how to learn from shifting distributions has been widely studied. Much past work has focused on single shift between training/test data (Lu et al., 2021; Wang & Deng, 2018; Fakoor et al., 2020b) as well as restricted forms of shift involving changes in only the features (Sugiyama et al., 2007a; Reddi et al., 2015a). labels (Lipton et al., 2018; Garg ct al., 2020; Alexandari al., 2020), or in the underlying relationship between the two (Zhang et al., 2013; Lu et al., 2018). Past approaches to handle distributions evolving over time have been considered in the literature on: concept drift Gomes et al. (2019); Souza et al. (2020), reinforcement learning (shift between the target policy and behavior policy) Schulman et al. (2015); Wang et al. (2016); Fakoor al. (2020a), (meta) online learning Shalev-Shwartz (2012); Finn al. (2019); Harrison et al. (2020); Wu et al. (2021), and task-free continual/incremental learning Aljundi et al. (2019); He et al. (2019), but to our knowledge, existing methods for these settings do not employ vime-varying data weights like we propose here. Lime-varying data weights have been considered in survival analysis which models longitudinal observations subject to censoring Lu (2005); Cox (1972), but our weights here are implemented differently (via deep learned estimates) and utilized for general supervised learning and reinforcement learning with drift.
APPROACH
Consider a standard learning problem where training data are drawn from a probability distribution p(x) and test data have a different probability distribution g(x). Our goal is to build a model that can We can do so by observing that
predict equally well the training and test data distributions.
B [[(=)]
dp(x) dp(x)
dq (x)((x) En-p(x)
dg(x)) dp(x)) ((x)) Ex-p(>)
(((x) [(x)]
(1)
where f(z) is any function, say the loss of our model. The propensity score ((x) dg(w) is the
dp(a)
importance ratio; this can also be seen the Radon-Nikodym derivative (Resnick, 2013) of the two distributions. The propensity score measures the likelihood that sample x came from distribution
against it coming from a distribution We will call ((x) the "standard propensity score".
Since densities q(x) and p(x) are often unknown, binary classifier (Agarwal et al., 2011) needs to be used to estimate ((x) by utilizing samples drawn from p and 9. In particular, want create
data set, D {x 24) N for the binary classifier such that 2; 1 z; ~ p and 2; if 2; g.
We want our training data have = 1 in half of the training examples (with p) and
(with x ~ g) in the other half. We fit the binary classifier by solving the following:
minimize
log (1+ exp(- (z+)))
(2)
(11...)
where can be either a linear or non-linear model parameterized by 6.
3.1 TIME-VARYING IMPORTANCE WEIGHT ESTIMATOR
Let us now consider time-varying probability distribution (x(x) with 4= 1,... T. We will assume that we have recorded tuples D {ar, f, 7 (,)) that can predict well on data from (x) using the historical data D, say seek to minimize
with each 2; ~ Pt, (x). Our goal to build a model
E Ex-pr-ae(or [((x))
2

---

Under review as a conference paper at ICLR 2023
Observe that seek to build model not for data from pr but instead from pr+di where will think of dt as small step in the future. This problem is challenging because we do not have data from Pr+
Let us define p(t) = N N:=NN=- "Eim (em,,) which is simply the empirical distribution of the time instants which we received our samples; here 8 is the Dirac-delta distribution. We can now define the marginal on the data as
((a)= = -f di p(1) pe(x))
Inspired by the above expression, we will liken pr p( (t.
Assumption (Data evolves gradually). If data evolves arbitrarily, then it is difficult to expect a model learned from past data to be able to predict accurately on future data. But observe that
E Eeppr [((x)) E pxppp(s [<(x)] / da (pT+ai(1) (r())))((())
(4)
S E Ex-pr(() (x) [<(x)] + 2TV Pr()),
ʃ dr aar - (r)) the total variational divergence be- tween the probability distributions time T and T + dt, and the integrand {(x) is upper bounded by in magnitude (without loss of generality, say after normalizing it). We therefore assume that the changes in our time-varying probability distribution pr(x) are upper bounded by a constant (uniformly over time I); this can allow us build model for 4 using data from {pther
where (V (z),
Validating and formalizing Assumption 1 This assumption is a natural one in many settings and we can check in practice using a two-sample test to estimate the total variation TV (p(++ddt- Pr) (Gret- ton et al., 2012). We do so for some of our tasks in $4 (see Fig. 2).We can also formalize this assumption mathematically as follows. The transition function Pt of a Markov process X, satisfies
- -f" ((,.))(X.,dd)).
E [xp(Xx)) =
for any measurable function so. The distribution of the random variable X, corresponding to this Markov process exactly our Pt- We can define the semi-group of this transition function as the conditional expectation K,y = dy (py) (z, dy) for any test function yp. Such semi-group satisfies properties that are familiar from Markov chains in finite state-spaces, e.g., Kets K,K,- The infinitesimal generator of the semi group K, is defined as Ap = lim, - Ktp)/t. This generator completely defines the local evolution a Markov process (Feller, 2008)
P: = e 1. Po
(5)
where e is the matrix exponential of the generator A. If all eigenvalues of A are positive, then the Markov process has a steady state distribution, i.e., the distribution of our data pt stops evolving (this could happen very large times and the time-scale depends upon the smallest non-zero eigenvalue of A). On the other hand, if the magnitude of the eigenvalues of A is upper bounded by small constant, then the stochastic process evolves gradually. Note it is not necessary for the stochastic process to have a steady state distribution for our varying importance weight be meaningful. Our method is expected to be effective so long as the process evolves gradually, which most processes do.
Remark (Evaluating the model learned from data (p)))) upon test data from ar+ We are interested in making predictions future data, i.e., data from pr+dtt- For all experiments, we will therefore evaluate on test data from "one time-step in the future".. The learner does not have access to any training data from pr+de in our experiments. This is reasonable the data evolves gradually. Note our setting is different from typical implementations of online meta-learning (Finn et al., 2019; Harrison et al., 2020) and continual learning (Hadsell et al., 2020; Caccia et al., 2020).
In the sequel, motivated by Assumption we will build a time-dependent propensity score for pr instead of pr+ Using similar calculation that of (1) (see Appendix A for derivation), can
3

---

Under review as a conference paper at ICLR 2023
write our objective
E (xpp(() (x) [e(x)] Rempe(t) [<(x, T,t) ((x)),
(6)
where
dpr
(xx T,t)=
(x)
dp:
time-varying importance weight estimator.
Remark (Is the time- varying importance weight equivalent to standard propensity (z, t))) One may be inclined to consider whether the development above remains meaningful if we define a new random variable 2 = (x, t) with corresponding distribution p(z) = pe (t)))()). However, doing is not useful for estimating (x))r(x) [<()) because our objective (6) involves conditioning on particular time T'
We again need to estimate this score using samples from our dataset D. To do so, will consider two models for (r).
Method 1: Modeling (x) using an exponential family.
We model the distribution of and as
Pr (x) = Po(x) exp (go t)).
(8)
for some continuous function ge parameterized by weights 0. This gives
w(x, T,t) exp (ge T) go (xx
(9)
Now observe that the left-hand side can be calculated using the same objective that we had in (2) to estimate (x, t). To train the binary classifier, we need to create a triplet (æ4, t', t) for each of samples in the dataset with label 1 if t' equals the time corresponding to 24, i.c., if t, t' and label -1 if equals the time corresponding to 24, i.c. t. These steps detailed in Algorithm 1.
Method 2: Modeling deviations from the marginal Observe that in the above method (ox) is essentially undefined. We can exploit this "gauge" to choose po(x) differently. We can model pt (x) as deviations from the marginal æ:
(((x) = p(z) exp (ge (2,
(10)
This again gives wxx T.t) exp (I, (o, t)). For each of the N samples in the dataset, create datum (2a,t, 4.) with label 1 and another datum (æ,, ty) for j & with label -1. The logistic regression in (2) is now fitted for the classifier ge(x, t) on this dataset.
The difference between the two methods is subtle. In Method 1, the classifier go f) fitted on tuples (xr, (', I) while in Method 2, the dataset only consists of tuples (x,, (). In the former case, think of the fitted model go (x. f) an estimate of log(p(ar, /2/p(, 0)) while in the latter case we think of the fitted model as the estimate of log(p(æ; t) pp)())). In both cases, we can use this function (a t) to calculate our varying importance weight. Because of this, we expect the two methods to perform similarly, as experiments will also show.
3.2 LEARNING MODELS USING DATA THAT EVOLVES ACROSS TIME
We next discuss a learning theoretic perspective how should learn models when the underlying data evolves with time. For the purposes of this section, suppose we have m input-output pairs D. = {(T,, f that are independent and identically distributed from pr(z, y) for = 1....,7 (note that these are data from each time instant, there can be correlations across time). We seek to learn hypotheses h (he))=1..... E H" using these samples that can generalize on new samples from distributions (p:) respectively. We have denoted the joint hypothesis space as H typical uniform convergence bound the population risk of one h, suggests that if
O 2VCC log
4

---

Under review as a conference paper at ICLR 2023
then (ha) I @e(he) + where (1() the population risk of h2 Ps and (1a)) is its average empirical risk on the m samples from D, (Vapmik, 1998). Here VC is the VC-dimension of H a hr.
Consider the setting when we want hypotheses that have a small population risk e e'(h) T-1 - (x(b))). To achieve this, we may minimize the empirical risk (h) = T E (14). Baxter (2000) shows that if
T. = ( (TC (VC
- log
(11)
then we have eh(h)= S (h) +e for any h. The quantity (T) is generalized VC-dimension that characterizes the number of distinct predictions made by the 7 different hypotheses; it depends upon the stochastic process underlying our individual distributions p.. Larger the time T, smaller the (T). Whether we should use the m samples from pr to train one hypothesis, or use all the mT samples to achieve small average population risk across all the tasks, depends upon how related the tasks are. If our goal is only the latter, then we can train on data from all the tasks because
VCH (T) S VCH, for all T 2 1.
This is fundamentally because all the tasks are together used by the learner to learn an inductive bias, and effectively a smaller hypothesis space to fit the hypotheses ì from. Such a procedure is effective if tasks are related each other and there are number of theoretical and empirical procedures to estimate such relatedness; see Baxter (2000); Gao & Chaudhari (2021); Achille et al. (2019) among others. This result is often used as the motivation for training on multiple tasks; it motivates one of our baselines (Everything). The procedure that only builds the hypothesis using data from pr is our second baseline (Recent).
Our goal is however not to obtain small population risk on all tasks, it is instead to obtain a small risk on Pr+de- proxy for being the latest task py. Even the average risk on all tasks of the hypotheses trained above is small, the risk on a particular task pr can be large. Ben-David & Schuller (2003) studies this phenomena. They show that if we can find a hypothesis h o fe such that h o fe can predict accurately on for all I, then this effectively reduces the size of the hypothesis space Te E H and thereby entails a smaller sample complexity in (11) at the added cost of searching for the best hypothesis E F for each task pa (which requires additional samples that scale linearly with VCr)). The sample complexity of such two-stage search can be better than training on all tasks, or training on pr in isolation, under certain cases, e.g., VCH 2 log [F for a finite hypothesis space F (Ben-David & Schuller, 2003).
While the procedure that combines such hypotheses to get h ft in the above theory is difficult to implement in practice, this argument motivates our third baseline (Finetune) which first fits a hypothesis on all past tasks (Pt)eST and then adapts it further using data from pr.
EXPERIMENTS
We provide a broad empirical comparison of our proposed method in both continuous supervised and reinforcement learning settings. We first evaluate our method on synthetic data sets with time- varying shifts. Next, we create continuous supervised image classification tasks utilizing CIFAR- 10 (Krizhevsky & Hinton, 2009) and CLEAR (Lin et al., 202 datasets. Finally, evaluate our method on ROBEL D' D'Claw (Yang et al., 2021) simulated robotic environments and MuJoCo (Todorov et al., 2012) simulator. See also Appendices C and for a description of our settings and more results.
4.1 CONTINUOUS SUPERVISED LEARNING
The goal in these experiments is to build model that can predict well on data from t using only historical data from {po..... Pt] } (see also Remark 2). Note we train all different models, including all baselines, completely from scratch per time step; we do not update the model continually because we focus understanding the performance of the time- varying importance weight only. Moreover, our method automatically detects shifts in the distribution of data without knowing them as priori. In the following, we first explain the baseline models, then the data sets and simulated shift setups. Finally, we discuss the results of these experiments.
5

---

Under review as a conference paper at ICLR 2023
4.1.1 BASELINE METHODS
We compare our approach to the following baseline methods:
Everything: To obtain model used at time t, we train on pooled data from all past times {p)(x): s s t), including the most recent data from P(()). More precisely, this involves minimizing the objective
(Φ)
Σ
Σ
((o;x;).
(12)
N
where the loss of the predictions of a model parameterized by weights ø is ((d; and we have defined (o) as the cumulative loss of all tasks up to time t.
Recent trains only on the most recent data, i... data from (z))). This involves optimizing
(Ø) N
Σ
((d;z;); where N'
Σ
1.
(13)
{
3. Fine-tune: first trains on all data from the past using the objective
arg min
Σ
((o xx)
N
(act,ss,stt)
and then finetunes this model using data from pt.
We can write this problem as
e
Σ (mgstt
((d:x:)) + 2(0 x²-),
(14)
N
where 0(0- 6²-) is penalty that keeps close to c.g. 0 this approach is called "standard online learning" (Finn et al., 2019) and it resembles (but is not the same as) Follow-The-Leader (FTL) (Shalev-Shwartz, 2012). We call "Finetune" make the objective explicit avoid any confusion due its subtle differences with FTL. In practice, implement finetuning without the penalty Ω by initializing the weights of the model to Ø in (14).
= -11---0- 00 Sometimes l|
Remark 4 (Properties of the different baselines). Recall that goal is to build model that
predicts on data from (z). It is important to observe that Everything minimizes the objective over all the past data; for data that evolves over time, doing so may be detrimental to performance at time f. If we think using a bias- bias-variance tradeoff, the variance of the predictions of Everything is expected to be smaller than the variance of predictions made by Recent, although the former is likely to have larger bias. For example, if the data shifts rapidly, then we should expect Everything to have a large bias and potentially a worse accuracy than that of Revent. But the amplitude of the variations in the data is small (and drift patterns repeat) then Everything is likely to benefit from the reduced variance and perform better than Recent. Fine-tune strikes a balance between the two methods and although the finetuning procedure is not always easy to implement optimally, e.g., the finetuning cryper-paramteen can be different for different times. As our results illustrate neither of these approaches show consistent trend in the experiments and highly depend on a given scenario which can be problematic in practice, e.g. sometimes Everything performs better than both but other times Fine-tune better (compare their performance in Fig. 1).
Remark 5 (How does the objective change when we have supervised learning or reinforcement learning?). We have written Eqs. (12) to (14) using the loss f(ő; x2) only for the sake of clarity. In general, the loss depends upon both the inputs x; and their corresponding ground-truth labels for classification problems. For problems in reinforcement learning, we can think of our data v; as entire trajectories from the system and thereby the loss E(); v;)) as the standard 1-step temporal difference (TD) error which depends the weights that parameterize the value function.

---

Under review as a conference paper at ICLR 2023
We incorporate our time-varying importance weight into (12) follows:
Σ
Σ (xp4=s)
(0,, (())
(15)
Note that the only difference between our method and Everything is introduction of w into (12) and all other details (e.g. network architecture, etc.) are exactly the same. w can be trained based on our algorithm explained in $3.1.
4.1.2 DRIFTING GAUSSIAN DATA
In this experiment, we design continuous classification tasks using a periodic shift where samples are generated from Gaussian distribution with time-varying means (1t)) and standard deviation of 1. In particular, we create data set, D {(of, 2f)), at each time step t where æf ~ [uu 1), H4 = 14-1 + d/10, Po = 0.5, and label 1 (0) = 1) is assigned if x > μr otherwise y = 0. We change direction shift every 50 time steps (i.e., set d + dd) and use N = 2000 in our experiments. We run this experiments for 160 time steps. Fig. displays that our method performs much better than Everything and similarly others. We note that both Recent and Fine-tune are like oracle for this experiment as test time data from Pt+1 (z) are more similar to the recent data than other historical data. This experiment makes a clear case for our method as it shows that our method allows the training selectively utilize data similar to the current time and importantly data that evolved in similar fashion in the past.
4.1.3 IMAGE CLASSIFICATION WITH EVOLVING LABELS
We adopt the setting of Wu et al. (2021) to design experiments for classification under continuous label shift with images from CIFAR-10 (Krizhevsky & Hinton, 2009) and CLEAR (Lin et al., 2021).
We split the original CIFAR-10 training data into 90% (train) and 10% (validation), and use original test split as is. Following Wu al. (2021), we pre-train Resent-18 (He et al., 2016) using training data for 100 epochs. In subsequent continuous label shift experiments, we utilize the pre-trained model and only fine-tune last convolution block and classification layer. We average all results over 3 random splits.
CIFAR-10.
CLEAR. This data set consists of 10 classes where each class has 3000 samples. CLEAR is curated over a large public data set only including data spanning from years 2004 to 2014. We utilize unsupervised pre-trained features provided by Lin et al. (2021) and only use a linear layer classifier As Lin et al. (2021) stated, the linear model using pre-trained features is far more effective than training from scratch for this data set (For more, see Lin et al. (2021)). We average all results over 3 random train-validation-tesl splits, of relative size 60% (train), 10% (validation), and 30% (test).
Continuous label shifts. We create a sequence of classification tasks where the label distribution predefined label distribution in slow repeating patterns (see
changes at each time according Appendix C.I for details). In particular, we train a model at time t from scratch using all training data seen so far and evaluate the model on test data drawn from label distribution time + In this setting, training data keeps growing while data stays the same size. This is challenging problem label shift occurs at each time step hence it is critical for method to learn how to only utilize data which are relevant for the current time step and task. Note we only use test data in the evaluation time and never use to train a model.
Results. We report results comparing our method against the baselines across different data sets and shifts in Fig. 1. These results clearly illustrate that our method performs better than others as our method obtains higher classification accuracy across all benchmarks. Importantly, our method shows consistent performances across various shifts and benchmarks whereas other methods do not. For instance, Everything fares poorly in Gaussian experiment than others but it works better than Fine-tune and Recent on continuous CIFAR-I0 benchmark. We also shows Fig. 2 that our method produces high quality estimates under shifted distribution.
7

---

Under review as a conference paper at ICLR 2023
Accurey
Timm
(a) Clusssaa
(b) CIFAR-10a
ye goo Gualo Lndd mnlth
Acsurac
Time
(c) CIFAR-I06
(d) CLEAR
Figure 1: Average test accuracy (higher is better) across runs for continuous supervised learning benchmarks comparing our method with others. The x-axes indicate different test time steps and y-axes display the classification accuracy on test data from time t. CIFAR-I00 10a and CIFAR-IOb refer two different shifts for this benchmark (see Appendix C.I for details). All models in these experiments are trained completely from scratch per time step (i.e. every point in these plots is different model) using exact same architecture and hyper-parameter parameters. sudden drop in plot is an indication that shift direction has changed. see that our method is the only approach that consistently works across different settings and shifts.
HOBEL OC
Halthhenn mide
Assragee
2.
Figure 3: Comparison of the average undiscounted return (higher is better) of our method (orange) against baseline algorithms ROBEL D'Claw and Half-Cheetah Wind environments. In all environments, method performs better than other methods terms sample complexity.
Maximum Mean Discrepancy
MMOL Aud
4.2 REINFORCEMENT LEARNING WITH CHANGING TASKS
as a.-
We use two different environments for exper- iments in this section. For all experiments in this section, we report the undiscounted return averaged across 10 different random seeds.
100
260 20
Time
Figure 2: Maximum Mean Discrepancy (MMD) of true distributions (Pe, Pola) and (Pts Pota) where Pota constructed from past data weighted by our method. Fach (blue) point in this plot shows distance between data from (cr) and all data from paa (Pt,(z) ). Estimates for Pau are based on 200 images sampled at each time step from a drifting version of CIFAR- 10. As the underlying distributions evolve, their MMD distance grows. To illustrate that method produces high quality estimates, data from Pt and all data from [Pe, (z))..< which are weighted by our w. The small gap between blue and orange plots shows that our method produces accurate estimates.
This is robotic simulated manipulation environment containing 10 differ- ent valve rotation tasks and the goal to turn the valve below 180 degrees in each task. All tasks have the same state and action spaces, and same episode length but each rotates a valve with different shape. We build a random sequence of these tasks switching from one to another (after an episode finishes) in repeating patterns. We utilize SAC (Haarnoja et al., 2018) () as our base- line. In order to incorporate approach into
ROBEL D'Claw.
also show distance between
8

---

Under review as a conference paper at ICLR 2023
Accorto
Scom
Now
Standaod Propertty Score
Sandtedinncoste Soom
Time
Time
(ab)
(b)
Figure 4: Comparing our method against standard propensity continuous CIFAR-10 benchmark. Fig 4a shows average accuracy and Fig 4b shows propensity scores for both methods across time during time. These results distinctly show our varying importance weight estimator weighs data the same fashion as they evolve and this can explain why our method performs better than standard propensity score where it can not identify when data evolves.
SAC, we only change Q-value update with w as follows:
Σ
w(s, a(s,a, (.,,,,tt) (:).... - -- + (ol@@sss))))oo@mmllllolo ta alogm((dlss') ) , TT(sss) (16)
(s,a,t,))
where a entropy coefficient, is the current episode, and (s, a) denote state and action, respectively.
Half-Cheetah Vel. We closely follow the setup of Xie al. (2020) to create Half- Cheetah agent keep changing but
WindVel environment where direction and magnitude of wind forces slowly and gradually. This is difficult problem as an agent needs to learn a non-stationary sequence of tasks (i.e. whenever direction and magnitude wind changes, constitutes as a new task). Since task information is not directly available the agent, we utilize the recent method of Caccia al. (2022) and Fakoor et al. (2020b) to learn and infer task related information using context variable (called SAC + Context here). Similar to (16), we only change Q-update to add our method.
Results. We can see Fig. that our method offers consistent improvements over the baseline method and can observe its effectiveness in correcting shifts across time. Note that the only difference between our method and baseline methods is the introduction of w term in the Q-update and all other details are exactly the same. See also Fig. 6 and appendix for additional experiments.
5 COMPARING STANDARD PROPENSITY WITH OUR TIME-VARYING METHOD
Now we compare our method with standard propensity scoring discussed in $3. Fig. 4 shows results of this experiment on CIFAR- 10 benchmark. This result provides an empirical verification that our method works better than standard propensity method and gives some insights why it performs better. Particularly, we can see in Fig. 4 that our method fairly detects shifts across different time steps, whereas standard propensity scoring largely ignores shifts across different time steps.
CONCLUSION
Here we propose a simple yet effective time-varying importance weight estimator to account for gradual shifts in data distributions. One practical property of our drift estimator that it is not specific partieular class of problems and can be utilized in all kinds of settings from supervised learning to reinforcement learning with combination of existing approaches. Importantly, our method automatically detects shift in data without knowing them as a priori. Comprehensive experiments on continuous supervised learning and reinforcement learning show the effectiveness our method. The broader impact of this work will potentially be to make machine learning methods more robust in the presence of continuous distribution shift, which is inevitable in the real world. Note that while we referred to /. time throughout this paper, our methodology also applicable to settings where data-generating shifts occur gradually over space any other continuous-vall valued index.

---

Under review as a conference paper at ICLR 2023
7 ETHICS STATEMENT
All datasets and benchmarks used in this work are publicly available. All code and documentation developed as part of this project will made public. We do not declare any conflicts of interest with respect the findings discussed in this work.
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pictro Perona. Task2vec: Task embedding for meta-lcarning. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6430-6439, 2019.
Deepak Agarwal, Lihong Li, and Alexander Smola. Linear-time estimators for propensity scores. In Proceedings of the Fourteenth International Conference Artificial Intelligence and Statistics, pp. 93-100, 2011.
Amr Alexandari, Anshul Kundaje, and Avanti Shrikumar. Maximum likelihood with bias-corrected calibration is murd-to-beat label shift adaptation. In Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pp. 222-232. PMLR, 13-18 Jul 2020.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11254-11263, 2019.
Jonathan Baxter. A Model of Inductive Bias Learning. Journal of Artificial Intelligence Research, 12: 149 198, March 2000.
Shai Ben-David and Reba Schuller. Exploiting task relatedness for learning multiple tasks. In Proceedings of the 16th Annual Conference on Learning Theory, 2003.
Steffen Bickel, Michael Brückner, and Tobias Scheffer. Discriminative learning for differing training and test distributions. Proceedings of the 24th International Conference on Machine Learning, ICML '07, pp. 81-88. Association for Computing Machinery, 2007. ISBN 9781595937933.
Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Page- Caccia, Issam Hadj Laradji, Irina Rish, Alexandre Lacoste, David Vázquez, and Laurent Charlin. Online fast adaptation and knowledge accumulation (osaka): a new approach to continual learning. In Advances in Neural Information Processing Systems, volume 33, pp. 16532-16545. Curran Associates, Inc., 2020.
Massimo Caccia, Jonas Mueller, Taesup Kim, Laurent Charlin, and Rasool Fakoor. Task-agnostic continual reinforcement learning: In praise of a simple baseline. arXiv preprint 1=XXi:2205.14495 2022.
Ewen Callaway. The coronavirus mutating - does it matter?, September 2020. URL https:
// nature wtomm//rticles//11155 0ttp:/4411660000022555 020-02040
Xiangli Chen, Mathew Monfort, Anqi Liu, and Brian D. Ziebart, Robust covariate shift regression. In AISTATS, 2016.
D. R. Cox. Regression models and life-tables. Journal of the Royal Statistical Society. Series (Methodological), 34(2):187-220 1972. ISSN 00359246.
Rasool Fakoor, Pratik Chaudhari, and Alexander Smola. P3o: Policy-on policy-off policy opti- mization. Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115, pp. 1017-1027, 2020a.
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander Smola. Meta-q-learning. In International Conference Learning Representations, 2020b.
10

---

Under review as a conference paper at ICLR 2023
Willliam Feller. An introduction to probability theory and its applications, vol 2. John Wiley & Sons, 2008.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning In Proceedings the 36th International Conference on Machine Learning, volume 97, pp. 1920-1930, 2019.
Yansong Gao and Pratik Chaudhari. An information distance on the space of tasks. In ICML, 2021.
Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary C. Lipton. A unified view of label shift estimation. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS' 20, 2020. ISBN 9781713829546.
Heitor Murilo Gomes, Jesse Read, Albert Bifet, Jean Paul Barddal, and João Gama. Machine learning for streaming data: state of the art, challenges, and opportunities. ACM SIGKDD Explorations Newsletter, 21(2):6 22, 2019.
Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bern- hard Schölkopf. Covariate Shift by Kernel Mean Matching. The MIT Press, 12 2008. ISBN 9780262170055.
kernel two-sample test. The Journal of Machine Learning Research, 13(1):723-773, 2012. Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf. and Alexander Smola. A
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International Conference on Machine Learning, pp. 1861-1870, 2018.
Raia Hadsell, Dushyant Rao, Andrei Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in Cognitive Sciences, 1a.1028-10400 12 2020. doi: 10.1016/j wiw.:i..sss..........
James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone. Continuous meta-learning without tasks. In Advances in Neural Information Processing Systems, volume 33, pp. 17571 17581. Curran Associates, Inc., 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 770-778, 2016.
doi: 10. 10.1109.CV....016.900
Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A Rusu, Yee Whye Tch, and Razvan Pascanu. Task agnostic continual learning via meta learning. arXiv preprint Siki::1906.05220 2019.
James J. Heckman. Sample selection bias as a specification error. Econometrica, 47(1):133-611, 1979. ISSN 00129682, 14680262.
Jiayuan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. Correcting sample selection bias by unlabeled data. In B. Schölkopf. J. Platt, and Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Zhiqiu Lin, Jia Shi, Deepak Pathak, and Deva Ramanan. The clear benchmark: Continual learning real-world imagery. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
Zachary C. Lipton, Yu-Xiang Wang, and Alex Smola. Detecting and correcting for label shift with black box predictors, 2018.
Bo Lu. Propensity score matching with time-dependent covariates. Biometrics, 61(3):721-728, 2005.

---

Under review as a conference paper at ICLR 2023
Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering, 31(12):2346-25 23663, 2018.
Nan Lu, Tianyi Zhang, Tongtong Fang, Takeshi Teshima, and Masashi Sugiyama. Rethinking 2021.
importance weighting for transfer learning. arXiv preprint arXiv:2 2021.
Sashank Reddi, Barnabas Poczos, and Alex Smola. Doubly robust covariate shift correction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015a.
Sashank Reddi, Barnabás Póczos, and Alexander Smola. Doubly robust covariate shift correction. In AAAI, 2015b.
Sidney Resnick. A probability path. Springer Science & Business Media, 2013.
Paul R. Rosenbaum and Donald B. Rubin. The central role of the propensity score in observational studies for causal effects. Biometrika, (0(1):41-55 04 1983. ISSN 0006-3444.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference Machine Learning, volume 37, pp. 1889-1897, 2015.
Shai Shalev-Shwartz. Online Learning and Online Convex Optimization. 2012.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of Statistical Planning and Inference, 90(2):227-244, 2000. ISSN 0378-3758.
Vinicius Souza, Denis M dos Reis, Andre G Maletzke, and Gustavo EAPA Batista. Challenges in benchmarking stream learning algorithms with real-world data. Data Mining and Knowledge Discovery, 34(6):1805-188 2020.
Masashi Sugiyama, Matthias Krauledat, and Klaus- Robert Müller, Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8(5), 2007a.
Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul Buenau, and Motoaki Kawanabe. Direct importance estimation with model selection and application to covariate shift adaptation. In Advances Neural Information Processing Systems, volume 20, 2007b.
Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul Von Bunau, and Motoaki Kawanabe. Direct importance estimation for covariate shift adaptation. 2008.
Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-basee control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Vladimir N. Vapnik. Statistical Learning Theory. Wiley-Interscience, 1998.
Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey, Nearocomputing, 312: 135-153, 2018
Ziyu Wang, Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. arXiv:1611.0122- 2016.
Junfeng Wen, Chun-Nam Yu, and Russell Greiner. Robust learning under uncertain test distributions: Relating covariate shift to model misspecification. In International Conference on Machine Learning, pp. 631-639, 2014.
12

---

Under review as a conference paper at ICLR 2023
Ruihan Wu, Chuan Guo, Yi Su, and Kilian Q Weinberger. Online adaptation to label distribution shift. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 11340-11351, 2021.
Annie Xie, James Harrison, and Chelsea Finn. Deep reinforcement learning amidst lifelong non-
stationarity, 2020.
Fan Yang, Chao Yang, Huaping Liu, and Fuchun Sun. Evaluations of the gap between supervised and reinforcement lifelong learning on robotic manipulation tasks. In 5th Annual Conference on Robot Learning, 2021.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift. In International conference machine learning, pp. 819-827, 2013.
13

---

Under review as a conference paper at ICLR 2023
A
DERIVATIONS
Using a similar calculation as that of (l), we can write our time-varying importance estimator objective follows
(x)) [(())] Ex-p(=lT) [(())]
( S.J. dp(() dp(())
dp(zlt)
((x) (17)
dp(xl[t)
J. dp(t) dp(x
dp(x(T)
((x)
dp(x)t)
J. aab(t) / dp(x)tt
dp(xl[))
((x)
dp(x)()
dd(xl2))
Ei-p(t) B Bx)m(arle)
{(x))
dp(x)t)
dpr
Eipp(t) B (am (cr)
((x)
dp.(x)
B Berpptt)Bapllle
(I, ())((()
(18)
B ALGORITHM DETAILS
Algorithm 1 describes the detailed implementation steps of our time-varying importance weight estimator. We use the exact same steps for both reinforcement learning and continuous supervised learning experiments. The main difference between these cases is using of (z. y, as inputs for supervised learning tasks whereas (s, a, t) are used as inputs in the reinforcement learning exper- iments. Note although can be parameterized differently considering its given task (c.g. image classification tasks require ge to convolutional deep network; however, continuous control tasks in reinforcement learning experiments need fully connected networks), its implementation details remain the same.
In order to implement Method 2 described in $3, we only need to change line Algorithm 1 as follow (all other details remain the same);
voJ+
Σ (x.12.2)EB
log (1 =(oo(aata)))))
(19)
For all the experiments in this paper, we utilize 19 in Algorithm I implementation. Fig. compares these methods using the continuous supervised learning benchmarks and shows that both method performs similarly.
Algorithm Train Time-varying Importance Weight Estimator
1: Input: D = fu 1
t
2: Input M: Number epochs
3: Initialize D,
4: for j=1...M do
5: 'D. + --enerateData())
6: repeat
7: Sample mini-batch B = [(,, 12, 11, 11, D.
8: VoJ - Vo
Σ
log 1 + (e0e-(lottt)))greeete
(2)
9:
V.J
10: until convergence
11:
end for
14

---

Under review as a conference paper at ICLR 2023
Accuracy
Accuracy
Merhad Merhod
Metholl
Mathoo
10
200
CIFAR-108
(b) CIFAR-10b
100
50-
Accuracy
75-
Method 1: t:cle/47--aen
Method (aio
100
260
300
400
500
Time
(e) CLEAR
Figure 5: Comparing our Method (see (8)) and Method 2 (see (10)) on continuous CIFAR-10 and CLEAR benchmarks. These results shows average test accuracy across time during test time. Both methods use exactly the same setting and the only difference between them is utilizing Method or Method 2 for time-varying importance weight estimator. These results show that both methods perform similarly. although Method has slight edge Method 2.
Algorithm 2 Generate Data
1: Input: D = {I, 4/
2: T= [1,..,
Initialize D. =
4: for j=l..N do
5: f, chosen uniformly at random from T such that t, #t,
6:
if random)) >= 0.5 then
7:
==
8:
D. < D. (13,,
9:
else
10:
-
11:
D. - Do u ((2,, tp,)
12:
end if
13:
end for
Return D.
C EXPERIMENT DETAILS
Implementation Details. Table 3, 2, and Table I show hyper-parameters, computing infrastructure. and libraries used for the experiments in this paper. We did minimal random hyper-parameters search for the experiments in this paper and we mostly follow standard and available settings for the experiments whenever it is applicable. We will open source the code of this paper upon publication.
15

---

Under review as a conference paper at ICLR 2023
Computing Infrastructure
Machine Type
AWS EC2 13.16xl
CUDA Version
11.0
NVIDIA-Driver
450.142.00
GPU Family
Tesla M60
CPU Family
Intel Xeon 2.30GHz
Library Version
Python
3.8.5
Numpy
1.22.0
Pytorch
1.10.0
Table Software libraries and machines used this paper.
C.I CONTINUOUS LABEL SHIFT SETTINGS
To simulate the continuous label shift process for image classification experiments in this paper. follow the setting of Wu et al. (2021) but adopt their settings to more challenging scenarios. In particular, we simulate label shifts for all classes (i.e. 10 classes here) as opposed to two classes in their setting:
t x+1
vi [1, A2 e [1, (1 Flai +
H 99c
(20)
where T is the number of shift steps per label pair i and + 1, [1, 7j. C denotes the number
of classes (C = 10 in our experiments), of Re is vector of class probabilities, and v E R xc For CIFAR-10 10 experiments, we use T (called CIFAR-10a). T = (CIFAR-10b), and T = 30 10c)). We also use 30 for CLEAAR experiment. Sample python implementation of (20) shown in Listing 1.
import numpy as np def create_shiff (T, ql, q2):
1amb = 1.0 (T-1)
return np. won.atenneteeeeeeeccceeeeeeee.ee (wp.expand_ddeellleeee _irddddddms_em (ql (1 1amb t) + q2 lamb
t, axis-0) for for t in range (T),, axis=C)
Listing Python code of (20) for given two classes
D MORE RESULTS
Number of seeds. We repeat some of supervised learning experiments of CIFAR-10 benchmark using 8 different random seeds (we increase the number of seeds from 3 to 8) and report results in Fig. As these new experiments show, results are consistent across and seeds. We also like to emphasize that all our reinfrocement leatning experiments are done for 10 seeds which is standard for MuJoCo environments.
We compare performance of our Method (8) and Method 2 (10) on continuous CIFAR-10 and CLEAR benchmarks. We can see Fig. that both methods perform similarly, although Method 1 has slight edge Method 2. Note experiments in the paper.
Method 1 vs Method 2.
Method 2 for all the
Here, we analyze how our method performs when there is no shift in data and see whether or not performs as good as Everything? To answer this question, we run new experiments on the CIFAR-10 benchmark without shift. We build continuous classification tasks from this dataset where we used past data to predict future data points. This is the same setting as other supervised learning experiments in the paper except we do not apply any shift to data. Results of this experiment are shown in Fig. 10. As this experiment shows, method and Everything perform similarly when
No Shift.
16

---

Under review as a conference paper at ICLR 2023
ROBEL DClaw
2000
Average Return
500
0-
-500
Ours
SAC
11000
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2,00
Time
1e6
Figure 6: Comparison the average undiscounted return (higher is better) of our method (orange) against baseline algorithm on ROBEL D'Claw. In this environment, our method performs better than standard SAC in terms of both sample complexity and stability (compare shaded area of our method with SAC, larger means less stable). We see that method has much more stable performance than SAC. Note the only difference between this experiment and the one Fig. that here we only move next after repeating each task for 215 time steps whereas cach task repeats only for 2 time in the experiment of Fig. Our method works better in both settings and this result again verifies the effectiveness of our method in practice.
ROBEL DClaw
1500
1000
Average Return
500
Ours
SAC Context
0.00
0.75
1.25
1.50
1.75
2,00
Time
166
Figure 7: Comparison of the average undiscounted return (higher is better) of our method with context (orange) against baseline algorithm with context on ROBEL D'Claw. In this environment, our method performs better than SAC Context in terms of both sample complexity and stability (compare shaded area of method with SAC Context, larger means less stable). Note that the only difference hetween our method baseline method is introduction of w term in the Q-update and all other details are exactly the same.
17

---

Under review as a conference paper at ICLR 2023
Hyper-parameters for Algorithm
value
Learning rate Number of Updates (M)
le-4
4
Batch Size
512
ge
4-FC layers with ReLU and BatchNorm
paraamtpraaeters of SAC
value
Random Seeds
{0, 1...... 9}
Batch Size
256
Number of Q Functions
2
Number of hidden layers (Q)
2 layers
Number of hidden layers (T)
layers
Number of hidden units per layer
256
Nonlinearity
ReLU
Discount factor (?)
0.99
Target network (+)) update rate (r)
0,005
Actor learning rate
3e-4
Critic learning rate
3e-4
Optimizer
Adam
Replay Buffer size
le6
Burn-in period
le4
w clip
1
HalfCheetah- number episodes to evaluate Robel's number of episodes to evaluate
50
5
Table 2: parameerasssers used for all reinforeement learning experiments. All results reported in our paper are averages over repeated runs initialized with each of the random seeds listed above and run for the listed number of episodes.
Hyper parameters for Algorithm
value
Learning rate
le-4
Number of Updates (M)
200
Batch Size
512
ge for CIFAR-10
5- S-Conv layers and linear layer
go for CLEAR
3-FC layers with ReLU and BatchNorm
Hyper paremntrrseserss of CIFAR-IO/CLEAR
value
Random Seeds
1,
CIFAR-10 10 Network
Resnet-18 18
CLEAR Network
Linear layer
Samples per time step Learning Batch Size
200
9e-4
128
Optimizer
Adam
clip
Number of training epochs per time step
CIFAR-10 (20), CLEAR (25)
Table 3: Hyper-parameters used for all continuous supervised learning experiments for both CIFAR- 10 and CLEAR tasks. All continuous supervised learning results reported our paper are averages over three random seeds.
there is shift in the data. This experiment provides further evidence about the applicability of our method and shows that our method works regardless of presence of shifts in the data and it has no negative effect on the performance.
18

---

Under review as a conference paper at ICLR 2023
100
Accurery
Score
Our Recert Phsture
Our
Time 10
250
Time 150
Figure 8: Average test accuracy (higher is better) across runs for continuous supervised learning on CIFAR-10c (see Appendix C.1 for details) benchmark comparing our method with others. The x-axes indicate different test time steps. The y-axis the left displays the classification accuracy test data and the the right shows time-varying importance weight scores. Note all models in these experiments trained completely from scratch per time step (i.e. every point in this plot is different model) using exact same architectures and
A sudden drop in plot is an indication that the shift direction has changed. We see that our method is the only method showing consistent behavior across different benchmarks and shifts, and importantly performs the best overall.
Mad your yoo
your
yoon
Anquracy
Accuracy
0-
(a) CIFAR-IOa (3 soods)
(b) CIFAR-IOB 1Ob (3 sceds)
Mad your your
y youre
Trom
Accuracy
Accurscy
30
Time
Time
CIFAR seeds)
seeds)
Figure Comparison average test accuracy (higher is hetter) running with seeds against seeds on continuous supervised learning benchmarks. The x-axes indicate different test time steps and the y-axes display the classification accuracy on test data from time f. CIFAR- 10a and CIFAR-10R 10b refer two different shifts for this benchmark (see Appendix C.I for details). We that results consistent across 3 seeds and 8 seeds.
19

---

Under review as a conference paper at ICLR 2023
Accuracy
82-
86
Everything
84
25
Time
Figure Average test accuracy (higher is better) across runs for continuous supervised learning on CIFAR- R-10 benchmark without label shift. The x-axis indicates different test time steps and the y y-axis displays the classification accuracy on test data, We see that our method and Everything perform similarly. This is an important observation showing our method does not negatively affect the performance when there shift.
20


Reviews:
[{"review_id": "p-QngO4Q2H", "reviewer": "Reviewer_Ytji", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This draft studies the problem of continual learning with streaming data in the presence of distribution change. The data distribution is changing over time. The proposed method uses a time-varying importance weight estimator to correct the distribution change. Experiments on online supervised learning and reinforcement learning show the effectiveness of their method.", "summary_of_the_review": "This draft studies an interesting but important problem in real-world machine learning tasks, especially with the explosion of big data, which are collected over time and the distribution can also change over time. The proposed method is simple, but it lacks novelty. Assumptions on how data evolve are strong and do not accurately reflect reality. Moreover, the proposed method requires the storage of past data, which limits its application to real-world tasks. While the experiments seem good, the datasets used do not match the assumptions made in the method section.", "strength_and_weaknesses": "Strengths:\n- This draft studies a well-motivated problem that is very common in real-world applications.\n\nWeakness:\n- The description of the proposed method is not very clear. Section 3 is difficult to follow. \n- The proposed method appears to need to first store all the data, then adaptively estimate the weights for all the past data, then optimize. This is not exactly a streaming data learning approach. How does the proposed method differ from offline methods like domain generalization?\n- The authors propose two models for estimating $p_t(x)$, the exponential family and deviations from the marginal. Assumptions on the data evolution process are strong. These generational models are not well-discussed. How are these scenarios used, for instance, in what applications? Moreover, these two data generation assumptions do not match the experiments in which the authors use a label shift dataset.\n- In the experimental part, the authors use a label shift dataset. It is suggested to compare with the baseline method in Wu et al. (2021).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "NuQelguUar", "reviewer": "Reviewer_LTNU", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method to automatically evaluate the contribution of a data point from some historical distribution to a new task without knowing the exact distributions of both old data and the new task. Experiental results show that the proposed wighting method works consitently better than several baseline methods on some simulated data sets.", "summary_of_the_review": "This paper proposes a neural network to estimate the importance sampling weight. But large variance is one critical issue of any importance sampling weight estimators, so it's better to further consider the effect of variance when using the proposed methods. Another point is that more realistic illustration of the effect of the proposed method is lacking.\n", "strength_and_weaknesses": "Strength: 1. it will be very convenient if we are able to evaluate the contribution of a data point to a new task without knowing exactly which distribution it were sampled from.\n 2. the idea of using a DL method for importance weight estimating is interesting.\n\nWeaknesses:\n1. the theoretical properties are lacking - The issue of how the variation of the estimator is influenced by the data and how to controll the variation is not discussed in the paper.\n\n2. there are many other methods to do the same thing (importance evaluating), e.g., using influence functions - some clarifying about the difference to those methods is needed.\n\n3.  the supervised learning experiments are mostly based on simulated data without real data shifting -  something like recommendation applications may be considered where people's interests are changing constantly.\n\n4. for the RL part - since the data is depended on what the underlying policy is, I don't think that learning a weighting network without considering the policy that generate the data is very appropriate - those policies are not necessarily changing graduately.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "L37X8w3LoJd", "reviewer": "Reviewer_UiAM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a novel time-varying importance weight estimator that can detect gradual shifts in the distribution of data. They evaluate their methods on synthetic image classification and reinforcement learning tasks that exhibit distribution shifts and show that the proposed method largely outperforms the standard methods (training on all data, \"recent\" data only, and fine-tuning on the recent data).", "summary_of_the_review": "The derivations are sound and the proposed model is very interesting, but the evaluation lacks a bit of depth and real-data experiments. ", "strength_and_weaknesses": "The method is very well described, and the results on synthetic data are very compelling. \n\nWeaknesses: \n\nI find that the \"CONTINUOUS LABEL SHIFT SETTINGS\" paragraph in the appendix is not detailed enough. It is hard to get a sense of the distribution shift that is induced in the data. Maybe adding some visualization here would be great. \nAlso, there is a public \"corrupted\" CIFAR10 dataset that allows to get CIFAR10 data, with different levels of corruptions. It might be useful (and more standard) to use this dataset for further analysis. \n \nAn important component of supervised learning models is their uncertainty representation. if the proposed method correctly approximates the distribution shift, it should be reflected in the uncertainty representation and in the calibration of the model. How well does the model perform in terms of ECE, for example, and how does the uncertainty evolves as distribution shifts? I think such analysis would strengthen the paper. \n\nFinally, I would have liked to see real data experiments. There are some \"common\" settings and datasets for distributional shifts. For example, in neuroscience, decoding neural data during learning would be one (the distribution of the input changes as the animal learns the task) or in climate science (the basic models do not accurately forecast weather or temperature as the CO2 emissions increase, for example)\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FWRDt9gjql", "reviewer": "Reviewer_TuzH", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper deals with handling changes in non-stationary environments. The authors assume that data change gradually over time: this idea is formally made explicit using a constant to upper bound the total variation divergence of the distributions of data in a small amount of time. The assumption thus allows predicting samples at time T+dT (in the future) using a time-dependent propensity score built on samples at time T. The authors try to reuse past samples (t<=T) in order to make predictions on samples at time T by using a mechanism based on importance weighting. The weights are computed using the output of a probabilistic binary classifier that maps each sample to the time it was seen, that coincides with a specific distribution of data. In the end, the authors test their approach in both continuous supervised and reinforcement learning settings comparing it with different baselines.", "summary_of_the_review": "The paper tackles the important problem of non-stationarity of data using the standard technique of importance weighting enriched with a time component in order to help the mapping of each sample to its associated time of arrival. The main ideas and assumptions presented in the paper are sound. The use of an importance-weighted estimator provides an unbiased estimate of the samples arriving at time T, which is a good proxy for the samples to come in the near future. Nevertheless, the concerns raised make me opt for a borderline negative score at the moment.", "strength_and_weaknesses": "**Strengths**\n- The approach is simple to understand\n- The approach is applicable to different settings from supervised to reinforcement learning\n\n**Weaknesses**\n- The approach seems computational (and possibly sample) demanding due to the need for retraining the binary classifier (for the weights) and of the prediction model at each time step.\n- The experimental evaluation does not succeed in showing a clear advantage of the proposed method", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CHIQW7sMkW", "reviewer": "Reviewer_FwNn", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper deals with the problem that the behavior of the system to be modeled may change during the time of data acquisition. For the case that the behavior changes only gradually, a method is proposed, theoretically justified and empirically tested, which is intended to achieve good model quality for the time shortly after the end of the data acquisition.", "summary_of_the_review": "A well-written paper on an important topic that proposes, theoretically justifies, and empirically tests a new method. An increase in significance through a larger number of experiments and an investigation of how the method performs in the case of stationary data is desirable.", "strength_and_weaknesses": "**Strengths**\n* The topic is relevant in practice\n* The method is theoretically justified and embedded in a broad context of previous work\n* Systematic studies on this topic are still rare, in my opinion.\n* The results are convincing\n\n**Weaknesses**\n* As far as I understand it, each experiment is repeated only three times, so the statistical significance of the results is somewhat unclear.\n\n**Further suggestions for improvement**\n* In some places boldmath should be used so that formula expressions are also written in bold, e.g. Remark 2, 3; Method 1; Figure 2 \n* Please check if the sentence \"The goal in these experiments is to build model that\" is really meant that way, or if it should be \"a model\" or \"models\".\n* Figure 2 partially covers the words \"sample cpmplexity\" above.\n* In the bibliography, the first names of the authors are not written out in some entries, as is the case in the vast majority. E.g. J. Baxter, D.R. Cox.\n* Please check the author list for \"Alex Krizhevsky, , and Geoffrey Hinton.\"\n\n**Further comments**\n* It would be interesting to apply the method in an example where there is NO shift, i.e., where all data stems from the same distribution and thus the baseline method \"Everything\" is optimal. Can the proposed method achieve the same performance in this setting?\n* The practice of exponentially downweighting older data relative to newer data is not new. Perhaps the authors can clarify this or provide a reference.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AAHUorBJiT5", "reviewer": "Reviewer_gsqz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the setting of training under a gradual distribution shift over time. To mitigate the effect of distribution shift on the performance on future data, they propose applying importance weighting to simulate training under the most recent data distribution. They motivate this by arguing that the latest distribution will be similar to the next future time step since the change in distribution happens gradually. The main contributions of this paper are \n- A formalisation of training under distribution shift\n- A method to estimate importance weights that depend on time\n- Experiments in a supervised and RL setting to showcase the benefits of the method", "summary_of_the_review": "The setting proposed is promising and an important one to study. The idea presented is simple (in a good way) and well motivated. However, the practical implementation of the algorithm seems to have too much overhead and the chosen benchmark datasets aren't different enough from already well established continual learning settings. ", "strength_and_weaknesses": "This paper focuses on the important setting of learning under continual distribution shifts. Indeed data naturally comes as a sequence and waiting until enough data is gathered to marginalize over time can be impractical in many applications. On top of that the authors make a strong case for the weakness of the marginal approach if one is interested in predicting the immediate future. I've found most of the paper to be clearly written; The transition from train/test propensity scores to time dependent ones was easy to follow, even when one isn't familiar with importance weighting and propensity scores. And the formalization of \"gradual\" shift seems to be coherent and valid in many cases. Therefore using p_T as a proxy for p_T+dt seems reasonable.\n\nHowever, the experimental setting seems a little limited since 1) it requires access to the whole histore of the data 2) starts training from scratch at each time step. I can hardly imagine a setting where this is feasible in practice or for bigger models / larger datasets. The benefits on the datasets chosen except Gaussian (which might be too toyish) might not justify the overhead of training the discriminative model to estimate importance weights. \nThe paper focuses on data that evolves through time but also assumes access to multiple samples per time step that are IID. The paper seems to try to be separate from the setting of continual learning but ends up using a sequence of datasets with known boundaries (the timestesp). I'm not 100% sure where the difference lies. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject