Time-Varying Propensity Score to Bridge the Gap between the Past and Present
Authors: Rasool Fakoor
Jonas Mueller
Zachary Chase Lipton
Pratik Chaudhari
Alex Smola
Venue: ICLR 2024 poster (2024)
Keywords: model adaptation to changing data
distribution shift
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: To keep models accurate in the real world, we must regularly update them with relevant data. This paper proposes a new method for doing so.

Abstract: Real-world deployment of machine learning models is challenging because data evolves over time. While no model can work when data evolves in an arbitrary fashion, if there is some pattern to these changes, we might be able to design methods to address it. This paper addresses situations when data evolves gradually. We introduce a time-varying propensity score that can detect gradual shifts in the distribution of data which allows us to selectively sample past data to update the model---not just similar data from the past like that of a standard propensity score but also data that evolved in a similar fashion in the past. The time-varying propensity score is quite general: we demonstrate different ways of implementing it and evaluate it on a variety of problems ranging from supervised learning (e.g., image classification problems) where data undergoes a sequence of gradual shifts, to reinforcement learning tasks (e.g., robotic manipulation and continuous control) where data shifts as the policy or the task changes.

Published as a conference paper at ICLR 2024
TIME-VARYIEE VARYING PROPENSITY SCORE TO BRIDGE THE GAP BETWEEN THE PAST AND PRESENT
Rasool Fakoor', Jonas Mueller Zachary C. Lipton", Pratik Chaudhari', Alexander J. Smola
I Amazon Web Services," 2 Cleanlab, Carnegie Mellon University," 2 Boson Al
ABSTRACT
Real-world deployment of machine learning models is challenging because data evolves over time. While no model can work when data evolves in an arbitrary fashion, if there some pattern to these changes, we might be able to design methods to address it. This paper addresses situations when data evolves gradually. We introduce a time-varying propensity score that can detect gradual shifts the distribution of data which allows us to selectively sample past data to update the model- not just similar data from the past like that of a standard propensity score but also data that evolved in similar fashion in the past. The time-vare varying propensity score is quite general: we demonstrate different ways of implementing it and evaluate it a variety of problems ranging from supervised learning (e.g-- image classification problems) where data undergoes sequence of gradual shifts, reinforcement learning tasks (e.g., robotic manipulation and continuous control) where data shifts as the policy or the task changes.
INTRODUCTION
Machine learning models are not expected to perform well when the test data is from different distribution than the training data. There are many techniques to mitigate the consequent deterioration in performance (Heckman, 1979; Shimodaira, 2000; Huang et al., 2006; Bickel al., 2007; Sugiyama et al., 2007b; 2008; Gretton et al., 2008). These techniques use the propensity score between the train and test data distributions to reweigh the data-and they work well (Agarwal et al., 2011; Wen et al., 2014; Reddi et al., 2015b; Fakoor et al., 2020c;a; Tibshirani et al., 2019) when dealing with single training and test dataset. But when machine learning models are deployed for real-world problems, they do not just undergo one distribution shift' (from train to test), but instead suffer many successive distribution shifts (Lu et al., 2019) e.g., search queries to an online retailer or a movie recommendation service evolve fashion and tastes of the population evolve, etc. Even for problems in healthcare, there are many situations where data drifts gradually, e.g., the different parts of the brain atrophy slowly as the brain ages for both healthy subjects and those with dementia; tracking these changes and distinguishing between them is very useful for staging the disease and deciding treatments (Saito et al., 2022; Wang al., 2023). Viruses can mutate and drift over time, which can make them resistant to existing treatments (Russell, 2016; Callaway, 2020; Harvey et al., 2021).
To build method that can account for changes in data, it is important to have some understanding of how data evolves. If data evolves in an arbitrary fashion, there is little that we can do to ensure that the model can predict accurately the next time instant in general. Settings in which the data distribution drifts gradually over time (rather than experiencing abrupt changes) are particularly ubiquitous 2012; Lu et al., 2019). Assuming observations are not statistically dependent over time (as in time-scries), how to best train models in such streaming/onling learning settings remains a key question (Lu et al., 2019). Some basic options include: fitting/updating the model only the recent data (which is statistically suboptimal if the distribution has not significantly shifted) or fitting the model to all previous observations (which leads to biased estimates if shift has occurred). Here we consider less crude approach in which past data are weighted during training with continuous-valued weights that vary over time. Our proposed estimator of these weights generalizes standard two-sample
In this paper, we will not distinguish between covariate, label, and concept shifts; they will all be referred to distribution shifts and the difference will be clear from context.

---

Published as a conference paper at ICLR 2024
propensity scores, allowing the training process to selectively emphasize past data collected at time t based on the distributional similarity between the present and time t.
This paper studies problems where data is continuously collected from constantly evolving dis- tribution such that the single train/test paradigm no longer applies and the model learns and makes predictions based already seen data to predict accurately test data from future where the learner does not have access to any training data from future. Our main contribution is to introduce simple yet effective time-varying propensity score estimator account for gradual shifts in data distributions. One important property of our method is that it can be utilized in many different kinds of settings- from supervised learning to reinforcement learning- and can be used in tandem with many existing approaches for these problems, Another key property of our method is that it neither has any assumption on data nor requires data to be generated from a specific distribution in order to be effective. We evaluate our proposed method in various settings involving a sequence of gradually changing tasks with slow repeating patterns. such settings, the model not only must continuously adapt to changes in the environment/lask but also learn how to select past data which may have become more relevant for the current task. Extensive experiments show our method efficiently detects data shifts and statistically account for such changes during learning and inference time.
APPROACH
For two probability densities p(æ) and ((x) that are absolutely continuous with respect to the Lebesgue measure on Rdd, observe that
E [((x))] = dp(x) dg (x){(x) dp(x)
dg(x))
= E
(1)) = E ((x) ((z)
(1)
amo dp(
where ((x) is any function. The propensity score ((x) aplz) is the Radon-Nikodym derivative
dg(x)
between the two densities (Resnick, 2013). It measures the relative likelihood of datum x coming from p versus q: in the sequel we will call (xx) the "standard propensity score". We may think of q(x)) as the probability density corresponding to the test distribution and p(z) as that of the training distribution. If E(x) is the loss of a datum, the identity here suggests that we can optimize the test loss, i.e., the left-hand-side, using 3-weighted version of the training loss, i.e., the right-hand-side. In practice, can estimate ((z) via samples from and (Agarwal et al., 2011) as follows. We can create a data set D {x > with = 1 if which can be linear or non- -ineer, parameterized by to classify these samples and obtain
and = -1 if x; ~ q. We fit model go.
i =
min
log (1 + exp((=200(x.)))
2.1 MODELING AND ADDRESSING DRIFT IN THE DATA
Let us now consider the situation when the probability density evolves over time: (Pe()))sTT where t E IR. Assume that we have a dataset D = {a,, tx) b where each datum I; was received at time and therefore I; ~ Pt. (x:) The dataset can contain multiple samples from any of the p, for t T. and there may even be time instants for which there are no samples. Our theoretical development will be able to address such situations. Let p(t) = N-1 Σ 5e, (t) be the empirical distribution of the time instants at which we received samples; here denotes the Dirac-delta distribution. We can now define the marginal on the data
=i=1
p(x) (am)= L-" T
dt p(t) pe()).
Inspired by this expression, we will liken pc(x) p( t). Assumption (Data evolves gradually). The probability density of data later time
px(x)t T
can be arbitrarily different from the past data and in the absence of prior knowledge of how the distribution evolves, cannot predict accurately far into the future. Suppose our goal is build
2

---

Published as a conference paper at ICLR 2024
model that can predict well data from Pr+di (2), i..., small time-step into the future using the dataset D. For the loss f(x), this amounts to minimizing
Ee-pr-ac [[(x)].
(3)
This problem is challenging because we do not have any data from pr+ at training time or for adaptation. But observe Ex-pr+a((s)) [[(x)] equals to
[(x)]+ + / dæ - -
E Pr (a))
{(x) S
E 2<pr(#)
[[=]]+ 2TV (PT (4)
where (V = 2 I dr (x)+a(()) (x))
I is the total variational divergence between the probability densities at time T and dt. and the integrand {(x) is upper bounded by in magnitude (without loss of generality, say after normalizing it). We therefore assume that the changes in the probability density pr are upper bounded by a constant (uniformly over time I); this can allow us to build model for Pr+ using data from (Pi)esT-
Validating and formalizing Assumption 1 This assumption is natural one in many settings and we can check it in practice using a two-sample test to estimate the total variation TV(p (rr) (Gret- ton et al., 2012). We do so for some of our tasks in Sec. 3 (see Fig. 9). We can also formalize this assumption mathematically as follows. Consider a stochastic process (Xe)teR with a transition function Pt. By definition of the transition function, we have that
E [x,()) - P( dy) S
for any measurable function y. The distribution of the random variable X, corresponding to this Markov process is exactly the distribution of data at time instant in our formulation. We can now define the semi-group of this transition function as the conditional expectation Kepp J d' (vy) Pr dy); this holds for any function şp. Such a semi-group satisfies properties that are familiar from Markov chains in finite state-spaces, e.g., K2+s K,K The infinitesimal generator /t. This generator completely defines
of the semi-group K2 is defined as Ap = lim,-- the local evolution of Markov process (Feller, 2008)
(5) where edA the matrix exponential of the generator A. all eigenvalues A are positive, then the Markov process has steady state distribution, i.e., the distribution of our data pt stops evolving (this could happen at very large times and the time-scale depends upon the smallest non-zero eigenvalue of A). On the other hand, if the magnitude of the eigenvalues of A is upper bounded by small constant, then the stochastic process evolves gradually.
Pt -.A Po
The above discussion suggests that our development in this paper, building upon Assumption 1, is meaningful, so long as the data does not change abruptly.
Remark (Evaluating the model learned from data from (P:)esT on test from pr+ We are interested in making predictions on future data, i.c., data from PT+dr- For all our experiments, we will therefore evaluate on test data from "one time-step in the future". The learner does not have access to any training data from 4r+ in our experiments. This is reasonable if the data evolves gradually. Our setting is therefore different from typical implementations of online meta-learning (Finn et al., 2019; Harrison et al., 2020) and continual learning (Hadsell et al., 2020; Caccia et al., 2020).
In the sequel, motivated by Assumption 1, we will build a time- dene-dependent propensity score for pr instead of dr+ Using similar calculation as that of Eq. (1) (see Appendix A for a derivation), we can write our objective as
E
Exwpr(*) [[(x))] Fewp(t)Eawwiee)) [~ T,t) ((x)),
B
where
dpt
w(x, T,t) dp: (z)
is the time-varying propensity score.
Remark 3 (Is the time-varying propensity score equivalent to standard propensity score on (x. t)?). We can define a new random variable & = (z,t) t) with a corresponding distribution p(z) = Pe (t)) However, doing so is not useful for estimating Ex-pr(x) [[(x)] because our objective Eq. (6) involves conditioning on a particular time T (see also Sec. 3.3 for numerical comparison).
3

---

Published as a conference paper at ICLR 2024
2.2 TIME-VARYING PROPENSITY SCORE
Let us first discuss simple situation where we model the time-varying propensity score using an exponential family (Pitman, 1936; Wainwright & Jordan, 2008)
(x po(x:) exp (2, t))
(8)
where go(x, is continuous function parameterized by weights 0. This gives
w(x, T,t) exp (go (zx, T) go (z.
(9)
The left-hand-side can be calculated using Eq. (2) to estimate ga(ar, 1) as follows. We create a modified dataset whose inputs triplets (1,, (1,() with label 1 if L' 11, i.e. the time at which the corresponding 2; was recorded, and a label -1 if = ty.
Remark (Modeling the propensity score using an exponential family does not mean that the data is from an exponential family). Observe that the exponential family in Eq. (8) is used model the propensity (T)/Po(z)) this is different from Pt (z) x exp(ge(r, t)) which would amount to modeling the data distribution as belong to an exponential family.
Observe that in the above method Po(x), which is the distribution of data at the initial time, is essentially unconstrained. We can exploit this "gauge" to choose Po(z) differently and build a more refined estimator for the vary-ig propensity score. We can model (x()) as deviations from the marginal on æ:
((())= = p(z) exp (z)
(10)
This gives
w(r, exp (go T) go(x,
In this case, we create a modified dataset where for each of N samples, there is an input tuple (I, t) with label if 4, i.e., the correct time at which æ; was recorded, and with label -1 if * 44- While creating this modified dataset, we choose all the unique time instances at which some sample was recorded. i.e., E {4+ : (zz, t.) E DJ. The logistic regression in Eq. (2) is fitted to obtain (z t) using this modified dataset. Algorithm 2 gives more details.
is important to note that the distinction between the two approaches is subtle. The Eq, (8) models the deviations of the probability density Pz(z)) from its value at the first time instant Po(z) while Eq. (10) models its deviations from the marginal p(x) = fo dt (()) The nuances between these approaches are subtle when deals with data with varying gradual shift, leading to comparable performance of both methods, as illustrated in Fig.
2.3 A LEARNING-THEORETE PERSPECTIVE
We next discuss a learning theoretic perspective on how we should learn models when the underlying data evolves with time. For the purposes of this section, suppose we have input-output pairs D. {(r,, y/ ) where H; are the inputs and yi E {0, 1} are the binary outputs, that are independent and identically distributed from Pe (z. y) for 1,..., T. Note that data from each time instant are IID, there can correlations across time. Unlike the preceding sections when time was real-vat valled, we will only consider integer-valued time.
Consider the setting where we seek to learn hypotheses To = (ht))=11....7 H that can generalize on new samples from distributions (pr) (P:),=1...., respectively. A uniform convergence bound on the population risk of one ht suggests that if
O 2VC 6)g
then (l() < (h1) + with probability at least 1 - 5; here e (he) the population risk of he on Pr, (1, is its average empirical risk on the m samples from D, (Vapnik, 1998) and VC is the VC-dimension of H he If we want hypotheses that have small average population risk e' (Thh) = T Et= (ht). across all time instants, we may minimize the empirical risk èt é'(h) T-1 ΣT (bx).. As Baxter (2000) shows, if - m ( (VC Tog
(11)
4

---

Published as a conference paper at ICLR 2024
then we have (h) I (h) + for any h. The quantity (T) is generalized VC-dimension that characterizes the number of distinct predictions made by the different hypotheses; it depends upon the stochastic process underlying VC+((). Whether we should use the m samples from pr to train one hypothesis, or use all the m samples achieve small average population risk across all the tasks, depends upon how related the tasks are. If our goal is only the latter, then we can just train on data from all the tasks because
individual distributions p.. Larger the time T, smaller the
(C) & VCH, for all T 2 1.
In the literature on multi-task or meta-leamm veta-learning. this result is often the motivation for learning on samples from all the tasks. We will use this as baseline in our experiments; we call this Everything in Sec. 3. In theory, this baseline effective if tasks are related to each other (Baxter, 2000; Gao & Chaudhari, 2021; Achille et al., 2019) but it will not be able to exploit the fact that the data distribution evolves over time, A related baseline that we can glean from this argument that the one that only builds the hypothesis using data from pr: call this Recent) in Sec. 3.1.1.
Our goal is however not to obtain a small population risk on all tasks, it is instead to obtain a small risk on pr+dr-our proxy for it being the latest task pr. Even if the average risk all tasks of the hypotheses trained above is small, the risk on a particular task pr can be large. (Ben-Dave -BeidDavid & Schuller, 2003) studies this phenomena. They show that if we can find hypothesis h o fa such that ho ft can predict accurately on Pt for all i, then this effectively reduces the size of the hypothesis space E H and thereby entails a smaller sample complexity in Eq. (11) at the added cost of searching for the best hypothesis fa F for each task pt (which requires additional samples that scale linearly with VCr). The sample complexity of such a two-stage search can be better than training on all tasks, training on pr in isolation, under certain cases, e.g., if VCH log FF| for finite hypothesis space F' (Ben-David & Schuller, 2003).
While the procedure that combines such hypotheses to get h ft in the above theory is difficult to implement in practice, this argument motivates our third baseline (Finetune) which first fits a hypothesis on all past tasks (es) and then adapts it further using data from pr
3 EXPERIMENTS
We provide a broad empirical comparison of our proposed method in both continuous supervised and reinforcement learning. We first evaluate our method on synthetic data sets with time-varying shifts. Next, we create continuous supervised image classification tasks utilizing CIFAR-IO (Krizhevsky & Hinton, 2009) and CLEAR (Lin et al., 2021) datasets. Finally, we evaluate our method on ROBEL D'Claw (Ahn et al., 2019; Yang et al., 2021) simulated robotic environments and MuJoCo (Todorov al., 2012). See also Appendices C and D for a description of our settings and more results.
3.1 CONTINUOUS SUPERVISED LEARNING
In these experiments, the goal is to build model that accurately predict future data (under continuous label shift) at t + using only historical data from {po..... Pt as described in Remark 2. It is important to note that all models, including baselines, are trained entirely from scratch each time step. Also, we do not update the model continually, as our main focus is on understanding the the performance of the varying propensity score. The following sections will explain the baseline models, the data sets and simulated shift setups, and the results of the experiments.
3.1.1 BASELINE METHODS
1. Everything: To obtain model used at time f, we train on pooled data from all past times (x)( : < t), including the most recent data from px))). More precisely, this involves minimizing the objective
1/2 1 Σ E(Ozz:). N
( d= (x,(,=s)
(12)
where the loss of the predictions of a model parameterized by weights ő is E(); and we have defined (0)) as the cumulative loss of all tasks up time l.
5

---

Published as a conference paper at ICLR 2024
2. Recent trains only the most recent data, i.e., data from (z))). This involves optimizing
(())) = N
Σ
((): where N'
Σ
(13)
(acitent)
3. Fine-tune: first trains on all data from the past using the objective
arg min
Σ
((6:
(actiss,et))
and then finetunes this model using data from Pt.
We can write this problem
(6)
Σ (oat==))
((d; + 2(Φ o² )
(14)
N
where 2(Φ -0-) is a penalty that keeps close "' e.g., Di(s ((((d-0--= 0(--) = - Sometimes this approach is called "standard online learning" (Finn et al., 2019) and it resembles (but is not the same as) Tollow-The-Leader (FTL) (Shalev-Shwartz, 2012). We call it "Finetune" explicitly state the objective and avoid any confusion due to its subtle differences with FTL. In practice, we implement finetuning without the penalty 52 by initializing the weights of the model to ø'- in Eq. (14).
Remark 5 (Properties of the different baselines). Recall that goal is to build model that predicts on data from Pt+ (I).. It is important to observe that Everything minimizes the objective over all the past data; for data that evolves over time, doing so may be detrimental to performance time t. If we think using a bias-variance tradeoff, the variance of the predictions of Everything, is expected be smaller than the variance of predictions made by Recent, although the former likely to have larger bias. For example. if the data shifts rapidly, then we should expect Every terithhin to have a large bias and potentially a worse accuracy than that of Recent. But if the amplitude of the variations in the data is small (and drift patterns repeat) then Everything is likely to benefit from the reduced variance and perform better than Recent. Fine-tune strikes a balance between the two methods and although the finetuning procedure is not always easy to implement optimally, e.g., the finetuning hyper- parameters can be different for different times. As our results illustrate neither of these approaches show consistent trend in the experiments and highly depend given scenario which can be problematic in practice, e.g. sometimes Everything performs better than both but other times Fine-tune is better (compare their performance in Fig. 1).
Remark 6 (How does the objective change when have supervised learning or reinforcement learning?). We have written Eqs. (12) to (14) using the loss E(0; x;) only for the sake of clarity. In general, the loss depends upon both the inputs z; and their corresponding ground-truth labels y; for classification problems. For problems in reinforcement learning, we can think of our data x; as entire trajectories from the system and thereby the loss ((0; x;) as the standard 1-step temporal difference (TD) error which depends on the weights that parameterize the value function.
We incorporate our time-varying propensity score into Eq. (12) as follows:
(0)
w(r,1,44 (():
(15)
(ast)==))
Note that the only difference between method and Everything is introduction of w into Eq. (12) and all other details (e.g. network architecture, etc.) are exactly the same. w can be trained based on algorithm explained in Sec. 2.1 and Algorithm 2. Important to note that, method automatically detects shifts in the data without knowing them as a priori. This means that our method able to detect these changes without being given any explicit information about when or where they occur.
3.1.2 DRIFTING GAUSSIAN DATA
Here, design continuous classification tasks using periodic shift where samples are generated from a Gaussian distribution with vime-ig means (pt) and standard deviation of 1. In particular, create data set, D {(a), vf) N at each time step i where æf NN4,, 1), pr 14-1 1 dd/10,

---

Published as a conference paper at ICLR 2024
20-
Gurt
Finettna Recent Everything
Relative securacy
Relative accuracy
1:00
2.6-
1.0
3.50
40
100
140
150
230
Time
Time
(a) Gaussian
(b) CIFAR-10a
1.00
Relative accuracy
Relative accuracy
0.50
1.06-
0.00-
1.00
3.50-
2.00
0.70
3.5
200
2000
400
Time
(c) CIFAR-10b
(d) CLEAR data
Figure 1: Relative test accuracy achieved by our method and others vs Everything in continuous supervised learning benchmarks. The x-axes indicate results on test data at each time step t as described in Sec. 3.1.3, and points above the line indicate better performance than Everything. We see that method stands out as the only approach that consistently maintains its effectiveness regardless of the specific setting the diversity of the past data. important note
models these experiments are trained entirely from scratch per time step and each point on these plots represents a different model resulted in large number of experiments (e.g. Fig. Id shows 500 experiments per method). To enhance the clarity, we only display the mean accuracy achieved by the Recent using dashed line, it performs the worst in the image benchmarks.
No 0.5, and label I (uf 1) is assigned if xt > H4, otherwise y O. We change direction of shift every 50 time steps (i.e., set d +- dd) and use N 2000 in our experiments. We run these experiments for 160 time steps. Fig. 1a displays that our method performs much better than Everything and similarly to others. We note that both Recent and Fine-tune are like oracle for this experiment as test time data from p++ (zx) are more similar to the recent data than other historical data. This experiment clearly illustrates how our method enables model training to selectively emphasize data similar to the current time and, importantly, data that evolved in similar fashion in the past.
3.1.3 IMAGE CLASSIFICATION WITH EVOLVING LABELS
We adopt the experimental setting of (Wu al., 2021) for classification under continuous label shift with images from CIFAR-10 (Krizhevsky & Hinton, 2009) and CLEAR (Lin et al., 2021).
CIFAR-10. We split the original CIFAR-10 training data into 90% (train) and 10% (yalidation), and use original test split as is. Following (Wu et al., 2021), we pre-train Resent-18 18 (He et al., 2016) using training data for 100 epochs. In subsequent continuous label shift experiments, we utilize the pre-trained model and only fine-tune its final convolution block and classification (output) layer. We use two different shifts, called CIFAR-100 and CIFAR-10b (see Appendix D.1 for details) and random data splits here.
The data set contains 10 classes with 3000 samples per class. CLEAR is curated from large public data set spanning 2004 to 2014. We utilize unsupervised pre-trained features provided by (Lin et al., 2021) and only use a linear layer classifier during continuous label shift experiments. As they stated, the linear model using pre-trained features is far more effective than training from scratch for this data set (for more, see (Lin et al., 2021)). We average all results over random lain-validation-tess splits, of relative size 60% (train), 10% (validation), and 30% (test).
CLEAR.
Continuous label shifts. We create a sequence of classification tasks wherein the label distributions predefined pattern (refer to Appendix D.1 for details). The
varies at each time step, according pattern repeats slowly, with each new task having a slightly different label distribution than previous task. Here, we train model from scratch each time step l using all training data encountered thus
7

---

Published as a conference paper at ICLR 2024
ROBEL DOOc
HeltChettah Windfel
25
-90
Average Retur
Average Retur
-75-
-100
566 Comed
Time
Time
(a)
(b)
Figure 2: Comparison of the average undiscounted return (higher is better) of our method (red) against baseline algorithms ROBEL D'Claw and Caef-Cheetahh Wend environments. Our method consistently outperforms other methods terms of sample complexity across all environments.
far and evaluate the model test data drawn from label distribution of time + 1. This resulted in significant number of experiments, typically ranging from 100 to 500 different models per benchmark illustrated in Fig. 1. This setup presents significant challenging scenario due to the occurrence of label shift at each time step. Consequently, it becomes crucial for a method to learn how to selectively utilize only data which relevant for the current time step and task. is important note that solely employ the test data during the evaluation phase and never use it to train a model.
method against the baselines across different data sets and shifts. To give further insight into our method, we show in Fig. that our method produces high quality estimates under shifted distribution. Also, in Fig. we present a comparison between our method and Meta- Weight-Net (Shu et al., 2019). All these comprehensive results illustrate that not only does our method outperform others by achieving higher classification accuracy, but it also consistently performs well across various shifts and benchmarks, unlike other methods. For instance, Everything fares poorly in Gaussian experiment than others but it works better than Fine-tune and Recent on continuous CIFAR- 10 benchmark. We also show the comparable performance of our method and Everything in the absence data shifts, highlighting their consistent applicability and effectiveness (see Appendix C for more).
Results. Fig. shows the results of these large experiments, where we compare
3.2 REINFORCEMENT LEARNING WITH CHANGING TASKS
In what follows, discuss how we adopt method into existing RL methods. We use two challenging environments with various settings for experiments this section. For all experiments, we report the undiscounted return averaged across 10 different random sceds.
ROBEL D'Claw. This is robotic simulated manipulation environment containing 10 different valve rotation tasks and the goal is to turn the valve below 180 degrees in each task. All tasks have the same state and action spaces, and same episode length but each rotates a valve with different shape. We build a random sequence of these tasks switching from one to another (after an episode finishes) in repeating patterns. We utilize SAC (Haarnoja al., 2018) as our baseline. In order to incorporate our approach into SAC, we only change Q-value update with w as follows:
Σ
ais,a,TT,, (.t,,,,,))@uu(s..) -r -100 a') + log (oggs(l''ss)) To(ddss)) " , mo(lss") (16)
(s,a.t,e))
where a is entropy coefficient, is the current episode, and (s, a) denote state and action, respectively.
Half-Cheetah Wind We closely follow the setup of (Xie et al., 2020) to create Kalf-Cheetah WindVel environment where direction and magnitude of wind forces on an agent keep changing but slowly and gradually. This is difficult problem as an agent needs to learn a non non-stationary sequence of tasks (i.e. whenever direction and magnitude of wind changes, constitutes as a new task). Since task information is not directly available to the agent, we utilize the recent method of Caccia et al. (2022) and Fakoor et al. (2020c) to learn and infer task related information using context variable (called SAC + Context here). Similar Eq. (16), only change Q-update to add our method. We emphasize that SAC + Context a strong method for non-stationary settings and leads to state-of-art performance in those scenarios (Fakoor et al., 2020c; Ni et al., 2022). Additionally, we use SAC in this experiment as well to underscore the challenging nature of the task.
8

---

Published as a conference paper at ICLR 2024
Accurscy
Score
Oucc
to
58
20
20
Time
(b)
Figure 3: Comparing our method against standard propensity on continuous CIFAR-10 benchmark. The x-axis indicates results on test data at each time step. Fig 3a shows classification accuracy and Fig 3b shows propensity scores. These results clearly demonstrate that our time-varying propensity score appropriately weighs the data as it evolves, providing plausible explanation for the better performance of our method compared the standard propensity score. The standard propensity score unable to identify the evolution of the data, which hinders its effectiveness adapting to changing conditions.
Results. We can see in Fig. 2 that our method offers consistent improvements over the baseline methods. This is particularly noteworthy that although the replay buffer contains data from various behavior policies, our method can effectively model how data evolve over time with respect to the current policy without requiring to have access to the behavior policies that were used to collect the data. It is particularly useful in situations where the data often collected from unknown policies. Note that the only difference between our method and baseline methods is the introduction of w term in the Q-update and all other details are exactly the same. See Fig. 10 and 11 for more results.
3.3 COMPARING STANDARD PROPENSITY WITH OUR METHOD
Now we compare our method with standard propensity scoring discussed in Sec. 2. Fig. shows results of this experiment on CIFAR- R-b benchmark. For a comparison with the Gaussian benchmark, please refer to Fig. 5. These result provide an empirical verification that method works better than standard propensity method and gives us some insights why it performs better. Particularly, we can see Fig. and 5 that our method fairly detects shifts across different time steps, whereas standard propensity scoring largely ignores shifts across different time steps. Moreover, one major issue with standard propensity is which portion of the data considered current and which considered outdated when creating binary classifier as Eq. (2). This challenge becomes more complex when data continuously changing, and what was considered current in one step becomes old in the next. However, method does not suffer from this issue as it organically depends on time.
DISCUSSION
We propose straightforward yet powerful approach for addressing gradual shifts in data distributions: a time-varying propensity score. Much past work has focused on a single shift between training/test data (Lu et al., 2021; Wang & Deng, 2018; Fakoor et al., 2020c; Chen al., 2016) as well restricted forms of shift involving changes in only the features (Sugiyama et al., 2007a; Reddi et al., 201 labels (Lipton et al., 2018: 80 Garg al., 2020; Alexandari al., 2020), or the underlying relationship between the two (Zhang et al., 2013; Lu et al., 2018). Past approaches to handle distributions evolving over time have been considered the literature on: concept drift (Gomes et al., 2019; Souza et al., 2020), survival analysis (Cox, 1972; Lu, 2005), reinforcement learning (shift between the target policy and behavior policy) (Schulman et al., 2015; Wang et al., 2016; Fakoor et al., 2020a;b; 2021), l), (meta) online learning 2021), and task-free continual/inerementt learning (Aljundi et al., 2019; He et al., 2019). However, to best our knowledge, existing methods for these settings do not employ time-varying propensity weights like we propose here. One of the key advantages of our method is its ability to automatically detect data shifts without prior knowledge of their occurrence. Additionally, our method is versatile and can be applied in various problem domains, including supervised learning and reinforcement learning and it can be seamlessly integrated with existing approaches. Through extensive experiments in continuous supervised learning and reinforcement learning, we demonstrate the effectiveness of our method. It is important to note that while we refer to "time" in this paper, our methodology can also be applied to situations where shifts occur gradually over space or any other continuous-value index. The broader impact of this research lies in enhancing the robustness of machine learning techniques in real-world scenarios that involve continuous distribution shifts.
2012; Finn et al., 2019; Harrison al., 2020; Wu et al.,

---

Published as a conference paper at ICLR 2024
REFERENCES
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. Task2vec: Task embedding for meta-learning In Proceedings of the IEEE International Conference on Computer Vision, pp. 6430-6439, 2019.
Deepak Agarwal, Lihong Li, and Alexander Smola. Linear-time estimators for propensity scores. In Proceedings of the Fourteenth International Conference Artificial Intelligence and Statistics, pp. 93-100, 2011.
Michael Ahn, Henry Zhu, Kristian Hartikainen, Hugo Ponte, Abhishek Gupta, Sergey Levine, and Vikash Kumar. Robel: Robotics benchmarks for learning with low-cost robots. In Conference Robol Learning (CoRL), 2019.
Amr Alexandari, Anshul Kundaje, and Avanti Shrikumar. Maximum likelihood with bias-corrected calibration is hard-to-beat label shift adaptation. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 222-232. PMLR, 13-18 Jul 2020.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Proceed- ings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11254-1 1263, 2019.
Jonathan Baxter. A Model of Inductive Bias Learning. Journal of Artificial Intelligence Research, 12: 149-198, March 2000.
Shai Ben-David and Reba Schuller. Exploiting task relatedness for learning multiple tasks. In Proceedings of the 16th Annual Conference on Learning Theory, 2003.
Steffen Bickel, Michael Brückner, and Tobias Scheffer. Discriminative learning for differing training and test distributions. Proceedings of the 24th International Conference on Machine Learning, ICML 07. pp. 81-88. Association for Computing Machinery, 2007. ISBN 9781 9781595937933.
Massimo Caccia, Pau Rodriguez, Oleksiy Ostapenko, Fabrice Normandin, Min Lin, Lucas Page- Caccia, Issam Hadj Laradji, Irina Rish, Alexandre Lacoste, David Vázquez, and Laurent Charlin. Online fast adaptation and knowledge accumulation (osaka): new approach to continual learning. In Advances in Neural Information Processing Systems, volume 33, pp. 16532-16545. Curran Associates, Inc., 2020.
Massimo Caccia, Jonas Mueller, Taesup Kim, Laurent Charlin, and Rasool Fakoor. Task-agnostic continual reinforcement learning: In praise of a simple baseline. arXiv preprint arXiv:2205.14499 2022.
Ewen Callaway. The coronavirus is mutating does it matter?, September 2020. URL https:
//www, nature. com/ a artiriiiiio http://rrclcleeetttoo 0001115---000000000 020-0254000000000000000000000000
Xiangli Chen, Mathew Monfort, Anqi Liu, and Brian D. Ziebart. Robust covariate shift regression. In AISTATS, 2016.
D. R. Cox. Regression models and life-tables. Journal of the Royal Statistical Society. Series B (Methodological), 34(2):187-2200 1972. ISSN 00359246.
Rasool Fakoor, Pratik Chaudhari, and Alexander Smola. P3o: Policy-on policy-off policy opti-
mization. In Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115. pp. 1017-1027, 2020a.
Rasool Fakoor, Pratik Chaudhari, and Alexander J Smola. Ddpg++: Striving for simplicity in continuous-control off-policy reinforcement learning. 1xXiv:2006.15199 2020b.
Rasool Fakoor, Pratik Chaudhari, Stefano Soatto, and Alexander Smola. Meta-q-learning, In International Conference Learning Representations, 2020c.
Rasool Fakoor, Jonas Mueller, Kavosh Asadi, Pratik Chaudhari. and Alexander J. Smola. Continuous doubly constrained batch reinforcement learning. Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
10

---

Published as a conference paper at ICLR 2024
Willliam Feller. An introduction to probability theory and its applications, vol 2. John Wiley & Sons, 2008.
Chelsea Finn, Aravind Rajeswaran, Sham Kakade, and Sergey Levine. Online meta-learning. In Proceedings of the 36th International Conference Machine Learning, volume 97, pp. 1920-1930, 2019.
Yansong Gao and Pratik Chaudhari. An information-geommtrcc distance on the space of tasks. In ICML, 2021.
Saurabh Garg, Yifan Wu, Sivaraman Balakrishnan, and Zachary Lipton. unified view of label shift estimation. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS' 20, 2020. ISBN 9781713829546.
Heitor Murilo Gomes, Jesse Read, Albert Bifet, Jean Paul Barddal, and João Gama. Machine learning for streaming data: state of the art, challenges, and opportunities. ACM SIGKDD Explorations Newsletter, (2 2019.
Arthur Gretton, Alex Smola, Jiayuan Huang, Marcel Schmittfull, Karsten Borgwardt, and Bern- hard Schölkopf. Covariate Shift by Kernel Mean Matching. The MIT Press, 12 2008. ISBN 9780262170055.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Muchine Learning Reseurch, 13(1):723-773, 2012.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International Conference on Machine Learning, pp. 1861-1870, 2018.
Raia Hadsell, Dushyant Rao, Andrei Rusu, and Razvan Pascanu. Embracing change: Continual learning in deep neural networks. Trends in Cognitive Sciences, 24:1028-1040, 12 2020. doi: 10.1016/j t/t.:///.tcs..22000000 2020.tc..222.0.00....
James Harrison, Apoorva Sharma, Chelsea Finn, and Marco Pavone. Continuous meta-learning without tasks. In Advances in Neural Information Processing Systems, volume 33, pp. 17571- 17581. Curran Associates, Inc., 2020.
William T Harvey, Alessandro M Carabelli, Ben Jackson, Ravindra K Gupta, Emma C Thomson, Ewan M Harrison, Catherine Ludden, Richard Reeve, Andrew Rambaut, COVID-19 Genomics UK (COG-UK) Consortium, al. Sars-cov-2 variants, spike mutations and immune escape. Nature Reviews Microbiology, 1977)409-424, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
doi: 10.1 D0.1109/VPP..201..999
Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. arXiv preprint arXiv:1906.05201 2019.
James J. Heckman. Sample selection bias as a specification error. Econometrica, 47(1):153-161, 1979. ISSN 00129682, 14680262.
Jaayan Huang, Arthur Gretton, Karsten Borgwardt, Bernhard Schölkopf, and Alex Smola. Correcting sample selection bias by unlabeled data. In B. Schölkopf, J. Platt, and T. Hoffman (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press, 2006.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009.
Zhiqiu Lin, Jia Shi, Deepak Pathak, and Deva Ramanan. The clear benchmark: Continual learning on real-world imagery. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
Zachary C. Lipton, Yu-Xiang Wang, and Alex Smola. Detecting and correcting for label shift with black box predictors, 2018.
11

---

Published as a conference paper at ICLR 2024
Bo Lu. Propensity score matching with time-dependent covariates, Biometrics, 61(3):721-723 2005.
Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering, 31(12)2336-2363, 2018.
Jie Lu, Anjin Liu, Fan Dong, Feng Gu, João Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering, 31(12):2346-3366 2019.
Nan Lu, Tianyi Zhang, Tongtong Fang, Takeshi Teshima, and Masashi Sugiyama. Rethinking importance weighting for transfer learning. arXiv preprint arXiv:2112.1015,, 2021.
Tianwei Ni, Benjamin Eysenbach, and Ruslan Salakhutdinov. Recurrent model-free RL can strong baseline for many POMDPs. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 16691 16723. PMLR, 17-23 Jul 2022.
E. J. G. Pitman. Sufficient statistics and intrinsic accuracy. Mathematical Proceedings of the Cambridge Philosophical Society, 32(4))567-579 1936.
Sashank Reddi, Barnabas Poczos, and Alex Smola. Doubly robust covariate shift correction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 29, 2015a.
Sashank Reddi, Barnabás Póczos, and Alexander Smola. Doubly robust covariate shift correction. In AAAI, 2015b.
Sidney I Resnick. A probability path. Springer Science & Business Media, 2013.
C.J. Russell. Orthomyxoviruses: Structure of antigens. In Reference Module in Biomedical Sciences. Elsevier, 2016. ISBN 978-0-12-801238-
Yuya Saito, Koji Kamagata, Peter A. Wijeratne, Christina Andica, Wataru Uchida, Kaito Takabayashi, Shohei Fujita, Toshiaki Akashi, Akihiko Wada, Keigo Shimoji, Masaaki Hori, Yoshitaka Masutani, Daniel C. Alexander, and Shigeki Aoki. Temporal progression patterns of brain atrophy in corticobasal syndrome and progressive supranuclear palsy revealed by subtype and stage inference (sustain). Frontiers Neurology, 13, 2022. ISSN 1664-2295.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I Jordan, and Philipp Moritz. Trust region policy optimization. In International Conference Machine Learning, volume 37, pp. 1889-1897, 2015.
Shai Shalev-Shwartz. Online Learning and Online Convex Optimization. 2012.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of Statistical Planning and Inference, 90(2):227-244 2000. ISSN 0378-3758.
Jun Shu, Qi Xie, Lixuan Yi, Qian Zhao, Sanping Zhou, Zongben Xu, and Deyu Meng. Meta- net: Learning explicit mapping for sample weighting. In NeurlPS, 2019.
Vinicius Souza, Denis M dos Reis, Andre G Maletzke, and Gustavo EAPA Batista. Challenges in benchmarking stream learning algorithms with real-world data. Data Mining and Knowledge Discovery, 34(6))1805-13 180--188. 2020.
Masashi Sugiyama, Matthias Krauledat, and Klaus-Robert Müller. Covariate shift adaptation by importance weighted cross validation. Journal of Machine Learning Research, 8(5), 2007a.
Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul Buenau, and Motoaki Kawanabe. Direct importance estimation with model selection and its application to covariate shift adaptation. In Advances in Neural Information Processing Systems, volume 20, 2007b.
Masashi Sugiyama, Shinichi Nakajima, Hisashi Kashima, Paul Von Bunau, and Motoaki Kawanabe. Direct importance estimation for covariate shift adaptation. 2008
12

---

Published as a conference paper at ICLR 2024
Ryan J Tibshirani, Rina Foygel Barber, Emmanuel Candes, and Aaditya Ramdas. Conformal prediction under covariate shift. In Advances in Neural Information Processing Systems, volume 32.
Curran Associates, Inc., 2019.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033. IEEE, 2012.
Vladimir N. Vapnik. Statistical Learning Theory. Wiley-Interscience, 1998.
Martin J. Wainwright and Michael I. Jordan. Graphical models, exponential families, and variational inference. Foundations and Trends0 in Machine Learning. 1(1-2):1-305, 2008. ISSN 1935-8237.
Mei Wang and Weihong Deng. Deep visual domain adaptation: A survey. Nearocompating, 312: 135-153, 2018.
Rongguang Wang, Guray Erus, Pratik Chaudhari, and Christos Davatzikos. Adapting machine learning diagnostic models to new populations using a small amount of data: Results from clinical neuroscience, 2023.
Ziyu Wang. Victor Bapst, Nicolas Heess, Volodymyr Mnih, Remi Munos, Koray Kavukcuoglu, and Nando de Freitas. Sample efficient actor-critic with experience replay. arXiv: 1.01224, 2016.
Junfeng Wen, Chun-Nam Yu, and Russell Greiner. Robust learning under uncertain test distributions: Relating covariate shift model misspecification. In International Conference on Machine Learning, pp. 631-639, 2014.
Ruihan Wu, Chuan Guo, Yi Su, and Kilian Q Weinberger. Online adaptation to label distribution shift. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 11340-11351, 2021.
Annie Xie, James Harrison, and Chelsea Finn. Deep reinforcement learning amidst lifelong non- stationarity, 2020.
Fan Yang, Chao Yang, Huaping Liu, and Fuchun Sun. Evaluations of the gap between supervised and reinforcement lifelong learning on robotic manipulation tasks. In 5th Annual Conference on Robot Learning, 2021.
Kun Zhang, Bernhard Schölkopf, Krikamol Muandet, and Zhikun Wang. Domain adaptation under target and conditional shift, International conference on machine learning, pp. 819-827, 2013.
13

---

Published as a conference paper at ICLR 2024
Appendix:
Time-Varying Propensity Score to Bridge the Gap between the Past and Present
A DERIVATIONS
Using similar calculation that of Eq. (1), write time-varying propensity based objective as follows
E (x))p(s) [((x)] Ex-p(=lr) [((x)]
J ddp(t)T) J.J dp(t)
dp(x)T) dp(()))(x)T)
dp(zlt)
((x) (17)
dy(x[()
J.I. dp(t) dp(a)t)
dp(x(T) ((x)
dp(xlt)
S.aott) / dp(x)t)
dp(xlT) ((x)
dp(zlt)
dp(x:17) dp(zlt)
((x)
Expp(t) E (grmp (cr)
f(x:)
dp.(x)
Eemp(t) (xmpp))))
(T, ()((z)
(18)
can be parameterized by neural network and is trained based on our algorithm explained in the Appendix B.
B ALGORITHM DETAILS
We learn by building binary classifier by utilizing samples from different time steps. In partieular, we need to create a triplet (æ,, t', t) for each of samples the dataset with label 1' equals the time corresponding to F,, i.e., if t' and label -1 if equals the time corresponding to Fg, Lee. t t. These steps are detailed in Algorithm 1.
Algorithm Generate Data
1: Input: D
{x, 1
2: T [1,.., 7
3: Initialize D. =
4: j=l..N do
5: 1,- chosen uniformly at random from T such that ł, + t,
6: if random() >= 0.5 then
7:
z ==1
8:
D. Do u ((2
9:
else
2 ==
11:
D. + D. u f(e
end if
13:
end for
14: Return D,
Once the triplet (xi, (',0) generated, fit the binary classifier by solving the following:
minimiie
Σ
log (1 (1+e- ((olttt)))))))) (acl:tta--o(artttt
(19)
(E)
14

---

Published as a conference paper at ICLR 2024
Neight
1,75-
Mett-teeggt-eett Everytling
Ours
Reletive accuracy
Relative accuracy
1,225-
1.200
0.220
0.925
0.900
102
50
100
20
Time
Time
(a) Gaussian
(b) CLEAR data
Figure 4: Relative test accuracy achieved by our method and others vs Everything on continuous supervised learning benchmarks, The x-axes indicate accuracy on test data at each time step as described in Sec. 3.1.3, and points above the line (-) indicate better accuracy than Everything. The setup of this experiment exactly the same that of Fig. I; Weta-Weight-NNt is added another baseline. We that method effective regardless the specific setting or diversity the past data; particular it performs better than Meta- Weight-Net.
where war, exp (go 7) go (x.(.) ()) indicates our time-varying propensity score described Sec. 2.1. Algorithm 2 presents the detailed implementation steps of our method.
It should be noted that we use the exact same algorithm (Algorithm 2) to learn for both reinforcement learning and continuous supervised learning experiments. The main difference between these cases using of (æ, f) as inputs for supervised learning tasks whereas (s, t) are used as inputs in the reinforcement learning experiments. Here s, a, I, and represent state, action, input data, label, and time respectively. Note although ge be parameterized differently considering its given task (e.g. image classification tasks require 9o to convolutional deep network; however, continuous control tasks in reinforcement learning experiments need fully connected networks), its implementation details remain the same. Moreover, in order to implement Eq. (10) described in Sec. 2.1, we only need change line 8 of Algorithm 2 as follow (all other details remain the same):
DoJ+ V
Σ
log (1 (1+ (1+e-2(00(0.a))
(20)
(z,f2.=)EB
Algorithm 2 Train Time-varying Propensity Score Estimator
1: Input: D (,,,t,}
2: Input M: Number epochs
3: Initialize D,=
4: for j1...MM do
5: D, GGnnerateData(D) (see Algorithm
6: repeat
7: Sample mini-batch B
{(xx t2, t1, 2)} Do log e- (2)
8:
V - V
Σ
9:
0 - a
10: until convergence
11: end for
It is important to point out that although Eq. (20) is used for the implementation of Algorithm 2 in all experiments presented this paper, either method could have been utilized as Fig. illustrates that both method performs similarly.
15

---

Published as a conference paper at ICLR 2024
C ABLATION STUDIES AND ADDITIONAL EXPERIMENTAL RESULTS
Method 1 vs Method 2. We compare performance of our Method Eq. (8) and Method 2 Eq. (10) on continuous CIFAR-10 and CLEAR benchmarks. We can see in Fig. that both methods perform similarly, although Method 1 has slight edge over Method 2. Note use Method 2 for all the
experiments in the paper.
Zoomed-in version of Fig. 1. We provide zoomed-in version of Fig. in Fig. 7 where we also remove Recent and Fine-tune make the plots less crowded. Note that the experiments in these plots are exactly the same as the ones in Fig. but only different visualizations.
No Shift. Here, analyze how our method performs when there is no shift in data and see whether or not it performs as good as Everything (Eq. (12))? To answer this question, we run new experiments the CIFAR-10 benchmark without shift. We build continuous classification tasks from this dataset where we used past data to predict future data points. This is the same setting as other supervised learning experiments in the paper except we do not apply any shift the data. Results of this experiment are shown in Fig. 14. As this experiment shows, our method and Everything perform similarly when there no shift in the data. This experiment provides further evidence about the applicability of our method and shows that our method works regardless of presence of shifts in the data and has no negative effect on the performance.
Number of samples. Our approach involves utilizing fixed number of samples (i.e. in Eq. (19)) to train our time-varying propensity score explained in Algorithm 1. Here, investigate the influence of the sample size on the overall performance of our approach. We conduct new experiments on the CIFAR-10 benchmark, where all other settings consistent with the other supervised learning experiments described in the paper. The results of this experiment can be observed in Fig. 13. These experiments demonstrate that our method consistently outperforms Everything across different sample sizes. This further validates the versatility and effectiveness of our approach in various settings and scenarios.
vary the sample size 50, 100, and 300, while keeping
Additional experiments with more shifts. In order to further validate the effectiveness of our method, conduct additional experiments on CIFAR-10 10c (see Appendix D.1 for more details). The results of this experiment are shown in Figure Fig. 6. Additionally, we run more experiments for our reinforcement learning settings, and the results of these experiments are illustrated in Fig. 10 and 11.
Number of seeds. To assess the sensitivity of supervised learning results to the number of seeds, we conduct supplementary experiments employing 8 different random seeds and compare them with the results obtained from seeds. The outcomes presented Fig. 12 maintain consistency with those derived from 3 seeds. It's worth noting that for all reinforcement learning experiments, we employ 10 seeds.
D EXPERIMENT DETAILS
Implementation Details. The dyper-parameee computing infrastructure, and libraries utilized in the experiments of this paper are presented in Table 2, 1, and Table 3, respectively. It is worth mentioning that we conducted minimal random hyper-parameter paper and mostly follow standard and readily available settings for the experiments whenever it is applicable. We intend to make the code for this paper publicly available upon its publication.
search for the experiments in this
D.1 CONTINUOUS LABEL SHIFT SETTINGS
In this paper, we simulate the continuous label shift process for image classification experiments by adopting the setting introduced in (Wu et al., 2021). However, we modify and adapt their settings to create more challenging scenarios for experiments. In particular, in contrast the setting of simulating label shifts for only two classes as done their study, we extend our label shift simulation encompass all classes, totaling 10 classes in this case:
16

---

Published as a conference paper at ICLR 2024
15-
Relative Score
8
0-
20
60
40
Time
(a)
(b)
Figure 5: Comparing our method against standard propensity on Gaussian benchmark. Fig Sa shows periodic shift which samples are generated from Gaussian distribution with means (14) that vary time; the standard deviation
all times. Fig shows the score assigned by standard propensity score method. We have calculated a propensity score for cach data point based on how similar it is to the samples at time 99, Data points that have similar shift to the data point at time 99 should get high score. For example, the data points times 84, 79, 74, and 69 should have the highest scores, they the most similar to the data point time 99. This experiment clearly demonstrates that time-varying propensity weighs
our
appropriately as it evolves, providing plausible explanation for superior performance of our method compared to that of standard propensity score which is unable to identify the distribution shift which hinders its effectiveness in changing conditions. See Sec. 3.3 for more details.
1.00-
1.0
Relative accuracy
1.00
0.5-
2200-
Spore
0.14
0.85-
0.00
5 102
150 200
d0
dd
250
Time
Time
(a)
(b)
Figure 6: Relative test accuracy achieved by our method and others vs Everything CIFAR-10c. The x-axes indicate results test data each time step L described in Sec. 3.1.3, and points above the line ( (-) on the left indicate better performance than Everything. The y-axis the right displays our time- ameevaryingg propensity scores. These results once again highlight the effectivencss of our method compared to others. is important to note that all models in these experiments are trained entirely from scratch per time step and each point on these plots represents different model and experiment resulted in large number of experiment. To enhance the clarity, only display the mean accuracy achieved by the Recent using dashed line, performs worst the image benchmarks.
L )gt+1
Vi [1,C] vt W
(21)
where T is the number of shift steps per label pair i and i + 1, E [1, C denotes the number of classes (C = 10 our experiments), of Rº is vector class probabilities, and v, TCC For CIFAR- 10 experiments, (CIFAR- 10c). We also use T = 30 for CLEAR experiment. Listing demonstrates simple Python implementation of the equation presented in Eq. (21).
use T (called CIFAR- 10a), T = 6 (CIFAR-1Ob), and T = 30
import numpy as np def create shift ql, q2):
lamb = 1.0 / (T-1)
return np. concateeeee ( [np..eeeee expand, dims (ql (1 lamb t) + q2 lamb
axis=0) for in range (T) axis=0)
Listing I: Python code Eq. (21) for given two classes
17

---

Published as a conference paper at ICLR 2024
Relative accuracy
Relative accuracy
1.00
1.04
1.00
1.01
1.00
50
103
150
200
100
150
200
Time
Time
(a) CIFAR- 1-a
(b) CIFAR-10b
Ours
Everything
Relative accuracy
1.10-
1.05-
1,00
6.5-
(c) CLEAR data
Figure 7: Relative test accuracy achieved by our method vs Everything in continuous supervised learning benchmarks. The experiments these plats are exactly the same as the ones in the Fig. but they are zoomed-in and we remove Recent and Fine-tune to make the plots less crowded. As mentioned before, points above the line (-) indicate better performance than Everything. These plots further highlight the effectiveness of method compared to Everything.
mPn
62-
M
Accuracy
Accuracy
50-
12
03
wlokk
(a) CIFAR-100
CIFAR-10b
100-
38-
80-
Accuracy
Nethod to
Nethod 2: planiing
200
450
Time
(c) CLEAR
Eq. (10) (called Method on continuous CIFAR-10 and CLEAR benchmarks. These results shows average test accuracy across time during test time. In each experiment, the same settings are used and the only variation is the utilization of Methad or Methad 2 for the time-iarying propensity score. These results show that both methods perform similarly, although Method has slight edge Method 2.
Figure Comparing our Eq. (8) (called Method
18

---

Published as a conference paper at ICLR 2024
x.10
1.0-
MMD((, Paul MMDD(P., Poo)
Maximum Mean Discrepancy
0.8-
i if
0.6-
Nrun Whant nans whnt
0.4-
0.2-
100
150
200 250
Time
Figure 9: Maximum Mean Diserepancy (MMD) of true distributions (Pts Pola) and (pt, Pota) where Pola is constructed from past data weighted by our method. Each (bluc) point this plot shows distance between data from (z) and all data from pau == (Pt,(.) ct, Estimates for fou are based on 200 images sampled at each time step from a drifting version of CIFAR-10. As the underlying distributions evolve, their MMD distance grows. To illustrate that our method produces high quality estimates, we also show distance between data from pi and all data from (pr, (x)] which weighted by The small gap between blue and orange plots shows that our method produces accurate estimates.
ROBEL DClaw
2000
1500
Average Roturn
1000
500
-500
Ours
-000
0.00
0.25
0.50
0.75
1.00
1.5
1.50
1.75
2.50
Time
1e6
Figure 10: Comparison of the average undiscounted return (higher is better) of method (red) against baseline algorithm on ROBEL D'Claw. In this environment, our method performs better than standard SAC in terms both sample complexity and stability (compare shaded area our method with SAC, larger means less stable). Note the only distinction between this experiment and the one in Fig. 2 is that in this experiment, we proceed to the next task after repeating each task for 25 time steps, whereas in the experiment of Fig. 2, each task is repeated only for 2 times. Our method works better both settings and this result again verifies the effectiveness of our method practice.
19

---

Published as a conference paper at ICLR 2024
ROBEL DClaw
1000
Average Return
500
-500
Ours
10000
0.00
0.25
0.50
0.75
1.00
125
1.50
1.75
2,00
Time
le5
average undiscounted return (higher better) of our method (red) against baseline algorithm on ROBEL D'Claw. this environment, our method performs better than SAC + Context in terms of both sample complexity and stability (compare shaded area of our method with SAC + Context, larger mcans less stable). Note that the only difference between method and baseline method the introduction of term in the Q-update and all other details exactly the same.
Figure 11: Comparison
1.35
1.00
Relative accuracy
1.00
1,20
2:06
15
0:35-
Relative
0.30
6.00
0.00
0.35-
Finstune
Everything
6.70
Fivsture Everythigg
(a) CIFAR- 10a (3 seeds)
(b) CIFAR 10b (3 sceds)
Relative accuracy
6.66-
6:00
Relative
0.00-
2.00
2.75
70
191
120
100
150
200
Time
Time
(c) 11FAR-10a (8 seeds)
(d) CIFAR- 10b (8 seeds)
Figure 12: Relative test accuracy achieved by our method and others vs Everything with 3 seeds against seeds on continuous supervised learning benchmarks. These results validate the consistency of our method across both 3 seeds and 8 seeds.
20

---

Published as a conference paper at ICLR 2024
Hyppee-asamettes for Algorithm
value
Learning rate Number of Updates (M)
le-4
4
Batch Size
512
4-FC layers with ReLU and BatchNorm
Hyper-parametes of SAC
value
Random Seeds
(0, 1,...9) 9)
Batch Size
256 2
Number of Q Functions
Number of hidden layers (Q)
2 layers
Number of hidden layers (т)
2 layers
Number of hidden units per layer
256
Nonlinearity
ReLU
Discount factor (?)
0.99
Target network (g)) update rate (T)
0.005
Actor learning rate
3e-4
Critic learning rate
3e-4
Optimizer
Adam
Replay Buffer size
1e6
Burn-in period
le4
w clip
1
HalfCheetah-Wind number of episodes to evaluate
50
Robel's number of episodes to evaluate
5
Table 1: Hlyper-paraeent used for all reinforcement learning experiments. All results reported in paper averages repeated initialized with each of the random seeds listed above and run for the listed number of episodes.
Sapeer-arammetrs for Algorithm 2
value
Learning rate
lc-4
Number of Updates (M)
200
Batch Size
512
go for CIFAR-10
5-Conv layers and 1 linear layer
9o for CLEAR
3-FC layers with ReLU and BatchNorm
of CIFAR-IO/CLEAR
value
Random Seeds
10, 2
CIFAR- 10 Network
Resnet-18 18
CLEAR Network
Linear layer
Samples per time step (N)
200
Learning rate
9e-4
Batch Size
128
Optimizer
Adam
clip
Number of training epochs per time step
CIFAR-10 (20), CLEAR (25)
Table 2: Hyper-parameters used for all continuous supervised learning experiments for both CIFAR- 10 and CLEAR tasks. All continuous supervised learning results reported in our paper are averages over three random seeds.
21

---

Published as a conference paper at ICLR 2024
Computing Infrastructure
Machine Type
AWS EC2 16xlare
CUDA Version
11.0
NVIDIA-Driver
450.142.00
GPU Family
Tesla M60
CPU Family
Intel Xeon 2.30GHz
Library Version
Python
3.8.5
Numpy
1.22.0
Pytorch
1.10.0
Table 3: Software libraries and machines used in this paper.
1.08
Relative accuracy
1.00-
Relative accuracy
1.00
1.00-
1.12-
1.
(a) 200 Samples
(b) 50 Samples
Ours Exeything
Relative accuracy
Relative acouracy
1.2
1.01
1.00
Time
100 Samples
(d) 300 Samples
Figure Relative test accuracy achieved by our method and others vs Everything with different sample sizes on continuous supervised learning benchmarks. Consistent results are evident across varying sample sizes, and irrespective of the size of the sample, our methods consistently outperform Everything.
94
82-
Accuracy
80-
Ours
Everything
81
10
25
Time
Figure 14: Average test accuracy (higher better) continuous supervised learning on CIFAR-10 benchmark without label shift. The x-axis indicates different test time steps and the y-axis displays the classification accuracy on test data. We see that our method and Everything perform similarly. This is an important observation showing our method does not negatively affect the performance when there is shift.
22


Reviews:
[{"review_id": "0R5e8vJWEk", "reviewer": "Reviewer_LkU5", "summary": "The authors tackle the problem of gradually evolving data which is an important problem in many real world settings. The paper introduces a time-varying propensity score that can detect gradual shifts in the distribution of the data. This helps with prioritizing samples from past data that evolved in a similar fashion. The proposed approach considers optimizing a beta-weighted version of the training loss. Previous research has mainly focused on handling single shifts between training and test data or limited forms of shift involving changes in features, labels, or the underlying relationship between them. \n\nOne key advantage of this method is its ability to automatically detect data shifts without prior knowledge of their occurrence.The proposed algorithm seems versatile in that it  can be applied in different problem domains, including supervised learning and reinforcement learning, and can be integrated with existing approaches. The authors conducted extensive experiments in continuous supervised learning and reinforcement learning to demonstrate the effectiveness of their method. The broader impact of this research is in enhancing the robustness of machine learning techniques in real-world scenarios that involve continuous distribution shifts.", "questions": "Please see some of the questions in weaknesses.  A few other questions/clarifications are:\n\nHow is the `Finetune` baseline different from typical online-meta learning approaches? Finetune  typically would continue learning  a hypothesis on all past tasks and then adapts it further using data from the new task at hand that is in round T $p_T$. for e.g. in [1] and [2]. \n Is it primarily that in your method the desiderata is essentially different i.e  to obtain a small risk on pT +dt — as opposed to obtaining a small population risk on all tasks. Is it possible to get the best of both worlds instead i.e. get small risk on the latest task/future task while maintaining low risk across ALL tasks that the agent has seen, as otherwise there is a recency bias to only caring about the immediate next data point. \n\nWhen you claim that “, existing methods for these settings do not employ time-varying propensity weights like we propose here.” Can you comment how does your work position with respect to online meta learning settings such as [1] and [2] where the methods are also able to “automatically detect data shifts without prior knowledge of their occurrence” without the knowledge of the number of tasks etc. \n\n\n[1] Nguyen, Nicolas, and Claire Vernade. \"Lifelong Best-Arm Identification with Misspecified Priors.\" Sixteenth European Workshop on Reinforcement Learning. 2023.\n\n[2]  Khetarpal, Khimya, et al. \"POMRL: No-Regret Learning-to-Plan with Increasing Horizons.\" arXiv preprint arXiv:2212.14530 (2022).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Typically the propensity score considered doesn’t account for the time varying aspect - the work offers a sound approach to do so. The method exploits the initial unconstrained distribution of data as a  “gauge” to choose $p_0(x)$ differently. With this, they build a model $p_t(x)$ as deviations from the marginal on $x$.\n\n*   The proposed method considers a learning theoretic perspective to minimize the risk on the immediate next task after T tasks until that point.  The work studies a simple situation where we model the time-varying propensity score using an exponential family. \n*    The experiments cover a broad empirical comparison in both continuous supervised and reinforcement learning including synthetic data sets with time-varying shifts, continuous supervised image classification tasks utilizing CIFAR-10 and CLEAR datasets, and a simulated robotic environment alongside MuJoCo.\n*    The empirical evaluation shows  1) Relative test accuracy achieved by the proposed  method is consistently better than other methods in continuous supervised learning benchmarks. 2) The performance in RL is particularly noteworthy wherein the proposed method can effectively model how data evolve over time with respect to the current policy without requiring to have access to the behavior policies that were used to collect the data. This is achieved by a simple introduction of a weighting scheme as proposed in existing loss.", "weaknesses": "Two major weaknesses/limitations of this work are: \n\n*    The assumption that data evolves gradually which might not be true in many real world situations and limits the scope of application of the work. Consider for e.g. the sudden shifts in the data distribution. Would the proposed approach be extendable to such settings? What do we need to cover this setting as well?\n*   Evaluating on test data seems very counterintuitive as typical settings would not have this privileged information except during training. Can you provide how this specific choice of evaluation can be overcome? I find it quite limiting to do so.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dR7GzyhVtL", "reviewer": "Reviewer_bXrx", "summary": "This paper proposes to learn a time-vary propensity score to combat distribution shift in learning problems in which the distribution of data evolves over time. At a high-level, this method learns a weighting function $\\omega$ which places greater weight on historic data that evolved similarly in the past. Empirically, the time-varying propensity score increases classification accuracy in supervised learning problems and outperforms other propensity score baselines. Moreover, it improves the data efficiency in RL tasks.", "questions": "1. Could the authors explain the CIFAR shifts described by Eq. 21? I'm hoping for a plain English description of the shift.\n\n1. Could the authors comment on the computation expense of each method? I imagine they're all similar in computation cost, since their losses have similar forms.", "limitations": "", "rating": 8, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-motivated and well-written. The problem, solution, and novelty, are all clearly discussed in the intro.\n1. The proposed method is sufficiently general to be applied to both supervised and reinforcement learning problems.\n1. The experimental setup is very clear and provides intuition or how different baselines should perform in different settings.", "weaknesses": "1. The baselines considered for the supervised learning experiments are reasonable, but the RL experiments don't consider these baselines. As written, it's unclear if the time-vary propensity score is any better than simpler methods. Also, ROBEL DClaw is missing the SAC + Context baseline. \n\n1. At the end of Section 2 \"Approach,\" it would aid the reader to include a brief summary of the line of thinking outlined in sections 2.1-2.3. I found Section 2 far easier to understand on my second read largely because I already had an idea of the arguments that followed.\n\n1. It would be useful to give a brief summary of the ablation findings (appendix C) in the main paper. For instance, mention that the proposed method performs just as well as Everything when there is no shift, \n\n1. The shaded regions for the RL curves should be darker; as of now, they are very difficult to see.\n\n**Typos:**\n1. Section 2.1: \"for some time large final time T\"\n2. Missing right parenthesis after Eq. 4: TV(p_{T+dt}, p_T)\n3. Section 2.2 last paragraph: \"is subtle when deals with data\"\n4. Just before Remark 5: \"We call it “Finetune” make the objective\"\n5. Section 3.1.3 last paragraph: \"We run this experiments\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LWvROSQZ99", "reviewer": "Reviewer_evvV", "summary": "This paper proposes a new formulation of propensity scores to handle a time-evolving data distributions. Specifically, they propose to approximate the propensity scores given the seen data, the time the data was received, and the current time. Because the propensity score is conditioned on time, the authors propose this will better weight how related past data is to the current data distributions. The paper then explores the effectiveness of the estimated propensity scores through several classification tasks and in reinforcement learning baseline.", "questions": "1. Is the first assumption the same as saying it is lipschitz smooth/continuous?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method is simple and easily understandable to anyone familiar with variance reduction through importance sampling which are quite common throughout machine learning and reinforcement learning. This approach is seemingly novel as far as I can tell, which shows the need to explore these methods in the continual learning setting.", "weaknesses": "After discussing with the authors, I believe they have adequately addressed my concerns. I am updating my score to account for their changes (and the one addition I just asked for).\n\n\n------ before edits -----\n\nThere are several weaknesses in this paper’s presentation and content. Some of these are less critical than others, but the overall value of the paper’s ideas are undermined by a series of confusing notation choices and overly confusing writing practices. There are also several issues with the empirical section which need clarification or addressing in subsequent submissions.\n\n**Section 2**\n\n1. The first weakness comes from how this work is situated in the literature, specifically the effort to distance this work from related work in continual learning. Specifically, on page 3 in Remark 2. I would argue that your setting is exactly encapsulated by continual learning frameworks. For a framework https://arxiv.org/pdf/2012.13490.pdf is a good resource. Specifically, definition 1 is a good starting point. While these definitions take a reinforcement learning perspective, they have counterparts in supervised settings as well, and can be readily adjusted to the supervised setting.\n2. I’m not sure I’m understanding the construction of your dataset D, and subsequently p(t). To me, each datum is received sequentially and therefore t_i is unique for each entry. This means p(t) is just a uniform distribution for the number of timesteps (of course this is not entirely accurate as t is assumed to be from the reals and not integers, but in effect this is what I understand). Given how you use this distribution, I’m assuming this is not the case. But it is not clear what t is here, how does t relate to how the data was generated? How is your dataset constructed? This confusion continues throughout the paper including all of section 2, and when defining objectives in section 3.\n3. The three lines before remark 4 are very confusing, and don’t clarify how $g_\\theta(x,t)$ is estimated. Do you mean t’ instead of s’? This comes up again after remark 4 under equation 10. It is not clear how the dataset is being generated to estimate g_\\theta, and the subtle difference from equation 8&9 is completely unclear (yes different distributions of the data are used, but how). It is very possible Algorithm 2 will clarify this, but I think a clarification on what the time in the original dataset (see 1) will deal with a lot of the confusion here.\n\n**Empirical section:**\n\n4. You did not explain how relative accuracy is defined. I’m left to assume from the first line of figure 1’s caption that they are normalized by the performance of the everything benchmark, but how is not stated in the main paper. If this is normalized according to the accuracy of the everything baseline, I think these results might be misleading. The paper would benefit from a discussion on how well the baseline is actually performing to give more perspective on the other method’s performance. It is also unclear how the models are being tested in these experiments. Are you testing the accuracy of the model on the data generated at t+1? How is this data gathered? Is this same data then added to the dataset. There are many details missing, likely related to the confusion mentioned above.understanding\n5. **Recent** and **Fine-tune** are not oracles given your problem definition. If you are testing on data generated from t+1, the oracle would be a model trained on data generated from this distribution (this model would represent the best we could hope for).\n6. Hyperparameters and other experimental details:\n   - How were the hyperparameters chosen for your method and for the competitors? This question applies to all experiments as I don’t see it mentioned in the paper. \n   - How does the amount of computation for your method relate to the amount of computation for the baselines (for both training and inference). I notice that your method has a larger batch size than the baselines for both the reinforcement learning and supervised learning settings. Why was this chosen? \n   - How does your method relate in number of updates compared to the baselines? Because of the different training regimes (i.e. the inner loop w/ M for your method), it is unclear how this compares to the epochs.\n   - Are the number of parameters in your models comparable to the baseline parameters.\n   - In the reinforcement learning experiment what are the shaded regions signifying? Also, these should be made darker. Currently it looks like there is significant overlap in both the experiments meaning there is not a statistically significant difference between your method and SAC. \n\n\n# Edits/Suggestions\n- Line right after equation 4 “TV(p_{T+dt}, p_T = “\n- After section 2.2, the paper shifts towards integer-valued time. I think the previous sections could benefit from this treatment as well, focusing on the simpler case of integer time and leaving real-valued time to the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w1MCqvyror", "reviewer": "Reviewer_A4rR", "summary": "The paper proposes a generalisation of propensity scores from a binary setting (i.e. one training distribution and then testing after a single discrete distribution shift) to a continuous one where data drift occurs gradually and thus the propensity scores vary over time. The method is able to detect and react to drifts without being aware of them a priori. The authors offer in-depth theoretical insights as well as an empirical analysis on both supervised as well as reinforcement learning tasks.", "questions": "see weaknesses", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method can deal with continuous data drifts without knowing about them a priori by naturally extending propensity scores to continuous drift settings. Since this type of drift scenario is more likely to occur in practice than prior considered settings such as a single discrete shift, the method is likely useful in real-world problems. In the empirical evaluation, the method performs as good as or better than all considered baselines.", "weaknesses": "I admit I am no expert on dealing with data drift, however I am sure methods other than the proposed one and the baselines (which are all quite naive) exist. While it is good to see that on the given data the developed method outperforms the baselines, it seems to me like a rather low bar and not very surprising. I am worried that a comparison to other relevant (and more sophisticated) approaches to dealing with data drift are left unconsidered, for example I know that a huge area of work is addressing data drift issues with calibration techniques. I think in order to really show the merits of the method, the authors need to consider other state of the art methods of dealing with data drift.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)