Online Continual Learning for Progressive Distribution Shift (OCL-PDS): A Practitioner's Perspective
Authors: Runtian Zhai
Stefan Schroedl
Aram Galstyan
Anoop Kumar
Greg Ver Steeg
Pradeep Natarajan
Venue: Submitted to ICLR 2023 (2023)
Keywords: Distribution shift
Continual learning
Domain Adaptation
Semi-supervised learning
Model drift
Online learning
Benchmark

Abstract: We introduce the novel OCL-PDS problem - Online Continual Learning for Progressive Distribution Shift. PDS refers to the subtle, gradual, and continuous distribution shift that widely exists in modern deep learning applications. It is widely observed in industry that PDS can cause significant performance drop. While previous work in continual learning and domain adaptation addresses this problem to some extent, our investigations from the practitioner's perspective reveal flawed assumptions that limit their applicability on daily challenges faced in real-world scenarios, and this work aims to close the gap between academic research and industry. For this new problem, we build 4 new benchmarks from the Wilds dataset, and implement 12 algorithms and baselines including both supervised and semi-supervised methods, which we test extensively on the new benchmarks. We hope that this work can provide practitioners with tools to better handle realistic PDS, and help scientists design better OCL algorithms.

Under review as a conference paper at ICLR 2023
ONLINE CONTINUAL LEARNING FOR PROGRESSIVE DISTRIBUTION SHIFT (OCL-PDS): A PRACTITIONER'S S PERSPECTIVE
Anonymous authors Paper under double-bling review
ABSTRACT
We introduce the novel OCI.-PDS problem Online Continual Learning for Pro- gressive Distribution Shift. PDS refers to the subtle, gradual, and continuous dis- tribution shift that widely exists in modern deep learning applications. is widely observed in industry that PDS can cause significant performance drop. While previous work in continual learning and domain adaptation addresses this prob- lem to some extent, our investigations from the practitioner's perspective reveal flawed assumptions that limit their applicability on daily challenges faced in real- world scenarios, and this work aims to close the gap between academic research and industry. For this new problem, we build 4 new benchmarks from the Wilds dataset (Koh al., 2.2T), and implement 12 algorithms and baselines including both supervised and semi-supervised methods, which we test extensively on the new benchmarks. We hope that this work can provide practitioners with tools to better handle realistic PDS, and help scientists design better OCL algorithms.
INTRODUCTION
In most modern deep learning applications, the input data undergoes continual distribution shift over time. For example, consider a satellite image classification task as illustrated Figure Ta In this task, the input data distribution changes with time due to the changes in landscape, and camera updates which can lead to higher image resolutions and wider color bands. Similarly, in a toxic language detection task on social media illustrated in Figure lb. the distribution shift can be caused by a shift in trends and hot topics (many people post about hot topics like BLM (Wikipedial contributors) 2022a) and Roe v. Wade (Wikipedia contributors 2022b) on social media), or a shift in language use (Röttger & Pierrehumbert] 2021] Luu et al. 2022). ). Such distribution shift can cause significant performance drop deep models, a widely observed phenomenon known as model drift.
A critical problem for practitioners, therefore, is how to deal with what we term progressive distri- bution shift (PDS), defined as the subtle, gradual, and continuous distribution shift that widely exists in modern deep learning applications. In this work, explore handling PDS with online continual learning (OCL), where the learner collects, learns, and is evaluated on online samples from continually changing data distribution. In Section 2
we formulate the OCL-PDS problem.
The OCL-PDS problem closely related to two research areas: domain udaptation (DA) and con- tinual learning (CL), in which there is rich body of academic work. However, through literature review and our conversations with practitioners, we find that there still remains gap between the settings widely used in academic work and in real industrial applications. To close this gap, commit ourselves to thinking from a practitioner's perspective, which the core spirit of this work. Our primary goal is to build tools for investigating the real issues practitioners are facing in their day-to-day work. To achieve this goal, we challenge the prevailing assumptions in previous work, and propose three important modifications to the conventional DA and CL problem settings:
Task- One point conventional DA and CL settings have in common is assuming clear boundaries between distinct domains (or tasks), but practitioners rarely apply the same model to very different domains in industry. In contrast, OCL studies the task-free CL setting (Aljundi et al. 2019b) where there is clear boundary, and the distribution shift is continuous. Moreover, in OCL both training and evaluation are online, unlike previous task-free CL. settings with offline evaluation, which is not as realistic in a "lifelong" setting.

---

Under review as a conference paper at ICLR 2023
BLM
Roe v. Wade
Distribution Shift with Time
Distribution
Shift with
Anather
Hot Topics
paper
church
heretical
fallures
Label
Airport
Stadium
Airport
Stadium
Stadium
Label
Toxic
Non-toxic
Toxic
Year
2002
2009
2016
Topic
Race
Religion
Gender
(a) FMo WPPS benchmark.
(b) CivilComments -antneeenttt--tSSSSS-- benchmark.
Figure 1: MMWW-WPDS and CivilComments- WPDS benchmarks which we build in this work.
2. Forgetting is allowed:
Avoiding forgeting is a huge topic in CL, which usu- ally requires forgetting all tasks. However, remembering everything is actually im- practical, infeasible and potentially harmful, so OCL-PDS only requires remembering re- cent knowledge and important knowledge which is described by regression (Sec. (.2].
3. Infinite storage: Previous work in CL usually assumes limited storage (buffer) size. However, storage is not the most pressing bottleneck in most industrial applications. Thus, in OCL-PDS, we assume infinitely large storage where all historical samples can be stored. However, the learner cannot replay all samples because it would be too inefficient.
To demonstrate the novelty and practicality of the OCL-PDS problem, in Section 2.3 we will discuss related work, compare OCL-PDS with common and similar settings used in previous work, and elaborate on the reasons why we believe these three key modifications align OCL-PDS more closely with industrial applications and practitioners' pain points. more thorough literature review can be found in Appendix A Then, in Section 3 we will build 4 new benchmarks for OCL-PDS, including both vision and language tasks. When building these benchmarks, we make every effort to make sure that they can reflect real PDS scenarios that practitioners need to deal with.
In Section 4 we will explore OCL algorithms and how to combine them with semi-supervised learning (SSL) as unlabeled data is very common in practice. In total, we implement 12 supervised and cupervuper OCL algorithms and baselines adapted to OCL-PDS, which we test extensively on our benchmarks in Section 5 Our key observations in these experiments include: (i) A task- dependent relationship between learning and remembering; (ii) Some existing methods have low performances on regression tasks; (iii) SSL helps improve online performance, but it requires critical virtual update step. Finally, in Section we discuss remaining problems and limitations.
Contributions. Our contributions in this work include: (i) Introducing the novel OCL-PDS prob- lem which more closely aligns with practitioners' needs; (ii) Releasing new benchmarks for this novel setting; (iii) Adapting and implementing 12 OCL algorithms and baselines, including both supervised and semi- semi-supervised, for OCL-PDS; (iv) Comparing these algorithms and baselines with extensive experiments, which leads number of key observations. Overall, we believe that this work is an important step in closing the gap between academic research and industry, and we hope that this work can inspire more practitioners and researchers to investigate and dive deep into real- world PDS. To this end, we release our benchmarks and algorithms, which are easy to use and we hope can help boost the development of OCL algorithms for handling PDS.
2 THE OCL-PDS PROBLEM
2.1 PROBLEM FORMULATION
We have stream of online data S1, where each S, is batch of i.i.d. samples from distri- bution D, that changes with time continuously, for which we assume that Div(D, D(+1) < for all for some divergence function Div. Online Continual Learning (OCL) goes as follows:
At = 0, receive labeled training set Sy, on which train the initial model fo For / = 1, .T,... do
1. Data collection: Receive a new unlabeled data batch S. {xx (1) 9i ())))) l.d. Dr-
2. Evaluation: Predict on S, with the current model fi-1. and get some feedback
3. Fine-tuning: Update the model ft + ft with all previous information
Evaluation metrics. An OCL algorithm is used for fine tine-tuning and is evaluated by three metrics:
1. Online performance: Denote the performance of fa on S, by A.. The online performance at time t (as computed in Step 2 Evaluation) is Af and the average online performance before horizon T' is defined as (A} +... + 1/T.
2

---

Under review as a conference paper at ICLR 2023
2. Knowledge retention: Unlike conventional CL that requires no forgetting on all tasks, in OCL- PDS the model only needs to remember two types of knowledge: recent knowledge and important knowledge. For a certain recent time window w, the recent performance is defined as (AІ_ +...+ //=1)/w. /w. The important data is described by a regression set, and the regression set performance is the model's performance on this set.
Training efficiency: This is measured by the average runtime of the fine-tuning step, which is very important for this online setting where the OCI. algorithm is run for many times.
2.2 DETAILS
Divergence function. For distributions P and Q, Div(P || Q) is the divergence, P to Q, and can be different from Div(Q || P). According to our reasoning in Appendix B an ideal divergence function for OCL-PDS should be asymmetric and bounded, so we cannot use popular functions such as total variation, Wasserstein distance, MMD. KL-divergence and JS JS-divergence. In this work, we use the E-KL-divergence (Eqn. (2) whose definition and properties can be found in Appendix B
Data batch. If |||| then this the conventional online learning setting where samples arrive one by one. However, industry practitioners seldom update the model over a single sample, and always collect a batch of samples before fine-tuning the model, so we consider data batches instead.
Feedback. Without any feedback from the evaluation process, the problem is fully unsupervised because we only have the unlabeled batches to fine-tune the model. This setting, however, is too hard and unrealistic. Indeed, for most deployed industrial systems, there are tools for evaluating online performance, either through some automated metries or feedback provided by end-users. Here, we consider the user error report model, where a fraction of the users provide feedback on incorrect model outputs. This model leads to Random Label Feedback (RLF), where the labels of a fraction of the samples in S₁ are provided feedback. In this scenario, correspond to the unsupervised, supervised and semi-supervised learning settings, respectively.
a 1 and a (0, 1)
All previous information. We allow the learner to store all previously seen samples and feedback (though the learner cannot really replay all samples as would be too inefficient), which is starkly different from most previous papers in CL. that assume limited storage size.
Recent knowledge. OCL requires no forgetting on recent knowledge, because (i) in general, practi- tioners expect the model not forget too quickly, and (ii) in many applications the same distribution repeats periodically, which makes recent knowledge useful. For example, satellite images in sum- mer and winter look very different (e.g. due to snow), but the images in two consecutive summers look similar, so in this case it is useful to remember the knowledge for at least one year.
Regression set. In the software industry, regression refers to the deterioration of performance after an update (Yan et al. 2021). The regression set contains the regression data on which making a mistake is more expensive. Moreover, the labeling function P(Y of the regression data changes very little over time if any (no concept shift). The two most common types of regression data in industry are: (i) Frequent data, which appears more often than other data; (ii) Critical data, which weighs more in the model evaluation and whose definition depends on the specific application.
2.3 RELATED WORK AND COMPARISON WITH PREVIOUS SETTINGS
This work is related to three areas: Domain adaptation (DA), continual learning (CL), and semi- supervised learning (SSL). DA provides learner with labeled samples from a source distribution P and (partially labeled, unlabeled, or no) samples from target distribution Q, and requires it to learn a good model on Q. Surveys on DA include Lu et al. (2018; ; Wang & Deng 2018); Ramponi & Plank (2020): [ang et al. (2022b). In CL., the learner needs to continually learn new knowledge from an online stream of data, and settings include task-incremental CL. (including domain-incremenn and class-incremennal CL), task-aware CL, task-agnostic CL, task-free CL and OCL. Surveys on CL include De Lange et al. (2021): I): Masana et al. (2020); Biesialska et al. (2020). SSL requires the model to learn from training set consisting of few labeled samples and many unlabeled samples. Surveys on SSL include Van Engelen & Hoos (2020); Ouali et al. (2020); Yang et al. (202T). A
more thorough literature review can be found in Appendix A
There are several differences between the OCL-PDS problem and conventional DA and CL settings. We now explain why formulation more relevant and thus useful for industry practitioners.
3

---

Under review as a conference paper at ICLR 2023
PDS vs DA/DG. Domain adaptation (DA) and its sibling domain generalization (DG) mostly study big, one-shot distribution shifts, i.e. training and testing on two very different domains. This has two problems: (i) Practitioners seldom directly apply a model to a very different domain in industry. Instead, they usually train one model for each domain, and train a domain classifier to distinguish among different domains; (ii) Even if model needs be applied would usually first collect some labeled data from the new domain and then fine-tune the model. It very rare to have no labels or samples at all from the target domain in industry.
different domain, practitioners
On the contrary, OCL-PDS is a very common scenario in industry. First, PDS has been widely reported to cause performance drop in industrial applications (Martinel et al. 2016 Jaidka et al. 2018 Huang & Paul 2019). and practitioners do not often train new models for PDS. Second, OCL- PDS studies the semi-supervised setting where the practitioners can collect a few labeled samples and many unlabeled samples, which is more realistic than having no labels or samples at all.
OCL vs CL. OCL falls under the task-free continual learning setting where there a fixed task and no clear task boundary. It is different from conventional musk-incremental CL in three ways:
1. iesseincremental CL has N distinct tasks and requires single model to learn them all, but what practitioners would usually do in this case is train N models, one for each task. In contrast, in OCL-PDS the task is fixed but the data distribution is gradually and constantly changing, so it is more reasonable to use and continually fine-tune a single model.
2. Conventional CL requires the model to remember all N tasks with a storage of size M. For studying PDS, this requirement has three problems:
(i) is not so practical as not all old knowledge is important A satellite image classifier in 2022 does not need to do very well on images in 2002 with different landscapes.
(ii) In the "lifelong" setting with N continually growing, we need M to also grow with N (like M = ONN). With a fixed M, it infeasible to remember everything.
(iii) Many applications have concept shift where P(Y|X) could change, so remembering old knowledge can be harmful to the performance on the current data distribution. For instance, languages that were not considered offensive 20 years ago are widely recognized as offensive today thanks to the recent civil rights movements.
Thus, OCL-PDS only requires remembering recent knowledge and important knowledge.
3. Our setting assumes infinite storage unlike previous settings. This is an over-optimistic assumption as there
real applications where the amount of data is so huge that it is impossible to store all data even for big companies. Real applications also have other con- siderations such as privacy restrictions so that the data cannot be stored forever. However, storage size is rarely the bottleneck of industrial applications. The point of making this assumption is to not put too much effort into utilizing every bit of storage. Instead, we want to focus on more practically relevant questions, such as how to leverage unlabeled data and how to improve training efficiency.
Other similar settings. First, OCL is different from the task-free CL. formulated in some previous work (Aljundi et al. 2019b] de Masson d'Autume et al. 22119) Wang et al. 2022d) where the training online but the evaluation offline. Previous work typically splits the data domain into different sections which the learner sees sequentially online, and in the end the learner is evaluated on the entire domain offline. On the contrary, both training and evaluation in OCL are online: For each new batch, the model is first tested and then trained on it. Thus, it is possible to have the real "lifelong" learning setting in OCL where the time horizon T = de but not in the previous setting.
Second, OCL-PDS is closely related to reinforcement learning (RL) and time series analysis. The difference from RL is that in RI,, the agent can learn from a number of episodes, while in OCL-PDS the evaluation is online and one-pass. The difference from time series analysis is that time series focuses on predicting on future data and does not care about forgetting. Morcover, though PDS naturally resides in time series data, in our literature review (Appendix A find little work about handling PDS with time series analysis. One such line of work is temporal covariate shift (TCS) (Du et al. 2021) that assumes that P(Y X) is always fixed, which is not assumed in OCL-PDS.
Third, there are two related settings, gradual domain adaptation (GDA) (Kumar et al. 2020) and gradual concept drift (Liu et al. 2017 Xu & Wang 2017), that also study gradual shift from one
4

---

Under review as a conference paper at ICLR 2023
domain to another with series of distributions Po. Pi.... Pr, where Po is the source domain, Pr is the target domain, and Pt and P.+1 is close for each t. Both settings only require good adaptation performance and do not consider forgetting. However, the concept of regression set widely exists in modern deep learning applications, and no forgetting on the regression set is a critical issue.
Finally, Cai et al. 2021 proposed similar OCL setting, where the model is also first evaluated on the new batch and then fine-tuned on it. However, OCL-PDS has three important distinctions: 2021 considered fully supervised setting while OCL-PDS mainly uses semi- supervised setting that is more common in practice; (ii) OCL-PDS uses different evaluation metrics, and in particular measures the critical regression set performance; (iii) Cai al.] (202T) evaluates knowledge retention only at three time steps whereas OCL-PDS is purely online and evaluates all metries at all time steps. A more detailed comparison can be found in Appendix A.2.1
(i) Cai
BENCHMARKS
In search of benchmarks for this novel OCL-PDS problem setup, we first investigate the benchmarks used in previous work on continual learning and find that most of them are one of the following: (i) Multiple datasets, egg. Task is MNIST (LeCun & Cortes 2010), Task 2 is SVHN (Netzer et al.) 2011 and so on; (ii) Perturbed image classification, where different tasks are constructed by rotating the images with different angles, shifting the colors with different scales or permuting the pixels with different random seeds; (iii) Split-class classification, e.g. split the 100 classes in CIFAR-100 (Krizhessky et al., 2009) into 20 groups with 5 classes per group, and make each group a 5-way classification task. None of them is realistic enough to represent the real PDS in practice.
There are two existing PDS benchmarks: CLOC (Cai et al. 2021) and CLEAR (Lin et al. 202
Both are image classification tasks and do not have regression sets. Thus, we build a new, more comprehensive suite of benchmarks that covers both language and vision tasks, and both classifi- cation and regression tasks, with intuitively defined regression sets. Since it is impossible for us to cover all existing tasks, we also provide our 3-step procedure to build our benchmarks, which can be used to construct PDS benchmarks on other existing datasets.
3.1 THE 3-STEP PROCEDURE TO BENCHMARK OCL-PDS
Here we provide the 3-step procedure we use to benchmark OCL-PDS on an existing dataset:
1. Separate the data into groups (domains). For example, group the data by year. Then, do an 0OD check which verifies that there is a significant distribution shift across the groups. |
2. Assign shifting group weights to the batches to create a distribution shift across the groups Then, do a shift continuity check which verifies that the shift is continuous (not abrupt).
3. Design a separate regression set which does not intersect with any online batch. Then, do regression check which verifies that na've methods have regression on this set.
Moreover, for each batch (including the regression set), we randomly divide the batch into a training batch and test batch. The initial training set contains both the first batch and the training regression set. The learner sees the training batches during OCL, and the separate test batches are used to eval- uate the recent and regression set performances. The model is never evaluated training samples it has already seen because it is not useful as the learner can store all these samples in its buffer.
Here we briefly demonstrate this procedure and detailed description can be found in Appendix C. The CivilComments dataset contains online comments with topic labels, and we want to model PDS with shifting hot topics on it. In Step 1, we divide the comments into four groups according to their topics, and verify that a model trained on any three groups does poorly on the fourth group; In Step 2, we construct a weight shift among the groups to simulate PDS, and verify that model trained on labeled samples from previous distributions can do well on the new distribution, so that the shift is continuous; Step 3, we define the regression set, and verify that catastrophic forgetting will happen if we only train on the new data.
Example: CivilComments-PPe
'This is not equivalent to group shift or subpepulation shift as studied in fair machine learning and long- tailed learning (learning with imbalanced classes). The group weights here are used to control the scale of the divergence from being too big. We simulate gradual shift by continuously shifting the group weights.
5

---

Under review as a conference paper at ICLR 2023
Table I: Benchmarks we build. 7 +
total number of batches. recent time window.
Benchmark
Description
Regression Set
T+1 1
CiviCoomennts-VV WPPD
Toxic language detection social media
Critical data
16
5
FMoW-WPHH
Satellite image classification for facilitics
Frequent data
25
6
Amazon- WPDS
Review sentiment analysis on Amazon.ccm com
Frequent data
18
6
Poverty-WPPD -WDD
Satellite image regression for wealth index
Critical data
14
5
3.2 OUR BENCHMARKS
We build 4 new benchmarks following the guidelines in Appendix See Table 1 for a summary. All 4 benchmarks are based on the Wilds datasets (Koh et al. 2021 Please refer to this paper for the potential leverage, broader context and ethic considerations of these datasets. Here we briefly describe the benchmarks we release in this work, and details can be found in Appendix C
Civlmmmeetts--Wi WPDS. This benchmark is based on the CivilComments Wilds dataset (Borkan iet al. 2019), which is toxic language detection task on social media (Figure[Ib) WPDS stands for Wilds-PDS. This benchmark models the shift in hot topics. The regression set contains critical data -oomments with severe harassment, including identity attack and explicit sexual comments.
This benchmark is based on the FMo Wilds dataset (Christie et al. 2018), which a satellite image facility classification task (Figure Tap. models the shift in time. The regression set contains frequent data Data from two highly populated regions: Americas and Asia.
EMoW-WPDS.
Amazon-WPDS, This benchmark is based on the Amazon-Wile dataset (Ni et al., (019), which is review sentiment analysis task shopping websites. This benchmark models the shift in language use. The regression set contains frequent data Data from 10 popular product categories.
Deverty-WPDS. This benchmark is based on the PovertyMap- Wilds dataset (Yeh al. 2020)
which is a satellite image wealth index regression task. This benchmark models the shift in time. The regression contains critical data Images from urban areas. Following Koh al. 202 performances here are measured by the Pearson correlation between outputs and ground truths.
4 OCL ALGORITHMS
An OCL algorithm consists of the following three components:
Continual fine-tuning: How to fine-tune the model the new data?
Knowledge retention: How to prevent forgetting?
Semi-supervised learning: How to leverage the unlabeled data?
In particular, an OCL algorithm is called supervised if does not have the semi-supervised learning does. In the rest of this section we will provide an
component, and called semi- supervisd overview of the OCL algorithms we implement: first baselines. then supervised, and finally semi- supervised. Implementation details of these algorithms can be found Appendix D
4.1 NAIVE BASELINES
The naive baselines are used measure the difficulty of OCL-PDS task for interpreting the performances of OCL algorithms. First we have First Batch Only (FBO), where we only train an initial model on the first batch So (including the training regression set) with empirical risk minimization (ERM) and use that model till the end, which serves as lower bound of the online performance as well as an upper bound of the regression set performance (because it directly trains the model on regression data without any forgetting). Then we have i.i.d. offline, where for each t we train a model on separate training set i.i.d. sampled from D, and test on S,, which serves an approximate upper bound of the online performance that reflects the generalization gap. Finally, we have New Batch Only (NBO), where we train the model on the new data alone and never care about forgetting, which serves lower bound of the knowledge retention performance.
4.2 SUPERVISED OCL ALGORIHTMM
Rehearsal based methods. Rehearsal was first introduced in Ratcliff (1990;; Robins (1995) to prevent catastrophic forgetting in CL, where historical data is stored in a memory buffer and replayed to the learner. The simplest method is ER-FIFFO (also called ring buffer (Chaudhry al. 2019b),

---

Under review as a conference paper at ICLR 2023
where ER stands for experience replay. There are three sources of data the model needs to learn or remember: new data, recent data and regression data. Thus, ER-FIFO simply fine-tunes the model over the union of these three sets of data. The buffer looks like the new batch replaces the oldest one (while the regression set is never removed).
(FIFO) queue as
There are some variants of ER-FIFO with different strategies of selecting replay samples. In ER- FIFO-RW where RW stands for reweighting. the three data sources are balanced so that they have the same probability of being sampled in stochastic gradient descent (SGD), which is useful when different batches have different sizes (the weights can also be customized to adjust between learning and remembering). In Maximally Interfered Retrieval (MIR) (Aljundi et al., 2019a), the model is first "virtually" updated on the new data only, and those previous samples on which the loss increases the most before and after the virtual update are selected. The model then recovered and fine-tuned with a real update on the selected samples together with the new samples. Similarly, in MaxLoss (Lin et al. 2022) there is also a virtual update step, and previous samples with the highest loss after virtual update are selected to be replayed during real update.
GEM-PDS. This method is a combination of Gradient Episodic Memory (GEM) (Lopez- Paz & Ranzato 2017) and Average GEM (A-GEM) (Chaudhry et al. 2019a), and we design it specially for OCL-PDCS Denote the gradients of the loss function on the new data, recent data and regression data by go. 91 and 92. respectively. Gradient descent along go might cause the model to forget recent and important knowledge, so instead we find a "pseudo gradient" g that is close to go. and gradient descent along g won't lead to forgetting. We solve the following convex optimization problem:
minimize llg 90l12 S.t.
(g. 2 O, (g. 92) 2 0
g
This problem always feasible, and the optimal g" can be found with a simple procedure described in Appendix D Eqn. (3) The constraints ensure that descent along g won't increase the loss on the recent and regression data, which can be shown with the Taylor expansion of the loss function.
Regularization based methods. The high-level idea of regularization is to keep the model weights close to the initial model which has a good regression set performance, so as to reduce forgetting. Denote the vectorized model weights at time by 0, In Online L2 Regularization (L2Reg), we add a penalty term 1|0, foll3 the loss function.
variant called Elastic Weight Consolidation (EWC) Kirkpatrick et al. 2017), the penalty term is À diagg (0, 8o) Bo) where F; is the Fisher information matrix (FIM) and diag(A?) only contains the elements on the diagonal of F.. The FIM ensures that weights that are more influential to the model output change less.
2
4.3 SEMI-SUPERVISED OCL ALGORITHMS
Pseudo Labeling (PI.. PL. was proposed in Lee 2013 and is also known as self-training. Since Dr close to D;-1, if fi-1 can do well on D:-1, then it is very likely that can also do well on Dr. Based on this observation, for unlabeled sample in S., define its pscudo label simply as (x). Then, we fine-tune the model ft + ft on the union of the labeled and the pseudo- labeled sets with any supervised OCL algorithm. We denote a PL method by adding suffix "PL" after this supervised algorithm, such as ER-FIFO-PL. Note that in classification, f=- is the "hard" label, so training fi-1 on this label minimizes its entropy (2022a) proved the effectiveness of PL in the GDA context. Moreover, pseudo-labeled samples are not replayed for knowledge retention as they could have large label noise.
Kumar et al. (2020); Wang et al.
Furthermore, in our implementation of PL we introduce an innovative virtual update step which we find very important in our experiments. The whole algorithm goes as follows:
1. Virtual update: Fine-tune the model fi + fi-1 with ERM on the labeled samples only.
2. Pseudo label the unlabeled samples with the updated model: -> (z.
3. Revert the model weights from fi,, back to ft-1, and fine-tune fi-1 J. with any supervised OCL algorithm on the union of labeled and pseudo-labeled samples.
FixMatch (FM). FixMatch (Sohn et al., 2020) is variant of PL. In FM, there are two types of data augmentation: a strong one and a weak one, and the pseudo labels are generated on the weakly
augmented samples while the model is fine-tuned on the strongly augmented samples, which leads to a consistency regularization effect that makes the model have consistent outputs on the weakly and strongly augmented samples. Currently Fixhhtch is only implemented for vision tasks. We denote FixMatch by adding suffix "FM" after supervised algorithm, such as ER-FIFO-FM.
7

---

Under review as a conference paper at ICLR 2023
Worst Recent Part
84
86
Worst Recent Perf
74
?2
Reg Perf
80
Worst Reg Per
62
82
70
Worst Reg
80
80
68
78
78
8g
78
86 88
70 Avg Online Perf
86
88
70
Avg Online Pert
Avg Online Perf
Avg Online Perf
(a) CivilComments.
(b) FMoW.
(c) CivilComments.
(d) FMoW.
Figure 2: Results of supervised OCL algorithms on DivilComments-PPSS (a = 0.5%) and FMoW-
WPDS (a= 50%). Each point corresponds to one pair of algorithm and hyperparameters.
EXPERIMENTS
We compare the algorithms on the 4 benchmarks we constructed for OCL-PDS. Fach experiment is run times with different random seeds. For the recent performance and the regression set per- formance (reg performance), we report both the average and the worst performances, which are the mean and minimum of the performance over = 1, T, respectively. The reason why we also report the worst performance is that knowledge retention is required for all t. For saving space, put detailed settings and results in Appendix E In this section we summarize five remarkable obser- vations make from our experiments, first supervised and then semi-supervised OCL algorithms.
5.1 RESULTS OF SUPERVISED OCL ALGORIHTS
Observation #1: Strong positive correlation between online and recent performances. On all benchmarks, methods that achieve higher online performances also achieve higher recent perfor- mances. In Figures 2a and 2b we plot the average online and worst recent performances achieved by different supervised OCL algorithms on the CivilComments-WPH WPDS (a 0.5%) and MMMo-WWPD (a = 50%) benchmarks (a is the fraction of labeled samples), where we can see a strong positive correlation between these two. The reason is that D, is very close D De-w by formula- tion, so model that performs well on D. naturally performs well on the w recent distributions too. This is also known as the accuracy-on-thes thieell phenomenon (Miller et al. 2021).
Observation #2: Correlation between online and reg performances differs among benchmarks. In Figure 2c and 2d] we plot the average online and worst reg performances on CivilComments- WPDS and F.PS. We can see that the trends on these two benchmarks are quite different. On CivilComments- -WWDn methods with higher online performances have lower reg performances, while there is no such trend on FMoW- SMoW--PDS. One possible reason is that it depends on how the regression set is defined, and how close it between the regression set distribution and the overall distribution. For MilloommentsWDDDS the regression set samples (severely offensive comments) are very different from the other samples (mostly normal comments), while for FMDWSWPDS the regression set samples from Americas and Asia closer to the samples from other regions.
Observation #3: Some existing methods do not work well on regression tasks. On the right, we plot the average online and worst reg performances of different supervised OCL algorithms on the Poverty-WPP WPDS benchmark (a 50%), which is a regres- sion task. We observe that while some methods like ER-FIFO and L2Reg do better than the FBO baseline, some others including MIR and EWC achieve lower online performance than FBO which should be a lower bound. Most existing methods have only been tested on classification tasks before, and this experiment shows that they might not work so well on regression tasks.
0.65
Reg Perf
EWC
GEM-PDS:
L2Reg
Worst Reg
0.6
ER-FIFO-RW
MaxLoeg
0.55
MIR
ERR
NBO
0.5
0.74
0.76 0.78 0.8
Avg Online Perf
Figure 3: Moverty-WPDS.
5.2 RESULTS OF SEMI-SUPERVISED OCL ALGORITHMS
Observation #4: Unlabeled data improves OCL performance. In Figure 4 we plot the perfor- mances of FR-FIFFPP and FR-FIRRRRW-PLLFMM (in red) along with the performances of all supervised OCL algorithms (in blue) on Civiloomment-WPID and FMow-WPDS. We can see that with the same worst reg performance, SSL methods achieve higher average online performances, and vice versa. SSL improves the online performance by learning more new data, but does not help knowledge retention. Thus, supervised OCL algorithms with high online performances (such as MIR) do not improve much with SSL. We also observe that FixMatch is slightly better than PL.
Observation #5: Virtual update in PL is important. In Figure 5 we plot the performances of ER- FIFO-PL and ER-FIFO-RW-PI CivilComments- WPDS with different epochs of virtual update
8

---

Under review as a conference paper at ICLR 2023
80
85
85
Worst Pert
Worst Reg Perf
Worst Reg Perf
00
F8O!
ERFIFFOPL
ER-FIFO-RW.PL
80
80
80
75
40
75
70
20
70
82
84
86
88
68
74
65
Avg Online Perf
Avg Online Perl
75
80
85
90
(a) CivilComments.
(b) FMoW.
Avg Online Perf
Figure 4: Performances of ER-FIFO-PL/FM and ER- FIFORRWPPLFMM (in red). FM is only used for FMoW.
Figure 5: PL with different epochs of virtual update (CivilComments).
as labeled near the points. We can see that without virtual update (0 epoch), the online performance of PL is even lower than the FBO baseline. However, with just epoch of virtual update. the online performance rises above FBO, and with more epochs of virtual update the online performance is higher (but with a lower reg performance). This shows the importance of virtual update though was not included in previous methods like gradual self-training (Kumar al. 2020
One explanation is that the virtual update step distills the knowledge of P(Y X) from the new distribution into the current model, so P(f (X) X) is closer to P(Y X, D,). Without virtual update. fz- only has knowledge from old distributions, so when the model is trained on the pseudo- labeled samples, it reinforces its old knowledge but learns little new knowledge, resulting in a low online performance but high reg performance. This also implies that unsupervised OCL is difficult and perhaps infeasible, because without new labels the learner cannot know how P(Y X) changes.
6 DISCUSSION
The application of deep learning has become so wide nowadays that is very difficult to cover all the tasks with single general problem formulation. Here we briefly discuss two additional problems that stem from OCL-PDS which practitioners in certain areas in industry might find useful.
Fine-grained PDS. In our problem formulation in Section 2.1 the learner can fine-tune the model at every However, in applications where new data comes in very fast and the distribution changes very quickly, this could be impractical. One typical example is time series analysis such as stock price prediction. We term this setting fine-grained PDS as the data batches are much smaller and are received much more frequently. In our benchmarks, the time horizon T is around 20, while in fine-grained PDS benchmark T shoud typically be 2,000 or 20,000.
One way to handle fine-grained PDS is to have two fine-tune procedures: A fast one that can be done at every t, and slow one that is run simultancously with the fast one. For instance, in an ensemble method like Mixture-of-Expee the weights of the base models, while the slow procedure trains a new base model with the new data.
(Masoudnia & Ebrahimpour 2014), the fast procedure only adjusts
Abrupt shift detection. We assumed that Div(D, D4+1) < p for all t. However, this might not always be true, and abrupt shifts could happen in practice. For example, for toxic language detection on social media (Figure when the US Supreme Court overturned Roe v. Wade, posts related to gender, religion, politics and civil rights flooded on social media, causing sharp spike in the data distribution and probably a significant drop the model's performance. Thus, practitioners need a mechanism to detect such abrupt shifts so that prompt human intervention could take place.
Some applications in industry have online metrics that keep track of the online performance, and significant performance drop triggers human intervention. However, in applications without online metrics or where feedback can be delayed, we need to detect distribution shift with unlabeled data alone, a problem known as 0OD detection (Hendryeks & Gimpell 2017: Lee et al. 2018). A more difficult recently proposed problem is 0OD performance prediction (Chen et al.] 2021 Jiang et al,, 2022 Garg et al., 2022, Back al.) 2022). i.e. predicting the model's performance
the new distribution with unlabeled data alone, as distribution shift does not necessarily hurt the performance.
Limitation. While we commit ourselves to studying the real PDS that exists in industrial applica- tions, in this paper for privacy reasons we are limited to working on public datasets, which (despite effort of making them realistic) are still different from real applications. We hope that in the future there could be more datasets containing real PDS in industry applications released for the better development of this field.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered re- trieval. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 11849- 11860. Curran Associates, Inc., 2019a. URL http::/ ccpp.ccppppcpccpppcpppcppppppppe pdf
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Pro- ceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 11254- 11263, 2019b.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in neural information processing systems, 32, 2019c.
Martin Arjovsky, Léon Bottou, Ishaan Guirajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint
2019.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
Christina Back, Yiding Jiang, Aditi Raghunathan, and Zico Kolter. Agreement-on-thee Predict- ing the performance of neural networks under distribution shift. arXiv preprint arXiv:2206. 13089, 2022.
David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach semi-supervised learning. Advances in neural informa- tion processing systems, 32, 2019.
Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà. Continual lifelong learning in natural language processing: A survey. In Proceedings of the 28th International Conference on Computational Linguistics, pp. 6523-6541, Barcelona, Spain (Online), December 2020. Interna- tional Committee on Computational Linguistics. doi: maain.55 574
URL
Gilles Blanchard, Aniket Anand Deshmukh, Urun Dogan, Gyemin Lee, and Clayton Scott. Domain generalization by marginal transfer learning. Journal of Machine Learning Research, 22(2):1-55, 2021. URL http: jmir. wttppappes///2/11/22.
html
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. In Companion Pro- ceedings The 2019 World Wide Web Conference, 2019.
Stephen Boyd, Stephen P Boyd, and Lieven Vandenberghe. Convex optimization. Cambridge uni- versity press, 2004.
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 872-881. PMLR, 09-15 Jun 2019.
Zhipeng Cai, Ozan Sener, and Vladlen Koltun. Online continual learning with natural distribu- tion shifts: An empirical study with visual data. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8281-8290, 2021.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in Neural Information Processing Systems, 33:9912-99244 2020.
Arslan Chaudhry, Mare' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a G-GEM. In International Conference on Learning Representations, 2019a. URL wtR.htttss://ppnneeee
10

---

Under review as a conference paper at ICLR 2023
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan. Puneet K Dokania, Philip HS Tomr and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint 1rX.::1902.10440 2019b.
Jiefeng Chen, Frederick Liu, Besim Avci, Xi Wu, Yingyu Liang, and Somesh Jha, Detecting errors and estimating accuracy unlabeled data with self-training ensembles. Advances in Neural Information Processing Systems, 34: 14980-14992, 2021.
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2018.
Aristotelis Chrysakis and Marie-Francine Moens. Online continual learning from imbalanced data. In International Conference on Machine Learning, pp. 1952-1961 PMLR, 2020.
Matthias De Lange and Tinne Tuytelaars. Continual prototype evolution: Learning online from non- stationary data streams. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2250-8259, October 2021.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting classification tasks. IEEE transactions on pattern analysis and machine intelligence, (4(7):3366-338) 2021.
Cyprien Masson d'Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory in lifelong language learning. In H. Wallach, H. Larochelle, A. Beygelzimer. F. d'Alché- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, vol- 32. Curran Associates, Inc., 2019. URL https: htt.::/tttssst.teeeeee http:::://ppcceeedengge nwwrins.rr..e ce/
Jacob Devlin, Ming- Wei bidirectional transformers for language understanding. arXiv preprint arXiv: 2018.
Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
Carl Doersch, Abhinav Gupta, and Alexei A. Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), December 2015.
Yuntao Du, Jindong Wang, Wenjie Feng, Sinno Pan, Tao Qin, Renjun Xu, and Chongjun Wang- Adamn:: Adaptive learning and forecasting of time series. In Proceedings the 30th ACM Inter- national Conference on Information & Knowledge Management, pp. 602-411, 2021.
Wenying Duan, Xiaoxi He, Lu Zhou, Zimu Zhou, Lothar Thiele, and Hong Rao. Hyper attention recurrent neural network: Tackling temporal covariate shift in time series analysis. arXiv preprint DrXiv:2202.10080. 2022.
John Duchi and Hongseok Namkoong. Learning models with uniform performance via distribution- ally robust optimization. arXiv preprint arXi::1810.08750 2018.
Enrico Fini, Victor G Turrisi da Costa, Xavier Alameda-Pineda, Elisa Ricci, Karteek Alahari, and Julien Mairal. Self-supervised models continual learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Jean-Christophe Gagnon-Audet, Kartik Ahuja, Mohammad Javad Darvishi Bayazi, G. Dumas, and Irina Rish. Woods: Benchmarks for distriiitttt generalization in time series tasks. ArXiv, abs/2203.00978 2022.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario March, and Victor Lempitsky. Domain-adversarial training of neural net- works. Journal of Machine Learning Research, 17(59):1-35, 2016. URL http: /mmrr. arq//papars/v17/11 2399. html
11

---

Under review as a conference paper at ICLR 2023
Saurabh Garg, Sivaraman Balakrishnan, Zachary Chase Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data to predict out-of distribbbti performance. In International Conference Learning Representations, 2022.
m
[ddo__Ha H8 HsiMPYh_
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In International Conference on Learning Representations, 2018. URL
https://openrett
net wew./forrnnndd-SssAA.....
Arthur Gretton, Dougal Sutherland, and Wittawat Jitkrittum. Advances in Neural Information Process-
Interpretable com-
parison of distributions and models.
ing Systems [Tulorial], 2019. URL https://slidest siteesliveee.m www.iieww../...mmmmm wtt.c//////39..... interpretable cete-cooparriii sor-ison-oo of distributiionss of -nde-ldeldds
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bemardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to supervsee learning. Advances in neural information processing systems, 33:21271 2211222712224 2020.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Reprexentations, 2021.
id=1odXeXDoRt
Tatsunori Hashimoto, Megha Srivastava, Hongseok Namkoong, and Percy Liang. Fairness with- out demographics in repeated loss minimization. In Jennifer Dy and Andreas Krause (eds.), International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1929-1938, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729- 2020.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In International Conference on Learning Representations, 2017.
URL httpt::////// htttps://cpenreiieeet
n wttffoorrmm////// heee:/oorrm22dd-HgggTT
Weihua Hu, Gang Niu, Issei Sato, and Masashi Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning, pp. 2029- 2037. PMLR, 2018.
Xiaolei Huang and Michael Paul. Neural temporality adaptation for document classification:
Diachronic word embeddings and domain adaptation models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4123, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. 10011863/vvl 1/P19-140/ URL https: 7/aclanthology www.:oy-yy....11--11111100
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kerne:: Convergence and gen- eralization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Kokil Jaidka, Niyati Chhaya, and Lyle Ungar. Diachronic degradation of language models: In- sights from social media. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 195-200, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 1...186//11P1S.. 2032. URL https:
//aclanthology wrg/ PP18 2112222222
Yiding Jiang, Vaishnavh Nagarajan, Christina Back, and J Zico Kolter. Assessing generalization of SGD via disagreement. In International Conference on Learning Representations, 2022. URL
httts:://ppenree net wttp:/neteeoooooooooooooooooooo
12

---

Under review as a conference paper at ICLR 2023
Xisen Jin, Arka Sadhu, Junyi Du, and Xiang Ren. Gradient-based editing of memory examples for online task-free continual learning. Advances in Neural Information Processing Systems, 34: 29193-29205, 2021.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. In International Conference on Learning Representations, 2018. URL httpsss///pppeeeelllee net/Eorum?id=SSIXne
Chris Dongjoo Kim, Jinseo Jeong, and Gunhee Kim. Imbalanced continual learning with parti- tioning reservoir sampling. CoRR, abs/2009.03633 2020. URL /RR..htts://aaxi..rrrr /ttpa/bb.ooogos////o/oo 2009. 03632
Taesung Kim, Jinhee Kim, Yunwon Tae, Cheonbok Park, Jang-Ho Choi, and Jaegul Choo. Re- versible instance normalization for accurate time-series forecasting against distribution shift. In International Conference on Learning Representations, 2022. URL httthtts:///peeeie Dett/oorumiid-ee /ttp
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Bareskkkaaa et al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):352133526 2017.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning. pp. 5637-5664. PMLR, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ananya Kumar, Tengyu Ma, and Percy Liang. Understanding self-training for gradual domain adaptation. In Hal Daumé IIII and Aarti Singh (eds.), Proceedings of the 37th International Con- ference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 5468 5479, PMLR, 13-18 Jul 2020. URL httts::/pprcceetee www.ggg.........m...m..........m wtt.pprs../.19///////
Preethi Lahoti, Alex Beutel, Jilin Chen, Kang Lee, Flavien Prost, Nithum Thain, Xuezhi Wang, and Ed Chi. Fairness without demographics through adversarially reweighted learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 728-740. Curran Associates, Inc., 2020.
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. In International Conference Learning Representations, 2017. URL.ttpp://oppeeeeveee...
id [d=BJ6oOfqge)
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL
lecun. wwwp////wwb//maeee
Dong-Hyun Lee. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. 2013,
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence--alibraatee classifiers for detecting out-of-diitributtion samples. In International Conference on Learning Representations, 2018. (ttt:hhtts////peeei
ew. nwt. wtt..t//oorrrmm.dryy.ka/.//
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-294) 2017.
Bill Yuchen Lin, Sida Wang, Xi Lin, Robin Jia, Lin Xiao, Xiang Ren, and Scott Yih. On continual model refinement in out-of-distributiin data streams. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3128-3139, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v 1/2022. acl- URL h / anthology wrg.ooy...o.g2222..//. acl ong. 223 223 htt.:/tttps///c...attoollll/////
13

---

Under review as a conference paper at ICLR 2023
Zhiqiu Lin, Jia Shi, Deepak Pathak, and Deva Ramanan. The clear benchmark: Continual learning on real-world imagery. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
Anjin Liu, Guangquan Zhang, and Jie Lu. Fuzzy time windowing for gradual concept drift adap- tation. In 2017 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE), pp. 1-6. IEEE, 2017.
Bing Liu. Learning on the job: Online lifelong and continual learning. Proceedings of the AAAI Conference on Artificial Intelligence, 34(09):135441133333 Apr. 2020, doi: 10. 10.1169//aaiiv3/4 v34i09. 7079. URL htt.http:... //ojs. htt./oossaaaiia... org www.g//iinx.xpppo
7779
Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, Antonio Carta, Gabriele Graffieti, Tyler L. Hayes, Matthias De Lange, Marc Masana, Jary Pomponi, Gido van de Ven, Martin Mundt, Qi She, Keiland Cooper, Jeremy Forest, Eden Belouadah, Simone Calderara, German I. Parisi, Fabio Cuzzolin, Andreas Tolias, Simone Scardapane, Luca Antiga, Subutai Amhad, Adrian Popescu, Christopher Kanan, Joost van de Weijer, Tinne Tuytelaars, Davide Bacciu, and Davide Maltoni. Avalanche: an end-to-end library for continual learning. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2nd Continual Learning in Computer Vision Work- shop, 2021.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Unsupervised domain adaptation with residual transfer networks. Advances in neural information processing systems, 29, 2016.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Jie Lu, Anjin Liu, Fan Dong, Feng Gu, Joao Gama, and Guangquan Zhang. Learning under concept drift: A review. IEEE Transactions on Knowledge and Data Engineering, 31(12):2346-288 2363, 2018.
Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Karishma Mandyam, and Noah Smith. Time
waits for no one! analysis and challenges of temporal misalignment. In Proceedings of the 2022
Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, pp. 5944-5958, Seattle, United States, July 2022. Association for woatthhology..oom www.nnthooooy..og/2222e2..2 naacl saaacllmaa -main.
Computational Linguistics. URI.
435
Divyam Madaan, Jaehong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. Representational continuity for unsupervised continual learning. In International Conference on Learning Repre- sentations, 2022. URL
http://wwwwwwwwweeeeeeeeeeeeeeee fentt/oorummiooo 9Hrka5PA7LM
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network to multi- ple tasks by learning to mask weights. Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
Niki Martinel, Abir Das, Christian Micheloni, and Amit K Roy-Chowdhurr. Temporal model adap- tation for person re-identification. In European conference on computer vision. pp. 858-877. Springer, 2016.
Marc Masana, Xialei Liu, Bartlomicj Twardowski. Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer. Class-ineremental learning: survey and performance evaluation on image classifi- cation. arXiv preprint arXiv:2010. 2020.
Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: a literature survey. Artificial Intelli- gence Review, 42(2):275-293, 2014.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
14

---

Under review as a conference paper at ICLR 2023
John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishall Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correla- tion between out-of-distriuution and in-distribution generalization. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7721-7735. PMLR, 18-24 Jul 2021. URL httpp::/pproceedingg.... mir www.sppessssvvee vttp/v139////// / htt//mmmmeeer wwereeb............ html
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and supervisd learning. IEEE transactions on pattern analysis and machine intelligence, ((8):197-11993 2018.
Saeid Motiann Marco Piccirilli, Donald A Adjeroh, and Gianfranco Doretto. Unified deep super- vised domain adaptation and generalization. Proceedings of the IEEE international conference
on computer vision, pp. 5715-5725, 2017.
Hyeonseob Nam, HyunJae Lee, Jongchan Park, Wonjun Yoon, and Donggeun Yoo. Reducing do- main gap by reducing style bias. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8690-8699 June 2021.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using -astantl--labeled reviews and fine-grained aspects. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNEL) 2019.
Yassine Ouali, Céline Hudelot, and Myriam Tami. An overview of deep semi semi-supervised learning. arXiv preprint arXiv:2006.0522 2020.
Alan Ramponi and Barbara Plank. Neural unsupervised domain adaptation in NLP-A A survey. In Proceedings of the 28th International Conference Computational Linguistics, pp. 6838- 6855, Barcelona, Spain (Online). December 2020. International Committee on Computational Linguisties. doi: 10.18653/w1/2020.com m00.inn. URLhhttss://aclaattholooo org/
main. 603
Roger Ratcliff. Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, (7(2)2285, 1990.
Sylvestre-AIvee Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Ger- ald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing terference. In International Conference on Learning Representations, 2019. URL https: //openreview wwtpeeewwwwwweee..eeeeeeeeeee fett/forummdddBllo
Mark Ring. Child: A first step towards continual learning. In Learning to learn, pp. 261-292. Springer, 1998.
Anthony V. Robins. Catastrophic forgetting, rehearsal and pseudorchearsal. Connect. Sci., 7:123- 146, 1995.
Elan Rosenfeld, Pradeep Kumar Ravikumar, and Andrej Risteski. The risks of invariant risk mini- mization. In International Conference on Learning Representations, 2021.
Paul Röttger and Janet Pierrehumbert. Temporal adaptation BERT and performance on down- stream document classification: Insights from social media. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2400-2412. 2412, Punta Cana, Dominican Re- public, November 2021. Association for Computational Linguisties. doi: 10.18653/v1/202.. -indings--mnlp.
findings-emmlp.2000 URL https: httpss////aaaattoooooooooooo
206
15

---

Under review as a conference paper at ICLR 2023
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint arXiv:1606.04677 04671, 2016.
Shiori Sagawa, Pang Wei Koh, Tatsunori B. Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generaliza- tion. In International Conference on Learning Representations, 2020a.
Shiori Sagawa, Aditi Raghunathan, Pang Wei Koh, and Percy Liang. An investigation of why over- parameterization exacerbates spurious correlations. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceed- ings of Machine Learning Research, pp. 8346-8356. PMLR, 13-18 Jul 2020b.
Shiori Sagawa, Pang Wei Koh, Tony Lee, Irena Gao, Sang Michael Xie, Kendrick Shen, Ananya Kumar, Weihua Hu, Michihiro Yasunaga, Henrik Marklund, Sara Beery, Etienne David, Ian Stavness, Wei Guo, Jure Leskovec, Kate Saenko, Tatsunori Hashimoto, Sergey Levine, Chelsea Finn, and Percy Liang. Extending the WILDS benchmark for unsupervised adaptation. In In- lernational Conference Learning Representations, 2022. URL httph:t/t///// /oppo/veeee.eeeeeeoeee 1 d= 72V6KROOVV
/
Joan Serra, Didac Suris, Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention the task. In Jennifer Dy and Andreas Krause (eds.). Proceedings the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 4548 4557. PMLR, 10-15 Jul 2018. URL https: sttps::/proceeddie
m1 www.sress/vee. v ceerrall Ba html
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):227-244, 2000.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran As- sociates, Inc., 2017. URL https: httttts:/pproeeeeee http:////oceeeeeissse..e newrssneurieee ips. ccc/paper/211//ED 0efbe c6c 73dba 25waae2e2202222ee2ee --- pdf
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying supe-super learning with consistency and confidence. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Bal- can, and H. Lin (eds.), Advances Neural Information Processing Systems, volume 33, pp. 596- 608. Curran Associates, Inc., 2020. URL 2020/Fi1e/06600
ing. neur ips. cc cc/paper/ .ddf]
9adob www.ccddee3e933733333333
Baochen Sun and Kate Saenko. Deep coral: Correlation alignment for deep domain adaptation. In ECCV 2016 Workshops. 2016.
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaged con- sistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
Eric Tzeng, Judy Hoffmmnn Trevor Darrell, and Kate Saenko. Simultancous deep transfer across domains and tasks. In Proceedings of the IEEE international conference on computer vision, pp. 4068 4068-4076, 4076, 2015.
Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv:1904.07734 2019.
Jesper E Van Engelen and Holger H Hoos. A survey on semi-supervised learning. Machine Learn- ing, 109(():373-44)) 440, 2020.
Haoxiang Wang, Bo Li, and Han Zhao. Understanding gradual domain adaptation: Improved analy- sis, optimal path and beyond. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepes- vari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Ma- chine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 22784-22801
16

---

Under review as a conference paper at ICLR 2023
PMLR, 17-23 Jul 2022a. URL htt.lttess//ppoeed..ggg.....o/ss
html
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip Yu. Generalizing to unseen domains: A survey domain generalization. IEEE Transactions on Knowledge and Data Engineering, 2022b.
Ke Alexander Wang, Niladri Shekhar Chatterji, Saminul Haque, and Tatsumori Hashimoto. Is im- portance weighting incompatible with interpolating classifiers? In International Conference Learning Representations, 2022c.
Mei Wang and Weihong Deng. Deep visual domain adaptation: survey. Neurocomputing, 312:
135-153, 2018.
Zhenyi Wang, Li Shen, Le Fang, Qiuling Suo, Tichang Duan, and Mingchen Gao. Improv- ing task-free continual learning by distributionally robust memory evolution. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Pro- ceedings of Machine Learning Research, pp. 22985-22998. PMLR, 17-23 Jul 2022d. URL ottps://proceecings presspress// vl 62/ mang22v.htm..
Wikipedia contributors.
Wikipedia, the free encyclopedia, 2022a. URL/attps://en wikipedia wwg.o//g//o..... htrg/w/indexx http://nee.pppppt..eeeeeee phwttteeeeeerrrree Floyd protestscolcidd119922220
George floyd protests -
[Online; accessed 27-July-2022).
Wikipedia contributors. Dobbs v. jackson women's health clopedia, 2022b. URL
Wikipedia, the free ency-
V Jackackso. Women827s Health accessed 27-July-2022).
[Online;
Qizhe Xie, Minh- Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.
Da Xu, Yuting Ye, and Chuanwei Ruan. Understanding the role of importance weighting for deep learning. In International Conference Learning Representations, 2021.
Shuliang Xu and Junhong Wang. Dynamic extreme learning machine for data stream classification. Neurocomputing, 238:433 449, 2017.
Sijie Yan, Yuanjun Xiong, Kaustav Kundu, Shuo Yang, Siqi Deng, Meng Wang, Wei Xia, and Ste- fano Soatto. Positive-congruent training: Towards regression-free model updates. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14299-14308, 2021.
Xiangli Yang. Zixing Song, Irwin King, and Zenglin Xu. A survey on seiisuuereed learning. CoRR, abs/2103.0053. 2021. /tt.ttts:://axiv..omm https/iwwwrrg/bb///10000 00550
Christopher Yeh, Anthony Perez, Anne Driscoll, George Azzari, Zhongyi Tang, David Lobell, Ste- fano Ermon, and Marshall Burke. Using publicly available satellite imagery and deep learning to understand economic well-being in africa. Nature Communications, 2020.
Haiyan Yin, Ping Li, et al. Mitigating forgetting in online continual learning with neuron calibration. Advances in Neural Information Processing Systems, 34: 10260-002222 202
Haiyan Yin, peng yang, and Ping Li. Mitigating forgetting in online continual learning with neuron calibration. M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 10260-10272. Cur- ran Associates, Inc., 2021b. URL htt httppppppppppceppiiiiie .ewriieeeepp...ee wat.eec/ppaa cc/paper/20217 .pdf
fi file/54ee290er 20ee4e220ee00 05leeee55882220 9a 2eeaaal22233333 0111111333555
Jachong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. In International Conference on Learning Representations, 2018. URL
htttps://openreve net http://wwww....eeooooooooooss//.
17

---

Under review as a conference paper at ICLR 2023
Jachong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coreset selection for rehearsal-basee continual learning. In International Conference on Learning Representations, 2022. URL http::/::////peeeeveeeee net/ wtw.e//ooommmaa--00----0 5WNG4NV
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In International Conference on Machine Learning, pp. 3987-3995, PMLR, 2017.
Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes. arXiv preprint arXiv: 10123, 2018.
Runtian Zhai, Chen Dan, Zico Kolter, and Pradeep Ravikumar. Doro: Distributional and outlier ro- bust optimization. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 12345-12355, PMLR, 18-24 Jul 2021a.
Runtian Zhai, Chen Dan, Arun Suggala, J. Zico Kolter, and Pradeep Ravikumar. Boosted evar clas- sification. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 21860-21871, Cur- ran Associates, Inc., 2021b. UR//tttps: neurips http:seccpppppppeee.. cc/pppper//2p//////// File:/660.34cc111000661117227223 Paperr..-pppppppp.
Runtian Zhai, Chen Dan, Zico Kolter, and Pradcep Ravikumar, Understanding why generalized reweighting does not improve erm. arXiv preprint arXiv:2201.12295. 2022.
Xiaohua Zhai, Avital Oliver, Alexander Kolesnikov, and Lucas Beyer. S4I: Self-supervised semi-
supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1476-1485, 2019.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond em- pirical risk minimization. In International Conference on Learning Representations, 2018. URI
https::/openzeviiie net/ terett/foummiddeoo r1Ddp1-DDDllp
Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: Learning to adapt domain shift. In M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 23664-23678. Curran Asso- ciates, Inc., 2021. URL httpppps//ppoocecinssseeeeer wttpp//wwwss.ccppppp///////aeeee
Paper. pdf
18

---

Under review as a conference paper at ICLR 2023
Appendix
Table of Contents
A Liferature Review
19
A Distribution Shift
19
A.2 Continual Learning
20
Domain Adaptation
23
A.4
Semi-supervised Leaminng
23
B
The Divergence Function
24
B.1
Existing Divergence Functions
24
B.2 Divergence Function for OCL-PDS
25
B.3 --L--Divegeence
25
C
Benchmark Details
26
C.1 CivilComments WPDS
26
FMoW-WPDS
28
Amazon WPDS
30
C.4
Poverty-WPDS
30
D A Agorittmm Details
30
E Experiment Details
33
E.1 Setup
33
Results
33
A LITERATURE REVIEW
A.1 DISTRIBUTION SHIFT
Distribution shift in machine learning refers to the scenario where the model tested on dis- tribution Q different from the distribution P which it trained, which is different from the conventional machine learning setting where the training and testing sets are i.i.d. sampled from the same distribution. Distribution shift is studied in number of areas in machine learning. including domain adaptation, continual learning, transfer learning, fair machine learning, long-tailed learn- ing, etc. See Table of Gulrajani & Lopez-Paz (2021) and Table 2 of Wang al. (2022b) for comparison among these areas.
In general, there are two types distribution shift problem: Domain shift and subpopulation shift (Koh et al. 2022 Sagawa et al. 2022). In domain shift, the training and testing distributions contain different domains, and the goal is to generalize to new domains. This problem also called Out- of-distribution (OOD) generalization. such that the training set is ID (in-distribution) and the test set is OOD. Related areas include domain adaptation, domain generalization, transfer learning, etc. In subpopulation shift, the training and testing distributions consist of the same domains, but the relative proportions are different. Related areas include fair machine learning, long- learning (learning with imbalanced classes), The difference between these is that in subpopulation shift, the supports of the training and testing data distributions are the same, while it is not for domain shift.
A lot of methods have been proposed to train models that are robust to distribution shift. Here introduce two general methods, and in the subsequent sections we will talk about methods for specific areas. The most classic method is importance weighting (Shimodaira 2000), which multi-
plies the loss sample with importance weight Q(x) because f(x)dQ P(æ)*
Q(2) d..... This
P(z)
19

---

Under review as a conference paper at ICLR 2023
method requires that P(x)= 0 for all z such that Q(x)) > 0. i.e. only works for subpopulation shift. Another general method that is widely used today is Distributionally Robust Optimization (DRO) (Duchi & Namkoong 2018), which assumes that Q (P)) where (P) is the uncertainty set that contains a family of distributions that are close to P. Then, DRO trains the model on the worst distribution in U (P) (with the highest empirical risk), so
ensure that the model can do well on any distribution in (P) which includes Q. A lot of variants of DRO have been proposed and being used today (Hashimoto al., 2018 Hu et al. 2018 Sagawa et al. 2020a Lahoti al. 2020, Zhai et al. 2021a b). However, there is also a recent line of work that points out some problems with these methods both empirically and theoretically (Byrd & Lipton| 2019: Sagawa et al., 2020bb
Gulrajani & Lopez-Paz, [021] Xu et al. 2021; Wang et al. (022c).. Remarkably, recent paper Zhai et al. (2022) proved the surprising result that generalized reweighting (GRW) methods, broad Tamily of methods including importance weighting, DRO and a lot more, cannot do better than ERM for linear models and NTK neural networks (Jacot et al. 2018). Thus, there is still a huge room of
improvement for methods for distribution shift.
A.2 CONTINUAL. LEARNING
Continual learning (CI.), also known as lifelong learning, comes from the philosophy that learning agent should be able to continually learn new knowledge and improve itself with new data on its own. As detailed in Ring (998). continual learner should be able to learn context-depende autonomously, incrementally and hierarchically. A more recent paper Liu (2020) introduced the concept of on-the-job learning, where the learner is required to detect new tasks its own, collect data for the new tasks and then learn with the data. Given these general philosophical ideals, the goal of CL is to mathematically formulate a problem setting that reflects these ideals. Here we point our readers to the Avalanche library (Lomonaco et al. 2022). a recently released Python library that contains many benchmarks and algorithms of continual learning.
knowledge
A.2.1 SETTINGS
There are bunch of continual learning settings, and here we discuss the most widely studied ones.
Task-innrrmennal CL. In this setting, N tasks Ti, .N are sequentially given to the learner, who is required to learn these tasks one by one without forgetting the previous tasks. The learner cannot see the old tasks while learning new one, but it has a memory buffer in which it can store data from previous tasks. The goal is to perform well on all N tasks, and the performance is usually evaluated with the average or the minimum of the performances maassinneremetall CL where new classes sequentially appear. fask-inncrmmnnal CL is the oldest setting that dates back to McCloskey & Cohen (9899, which trained feed- forwarr network to first learn addition with one and then learn addition with two, and found that when learning the second task the network forgot the first one. This work pinpointed the catastrophic forgetling problem, and as a result many researchers today believe that "the central problem of continual learning is to overcome the catastrophic forgetting problem" (Aljundi et al. 2019c).
all tasks. A related setting is
Two related problems are Nomain-incremental CL and class-incremental CL. In domain-incremmnnta CL, the data of each task comes from a different domain. In ctassmincrementll CL, samples from new classes appear one by one.
Task-aware CL. In this setting, the tasks are not sequentially given. Rather, each sample has "task descriptor" indicating which task it belongs to. This setting dates back to the early paper Ratcliff (1990) which trained a multi-layer encoder model to learn four vectors A, B, C and D, which were provided in a cyclic fashion: ABCDABCDABCD... Then, Lopez-Pap & Ranzato (2017) studied task-aware CL in the online learning setting where samples come in an online stream. The goal of task-aware CL is the same as inseeencreettal CL: To perform well on all N tasks.
Task- agnostic CL. In task-agnostic CL, each sample still belongs to certain task, but the "task descriptor" is not provided. The model is still required to do well on all N tasks, and it is still evaluated by the average or the minimum of the performances setting usually do task inference, iee inferring the task from the input (Van de Ven & Tolias 2019). where they assume that samples from the same task are closer each other. Note that our definition
all tasks. Algorithms for this
20

---

Under review as a conference paper at ICLR 2023
is different from some previous work, and some previous work on task-agnostic CL like Zeno et al. 2018 is in fact under the task-free setting.
Task-free CL. In task-free CL. there is task all. The data comes in online stream and each sample only appears once. This setting was first introduced in [Aljundi et al.] (2019b), where they split a data domain into several partitions, and sequentially present each partition to the model. At the end of training, the model is evaluated on the entire data domain, ie. the union of all the partitions. Thus, if the learner is allowed store all samples and train the model on them, then the problem becomes equivalent to supervised learning. However, the model cannot store all samples because the storage size assumed be limited. This problem is also called the data incremental learning problem in De Lange et al. (2021). We can see that in this problem, the training is online (partitions are sequentially given), but the evaluation is offline (test once on the entire domain).
Online Continual Learning (OCL). We study OCL in this work. It is a variant of task-free CL as there is no task in OCL. The key characteristic of OCL is that both training and evaluation are online, which is different from the previous task-free CL problem. In OCL, for each new data batch, the model is first evaluated on it and then trained on it. Thanks to the online evaluation, we can study the real "lifelong" learning setting in OCL where the time horizon is infinite, which is not possible in the previous task-free CL problem. Note that the term "online continual learning" was used in quite few previous papers, but most of them refer to "continual learning with online data" (such as Yin et al. (02 which is not the OCL setting we define in this work.
Time series analysis. related area is time series analysis, where the data comes from non- stationary online distribution, and the task is to predict future data with the current and past obser- vations. We can see that distribution shift naturally exists in the formulation of time series analysis, but we fail to find much work about dealing with distribution shift with time series analysis.
One setting in time series analysis that is similar to OCL is temporal covariate shift (TCS). which was first introduced in Du et al. (2021). TCS makes the covariate shift assumption: P(Y is fixed while P(X) shifts with time. The goal of TCS is r-step ahead prediction, i.e. predicting the labels for inputs whose ground truth labels will be revealed steps later. Other recent papers that study distribution shift with time series analysis include Duan et al. 2022); Gagnon-Auded et al. (2022);
Kim et al. (2022).
Comparison with a previous work. One previous work/Car et al. 2021 proposed an OCL frame- work similar ours. In both settings, the model is evaluated new online batches before fine-tuned on them. The differences between our work and Cai et al. (2021) are the following:
1. Cai et al. 2021 considered fully supervised setting: The environment reveals all true labels to the learner after evaluation. In contrast, in OCL-PDS, we use the random label feedback which randomly selects a fraction of the new samples and provides their labels. In particular, we mainly study semi-supervised learning setting where the environment only reveals fraction of the labels, which is a more common scenario in industrial applications.
2. The evaluation metrics used in Cai et al. 2021 were: Average online accuracy, backward transfer and forward transfer. The first two metrics correspond to the average online per- formance and the average recent knowledge retention in OCL-PDS. In addition to these metrics, OCL-PDS also considers the regression set performance which is a very important metric in real applications, the worst (recent/important) knowledge retention performances since knowledge retention is required for all t, and the training efficiency which is very important an online setting where the algorithm run for many times.
3. Cai et al. 12021) only evaluates knowledge retention performances at three fixed time steps: H/3, 2H/3 /3 and H, where H is the total number of time steps. In contrast, OCL-PDS is purely online: all metrics including knowledge retention are evaluated at each step.
4. Cai et al. (202T) proposed the CLOC benchmark which is indeed a PDS benchmark. How- ever, CLOC only covers vision classification, while our benchmarks cover both language
and vision tasks, and both classification and regression tasks.
5. Cai et al. 202T) only run experience replay (ER) on their benchmark, while our work also compares regularization based methods such as EWC, variants of ER such as GEM,
21

---

Under review as a conference paper at ICLR 2023
MIR and MaxLoss, well semi-supervised learning methods like pseudo labeling and FixMatch.
A.2.2 METHODS
Rehearsal based methods. The concept of "rehearsal" was first introduced in Ratcliff (990);; Robins (1995). Rehearsal-based methods store samples in an operational memory (a memory buffer). When the learner learns new task, it also reviews the old samples this buffer order to prevent catastrophic forgetting, known as experience replay. Almost all previous work assumed that the buffer has a fixed size, so two key questions for these methods are: (i) Which samples to be store in the buffer, and (ii) Which samples to be replayed to the learner.
For problem (i), the most widely used strategy is Reservoir sampling (Isele & Cosgun 2018) which maintains the buffer distribution to be the average of all task distributions. Other variants include Chrysakis & Moens 2020); Kim et al. (2020). For problem (ii), MIR (Aljundi et al. (019a) selects the samples with the highest loss increase after "virtual update" step to be replayed, MaxLoss (Lin) et al.
2022) selects the samples with the highest loss after virtual update, and OCS (Yoon et al,
2022 selects an online coreset that diverse and has high affinity to previous tasks. Moreover, experience replay (ER) can be combined with other methods. For example, MER (Riemer et al., 2019) combines ER with meta learning, GMED (Jin al. 2021) combines ER with adversarial attack, and Wang et al. (2022d) combines ER with DRO.
There are alternative ways to use this buffer. For example, iCaRL (Rebuffi al. (017) stores "exemplars" in the buffer and uses a nearest neighbor classifier with these exemplars. In other words, samples the buffer are not used for training, but used for inference. Similarly, Continual Prototype Evolution (De Lange& & Tuytelaars] 202I) maintains and continually updates prototype for each class, and uses a nearest neighbor classifier for inference. Another way is introduced in GEM (Lopez-Paz & Ranzato 2017), where the buffered samples are not directly replayed for training, but instead used to find a "pseudo gradient" with a convex optimization problem so that the loss on previous tasks won't 't increase. Variants of GEM include A-GEM (Chaudhry et al. 2019a) and GSS (Aljundi et al., 2019c).
Since the buffer size is assumed to be limited, bunch of previous papers studied how utilize the buffer space more efficiently. One line of work proposes to learn the distributions of previous tasks with generative model, which includes DGR (Shin et al. (017). FearNet (Kemker & Kanan 2018) and so on.
Regularization based methods. The high-level idea of these methods not change the model weights too much so that the model's performance on previous tasks won't drop too much. One type of methods directly add to the objective a penalty term which keeps the model weights close to the old weights, such as EWC (Kirkpatrick et al.) 2017). Another type of methods use synaptic regularization (Zenke et al., 2017 Aljundi et al. 2019b) which controls the learning rate of each weight, so that more influential weights change more slowly.
Architecture based methods. These methods continually update the model architecture in order to learn the new tasks. One type of method changes the architecture by adding "mask" some weights, such as Piggyback (Mallya et al.] 2018) which learns a mask for each task, learning a hard attention mask with gradient descent (Serra al. 2018), and NCCL that adds weight calibra- tion modules and feature calibration modules to balance between stability and plasticity (Yin et al.) 20216). Another type of method uses the isolation approach where the model has two parts A shared part and a task specific part. When a new task arrives, the shared part is updated very little, and a new task specific part is learned on the new task. Examples of isolation methods include progressive neural network (PNN) (Rusu et al., 2016), Learning without Forgetting (LwFF) (Li & Hoiem 2017), dynamically expandable network (DEN) Yoon et al. 2018). etc.
continual learning. There are also some existing continual learning methods that work under semi-supervised or unsupervised setting. For instance, Lump MMadaan et al.] 2022) uses Mixup to interpolate between the current task and previous tasks' in- stances to alleviate catastrophic forgetting, and Fini et al. (202)) combines continual learning with self-supervised learning. leannig
22

---

Under review as a conference paper at ICLR 2023
A.3 DOMAIN ADAPTATION
Domain adaptation (DA) is a type of distribution shift where "the tasks are the same, and the differ- ences are only caused by domain divergence" (Wang & Deng 2018). There is a source distribution Pand a target distribution Q with distribution shift from P to Q. During training, the learner is pro- vided with samples from both the source and the target distributions, and conditioning on whether the samples from the target distribution is labeled, partially labeled or unlabeled, DA is classified as supervised, semi-supervised and unsupervised DA. Particularly, in supervised DA, the number of target samples is usually very small that training on these target samples alone cannot lead to a good model. A related area is domain generalization (Gulrajani & Lopez-Paz [202]] Blanchard et al. 2021 where the learner does not have any samples from the target domain, even unlabeled ones.
In general, a DA algorithm consists of two parts: Feature alignment and class alignment. The goal feature alignment is to train feature encoder & that can encode invariantfeatures, i.e. the images of the source and target domains in the feature space are close to each other, or (P)) s ((Q). As summarized in Wang & Deng (2018), there are two common ways to achieve feature alignment. The first one is adversarial- the source and target domains, and the features are aligned if this discriminator cannot achieve high performance. Methods of this type include DANN (Ganin et al.) 2016), SagNet (Nam et al.) i.e. minimizing a divergence function between
i.e. training a domain discriminator to distinguish features from
202T]. etc. The second one is discrepancy-basee (P) and ((Q),, also called the "confusion alignment loss" in Motiian et al. (2017). Methods of this type include CORAL (Sun & Saenko| 2016), IRM (Arjovsky et al., 2019). ARM (Zhang al. 2021 1 etc. However, there are also some papers that point out the problems within these methods RRoeenelld et al.. 202T; Gulrajani & Lopez-Paz 2021).
Moreover, even if we have learned a feature encoder & such that ((P) = ((Q), we cannot be sure that the same classifier works for both domains, because samples of different classes in P and Q might be mapped to the same latent feature. Thus, the goal of class alignment is either to make sure that samples of the same class are mapped together, or to train a new classifier w' which works for & ((Q). Note that class alignment requires labels from the target domain which are unavailable in unsupervised domain adaptation or domain generalization. For instance, Tzeng al. (2013) used soft labels for class alignment, [ong et al. (2016) minimized the cross entropy on the target data while using a residual block keep the source and target classifiers close, and Motiian et al. (2017) used a similarity penalty between samples from different classes.
A.4 SEMI SUMERVSEE LEARNING
In many applications. the number of labeled samples are limited, but there are also a large number of unlabeled samples. For example, for image classification tasks, while the labels are hard to obtain, free images can be very easily retrieved from the internet. Semi-supervised learning studies how to train model on a small set of labeled samples and large set of unlabeled samples. This is an old area of research and there is a very rich body of work, on which we do not intend to make an exhaustive survey here. Here we briefly discuss several types of methods that are widely being used today, and more methods can be found in surveys such as Van Engelen & Hoos 2020)..
Generating labels for unlabeled samples. Perhaps the most direct and intuitive way of leveraging the unlabeled samples is to try to generate labels for them, and then train the model over all samples as if they were all labeled. The simplest of such methods is pseudo-labeling (Lee) 2013). which first trains a model over the labeled samples alone and then uses this model to pseudo-label the unlabeled samples. This method assumes that the labeled and unlabeled samples come from the same underlying distribution, so if model can do well on the labeled samples, should be able to generate good pseudo labels for the unlabeled samples. However, the quality of pseudo labels depends on the generalization ability of the model, and if the number of labeled samples is too small, then the pseudo labels could contain large systematic label noise. Thus, number of techniques have been proposed to improve pseudo- labeling.
One such technique is consistency regularization, which is based on the following observation: Given two transformations x' and x" of the same sample x. their labels should be the same. For instance, FixMatch (Sohn al. 2020) uses two augmentation methods: a weak one and strong
23

---

Under review as a conference paper at ICLR 2023
one. It generates pseudo labels on the weakly augmented sample x', and trains the model on the strongly augmented one x". Similarly, Noisy Student (Xie et al. 2020) also uses a weak and a strong augmentation, but it alternates between teacher phases which generate pseudo labels and student phases which learn these labels, until convergence. Some other work defines x' and x" as the outputs of the model at different epochs, such as Temporal Ensembling (Laine & (017) and Mean Teachers (Tarvainen & Valpola, 2017). There is also line of work that leverages adversarial attack, such as Virtual Adversarial Training (VAT) (Miyato al. 2018
Interpolation based methods. These methods train the model on the interpolation between la- beled and unlabeled samples. This type of methods was initially introduced in MixUp (Zhang et al. 2018). which interpolates between two labeled samples combat label noise and adversarial attack as it makes the model have linear behavior in between samples. This method is then applied to remi-supervised learning in MixMatch (Berthelot et al., 2019), which combines MixUp with a lot of other techniques including pseudo labeling.
Self-supervised learning. The high-level idea of self self-supprvised learning (also known as repre- sentation learning) is to train a good feature extractor on the unlabeled data set with an auxiliary task (upstream task), and then train classifier on top of it the labeled data set (downstream task). The most famous and widely-used self-supervised learning technique is masked language modeling (MLM) in NLP (Devlin et al. 2018), where the auxiliary task is predicting masked word within sentence. MLM has achieved great success in NLP as the feature extractor it learns can be applied to almost any language task and lead to good performance.
Inspired by the success of MLM, people also try to apply self-supervised learning to vision tasks. For example, Doersch et al. (2015) extracts random pairs of patches from each image where the auxiliary task is to learn the relative position between each pair of patches, Gidaris et al. (2018) rotates each image with different angles where the auxiliary task is learn this rotation angle, which is applied to semi-supervised learning in S (Zhai et al. 2019). Today, the most widely used technique is contrastive learning, which extracts different views from each image, and the auxiliary task is to learn which views come from the same image. The feature extractor is trained to learn the similarity between two views: similar if they come from the same image, and different if they do not. This idea was first introduced in Bachman et al. (2019). Currently the most popular contrastive learning methods include SimCLR (Chen et al. 2020). MoCo (He et al. 2020), BYOL (Grill et al. 2020). SwAV (Caron et al. 2020), etc.
B THE DIVERGENCE FUNCTION
In this section, we dive deep into one important problem in the formulation of the OCL-PDS prob- lem: How to choose the divergence function Div(P Q) that guarantees the continuity of the distribution shift? As mentioned in Section 2.2 Div(P Q) refers to the divergence from P to Q and ideally we want it to reflect the performance of a model which is trained on P and tested on Q.
To study this problem, first we will review existing divergence functions, and then we will show that an ideal divergence function for OCL-PDS should be asymmetric and bounded, which is un- fortunately not satisfied by any popular divergence function. Finally, we will introduce the e-KL- divergence which is used in this work.
B. EXISTING DIVERGENCE FUNCTIONS
This part is based on the NeurIPS tutorial by Gretton al. (201)) Generally speaking, there are two types of existing divergence functions: Integral probability metrics (IPMs) and @-divergences. To quickly understand these two types of divergence, think about how to determine whether two distributions P and Q are equal. There are two ways in general: (a) Compute P - Q and see if is zero almost everywhere, and (b) Compute P/Q and see it is one almost everywhere. IPMs correspond to method (a) and d-divergences correspond to method (b).
IPMs. An IPM is defined as Div(P Q) sup Exxrf(X) - Bx)) for some func- tion family F. Examples include total variation (TV) defined as Div(P Q) 1 P(x)
24

---

Under review as a conference paper at ICLR 2023
Q(z)]dx, MMD defined Di || Q) [xx)][[(]] - By~ (Y for some fea- ture mapping æ and reproducing kernel Hilbert space H. and the Wasserstein distance defined as Div(P Q) inf inf D(z, (x, y) for some distance function -)
d-divergences. A d-divergence defined as Div (P Q) - fó dQ. For example, when log, then the o-divergence becomes the reverse KL-divergence, where the popular KL- Q(z) de (note that P and Q are reversed). Total variation is the only non-trivial function that is both an IPM and g-divergence.
divergence is defined as (Q P) Sa Q(x) log
P(z)
B.2 DIVERGENCE FUNCTION FOR OCL-PDSS
In this part, we show that an ideal divergence function for the OCL-PDS problem should be asym- metric and bounded.
Asymmetric. Suppose we have two very different data domains and B. Let P and Q = 0.5.4 + 0.5B. A model trained on P would have a very poor performance on Q. because it has never seen any samples from domain B. On the other hand, a model trained on Q could have a good performance on P. because it has seen samples from both A and B. Thus, in this example, would like to have Div(P || Q) > Div(Q || P). so Div should be asymmetric.
Bounded. The KL-divergence is widely used in machine learning literature, but one problem is that it is unbounded. Recall that in the OCL-PDS problem formulation, assume that Div(D, D++1) < for all f. Now consider what would happen the function Div is unbounded, and we want to introduce data from new domains into the problem. Specifically, Q contains samples from new domains that are not in P. i.e. there exists such that Q(r) > 0 and P(x) = 0. In this situation, must have Div(P || Q) = cc. matter how small Q(x) is. Therefore, if Div is unbounded like the reverse KL.ddivergence, then we could never introduce new domains into the problem, which not desirable. There variant of the KL-divergence called JS -S-divergence, which is defined as Div(P || Q) 2 [Dxu (P || M) + Dx (Q || M)] where M = } + Q. Although is bounded, also symmetric, so it not ideal for OCL-PDS.
B.3 e- -KL-DIVERGENCE
In this work, we use the e-KL-divergence defined as follows:
Div(P || [] = [Exxplo(X]] + Ey
[oo( (Y()]] (y=g(log(-g({))]
Q(x)
(2)
where g(x)
max Inax(P(an) cP
This divergence functions has the following properties:
This function is lower bound of the reverse KL-divergence, and it is bounded.
(ii) If g(x) (x)) P(zj* then this function equivalent the reverse KL-divergence (which is its dual formulation). Thus, for any x such that Q(x) > O, we have P(x) 2 c, then the c- C-KL-divergence is equivalent the reverse KL-divergence.
(iii) In the case where Q contains new domains that are not in P. for example Q = 00=101100 (1- (1-
for some P 1 P. then Div(P Q) = 3 + 3 log & + (1 38) log(1 - 3) R2 3 1 (K (1 P
From the properties, an e lower bound to the denominator of g(x)) so that the function becomes bounded.
that what are doing in the <-KL--ivereence is essentially adding
Example. Given two groups A and B, let p = 0.17 and e = 0.02. Then, we have the following group weight allocation schedule which is widely used in our benchmarks:
25

---

Under review as a conference paper at ICLR 2023
Table 2: Sample group weight allocation schedule.
t
A
B
Div(D; Du+1)
0
1.00
0.00
0.1661
1
0.90
0.10
0.1674
2
0.69
0.31
0.1663
3
0.41
0.59
0.1595
4
0.15
0.85
0.1625
5
0.00
1.00
Limitations. The major limitation of the <-Kl--divergence is that it cannot measure how similar two domains are. For example, suppose have three domains: A. B and C, and they do not overlap with one another. Samples in and B are very similar, but they are vastly different from samples in C. In this case, model trained on A can have good performance but a poor performance on 0.5AA + 0.5C. However, for the c-KL-divergence, we have Div(A 0.5A + 0.5B) = (iv(A A| 0.5A 0.5C). We similarity between A and B or and C. Another limitation is that the choice of e is arbitrary and affect the function value. Nevertheless, even with these two limitations, c-KL-divergence is suitable for the OCL-PDS problem. Finally. keep in mind that no divergence function can cover every facet of real problems practice, and that's 's why we have three important checking steps in our benchmarking procedure described in Section 3.1
0.5AA + 0.5B,
that the c-KL-divergence cannot measure the
still believe that the
C BENCHMARK DETAILS
We build new benchmarks for OCL-PDS following the guidelines listed below:
We only use public datasets in this work.
The benchmarks should cover a wide variety of tasks.
The benchmarks should be realistic and can reflect the real PDS in industry.
Naïve methods should be poor on these benchmarks, so special methods are necessary.
In this section, we will first demonstrate in detail how to use the 3-step procedure described in Section 3.1 with the CivilComments- -enne WPDS benchmark an example. Then, we will present the details for all other datasets, but without the detailed benchmarking procedure.
C.I CIVI.COMMENTS WPDD
Here present how construct CivilComments-Wen WPDS from the CivilComments Wilds dataset with the 3-step procedure including the important checking steps.
Separate the data into groups. First, we investigate the metadata we have in this dataset. In CivilComments- Wilds, apart from the target label, each sample also has the following annota- tions: whether it contains certain topic (male. female, LGBTQ, christian, muslim, other religions, black, white), and whether it contains a certain type of toxicity (identity attack, explicit sexual, etc.). model the distribution shift with the shift in hot topics. We separate
Step 1:
Based this metadata, the samples into four groups according to their topics, as shown in the following table.
Table 3: Samples in CivilCommens-Wilde are separated into groups by their topics.
Group
Topic
Samples
# Toxic
# Non-toxic
0
Normal
269,422
21,297
248,125
Race
37,459
10,820
26,639
2
Religion
65,816
8,106
57,710
Gender
75,303
10,571
64,732
We can observe that: (i) There are less toxicity in normal comments than comments about a specific topic; (ii) The portion of toxic comments about race is much higher than that of other topics.
26

---

Under review as a conference paper at ICLR 2023
Then, do the OOD check, where verify that there is distribution shift across the groups. We perform this check in the following way: For each Group k. we train a model on a training set sampled from all groups except Group k, and then test this model all groups except Group k and a test sampled from Group k. The results are the following:
validation set sampled from
Table 4: OOD check results of CivilComments WPDS (%)
Group k
Topic
Validation Accuracy
Test Accuracy
OOD Gap: Val Test
0
Normal
87.14
94.52
-7.38
1
Race
92.90
78,86
14.04
Religion
92.11
89.88
2.23
3
Gender
92.20
89.80
2.40
From this table, can see that Group (race) has the largest OOD performance gap, much larger than other groups. Thus, when allocating the groups in Step 2, we will make sure that the shift to Group is slower. Moreover, the OOD performance is better than the ID performance (i.e. A model trained on Groups 1-3 has higher accuracy on Group 0 than Groups 1-3). This a counter-intuitive phenomenon as is usually taken for granted that OOD performance should be lower than ID performance. The cause of this phenomenon might be that Group 0 much easier to learn than the other groups (for example, data is more concentrated and depends fewer features).
observe that on Group 0, the gap is negative, which means that
Step 2: Assign shifting group weights to the batches. We want to model the distribution shift with the shift hot topics on social media. At each time D, contains normal comments well as comments about the current hot topic, i.e. samples from Group exists in every batch. We design the schedule listed in the following table:
Table 5: Group weight schedule for CivilComments-WPSS, For each f, the weights add up to 1.
Group 0
Group
Group 2
Group 3
0
1.00
0.00
0.00
0.00
0.90
0.10
0.00
0.00
2
0.75
0.25
0.00
0.00
0.60
0.40
0.00
0.00
4
0.40
0.60
0,00
0.00
0.40
0.30
0.30
0.00
6
0.40
0.00
0.60
0.00
0.40
0.00
0.30
0.30
0.40
0.00
0.00
0.60
27

---

Under review as a conference paper at ICLR 2023
Then, we do the shift continuity check, whose point is to make sure that the distribution shift in this schedule continuous. This check goes as follows: For each we sample set from Dr and split into a training set and validation set, and then sample test set from D++1- We want to make sure that the gap between the validation and the test performances is not too large for each t. The results the following:
Table Shift continuity check results of CivilComments-PPD (%).
Validation Accuracy
Test Accuracy
OOD Gap: Val Test
94.87
92.99
1.88
1
93.49
91.31
2.18
90.67
88.10
2.57
3
87.56
83.59
3.97
85.60
89.18
-3.58
5
89.08
91.59
-2.51
91.63
91.69
-0.06
92.05
91.47
0.58
We can see that the OOD gaps are kept under 4%, opposed to the 14.04% gap we got in the OOD check. And we can observe the same phenomenon again: For some t, the OOD accuracy is higher than the ID occuracy.
Then, based on these results, we design the following sample allocation schedule listed in Table g The first batch contains 50000 samples, while all the other batches contain 10000 samples each.
Step 3: Design a separate regression set. We notice that each toxic comment is annotate which types of toxicity the comment contains in the CivilComments-Wiie regression set as the set of comments with two types of severe toxi- city: identity attack and explicit sexual. Such comments are critical toxic comments that good detector should be able to detect with high success rate.
Reg Performance
80
dataset. Thus, we design the
60
20
0
5
10
15
Then, do the regression check, where verify that na've meth- ods have regression on the regression set we constructed. Here use the NBO baseline described in Section 4.1 the "naïve method"" As shown in Figure 6 the regression set performance of NBO quickly declines to under 50% The performance of random guessing. Thus, the benchmarks passes this check.
Figure 6: Regression check.
Train/test split. For each f, we split Sy into a training set and a test set, and the regression set also split into training and test These test are used evaluate the recent and regression set performances. We never evaluate the learner on data it has seen, because the learner has an infinitely large buffer where it can store the data. Instead, evaluate the learner separate i.idd test sets. Note that the numbers listed in Table g are all sizes of the training sets.
C.2 FMOW-WPDS
This benchmark is built from FMoW-Wilds First, we separate the data into groups by the year:
Table 7: Samples in FMoW-WPDS are clustered into 5 groups by their year.
Group
Year
Samples
0
2002-2013
132,948
1
2014
54,575
2
2015
87,358
3
2016
140,459
4
2017
54,746
28

---

Under review as a conference paper at ICLR 2023
Table 8: Sample allocation schedule for CivilComments WPDS.
/
Group
Group 1
Group 2
Group
50000
0
1
9000
1000
0
2 9000
1000
3 7500
2500
O
7500
2500
5 6000
4000
0
6000
4000
7
4000
6000
0
8
4000
3000
3000
9
4000
3000
3000
10
4000
0
6000
0
11
4000
0
6000
12
4000
0
3000
3000
13
4000
0
3000
3000
14
4000
0
0
6000
15
4000
0
6000
Total
131000
27000
24000
18000
Table 9: Sample allocation schedule for FMoW-WPDS.
Group
Group
Group
Group 3
Group
120000
0 10000
0
0
9000
1000
0
0
6900
3100
O
0
6900
3100
O
0
4100
5900
O
6
0
4100
5900
O
0
1500
8500
0
8
0
10000
0
9000
1000
10
0
6900
3100
6900
3100
12
4100
5900
13
4100
5900
14
1500
8500
15
1500
8500
10000
17
0
9000
1000
6900
3100
19
0
0
6900
3100
0
4100
5900
21
0
0
4100
5900
22
0
1500
8500
23
0
0
1500
8500
24
0
O
10000
Total
120000
42500
71500
80000
46000
Then, allocate the groups with the schedule listed in Table We put 120000 samples from Group into batch for pretraining, and all other batches contain 10000 samples each. The regression set is defined as the set of samples in two highly populated regions: Americas and Asia.
29

---

Under review as a conference paper at ICLR 2023
C.3 AMAZON-WPDS
This benchmark built from the Amazon-Wilds dataset. In the original paper (Koh et al. 2021), the ID/OOD sets divided by the reviewers, and the dataset also contains some other metadata such as year and product categories. However, we find in our experiments that these groups cannot create a sufficiently large distribution shift. Thus, we use the following class split method, which has been widely used in the continual learning literature: We divide the 5 classes of this datasets (corresponding to the 5 stars rating) into groups and classes positive and negative reviews, as shown in the following table:
Table 10: In Amazon-WPDS, the 5 stars rating are divided into groups and 2 classes.
Group 0 Group 1
# Samples
Group 0
Group 1
y
2 stars
stars
y
193,900 1,087,385
377,039 2,343,846
y=
4 stars
5 stars
==1
Then, we allocate the groups the batches with the schedule listed in Table T2 This schedule based on the weight schedule obtained in Table The first batch has 50000 samples from Group I, and then we model a group shift from Group to Group 0. Each subsequent batch has 5000 samples.
Finally, the regression set is defined as the set reviews from 10 popular product categories, in- cluding books, fashion, etc.
C.4 POVERTY-WPDD
This benchmark is build from the PovertyMap- Mai-Wils dataset, which is a image regression task. First, we cluster the data into 4 group by the year:
Table 11: Samples in Poverty- -PDD are clustered into 4 groups by their year.
Group
Year
Samples
0
2009-2011
7129
2012-2013
5005
2
2014
3494
3
2015-2016
4041
Then allocate the groups with the schedule listed in Table 13 We put all samples from Group 0 (except test samples) into the first batch for pretraining, and each subsequent batch contains 800 is defined the samples from urban areas, which better
samples. Finally, the regression reflect the overall wealth index of each country. Note that this is regression task, and we evaluate the model performance with the Pearson correlation following Koh et al. (2021)
D ALGORITHM DETAILS
First of all, at t O, all OCL algorithms train the initial model on the first labeled batch and the training regression set with empirical risk minimization (ERM). In particular, following Koh let al. (2021), we use the cross entropy loss for classification tasks and the mean squared error for regression tasks. The differences among the methods only appear after / 0.
First, note that the upper bound baseline i.i.d offline is not OCL algorithm because assumes access to D, at time For each f, trains a model on a set sampled from D, with sufficiently large size and different benchmarks, but we always make sure that it is at least as large as the union all batches before time f. i.i.d. offline is only approximate upper bound of the online performance because
Baselines.
overlapping with S. The size of the training set is different for
30

---

Under review as a conference paper at ICLR 2023
Table 12: Sample allocation schedule for Amazon- Amazon-WPDS.
Group 0
Group
0
50000
500
4500
500
4500
1550
3450
1550
3450
5
1550
3450
2950
2050
2950
2050
2950
2050
2950
2050
10
4250
750
4250
750
12
4250
750
4250
750
14
5000
0
5000
16
5000
0
5000
Total
54450
80550
Table 13: Sample allocation schedule for Poverty- Poverty-WPDS. WPDS.
t
Group
Group 1
Group 2
Group
0
6359
0
0
800
2
0
720
80
0
3
720
80
4
0
552
248
0
5
0
552
248
0
6
0
328
400
72
7
0
328
400
72
120
400
280
9
0
120
400
280
10
0
400
400
11
0
200
600
12
0
800
13
0
0
0
800
Total
6359
4240
2856
3304
for a distribution P, training on multiple distributions other than might even achieve a better performance on P than training directly, as in benchmarking CivilComments-WPP However, this baseline gives us a sense of the generalization error, and thus help us interpret the performances of other OCL algorithms.
Then, for FBO, the algorithm does not do anything after t > 0, and the initial model is used till the end. For NBO, for each new batch S,, the model trained on the labeled portion of only with fixed number of epochs of ERM, and no previous sample is replayed.
ER-FIFO and ER-FIFO-RW. In ER-FIFO, for each > 0, the model is fine-tuned on the union of the new labeled batch, the recent labeled batches, and the training regression set, for fixed num- ber of epochs (named epochs) of ERM. Specifically, the recent labeled batches and the training regression stored in the memory buffer. One issue of this approach is that it does not take
31

---

Under review as a conference paper at ICLR 2023
into account the sizes of the batches and the regression set. For example, if the regression set is much larger than the batches, then it will be hard for the model to learn the new knowledge. Thus, ER-FIFO-RW alters the weights of the three data sets: the new data, the recent data and the regres- sion data. Specifically, uses uniform sampling over these three sets, so that each set has the same probability of being selected.
MIR and MaxLoss. When there are too many previous samples in the buffer, we cannot replay all of them for efficiency. ER- ER-FIFO-RYE solves this problem by randomly sampling previous samples to replay. However, this might not be the most efficient method, some previous samples might be more useful than the others for preventing catastrophic forgetting. MIR and MaxLoss are two strategies of selecting replay samples, which operate follows: For each iteration,
1. Sample new samples and fer previous samples that require knowledge retention (KR),
2. Virtual update: Fine-tune fi + fi with ERM on the new samples.
3. Select n samples from the Aer previous samples for replay. Specifically, MIR selects the samples whose loss increase the most before and after virtual update, while MaxLoss se- lects the samples with the highest loss after virtual update.
4. Recover the model to fi-1, and fine-tune ft-1 fe with ERM the new samples and the n selected previous samples.
The ratio nkr n is named as kr_size in our code, which should be greater than The larger nar is, the more likely we can select "useful" replay samples, but the slower the algorithm.
GEM-PDS, In this method, for each iteration, we first sample n new data, Agr recent data and Wkr regression data, which we estimate the three gradients of the loss function go- 91 and g2. respectively. A larger fkr allows the learner to estimate gi and d2 more accurately. The ratio fikr /n still named as kr-size. Then, the pseudo gradient g" is the optimal solution of the convex optimization problem Eqn. L which can be solved with the following procedure:
1. (9o.91 b (91.91), c (g1.92) (0i.92),4r (go, g2), e = (92.92).
2. If a 0, then return go-
3. p = ed ac, q = ac bd, r be e²
4. 91 9o
91. If S and g S then return ś1.
(3)
d
5. 02 = g0-
992-
If d < and p then return 92.
p 9 92-
6. Return 93 9o
91
We can verify with the KKT conditions that this procedure returns the correct solution of Eqn. ! (e.g. see Section 5.5.3 Boyd et al. (2004)) The model is then updated with gradient descent along g".
Online 1.2 Regularization and EWC. Let the loss of model fa parameterized by 0, (which is the vectorized model weight) on the labeled batch of S. be (0,). In online 1.2 regularization, we minimize the following objective function with fixed number of epochs of ERM:
min Ee().) + 2 110, 0u
(4)
where we add a La penalty between the new model weight O, and the initial model weight Og. The reasons why we use 1|0, full2 instead of ||e, 0 are:
1. Og has very good regression set performance, which is not guaranteed for p--1. So using |0: 00113 ensures a higher regression set performance.
2. If T is very large, then the model weight can still change a lot if we use ||0 - 0.- 113 (as the change accumulates with !), knowledge retention cannot be guaranteed.
32

---

Under review as a conference paper at ICLR 2023
In EWC, the objective function is the following:
min (((:)) + (diia((F))0 -0))l1
(5)
where F. the Fisher information matrix (FIM), and diag(Ft) only contains the elements on the diagonal of F2. Following Kirkpatrick et al. (2177), estimate diag F?) from the first-order deriva- tives of the loss function on Agy samples for knowledge retention. The ratio nkr/ n is still named as sizsize. With larger Tkr,, we can estimate F; more accurately. is named as 1bd.
Pseudo Labeling (PL) and FixMatch (FM). In PL and FM, there are two hyperparameters: the number of epochs of the virtual update step epochs_v, and the number of epochs of the real fine- tuning step epochs_r. Virtual update is done with ERM on the labeled samples only, while real fine-tuning is done with supervised OCL algorithm on the union of labeled and pseudo-labeled samples. Thus, PL. and FM can be combined with any supervised OCL algorithm, which we denote by adding the suffix "PL." or "FM" to the algorithm, such as ER-FIFO-PE and ER-FIFO-FM.
E EXPERIMENT DETAILS
E. SETUP
Following Koh et al. 202I)) we use DistilBert-bas--unaasd for CivilComments-WPS and Amazon- WPDD, a DenseNet-12 121 for FMoW-WPDS and ResNet-18 for Poverty- WPDS. For the training hyperparameters, we generally
the same ones as in Koh et al. (20T)) with one exception that we use a multi learning rate decay scheduler for the two vision benchmarks as we find that it produce better performances than the old scheduler.
For each experiment we report the following metrics:
For online performance, we report the average online performance (avg online) within finite T'
For knowledge retention, we report the average recent performance (avg recent) and the worst recent performance (worst recent), which the average and the minimum of the recent performance over = w..... .T. We also report the worst performance because knowledge retention required for every I. Likewise, we also report the average reg performance (avg reg) (regression set performance) and the worst reg performance (worst reg). As mentioned in Section 3.1 we use separate test batches to evaluate the recent and reg performances.
For training efficiency, we report the average runtime (avg time) of the method over 1,... Note that the time used train the initial model (t 0) not computed in the average runtime.
Each experiment is run on single NVIDIA V100 GPU. Each experiment is run times with differ- ent random seeds, and the mean and the standard deviation of the results reported. In particular, for each benchmark we use 5 fixed initial models: We train 5 initial models on the first batch and the training regression set with ERM for fixed number of epochs with different random seeds, and then use these 5 models as initial models for all methods. This both alleviates the effect of randomness the initial models and saves time. Under this setting, FBO can also serve as an upper bound of the worst regression set performance, which is the regression set performance at / = for every method.
E.2 RESULTS
In Table 14 we list the notations in the results. The results reported in Tables 15 18
33

---

Under review as a conference paper at ICLR 2023
Table 14: Notations of algorithms.
Algorithm
Hyperparameters
Notation
Example
NBO
epachs
NBO-epochs
NBO-5
ER-FIFO
epochs
ER-FIFO-epochs
ER-FIFO-S
ER-FIFO-RW
epochs
UR-FIFO-RW-epochs
ER-FIFO-RW-S
MTR
epochs, kr size size
MTR MIR-epochs-it si
MIR-5-4
MaxLoss
epochs, kr_size
MaaLossepooch-<<r.sizz
MaxLoss-5
GEM-PDS
epachs, kr.size epachs, 1bd epocha. Lod, kr.size epacha_y, epochs_r epocha_v, epochs_r
GEM-PDS-epochs<<rr....
GEM-PDS-5-4
1.2Rgg
I.Ree--pochhs-Idd
1.2egg-5-0.1
EWC
L.Kgg-pocchs--bakk.ssee
EWC-5-0.1-4
ER ER-FIFO-PL
ER ER-FIFO-PI -iiF--LL-epoccssseeeccee
ER-FIFO-PL--10
ER-FIFO-RW-PI
ER-FIFO-REE HILOO-RW-L--ppochs.--eooo
ER-FIFO-FM
epocha v. epochs r epocha_ye epochs_r
OR-FIIO-FM-ep-n v-epocha
ER-FIFO-FM-IOG
ER-FIFO-RW-FM
ER-FIFO-RW-FME
Table 15: Results on CivilComments (a 0.5%). Accuracies in %.
Algorithm
Avg Online
Avg Recent
Worst Recent
Avg Reg
Worst Reg
Avg Time
FBO
81.45 (0.37)
84.79 (0.96)
i.i.d. offline
90.03 (0.19)
NBO-100
89.19 (0.39)
89.08 (0.23)
85.12 (1.28)
55.20 (7.54)
30.07 (11.67)
95.61 (0.85)
ER-FIFO-1
82.62 (0.34)
82.92 (0.30)
77.42 (0.97)
83.81 (0.41)
81.46 (1.00)
309.04 (1.12)
ER-FIFO-RW-3
85.27 (0.70)
85.87 (0.54)
81.42 (1.44)
80.93 (0.57)
75.90 (1.71)
7.75 (0.04)
ER-FIFO-RW-10
83.94 (0.31)
84.09 (0.47)
79.48 (0.64)
83.32 (0.43)
80.63 (1.09)
24.48 (0.04)
ER-FIFO-RW-100
81.84 (0.59)
81.95 (0.78)
76.42 (1.54)
84.48 (0.68)
82.64 (1.10)
248. 83 (3.52)
MIR-10-4
87.33 (0.30)
86.70 (0.51)
80.99 (1.30)
75.94 (1.98)
57.57 (3.95)
40,99 (0.12)
MIR-100-4
84.16 (0.15)
83.36 (0.38)
77.36 (1.11)
82.29 (0.97)
74.65 (0.95)
419.84 (2.08)
MIR-10-10
86.85 (0,29)
86.06 (0.58)
79.57 (1.42)
77.71 (1.43)
60.21 (4.29)
75.31 (0.37)
MIR-100-10
83.72 (0.
83.09 (0.28)
77.21 (0.48)
83.00 (0.65)
78.96 (1.60)
757.66 (5.90)
MaxLoss- 10-4
87.32 (0.31)
86.76 (0.51)
81.04 (0.80)
76,25 (1.90)
$7.15 (2.91)
31,27 (0.09)
MaxLoss- 100-4
84.18 (0.44)
83.36 (0.40)
77.22 (0.79)
82.39 (1.32)
74.31 (1.74)
317.14 (1.47)
MaxLoss- 10-10
86.81 (0.34)
86.05 (0.46)
79.96 (1.11)
77.63 (1.31)
60.35 (2.31)
52.46 (0.41)
MaxLoss- 100-10
83.71 (0.18)
83.12 (0.35)
77.66 (0.58)
82.73 (0.95)
(1.03)
502.09 (6.19)
GEM-PDS-
87.59 (0.39)
87.64 (0.28)
82.96 (1.01)
76.13 (1.38)
66.78 (4.01)
68.15 (0.07)
GEM-PDS-30--
87.61 (0.55)
87.58 (0.45)
(1,01)
74.92 (2.53)
54)
195.42 (0.29)
GEM-PDS-10-
86.95 (0.43)
86.86 (0.46)
82.22 (0.59)
78.33 (1.37)
71.28 (2.31)
149.77 (0.14)
GEM-PDS-30-IE
86.72 (0.43)
86.60 (0.62)
80.60 (1.61)
77.52 (1.71)
64.50 (6.38)
456.54 (0.72)
L2Reg-10-1 10-1.0
88.35 (0.38)
88.51 (0.42)
84.87 (0.81)
70.51
59.10 (3.88)
10.33 (0.13)
1.2Reg-10-1 10-10.0
87.13 (0.48)
87.20 (0.61)
82.09 (0.62)
75.63 (2.90)
67.15 (4.56)
9.82 (0.12)
L2Reg-10- 10-100.0
86.11 (0.35)
86.03 (0.51)
80.18 (0.97)
78.07 (2.07)
69.46 (5.63)
(.. (0.12)
L-2ccg-100-100
87. 87.54 (0.39)
87.35 (0.42)
82.04 (0.71)
74.56 (2.21)
65.30 (3.71)
94.03 (0.67)
1.2Reg-100-11 100-1000
86.91 (0.37)
86.65 (0.49)
80.94 (0.91)
76.30 (2.21)
68.32 (2.54)
91.92 (0.71)
L2Rcg-100-100.0
86.18 (0.35)
85.93 (0.46)
79.98 (1.04)
78.05 (1.87)
69.80 (3.40)
93.58 (0.39)
EWC-10-1.04
88 50 (0.45)
88.54 (0.55)
84.49 (1.28)
68.74 (3.76)
$4.83 (5.17)
38.86 86 (0.37)
EWC-10-104. 10-00--00.0-
87.51 (0.25)
87.39 (0.45)
82.06 (1.16)
74.67 (2.09)
65.57 (2.81)
39,00 (0.44)
EWC-10-1000 10-100-100.0-4
85.62
85.38 (0.38)
79.34 (0.79)
(1.46)
(3.14)
38.81 (0.08)
EWC-100-1.0-4
86.56 (0.32)
86.36 (0.45)
80.70 (0.98)
77.53 (2.04)
71.48 (3.50)
371.97 (1.27)
EWC-100-10.0-4
86.24 (0.31)
86.01 (0.48)
80.24 (1.01)
78.21 (1.80)
72.01 (3.56)
371 (0.86)
EWC-100-100.0-4
85.20 (0.21)
84.79 (0.43)
78.73 (0.82)
80.44 (1.44)
75.44 (3.06)
368.17 (3.44)
ER-FIFO-PL-0-1
77.22 (1.83)
76.54 (2.06)
69.66 (2.17)
85,66 (1.31)
83.76 (0.87)
510.40 (8.05)
RR-FIOO-PL--11
85.21 (0.57)
86.46 (0.48)
82.81 (0.79)
79.54 (0.72)
73.69 (2.45)
$26.34 (4.13)
ER-FIFO-PL-3-1
87,28 (0.33)
88.22 (0.38)
84.59 (1.02)
76,26 (1.49)
68.86 (2.63)
514,51 (4.14)
ER-FIFO-PL-IO-1
87.94 (0.35)
88.76 (0.32)
85.72 (0.44)
73.82 (2.14)
64,97 (4.44)
501.86 (1.50)
ER-FIFO-PL-RW.CE
76.55 (1.05)
75.92 (1.97)
69.13 (2.96)
86.39 (U.76)
84,36 (1.08)
499.08 (1.22)
ER-FIFO-PL-RUU
PL-RW-1--
84.94 (0.85)
85.97 (0.91)
82.32 (1.44)
80.26 (1.40)
75.84 (1.36)
$31.59 (7.74)
ER-FIFO-PL-RW-3
87.08 (0.30)
88.29 (0.28)
84.79 (0.97)
76.60 (0.94)
68.59 (0.56)
522.92 (8.41)
ER-FIFO-PI-RRE 10-1
87.65 (0.46)
88.64 (0.37)
85.57 (0.63)
75.28 (2.13)
67.58 (3.41)
512.99 (2.48)
34

---

Under review as a conference paper at ICLR 2023
Table 16: Results FMoW-WPDS (a 50%). Accuracies in %.
Algorithm
Avg Online
Avg Recent
Worst Recent
Avg Reg
Worst Reg
Avg Time
FBO
68.00 (0.24)
84.54 (0.21)
Li.d. offline
83.04 (0.25)
NBO-5
71.78 (0.08)
73.08 (0.06)
71.27 (0.26)
76.84 (0.23)
72.71 (0.47)
161.90 (2.11)
ER-FIFO-2
74.01 (0.10)
76.04 (0.12)
74.38 (0.34)
83.91 (0.09)
80.82 (0.49)
1458.19 (239.67)
ER-FIFO-RW-2
72.77 (0.04)
74,56 (0.16)
73.07 (0.15)
81.84 (0.17)
80.34 (0.30)
226.31 (10.83)
ER-FIFO-RW-5
73.33 (0.11)
75.21 (0.06)
73.72 (0.28)
82.33 (0.12)
80.27 (0.34)
480.61 (20.08)
ER FIFFORF-R--10
73.56 (0.09)
75.80 (0.08)
74.31 (0.29)
82.63 (0.08)
80.21 (0.19)
953.56 (96.59)
MIR-2-2
69.75 (0.09)
70.74 (0.25)
68.68 (0,57)
79.25 (0.25)
76.48 (0.66)
357.74 (3.26)
MIR-2-4
71.29 (0.06)
72.36 (0.10)
70.44 (0,20)
80.81 (0.16)
79.19 (0.52)
460.35 (9.11)
MIR-2-10
72.59 (0.08)
73.95 (0.08)
72.27 (0.15)
81.90 (0.25)
80.92 (0.20)
710.30 (54.07)
MaxLoss-2-22
69.66 (0.21)
70.53 (0.30)
68.15
79.23 (0.13)
76.40 (0.56)
318.70 (3.33)
MaxLoss-2444
71.34 (0.07)
72.46 (0.05)
70.52 (0.32)
80.86 (0.12)
79.40 (0.35)
361.38 (25.58)
MaxLoss-2-110
72.64 (0.08)
74.00 (0.10)
72.37 (0.21)
81.90 (0.30)
80.90 (0.32)
553.68 (18.74)
(RM-PDS-2-2
72.07 (0.08)
73.21 (0.19)
71.49 (0.33)
79.10 (0.15)
75.19 (0.45)
240.80 (3.06)
GEM-PDS-24 PDS-2-4
72.04 (0.11)
(0.09)
71.54 (0.34)
79.16 (0.23)
75.54 (0.75)
502.77 (41.03)
GEM-PDS-2-10
72.05 (0.07)
73.21 (0.13)
71.45 (0.37)
79.17 (0.21)
75.86 (0.38)
969.29 (5.54)
L2Rcg-2-0.1
69.61 (0.11)
70.14 (0.19)
66.43 (0.28)
81.59 (0.17)
78.84 (0.95)
76.42 (1.15)
L2Rcg-2-1.0
67.90 (0.14)
68.32 (0.20)
63.78 (0.61)
82.30 (0.21)
80.30 (1.00)
79.84 (1.27)
1.2Reg-2-10.0
68.02 (0.12)
(0.26)
63.34 (0.43)
83.74 (0.19)
(0.30)
79.94 (0.99)
EWC-2-0.1-4
(0.16)
70.76 (0.24)
67.72 (0.43)
81.94 (0.10)
80.20 (0.60)
239.01 (1.32)
EWC-2-1.0-4
68.51 (0.16)
(0.27)
64.84
82.93 (0.14)
81.73 (0.42)
288.53 (26.6)
EWC-2-10.0-4
68.13
68.84 (0.20)
63.96 (0.35)
83.69 (0.11)
83.21 (0.14)
287.97 (26.06)
ER-NIFO-PL-3-3
74.25 (0.15)
(0.22)
74.49 (0.48)
84,00 (0.18)
80.34 (0.43)
1881.01 (6,93)
ER-FIFO-PL-5-3
14.3600.08
76.47 (0.23)
75.00 (0.58)
83.95 (0.13)
80.20 (0.50)
1924.03 (15.96)
ER-FIFO-PL-3-S
74.55 (0.06)
76.54 (0.24)
74.96 (0.42)
84.40 (0.10)
80.90 (0.60)
3071.01 (28.59)
ER-FIFO-PL-5-5
74.62 (0.05)
76.67 (0.13)
74.91 (0.21)
84.29 (0.11)
80.65 (0.23)
3128.51 (54.12)
ER-FIFO-PL-RWA:
73.40 (0.04)
75.28 (0.25)
73,80 (0.40)
82.12 (0.16)
79.80 (0.37)
573.71 (9.83)
ER-FIFO-PL-RW-S.
73.49 (0.09)
75.43 (0.18)
73.90 (0.34)
82.10 (0.19)
80.07 (0.24)
622.28 (6.41)
ER-FIFO-PL-RW-
(0.15)
75.41 (0.26)
73.70
82.27 (0.08)
(0.14)
891.74 (9.63)
ER-FIFO-PI-RRY
73.46 (0.04)
75.42 (0.14)
73.89 (0.21)
82.26 (0.06)
79.40 (0.31)
944.21 (6.73)
ER-PIFO-PM-3-3
74.81 (0.09)
76,91 (0.15)
75.36 (0.62)
84.37 (0.11)
80.80 (0.18)
2022.45 (179.49)
ER-FIFO-FM- 3-3
74.80 (0.06)
76.86 (0.13)
75.27 (0.17)
84.32 (0.16)
80.36 (0.42)
2084.54 (183.02)
ER-FIFO-FM-3.
75.10 (0.07)
77.19 (0.13)
75.46 (0.15)
84.70 (0.16)
80.96 (0.27)
3056,44 44 (52.54)
ER-FIPO-FM-5-S
75.16 (0.05)
77.33 (0.19)
75.75 (0.60)
84.71 (0.10)
80.88 (0.39)
3268 89 (307. 79)
73.44 (0.05)
75.37 (0.30)
73.80 (0.37)
81.87 (0.19)
79.62 (0.35)
$72.95 (2.68)
ER-FIFO-RW
--F---
7340 (0.08)
75.31 (0.12)
73.83 (0.30)
81.80 (0.05)
79.52 (0.41)
642.91 (37.83)
ER-FIFO-RW-FM-335
73.71 (0.05)
(0.15)
74.04 (0.41)
82.27 (0.08)
(0.44)
896.77 (4.61)
ER-FIFO-RW-PM-5-5
73.75 (0.15)
75.68 (0.13)
74.32 (0.25)
82.31 (0.05)
(0.33)
971.52 (39.10)
35

---

Under review as a conference paper at ICLR 2023
Table 17: Results on Amazon- (a = 0.5%). Accuracies in %.
Algorithm
Avg Ouline
Avg Recent
Worst Recent
Avg Reg
Worst Reg
Avg Time
FBO
84.85 (0.20)
93.78 (0.26)
i..dd offline
94.15 (0.19)
NBO-100
90.49 (0.41)
90.68 (0.34)
88.58 (0.95)
92.22 (0.43)
88.63 (0.76)
105.68 (1.74)
ER-FIFO-1
87.12 (0.34)
88.06 (0.28)
84.70 (0.40)
93.58 (0.16)
93.02 (0.28)
314,6 64 (1 60)
IR-FIFO-RW-I0
88.47 (0.67)
89.32 (0.73)
86.85 (1.62)
93.63 (0.30)
17.09 (0.06) 93.00 (0.37)
BR-FIFO-RW-100
86.65 (0.21)
87.39 (0.31)
83.22 (1.07)
93.68 (0.20)
93.11 (0.37)
181.74 (1.46)
MIR-10-4
89.46 (0.46)
90.19 (0.51)
88.00 (1.55)
93.31 (0.34)
91.82 (0.94)
28.88 (0.23)
MIR-100-4
86.99 (0.48)
87.87 (0.59)
84.28 (0.87)
93.62 (0.38)
92.86 (0.67)
286.59 (2.91)
MIR-10-10
89.15 (0.61)
89.95 (0.66)
87.28 (2.16)
93.45 (0.36)
92.21 (0.82)
47.99 (0.24)
MIR-100-10 10
87.16 (0.36)
88.07 (0.42)
84.68 (0.38)
93.78 (0.26)
93.24 (0.29)
479, 84 (11.69)
MaxLoss-10a
89,42 (0.43)
90.14 (0.54)
87.52 (2.01)
93.33 (0.28)
91.82 (0.84)
24.37 (0.17)
Max 100-4
87.06 (0.69)
87.88 (0.75)
84,42 (1,17)
93.72 (0.33)
93.13 (0,41)
247.32 66)
MaxLoss- 10-11000-00-------00000
89.32 (0.57)
90.06 (0.61)
87.78 (1.78)
93.49 (0.27)
92.41 (0.57)
34.72 (0.24)
MaxL 100-10
87.11 (0.45)
88.06 (0.66)
84.73 (1.18)
93.73 (0.30)
93.18 (0.30)
358.98 (3.47)
GEM-PDS- 10-4
89.94 (0.46)
90.50 (0.41)
88.74 (1.11)
92,94 (0.46)
90.65 (0.47)
37.30 (0.18)
GFM-PDS--3-
90.32 (0.47)
90.84 (0.35)
89.22 (1.08)
92.55 (0.43)
89.64 (0.66)
111.27 (0.22)
GEM-PDS-JU
89.77 (0.61)
90.42 (0.51)
88.20 (1.79)
93.10 (0.46)
91.30 (0.45)
82.24 (U.20)
GEM-PDS-30-10
90.26 (0.58)
90.81 (0.55)
89.32 (1.23)
92.80 (0.26)
89.93 (1.16)
239.08 (0:97)
L2Reg-10-0.1
89.01 (0,43)
89.79 (0.45)
86.72 (2.43)
93,23 (U.47)
91.46 (1.00)
11,06 (U.16)
12Reg-10-1.0
88.00 (0.51)
88.83 (0.57)
85.14 (1.48)
93.62 (0.20)
92.94 (0.35)
8.52 (0.05)
L2Reg-10-10.0
86.67 (0.30)
87.46 (0.35)
82.21 (1.26)
93.75 (0.18)
93.26 (0.18)
8.53 (0.07)
EWC-10-0.1-4
89.32 (0.43)
90.05 (0.46)
87.06 (2.60)
93,20 (0.35)
91.21 (1.21)
26. (0.41)
EWC-10-1.04
88.94 (0.45)
89.77 (0.49)
86.48 (2.77)
93.34 (0.27)
91.60 (1.27)
23.02 (0.21)
AWC-10-10.0-4
88.52 (0.48)
89.32 (0.57)
85.75 (2.05)
93.58 (0.24)
92.72 (0.34)
23.34 (0.24)
ER-PIFO-PL-O-1
85.75 (1.09)
86.39 (1.27)
80,91 (2.85)
93,94 (0.10)
93.39 (0.17)
428.25 (11.42)
ER-FIFO-PL-I-1
87.84 (0.57)
88.94 (0.69)
86.21 (1.57)
93.77 (0.22)
93.17 (0.27)
427.75 (7.14)
ER-FIFO-PL-3-1
88.97 (0.36)
89.98 (0.26)
87.72 (0.48)
93.66 (0.28)
92.92 (0.42)
422.17 (2.50)
ER-FIFO-PL-I0-
89.47 (0.40)
90.39 (0.38)
88.42 (1.43)
93.20 (U.13)
92.07 (0.36)
419.94 (1.45)
RE-FIFOW--LLOO
85.28 (1.37)
85,96 (1.41)
80.30 (2.62)
93.66 (0.44)
92.97 (0.58)
277.10 (0.58)
ER-FIFO-RW- ----RW-P
88.21 (0.95)
89.03 (1.09)
85.95 (2.30)
93.79 (0.25)
93.15 (0.33)
278.52 (1.28)
D-FFIFOR--PL.S
89.19 (0.43)
90.03 (0.40)
87.56 (1.09)
93.68 (0.15)
92.88 (0.38)
274.58 (0.62)
R--FIFO-RW-PL-10-
89.84 (0.54)
90.56 (0.49)
88.95 (1.30)
93.39 (0.31)
91.80 (0.59)
280.12 (1.85)
Table 18: Results on Poverty- -PDS (a 50%). Results are Pearson correlations.
Algorithm
Avg Online
Avg Recent
Worst Recent
Avg Reg
Worst Reg
Avg Time
FBO
0.781 (0.024)
0.638 (0.017)
i.i.d. offline
0.832 (0.010)
NBO-100
0,813 (0.003)
0.831 (0.004)
0.780 (0.010)
0.583 (0.012)
0.520 (0.035)
204.69 (2.12)
ER-FIFO-50
0.791 (0.007)
0.806 (0.008)
0.748 (0.050)
0.633 (0.018)
0.549 (0.109)
881.86 (57.91)
ER-FIFO-RW-SS
0.793 (0.006)
0.807 (0.009)
0.753 (0.028)
0.641 (0.016)
0.612 (0.024)
244.83 (3.85)
ER-FIFO-RW-101
0.798 (0.007)
0.812 (0.009)
0.765 (0.045)
0.641 (0.018)
0.611 (0.032)
426.77 (2.04)
ER-FIFO-RW-150
0.799 (0.004)
0.813 (0.007)
0.765 (0.044)
0.643 (0.015)
0.608 (0.021)
629.51 (8.50)
MIR-30-4
0.737 (0.021)
0.759 (0.016)
0.691 (0.050)
0.579 (0.015)
0.502 (0.036)
256.05 (31.52)
MIR-100-4
0.750 (0.007)
0.773 (0.015)
0.702 (0.023)
0.585 (0.007)
0.524 (0.043)
847.5 82 (27.40)
MIR-30-10
0.759 (0.010)
0.779 (0.016)
0.694 (0.060)
0.602 (0.008)
0.561 (0.019)
719.60 (42.89)
MIR-100-10
0.774 (0.007)
0.793 (0.011)
0.747 (0.029)
0.611 (0.014)
0.568 (0.031)
2148.85 (137.06)
MuxLoss-30-4
0.727 (0.023)
0.752 (0.020)
0.688 (0.038)
0.575 (0.014)
0.509 (0.024)
274.64 (25.15)
MaxLoss- 100-4
0,740 (0.019)
0.760 (0.023)
0.677 (0.039)
0,584 (0.017)
0.471 (8.182)
818.20 (19.62)
MaxLoss-30 30-10
0.754 (0.012)
0.776 (0.016)
0.695 (0.053)
0.601 (0.008)
0.561 (0.015)
557.77 (23. 23)
MaxLoss- 100-10
0.770 (0.010)
0.790 (0.015)
0.734 (0.054)
0.610 (0.017)
0.569 (0.029)
1643.23 (116.68)
GEM-PDS-30-
0.771 (0.005)
0.800 (0.009)
0.783 (0.009)
0.634 (0.014)
0.622 (0.015)
338.23 (7.42)
GEM-PDS-1001
0.777 (0.006)
0.809 (0.005)
0.793 (0.013)
0.640 (0.010)
0.627 (0.012)
1125.91 (36.97)
GHM-PDS-30-10
0.761 (0.004)
0.786 (0.009)
0.773 (0.013)
0.634 (0.014)
0.622 (0.019)
750. (4.47)
GEM-PDS-100- 10
0.767 (0.005)
0.798 (0.006)
0.781 (0.012)
0.640 (0.009)
0.629 (0.011)
3247 20 (472.93)
L2Reg-100-01
0.797 (0.007)
0.804 (0.004)
0.743 (0.027)
0.614 (0.012)
0.559 (0.032)
249 (19.51)
L22eg-110-11
0.792 (0.006)
0.807 (0.004)
0.783 (0.008)
0.642 (0.005)
0.619 (0.010)
247.66 (6.89)
L2Reg-100 100-10.0
0.780 (0.010)
0.799 (0.006)
0.778 (0.013)
0.641 (0.004)
0.616 (0.008)
247.39 (7.83)
EWC-100-0.1
0.726 (0.009)
0.744 (0.012)
0.719 (0.018)
0.625 (0.010)
0.601 (0.013)
884.09 (12.90)
EWC-100-1.0
0.735 (0.004)
0.753 (0.007)
0.734 (0.007)
0.648 (0.003)
0.628 (0.007)
714.14 (61.72)
EWC-100-10.0
0.716 (0.004)
0.731 (0.007)
0.718 (0.006)
0.652 (0.006)
0.636 (0.015)
719.71 (66.79)
36


Reviews:
[{"review_id": "glDPFGHfzge", "reviewer": "Reviewer_UPWo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "* The paper attempts to bridge the gap between the continual learning settings used in academic work and industrial applications. It introduces the setting of Online Continual learning for Progressive Distribution Shift (OCL-PDS) that considers subtle, gradual, and continuous distribution shifts. \n* The paper proposes three modifications to existing CL settings:\n  * Online evaluation and training.\n  *  Remember recent and important knowledge described by regression set instead of the complete past knowledge.\n  * Infinitely large storage without the replay of all samples. \n* The evaluation is conducted on four benchmarks proposed for this setting based on the WILDS benchmark for supervised and semi-supervised settings.\n", "summary_of_the_review": "While the paper is interesting and tackles a significant problem, it is not ready for acceptance in its current form. Remarkably, the paper lacks a comparison with existing benchmarks and CL methods, the modifications to the CL settings could be better-motivated and appropriate for many practical scenarios, and the paper's clarity (especially the results) needs to be significantly improved.", "strength_and_weaknesses": "I am well-familiar with the literature and read the full paper in detail. Accordingly, now I will describe the strengths and weaknesses of the paper in the order of originality + significance, quality of the paper, and clarity. \n\n### Originality\n##### Strengths\n* The paper takes an important step to develop CL benchmarks based on the issues faced by the practitioners. The proposed benchmarks model the shifts in hot topics, time, and language use and would be of interest to the CL community.\n* It provides various insights into the existing methods in the proposed OCL-PDS setting. In Particular, observation two, three, and five would be valuable for developing future methods in this setting.\n\n##### Weaknesses\nThe paper lacks the positioning and comparison of the proposed benchmarks and observations with many prior works:\n* **Proposed benchmarks:** The proposed benchmarks are a direct extension of the WILDS benchmarks. Cai et al. [1] proposed Continual LOCalization (CLOC) – 39 million images over nine years, with 712 classes. The paper refers to Lin et al. [2], which introduced CLEAR with 7.8M images based on the temporal evolution of visual concepts of Internet images in the footnote; however, it does not clarify the challenges in the creation of the regression set for CLEAR following the proposed approach. Both these benchmarks have a much larger scale than the benchmarks proposed in the paper. I would also suggest comparing the CL video benchmarks in the paper benchmarks section, providing the limitations of these already existing benchmarks and the necessity for the new benchmarks.\n* **Supervised and unsupervised methods:** The paper does not compare or refer to current online CL methods [3,4]. Further, while the paper does not consider many recent unsupervised/self-supervised CL methods [5,6].\n\n---\n\n### Quality\nThe paper evaluates multiple benchmarks that reciprocate real-world distribution shifts. It focuses on both the supervised and semi-supervised settings, strengthening the overall experimental evaluation. However, I have various concerns regarding the modifications and the proposed setup that I highlight below:\n* The assumption of infinite storage is impractical for most applications. Especially in the online setting, where a large number of instances are collected every second, it is not feasible to store the complete dataset. Furthermore, data storage is often restricted for many applications due to privacy restrictions.\n* The choice of the time window for knowledge retention needs to be clarified. There are no guidelines to select this hyper-parameter, and it will lead to the formulation of methods that can only remember the recent knowledge, where “recent” is artificially curated for each benchmark. Further, the importance/critical data for the regression set is not well-defined. It is also unclear whether the regression set should change or update depending on the change in the criticality of the data.\n\n---\n\n### Clarity\nThe paper was overall clear and well-written. Following are a few suggestions that highlight the missing details and should help improve the overall presentation:\n* ***Details about regressions set:*** The selection of the regression set should be elaborated in the paper. For instance, for FMoW-WPDS and Amazon-WPDS, how is the regression set constructed? Does it contain all the samples from America, Asia, and ten product categories for FMoW-WPDS and Amazon-WPDS, respectively?\n* ***Clarity of the figures:*** The figures for the experimental results are not interpretable. For instance, in Figure 2, each point corresponds to one algorithm according to the caption. However, the figure has no labels, and it is challenging to analyze the performance of any method in the figure. The tables in the supplementary material are much easier to read than the figures; however, the notations could be more straightforward and concise.\n\n---\n\n### Reproducibility\nThe paper included the code in the supplementary, which would promote the usage of these benchmarks for future CL papers.\n\n---\n\n### References\n[1] Cai et al. Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data. ICCV 21.  \n[2] Lin et al. The CLEAR Benchmark: Continual LEArning on Real-World Imagery.  NeurIPS 21.   \n[3] Yin et al. Mitigating Forgetting in Online Continual Learning with Neuron Calibration. NeurIPS 21.   \n[4] Yoon et al. Online Coreset Selection for Rehearsal-based Continual Learning. ICLR 22.   \n[5] Madaan et al. Representational Continuity for Unsupervised Continual Learning. ICLR 22.   \n[6] Fini et al. Self-Supervised Models are Continual Learners. CVPR 22.   \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YQb5bpPmek", "reviewer": "Reviewer_6758", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is intended to formally introduce the online continual learning problem with progressive distribution shift, and also presented four benchmark datasets and 12 algorithms (adapted from existing algorithms) for this setting, and finally presented empirical results of these algorithms on these datasets. ", "summary_of_the_review": "This paper attempts to establish some benchmark datasets and algorithms for online continual learning under progressive distribution shift. While a practitioners’ perspective is emphasized, I failed to see much novelty of this perspective, and the empirical results are not presented in a convincing enough way to show the benchmarks are good enough for wide use for the community. \n\nI would recommend the authors to check out this paper and explain the differences: Cai, Sener, Koltun, “Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data”, ICCV 2021. \n", "strength_and_weaknesses": "* The paper tries to be practical and bridge the gap between academic and industry, but I don’t really see anything novel. Missing citation: Cai, Sener, Koltun, “Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data”, ICCV 2021. \n* An important contribution is the benchmark datasets, and a 3-step procedure is described, but no examples are shown in the main text how these 3 steps are executed, and how the distribution is shifting with some metrics to demonstrate. I’m not very convinced these benchmark dataset are good. \n* Most of the results are presented in the format of scatter plot of “worst reg perf” vs  “avg online perf”, but it’s not well motivated why this is a good way of demonstrating the benchmarks are good.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UubSCn7T5V", "reviewer": "Reviewer_FxrK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Authors introduce the novel OCL-PDS problem - Online Continual Learning for Progressive Distribution Shift. Authors contrast their problem with that of continual learning and domain adaptation. Authors build 4 new benchmarks and implement 12 algorithms to test these benchmarks.\n", "summary_of_the_review": "\n1) From Appendix B, one can understand why divergence should be asymmetric. But from the example given, it looks like one needs structured asymmetry. Is just having asymmetry enough? \n\n2) Figure 1: It is not clear what distribution shift we are talking about in \"FMoW-WPDS benchmark.\" Are most images in 2002 of prison? Are most images in 2009 of Helipads? \n\n3) Section 2.3: Work is also very related to domain generalization (DG) which is a harder problem compared to domain adaptation (DA) [1]. If one wants to be truly \"online\" then one DG might be a better setting than DA? Authors should add more discussion comparing and contrasting with DG too. \n\n4) In general, real world datasets do not have shift continuity. Or there could be too much non-stationarity. How would ODD check or shift continuity check work in that case?\n\n5) How does one decide threshold on Div (D_t || D_t+1)? In the real world dataset, this could be even harder to determine. \n\n6) I checked the hyperparameters considered In Table 14(in appendix). But the list of hyperparameters tuned does not look complete. The training data generation (domains, dist shift and threshold on div for continual shift) should be treated as hyperparameters too. Can authors comment more on this? \n\n\n\n\n[1] Blanchard, Gilles, Aniket Anand Deshmukh, Ürun Dogan, Gyemin Lee, and Clayton Scott. \"Domain generalization by marginal transfer learning.\" The Journal of Machine Learning Research 22, no. 1 (2021): 46-100.\n", "strength_and_weaknesses": "Strengths:\n1) Paper is clearly written and the authors point to practical issues with the current approaches. \n2) Authors give very detailed justification of their choices, give details of experiments which will be helpful for other researchers to build on. \n\nWeaknesses: \n1) The proposed approach still has some practical issues (e.g. how does one decide the divergence threshold, how does one make sure that 3-step procedure benchmark OCL-PDS is applicable to real world problems? \n2) The final takeaway is not clear. Authors should discuss some of the real world problems and give some conclusions based on it. It is not clear what part of the discussion is helpful if a real world practitioner wants to use any of the knowledge in the paper. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GjblZm5S13", "reviewer": "Reviewer_FcFA", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a novel “Online Continual Learning for Progressive Distribution Shift (OCL-PDS)” problem that widely exists in industrial applications from the practitioner’s perspective, aiming to close the gap between academic research and industry. They propose three modifications to the related conventional settings, build 4 new benchmarks from the Wilds dataset, and implement 12 algorithms and baselines including both supervised and semi-supervised methods. Extensive experiments on the new benchmarks also bring some observations. ", "summary_of_the_review": "The proposed setting, concepts, evaluation paradigms, and datasets are realistic and may have a large impact on industrial applications.", "strength_and_weaknesses": "Strengths:\n1. This work first introduces and investigates the novel OCL-PDS problem, which more closely aligns with practitioners’ needs in the industry;\n2. This work improves the setting of conventional DA and CL problems at three points, including task-free, forgetting-allowed, and infinite-storage, which is reasonable and more practical, and thus can close the gap between academic work and real industrial applications to some degree.\n3. This work requires only remembering the “recent knowledge” and “important knowledge” rather than the knowledge on all tasks and designs a “regression set” to describe the important knowledge.\n4. This work releases 4 new benchmarks using the 3-step procedure for the OCL-PDS setting, while adapting and implementing 12 OCL algorithms and baselines, including both supervised and semi-supervised. Their observations based on extensive experiments may help boost the development of OCL algorithms for handling PDS.\n\nWeaknesses:\n1. The “infinite storage” this work proposes is somewhat inconsistent with the actual situation, especially in scenarios where the distribution changes gradually over time, such as a recommendation system. The space cost that increases linearly with time T will bring a heavy burden when T becomes large even for the industrial scenarios. \n2. According to observation 2, the online performance depends on how the regression set is defined, and how close it is between the regression set distribution and the overall distribution. So besides the proposed “frequent data” and “critical data”, is there a more general way to construct an appropriate regression set to describe the important knowledge? \n3. It is expected to add First Batch Only (FBO) without the training regression set as a baseline, which can serve as a better lower bound of the online performance than FBO. Because the critical data may not show in the first batch.\n4. The proposed i.i.d. offline may not be the upper bound of the online performance, because jointly learning multiple datasets may outperform learning each dataset independently. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject