Spurious Features in Continual Learning
Authors: Timothee LESORT
Venue: Submitted to ICLR 2023 (2023)
Keywords: Spurious Features
Continual Learning
Plasticity

Abstract: Continual Learning (CL) is the research field addressing learning without forgetting when the data distribution is not static. 
This paper studies spurious features' influence on continual learning algorithms.
We show that continual learning algorithms solve tasks by selecting features that are not generalizable. 
Our experiments highlight that continual learning algorithms face two related problems: (1) spurious features (SP) and (2) local spurious features (LSP). The first one is due to a covariate shift between training and testing data, while the second is due to the limited access to data at each training step.
We study (1) through a consistent set of continual learning experiments varying spurious correlation amount and data distribution support.
We show that (2) is a major cause of performance decrease in continual learning along with catastrophic forgetting. 
This paper presents a different way of understanding performance decrease in continual learning by highlighting the influence of (local) spurious features in algorithms capabilities.

Under review as a conference paper at ICLR 2023
SPURIOUS FEATURES IN CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Continual Learning (CL) is the research field addressing learning without forget- ting when the data distribution is not static. This paper studies spurious features' influence on continual learning algorithms. We show that continual learning algo- rithms solve tasks by selecting features that are not generalizable. Our experiments highlight that continual learning algorithms face two related problems: (1) spuri- ous features (SF) and (2) local spurious features (LSF). The first one is due to a covariate shift between training and testing data. while the second is due to the limited access data at each training step. We study (1) through a consistent set of continual learning experiments varying spurious correlation amount and data distribution support. We show that (2) is major cause of performance decrease in continual learning along with catastrophic forgetting (CF). This paper presents a different way of understanding performance decrease in continual learning by highlighting the influence of (local) spurious features in algorithms performance.
INTRODUCTION
Feature selection is a standard machine learning problem. Its objective is to improve the prediction performance, provide faster and more effective predictors, and provide better understanding of the underlying process that generated the data (Guyon & Elisseeff, 2003). In this paper, we are interested in improving prediction performance in the presence of spurious features. Spurious features arise when features correlate well with labels in training data but not in test data. Learning algorithms that rely on spurious features will generalize badly to test data.
In continual learning (CL., the training data distribution changes through time. Hence, we could expect that spurious features (SFs) one time-step of the data distribution will not last. continual learning algorithm relying spurious feature to solve task can then be resilient and learn better features later, given more data, Algorithms can aim to detect and ignore spurious features learned in the past (Javed et al., 2020). An example of task with spurious features could be classification task between cars and bikes, but in the training data, all cars are red, and all bikes are white, but test data, both are in a unique blue not available in train data. A model could easily overfit the color to solve the task while it is not discriminative in the test data. This problem is notably caused by a covariate shift between train and test data. In CL., we would expect future tasks to bring pictures of cars and bikes of other colors to learn better features.
On the other hand, in CL, the second type of spurious feature can be described: local spurious features. Local features denote features that correlate well with labels within a task (a state of the data distribution) but not in the full scenario. In opposite the usual spurious features, this problem is provoked by the unavailability of all data, for example, because only red cars and white bikes are currently available in train data, but in the test data, there are also cars and bikes of colors not seen yet or seen in the past. Overall, there is no significant covariate shift between all train data and test data. is, therefore, a problem specific to continual learning.
This paper investigates both the problem of spurious features (with covariate shift) and local spurious features (without covariate shift) in CL as shown in Fig. 1. Our contributions are: (I) We propose a methodology to highlight the problems of spurious features and local spurious features in continual learning. (2) We create a binary CIFAR1O scenario SpuriousCIFAR2 inspired by colored MNIST to experiment with spurious correlations. (3) We propose modified version of Out-of- Diittr---Ditiittinnn (OOD) generalization methods for continual learning and evaluate them on SpuriousCIIP CIFAR2. (4) We identify local spurious features core challenge for continual leaming algorithms along with CF

---

Under review as a conference paper at ICLR 2023
Terk
(a) Train (Spurious)
(b) Test (Spurious)
(c) Train (local spurious) (d) Test (local spurious)
Figure 1: Spurious features and local spurious features. If the task is to distinguish the squares from the circles. In Fig. la and 1b, the color is a spurious feature because there is a covariate shift between train and test data. In Fig. Ic and 1d, observe two tasks of a innremmnaanem scenario, the colors are locally spurious in tasks and 2. Even if there is no significant covariate shift between train and test full data distribution, colors appear discriminative while looking data within task.
We expect this paper to improve the understanding of the continual learning problem.
2 RELATED WORK
In large part of the continual learning bibliography. algorithms assume that to avoid catastrophic forgetting (CF), they should not increase the loss past tasks (Kirkpatrick et al., 2017; Ritter et al., 2018). It leads to the definition of interferencefforgetting of (Riemer et al., 2019): @L(Z. 89
2L(x,.0;) < O, V(I;; 36) E Ti and v(zj. Vj) E Ti with j > i, <.>>>>> is the dot product operator. Following this definition, increasing the loss on past tasks necessarily leads to a performance decrease. However, the algorithm might have learned spurious/local features that need to be forgotten to improve performance. Hence, the loss needs to be temporarily increased to reach a more general solution, and optimizing the interference equation could be counterproductive. On the same line, the presence of spurious features could be adversarial to most continual regularization strategies. For example, if measure weights importance with Fisher information, high importance will be given to weights using spurious features, and regularization will penalize their modification. Regularization could protect features that generalize poorly the test set.
Vanilla rehearsal or generative replay is good solution to not forget meaningful information and to deal with spurious features. By replaying old data, algorithms simulate independent and identical distribution (iid) and avoid local spurious feature problems. Replay methods have been shown in the bibliography be efficient and versatile even in their most straightforward form (Prabhu al., 2020). Notably, CL state-of-the of-the-art on ImageNet uses replay (Douillard et al., 2020; Zhao et al., 2020).
The research field that usually deals with spurious correlations is the ofi-t--distribution (OOD) generalization field. This field has received lot of attention in recent years, especially since the Invariant Risk Minimization (IRM) (Arjovsky et al., 2019) paper. OOD approaches target training scenarios where there are several training environments within which different spurious features correlate with labels. The goal then is to learn invariant features among all environments to build an invariant predictor in all training environments and potentially any other (Arjovsky et al., 2019; Ahuja et al., 2021; Sagawa et al., 2019; Pezeshki et al., 2020). This paper will adapt some those approaches for continual learning to evaluate how those approaches can deal with sequences of tasks.
PROBLEM FORMULATION
This section introduces the spurious features problems in a sequence tasks. The goal is to present the key types of features, namely: general, local, and spurious features.
General Formalism: We consider a continual scenario of classification tasks. We study a function fo(.). implemented is is the number of parameters) representing the set of weight matrices and bias vectors of deep network. In continual learning, the goal is find a solution 0' by minimizing loss on a stream of data formalized as a sequence of tasks [To. .i..., Tr- such that V(I,, y.) ~ Te (t E [0, 1)) for (z) y. We do not use the task index for inferences (i.e. single head setting).
neural network, parameterized by vector of parameters IRP (where p
2

---

Under review as a conference paper at ICLR 2023
To describe the different types of features, let be a feature and 2 ~ D a datum point in dataset D. We define w.)) function which returns if z is in x and 0 if not. w())'s output is binary for simplicity. Then, for all data with a label y in the dataset D, we can compute the correlation (rreeation((((,2)2
Table Summary of characteristics of the types of features. For feature of class we denote if it verify (1) on different data setting, a single task Tr, the whole scenario Cr, the test set Die.
Name
T
Cr
Die
1, = y), which estimates how a feature correlates with the data of a given class. We can then define discriminative features as:
c(D,2,y)
Good Feature (3.))
r
Spurious Feature (Eapur)
Local Feature (Sioe)
?
Local Spurious Feature (Zspurit)
x
x
is discriminative for class y in D if:
Vy EV,y + v/ c(D,z.9) > (D, -,, 2,1y)
(1)
y is the set of classes in D. other words, Z discriminative for if it correlates significantly more to y is data than the data of any other class. Then good feature 24 for class y respects (1) for training data Dr, and test data Die-
Spurious Features (SF) and Local Spurious Features (LSF))
spurious feature zspur for class respects (1) for training data Dr, but not for test data Die- A spurious feature is well correlated with labels in training data but not with testing data. Hence, learning from Expur may offer a low training error but high test error. The presence of žspur is due to a covariate shift between train and test distribution which changes the feature distribution.
In continual learning, the covariate shift between train and test Espur may also lead to poor general- ization. Further, the features can be locally spurious, e-g., they correlate well with labels within task but not within the whole scenario. We name them local spurious features (LSF). We illustrate the difference between spurious features and local spurious features in Figure
At task f, A local spurious feature Espur:t respects (1) for class in task Tr, but not for the whole scenario Cr. is LSF for a class y in To ~ Cr, with E [0, 1]]
if v' E Vo, + ((T,,,)) > 2(T,, and 3 n" E V. v (CTT,2,) 2 cCCT, 2,1"
(2)
3%. is the classes set in task T; and y is the classes set in the full scenario Cr composed o tasks.
A LSF žspur: correlates well with label on the current task but not the whole scenario. Espur;t can be extended from a single task Te to all task seen so far Tox without loss of generality.
Global Local Solution: We assume that machine learning models solve tasks by learning to detect (or select) features that correlate well with labels. Then, while learning on a task t, we can distinguish local solution 07, satisfying for the current task Tr, from global solution 05r that is satisfying for whole scenario Cr (past, current, and future tasks).
Similarly, can differentiate local and global features, contributing to local and global solutions. The global features are the good features 2, that are useful for the solution of the scenario. Unfortunately. time 1, can not know if feature is part of 24 without access to the future. Therefore, algorithms should learn with their current data but update their knowledge afterwards, given new data. For example, in classification, the discriminative features for given class depend on all the classes. Therefore, when new classes arrive, discriminative features can become outdated in class-incremental scenarios.
INFLUENCE OF SPURIOUS FEATURES (SFs) ON CONTINUAL LEARNING
This experimental section studies how continual learning algorithms can deal with spurious features. We design a scenario with spurious features that change at each task. We create a set of scenarios with gradual correlations between spurious features and labels. We evaluate various baselines to assess continual learning capabilities (Sec. 4.3.2) in such scenarios. We also experiments with potential solutions to deal with spurious correlation.
4.1 SETTING
3

---

Under review as a conference paper at ICLR 2023
We propose benchmark "SpuriousCIFAR2" inspired by col- ored MNIST (Arjovsky et al., 2019) with CIFARIO but without label noise. We convert the ten-way classification dataset into binary dataset. The new classes are "transportation means ver- not transportation means", i.e., cars, trucks, ships, airplanes, and horses versus the other classes: birds, cats, dogs, deers, frogs. The goal is have simple setting more challenging than colored MNIST with features that are built upon all three color channels.
t
sspu
X
Y
2+
The spurious feature is square of color. We sample two colors randomly (one per class) and add a x pixels square randomly positioned in the images (Ap. E). We can vary the percentage of images with the spurious feature reduce or grow the correlation between the spurious feature and the labels. correlation means that all images have a colored square.
Figure 2: of SpuriousCIFAR2 scenario. The good features 24 are generated from the labels and 2spu are gen- erated from the task label.
Data generation model
The scenario is then sequence of SpuriousCIFAR2 datasets
with different colors of spurious features. The test set of each scenario is the binarized version of CIFARI0 test set without any spurious features. We created the sequence of tasks using the Continumm library (Douillard & Lesort, 2021). We illustrate two environments and the test set in Sec. E, and the data generation process in Fig.
Setting Goal: This setting is made to highlight how spurious features can disrupt CL algorithms and discuss the problems that spurious correlation can bring to existing approaches. Moreover, evaluates the capacity of algorithms to modify their past knowledge to improve test accuracy.
4.2 APPROACUES
First, we experiment with a classical vanilla replay (rehearsal) method and simple finetuning. Finetun- ing baseline consists of training with Adam optimizer without any CL mechanism. For rehearsal, the replay buffer is constructed by randomly selecting samples per class. The buffer then sampled to keep class distribution uniform over all classes so far. Balancing samples distribution over classes is made to avoid the challenge of training on imbalanced datasets (cf Sec. I in the appendix for details).
For the existing OOD approaches, we compare continual versions of IRM (Arjovsky et al., 2019) and the state-of-te thr-ttt- OOD classification methods IB-ERM, IB-IRM (Ahuja al., 2021), GroupDRO (Sagawa et al., 2019) and Spectral Decoupling (Pezeshki et al., 2020). OOD approaches are algorithms designed to be trained on multiple environments in multi-task fashion. IRM (Arjovsky al., 2019) (invariant risk minimization) is an approach that uses multiple environments to learn invariant features and improves empiric risk minimization (ERM) generalization. IB-ERM augments this approach with a regularization terms based on an information bottleneck constraint. GroupDRO (group Distributionally Robust Optimization) learns models that minimize the worst-case training loss over the set of environments. (Sagawa et al., 2019) proposes to couple group DRO models with a strong regularization term such as L2 regularization or early stopping to improve generalization. Spectral Decoupling (Pezeshki et al., 2020) proposes regularization term that maximizes the number of features an algorithm learns to avoid relying only on spurious features. The continual version of OOD approaches simulates multiple environments by replaying data of past tasks. The adaptation of all those methods is then to add a replay buffer to the algorithms to train continually the baseline through the sequence of tasks. The replay buffer simulates the growing number of environments for the OOD approaches. In such context, replay (rehearsal) is equivalent to the ERM (empiric risk minimization) baseline in OOD literature. We choose empirically replay buffer storing 100 samples per class for both OOD approaches and vanilla replay (cf appendix F for HPs selection).
4.3 EXPERIMENTS
4.3.1 PROBLEM HIGHLIGHTS
Overfitting the spurious features: To assess that algorithms overfit on the spurious features, we train the model on a single task with spurious features. We compare the test accuracy data without
4

---

Under review as a conference paper at ICLR 2023
spurious features (final test set) with the test accuracy on data with spurious features (evaluation set). the test accuracy is good with spurious features and bad without, the algorithm overfits the spurious features. Fig. 3, show exactly this phenomenon, test accuracy task 0 is near- near-perfect accuracy, while on the final test set, the accuracy is near-random prediction. This figure shows that the artificial spurious features cause the expected learning behaviors: the model overfits and generalizes poorly.
Instability: In Fig. 4, we assess the test accuracy A at each epoch over the whole sequence of 10 tasks. This figure indicates two interesting pieces of information. First, even in the 100% spurious correlation, i.e. all images have a square of colors, baseline models can learn at some point a good solution. Secondly, even when they learn a good solution, they are very unstable and can easily forget good solution. To lighten the influence of instability on the evaluation metrics, in the later experiments of this section, we report the average test accuracy after each task, which
Rinal
Eval
1.0
0.8
0.6
0.4
0.2
we note Q = + Σ == Ar-1-
7-1 A; instead reporting the final test accuracy
finetuning
Figure 3: Overfitting the spurious features sin- gle task setting.
Comparing CL Baselines with OOD Baselines: We now assess how baselines adapted from the OOD field behave in the 100% spurious correlation setting. In our experiments, IB-ERM, SpectralDecoupling and GroupDRo show some interesting improvement over rehearsal and finetuning baselines ( Fig. 5). On the other hand, IRM and IB-IRM performed poorly.
rep lay finetuning
Test Necurac
0.8
0.7
Those experiments show us that we can have some improvement over replay and finetuning baselines when there 100% correlation between spurious features and labels. Note that there is much variance in results because of the instability mentioned earlier. Moreover, the improvements stay far from satisfying accuracy. Indeed, the average performance of the baselines on SpuriousCIFAR2 is below 70% of average accuracy. Meanwhile, trained model on the CIFAR2 dataset without spurious features and reached 96.73% 73% of accuracy. The next experiments will analyze performance while gradually growing the spurious correlations.
8.6
100
200
epach
Figure 4: Generalization and forgetting: test ac- curacy in sequence of tasks (25 epochs per task).
4.3.2 INFLUENCE OF SPURIOUS CORRELATION
In this section, we aim to answer the question "how does the level of spurious correlation influence learning algorithms?". From a stream of tasks, we can expect that the 100% spurious correlation is quite rare. Hence, we will investigate the setting lower spurious correlation in this set of experiments. We study the 25%. 50% and 75% spurious correlation cases along the 100% correlation.
0.8
Test Accuracy
0.6
Fig. 6 show that lowering the spurious correlation makes rehearsal and finetuning (baseline) the best approaches. Those results indicate that the OOD baselines are most interesting in the very high spurious correlation setting but are not very interesting when the spurious correlation is lower or equal to 75%. It might seem counter-intuitive that finetuning is over the best baseline in continual learning setting. Still, only the spurious correlation change our scenario, so the global features needed to solve all tasks are in each task. It is then reasonable that with a lower spurious correlation finetuning works well.
0.4
em
ibb mm
im
reylay
baneine
SpeotDie
GroppRd
Figure 5: Accuracy with local 100% correlation between fea- tures and labels
The scenario could be made harder by only keeping subset of the dataset in each task, i.e., lowering the support the data distribution (cf in appendix B). However, in the experiments, we can note that lowering the support makes the task harder because there are fewer data per task and removes the full observability of data. It means that good/global features are no longer always observable and that some good features can be available in some tasks but not others. Nevertheless, on the support showed that it has a low influence on the results. This is probably because replay makes everything observable at the last tasks and overcomes partial support challenges. It is similar
results
5

---

Under review as a conference paper at ICLR 2023
ib erm
im rehearsal
baseline
GroupORO
1b jim
Decoupling
0.9
Average Accuracy
0.8
0.7
0.6
0.25
0.5
0.75
1.0
spurious corr
Figure 6: Averaged accuracy Ω on 10 tasks over various amount of spurious correlation between spurious features and labels.
to creating several environments with separate splits of the full distribution as in DomainBed datasets (Gulrajani Lopez-Paz, 2020).
4.3.3 POTENTIAL SOLUTIONS TO LOWER IMPACT OF SPURIOUS FEATURES
A solution to prevent models from relying too much on spurious features is (1) to force them learn/use more features (2) try to ignore the spurious features. We experiment with (1) by using regularization strategy to maximize the number of features selected. This regular- ization is different from regularization methods designed to not forget. We experiment with (2) by using model pre-trained on trusted task. The idea of the pre-trained model is that it will ignore noisy features, such as SFs.
33
test socurare task
23
0.6
Yel
No
(a) Pre-training
(b) Dropout
Figure 7: Canceling noisy spurious features with pre-trained models.
We can use a pre-trained model on a trusted data source to ignore spurious features. For example, the spurious feature our setting is noise, so if we use a pre-trained model on known dataset such as CIFARI00, the model filter out noisy features and can significantly improve results. This approach, experimented in Fig. 7a, shows clearly that using pre-traines model erase the problem of noisy spurious correlation. This solution is convenient, but it assumes have a compatible trusted set of data (or a trusted model) and that the spurious features are noisy. More details on noisy features in appendix C.
Using a pre-trained model:
we can not use the previous solution, another potential solution learn select as many features as possible that could help solve the problem. A famous solution to maximize the features learned is dropout. Dropout randomly replaces some activations value with zero for inference force the model learn more robust features. It has notably been widely experimented in continual learning (Goodfellow et al., 2013; Mirzadeh et al., 2020). We experimented 0.25, 0.5, and 0.75 amount of dropout just before the last linear layer. However, in our experiments, it did not show any improvement (cf. Fig. 7b).
Maximizing the amount of features selected:
Nevertheless, on similar idea as dropout, the spectral decoupling approach designed to address the gradient starvation problem. The gradient starvation problem arises when the loss is minimized by capturing only a subset of features relevant to the task, despite the presence of other predictive features that fail to be discovered (Pezeshki et al., 2020). Spectral decoupling is designed to discover supplementary features even with minimal train error. As dropout, it enables the possibility to learn additional features that could help to improve the test error. The experiment in section 4.3.1 illustrated in Fig. 5 indeed shows that in the 100% spurious correlation experiment, this strategy greatly improves simple rehearsal proving the potential of the idea.

---

We train linear classifier per task (multi-head) Hence, while training, inference considers one head at a time for classification. The softmax function is applied only to the classes of a single head. We note the test performance in multi-head Ate-tocal_en too-toool_sstaaa softmar- It estimates the capabilities of the features selected to solve tasks independently. After each task, we freeze the weights of past heads to avoid forgetting. After training, we concatenate all the heads into a single linear classifier and evaluate this classifier the whole scenario (the test performance with single This second evaluation estimates the generalization capabilities of Under review as a conference paper at ICLR 2023
Conclusion We proposed some trivial solutions to illustrate how supplementary knowledge or assumptions on the spurious features might help prevent or fix bad learning behavior. However, would probably not work as easily in setting with more complex spurious features.
Experiments in sections 4.3.2 and 4.3.3 investigate how algorithms learning continually can deal with spurious features. We created benchmark with spurious features and empirically investigated algorithms' performance when varying the correlation of spurious features with labels. In the next section, will investigate local spurious features, which are type of spurious feature specific to CL. We will investigate if those features may cause a performance decrease in CL algorithms. We shown that models can overfits spurious features easily, in the next section, we investigate if local spurious features (LSF) can exist in exisiting original datasets, and if they can be easily be overfit and influence models performance.
INFLUCENCE OF LOCAL SPURIOUS FEATURES (LSF) IN CONTINUAL LEARNING
In the previous section, we study how spurious features impact continual learning algorithms. this section, we show that local spurious features (LSFs) may also lead to performance decrease, even scenarios composed of usual data.
5.1 LOCAL SPURIOUS FEATURES SETTING
Our goal to investigate if some LSFs exist in continual learning scenarios without being manually added. To demonstrate their existence, we will show that they influence the results. Their influence is incarnated by a model that learns independent tasks correctly, but that can not generalize to tasks together.
Data: We experiment with classical datasets without modification to create glass-incrementer sce- narios in most continual learning literature. We use CIFARIO. 10, OxfordPet, OxfordFlowers and CUB200 datasets with pretrained models: a resnet model on CIFAR 100 for CIFAR 10 and for the other datasets
VGG, Resnet, Alexnet and googlenet pretrained on Imagenet from torchvision library. We create the scenarios by splitting each dataset into 5 tasks with disjoint sets of classes.
Model: We use frozen pretrained models: resnet model CIFAR100 for CIFARIO and for the other datasets we use VGG, Resnet, Alexnet and googlenet pretrained on Imagenet from torchvision library. We assume that the pre-trained models provide feature space sufficient for the continual downstream scenarios. They are used frozen with a linear classifier on top. We also assume that algorithms select features that correlate well with labels to solve the current task. The goal is then to show that classifiers rely on local spurious features while learning continually. If it is the case, this will lead to a good performance on the tasks but not the full scenario. The training realized with simple finetuning without any replay or regularization.
Approach:
head is noted gioballooolfftaaaa the features selected.
Hence, classifiers select local spurious features, Ate- sof tmax should be significantly bigger than Ate-global so ftmas-
5.2 DISENTANGLING FACTORS OF INFLUENCE
Apart from the problem of feature selection, a gap between Ate-local softmaz and Ate-global so f'mmo could be explained by (1) the difference in difficulty of both evaluation (multi-head and single head), (2) unbalance of bias and norms from different heads that could lead to bad performance in single
7

---

Under review as a conference paper at ICLR 2023
head (Lesort et al., 2021) (more details about this problem in appendix G). However, if neither (1) (2) are sufficient to explain the performance gap, then we can conclude that LSFs exists.
(1) Difficulty gap: the comparison between single head and multi-head is biased because the first is one 10-way classification (harder) while the latter is the addition of five binary classifications (simpler). To estimate the difference of difficulty between both, MeanLayer as in (Lesort et al., 2021) which is an nearest mean classifier (NMC). There is no feature selection in MyanLayer. The classifier only uses the mean of the features of each class. There is no feature selection then no problem with local features selection. The difference in performance with MeanLayer in multi-hear and the single head is then a good proxy to estimate the difference in the difficulty of both evaluations.
added non-parametric method,
(2) Unbalance of bias and norms: we compare the linear layer performance with the weightnorm layer from (Lesort et al., 2021). This layer does not use norm and bias for inference and is then insensitive to such imbalance (details in appendix G).
We note that forgetting is by design impossible here since the features extractor and the other heads are frozen. Hence, forgetting can not explain drops in performance.
5.3 LOCAL SPURIOUS FEATURES EXPERIMENTS
Mick
Figure 8: Comparison of final test accuracy Ar-1 in multi-head and the same classifier without task label information (In this order: CIFAR10, CUB200, OxfordFlower 12, OxfordPet). The results averaged among several pretrained models. The performance gap in the linear layer indicates how local features selected are generalizable. We use the weightnorm to estimate if bias or norm imbalance phenomena (Lesort al., 2021) play role in the performance gap The MeanLayer baseline indicates the difference of difficulty of both evaluations. The results of this figure indicate that classifiers rely on local spurious features solve tasks.
We report results in Figure 8. We remind that only the inference method changes between single head and multi-heads results. The parameters of the classifiers are the same (cf Sec. 5.1). The details of the results are in appendix H. We see that there a considerable accuracy gap between multi-head (MH) Ate-local softmaz and single head (SH) Aleobl softmez performance in the linear layer. Moreover, the gap for MeanLayer is significantly smaller than the gap for the linear layer. Hence, the difference in difficulty is not sufficient to explain the accuracy gap of the linear layer. And finally, the results are similar for linear layer and weightnorm which show that the norm/bias unbalance is not a problem that could explain the performance gap. Therefore, as described in Sec 5.1, we can conclude that the classifier selected local spurious features, which leads to a poor generalization and a low single-heat performance.
These phenomena prove that the drop in performance in dlass-incremental can be due to a bad feature selection and not necessarily to forgetting. An explanation for this that in CL, the unavailability of data from future tasks makes it impossible to know the features that generalize to the whole scenario. Therefore the model solves the tasks based on currently available data and can only verify later if the feature selected are correct. This is fundamental observation for future continual learning approaches as it means that the features learned and selected should not necessarily be preserved for later tasks. While the continual learning bibliography focused mostly on forgetting or transfer, the LSF problem is also a great challenge to deal with to create efficient CL.
Another interesting insight that this experiment gives is that contrary to experiments in section 4.3.3 which show that using a pre- pre-trained model can help for spurious moariuuseaturee features, it does not offer a solution to local spurious features. A good feature extractor is then not sufficient for good feature selection.
8

---

Under review as a conference paper at ICLR 2023
5.4 EFFECT OF NUMBER OF CLASSES PER TASKS
We have seen in the previous experiments that local spurious features are due to wrong selection of features. Nevertheless, if increase the complexity of a task, for example, by increasing the number of classes per task, then the model is forced to learn more about the data and select more features, building more complete representation of the data. In this experiments, we investigate the influence of local spurious features with various number of classes per tasks. The hypothesis is that increasing the number of classes should reduce the influence of local spurious features. We create scenario of 5 tasks with various number of classes per tasks from tiny-Imagenet and CUB200 and train only classifier on top of a pretrained model in a similar way as in previous experiments (Sec. 5.1).
Fig. 9 reports the difference Head for
pre-tocol_son softmoz - Ate-gloool_snffte various number of classes per task. For cach scenario, one task represents 20% or the total number of classes. Hence, the difference of diffi- culty between solving tasks separately (MH) and together (SH) is assumed to be marginal. The results with MeanLayer in Fig. confirm this hypothesis; the gap stays quite stable with num- ber of classes per task. On the other hand, Fig. 9 shows that increasing the number of classes per task reduces A Head, which means that the in- fluence of local spurious features is reduced, In practice, the algorithms should be more careful about local spurious features when the number of classes in a task is low.
Outlayer
Deta Test Acourast
0.5
QutLayer
Dela Jest/Acurag
Uiner
0.4
Linear
0.4
Me Layer
Meanlayer
0.3
0.2
0.2
0.1
0.0
Tasks
Taskcs
Figure 9: Delta accuracy between MH and SH ac- curacy. The effect of LSF reduces with the number of classes per tasks. CUB200 (left), Tinylmagenet (right), 5 tasks per scenario.
Conclusion: In this section, we have shown that local spurious features (LSF) exist. We proved it by highlighting their influence on a classifier on top of frozen pretrained models. We consider that is sufficient to assume that the same problem would also exist when training the models end-to-end. We also showed that the influence of LSF depends on the number of classes inside a task. The more classes there are inside task, the fewer influence LSFs have.
6 CONCLUSION
Continual learning algorithms are built to learn, accumulate and memorize knowledge through time to reuse them later. Memorizing bad features can have catastrophic repercussions on future performance. Then, to learn general features, algorithms need to deal with spurious and local spurious features.
This paper first investigates the impact of spurious features on continual learning. Algorithms easily overfit spurious features for or several tasks, leading to poor generalization. Our goal to investigate if continual learning algorithms can benefit from the variation of the spurious features through time ignore them. Our results show that classical continual learning approach such as rehearsal can deal with spurious features until certain level of correlation with labels. Secondly, we investigated if local spurious features exist in usual CL scenarios and can compromise continual learning. Those features are supposed to correlate spuriously with labels within task only. Our results show that they are indeed present, and they can influence performance in CL significantly.
In the continual learning literature, performance decrease is generally attributed to catastrophic forgetting. Our results show that the problem of local spurious features also plays a major role. More research is needed to understand better the impact of local spurious features along with catastrophic forgetting. Understanding this phenomenon is critical to better address forgetting and feature selection and enable efficient continual learning.
We recommend using replay as a practical solution to reduce SPs and LSPs influence. Indeed, the replay process can simulate an identically and independent distribution of tasks seen so far. It makes it possible to fix potential mistakes in past feature selections. Nevertheless, even if this solution already exists, it significantly increases the computational cost when the number of tasks grows. Therefore, better understanding of forgetting and spurious feature in CL could help to optimize it.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Kartik Ahuja, Ethan Caballero, Dinghuai Zhang, Yoshua Bengio, Ioannis Mitliagkas, and Irina Rish. Invariance principle meets information bottleneck for out-of-distribution generalization. arXiv preprint arXiv:2106.000 06607, 2021.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:197.0028900 2019.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL hattps://www wandb. com/. Software available from wandb.com.
Arthur Douillard and Timothée Lesort. Continuum: Simple management complex continual learning scenarios. 2021. URL https: http//axxivvv..///. wwt.abs////oooooooooooo....... 220.. 2212..0653...
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled
outputs distillation for small-tasks incremental learning. In Proceedings of the IEEE European Conference on Computer Vision (ECCV), 2020. URL https://////// /ttp:///wwwwwww// www..ecva.net. www.aa..eeaaaeeeeeeeeeeess eccv_2020/ ppppppppcce htt.apapperr/1350065555 123650086. pdf.
I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, and Y. Bengio. An empirical investigation of catastrophic forgetting gradient-based neural networks. ArXiv e-prints, dec 2013. URL
https: /tttss:/arrivv org//bgg/s... abs//311 1112. 6211.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. arXiv preprint arXiv:2007.01434, 2020.
Isabelle Guyon and André Elisseeff. An introduction variable and feature selection. Mach. Learn. Res., 3(null): 157-1182, mar 2003. ISSN 1532-4435.
Khurram Javed, Martha White, and Yoshua Bengio. Learning causal models online. arXiv preprint arXiv:2006.07461, 2020. URL htw.t:ttp//// http//axvvvv///aa///v///eeaa.aaa wwwp abo////....///./////////// 07461.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho. Agnieszka Grabska- Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proc. of the national academy of sciences, 2017. URL
https: www. pnas org/ wwwtent/ttttccccccccccc pwtpt//aaa///aaaaa / 114 /1///33/5222 full. full.pdf.t
Timothée Lesort, Thomas George, and Irina Rish. Continual learning in deep networks: an analysis of the last layer. arXiv preprint arXiv: 106::2206.008344 2021. URL https::s: httppps::///aiiiie http//axivvvc///// aba/ 21 2..66.0 01834.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, and Hassan Ghasemzadeh. Dropout implicit gating mechanism for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 232-233, 2020.
Mohammad Pezeshki, Sékou- Oumar Kaba, Yoshua Bengio, Aaron Courville, Doina Precup, and Guillaume Lajoie. Gradient starvation: learning proelivity in neural networks. arXiv preprint arXiv:2011.09468, 2020.
Ameya Prabhu, Philip HS Torr, and Puneet Dokania. Gdumb: A simple approach that questions our progress in continual learning. In European Conference on Computer Vision, pp. 524 540. Springer, 2020.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. In International Conference Learning Representations, 2019. URL /ttpp:://penrest met/fose http:/oormmmddddmm (it.rrmmiidsllgShhhhtt....
Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured laplace approximations for overcoming catastrophic forgetting. In Advances Neural Information Processing Systems, pp. 3738-3745 2018.
10

---

Under review as a conference paper at ICLR 2023
ib_es _0_erm ib_im
irm
baseline Spectralleccupping
GroupDRO
rehearsal
0.9
0.8
Average Test Accuracy
0.7
0.6
0.5
0.4
0.2
0.4
0.6
0.8
1.0
support
Figure 10: Averaged accuracy Ω on 10 tasks over various amount of support, the spurious correlation set to 0.75.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang, Distributionally robust neural networks. International Conference on Learning Representations, 2019.
Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and fairness in class incremental learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 13208-13217, 2020.
Hattie Zhou, Ankit Vani, Hugo Larochelle, and Aaron Courville. Fortuitous forgetting in connectionist networks. In International Conference on Learning Representations, 2022. URL https:/ openreview n forum? dd 2
A COMPUTE
The experiments were run on internal cluster Quadro RTX 8000 GPUs. The total time of compute for an approximate period 100 days of GPU use with hyper-parameter selections and experiments.
B LOWERING THE SUPPORT OF THE DISTRIBUTION AT EACH TASK
B.0.1 LOWERING SUPPORT AMOUNT
In experiments Sec. 4.3.2, the same data support was used for all tasks. This means that the shared support of original data is 100%. However, expect continual algorithms learn with less support. The support of task is the percentage the full original distribution within the task. We investigate in those experiments the influence that the support amount has learning algorithms. We set the spurious correlation to 75% in those experiments.
As it is designed in the SpuriousCIFAR2 scenario, each class (0 and 1) has 5 distributional modes corresponding to the original classes. In the previous experiments, all the data for all those modes available in all tasks only the spurious features changed from one task another.
11

---

Under review as a conference paper at ICLR 2023
In those experiments, reduce the support by subsampling the original data. We select the data of a subset of the CIFARIO 10 classes for each task. For example, we select only cars for class 0 and only deers for class 1, instead of airplanes, cars, trucks, ships, and horses for class 0 and birds, cats, dogs, deers, and frogs for class 1. Hence, if we use the support of 0.2 (i.e. 20%), we will only select the data of * 0.2 1.0 original classes for class 0 and one other for class 1. For simplicity's sake, we will use only support compatible with the number of classes to have a round subset, i.c., [0.2, 0.4, 0.6, 0.8, 1.0]. At each task, the support is randomly sampled, hence the same original data can be in several tasks, but the spurious features will still be different for all tasks.
The support experiments results, illustrated in Fig. 10, show that all support settings, the finetuning baseline and rehearsal are the best performing methods. On the other hand, contrary to what would be expected, the amount of support seems to not play an important role in the final performance, at least in the range of supports possible in our scenario. This is probably because doing replay converts a partially observable setting into a fully observable setting by simulating an iid distribution.
C SPURIOUS CORRELATIONS: DIFFERENT CASES
We have defined the different types of features in Sec. 3. We now identify different cases among the spurious correlation between features and labels. Those different cases can either apply to SF or LSF.
Data Observability: (1) Fully observable data (Javed et al., 2020) The global features are always present and always observable. In this case, we assume that features in data that do not last are spurious, and we can learn to ignore them. (2) Partially observable data The global features' presence is not invariant. In this case, features that do not last can either be spurious or good.
Noisiness of Spurious Features: (1) SFs are irrelevant for other classes they can be ignored completely without affecting the learning process. We will refer to them as noisy SFs. (2) SFs are good features for other classes, e.g., for classification, the color can be a spurious feature for some classes and valuable for others. We can not ignore those features since it could lead to poor performance in other classes.
In our experiment, we propose settings with fully observable and partially observable data. Our spurious features are noise in our domain incremental experiments and true features in our class- incremental experiments. Nevertheless, we do not exploit information about spurious feature types to design approaches.
D DISCUSSION
Spurious Features vs Local Spurious Features: Spurious features and local spurious features lead to the same problem for learning algorithms: overfitting features that not discriminative. The difference is that spurious features result from a covariate shift between training and test data. In contrast, local spurious features are due to the unavailability of all data while learning. Local spurious features are then, more specifically, a continual learning challenge, and we showed that they could have a significant impact classical continual learning scenarios.
Solutions to spurious features: The problems of spurious features and local spurious features lead to phenomena where forgetting helpful improve final performance (Zhou al., 2022) also denoted as graceful forgetting. Indeed, forgetting by reinitializing some weights allows escaping spurious local minima and relearning a better solution taking new data and knowledge into account. On a more general note, spurious features and local spurious features make ineffective approaches that are too rigid and unable to modify and fix previously learned features/knowledge. The replay methods have been known to be good solution for many continual learning problems. Our findings do not challenge this approach. However, it could be of some use to make replay approaches more effective and reduce the need for replay.
The scenario EpuriousCIFARE proposed in this paper has been designed to highlight the problem that spurious correlation might create in continual learning. This scenario plainly fulfills its task of disturbing CL algorithms, particularly in the 100% correlation setting. However, it can not be used as benchmark to evaluate the robustness of algorithms. The spurious features are very
Benchmark:
12

---

Under review as a conference paper at ICLR 2023
simple, and a simple ad hoc processing of data could solve this scenario, i.e., encoding data with pre pre-trained model as in Fig. 7a. proper benchmark to assess robustness to spurious correlation would propose spurious features easy to learn by the model but harder to detect ignore than simple squares color. DomainBed datasets (Gulrajani & Lopez-Paz, 2020) are an interesting set of benchmarks for spurious correlation investigation. However, the amount of features is not controllable, making it harder to evaluate the limits of algorithms.
E SAMPLES SUPPORT EXPERIMENTS
(a) Train Valid data
(b) Train / Valid data task
(c) Final Test
Figure 11: Samples for support experiences, here with 20% support, i.c. the data of only two of the original classes in each task.
F
SELECTION
For fair comparison between algorithms originally designed for continual learning, such as replay, and OOD algorithms, we conduct the hyper-parameter search more intensively for OOD approaches.
For each OOD approach, search for the best hyper-parameters in the range proposed in the DomainBed GitHub repository We also searched for the best learning rate with the bayesian method of wandb (Biewald, 2020). We used approximately 100 runs for each OOD baseline to select typer-parameters. The scenario for hyper-parameters selection was an OOD setting with environments SpuriousCIFAR2 with 75% correlation but all simultaneously available (with no continual stream of tasks).
The hyper-paramend parameters for rehearsal and finetuning (baseline), have been selected on finetuning training in single task setting with 75% of correlation. The number of samples per class for replay has been selected on task SpuriousCifar2 scenario with 75% of correlation. The optimizer used was Adam with a learning rate of 0.06 (others HPs are default ones in pytorch library).
G BIAS NORM IMBALANCE
As described in (Lesort al., 2021): A linear layer is parameterized by a weight matrix and bias vector b, respectively of size N x h and N, where h is the size of the latent vector (the activations of the penultimate layer) and the number of classes. For latent vector, the output layer computes the operation where (-) the euclidean scalar product, A, is the ith row of the weight matrix viewed as a vector and 6; is the corresponding scalar bias.
A= + b. We can formulate this operation for single class i with (z, A,) + by Ox-
can be rewritten:
| |+ cos( (((2, + by o,
(3)
Where ((,-)) the angle between two vectors and ||-|| denotes here the euclidean norm vector.
'https://gii thw.ihhhbb.... com htt.c/////ccccco.eeeeeeeeeeeeeee Doma niedd
13

---

Under review as a conference paper at ICLR 2023
Then, at inference time, y/4 argmazz(o;) rely the norm of ||AA and on the bias bj- Within single task, ie. within a single head in multi-head setting. ||A,]| and by are balanced to predict class conrectly. However, we can not ensure that ||A:]| and b, will are not biased from one head to another.
To avoid unbalance for bias and norm for inference, (Lesort et al., 2021) proposed the weigthnorm layer where: ||= (2s, A;)) o, and show that this layer in efficient in learning in incremental and lifelong settings.
H DETAILS MULTI-HEAD EXPERIMENTS
Wat--aad Segainsc
(a) alexnet OxfordFlower102
(b) vgg
(c) googlenet
(d) resnet
OxfordFlower102
OxfordFlower100
OxfordFlowerr 102
Uiear
Negyslor
Libarr
7.0 Meggtaore
Waggstorr
2.0
69
Seporew
00
86
00
(e) alexnet OxfordFlower102
(f) vgg OxfordPet
(g) googlenet OxfordPet
(h) resnet OxfordPet
Uiaa
Negysemm
Meggacrr
Maggstrr
Waggstrr
000
Verieble
0.
Matt-aad
Organinc
00
Vartuble
Sira-inn:
vottabl
(i) alexnet
(i) vgg
(k) googlenet
(1) resnet
CUB200
CUB200
CUB200
CUB200
Figure 12: Local spurious features experiments: after training in multi-head way, we compare accuracy between multi-head (soft-max applied on a subset of classes' outputs) and single head (softmax applied on all classes' outputs). The performance differences assess how the model selected local spurious features to solve tasks. Two baselines are added, (1) meanlayer, which assesses the difference in the difficulty of the two evaluations, and (2) weightnorm, which assesses if the performance difference is due to an imbalance in norm or bias. The results in this figure show that models indeed rely on local spurious features to solve tasks.
I SAMPLING ALGORITHM
Algorithm 1 describes the sampler used train the model with replay. The input dataset D is concatenation of the buffer with the current data. The goal is to make the probability of sampling on each class, the same whatever the number of sample for each class in D.
14

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Balanced Sampling of Data Mixture.
1: procedure SAMPLER()))
2: < '.yy
D Cett data class labels
3: nb per class = binsouutt))))(
4: weights per class nb-per_closs
5:
wemple-weights = weights..er..clas// _lr_classly]
D sample probubility weight each data create sumpler to sample accordingly to the
6:
sampler = Sampler(D, sample.weights, weights, replacement-True)
neighe.weights
return sampler
8: end procedure
15


Reviews:
[{"review_id": "evGrEkHEDn", "reviewer": "Reviewer_mcta", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper offers an extensive empirical evaluation of different methods for continual learning (CL). It shows that some methods are unable to find what the paper calls good features. That is features that generalize across tasks and also to the test set. The algorithms are evaluated on various datasets and under different amounts of spurious correlation between spurious features and labels. Insights into why some algorithms work better than others and under what conditions are also given. Moreover, the paper distinguishes a new type of correlation that the authors call local spurious correlation. The consequences of this on model performance are also investigated empirically.\n", "summary_of_the_review": "I already reviewed this paper for NeurIPS. There have been no major changes to the paper aside from an additional section (5.3). Hence, I am copy-pasting my previous review and adding a few comments about Section 5.3. Overall, I think the paper is still not ready for publication. Although there are some interesting insights the paper does not have a clear structure which makes it hard to parse.", "strength_and_weaknesses": "### Strengths\n\nThe paper does a nice job of classifying the different types of features you can find on CL tasks (spurious vs. good, and local vs. global). These are not only described with intuitive examples but also defined mathematically.\n\nThe experimental evaluation is thorough. The paper also provides valuable insights that might be useful for the community. Such as the fact that using a rich enough replay buffer is sufficient when the amount of spurious correlation is low enough; or the idea of using regularization in the opposite way (forcing the model to use more features). The paper also investigates simple tricks that help mitigate the problem of spurious correlations. Such as using a pre-trained model when possible, or using spectral decoupling to maximize the number of features selected.\n\n### Weaknesses\nThe paper does not contribute any new solution to the problem of spurious correlations in CL. I can foresee that this is also going to be the main point of criticism from other reviewers. In my opinion, proper analysis of the problem and empirical evaluation of existing methods should be sufficient for acceptance. That being said, CL is not my area of expertise. Some of the insights that the paper brings forward were interesting and valuable for me but I am not sure to what extent these were already known by the community.\n\nGiven that the main contribution of this paper is the empirical evaluation of existing approaches and solutions to the problem of spurious correlations, I feel like the experiments section should be very clearly structured such that future researchers and practitioners can use it as a reference when facing similar types of problems.\n\nSection 5 is very hard to follow. A new approach (multi-head), a set of experiments, and problems with the comparison (single head vs. multi-head, bias, and norm unbalance) are all described in the same place. I suggest breaking things down into multiple subsections such that the structure is more clear. There are also a lot of repetitions. My suggestion would be to first list what things you are investigating, then explain how these are investigating (maybe also explaining why the comparison is hard), then describe the results, and finally summarize the key points.\n\n**Comments on Section 5.3:**\n\nI'm having trouble understanding the difference between multi-head (MH) and single-head (SH). Where is this explained? The first reference to this is in Section 5.1: \n\n*\"We use a multi-head approach to see if the classifier learns local features. Hence, while training, we only apply the softmax function to current classes outputs. (...)  After each task, we freeze weights of past heads to avoid forgetting. After training, we compare the test performance with the same classifier (i.e., the classifier trained with multi-head) but with the softmax applied on all\noutputs Ate−global softmax,\"* What does this mean? \n\nDo I understand correctly that in single-head you use the same layer for all tasks, while for multi-head you replace the layer at each task during training?\n\nIf I understand correctly, the takeaway from this experiment is that if you increase the number of classes that the NN needs to predict in each task then it is less prone to overfitting to spurious features and can perform better on the test task. I feel this is somewhat obvious. Am I missing something?\n\nFigure 9: which one is single-head and which one is multi-head?\n\n\"For each scenario, one task represents 20% or the total number of classes\" What does this mean?\n\n**Minor details:**\n\nMove Figure 5 before Figure 6.\n\nI suggest moving Section 4.3.3 before the experiments. It is weird that the methods are described after the results are reported. Also, including a brief description of all the other baselines would be useful.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Y3CAjTZvPu", "reviewer": "Reviewer_g5vD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is an analysis paper. It experimentally shows that spurious features and local spurious features harm continual learning.  ", "summary_of_the_review": "This is purely an analysis paper, but I fail to see any new insights from the analysis. The main issue is that the authors are unaware of many existing continual learning papers, which have already discussed and dealt with the problems that they pointed out. ", "strength_and_weaknesses": "Strength\nThe paper experimentally demonstrates spurious features harm continual learning, in particular class-incremental learning. This is valid. \nThe writing is not bad. \n\nWeaknesses\n1. Your examples of spurious features seem confusing. For the general spurious features, you give the example, “… but in the training data, all cars are red, and all bikes are white, but in test data, both are in a unique blue not available in train data. A model could easily overfit the color to solve the task while it is not discriminative in the test data. In CL, we would expect future tasks to bring pictures of cars and bikes of other colors to learn better features.” When you present local spurious features, you said “because only red cars and white bikes are currently available in train data, but in the test data, there are also cars and bikes of colors not seen yet or seen in the past,” I fail to see the difference between the two examples for different types of spurious features. \n\n2. The above examples have two problems:\n\n(i). Spurious features apply to any supervised learning, not specific to continual learning. The problem has been investigated extensively in supervised learning and thus it has not been the focus of the continual learning except (maybe) in online continual learning. \n(ii). In the continual learning context, you never know what classes will come in the future. The color features may not be the best for the current task but may be useful to some future tasks. Thus, calling the color features spurious is not justified as continual learning is different from traditional supervised learning. This problem has been discussed in [a] and many approaches have been also proposed to deal with it [a, b, c], \n\n(a) Guo et al. Online Continual Learning through Mutual Information Maximization. ICML, 2022. \n(b) Zhu et al. Class incremental learning via dual augmentation, NeurIPS, 2021.\n(c) Zhu et al. Prototype Augmentation and Self-Supervision for Incremental Learning, CVPR-2021.\n3. I don’t think that the types of features you analyzed in your examples should be called spurious features because they are not un-useful. They are just not enough. True spurious features should be ignored. \n\n4. In section 4.3.3, you give two recommendations, (i) using pre-trained model and (ii) maximizing the amount of features selected. These have been done. For (i), please see the following two papers and for (ii), please see [a, b, c]. \n\n(d) Kim et al. A Multi-Head Model for Continual Learning Via Out-of-Distribution Replay, Collas’2022. \n(e) Wang et al., Learning to prompt for continual learning. CVPR’2022\nIn fact, all continual learning papers in natural language processing use pre-trained models. In computer vision, papers are starting to use pre-trained models too. \n\n5. The paper proposed a modified version of Out-of-Distribution (OOD) generalization methods for continual learning and evaluated using the SpuriousCIFAR2 data, but this has been done and successfully used in continual learning, please see [d, f]. \n\n(f) Kim, et al. Continual Learning Based on OOD Detection and Task Masking. CVPR workshop, 2022. \n6. For the analysis of local spurious features, your setting is not a continual learning setting as you treat each task independently, which clearly has no forgetting, but that is not continual learning. Your pre-trained model has label and information leak because some of your datasets are closely related to or overlapping with the ImageNet data. In your multi-head setting, the local spurious features are obviously. \n\n7. For the local spurious features, which are the main problem of CIL, you did not give any new recommendations to solve the problem except saying replay is a practical solution. That is true, but that is what replay methods are designed for. I don’t see any new insights. \n\n8. Please state explicitly which problem you are dealing with, class-incremental learning (CIL) or task-incremental learning (TIL). I believe that you are dealing with CIL based on your problem formulation and later analysis. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ANCmDpvwjI0", "reviewer": "Reviewer_9EtC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study the problem of spurious correlations during continual learning. They divide the problem into two cases: the typical spurious correlation setting, and a local spurious correlation setting where the spuriousness is due to limited data within each task. They conduct experiments for each of the two cases, and compare with baselines from domain generalization. They conclude that spurious correlations may lead to, and may be responsible for, large performance drops during continual learning.", "summary_of_the_review": "The paper examines an interesting and previously unstudied problem. However, the scope of the paper is rather limited (Weakness #6), and there are major issues with the empirical evaluation (Weaknesses #1-3), as well as with the clarity of the paper (Weaknesses #4-5). For these reasons, I recommend rejection. ", "strength_and_weaknesses": "Strengths:\n- The paper studies the intersection of two important machine learning problems (spurious correlations + continual learning), which has not been examined in prior work to the best of my knowledge.\n\n\nWeaknesses:\n1. The paper misses a large set of recent works in the spurious correlation literature, including but not limited to [1-5]. Many of these works should be included as benchmark methods, and the result in [4] specifically might be useful to explain some of the findings in Section 5.\n\n2. The experimental setting in the paper is rather limited. The experiment in Section 4 is only conducted on a single semi-synthetic dataset, which is not one of the common spurious correlation datasets from the literature. The authors should expand this experiment to standard datasets like Waterbirds and CelebA, where the spurious correlation can vary across tasks by supersampling or by introducing it semi-synthetically. \n\n3. The experiment in Section 5.1 is confusing to me. As the authors do not know the level of spurious correlations in base CIFAR10 or CUB200, it is unclear to me how the authors are disentangling the effect of the local spurious correlation. The authors should consider exploring the effect of introducing semi-synthetic spurious correlations in these datasets. The authors should also evaluate mitigating algorithms in this setting.\n\n4. The clarity of the paper is poor in general. In particular, the setup of the experiment in Section 5.1 is hard to understand, and the formulation of spurious features in Section 3 can be improved. For example, $C_T$ should be defined formally along with the correlation function, and the definition of $z_+$ (Eq. 1) seems restrictive as it does not take non-linear feature interactions into account. The authors should consider defining their problem setting more formally, such as the formulation in [6].\n\n5. There are many spelling and grammar mistakes throughout the paper. The language used in the paper should also be made more formal in general. For example, \"good feature\" should be replaced with \"invariant feature\". The \"test data\" is also overloaded (as is clear in the first paragraph of Section 4.3.1). The authors should also clearly state which methods require knowledge of the spurious attribute, and which do not.\n\n6. The scope of paper is quite limited. The authors evaluate existing methods in a couple of scenarios, but do not provide any novel methods for addressing the problem.\n\n[1] https://arxiv.org/abs/2107.09044\n\n[2] https://openreview.net/forum?id=jphnJNOwe36\n\n[3] https://proceedings.neurips.cc/paper/2020/file/f1298750ed09618717f9c10ea8d1d3b0-Paper.pdf\n\n[4] https://arxiv.org/abs/2204.02937\n\n[5] https://arxiv.org/abs/2201.00299\n\n[6] https://arxiv.org/abs/2010.15775", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "20shRxzKI8c", "reviewer": "Reviewer_22hc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper mainly studies about the spurious features (SF) and local spurious features (LSF) in continual learning. It is an analyses paper without a method to address the observed problem. The paper considers multiple settings involving SF and LSF and claims that the LSF problem is the main cause for the performance degradation in continual learning.", "summary_of_the_review": "Please see the comments above. Overall, I do not think the paper meets the high bar of ICLR since several details and justification are missing. ", "strength_and_weaknesses": "Strength:\n- The paper proposes to consider new notion of LSF in continual learning problems. \n- The paper attempts to produce various data settings to validate their claim. \n\nWeakness:\n- First of all, since this is an analysis paper, I think the experimental setting for each figure should have been more clearly presented. \n   - For example, in each figure, are test accuracies all obtained from the final test set? \n   - How are tasks formed in the continual learning experiments?\n   - When the average accuracy is computed, how did you select the hyperparameters? (based on the final acc? this may not make sense in the continual learning setting.)\n- It is not entirely clear whether the SF and LSF really exist in practice. When exactly does this features exist? Can you characterize them?\n- Setting for the SF experiments\n   - Only the setting with same correlation level across tasks is considered. What about when the correlation levels are different across tasks? What happens then? \n- Trivial results in Table 5\n   - When the spurious correlation is low, the baselines without bias remedy perform well, while when the correlation is high, group dro is performing well. This might be a common result even for the single task, and what information does this result additionally give for the continual learning setting? \n- Unclear definition of LSF? \n   - LSF is defined as features that can be useful for a specific task, but are spurious for *all* tasks. But, this definition would also classify the useful features as LSF as well. For example, when a feature is useful for a certain task of classifying cat from dog, it could become LSF when the all tasks involves two breeds of dogs. Is it really LSF? \n- LSF results are unclear\n   - The paper considers Task-IL setting and compares the performance between single-head and multi-head structures. The conclusion is that since Multi-head structure has high accuracy, LSF is causing a problem in single head. But it seems like there is some logical gap in this result -- multi-head might be using more information to achieve high acc, but there is not consideration for this aspect. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject