Continual Learning in the Presence of Spurious Correlations: Analyses and a Simple Baseline
Authors: Donggyu Lee
Sangwon Jung
Taesup Moon
Venue: ICLR 2024 poster (2024)
Keywords: continual learning
bias
spurious correlation
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: We first consider continual learning problem in the presence of spurious correlations.

Abstract: Most continual learning (CL) algorithms have focused on tackling the stability-plasticity dilemma, that is, the challenge of preventing the forgetting of past tasks while learning new ones. However, we argue that they have overlooked the impact of knowledge transfer when the training dataset of a certain task is biased — namely, when the dataset contains some spurious correlations that can overly influence the prediction rule of a model. In that case, how would the dataset bias of a certain task affect the prediction rules of a CL model for future or past tasks? In this work, we carefully design systematic experiments using three benchmark datasets to answer the question from our empirical findings. Specifically, we first show through two-task CL experiments that standard CL methods, which are oblivious of the dataset bias, can transfer bias from one task to another, both forward and backward. Moreover, we find out this transfer is exacerbated depending on whether the CL methods focus on stability or plasticity. We then present that the bias is also transferred and even accumulates in longer task sequences. Finally, we offer a standardized experimental setup and a simple, yet strong plug-in baseline method, dubbed as group-class Balanced Greedy Sampling (BGS), which are utilized for the development of more advanced bias-aware CL methods.

Published as a conference paper at ICLR 2024
CONTINUAL LEARNING IN THE PRESENCE OF SPURI- OUS CORRELATIONS: ANALYSES AND A SIMPLE BASE- LINE
Donggyu Lee'*, Sangwon Jung" Tacsup Moon
Department of Electrical and Computer Engineering. Sungkyunkwan University 2Department of Electrical and Computer Engineering, Scoul National University 3 *ASRI/ INMC IPAI / AIIS, Scoul National University edu, 1dk3080skkuu edu, {s jung, tsmoon ) & snu. ac.
ABSTRACT
Most continual learning (CL) algorithms have focused on lackling the stability- plasticity dilemma, that is. the challenge of preventing the forgetting of past tasks while learning new ones. However, we argue that they have overlooked the impact of knowledge transfer when the training dataset of certain task is biased - namely, when the dataset contains some spurious correlations that can overly influence the prediction rule of model. In that case, how would the dataset bias of certain task affect the prediction rules of a CL model for future or past tasks? this work, we carefully design systematic experiments using three benchmark datasets to answer the question from our empirical findings. Specifically, we first show through two-task CL experiments that standard CL methods, which are oblivious of the dataset bias, can transfer bias from one task to another, both forward and find out this transfer is exacerbated depending on whether
backward. Moreover, the CL methods focus on stability or plasticity. We then present that the bias is also transferred and even accumulates in longer task sequences. Finally, we offer a standardized experimental setup and simple, yet strong plug-in baseline method, dubbed as group-class Balanced Greedy Sampling (BGS), which are utilized for the development of more advanced bias-aware CL. methods
INTRODUCTION
Continual learning (CL) is essential for a system that needs to learn (potentially increasing number of) tasks from sequentially arriving data. The main challenge of CL is to overcome the stability
dilemma (Mermillod et al.) 2013); when a CL model focuses too much on stability for remembering past tasks, would suffer from low plasticity for learning new task (and vice versa). Recent deep
neural networks (DNNs) based CL methods (Kirkpatrick et al. /017 et al. 2020 Li & Hoiem 2017) attempted address the dilemma by devising mechanisms to attain stability while improving plasticity thanks to the knowledge trunsferability (Tan et al. 2018), which is one of the standout properties of DNNs. Namely, while maintaining the learned megle the performance on new task (resp. past tasks) is improved by transferring knowledge of past tasks (resp. new task). Such phenomena are called forward and backward transfer, respectively.
However, most of such DNN-based CL approaches have not explicitly considered more realistic and challenging setting in which the dataset bias (Torralba & Eiros, 2011 exists; i.e., a training dataset contains unintended correlations between some spurious features and class labels. In such case, it is widely known that DNNs often dramatically fail to generalize to the test data without the correlation due to overly relying on the spurious features (Geirhos al., (020). Since the failure cases by such so-called shortcut learning are being commonly observed in various application areas (Geirhos et al. 2020), is also necessary thoroughly study the effect of spurious correlations the CL setting.
"Equal contribution
Corresponding author
Code is available at
e38

---

Published as a conference paper at ICLR 2024
To that end, claim that the effect of spurious relations becomes significant in CL due to the bias transfer issue. In recent study (Salman et al. 2022). is shown that during transfer learning. the bias learned by pre-trained model is (forward) trans- ferred to the downstream model even when it is fine- tuned with unbiased downstream task data. In CL, this issue can be potentially exacerbated since typical CL methods, which are oblivious of the bias transfer, would prevent the forgetting of learned tasks that had the dataset bias. Moreover, the bias transfer can oc- cur in both forward and backward directions as CL involves learning a sequence of tasks. For instance, consider example of (Domain-IL) shown in Figure in which the goal is to incrementally learn the classification model that predicts whether the face in the image is young or old. Now, assume that only the training dataset for T3 possesses the gender bias, that is, the spurious correlations exist between young and female as well as old and male. We then argue that when a naive CL method is used to update the model hy from Tr. the gender bias picked up by the model can encourage to make gender-dependant predictions in the previous or future tasks (e.g., Ti-1 or T++1) even if those tasks do not contain any dataset bias.
Unbiased
(Gender) Slosed task
Unbiased task
y: Young
Model Ag-1
Model hg
Model hep1
Forword
learning
yo Young Old
ya Young
ya Youngg Old
D
O:lld D
D
Figure I: Bias transfer in CL. Three tasks se- quentially arrive, in which each task has different distributions such face images having blond hair, black hair, and hat. The CL models, ht, and hu+1, may falsely classify young man images in T T--1 and Te+1 to "Old" due to the bias transfer from Tr.
One might reason that the risk of such bias transfer can be expected since spurious features may be transferred to other tasks in similar fashion as regular features. Nonetheless, to the best of our knowledge, an extensive investigation on this issue has not been carried out in the CL setting. Therefore, in this paper, conduct thorough and comprehensive experiments demonstrate the substantial quantitative evidence that the bias transfers in CL. both forward and backward, indeed exist and significantly affect model predictions. Specifically, we design three representative CL experiment settings with several benchmark datasets containing spurious correlations and employ well-defined metries for CL and bias, as well as various CL baselines (Section 33. We then carry out simple two-task CL experiments and identify that when a typical CL method focuses on the stability (resp. plasticity), the forward (resp. backward) transfer of bias occurs (Section 4 Furthermore, for longer sequence of tasks, we show that the bias is also transferred to much earlier or later tasks and even accumulates when naively applying CL. methods (Section 5). Finally, we provide a standard experimental setting including benchmark bias-aware CL scenarios and evaluation protocols, along with a simple yet strong plug-in baseline method, dubbed group-class Balanced Greedy Sampling (BGS), which can aid in developing a more advanced bias-aware CL method.
2 RELATED WORK
Continual learning (CL). Assuming that tasks are well-separated, CL scenarios are typically divided into three categories (Van de Ven & Tolias 2019): task-incremental learning (Task-IL), domain- incremental learning (Domain-IL), and classmnnremmentll learning (Class-IL). Both the Task-IL and Class-IL assume each task has disjoint set of labels, and the task identity is provided during training, but they differ in whether the identity is available (Task-IL) or not (Class-IL) at test time. In Domain-II, the class set of tasks always remains the same, but only input distributions vary as the number of tasks increases.
Recent CL methods can also be classified into three groups based on how they prevent forgetting of the previously learned tasks (De Lange et al. 2021]; regularization-base parameter isolation-based methods. Regularization-basee methods add regularization terms for penalizing deviation from past models (Kirkpatrick et al. 2017 [i & Hoiem 2017 Jung et al. 2020). Rehearsal-bes based methods store some data points from past tasks in small exemplar memory and replay them while learning the current task (Chaudhry et al. 2019 Lopez- Pazz & Ranzato 2017).
rehearsal-based and
Parameter isolation- based methods (Mallya & Lazebnik 2018 Ye & Bors 2021) allocate model
2

---

Published as a conference paper at ICLR 2024
parameters separately for each task by masking out previous task parameters and updating only the remaining parameters for learning a new task.
Spurious correlations and debiased learning. The problem of addressing spurious correlations in machine learning models has recently become a popular research topic (Geirhos et al., 2020). Enormous studies have identified the issue of spurious correlations in various domains (Geirhos et al. 2019 Garrido- et al., 2021) and strived to find out the source of the correlations Scimeca et al. (2021). In addition, numerous techniques to solve these issues have been developed by utilizing additional annotations for spurious correlation (Sagawa el al. 2020) or prior knowledge for the learning mechanism of spurious features (Liu et 2021 Kirichenko et al.) 2023). Nonetheless, all the works above have not considered the existence of spurious correlations in CL.
CL considering spurious correlations. While most state-of-the-ar CL methods have not considered the effect of spurious correlations, there are two notable exceptions (Lesort) 2022 Jeon et al., (203)) Lesort (2022) considers the issue of local spurious features (LSF) which correlates with class labels in particular task but does not correlate when data samples from all tasks are considered. [Jeon] jet al. (2023) find out that forgetting the biased knowledge obtained from previous tasks conducive to LSF problems, thereby devising method that encourages such forgetting while keeping other information maintained. However, both of the works only focus on Domain- IL and lack in- analyses for more general bias transfer problems in CL.
3 EXPERIMENTAL SETUP
3.1 PROBLEM SETTING AND DEFINITION
Notation. We consider the bias-aware CL setting, which is composed of sequence of classification tasks in which dataset biases may exist. Each data sample in the (-th task, Tú, consists of the tuple (x,, as y4) where a, E AA and ys E Vt are the group and class label of an input x,, respectively. The group label is defined by attribute of x; that may be spuriously correlated with the class label, Le., cause dataset bias. Unless otherwise noted, we consider a single type of dataset bias for the simplicity of analysis. In addition, the sets of class labels, {)i}, can be identical or not across t, depending on the CL scenarios.
Metries for model bias. We measure the extent of bias of a classification model with respec to group label by computing the number of prediction flips given counterfactual inputs with a flipped group label given correctly classified inputs. Formally, Bias-flipedd Mis- Mis-classificatton Rate (BMR) is defined as a metric for the bias of a model h:
BMR(h) (h) EAaEEDINa I(h(x;) # v.)
(1)
Mr, E DIM(2.) = MH1
in which 2 is the counterfactual input with a flipped group label. While BMR is an ideal bias metric, can be challenging generate counterfactual samples in some real-world datasets like facial images. In such case, we employ the Difference of Classwise Accuracy (DCA) (Berk et al.) 2021) as a surrogate metric for BMR. inddinneestes the gap in accuracy among different groups, which can be roughly considered an approximation of BMR at the group level. Note that high BMR and DCA indicate a high degree of existence of spurious correlations/bias a model.
Metric for the relative focus the trade-off, we also employ Normalized F F-I as a metric to gauge the relative emphasis on stability and plasticity of a CL method. For concrete definition, let h4 and h; be the models learned up to T, by a CL. baseline and the naive fine-tuning, respectively. Then, we introduce the forgetting and intransigence measures, F, and I,, (Cha al. (02T) which are commonly used for quantifying the stability and plasticity of he, respectively, as follows:
Since there exists the stability-
max A(h,, D2) A(he, D,)). I == 1 JEB.t-1]
F
-
- Dz)], (2)
in which A(h, Dr) denotes the accuracy of model h on the test set D; for the task T,. Then, with some pre-selected hyperparameters, we compute Fy Ir for each hyperparameter of each CL
Thh formal definition of DCA is provided in Appendix
3

---

Published as a conference paper at ICLR 2024
method after learning the final task Tr. Since CL methods have different degrees of Fr- Ir. re-scale these values with the min-max normalization for each CL method to obtain the "Normalized F-I" I" metric. From the definition, we can observe that the small (resp. large) Normalized F I value indicates more focus on stability (resp. plasticity).
How to investigate bias transfer. As illustrated in Figure when the dataset bias of certain task adversely affects the other tasks in CL scenarios, we refer to this phenomenon as bias transfer. We can systematically assess the presence and magnitude of bias transfer with the metrics defined above. Concretely, assume CL model h learns a sequence of tasks via a CL method. Next, we intentionally intensify the dataset bias of certain task, I₁ and learn the modified sequence of tasks to obtain another model h'. Then, the degree of forward and backward bias transfers can be measured by comparing the biases of h and ht' on the tasks other than T, We note that the changes in the bias of the model should be evaluated under fixed Normalized F - I metric of CL method since the knowledge transfer would vary depending on the relative focus on the stability and plasticity.
3.2 BENCHMARK DATASETS
We use three benchmark datasets, Split 100S, CelebA? (or CelebA and Split ImageNet- 100C, which are applied for Task-II., Domain-II, and Cl. respectively. The following is the description of our datasets (for more details, see Appendix B.4).
Split CIFAR-100S is a modification of Split CIFAR-100 (Chaudhry et al. 2019). which randomly divides CIFAR- 100 (Krizhevsky 2009) into 10 tasks with 10 distinct classes. As in Wang et al. (2020). we introduce the "color" bias into each task by disproportionately converting some color images to grayscale. Specifically, within each task, half of the classes are skewed towards the grayscale group
and the other half towards the color group, with given skew-ratio 2 0.5. We set seven levels of bias (0-6) by dividing the range of skew-ratio from 0.5 to 0.99 evenly on a log scale for systematic control of the degree of bias.
CelebA (Lu et al., 2015) contains more than 200K face images, each annotated with 40 binary attributes. It is notorious for containing representation biases towards specific attributes, such race, age, or gender (Torfason et al.) 2017: Fabbrizzi et al.) 2022). Unless otherwise specified, we use "male" and ""ounng" attributes as the group and class labels, respectively (please refer Table B.2)). We additionally select one or three other attributes to divide CelebA into two or eight tasks (which are used in Section 4 and Section 6 respectively). denoted as 2 and CelebA respectively. We again set seven levels of the dataset bias as in Split CIFAR-100S
Split ImageNNe-1100C is made from Split ImageNet- 100 that consists of 10 tasks with disjoint 10 classes, randomly sampled from ImageNet-1K (Russakovsky et 2015). We disproportionately injected Gaussian noise into (a 100)% of the samples for one randomly selected class and only 1 % of samples for other classes, given skew-ratio 0.01. We similarly set seven levels of bias depending on the skew-ratio.
3.3 CONTINUAL. LEARNING AND DEBIASING BASELINES
On the above datasets, we first evaluate two naive baselines, freezing that freezes the model learned from the previous task and fine-tuning that simply fine-tunes the model with new task data. Note they are the two extreme schemes that only focus on stability and plasticity, respectively. addition. we compare six representative CL methods: LWF (Li & Hoiem 2017) and EWC (Kirkpatrick et al.) 2017) for regularization based methods, ER (Chaudhry et al. 2019). iCaRL (Rebuffi et al., 2017). and EEIL (Castro al.] (018)) for rehearsal- baasedd methods, and PackNet (Mallya a & Lazebnik (0188) for a parameter isolation- based method. We note that each CL method can control the cuabillity-pasttion trade-off by adjusting its own hyperparameters such the regularization strength, the size of the exemplar memory, or the pruning ratio. We further note that PackNet can be applied only for Task-Il. and, iCaRL and EEIL for Class-IL. Additionally, we employ a widely used debiasing technique, Group DRO (Sagawa et al.) 2020). as a baseline to naively combine with CL techniques to reduce the dataset bias. More implementation details are provided in Appendix B
With this setting, we have not considered the issue of L.S in Domain- Lesont 2022 Le., for the case that the skew-ratio ranges from 0 to 0.5. For further discussion, please refer to Appendix D
Jeon et al. 2023
4

---

Published as a conference paper at ICLR 2024
Level O,:0
level 7y:
TE:
T;:
Fine-tuning Freezing
BWC
LWF
Rne-tuning Freezing
EWC
UNF
Fine-tniigg
iGaRL
LWFF
Parkkett
REIL
1.00
100
Pastchyy
Normalieed p- 1.550
80
80
(16
80
60
80
60
Norallzed
Normalized
40
40
40
20
20
20
Suatilty
Subeltyy
Slabely?
40
20
80
60
40 20
25
20
10
BMR Ty 1%6)
DCA of Tr (%)
BMR of (%)
(a) Split CIFAR-10OS Task-IL
(b) CelebA in Domain-IL
(c) Split ImageNet-100C Class-IL
Figure 2: Forward transfer of bias in two tasks-continual learning.
mmm Level
Level
Level
Bias of T
Fine-tuning freezing
EWC
line-tuning Freazing
EWC
UNF
Fine-tuning
ItaRL
UWF+ER
ER
ER
EEIL
ER
1.00
100
100
Hasschy
Plarrcyy
Normalized p- -11%)
80
80
lile
80
60
60
60
Normalized
Normalized
40
4:0
40
20
Suabilyy
Subllty
40
20
80
60
20
100
1o
20
BMR 71 196)
DCA of (%6
BMR 72 (%)
(a) Split 1O0S in Task-IL
(b) CelebA 2 Domain-IL
(c) Split ImageNet-100C in Class-IL
Figure 3: Backward transfer of bias in two tasks-continual learning.
4 CASE FOR CL WITH TWO TASKS
We begin our analysis by examining a simple setting: continual learning of two tasks. We aim to identify forward and backward transfers of bias for CL model through quantitative analyses using our metrics. We also aim to understand how the CL methods aggravate each of these transfers.
4.1 FORWARD TRANSFER OF THE BIAS
To investigate the forward transfer of bias, we evaluated CL methods for two different bias levels for Ti (i.e., level 0 and 6), while that of T₃ is fixed to level 09] i.e., be unbiased. 2 the bias metric values for T2 along with Normalized FF - on the three datasets, each with a different CL scenario, after learning T2. In the figure, we plot the results of each CL method by varying their trade-off. Namely, the upper points on
own hyperparameters for controlling the stability- each plot represent lower regularization strength, smaller memory size, or a lower pruning ratio for each method, hence putting more emphasis on plasticity. and vice versa.
From the figures, we make the following observations. First, from the BMR gap of the two blue triangles in Split CIFAR-100 results (1 11.5% vs 10.72%), we observe that even with simple fine- tuning, the bias of T, adversely affects that of T2, i.e., forward transfer of bias exists, which is consistent with Salman al.) (2022). Second, for all CL methods, the gaps between colored and uncolored points for similar Normalized F are larger than fine-tuning, and mostly increase as Normalized F I becomes lower. In other words, when Tj is severely biased, the CL methods transfer such bias more than fine-tuning since they are designed to remember past tasks, and the level of transfer becomes severer the CL methods more focus on stability. We additionally report the accuracy of 72 in Appendix E.2 and observe similar accuracies between colored and uncolored points, especially for Split CIFAR-TOOS and Split ImageNet- 100C, meaning that the bias gaps not caused by the accuracy gaps. Finally, we notice that the bias transfer occurs even when the dataset
WW additionally report the results on Split CIFAR- when the hias level of Ta is 2 or 4 in Appendix F.1
and observe similar trends from Figure 2 and
5

---

Published as a conference paper at ICLR 2024
Dios Lovel 0
Dlm Lend ofF
50
BMR of (5)
30
ONe of ('5)
30
20
10
Low Stability
Mid
liih Plasticity
Law Stability
Mid Normalized
High Plasticity
Normalized
F-I
P-1
(a) Forward transfer of bias
(b) Backward transfer of bias
Figure 5: Bias transfers sequence 10 tasks on OpERtCCIFARIOS.. The BMRs Tio and are shown after learning up to Tio by CL methods.
bias in Ti is relatively mild, demonstrating that the bias transfer is not confined to cases where T1 exhibits heavy bias. This finding is included in Appendix E.
4.2 BACKWARD TRANSFER OF THE BIAS
Now, we study the backward transfer of bias. Figure compares the bias of model for task Tı, of which the bias level fixed to 0, after learning T2. Similarly as Section 4.1 the two different bias levels for 72, i.e., level 0 and 6, are tested. We omit the results for PackNet since PackNet does not change the predictions for Ti after learning T2 and hence, there is no backward transfer.
We clearly observe from Figure 3 that the trend is now opposite to Figure 2 Firstly, we observe that the bias gap for Ti under similar Normalized - is maximized by fine-tuning" Also, for each CL method, the gap becomes severe the Normalized J - I increases. This means that the more CL methods prioritize plasticity over stability, the more bias obtained from T2 is transferred to T1, i.e., the backward transfer of bias occurs more. We report the accuracy of Tı, and results for the milder bias of Ti and T2, in Appendix E.2 and E.
4.3 FEATURE REPRESENTATION ANALYSIS
In our pursuit to provide concrete evidence of the bias transfer, we analyze feature representations from the penultimate layer of a DNN- based model using the centered kernel alignment (CKA) with the linear kernel (Kornblith al. 2019). CKA is isotropic scaling- invariant metric, making it suitable for quantifying the similarity between two representations derived from DNN model. Figure Acompares the CKA values on Split CIFAR- 100S for EWC under the two-tasks settings similar Figure 2a Namely, we evaluate models after learning T2 by varying the regularization strength and the bias level of Ti. We then compute CKA between color and grayscale images in the test dataset of Ty. Therefore, lower CKA value implies a greater dissimilarity in representations between two groups, signifying higher color bias in the model. Figure Angure/elear clearly shows that as the regularization strength increases and the bias of T; is more severe, CKA value decreases. Thus, the feature representation analysis again corroborates the forward transfer of the bias. For the case of backward transfer, please refer to Appendix E.3
Basieve
Baz Level TL
0.6
85
0.4
03
Ane tuniig
.nng
Regubnizstion Strength
Figure 4: CKA values 72 on Split CIFAR-IO01
5 CASE FOR CL WITH A LONGER SEQUENCE OF TASKS
We now further investigate whether the bias transfers shown for two-task CL scenarios also generalize and aggravated for longer sequence of tasks. In order to enhance the clarity of the comparison,
3 We observed that the models trained by fine-tuning in Class-IL classified the classes in 222 and hence, is infeasible to compute BMR since accuracy for Ti is O. Thus, we instead marked its BMR as 0% in the figure. In addition, we exclude the result of freezing in Class-IL IL since when learning the second task through freezing, only the linear classifier is learned for the second task while the classifier for the first task is fixed and hence, the leamned model produces unexpected low stability and high plasticity.
samples in Ti into one of

---

Published as a conference paper at ICLR 2024
BMi (F5 (9
30-
of biased:
blased:
60
70
Bis Level of,, &.7: Lovel
D02 1ood 15 for smill ng
25
Blos Lovdl of 12-6
Lovel
EWC
blased:
50
20
20
blased:
blased:
40
LAT
EWC
30
10
20
10
10
Low
Mid
High Plasticity
Low
Mid
High
Low
Mid
High
Stability f
Normalized P- Z
Stability
Normalized F- Plasticityy
Stability
Normalized F- I Plasticity
Figure 6: Accumulation of same
Figure Accumulation the different types bias on CelebA DCAs of T3 for "gender" and "smiling" attributes are reported.
type of bias on Split CIFAR-10OS.
we simplify the visual format of all figures in this section. We divide range of the Normalized F I into three equal intervals, and provide one result for each interval and CL method: the result for hyperparameter that leads to the lowest Normalized F I within an interval.
5.1 BIAS TRANSFER IN LONGER SEQUENCES
Firstly. we consider a sequence of 10 tasks with similar settings as in Figure 2 and on Split CIFAR- 100S. In Figure 5a and 5b] we vary the bias level of Tı and T1o to 0 or 6, respectively, while the levels of all other tasks are fixed to O. Correspondingly, the two plots show BMR of Tio (resp. Ti) according to the bias of Ti (resp. Tio) for each CL method, Thus, Figure 5a and 5b are for the forward and backward transfers, respectively.
The plots reveal an analogous trend with our two-task analyses. Namely, we observe that BMR of Tio (resp. Ti mostly higher when the bias level of Ti (resp. Tio) is 6, compared to when that of Ti (resp. Tio) is O. Furthermore, the gaps of BMR between colored and uncolored bars are at their widest when the Normalized obviously shown that both types of bias transfers also occur in longer sequences of tasks and can be aggravated by CL methods. We additionally display the accuracy Too and Ti for each plot in Appendix and notice that the accuracies are roughly the same, meaning that the gaps of BMR are due to the bias transfer, not due to accuracy gaps. Finally, we emphasize that the bias of a CL. model indeed can persist even if the tasks in the middle are not biased, depending the relative focus on stability and plasticity.
I low (for Figure 5a and high (for Figure [b)). Namely, is
5.2 ACCUMULATION OF THE SAME TYPE OF DATASET BIAS
We now examine whether the same type of dataset bias for each task could get accumulated by CL methods. To this end, we take sequence of 5 tasks with a bias level of 0 in Split CIFAR-I0OS. We then randomly select some of the four tasks except the last task and change the bias level of them to Loo to make biased tasks. Figure 6 presents the BMR values according to the varying number of biased tasks when LWF is applied. From the results, it is evident that BMR of T3 becomes higher as the number of biased tasks increases. This observation indicates that when tasks with the same type of dataset bias are continually added, their bias accumulates within the CL. model, resulting in a greater negative impact on other tasks. Also, note that the bias accumulates more when CL focuses more on stability. Accuracy of T3 and the results for backward transfer are provided in Appendix B.4 and E.5
5.3 ACCUMULATION OF THE DIFFERENT TYPES OF DATASET BIAS
In order investigate whether different types of dataset bias also accumulate, consider sequences of three tasks, each randomly selected from C Specifically, in the training datasets of Ti and T2, we use "gender" and "smiling" attributes as group labels, respectively. Then, while fixing the bias levels of T3 for both group labels to 0, we evaluate DCA of T3 in terms of both group labels by varying the bias level of Ti and Tz.
Figure 7 displays the degree of gender and smiling bias of models at T3 after CL of three tasks with LWF, EWC, and ER. We observe that when Ti and T2 are biased towards gender and smiling,
We used a lower level of hias (i.e., 4), instead of 6, in order to more clearly display how BMR varies with the number of biased tasks.
7

---

Published as a conference paper at ICLR 2024
respectively, DCA of T3 gets mostly higher in terms of both group labels. Furthermore, again notice from the left plot of Figure 7 that when CL methods more focus on stability, the forward transfer of bias becomes stronger. In contrast, in the right plot, the gap between colored and uncolored bars for LWF and EWC is larger when Normalized F 'I is in the middle regime, compared to the low regime. We infer that this might be because the dataset bias of T2 is less likely to be picked up under lower plasticity when learning T2, resulting in the bias being less transferred to T3.
AWARE CONTINUAL LEARNING
We demonstrated that the dataset bias of each task can be transferred by CL methods various scenarios, strongly highlighting the need for new CL approach that considers the dataset bias of cach task. To that end, we provide a standardized experimental setup for developing a novel awas- CL method, including evaluation metrics, benchmark bias-aware CL scenarios, and hyperparameter selection rule, together with a simple, yet effective plug-in baseline method, BGS. Based on these resources, we display the performance for typical CL methods and BGS in bias-aware CL scenarios, and pose discussions about future directions of bias-aware CL studies.
6.1 EXPERIMENTAL SETUP FOR DEVELOPING BIAS-AWARD CL METHOD
We construct three kinds of bias-aware CL scenarios for evaluation: 10-task sequences Split CIFAR-100S and Split ImageNet- and an 8-task sequence on CelebA The degree of bias for each task in Split CIFAR-100S and Split ImageNet-100c is controlled with a uniformly sampled skew-ratio by injecting the color or noise bias. On the other hand, we use CelebA as it is without artificially adjusting the number of samples, in order to reflect more realistic bias-aware CL scenaric For evaluation metrics, we employ average accuracy and BMR (or DCA) across all tasks in each scenario. Using the bias-aware CL scenarios and metries, we analyze the performance of naive CL methods and combinations between CL methods and the two debiasing techniques, Group DRO and BGS. We note that Group DRO combined with CL method employs an integrated loss function of both methods. To consider practicality, all hyperparameters for each CL method and Group DRO are tuned based on the average accuracy and bias metric up to T3, as presented in (Mai ct al. 2022). We choose the best hyperparameters with the average accuracy for naive CL methods. For Group DRO combined with a CL method, we fix the best hyperparameters for applying the CL method and search for the best ones of Group DRO with the best bias metric.
6.2 GROUP-CLASS BALANCED GREEDY SAMPLING
BGS is inspired by two recently developed methods for CL and debiasing, GDumb (Greedy Sampler and Dumb Learner) (Prabhu et al., (020) and DFR (Deep Feature Re-weighting) (Kirichenko et al.) 2023). The main idea of both methods is to simply train a model using a small amount of balanced data. Specifically, GDumb utilizes greedy sampler collect balanced data over all classes in the whole tasks during a CL scenario. It then trains a model from scratch with the data collected over all tasks at inference time. On the other hand, DFR trains a DNN model with an imbalanced training dataset and re-trains only the last layer with the validation data that is balanced over both class and group labels. Empirical results presented in Prabbu et al. (2020); Kirichenko et al. (002) show that using a small number of balanced data can be sufficient to achieve comparable performance in terms of CL and debiasing, respectively, compared to more complex methods currently use.
Following the same principle, we design the BGS algorithm to perform two key steps. First, BGS stores group-class balanced data across all classes and groups encountered in trained tasks in greedy manner, while concurrently learning sequence of tasks using any existing CL method. BGS then re-trains only the last layer of the DNN model which has been trained by the CL method, using the group-class balanced exemplar memory. By doing so, we expect that BGS can mitigate the bias of the model akin to DFR while also preventing the forgetting of previous tasks. Notably, BGS is simple and flexible in that BGS is compatible with any existing CL methods without any additional hyperparameters, which makes BGS broadly available as a baseline method. The sampling procedure of BGS is illustrated in Appendix A
In this section, we "blond_ hair" as the class label instead of "young" to create more biased tasks.
8

---

Published as a conference paper at ICLR 2024
Table 1: The comparison of methods on different CL scenarios. The average accuracy and BMR or DCA (%) over all tasks are shown. The numbers in parentheses represent the size of the exemplar memory. We bold the results improved by BGS.
(c) Split ImageNet-100C in Class-IL
(a) Split CIFAR-IOOS in Task-IL
(b) CelebA in Domain- IL
Method
Acc.
BMR
Method
Acc.
BMR
ER (1000)
23.49
36.53
Fine-tuning
20.47
52.86
Method
Acc.
DCA
BGG
26.33
26.18
BGS (1000)
30.98
46.47
Fine-tuming
94.56
27.00
ER (2000)
31.89
28.93
BGS (2000)
30.23
37.38
BGS (160)
94.30
26.32
BGS
33.43
21.83
LWF
67.81
28.25
BGS (320)
94.09
24.99
LWF + ER (1000)
28.03
33.60
BGS (1000)
70.46
20.60
BGS
31.57
26.10
BGS (2000)
70.80
17.97
LWF
95.54
29.41
LWF (2000)
34.79
29.85
BGS (160)
94.87
25.45
BGS
37.68
21.84
EWC
41.09
46.21
BGS (320)
94.21
23.12
BGS (1000)
47.53
37.17
iCaRL (1000)
30.15
38.30
BGS (2000)
49.79
31.89
EWC
94.99
28.41
BGS
34.79
14.84
BGS (160)
88.48
15.97
iCaRL (2000)
36.79
34.55
ER (1000)
61.27
29.19
BGS (320)
89.66
16.65
BGS
41.77
12.92
BGS
60.78
21.79
ER (2000)
67.90
25.62
ER (160)
94.82
27.11
EEIL (1000)
37.95
32.67
BGS
66.30
18.12
BBS
94.51
26.42
BGS
37,99
22.26
ER (320)
94.88
27.31
EEIL (2000)
46.45
26.49
PackNet
55.01
33.75
BGS
43.03
17.53
BGS
94.67
24.78
BGS (1000)
55,49
31.76
BGS (2000)
55.06
29.16
LWF+
94.26
23.67
EEIL (1000)
Group DRO
27.20
31.37
LWF
Group DRO
65.64
26.10
Group DRO
Group DRO
EEIL (2000)
33.94
28.78
6.3
PERFORMANCE COMPARISON
We apply our BGS to standard CL methods with two different sizes of exemplar memory on the three bias-aware CL scenarios. Table I presents the performance for naive CL methods, Group DRO combined with the best- pest-performing CL method, and CL methods with BGS plugged in g From the tables, we observe that the naive CL methods exhibit substantial bias in CL models, possibly due to the bias transfer. Naively combining Group DRO with CL methods mitigates the bias of CL models except for Split ImageNet- 1000,, but still sub-optimal terms of bias metrics when compared to results achieved by incorporating BGS with the same CL method. It could be because of the difficulty of tuning both hyperparameters for CL method and Group DRO. In contrast, integrating BGS into CL methods consistently yields notable improvements in terms of BMR or DCA across all scenarios, without requiring any additional hyperparameter tuning. Moreover, BGS also enhances average accuracy in most cases on Split CIFAR-100S and Split ImageNet-100C.
In spite of the promising performance of BGS, there remains room for devising more advanced bias- aware CL method. First, although BGS seeks to remove the influence of spurious features in model predictions by re-training only the last layer in a CL model, the underlying feature representation may still be biased. Namely, BGS may not be fundamental solution to the issue of bias transfer at its core. Furthermore, when a task contains multiple or continuous types of bias, obtaining the balanced exemplar memory over group labels remains nontrivial. Nevertheless, we emphasize that despite these limitations, our experimental setup and BGS can serve as a crucial stepping stone in developing more advanced bias-aware CL methods.
CONCLUDING REMARK
With systematic analyses for two-task and multiple-task CL scenarios, showed the bias can be transferred both forward and backward by typical CL methods that are oblivious to the dataset bias. We pointed out that the bias-aware CL problem is a general and important problem by exhibiting in-depth analyses and proposed simple baseline. For future work, we will develop a more principled approach that can accomplish continual learning and debiasing simultaneously.
We report the standard deviation and additional results including more recent CL. methods in Appendix F.7

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work was supported in part by the National Research Foundation of Korea (NRF) grant 200222/RRA222000885 and by Institute of Information & communications Technology Planning Evaluation (ITP) grants (No.2021-0-01 1343, 2022200-22008 No.2022-0-001 No. 2022002-0-00959 funded by the Korean government (MSIT). was also supported by SNU-LG AI Research Center.
REFERENCES
Hongjoon Ahn, Jihwan Kwak, Subin Lim, Hyeonsu Bang, Hyojun Kim, and Taesup Moon. Ss-il: Separated softmax for incremental learning. In IEEE/CVE International Conference Computer Vision (ICCV). pp. 844-853, 2021.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) forget. European Conference on Computer Vision (ECCV), pp. 139-154, 2018.
Richard Berk, Hoda Heidari, Shahin Jabbari, Michael Kearns, and Aaron Roth. Fairness in criminal justice risk assessments: The state of the art. Sociological Methods & Research, 50(1):3-44, 2021.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. In Advances in Neural Information Processing Systems (NeurIPS), pp. 15920-15930, 2020.
Francisco M Castro, Manuel Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In European Conference on Computer Vision (ECCV), pp. 233 248, 2018
Sungmin Cha, Hsiang Hsu, Taebaek Hwang, Flavio Calmon, and Taesup Moon. Cpr: Classifier- projection regularization for continual learning. In International Conference on Learning Repre- sentations (ICLR), 2021.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. Continual learning with tiny episodic memories. arXiv preprint 19X.v:1902. 2019.
Matthias De Lange, Rahaf Aljundi, Marc Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, (4(7):3366-3385 2021.
Simone Fabbrizzi, Symeon Papadopoulos, Eirini Ntoutsi, and Ioannis Kompatsiaris. A survey on bias in visual datasets. Computer Vision and Image Understanding, 223:103552, 2022.
Ismael Garrido-Muñoz, Arturo Montejo- Ráez, Fernando Martinez-Santiago, and I. Alfonso Ureña- López. A survey on bias in deep nlp. Applied Sciences, 11(7):3184, 2021.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-rrained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations (ICLR), 2019.
Robert Geirhos, Jörn- Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 11:665)666777 2020.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF IonfeeeeeCConfeeenneceee on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In International Conference on Learning Representations (ICLR), 2018.
10

---

Published as a conference paper at ICLR 2024
Myeongho Jeon, Hyoje Lee. Yedarm Seong, and Myungjoo Kang. Learning without prejudices: Continual unbiased learning via benign and malignant forgetting. In International Conference on Learning Representations (ICLR), 2023.
Sangwon Jung, Hongjoon Ahm, Sungmin Cha, and Taesup Moon. Continual learning with node- importance based adaptive group sparse regularization. In Advances in Neural Information Processing Systems (NeurIPS), pp- 3647-3658, 2020.
Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson.
Last layer re-training is sufficient for robustness to spurious correlations. In International Conference on Learning Representations (ICLR), 2023.
J Kirkpatrick, R Pascanu, N Rabinowitz, Veness, G Desjardins, AA Rusu, K Milan, J Quan, T Ramalho, A Grabska-Barwinske et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences of the United States of America, 114(13): 3521-3526, 2017.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Bal- subramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, al. Wilds: benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML), pp. 5637-5664. PMLR, 2021.
Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning (ICML), pp- 3519-3529. PMLR, 2019.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
Timothée Lesort. Spurious features continual learning. In Continual Causality Bridge Program at AAAI23, 2022.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40 40(12):2935-2947. 2011
Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. Wanli: Worker and ai collaboration for natural language inference dataset creation. arXiv preprint arXiv:2201.059%0 2022.
Evan Z Liu, Behzad Haghgoo, Annie S Chen, Aditi Raghunathan, Pang Wei Koh, Shiori Sagawa, Percy Liang, and Chelsea Finn. Just train twice: Improving group robustness without training group information. In International Conference on Machine Learning (ICML), pp. 6781-6792. PMLR, 2021.
2015. Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In IEEE/CVF International Conference Computer Vision (ICCV), pp.
David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Informaionn Processing Systems (NeurIPS), 2017.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- on Learning Representations (ICLR), 2019.
Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online continual learning in image classification: An empirical survey. Neurocomputingg 469:28-51 2022.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks to single network by iterative pruning. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7765-7773, 2018.
lacopo Masi, Yue Wu, Tal Hassner, and Prem Natarajan. Deep face recognition: A survey. In SIBGRAPI Conference Graphics, Patterns and Images, 2018.

---

Published as a conference paper at ICLR 2024
Martial Mermillod, Aurélia Bugaiska, and Patrick Bonin. The stability- plasstityy dilemma: Inves- tigating the continuum from catastrophic forgetting to age-limited learning effects. Frontiers in Psychology, 4:504, 2013.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performmnce deep learning library. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In European Conference on Computer Vision (ECCV), pp.
2020.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2001-2010, 2017.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision, 15(3):: 211-252, 2015.
Shiori Sagawa, Pang Wei Koh, Tatsunori B Hashimoto, and Percy Liang. Distributionally robust neural networks. In International Conference on Learning Representations (ICLR), 2020.
Hadi Salman, Saachi Jain, Andrew Ilyas, Logan Engstrom, Eric Wong, and Aleksander Madry. When does bias transfer in transfer learning? arXiv preprint arXiv:2207.02842 2022.
Jonathan Schwarz, Wojeiech Czarnecki, Jelena Luketina, Agnieszka Grabska- Barwinkkk, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell, Progress & compress: A scalable framework for continual learning. In International Conference on Machine Learning (ICML), pp. 4528 5528-4537. PMLR, 2018.
Luca Scimeca, Scong Joon Oh, Sanghyuk Chun, Michael Poli, and Sangdoo Yun. Which shorteut cues will dnns choose? study from the parameter-space perspective. arXiv preprint arXiv:21 00.03 2021.
Chuanqi Tan, Fuchun Sun, Tao Kong, Wenchang Zhang, Chao Yang. and Chunfang Liu. A survey on deep transfer learning. In International Conference on Artificial Neural Networks, 2018
Robert Torfason,
Eirikur Agustsson, Rasmus Rothe, and Radu Timofte. From face images and attributes to attributes. In Proceedings of the Asian Conference on Computer Vision, 2016, pp. 313-329, 2017.
Antonio Torralba and Alexei A. Efros. Unbiased look at dataset bias. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1521-1528, 2011.
Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint 19XXvv/904.07734 2019.
Jeffrey S Vitter. Random sampling with reservoir. ACM Transactions on Mathemutical Software, 11(1):37-57, 1985.
Zeyu Wang, Klint Qinami, Ioannis Christos Karakozis, Kyle Genova, Prem Nair, Kenji Hata, and Olga Russakovsky. Towards fairness in visual recognition: Effective strategies for bias mitigation. In IEFF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8916 8916-8925, 2020.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprini arXiv: 1704.05466. 2017.
Fei Ye and Adrian G Bors. Lifelong infinite mixture model based on knowledge-driven dirichlet process. In IEEE/CVF International Conference on Computer Vision (ICCV), pp. 10795-10704, 2021.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
We offer supplementary materials in this document. Specifically, we provide the detailed algorithm of BGS in Section A In Section B present implementation details including model architecture, optimization, implementation of baseline methods, the range of hyperparameters used, and description for datasets. In Section C we give the formal definition of DCA. In Section [ we discuss possible issue with LSF in Domain-IL with further experiments using CelebA². In Section we display additional results for accuracy, other various settings, and more baselines. Finally, add case studies of the bias transfer in more realistic scenarios using satellite images and natural language processing (NLP) datasets.
A BGS ALGORITHM
Algorithm 1: Group-class Balanced Greedy Sampling
Init :Memory M = -()....;) with capacity K, labels L. = 0. count C' = ..,...)
Input :A data sample (z4, a, yi). task id
Output:M
K
Ca Vi, t)] then
if E C Ca Vi.tt) t)] == 0 then
L L (W; end
E M V1. = Ma v.. (2 ae V6,
AEEAAANNECCC (.,y, K then
else
(ag,Wi,tt) (a..) vl(a,,,t)]
M((aj; N5, pop(()
10
M((a; ys. t) = Mi(ai. yi, t)] (Ii, az Us. t);
11
12
Cl(a,, Uy, Cl(ag, t,)l -
13
end
(a,, yo. 0] = C((a ur. 0) + 1;
14
1s
end
B MORE IMPLEMENTATION DETAILS
B. MODEL ARCHITECTURES AND OPTIMIZATION
For all datasets, we used the AdamW optimizer (Loshchilov & Hutter 2019) with the following hyperparameters: learning rate of 0.001, weight decay of 0.01, 31 of 0.9, 32 of 0.999, and of 10 For Split CIFAR-10OS, we trained ResNet-56 (He et al. 2016) from scratch for 70 epochs using batch size of 256. For 2. and Split 1 we trained ResNet- 18 from scratch for 50 and 70 epochs, respectively, using a batch size of 128. We incorporated the cosine annealing learning rate scheduler, with the maximum number of iterations set to the same the number of training epochs. We use PyTorch (Paszke et al.) 2019) and conduct experiments with servers equipped with AMD Ryzen Threadripper PRO 3975WX CPUs and NVIDIA RTX A5000 GPUs. All experimental results are obtained by averaging the results from different runs.
B.2 IMPLEMENTATIONS OF CONTINUAL LEARNING METHODS AND BGS
For ER (Chaudhry 2019). iCaRL (Rebuffi 2017). EEIL CCsstro al. 2018 DER (uzzegaa et al.) 2020). SSIL (Ahn et al. 2021). GDumb (Prabhu et al. 2020), MAS (Aljundi et al. 2018). and DFR (Kirichenko et al. 2023 we implemented the same their original versions.
Results of DER, SSIL,, GDumb, MAS, and DFR reported in Section F.7
13

---

Published as a conference paper at ICLR 2024
Table B.1: Hyperparameter search ranges.
Method
Hyperparameter
Scarch range
EWC
Kirkpatrick et al
2017
Regularization strength À
[10", 10º]
LWF
Li & Hoiem
2017)
Regularization strength À
[10 x 1021
ER
Chaudhry et
2019
Memory-to-dattsses size ratio r
[10 _10
iCakL.
Rebuffi et al.
2017)
Memory-to-datasets size ratio
n0 , 10
EEIL
Castro et al.
2018
Memory Mlemory-todalle to-dstsssss size ratio r
[10 [100 1
PackNet Mallya & Lazebnik
2018)
Pruning ratio r
[10 1,8 x 10-"1
Group DRO (Sagawa et al.
2020
Learning rate of q
[10 [10 1,0
MAS Masi et al.
2018
Regularization strength À
[10-1, 10'1
DER BBuzzeaa et al.
2020
Regularization strength À
[10- 10")
The following describes the CL methods that have been implemented differently from their original versions, in order to improve the practicality, performance, and consistency of our analysis.
In the original EWC algorithm (Kirkpatrick et al., 2017), the snapshot of CL model should be stored whenever the model is updated from new task. The snapshots are then used to calculate the importance scores of model parameters in the new task. Namely, the algorithm requires a linearly growing amount of memory to store sequence of models, which is space-inefficient To address this issue, we implemented online FWC, proposed in (Schwarz et al. 2018). which averages the importance scores in an online manner without storing a set of models.
For LWF (Li & Hoiem 2017), we used average of the distillation losses for each head, not sum of them, to balance between the cross entropy loss and the distillation losses.
For PackNet (Mallya & Lazebnik 2018), we pruned and re-trained models for 10 epochs with tenth of the initial learning rate. While the original algorithm of PackNet uses the fixed pruning ratio during learning all tasks in CL, use different pruning ratios for each task regarding experiments in Sections 4 and 5. Namely, we fix the pruning ratio at the first task to 0.5 and then vary the pruning ratio for the other tasks in order to control the stability and plasticity.
For all rehearsal based methods, we employed the Reservoir sampling (Vitter 1985) as a strategy for updating the exemplar memory.
For DFR in CL scnearios, we retrained the last layer with a small subset (20%) of training data of a task after every task.
When BGS is combined with a rehearsal-based method, we do not use an exemplar memory of the rehearsal-based method and adopt only the exemplar memory of BGS. Furthermore, when BGS is applied to iCaRL, re-training with the collected balanced exemplar memory proceeds for the whole layers rather than only for the last layer since iCaRL utilizes the pearesttmeennoo---eeempaaa classification based on the features of exemplars (Rebuffi et al. (017).
B.3 HYPERPARAMETER SELECTION FOR EACH RESULT
For each experiment, we evaluated CL methods several times by varying their hyperparameters based on the sets of pre-selected candidates. The candidates uniformly distributed on logarithmic scale within a given range, which is reported in Table B.]] For Figure 2 and 3, we discard overlapping results and report the remaining ones among all results for pre-selected hyperparameter candidates.
In Section 4 and 5, we consider the proportion r of the number of training samples of all past tasks as the hyperparameter of rehearsal-basee methods with respect the size of their exemplar memory. To be precise, in Section 4, we store r% of T, training samples. For the proportion in Section 5.1 and 5.3, we consider the total number of training samples from Ti to Ty and 7, to T2, respectively.
14

---

Published as a conference paper at ICLR 2024
(a) Training samples
(b) Test samples
Figure B.1: Samples in certain task with bias level of in Split CIFAR-I0OS. Each represents specific class within task. top three rows represent classes biased toward the grayscale samples, while the bottom three rows contain classes biased toward the color samples. The test dataset includes pairs of images, where cach pair contains one grayscale and one color version of the same image.
(a) Training samples
(b) Test samples
Figure B.2: Samples in certain task with bias level of 0 in Split ImageNet-100C. Fach row represents
specific class within the task. The last is for the class which is biased towards the Gaussian noise. Red boxes indicate the images with the Gaussian noise. The test dataset includes pairs of images, where each pair contains one original and one noisy version of the same image.
B.4 DATASETS
Split CIFAR-100S. In the training dataset of each task, the first five classes skewed toward the color group and the latter skewed toward the grayscale group, given the skew ratio. For the test dataset of each task, we have pairs of the same images; ones in color, and ones in grayscale. Figure B.Iiullusrate some training and test samples in Split CIFAR-100S. For the experiment in Section 6, the skew-ratio of each task was uniformly randomly sampled from [0.5, 0.99].
CelebA. Domain-Il IL typically assumes the different input distributions with the same label space for each task. To construct such Domain-ll. scenario using CelebA, we first select two attributes for group
15

---

Published as a conference paper at ICLR 2024
and class labels, and divide the dataset into several tasks based on other attributes except the two attributes such that each task has different facial attributes, L.e, distinct distributions of non-spurious features. In detail, the two tasks in CelebA² are divided depending attribute. For CelebA used in Section 5, we adopt "mouth Srruuuttssgghl____t slightly_open" "black_hair" and loval_facer attributes to split CelebA into disjoint tasks. In Section we the "blond hair" attribute for the class label and "young", "smiling" and "straight hair" attributes for dividing the CelebA datasets. We note that we use 5,000 samples per each class and group for CelebA and subsample the samples with a given skew-ratio a, as shown in B.2
the "mouth slightly open"
Table B.2: Dataset construction of CelebA²
Y = 0
Y = 1 ("young")
G
5000 (1 a)
5000xa
G 1 ("male")
5000
5000x -1
Split AmageNet-100C. For this dataset, we assume a specific class at each task is heavily biased toward Gaussian noise. Thus, for each task, we randomly select one class and inject Gaussian noise into 1% of the training images of the selected class. For the concrete method to corrupt images, we follow the protocol of [enddrycs & Dietterich (2018) with noise severity level of 1. The proportions of the noisy images for the other classes set 1% (5% in Section 6). Similar Split CIFAR-1005 we doubled the original test dataset and added Gaussian noise to half of it for the test set. Examples of training and test samples of Split ImageNet- 100C are illustrated in Figure B.2a and B.2b For the experiments in Section 6, was uniformly randomly sampled from [0.5, 0.95].
C FORMAL DEFINITION OF THE DIFFERENCE OF CLASSWISE ACCURACY (DCA)
DCA of a model at task T₃ is defined as follows:
DCA(h)=
1
max AAh,, A(h, D)' Σ yEJ.
(C.1)
D
a.a'EA
in which D V is the subset of Dr that are confined to the samples with group label a and class label y. DCA indicates the average (across the class) of maximum accuracy gaps between different groups
D ON THE ISSUE OF LOCAL SPURIOUS FEATURES (LSF) IN DOMAIN-IL
In our experiments for Domain-IL. in our manuscript, we did not consider the issue of LSF (Lesort 2022). LSF is spurious feature in certain task that can be mitigated when datasets from the whole tasks are collected; for instance, a two-task Domain-ll. scenario on CelebA 2 in which the skew-ratios of the first and second tasks are 1 and a, respectively. In such setting, Lesort (2022) Jeon et al. (2023) demonstrate that the rehearsal-basee data in their exemplar memory and current data alleviates the dataset biases, especially when the size of the memory is sufficiently large. Consequently, our claim that naive CL method promotes bias transfers depending on the relative focus on stability and plasticity may not hold in that case.
methods can solve this LSF issue because mixing past
However, we claim that rehearsal- methods could not be solution for Domain- IL scenarios assuming different distributions of non-spurious features for each task, which /eecot/ p: JJon al. (2023) have not addressed. Namely, even if mixing the past and current samples properly to eliminate the spurious correlation between group and class labels, the correlation is still valid when conditioned on the features of which the distribution is shifted across tasks.
To empirically verify this, we additionally consider the six levels bias from -6 - by dividing the range of skew-ratio a between the class and group label from 0.01 to 0.5 (refer to Table B.2 evenly on a log scale. Then, we carry out a further experiment using two-task task scenarios on CelebA where the bias levels of Tj and Tr 6 and -6, respectively. We use two CL scenarios depending whether there exists a distribution shift of non-spurious features (beyond the bias level change) or not. Specifically, the first scenario assumes that the attribute values for "mouth "mmothhslighhil"" slightlyyopen" in training
16

---

Published as a conference paper at ICLR 2024
Table D.I: Performance and confusion matrix for each task. Each value computed after learning T22 by ER with the maximum size of memory.
(a) Accuracy and DCA for each task
(b) Confusion matrix for each group and task in the presence of shift of input distribution
Ti (bias level: 6)
Ta (bias level: -6)
Task I
Task 2
Dist.
shift O
Dist, shift X
Group
y 0
y
0
Y
Ti
T2
Ti
T2
G=D
y = 0
87
163
226
24
Y 1
10
240
158
92
Accuracy
95.52
88.96
79.28
75.87
DCA
63.15
44.45
3,45
3.20
G=1
Y=0
241
9
133
117
Y 1
172
78
29
221
samples of the first and second tasks are different, i.e., 0 and 1, respectively. For the second scenario, there is no specific shift of the attribute, namely, the two tasks composed of the training samples which are divided uniformly randomly over all attributes (except for the gender bias). Note that both scenarios correspond the LSF setting because there does not exist correlation between the class and group labels when the entire training data is collected from the two tasks.
Two tables in D.I display results for each task after learning T2 using ER with the maximum size of memory. Table D. compares accuracy and DCA for each task of the two different CL scenarios. From the results in Table D.1a we observe that ER does not always mitigate the bias in LSF settings, contrary to what would be expected in (Lesort 2022). We clearly observe that the bias cancellation only occurs when the input distributions regarding non-spurious features do not shift. We infer that this would be because CL model can utilize both spurious correlations for each task based on the estimation of different input distributions. The confusion matrices for each group and task in Table D.Ib show more clearly that the model relies on the different correlations for prediction in T'i and T2. Namely, for task Ti, samples with G c= (resp. = 1) are more likely to be predicted to be (resp. 0) and vice versa for task T2. In conclusion, the issue of the bias in the LSF settings could not be solved by the typical rehearsal- based CL methods when input distributions regarding non-spurious features are different across tasks, which could also lead to bias transfer to subsequent tasks. In short, although our findings of bias transfer may not hold for settings with LSF based on the same input distributions, we claim that the settings are somewhat non-realistic and our finding could be still valid for the case of LSF with different distributions regarding non-spurious features.
E ADDITIONAL EXPERIMENTAL RESULTS
E.1
TWO TASK SCENARIOS WITH DIFFERENT BIAS LEVELS AND NOISE TYPE
Figure E.
and F.2 display the forward and backward transfer of the color bias Split CIFAR-
100S with two task- CLs. For each figure, we vary the bias levels of T2 (resp. Ti) with 2 or 4 and report BMR of Ti (resp. T2) for each CL method and hyperparameter. The results mostly show the analogous trends with the results in Figure 2 and 3. Moreover, it is apparent from the figure that compared to the results in Figure and 3, which are for the bias level of 0 in T2, BMR of Tr is much larger and even the difference in BMR between colored and uncolored points becomes more pronounced. This would be because previously learned biases CL model tend to facilitate learning of the dataset bias of the current task more.
In addition, we investigate whether the bias transfer occurs even the degree of bias of certain task much smaller. To that end, we explore the forward (resp. backward transfers) using other two-task CLs with Splüt CIFAR- 100S where the bias level of Ti (resp. T2) is 2, which is much milder than the one used in Figure 2(a) (resp. Figure 3(a).. Again, we observe the same trend from E.3 in Figure 2(a) and 3(a), demonstrating a small amount of bias can be transferred.
Furthermore, in Figure E.4 we report the results for frost noise instead of Gaussian noise in the same setting of Split Image.et- 100C. From the Figure, we demonstrate that the bias associated with another type of noise can also be transferred.
17

---

Published as a conference paper at ICLR 2024
Blas Level of Tx:
Iiias Level of Ta:
Fine tine-tuning
EWC
LWF
Fine-tuning
LIVF PackNet
Freezing
ER
PackNet
Freezing
ER
100
Pactrity
Plasticity
Normalized P (%6
80
Normalized P [(6
80
60
6
40
40
20
20
*
Stality
Suabiity
70
60
50
40
30
20
10
80
70
60
50
40
30
BMR of T2 (%6
BMR of T2 (%)
(a) Bias level of T2: 2
(b) Bias level of T2: 4
Figure E.I: Forward transfer of bias two tasks-continual learning on Split CIFAR-I00S with different bias levels of T2.
Blas Level of Tg:
JBias Level of
I Bias Level of T2: Rine-tuning Froszing
18las Level of Ty:
Fine-tuning
EWC * LWF
EWC
LWF
Freezing
ER
100
Pastioty
Masticty1
(96)
80
Normalized P (%k)
60
60
40
20
Stallity
Subilly
0
40 BMR Ty (%)
BMR of (%)
(a) Bias level of
(b) Bias level of Ti:
Figure E.2: Backward transfer of bias in two tasks-continual learning Split CIFAR- 100S with different bias levels of Ti.
TB Blas Level of Ty: 2
Blas Level of Ts:
IB: as Level of Ts:
BBlas Level of T2:
Fine-tuning Froezinng
EWC
LWV
Fine-tuning Frozzing
EWC
LWF
ER
Packiint
100
100 Plustcity
Pastidty
Normalized r- (%6
80
60
Normalized F
60
40
40
20
20
Stabilty?
50
40
30
20
10
50
40
30
20
10
BMR of T2 (%)
BMR of T: (%)
(a) Forward transfer. Bias level of T2: 0
(b) Backward transfer. Bias level of Ti: 0
Figure E.3: Effects of bias transfer with mild bias in two tasks-CI. on Split CIFAR-100S.
E.2 ACCURACY FOR CL WITH TWO TASKS
Figure E.5 and E.6 show the accuracy for each point plotted in Figures and 3. Similar accuracy between uncolored and colored points Split CIFAR-100S and Split ImageNet- 100C corroborates
18

---

Published as a conference paper at ICLR 2024
Blas Level of Ty: BBias Level of Ta:
Bias Level of Ty. 6 1iias Level of 72:
Fine-tuning
iCaRL
LWF++ER
Fine-tuning
ICaRL
LWFFER
EEIL
EEIL
100
100
Pasticity
Passhiity
80
Cormalized r. (1%)
*
Normalized p
60
60
40
40
20
20
Siassity?
Suabllyy
25
20
15
10
100
80
60
40
20
BMR of T2 (%)
BMR of T1 (%)
(a) Forward transfer. Bias level of T2: 0
(b) Backward transfer. Bias level of Ti: 0
Figure E.4: Transfer bias two tasks-continual learning on Split ImageNet-100C with frost noise.
6
Level 7::
Blas level of Tu: 6
Blas level of
Blas Level of 6 =
Blas Level of T1: UVF+ER
Fine tuning
EWC
UNF
tuning
tuning
Freezing
ER
EEIL
ER
100
100
Pusiulyy
Paackly
Normalized (80
80
80
Normalized F-1 (99)
80
W
60
60
Normalteed
40
40
Suaiily
*
*
20
40
60
80
60
70
90
70
85
90
Accuracy of Ta (%)
Accuracy Ta (9)
Accuracy of Ta (%)
(a) Split CIFAR-100S in Task-II.
(b) CelehA? in Domain-II.
(c) Split ImageNet-1008 in Class-II.
Figure E.5: Accuracy of 22 for the forward transfer of bias in two tasks-continual learning.
Level
Biaas Level
of Finestuiing Freezing
Level 1Bias
Fine-tuning
EWC * LWF
EWC
LWF
Rina-tuning
ICoRI.
LWF
Freezing
EEIL
100
Panticly
Mosiccyy
Panticty
2,033
Normal lood 1.00
8:0
Normalized lized p- (.0
0/x
60
60
III
60
Norma
40
40
40
20
20
20
Stabily
Stabilty
Accuracy Tı (%6
Accuracy of (%)
Accuracy Tı (%)
Split CIFAR-100S in Task-IL.
(b) CelebA? in Domain-II.
Split ImageNet- in Class-IL.
Figure E.6: Accuracy of Ti for the backward transfer of bias in two tasks-continual learning.
that the gaps in BMR under similar Normalized JF - in Figures 2 and 3 not due differences in accuracy. Furthermore, although the accuracy gaps in CelebA have similar pattern with the DCA gaps in Figures 2b and 3b, the accuracy gaps are smaller in magnitude compared the DCA gaps.
E.3
FEATURE REPRESENTATION ANALYSIS FOR BACKWARD TRANSFER OF BIAS
We exhibit the results of CKA analysis for the backward transfer on Split CIFAR- CIFAR-1OOS. From Figure is obviously shown that the CKA value decreases the regularization strength decreases and
E.7
19

---

Published as a conference paper at ICLR 2024
Bias level of Ty:
Blas Level of T>
Fine- tuning
2
Free-
zing
Regularization Strength
Figure E.7: CKA Split CIFAR-IOUS. The CKA values between color images and grayscale images in 'i are shown. Each value calculated after learning 722 by EWC.
the bias level of T2 increases. This again demonstrates that when a CL method focuses on learning biased current task, i.e,, plasticity, the backward transfer of bias by a CL. method becomes more obvious.
E.4 RESULTS FOR ACCURACY WITH LONGER SEQUENCE TASKS
Figures E.8 and E.9a display accuracy corresponding to each point plotted in Figures 5 and 6. It is observed from the figures that the accuracy in the same intervals roughly the same, so we infer that the gaps of BMR shown in Figures 5 and are due the bias transfers, not due the accuracy gaps.
100
100
Londl Tu
Level Tax
90
Load
90
Accuracy of Tis (8)
80
LIN
of
70-
60
50
40
50
30
40
20
30
10
LOW Stability
Mid Normalized 7-
High Plaiticcity
Low
High Plasticity
Stability
Normalized F-
Figure E.8: Accuracy longer sequences of Split CIEAR-10OS. The experimental settings the two plots are the same as in Figure 5a and 5b, respectively.
of
biased:
of biosed:
of blased:
Accuracy of (9))
of blased:
80
biased:
of biosed:
of blased:
blasse:
Accuracy of T
70
blased:
60
70
50
40
60
20
Low Stability
Mid Normalized y-1
High Plasticity
Low
Mid Normalized y-z
High Plasticity
Stability
(a) Accuracy Tf
(b) Accuracy of Ti
Figure E,9; Accuracy depending on the number of biased tasks, The experimental settings are the same as
in Figure 6.
20

---

Published as a conference paper at ICLR 2024
E.5 ACCUMULATION OF THE SAME TYPE OF DATASET BIAS BY BACKWARD TRANSFER
We investigate whether the same type of dataset bias for each task can be accumulated by CL method by backward transfer. For this experiment, we randomly selected some of the four tasks from T2 to 75 and changed the bias levels of them to 4, Le., make them more biased. The bias levels of the remained tasks are set to O. Figures E. and E.9b show the BMR and aceuracy T2 depending on the varying number of biased tasks when LWF is applied. Again, we observed the BMR of I becomes higher as the number of biased tasks increases especially when CL focuses on high plasticity. This demonstrates that accumulation can happen also backward.
* of biased: D
35
of biased: 2
of blased: 3
25
biased: 4
10
Low Stabillit
Mid Normalized
High Plasticity
Figure E. 10: Accumulation of the same type of dataset bias on Split CIFAR-100S by backward transfer.
E.6
ACCUMULATION OF THE DIFFERENT TYPES OF DATASET BIAS BY BACKWARD TRANSFER
50
22.5
Blas Level T2 & Ts:
LWF
45
Bias Level Tg
LWF
DCA of T tot gender
20.0
Blas Level T2 Ty:
40
Bias Level
EWC
17.5
35
ER
15.0
30
12.5
- 25
10.0
n
20
7.5
σ 15
5.0
3
2.5
Low
Mid
High
5
Low
High Normalized F- Plasticity
Mid
Stability
Normalized P- I Plasticity
Stability
(a) DCA of for gender bias
(b) DCA of Ti for smiling bias
Figure E. Accumulation of the different types of bias on CelebA by backward transfer.
We consider sequences of three tasks, similar Figure 7, to show the different types of dataset bias that can also be accumulated by wark transfer. We assume spurious correlations regarding "gender" and "smiiling" attributes for Tr and Ta as "gender" and "smiling" attributes, and vary the bias level of the two tasks while fixing both types of the bias levels T' to O. We again confirm from Figure E..T that the accumulation of the two types of bias similarly occurs by backward transfer. Note that backward transfer for the bias of the "gender" attribute can happen more in the low or mid regime than the high regime because the knowledge for the bias picked up obtained from the second task can be forgotten after learning T3 in the high regime.
E.7 PERFORMANCE COMPARISON WITH MORE BASELINES AND ADDITIONAL SETTING
In Table E.1 report the standard deviation for the results in Table 1. We also added results for GDumb (Prabhu et al. 2020), DFR (Kinchenko et al. 2023). and more recent CL methods: MAS (Aljundi al. 2018). DER (Buzzega et al. 2020) and SSIL (Ahn al. 202T). Additionally,
21

---

Published as a conference paper at ICLR 2024
Table E.1: The comparison of methods on different CL scenarios. The average accuracy BMR DCA (%) over all tasks are shown. The numbers in parentheses represent the size of the exemplar memory. We bold the results improved by BGS.
Split CIFAR-100 in Task-II.
(b) CelebA in Domain- IL
Method
Acc.
BMR
Fine-tuning
20.47=1 29
52.80 12.86111.15
Method
Acc.
DCA
BGS (1000)
30.98=1991
46.47:1.335
Fine-tuning
94.56++0.47
27.00++22.2
+ BGS (2000)
40.23 0.96
37.38.16.39
BGS (160)
94.30110.26
26.32:11.53
LWF
67.81:11.00
28.25::2.35
+ BGS (320)
94.09+00.00
24.99++1.99
BGS (100D)
70.46
20.60.1..04
BGS (2000)
70.80
17.97-40.44
LWF
95,541:0.0
19.41=1.60
BGS (160)
94.87:+0.48
25.45-+3.12
FWC
41.09-2.554
46.21 2.000
+ BGS (320)
94.21::0.73
23.12:13.16
BGS (1000)
47.53=11003
37.17:42.24
+ BGS (2000)
49.79=0.79
31.89:12.28
EWC
94.99:10.18
28.41:1:.44
BGS (160)
88,48-+3.35
15.97 3.83
ER (1000)
61.27=0.63
39:3
+ BGS (320)
89,6611.55
16.65
+ BGS
60.78:10.63
21.79:12.22
ER (2000)
67.90=0.96
25 15.622:1.440
ER (160)
94.824-0.00
+ BGS
66.30:20.46
18,12:11.32
BGS
94,51+00.11
1.542++1.55
ER (320)
94.88:1.113
27.31
PackNet
55.01 -.556
33.75 5.886
+
94.67::0.06
24.78++1.39 1,39
BGS (1000)
55,49
31.76.13.01
+ BGS (200D)
55.06- 1,25
29.16 2.333
MAS
95.07:10.24
08.83111.0 68
+ BGS (160)
91.16:12.05
18.62:12.67
MAS
50.61=0.72
41,06-+-669
+ BGS (320)
90.92:1:.95
17.43::1.22
BGS (1000)
49.97-00511
14..66--4.59
+ (2000)
49.23==0.11
31.72++3.15
DER (160)
94.54110.15
27.19:12.06
+ BGS
94.31++0.38
25.99- 2,05
DER (1000)
50.6420.177
39.16116277
DER (320)
94.51#:0.42
26. 26.28-1.64
+ BGS
48. 18.15=1.03
32.50:13.64
DER (2000)
$1.6220.57
37.78:16.44
BGS
94.76+H0.15
28. 2..166+22.2
+ BGS
48.65:10.72
12.90
GDumb (160)
84,75:t1.68
17.47=22.04
GDumb (1000)
36.11.11.70
35.36.19.83
GDumb (320)
85.71:10.70
18.31
GDumh (2000)
43.70-00.57
10.40
93.63-+0.94
23.3442.11
DFR
20.38=1144
15.01=11.00
LWF Group DRO
94.26110.35
23.67:12.23
LWF Group
65.6420.67
26. 26.10:2
(c) Split ImageNet-100C Class-IL
(d) Split CIFAR-100S in Class-IL
Method
Acc.
DMR
(1000)
23.49::0.67
36 16+1171777
BGS
26.33:0051
26.18:11.48
ER (1000)
1899#1.11 17
50.88:3115
ER (2000)
31.89:11.32
28.93:00722
RGS
28.67-00.6
4087-133
BGS
1.14
21.831..222
(2000)
25.74+0.99
47:24:041
BOS
26.85:0.73
2.785:0.73
FR. (1000)
28.03-11.28
33.60+-2.41
BGS
31.57=1.599
26.10+-2.21
LWF + ER (1000)
21.52+1.30 30
51.63:1144 48
LWF ER (2000)
14,79:11.12
29 85::1.55
LWF
BGS
27.29:1:133
24.82.1.233
41.26.11133
22:12221
DGS
30.28:10.73
30.28:10.73
SCaRL (1000 (1000)
30.15:10.55
38.90.12.96
BGB
34.79:4033
14.84=1443
ICaRL (1000)
21.29:10.52
49.72:14:53
TERRLCZ000 RGS
41.77-0.75
RGS (2000)
25.36+1.34
28.26::-0844
31.65- 2.78
ROS
32.42+1.18
25.40 2.56
EEIL (1000)
BGS
BGS
(2000)
16.45.11.30
43.03-#1.00 1.02
37.95.0.117
1.,99:11.20
22.26::1.21
32.67.22.70
07.53+-0.37
BEIL (1008)
EEIL (2000)
BGS
BOS
28.04+0.55 59
35.1910.53
27.41::0499
33.03+00.90
43 43.58+11 56
57
6LX3--0.30
SSIL (1000)
BGS
43.76200.31
21.05 1.24
32.25:1.33
SSIL (1000)
BOS
31.37110.46
27.9512.03
41.49125
49. 3.777
SSIL (2000)
45.73.20.41
31.044.2.56
SSIL (2000)
29.30:1:.10 13
48, 2889
BGS
44. 12:00.0000
19.22 1.28
BGS
32.26100.11
38.25 2.19
(1000)
37.15:20.66
11:21113
DER BGS
26.08-+0091
53.87
HGB
37. 17.10::0.88
25.23::0.79
42.72-1.00
DER (2000)
47,20.1..88
31.9011..50
DER (2000)
20 00.19:0017 76
52.02::3.20
BGS
45.62:::1115 1.15
19.33::079
RCSS
34.3--0.75
37.35+2.28
GDumb (1000)
12,88:20.22 28
16. 1..0014488 88
(1000)
12.62.10.22
64.5611.61
GDumb (2000)
21.26-50.55
20.28+t11 65
GDumb (2000)
19.24t1 36
58.3211.71
EEIL. (1000) + Group DRO
27.20= 1.45
31.374-3.23
EEIL. (1000) Group
22.51=0:95
4..56::1.79
EEIL EE000)000 Group DRO
1.03
28.78++2.84
FEIL. (2000) Group DRO
27.75++0.80
38.28+2266
we compare the baselines in the Class-IL setting of Split CIFAR-I00S to provide a more unified evaluation based on Split CIFAR-10OS.
22

---

Published as a conference paper at ICLR 2024
F ANALYSIS FOR BIAS TRANSFER USING MORE REALISTIC CL SCENARIOS
BBC
Blas level of Ts: 4
Blas level of Ts:
Blas level of Ts:
Blas level of Ts:
Fine-tuning
EWC * LNV
Fine-tuning
EWC *
LNV
Freezing
Freezing
100
100 Plastiiity
Pasticityy
*
(1%)
80
(%)
80
60
60
Normalized
40
*
20
20
Stasiity?
0
20
15.0
12.5
10.0
7.5
DCA of 72 (%)
DCA of 71 (%)
(a) Forward transfer of bias
(b) Backward transfer of bias
Figure F.l: Bias transfer in tasks-CL CL (Domain-IL) on FMoW-WILDS dataset.
Blas level of
Blas level of
Blas level of
Blas level of
Fine-tume fine-tuning
EWC *
LWF
Fine lino-tuning
EWC * LWF
Freezing
ER
Freezing
ER
Pasticity
Plsstcity
(%6)
80
Normalized P. (96)
80
*
60
60
40
40
*
20
20
Stallity?
Stallly
20
15
10
40
30
20
10
DCA of Tz (%)
DCA of 71 (%)
(a) Forward transfer of bias
(b) Backward transfer of bias
Figure F22: Bias transfer two tasks-CL CL (Domain-IL) -nL on NLP datasets.
To extend our findings of bias transfer to more realistic CL scenarios, we constructed two types of two-task CL scenarios: the first one is made from FMoW-WILDS dataset (Koh et al., [021] and the other one consists of two types of NLP dataset, i.e., MultiNLI (Williams et al. 2017) and WanLI (Liu al. 2022).
FMOW-WILDS dataset consists of 141K RGB satellite images annotated with building or land use categories, the year the images were taken, and their geographical regions. Koh et al. (202I) exhibited that a classifier trained from the FMoW-WILDS dataset only performs well on data-rich regions due the disparities in the number of available data between regions. We assume two-task CL scenarios in time order, where the two tasks consist of subsets of images taken in 2010 and 2016-2017, respectively. We set the binary group label with regions, America and Asia, and the binary class label with two categories, recreational facility and toll booth, which are spuriously correlated with the group label.
We consider another two-task CL scenario where model is trained with two kinds of NLP datasets, MultiNLI and WaNLI. sequentially. The two datasets are natural language datasets where the task is to determine if given hypothesis is entailed by, neutral with, contradicts given premise. (Sagawa et al. (2020) represents that MultiNLI dataset possesses a dataset bias; among premise- hyyphhesss pairs annotated with the contradiction, most hypothesis sentences contain negation words such as "never" or "no". Based on the findings, we set the group label to the presence of negation words in the hypothesis such that the class labels in each task be spuriously correlated with group labels.
23

---

Published as a conference paper at ICLR 2024
The figures F.1 and F.2 display how much the dataset bias at the first or second task adversely influences the other task depending on the relative weight on stability and plasticity. The results indicate similar tendency with 2nd 3] Specifically, transfer of bias exists and is significant, and its degree is influenced by the relative focus on stability and plasticity.
observe again that the forward and backward
24


Reviews:
[{"review_id": "PUnffJOs1J", "reviewer": "Reviewer_UUYC", "summary": "While most continual learning (CL) algorithms focus on the stability-plasticity trade-off, this study highlights the overlooked impact of dataset bias on knowledge transfer. Through systematic experiments on three benchmark datasets, the authors show that standard CL methods can transfer bias between tasks, affecting both forward and backward learning. The degree of bias transfer depends on whether the CL methods prioritize stability or plasticity. Bias transfer accumulates in longer task sequences. To address this issue, the authors propose a standardized experimental setup and introduce a simple yet effective baseline method called Group-class Balanced Greedy Sampling (BGS).", "questions": "- In figure 4, for row 1, why does the CKA value drop even when bias of T1 is zero?\n- In figure 5, how does PackNet do on backward transfer?\n- Also can authors comment on why ER does much better than LWF and PackNet in general?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Amongst the works that look at CL and distribution shift, this is the one of the first papers to do an empirical study on impact of dataset bias on three different forms of CL: task-IL, domain-IL and class-IL. \n\n- The authors do a good job of structuring their findings, by first illustrating results in the two task case, followed by multiple tasks in a sequence. The metrics for bias (BMR/DCA) and CL (normalized $\\mathcal{F}-\\mathcal{I}$) are clearly defined.  In general the presentation is good, and the writing is clear. Particularly, experiment results in section 4 and 5 are well presented with easy to read plots. \n\n- The CKA analysis done on representations of penultimate layer (Sec 4.3) strengthens the empirical results observed.\n\n- The proposed baseline (group-class balanced greedy sampling) is fairly simple and combines ideas from GDumb and DFR. The algorithm presents significantly lower BMR over CL baselines in class-IL and task-IL settings.", "weaknesses": "- The datasets used for the experiments are small scale and synthetic. Using more natural datasets that evolve over time, e.g. FMoW dataset from Wilds benchmark, or other datasets from the Time-Wilds paper would be helpful. In some of these datasets, there is also group/attribute information that can be used to measure bias. \n\n- The paper can be improved with experiments that use a pretrained model (like CLIP), and then perform continual learning. It would be interesting to see if the same trends hold, or are they amplified/diminished with respect to forward/backward bias transfer.\n\n- I understand that theoretical analysis is not always feasible or even helpful, but for spurious correlations there exist simple settings/distributions in the SC literature where the SC induces failure even in linear models (see [1, 2, 3]). Extending these frameworks to the CL setting and then proving formal claims about forward/backward bias transfer can make the claims in this paper much stronger and build understanding to develop mitigation strategies and algorithms.\n\n[1] Nagarajan, Vaishnavh, Anders Andreassen, and Behnam Neyshabur. \"Understanding the failure modes of out-of-distribution generalization.\" arXiv preprint arXiv:2010.15775 (2020).\n\n[2] Ghosal, G. R., Setlur, A., Brown, D. S., Dragan, A., & Raghunathan, A. (2023). Contextual Reliability: When Different Features Matter in Different Contexts.\n\n[3] Sagawa, Shiori, et al. \"An investigation of why overparameterization exacerbates spurious correlations.\" International Conference on Machine Learning. PMLR, 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qIj4dxIOju", "reviewer": "Reviewer_6w81", "summary": "The paper revolves around the subject of bias transfer in continual learning (CL). The authors develop an experimental framework examining six CL strategies using two evaluation metrics across three scenarios, ensuring comprehensive analysis of the problem. Their findings show that CL techniques, unaware of dataset bias, can transfer such biases in both forward and backward directions. In response to this issue, they suggest a novel approach, Group-class Balanced Greedy Sampling (BGS), aimed at mitigating bias transfer. This paper is unique in the sense that it deliberates on the existence of spurious correlations in the CL context and calls for attention to develop bias-aware mechanisms in CL.", "questions": "- The role and impact of the stability-plasticity trade-off on bias transfer in CL could be more thoroughly explored.\n- The experimental design lacks uniformity across all three CL scenarios and should strive for a standardized evaluation protocol.\n- It would be insightful to know if the bias transfer problem exists in substantial real-world applications (as shared in weaknesses) and what limitations exist in the experimental setup.\n- Why did you use LWF + Group DRO as a comparison in Tables 1(a) and 1(b) when ER is better performing than LWF? Could you share your rationale here?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors address an often-overlooked issue of bias transfer in CL and provide a well-motivated argument adopting a fresh perspective in the field of CL.\n- Their analysis is fairly extensive, using six CL methods with three different scenarios.\n- The novelty of the proposed BGS method to mitigate bias without requiring any additional hyperparameter tuning enhances the paper's contribution.\n- The empirical evidence that reveals the existence of bias transfer in CL and its subsequent impact on the tasks makes a significant contribution to the field.", "weaknesses": "- While the empirical investigation of CL provides valuable insights, the novelty of findings and proposed BGS approach could be more explicitly addressed considering existing similar work in the domain.\n- The main limitation of this paper lies in the diversity of the samples used. The authors base their experiments on three benchmark datasets, all of which are synthetically created for Continual Learning (CL). From personal observation, it has been noted that such models can perform adequately even with a few hundred examples, contradicting the need for the 2000 or 4000 examples that BGS involves. By conducting tests in more authentic scenarios (such as shift in time/linguistic diversity, or the nature of the sequence to sequence task (like going from question answering to machine translation to paraphrasing etc)), a stronger foundation of support for the results could be established.\n- The paper's dual research goals are compressed into a limited amount of space, making comprehensive comprehension challenging. More in-depth discussion or a more detailed layout for the proposed BGS method would be beneficial.\n- The underlying mechanisms contributing to bias transfer are not entirely delved into in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9AZnaWke9x", "reviewer": "Reviewer_A8St", "summary": "This paper investigates the impact of spurious correlations in the CL setting. Through comprehensive experiments, it confirms the existence of bias transfer in CL, affecting model predictions in both forward and backward directions. Then, they establish standard experimental settings, bias-aware CL scenarios, and evaluation protocols and introduce a practical baseline method called \"Group-class Balanced Greedy Sampling (BGS)\" for advancing bias-aware CL techniques.", "questions": "1.    Can the paper provide a more detailed comparison between its proposed methods BGS and the GDumb and DFR CL methods to highlight their differences in addressing bias-aware CL scenarios?\n2.    Does Table 1 in the paper, which indicates a significant improvement in bias-aware CL scenarios when using the BGS method, imply that BGS may be less robust in scenarios with larger datasets, as increasing data has a limited impact on model improvement in all three scenarios?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.    This article has a clear motivation. I also agree that investigating spurious correlations is a highly worthwhile topic in the CL setting.\n2.    The related work section of the article provides an excellent summary of the relationship between CL methods and spurious correlations.\n3.    The paper's approach to investigating bias awareness from three distinct angles, i.e., model bias, the relative focus on the Stability-Plasticity trade-off, and bias transfer, is comprehensive and convincing.", "weaknesses": "1.    The readability of the paper could be improved. The abstract and introduction should be revised to provide a more engaging and clearer overview of the research. \n2.    The paper introduces the Bias-flipped Mis-classification Rate (BMR) and the Difference of Classwise Accuracy (DCA) as metrics, but it lacks a detailed comparison of these metrics on the proposed benchmark datasets. It would be valuable to provide an in-depth analysis of how these metrics perform under different scenarios.\n3.    Table 1 should include further comparisons of CL baselines. A more comprehensive analysis of the performance of other CL baselines on the proposed benchmark datasets would provide a stronger basis for evaluating the proposed method.\n4.    While this paper introduces the BALANCED GREEDY SAMPLING (BGS) method, its novelty appears to be limited. The paper could benefit from a more thorough exploration of innovative techniques in the bias-aware CL domain.\n5.    The experimental analysis and the overall structure of the paper should be enhanced. This paper reads more like a forward-looking exploration of bias-aware scenarios rather than a comprehensive research work. It would be beneficial to present a more detailed and rigorous experimental analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6W3Zce2r8m", "reviewer": "Reviewer_C1FY", "summary": "The paper investigates the impacts of spurious correlations in the continual learning setting.\n\nFirst, the paper defines the experimental setups and evaluation metrics for investigating the issue of bias transfer.\n\nThen, the paper conducts experiments, arguing that bias exists and existing CL approaches cannot handle the bias. This is done first for the case of two tasks, and then generalized to longer task sequences.\n\nNext, the paper proposes a method to address the bias. The method consists of retraining the last linear layer using a balanced set of data samples.", "questions": "I have no specific questions", "limitations": "", "rating": 5, "confidence": 3, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "Omitted.", "weaknesses": "In my humble opinion, the paper has several weaknesses:\n\n- The paper has limited novelty. In particular, the proposed problem is a basic combination of continual learning and bias of machine learning models. The proposed approach is a direct variant of Kirichenko et al. (2023). An analogy would be that the paper derives a corollary from a theorem in prior work. \n- In the experimental setting, the paper changes the input data by skewing them towards gray-scale images (or colored) images. This creates several issues:\n  - This setup makes the experiments artificial, human-made, and synthetic. The practical relevance is to be evidenced.\n  - The notation of bias is subjective. The way the paper changes the input data is just to make a shift of distribution or create subclasses from existing classes, rather than creating any bias. Retraining the last layer using a balanced set of data would of course improve in this case as it reduces the performance gap between (sub)classes and improves DCA. Therefore, the story the paper tries to convey is in my eyes a word game that is not convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)