Learning without Prejudices: Continual Unbiased Learning via Benign and Malignant Forgetting
Authors: Myeongho Jeon
Hyoje Lee
Yedarm Seong
Myungjoo Kang
Venue: ICLR 2023 poster (2023)
Keywords: representation learning
continual learning
unbiased learning

Abstract: Although machine learning algorithms have achieved state-of-the-art status in image classification, recent studies have substantiated that the ability of the models to learn several tasks in sequence, termed continual learning (CL), often suffers from abrupt degradation of performance from previous tasks.  A large body of CL frameworks has been devoted to alleviating this issue. However, we observe that forgetting phenomena in CL are not always unfavorable, especially when there is bias (spurious correlation) in training data. We term such type of forgetting benign forgetting, and categorize detrimental forgetting as malignant forgetting. Based on this finding, our objective in this study is twofold:  (a) to discourage malignant forgetting by generating previous representations, and (b) encourage benign forgetting by employing contrastive learning in conjunction with feature-level augmentation. Extensive evaluations of biased experimental setups demonstrate that our proposed method, Learning without Prejudices, is effective for continual unbiased learning.

Published as a conference paper at ICLR 2023
LEARNING WITHOUT PREJUDICES: CONTINUAL UNBI- ASED LEARNING VIA BENIGN AND MALIGNANT FOR- GETTING
Myeongho Jeon', Hyoje Lee", Yedarm Seong, Myungjoo Kang
Seoul National University
(andyjeon, byoje42, mybirth0407, mkang) 8snu.ar ac. kr
ABSTRACT
Although machine learning algorithms have achieved oftte-of-the-att status in image classification, recent studies have substantiated that the ability of the models to learn several tasks in sequence, termed continual learning (CL), often suffers from abrupt degradation performance from previous tasks. A large body of CL frameworks has been devoted to alleviating this forgetting issue. However, we observe that forgetting phenomena in CL are not always unfavorable, especially when there is bias (spurious correlation) in training data. We term such type of forgetting benign forgetting. and categorize detrimental forgetting as malignant forgetting. Based on this finding, our objective in this study is twofold: (a) discourage malignant forgetting by generating previous representations, and (b) encourage benign forgetting by employing contrastive learning conjunction with feature-level augmentation. Extensive evaluations of biased experimental setups demonstrate that our proposed method, Learning without Prejudices, is effective for continual unbiased learning.
INTRODUCTION
In continual learning (CL), model learns a sequence of tasks to accumulate existing knowledge for a new task. This is preferable in practice, where a model cannot retrieve previously used data, owing to privacy, limited data capacity, or an online streaming setup. The main challenge in CL is to alleviate "catastrophic forgetting." whereby model forgets prior information while training on new information (McCloskey & Cohen, 1989.. line recent works has been dedicated to mitigating methods force a current model not to be far from the previous one by
this issue. Regulariiaaionnbassee penalizing changes in the parameters learned in previous tasks (Kirkpatrick et al. 2017 Chaudhry
et al. 2018: Aljjuni al. 2018 2019a Ahn et al.) 2019 Dhar et al. 2019: Douillard et al. 2020).
Replay-based methods store samples prior tasks in a butfer and employ them along with present
samples (Robins 1995| Lopez-Paz & Ranzato 2017
Buzzega et al.
2020
Aljundi al. 2019b
Mai et al. 2021] Lin et al. 22021||Mddaa et al. 2021
Chaudhry et al.
2021
Bonicelli et al.
2022).
Generator- based methods generate prior samples and input them into current tasks (Shin et al.
2017
Kemker & Kanan 2017 Xiang et al. 2019 Ostapenko et al., [2019] Liu et al.
2020 Yin et al. 2020).
common assumption of the above-mentioned existing methods is that the training dataset is well- distributed. However, source dataset is often biased, and machine learning algorithm could perceive the bias meaningful information, thereby leading to misleading generalizability of the model (Kim et al.] 2019; Jeon et al. 2022). In the experiment in Section 3.1 we show that biased distributions are detrimental the robustness of models in existing CL. scenarios. Thus, we propose a new type of CI., termed "continual unbiased learning (CUL)". in which the dataset of each task has a different bias. With CUL, we aim to make any model trained on any task unbiased, considering all models as candidates for application. This is particularly desirable in practice, whereby model designed for a specific purpose is deployed for long periods and training datasets with divergent distributions are fed sequentially update the model.
Equal contribution

---

Published as a conference paper at ICLR 2023
Even with CUL, forgetting past information ("malignant forgetting") degrades the generalizability of a model. For instance, with Biased MNIST in Figure I the classifier perceives color as meaningful information for prediction, although it is not natural meaning associated with the number. If the model clearly memorizes prior information that there are (red, 0) and (gray, 0) samples, it could know that color is not the key factor for predicting numbers. Furthermore, we observe that forgetting is not always malignant through the experiment in Section [3.2] Although information (derived from model's I generalizability. is beneficial forget the misguidance
prior data) itself can contribute learned from biased datasets, and hence we term such a forgetting "benign forgetting". As an example, suppose classifier trained on the MNIST dataset extremely biased toward the background color, as in Section 3.2 It is unfavorable for the classifier to make logic that color numbeer and thus bet all the 'blue" images on '3', 3,, for instance.
Therefore, we aim to discourage malignant forgetting and encourage benign forgetting. Toward this, we design novel method, named Learning without Pre judices (Iw2). which employs feature generator and contrastive learning. (i) Inspired by the research in Section [.1 that the model trained with a set of data from all the tasks does not suffer from malignant forgetting, we exploit the capabilities of a feature generator. The feature generator generates feature maps containing previous information via a generative adversarial network (GAN). Feature maps provide a larger range of feature space (to be referenced to) than images, making the classifier more robust. (ii) The generated features are fed into the model by contrastive learning (Grill et al. 2020), and then current data are used for training in supervised mode. Because bias means a spurious correlation between some particular attribute variables and label space, the model can learn representations free of bias, with supe-supe learning that does not require labels. (iii) To optimize the classifier with generated features effectively, we propose feature-level uugmentation that spatially and channel-wise transforms features. An extensive evaluation of biased datasets shows that our proposed framework effective for CUL. The main contributions of this study summarized as follows:
forgetanttorgeetinng
We present novel framework, termed "continual unbiased learning", address bias in CL. Additionally, we propose continual unbiased learning benchmarks and an evaluation protocol for future research.
We find that forgetting phenomena in CL is not always catastrophic when the training dataset exhibits the non-uniform distribution of features, e.g., biased dataset, and hence categorize them into malignant forgetting and ferignforgetinn
We propose a novel method, Learning without Prejudices (LwP), that em- ploys a feature generator and contrastive learning, presenting feature-level augmentation to bridge them. LwP contributes to models' generalizability significantly.
PRELIMINARIES
2.1 PROBLEM STATEMENT
Bias. Let X' be an input space and y be a label space. We define an attribute variable allr as an informative data feature of x X, possibly ranging from fine details (e.g., the pixel at (0, 0) is black) to high-level semantics of the image (e.g., there is a cat). Thus, a set of attributes can represent data x. Formally, let A be an attribute space and a X' + 2-4, where 2 denotes the power set of A. A function a extracts attribute variables aftr E A from input space X, i.e. axx) {aftr], attr. ... aftr,). }. Among these attr, some might be very correlated y while they are irrelevant to the natural meaning of the target object. We define this attr as "bias". As machine learning al algorithms (e.g., convolutional neural networks (CNNs)) are overly dependent on training data distribution, the model could be biased, potentially leading to misleading generalizability (Torralba & Efros, 201T: Tommasi et al., 2017; Jeon et al.) 2022)). For instance, according to Bahng et al. (2020), the majority of frog images are captured in swamp scenes and many bird images are captured in the sky, making the model consider the background as a dominating cue that often fails to infer (frog, sky) and (bird, swamp) images correctly.
Continual learning. Consider a dataset = [(x v) X E X, ))} for a classification problem. "continual learning" is learning type with sequence of Ds = (D, (X, )1) where each X and D/ implicitly changes, expecting that f: X; + Vd accumulates previous information without forgetting while learning new tasks. Here, T means the number of tasks. A task & is predicting the target label y with unseen feature variable x and learning a task means the procedure of optimizing a classifier f : Xx= + Yi, with D, to make discriminative logic.
2

---

Published as a conference paper at ICLR 2023
O
2
3
4
5
6
7 8 9
Task 1
0
2
3
4
5
6
7
8
9
Task 2
:
O
1
2
3
4 5 6 7 8 9
Task 10
(a)
(b)
Figure I: Our experimental setup. (a) Biased MNIST. each task, we change the bias (background color) sliding each one by +1 for the label (0-9), and the color corresponding to 9 moves to O. (b) Distorted test set Ds. We randomly choose pixels of ratio single-colored image and then paint them to 'white" setting a as the average ratios all the number pixels, which are represented by 'white", in the MNIST. A more detailed dataset configuration is provided in the Appendix.
Continual unbiased learning. In addition to CL, suppose 3/4 different bias. We aim at making the classifier f unbiased toward any of Dr in the sequence, and term such type of learning as "continual unbiased learning". In practice, inconsistent distributions with different biases could be fed over time for a model that is applied for long periods for the same purpose. In this scenario, is desirable for the model be unbiased toward any of the datasets encompassing all of these domains.
= 3rr and each D, has
2.2 EVALUATING PROTOCOL AND METRICS
We set bias attribute and define dataset that has biases as "biased dataset" and dataset in which the bias attribute is uniformly distributed across the labels as an "unbiased dataset". We randomly split each biased dataset D. (Dirain Deal) Ds into train and validation set to have the same ratio of biased samples. We use one unbiased test data Drest for evaluation. And, we define fa as the model trained with {Df 1 sequentially and denote
(ff...... fr).
After training the model with a sequence of the differently biased datasets in favor of one attribute or another, we evaluate its average accuracy following the conventional CL protocol: Ace(fr, Dgrt) I E acc(Sr, D where T denotes the number tasks and acc(Jr, Dral) denotes the accuracy of the 7'tth model on the 1-th task of the biased evaluation set. Additionally, we suggest an average unbiased accuracy for an unbiased test set whenever each task is trained to estimate the generalizability of all the models: Accub(FF Drest) T Ei_ acr(Se, Diest), where ace( [t, Dleat) denotes the accuracy of ft with unbiased test dataset. Using this metric, we can evaluate = {f1.... fr... considering every model as a candidate for deployment.
MOTIVATING EXPERIMENTS
3.1 CONTINUAL LEARNING ON BIASED DATASETS
As a first step toward CUL, we investigate the learning tendency of a CNN-based classifier on biased datasets. We exploit biased MNIST (Bahng et al. 2020) toward background color, as shown in Figure I The unbiased test set has a uniform distribution for background color overall targets. Using the biased MNIST, we compare the average unbiased accuracy of the models trained with biased datasets UISIST (D;) simultaneously. and in the sequence [D1,-.. Dr Figure (a) shows a significant performance gap between these two learning scenarios. We suggest that this is because the prior information is forgotten, e,g. although the model refers to (red, 0) in the first task, it still makes
biased decision that gray = 0, considering only the (gray, 0) samples the second task. We term this type of forgetting "malignant forgetting".
3.2
CLOSER LOOK AT FORGETTING
Setup. With our motivation for the study being that a classifier could learn unintended information in biased conditions, hence forgetting such logic being desirable, further conduct another moti- vating experiment. We additionally construct shape-absent color images and their label (x, y) E D to investigate the model's adaptability to sequence of biased sets. The distorted dataset D is shown in Figure (b). We estimate the benign forgetting rate (BFR), which quantifies the model's generalizability by calculating the forgetting rate of the previous biased logic during the training of
3

---

Published as a conference paper at ICLR 2023
B
upper bound
-00.5
-00.
0.95
Base - EWC -LwF DGR @ GFR
ABD L.WP (Ours)
100
100
80
XL.PP (Cuis)
DGGg-WF
97, %
60
93.95%
40
EWC
20
0
Task number
Benign lorgelting rale
(a)
(b)
Figure 2: Motivating experiments. (a) Investigation for CUL. Each line in the graph means the performance of vanilla CNN for sequence Biased MNIST tasks with several different biased degrees B. Biased degree 3 denotes the percentage of biased images in the dataset meaning '0.1' as an unbiased set with uniform distribution and '1' as a completely biased one. train all the tasks from to 10 concurrently and then represent its Accuub by red line indicating the upper bound. (b) Benign forgetting rate of models. We estimate BFR for the conventional CL model, the baseline, and model (LwP).
new biased tasks. With all the models F and all the datasets Ds - -1D1,... ,D BFR BFF, Ds) can be defined as
T B(F Ds) T Σ
acelSe, D.)
(1)
acc(ft, D.) -
where ft, D.) denotes the accuracy of ft E F for D., and D, is the augmented images having the same correlation between the background and target labels as the original colored MNIST samples DD. The function n)))) denotes the number of target labels, displaying 10 for MNIST (0-9). Intuitively, the performance distance acc(fr, f, D.) 1/m()²) is the degree of the model's dependence on the bias attribute (color) for predicting Vd because there is only 'color' information in D, (Note that 1/n())) is the lower limit of the model's discrimination, meaning fo takes guess with unclear confidence). Thus, for the numerator acc D.) aco(fe-1 D.) with offset 1 1/n())). if the classifier f++1 still sustains the biased logic of f. (e.g., red = 0, ..... grey = 9 for fi in Figure D.. the difference acc( ft. D.) - acc(Se+1. D.) Rs O. whereas acc(Se. D.) acc(S++1, D.) increases towards otherwise. Denominator ace(ft, D.) - 1/mo)i) is the scaling factor. This is because if the model is relatively unbiased and thus acc(fi, D.) is initially small, the numerator takes penalty, making ace(fft D.) acc(fr+1, D.) small (and (FF, Ds) becomes small), even though forgets the biased logic well.
We set the baseline using a simple CNN model and compare it with conventional CL models, regularization- based (Kirkpatrick et al. 2017 Li & Hoiem 2017), generator-based (Shin et al. 2017 Liu et al. 2020; Smith et al. 2021) methods. We calculate Accub and BFF, Ds) with 8 0.95.
Forgetting is not always malignant. First, by performing the experiment displayed in Figure 2 (b). we find that the BFR and generalization on biased datasets have correlation; meaning that forgetting is not always catastrophic. Confirming our assumption that forgetting previous biased logic is preferable in CUL, we define such forgetting as "benign forgetting" Second, it is notable that conventional CL methods are limited to CUL with insufficient BFR on biased MNIST, and unsatisfactory accuracy on the unbiased test set, even when compared to the baseline. This because they are designed only to mitigate malignani forgetling and hence cannot adequately utilize benign forgetting.
4 METHODOLOGY
We discourage malignant forgetting by generating previous representations and encourage benign forgetting via contrastive learning. Although feature generator (Liu et al.) (020)) and contrastive learning (Cha et al.] 2021: Fini al., 2002) dedicated to CL, we first employ both of them, proposing feature- augmentations to bridge the two methods. Additionally, we qualitatively and quantitatively suggest that these methods are effective for CUL..
4

---

Published as a conference paper at ICLR 2023
4.1 DISCOURAGING MALIGNANT FORGETTING
Let f ; + y be a classifier with L layers. For 1 < 1 < L, classifier f can be split into three sub-modules as f Up.....)) fit+1.....-1 fun],, where [:] denotes the indices of the layers included the sub-modules. For convenience, we denote /n..... as I and fI++1.-..L-- L-1] as s i.e.
f f= ( [(
To address malignant forgetting, we intend to make the latent feature vector of (-th laver IR xW xC that include prior information. Following the adversarial training of the GAN (Gulrajani al. 2017), we train (fealure) generalor G 2 - IR H W xC that maps the noise vector z P2 into 1" W xc, and discriminator D R XC + [0. 1] that distinguishes real samples from P. and fake samples (zz). Thus, with fake features vy := (zz) and real features v, R : (z).
feature generator G and discriminator 'D are optimized by
min max Ex- [D(J' - E. [D(G(z)]. D
(2)
With G trained with prior tasks, fo receives both v, (current) and v (previous) as input and hence can memorize previous knowledge, L.e, discouraging malignant forgetting. When generating fake images and inputting them as [Shin et al. (2017); Kemker & Kanan (2017); Xiang et al. (2019); Ostapenko et al. (2019), because they are only 3-channel aggregated features, the classifier may be confused if the generator does not work well. However, the generated feature maps have a larger range feature space, providing the classifier with more information to be utilized selectively (please consult the comparison experiment in the Appendix.). This is favorable for CUL because biased classifier results when the model is overly dependent on a certain few features, especially the bias attribute variable.
4.2 ENCOURAGING BENIGN FORGETTING
To train fr, we apply bootstrap your own latent (B YOL) contrastive learning (Grill et al., 2020) because the generator G does not provide target labels for fake features. Although pseudo- labels can be obtained through a discriminator (Shin et al. 2017). auto-encoder-based generators (Kemker & Kanan 2017). 7. or conditional GANs (Xiang et al. 2019; Ostapenko et al. 2019), under biased conditions, few mislabeled samples could cause large degradation in performance. Furthermore, bias is fundamentally based on the correlation between label and attribute space. Thus, label-free contrastive learning encourages the classifier to make representations independent of bias and hence to be robust for bias by forgetting the biased logic, i.e., encouraging benign.forgetin forgetting.
Following the two network designs of BYOL, let gg == {f Pa. }a and 9 target := {.f Pe} be the online and target networks, where p, q denote additional embedding layers and 0, are the parameters of the online and target networks, respectively. The embedding layers p encode and g receives the output of p as input. We aim to train our objective network, 96 the representations made by 9g target i.e., knowledge distillation. The f and pe networks have the same architecture as S38 -1+11...L-11 and pe. respectively, but with different weight parameters
gonline by learning to predict
Since augmentations of input image space are not directly applicable in the feature space due to distribution shift, we propose fraposefeature-le feature-level augmentation. For given feature vector v. the transformed features v' = Dropout (v +E', 7) and v = Dropout(v from (p, Σ) are Gaussian noise vectors with the same shape as the feature v, and 7 is the dropout ratio. By adding noise vectors, the feature v is augmented spatially because adjacent pixels of feature maps contain spatial information and are closely correlated. And, we apply the channel-wise dropout technique (Tompson et al.) 2015), which means channel-wise sampling of feature maps. With augmented features v' and v", contrastive loss can be formulated as:
7), where e',e" E H W xC sampled
go(y') Coontra (v' v") [v')l C
ge Je(v") [lge(v")ll
where || denotes norm For the real and fake features, we optimize the online network go using Ceuntra (ypnttr(//,, v"). The target network g& is updated via the
the contrastive loss Crontra (v'g. v²) + C C moving average technique expressed 7 + (1 (1-77 -),, where 0 < T < 1 denotes the increase in the target decay rate during training to adjust the weight updating ratio between gonline and gg
target, After
5

---

Published as a conference paper at ICLR 2023
contrastive learning, only f0 is used in conjunction with f and fiz-- Then, we train the classifier by cross-entropy loss with samples of the current task in supervised mode. The overall procedure for the proposed approach is presented in Algorithm[[
Algorithm 1: LwP: Learning without Prejudices
Inputs Datasets for T tasks (D, (4),)))) classifier f= (f9, [(n]], online network 96 go target network 9g
target, dropout ratio y, target decay rate T, (u,,,
for f = 1 to do
while not converged do
// Train with GAN
(x,y) ~ (K. and z (x) and V +- G(2) Optimize 9 and D by 2 using Vr, Vf // Train with Contrastive Learning
sample real data and noise // obtain real and fake features
for v v do c. NN(,, v' t -rropoutyy + c,7) v" Dropout(v (v + 77)
11 sample noises for augmentations // augment the feature vector // augment the feature vector
Calculate L Coontra (v', v") in
// calculate contrastive loss
A Optimizer( V VoLoge
Coontra
0) // optimize only 0 and not & update & by the moving average
11
12
+ 71 end
18
// Train with classifier Optimize classifier f using CrossEntropy (f(x),
14
15
end
end
Output classifier f
EXPERIMENTS
In this section, we experimentally evaluate the proposed method and compare with several state-of- the-art models. We used three biased datasets: Biased MNIST (Bahng al., 2020). Biased CIFAR-10 (Hendrycks Dietterich] 2019), and Biased CelebA-Ho modified from (Krras et al. 2017).
5.1 DATASETS
3.1 to evaluate the model's For Biased MNIST, we use the experimental setup in Section generalizability. Following the bias planting protocol proposed by Nam et al. (2020), we create the Biased CIFAR-10. For the types of corruption (Snowr, Frost, Fog, Brightness, Contrast, Spatter, Flastic, JPEG, Pixelale, Saturate). we set each as bias corresponding to the target object. The biases are changed for each task in exactly the same way as the Biased MNIST, indicating (airplane, snow), (automobile, frost). .... (truck, saturate) in the first task, and (airplane, frost), (automobile, fog), exhibited a uniform distribution of corruption types along the target object. Thereby, the number of training image samples for each task is 4,000, and the test set has 10,000 images. We set 3 = 0.85 for Biased MNIST and Biased CIFAR-10. Among the attributes of images in CelebA-HQ (Karras) et al. 2017), we set 'gender' as the target label and select 'makeup' and 'hair color' as the bias of the first and second task. respectively, because they have a significant correlation with 'gender' in the dataset. We name this sampled dataset as Biased CelebA-HQ. Thus, randomly sampled images for training are ((HearyMakenn akeup, Female), (NoMakeup, Male)) and {BBlondHais aár, Female), (BackHHais Male)} for each task, respectively. We additionally set the bias of the third task by "hair length' utilizing the public annotation provided by Jeon al. (2022); hence, the training set consists of {Longlll F Female), SShorffaais ] images. All pairs of (attribute, gender) for training have 2,000 samples, and the unbiased test set is composed of 100 images to evenly distributed. For CelebA-HO. we do not split biased data for validation because there not enough biased pair samples. The detailed distribution the training and test sets presented in the Appendix.
(truck, snow) in the second task. The unbiased test set

---

Published as a conference paper at ICLR 2023
Table 1: Average accuracy Ace and average unbiased accuracy Accub all along the datasets. Base denotes simple CNN for Biased MNIST and ResNet- for Biased CIFAR-10, and Biased H without any additional regularization for forgetting unbiasing. Replay (200) means replay-based approaches with 200 previous samples buffer. We display the best performance by bold and second performance by underline. All experiments run three different random seeds and we report the means and standard deviations.
MNIST
CIFAR-10
CelebA-HQ
Dataset
Accub
Acc
Acc.b
Ace
Acc,s
Continual Unbiased Learning
Base
84.66(+2.41)
26.58(+0.45) 89.22(#6.05)
30.98(+0.41)
14.67(=1.15)
Regularization
EWC
67.56(12..5) 68.32(224335)
27.03(10.68)
77.33(=1.45) 31.73(11.77)
LwF
83,58(10.71)
19,95(11.80)
25.24(10.57)
27.16(+1..5)
74.56(10.51)
DGR
85.04(:: 1,32)
92.77(+:.116
28.94(+:2.11)
32.08(+7.000
71.56(=2.87)
Generator
GFR
78.49(11.90) 77.50(12.78)
29.11(10.488
33.95(+1100)
64.78(=4.07)
ABD
82.78(+ 1.05)
87.44(+0.722)
29.67(+0.61)
34.40(+0.89)
79.11(+4.17)
Unblasing
LFF
82.48(+)2.45)
86.85(+3.89)
25. (.(
27,50(+2.522
72.44(+3.33)
Lw? (w/o buffer)
91.57(
95,58(10.44)
31.18(10.29)
35.16(12.88)
19.78(11.90)
Continual Unbiased Learning with Replay Buffer
HAL
77.08(+9.32) +2.40(+2.21)
23.50(+8.00)
26.48(+14.222 14.26)
74.56(=4.86)
Replay (200)
DER
90.37 1.33)
95.05(10:29)
38.42(110.4) (1.07(
75.00(
LIDER
89.44(16.07)
96.88(=0.433
28.02(+0.48)
30.89(+0094))
77.44(=2.34)
LxF (w/ buffer)
92.34(10.555
96.02(10.55)
32.06(10.59)
30.99(111.8))
80.89
5.2
EXPERIMENTAL SETUP
Competing models. Previous approaches to CL can be categorized into regularization. replay, and generator-based approaches. We set all the categories on a competing line. We add an unbiasing method, LfF (Nam al.
2020), although it is not designed for CL. can be applied for CUL. Thus,
we compare the proposed method with EWC (Kirkpatrick et al. 2017). 7). LwF (Li & Hoiem 2017 (017)..0 DGR
(Shin al. 2017). GFR (Liu et al. 2020), ABD (Smith et al. (021)) HAL (Chaudhry al. 2021)
DER (Buzzega et al. 2020). LiDER (Bonicelli et al. (022) and LfF for all the datasets. We do not consider previous samples during training for new task our model design. Because it cannot be deployed when access to prior data is strictly limited due to privacy problems, c.g., personal medical or credit information. Nonetheless, for a fair comparison to the replay-based method, we evaluate our method with buffer. We set the buffer size as 200.
Implementation details. For the Biased MNIST, we employ a simple CNN composed of four convolutional layers and three fully connected layers the baseline model and backbone network of all competing models. For Biased CIFAR-10 and Biased CelebA-HQ. ResNet-18 (He et al. 2016) used the backbone. For each experiment, we used the Adam optimizer (Kingma & Ba 2014). grid search learning rate (initial value and decay schedule), stopping criterion, and batch size. For hyperparameters in competing models, we follow the same setting as that presented in this paper. However, as we experimentally found that the learning rate is important in CUL, we fine-tune it and display the result if better than the original setting. More implementation details are provided the Appendix. We set / by experiments reported in the Appendix.
5.3 EXPERIMENTAL. RESULTS
Table I exhibits the evaluation results of LwP and state-of-ther the-hart art models on CUL, From the experi- ments on three intentionally biased datasets, we find that regularization-ben based models are limited to CUL, showing performance degradation from the Base model on some datasets. It is notable that the generator-based methods show remarkable generalizability for some experiments. This implies that generating prior samples and feeding them contribute to CUL. Nonetheless, utilizing only the gener- ated prior samples is limited to encouraging benign forgetting (experiments on Biased CelebA-HQ). Based on the generator- based approach, LwP significantly increase performance with contrastive learning. This quantitatively demonstrates contrastive learning contributes for benign forgetting (It is also demonstrated in Table 2). Further, it is noteworthy that IwP (wo buffer) outperforms
7

---

Published as a conference paper at ICLR 2023
replay-based methods some datasets (Biased MNIST, Biased CelebA-HQ). The unbiased learning method, LÉF, deteriorates the performance for all the datasets, exhibiting performance degradation compared to Base.
5.4 ANALYSIS
Table 2:
Ablation study on components of LwP. We use Biased MNIST for evalua- tion. The model with none of them applied the base model, simple CNN.
Table 3: Ablation study on self-supervised learning. We use Biased MNIST for evaluation. All the architecture and experimental setup except for SSL are exactly the same.
SSL GAN
SSL
MoCo
DINO
Barlow Twins
BYOL (Ours)
Accio 84.66(1241) 19.79(11.266 (1.87()
Accuo
83.86(+0.71) 84.23(+0.39)
86.66(=0066)
91.57(=0.88)
We conduct an ablation study to evaluate the contribution of each presented component section + Table shows that the self-supervised technique (BYOL) and feature generator contribute to the generalizability of the model.
Analysis of LwP components.
Choice of self-supervised learning. We compare the performance of the models using different self-supervised learning (SSL) approaches (He al., 2020 Grill et al. 20200] Caron et al. 202T
Zbootar et al. 202T). MoCo (He et al. 2020) requires large memory banks with contrastive loss,
called infoNCE (Oord et al. 2018). BYOL (Grill et al. 2020) proposed metric learning approach
trained using momentum encoder. DINO (Caron al. 2021 complemented BYOL with similarity
matching loss and mean-teacher self-distillation Tarvainen & Valpola 2017). Barlow Twins (Zbontar et al. 202T) proposed objective function that measures the correloornl matrix between two embeddings via an identical network from two different distorted samples. MoCo is limited in its use because of the unacceptable computational costs of maintaining (posilive, negative) pairs. We decide on BOL with our SSL technique through an experimental comparison between B BYOL, DINO, and Barlow Twins. Table Jshless shows that the BYOL method outperforms other SSL methods.
LwF
Lw? (Ouns)
HEL
DER
LIDER
LwP (Ours)
90
50
80
60
60
40
30
30
0.8
0.85
0.9
0.95
0.8
0.85
0.9
B
(b)
Figure 3: Evaluations on various B. (a) Continual unbiased learning. (b) Continual unbiased learning with replay buffer. We evaluate all the experiments with Biased MNIST.
Generalization of LwP on various 3. In reality, the training dataset is biased to varying degrees. Thus, evaluate LwP several 0 and compare it state-of-the-ar methods. In Figure 3 exhibits the best generalizability for all the / in both cases.
LwP
6 RELATED WORK
6.1 CONTINUAL LEARNING
Recent literature on continual learning can be categorized into regularization-basee replay-based, and generator-based methods.
Regularization-basse methods. EWC (Kirkpatrick et al., 2017) approximated the importance of parameters from probabilistic perspective and regularized the update of decisive ones training new task. LwF (Li & Hoiem 2017) has multiple task-specific heads. It records probabilities obtained from the previous task heads and uses them as targets of surrogate loss when learning a task.
Chaudhry et al. (2018) regularized parameter updating in a current task such that the new conditional likelihood is close the previously learned one in terms of KI KL-ddivegeecce Aljundi et al. (2018)
8

---

Published as a conference paper at ICLR 2023
presented memory-awart synapses (MAS) that estimate the importance of the parameters in model and then penalize changing them in new tasks. In further work. (Aljundi et al. (2019a) investigated how to transform MAS into an online setup where the data distribution changes and the tasks are not specified. Dhar et al. (2019) presented an attention-based approach that incrementally learns new classes by restricting divergence. Ahn et al. (2019) presented incertainty-regulariiee continual learning framework based on Bayesian online learning. Douillard et al. (2220) proposed spatial-base -basd distillation loss applied throughout the model.
Replay-based methods. Inspired by the first replay-based approach for catastrophic forgetting and experience replay (Robins] 1995), the majority of related studies have heen devoted to replay-based methods. Lopez-Paz & Ranzato 2017) united rehearsal methods with knowledge distillation and regularization, making the model sufficienlly close to the previous model. [Aljundi et al. (2019b) considered a sampling prior tasks as a constraint reduction problem, resulting in maximizing the diversity of samples. Mai et al. (2021) suggested that softmax classifiers could cause recency bias in continual learning and hence exploited the nearest-class mean classifier, instead. For preferable clustering, data samples from previous tasks are saved and utilized in the current training step. Lin et al. (2021) suggested continual contrastive self-supervised learning via a rehearsal method, which preserves the feature vectors using k-means clustering from the previous dataset. Madaan al. (202I) presented technique interpolating previous and current instances. DER (Buzzega et al. 2020) exploited logits sampled during the optimization trajectory encourage consistency with its past. HAL (Chaudhry et al. (02T) complemented a new objective termed anchoring, where the model is bi-level optimized. LiDES DER (Bonicelli et al., 2022). constrained the backbone network by its layer-wise Lipschitz constants with respect to replay samples.
Generator-based methods. Although the replay-based method is an intuitive approach for tackling catastrophic forgetting, it cannot be deployed if there is a privacy issue regarding data. As an alternative, baaee methods that do not directly store prior samples were presented. DGR (Shin et al. 2017) obtained past information via a GAN, making pseudo-labels from the discriminator: Kemker & Kanan (2017) generated prior informative images via auto-encoder, where the encoder approximates pseudo labels, and the decoder makes images guided by reconstruction loss. Xiang et al.. (2019) exploited GAN conditioned the labeled features embedded by the discriminator, which allows explicit supervised learning for the classifier. Ostapenko et al. (2019) presented dynamic generative memory that employs an auxiliary classifier GAN with an increased number of parameters. In each task, binary masks were applied to concentrate on influential parameters. GFR (Liu et al., 2020) proposed a feature generator, reducing the complexity of generative replay and preventing the imbalance problem. Yin et al.) (2020) and ABD (Smith et al. (2021) applied 'inversion', , which makes olass-conditional input images from random noise via a trained network.
6.2 UNBIASED LEARNING
Unbiased learning is a branch of robustness in machine learning. As machine-learning algorithms are overly dependent on the distribution of training datasets, models are often biased, causing unreliable generalization at inference (Torralba & Efros 2011). A line of recent works has been dedicated to mitigating this issue. Most of the studies assumed that the biases in datasets are known (c.g..
color, texture) and exploited this information by designing various models (Kim et al. 2019 Geirhos al., 2019: Bahng al., 2020 Gong et al.] [0200] Adeli et al.) [02T]] Dhar et al., 2021). However, with the motivation that mechanism based on bias predefined by human knowledge is unsuited for image datasets including countless sensory attributes, LFF (Nam et al. 2020) and Jeon et al. (2022)
addressed the challenge of unknown biases.
7 CONCLUSION AND FUTURE WORK
A large body of literature suggests that forgetting while learning a sequence of tasks is catastrophic. However, in this study, we found that forgetting could encourage the generalizability of model if the dataset has bias. Based on this motivation, our proposed method, LwP, encourages benign forgetting and regularizes malignant forgetting for continual unbiased learning via a a.eeature generator and contrastive learning in conjunction with feature-level augmentation. Experimentally, the LwP contributes to generalization, while conventional CL methods are limited to unbiasing. In terms of further work, extensive exploration of the relationship between forgetting and various incomplete data distributions, e.g., imbalanced or mislabeled data distributions, are potentially value-adding future research directions.

---

Published as a conference paper at ICLR 2023
This work was supported by the NRF grant [2012RIA2C3010887 and the (STT (2010-000770) [No. 2021-0-01343, Artificial Intelligence Graduate School Program (SNU)I..
Acknowledgement.
REFERENCES
Ehsan Adeli, Qingyu Zhao, Adolf Pfefferbaum, Edith V Sullivan, Li Fei-Fei, Juan Carlos Niebles, and Kilian M Pohl. Representation learning with statistical independence to mitigate bias. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2515 2513- 2523, 2021.
Hongjoon Ahn, Sungmin Cha, Donggyu Lee, and Taesup Moon. Uncertainty- based continual learning with adaptive regularization. Advances in neural information processing systems, 32, 2019.
Rahaf Aljundi, Francesca Babiloni, Mohamed Elhoseiny, Marcus Rohrbach, and Tinne Tuytelaars. Memory aware synapses: Learning what (not) to forget. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 139-154, 2018.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12263,
2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in neural information processing systems, 32, 2019b.
Hyojin Bahng, Sanghyuk Chun, Sangdoo Yun, Jaegul Choo, and Seong Joon Oh. Learning de-biased representations with biased representations. In International Conference on Machine Learning, pp. 528-539. PMLR, 2020.
Lorenzo Bonicelli, Matteo Boschini, Angelo Porrello, Concetto Spampinato, and Simone Calderara. On the effectiveness of lipschitz-driven rehearsal in continual learning. Alice Oh, Alekh Agar- wal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: strong, simple baseline. Advances in neural information processing systems, 33: 33:15920-15930 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Hyuntak Cha, Jaeho Lee, and Jinwoo Shin. Co2l: Contrastive continual learning. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 9516-9525, 2021.
Arslan Chaudhry, Puneet Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 532-547, 2018
Arslan Chaudhry, Albert Gordo, Puneet Dokania, Philip Torr, and David Lopez-Par Using hindsight to anchor past knowledge in continual learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 6993-7001, 2021.
Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without memorizing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5138-51844 2019.
Prithviraj Dhar, Joshua Gleason, Aniket Roy, Carlos D Castillo, and Rama Chellappa. Pass: Pro- tected attribute suppression system for mitigating bias in face recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15087-15096. 2021.
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled
outputs distillation for small-tasks incremental learning. In European Conference on Computer Vision, pp. 86-102. Springer, 2020.
10

---

Published as a conference paper at ICLR 2023
Enrico Fini, Victor G Turrisi da Costa, Xavier Alameda-Pineda, Elisa Ricci, Karteek Alahari, and Julien Mairal. Self-supervised models are continual learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9621-9630, 2022.
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness. In International Conference Learning Representations, 2019. URL
Sixue Gong, Xiaoming Liu, and Anil K Jain. Jointly de- de-biasing face recognition and demographic attribute estimation. In European conference on computer vision, pp. 330-347. Springer, 2020.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21255 2020.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in neural information processing systems, 30, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9726-9735, 2020. doi: 10. 1.L10///VR222000220000000
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
net Htte://oooumm11-HH2SS1S-1C
Myeongho Jeon, Daekyung Kim, Woochul Lee, Myungjoo Kang, and Joonseok Lee. A conservative approach for unbiased learning on unknown biases. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16752-16760, 2022.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lchtinen. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv: 2017.
Ronald Kemker and Christopher Kanan. Fearnet: Brain- irainninspired model for incremental learning.
arXiv preprint arXiv: 1.11. 105663, 2017.
Byungju Kim, Hyunwoo Kim, Kyungsu Kim, Sungjin Kim, and Junmo Kim. Learning not learn: Training deep neural networks with biased data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9012-9020, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412,6680, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barnskaaaariiska,, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114 (13):3521-3526,, 2017.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-22) 2947, 2017.
Zhiwei Lin, Yongtao Wang, and Hongxiang Lin. Continual contrastive self-supervised learning for image classification. arXiv preprint arXiv:2107:01776, 2021.
Xialei Liu, Chenshen Wu, Mikel Menta, Luis Herranz, Bogdan Raducanu, Andrew D Bagdanov, Shangling Jui, and Joost van de Weijer. Generative feature replay for class-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 226-227, 2020.
11

---

Published as a conference paper at ICLR 2023
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Divyam Madaan, Jachong Yoon, Yuanchun Li, Yunxin Liu, and Sung Ju Hwang. Rethinking the repre- sentational continuity: Towards unsupervised continual learning. arXiv preprint arXiv:21 Ttf::2100006766 2021.
Zheda Mai, Ruiwen Li, Hyunwoo Kim, and Scott Sanner. Supervised contrastive replay: Revisiting the nearest class mean elassifier online iaasminncrmmnttl continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 3589-3599, 2021.
Michael McCloskey and Neal J Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jacho Lee, and Jinwoo Shin. Learning from failure: De-biasing classifier from biased classifier. Advances in Neural Information Processing Systems, 33:20673-206844 2020.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.0372 2018.
Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi. Learning to remember: A synaptic plasticity driven framework for continual learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 11321-11 11329, 2019.
Anthony Robins. Catastrophic forgetting, rehearsal and pseudorehearsal. Connection Science, 7(2):
123-146, 1995.
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay, Advances in neural information processing systems, 30, 2017.
James Smith, Yen-Chang Hsu, Jonathan Balloch, Yilin Shen, Hongxia Jin, and Zsolt Kira. Always be dreaming: A new approach for data-free class inssemnncreetttl learning. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. 9374-9384, 2021.
Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems, 30, 2017.
Tatiana Tommasi, Novi Patricia, Barbara Caputo, and Tinne Tuytelaars. A deeper look dataset bias. In Domain adaptation in computer vision applications, pp. 37-55. Springer, 2017.
Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler. Efficient object localization using convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 948-656,77 2015.
Antonio Torralba and Alexei A Efros. Unbiased look at dataset bias. In CVPR 2011, pp. 1521-1528. IEEE, 2011.
Ye Xiang, Ying Fu, Pan Ji, and Hua Huang. Incremental learning using conditional adversarial networks. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6619-666 6628, 2019.
Hongxu Yin. Pavlo Molchanov, Jose M Alvarez, Zhizhong Li, Arun Mallya, Derek Hoiem, Niraj K Jha, and Jan Kautz. Dreaming distill: Data-free knowledge transfer via deepinversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
8715 8724, 2020.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310 12320. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2023
A EXPLORATION ON THE DATA DISTRIBUTION
We construct the biased training set and unbiased test set from Background Colored MNIST, Corrupted CIFAR-IO, and CelebA-HQ set up continual unbiased learning. Figure | 7 exhibit the overall
distribution of each task on each dataset. And, the examples of Biased CIFAR- 10 and Biased CelebA displayed Figure
600
Bins
500
Rod
Count
400
Green
Blue
300
Yellow
200
Violct
Cyan
Orange
Magenta
0
1
2
3
4
Purple
Gray
Label
(a)
1200
Bias
1000
Red
Count
800
Green
600
Bluc
Yellow
400
Violet
200
Cyan
Orange
0
Magenta
1
2
3
4
5
6
7
8
Purple
Label
Grey
Figure 4: Distribution of Biased MNIST. (a) Training set (3 0.85). We randomly sampled images from MNIST to make ten subsets allocating them as training sets for ten tasks. Following the same sampling scenario, we just slid bias (color) whenever the task is changed from 10. (b) Test set. The colors are uniformly distributed for each label, denoted by unbiased set.
450
400
350
Brightness
300
8 Contrast
Count
Spaiter
200
Elastic
JPEG
Salurate
Snow
Airplane
Automobile
Bird
Doer
Dog
Frog
Hons
Ship
Truck
Label (a)
1200
Frost
1000
Fog
Brighiness
Contrast
Count
600
Spwiter
400
JPEG
Pixclato
Saturate
Snuw
airplane
autoenobile
bird
deer
horse
ship
Label
(b)
Figure 5: Distribution of Biased CIFAR-10. (a) Training set (3 0.85). We construct datasets in exactly the same way as Biased MNIST. (b) Unbiased test set.
13

---

Published as a conference paper at ICLR 2023
2500
1982
1898
2000
1500
Count
1000
500
369
545
Henvy
No
Black
Blond
Sboct
Long
Heavy
No
Black
Blood
Short
Long
Makesp
Hair Color
Hair Length
Makeup
Hair Color
Hair Length
Female
(a)
2500
2000
1957
2000
2000
1917
2000
1542
1500
Count
1000
458
43
Henvy
No
Black
Blond
Shoot
Long
Heavy
No
Black
Blood
Short
Long
Makeup
Length
Makenp
Langth
Feraaee
(b)
Male
2500
2000
2000
2000
2000
1458
1,500
Count
1000
500
542
381
520
$/19
Heavy
Black
Blond
Short
Long
Heavy
No
Black
Blood
Short
Long
Makeup
Hhir Color
Hair Length
Makeup
Hair Color
Hair Length
(c)
Figure 6: Distribution of Biased CelebA-HQ training set. (a) Task 1. 2,000 (Female, HaavyMakeup) and 2,000 (Male, NoMakeup) images are randomly sampled. (b) Task2. 2.000 (Female. BloncHair)
and 2,000 (Male, BlackHair) images. (c) Task3. 2,000 (Female, LongHair) and 2,000 (Male, ShortHair) images. However, as face image include several attributes, we display all these pairs.
33
20
20
16
17
10
Heavy
No
Black
Blond
Short
Long
Heavy
No
Black
Blond
Short
Long
Makeup
Hair Color
Hair Length
Makcup
Hair Color
Hair Length
Fomale
Male
Figure 7: Distribution of CelebA-HQ test set. We randomly sampled test set to be uniform for all the pairs. However, as the limited number of image samples, e.g., (Male, HeavyMakeup), and such samples are biased, e.g., all the (Male, HeavyMakeup) have short hair, the distribution could not be strictly uniform.
14

---

Published as a conference paper at ICLR 2023
(b)
Figure 8: Experimental dataset. (a) Biased CIFAR-10. 10. (b) Biased CelebA-HQ. We display image samples the first task. For CIFAR-10, 10, corruption the bias attribute, egg. all the airplanes are comppted by snow noise. Each time the task changes, the bias slides one by one. For CelebA-HQ, all the women put on makeup, while the men do not. Hair color and hair length are bias for the second and third tasks, respectively.
B EXPERIMENT DETAILS
B. ARCHITECTURE
Biased MNIST. The architecture of the classifier for the Biased MNIST is composed of four convolutional (CONV) layers, a global average pooling (GAP) layer, and 3 fully-connected (FC) layers (please see Table 4. this paper, We set I splitting f into (/1.2), J(3.1.5.6.71 J(S) Hence, fe and f are f|1.2] and J13.4.5.6771 respectively. ggmliue = { Pa, qe} and g used for contrastive learning, where p and are FC layers with 128 units. Since the generator, G makes fake features with the same shape as the output of encoder fa, the generated features / via belong to IR 2828x28x33
target { Pc} are
feature maps activated from the input image by 4 CONV layers. Note that / denotes an 1-th layer, e.g. I means GAP layer and 7 is the second FC layer.
Table 4: Classifier f, Simple CNN. CONV denotes the CONV layer and FC_n n denotes the FC layer for Simple CNN. H and W mean the height and width of input images.
Layer Name
Type
Output Size
CONV 1
x 16
W
x 16
2
CONV_2 2
3 3,32
H X W x 32
3
CONV 3
x 3,64
H W 64
CONV 4
[3 3 3, 128]
X
x 128
5
GAP
128
6
FC_
128 (units)
128
7
FC_2
128 (units)
8
FC_3
10 (units)
10
Biased CIFAR-10 and Biased CelebA-HQ. For Biased CIFAR-10 and Biased CelebA-HQ, use ResNet-15 18 as our classifier f as depicted in table[]] We modify the first CONV of ResNet-18 18 with instead of
the kernel size f(g) resulting in f" = fly and f f22.3.4.6] for Biased CIFAR- 10 with and set | for Biased Ce. We choose by ablation study. We set gonline {f). pe, qe} and 0g target [ Pe), where p and g FC layers with 512 units. Thus, the size of vy generated by H W 64 for Biased CIFAR-10 and H/2 x W/2 128 for Biased CelebA-HQ, where H and W denotes the height and width of input images.
For convenience, we consider a block as layer (. We split
into (f,
Generator and discriminator D. We set the architecture of the feature generator G by one FC layer and three CONV layers. During generating features, we increase the size of feature maps in G with two interpolation layers. The discriminator D consists of four CONV layers and an FC layer to distinguish whether input features real fake. The leaky ReLU activation layer follows every CONV layer. We adjusted the number of units in the FC layer for & to fit the different shapes of the target feature maps for the three datasets while maintaining the overall architecture.
15

---

Published as a conference paper at ICLR 2023
Table Classifier f, ResNet-18. Block denotes the basic building block for ResNet. H and W mean the height and width of input images.
Layer Name
Type
Output Size
3, 64
Block_
3x3, 128
II x x 64
x2
3x3, 128
3x3, 128 3x3, 128
2
Block_2
x2
II/2 x W/2 x 128
3x3, 256 3x3, 256
3
Block 3
x2
H/4 X W/4 x 256
3x3, 512 3x3, 512
4
Block 4
x2
H/8 W/8 512
5
GAP
512 of classes
6
FC
of classes (units)
B.2 TRAINING PROCEDURE
To train the classifier f. for both supervised learning with samples of the current task and contrastive
learning with previous samples, we use Adam optimizer with learning rate 10 10-4, weight decay 5 x 10-4, and (31, (22 (0.9, 0.999). To train the generator G and discriminator D, we use Adam optimizers with learning rate x 10-6 for G and 2 10-4 for D. We (31, (22 (0.5, 0.999) for Adam in G and D. To optimize the generator G and discriminator D, we use the WGAN-GP, We set input image size as 28 x 28 for Biased MNIST, 32 x 32 for Biased CIFAR-1O, and 128 x 128 for Biased CelebA-HQ. The batch size and epochs per task are 32 and 20, respectively for all the experiments.
For feature-level augmentation, set channel-wist applied dropout rate y 0.2, and μ 0 and = 0.005 * I for (p,, 2) when spatially augmenting. Following exactly the same setup of BYOL, variable := 1 0.996) rr/ K) + 1)/2, where i denotes the current training epoch and denotes the total number training epochs.
C EXPLORATION ON LwP
C.I INVESTIGATION FOR ARCHITECTURAL PARAMETER, /
Splitting layer / an important parameter in our model design. From the feature generator to online and target networks, all the architectures of the networks are decided by the parameter 1. Therefore, we conduct a comparing experiment for l. By experiment in Table 6 and Table we found all the I show competitive generalizability compared state-of-the-ut CL methods. Among them, chose the best one, = 2, for our model. We conducted the same ablation experiments for Biased CIFAR-10 to choose l.
Table 6: Comparison for splitting layer l. We evaluate the models on Biased MNIST three times and display the average of them with standard deviation.
3
2 (ours)
Accus 90.52(=0.15) 91.08(+1.24)
90.24(++090)
91.57(++0.22)
16

---

Published as a conference paper at ICLR 2023
Table 7: Comparison for splitting layer I. We evaluate the models on Biased CIFAR-10 three times and display the average of them with standard deviation.
(ours)
Accub 29.79(+0.53) 26.39(+0.54) 25.59(-0.87)
31.18(+0.29)
C.2 CONTRIBUTION OF FEATURE GENERATOR
Although several conventional CL approaches employed image generators to memorize previous samples, they are only 3-channel aggregated features. Thus, if the generator does not work well, generated input image could disturb the desirable optimization of the classifier. Instead, generated features can provide more information robust. Table eexperimentally demonstrates our intuition.
used selectively and hence make the classifier more
Table 8: Comparison for image and feature generation. We evaluate all the models on Biased MNIST three times and display the average of them with standard deviation.
Dataset
Image
Feature (ours)
Accub Accub
MNIST
88.80(+1.36)
91.87(1
CIFAR- 10
25.33(=0.74)
31.18(+0.29)
Choice of self-supervised learning. We compare the performance of the models using different self-supervised learning (SSL) approaches MoCo, DINO, Barlow Twins, and BYOL for Biased CIFAR-10 following the exactly same way as Biased MNIST.
Table 9: Ablation study on components of LwP. We use Biased CIFAR-10 10 for evalua- tion. The model with none of them applied is the base model, ResNet-18.
Table 10: Ablation study on self-supervised learning. We use Biased CIFAR-10 for evaluation. All the architecture and experimental setup except for SSL are exactly the same.
SSL
SSL
McCo
DINO
Barlow Twins
BYOL (Ours)
GAN
Accua
26.58(10.45) 30.36(10.29) 31.18(10.29)
Aoca
31.00(++0.57) 27.81(+)0.33
30.61(+0.36)
00.29
C.3 MODEL SIZE
Generally, the number of parameters affects the performance of the model. Although simple compari- son for the number of parameters is not fair because many methods for CL address malignant forget- ting by regularization-basee (additional operation for regularization) and replay-based (more training iterations with buffer) approaches, which are not related to model size. Nonetheless, generator-basee methods can be compared for model size. Therefore, we compare the model size and present a tiny It is notable that LwP (tiny) also exhibits competitive performance.
version of LwP in Table [1
C.4 REPLAY BUFFER
Learning with replay buffer can help the model to memorize previous information, discouraging malignant forgetting. Although there are some special cases, where access to prior samples is strictly limited (e.g., privacy data such as medical or financial data, with short storage period), a replay buffer is often available. Therefore, methods with small-sizes (200) and big-sized buffers (2,000). As a result, Table 12 shows that our method achieves the best performance in both scenarios.
compare the generalizability of LwP and replay-basee
17

---

Published as a conference paper at ICLR 2023
Table 11: Model size. We investigate the number of parameters for senerator-based methods. Since
set the classifier for all the models as ResNet18, compare GAN models, generator G, and D.
Dataset
DGR
GFR
A3D
LwF
LwF (tiny)
g
2.30 M
0.44 M
6,50 M
6.52 M
0.38 M
D
2.76
0.34 M
2.77 M
0.18
MNIST
Total Arruo
0.78
6,50
85.04(+
77.50(-2.78)
(2.78(+1005)
91.57(+0.82)
91.33(+0.63)
G
3.58
0.63 M
8.42
2.07
D
CIFAR-10
2.76 M
0,54 M
2.82 M
0.72 M
Total
6.34
1.17 M
8.42 M
11.28 M
2.79
Accub
28. (..4442266)
29.11(=0:48)
29.67(#0.61)
31.18(=0:29)
30.65(::1.33)
Table 12: Buffer size. With small (200) and big (2,000) buffers, we estimate Accus and report them.
MNIST
CIFAR-10
200
2,000
200
2,000
HAL
82.40(=2.211
84.28(=0.75)
23.50(=8.00)
29.24(=0.48)
9..37((11.33)
31.07(10.53) 89.84(10.72)
34.15(10.21)
LIDER
89.44(-6.07)
92.61(+2.14)
35.72(20.76) 32,06(
28.02(10.48)
TwP
92.34( 0.55)
93.14(= 0.95)
40.18(+0.37)
C.5 INVESTIGATION ON MORE VARIOUS 8
Collecting data from the real world, the degree of bias B could be various. To investigate the generalizability of LwP and competing models, 0.8 ~ 0.95 at intervals of 0.05. Figure C.3 shows that our method generalizes best for all 8
3 0.1 0.8 intervals of 0.1 along with
Base
EWC
LwF
DGR
GER
+ ABD
+ LIF
* LwP (Ours)
ACC.B
50
30
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.85
0.9
0.95
B
(a)
HAL
DER
LiDER
* LwLP (Ours)
100
90
80
ub
70
0.1
0.4
0.5
0.6
0.7
0.9
0.95
(b)
Figure 9: Evaluations on various B. (a) Continual unbiased learning. (b) Continual unbiased learning with replay buffer. We evaluate all the experiments with Biased MNIST.
18


Reviews:
[{"review_id": "_eymgCfPDc_", "reviewer": "Reviewer_kjJH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper coins the new problem of Learning without Prejudices in a continual learning (CL) setup. Unlike most CL works that assume catastrophic forgetting, which is considered bad, the paper is the first to point out that there are actually both malignant and benign forgettings and that one must actually encourage the benign forgetting. Benign forgetting happens when existing biases (correlations between the label and attributes) are gradually unlearned. Two key techniques are proposed: a GAN-like feature generation is used to discourage malignant forgetting by reducing dependencies to a few biased attributes and feature-level augmentation and contrastive learning, which encourages benign forgetting by bypassing the label-attribute correlation. Experiments show that the proposed method outperforms various baselines on several real datasets.", "summary_of_the_review": "This paper proposes the novel problem of learning without prejudices by making the key observation that forgetting can be benign. The proposed techniques for encouraging benign forgetting and discouraging malignant forgetting are effective and go well with each other. Experimental results look strong. The presentation can be improved a bit.", "strength_and_weaknesses": "Strengths\n- Defines and solves a novel and important problem.\n- The problem is well formulated.\n- The two key techniques are effective in handling malignant and benign forgetting and go well together.\n- Experiments show clear benefits of the proposed method.\n\nWeaknesses\n- In Section 3.2, the explanation right after Equation (1) is a bit cryptic. Perhaps the authors can explain while making direct references to Figures 1 and 2? It also took a while to understand how $\\beta$ plays a role when using the images in Figure 1.\n- How generally applicable is Equation (1)? I don't think it's always the case that accuracy decreases as bias decreases. Usually one would expect lower bias to lead to higher accuracy.\n- In Section 4.1, it is not clear why a GAN training on lower layers of a CNN discourages malignant forgetting only. Expanding the feature space seems to discourage both malignant and benign forgetting. What is the evidence that only malignant forgetting is affected? Also how does one set $l$? \n- Why should contrastive learning be performed on $f_\\theta^b$ instead of $f_\\theta^a$ and $f_\\theta^b$ combined (i.e., $f_{[1,\\ldots,L-1]}$)? \n- The contrastive learning seems effective when bias is defined as a correlation between label and attribute space. Although out of the scope of this paper, if bias is defined as some biased distribution in the attribute space, how would this technique perform? \n- Section 4.2: \"Since augmentations of input image space are not directly applicable in the feature space due to distribution shift\" -> can you elaborate a bit more on why this is the case?\n- In Section 5.2, please briefly explain how the baselines (LfF, EWC, LwF, DGR, GFR, ABD, HAL, DER) work.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "VDGdPGpskq", "reviewer": "Reviewer_ZsCC", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the bias in continual learning. According to the bias, this paper designs a novel method, named learning without prejudices (LwP) to discourage malignant forgetting and encourage benign forgetting. The main contributions can be summarized as:\n\n(a) Presenting a novel framework, termed \"continual unbiased learning\", continual unbiased learning benchmarks and an evaluation protocol.\n\n(b) Categorizing the forgetting into malignant forgetting and benign forgetting.\n\n(c) Proposing a novel method, Learning without Prejudices (LwP).", "summary_of_the_review": "I recommend accept, good paper.", "strength_and_weaknesses": "Strengths:\n\n(a) This paper is well-written and easy to read.\n\n(b) The phenomenon found in this work is easy to follow.\n\n(c) I appreciate the method proposed in this paper for its motivations and great performance in extensive experiments.\n\nWeaknesses:\n\n(a) Please describe BYOL in details.\n\n(b) The proposed method splits the classifier into $[f^a,f^b,f_{{L}}]$. Why training $f^a$ by GANg can address malignant forgetting? Why training $f^b$ by BYOL contrastive learning can encourage benign forgetting? \n\n(c) In the final step of Algorithm 1, $f^a$ and $f^b$ will be retrained?\n\n(d) In practice, how to determine $l$ and $L$ in Section 4.2?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EURGzukgx", "reviewer": "Reviewer_K36S", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces concepts of benign forgetting and malignant forgetting in continual learning, and it conducts a study to discourage malignant forgetting and encourage benign forgetting. The performance of the proposed method is compared to previously proposed continual learning algorithms. ", "summary_of_the_review": "The paper is generally clear; however, the scope of the problem is very narrow as the task share a common class set with the distribution of the test data being identical. The paper does not compare with more recent CUL papers and should compare it. It is not clear whether the experiment results are reproducible.", "strength_and_weaknesses": "(+) The paper proposes an interesting problem. To categorize forgetting as malignant and benign and propose an algorithm to discourage malignant forgetting and encourage benign forgetting sound interesting. Essentially there is a distribution mismatch between the training and test dataset. \n(-) In CL, the problem is generally defined assuming non-intersecting class labels; however, CUL is studied assuming that the class labels are equal. This restricted problem considered in this paper makes the CUL problem less interesting and very restrictive.  Since the test data in this paper for every task have same distribution, there does not seem to be any catastrophic forgetting in the usual sense.\n(-) The readability of the paper could be improved with examples and figures.\n(-) Comparative experimental results do not include more recent CL algorithms and the datasets used for evaluation are small and not extensive. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7nwZSYbLFkE", "reviewer": "Reviewer_vbj9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes the setting of ``continual unbiased learning\" (CUL), where for each task of continual learning, labels are strongly correlated with specific but different input features. It also proposes to use self-supervised learning methods to address the model bias issue during CUL.", "summary_of_the_review": "As explained in detail in ``Strength And Weaknesses\". I agree that the paper proposes a new setting and effective method under this setting. However, according to the weaknesses part, I think the paper needs further motivation of the proposed setting and supporting evidence of the method effectiveness in general. And the novelty is somewhat limited.\n", "strength_and_weaknesses": "Strength:\nThe setting and method are clearly presented. \nThe effectiveness of the proposed method under the proposed setting is demonstrated in several synthetic datasets.\n\nWeaknesses:\nThe setting of the paper seems artificial. Specifically, this work considers the case where strong correlations (large beta) exists between the label space and feature space. And each task contains data with different yet strong correlations. And the training is under continual learning case where old data cannot be re-visited hence strong correlations cannot be regularized using other data. Though I agree that this setting is new, but it is not very convincing to me that it is practical. For example, if we know that each task has strong bias, why would we completely drop previous data? \n\nIn the meantime, the algorithm design seems particularly-biased to the proposed setting, and not very unexpected. Specifically, in the proposed setup, each task has a strong correlation between labels and features, hence using self-supervised learning where the model is regularized without labels seems somewhat natural to me, making it less surprising. Though for this point, I agree that at least to combine GAN and BYOL, there are some technical contributions made in this work.\n\nMoreover, all experiments are conducted with synthetic data, in small scale (MNIST/CIFAR level, with < 10 tasks), and with strong artificial bias. I agree on the motivated examples that maybe frogs are taken frequently with swamp, but I can hardly image that in practice similar cases happens as in the proposed setting, for example, frog sometimes mostly taken with swamp, yet later taken with sky etc. Hence, I believe to better motivate the practicality of the proposed setting and the effectiveness of the proposed methods, two things need to be done:\n1) Show that in practice, the proposed setup actually happens, using real-world datasets.\n2) In practice, it is at least often unknown whether strong biases occurs in all tasks in the sequence. Hence, the proposed method must be also effective when no strong bias is presented in continual learning, a fair comparison against other CL methods is required on such setting (maybe normal CL problems).\n\nAlso, there is no notion of computation/parameter increase during the experiments. It would be clearer if the increase on the model parameters and computations during training can be explicitly reported, since two new branches are required for GAN and BYOL. And maybe for fairer comparisons, one can compare with baseline models with the same model size etc.. But I do understand that it might be hard to compare completely fairly on this point. \n\nFinally, for replay-based methods, only 200 images are used in the buffer, I wonder whether all methods are still much worse than the proposed one when the replay buffer is enlarged? Say at least 10 times larger?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster