Online Bias Correction for Task-Free Continual Learning
Authors: Aristotelis Chrysakis
Marie-Francine Moens
Venue: ICLR 2023 poster (2023)
Keywords: Task-Free Continual Learning

Abstract: Task-free continual learning is the machine-learning setting where a model is trained online with data generated by a nonstationary stream. Conventional wisdom suggests that, in this setting, models are trained using an approach called experience replay, where the risk is computed both with respect to current stream observations and to a small subset of past observations. In this work, we explain both theoretically and empirically how experience replay biases the outputs of the model towards recent stream observations. Moreover, we propose a simple approach to mitigate this bias online, by changing how the output layer of the model is optimized. We show that our approach improves significantly the learning performance of experience-replay approaches over different datasets. Our findings suggest that, when performing experience replay, the output layer of the model should be optimized separately from the preceding layers.

Published as a conference paper at ICLR 2023
ONLINE BIAS CORRECTION FOR TASK-FREE CONTINUAL LEARNING
Aristotelis Chrysakis & Marie-Francine Moens*
Department of Computer Science
KU Leuven
Leuven, Belgium
ABSTRACT
Task-free continual learning is the machine- learning setting where model is trained online with data generated by a nonstationary stream. Conventional wis- dom suggests that, in this setting, models experience replay, where the risk is computed both with respect to current stream observations and to small subset of past observations. In this work, we explain both theoretically and empirically how experience replay biases the outputs of the model towards recent stream observations. Moreover, we propose a simple ap- proach to mitigate this bias online. by changing how the output layer of the model is optimized. We show that our approach improves significantly the learning per- formance of experience-replay approaches over different datasets. Our findings suggest that, when performing experience replay, the output layer of the model should be optimized separately from the preceding layers.
trained using approach called
INTRODUCTION
In broad terms, continual learning the process of incrementally aggregating knowledge from data that are generated by nonstationary distribution (Lee et al., 2019; Riemer et al., 2019). The main motivation for studying continual learning is to give artificial learners the ability to learn as biologi- cal learners do po--erretuulll updating and refining their body of knowledge under changing external conditions (Silver et al., 2013). The inability of artificial learners to learn continually stems from the fact that they overwrite previously learned knowledge whenever they encounter new information. This phenomenon called catastrophic forgetting (McCloskey & Cohen, 1989; French, 1999).
In this paper, we focus specifically on lask-free continual learning (Aljundi al., 2019b). In this setting, the data are presented to the learner in small minibatches, and this setting is agnostic to the way the data distribution changes over time. In other words, assume knowledge about whether the distribution is piecewise-stationary (that is, when there are distinct tasks being learned), or whether the distribution changes continuously over time (Aljundi et al., 2019b). Most task-free continual learning approaches make use of a memory which can store a small percentage (typically 10% or less) of all observed data instances. The data instances stored in memory are subsequently replayed in order to mitigate catastrophic forgetting. This simple paradigm, called replay-base continual learning, is surprisingly effective in task-free settings. Furthermore, also supported by findings from the field of neuroscience, in relation to how biological learning takes place (Marr, 1971; & Wilson, 2007; Liu et al., 2019).
A number of continual learning methods tend to make predictions that are biased towards recently observed data (Buzzega et al., 2021; Mai et al., 2021). Several strategies have been proposed to deal with this prediction bias (also called recency bias). Unfortunately. most of them not applicable to task-free continual learning, since they have been designed for continual learning settings that consist of task sequence, and they require knowledge of which classes the current task comprises (Wu et al., 2019; Belouadah & Popescu, 2019; Buzzega et al., 2021). . One approach which is appli- cable in task-free continual learning is proposed in Mai al. (2021). but it can only be performed after the end of training, hence the learner's 's predictions during training would remain biased.
* Please address your correspondence aristotelis.chrysais @ kuleuven.he.

---

algorithm called Published as a conference paper at ICLR 2023
In this paper, we propose simple approach that performs online bias correction for task-free con- tinual learning. Our contributions are as follows: a) We formally illustrate that the conventional paradigm of model training in free continual learning overweights the importance of current stream observations (Section 3.2), and we speculate that this overweighting is a cause of prediction bias of continual learners; b) We propose novel metric quantify prediction bias (Section 3.3), and we show that this bias can be effectively mitigated by appropriately modifying the parameters of only the final layer of the model, after the end of training (Section 3.4); c) We propose novel approach called Online Bias Correction (OBC; Section 3.5), which maintains an unbiased model on- line, throughout the entire duration of learning (see Figure for an illustrationn; d) We evaluate the performance of OBC extensively, and we show that it significantly improves number of task-free continual learning methods, over multiple datasets (Section 4).
2
BACKGROUND
2.1 TASK-FREE CONTINUAL LEARNING
We define task-free continual learning as the online optimization of a model via small minibatches that are sampled from a nonstationary stream. In task-free continual learning, no strong assumptions are made about the nature of the distributional nonstationarity of the stream (Aljundi et al., 2019b). Other continual learning settings, such as maskinncemental and dlass-incrementie continual learning assume a data distribution that is piecewise stationary, hence one that only changes at discrete points in time (Van de Ven & Tolias, 2019). The objective of continual learning is to learn from all observed data despite the nonstationary nature of the distribution (Jin et al., 2021), and, in general, previous work assumes no distributional mismatch between training and evaluation data.
Previous work in task-free continual learning mostly focuses on replay-based methods (Aljundi et al., 2019c; Jin et al., 2021). The prevalent replay paradigm is called experience replay (ER) (Isele & Cosgun, 2018; Chaudhry et al., 2019). According to the ER paradigm, each minibatch of obser- vations received by the learner is combined with another minibatch of equal size sampled from the memory. The model is then trained for one step with the combined stream-and -araa---dd-mmmory minibatch. Moreover, the memory is typically maintained by online memory-populan reservoir sampling (Vitter, 1985).
There are multiple variants of the ER method. For instance, one approach called Maximally- Interfered Retrieval (MIR) replays instances that are going to be interfered the most by the cur- rent minibatch of new observations. Another approach called Maas--Balnncing Reservoir Sampling (CBRS) (Chrysakis & Moens, 2020) modifies the memory population algorithm to ensure that the memory remains balanced. There also other approaches that deviate from the ER paradigm, such as Greedy Sampler and Dumb Learner (GDUMB) (Prabhu et al., 2020), which only trains the model using data stored in memory, or Asymmetric Corss-Entropy (ACE) (Caccia et al., 2022), which uses modified version of the cross-entropy loss to prevent the drift of latent representations.
2.2 COMPUTATIONAL COST
An important issue in task-free continual learning is computational cost. Since practical applications will likely involve large amounts of data, the design of task-free continual learners should ensure they are tractable. In practical terms, let assume that a model has to learn from stream of instances. Moreover, we assume that applications with larger streams will likely involve memory storages of larger size m. In real-world applications, the difference between an O(n) learning algorithm, and an 0(mn) algorithm could be enormous. Hence, in this work we will only focus on learning algorithms whose computational cost per incoming batch is independent of the memory size m, so that the computational complexity of learning from the entire stream is O(n).
2.3 BIAS CORRECTION IN TASK- -REE CONTINUAL LEARNING
To the best of our knowledge, there is only one approach explicitly designed to correct for prediction biases in task-free continual learning. Mai et al. (2021) propose learning model using conventional experience replay, and after the entire stream has been observed, they replace the final linear layer of the model with mearest-class-menn (NCM) classifier computed using all data stored in mem-
2

---

Published as a conference paper at ICLR 2023
ory. Moreover, they demonstrate that this approach is effective in increasing the final accuracy of the model. However, in many real-world applications, there is a need for a model that learns and performs inference at the same time, and such a model should ideally be unbiased all the time. To achieve this goal, the NCM approach would have to be applied after every update of the model, and since it needs to make full pass over the memory, it would be computationally very expensive.
3 METHODOLOGGY
3.1 NOMENCLATURE
Previous work in continual learning typically views the entire neural network one learning com- ponent. In contrast, we adopt more modular view of a neural network that consists of two learning components. Specifically,
will call the output layer of neural network the classifier, and will denote it as the parameterized function c(z; 0c). Moreover, we will refer to the set of layers preceding the classifier the feature extractor, and we will similarly denote it as g(x; 0g). Using this notation, we can represent the full neural network h as the composition of and c, that is
r(:; B,) - c(g(x; 0z). where O & {0g, 0c)
(1)
An important distinction between the classifier and the feature extractor, is that classifier has a low learning capacity because consists of only one linear layer, while the feature extractor has a high learning capacity because it is composed of multiple nonlinear layers. Hence, given enough data, the feature extractor can learn more complex representations, but, in low-data scenarios, it is also more prone overfitting, in comparison the classifier.
3.2 DATA-SAMPLE BIAS
At a high level, the optimization process during task-free continual learning is very simple. At each step the learner receives a small minibatch of b observations S, = { v:) from the stream. The learner then samples another minibatch R. of equal size from its memory M = {(z;,, 3: 1mm and performs an update step over the model parameters using the gradient calculated with respect to S.U.. Finally, the learner updates its memory with respect to S,, using reservoir sampling (Vitter, 1985), or another memory population algorithm. This training paradigm called experience replay (Isele & Cosgun, 2018; Chaudhry et al., 2019). We will now explain why this paradigm has data-sampling bias that favors new observations over the instances stored memory.
An intuitive way to understand this data- data-sampling bias is to consider learner with an infinite mem- ory. Let us assume that the learner has encountered a data instances in the past-all all of which have been stored in the memory for replay- and that the learner now receives minibatch S, of b new observations. Given an Occam's s-razor assumption that all observations, current and past, equally important, we will say that the data-sampling is unbiased if all instances are equally likely to contribute to the upcoming optimization step of the model. In other words, the probability of using any instance in the optimization step should be the same, regardless of whether that instance is in the current stream minibatch S,, or stored in memory. Accordingly, the model should be updated with minibatch B sampled unffrrmy--tt-anddon from the set of all + observed instances (that is, the concatenation of the memory and the stream minibatch M U S,).
observed data
Now let us contrast this unbiased data sampling with experience replay. Under experience replay, the b new observations are included in the minibatch B with a probability of 1, and, by definition, from the memory (which contains
an equal number of instances are sampled uniformly-at-rande instances). Hence, each memory instance has a probability of b/a < 1 of being sampled for the model update. Therefore, unlike in the unbiased case described above, new observations are guaranteed to participate in the model update, but an arbitrary memory instance is very unlikely to. In essence, this is a data-sampling bias that favors current observations over past ones, and, in turn, leads to the predictions of the model being biased towards recent observations." ?
'iice the new observations arrive in small minihatches, we generally assume that a >> b. Put another way, the instances in memory vastly outnumber the instances in the newly observed minibatch. To further elucidate the concept of data-sampling bias, we include numerical example in the appendix.
3

---

Published as a conference paper at ICLR 2023
Table 1: We compare the final accuracy (Ace.) and the bias of ER, MRO, and ER with post-training bias correction (ER+BC), over four datasets. All entries are 95%-confidence intervals over 15 runs.
FashionMNIST
CIFAR-I0
CIFAR-I00 100
tinyImageNet
Acc.
Pred.
Bias
Ace.
Pred. Bias
Acc.
Pred. Bias
Acc.
Pred. Bias
FR
0.5
0.9 + 0.2
45.5 + 1.7
17.0 + 2.4
19.3 + 0.7
33.8 + 2.0
13.5 + 0.5
25.3 + 1.7
MRO
0.5
0.4 = 0.3
38.9 + 1.6
8.7 2.7
14.7 0.6
2.2
10.7 0.4
12.6 # 1.3
ER+BC
84.8 0.2
0.4 0.1
54.4
6.6 1 0.5
27.4
1..4
6.3 1 0.4
20.4 1.3
5.1 1 0.3
A hypothesis arising from this discussion is that prediction biases could be negated if the learner had a memory of infinite size, and if it constructed the minibatch B, used to update the model, by sampling instances uniforml--at-randonm would expect that, on average, a ratio of b/(a + b) of the instances the minibatch B will be from (a + b) of the instances will be from M. Unfortunately, in practical applications, it is not S,, and a/(a
from all observed data, current or past. In this case, we
possible to have a memory of infinite size. As the learner observes more and more data, it is a > b, which means that a/(a + 6) = 1 and b/(a + b) = O. Therefore, on average, the minibatches B used to train the model will contain almost exclusively data from the memory, and almost no data from the current stream batch Sy. Since in practical applications the size of the memory is much smaller than the size of the stream, this weighting scheme will arguably lead to overfitting the memory data.
3.3 QUANTIFYING PREDICTION BIAS
At this point, we will propose metric to quantify prediction bias with respect set of unseen data. Let T { } be this set, and by yi. we will denote the one-hot representation of the label ys. We define the expected vector of prior class probabilities as p E/y.]. Moreover, we define the expected vector of model predictions as q IF [M(,,;;; 0₂)] Note that both expectations are taken with respect to the data distribution of T, and are computed by averaging over the test-set instances.
We quantify the prediction bias of the model by measuring the discrepancy between the expected ground truth p and the expected prediction q. Since both p and q are vectors of probabilities, we propose the use of the -Sasnonn divergence (Lin, 1991), which is a symmetric divergence measure between two probability distributions. It is defined as
IS(p 11 q) 1)
1 Σ
Pj log
Σ j=1
9j
log Hz qj
(2)
Pj
where μ = 1/2(p + q) is the average of p and q, and j is the index over the classes I,..., e, that are present in the test set. At high level, when the model predictions are not biased, p and will be very similar, and the divergence will be close to zero. Conversely, the more dissimilar p and are, the more the divergence will increase.
3.4 POST-TRAINTE TRAINING BIAS CORRECTION
At this point, we will show that we can significantly reduce the prediction bias of a task-free contin- ual learner if we appropriately change the parameter vector O, of the classifier after the entire stream has been observed. We will compare three approaches over four datasets (we follow the experimen- tal settings described in Section 4.1). We report both the accuracy and the prediction bias (Eq. 2) after the end of the stream. We compare regular experience replay (ER), which weights equally its present and past, and memory-replay only (MRO), which uses only data stored in memory to train the model. Moreover, we also evaluate a version of ER with post-training bias correction (ER-BC). To correct the bias of the model, we use the data stored in memory to train only the classifier of the model, until convergence. The feature extractor of the model remains unchanged.
The results of this comparison are presented in Table 1. We observe that MRO is less biased than ER in all four datasets. On the other hand, despite being more biased, ER outperforms MRO in CIFAR- 10, CIFAR-100 100, and tiny ImageNet. As we hypohtesized at the end of Section 3.2, training a model using only the data stored in memory will reduce the prediction bias of that model, but it could also
4

---

Published as a conference paper at ICLR 2023
Streem
Streem
Batch
Batch
Fealure Extractor
Feature Extractor
Memory
Memory
Batch
Batch
Figure 1: An illustration of Online Bias Correction (OBC). At first (left), the feature is extractor trained using both a stream and replay minibatch (as in conventional experience replay), with the help of surrogate classifier (SC). Afterwards (right), the classifier (C) is trained using only minibateh sampled from the memory, while the feature extractor remains unchanged. This two-step process is repeated for every incoming stream minibatch. The color of a model component illustrates the instances with respect to which its gradients are calculated (i.e., both stream and memory or only memory instances). Lack of color for a component means that it is not currently trainable.
lead to overfitting. On the contrary, ER+BC is less biased than ER, and achieves significantly higher accuracy values on all four datasets.
These results are evidence that prediction bias can be mitigated just by changing the way the clas- sifier is trained. Moreover, because the classifier consists of just one layer, it has lower learning capacity, and, therefore, overfits much less than if we were to train the entire model using only the memory data (as is the case with MRO). In the following section, we exploit the findings of this section to propose an algorithm that maintains an unbiased model throughout the entire stream.
3.5 ONLINE BIAS CORRECTION
Our approach is called Online Bias Correction (OBC) and acts wrapper around other task-free continual learning methods that perform experience replay. At first, a generic task-free continual learning method performs its training step. Such training step. typically includes receiving minibatch of observations S, from the stream (Line 1), sampling another minibatch R. from memory (Line 2), and performing training step using the combination of the two minibatches (Line 3). Note that only the parameter vector of the feature extractor (FF), and that of the surrogate classifier (SC) are updated during this training step. Next, the learner decides which of the new observations in S, to store in memory, and which ones to discard (Line 4).
The following four steps (i.e., Lines 5-8) are the ones that OBC introduces. First, the fea- ture extractor frozen (Line 5hathat is, in the update step that follows, there will be gra- dients with respect to its parameter vector cal- culated in the backward pass, and thus its pa- rameter vector will not be updated. Next, new minibatch R" is sampled from the memory (Line 6). This memory minibatch is feedfor- warded through the feature extractor and then the classifier (C), and only the parameter vec- tor of the classifier, is updated with the result-
Algorithm Online Bias Correction
Stream minibatches S,, = 1,2, 2,.,....
Memory M. Feature Extractor (FE),
Classifier (C), Surrogate Classifier (SC)
1: for each stream minibatch S, do
2: Sample memory minibatch R. M
3: Train the FE and the SC using R U
4: Perform memory population using S;
ing gradients (Line 7). Since the classifier is trained using only memory data, order re- make use of label smooth-
5:
Freeze the FE
6:
Sample memory minibatch R M
7: Train the C using R
duce overfitting, ing (Szegedy et al., 2016), which a technique that adds noise the ground-truth labels, in der to discourage overconfident predictions. Fi- nally, the feature extractor is unfrozen, meaning that its parameter vector is trainable again (Line 8).
8:
Unfreeze the FE
9:
end for
The design of OBC was motivated by the preliminary experiments of Section 3.4. In particular, we exploit the fact that, as we saw earlier, it is possible mitigate prediction bias by just optimizing the
5

---

Published as a conference paper at ICLR 2023
weights of the classifier with respect to the data in memory. Moreover, the classifier, in the context discussed here, consists of a single layer, hence it is more resistant to overfitting compared to the feature extractor, which is typically deep neural network of high learning capacity. Therefore, by training the classifier only using memory data, we mitigate the data-sampling bias, and hence, its prediction bias. Regarding the feature extractor, as we saw in Section 3.4, it is beneficial for it to be trained via experience replay. possibly because this approach leads to less overfitting compared to training the feature extractor only with memory data. Therefore, we introduced a surrogate classifier in order to ensure that the feature extractor is trained in exactly the same way as in experience replay (that is, combination with a biased classifier). We have also validated this design choice experimentally in Section 4.4.
In short, OBC attempts to capture the benefits of both experience replay (less feature-extractor over- fitring), and training the classifier only with memory data (less prediction bias), in a best-of-both worlds manner.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Datasets and Experimental Setup We use four datasets of varying difficulty. The Mashi dataset (Xiao et al., 2017) contains 60,000 grayscale images of clothing items split in 10 classes. CIFAR-10 and CIFAR- 100 (Krizhevsky. 2009) each contain 50,000 color images, with the ones in CIFAR-10 being divided in 10 classes, and the ones in CIFAR- in 100 classes. Finally, tinylma- geNet (Le & Yang, 2015) contains 100,000 color images of 200 different classes. The tinylmageNet dataset is the most challenging dataset widely used in the task-free continual learning literature, mainly due to its large number of classes (200) and the small number of data instances per class (500). Our experimental setup closely follows previous work (Aljundi et al., 2019a; Jin et al., 2021; Caccia et al., 2022). We use class-incremental streams that are presented to the learner online, in small minibatches. For FashionMNIST and CIFAR-10, we use five binary tasks; for CIFAR-100 and Net,, we use and twenty tasks, respectively, each containing ten classes. We follow the majority of past work (Aljundi et al., 2019c; Chrysakis & Moens, 2020; Aljundi et al., 2019a) by not using data augmentation in our experiments.
Experience replay (ER) (Isele & Cosgun, 2018; Chaudhry et al., 2019) is the most fun- damental continual learning baseline. It performs replay using a memory that populated using reservoir sampling (Vitter, 1985). Memory-replay only (MRO) also uses reservoir sampling to pop- ulate the memory, but instead of using both stream and memory data, trains the model using only data from the memory. Maaximlll-inteeffee of ER that replays the instances which would experience the largest loss increases, if the model were to be updated using only the current minibatch of observations. Class-balancing reservoir sampling (CBRS) (Chrysakis & Moens, 2020) uses a memory population algorithm that attempts to maintain the memory as class-balanced as possible at all times. Greedy sampler and dumb learner (GDUMB) (Prabhu et al., 2020) also uses a class- balancinl memory population algorithm and trains the model using only data stored in memory. Gradient- based memory editing (GMED) (Jin et al., 2021) edits the data stored in memory in order to increase their loss values and make them more challenging. Finally, asymmetric cross entropy (ACE) (Caccia al., 2022) modifies the traditional cross-entropy loss with class-masking, which reduces representation drift during continual learning.
Methods
retrieval (MIR) (Aljundi et al., 2019a) is an extension
Architectures and Hyperparameters Similar previous work (Lopez-Paz & Ranzato, 2017; Aljundi et al., 2019a), we use a reduced R8 architecture (He et al., 2016) for CIFAR-IO, CIFAR-100, and tinylmageNet. For the simpler FashionMNIST, we use a simple convolutional neural network (CNN). For more information on these architectures, please refer to the appendix. Following previous work (Aljundi al., 2019a; Jin al., 2021), we use learning rate of 0.1 when using the reduced ResNet- 18 architecture. When using the simpler CNN, we use a learning rate of 0.03. The stream and replay batch sizes were both set to 10, in accordaace with past work. The batch size of OBC was set to 50 (please refer to the appendix for sensitivity analysis.) Past work typically
We adapt GDUMB for use in task-free continual learning (in this setting. the learner should always be available for inference, while in (Prahhu et al., 2020), the model is trained only after the entire stream is observed).

---

Published as a conference paper at ICLR 2023
Table 2: Comparison of various task-free continual learning methods with and without OBC, over four datasets. We report the accuracy after training (Acc.) and the information retention averaged over the stream (Av, IR). All entries are 95%--cnffidence intervals over 15 runs.
FaSHionMNIST
CIFAR-10
CIFAR-100
tinylmageNet
Acc.
Av. IR
Acc.
Av. IR
Acc.
Av. IR
Acc.
Av. IR
FR
83.4 0.5
86.5 0.2
45.5+
60.0 + 0.4
19.3 0.7
27.0
13.5
15.4 0.2
+OBC
0.3
8.
14.
0.4
0.6
3.4
0.5
21.7 0.2
MIR
0.2 0.4
47.1
0.7
1.7
27.4104
1.74
1.2
+OBC
8.3
8.17+0.1
53.4+
0.4
23.8
3.49
1..5
0.2
CBRS
0.5
84
4..9
0.4
19.2
0.3
14.8 13.4
+OBC
8.3
8.
53.4111
6..7
26.4
3.33
1..0
21.6 1
GMED 8.7
8.2
46.2
0.5
19.8
2.4
13.7
+.3
+OBC
8..1
90.1 0.3
5..7
1.5
2..4
35.4
17.1
21.9 1 0.3
uses memory sizes in the range of 1%-10% of the size of the stream. Hence, unless explicitly mentioned otherwise, we set the memory size to 1000 for both FashionMNIST (approximately 2%) and CIFAR-10 (2%), and to 2500 for CIFAR- (5%) and 5000 for tiny ImageNet (5%).* For OBC, and only when training the classifier, we use batch size of 50, and a sabel-smoothing" factor of 0.5 (we perform a sensitivity analysis of these two hyperparameters in the appendix). Method-specific hyperparameters were set to the values given in their respective papers.
Evaluation Metrics We calculate the accuracy and the proposed prediction bias metric we pro- posed in Section 3.3, after the entire stream has been observed. Both of these metries are calculated with respect to the unseen test set. These metrics inform about how well the model has learned after the end of learning. We also use the information retention metric (accuracy computed with respect to past observations) proposed in Cai et al. (2021), averaged over the entire stream. This metric is a form of continual evaluation that evaluates how well each method performs, not only after the end of learning, but during its entire length. Continual evaluation is crucial for real-world applications where training and inference take place interchangeably.
4.2 APPLYING OBC
At first we apply OBC four state-of-the-art, task-free continual learning methods. Namely, we compare the performance (final accuracy and information retention averaged over the stream) of ER, MIR, CBRS, and GMED, with and without OBC on four different datasets (see Table 2). (Due to of lack of space, we report the prediction bias numbers of these experiments in the appendix.) We observe that OBC improves both the learning performance of each method over the continuum (Av. IR), and their final accuracy (Acc.), for all four datasets. The learning benefit that OBC provides especially prominent in CIFAR-10, CIFAR- 100, and tinylmageNet. We note that methods that perform post-training bias correction (as the proposed by Mai et al. (2021)) would only affect the final accuracy of each method, but not the average information retention over the stream, because, by definition, the bias correction takes place after the end of training.
proposed in Section 3.4, the NCM approach
Since OBC only modifies the way the classifier is trained, we argue that when applying bias rection to various methods, we can essentially compare the quality of the representations that their feature extractor learns. In the results presented here, there is no clear best method in that respect, but future work could specifically focus on improving feature-extractor representation learning.
"To avoid potential confusion, we would like to point out that some previous works (e.g.. Aljundi et al. (2019a)) do not report the size of the entire memory, but the memory size divided by the number of classes instead. Along similar lines, the memory sizes used here 100, 100, 25, and 50, respectively.
*e found that when label smoothing is applied in combination with other methods (ER, GDUMB, ACE, etc.) it invariably leads to reduced performance, possibly because is not applied specifically to the training of the classifier, but to the entire model.

---

Published as a conference paper at ICLR 2023
Table 3: Comparison of various bias-correction methods for three memory sizes CIFAR-10. We report the accuracy after training (Acc.), the information retention averaged over the stream (Av. IR), and the prediction bias (Pred. Bias) of each approach. All entries are 95%-confidence intervals over 15 runs.
500
1000
2500
Bias
Acc.
Bias
ER
37.4 + 1.4
55.0 + 0.3
28.9 2.4
45.5
60.0 + 0.4
18.1 288
56.1 + 1.8
64.3 + 0.5
8.3
1.7
GDUMB
37.5 1.2
0.3
16.8 2.2
42.2 =
1.7
57.9
0.3
10.8 - 2.7
49.3
61.3 0.4
0.4
6.6
1.6
MRO
31.
0.3
38.2
0.3
10.5=27
46.0
616+033
7.7
1.7
ACE
46.7 1.4
63.3
4.6
51.6
1.9
65.3 0.5
5.3
53.8+1.6
67.5 0.4
5.0 1.4
ER+OBC
16.8 =
63
13
51.3
67.6 0.4
0.9
61.5 1.3
7100
1.0 0.1
Table Comparison of various bias-correction methods for three memory sizes on CIFAR-100. We report the accuracy after training (Acc.), the information retention averaged over the stream (Av. IR), and the prediction bias (Pred. Bias) of each approach. All entries are 95%--confidenc intervals
over 15 runs.
1000
2500
5000
Pred.
IR
ER
12.8 1: 0.6
23.4 1 0.3
57.2 2.4
19.3 0.7
26.9 1 0.3
34.7 1 2.0
23.1
1: 0.8
27.8 : 0.4
25.5 1.6
GDUMB
0..2++0.
19.2
+ 0.2
26.2
2.6
15.3 + 0.7
21.5 0.3
16.4
2.3
18.6 +
22.4 0.3
10.2
MRO
0.5
23.8
17.0
1.6
22.4
ACE
17.8+00.
27.0 0.3
8.7 + 1.2
22.4
29.3 0.4
6.6
25.7 +-7
30.2 + 0.4
4.8 + 0.5
ER+OBC
1..
3.3
2.5
25.1 0.5
3.
2.4
3..7
37.1
1.8 0.3
4.3 COMPARISON WITH OTHER BIAS CORRECTION APPROACHHES
At this point, we will compare OBC to three other methods that perform implicit bias correction (in the sense that they were not designed specifically to correct for prediction bias, but they do so anyway). In particular, we compare OBC to GDUMB, MRO (they are not biased by the stream in the first place since they both train models using memory data only), and ACE (it uses both stream and replay data, but with a masked cross-entropy, which is aimed to reduce representation drift, but also does not lead biased predictions). We also include ER in this comparison, baseline. The comparison takes place on CIFAR-10 (see Table 3) and CIFAR- 100 (see Table 4). We memory sizes that correspond to 1%, 2%, 5% of the size of CIFAR-IO. and 2%, 5%, 10% of the size of CIFAR-100. For CIFAR-10, we can see that OBC outperforms GDUMB and MRO, and is competitive with ACE for very small memories OBC outperforms GDUMB, MRO, and ACE, with respect to all three metrics used. In CIFAR- 100, OBC outperforms GDUMB, MRO, and ACE, for all three memory sizes. Interestingly, we point out that for both CIFAR- 10 and CIFAR- 100, the gaps in accuracy and average information retention between OBC and the other three bias-correction methods increase with larger memory sizes.
biased
500 instances. For the other two memory sizes,
Table 5: Comparison of OBC with and without (OBC SC) a surrogate classifier over two datasets. We report the accuracy after training (Acc.), the information retention averaged over the stream (Av. IR), and the prediction bias (Pred. Bias) of each approach. All entries are 95%-ccnfidence intervals over runs.
CIFAR-10
CIFAR 100
Aec.
Av. IR
Pred. Bias
Acc.
Av. IR
Pred. Bias
ER
45.5 + 1.7
60.0 +
18.1 + 2.8
19.1 + 0.7
27.0 0.3
34.7 2.0
+OBC
54.3 + 1.0
67.6 0.3
0.9 4 0.2
25.1
1 0.5
35.2 0.3
2.7 0.4
ABCC-SS SC
42.8 1 2.3
58.3 1 0.4
6.1 4 1.3
21.0 1 0.6
25.6 0.5
6.1 0.5
8

---

Published as a conference paper at ICLR 2023
4.4 THE NEED FOR A SURROGATE CLASSIFIER
Finally, we experimentally motivate the introduction of a surrogate classifier in the OBC paradigm. We compare the performance of OBC when applied to ER, with and without the use of a surrogate classifier. In the latter case, the computational graph in the left part of Figure flows from the feature extractor to the classifier, but the classifier is not updated with the resulting gradients. The right part of Figure I remains unchanged.
The results are presented in Table 5, We observe that both for CIFAR-I0 and CIFAR-I00, the alternative formulation of OBC without surrogate classifier (OBC- SC) performs worse than OBC with a surrogate classifier. These results validate the use of the surrogate classifier. Since OBC is applied only for the optimization of the classifier in both cases, we can only conclude that not using surrogate classifier negatively affects the feature extractor. In other words, it appears that the feature extractor learns better representations when it is trained in combination with the biased surrogate classifier than when it is trained combined with the unbiased classifier. Future work focusing on better representation learning in task-free settings could further interpret this finding.
4.5 DISCUSSION
As we saw in Section 4.2, OBC performs reliably for multiple methods and over multiple datasets. In Section 4.3, we showed that it also outperforms other methods that perform bias correction im- plicitly, for different memory sizes, and for two different datasets.
In our view, one important point this work makes is that, in the context of fask-free continual learn- ing, we should not necessarily be viewing a neural network as single black box. Instead, we should make a distinction between the feature extractor and the classifier, because their different learning capacities. In particular, our OBC approach trains the classifier of the neural network online using only memory data, in order to mitigate its prediction bias. Moreover, the feature extractor and the surrogate classifier are trained via experience replay, in order to prevent the overfitting that would result if the feature extractor were to be trained only using memory data. An interesting direction for future work would be investigate the extent which the lata-sampling bias affects the feature extractor of the learner.
One potential limitation of OBC is that it does not improve the feature extractor of the model. Conse- quently, the final performance after applying OBC will depend on the quality of the representations learned by the method being wrapped. Therefore, another interesting avenue for future work, would be to focus on how to improve task-free continual representation learning.
Finally, we hypothesize that OBC can be used disentangle the confounding factors of prediction bias and representation learning. In particular, when two task-free continual learning methods use the same memory population algorithm and their classifiers are both trained using OBC, any dif- ferences in learning performance can be explained by the quality of the representations learned by their respective feature extractors. For example, a method learns better representations but is also more biased than approach B, it is possible that method B would achieve higher accuracy. But, we correct for the bias in both methods using our proposed bias-correction approach, we expect method A to outperform method B, since A learns better representations than B.
5 CONCLUSION
To summarize, this work discusses the issue of prediction bias in task-free continual learning. In par- ticular, we provided a concrete explanation of how this bias is caused-namely, by the experience- replay paradigm favoring current stream observations over the past data stored in memory. Sub- sequently, we proposed an evaluation metric that quantifies the prediction bias of a model on an unseen test set. More importantly, we proposed simple approach called Online Bias Correction (OBC) that can correct this prediction bias online throughout the entire duration of learning. Be- cause of its online nature, OBC is especially useful for real-world applications in which model needs to learn and perform inference at the same time. Also, OBC is trivial to implement and can be applied as wrapper over other task-free continual learning methods in order increase their accuracy and their information retention, and reduce their prediction bias.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
This work part of the CALCULUS project (fttps://calculussppojjeeee CALCULUS project is funded by the ERC Advanced Grant H2020-ERC-201--ADG 788506.
The
REPRODUCIBILITY
Our code can be found at https: http:::://ggthbbbb/ wtt.//////ccc//eeeeeeeeee com/chrysakis/OBG
REFERENCES
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. In Advances in Neural Information Processing Systems, pp. 11849-11860 2019a.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11254 11263, 2019b.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems, volume 32, pp. 11816-11822 2019c.
Eden Belouadah and Adrian Popescu. 112m: Class incremental learning with dual memory. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 583-592, 2019.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, and Simone Calderara. Rethinking experience replay: bag of tricks for continual learning. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 2180-2187. IEEE, 2021.
Lucas Caccia, Rahaf Aljundi, Nader Asadi, Tinne Tuytelaars, Joelle Pineau, and Eugene Belilovsky. New insights on reducing abrupt representation change in online continual learning. Interna- tional Conference on Learning Representations, 2022.
Zhipeng Cai, Ozan Sener, and Vladlen Koltun. Online continual learning with natural distribu- tion shifts: An empirical study with visual data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8281-8290, 2021.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. Continual learning with tiny episodic mem- ories. In Workshop Multi-Tan and Lifelong Reinforcement Learning, 2019.
Aristotelis Chrysakis and Marie-Francine Moens. Online continual learning from imbalanced data. In International Conference on Machine Learning, pp. 1952 1961. PMLR, 2020.
Óscar Fontenla-Romero Bertha Guijarro-Beriihas, David Martinez-Rego, Beatriz Pérez-Sánchez. and Diego Peteiro-Barne Online machine learning. In Efficiency and Scalability Methods for Computational Intellect, pp. 27-54, 2013.
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 144):128 1999.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, no. 1, 2018.
Daoyun Ji and Matthew Wilson. Coordinated memory replay in the visual cortex and hippocampus during sleep. Nature neuroscience, 10(1) 100-1700 2007.
10

---

Published as a conference paper at ICLR 2023
Xisen Jin, Arka Sadhu, Junyi Du, and Xiang Ren. Gradient- editing of memory examples for online task-free continual learning. Advances in Neural Information Processing Systems, 34: 19193-29205, 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7,7):3, 2015.
Soochan Lec, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for task-free continual learning. In International Conference on Learning Representations, 2019.
Jianhua Lin. Divergence measures based the shannon entropy. IEEE Transactions on Information theory, 37(1):145-1511 1991.
Yunzhe Liu, Raymond Dolan, Zeb Kurth-Nelson, and Timothy EJ Behrens. Human replay spon- taneously reorganizes experience. Cell, 178(3):640-652, 2019.
Vincenzo Lomonaco and Davide Maltoni. Core50: a new dataset and benchmark for continuous object recognition. In Conference Robot Learning, pp. 17-26. PMLR, 2017.
David Lopez-Paz and Marc' Aurelio Ranzato, Gradient episodic memory for continual learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pp. 6470-6479, 2017.
Zheda Mai, Ruiwen Li, Hyunwoo Kim, and Scott Sanner. Supervised contrastive replay: Revisiting the nearest class mean classifier in online dass-incremental continual learning. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3589-3599, 2021.
Marr. Simple memory: a theory for archicortex. Philosophical Transactions of the Royal Society of London, Series B, Biological Sciences, 262(841):23-81, 1971.
Michael McCloskey and Neal J Cohen. Catastrophic interference in comnectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. Gdumb: A simple approach that questions our progress in continual learning. In European conference on computer vision, pp. 524-540. Springer, 2020.
levgen Redko, Emilie Morvant, Amaury Habrard, Marc Sebban, and Younes Bennani. Advances in Domain Adaptation Theory. Elsevier, 2019.
Matthew Riemer, Tim Klinger, Djallel Bouneffouf, and Michele Franceschini. Scalable recollections for continual lifelong learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, no. 01. pp.
2019.
Daniel L. Silver, Qiang Yang, and I. Lianghao Li. Lifelong machine learning systems: Beyond learning algorithms. In 2013 AAAI spring symposium series, 2013.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and puttern recognition, pp. 2818 1816-2826, 2016.
Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning, arXiv preprint 1rXiv:1904007733 2019.
Jeffrey S Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS), 11(1:337-57, 1985.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 374-382, 2019.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist. a novel image dataset for benchmark- ing machine learning algorithms. arXiv preprint arXiv: 1708.07777, 2017.
11

---

Published as a conference paper at ICLR 2023
A HYPERPARAMETER SENSITIVITY
We now examine how sensitive OBC to hyperparameters. (Note that other hyperparameters such the stream batch size are applicable, not directly to OBC, but only to the method that OBC wraps.) In Figure 2, we present the final accuracy ER+OBC for various values of the OBC batch size (10, 20, 50, 100, 200), and the label-smoothing factor (0.2... .08). The accuracies correspond to CIFAR-100 for memory sizes 1000 and 2500, and are presented as 95%-confidence intervals. The batch-size curves very similar for both memory sizes. Higher batch sizes correlated with higher accuracies, but with diminishing returns. For the label-smoothing factor, we can see that relatively large values (0.7 0.8) lead to lower accuracies for both memory sizes. In addition, only for the smaller memory of 1000 instances, we also observe that a relatively small -moothin factor of 0.2 0.3 also leads to slightly reduced accuracies.
CIAAR-00 100-1000mP momory of 1000
CFAR 100 2500
3-
260
233
DOC Butch
14.5
23.
265
380
0.2
Figure 2: We examine the sensitivity of OBC with respect to its two hyperparameters, namely, the label-smoothing factor, and the batch size. The comparison performed on CIFAR-100 for two memory sizes. The results are presented as 95%-confidence intervals over 15 runs.
Table 6: (left) A simple convolutional block; (middle) The Convolutional Neural Network (CNN) architecture used in the FashionMNIST experiments. (right) The reduced ResNet-18 18 architecture used for CIFAR-IO, CIFAR-100, and tinyImageNet, is built using the BasicBlock(Ar, 7b. n2) from (He et al., 2016), where ny is the number of convolutional filters, np is the number sub-blocks per block, and n, the stride of the layer.
ConvBlock
CNN
Reduced ResNet-188
Conv2D(1a, Pout)
(oovBlock(1,33 ConvBlock (32, 64)
BasicBlock(20, 2, 1)
ReLU
BasicBlock(40, 2, 2)
BatchNorm2
Linear(64) (64,
BasicBlock(80., 2,
Conv2D (ooat Tlout)
BasicBlock I (60, 2, 1)
ReLU
AveragePooling
(DcchNorm2D(Moou)d
Linear [100, c)
MaxPoolinggDD(2.2)
B ARCHITECTURES
In Table 6, we report the architectures used in our experiments.
C RESULTS ON C0RE50
In Table 7, we present results on the CORe50 dataset (Lomonaco & Maltoni, 2017). This dataset contains sequential video frames 50 objects filmed during 11 different sessions, with each session
'We remind the reader, that for all previous experiments use OBC hatch size of 50 and label- smoothing factor of 0.5.
12

---

Published as a conference paper at ICLR 2023
Table 7: Comparison of four methods with and without OBC using the CORe50 dataset. All entries are 95%-confidence intervals over 15 runs.
Acc.
Av. IR
Pred. Bias
ER
26.8 1.2
76.9 + 0.7
24.8 + 1.7
+OBC
1.2
8..0
2.2 0.5
MIR
1.6
8.7
3.6
+OBC
0.9
8..2
2.1
CBRS
2.21
7.7
2.70
+OBC
3..2
8.2
2.5
GMED
1.3
0.9
1.7
+OBC
3.2
0.5
0.5
Table 8: Comparison the prediction bias four task-free continual learning methods with and without OBC, over four datasets. The prediction bias is always calculated after the end of the stream, with respect to the unseen test set. All entries are conficneeee intervals over 15 runs.
FashionMNIST
CIFAR-10 10
CIFAR-100
tinylmageNet
ER
0.9 + 0.2
17.0
33.8 2.0
25.3 + 1.7
+OBC
0.2 1:2+0.1 0.1
0.9
2.7
2.4 0.5
MIR
0.2
16.0
3.0
28.3
+OBC
0.2
1.0
0.5
3.5 0.7
CBRS
1.2
18.7
33.7 2.1
25.3 + 1.9
+OBC
0.2
1.0
2.5
2.4 0.4
GMED
0.3
1..9
3.06
25.1 1.7
+OBC
0.1
1.0 0.5
0.5
2.3 0.6
consisting of approximately 300 frames. As Lomonaco & Maltoni (2017) suggest, sessions 3, 7, and 10 are used for evaluation purposes (approximately 45,000 images), and the remaining 8 sessions used construct the stream (approximately 120,000 images). The stream consists of ten tasks of five objects each. We used the reduced ResNet- 18 architecture described earlier, with a learning rate of 0.01 and memory size of 2400 (2% of the size of the stream). The remaining hyperparam- eters are identical to the ones described in the main paper. As before, we compare four task free continual learning methods, with and without OBC, and we report the accuracy and the prediction bias calculated with respect to the unseen test set, and the information retention (accuracy on past observations) averaged over the stream. We observe that the use of OBC results to higher accuracy and average information retention, and lower prediction bias, for all four methods.
D REDUCTIONS IN PREDICTION BIAS WHEN USING OBC
In Table 8, report the prediction bias results from the experiments of Section 4.2, which included in the main text due to lack of space. These results suggest that the use of OBC leads to significant reductions in prediction bias. These reductions are consistent for all four methods and all four datasets.
E A NUMERICAL EXAMPLE OF DATA-SAMPLING BIAS
Consider learner that has previously observed data 990 instances, and these instances are currently stored the learner's 's memory. A new minibatch of 10 instances is now observed by the learner. If the leamner wants to update the model in unbiased way, all 990 + 10 data instances should be equally likely to participate in this update. Assuming that instances will be used in the update, the probability that any instance participates is therefore 20/(99) 10) 0.02. In experience
13

---

Published as a conference paper at ICLR 2023
replay, however, the 10 instances of the new minibatch guaranteed to participate in the update (therefore with a probability of 1), while an equal number of instances would be randomly sampled from memory. Hence, each memory instance is sampled with probability of 10/990 Therefore, in experience replay, the probability that a new instance will participate in the update is 1/0.01 100 times larger than that of random memory instance, while in the unbiased case, all 990 + 10 are equally likely to be sampled, regardless of whether they are new observations or stored in memory.
0.01.
F CONTINUAL LEARNING, ONLINE LEARNING, AND DOMAIN ADAPTATION
For the sake of clarity provide a high-level comparison of continual learning, online learning, and domain adaptation. In continual learning, the data distribution changes over time and, in gen- eral, previous work assumes that all observed instances are equally important, and that there is no distributional mismatch between the training and testing data (Jin et al., 2021). Online learning is similar continual learning in the sense that the data distribution changes over time. Yet, the goal in online learning is for the model to appropriately adapt to the current state of the data distribution (Fontenla-Ronee Romero et al., 2013). Therefore, currently observed instances are essence more impor- tant than previous ones. Finally, in the problem of domain adaptation, model is trained offline using a set training data. Subsequently, the model needs to perform inference on data generated by data distribution that is different (to degree) from the one that generated the training data (Redko et al., 2019).
14


Reviews:
[{"review_id": "ZzxlSDt4p4", "reviewer": "Reviewer_cbbB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method for correcting bias of continual learning method using experience replay. The proposed method uses a memory from past examples to train the last layer of a neural network trained by an experience-replay-based method. This prevents the model from being biased to recently observed samples. By fixing layers other than the last, it also prevents overfitting to the chosen samples from the memory. The authors show that the proposed method indeed reduces the bias and improves the overall performance.", "summary_of_the_review": "Although the work is well-motivated, the paper is easy to read, and the experiments show the effectiveness of the proposed method, some important details are not clear, and the theoretical part might have a weakness. Also, there is no evidence that the proposed method really corrects the bias theoretically or even empirically because the measure of the bias does not seem to capture the bias that the authors are trying to mitigate. I vote for weak reject.", "strength_and_weaknesses": "# Strengths\n- The paper points out an important weakness of experience replay that seems relevant to the research of continual learning.\n- The proposed method is simple and widely applicable without big implementation or computational overhead.\n- The experiments consistently show its effectiveness.\n\n# Weaknesses\n- The problem setup on the mathematical level is not clear to me. The paper says that it does not assume strong assumptions, but without any assumptions on how the distribution can change, I don't think we can say much about the learnability. One possible direction may be considering some form of cumulative regret as in online learning, but the paper does not seem to be considering such a setup either.\n- The setups of the experiments are also not clear to me. How the samples are drawn in the stream from the dataset (e.g., gradually including new classes) seems important, but I could not find any part mentioning that.\n- I could not understand how the proposed method makes the right amount of bias correction. If the effect is too strong, it can be biased in the opposite way, to the past experience side.\n- The measure of the \"bias\" is misleading because it does not quantify the bias in the sense of the standard statistics. (The expectation is not taken over all possible draws of the training samples.) It is also a quite weak measure because it averages over the samples.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Y4JHTBvU6P_", "reviewer": "Reviewer_S1Ps", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on online bias correction during task-free continual learning. The paper first shows, both theoretically and empirically, why simple experience replay biases on the results of the recent stream observation. Second, the paper introduces the metric to quantify prediction biases. Using the observation from this, the paper proposes a simple approach called Online Bias Correction which appropriately modifies the final layer of the network to correct for the biases online. Finally, the paper concludes with extensive experimentation of the OBC showing significant improvement on a number of task-free continual learning methods on multiple datasets.", "summary_of_the_review": "The paper focuses on a problem that is underexplored. The ideas presented are novel. The contribution is significant in the area of Bias correction, especially in online settings of task-free continuous learning. Even though the proposed method is a  wrapper around the existing methods, I think it is a significant and valuable contribution in this area as it is also evident from the rich set of results included.\n", "strength_and_weaknesses": "Strengths:\n1. Online task-free continual learning has recently drawn more attention because it focuses on solving practical challenges. The direction of this paper is promising as it focuses on correcting the bias incurred during training, addressing a major problem in this area.\n2. The claim of correcting the prediction bias by changing the way the classifier is trained is well supported by extensive experimentation of various methods on a good amount of datasets.\n3. The proposed method, Online Bias Correction, is simple but very effective. Also, it acts as a wrapper around other task-free continual learning methods. The main advantage of this is that it can be applied to existing methods without any major change in existing architecture and training procedure.\n4. The paper also includes the ablation study on surrogate classifier which was introduced by the proposed method. Its effectiveness is well supported by extensive experimentation.\n\nWeaknesses:\n1. Even though the paper has used a variety of datasets, many recent papers in this area also use CORe50 [1]. I believe that it would strengthen the experiments and would make it easier to compare against some other work.\n2. In section 3.2, the explanation on data sampling bias is unclear. First, they state that past and future observations have contributed equally. Then it is changed to not contributing equally, making it confusing. I believe that a clear mathematical definition would likely remove the confusion.\n3. Main concern of the method is that it is a wrapper over the existing methods which makes it dependent on them and also might not guarantee effectiveness on the methods outside the ones used in the experiments.\n4. Table 1 includes a column called “Bias”. Is this the result of the Jensen-Shannon divergence? \n\n[1] Lomonaco, Vincenzo, and Davide Maltoni. \"Core50: a new dataset and benchmark for continuous object recognition.\" Conference on Robot Learning. PMLR, 2017.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "QSNuckm80R", "reviewer": "Reviewer_yCcF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the problem of recency bias in the task-free continual learning setup. It proposes a new metric to quantify prediction bias and a method to mitigate it through adjusting parameters of the final model layer post training. The authors then propose a online procedure for training and unbiasing the model over the data stream via an experience replay mechanism. They test the performance of the algorithm over multiple datasets comparing favourable to several baselines.", "summary_of_the_review": "The topic of the paper - continual learning - is an important one and novel methods in this area would be of great interest to the community. However, I find the current paper lacking significantly in clarity, quality and novelty to be recommendable for publication at ICLR. I therefore recommend rejection from the conference.", "strength_and_weaknesses": "Strength:\nThe paper looks into the very important problem of continual learning. It is very readable and easy to follow.\n\nWeaknesses:\nThe paper lacks on clarity and quality at several major points. These are concretely listed in the next section. It is also of limited novelty - also clarified in the following section.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "EZUZXKUDNiT", "reviewer": "Reviewer_zZ8e", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Paper proposes a method to correct a recency bias in replay-based task-free continual learning, but separately optimising the final connected layer  of network from the rest of the network. Focuses on continual learning in vision with evaluations in the area.", "summary_of_the_review": "A simple method to mitigate the effect of the recency bias in task-free continual learning. Well written paper. Accept.", "strength_and_weaknesses": "Strengths\n- A very simple but highly effective method for bias correction in task-free continual learning\n- Ablates the effect of various components (e.g. utility of surrogate classifier).\n- Well written paper with decent set of evaluations in the vision space.\n- Good comparison to baselines and shows particularly good performance with respect to the bias metric proposed, and another metric proposed by Cai et al (2021).\n\nWeaknesses\n- Lack of understanding into why various components work despite ablations. This makes it difficult to find as much value in the paper, given the focus on toy datasets. Concretely, in section 4.5 it details why a surrogate classifier is necessary. However, there is no exposition on why surrogate classifier is necessary. There is a reference to gradient flow and figure 1 but nothing beyond that.\n\nMinor Comments & Questions\n- Provide a name for the bias metric, and then use that name in the tables\n- Highlight performance of OBC in tables.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster