Populating memory in Continual Learning with Consistency Aware Sampling
Authors: Julio Hurtado
Alain Raymond
Vladimir Araujo
Vincenzo Lomonaco
Alvaro Soto
Davide Bacciu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Continual Learning
Learning Consistency
Populating Memory
Memory-based Continual Learning

Abstract: Continual Learning (CL) methods aim to mitigate Catastrophic Forgetting (CF), where knowledge from previously learned tasks is often lost in favor of the new one. Among those algorithms, some have shown the relevance of keeping a rehearsal buffer with previously seen examples, referred to as $memory$. Yet, despite their popularity, limited research has been done to understand which elements are more beneficial to store in memory. It is common for this memory to be populated through random sampling, with little guiding principles that may aid in retaining prior knowledge. In this paper, and consistent with previous work, we found that some storage policies behave similarly given a certain memory size or compute budget, but when these constraints are relevant, results differ considerably. Based on these insights, we propose CAWS (Consistency AWare Sampling), an original storage policy that leverages a learning consistency score (C-Score) to populate the memory with elements that are $easy$ $to$ $learn$ and $representative$ of previous tasks. Because of the impracticality of directly using the C-Score in CL, we propose more feasible and efficient proxies to calculate the score that yield state-of-the-art results on CIFAR-100 and Tiny Imagenet.

Under review as a conference paper at ICLR 2023
POPULATING MEMORY IN CONTINUAL LEARNING WITH CONSISTENCY AWARE SAMPLING
Anonymous authors Paper under double-blind review
ABSTRACT
Continual Learning (CL) methods aim to mitigate Catastrophic Forgetting (CF). where knowledge from previously learned tasks often lost in favor of the new one. Among those algorithms, some have shown the relevance of keeping re- hearsal buffer with previously seen examples, referred to as memory. Yet, despite their popularity, limited research has been done to understand which elements are more beneficial to store in memory. It is common for this memory to be populated through random sampling, with little guiding principles that may aid in retain- ing prior knowledge. In this paper, and consistent with previous work, we found that some storage policies behave similarly given certain memory size pute budget, but when these constraints are relevant, results differ considerably. Based these insights, we propose CAWS (Consistency AWare Sampling). original storage policy that leverages a learning consistency score (C-Score) to populate the memory with elements that are easy to learn and representative of previous tasks. Because of the impracticality of directly using the C-Score in CL, we propose more feasible and efficient proxies calculate the score that yield state-of-the-an art results on CIFAR-100 and Tiny Imagenet.
INTRODUCTION
Deep Learning models have repeatedly shown state of the art performance in numerous tasks, in- cluding image recognitior He et al. (2016): Dosovitskiy et al. (2020), Natural Language Processing et al. (2018): Brown et al. (2020) or games previously thought to be intractable to solve, such as Go Silver et al. (016) and Starcraft II Vinyals limitation, all these models lack versatility: when trained to perform novel tasks, they rapidly for- get how to solve previous ones. This condition is known as catastrophic forgetting and is the main problem tackled by Continual Learning methods Parisi et al. (2019) Delange et al. (202T).
al. (019). However, as common
variety of methods have been proposed to approach this problem. Some have focused on allocating parameters sub-spaces for each new task Rusu et al. (2016); Mallya et al. (2018). others define restrictions on gradients learned Kirkpatrick et al. (2017] Lopez-Paz & Ranzato (2017), while others meta-learning to learn reusable weights for all tasks Rajasegaran al. 2020); Hurtado et al. (2021). Among these, memory-based methods like Experience Replay Chaudhry et al. (209)) Kim al. (2020) have consistently exhibited greater performance while being easy understand. In these methods, a memory of samples from previous tasks is kept during training of the current task avoid forgetting how solve previous tasks. Notwithstanding the popularity and effectiveness of memory-based methods, few studies have been conducted on how populating the memory affects the performance of CL methods. In particular, [Chaudhry et al. (2018a); Wu al. (2019); [Hayes et al. (2000): Araujo et al. (2022) show that when populating the memory by focusing solely on sample diversity or class balance, random selection of elements ends up performing nearly or just as well without adding extra computation.
clear that having a representative set of examples the underlying distribution critical for preserving previous knowledge. Ideally, one would like to save a large number of samples. Unfor- tunately, since saving large amounts of data results in computational overhead, we have to limit the memory size and choose which elements keep. In this paper, we argue that this memory must satisfy two fundamental requirements in order to perform reliably. The first is to have elements that easy to remember, that the model can learn quickly. The second is have elements that are

---

Under review as a conference paper at ICLR 2023
suitable representation of the distributions of past experiences, having diversity but being careful to avoid outliers. We refer to these two ideas as Fast Learning and Diversity.
For the first requirement, we leverage concept called learning consistency (Jiang al. 2021 to measure how consistently a sample is learned by a given class of models. Specifically, we populate the memory with elements with higher consistency values, which have also been shown to be the fastest to learn. Thus, sample efficiency is improved for memory samples. However, by selecting only those samples with the highest consistency values, the model learns only limited set of patterns, reducing the diversity of samples stored in the memory and limiting how much of the decision boundary the model is capable of representing. To overcome this, we propose Consistency AWare Sampling (CAWS) as new populating strategy that incorporates sampling from a broader group of high consistency elements. This new proposal adds diversity to the memory while allowing the model to learn far more detailed decision boundary.
One of the limitations of the consistency score (C-Score) that it requires training multiple models with the entire training distribution in order to find out how easy or difficult it is to train example, which, in addition to being expensive, is impractical for CL. mitigate this problem, we propose proxies to calculate the consistency of an example, achieving similar performance without the need of training multiple models on the entire training set beforehand. These proxies are not limited to CL scenarios, they also be used in environments where these types of scores commonly used, such as in Curriculum Learning (Bengio et al. 2009)..
Thus, contributions be summarized follows:
Taking a step towards understanding how the memory should be populated based on the eflectiveness-effiiienny trade-off of the scenario.
In Section 3. we propose a novel method Consistency AWare Sampling (CAWS)- for populating the memory for Continual Learning based on the idea of learning consistency.
This method equals or outperforms state-of-the-art memory selection methods.
Since learning consistency requires trained models on the same training data to be esti- mated, in B we propose practical proxies that require no extra training and achieve similar results. Moreover, these proxies could be used for other scenarios where C-Scores are required, such as Curriculum Learning.
2 FAST LEARNING
Multiple ways to populate memory in CL have been proposed. However, few studies have explored when different approaches work better than others. Some studies have shown that, under certain conditions, there is no significant difference between the proposed methods, showing how limited our understanding of how replay strategies this work, we will consider the following methods as baselines:
(a) Reservoir. A reservoir (ittee 1985 ) strategy allows sampling elements from a stream without knowing how many instances to expect. The method selects each sample with probability ad where N is the number of elements observed so far, and M is the memory size. This way, acts randomly to maintain uniform sample from the already seen stream.
(b) Class Balance. As the name states, each class has an equal proportion of the buffer size Chrysakis & Moens, 2020). We use dynamic assignment, meaning that the memory is always complete. Samples of new classes replace instances of old classes to maintain equal distribution in the memory.
(c) Task Balance. Similar to Class Balance, but instead of an equal proportion of classes, the memory is divided by the number of tasks the sequence has (Lopez-Pat Paz & Ranzato 2017h.
(d) Mean of Features (MF). Proposed by Rebutfi et al. 2017). calculates an average class fea- ture vector, based the representations of the elements in memory for a given class. If the distance of the new vector to the corresponding class vector is smaller than the farthest in the memory, we replace the new example with the farthest one.
2

---

Under review as a conference paper at ICLR 2023
These methods perform rather similar to one another. Table shows the mean accuracy the sequence of tasks obtained by these baseline methods. MF behaves similarly to Reservoir, Class Balance, and Task Balance when we train each task for limited amount of epochs (1). However, when increasing the number of epochs, we see that MF outperforms the other methods considerably, showing the importance of correctly populating the memory in some settings. The behavior is not limited to a simple environment, as it can be replicated when changing the memory size and the model used.
The similarity between methods when we have limited computation suggests that elements selected are unsuitable when training for small number of epochs. Given the selected memory. the model cannot correctly identify relevant patterns from previous tasks, We hypothesize that: when we face restrictions on the number of epochs that a model can train or the memory size, we need to focus only on those elements that are faster to learn.
Given this implication, and taking a page from the Curriculum Learning literature, we propose prioritizing samples that are faster to learn based on learning consistency. In particular, propose to use a metric developed for Curriculum Learning called Consistency Score (Jiang et al. 2021) This metric has shown benefits for faster learning under compute restrictions while also providing more robust learning to noise (Wu et al., 2020) in Curriculum Learning.
Learning Consistency C-Score measures how learnable specific sample can be concerning set of models. It has been proposed as a task agnostic measure of the difficulty used in Curriculum Learning. We define the C-Score as follows: let D be a dataset of size n sampled from an underlying distribution P. Let f(, D) be a model trained on D. Then, for an instance (æ, y) of D, Learning Consistency defined as:
Cp (I, v)
E [I; (2, = DMP
In practice, estimating this score for each sample is computationally intractable, thus, approxima- tions need to be made. Current approximations, however, still require training thousands of models over training set to acquire the score of each sample. Thankfully. the authors of the original pa- per provide precomputed C-Scores for MNIST, CIFAR- and CIFAR-100. For other datasets, in contrast, we need more efficient proxies. In particular, in the same work, the authors propose to approximate the C-Score based on the learning speed of samples. However, this still requires registering the percentage of iterations where sample has been correctly classified.
To test the importance of our hypothesis, we devise two different methods where we populate mem- ory based on learning consistency. The first one is High C-Score (High-C), which samples only the top N most consistent elements of each class. The second one is Low C-Score (Low-C), which, contrary to High-C, selects only elements with the lowest C-Score. This division helps us verify that selecting easier (High-C) or difficulttto-lear (Low-C) elements can strongly impact the sequence's final accuracy.
2.1 EXPERIMENTAL SETUP
To provide an empirical validation to our hypothesis, we focus on a Class-Incremental setting, as has been the main focus of recent Continual Learning scientific endeavors. Such scenario is much more challenging and more realistic than the traditional Task Incremental setting (Van de Ven & Tolias 2.19 In Class Incremental scenarios, each task t consists of a new data distribution D' (X". where X denotes the input instances and Y denotes the instance labels. The goal is to train classification model X - Y using data from a sequence of tasks: D = {D1, D") Each task is presented sequentially to the model and trained for E epochs. Crucially, unlike the Task Incremental setting, a task descriptor is only available during training.
2.1.1 DATASETS
We train our models on different sequence of CIFAR-10 and CIFAR-100 (Krizhevsky & Hinton 2009) splits in 5 tasks, and TinyImageNet (Le & Yang 2015) split in 10 and 20 tasks. CIFARIO and CIFAR-100 data-sets traditionally used in Continual Learning and have the advantage of
3

---

Under review as a conference paper at ICLR 2023
Table I: Accuracy for models trained with different memory population methods for and 10 training epochs. Selecting only from the most consistent samples improves accuracy when we have limited budget (1 epoch). However, as we increase the number of epochs, we can see that Mean Features achieves better results. This behavior holds for different memory sizes and architectures.
CIFAR100-500
CIFAR100-1000
CIFARI00-500-RS
E
5
10
5
10
5
10
Reservoir
11.0%
14.9%
15.6%
11.1%
18.3%
19.3%
15.4%
22.8%
25.5%
Class Bal
11.5%
14.9%
15.9%
11.5%
18.2%
18.7%
15.7%
23.1%
26.0%
Task Bal
11.0%
14.9%
15.6%
11.5%
18.4%
19.3%
15.0%
22.9%
25.5%
MF
12.2%
16.4%
17.7%
11.7%
20.9%
20.6%
16.28%
25.5%
28.0%
Low-C 6.9%
9.9%
11.3%
7.0%
9.7%
11.4%
9.0%
14.0%
16.2%
High-C 14.5%
17.7%
18.6%
15.2%
19.9%
20.8%
18.2%
25.3%
27.3%
having available precomputed C-Scores. On the other hand, for TinyImageNet, compute the C- Score using the approximation proposed in (Jiang et al. (021). These datasets also provide different distributions of C-Scores, with CIFAR-I0 and Tiny Imageett having highly skewed distributions, while CIFAR- 100 shows much more uniform distribution of C-Scores, as shown in Figure in the Appendix.
2.1.2 IMPLEMENTATION DETAILS
with 3 different seeds, each inducing different ordering of sequences. In the case of CIFAR-10 and CIFAR-100, we use a simple convolutional architecture proposed in Mirzadeh et al. (2022). To better understand the behavior of our proposal, also use reduced Resnet- 18 (RS) Rebuth et al. 2017) for CIFAR-10M 100. This last model is also used for
All experiments
experiments. The optimizer is SGD with a learning rate of 0.001, momentum 0.9, and batch size 32, unless otherwise mentioned. All methods are trained using Avalanche (Lomonaco et al. (021), and the proposed methods' plugin will be released and integrated in the library upon acceptance. proposed by Lopez-Paz & Ranzato (2017). the sequential learning (Acc), and the forgetting (For) measured by how much performance lost on previous tasks after sequential learning. Equation 2 shows the formulas for the metrics, where As, is the accuracy of task after training task j.
the average performance over the T tasks after
1 T
Acc
Σ
Acora
For T-1
Accr,i Acci,
2.2 RESULTS
Since populating the memory with only the most consistent elements, expect accuracy to increase when the number of epochs is limited. Looking at the results Table [] we can indeed confirm this overall trend. These results occur not only with different memory sizes and models in CIFAR-100, as indicated in the columns of Table I but also in CIFAR-10, as shown in Figure [ Despite highly increasing the memory size in CIFAR-10, the effect on accuracy still considerable when we train for a single epoch. Focusing on Figure [b] the primary reason for the increase in accuracy is the mitigation of forgetting when populating with the High-C strategy. On the other hand, we see that the opposite effect happens when we populating the memory with only the least consistent samples. These results are expected, as these are the most difficult elements learn.
We hypothesize that highly consistent samples are easily classified because they represent common patterns in the dataset. Thus, training with those samples rapidly reinforces learning patterns that can be applied to broader set of samples from the same dataset, increasing the learning speed. On the other hand, highly inconsistent samples would depend on patterns that are specific to fewer samples outliers. While High-C samples common patterns, it still represents a limited spectrum of total dataset patterns, which limits its ability to remember past experiences. We find supporting evidence for this hypothesis in Figure 1c In it, that increasing the number of epochs used
4

---

Under review as a conference paper at ICLR 2023
2003
2000
4033
10
6003
20
Forgettigg
Mean Features
Low-C
High-C
Moan Feetures
Low-C
High-C
Mean Features
Low-C
High-C
(a) Accuracy Memory Size
(b) Forgetting Memory Size
Accuracy Training Epochs
Figure 1: Different metrics for Mean of Features, Low-C, and High-C methods when trained on CIFAR-10. 10. Figure (a) shows that when increasing sample diversity by increasing the memory size, one can observe that populating with High-C values can improve performance when training for epoch. These results are explained because High-C can better mitigate forgetting. On the other hand, by keeping the memory size at 2000 but changing the number of epochs, the performance of High-C almost does not change, showing the limitation of the method.
Selecting Memory
C Score Distribution
Lowic
High-s
CAWS 0.7
Figure 2: Difference in how the evaluated methods select samples from different sections of the C-Score distribution for memory of size 3. Low-c only selects elements with the lowest C-Score. High-c selects only those with the highest C-Score. On the other hand, CAWS samples from sub- group of elements with C-Scores higher than given threshold, selecting randomly from this group.
in each task does not result in better performance when training on highly consistent samples as in other methods. The low diversity from selecting only the highest consistency elements negatively affects the performance since the memory cannot fully represent previous distributions.
DIVERSITY
As noted in the previous section, by only selecting the most consistent elements, the model can learn effectively when given a few epochs per task. However, it can suffer from a lack of diversity when training for more epochs. To avoid this problem, and part of our current hypothesis, we want to increase the diversity of the samples in memory.
With previous results, we had shown that selecting samples that easy learn help during training. At the same time, we learn to avoid less consistent elements since these tend to represent outliers of each task. Given these insights, propose Consistency Aware Sampling (CAWS), which populates the memory by randomly selecting from the top X % of C-Score samples. We call this percentage the Sampling Ratio of CAWS.
Both the High-C and Low-C scenarios emphasize learning consistency without diversity as they only have access a smaller range of input values. On the other hand, CAWS gets to choose from a pool of high learning consistency samples over a wider spread of the training distribution. In Figure 2 there is visual explanation of how CAWS populate the memory. the Appendix (Algorithm) an algorithm explaining the procedure.
3.1 RESULTS
One of the problems we had with selecting only the most consistent elements was that the model quickly forgot when increased the number of epochs. Unlike High-C, CAWS can better repre-

---

Under review as a conference paper at ICLR 2023
6.35
Mean Features CAWS
0.20
Accuracy
Accuracy
Accuracy
0.25
0.30
0.36
0:34
Mean Fediures High C.coee
0.20
Mean Fediures
High Ccore
CANS
1
2
20
20
Epochs
Epochs
Epochs
(a) CIFAR-IO 100
(b) CIFAR-100 500
(c) CIFAR- 100 500 Resnet
Figure 3: Model accuracy for different values of training epochs per task. Models trained on CIFAR- 10 and CIFAR-100 100 with memory size of 100 and 500, respectively. Only selecting samples with high Ccore does not work as well as other methods. However, when mixed with random selection, as with CAWS, it equals or outperforms the baselines.
Accuracy
40
25
500
30
1004
5006
6.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
6.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
Threshold
Threshold
(a) CIFAR-10- 10 Epochs
(b) CIFAR-100 10 Epochs
Figure 4: Accuracy for models trained using CAWS for different sampling ratios. Accuracy behaves as an inverted U-shape between two extremes. When the sampling ratio is 0.0, behavior is the same as randomly selecting elements with maximal diversity. When close 1.0 we are choosing a small selection of the training data that is easy to learn but with little diversity. In between, there is an optimal point where diversity and rapid learning balance each other.
sent previous tasks in memory, achieving better accuracy and less forgetting in different scenarios. Figures 3b and 3c indicate that CAWS can add better representation of previous experiences to memory, achieve even better accuracy than MF when training for multiple epochs.
The compactness of some datasets' C-Score distribution makes it difficult for High-C C. to represent them with only the most consistent elements. This effect does not occur as drastically in datasets where the distribution the C-score is more uniform. For scenarios where High-C inefficient, CAWS performs similarly to the previous methods when adding diversity, shown in Figure 3a
3.2 UNDERSTANDING THE IMPORTANCE OF SAMPLING RATIO
Figure 4 shows how altering the sampling ratio affects the performance of CAWS. The performance moves in an inverted U-Shape. When close to 0.0 it is the same as randomly selecting memory elements. Ideally, this where our memory may have access samples with the greatest diversity. However, this diversity comes with the price of being costly to learn in both compute and sample efficiency. When we move closer to 1.0, get efficient samples but lack enough diversity model the decision boundaries of the classifier accurately. We observe that sampling ratio between 0.6 and 0.8 produces the best results in different datasets, where balance of sample diversity and learning speed is attained.
One can observe the relationship between memory size and the behavior Sampling Ratio to study further the diversity obtained using CAWS. Figures 4 shows that when the memory capacity in- creases, the optimal value tends to decrease, which agrees with our CAWS diversity hypothesis. When the memory capacity is low, it is preferable to have easy-to-learn examples that can correctly represent a small group of data from the previous tasks. On the other hand, when memory grows, a lower Sampling Ratio helps increase diversity, improving the representativeness of multiple patterns. These results imply that if memory limited, is better to have less variety but well-represented patterns than to have poorly represented patterns with high diversity.

---

Under review as a conference paper at ICLR 2023
Score: 0.50
Model
Score: 0.83
Figure 5: To get the proposed proxies for the consisteney score, first (1) we must obtain feature vector for each sample of the dataset. These vectors come from pre-trained model or the current model depending on the proxy used. Then, for each sample, the N nearest neighbors (2) are retrieved to count how many of those belong to the same sample class (3).
3.3 LIMITATIONS OF USING C-SCORES
Though CAWS outperforms other memory population methods, understand that it requires hav- ing access to C-Scores, which are highly costly to compute. Moreover, they require training models on the same training distribution one hopes to learn. This assumption does not hold the Contin- ual Learning scenario. Thus, we propose few proxies for the C-Score, which perform similarly without any of its current limitations. The following section describes how they work.
C-SCORE PROXIES
To develop proxies for the C-Score, we base ourselves on the Critical Sample Ratio (CSR) from Arplt et al. (2017) and the relative local-densit scores proposed in Jiang al. (2021). These metrics given sample with neighboring samples in feature space. relate fast learning with the relation surfaces have greater mixture of samples with different The idea is that harder-to-leam decision labels than easier ones. They find that pairwise distance proxies work in feature space after having model trained the training distribution. These findings, of course, when using pre-trained model, one might get reasonable estimate of the C-Score without training on the training data. Thus, our proxy consists of counting the ratio of neighbors of given sample that are from the same class:
impractical. However,
1y yil
Where {xx1, 31). (I2, .2,, (Ir, yNv) are the closest neighbours of sample (ar, y) in the pre- trained model's feature space. We use cosine distance our distance metric and test using different values of N. Empirically we found that between and 200 shows similar results, so we decided to use value of N equal to 100.
We test progressively easier to calculate versions of the proxy in our experiments:
Proxy 1: using data of the complete sequence, we use pre-trained model to calculate the embedding of each sample. Then we calculate neighbors. This proxy is still impractical for the Continual Learning Scenario but can be used strong baseline.
Proxy 2: using data only from the current task and using pre-trained model, we calculate the neighbors of the samples of the current task.
Proxy 3: we use data only from the current task to calculate neighbours but use the current model's embeddings.
For simplicity, the pre-trained model used a ResNet-18 18 with the weights obtained from PyTorch, but even models pre-trained in self-supervised way can be used.

---

Under review as a conference paper at ICLR 2023
DMGM
MY
Accunng
12
Epochs
Epochs
Epochs
(a) CIFARIO Memory: 50
CIFARIO0 Memory: 500
Tinylmagenet Memory: 1000
Figure 6: Accuracy for models trained with CAWS using increasingly more relaxed versions of proxy C-Scores compared against the baseline Mean Features method for different number of train- ing epochs. As can be seen, CAWS using any proxy outperforms the baseline significantly. Perfor- mance differences between proxies are far less significant suggesting they can indeed practically used for Continual Learning Scenarios. NP: No Proxy.
60
80
CANS 73
CANE 72
CANS
CANS
CANS
CAWS r3
Forgeting
60
20
Epochs
Epochs
Epochs
(a) CIFARIO Memory: 50
(b) CIFAR100 Memory: 500
(c) Imagentee Memory: 1000
Figure 7: Similar to Figure 6 but with Forgetting. most cases, forgetting decreases when we approximate the C-Score. The results agree with section 1, where forgetting is much lower in our proposal than MF, when training for fewer epochs. Although the gap narrowed as we increased the epochs, Forgetting is still lower in CAWS than in MF.
4.1 RESULTS
We compare the Mean Features baseline with the different proxies of CAWS in Figure The first thing that stands out is that proxies continue outperforming MF. By utilizing our approximations, we achieved value that can be compared the consistency of each task's samples. However, direct comparison between the consistency values is not possible since they represent different ways of calculating consistency. These results also suggest that CAWS can be used in practical scenarios for Continual Learning, both Proxies 2 and 3's requirements are easy to meet in most applications.
Sometimes, proxy- based CAWS achieves better results than using the ground truth C-Scores. Two complementary reasons can explain this. The first is that these approaches can help improve mem- ory diversity without weakening the previous tasks representativeness.. Second, these approaches achieve a better consistency score than the C-Score. It is beyond the scope of this paper to verify this hypothesis, as more studies are needed, focusing on different scenarios, not only CL. Nonethe- less, we consider it an interesting topic for future research.
Based on Figure we observe that CAWS with the ground truth C-Score less effective than other simpler proxy methods in the case of Tiny ImageNet. In fact, its performance is even lower than MF. As previously noted, Tiny Imagenet's C-Score is approximated using an approximation proposed in the original paper (Jiang et al. 202T), which we suspect may not be strong its ground truth C-Score.
RELATED WORK
Memory-bast based methods mitigate CF by inserting data from previous tasks into the training process of the current one (Ebrahimi al. 2021 Buzzega et al. 2021). These approaches either use raw samples(Rebuffi et al. 2017] Chaudhry et al. 2019), minimize gradient interference (Lopez-Paz & Chaudhry et al. 2018b) train generative models such GANs autoencoders
Ranzato 2017
8

---

Under review as a conference paper at ICLR 2023
Lesort al. 2019 Shin et al. 2017 Kemker & Kanan 2018 generate elements from previously-
seen distributions.
Multiples approaches has been propose to populate the memory. One simple but efficient method is the Reservoir strategy (Vitter 1985). which randomly select the elements that go into the memory buffer. Other strategies have been proposed by adding different metrics to populate the memory with more representative elements (Chaudhry et al., 2019; Hayes et al.] 2020 Hayes & Kanan, 2021 Aljunndi et al., 2019b). Other works have focused in measuring the impact of hyperparameters on certain methods (Melinn et al. (022). or studied the effect that rehearsal methods have on the loss functions /Verwimp al., 2021). A different line of work has focused on how to select elements from the memory, either by how much the loss of an element is affected (Aljundi et al., (2019a) or by a ranking based on the importance of preserving prior knowledge (lsele Cosgun 2018).
Yet, in spite of the popularity of memory- based methods, little has been studied about the impact of memory composition on Continual Learning (Tiwari et al. 2022). Some proposals along this line are based applying reservoir strategies (Chrysakis & Moens, 2020), while others have proposed to use entropy-based functions to increase memory Wiveesitt(Wiews. & Yang 2021 Sun et al. 2021). I). Others have increased diversity by minimizing the angles of the gradients between different elements (Aljundi et al., 2019b). Despite improving performance in certain scenarios, few studies have been done targeting how to improve memory representativeness.
Other definitions of consistency have also been used in Continual Learning: Bhat et al. (2022a) and Bhat et al. (2022b). In the first one, the author proposes to add regularization term that minimizes the Lp norm between representations of a pair of samples. In the second, the authors proposes a self-supervised learning strategy consolidate the knowledge of different tasks. However, both definitions of learning consistency differ strongly from the one used in this work.
5.1 LEARNING CONSISTENCY
Learning Consistency and the C-Score (Jiang et al. 2021) come from a line of work analyzing deep neural network training dynamics. One landmark study (Zhang et al. (2019) showed that deep neural networks had the capacity to learn even random noise. Later studies (Arplt al. 2017), showed that natural images were learned faster than noise. Others analyzed how examples are forgotten during training (Toneva et al. 2019). Other metrics have been proposed for measuring learning dynamics such as model confidence, learning speed, holdout retraining and ensemble agreement (Carlini et al.) 2019) which correlate well with each other. Learning speed in particular has been shown to correlate well with C-Score. Finally, a recent alternative for understanding per sample difficulty from the model's perspective is measure the prediction depth in which a sample correctly predicted at (Baldock et al. (021).
CONCLUSIONS AND FUTURE WORK
In this work, we have analyzed how a memory population criterion based on learning consistency affects Experience Replay. We find that using only the most consistent samples in the memory is useful solely when having a limited compute budget. Otherwise, using Means of Features to populate the memory remains a strong baseline. However, selecting elements randomly from a set of the most consistent elements procedure we named Consistency AWare Sampling (CAWS) does help and outperforms all baselines. This method relies on an accurate estimation of a measure called C-Score that is non trivial to calculate and would usually be impractical for Continual Learning scenarios. We propose proxies based on previous works that are easy to calculate and show little deviation from using ground truth C-Scores and still outperform our presented baselines. These proxies are not only useful for Continual Learning but can also be used for creating C-Score proxies for Curriculum Learning. In future work, would like to find ways to determine priori what is the right sampling ratio for a given task and how it relates to the C-Score distribution of a given dataset.

---

Under review as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
We will make our code public upon acceptance, along with examples of how to run the different experiments.
REFERENCES
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. Advances in neural information processing systems, 32, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in neural information processing systems, 32, 2019b.
Vladimir Araujo, Helena Balabin, Julio Hurtado, Alvaro Soto, and Marie-Francine Moens. How relevant is selective memory population in lifelong language learning? In Proceedings of the 2nd Conference of the Asia- Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing, Online, November 2022. Association for Computational Linguistics.
Devansh Arplt, Stanislaw Jastrzebskl, Nicolas Bailas, David Krueger, Emmanuel Bengio, Maxin- der S. Kanwal, Tegan Maharaj, Asja Fischer, Aaron Courville, Yoshua Benglo, and Simon Lacoste- A closer look at memorization in deep networks. 34th International Conference Machine Learning, ICML 20/7, 1:350-359, 2017.
Robert John Nicholas Baldock, Hartmut Maennel, and Behnam Neyshabur. Deep learning through the lens of example difficulty. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL https: //openreview.co www.::/wweeeootttttoee Www..aLI11...EEE111111111
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference on Machine Learning, ICML '09, pp. 41 41-48, New York, NY, USA, 2009. Association for Computing Machinery. ISBN 9781605585161. doi: 155444//15533744553888 URL pttps:///ci.ore 1145/
1553374 1553380
Prashant Bhat, Bahram Zonooz, and Elahe Arani. Consistency is the key to further mitigating catastrophic forgetting in continual learning. arXiv preprint Sttiv:2200.04999: 2022a.
Prashant Bhat, Bahram Zonooz, and Elahe Arani. Task agnostic representation consolidation: self-supervised based continual learning approach. arXiv preprint arXiv:2207.06667, 2022b.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, and Simone Calderara. Rethinking experience replay: bag of tricks for continual learning. In 2020 25th International Conference on Pattern Recognition (ICPR), pp. 2180-2187. IEEE, 2021.
Nicholas Carlini, Úlfar Erlingsson, and Nicolas Papernot. Distribution density, tails, and outliers in machine learning: Metrics and applications. arXiv, (1), 2019.
Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. In Proceedings the European Conference on Computer Vision (ECCV), pp. 532-547, 2018a.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In International Conference Learning Representations, 2018b.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Tom, and M Ranzato. Continual learning with tiny episodic memories. Inter- national Conference Machine Learning, 2019.
10

---

Under review as a conference paper at ICLR 2023
Aristotelis Chrysakis and Marie-Framn Fraiiine Moens. Online continual learning from imbalanced data. In International Conference on Machine Learning, pp. 1952-1961. PMLR, 2020.
Matthias Delange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: /81.:0v810000455,, 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x 16 words: Transformers for image recognition at scale. International Confer- on Learning Representations, 2020.
Sayna Ebrahimi, Suzanne Petryk, Akash Gokul, William Gan, Joseph E. Gonzalez, Marcus Rohrbach, and trevor darrell. Remembering for the right reasons: Explanations reduce catas- trophic forgetting. In International Conference on Learning Representations, 2021. URL
net
Tyler L Hayes and Christopher Kanan. Selective replay enhances learning in online continual ana- logical reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3502-3512, 2021.
Tyler L Hayes, Kushal Kafle, Robik Shrestha, Manoj Acharya, and Christopher Kanan. Remind your neural network to prevent catastrophic forgetting. In European Conference on Computer Vision, pp. 466 483. Springer, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Julio Hurtado, Alain Raymond, and Alvaro Soto. Optimizing reusable knowledge for continual learning via metalearning. Advances in Neural Information Processing Systems, 34, 2021.
David Isele and Akansel Cosgun. Selective experience replay for lifelong learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Ziheng Jiang, Chiyuan Zhang, Kunal Talwar, and Michael C Mozer. Characterizing structural reg- ularities of labeled data in overparameterized models. In International Conference Machine Learning, pp. 5034-5044. PMLR, 2021.
Ronald Kemker and Christopher Kanan. Fearnet: Brain-inspired model for incremental learning. In International Conference on Learning Representations, 2018.
Chris Dongjoo Kim, Jinseo Jeong, and Gunhee Kim. Imbalanced continual learning with partition- ing reservoir sampling. European Conference on Computer Vision, pp. 41 411-428. Springer, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barkkiskaaaiikkaa..a. al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. http://wwwwcssttromtooddde kriz/eifar.hml, 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7)33 2015.
Timothée Lesort, Hugo Caselles-Dupré, Michael Garcia-Ortiz, Andrei Stoian, and David Filliat. Generative models from the perspective of continual learning. In International Joint Conference Newral Networks. IEFE, 2019.
11

---

Under review as a conference paper at ICLR 2023
Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, Antonio Carta, Gabriele Graffieti, Tyler L. Hayes, Matthias De Lange, Mare Masana, Jary Pomponi, Gido van de Ven, Martin Mundt, Qi She, Keiland Cooper, Jeremy Forest, Eden Belouadah, Simone Calderara, German I. Parisi, Fabio Cuzzolin, Andreas Tolias, Simone Scardapane, Luca Antiga, Subutai Amhad, Adrian Popeseu, Christopher Kanan, Joost van de Weijer, Tinne Tuytelaars, Davide Bacciu, and Davide Maltoni. Avalanche: an end-to-end library for continual learning. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, 2nd Continual Learning Computer Vision Work- shop, 2021.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in neural information processing systems, 2017.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting single network multi- ple tasks by learning to mask weights. In Proceedings of the European Conference on Computer Vision, 2018
Gabriele Merlin, Vincenzo Lomonaco, Andrea Cossu, Antonio Carta, and Davide Bacciu. Practical recommendations for replay- replay-based continual learning methods. arXiv preprint arXiv:2203. 10317, 2022.
Seyed Iman Mirzadeh, Arslan Chaudhry, Dong Yin, Timothy Nguyen, Razvan Pascanu, Dilan Gorur, and Mehrdad Farajtabar. Architecture matters in continual learning. arXiv preprint SXXiv22002.002755 2022.
German Parisi, Ronald Kemker, Jose Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 1 113:54-71, 2019.
Jathushan Rajasegaran, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Mubarak Shah. itaml: An incremental task-agnostic meta-learning approach. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, 2020.
Sylvestre-Alvee Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Laamper. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2017.
Andrei A Rusu, Neil C Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. arXiv preprint crXiv:1606.04677 2016.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in neural information processing systems, 2017.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittweeser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game go with deep neural networks and tree search. nature, 529(7587):484 4899, 2016.
Shengyang Sun, Daniele Calandriello, Huiyi Hu, Ang Li, and Michalis Titsias. Information- theoretic online memory selection for continual learning. In International Conference on Learn- ing Representations, 2021.
Rishabh Tiwari, Krishnateja Killamsetty, Rishabh Iyer, and Pradeep Shenoy. Ger: Gradient coreset based replay buffer selection for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 99-108, 2022.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey Gordon. An empirical study of example forgetting during deep neural network learning. In ICLR, 2019.
Gido M Van de Ven and Andreas S Tolias. Three scenarios for continual learning. arXiv preprint arXiv: /004:09940077.. 2019.
Eli Verwimp. Matthias De Lange, and Tinne Tuytelaars. Rehearsal revealed: The limits and mer- its of revisiting samples in continual learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9385-9394, 2021.
12

---

Under review as a conference paper at ICLR 2023
Oriol Vinyals, Igor Babuschkin, Wojciech M Czamecki, Michaël Mathieu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft ii using multi-agent reinforcement learning, Nature, 575(7782):350-3554 2019.
Jeffrey S. Vitter. Random sampling with a reservoir. ACM Trans. Math. Sofiw,, 11(1):37-57, 1985.
Felix Wiewel and Bin Yang. Entropy-based sample selection for online continual learning. In 2020
28th European Signal Processing Conference (EUSIPCO), pp. 1477-1481. IEEE, 2021.
Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? In International Conference on Learning Representations, 2020.
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. In Proceedingy of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 374-382, 2019.
Chiyuan Zhang, Benjamin Recht, Samy Bengio, Moritz Hardt, and Oriol Vinyals. Understanding deep learning requires rethinking generalization. 5th International Conference on Learning Representations, ICLR 2017 Conference Track Proceedings, 2019.
13

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1 C-SCORE DISTRIBUTIONS
The distribution that follows the C-Score for the dataset used the experiments can be found in Figure 8 As mentioned in the main text, each dataset follows different distribution, CIFAR-10 and Tiny Imagenet are more skewed than -OFAR- This distribution affects the results obtained by the methods proposed in the paper.
2000
200
203
50
(a) 10
(b) CIFAR-100 100
(c) Tiny Imagenet
Figure 8: C-Coore distribution for MNIST, CIFAR- 10 and CIFAR- We can see clear difference between the different datasets, where the sets with the most data per class tend to cluster the C-Scores in the upper part, indicating that more examples are easy to train. On the other hand, CIFAR-100 shows a more uniform distribution.
A.2 ALGORITHMS
The Algorithm 1 shows the details of CAWS. The only difference when applying the proxies, is that the Original C-Score is changed with the consistency approximation.
Algorithm 1: CAWS
Components:
Dí: Dataset for task f.
M: Memory.
No: of elements in memory of class c.
N of elements to add per class.
5: C-Score threshold.
C: C-Score
for classes in M do
M + remove N - No elements
end
for classes in D' do
x Sample N2 from Dj where C(x) 2 8 M.odd(x)
end
Output: Populated memory M.
A.3 TABLE WITH RESULTS
Below you can see the details of all the results obtained. These the same ones used generate the figures in the main text.
14

---

Under review as a conference paper at ICLR 2023
Table 2: CIFARIO Split 5 Memory size 50 - Model simple cnn
Epochs
2
10
Reservoir
26.3%
25.6%
22.6%
23.7%
22.1%
22.9%
Class Balance
26.7%
24.8%
24.1%
23.9%
22.4%
23.1%
Task Balance
25.8%
24.5%
23.0%
23.2%
22.2%
23.1%
Mean Features
29.4%
27.0%
25.1%
23.9%
25.2%
24.7%
Proxy
C-score Lower
20.5%
19.3%
19.7%
18.8%
19.4%
19.4%
Proxy
C-score upper
28.4%
28.8%
27.9%
27.0%
26.3%
25.9%
Proxy
CAWS
30.0%
29.2%
28.5%
29.1%
28.3%
27.2%
Proxy
C-score Lower
19.9%
19.7%
19.8%
19.6%
19.6%
20.0%
Proxy
C-score upper
28.3%
27.9%
27.2%
26.1%
26.3%
25.4%
Proxy
CAWS
28.6%
28.1%
27.8%
25.9%
26.0%
25.2%
Proxy
C-score Lower
18.6%
19.1%
19.0%
18.6%
18.8%
18.4%
Proxy
C-score upper
28.3%
26.8%
27.0%
25.8%
25.0%
25.3%
Proxy 3
CAWS
29.6%
28.6%
26.4%
25.8%
25.4%
24.7%
No Proxy
CAWS
29.5%
27.9%
26.0%
25.7%
25.1%
24.9%
Table CIFARIO Split Memory size 100
Model simple cnn
3
4
5
10
Reservoir 30.44%
32.23%
28.26%
28.38%
27.88%
26.85%
Class Balance
31.80%
31.40%
29.61%
28.11%
27.60%
28.95%
Task Balance
30.45%
32.62%
29.41%
28.18%
28.22%
26.89%
Mean Features
34.09%
34.92%
32.70%
31.31%
31.06%
29.42%
Proxy
C-score Lower
22.99%
23.12%
22.89%
21.78%
21.98%
20.64%
Proxy
C-score upper
31.23%
32.98%
31.69%
30.96%
30.28%
29.14%
Proxy
CAWS
33.62%
34.73%
34.09%
33.68%
33.43%
31.80%
No Proxy
CAWS
32.45%
34.22%
31.46%
31.52%
31.75%
30.01% %
Table 4: CIFARIO Split Memory size 250
Model simple cnn
4
5
10
Reservoir 34.80%
39.99%
40.32%
39.30%
38.82%
35.00%
Class Balance
33.39%
40.30%
40.00%
38.61%
38.51%
36.91%
Task Balance
34.56%
38.48%
39.66%
38.73%
38.79%
35.47%
Mean Features
35.35%
42.43%
43.34%
41.92%
41.97%
37.95%
Proxy
C-score Lower
24.87%
30.34%
31.04%
29.26%
29.09%
27.48%
Proxy
C-score upper
36.25%
38.93%
40.22%
40.00%
40.17%
38.31%
Proxy
CAWS
37.51%
41.61%
42.16%
42.89%
43.54%
40.46%
No Proxy
CAWS
37.60%
41.31%
42.11%
41.77%
40.94%
39.71%
Table 5: CIFARIO Split 5 Memory size 500
Model simple cnn
# Epochs
2
10
Reservoir 35.0%
42.2%
44.7%
46.5%
46.6%
46.2%
Class Balance
35.5%
41.7%
45.2%
46.1%
47.7%
46.1%
Task Balance
35.2%
42.6%
44.8%
46.4%
47.3%
46.2%
Mean Features
35.8%
43.0%
46.8%
48.5%
48.9%
50.2%
Proxy
C-score Lower
25.0%
31.2%
34.5%
35.0%
34.0%
32.7%
Proxy
C-score upper
39.0%
42.3%
45.1%
46.7%
47.4%
46.5%
Proxy
CAWS
39.6%
44.3%
47.1%
48.6%
49.0%
49.6%
Proxy
C-score Lower
21.7%
25.1%
27.3%
28.8%
29.1%
29.3%
Proxy
C-score upper
37.8%
42.5%
45.0%
46.2%
47.9%
47.3%
Proxy
CAWS
37.9%
42.7%
46.2%
48.0%
50.0%
49.5%
Proxy
C-score Lower
19.7%
21.5%
21.8%
22.5%
23.0%
23.3%
Proxy
C-score upper
37.7%
41.4%
44.7%
45.0%
45.9%
47.3%
Proxy
CAWS
38.4%
43.2%
46.2%
47.4%
48.4%
48.9%
No Proxy
CAWS
39.0%
44.1%
46.9%
48.7%
49.7%
49.8%
15

---

Under review as a conference paper at ICLR 2023
Table CIFARI00 Split
Memory size 500
Model simple enn
# Epochs
2
3
5
10
Reservoir
11.0%
13.3%
14.7%
14.8%
14.9%
15.6%
Class Balance
11.5%
13.2%
14.8%
14.5%
14.9%
15.9%
Task Balance
11.0%
13.3%
14.4%
14.8%
14.9%
15.6%
Mean Features
12.2%
15.3%
15.5%
16.5%
16.4%
17.7%
Proxy
C-score Lower
8.7%
10.5%
11.1%
12.0%
11.9%
13.0%
Proxy
C-score upper
14.2%
15.9%
17.1%
17.6%
17.8%
18.7%
Proxy
CAWS
14.5%
16.3%
17.5%
17.8%
18.2%
19.0%
Proxy2
C-score Lower
8.1%
10.1%
10.6%
11.2%
11.6%
12.6%
Proxy
C-score upper
13.7%
16.1%
17.1%
17.4%
18.2%
18.7%
Proxy
CAWS
13.9%
16.5%
17.4%
17.9%
18.2%
18.9%
Proxy
C-score Lower
9.1%
10.5%
11.2%
11.6%
11.9%
13.0%
Proxy
C-score upper
13.5%
15.6%
16.6%
17.3%
18.0%
19.0%
Proxy
CAWS
13.7%
16.0%
17.4%
17.7%
18.2%
19.1%
No Proxy
CAWS
14.7%
17.3%
18.1%
18.2%
19.0%
19.8%
Table 7: CIFARI00 S Memory size 1000
Model simple enn
Epochs
2
3
10
Reservoir
11.1%
15.5%
17.8%
18.3%
18.7%
19.3%
Class Balance
11.5%
15.3%
17.9%
18.2%
18.3%
18.7%
Task Balance
11.5%
15.2%
17.5%
18.4%
18.5%
19.3%
Mean Features
11.7%
16.9%
19.6%
20.9%
21.1%
20.6%
Proxy
C-score Lower
8.7%
11.6%
13.1%
13.5%
14.4%
14.5%
Proxy
C-score upper
15.4%
18.2%
19.8%
21.2%
21.5%
21.8%
Proxy
CAWS
15.5%
18.6%
20.6%
21.8%
21.9%
22.4%
Proxy
C-score Lower
7.6%
10.7%
11.8%
12.6%
13.0%
13.3%
Proxy
C-score upper
14.2%
17.8%
19.7%
21.0%
21.1%
22.1%
Proxy
CAWS
14.5%
18.4%
20.2%
21.2%
21.7%
22.7%
Proxy
C-score Lower
8.8%
11.5%
12.9%
13.3%
13.8%
14.3%
Proxy
C-score upper
14.4%
17.2%
18.6%
19.8%
20.7%
22.1%
Proxy
CAWS
14.3%
17.5%
19.8%
21.1%
21.0%
22.4%
No Proxy
CAWS
15.4%
19.5%
20.8%
22.4%
22.4%
22.8%
Table 8: CIFARI00 Split 5
Memory size 500
Model Resnet
Epochs 15.4%
2
10
Reservoir
19.1%
21.3%
21.8%
22.8%
25.5%
Class Balance
15.7%
19.3%
21.1%
22.1%
23.1%
26.0%
Task Balance
15.0%
19.2%
21.6%
22.0%
22.9%
25.5%
Mean Features
16.3%
21.0%
22.8%
24.2%
25.5%
28.0%
No Proxy
CAWS
19.8%
23.7%
25.8%
26.9%
27.8%
30.1%
Table CIFARI00 Split 5 Memory size 1000
Model Resnet
# Epochs
2
3
10
Reservoir 15.3%
24.3%
26.2%
28.4%
29.1%
31.7%
Class Balance
17.5%
24.1%
26.5%
27.7%
29.6%
30.8%
Task Balance
15.3%
22.3%
25.6%
27.7%
29.1%
31.3%
Mean Features
17.9%
25.4%
29.3%
30.9%
31.0%
34.3%
No Proxy
CAWS
21.3%
27.8%
31.1%
31.6%
32.6%
35.9%
16

---

Under review as a conference paper at ICLR 2023
Table 10: Tiny Imagenet Split Memory size 1000
Model Resnet
Epochs
5
10
Reservoir
9.2%
12.3%
12.7%
Mean Features
10.3%
14.7%
14.5%
Proxy
C-score Lower
9.1%
12.7%
13.3%
Proxy
C-score upper
9.5%
13.2%
13.3%
Proxy
CAWS
9.8%
13.4%
13.5%
Proxy
C-score Lower
8.8%
12.5%
11.7%
Proxy
C-score upper
10.8%
14.9%
15.8%
Proxy 2
CAWS
11.0%
15.5%
15.7%
Proxy
C-score Lower
9.0%
12.3%
12.0%
Proxy
C-score upper
9.6%
14.5%
15.0%
Proxy CAWS
9.9%
14.9%
15.2%
No Proxy
CAWS
9.8%
13.0%
13.3%
Table 11: Tiny Imagenet- Split 20 Memory size 1000
Model Resnet
# Epochs
10
Reservoir
7.7%
9.7%
8.9%
Mean Features
8.3%
11.9%
11.2%
Proxy
C-score Lower
7.9%
10.3%
9.7%
Proxy
C-score upper
7.4%
9.8%
10.0%
Proxy
CAWS
7.8%
10.3%
9.9%
Proxy
C-score Lower
6.4%
8.2%
8.1%
Proxy
C-score upper
8.6%
11.7%
12.0%
Proxy
CAWS
9.0%
12.5%
12.6%
Proxy
C-score Lower
6.6%
8.6%
9.0%
Proxy
C-score upper
7.7%
11.7%
11.5%
Proxy
CAWS
8.0%
1 11.8%
12.2%
No Proxy
CAWS
7.6%
10.3%
10.0%
Table Tiny Imagenet Split Memory size 2000
Model Resnet
5
10
Reservoir
9.39%
17.23%
17.51%
Class Balance
8.84%
17.82%
17.24%
Task Balance
9.28%
17.89%
17.31%
Mean Features
10.50%
20.25%
19.91%
No Proxy
C-score Lower
9.11%
17.69%
17.58%
No Proxy
C-score upper
8.96%
17.68%
16.24%
No Proxy
CAWS
10.12%
18.29%
17.95%
Table 13: Tiny Imagenet- Split 20 -0mmory size 2000
Model Resnet
5
10
Reservoir
7.12%
14.49%
14.64%
Class Balance
6.19%
15.70%
15.02%
Task Balance
7.01% %
14.79%
13.79%
Mean Features
7.79%
17.79%
16.45%
No Proxy
C-score Lower
7.04%
15.77%
14.92%
No Proxy
C-score upper
7.22%
15.33%
15.15%
No Proxy
CAWS
8.15%
15.43%
15.08%
17


Reviews:
[{"review_id": "yqHmD577aM", "reviewer": "Reviewer_1H7t", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents an idea to select important examples to be buffered in continual learning. The proposed Consistency Aware Sampling (CAWS) considers examples that are easy to learn and representative of previous tasks. Empirical investigation has been conducted to evaluate the effectiveness of the proposed method.", "summary_of_the_review": "Although the topic of selective buffering is interesting, the proposed method is not well justified neither intuitively or empirically. Therefore, I recommend rejection of the paper.", "strength_and_weaknesses": "The main strength of the paper is the topic itself, selective buffering in continual learning.\n\nHowever, the weaknesses of the paper overweigh the strength by a large margin:\n- The organization of the paper is scattered and hard to follow. Interleaving methods and experiments are not clear. Moreover, the writing style makes it hard to distinguish the motivation and original parts of the paper.\n- Saving the \"easy to learn\" examples in the buffer is counter-intuitive. An easy-to-learn example should already been learned by the model in early iterations, why bother saving it in the replay buffer instead of giving the model more exposure to hard examples? \n- Lack of latest competing methods. As far as I am concerned, there has been recent work that studies the same topic as this paper, including but not limited to GSS [1], LARS[2] and DDR [3]. The authors should definitely compare the proposed method against these recent methods.\n\n\n[1] Aljundi, Rahaf, et al. \"Gradient based sample selection for online continual learning.\"NeurIPS 2019.\n\n[2] Buzzega, Pietro, et al. \"Rethinking experience replay: a bag of tricks for continual learning.\" ICPR 2021.\n\n[3] Wang, Zifeng, et al. \"SparCL: Sparse Continual Learning on the Edge.\" NeurIPS 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "X-_69BkmK4", "reviewer": "Reviewer_gaRy", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates continual learning under limited memory setup  and proposes a storage policye CAWS (Consistency AWare Sampling), that leverages a learning consistency score (C-Score) to enable effective and efficient learning. CAWS adds diversity to the memory while allowing the model to learn a far more detailed decision boundary. Practical proxies that require no extra training can still achieve similar results according to some experimental results.", "summary_of_the_review": "Overall, even though the problem is important, I find the structure of the paper needs further modifications. The method CAWS they proposed seems like just sampling from the top C-Score samples under the continual learning framework, I think the authors could put more emphasize on the novelty of this method and how much of the memory size is saved or the performance boost in using CAWS sampling. \nOn top of this, there are a few technical aspects (inconsistent results, hyperparameter thresholds etc) I do not completely agree with, therefore, would request authors to answer above questions for clarity.", "strength_and_weaknesses": "Pros: \nThe paper focused on a very interesting scenario in continual learning. The problem setting of continual learning with efficient learning is well motivated. It is reasonable to apply C-score in evaluating the learnablity of a sample and this method could be extended to other areas including Curriculum Learning.\n\nCons:\n1. The structure of the paper is not clear. The authors proposed to use C-Score to measure how learnable a specific sample can be concerning a set of models, and C-Score of some datasets have already be calculated. They mentioned they needed a proxy to evaluate the score for the other datasets, but the definition of the proxy appear after several sections. The writing order will put extra reading burden for the readers. Also the original paper appears in Section 4.1, while the C-Score definition appears in Section 2.\n\n2. For the datasets we already have the C-Score computed by Jiang 2021, does this C-Score rely on the model architecture? If we switch to a new architecture, does that mean this C-Score does not work any more? \n\n3. There are a few proxies proposed by Jiang 2021, what is the difference between the proxy in this paper and the proxies in Jiang 2021?\n\n4. How to determine the threshold of high C-score and low C-score? How to determine the X in CAWS? \n\n5. The results in Figure 4 a and Figure 4 b are leading to two different directions when we increase the threshold. Could you please explain why the 1000 memory capacities would increase with threshold = 0.9 in the CIFAR100 and all the other curves seens going downwards?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IofZ7ljXGv", "reviewer": "Reviewer_zPPV", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to use C-scores as a score to select representative samples to insert into a memory buffer in continual learning. More concretely the authors:\n- describe current limitations of memory selection methods\n- propose to use C-score, an expensive but accurate score capable to identify learnable samples, to select a memory that is highly representative of the class and can be efficiently learned\n- identify two limitation in a naive use of C-scores and try to alleviate them:\n    - C-scores are expensive to compute, and the authors propose more efficient proxies to approximate them\n    - selecting a memory greedily w.r.t. C-scores can result in lack of diversity, so the author propose to select uniformly at random (promotes diversity) but only out of samples that achieve a C-score above a certain threshold (promotes relevancy) \n\nThe proposed method is evaluated using several ablations and baselines, but only within the context of other memory-based CL methods.", "summary_of_the_review": "Overall the paper presents an interesting approach and is overall acceptable, but the empirical evaluation undermine the message that C-scores are an interesting metric in CL. In particular\n- the quality of the proposed proxies are never truly investigated\n- it is not clear which computational regime the authors are targeting for their approach, and they should justify their choiches for memory size, number of epochs, and size of datasets in each epoch in terms of memory and compute, providing back-of-the-envelope estimations supporting their choice (e.g. if this approach should target compute on a phone, a consumer desktop, an industry server, or a whole cluster).", "strength_and_weaknesses": "The strong points of the method are its conceptual simplicity (C-scores are an intuitive metric of relevance) and good performance compared to other memory-based baselines. The motivation section is especially insightful and helps to highlight how the memory selection problem is still very open and hard in general. The algorithm exposition is clear and its most relevant hyperparameter are ablated. The experimental section covers multiple datasets.\n\nHowever the work also has some clear weaknesses\n- One of the main contributions is the derivation of new proxies for C-scores, but the approximation accuracy (e.g. proxy C-score/actual C-score) of these proxies w.r.t. the real score or Jiang et al.'s approximation. The only metric reported is downstream accuracy of the classification task, but without checking the quality of the C-score approximation it is possible that the proxies are injecting additional biases that could actually be helpful for these specific tasks.\n- The memory budgets considered in the experimental section are quite small (i.e. ~100-1000), and much smaller than what off-the-shelf hardware is capable of handling currently (i.e. 10k and up). It would be interesting to see how the various proxies and the baseline behave with a larger memory, and if the gain are preserved.\n- For the real algorithm (i.e. the one using C-score proxies) only accuracy is reported, while usually the more relevant metrics in CL are forgetting or some sort of transfer.\n- None of the experiments are reporting error bars or uncertainty, and it is not clear if the experiments with proxies have also been run with 3 seeds.\n\nMinor comments:\n- The baselines considered are only memory-based. While this is acceptable (CL is a broad field) it is a bit of a limitation of the comparison.\n- The paper seems to only consider the setting where sample are explicitely labeled with their tasks during training. This is acceptable but it should be stated that it is a special case of the more general CL problem without explicit tasks or task boundaries.\n- Accuracy and forgetting are never defined as metrics in the paper.\n- In A.2 CAWS should include points with C(x) >= delta", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2RRtmcjaNpQ", "reviewer": "Reviewer_Yna5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper explores the idea of addressing catastrophic forgetting through rehearsal using a memory buffer. It studies strategies to select the most important samples for storage in a memory buffer. The core idea is to design a consistency score that ranks samples for storage according to how easy they are to learn and at the same time are representative of previous tasks. Experiments using three common continual learning benchmarks are performed to demonstrate that the proposed method is effective.", "summary_of_the_review": "The idea of experience replay using selected samples is not new and many existing works have been proposed. The proposed idea is a heuristic idea without no theoretical justification. Experiments also do not demonstrate that the proposed method leads to state-of-the-art performance. Hence, the paper does not provide theoretical novelties or sufficient empirical results to demonstrate its effectiveness. In conclusion, it is not ready for publication.", "strength_and_weaknesses": "Strengths:\n\n1. The paper is clear and easy to follow.\n\n2. Experiments on three major continual learning frameworks are provided.\n\nWeaknesses:\n\n1. The idea is incremental.\n\n2. The approach is heuristic and without much theoretical justification.\n\n3. Comparison against prior methods is extremely limited.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject