Class-Incremental Learning with Repetition
Authors: Hamed Hemati
Andrea Cossu
Antonio Carta
Julio Hurtado
Lorenzo Pellegrini
Davide Bacciu
Vincenzo Lomonaco
Damian Borth
Venue: Submitted to ICLR 2023 (2023)
Keywords: continual learning
lifelong learning
class-incremental learning
incremental learning

Abstract: Real-world data streams naturally include the repetition of previous concepts. From a Continual Learning (CL) perspective, repetition is a property of the environment and, unlike replay, cannot be controlled by the user. Nowadays, Class-Incremental scenarios represent the leading test-bed for assessing and comparing CL strategies. This family of scenarios is very easy to use, but it never allows revisiting previously seen classes, thus completely disregarding the role of repetition. We focus on the family of Class-Incremental with Repetition (CIR) scenarios, where repetition is embedded in the definition of the stream. We propose two stochastic scenario generators that produce a wide range of CIR scenarios starting from a single dataset and a few control parameters. We conduct the first comprehensive evaluation of repetition in CL by studying the behavior of existing CL strategies under different CIR scenarios. We then present a novel replay strategy that exploits repetition and counteracts the natural imbalance present in the stream. On both CIFAR100 and TinyImageNet, our strategy outperforms other replay approaches, which are not designed for environments with repetition.

Under review as a conference paper at ICLR 2023
CLASS-INCREMENTA LEARNING WITH REPETITION
Anonymous authors
Paper under double-blind review
ABSTRACT
Real-world data streams naturally include the repetition of previous concepts. From Continual Learning (CL. perspective, repetition is property of the en- vironment and, unlike replay, cannot be controlled by the user. Nowadays, Class- Incremental scenarios represent the leading test-bed for assessing and comparing CL strategies. This family of scenarios is very casy to use, but it never allows revisiting previously seen classes, thus completely disregarding the role of repe- tition. We focus on the family of Class- Incremetal with Repetition (CIR) sce- narios, where repetition is embedded in the definition of the stream. We propose two stochastic scenario generators that produce a wide range of CIR scenarios starting from single dataset and a few control parameters. We conduct the first comprehensive evaluation of repetition in CL by studying the behavior of existing CL strategies under different CIR scenarios. We then present a novel replay strat- egy that exploits repetition and counteracts the natural imbalance present in the stream. On both CIFAR 100 and TinylmageNet, our strategy outperforms other replay approaches, which are designed for environments with repetition.
INTRODUCTION
Continual Learning (CL) requires a model to learn new information from a stream of experiences presented over time, without forgetting previous knowledge (Parisi al. 2019 Lesort al. 2020). The nature and characteristics of the data stream can vary lot depending on the real- world en- vironment and target application. Class- Incremental (CI) scenarios (Rebufh et al.) 2017) are the most popular ones in CL. CI requires the model to solve a classification problem where new classes appear over time. Importantly, when a set of new classes appears, the previous ones are never seen again. However, the model still needs to correctly predict them at test time. Conversely, in Domain-Inccrmmente (DI) scenario (van de Ven & Tolias 2019) the model sees all the classes at the beginning and continue to observe new instances of the classes over time.
The CI and DI scenarios have been very helpful to promote and drive CL research in the last few years. However, they strongly constrain the properties of the data stream in way that it sometimes considered unrealistic or very limiting (Cossu et al., (021). Recently, the idea of Mlass-Incremental with Repetition (CIR) scenarios has started gather some attention in CL (Cossu al.) (021). CIR scenarios are arguably more flcxible in the definition of the stream, since they allow both the introduction of new classes and the repetition of previously seen classes. Crucially, repetition is a property of the environment and cannot be controlled by the CL agent. This is very different from Replay strategies (Hayes et al. (02T), where the repetition of previous concepts is heavily structured and can be tuned at will.
CIR defines a family of CL scenarios which ranges from CI (new classes only, without repetition) to DI (full repetition of all seen classes). Although appealing, currently there exists neither quantita- tive analysis nor an empirical evaluation of CL. strategies learning in CIR scenarios. Mainly, because it is not obvious how to build stream with repetition, given the large amount of variables involved. How to manage repetition over time? How to decide what to repeat? What data should we use? In this paper, provide two generators for CIR that, starting from single dataset, allow to build customized streams by only setting few parameters. The generators are as easy to use as CI or DI ones.
We leveraged our generators to run an extensive empirical evaluation of the behavior of CL strategies in CIR scenarios. We found out that knowledge accumulation happens naturally in streams with

---

Under review as a conference paper at ICLR 2023
repetition. Even naive fine-tuning, subjected to complete forgetting in CI scenarios, is able to accumulate knowledge for classes that are not always present in an experience. We observed that Replay strategies still provide advantage in terms of final accuracy, even though they are not crucial to avoid catastrophic forgetting. On one side, distillation-based strategies like LwF (Li & Hoiem 2018) are competitive in streams with a moderate amount of repetition. On the other side, existing Replay strategies are not specifically designed for CIR streams. We propose a novel Replay approach, called Frequency-Aware Replay (ER-FA) designed for streams with unbalanced repetition (few classes appear rarely, the other very frequently). ER-FA surpasses by a large margin other Replay variants when looking at infrequent classes and it does not lose performance in terms of frequent classes. This leads to moderate gain in the final accuracy, with a much better robustness and reduced variance across all classes. Our main contributions are:
1. The design of two CIR generators, able to create streams with repetition by only setting few control parameters. We built both generators with Avalanche (Lomonaco al. 2021 and we will make them publicly available to foster future research. The generators are general enough to fit any classification dataset and are fully integrated with Avalanche pipeline to run CL. experiments.
2. We perform an extensive evaluation of the properties of CIR streams and the performance of CL strategies. We study knowledge accumulation and we showed that Replay, although still effective, is not crucial for the mitigation of catastrophic forgetting. Some approaches (e.g., LwF) look more promising than others in CIR scenarios. We consolidate our results with analysis of the CL models over time through Centered Kernel Alignment (CKA) (Kornblith et al.) 2019) and weights analysis.
3. We propose a novel Replay variant, ER-FA, which is designed based the properties of CIR scenarios. ER-FA surpasses other Replay strategies unbalanced streams and provide more robust performance on infrequent classes without losing accuracy on the frequent ones.
2 CLASS-INCREMENTAL LEARNING WITH REPETITION GENERATORS
Concepts:
(6A
CL requires a model to learn from a stream of N expe- riences S = {e1, e2, ]. where each experience brings a dataset of examples De: = {X,, Y. Many CL scenarios, like CI or DI, are generated from fixed dataset D = {(z, y); E X,y E )). where x is the in- put example, y is the target and Y {1..... C) is the la- bel space (closed-wor world assumption). Depending on how classes from the entire dataset 'D are shown revisited in the stream, this configuration can lead to CI, CIR or DI scenarios (Figure J. In Table I we formally present and compare the properties of the three scenario types.
e2
C3
New lnst.
CI
+
Newolld Inst.
e1 8+
C2
C
CIR
m
+
62
eg
e4
DI
In CIR, streams with repetition are characterized by mul- tiple occurrences of the same class over time. To study this scenario, we propose two stream generators designed to create a stream from finite dataset: the Slot-Based Generator (Gstot) and the Sampling- hampling-Based Generator (G (amp)). Golot generate streams by enforcing constraints on the number of occurrences of classes in the stream us- ing only two parameters. Getot does not repeat already observed samples, therefore the stream length is limited by the number of classes. However, it guarantees that all samples in the dataset will be observed exactly once during the lifetime of the model. Instead, Gsamp generates streams according to several parametric distributions that control the stream properties. It can generate arbitrarily long streams in which old instances can also re-appear with some probability.
Figure 1: Illustration of scenario types that can be generated with episodic par- tial access a finite set of concepts. The shape colors indicate whether in- stances are new in each episode or can be mixture of old and new instances.
2.1 SLOT-BASED GENERATOR
The Slot-Based Generator Gstot allows to carefully control the class repetitions in the generated stream with a single parameter K. Gstat takes as input dataset D, the total number of experiences
2

---

Under review as a conference paper at ICLR 2023
Property
C1
DI X =
CIR
Instance Repetition $
Xx, NX, =
x 20
Domain Coverage
U =x=
U X; -
U X P
Concept Repetition *
YinY, =0
= =YN= =Y=Y
Mnyl 20
Codomain Coverage
U =Y
U =
U Y (
Comparison of scenario properties in CI, DI and CIR. P(A) and |A| represent the power set and the cardinality of set A. *: V1 S i.j S N j.
Table 1:
N and the number slots per experience K. It returns CIR stream composed by N experiences, where each of the K slots in each experience is filled with samples coming from single class.
Gstor constrains the slot-class association such that all the samples present in the dataset are exactly once in the stream. Therefore, Getot considers repetition at the level of concepts. To implement this logic, Gtto first partitions all the samples in the dataset into the target number of
Classiincremental
Domain Incremental
Number of Slots (K)
slots. Then, it randomly assigns without replacement K slots per experience. At the end, the N mod K blocks
20
CIR
remaining assigned to the first experience, such that
the rest of the stream is not affected by variable number of slots.
10
The Slot-Based Generator is useful to study the transition from CI scenarios to DI scenarios, obtained by simply changing the parameter K (Figure ( For example, let us consider a dataset with classes such as MNIST. By choosing N = 5 and K 2 we obtain the popular Split- MNIST, scenario with no repetition and 2 classes for each experience. Conversely, by setting N and K 10 we obtain a DI stream where all the 10 classes appear in each experience with new unseen samples. More in
1
10 20
Number Experiences (N)
Figure 2: Illustration of how various
scenarios can be generated by Gator by changing K and N. The red area under the blue curve represents invalid scenar- ios.
C (N must divide general, given a dataset with C classes, we obtain a CI scenario by setting K = N C. In Appendix [ illustrate the overall steps of stream generation (Figure 12 and provide a step-by-step formal definition of Gstor (Algorithm) 2
C).. We obtain DI scenario by setting
2.2 SAMPLING- DAMPEING-BASED GENERATOR
The Sampling- -aseed Based Generator sammp) generates arbitrarily long streams and controls the repe- titions via probability distributions. The stream generator allows to control the first occurrence of new classes and the amount of repetitions of old classes. Unlike Gsiot, it allows to generate infinite and even unbalanced streams.
G Gsamp parameters:
N: Stream length, i.e. number of experiences in the stream.
S: Experience size which defines the number of samples in each experience.
PPy(S): Probability distribution over the stream used for sampling the experience ID of the first occurrence in each class.
. P,: List of repetition probabilities for dataset classes.
Note that Py probability mass function over the stream which means sums up to 1.0 and determines in which parts of the stream it is more likely to observe classes for the first time. However, the list of probabilities {P1, P2. {P1,P2.... Pc} } in P. are independent and each probability value 0.0 S P: < 1.0 indicates how likely is for each class repeat after its first occurrence.
For each experience, Gsamp samples instances from the original dataset D according to a two step process. First, G Gsamp defines C' N binary matrix T called Occurrence Matrix that determines which classes can appear in each experience. Then, for each experience e,, < < N we use the i-th column of 7 to sample data points for that experience. The generator uses the inputs N. PS
3

---

Under review as a conference paper at ICLR 2023
Scenario Matrix
Comcapts
CL Scenario
Gsump Parameters
N
Scenario
5 trate
K (5) P+
Matrix
Generutor
Stest
Sampler
Figure 3: Schematic view of G Samp generator. Each concept is shown with a different color.
and P, to generate T. Therefore, it first initializes T as a zero C' x N matrix. Then for each class e in the dataset, it uses Py(S) sample the index of the experience in which class c appears for the first time. Different probability distributions can be used to either spread the first occurrence along the stream or concentrate them at the beginning, which allows a good flexibility in the stream design. After sampling the first occurrences, the classes are then repeated based on P, probability values finalize matrix T. In the simplest case, P, can be fixed the same value for all classes to generate a balanced stream.
Once the matrix T is constructed, a random sampler used to sample patterns for each experience. Since each experience may contain an arbitrary number of classes, another control parameter that could be added here is the fraction of samples per class in experience size S. For simplicity we keep the fractions equally fixed and thus the number of datapoints sampled from each class in experience
e, is L 5 where |C*| indicates the number of classes present in c₁. Since the sampler is stochastic, each time we sample from a class, both new and old patterns can appear for that class. Given large enough stream length N, the final stream will cover the whole dataset with measurable amount of average repetition. In Figure 3 we demonstrate the schematic of the generator Samp. We provide the pseudo-code for Gaamp in Appendix C (Algorithm 2
[
Although we assume a fixed number of instances per class in D, Gramp can be easily extended to settings where the number of instances can grow over time. Moreover, the sampler can also be designed arbitrarily depending on the stochasticity type, e.g., irregular or cyclic.
3 FREQUENCY-AWAR REPLAY
Experience Replay (ER) is the most popular CL strategy due to simplicity of use and high performance in class- incremental scenarios. The storage policy, which deter- mines which samples to keep in a limited buffer, is the major component of ER methods. Class-Balanced (CB) and Reservoir Sampling (RS) Vitter (1855) are the most popular storage policies in ER methods. CB keeps fixed quota for each class, while RS samples randomly from the stream, which leads to the class frequency in the buffer being equal to the frequency in the stream. CB and RS are great choices for balanced streams such class incremental scenarios, where the number of samples per class is the same over the whole stream. However, as in most real-world scenarios, CIR scenarios are naturally unbalanced, and different classes may have completely different repetition frequencies. Accordingly, CB and RS storage policies may suffer big accuracy drop in the infre- quent classes of an unbalanced stream. For example, in highly unbalanced streams, RS will store an unbalanced buffer replicating the the original distribution of the stream, which is sub-optimal be- cause the less frequent classes will require more repetition to prevent forgetting, while the frequent classes will be repeated naturally via the stream occurrences.
0.6
Storage Policy
(Oum)
Ratio
CB
20
40
60
80
100
Experience
Figure 4: Ratio of buffer slots for infre- quent classes for three random seeds.
We propose Frequency-Aware (FA) storage policy that addresses the imbalance issue in CIR streams by online adjustment of the buffer slots in accordance with the amount of repetition for each class. Given a buffer with a maximum size of M, a list of previously observed classes P initialized as P O with corresponding list O indicating the number of observations per class c in C, and dataset D, from experience e, the algorithm first checks the present classes P. in D; and adds them to P (P Pu P.). Then, for each class e in P, it increments the number of observations O(c) by
4

---

Under review as a conference paper at ICLR 2023
ER-RS
+ LwF
EWC
A.GEM
Naive
MNIST
CIFAR-100
Tiny- Imagllta
1.0
0.5
Average Test Accuracy
0.6
0.0
0.4
0.6
0.4
0.4
0.2
0.2
0.2
0.1
0.0
0.0
K
K
K
Figure 5: Average Test Accuracy for different values of K in CIR scenarios generated with Gstat- Ilaremnncrementall scenarios are represented by the left-most point of each plot, scenarios by the right-most point. Results averaged over seeds.
1 if the class was previously seen, otherwise it initializes Old] = After updating the number of observations, FA computes the buffer quota Q for all observed classes by inverting the number of Tooo
observations (Q = ooc Vc E CT) and normalizes it. This way, the algorithm offers the less frequent classes a larger quota. Finally, a procedure ensures the buffer is used to its maximum capacity by filling unused slots with samples from more frequent classes sorted by their observation times. This a crucial step since it is possible that an infrequent class which is not present 6; will be assigned with a larger quota than its current number of samples in B, and therefore the buffer will remain partially empty. In Figure @eureewee show how our method assigns higher ratio of samples for infrequent classes overcome the imbalance issue in the stream. For further analysis and pseudo-code of FA policy refer to Appendix E. We present examples of unbalanced scenarios in Appendix [
EMPIRICAL EVALUATION
We study CIR scenarios by leveraging our generators Gaamp and Gstot- First, by using Gstor we provide quantitative results about forgetting in CL strategies when transitioning from CI to DI sce- narios (Sec. 4.1. Then, by using G Gammp we focus on long streams with 500 experiences and study the performance of Replay and Naive (Sec. 4.2). The long streams give us the opportunity to study knowledge accumulation over time in the presence of repetition. We also provide an intu- itive interpretation of the model dynamics over long streams (Sec. 4.3). Finally, we show that our Frequency- Aware Replay is able to exploit the repetitions present in the stream and to surpass the performance of other replay approaches not specifically designed for CIR scenarios (Sec. 4.4).
The experiments were conducted using the CIFAR- Krizhevsk et al. (2009) and Tiny- Intgeee LeCun et al. (1998) datasets with the ResNet- 18 model. For Gstot. we run experiments for Naive (in- eremental fine tuning), LwF & Hoiem (2013 &). CWCKkirkparrick et al. (017). Experience Replay with reservoir sampling Kim et al. (2020) (ER-RS) and AGEM Chaudhry et al. (2018) strategies. For Gammp run experiments for Naive and ER (CB/RS/FA) strategies. We the default buffer size for CIFAR- 100 to 2000, and for Tiny-Imagon ImageNet to 4000 in the replay strategies. We evaluate all strategies on the Average Test Accuracy (TA).
4.1 TRANSITION FROM CLASS-INCREMENTAL TO DOMAIN INCREMENTAL
DI and CI scenarios are heavily studied in the CL literature. However, little is known about what happens to the performance of popular CL strategies when gradually transitioning from one scenario to the other. By changing the value of K in Gelot, we provide quantitative analysis of such behavior in CIR scenarios. Figure [ouure]]shhwe shows the Average Test Accuracy over all classes for different CL strategies when transitioning from CI (left-most point of each plot) to DI (right-most point of each plot).
Replay is one of the most effective strategies in CI scenarios. As expected, in CIR scenarios the advantage provided by ER-RS with respect to other CL strategies diminishes as the amount of rep- etition increases. However, in order for the other strategies to match the performance of ER-RS, the environment needs to provide a large amount of repetition.
LwF guarantees consistent boost in the performance, both in CIFAR- 100 and Tiny-Im In particular, and quite surprisingly, on -mny-Ima LwF is able to quickly close the gap with ER-RS and even surpass it as the amount of repetition increases. The positive interplay between
5

---

Under review as a conference paper at ICLR 2023
Class Status
Presont Missing
Class Accuras a
6.1
0.3
0.2
01
150
150 200
250
300
250
4.00
450 500
Experience
Figure 6: Accuracy of a particular class over the stream. The target class is either present or absent in the experiences indicated by the blue and orange points, respectively.
distillation and repetition provides an effective way to mitigate forgetting in CIR scenarios, without the need to explicitly store previous samples in an external memory. EWC showed different sen- sitivity the regularization hyyer-paramen on MNIST we did not see any difference in performance, on CIFAR- 100 and Tiny-ImageNet large values of À lead to dramatic decrease, dropping low Naive. We found 0.1 to be the best value on both CIFAR-100 and Imay-ImageNet. This configuration only provides a low amount of regular- ization. Overall, the role played by the natural repetition already guarantees a sufficient stability of the model, which is additionally boosted only in the case of LwF.
A. We experimented with À 0.1, 1, 10, 100. While
4.2 IMPACT OF REPETITION IN LONG STREAMS
We investigate the impact of repetition in long streams (N 500) generated with Gramp- For the long-stream experiments we also report the missing-classes accuracy (MCA) and seen-classes accuracy (SCA). MCA measure the accuracy over the classes that were seen before but are missing in the current experience, and SCA measure the accuracy over all seen classes up to the current experience.
Missing Class Accuracy Increases Over Time In CI scenarios, Naive strategy catastrophically forgets each class as soon as it starts learning on new classes. Surprisingly, we found that in CIR scenarios there is knowledge accumulation over time for all the classes. Figure the accuracy of single class over time, highlighting whether the class is present or not in the current experience. At the beginning of the stream missing classes completely forgotten, which be noticed by the instant drop of the accuracy to zero. However, over time the model accumulate knowledge and the training process stabilizes. As a result, the accuracy of missing classes tends increase over time, suggesting that the network becomes more resistant to forgetting. Notice that this is an example of continual learning property that is completely ignored when testing on CI scenarios. This finding prompts the question, "What is happening to allow knowledge accumulation even for Naive finetuning?" We investigate this question by analysing the model's accuracy over time and the properties of the learned model in the next experiments.
Accuracy Gap Between Naive and Replay Decreases Over Time To study the impact of long streams with repetitions monitor the accuracy gap between ER and Naive fine-tuning by com- paring their accuracy after each experience. For the scenario configuration, we set Py(5) as a Geometric distribution with coefficient of 0.01 and fix the probability of repetition P, 0.2 for all classes. For more details and illustrations on distribution types refer to Appendix C In such scenarios, the majority of classes occur for the first time in the first quarter of the stream, and then repeat with a constant probability of 0.2 which makes them appropriate for our experiments since all classes are observed before the middle of the stream and the repetition probability is low enough.
Average Test Accuracy
Average Seen Class Accuracy
Average Missing Class Accuracy
0.6
0.5
0.5
Accuracy
0.4
Accuracy
Accuracy
0.3
0.3
0.3
Strategy ER-CB
Strategy ER-CB
Strategy
0.1
Nalve
0.1
Naive
0.1
Naive
ER-CB
200 300
300 Experience
400
200
500
Experience
Experience
Figure 7: Average test accuracy and average missing class accuracy plots for long streams streams with 500 experiences.

---

Under review as a conference paper at ICLR 2023
As can be seen in Figure while the accuracy of ER saturates over time, the accuracy of Naive increases, closing the gap between the two strategies from around 25% in experience 100 to 7% in experience 500. This supports our hypothesis that neural network's ability to consolidate knowledge is significantly influenced by "natural" repetition in the environment.
The Role of Repetition The amount of repetition is one of the key aspects of CIR scenario. To find out how strategies perform under different repetition probabilities, we consider a setting where all components of a scenario are fixed except for P.. For this experiment, we set (P((S) as geometric distribution with p 0.2 and let P. change. In Figure S we demonstrate the seen class accuracy (SCA) for the Naive and ER-CB strategies in CIFAR-100. It is clear from the plots, that the model's rate of convergence can be significantly affected by the amount of repetition the scenario. Although, it may seem obvious that higher repetition leads to less forgetting, it is not very intuitive to what extent different strategies may gain from the environment's repetition. While the Naive strategy gains lot from increased repetition, the replay strategy saturates after some point for higher repetitions and the gaps close between different repetition values.
Naive
0.7
0.7
0.6
Accuracy
0.5
Accuracy
0.5
p
0.4
0.2
0.3
0.2
0.3
0.7
0.7
0.1
1.0
0.0
0.0
Experience
Experience
Figure 8: Retained aceuracy for different values of p in Py.
4.3 MODEL. SIMILARITY AND WRIGHT SPACE ANALYSIS
Weight Interpolation Based on the "gradual loss drop" observation in missing classes, we study how the loss surface changes over time if we perturb the weights. We interpolate between the model weights from two consecutive checkpoints with an interval of 10 experiences in various segments of the stream. Assuming that wi and wi+10 are the obtained solutions for experiences t and t + 10 respectively, we generate eight in-between models wh a wi - (1 a) * w by by increasing from zero to one, and then compute the accuracy of w/ for the data of experience
We show the interpolation accuracy for various pairs of experiments in different segments of the stream for the Naive strategy Figure 9 (left). In the beginning of the stream, the accuracy of experience in each pair drops significantly, while we observe a milder loss drop towards the end of the stream. The findings suggest that, towards the end of the stream, even a relatively big perturbation does not have large negative effect on the model's accuracy and the optimal solutions of the consecutive experiments are connected with a linear low-loss path.
Accuracy over Linear Path
Difference the Experience
1.0
0.175
Blocks
0.150
Bock
0.0
0.125
Bock
Accuraoy
0.6
0.100
Block
Start End
0.4
50-100
0.075
0.050
250
300 350
0.025
400 450
0.000
0.4
0.6
0.9
1.0
100
300
400
Interpolation Weight
Experience
Figure 9: (left) Interpolation accuracy. (right) Weight changes in each block. The difference used in (right) calculated as Dj = pol Σ. 9 with the initialization fg for each block
(0..)
where the weights of experience j are compare
Po.
Weight changes Another approach analyzing the gradual drop of the accuracy is by dissecting how much when, and where the weight changes occurs. As shown in Figure 9 (right), we can observe that within the first experiences, there significant difference for blocks 1, and 2. This
7

---

Under review as a conference paper at ICLR 2023
Fraction= 0.1
Fraction 0.3
Fraction= 0.5
DS
Strategy
MCA
MCA
TA
MCA
Naive
5.0
58.0 0.1
2.2
4.8
8.0
40.8 1.4
C-100
ER RS
11.4
0.7
16.7 3.6
51.1
20.5
4.8
ER-CB
2..7
0.1
34.5
55.3
0.5
52.0 1.5
ER-FA
52.2 +.1
6.3
44.7 1.5
57.8
40.9
54.2
Naive
0.4
0.1
29.1
2.0
2..0
ER RS
3.7
1.2
0.7
28.1
6.0
24.0
ER-CB
10.4 0.2
32.2 0.7
10.0
28.8 28.8+022
1..0
26.0 + 0.3
ER-FA
22.0 1.0
33.0 0.9
15.3 + 1.0
30.4+0.1 0.1
13.6 0.1
27.0 0.1
Table 2: Unbalanced scenario results for the CIFAR-100 (C-100) and TinylmageNet (TIN) dataset. "Fraction" refers the fraction of infrequent classes having repetition probability of only 10%.
difference then stalls, showing that as we continue training experiences, the weights of these blocks stabilize. On the other hand, blocks and show a linear increase in the difference with the number of experiences. An explanation for this phenomenon, is that the first layers of the model capture knowledge that can useful for several classes (more general), after several experiences. On the other hand, the last blocks are the ones that memorize or learn more specific patterns, so they adapt to each experience.
unnecessary to change them
CKA Analysis Finally, we show the CKA Korbblith et al. (2019) the model in the beginning, middle and the end of the stream with interval difference of 50 experiences. As shown in the visualizations in Figure TO the longer the model is trained on more experiences, the less significant the changes in the representations become especially for the final layers. We can that the diag- onal of the CKA becomes sharper propagating forward with more experiments. This indicates that although the model is trained different subsets of classes in each experiment, the representations change less after some point in the stream.
5 15
25 35
55 65
190-+200
190
Experience-
Exper Sierreene--2
Experienc PlperienceS5
-84
004
Experience-15
Experience-35
Experience-65
Experience
Figure 10: CKA of the model in different parts the stream.
4.4 FREQUENCY-AWARE REPLAY UNBALANCED SCENARIOS
We conduct experiments for bi-modal unbalanced scenar- ios where classes have high frequency of 1.0 or low frequency of 0.1. We use fraction factor that de- termines the amount of infrequent classes in the scenario,
0.5
Strategy
0.4
e.g., Fraction=O.3 - means that 30% of the classes are infre- quent. In Table we compare ER-FA with the Naive, ER- RS and ER-CB strategies. The numbers show the MCA and average Test Accuracy (TA) metrics for each strategy in the end of the stream averaged over three runs. Our strategy outperforms all other scenarios in almost all set- tings in both CIFAR-100 and TinylmageNet datasets in terms of TA. and significantly outperforms other methods in terms of MCA (in the last experience). Moreover, we illustrate the accuracy of infrequent classes in CIFAR-100 100 experiments for Fraction=0.3 in Figure
Accuracy
0.3
Nave
ERRRS
0.2
0.1
0.0
40 Experience
11: Accuracy of Infrequent Classes.
Figure
Il where ER-FA achieves considerably higher accuracy in the whole stream by assigning larger quota to infrequent classes without losing its performance on frequent classes (refer Appendix for further illustrations).
8

---

Under review as a conference paper at ICLR 2023
5 RELATED WORK
Current CL methods are mainly focused on two types of benchmarks namely, Multi Task (MT) and Single Incremental Task (SIT) Maltoni & Lomonaco (2019). MT divides training data into distinct tasks and labels them during training and inference. SIT splits single task into sequence of unlabeled experiences. SIT can be further divided into Domain-Incremental (DI) where all classes seen in each experience, and Incremennll (CL) where each experience contains only new (unseen) classes vaa de Ven & Tolias (2019). Both DI and CI are extreme cases and are unlikely to hold in real-world environments Cossu et al. (202T). In more realistic setting, the role of natural repetition in CL scenarios was studied in the context of New Instances and Classes (NIC) scenario Lomonaco al. (2020) and the CRIB benchmark [Stojanov al. (2019). NIC mainly focuses on small experiences composed of images of the same object, and repetitions in CRIB are adapted to a certain dataset and protocol. The Incremenall with Repetition (CIR) scenario was initially formalized in Cossu et al. (202I) however the work lacks systematic study of CIR scenarios as the wide range CIR scenarios makes them difficult to study.
To counter the lack of repetition in CI. replay has been extensively used as CL strategy (Rebuffi)
cet al. 2017; Lopez-opprrrzzz & Ranzato 2017, Chaudhry et al. 2018 Wu et al. 2019 Castro et al, Belouadah & Popescul 2019: Kim et al., [2200] Douillard et al. 2020). In such methods,
2018
natural repetition is artificially simulated by storing past data in an external memory, and replaying them alongside the scenario stream data. Repetition reduces catastrophic forgetting through implicit regularization of model's weights Hayes et al. (2021). In CI benchmarks, replay seems to be the only working strategy |van de Ven et al. a (2020). other words, replay seems to be necessity when no natural repetition happens.
Although replay can be seen as method to simulate natural repetitions artificially, the two concepts are fundamentally different. Repetition in replay strategies occurs with the same data seen in pre- vious experiences. which is neither realistic nor biologically plausible Gupta al. (2010). On the other hand, natural repetitions of already seen objects occur in different real- environments, and better fit the CIR scenario studied in this paper. Recently, Lesort et al. (2022) scaled the number of tasks in a finite world setting (Boult et (019 | Mundt et al. (002)) where the model has access to a random subset of classes in each experience. The authors proposed naive fine-tuning with masking techniques to improve retained accuracy. Our work is different in the sense that we compare among different strategies and study various types of repetitions with two flexible generators.
6 DISCUSSION AND CONCLUSION
We defined CIR scenarios which represent CL environments where repetition is naturally present in the stream, Although the concept of repetition is quite intuitive, it is not obvious how to realize it in practice for research purposes. Therefore, we proposed two CIR generators that can be exploited to address this issue. Through empirical evaluations, we showed that, unlike CI scenarios, knowledge accumulation happens naturally in CIR streams, even without applying any CL strategy. This raised the question of whether the systematic repetition provided by Replay critical in all CIR scenarios. With several experiments in long streams, we demonstrated that although Replay provides an ad- vantage in general, even random repetition in the environment can be sufficient to induce knowledge accumulation given a long enough lifetime.
Moreover, found that existing Replay strategies are exclusively designed for classical CI scenar- ios. Thus, we proposed novel strategy, ER-FA, to exploit the properties of CIR scenarios. ER-FA accumulates knowledge even in highly unbalanced stream in terms of class frequency. ER-FA out- performs by a large margin other Replay approaches when monitoring the accuracy for infrequent classes while preserving accuracy for the frequent ones. Overall, ER-FA guarantees a more ro- bust performance on wide range of real-world scenarios where classes are not homogencously distributed over time.
The framework defined in this work opens new research directions which depart from the existing ones, mainly focused on the mitigation of forgetting in CI scenarios. We hope that our experiments and results will promote the study of CIR scenarios and the development of new CL strategies, able to exploit the inner semantics of repetition, a natural trait of werl-world data streams, in many other clever and imaginative ways.

---

Under review as a conference paper at ICLR 2023
7 REPRODUCIBILITY STATEMENT
We will provide an open-source implementation of our generators and algorithms with the scripts needed to reproduce the results reported in the paper.
REFERENCES
Rahaf Aljundi, Min Lin, Baptiste Goujand, and Yoshua Bengio. Gradient based sample selection for online continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buse E. Fox, and R. Garnett (cds.). Advances in Neural Information Processing Systems, volume 32. Cur- ran Associates, Inc., 2019. URL ltt pttpss:::........... http::::s///p.ceeeieeeeeooooeeee nwwriss.errrpp..p cttppp/s/cccpppppp//./////
Jihwan Bang. Heesu Kim, YoungJoon Yoo, Jung-War Woo Ha, and Jonghyun Choi. Rainbow mem- ory: Continual learning with memory of diverse samples. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8218-8227, June 2021.
Eden Belouadah and Adrian Popescu. I12m: Class incremental learning with dual memory. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 583-592, 2019.
Terrance E Boult, Steve Cruz, Akshay Raj Dhamija, Manuel Gunther, James Henrydoss, and Wal- ter J Scheirer. Learning and the unknown: Surveying steps toward open world recognition. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 9801-9807, 2019.
Francisco M Castro, Manuel Marín-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pp. 233-248, 2018.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. In International Conference on Learning Representations, 2018.
Andrea Cossu, Gabriele Graffieti, Lorenzo Pellegrini, Davide Maltoni, Davide Bacciu, Antonio Carta, and Vincenzo Lomonaco. Is class-incremental enough for continual learning? arXiv preprint arXiv:2112.02925,, 2021.
Matthias De Lange and Tinne Tuytelaars. Continual prototype evolution: Learning online from non- stationary data streams. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 8250-8259, October 2021.
Arthur Douillard, Matthicu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In European Conference on Com- puter Vision, pp. 86-102. Springer, 2020.
Anoopum S Gupta, Matthijs AA van der Meer, David S Touretzky, and A David Redish. Hippocam- pal replay is not a simple function of experience. Neuron, 6555)65557 2010.
Tyler L Hayes, Giri P Krishnan, Maxim Bazhenov, Hava T Siegelmann, Terrence J Sejnowski, and Christopher Kanan. Replay in deep learning: Current approaches and missing biological elements. Neural Computation, 33(1) (111))))28-295555 2021.
Chris Dongjoo Kim, Jinseo Jeong, and Gunhee Kim. Imbalanced continual learning with partition- ing reservoir sampling. In ECCV, 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinste Demis Hass-
abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forget- ting in neural networks. Proceedings of the National Academy of Sciences, 1 114(13):32113356 2017. ISSN 0027-8424. doi: 10.1073/ppas.161113511444 URL https: www. pnas org/
Hyunseo Koh, Dahyun Kim, Jung- Woo Ha, and Jonghyun Choi. Online continual learning on class incremental blurry task configuration with anytime inference. In The Tenth International Confer- ence on Learning Representations, ICLR 2022, Virtual Event, April 25- 2022. OpenReview.net, 2022. URL htt.httts///opeeevvieeee net. forum? www. GGG
10

---

Under review as a conference paper at ICLR 2023
Simon Kornblith, Mohammad Norouzi, Honglak Lee. and Geoffrey Hinton. Similarity of neural network representations revisited. In International Conference on Machine Learning, pp. 3519- 3529. PMLR. 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2224 1998.
Timothée Lesort, Vincenzo Lomonaco, Andrei Stoian, Davide Maltoni, David Filliat, and Natalia Blaz-RRoríguez. Continual learning for robotics: Definition, framework, learning strategies, op- portunities and challenges. Information fusion, 58:52-68 68, 2020.
Timothée Lesort, Oleksiy Ostapenko, Diganta Misra, Md Rifat Arefin, Pau Rodríguez, Laurent Charlin, and Irina Rish. Scaling the number of tasks in continual learning. arXiv preprint arXiv:2207.04544 2022.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Puttern Analysis and Machine Intelligence, 40(12):2935-2947 Dec 2018. ISSN 1939-3539. doi: 10.1109TTPAMI. 2017.277308
Vincenzo Lomonaco, Davide Maltoni, and Lorenzo Pellegrini. Rehearsal-fre continual learning small non-iid batches. In CVPR Workshops. pp. 989-998, 2020.
Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, Antonio Carta, Gabriele Graffieti, Tyler L. Hayes, Matthias De Lange, Mare Masana, Jary Pomponi, Gido M. van de Ven, Martin Mundt, Qi She, Keiland Cooper, Jeremy Forest, Eden Belouadah, Simone Calderara, German I. Parisi, Fabio Cuzzolin, Andreas S. Tolias, Simone Scardapane, Luca Antiga, Subutai Ahmad, Adrian Popescu, Christopher Kanan, Joost van de Weijer, Tinne Tuytelaars. Davide Bacciu, and Da- vide Maltoni. Avalanche: An end-to-end library for continual learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR) Workshops, pp- 3600-3610, June 2021.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
Davide Maltoni and Vincenzo Lomonaco. Continuous learning in single-incrementalttas scenarios.
Neural Networks, 116:56-73. 2019.
Martin Mundt, Iuliaa Pliushch, Sagnik Majumder, Yongwon Hong, and Visvanathan Ramesh. Uni- fied probabilistic deep continual learning through generative replay and open set recognition. Journal of Imaging, 8(4):93, 2022.
German Parisi, Ronald Kemker, Jose I. Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Stefan Stojanov. Samarth Mishra, Ngoc Anh Thai, Nikhil Dhanda, Ahmad Humayun, Chen Yu, Linda B. Smith, and James M. Rehg. Incremental Object Learning From Contiguous Views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 8777-8786, 2019.
Gido M van de Ven and Andreas Tolias. Three scenarios for continual learning. arXiv preprint 19X4::1904.0773,, 2019,
Gido M van de Ven, Hava T Siegelmann, and Andreas S Tolias. Brain-inspired replay for continual learning with artificial neural networks. Nature communications, 11(1):1-14, 2020.
Jeffrey S Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS), 11(1):37-57, 1985.
11

---

Under review as a conference paper at ICLR 2023
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye, Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 374-382, 2019.
12

---

Under review as a conference paper at ICLR 2023
A RELATED WORK (CONTINUED)
Considering benchmark formalization frameworks, De Lange & Tuytelaars (202T) recently pro- posed subdivision aimed at framing continual learning setups by categorizing them based on the batch and observable horizon that the learning agent able to access at each time. With this frame- work, the authors aim to better formalize the online learning setup. While the concept of observ- able horizon may be useful in evaluating the significance and local (in time) usefulness of natural repetition in a training stream, this work does not consider the concept of natural repetition in its framework.
Recently, Koh al. (2022) proposed to introduce blurry task boundaries in class incremental bench- marks. Their proposal is based on previous works Bang et al. (202T); Aljundi et al. (2019) that tried to produce more realistic benchmarks by blurring the class incremental scenario, which however resulted in a setup in which no classes are added to new tasks. They argue that this idea moves the focus too far away from the class-incremental setup and it is still not quite realistic. The resulting setup, named i-Blurry, aims at resolving the aforementioned issues and moving toward more re- alistic scenario by partitioning the classes available in the source dataset into two groups: Disjoint and Blurry. Classes of the disjoint group are gradually added in successive experiences while sam- ples of classes from the blurry group always appear in all experiences with their numerosity being controlled through blur ratio M. The authors show that, based on the degree of disjunction N and blurry M, this framework can produce class-incrementa (no blurring), domain-incremennal (no disjunction), and blurred setups. This setup is the one that most moved towards the direction of introducing repetition in Continual learning benchmarks in a controlled way so far. However, the proposed blurring mechanism is too coarse-grained to simulate natural repetition of concepts as the significance of the repetition introduced by blurring relies too much on a random (uniform) sampling of the concepts to be repeated, ii) the static subdivision of classes in the Disjoint and Blurry groups.
B SLOT-BASED GENERATOR
Following the properties of CIR scenarios in Section hold the assumptions below in the defined properties:
2
Gstot generates a subset of CIR streams that
X, n X,I 0: new instances appear in each experiences
U X; = X: all samples are used
IY Y3 2 0, v1 S iÍ S N where i j
U Y = Y: all classes are used
is constant.
These assumptions allow transitioning through different CIR scenario types between the two ex- tremes of CI and DI.
B. ALGORITHM
The overall steps of Gstot are illustrated in Figure 12] Algorithm 2 we present all steps of Gstot used to generate arbitrary CIR scenarios given dataset D, number of experiences N and number of slots K. The output of the algorithm is CL. stream.
B.2 TRANSITIONING
Transitioning in Gotot for a scenario with fixed number experiences can be done by increasing K. When K 1 the generated scenario will be class- insseincrementll and K get closer to the total number of classes D. the scenarios moves towards a domain incremental setting. In Figure 13 we show example of how generated scenarios change by increasing K.
13

---

Under review as a conference paper at ICLR 2023
Concepts: 4 0
Replicate each concept and shuffle the sequence.
KKN
times
Example:
G 0
e2
E3
e
New Instances
Slot
Strain
S
d
Slot
L
Slot
Figure 12: Illustrations of the overall steps of Gtot Each shape represents a concept, and the green color means that new instances of that concept are used in each experience.
3::
Class
Experience
Figure 13: From left to right: transitioning from CI to DI in Gstor. Each class is represented with unique color.
C SAMDLING-BASED GENERATOR
Following the properties of CIR scenarios in Section 2 that hold all defined properties. Additionally, for any stream S {er E2, ENy ). G Gsamp defines probability distribution for the first occurrence of concepts over S and per-class probabilities for each concept e Y. G sampp can generate arbitrarily long stream (N 1) and even from a growing set of samples X where Y remains constant.
Ssamp generates a subset of CIR scenarios
C. ALGORITHM
The overall steps the Getor are shows in Algorithm 2
C.2 DISTRIBUTION TYPES
In this section show examples of different discrete distributions that can be used for (S) and P,. For P, we use the unnormalized version of the final distribution. Distributions used for Gsamp can be any arbitrary discrete distribution and limited the ones describe here.
C.2.1 ZIPFIAN
Given the number of elements N and scalar 2 0 the probability mass function of Zipfian distribution over a list N elements defined in Equation [ When used for the probability of first occurrence, the distribution can be defined over the experiences of stream. For example, N can be considered as the number of experiences and i can indicate the ith experience in the stream. By increasing the distribution over the stream will be skewed towards the beginning. In figure 14 we demonstrate some examples first occurrence probabilities over stream of length 10 generated with Zipf distribution with increasing values of Many natural distribution follow Zipf distribution
14

---

Under review as a conference paper at ICLR 2023
Algorithm Slot-Based Generator (Gator) Pseudo-Code.
Require: Dataset D {(2; I=........ with C' classes, number of experiences N, experience size present in each experience.
Ensure: K S N
Ensure: C mod N ==
Ensure: NK mod C' =
cls-idxs = 0
Empty dictionary
for E set (ee do
cls-idxs[y] = =0
Empty list init
end for for l, P
cls-idxs[y, Jappend()) end for slots={}
Empty dictionary
for y cls-idxs do slots[y] = [
ksample int(Len(cls-idxslg)) /K)
for k 1,..., N do
subset-idxs pop(cls-idxslyy) ksample)
subset-samples (dids for idx subset-idxs)
end for end for stream =
for n , N do
experience = dataset()
seen-classes = []
for k=l.....KK K do
repeat
y = sample(slots)
until & seen-classes
geen-classes.appendgg
experieence.dd(ppppssotttll 1))
end for
stream.appendlexperinnee))
end for
return stream
0.0
e 0.5
e=0.0
e 2.0
1.0
1.0
1.0
Probability
0.5
0.5
0.5
0.0
0.0
0
5
10
10
0
5
10
10
5
10
Experience
Figure 14: Zipf distribution with varying values of
and it can be used to generate highly skewed distributions both for first occurrence and repetition probabilities.
f(ie,N) = 50
(1)
Σ n
15

---

Under review as a conference paper at ICLR 2023
Algorithm 2 Sampling-Basse Based Generator (G samp)) Pseudo-Code.
Require: Dataset
{(z, y=) with C classes, number of experiences N, number of slots K, probability distribution for first occurrence P (S), and list of repetition probabilities P.- b Initialize occurrence matrix with zeros
T [0)
for c E 10. C} do
8 ~~ (S))
Samples the first occurrence of class c
T[c
for j E {i, + 1,...... N) do
r~ (0,1)
U: uniform distribution over [0, 1.0]
ifr < Pr|c then [(c.j] =
end if
end for
end for
E = 0
for
e; E f1.2.... N) do
C = RetrieveClassee(c)
De a, Sample DD, C.
Sample S instances from dataset for classes C;
E 4 EU De:
end for
Strain: Stest Generote return Strains Stest
Generate streams using E
C.2.2 POISSON
The PMF for Poisson distribution is given Equation 2 where μ 2 Poisson with larger values of μ can be used for distributions where the probability of occurrenccrrepetttion first rises and then gradually decreases over time.
(i; Wee
i!
μ 0.0
== 0.5
== 1.0
1=1.5
μ 2.0
1.0
1.0
1.0
1.0
1.0
Probability
0.5
0.5
0.5
0.0
0.0
0.0
Experience
Figure 15: Poisson distribution with varying values of
C.2.3 GEOMETRIC
Another useful distribution that can be used for the first occurrence probabilities over a stream is Geometric distribution with its PMF given in Equation 3] This distribution is in particular inter- esting for transitioning from domain incremental class incremental. By setting p only the probability of experience probability will spread over the stream. figure 17 we show examples for generated scenarios with G Gampp with Geometric first occurrence and fixed probability of repetition.
will be equal to 1.0 and the rest will be zero, and by decreasing p.
f(ip) (1 - -p)
(3)
16

---

Under review as a conference paper at ICLR 2023
0.01
0.2
0.5
0.7
1.0
1.0
1.0
Probability
0.5
0.5
0.5
0.0
0.0
5
5 Experience
Figure 16: Geometric distribution with varying values of p.
Class
Class
C.
Experience
Exper ence
Experience
Figure 17:
Scenarios generated with Geometric first occurrence and probability of repetition equal 1.0 for all classes. The p values for the Geometric distributions from left to right are 0.01, 0.2 and 1.0 respectively.
D UNBALANCED SCENARIOS
In this section we present particular type of unbalanced scenarios where a subset of classes in the stream have low probability of repetition and the rest repeat very often. We refer to such scenarios bi-modal scenarios, where each mode refers to a subset classes with distinct repetition probability. More specifically,
have stream of experiences S {ei, E2, .... EN where y UN Ye, indicates the set of all available concepts in S. In bi-modal scenarios yiuu' where y'd and frr are the set of frequent and infrequent concepts respectively, and yif U yfr O. In Figure 18 we show examples of unbalanced bi-modal scenarios.
Class
Class
Experience
Figure 18: Unbalanced scenarios with two modes of repetition. The fractions of infrequent classes from left to right are 0.2, 0.4 and 0.6 respectively. Repetition probabilities for frequent and infre- quent classes are set 0.2 and 0.9 accordingly.
E FREQUENCY-AWARE REPLAY
E. ALGORITHM
In Algorithm 3 we present the steps for updating the buffer in FA storage policy.
E.2 ANALYSIS: VARYING THE FRACTION OF INFREQUENT CLASSES
In this section, we study the behavior of FA, CB, and RS storage policies by changing the fraction of infrequent classes. In analysis, consider unbalanced stream generated with G Geamp where N 100 and the probability of repetition for frequent and infrequent classes are 0.9 and 0.1, respectively. In such streams, the large probability gap between frequent and infrequent classes
17

---

Under review as a conference paper at ICLR 2023
Algorithm 3 Frequency-Aware Buffer.
Require: Current Buffer Set B, Maximum buffer size M, List of Seen Classes C, Number of
Observation per Seen Class O
D = GedExxeerienceeetttseeee]
P = DetecdPreeenntClsssss(()
C' - C
for c: E P do
> For each present class, increment the number of observations
c O then O[d]+
else Olc] =1 end if
end for
Q lola Vc E CI
Calculate quota per class
Q Q
> Normalize quota values I Calculate buffer slot size for each class Update assigned slots according to the current state of B
S = UlQ vc E C}
Upoate S)
UpdateBuffert B, D, S) return B,M,C.O
Storage Policy
Storage Policy
(Durs)
(Surs)
0.8
CB
0.8
C8
0.8
RS
R5
(ours)
0.2
0.2
0.2
CB
RS
Experience
Experience
Experience
0.8
0.8
Storage Policy
FA (Ours)
0.6
Storage Policy (Gurs)
0.2
CB
0.2
R5
Experience
Experience
Figure 19: Ratio of samples for infrequent classes in unbalanced scenarios for the FA, CB and RS policies. Fraction of infrequent classes from top-left to bottom-right are 20%, 40%, 60,, 80%, 100%.
helps observe the difference more clearly. We report the ratio of samples assigned to infrequent classes in the buffer in the lifetime of the model in the stream for scenarios where the fraction of infrequent classes is equal to {20%, 40% 60%, 80%, 100%). }. For this experiment, we set the buffer size 500 for all methods.
As demonstrated in Figure 19 when the fraction of infrequent classes is equal to 20%, i.e. only 20% of classes are infrequent, the ratio is very low for RS policy tries replicate the distribution of the stream while CB assigns exactly 20% of the buffer space to the infrequent samples. However, we can observe that FA starts to assign more samples over time to the infrequent classes over time as it adapts the buffer slots based on the frequency of repetition. Moreover, it is evident in the plots that, by increasing the fraction of infrequent classes, the ratio gap between FA and CB gets smaller as the quota for CB stays the same while the number of infrequent classes increases. Eventually, when the fraction of infrequent classes is equal to 100%, i.e. all classes have the same (low) probably of repetition, all buffers have exactly the same ratio since all classes are infrequent.
In conclusion, FA buffer slots can be very helpful in highly unbalanced streams where a smaller fraction of classes have low probability of repetition. When the stream moves towards becoming
18

---

Under review as a conference paper at ICLR 2023
balanced, the FA and CB get closer, and all methods become similar in the extreme case of fully balanced stream with similar probability of repetition.
F CHANGING Py(S)
ER.CB
Naive
0.7
0.7
0.5
0.5
0.3
0.3
0.3
0.3
0.0
0.0
Experience
Experience
Figure 20: Average test accuracy for different values of p in first occurrence.
We conduct experiments find the differences between situations in which all classes versus those which new classes also appear late the stream in order to analyze the role of first occurrence type. How early late in the stream observe all classes of dataset, depends the the parameters that control (S) In this experiment, we fix the probability of repetition P. and change PS parameters. In particular, we opt the geometric distribution for P/ and choose the values {0.1, 0.3, 0.7, 1.0] for its only parameter ( < S. 1.0. Increasing p is inversely proportional the spread factor in the first occurrence distribution, i.e. when p is close to all classes happen in the first experience and as we move toward 1.0 the classes start to spread along the stream. Figure s0oshows the CIFAR-100 results for the Naive and ER-CB strategies. The results suggest that when the spread factor is low, the model initially has difficulty to learn since there are more classes the initial experiments thus model has to learn from fewer instances. However, with more experiences, all first occurrence types, reach almost the same SCA.
early
G ER-FA RESULTS
Results in Figure 21 illustrate the total test accuracy and accuracy of frequent classes over time. Although the discrepancy between the accuracies of frequent classes is very small, the total test curacy can significantly vary due to the difference in the accuracy of infrequent classes as presented in Section
4.4
0.7
0.6
Accuracy
0.4
Accuracy
0.4
0.3
Strategy ER-FA
0.3
Strategy
0.2
0.2
ER-FA
0.1
ER.S
ER.R
Naive
Naive
40
80
100
20
40
80
100
Experience
Experience
Figure 21: TA over all classes (left) and frequent classes (right) in a bi-modal unbalanced scenario with Fraction=0.3.
19


Reviews:
[{"review_id": "OmuUC31DAjz", "reviewer": "Reviewer_U14g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "For the possible repetition in Continual Learning, the paper makes contributes in the following three areas: (1) The author proposes two stochastic scenario generators that produce a wide range of CIR scenarios starting from a single dataset and a few control parameters. (2) The author conducts a comprehensive evaluation of repetition in CL by studying the behavior of existing CL strategies under different CIR scenarios. (3) The author presents a novel replay strategy that exploits repetition and counteracts the natural imbalance present in the stream.", "summary_of_the_review": "Class-Incremental with Repetition (CIR) scenarios are important in realistic scenarios but have not been sufficiently studied so far. This paper proposes two practical CIR generators and performs a comprehensive series of comparative experiments of existing CL methods, which provides the CL area with a referenceable basis for further research. Therefore, it is recommended to accept this paper if the questions about the weaknesses can be answered.", "strength_and_weaknesses": "Strengths: The paper proposes two practical CIR generators and performs a comprehensive series of comparative experiments of existing CL methods, which provides the CL area with a referenceable basis for further research. \n\nWeaknesses: The analysis of the FA storage policy is not sufficient. In the paper, the FA method is only proposed based on ER. Can the FA method be applied to other CL methods? How does the ER-FA method perform when the dataset is not unbalanced? Does FA cause a decrease in class accuracy for frequent classes? The author may want to respond to the weakness the reviewer raised.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LPxpGV8u8CC", "reviewer": "Reviewer_MpJD", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on class-incremental with repetition (CIR), which ranges from class- and domain-incremental learning. For building streams and managing repetition over time, the authors propose two CIR generators (i.e., slot-based and sampling-based), which are from two aspects to generate data streams. Moreover, the authors propose a frequency-aware (FA) storage policy tailored to CIR scenarios. Further, the authors perform the first comprehensive evaluation on CIFAR 100 and TinyImageNet to compare other replay approaches in the CIR scenario, and then show the effectiveness of FA.\n", "summary_of_the_review": "This is not a high-quality paper without a clear expression and lack of solid theoretical analyses, which could not stimulate other research works. So, I recommend rejection.", "strength_and_weaknesses": "Strengths\n1. The idea of CIR is absorbing and reasonable. Because for realistic data streams, it is natural to occurrences previously seen classes in new coming samples. Further, they designed a paradigm to build a stream with repetition via the proposed generators. \nWeaknesses\n\n1. The novelty of this paper is insufficient. In my view, this paper conducted a sampling procedure via two generators. As for frequency-aware replay, its improvement is dealing with unbalanced data. What is neglected in this paper, however, is that there have been numerous works on adaptative store policy in continual learning.\n\n2. The writing of this paper is not clear enough to follow. Firstly, the explanation of the discrepancy between CIR and traditional incremental learning (i.e., class- and domain-incremental) is not clear. Figure 1 replaced with actual objects may give a more intuitive understanding of the authors' motivation. Moreover, it is necessary to use a coherent algorithm to introduce the overall procedure, however, absent in the main text. Finally, the conclusion part is tedious and should be refined.\n\n3. The lack of formulas in this paper makes comprehending have more barriers. For example, the generators build streams part lacks the complementary mathematical explanation. \n\n4. The authors should provide comprehensive theoretical analyses of their method, in my view, which is necessary for solid work.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EmqkcFDinBy", "reviewer": "Reviewer_rWm4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a study on a different flavor of continual learning scenarios than previously considered in the literature. Precisely, one in which data separation is not strict among concepts, and later tasks can revisit one of the previously introduced classes. The authors name this \"class incremental with repetitions\" (CIL).\n\nThe authors show that in these scenarios storing and replaying data is less important (which is not surprising), unless it corrects for imbalanced exposures to the different classes. Therefore the authors propose a sampling method which balances the class representation during training.\n\nExperiments on the standard computer vision tasks show comparisons between domain incremental, class incremental and class incremental with repetitions.", "summary_of_the_review": "The CL community would benefit from an analysis of a more natural stream of tasks than the extreme cases considered so far. Due to the clear exposure, interesting analysis and experiments I suggest the paper should be accepted.", "strength_and_weaknesses": "Strengths:\n - novelty of the analysis\n - relevant CL algorithms analysed under the different scenarios\n - well written paper\n\nWeaknesses:\n - the results are not necessarily surprising; e.g. it is known that balancing classes in training batches benefits performance / optimization", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject