Generative Negative Replay for Continual Learning
Authors: Gabriele Graffieti
Davide Maltoni
Lorenzo Pellegrini
Vincenzo Lomonaco
Venue: ICLR 2022 Submitted (2022)
Keywords: Continual Learning
Generative replay
Lifelong learning

Abstract: Learning continually is a key aspect of intelligence and a necessary ability to solve many real-world problems. One of the most effective strategies to control catastrophic forgetting, the Achilles’ heel of continual learning, is storing part of the old data and replay them interleaved with new experiences (also known as the replay approach). Generative replay, that is using generative models to provide replay patterns on demand, is particularly intriguing, however, it was shown to be effective mainly under simplified assumptions, such as simple scenarios and low-dimensional benchmarks. 
In this paper, we show that, while the generated data are usually not able to improve the classification accuracy for the old classes, they can be effective as negative examples (or antagonists) to learn the new classes, especially when the learning experiences are small and contain examples of just one or few classes. The proposed approach is validated on complex class-incremental and data-incremental continual learning scenarios (CORe50 and ImageNet-1000) composed of high-dimensional data and a large number of training experiences: a setup where existing generative replay approaches usually fail. 

Under review as a conference paper at ICLR 2022
GENERATIVE NEGATIVE REPLAY FOR CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Learning continually is a key aspect of intelligence and a necessary ability solve many real-world problems. One of the most effective strategies to control catastrophic forgetting, the Achilles' heel of continual learning, storing part of the old data and replay them interleaved with new experiences (also known as the replay approach). Generative replay, that is using generative models to provide replay patterns on demand, is particularly intriguing, however, it was shown to be effective mainly under simplified assumptions, such as simple scenarios and low-dimensional benchmarks. In this paper, show that, while the generated data are usually not able to improve the classification accuracy for the old classes, they be effective as negative examples (or antagonists) to learn the new classes, especially when the learning experiences are small and contain examples of just one few classes. The proposed approach is validated on complex class-incremental and data-incremin inteennncrmeetal continual learning scenarios (CORe50 and ImageNet- composed of high-ddmennsioal data and large number of training experiences: a setup where existing generative replay approaches usually fail.
INTRODUCTION
The majority of neural networks training approaches assume that is feasible build set of indepen- dent and identically distributed (i.i..)) samples to train the model. This assumption is in contrast with biological learning since intelligent beings observe the world as an ordered sequence of highly corre- lated data. When state-of-the-art deep neural networks are trained continually, and the whole data cannot be accessed at once, the model suffers from the catastrophic forgetting problem (McCloskey & Cohen, 1989), and the knowledge about old data (old experiences) tend to be overwritten by new examples.
Storing part of past data and replaying them interleaved with new data proved to be effective approach to mitigate forgetting (see Hayes al. (202T) for comprehensive survey). However, in some applications, the storage overhead together with privacy issues make replay techniques unfcasible. Therefore, generative replay has been recently explored, where a generative model is trained to produce data from past experiences (see Lesort et al. (2018) and (Shin et al. (2017)) Besides solving the replay memory issue, generative replay can theoretically be capable of generating more general and novel examples not included in past experiences, thus potentially overcoming replay methods. Unfortunately, generative replay introduces much complexity due to the need for an interleaved incremental training of both a classifier and generator. Moreover, generative models usually complex and unstable to train, especially in incremental scenarios. Several researchers have shown that generative replay fails complex CL scenarios with high-dimensional data (see Aljjundi et al. (2019); Lesort et al. (2018) and van de Ven et al.) (2020)) mainly due to the inaccuracies in the data generation that progressively grows across the experiences if single generator is incrementally updated (see related works in infsection for more details). The photocopy example helps to understand why. Let us consider a high-quality photocopy machine: when picture is initially copied the output looks very similar to the original, but if the process is repeated several times by using input the output of the previous step, some artifacts will soon appear and, after many iterations, the result will be highly compromised. Hence, even if some ofte-oo-the--nn models have been proved to be effective in generating also high dimensional data (Huang et al. (2018) Karras et al. (2019), the continual training of such generators remains challenging problem.

---

Under review as a conference paper at ICLR 2022
CORe50 NC
ImageNet-1000
0.8
0.9
0.7
0.8
Generative negaive replay
Generative posilive replay
0.7
0.6
Accuracy
Accuracy
0.8
0.5
0.5
0.4
0.4
0.3
0.3
Generative pasitive replay
0.2
Generative negaive replay
0.1
0.0
15
20
Experience
Experience
Figure 1: The proposed generative negative replay is compared with classical generative replay on two complex class incremental CL benchmarks (details in infsection 4.2p. In both the benchmarks, using the same classifier, generator, and training procedure, negative replay performs significantly better.
Although generative models are hot research topics and we can expect improved methods in the future, as of today we must deal with imperfect generated data and try to exploit them at best when a classifier is incrementally trained. The proposed approach, denoted Generative Negative Replay, does not attempt to improve the knowledge of old classes using the generated data because it assumes that the data quality is not enough for this purpose. Nevertheless, makes of generated (latent) data as negative examples to better learn the classes of current experience, especially when the number of classes per experience is small and we incur in the "learning in isolation" problem.
We experimentally demonstrate, on complex benchmarks such as CORe50 and PrageNet-1000, where (positive) generative replay fails, that negative replay is effective to contrast the learning in isolation problem, allowing to train a classifier incrementally across high number of experiences (see Figure Ip. We also investigate the impact of data quality on negative replay with an ablation study (section 4.4) where negative examples are sampled from original past patterns (upper bound) and randomly generated.
PROBLEM FORMULATION
A continual learning (CL) problem consists of number NE of experiences, each containing subset of data that is only accessible during the corresponding experience:
CL {ei, e2, ..., eNg).
(1)
each of them is composed of several data points and the corresponding labels:
= (Nx, Va), X = {-f 2 & J. 34- = {vi v2,,.., v..
(2)
where of and w are the data points and the associated labels contained the k-th experience and N& is the number of samples in the k-th experience.
Let D (X) be the entire dataset, then X' = =U X; and - V.
We can define three different scenarios for supervised continual learning (Maltoni & Lomonaco 2019 van de Ven & Tolias 2018): New Instances (NI), New Classes (NC), and New Instances and Classes (NIC). In NI (also known as domain incremental) all the classes are introduced in the first experience, and only new examples of the same classes are included in the following experiences. In NC (also known as class incremental) each experience contains only examples from classes never seen before. NIC the combination of NI and NC, so each experience can be composed of new examples of already seen classes and/or examples from new classes. A formal definition of the above scenarios can be found in Appendix A
Given the above definitions, our goal is to fit a function S. parametrized by θ, to the sequence of experiences. naive approach is finding the best parameters @* that minimizes:
- = arg min L(fe (K,), for i = [1....NE5)
(3)
2

---

Under review as a conference paper at ICLR 2022
where [() is a loss function (e.g. cross entropy loss).
As first pointed out by McCloskey & Cohen (1989), this simple approach is prone to catastrophic forgetting, thus the model fe is not able to learn the experiences {c1, ENg sequentially.
2.1 CONTINUAL LEARNING WITH GENERATIVE REPLAY
Generative replay requires to train simultaneously and incrementally classifier and a genera- tive model (Shin al. 2017: Wu et al. 2018; Thandiackal et al. 202I) The generative model g. parametrized by 52 provides surrogate data similar to the past experiences" data. In the case of conditional generative model (in which we can control the class of the generated data), the optimal parameters of the classifier can be derived using replay memory as follows:
θ = arg min (X U MM Dr M for [l,..,,
(4)
where Mj and M are the datapoints and labels contained in the replay memory during the training on experience when the replay memory is populated as:
M + sa(z))e) My + cs oj U j= (1,,,..,
(5)
where R is the number of generated replay patterns (size of memory), is latent random input vector to the generative model, e is a label sampled from the set of labels encountered in the past experiences, and "+"" indicates the insertion of element in the memory.
The same generated data fed to the classifier can be used to control forgetting in the generative model as well. Instead of generic generative model, suppose we have conditional generative model composed of an encoder q, parametrized by ty and a decoder Pe parametrized by &. such that In = Pc 97 = (7, 5) The optimal parameters of the generative model can be obtained requiring that the generated data are similar (L2 loss) to the original ones:
,", = arg min pp (4, (x, U MID M - X, U M for i = {1, Neh.
(6)
7.6
where dy (X,) is forced to follow target distribution, typically (0 1).
GENERATIVE NEGATIVE REPLAY
As discussed before, generative replay is an appealing strategy for continual learning, but, to exploit it in complex scenarios with many experiences, we need to overcome the data degradation issue. Since this problem is not easily addressable on the generator side, propose to circumvent it by changing the way the classifier makes use of generated data.
Let us suppose the classifier fe be divided into feature extractor far parametrized by é and classification head ce parametrized by vh, so that fe = CR. o fa. θ = (Ø. v) The parameters e of the classification head can be divided into C groups, where is the number of classes. The groups, denoted as (182, 1/22 .... C) represents the parameters associated to the connections between the features extracted by fo and the output neuron of the corresponding class.
For simplicity, let us assume that the feature extraction weights & are frozen (after an initial pre- training) and, across the experiences, we only learn the classification head weights vb. As explained in Section Section 3.3) this assumption is not necessary and our experiments were carried out by learning both 6 and db.
3.1 LEARNING CLASSES IN ISOLATION
Learning in isolation is one of the main causes of catastrophic forgetting, especially in the NC or NIC scenarios where only a limited number of classes are present in a single experience, and the parameters of the classification head learned without negative examples that counteract the "greediness" of the optimization. As an example, let us consider an NC scenario where only one class is present in each experience. Suppose that is the only class in the experience then the
3

---

Under review as a conference paper at ICLR 2022
best way to optimize the model is to change the parameters vf to maximize the output of the output neuron с for every input and change the rest of v/,, j * с to minimize the output for remaining classes. This still holds if in the experience are present only few classes, since the model is only optimized to discriminate between the present classes and has no interest in maintaining the past acquired knowledge.
3.2 POSITIVE AND NEGATIVE REPLAY
Replay can be used to counteract the learning in isolation problem, however, when the replay data comes from a generative model, the data quality degradation has negative impact on the classifier training. The aforementioned problem typical of the standard generative replay approach (hereafter denoted as generative positive replay), where replay data is used by the classifier in the same manner of the current experience's 's data, and therefore the classification head's weights associated to the replay classes are optimized based on the replay data.
On the contrary, in the proposed generative negative replay approach the replay patterns are used to counteract the detrimental effects of the training in isolation but they are not used to modify the parameters 4 associated with the replay classes. The key idea (validated experimentally) is that the generated patterns are valid antagonists to mitigate the learning in isolation problem, but their quality is not enough to improve the knowledge of classes originally learned on real data. is well known in general less effective than discriminative learning because
that one class learning approaches the presence of both positive and negative examples allows to better characterize the classification boundaries (Hempstalk & Frank 2008). Therefore, the proposed approach exploits generated data to constrain the classification boundary and to avoid that the real data in the current experience pull it too much in their direction.
3.3 TRAINING A CLASSIFIER WITH GENERATIVE NEGATIVE REPLAY
The idea of generative negative replay is quite general can be used in conjunction with different continual learning classification approaches and scenarios (NI, NC, NIC). To avoid replay data (i.c. negative examples) alter the knowledge of the already learned classes, the gradient accumulation can be selectively blocked during the backward pass. The general idea is illustrated in Figure 2 While the original examples (X₂) normally flows forward and backward throughout the model, the replay examples (MІ) are passed forward, but, before the backward pass, the loss tensor is masked at the class level by resetting the gradient components corresponding to the classes in .M The negative replay implementation illustrated in FFiuure 2)is discussed in more details in Appendix C where preliminary tests on Core50 NC are also included.
Hereafter, we provide an alternative implementation embedded in ARI algorithm MMaltoni & Lomonaco 2019). whose update mechanism for the classification head weights allows very simple and efficient integration of negative replay. AR1 is flexible continual learning approach that can achieve state-of-the-art accuracy on complex CL benchmarks. In Appendii E, AR1 is shown to outperform several recent CL. algorithms of the difficult ImageNet- 1000 benchmark proposed by Masana et al. (2020).
ARI uses different mechanisms learn the classification head and the feature extractor weights. The feature extraction weights e are protected against forgetting: i) through the Synaptic Intelligence regularization technique (Zenke al. 2017) or ii) using a replay memory with a small learning rate (denoted as AR free in PPellegrini et al, (222)). The classification head weights & are managed by CWR. CWR a simple method aimed at addressing the score bias problem produced by imbalance learning during continual learning (Belouadah et al. 2020). CWR (Maltoni & Lomonaco 2019) maintains copy of the weights the classification head of the previous experience (1²²) and the start of each experience the classification head is reset and only weights of classes of the current experience are loaded from V. At the end of the experience, a weight consolidation phase takes place, where the weights & learned in the current experience consolidated with the weights vi" This is the point where positive and negative replay behaves differently.
In particular, during the consolidation phase, for each parameter group e associated to a class belonging to the current experience (c E Vi U UM), the mean of all the parameter group (((()) is calculated, and subtracted to all the parameters in the group, in order to force zero mean: th
4

---

Under review as a conference paper at ICLR 2022
Replay
D Salpere
Replay Boskhol
Original examples
Original exeippes
Feature extractor
Feature extractor
Feature extractor
hh)
(c)
Figure 2: Graphical representation of the negative replay idea. Green output neurons represent the classes present in the current experience, while red output neurons represent the replay classes, During forward (a) both the replay data and the original data from the current experience flow through the network. During backward, the original data flow through all the neurons of the classification head (b), while the replay data contribution is masked and data only flows through the neurons of classes belonging to the current experience (c).
2/ - ((b)). This prevents class bias problems due to the different magnitudes of the weights. Then, there are three possibilities, based on c:
1. c is a new class never seen before (c E V&s A U V/): in this case d is maintained as
2. is class seen before (c E V4. A U V:): the consolidation step is applied, so v to0 where Wpast, is parameter that balances the contribution of the past "laste w.r.t. the present, calculated as follows:
paste
posst
current,
where paste is the number data points of class e encountered in past experiences, while current, is the number of data points of class e present in the current experience.
3. is not in the current experience but is replay example (c & Vk A E MD:
in case of positive replay apply consolidation (step 2).
in case of negative replay vh is substituted with t (no contribution to the parameters d/e from replay examples).
The pscudo-code of the above weigh consolidation algorithms is reported in Appendix B noting, that the proposed embedding of negative replay ARI, the replay pattern can alter the feature extraction weights since CWR weight consolidation only "protects" the classification head However, in our experiments, we found that a more complex embedding of negative replay in AR1 where block the gradient propagation for negative patterns throughout the feature extraction layers performs very similarly, and therefore we opted for simplicity.
It is worth
4 EXPERIMENTS AND RESULTS
In this section, we describe the experimental setup used to validate the proposed negative replay. We focus on difficult continual learning scenarios, where data is high-dimensional, non-i.i.d. and the number of experiences is very large. Negative replay is compared with alternative strategies (e.g. positive replay) and the role of quality of generated data is investigated by also using, as negative replay patterns, real and random data.
4.1 EXPERIMENTAL SETUP
Datasets We performed our experiments on the CORe50 dataset (Lomonaco & Maltoni 2017) and ImageNet-1000 dataset (Deng et al. 2009). CORe50 dataset was specifically collected for continual learning (NI, NC, and NIC scenarios) and is composed of small video sessions (about 300 frames) of 50 objects taken from the point of view of person that handles them in the hand. Every class has 11 video sessions (a total about 3,300 images) with different backgrounds and illuminations. Eight video sessions for each class used for training, and 3 for testing. Images have size 128 128
5

---

Under review as a conference paper at ICLR 2022
pixels. ImageNet is composed of 1, 1,000 classes with about 1,000 patterns per class for training and 100,000 images for testing. All images are resized to x24x224 pixels.
Classifier architecture ARI algorithm was used with Synaptic Intelligence (SI) regularization
when trained without replay, and without protection on the feature extraction weights (AR free)) in case of positive and negative replay. In the experiments with CORe50 dataset we follow Maltoni & Lomonaco (2019) and Lomonaco et al. (2020) by employing a MobileNetVI network (Howard ct al.) As in Pellegrin al. (2020) and van de Ven et al. (2020), we opted for latent replay, that is replaying latent activations instead of input data. As described in Pellegrini et al. 22220) the
choice of the latent replay layer is related to tradeoff between accuracy and efficiency. For CORe50 experiments, as in Pelllegrini et al. (020), we used the conv5_4 _4 layer as latent replay layer, and the classifier was pretrained on ImageNer-100M 1000. We also substituted all the batch normalization layers of the network with batch renormalization (Loffe] 2017). For ImageNet-
we use ResNet- 18 (He) ct al., 2016) architecture. Following the benchmark proposed by Masana et al./ (2020) the model was not pretrained. To maintain compatibility with the experiments on CORe50, even on ImageNet-1000 we use latent replay, setting the replay layer on the fourth residual block of the network (after conv4_x using He et al. (2016) nomenclature).
Generative model architecture For the choice of a generative model we initially focused on two offe-offthh-arr approaches whose implementations are open source (van de Ven et al. [020 Shin 2017 Ayub Wagner 202T). However, since they were designed work in simpler settings (with a lower data dimensionality and a smaller number of experiences), we were not able to port and scale them to our complex setups. Therefore, we implemented a generative model by trying to combine the most promising techniques and ideas from different sources and control its overall memorylcomputttion complexity. In particular, taking inspiration from van de Ven et al. (2020) use a Variational Autoencoder (V (VAE) Kingma & Welling (2014) model, but unlike van de Ven let al.] (2020) we opted for a conditional VAE (cVAE) configuration /Shnn et al.] (2015). Moreover, partially blend the generator (encoder) with the classifier model: both the networks share the same feature extractor fg. Finally, instead of generating raw data, we generate activations at an intermediate "latent" level as suggested by Vvan de Ven et al. (2020). A detailed discussion on the architecture of the generator is provided in [[Appendix D including a pseudo-code that highlights the details of the interleaved training of the generator and the classifier.
al.
4.2 EXPERIMENTS ON THE NC SCENARIO
The first round of experiments has been performed on the NC scenario using CORe50 and ImageNet- 1000. For CORe50 the benchmark is composed of experiences: the first one contains 10 classes while the following contains five classes each. We used replay memory of 1,500 patterns, and (for generative replay) we inserted in each minibatch, of size 128, 14 replay patterns, and 114 patterns from the current experience. We train both the classifier and the generator for 4 epochs for each experience. Hypper-parameters of the classifier and generator are reported in Appendix F Fland Appendix G respectively.
experience
For ImageNet-1000 the benchmark follows the one proposed by Masana et al. (2020): the dataset is divided into 25 experiences of 40 classes each. We used a replay memory of 20,000 patterns, and (for generative replay) we inserted in each minibatch, of size 128, 36 replay patterns, and 92 patterns from the current experience. We did not expect negative replay to perform well in this setup, because each experience already contains 40 classes and, therefore, the learning-ing in in-isolation problem is here marginal. Nevertheless, we were interested in understanding in this setup, negative replay hurts the learning process or still provides minor benefits
The results are shown in Figure 3 and Table I In CORe50 the baseline with no replay (using the AR1 algorithm) reaches a final accuracy of about 60% while using replay raises the accuracy to more than 70% (Positive Replay Original Data- PR-OD). These were expected to be the lower and upper bounds of this experiment, respectively. However, because of the data degradation problem, performing positive replay with generated data (Positive Replay Generated Data PR-GD) performed significantly worse than the case with replay. Using replay in negative manner with generated data, as proposed in this work (NR-GD), only slightly decreases the final accuracy with respect to the upper bound PR-OD.

---

Under review as a conference paper at ICLR 2022
replay
Replay NR-GD (Negative Replly Generated Data)
PR-OD (Postive Replay Original Data)
CORe50
Imagenet-1000
0.8
0.9
0.40
0.8
0.7
0.35
0.7
0,80
0.6
Accuracy
Accuracy
0.6
5
10
0.4
0.4
0.3
0.3
0.2
0.2
0.1
Expertence
Experience
Figure 3: Overall accuracy on CORe50 NC scenario, using the whole test set (even at intermediate experiences) as defined in the CORe50 protocol (Lomonaco & Maltoni 2017) (left), and on ImageNet- 1000 using a growing test set as defined by Masana et al. (2020) (right). For a direct comparison of the two benchmarks, plot of the experiments on CORe50 NC using a growing test set is included in Appendix H Every experiment is averaged over runs using different seeds and class order. The standard deviation is reported in light colors. Better viewed on computer monitor.
Method
CORe50
ImageNeet-1000
No Replay
41.68 + 0.62
31.91 + 0.17
PR-OD (upper bound)
47.02 + 0.45
38.02 + 0.08
PR-GD
34.05 + 0.29
18.29 + 0.07
NR-GD
44.63 = 0.77
32.74 0.17
Table 1: Average accuracy all the experiences for the CORe50 and ImageNet- 1000 NC scenarios.
For ImageNet-1000, due to the complexity of the experiment and the fact that the network is fully trained only during the first experience (blocked after conv4_x in the following experiences) the final accuracy quite similar for all the methods (except PR-GD that performed far worse). However, in the first 10 experiences some differences can be appreciated: see the insect view in Figure right. The impact of the generated data quality negative replay is more evident in Table 1 using negative replay with generated data (in this case highly degraded) improve the average accuracy (calculated as the mean of the accuracy after each experience) of more than 24 points and the final accuracy of more than 10 points w.r.t. using replay data in positive manner. Furthermore, even if in this scenario the advantage of negative generative replay is little with respect the no replay case, note that negative replay is not hurting the training process even in scenarios where learning in isolation is not an issue.
4.3 EXPERIMENTS ON THE NIC BENCHMARK
CORe50 NIC-391 protocol is composed of 391 learning experiences, each containing examples of single class (300 frames of short video). This scenario is particularly challenging and prone learn-in isolation issues, hence we may expect the role of replay be more important here. In this scenario, we used replay memory of only 300 patterns. The minibatch size is 128, and when generative replay employed, we generate 64 patterns for every mini-batch (plus 64 from the current experience). flyper-parameter. of the classifier and generator reported in Appendix F and Appendix G respectively.
The results are shown in Figure 4 and they are quite in line with the previous experiment, but here the accuracy gaps grow and the benefit of replay is more evident. The proposed negative replay with generated data (NR-GD) performs quite well. about 10 points better than with replay and just less than 5 points worse than positive replay with real data, the upper bound. Using generated data in positive manner (PR-GD) is here even worse than in the NC case, because the data degradation
7

---

Under review as a conference paper at ICLR 2022
No replay
PR-GD (Positive Replay Generated Data] (Negative Replay Data)
(Pasilive Replay Original Data)
Appuracy
50
100
160
200
250
300
350
Experience
Figure 4: Overall accuracy CORe50 NIC391 scenario. using the whole test set defined in the CORe50 protocol (Lomonaco & Maltoni 2017). Every experiment is averaged over 3 runs using different seeds and class order. The standard deviation is reported in light colors. better viewed computer monitor.
is amplified with so many iterations: PR-GD is losing 30 points w.r.t. not using replay at all, and performs about 40 points worse than using the same replay data with the proposed generative negative replay approach.
4.4 ABLATION STUDY
The effect of generated data quality on negative replay is investigated by performing two further experiments: NR-OD uses original data (max. quality) for negative replay, while NR-RD uses randomly generated replay data, obtained by uniform random sampling in the latent replay layer and assigning to each data point a random class label. Since in our experiments we replay hidden features, order to produce reasonable replay data we first calculated the range of latent activations on sample dataset, and then we set our random generator to produce values in the range: 0 (since we use ReLU activation functions) 90th percentile of the real activation values. We used CORe50 NC and CORe50 NIC in these experiments.
The results are reported in Table 2] Surprisingly, even with random replay data (that we assume to be the worst degradation possible), negative replay is still able to perform better than replay. Furthermore, the difference between original and generated data is minimal, thus proving that negative replay is tolerant in terms of data quality. Note that in both the experiment using random data with negative replay performs way better than using generated data in a classical (positive) manner (PR-GD in previous figures). Comparisons in all the benchmarks of all the experiments (positive and negative replay with original, generated, and random data) are reported in Appendix H
Method
CORe50 NC
CORe50 NIC
No Replay
60.99 + 0.49
52.71 1.02
NR-OG
68.60 + 1.38
67.93 + 0.31
NR-GD
68.87 + 0.88
61.46 + 0.67
NR-RD
64.05 + 0.71
58.85 + 0.58
Table 2: Final accuracy on CORe50 NC and NIC using original (NR-OD), generated (NR-GD), and random (NR-RD) data with negative replay. The results with no replay reported references. Every experiment is averaged over runs using different seeds and class order.
8

---

Under review as a conference paper at ICLR 2022
5 RELATED WORKS
The of negative examples learn more discriminative class boundaries can be traced back to one-class support vector machines (SVM) (Chen et al. 2001), where the data points belonging to the other classes in the training set are used as negative examples. Malisiewicz et al. (201T) proposed using an ensemble of one-class SVMs instead of single multi-class classifier. This approach operates in scenario which is similar to the experiments on the CORe50 NIC benchmark, whose experiences contains only one class and all the replay data points (possibly belonging to many past encountered classes) are used as negative examples. The use of negative examples can also be seen as kind of contrastive learning (Khosla al. 2020). where negative examples are used to cluster embeddings of data points of the same class while moving away embeddings of data from different classes.
Masking parts of a neural network has been experimented before in continual learning. Wortsman et al. (2020) masked the weights of a randomly initialized neural network in order to find a sub-network that yields good performance for a particular task. The loss masking proposed for standalone negative replay introduced in section 3.3 and Appendix Cl (without using any continual learning strategy) is similar to the masking method proposed by Masana al. (2021). In that work, each feature can be used normally, masked (not used), or used only during forward (no modification of the related parameters during network update).
Generative replay for continual learning was first introduced by Shin et al. (2017) who proposed Deep Generative Replay (DGR), using a generative adversarial network (GAN) (Goodfellow et al.) 2014). Many works on generative replay (Wu et al., 2018 Ostapenko et al., 2019) use GANs as generative models, but GANs are usually slow and complex to train, even in non-incremental scenarios, Ayub & Wagner (2020) proposed to use autoencoders, however that approach requires maintaining a generative model for every experience, making it not scalable to long incremental sequences. |emkker& & Kanan (2018) and van de Ven et al. (2020) proposed continual learning framework inspired to biological brain functionalities and memories. In particular, (van de Ven et al. 2020) showed significant results in continual learning scenarios with dozens of experiences. However, this approach was not tested on high-dimensional data and in much bigger scenarios.
CONCLUSION
In this paper, we addressed the problem of continual learning with generative replay, focusing on the obstacles of generative replay in complex scenarios. Our experience confirms that incrementally training generator over a long number of experiences with high dimensional data is very challeng- ing problem and remains an open issue. Therefore, instead of trying to design a better generative model, we focused on classifier training. We found that even inaccurate replay data can be useful to contrast the learning in isolation problem, especially in scenarios where only a limited number of classes is present in each experience. We called this approach negative replay since the replay data is used as negative examples when the model is trained with data from the current experience. We validated negative replay using complex continual learning scenarios, with high dimensional data and hundreds of incremental experiences. The results show that using negative replay largely improves classification performances w.r.t. using the generated data in a traditional fashion. We also investigated the impact of generated data quality, by considering the two extremes of using original data and random data for negative replay, and, surprisingly, we found that negative replay effective even using random replay data.
Since negative replay can be easily applied to other continual learning strategies (besides AR1), we believe that many other CL approaches may benefit from our proposal, especially when complex scenarios are employed. Moreover, negative replay could be used in the pre-training phase of large models, possibly making them more robust to noise or degraded data. Finally, is worth noting that dealing with imprecise replay data can be viewed as a biological feature since human's 's memory is far from being accurate, but is thought to be essential to consolidate learning (van de Ven et al. 2020). therefore investigating the role of negative replay- mechanisms in biological learning could be an interesting research direction for computer scientists and neuroscientists.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
The source code of the project, alongside configuration files for the reproducibility of experiments, is included in the additional materials. A public version of the code, based on the Avalanche framework (Lomonaco et al., 202I) will be released upon publication. The datasets used are publicly available and can be downloaded from the respective official websites! 1.2, Both the datasets are preprocessed normalizing them using statistics derived from ImageNet- 1000. Images from ImageNet- 1000 were randomly cropped and resized to the final dimension of 224 224, then horizontally flipped with probability of 50%.
REFERENCES
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. H Wallach, H Larochelle, A Beygelzimer, d'lché-Bir Buc, E Fox. and R Gamett (eds.). Advances in Neural Information Processing Systems 32, pp. 1 11816-18255 Curran Associates, Inc., 2019.
A. Ayub and A. R. Wagner. EegnitvvelyInsspree Model for Incremental Learning Using Few Ex- amples. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, 2020.
Ali Ayub and Alan R. Wagner. EEC: Learning to encode and regenerate images for continual learning.
In International Conference on Learning Representations, 2021.
Eden Belouadah and Adrian Popescu. 112m: Class incremental learning with dual memory. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 583-592, 2019.
Eden Belouadah, Adrian Popescu, and Ioannis Kanellos. A comprehensive study of class incremental learning algorithms for visual tasks. Neural Networks, 2020.
Francisco M Castro, Manuel Marin-Jiménez, Nicolás Guil, Cordelia Schmid, and Karteek Alahari. End-to-end incremental learning. In Proceedings of the European conference on computer vision (ECCV), pp. 233 233-248, 2018.
Arslan Chaudhry, Puneet K. Dokania, Thalaiyasingam Ajanthan, and Philip H. Torr. Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 532-547, 2018
Yunqiang Chen, Xiang Sean Zhou, and Thomas Huang. One-class svm for learning in image re- trieval. In Proceedings 2001 International Conference on Image Processing (Cat. No. 01 01CH37205), volume 1, pp. 34-37. IEEE, 2001.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: 10. 109/CVPR.2009 5506848.
Prithviraj Dhar, Rajat Vikram Singh, Kuan-Chuan Peng, Ziyan Wu, and Rama Chellappa. Learning without Memorizing. In CVPR, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Tyler I. Hayes, Giri Krishnan, Maxim Bazhenov, Hava T Siegelmann, Terrence J Sejnowski, and Christopher Kanan. Replay in deep learning: Current approaches and missing biological elements. arXiv preprint arXiv:2104.041. 2021.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778, 2016.
hnttps://vloooncco.iiiii wtt..aiithh......eettteeeee
httpp:://ww.tt www.::wwwmmmaaaee mage-net.orr.
10

---

Under review as a conference paper at ICLR 2022
Kathryn Hempstalk and Eibe Frank. Discriminating against new classes: One-class versus multi- classification. In Wayne Wobeke and Mengjie Zhang (eds.), Al 2008: Advances in Artificial Intelligence, pp. 325 336, Berlin, Heidelberg, 2008. Springer Berlin Heidelberg. ISBN 978-3-540- 89378-3.
Higgins Matthey, Pal, Christopher P. Burgess, Xavier Glorot, M. Botvinick, Mohamed, and Alexander Lerchner. bet:-vae: Learning basic visual concepts with constrained variational framework. In Proceedings of the 5th International Conference on Learning Representations, 2017.
Saihui Hou, Xinyu Pan, Chen Change Loy, Zilei Wang, and Dahua Lin. Learning a unified classifier incrementally via rebalancing. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 831-839, 2019.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint MXXXv:1704.04880 2017.
Huaibo Huang, Zhihang Li, Ran He, Zhenan Sun, and Tieniu Tan. Introvae: Introspective variational autoencoders for photographic image synthesis. arXiv preprint 1rXiv:1807.0663 2018.
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in normalnodd models. arXiv preprint arXiv:/702.0333 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pultern Recognition, pp. 4401-4410, 2019.
Ronald Kemker and Christopher Kanan. FearNet: Brain-Inspired Model for Incremental Learning. In ICLR, 2018.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint arXiv:2004. 1162, 2020.
Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinske Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. PNAS, 114(13)3521-3555 2017.
Timothée Lesort, Hugo Caselles- Dupreé, Michael Garcia-Ortiz, Andrei Stoian, and David Filliat. Generative Models from the perspective Continual Learning. Proceedings of the International Joint Conference on Neural Networks, 2018. doi: 10.1109/JCCNN.200 20t../8199..8.19998888 .8851986.
Zhizhong Li and Derek Hoiem. Learning without Forgetting, In European Conference on Computer Vision, Springer, pp. 614-629, 2016.
Vincenzo Lomonaco and Davide Maltoni. CORe50: A New Dataset and Benchmark for Continuous Object Recognition. CoRL. 2017.
Vincenzo Lomonaco, Davide Maltoni, and Lorenzo Pellegrini. Sihearsal-Free Continual Learning over Small Non-L.L.D Batches. CVPR Workshop on Continual Learning for Computer Vision, 2020.
Vincenzo Lomonaco, Lorenzo Pellegrini, Andrea Cossu, Antonio Carta, Gabriele Graffieti, Tyler L. Hayes, Matthias De Lange, Mare Masana, Jary Pomponi, Gido van de Ven, Martin Mundt, Qi She, Keiland Cooper, Jeremy Forest, Eden Belouadah, Simone Calderara, German Parisi, Fabio Cuzzolin, Andreas Tolias, Simone Scardapane, Luca Antiga, Subutai Amhad, Adrian Popescu, Christopher Kanan, Joost van de Weijer, Tinne Tuytelaars, Davide Bacciu, and Davide Maltoni. Avalanche: An End-to-Ent Library for Continual Learning. CLVision Workshop at CVPR 2021, 2021.
11

---

Under review as a conference paper at ICLR 2022
Tomasz Malisiewicz, Abhinay Gupta, and Alexei A Efros. Ensemble of exemplar-svmn for object detection and beyond. In 2011 International conference on computer vision, pp. 89-96. IEEE, 2011.
Davide Maltoni and Vincenzo Lomonaco. Continuous Learning in Single-Incremen Task Scenarios. Neural Networks, 1 116:56-73, 2019.
Marc Masana, Xialei Liu, Bartlomiej Twardowski, Mikel Menta, Andrew D Bagdanov, and Joost van de Weijer. inssemnnrrme learning: survey and performance evaluation. arXiv preprint arXiv:2010.15277, 2020.
Mare Masana, Tinne Tuytelaars, and Joost van de Weijer. Temary feature masks: zero-forgetting for task-incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3570-3577 2021.
Michael McCloskey and Neal J. Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. Psychology of Learning and Motivation. Academic Press, 1989. doi: http:://c..omg111100006677777777
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxiliary classifier gans. In International conference on machine learning, pp. 2642-2651. PMLR, 2017.
Oleksiy Ostapenko, Mihai Puscas, Tassilo Klein, Patrick Jahnichen, and Moin Nabi. Learning to remember: A synaptic plasticity driven framework for continual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11321-11329, 2019.
Lorenzo Pellegrini, Gabriele Grafficti, Vincenzo Lomonaco, and Davide Maltoni. Latent replay for real-time continual learning. IROS, 2020.
Sylvestre-Alvin Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCaRL: Incremental Classifier and Representation Learning. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual Learning with Deep Generative Replay. In Guyon, U V Luxburg. S Bengio, Wallach, R Fergus, Vishwanathan, and R Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 2990-2999. Curran Associates, Inc., 2017.
Kihyuk Sohn, Honglak Lee, and Xinchen Yan. Learning structured output representation using deep conditional generative models. In Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc., 2015.
Kevin Thandiackal, Tiziano Portenier, Andrea Giovannini, Maria Gabrani, and Oreun Goksel. Match what matters: Generative implicit feature replay for continual learning. arXiv preprint 20X::22006.05300, 2021.
Gido M van de Ven and Andreas S Tolias. Three scenarios for continual learning. In Continual Learning Workshop NeurIPS, 2018.
Gido M. van de Ven, Hava T. Siegelmann, and Andreas Tolias. Brain-inspired replay for continual learning with artificial neural networks. Nature Communications, 11, 2020. doi: 10.1038/ 1414677-00-178666.
Mitchell Wortsman, Vivek Ramanujan, Rosanne Liu, Aniruddha Kembhavi. Mohammad Rastegari, Jason Yosinski, and Ali Farhadi. Supermasks in superposition. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 15173-15184
Curran Associates, Inc., 2020.
Chenshen Wu, Luis Herranz, Xialei Liu, Joost van de Weijer, Bogdan Raducanu, et al. Memory replay gans: Learning to generate new categories without forgetting. Advances in Neural Information Processing Systems, 31:5962-5972, 2018.
12

---

Under review as a conference paper at ICLR 2022
Yue Wu, Yinpeng Chen, Lijuan Wang, Yuancheng Ye. Zicheng Liu, Yandong Guo, and Yun Fu. Large scale incremental learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pp. 374-382, 2019.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual Learning Through Synaptic Intelligence. In International Conference on Machine Learning, pp. 3987-3995, 2017.
A NI, NC, AND NIC DEFINITIONS
Given the definitions of section 2 the NI (New Instances), NC (New Classes), and NIC (New Instances and Classes) continual learning scenarios can be defined based on the labels 3 contained in the experiences (k (1,.., Ng) with Ng the total number of experiences) as follows:
New instances (NI) also known as domain intremmentee learning, where all the labels are known from the first experience, and the successive experiences, only new instances of the same classes are included. Formally, we could define the NI scenario as:
Y n 3/+ = V) for k: = {1....,NM Ng).
meaning that every possible label of the entire dataset must be present in the first experience. New classes (NC) also known as class- daas-incremmntall learning, where each experience includes data of classes not present in any other experience. Formally, we can define the NC scenario as:
k-1 34. U y. =0 0 for k = NE}. 1=1
New instances and elasses (NIC) where a new experience can contain already seen classes, new classes, or mix of the two. This is the most natural scenario since in the real world an agent may sense both known and unknown objects. Formally the NIC scenario can be defined as:
R-1
36 : D&e O Ut: + and Bj V ny, + 3/3.
(10)
Meaning that there least one experience that contains classes already seen in the past (left part) and at least one experience that contains classes not present in the first experience (right part).
B PSEUDO-CODE OF THE WEIGHT CONSOLIDATION PHASE
Algorithm 1 Weight consolidation
Require: th v,, Ve. M!
1: for each class E Ve U UM! do
2: we = -1(0))
3: if Ac U y then
1/e 6ec ue paart t Wpast.
4:
5:
end if if c & Venc ACE My then
6:
7:
if positive replay then e/
1/" Wppaste tip
8:
Wpasle
9:
end if negative replay then d th
11:
12:
end
13:
end if
14:
end for
15: e
13

---

Under review as a conference paper at ICLR 2022
C ST NEGATIVE REPLAY IMPLEMENTATION
To validate the proposed negative replay approach, we implemented version of negative replay that does not depend on any specific continual learning strategy. As discussed in the main text, the main idea behind negative replay is to avoid replay data altering the knowledge of the already learned classes. In other words, the replay data cannot change the weights of the already learned classes.
This behavior can be obtained by selectively blocking the gradient accumulation during the backward pass. The original examples (22) flow normally through the network, accumulating the gradient in all the feature extractor weights (1²). On the other hand, the replay data points (MF)) participate in the loss calculation, but their backward contribution limited to the weights associated with the output neurons of the original data classes. The loss tensor associated to replay data is thus masked class level, resetting the gradient contribution zero for all the classes in M.. The procedure is graphically explained in Figure 5]
Note that the proposed standalone negative replay implementation does not provide any specific mechanism to counteract catastrophic forgetting problem, and since it applies negative replay on top of the naive CI. strategy we denote the resulting strategy as naive negative replay.
In line with the experiments included in the main text, we compared naive negative replay (NR-GD) against naive with no replay (lower bound), naive with positive generative replay (PR-GD), and naive with positive replay with original data (PR-OD, upper bound). We used the CORe50 NC benchmark to perform this experiment.
The results are shown in Figure 6) ARI, but the relative ranking of the different replay approaches in maintained. In particular, negative replay performs better w.r.t. replay (naive) using the generated data in a positive manner (naive PR-GD). In this scenario the gap between negative replay with generated data and positive replay with real data is larger than when using AR1, because replay is the only methods contrast forgetting and the replay data quality is more relevant.
As expected, the overall accuracy with naive is lower than with
Loss original data
Loss replay data
0
Feature extractor
Feature extractor
(a)
(b)
Figure 5: Graphical representation of the naive negative replay. The output neurons are depicted in green for neurons associated with original data, and in red for output neurons associated to replay data. The loss vector of original data is backpropagated through all the weights of the network (yellow arrows), so the gradient accumulated by every weight. On the other hand, the loss vector replay data is masked, and the loss is only backpropagated through the output neurons corresponding to the original classes. Thus, the gradient accumulated only on the classification head weights associated with the classes present in the current experience.
D DETAILS OF THE GENERATIVE MODEL IMPLEMENTATION
We designed our generative model using different insights from previous works in the fields, bringing together different ideas and proposals. We extensively tested the generative model alone find the better combination of building blocks that yield the best performance. Our design choices have been also influenced by the computation complexity since aim is develop a (near) real-time system.
14

---

Under review as a conference paper at ICLR 2022
Nalve
Nalve PR-GD
Nalve PR-OD
Nalve NR-GD
0.3
02
0.1
Experience
Figure 6: Overall accuracy CORe50 NC scenario of the naive negative replay, using the whole test set as defined in the CORe50 protocol Lomonaco & Maltoni (2017). Every experiment is averaged over 3 runs using different seeds and class order. The standard deviation reported in light colors. Better viewed on a computer monitor.
This is particularly hard constraint since many incremental generative replay methods are based on generative adversarial networks (GANs) (Goodfellow et al. 2014), which notably have long training phases and often suffer from instabilities due to the adversarial nature of the training procedure.
As discussed in the main text, took inspiration from some Mate-of-the-arr methods, trying to combine promising techniques and ideas from different sources. Taking inspiration from van de Variational Autoencoder (VAE) Kingma & Welling (2014) model, but
Ven al. (2020) unlike van de Ven et al. (2020) we opted for conditional VAE (cVAE) configuration (Shnn et al. (2015). So, while in van de Ven al. (2020) mixture of Gaussian is used to sample latent vectors and soft labels are provided to the classifier itself, in our approach the latent vector is sampled from the normal distribution ((0, 1) and conditioned the desired class. This results in faster and less complicated sampling of replay pattern. Moreover, as in van de Ven et al. (2020) we partially blend the encoder part of the generative model with the classifier model: in fact both the network share same feature extractor f4. For the classifier, this branch is connected with the classification head cy while, for the generator, is connected with some other layers that transform the feature into a latent vector The bifurcation is located in the latent replay layer. The resulting on- the-loop training of the generative model consistent with brain structures and neuroscience's findings (van de Ven et al. 2020).
Since we use a cVEE, the objective for the generative model can be expressed as:
-" arg min - (x (log Pe (=lv²)) DEL (xc
(11)
7.6
where (af; v are the data point and the label of the i-th pattern of the k-th experience, and the DKL term represents the Billbackk-Libble divergence between the latent space distribution and the target distribution p(2) = =N(0
The terms of Equation II determine two losses:
Lrecon = || - (a(
(12)
CKL. (x
(13)
We also add another loss term, denoted as classification loss, which is similar to the classifier loss adopted the AC-GAN model (Odena et al. (2017). The rationale is to guide the generative model to produce data that are not only visually similar to the original ones (1.2 loss) but that is also classified
15

---

Under review as a conference paper at ICLR 2022
Prcoon
Encoder (gx)
Decoder (pc)
"Cat"
LKL
Ldass
Classifier (fo)
Predicted label
Figure 7: A visual schema of the generative model training. Losses are represented by dashed arrows. The shared branch of the classifier and the encoder are depicted using the same color (yellow). The encoder and the classifier's additional layers are drawn in red and blue respectively.
by the current classifier in the same way. Hence, use fe as "auxiliary"" classifier, adding the following term to the generator's 's loss:
Lelass - log (e
(14)
which represents typical negative log-likelihood classification loss. Note that the parameters O of the classifier are not trained in this phase, since only the generative model is updated. Overall, the generative model is trained using the following loss function:
LGM = Crecon + 3BKL + nClaase
(15)
where B is a hyper-parameter inspired to the 3-VAE framework (Higgins et al. 2017). and v is hyper-parameters that weights the importance the classification loss.
A visual representation of generative model training is shown in Figure
To keep notation light, in the equations above the replay memory not used, but it is trivial to include patterns from the replay memory, since there is no distinction in the generative model training procedure between current and replay data.
Note that the utilization of raw images is not mandatory for the method, and any intermediate (or latent representation) can be used, making our proposal compatible with latent replay methods (Pellegrini et al. 2020; van de Ven al. 2020). In fact, in the of latent replay, the data points x in the above equations can be simply substituted with for (æt), where for is the set of feature extraction layers before the latent replay layer.
The blending of part of the generative model into the classifier poses some difficulties in the training, especially regarding the balancing of the two models and how to train each of them without destructive opted for blocking model parameters when
inference the other. After some initial experiments, the other model is trained. Detailed pseudo-code for the proposed negative generative replay strategy provided in Algorithm 2
16

---

Under review as a conference paper at ICLR 2022
Algorithm 2 Generative negative replay
1: fe +- RANDINIT or PRETRAINED
2: 9a < RANDINIT or PRETRAINED
3: .M < 0, M +- 0
4: memory size
5: for cach & from 1 tp Ne do
6: if > then
7: SAMPLE (21, 2nd (0, 1)
8: SAMPLE (cI, CR) ~ UK=1 }4.
9:
BLOCK generator parameters (7,5)
10:
POPULATE Mi = pe(zylo,)) j = {,, .....R R}
11:
POPULATE M (c1,-,c)))
12:
end
classifier training
14:
6'=
BLOCK generator parameters (7,5)
16:
UNLOCK classifier parameters (6. 1
o², 15* OPTIMIZE(fe. Xr U UM M 3% M using Equation 4
18:
WFEIHTTCONSSLDDDATTONN v Jkkk M (see Algorithm
generator training
20:
BLOCK classifier parameters (Ø, ./)
21:
UNLOCK generator parameters (7; E)
22:
2-5" OPTIMIZE(P0 X U M D'k U M using Equation 15
23:
end for
E VALIDATION OF ARI ON IMAGENET-1000
To validate the chosen AR1 algorithm we performed a test on competitive benchmark on ImageNet- 1000, following the NC benchmark proposed by Masana et al. (2020), which is composed of 25 experiences, each of them containing 40 classes. The benchmark is particularly challenging due to large number of classes (1,000), the incremental nature the task (with 25 experiences), and the data dimensionality of 224 x 224 (as with ImageNet protocol).
With this experiment we want to assess the performance of ARI in a complex continual learning scenario, validating the choice of ARI as the baseline algorithm which the tests on negative replay are conducted. In this experiment, we tested ARI against both regularization. based methods (Dhar Kirkpatrick et al.) 2017; & Hoiem 2016) and replay- based approaches (Belouadah &
let 2019
Popescul 2019 Castro et al.] 2018 (018|CCoe Chaudhry et al. 2018 Hou et al.) [019 Rebuffi et al. 2017] Wu al. (019)). We the same classifier (ResNet- (He et al.) (016)) and the memory size for all the tested methods (20,000 patterns, 20 per class); for the regularization- babsed approaches, the replay is added as an additional mechanism.
For ARI, we trained the model with an SGD optimizer. For the first experience, we used an aggressive learning rate of 0.1 with momentum 0.9 and weight decay of 10-4 We multiply the initial learning rate by 0.1 every 15 epochs. We trained the model for a total of 45 epochs, using a batch size of 128. For all the subsequent experiences we used SGD with a learning rate of 5 10 for the feature extractor's 's parameters & and 5 10-2 2 for the classifier's parameters vh. We trained the model for 32 epochs for each experience, employing a learning rate scheduler that decreases the learning rate as the number of experiences progresses. This was done to protect old knowledge against new knowledge when the former is more abundant than the latter. As the first experience, the batch size was set to 128, composed of 92 patterns from the current experience and 36 randomly sampled (without replacement) from the replay memory.
The results shown in Table 3
Replay-based methods exhibit the best performance, with iCaRL and BiC exceeding a final accuracy of 30%. AR1 outperforms all the baselines (33.1%) demonstrating the validity of this approach also in difficult continual learning benchmarks. However, considering
17

---

Under review as a conference paper at ICLR 2022
Method
Final Accuracy
Fine Tuning (Naive)
27.4
EWC-E
(Kirkpatrick al.
2011
28.4
RWalk
Chaudhry et al.
2018
24.9
LwM
(Dhar al.
2019
17.7
LwF
Li & Hoiem
2016
19.8
iCaRL
RReuuffi et al.
2017)
30.2
EEIL
Castro et al.
2018
25.1
LUCIR (Hou al.
2019)
20.1
11.2M
(Belouadah & Popescu
2019)
29.7
(Wu al. 2019)
32.4
AR1 (Maltom & Lomonaco
2019)
33.1
Table 3: Final accuracy on ImageNet-1000 following the benchmark of Masana et al. (2020) with 25 experiences composed of 40 classes each. For each method, replay memory of 20,000 pattems used (20 per class at the end of training). Results for other methods reported from Masana et al. (020))
that top-1 ImageNet accuracy for ResNet-18 18 when trained on the entire dataset is 69.76% even the best methods the accuracy gap the continual learning setup is very large. This suggests that continual learning, especially in complex scenarios with a large number of classes and high dimensional data, is far be solved, and further research should be devoted to this field.
Accuracy taken from the torchvision official page:
model hhml
18

---

Under review as a conference paper at ICLR 2022
F
CLASSIFIER HYPER-PARAMETERS
F.1
CORE5O NC
Hyper-panee Saper-parrmettr
Value
optimizer
SGD
momentum
0.9
weight decay minibatch size
10-1
Common
128
SI (Synaptic Intelligence)
8 10
SI Fisher matrix clip value
10
SI Fisher matrix multiplier
10
epochs
1st experience
Ir ø (feature extractor)
3 10-1
Ir v (classification head)
3 10
nr. epochs
4
Following experiences
Ir á (feature extractor)
3 10
Ir v (classification head)
3.10
Table 4: prrperpprrmmttes the model trained with no replay. Common syper-parameten are the same for each experience, 1st experience hyper-parameters are used in the first experience, the following experience hyper-parameters are used in all the following experiences.
Hlyer-ppaaaeee
Value
optimizer
SGD
momentum
Common
weight decay minibatch size
10-4
128
SI (Synaptic Intelligence)
disabled
nr. epochs
1tt experience
Ir ø (feature extractor)
3 10 -2 30-0
Ir v (classification head)
epochs
Ir 6 (feature extractor)
5 10 - 5 10
Ir (classification head) memory size
Following experiences
1,500
replay pattern in the minibatch
latent replay layer
conv5.4
Table Syyper-parameter of the model trained with replay (generative replay, random data, and real data). Common parammerssss are the same for each experience, 1st experience hyper-paramete parameters are used the first experience, the following experience experiences.
are used in all the following
19

---

Under review as a conference paper at ICLR 2022
F.2
IMAGENET- 1000 NC
Hyper-parameter
Value
optimizer
SGD
momentum
0.9
Common
weight decay minibatch size
10-4
128
SI (Synaptic Intelligence)
disabled
nr. epochs
45
Ir é (feature extractor)
10-1
1st experience
Ir ga (classification head) Ir scheduler
10-1
Ir 0.1 every 15 epochs
nr. epochs
32
Ir & (feature extractor) Ir và (classification head) scheduler
5 10
Following experiences
5 10-2
see Equation 16
Table 6: Hyper-panne parameters of the model trained with replay. Common hyper-parameter are the same for each experience, 1st experience hyper-pane parameersseeeess are used in the first experience, the following experience hyper-parameters are used in all the following experiences.
Hyper-parameter
Value
optimizer
SGD
momentum
0.9
Common
weight decay
10 10-4
minibatch size
128
SI (Synaptic Intelligence)
disabled
nr. epochs
Ir (feature extractor)
10-1
1st experience
Ir & (classification head) scheduler
10-1
Ir every 15 epochs
nr. epochs
Ir (feature extractor)
5 10
Ir d (classification head)
5 10
Following experiences
scheduler memory size replay pattern in the minibatch latent replay layer
see Equation 16
20,000
36
layer4 (4th resnet block)
Table 7: Hyper-parameters the model trained with replay (generative replay and real data).
Common hyper-parameters are the same for each experience, 1st experience hyper-parameters used in the first experience, the following experience hyrer-pasameters are used in all the following experiences.
Due to the complexity of the ImageNet- 1000 scenario, we found it useful to use a learning rate scheduler that decreases the learning rate as the number of experiences progresses. The scheduler can be formalized as:
Ir Iraitt (-I+e9. 1.544 +1).
0.9
(16)
where indicates the index of the current experience.
20

---

Under review as a conference paper at ICLR 2022
F.3
CORE50 NIC
Alyper-paramenn
Value
optimizer
SGD
momentum
0.9
weight decay
10
Common
minibatch size
128
SI (Synaptic Intelligence) A
2.3 10
SI Fisher matrix clip value
10
SI Fisher matrix multiplier
2 10
nr. epochs
1st experience
Ir é (feature extractor)
10
d (classification head)
10-3
epochs
4
Following experiences
Ir é (feature extractor)
10
(classification head)
10
Table 8: Hyper-paane parameters of the model trained with replay. Common the same for each experience, 1st experience nyper-parametens are used in the first experience, the following experience hyper-parameters are used in all the following experiences.
are
Hyper-paramete
Value
optimizer
SGD
momentum
0.9
Common
weight decay
10-4
minibatch size
128
SI (Synaptic Intelligence)
disabled
nr. epochs
1st experience
Ir é (feature extractor)
10
Ir 1 (classification head)
10
epochs
Ir ó (feature extractor)
10 10-4
Ir v (classification head) memory size replay pattern the minibatch latent replay layer
10
Following experiences
300 (N/A for random data)
64 (21 for random data) conv5_4
Table 9:
data). Common of the model trained with replay (generative replay, random data, and real are the same for each experience, 1st experience hyper- hyper-paramed parameters are used the first experience, the following experience hyper-parameters are used in all the following experiences.
F.4 ON THE AMOUNT OF REPLAY DATA IN THE MINIBATCH
The amount of replay data included in each minibatch has a direct impact on the performance of the continual learning strategy adopted. We observed that the optimal value changes with the quality of the replay data, and that large amount of degraded replay data in each minibatch may decrease disruptively the performance of the model.
We compared different original/replay proportions, finding that when using real replay data, the model is not much sensitive the amount of replay data in the minibatch and different proportions work well: we empirically noticed a peak of performance around a 50-50 split. Using generated (degraded) random data quite different. We noticed that if the data used is highly degraded the
21

---

Under review as a conference paper at ICLR 2022
maximum gain in performance is when 10-30% replay data are added. Exceeding 30% usually leads to degradation of performance, and if the amount of replay data is still higher (depending on the replay data quality) the accuracy of the model can be lower than not using replay data.
G GENERATIVE MODEL HYPER-PARAMETIE
G.1
CORE5O NC
Hyper-parameter
Value
optimizer
Adam
betas
0.9 0.999
weight decay minibatch size latent space dim. B
0
128
100
Common
0.1
7 Ir Ir scheduler nr. epochs
0.01
10-3
None
4
Following experiences
replay patterns in the minibatch
27
Table 10: prreeeppraaeters of the generative model trained CORe50 NC. Common hyper-
parameters are the same for each experience, while following experience cyper-parameters are used in all the experiences except the first one.
G.2 IMAGENET- 1000 NC
Hyper-parameter
Value
optimizer momentum weight decay minibatch size latent space dim. B
SGD
0
128
100
Common
0.25
0.01
Ir Ir scheduler epochs
see Equation 16
32
Following experiences
replay patterns the minibatch
36
Table 11: ppper-parameters of the generative model trained on ImageNet- 1000 NC. Common hyper- parameters are the same for each experience, while following experience in all the experiences except the first one.
are used
22

---

Under review as a conference paper at ICLR 2022
G.3
CORE50 NIC
Hy plrra-errreeter
Value
optimizer
Adam
betas
0.9 0.999
weight decay minibatch size
0
128
latent space dim.
100
Common
B
0.1
T
0.01
Ir
10-3
scheduler nr. epochs
None
4
Following experiences
replay patterns the minibatch
64
Table 12: Tlyee-parameters of the generative model trained on CORe50 NIC. Common hyper-
parameters are the same for each experience, while following experience hyper- syper--arameters are used in all the experiences except the first one.
H ADDITIONAL PLOTS
replay PR-OD (Posilive Repley Original Data)
Replay Generated
NR-GD (Negalive Repley Genersted Data)
1.0
0.9
Apsuracy
2
3
Experience
Figure 8: Overall accuracy on the CORe50 NC scenario, using a growing test set. After each experience, the model evaluated using test composed of only data belonging the classes so far, similar to the benchmark proposed by Masana et al. (2020). Every experiment is averaged over 3 runs, with different seeds and class order. The standard deviation is reported in light colors. Better viewed if zoomed on computer monitor.
23

---

Under review as a conference paper at ICLR 2022
...... No repley
NR-GD (Négalive Repley Generated Data)
PR-OD (Poeilive Repley Original Data)
NR-OD (Negalive Repley Original Data)
PR-GD (Poeilive Repley Generated Data)
NR-RD (Negative Replay Random Data)
Accuracy
Experiance
Figure 9: Overall accuracy on the CORe50 NC scenario for all the experiments performed this work (included random data and negative replay with original data). After each experience, the model was evaluated using the cumulative test set as proposed by Lomonaco & Maltoni (2017). Every experiment is averaged over 3 runs, with different seeds and class order. The standard deviation is reported in light colors. better viewed if zoomed on computer monitor.
No replay
NR-GD (Negative Raplay Generated Data)
PR-OD (Positive Replay Original Data) (Posilive Repley Data)
NR-OD (Negalive Repley Ofiginal Data)
(Negative Replay Dats)
0.9
0.400
0.375
0.7
0.325
0.300
0.275
Accuracy
7
10
0.4
0.1
10
15
20
Experience
Figure 10: Overall accuracy on the ImageNet- 1000 NC scenario all the experiments performed in this work (included random data and negative replay with original data). After each experience, the model was evaluated using the whole test set as proposed by Masana et al. (2020). Every experiment is averaged over 3 runs, with different seeds and class order. The standard deviation is reported in light colors. better viewed if zoomed on computer monitor.
24

---

Under review as a conference paper at ICLR 2022
No replay
NR-GD (Negative Generated Dats)
PR-OD (Posilive Repley Original Data)
NR-OD (Negative Replay Original Data)
PR-GD (Pesilive Replay Generated Duta)
NR-RD (Negalive Replay Random Data)
Accuracy
50
100
150
200
250
300
350
Experience
Figure 11: Overall accuracy on the CORe50 NIC scenario for all the experiments performed this work (included random data and negative replay with original data). After each experience, the model was evaluated using the cumulative test set as proposed by [omonaco & Maltoni (2017). Every experiment is averaged over 3 runs, with different seeds and class order. The standard deviation is reported in light colors. better viewed if zoomed on a computer monitor.
25


Reviews:
[{"review_id": "txFe-Og7z8t", "reviewer": "Reviewer_9MUi", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The paper is well-written and states the problem setting quite well.\n2. Using generated examples as negative examples is interesting and novel (for generative replay based continual learning methods).\n\nWeakness:\n1. The main method is based on AR1, is there a particular reason of using this AR1? Could the method still work well under the case that only generative replay is allowed (no additional components as in AR1)?\n2. To my understanding, the main purpose of the paper is to learn new data better, instead of reducing forgetting on the old data. However, according to Figure 2 (c), the gradient will also flows back to the feature extractor (when it is not frozen). This may leads to forgetting issue, when no other techniques (Synaptic Intelligence or Experience Replay using real data) are applied. \n3. There are actually many tricks for learning a better classifier in the continual learning literature, see Labels Trick [1], Separated Softmax [2], etc., which also works without any replay techniques. It would be idea to compare with those methods.\n4. (Minor) The novelty and practical usage of the method is limited by the fact that a generative model is required.\n\n[1] Zeno et al. Task agnostic continual learning using online variational bayes. Arxiv 2018.\n[2] Ahn et al.  A simple class decision balancing for incremental learning. Arxiv 2020.", "summary_of_the_paper": "This paper proposes a novel learning scheme for generative replay methods in continual learning. Different from the original generative replay method, the generated examples are treated as negative examples for new classes. Experimental results on CORe50 and ImageNet-1000 demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "The idea is interesting, but the novelty is limited, and some details and comparison experiments are missing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "q8z8wOyXRUH", "reviewer": "Reviewer_mn5p", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nThe paper presents an interesting method that could improve how to use replay in continual learning.\nThere is a high amount of experiments with various baselines.\nUnfortunately, the baselines are not run to estimate the interest of the replay strategy but to select only one model that will use the replay strategy.\nIt would make more sense to run the different baseline approach with and without the replay strategy to estimate the interest of the proposed method.\n\nThe authors beat the state of the art in ImageNet1000 and Core50 with a combination of their method and a model inspired by several papers.\nBeating state of the art is good; however, it does not provide a good evaluation of the replay strategy.\nThe sota experiments would be interesting as a conclusion experiment after evaluating negative replay on several baseline strategies.\n\n\nMasking past output neurons while applying backward pass for generative replay is new, however, it has already been experimented in continual learning papers in other contexts such as \"Continual Learning in Deep Networks: an Analysis of the Last Layer\" Lesort et al. \nThe related works should also mention how the proposed masking strategy is related to existing works.\n\n\nOther Remarks:\n\n- the approach is designed to make low-quality replayed data useful. However, no assessment of the replay quality is presented. The authors compare the results of replay with generated data, real data, and random data without actually evaluating the data quality.\n- the approach for Core50 and Imagenet is not the same (one pretrained feature extractor and one feature extractor trained only on task 1), making it difficult to assess the results.\n- the paper should be clear from the beginning that the replay process is realized in the latent space.\n- in conclusion : \"using negative replay largely improves\" is maybe a bit over enthusiastic (+2.95% for Core50, +0.83% Imagenet)\n- Table 3: The accuracy reported from [Masana et al] is not EWC but from EWC-E, which is a version of EWC with Replay. \n- Figure 2: The mask should appear more clearly and annotate into the figure. Why the backward pass of new samples goes to past classes? New data does not bring any information about past data.\n- The related works section presents a bibliography of generative replay but do not provide perspective on the link between the presented approach and the bibliography. The context described in the related works section should be linked to the paper approach.", "summary_of_the_paper": "The paper proposes a method to make generative replay for continual more effective even if generated data quality is not perfect. The method uses replayed data only as negative samples for current tasks and not as training samples to remember.\nThey apply their method on one model on Core50 and ImageNet data.\nThe replay process is realized in the latent space.", "summary_of_the_review": "Negative Replay looks like a promising approach to continual learning with approximate generative models. The papers propose interesting results with several baselines.\nHowever, the experiments are not designed in a convincing setup to evaluate negative replay. The evaluation of negative replay would benefit to be applied on several classical approaches before being applied on a specific/complex training procedure.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "RxtJAr5xhTs", "reviewer": "Reviewer_SNYr", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### **Strengths**\n- The key idea of using generative patterns/samples only as negative samples is simple, yet effective. The weight updates by the generated imperfect samples can be derogatory for the classes learnt in the past experiences. But these samples are still important for the new classes being learnt in the current experience due to the \"learning in isolation problem\".\n\n- Empirically, this approach is found to significantly outperform both positive generative replay and no-replay paradigms. Also, when compared to the PR-OD (Positive replay with original dataset), which is an upper bound for continual learning with replay approaches, the generative negative replay comes reasonably close in performance\n\n- The datasets used in this paper such as ImageNet-1000 indicate that this approach could be scalable to more complex or real-world scenarios of continual learning for classification.\n\n- The paper is generally well written and is an easy read.\n\n### **Weaknesses** and **Suggestions**\n- One of my concerns with that approach is that splitting the model into classification and feature extraction heads adds another hyperparameter. How does one decide where the split will be? Readers might want an answer to this question before using this approach.\n- It is a little surprising to see the accuracy on the ImageNet-1000 decreasing with the number of experiences, while the accuracy of CORe50 increases with the number of experiences. Is it because they are calculated differently? Maybe for CORe50, the accuracy is calculated across all classes even though they haven't been trained yet, while this is not the case for ImageNet-1000.\n  - If yes, it would be better to use the same way to calculate the accuracy, otherwise it is quite confusing\n  - If no, then it would be interesting to understand why this is happening?\n\n- Is PR-OD the same as Cumulative scores (Pelligrino et. al. 2020)? One of the goals of continual learning is to learn models (that are incrementally exposed to all the training data) that are comparable to a model that was trained on the full dataset. Hence, instead of PR-OD cumulative scores would probably be a better upper bound. \n- For experiments on CORe50, the MobileNetV1 was first pre-trained on ImageNet-1000 (to probably get a good feature extractor). But what about experiments on ImageNet1000; was the ResNet-18 pretrained? If yes, on which dataset? \n  - Another question that could be raised here is the choice of pretrained model as the feature extractor. Does it matter how the pre-trained model was trained? For example, if an unsupervised/self-supervised approach is used that uses negative examples and learns linearly separable embeddings, will it reduce the need for generative negative replay for continual learning?\n- The definition of the generative model $g_\\Omega$ is slightly confusing. In eq 5, the input to $g_\\omega$ is $z$, while in equation 6 the input now becomes a sample from the input space $\\mathcal{X}$.\n\n### Typos\n- Table 1: PR-OD instead of PR-OG\n", "summary_of_the_paper": "The key idea proposed in this paper is to use generative replay as negative samples to improve performance in class-incremental scenarios of continual learning. The authors argue that samples from a generative model have a lot of artefacts due to challenges in training/adapting a low resource generative model. Hence, its use as a positive sample for future experiences fails. But, these imperfect samples can still be relied on as negative samples for classes being trained in the current experience to prevent \"learning in isolation\" problems. The proposed approach is compared to positive replay, negative replay and no-replay baselines for two complex datasets in both NC and NIC scenarios.", "summary_of_the_review": "Overall the key idea proposed in the paper is sound and reasonably well justified with experiments. That being said, I have some concerns regarding some of the design choices and is the approach robust to these design choices. Hence, I have given a borderline rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "11FSS04Yjj", "reviewer": "Reviewer_onET", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Using generated data as only negative examples is interesting. It reminds me of SVM Examples [1], where they train N binary classifiers (where N is the amount of data). To train each classifier, this method uses only 1 positive example and the others as negative examples. In Exemplar SVM, the model learns very well what not to classify. I understand that the idea of Generated Negative Examples is different, but maybe it could be helpful.\n- I find it an ingenious idea to use CWR to train the sorting head. The model only focuses on making controlled changes, ignoring changes that can generate unwanted noise.\n- I liked how the paper is structured, introducing the current problem of methods that use generated inputs. Then, how this can be alleviated by using negative replay as negative examples.\n\nComments and Questions to the authors:\n- Acknowledging that one of the problems for not using data from past experiences is data privacy. Instead of generating negative examples, you could use the same proposed technique but actual data (extracted from other parts without security problems). Since you only train the columns of the classifier of current experience, there is no supervision needed. Maybe this brings the results a little closer to the real examples, but without security problems?\n- On page 5, just below Eq 7. What do you mean by “patterns” is similar to a data point?\n- In Figure 4, why do you think that after the 350 experience, the accuracy of the PR-OD starts to drop, but using NR-GD it remains stable? Do you think it could be due to the poor representation of the data from past experiences during the training? Is this drop due to the number of items from past experiences in the batch?\n- How does the proportion of elements generated in the batch affect it? I imagine that if you place fewer elements, the performance drops, but how sensitive is it to this number?\n- I understand that this method only works when we have little data in each experience. This is since there would not be enough data to correctly train the columns of the classifier of the classes that are not present in the experience. Could this technique be used in other scenarios? Not necessarily when we have few classes from experience. Perhaps using techniques similar to ACL to train the adversarial model?\n\nTypos:\n- Table 1 and 2, said NR-OG. I think it should have said NR-OD\n\n[1] Malisiewicz, Tomasz, Abhinav Gupta, and Alexei A. Efros. \"Ensemble of exemplar-svms for object detection and beyond.\" 2011 International conference on computer vision. IEEE, 2011.", "summary_of_the_paper": "In Continual Learning (CL) scenarios, storing data from previous experience is helpful to mitigate catastrophic forgetting. However, privacy issues or storage overhead makes replay methods impractical. Aware of this problem, generative models have been proposed in previous work to generate data that represent previous experiences. Still, these methods suffer from low performance because of the continual training of the generating model and/or the generation of high dimensionality data. In this paper, the authors propose Generative Negative Replay. Instead of using the generated data as a positive example of previous classes, they used it as a negative example for the classes present in the current experience. While training with the negative examples, the authors propose to freeze the weights of the classification head corresponding to the previous classes, similar to what happens in CWR. This solution is particularly relevant in situations where there are not as many classes from experience.", "summary_of_the_review": "The authors do not present a new idea but a very ingenious solution for a recurring problem. Having little data to train the tasks does not impede training well if you have a little imagination. By generating data that cannot be used directly as positive examples (because of the limitations of generation methods), they use this data as negative examples for other tasks. Although I still doubt if this can be extended to other scenarios, for example, when tasks are more similar or different. I think the authors present surprising results.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper suggests a new technique to utilize generative replay for continual learning. Specifically, the authors claim that even though the generated samples are imperfect (thus cannot be used as positive samples for old classes), they can still be used as negative samples for the current class. 3 reviewers are negative and 1 reviewer is positive. The main concerns of negative reviewers are (a) non-ablated effects of baseline and proposed components, (b) insufficient analysis of negative replay, and (c) no assessment of generated data quality. The rebuttal provides an additional experiment to address the issue (a), but the reviewers and AC think the experiments should be better polished. Also, AC believes the issues (b) and (c) should be better analyzed. The rebuttal claims that issue (c) is not applicable as they generate samples on the latent space. However, the main motivation of the paper is the low quality of generated samples, and the paper should provide a quality measure to support their claim. For example, an update of the feature extractor may move the latent space generative replay to the wrong class (i.e., low quality), and thus one should not use it as positive but only as negative, as suggested in this paper. Here, the negative replay would increase the margin of current and old classes, enhancing the accuracy of the current class. To analyze the source of benefits (old vs. current classes), the authors could report the task-wise accuracy trends, not only the overall accuracy. It would be a nice addition to the issue (b). Due to these unresolved concerns, AC tends to recommend rejection.
Decision: Reject