Make Memory Buffer Stronger in Continual Learning: A Continuous Neural Transformation Approach
Authors: Zhenyi Wang
Li Shen
Qiuling Suo
Tiehang Duan
Yanjun Zhu
Tongliang Liu
Mingchen Gao
Venue: Submitted to ICLR 2023 (2023)
Keywords: Continual Learning

Abstract: Continual learning (CL) focuses on learning non-stationary data distribution without forgetting previous knowledge. However, the most widely used memory-replay approach often suffers from memory overfitting. To mitigate the memory overfitting, we propose a continuous and reversible memory transformation method so that the memory data is hard to overfit, thus improving generalization. The transformation is achieved by optimizing a bi-level optimization objective that jointly learns the CL model and memory transformer. Specifically, we propose a deterministic continuous memory transformer (DCMT) modeled by an ordinary differential equation, allowing for infinite memory transformation and generating diverse and hard memory data. Furthermore, we inject uncertainty into the transformation function and propose a stochastic continuous memory transformer (SCMT) modeled by a stochastic differential equation, which substantially enhances the diversity of the transformed memory buffer. The proposed neural transformation approaches have significant advantages over existing ones: (1) we can obtain infinite many transformed data, thus significantly increasing the memory buffer diversity; (2) the proposed continuous transformations are reversible, i.e., the original raw memory data could be restored from the transformed memory data without the need to make a replica of the memory data. Extensive experiments on both task-aware and task-free CL show significant improvement with our approach compared to strong baselines.  

Under review as a conference paper at ICLR 2023
MAKE MEMORY BUFFER STRONGER IN CONTINUAL LEARNING: A CONTINUOUS NEURAL TRANSFORMA- TION APPROACH
Anonymous authors Paper under double-blind review
ABSTRACT
Continual learning (CL) focuses on learning non-stationary data distribution with- out forgetting previous knowledge. However, the most widely used memory-replay approach often suffers from memory overfitting. To mitigate the memory over- fitting, we propose continuous and reversible memory transformation method that the memory data is hard to overfit, thus improving generalization. The transformation achieved by optimizing a bi-level optimization objective that jointly learns the CL model and memory transformer. Specifically, propose deterministic continuous memory transformer (DCMT) modeled by an ordinary differential equation, allowing for infinite memory transformation and generating diverse and hard memory data. Furthermore, we inject uncertainty into the transfor- mation function and propose a stochastic continuous memory transformer (SCMT) modeled by a stochastic differential equation, which substantially enhances the diversity of the transformed memory buffer. The proposed neural transformation ap- proaches have significant advantages over existing ones: (1) we can obtain infinite many transformed data, thus significantly increasing the memory buffer diversity; (2) the proposed continuous transformations are reversible, i.e., the original raw memory data could restored from the transformed memory data without the need make replica of the memory data. Extensive experiments on both task-aware and fase-free CL show significant improvement with our approach compared to strong baselines.
INTRODUCTION
Continual learning (CL) aims to learn non-stationary data distribution without forgetting previous knowledge. Depending on whether there are explicit task definitions (partitions) during training, CL can be categorized into task-aware and task-free CL. For task-aware CL, there explicit tasks and class splits during training; according to whether the task identities are known or not during testing, be further categorized into aask/domainclass-nncrer task-free CL (Aljundi et al.) 20195), there no explicit task definition, and data distribution shift could happen any time.
CL. (van de Ven & Tolias 2019). For
Memory replay is an effective way to mitigate forgetting and has been widely used in CL. One major problem of the memory-based methods that the effectiveness of memory buffer data could gradually decay during training (Delange et al. 2021 Jin et al. 202T), i.e., the CL model may overfit the limited memory data and could not generalize well to the previous tasks. Recently, gradient- based memory editing (GMED) (Jin et al. 2021) has been proposed to mitigate memory overfitting by editing memory data with hard examples in a way similar adversarial data augmentation (ADA) (Madry et al. 2018). Specifically, it creates hard examples that increase model losses at each gradient step but restricts to few (less than three) discrete gradient-based editing steps. With more editing steps, similar to ADA, GMED would make the memory even harder but cause less data diversity since the adversarial force will drive the feature space of different classes overlap and cluster together (Madry et al.) 2018, Wang et al.) 2021). However, studied in previous work (Gontijo-Lopes et al. 2021). improving the diversity of training data is crucial to improving model generalization. An illustration of this phenomenon is shown in Figure (b) and (c). This naturally leads to new

---

Under review as a conference paper at ICLR 2023
(a) experience replay (ER) (b) GMED (3 edit steps) (c) GMED (10 edit steps)
(d) DCMT(Ours)
Figure 1: T-SNE visualization of existing memory-replay and proposed methods on CIFARIO. We use features extracted from the last layer output of ResNer18 as the input to T-SNE. We use four classes of memory data to illustrate the difference. T-SNE embeds each data point, and each color denotes one class of memory buffer data. (a): ER is very easy to overfit and easy to classify; (b): GMED with smaller editing steps has limited effectiveness due to the limited hardness; (c): GMED with larger editing steps creates memory examples harder to classify but lack of diversity; (d): DCMT (our method) with better diversity and transformed memory data is hard to classify and overfit.
problem: how can we increase the diversity of the edited memory data and maintain its hardness at the same time?
To address this problem, we present a continuous, expressive, and flexible memory transformation method to obtain a diverse set of memory data and make the memory buffer harder to memorize at the same time. We first model the gradual and continuous memory transformation as a deterministic neural ordinary differential equation in the time interval [o, T). named Deterministic Continuous Memory Transformer (DCMT). There are several advantages compared to existing methods. First, we can obtain infinite time steps of transformed memory data for any [0, 7' and thus significantly improve the diversity in the transformed memory data. Second, we do not need to make a replica of the raw memory data since the transformation process reversible. We can restore original raw memory data from the transformed memory data. As shown in Figure dd), DCMT diversifies memory data while maintaining hardness.
The proposed DCMT considers single transformation function. However, there are infinite pos- sible transformation functions for transforming the memory data, and beneficial to model the uncertainty in the transformation function to further avoid overfitting (Lu et al. Liu et al. (219)). To model the underlying various transformation functions and further improve the data diversity, we thus generalize the methods in probabilistic manner to model the memory transformation stochastic process with neural stochastic differential equations, named Stochastic Continuous Mem- ory Transformer (SCMT). This enables us model infinite transformation functions and significantly improves the diversity of the transformed data with some increased computation cost compared to DCMT. The overview of the proposed methods is presented in Figure
Memory transformation by DCMT or SCMT
Current batch
model
Memory buffer
to
Cg
Ex
Figure 2: Overview of the proposed approach for memory transformation. DCMT and SCMT continuously and gradually transform the memory data to be diverse and hard to memorize. Note that the transformed data could be obtained at any continuous time step, thus providing significantly larger diversity.
We propose bi-level optimization to jointly learn the memory transformer and CL model. The memory transformer can generate diversified memory data that is hard to memorize. Concretely, after continuous interval [0, transformation, optimize the loss increase before (t 0) and after
2

---

Under review as a conference paper at ICLR 2023
(t T) the transformation to ensure the hardness of the transformed data. To ensure that the network embeds similarly for both the original raw memory data and the transformed memory data, we adopt Jensen- -hasnnnn divergence consistency loss regularize the network output to ensure smoother neural network responses for both data. Furthermore, our proposed method is general, versatile, and can be seamlessly applied both task-aware and task-free CL., Extensive experiments both task-aware and task-free CL demonstrate the effectiveness of the proposed methods. We summarize our contributions three-fold:
We propose bi-level optimization framework with continuous memory transformer address memory overfitting issues. The continuous memory transformer can make the transformed memory data substantially more diverse and harder memorize.
We instantiate the continuous memory transformation with deterministic and stochastic memory transformation, which can be seamlessly applied to both task-aware and task-free CL.
We perform extensive experiments both task-aware and task-free CL, showing significant improvements compared to strong baselines. Furthermore, we provide detailed ablation studies.
2
PRELIMINARY AND RELATED WORK
2.1 TWSK-AWARE CL
Problem setup. Task-aware CL focuses on the case where there are explicit task definitions dur- ing CL. Taak/domain/clas-inereeee
learning (van de Ven & Tolias, 2019) are the three most representative CL scenarios. We consider the problem of learning sequence of tasks denoted as D = {Dir, D2 Drr consists of a set of triplets {( vf Th) J. where oF is the i-th data example in the task, of is the corresponding data label, and Ta is the task identifier. The goal is to learn a model fo on the training task sequence Dd' so that it performs well on the test of all the learned tasks Dee (Dfr, D2r, } without forgetting previously Iearned knowledge.
D where N is the number of training tasks. The k-th task training data
Existing work. The proposed approaches for task-aware CL can be categorized into: 1) maintaining a memory buffer that stores previous examples for future replay (Lopez-Paz & Ranzato 2017| Shin
et al. 2017] Chauddry et al.
Aljundi et al. [0199a] 2019a Riemer et al. 2019 Chaudhry et |
Pourkeshavarzi et al.) 2022
Arani et al. (002); 2) using dynamic network architectures (Rusu et al.
Fernando et al. 2017
Yoon et al. 2018 Qin et al. 2021; Miao et al., 2022) and remembering
past knowledge by d dynamically updated architectures; 3) enforcing regul arization to slow down
forgetting (Kirkpatrick et al.] 2017 Zenke et al. 201 von Oswald et al.) 2020: Liu & Liu 2022
Raghavan & Balaprakash (021);; and 4) modeling the parameter update uncertainty with Bayesian
methods (Nguyen et al.] [br]) Ebrahimi et al., 2020 Henning et al. (02T). In this paper, we focus on memory-replay cemory-replay-basede based methods since they often achieve SOTA performance.
2.2 TASK-FREE CL
Problem setup. Task-free CL (He et al. 2019 Zeno et al. 2019 Aljundi et al. 2019b; Chrysakis &
Moens 2020 Lee et al. 2020 recent generalization of CL to the more complex cases, where
data distribution shift could happen at any time during CL without explicit definition of tasks.
Existing work. Most existing works in task-free CL are memory-replay-basee methods (Chaudhry et al. 2019b a). Our works share similar goal with GMED (Jin et al., (202T) which edits the memory buffer based on ADA, making the memory data harder but lacks diversity (Madry et al. 2 et al. (202)). There are several significant differences. First, GMED is gradient-based discrete step memory editing method. Our memory transformer can obtain infinite continuous time steps transformation of memory data, which improves the memory diversity significantly. Second, GMED overwrites the memory data with the edited ones making the memory buffer data distribution significantly deviate from the original raw memory data distribution after many epochs of editing, especially in taske CL, which would decrease the performance. In contrast, our transformation process is reversible, i.e., the original raw data can be recovered from the transformed data. Thus, do not need to overwrite the memory buffer data or keep an additional mini-batch transformed data,
2018: Wang
We provide more detailed discussions of related work in Appendix C due space limitations.
3

---

Under review as a conference paper at ICLR 2023
3 METHODOLOGY
In this section, we first present standard memory replay of CL in Section 3.1 our proposed deter- ministic continuous memory transformer (DCMT) in Section 3.3 and stochastic continuous memory transformer (SCMT) in Section 3.4 Then, we present the training objectives in Section 3.5 The overall description of the proposed method shown in Figure
3.1 CONVENTIONAL MEMORY REPLAY
Standard memory replay for CL (Chaudhry et al, 2019b) is to optimize risk of data from both memory buffer M and current mini-batch. Formally speaking, the optimization can be formulated as:
min VOE
[(O, [(0, 2,, Uk) +
E C(O,
(1)
(m.g)m-M
where l: is the CL timestamp (k.hh CL step), O are model parameters, and L(O, 20, y) is the loss function associated with the data (æ. y). Conventional memory replay would make the memory buffer data gradually become less effective for mitigating forgetting when training for a long time, as it is easy to overfit the limited memory buffer data (Delange et al. [202]] Jin et al. (02T). Thus, the previously learned knowledge would get lost, and the CL. model may not generalize well the previous tasks. We thus propose a continuous memory transformation method to generate diversified memory buffer data that hard to memorize in the following sections.
3.2 A PRELIMINARY APPROACH TO INCREASE MEMORY DIVERSITY
A preliminary way to increase the memory diversity is to transform the memory data with neural network function parameterized by Φ. A sequence of transformations can be applied on the original raw memory data (1) by:
xx' + (i 1)A,) x ii,)
d)A, i= 0, 1.... n
where Ar is the step size. By repeating this discrete transformation process, we can obtain diverse collection of transformed memory data. However, when we updating the function g by backpropagation. we need to store all the intermediate transformations, i.e., {xe +i,), 0,... .n 1). Thus, the memory cost scales linearly with the number of memory transformation steps, i.., On). This would bring lot of memory cost especially if we transform the memory data by a large number of transformation steps. Furthermore, the neural network function g generally not invertible. We thus also need to store both the raw memory data and the transformed data. In the following, we simultaneously addressed the above issues by viewing the memory transformation process as continuous dynamic system. This brings several benefits: (1) do not need to store any intermediate transformation results, i.e., the memory cost is constant, i.e., O(1): (2) we do not need to store both the memory data and transformed data since the entire transformation is invertible even if the function g is not invertible (More elaboration on this is provided in Appendix (B.3). But the above discrete transformation needs do so since g generaly not invertible; (3)
method brings infinite amount of transformed memory data vs. the discrete steps of transformations. We name this our proposed preliminary method as Discrete Transformation (DT).
3.3 MEMORY TRANSFORMER AS A DETERMINISTIC CONTINUOUS DYNAMIC SYSTEM
In this section, we first view the memory transformation deterministic dynamic system. We transform the raw memory data into a continuous system in the time interval [0, T). Suppose at cach CL timestamp k. and sample a mini-batch data (æ"" r from the memory buffer M. Since we perform similar continuous mini-batch memory transformation operations at each CL timestamp k, we thus omit & for presentation clarity. We model the gradual and continuous memory data transformation by the following differential equation:
du (1) g(x (t), Φ),
(0) x:
(3)
dt
where the memory transformer is parametrized by function g with parameters & and g represents the instant time transformation rate of memory data. By integrating both parts of the Eq. (3) over the
4

---

Under review as a conference paper at ICLR 2023
time interval [0, T), we can obtain the solution to Eq. 33) for the transformed memory data at time T:
= =0) + / g(x (f),t Φ)
Th =T)
(4)
where the transformed memory data at any time T, i.e., x" (T) is a continuous function of T. For any E T), x=' (t) is transformation of original memory data, thus we can obtain set of infinite transformed memory data, i.e., (() E [0,7]] This is in contrast to GMED (Jin et al. 2021) which works with small number (less than three) of discrete-time steps of memory editing. With longer editing steps, similar to ADA, the edited data examples by GMED become harder but decrease the data diversity; thus, performance drops (Wang et al., (02I). Therefore, the design principle restricts the expressiveness and effectiveness of GMED. However, our memory transformation is significantly more expressive and provides substantially more diverse transformed memory data than GMED. We name our method as Deterministic Continuous Memory Transformer (DCMT), In practice, we can use a numerical integration scheme, such as the Sunge-Kutta method (Schober et al. 2014), similar to the implementation in (Chen et al. 2018) to solve the Eq. 4 We provide the algorithm details in Algorithm 3 in Appendix B. 12 The above transformation process is reversible
because can obtain the raw memory data by the following reverse integration:
(0) x (T) + [[ g(0 (1) d)dt.
(5)
Eq 5 transforms from x (T) into raw memory data (0) by integrating over the reverse time interval [T, 0J. We can thus discard the raw memory data and only keep the transformed memory data. After the replay, we can invert the transformed memory data into original data. The transformation function g does not need to be reversible and only needs to be uniformly Lipschitz continuous in (t) and continuous in (This condition is to make sure the solution to Eq. (3) exits and is unique), thus providing great flexibility for the transformation function design (More elaboration this is provided in Appendix B.13). Thus, our method has no extra memory cost to store additional raw memory data.
3.4 MEMORY TRANSFORMER AS A STOCHASTIC CONTINUOUS DYNAMIC SYSTEM
The DCMT method in Section 3.3 i.e., Eq. only considers a single deterministic transformation function g. but there are infinite possibilities of available memory transformation functions. Thus, single deterministic transformation is insufficient for modeling the underlying high diversity in the memory transformation functions. Furthermore, (Lu et al. [iu et al. 2019) show that adding uncertainty modeling for the network is beneficial to avoid overfitting.
We thus model the memory transformation process as a stochastic dynamic system with a path-valued random variable X [0, T + R. where each random variable at time t, i..., X,, is to model the distribution of the transformed memory data x"' (t) at time t. We use d to denote the memory data dimension. Let W [0, T] n be w-dimensional Brownian motion (Øksendal 2014) which a continuous time stochastic process such that W - W, follows a Gaussian distribution with mean 0 and variance Let μ [0, T] Rd + Rdd be the network for modelling the drift term, and
T [0, 7] x R Rdx be the network for modeling the diffusion term. They parameterised together by Φb. For notation and presentation clarity, we still use the same notations, Φ as DCMT to denote the parameters of memory transformer. The memory transformation stochastic process can be modeled as the following stochastic differential equations (SDE):
(6) where the initial values of the SDE the raw memory data that be viewed samples from the initial random variable Xo- For all t E [0, let X [0, - Rd denote the solution to Eq. (6) and odW, denotes Stratonovich integration (defined in Appendix B.III The stochastic process [Xceee jo.7] determined by Eq. (6) can be equivalently expressed as following:
dX.- = Xt) + (t, o dW,,
x Xa.
Xx = Xo + 1-" polt, Xi)dt + J' as (t) dW,
Each random variable X, of (Xelrepo.r models a transformed memory data distribution. Thus, we obtain infinite transformed memory data distributions. This is in contrast to DCMT, where each
5

---

Under review as a conference paper at ICLR 2023
(t) is deterministic.. To solve this SDE and achieve cheap backpropagattin, method, i.e., the reversible Heun method proposed in (Kidger et al. 2021). (algorithm details shown in Algorithmm3 [S] in Appendix 1.I.IIp. We name this method as Stochastic Continuous Memory Transformer (SCMT). Therefore, similar to DCMT, SCMT does not need to make a replica of the raw memory data.
the adjoint
3.5 TRAINING OBJECTIVES FOR CONTINUOUS NEURAL MEMORY TRANSFORMER
The goal of the memory transformer is to make the memory data hard to be memorized for the CL model. Our overall learning objective is the following bi-level optimization:
min (a yk. 0) + ( (T)), y" 0.).
(8)
s.t. Φ. arg max C(x (T), "")"""" 0, Φ) - -(x ,y"", yy 0) AJS(x" """ (T))
(9)
where x= xT)(T) xx'h (0) / g(x= ,)dl, am xx' x0' ['" gla" (.,,,,)) d.)dl
(10)
where x=t x²((7) (T) could be either from DCMT or samples from the terminal state of SCMT, Φb denotes the parameters of either DCMT SCMT. The lower-level optimization, i.c., Eq. (9). is to make memory buffer data hard to be memorized, and Φ, is the obtained optimal solution. The last term IS(x" x (T)) is to ensure smoother model responses data; where À is the regularization strength. We will discuss this in detail in the following. The upper-level optimization, i... Eq. 8 is to replay the transformed memory data. Note that x'' the transformed memory data by the optimal memory transformer with parameter Φ.. defined in Eq. 10 (right). The above bi-level optimization is for DCMT, but the method can also be directly applied to SCMT. In practice, besides the obtained x= (T), we can obtain infinite time steps transformed data any time in the interval [0, T). To make the computation tractable,
the original raw and transformed
can randomly sample time steps, i.e., 0 40 < 4 < 12 .....,y, T and obtain the transformed x= without additional cost since they are already in the integration interval [0, T). which be used for memory replay. It worth noting that the transformed data at different time stamps are not combined and they will not be added to the memory buffer. First, previous transformed data has already been learned by the CL learner. For new tasks, we need to transform the memory data adaptively. Second, storing those data would increase lot of memory storage cost. To maintain the diversity of the transformed memory buffer and reduce computation cost, we randomly sample n from [1, 5] at each CL step. Note that the number of parameters in DCMT or SCMT, i.e. Φ,, is much smaller than that of CL model, 0, thus negligible compared to CL. model backbone.
memory data a= (lo), ah (41). n. (12),
Consistency loss. The memory transformer could generate diverse memory data. To ensure that the network embeds similarly for both the original raw memory data and the transformed memory data, we use a similar consistency loss (Hendrycks et al. 2020) to regularize the network output to ensure smoother CL model responses. The goal is to make the CL model respond similarly x (T) and æm" thus minimize the Jensen-Shannon divergence among the posterior distributions of the original sample n and its transformed variants "h (T)) The consistency loss is defined below:
JS(x", x m (T)) = (pu (mmean) KL (pa)( (P
(Pm)))a()))PP ()mmean)))/22 /2) where Pmean = (Pam + Pam (T2)/2
where KL denotes the KL divergence between two distributions, Par. fe(x"") is the network output probabilities of each class for original raw data x" Similarly, we can define (T) = fo(x"" (T)).
The CL model parameters are then updated using the transformed memory data and the mini-batch data received at timestamp as follows:
e 0 = - Vol Tth (T), "d + C(Ox: 34, yR)]
where 17 is the learning rate. The learning process alternates between updating the memory transformer parameters (using adjoint method (Pontryagin et al. 1962) to update the parameters in DCMT or using the reversible Heun method (Kidger et al. (201]) to update the parameters in SCMT) and the CL. model parameters O. The complete memory transformation algorithm is shown in Algorithm T

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Continuous Memory Transformation.
1: REQUIRE:
model parameters O, memory transformer parameters d, CL model learning rate v, memory transformer learning rate S; transformation time I at each iteration, memory buffer M: K is the number of iterations during the training process for both task-aware and task free CL.
for 1 to K
3: mini-batch data (a+;, U&) arrives.
4: sample mini-hatch data from memory buffer, i.c., (x -y ~ M
5: x=" (T) OD "":"" 0, T') (Eq. (4) by Al Algorithm 3 in Appendix B.12
77) by Algorithm in Appendix B.12 calculate loss function (( (b) [C("" (T') y Φ)
or 2d (7) Xr SDEsolver(a" 0,7) (Eq.
C("
0, +)
A AJ
x (T))
we calculate gradient (C(0.4) with details provided Appendix B.3 and Algorithm
8:
update memory transformer parameters & by gradient ascent to maximize max C(,,
9:
transform memory data at randomly sampled time steps 0= <h t2 12 in = T, i.c xm by Eq.
T0 for memory replay.
10:
update CL model parameters 0: 0s+1 0% V/VolC(O,, Tt (4)). v) [(0,, ak, yk)] restore then
12:
restore 2D with Eq.
13:
end if
14:
update memory buffer by reservoir sampling (RS) RS(M, (Ik, yx))
Vitter
1985
Riemer al. 2019 M
15: end for
4 EXPERIMENTS
We evaluate our methods for task-aware CL in Section 4.1 and task-free CL in Section 4.2
Datasets. We compare different methods on CIFARIO (Krizhevsky) 2009) with 10 image classes, CIFAR100 (Krizhevsky) 2009) with 100 image classes, MinImageNet (Vinyals et al. 2016 with 100 image classes and Imag-lm (Stanford 2015 with 200 classes.
4.1 TASK-AWARE CL
We perform experiments on both task-incremental (Task-IL) and class-incremental (Class-IL) CL (van de Ven & Tolias 2019). Task-IL IL provides task identities to the CL learner and is the easiest scenario. Class- does not provide task identities and is the hardest scenario in task-aware CL.
Baseline. We compare to various SOTA CL methods, including: 1) regularization-be based methods, Classifier-Proecction Regularization (CPR) (Cha et al.) 202T). PASS (Zhu et al., (02TT. Gradient Projection Memory (GPM) (Saha et al. 2021) oEWC (Schwarz et al. 2018). synaptic intelligence (SI) (Zenke et al.) (017a) and Learning without Forgetting (LwF) (Li & Hoiem 2018): 2) Bayesian- based methods, UCB (Ebrahimi al. (020):: 3) architecture-based methods, HAT (Serrá et al. 2018
4) memory-based CL methods, including ER (Chaudhry et al. (019b). A-GEM (Chaudhry et al. 2019ah, GSS (Aljundi al., 2019c). HAL(Chaudhry et 2022)). DER+ (Buzzega et al. 2020) and GMED (Jin et al. 20221) The implementation for those basclines (Buzzega et al.) 2020) already applies data augmentation, such as random crops and horizontal flips, etc. We apply the proposed method on top of those implementations. We adapt PASS et al.) (022) to standard CL by adding noise to memory data, We provide detailed baseline deseriptions Appendix [B.2
Evaluation metrics. We evaluate the performance of the proposed methods and the compared methods with average accuracy (ACC) and backward transfer (BWT) at the end of CL training to measure the final performance and the degree of forgetting for different methods. We denote AN. as the testing accuracy on task & after learning on task N. The overall accuracy for all the tasks is ACC 인 -== measures the extent of forgetting on previous tasks after learning new ones. Formally, BWT defined 1
aN.K. To measure catastrophic forgetting, we also evaluate BWT, which
as: BWT- = n >k=n K= (ax - a4.4)- BWT < 0 indicates forgetting of previous tasks, and BWT > 0 indicates that learning new tasks is helpful on previous tasks.
Implementation details. We follow (Buzzega et al. 2020) use ResNet18 (He et al. 2016) the classifier for all datasets. Following (Buzzega et al. 2020), we split the CIFAR- 10 dataset into disjoint tasks, where each task consists of classes. We split Minilmagenet (Vinyals et al. 2016)
7

---

Under review as a conference paper at ICLR 2023
Table 1: Task-IL and class-IL results on CIFARIO, CIFAR-100 and Minilmagenet, respectively with memory size 500. '-' indicates not applicable.
Algorithm Method
CIFAR-10
CIFAR-100
Minilmagenet
Class-II.
Task-II.
Class-II.
TIsk-II.
Class-II.
Task-II.
fine-tuning Joint train
19.62 1 0.05
61.02 1 3.33
9.29 1 0.33
33.78 1 0.42
8.59 = 0.29
27.48 1 0.38
92.20 + 0.15
98.31
= 0.12
71.32 1: 0.21
91.31 + 0.17
65.56 0.18
57.74 + 0.15
oEWC
19.49 1: 0.12
68.29
3.92
8.24 1: 0.21
21.2 1 2.08
7.32 = 0.16
22.54 1: 1.78
SI
19.48 1 0.17
68.05
5.91
9.41 1 0.24
31.08 1 1.65
8.07
0.26
30.16 1: 1.72
LWF
19.61 0.05
63.29
2.35
9.70
+ 0.23
28.07 + 1.96
7.65 1.31
21.49 2.06
CPR
21.38 1: 0.19
72.37 + 3.51
10.31 + 0.24
31.93 1: 2.31
9.85
0.23
31.87 + 1.91
GPM
90.68 3.29
72.48 1 0.40
60.41 1 0.61
UCB
79.28 + 1.87
57.15 + 1.67
49.36 + 1.09
HAT
92.56 0.78
72.06 +: 0.50
59.78 +: 0.57
A-GEM
22.67 +: 0.57
89.48 + 1.45
9.30 1: 0.32
48 1 0.37
7.76 = 0.12
39.28 + 0.43
GSS
49.73 + 4.78
91.02 1.57
13.60 + 2.98
57.50 + 1.93
+.177
50.19 1.78
HAL
41.79 1: 4.46
2.36 84.54
9.05 1: 2.76
42.94 1: 1.80
1.72 4.46
31.97 1.57
ER
0.27
93.61
0.27
20.98 + 0.35
73.37 + 0.43
11.76
11.76+00.75 0.75
61.58 + 0.31
ER+GMED
57.89 + 0.38
0.39 93.45
21.07 + 0.43
73.42 + 0.49
0.81 11.85
61.76 0.37
ER+DCMT
62.07 1 0.43
0.46 95.16
23.54 0.51
7..93
118811000 0.85 14.28
64.37 1 0.53
ER+SCMT
64.49+0072
95.35+0.53
24.19+0.63
75.51 0.67
14.95+0.93
64.91+0.67
DER++
72.70 1: 1.36
93.88
0.50
36.37 1: 0.85
75.64 0.660
22.09 0.63
61.26 1: 0.57
DER++GMED
72.82 + 1.79
93.94 0.70
36.25 0.69
75.49
22.21 = 0.81
61.42 + 0.64
DER++noise
72.41 + 1.83
93.96 + 0.74
36.02
0.91
75.38+0.68
22.32 + 0.87
61.51 + 0.71
DER++DT(Ours)
73.77 +: 1.79
0.82 94.68
37.53 4 0.97
77.21
23.45 0.93
63.03 t 0.75
DER++DCMT(OUrs)
74.86 + 1.53
00.29=0.54 0.54
38.68=0.81
78.56 0.82
24.53 0.89
64.08 + 0.78
DER+-SCMT(OUr))
7..12+1.62
95.67+0.67
38.56 + 0.93
7.875+0.87
24.81+0.97
64.73++0.86
into 10 disjoint tasks, where each task has 10 classes. We also split CIFAR-100 into 10 disjoint tasks, where each task has 10 classes. The transformation time T at each CL step is T = 0.05. Other hyperparameter settings follow (Buzzega et al., 2020). The memory buffer has a size of 500 data points by default. For the DCMT architecture, it is a bourkblock Resnet with a filter size of 8. For the SCMT architecture, the drift and diffusion networks are both four-block Resnet with a filter size of 6. DT (Section 3.2) uses the same architecture as DCMT, Thus, for DT. DCMT and SCMT, they have a much smaller number of parameters compared to the ResNet18 HH et al.) 2016). It only accounts for about 0.4% parameters of ResNet18, thus is negligible. To exclude the influence of the number of parameters in performance comparison, we reduce the number of parameters in our base model ResNet18 to offset the parameters in our transformation component. This ensures that all the compared models have the same number of parameters. All reported results in our experiments are the average accuracy and standard deviation with ten runs. The compared methods and our proposed methods are based on the public implementation [
Result. We compare the proposed methods to various CL. baselines in Table 1 Due space limitations, we put the results on Tiny-ImageNNet in Appendix [B.9 and backward transfer results in Appendix B.5 We can observe that our method outperforms those baselines. In particular, for class-IL, combining ER with DCMT or SCMT outperforms ER by 3.2%, 3.1%, and 6.7%, on MinilmageNet, CIFAR- 100 and CIFAR1O, respectively. For task-IL, integrating ER with DCMT or SCMT outperforms ER by 3.%%, 2.6% and 1.7% on MinilmageNet. CIFAR- 100, and CIFAR10. respectively. Furthermore, for class-IL, combining DER++ with DCMT or SCMT, outperforms DER++ by 2.7%, 2.3% and 2.4% on MinilmageNett CIFAR- 100 and CIFARIO 10 respectively. For task-IL, integrating DER+ with DCMT or SCMT, outperforms DER++ by 3.5%. 3.1%, and 1.8% on MinilmageNet. CIFAR- 100 and CIFAR10 respectively. SCMT can further improve over DCMT due to the increased data diversity. GMED brings little or even worse performance, consistent with the observations of (Tin et al. 2021). We believe this is because, in task-aware CL. memory buffer data could be replayed many epochs, and GMED may edit the memory data too much so that they may significantly deviate from the original raw data, Adding random noise marginally helps in some cases, but some cases brings worse performance. We believe that adding noise only adds a simple transformation pattern, and still lacks diversity. When training for a long time, the network could also memorize the noise pattern. DT improves the performance compared to basclines since discrete step transformation by neural network can be viewed as discrete approximation of our continuous transformation. The performance of DT is upper bound by our continuous transformation. Also, DT needs to store intermediate transformations. In contrast, DCMT and SCMT do not need to store them. Our method outperforms baselines because the memory transformation function is highly expressive
httg://gthhb..cmmmmmgggaaaaannn
8

---

Under review as a conference paper at ICLR 2023
and flexible and provides significantly more memory diversity. The transformed memory buffer is more difficult for the CL model to overfit.
Ablation Study (1) Due space limitations, put hyperparameter analysis, including X, T, etc. in
Appendix B.6. (2) Memory size 2000 in Appendix B.4 (3) Memory visualization in Appendix [B.7
4.2 TASK-FRRE CL
Baseline. We performed experiments by compar- ing to the following fask-free CL baselines, includ- ing ER (Chaudhry et al. 2019b), MIR (Aljundi
Table 2: Task-free CL results CIFARIO. CIFAR-100 and Minilmagenet, respectively.
et al. 2019a). AGEM (Chaudhry et al. 2019a).
Algorithm
CIFARIO
CIFAR-100
MiniImagenet
GSS-Greedy (Aljundi et al.) 2019c) and GMED (Jin et al. 2021) with single epoch training. Fur- thermore, following (Jin et al. 202I), we also compare data augmentation, such as random ro- tations, scaling, and horizontal flipping applied to ER method, named aagg More descriptions of baselines are placed in Appendix B.22
fine-tuning A-GEM GSS-Greedy
18.9
3.1
0.5
0.3
2.1
3.0
2..9
19.5 1.3
17.
ER
33.3+288
20.1 1.2
24.8 1.0
ERADCMT ER+SCMT
365
21.94:1.3
27.6
37,3+3.0
21.6 21.6+11.5
1..
MIR
314124
20.0
25.3 1.7
MIR+DCMT
36.8
21.3 21.3+ 1.8 1.8
27.5 1.9
MIR+SCMT
37.6:299
21.7:12.11
27.8:2200
Implementation details. We use Resnet-18 18 (Aljundi et al. (2019)). The transformation time T each CL step is T 0.03. By default, following (Jin et al. 202T), we set the memory buffer size be 500 for CIFAR-10. 10K for Minilmagenet, and 5K for CIFAR- 100. Other hyperparameters are the same as (Aljundi et al. 2019a). All reported results are the average accuracy and standard deviation with ten runs.
GMED (ER)
3-28
20.9 1.6
27.3 1.8
GMED+DCMT
37.1
21.61:1..
28.1 ::1:::0 1.7
GMED+SCMT
37.9:2.77
21.8:119
18
ER.
4..3
18.3
30.8 :
ER..g +DMM
48.2:12.99
2.11
31.7 2.4
ER SCMT
47.9
20.6:1:.55
32.1+2.3
Result. We compare to various CL baselines and combination with ER, MIR, and GMED in Table 2 We observe that our method outper- forms these basclines. Particularly, combining ER with DCMT or SCMT outperforms ER by 4.0%, 3.1%, and 1.8% CIFAR10, Minilma- geNet and CIFAR- 100, respectively. Combining MIR with DCMT or SCMT outperforms MIR by 3.2%, 2.5% and 1.7% on CIFAR 10, Minilma- geNet and CIFAR- DCMT or SCMT outperforms GMED by 3.1%, 1.2% and 0.9% on CIFARIO, MinilmageNet and CIFAR-100. Our method outperforms baselines for reasons similar to task-aware CL.
Table 3: Task-free CL results CIFAR-100 and
Minilmagenet with different memory size
Algorithm
CIFAR-100
Minilmagenet
Memory size
2000
ER
11.2 = 1.0
15.0 1.9
11.0 0.3
17.9 1: 1.6
ER+ DCMT
14.5 1.1
17.1 1.11
14.1.00..
21.0 1.7
ER+ SCMT
14
17,6+14
14.5:0066
21.3=119
MIR
11.6 -
15.6 1.0
11.2
17.4 1.2
MIR+DCMT MIR+SCMT
14.8:110
16.7
20.6 146
Combining GMED with
14.6 1.1
17,0:122
15.0:1111
21.0=166
Ablation Study: The smaller memory size of 2000 and 3000 for 10FAR100; memory sizes of 2000 and 5000 for Mini-imageNet are provided in Table This further shows the significant improvement of DCMT and SCMT with more than 3% improvements in many cases.
Complexity analysis. Our current implementation has comparable computation cost compared to GMED. We put our efficiency improvement techniques and detailed complexity analysis in Appendix B. We put the formal complexity analysis in Table I0 and run time versus performance evaluation in Table in Appendix [B.8)
CONCLUSION
This paper explores the memory overfitting issues and proposes a novel continuous memory trans- former. We apply the proposed method both task-aware and task-free CL. Compared to existing works, our proposed methods are very flexible and can make the memory data diverse and hard to overfit. Extensive experiments with both strong baselines of task-aware and task-free CL demon- strate the effectiveness of the proposed methods. Future work includes automatically learning the transformation time interval obtain the optimal transformed memory data.

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
We provided detailed implementation details and codebase we used to implement our methods.
REFERENCES
Rahaf Aljundi, Eugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. Advances in Neural Information Processing Systems 32, pp. 1 1184-118600 2019a.
Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Task-free continual learning. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019b.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. In Advances in Neural Information Processing Systems 30, 2019c.
Elahe Arani, Fahad Sarfraz, and Bahram Zonooz. Learning fast, learning slow: A general continual learning method based on complementary learning system. International Conference on Learning Representations, 2022.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark experience for general continual learning: a strong, simple baseline. 34th Conference on Neural Information Processing Systems, 2020.
Sungmin Cha, Hsiang Hsu, Taebaek Hwang, Flavio Calmon, and Taesup Moon. {CPR): Classifier- projection regularization for continual learning. In International Conference on Learning Repre- sentations, 2021.
Arslan Chaudhry, Mare' Aurelio Ranzato, Mareus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-gem. Proceedings of the International Conference on Learning Represen- tations, 2019a.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, and Marc' Aurelio Ranzato. Continual learning with tiny episodic memories. https://arxivvorg/abs////
10486, 2019b.
Arslan Chaudhry, Albert Gordo, Puneet K Dokania, Philip HS Torr, and David Lopez-Paz. Using hindsight to anchor past knowledge in continual learning. Association for the Advancement of Artificial Intelligence (AAAI), 2021.
Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in Neural Information Processing Systems, 2018
Aristotelis Chrysakis and Frani-Francine Moens. Online continual learning from imbalanced data. Proceedings of the 37th International Conference on Machine Learning, 191199922911,, 2020.
M. Delange, R. Aljundi. M. Masana, S. Parisot, X. Jia, A. Leonardis, G. Slabaugh, and T. Tuytelaars. A continual learning survey: Defying forgetting classification tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach. Uncertainty-guidee
continual learning with bayesian neural networks. Proceedings of the International Conference on Learning Representations, 2020.
Chrisantha Fernando, Dylan Banarse, Charles Blundell, Yori Zwols, David Ha, Andrei A. Rusu, Alexander Pritzel, and Daan Wierstra. Pathnet: Evolution channels gradient descent in super neural networks. https://arxiv.or//abb/// 1ot.p///////110774444 2017.
Raphael Gontijo-Lopes, Sylvia Smullin, Ekin Dogus Cubuk, and Ethan Dyer. Tradeoff's data augmentation: An empirical study. International Conference Learning Representations, 2021.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2016.
10

---

Under review as a conference paper at ICLR 2023
Xu He, Jakub Sygnowski, Alexandre Galashov, Andrei A. Rusu, Yee Whye Teh, and Razvan Pascanu. Task agnostic continual learning via meta learning. https//aarav.orgg/////990000202 2019.
Dan Hendryeks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: A simple data processing method to improve robustness and uncertainty. Proceedings of the International Conference Learning Representations (ICLR), 2020.
Christian Henning, Maria Cervera, Francesco D' Angelo, Johannes Von Oswald, Regina Traber, Benjamin Ehret, Seijin Kobayashi, Benjamin Grewe, and Joao Sacramento. Posterior meta- replay for continual learning. Advances in Neural Information Processing Systems, 2021.
Xisen Jin, Arka Sadhu, Junyi Du, and Xiang Ren. Gradient-based editing of memory examples for online task-free continual learning. Advances in Neural Information Processing Systems, 2021.
Patrick Kidger, James Foster, Xuechen Li, and Terry Lyons. Efficient and accurate gradients for neural SDES. Advances in Neural Information Processing Systems, 2021.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Demis Hassabis,
Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Soochan Lee, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural dirichlet process mixture model for task-free continual learning. In Proceedings of the International Conference on Learning Representations, 2020.
Xuechen Li, Ting-Kam Leonard Wong, Ricky T. Q. Chen, and David Duvenaud. Scalable gradients for stochastic differential equations. International Conference on Artificial Intelligence and Statistics, 2020.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018.
Hao Liu and Huaping Liu. Continual learning with recursive gradient optimization. International Conference Learning Representations, 2022.
Xuanqing Liu, Tesi Xiao, Si Si, Qin Cao, Sanjiv Kumar, and Cho-Jui Hsich. Neural sde: Stabilizing neural ode networks with stochastic noise. http:///anthhor/aab//9000023333 0220000 2019.
David Lopez-Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in Neural Information Processing Systems, 2017.
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks: Bridging deep architectures and numerical differential equations.
Aleksander Madry. Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. International Conference on Learning Representations, 2018.
Zichen Miao, Ze Wang, Wei Chen, and Qiang Qiu. Continual learning with filter atom swapping. International Conference Learning Representations, 2022.
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational continual learning. Proceedings of the International Conference on Learning Representations, 2018.
Bernt Øksendal. Stochastic Differential Equations: An Introduction with Applications (Universitext). Springer, 6th edition, 2014.
Lev Semenovich Pontryagin, EF Mishchenko, VG Boltyanskii, and RV Gamkrelidze. The mathemat- ical theory of optimal processes. 1962.
11

---

Under review as a conference paper at ICLR 2023
Mozhgan PourKeshavarzi, Guoying Zhao, and Mohammad Sabokrou. Looking back on learned expe- riences for class/task incremental learning. International Conference on Learning Representations, 2022.
Qi Qin, Wenpeng Hu, Han Peng, Dongyan Zhao, and Bing Liu. BNS: Building network structures dynamically for continual learning. Advances in Neural Information Processing Systems, 2021,
Krishnan Raghavan and Prasanna Balaprakash. Formalizing the generalization- forgetting trade-off in continual learning. Advances in Neural Information Processing Systems, 2021.
Matthew Riemer, Ignacio Cases, Robert Ajemian, Miao Liu, Irina Rish, Yuhai Tu, and Gerald
Tesauro. Learning to learn without forgetting by maximizing transfer and minimizing interference. International Conference on Learning Representations, 2019.
Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, and Raia Hadsell. Progressive neural networks. https://arxiv.orgaab///66666 04671, 2016.
Gobinda Saha, Isha Garg, and Kaushik Roy. Gradient projection memory for continual learning. Proceedings of the International Conference on Learning Representations, 2021.
Michael Schober, David Duvenaud, and Philipp Hennig. Probabilistic ode solvers with runge-kutta means. Advances in Neural Information Processing Systems, 2014.
Jonathan Schwarz, Jelena Luketina, Wojciech M. Czarnecki, Agmieszka Barriikk, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Progress and compress: A scalable framework for continual learning. In Proceedings of the International Conference on Machine Learning, 2018.
Joan Serrá, Dídac Surís. Marius Miron, and Alexandros Karatzoglou. Overcoming catastrophic forgetting with hard attention to the task. In Proceedings of the International Conference on Machine Learning, 2018
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in Neural Information Processing Systems, 2017.
Stanford. Tiny imagenet challenge (cs23 In). http://ttny--mmggeeett hwwp:/ggeeeeetteeroucccca.ppoooo 2015.
Gido M. van de Ven and Andreas S. Tolias. Three scenarios for continual learning. 2019.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, koray kavukcuoglu, and Daan Wierstra. Matching networks for one shot learning. Advances in Neural Information Processing Systems, 29, 2016.
Jeffrey S. Vitter. Random sampling with a reservoir. ACM Trans. Math. Softw., 1985.
Johannes von Oswald, Christian Henning, João Sacramento, and Benjamin F. Grewe. Continual learning with hypernetworks. International Conference on Learning Representations, 2020.
Haotao Wang, Chaowei Xiao, Jean Kossaifi, Zhiding Yu, Anima Anandkumar, and Zhangyang Wang. Augmax: Adversarial composition of random augmentations for robust training. Advances in Neural Information Processing Systems, 2021.
Weixiang Xu, Xiangyu He, Ke Cheng, Peisong Wang, and Jian Cheng. Towards fully sparse training: Information restoration with spatial similarity. The Thirty-Sixth AAAI Conference on Artificial Intelligence, 2022.
Jaehong Yoon, Eunho Yang, Jeongtae Lee, and Sung Ju Hwang. Lifelong learning with dynamically expandable networks. International Conference on Learning Representations, 2018.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. In Proceedings of the 34th International Conference on Machine Learning, volume 70, pp. 3987- 3995, 2017a.
Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelligence. http:://arivorr/laa////....2 2017b.
12

---

Under review as a conference paper at ICLR 2023
Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task agnostic continual learning using online variational bayes. http:://axii..rg/aass/// www.:or//bbs........... 10123, 2019.
Fei Zhu, Xu- Yao Zhang, Chuang Wang, Fei Yin, and Cheng-Lin Liu. Prototype augmentation and self-suuprvision for incremental learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5871-5880, June 2021.
A APPENDIX
We provide more experimental details and results in Section and theoretical analysis Section ??.
B EXPERIMENTS
B. MORE IMPLEMENTATION DETAILS
We use NVIDIA RTX A6000 GPU to do the experiments. For the hyperparameters values of DER++ ER and all the other baselines, we follow the implementation in (Buzzega al. 2020).
B.2 BASELINE DESCRIPTIONS
Experience Replay (ER) (Chaudhry et al. 2019b) stores small subset data from previous tasks
with reservoir sampling (Chaudhry al. 2019b)). When training with new tasks, we randomly sample a subset of examples from the memory buffer to train with the received new mini-batch data together mitigate forgetting.
Maximally Interfering Retrieval (MIR) (Aljundi et al. 2019a). the goal of MIR is to select the examples that are most easily forgettable for replay. We follow similar setting GMED (Jin et al.) 2021) for fair comparison. We evaluate the CL model forgetting with 25 memory examples for Mini-ImageNett dataset, and 50 memory examples for other datasets.
Averaged Gradient Episodic Memory (AGEM) (Chaudhry et al. 2019a). At every CL training step, AGEM ensures that the average memory buffer data loss over the previous tasks does not increase. AGEM projects the gradient update direction to the closest gradient direction in Le space that keeps the gradient angle less than 90 degree to ensure the memory data are less interfered with current data.
Cradient-Based Sample Selection (GSS-Greedyy Alljundi et al.) 2019c) encourages storing diverse examples in the memory buffer. We GSS-Greedy, which is efficient and performs the best in the variants proposed in (Aljundi et al. 2019c).
Gradient based Memory Editing (GMED) (Jin al. 2021 The goal of GMED is edit the memory buffer data with gradient information so that they are harder to be memorized, which shares the similar goal (IRuunj al. 2019a).
Dark Experience Replay (DER) (Buzzega et al. 2020 is a memory replay-based methods that combines memory replay with knowledge distillation mitigate forgetting. DER++ one of the state-of-art methods in CL.
Hindsight Anchor Learning (HAL) (Chaudhry et al. 2021) regularizes the training objective with one data point per class per task, named anchors by maximizing its estimated forgetting. Keeping the model prediction fixed on those anchor points preserves the performance of previous tasks.
Task-free CL additional baselines Following (Aljundi al. 2019a), we also additionally compare:
(1) iid online: which trains the model with a single-past paggle-passsae through the iid sampled data on the same set of samples; (2) fid offline: which trains the model with multiple epochs through the iid sampled data.
We train the model with epochs for this baseline and the performance serves as upper-bound bound. Table 4 shows the results of these two baselines.
13

---

Under review as a conference paper at ICLR 2023
Table 4: Results for Task-free CL iid online and iid offline training on CIFARIO, CIFAR-100 and Minilmagenet, respectively.
Algorithm
CIFAR10
CIFAR-100
Minilmagenet
iid online
60.3 1.4
18.7 42.0 0.9
17.7 1.5
iid offline
78.7
39.8 + 1.4
B.3 BI-LEVEL OPTIMIZATION
(T) = xx + f (f),
(II)
We first define adjoint-state time l as a(l)
(T)
(a) here abbreviate the
da (1)
notation ((x" (T), y) as L, which follows the following differential equation:
da(l) dt
0g(a:")))) x²"
dc
aT
dx (T)
(12)
The adjoint-state equation Eq. 12 can be viewed as the continuous version of backpropagation (for discrete number of layers). Therefore, we can obtain (0) d by the reverse time ODE the time interval [2', 0] with initial state, i.e., ar dee the memory transformer parameters & as the following equation:
dC We can compute the gradient with respect to
dc = - [-- a(1) 0g(x" (t).t, σΦ
[a(1)r0g(a'
(13)
OΦ
The Eq. 111) serves as for transforming the memory data. Eq. serves as for calculating the adjoint state a()). Eq. (13) calculates the gradient with respect to the memory transformer Φ. Those three integration can be jointly solved together in single memory data, the adjoint state and derivatives with respect to the parameters. The entire algorithm for calculating the derivative with respect to the memory transformer parameters is shown in Algorithm 2 in the following.
by concatenating the transformed
Algorithm 2 Reverse- mode derivative for calculating the gradient w.r.t of DCMT.
I: REQUIRE: memory transformer parameters Φ. transformation time interval [0, T). final transformation state at time T, i.c., x=" (T), loss gradient @s""
a.
2: (T",, (2y +1]: calculate adjoint state
de
3: d aug-dynamics( (1 (L), a.,
4: return(g(2"" (t), t, Φ), -a m
8g
a
0g σΦ
5: (0), 3mm
ac
ac
ODEsolver(so, aug dynamics, T,0)
6: Return Dm (0) 06
ac
B.4 TASK-AWARE (CLASS-IL AND TASK-IL) MEMORY SIZE 2000
Table Sshows the results for task-Il IL and class-IL I on CIFAR-100 and Minilmagenet, respectively with memory size 2000. Our methods still significantly outperform strong baselines by large margin.
14

---

Under review as a conference paper at ICLR 2023
Table Task-IL and class-IL results on CIFAR-100 and Minilmagenet, respectively with memory size 2000
Algorithm Method
CIFAR-100
Minilmagenet
Class-IL
Task-IL
Class-IL
Task-IL
A-GEM
9.36 9.36
49.62 + 1.08
8.10 0.28
38.75 0.89
GSS
16.31 3.76
65.52 + 1.86
14.01 3.64
55.39 2.03
HAL
15.61 + 3.94
18.28 + 2.04
7.88 + 2.78
32.91 1.75
ER
36.87 0.64
82.07
22.78 0.57
72.25 0.32
ER+ GMED
36.35 + 0.77
$1.38+ 0.45
22.86 0.65
72.16 0.39
ER+ DCMT
38.90 0.71
83.45 1.55
74.57-0.66 0.78
ER+ SCMT
39.2810.89
83.79 10.68
25.3710.87
74.31 + 0.57
DER++
50.72 0.71
8
37.82 0.79
72.02 0.31
DER+
51.03 1 0.68
82.06 0.41
37.98 + 0.85
71.68 + 0.41
DEM+
52.93 + 0.94
83.81 + 0.52
40.16
74.06 + 0.49
DER++SCMT
52.97=0.77
84.26 0.69
39.91 = 0.89
74.29=0.65
B.5 BACKWARD TRANSFER (BWT)
Thhoe the backward transfer results with memory size 500 on various datasets. BWT measures the forgetting of CL model. Note that if one method restrains learning the current task would preserve the past knowledge with high BWT but achieves overall low accuracy. This would make the current task not learned well. The results indicate the significant improvement of the proposed methods (DCMT and SCMT) for mitigating forgetting compared to baseline ER and DER++. The baseline HAL achieves higher BWT in some cases because HAL does not learn the new task well and achieves overall much lower accuracy shown in Table | (main text).
Table 6: Backward Transfer of various methods with memory size 500.
CIFARIO
CIFAR100
MinilmageNet
Method
Class-IL -LL
Task-IL
Class-IL
Task-IL
Class-IL
Task-IL
finetuning
-96.39 + 0.12
-46.24 + 2.12
-89.68 + 0.96
-62.46 0.78
-82.48
1.17
-61.5 + 0.82
AGEM
-94.01 + 1.16
-14.26 1.18
-88.5 = 1.56
-5.433
-79.86
# 1.43
-44.84 1 2.81
GSS
-62.88 + 2.67
-7.73=3.90
-82.17 + 4.16
-33.98
1.54
-77.21
3.97
-34.12 = 1.85
HAL
62.21 +: 4.34
1.10
49.29 + 2.82
-13.00 1.04
-40.33
1.533.2..99
-10.50
11.50:1.77 1.70
ER
43.35
-3.54=0.30
-74.84 + 1.38
#.97
-78.21
1.57
-2.96 = 0.32
ER+GMED
0.33
-3.49 0.33
-74.60 + 1.26
-6..77
-77.76
1.38
222.10 0.27
ER+DCMT
43.91
3.15=0.45
-73.31 + 1.51
-15.23
-76.74
1.72
20.57 0.43
ER+SCMT
43.65
-3.12 0.53
-73.19 1.67
-15.02 1.21
-66.92
1.89
-20.43 + 0.58
DER+
22.38
4.66 1.15
53 80 + 1.85
14.72
2.773
22.93 0.91
DER+
-22.53 4.21
-.57
1.15
53.677
1.71
14.86
2.67
22.
0.87
DDMM
-20.61 4.58
-4.21
1 1.26
-52033 + 1.93
-13.92
-99.49 2.89
20.51 = 0.98
DER+
-20.27 1 4.67
4,08 1 1.37
52.31 1: 2.06
-13.61 1 1.09
-59.03 1: 2.97
-20.09 1 1.07
B.6 HYPERPARAMETER ANALYSIS
Table 7 shows the effect of integration (transformation) time on the performance. We can observe that the performance becomes better with a longer integration time T'. This is because with a longer integration time T, we can generate more diverse transformed memory data so that the proposed methods generalize better to previous tasks. If the T' is too large (T = 2.0), the transformed data would significantly deviate from the original data distribution, thus the performance could become worse. To maintain the transformed memory data distribution not deviate from the original data distribution too much and reduce computation cost, we use moderate time T'. Table & shows the effect of À on the model performance. The model achieves best performance at X 3.0. Table 9 shows the effect of the number editing steps N on the performance for combining GMED Jin
et al. 2027) with + We can observe that with more editing steps, the performance of GMED drops significantly because the memory data becomes much harder but lacks data diversity, thus generalizing worse with more editing steps.
15

---

Under review as a conference paper at ICLR 2023
Table 7: Effect of integration (transformation) time T for combining DER++ with DCMT on CIFARIO, CIFAR-100 and Minilmagenet, respectively.
Integration time
T 0.03
0.05
2.0
CIFAR10
04.78 0.42
95.29 + 0.54
94.42 + 0.64
Task- IL
Minilmagenet
63.42 0.67
64.08 0.78
63.26 + 0.90
CIFAR10
74.51 + 1.37
74.86 + 1.53
74.17 + 1.49
Class-IL
Minilmagenet
24.16 0.72
24.53 + 0.89
23.93 + 0.96
Table Effect of regularization weight combining DER++ with DCMT on CIFARIO, CIFAR- 100 and Minilmagenet, respectively.
regularization weight
À = 0.0
1.0
3.0
À = 5.0
CIFAR10
94.62 0.49
94.98
0.54
95.29 + 0.54
94.85 + 0.57 64.11 + 0.73
Minilmagenet
63.71 0.72
64.37
0.53
64.08 + 0.78
Table 9: Effect of the number of editing steps N for DER++GMED (Jin et al. 2021 CIFAR10 10, CIFAR-100 and Minilmagenet, respectively. We can observe that with more editing steps, the performance of GMED drops significantly because the memory data becomes much harder but lacks data diversity, thus generalizing worse with more editing steps.
number of editing steps
N
N
N 5
N 10
CIFAR10
93.94
0.70
93.47 + 0.67
90.92 + 0.61
88.68 + 0.79
Task-IL
Minilmagenet
61.42 + 0.64
58.36 + 0.61
52.19 + 0.57
46.52 1 0.42
CIFARIO
72.82 + 1.79
70.59 + 1.62
60.96 + 1.51
59.40 + 0.57
Class-IL
Minilmagenet
22.21 1 0.81
19.83 0.71
15.96 + 0.62
14.54 1 0.53
B.7
MEMORY TRANSFORMATION VISUALIZATION
Figure and show the transformation results different time t. With longer transformation time, the transformed data becomes diverse when viewing all the data in the transformation interval [o. T]. SCMT brings diversity than DCMT in terms of appearance. This suggests that with gradual and continuous transformation of memory data can significantly improve the memory data diversity.
(a) = to
(b) t f
(0) f2
( to
(e) f4
(8) f fs
Figure 3: Gradual memory transformation by DCMT on CIFAR10.
16

---

Under review as a conference paper at ICLR 2023
(a) to
(b) (o)== 4
(d)=
(e) 4
(0)
Figure 4: Gradual memory transformation by SCMT on CIFARIO.
B.8 COMPLEXITY ANALYSIS AND EFFICIENCY EVALUATION
According (Xu al. 2022), the computation cost of backward propagation is two times forward propagation.
DER++: it needs forward and backward pass replay the memory data, which equiva- lently requires 3 + 1*2 =5 forward computation.
DER++GMED: at each CL step, GMED additionally needs 3 forwards and backward passes (for two mini-batch data) to edit the memory examples for each editing step. So, the complexity is 6+2*2= 10 forward propagation calculations.
DER++DCMT, since uniformly sample the number from [0, 5], the amortized cost across different CL steps is to use 3 discrete time points at each CL step in the interval [0, 7J. First, the time complexity of forward ODE calculation those time points is 3 forward propagation. Then, we need to calculate the gradients for model parameters. The time complexity backward gradient calculation at those time points. Thus, the computation cost of DCMT is equivalent to 6+3*2 forward propagation. We can further reduce the computation cost of DCMT by backward propagation into the memory transformer for every S CL. step instead of backpropagating into the memory transformer for every CL step. For example, suppose we have CL steps, i.c., 1, 2, 3; we backpropagate into the memory transformer at step 1, but use the same memory transformer in the step and 3 to transform the memory data. This method further reduces the cost of DCMT and SCMT into 6 = 8. Additionally, the integration and backward computation time for the memory transformer in the interval [0, 7') is empirically equivalent to 1.6 forward computation of the ResNet backbone since the memory transformer network is very small. Thus, the total computation cost is 9.6 forward computation.
DER++SCMT, the complexity similar, but requires evaluating both the drift and diffusion terms with the corresponding gradient. Thus, the computation cost doubles the cost of DCMT, and therefore it equivalent to 6+ 6 8 forward propagation. Additionally, the integration and backward computation time for the memory transformer in the interval [0, T is empirically equivalent to 4.2 forward computation of the ResNet backbone since the memory transformer network is very small. Thus, the total computation cost is 12.2 forward computation.
We summarize the computation complexity of the above methods in table TO We set DER+ as the baseline with a running time unit of 1; we compare the computation complexity of the other methods with respect to DER++.
Table II shows the rumning time evaluation of different methods on CIFAR-100 for one epoch training.
17

---

Under review as a conference paper at ICLR 2023
Table 10: Formal analysis of computation complexity of different methods.
Method
running time (units)
DER++
1.0
+ER++MED
2.0
DER++DCMT
1.9
DER++SCMT
2.4
Table 11: Performance verses efficiency evaluation (wall clock time) on CIFAR-100 for one epoch training. indicates not applicable.
Method
Class-IL
Task-IL
running time (wall clock)
GSS
13.6 + 2.98
57.50 1.93
50.6
HAL
9.05 + 2.76
42.94 + 1.80
27.9
UCB
57.15 1.67
83.6
DER+
36.37 + 0.85
0.85
75.64 + 0.60
25.8
DER++GMED
16.25:0.60
41.7
DEM+
38.68
78.56
38.9
DER+SCMT
38.56 + 0.93
78.75 +
59.1
B.9 RESULTS ON TINY-IMAGENET
Following (Buzzega et al. 2020 we split Tiny-ImageNet with 200 classes into 10 tasks. Each task has 20 classes. The results shown in Table
Table 12: TIsk-IL and class- results on Tiny-ImageNet, respectively with memory size 500
Algorithm Method
Tmny-ImageNet
Class-IL
Task-IL
oEWC
7.58 0.10
19.20 0.31
SI
6.58 0.31
36.32 0.13
LwF
8.46 0.22
15.85 0.58
CPR
8,91 0.15
20.71 0.35
UCB
46.89 + 0.42
A-GEM
8,06 0.04
25.33 0.49
ER
9.99 0.29
48.64 + 0.46
DER+
19.38 + 1.41
51.91 0.68
DER++GMED
18.93 + 1.56
51.53 0.76
DEM+
21.06 1 1.26
54.02 0.57
DER++SCMT
21.27 + 1.42
53.67 + 0.73
B.10 T-SNE VISUALIZATION
More T-SNE visualization of ER is shown in Figure
GMED shown in Figure 6 our method is
shown in Figure 7
18

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(c)
(d)
Figure 5: Experience Replay (ER) T-SNE visualization on CIFARIO. We use features extracted from the last layer output of ResNet18 the input to T-SNE. We use four classes of memory data to illustrate the difference. T-SNE embeds each data point, and each color denotes one class of memory buffer data. Each figure is plot at different times.
(a)
(b)
(c)
(d)
Figure 6: GMED T-SNE visualization on CIFAR10. We use features extracted from the last layer output of ResNet18 as the input to T-SNE. We use four classes of memory data to illustrate the difference. T-SNE embeds each data point, and each color denotes one class of memory buffer data. Each figure is plot at different times.
(a)
(b)
(c)
(d)
Figure 7: DCMT (our method) on CIFARIO with T-SNE visualization. We use features extracted from the last layer output of ResNet18 8 the input to T-SNE. We use four classes of memory data to illustrate the difference. T-SNE embeds each data point, and each color denotes one class of memory buffer data. Each figure is plot at different times.
B. MATH BACKGROUND
Definition (Stratonovich stochastic integral) Suppose [Fs.tlecttatefo..] be two-sided filtra- tion, where Fs,t is the o-ulgebra generated by {W., W. S S S 1} for s, E [0, T] such that s S For continuous semi momiiimrtinggle (Xiheforr adapted to the forward filtration, (Fo,heejo,T) Stratonovich stochastic integral
f' X. dW, lim Σ
Xtx + X (W We..))
(14)
where = ... <N = T is a partition of the interval [0, T) and II] max; to tx denotes the size of largest segment of the partition, and the limit be interpreted in the Tog space.
19

---

Under review as a conference paper at ICLR 2023
B.12 TRAINING ALGORITHMS FOR DCMT AND SCMT
We present the Runge- MutteeKKuttaaao Method in Algorithm 3 for integrating ODE to obtain its solution. We present the adjoint method for calculating the gradient with respect to the parameters of DCMT ø in Algorithm 2 We then present the algorithm for calculating the forward pass of SCMT (Eq. 7 in Algorithm 4 We then present the reversible Heun algorithm for calculating the gradients with respect the parameters of SCMT in Algorithm
Algorithm 3 Runge--Kuui Kutta Method for integrating ODE.
1: REQUIRE: dt"
g(x (t), p), x²'(0) dividing the time interval [0, T] into sub-intervals, (fo, t1l, 1 " tat)) 1 (fx-1, where
[f+1 - tx| = h; h is the sep size.
3: for 1 to N do
4: K1 g(= Lm lh
K2 ((=
+
K1, bn + hh)h
6: Ks g(x
7 K2,
g(=
+ Kn + 11)h
aa+1 æn
+ (K, + 2K2 + 2K3 K4)
9: end for
10: Return 2.
Algorithm 4 SCMT (SDE) solver (forward pass).
1: REQUIRE: L,, x 2. foo on, At. Brownian motion W.
l.41 = ln + A!
3: AW, = Wt,
Wtn
4: - = -xx xth + pm At on
5: Pat1 =M(tn+1, p.t1)
6: On+1 =0((n+1, 1,
7: 2=+1 = -. + 3 (M + Pat1) + [a,, + aw,
8: Return 1o41, 2 1: 2 Pat1, 0n41
B.12.1 REVERSIBLE HEUN (KIDGER ET AL.. 2021 FOR SOLVING THE SCMT GRADIENTS
Suppose we have the loss L on the terminal random variable Xr, then the adjoint process A; dC(Xr) E Rd is the solution to the following SDE: dx.
dAi AA. 0pe Ox X.)dt Ai
0o ax. Xt, dw?
(15)
where Ao dc(Xx) dXa is the obtained backpropagated gradient. The gradients with respect to the parameters of the drift and diffusion networks can be obtained by viewing them as additional part of the state whose dynamics has zero drift and diffusion (Li et al. 2020). Furthermore, the adjoint method of reversible Heun (Kidger al. 2021) utilizes the reversibility of differential equation: intermediate computations such as X, for are restored from Xr,, so that they do not need to be held in memory.
20

---

Under review as a conference paper at ICLR 2023
Algorithm 5 Reversible Heun method of Backward computation calculate the gradients w.r. the previous state of SCMT.
1: REQUIRE:
2p+1 a- Mm+ Out1, At,
Brownian
motion
W.
ac(
a2(27))
++1
0p++1
00a+1
Al
3: AWn W WWn
4: æn 2
Un+ A1 ont1AWn
5: j = pln: FR
6: Oa (Ln, 1t
7: In an+1
(No + 1n+1) - 200n + 10+1
8: Tatl. T. Mm+1, 0u+1 = Forward(tn, The ai on. At, W)
9: Local Backpropagation
10:
(C(W5)
oc(=7)
S( ni 1. 1i
8( An.Pn)
0lam 2 knlan)
11: Return bu 2. L Mr, Ons
(0(m))
aC(a))
0C(77)
(C(=7)
80
B.13 WHY THE ENTIRE TRANSFORMATION INVERTIBLE AND THE VELOCITY TERM DOES NOT NEED TO BE INVERTIBLE
The forward integration is as the following:
ac (T) = (0) + { (t,,t d)
(16)
We then multiple -1 for both sides of the above equations, obtain the following equation:
(T) xx= L0" (t), Φ)dt
(17)
We then rearrange the above equation as following:
x: (0) x (T) - S/" ((), d)dtt
(18)
Then due to the following property of integration
L g(x
d) = [ [ (()
(t), g(x"((1))
(19)
We can obtain the following reverse integration:
(() [
x Th (0) x nh (T)
(20)
While we can observe that in the above derivation, there is no restriction that should be invertible.
C PRELIMINARY AND RELATED WORK
C.I TASK-AWARE CL
Problem setup. Task-aware CL focuses on the case where there are explicit task definitions dur- ing CL. cattoomain/clss--nncmeeen
learning (van de Ven & Tolias 2019) the three most representative CL scenarios. We consider the problem of learning a sequence of tasks denoted as Dir [Dfr, Dr..... DDr), where N is the number of training tasks. The k-th task training data
21

---

Under review as a conference paper at ICLR 2023
DE consists of set of triplets {(0,, vx. Tx) ), where of is the i-th data example in the task, of is the corresponding data label, and TR is the task identifier. The goal is to learn model fe on the training task sequence D so that it performs well on the test set of all the learned tasks Dee (Df Di ....... ) without forgetting previously learned knowledge.
Existing work. The proposed approaches for task-aware CL can be categorized into: 1) maintaining
memory buffer that stores previous examples for future replay (Lopez- Paz & Ranzato 2017] Shin
ct al. al.]|2201111 Chauddryy et al. 2019a, Riemme et al.] 2019) Chaudhry et 2019b [Aljundi et | 2019
Pour Keshavazi al. 2022
Arani al. 2022); using dynamic network architectures (Rusu et al.
Fernand et al. 2017
Yoon et al. 2018 8 Qin et al. 2021; Miao et al., 2022) and remembering
past knowledge by d dynamically updated architecture;; 3) enforcing regularization to slow down
forgetting (Kirkpatrick et al. 20177 Zenke et al. 2017b] von Oswald et al. 2020: Liu & Liu 2022
Raghavan Balaprakash (2021); (20t1) and 4) modeling the parameter update uncertainty with Bayesian |Ebrahimi et al. [2020] Henning et al.) (021). In this paper, we focus on memory-replay-basse methods since they often achieve SOTA performance. Memory-based methods include experience replay (Chaudhry et al. 2019b), which jointly trains the memory buffer data with current mini-batch, Meta Experience Replay (MER) (Riemer et al.) 2019) adopts meta-learning to maximize transfer from previous examples and minimize interference. Hindsight Anchor Learning (HAL) (Chaudhry et al. (20TT) using anchor points to mitigate forgetting on previous tasks. GEM (Lopez-Paz & Ranzato 2017) and A-GEM (Chaudhry al., (209a) use the losses the memory buffer data as inequality constraints, avoiding their increase but allowing their decrease to avoid forgetting. DER (Buzzega et al. 2020) further combines rehearsal with knowledge distillation.
methods (Nguyen et |
C.2 TASK-FREE CL
Problem setup. Task-free CL (He et al. 2019 Zeno et al. [019 Aljundi et al. 2019b; Chrysakis & Moens 2020 Lee et al. 2020 is recent generalization of CL to the more complex cases, where data distribution shift could happen at any time during CL without explicit definition of tasks. A sequence of mini-batch labeled data (æt 3/k hig) sequentially arrives at each timestamp k and forms nonn-ttatonnary data stream; where Iy. denotes the mini-batch data received at timestamp is, y4 is the data label associated with Zg. and h2 is the hidden task identity associated with æg. During both the training and testing time, the task identity ha is not available to the learner. A more general definition of task-free CL in (Aljundi et al., (209b)) assumes no explicit partitions of tasks, and the data distribution can change arbitrarily. However, our proposed methods can be seamlessly applied to those more general scenarios.
Existing work. Most existing works in task-free CL are memory-replay-basse methods (Chaudhry et al. 2019b a). They directly perform memory replay on the raw data without any transformation. MIR (Aljundi et al. 2019a) proposes to replay the samples with which are most interfered. GEN- MIR (Aljundi et al. 20) further uses generative models to synthesize the memory examples. Gradient-based Sample Selection (GSS) (Aljundi et al., 2019c) focuses on storing diverse examples which is completely different from our method. Our works share a similar goal with GMED (Tin (et al. 202I) which edits the memory buffer based on ADA, making the memory data harder but lacks diversity (aadry et al.) 22188: Waang et al.) 2021). There are several significant differences. First, GMED is a gradient- gradient-based discrete step memory editing method. Our memory transformer can obtain infinite continuous time steps transformation of memory data, which improves the memory diversity significantly. Second, GMED overwrites the memory data with the edited ones making the memory buffer data distribution significantly deviate from the original raw memory data distribution after many epochs of editing, especially in task-aware CL. which would decrease the performance. In contrast, our transformation process reversible, i.e., the original raw data can be recovered from the transformed data. Thus, we do not need to overwrite the memory buffer data or keep an additional transformed data with extra cost.
22


Reviews:
[{"review_id": "xnqORPzaGmN", "reviewer": "Reviewer_vLJC", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to augment the memory buffer, aiming to increase the data diversity. Specifically, this paper chooses to use differential equations to model the memory transformation. The experimental results on standard continual learning benchmarks show some improvements of the proposed method.", "summary_of_the_review": "Overall, I find this paper below the bar of acceptance, especially because of the lack of baselines (other perturbations for the memory buffer). It requires more ablation study and more justifications for the usage of differntial equation modeling.", "strength_and_weaknesses": "Strength:\n\n- The idea of augmenting the memory buffer with a transformation is straightforward and interesting.\n\n- The paper is overall easy to follow.\n\nWeakness:\n\n- Why using differential equations to model the memory transformation is poorly motivatied and also lack of justificaitons.\n\n- Despite the gain of the method, there is more must-to-compare baselines. For example, how about using a standard neural network to serve as the transformation, or use standard noise to augment the memory, similar to [Prototype augmentation and selfsupervision for incremental learning. In CVPR, 2021].", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "nZrQzN6ruC", "reviewer": "Reviewer_A5Re", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In the paper, the authors propose a continuous and reversible memory transformation method to prevent overfitting on the memory in continual learning. The main idea is to increase the diversity of the data in the memory buffer while maintaining hardness. The authors propose Deterministic Continuous Memory Transformer to modify the data with infinite transformation functions to encourage diversity. The proposed method is compared with several existing memory editing methods. The results show the proposed method is better than existing ones. ", "summary_of_the_review": "The paper proposes to edit memory buffer with continuous transformation. However, there are several confusing pieces which make it hard to understand the paper. Also, the additional complexity is also an issue. ", "strength_and_weaknesses": "Strength:\n\n1. The problem of preventing overfitting on memory is interesting and important for continual learning.\n2. The idea of introducing continuous transformation to the memory is also interesting. \n\n\nWeaknesses:\n\n1. What's the computational complexity of the proposed approach? The transformations seem expensive which is infeasible in online continual learning. Also, the training of the memory transformer also takes time. \n\n2. Section 3.4 is confusing. Are all the transformed data at different time stamps are combined? Will the transformed data be added to the memory buffer?\n\n3. In ER, are any random augmentations applied to the memory data?\n\n4. For ER, if the buffer size increases such that it consumes the same memory as the proposed method, how the performance will change? The proposed method introduces a memory transformer which takes more space than ER. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TeBXeDcvTE", "reviewer": "Reviewer_JHx6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a novel method modifying the memory data in exemplar memory when performing experience replay in continual learning (CL). To address the problem of memory overfitting, the authors used deterministic and stochastic differential equations to perturb the memory data by increasing the cross-entropy loss to make the memory data hard to be classified. Furthermore, since differential equation based generative models are used, they can produce diverse memory data. In the experiments, the proposed methods achieves much higher accuracy than baselines. ", "summary_of_the_review": "I vote to reject this paper. Though the proposed methods can be novel, authors does not verified the effectiveness of DCMD or SCMT by analyzing the important components of proposed methods.", "strength_and_weaknesses": "**Pros:**\n\nP1. The proposed method achieves state-of-the-art performance compared to the baselines, and using differential equation based perturbation in continual learning is a novel approach. \n\n**Cons:**\n\nC1. Though authors said the proposed methods can generate diverse examples, there is no quantitative results to verify this statement. It would be better to report the numbers using a metric such as precision-recall used in evaluating a generative models.\n\nC2. The addition computation cost for performing single stochastic gradient step is not negligible. The proposed algorithm should execute the ODE or SDE solver and optimize the parameters for the generator during the learning process. \n\nC3. There is no analysis on this method for verifying the proposed approach. For example, how much DCMT or SCMT affect the forgetting measure to the previous task? Since the memory data is perturbed, this rather produces negative impact on previous tasks.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CmdoTsZB_4R", "reviewer": "Reviewer_XsdS", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on memory-replay-based approaches for Continual Learning (CL). The authors justifiably argue that a continual learner could overfit to the memory buffer reducing the generalizability of the model on previously learned tasks. Hence, one could benefit from adjusting the samples in the memory to avoid such overfitting. The authors propose to update the samples in the memory buffer via first an Ordinary Differential Equation (ODE) approach, denoted as Deterministic Continuous Memory Transformer (DCMT), and second through a Stochastic Differential Equation (SDE) based approach, denoted as Stochastic Continuous Memory Transformer (SCMT). DCMT is identified via a parametric velocity/drift term $g(\\cdot ; t, \\phi)$, with parameters $\\phi$, while SCMT uses both a drift and a diffusion term that are parameterized by $\\phi$. Then, the authors propose a bilevel optimization where the lower level optimization finds the optimal parameters $\\phi$ that provide memory updates that are adversarial/challenging (i.e., maximize the loss). At the same time, they maintain the class information (i.e., the CL provides consistent labels for the current and updated memory samples). Lastly, the authors show that DCMT and SCMT boost the performance of memory-replay-based CL approaches in task-aware (for task and class incremental learning) and task-free learning.\n", "summary_of_the_review": "In summary, the paper addresses an important and interesting problem, and the proposed approach is original. However, many details are missing from the paper, and some critical points are completely ignored (please refer to weaknesses). ", "strength_and_weaknesses": "## Strengths \n\n* The paper addresses an interesting challenge and important problem with memory-replay-based approaches in CL.\n\n* The proposed idea of using ODEs/SDEs for updating memory samples is novel and interesting.\n\n* The empirical results of the paper indicate that DCMT/SCMT consistently improves the performance of memory-replay-based methods. \n\n## Weaknesses\n\n* The paper's flow could be significantly improved. For instance, Equation (11) seems to be the actual objective function used for the lower optimization problem proposed in Equation (9). It would be better to add the consistency loss directly to Equation (9) and then expand on why it is needed. \n\n* Much of the presented details are unnecessary, while many critical details are completely missing from the paper. Most importantly, the velocity term $g(\\cdot;t,\\phi)$, which is at the heart of the proposed method, is never discussed in the paper (same goes for $\\mu_\\phi$ and $\\sigma_\\phi$). What classes of functions are used to update memory samples? From the results, it seems like $g(\\cdot;t,\\phi)$ acts on the color histograms, while one can use a large variety of transformations (e.g., the auto-augment approaches that also provide geometric transformations) as the parametric velocity term. \n\n* The details of how the bilevel optimization problem is solved is entirely missing from the paper. \n\n* The consistency loss in Equation (11) makes a lot of sense, and I believe it is critical for the success of the proposed methods. While the JS divergence regularizes for obtaining consistent labels, it does not guarantee it. Hence, if the regularization parameter $\\lambda$ is not carefully selected, then one can end up with memory updates that cross the decision boundaries leading to harming the performance of previous tasks. The ablation study on $\\lambda$ in Appendix B5 does not reflect this point, possibly due to the small range of $\\lambda$ used in the ablation study. \n\n* On Page 5, the authors state that even if  $g$ is not invertible they can recover $x(0)$ from $x(T)$. If I understand correctly, having $g$ we can perform forward propagation through time, i.e., naively $x(t+1)=x(t)+g(x(t);\\phi)$ (or using Runge-Kutta as described in Algorithm 2 in the Appendix). However, one cannot retrieve $x(t)$ from $x(t+1)$, unless $g$ is invertible. Could you please elaborate on your statement? ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject