Task-Free Continual Learning via Online Discrepancy Distance Learning
Authors: Fei Ye
Adrian G. Bors
Venue: NeurIPS 2022 Accept (2022)
Keywords: Task-free continual learning
Variational autoencoder
Theoretical analysis for continual learning

Abstract: Learning from non-stationary data streams, also called Task-Free Continual Learning (TFCL) remains challenging due to the absence of explicit task information in most applications. Even though recently some algorithms have been proposed for TFCL, these methods lack theoretical guarantees. Moreover, there are no theoretical studies about forgetting during TFCL. This paper develops a new theoretical analysis framework that derives generalization bounds based on the discrepancy distance between the visited samples and the entire information made available for training the model. This analysis provides new insights into the forgetting behaviour in classification tasks. Inspired by this theoretical model, we propose a new approach enabled with the dynamic component expansion mechanism for a mixture model, namely Online Discrepancy Distance Learning (ODDL). ODDL estimates the discrepancy between the current memory and the already accumulated knowledge as an expansion signal aiming to ensure a compact network architecture with optimal performance. We then propose a new sample selection approach that selectively stores the samples into the memory buffer through the discrepancy-based measure, further improving the performance. We perform several TFCL experiments with the proposed methodology, which demonstrate that the proposed approach achieves the state of the art performance.


Task-Free Continual Learning via Online Discrepancy Distance Learning
Fei Ye and Adrian G. Bors
Department of Computer Science University of York York, YO10 5GH, UK (fy689, adrian. borrk ac. uk
Abstract
Learning from non-stationary data streams, also called Task-Free Continual Learn- ing (TFCL) remains challenging due to the absence of explicit task information most applications. Even though recently some algorithms have been proposed for TFCL, these methods lack theoretical guarantees. Moreover, there are no theoreti- cal studies about forgetting during TFCL. This paper develops analysis framework that derives generalization bounds based on the discrepancy distance between the visited samples and the entire information made available for training the model. This analysis provides new insights into the forgetting behaviour in classification tasks. Inspired by this theoretical model, we propose a new approach enabled with the dynamic component expansion mechanism for a mixture model, namely Online Discrepancy Distance Learning (ODDL). ODDL estimates the discrepancy between the current memory and the already accumulated knowledge as an expansion signal aiming to ensure a compact network architecture with optimal performance. We then propose a new sample selection approach that selectively stores the samples into the memory buffer through the discrepancy- based measure, further improving the performance. We perform several TFCL. experiments with the proposed methodology, which demonstrate that the proposed approach achieves the state of the art performance.
theoretical
Introduction
Continual learning (CL.) and its extension to lifelong learning, represents one of the most desired functions in an artificial intelligence system, representing the capability of learning new concepts while maintaining the knowledge of past experiences [32]. Such an ability be used in many real-time applications such as robotics, health investigative systems, autonomous vehicles [20] or for agents exploring artificial (meta) universes. Such applications require to continuously adapt to changing environment. Modern deep learning models suffer from degenerated performances on past data after learning novel knowledge, phenomenon called catastrophic forgetting [3].
popular attempt relieve forgetting CL is by employing small memory buffer to preserve few past samples and replay them when training on a new task 3 However, when there are restrictions on the available memory capacity, memory-based approaches would suffer from degenerated performance Recently, the Dynamic Expansion Model (DEM) [51] has shown promising results CL, aiming to guarantee optimal performance by preserving the previously learnt knowledge through the parameters of frozen components trained on past data, while adding a new component when learning a novel task. However, such approaches require knowing where and when the knowledge associated with given task is changed, which is not always applicable in a real environment.
past tasks, especially when aiming to learn an infinite number of tasks.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

In this paper, we address more realistic scenario, called Task-Free Continual Learning (TFCL) (2]. where task identities are not available while the model can only access small batch of samples at a given time. Most existing CL methods may be adapted to TFCL. by removing the task information dependency. For instance, memory-basee stream at each training time and replay them later during the training [8] 12 However, such an approach requires to carefully design the sample selection criterion to avoid memory overload. The key challenge for the memory-base cemoor-bbaeed approaches is the negative backward transfer caused by the stored samples which interferes with the process of model's updating with incoming samples [6]. This issue can be relieved by employing DEM in which previously learnt samples are preserved into frozen components and would not interfere with the learning of new probabilistic data representations 24 38.
approaches can store few past samples from the data
However, these approaches do not provide any theoretical guarantees and there are no studies analysing the trade-off between the model's generalization and its complexity under TFCI..
There are some recent CL. theoretical analysis studies from different perspectives including for the risk bounds [46 51), NP-hard problems 17. Student frameworks 23 57 or for the game theory (37] applications. However all these approaches require strong assumptions, such as defining task identities, which are not available under the TFCL. This inspires us to bridge the gaps between the underlying theory and the algorithmic implementation for TFCL. In this study we propose theoretical classification framework, which provides new insights in the forgetting behaviour analysis and guidance for algorithm design addressing catastrophic forgetting. The primary motivation behind the proposed theoretical framework is that we can formulate forgetting as a generalization error in the domain adaptation theory. Based this analysis we extend the domain adaptation theory (2] to derive time-dependent generalization risk bounds, explicitly explaining the forgetting process at each training step. Inspired by the theory, we devise the Online Discrepancy Distance Learning (ODDL) method which introduces a new expansion mechanism based on the discrepancy distance estimation for implementing TFCL. The proposed expansion mechanism detects the data distribution shift by evaluating the variance of the discrepancy distance during the training. This model enables a trade-off mechanism between the model's generalization and complexity. We also propose a new sample selection approach based on the discrepancy-basee criterion, which guides storing diverse samples with respect to the already learnt knowledge, further improving performance. Our contributions are
This paper is the first research study to propose a new theoretical framework for TFCL., which provides new insights into the forgetting hehaviour of the model in classification tasks.
Inspired by the theoretical analysis, we develop novel dynamic expansion approach, which ensures a compact model architecture enabled by optimal performance.
We propose a new sample selection approach that selects diverse data samples, with respect to the existing knowledge of the model, for the memory buffer, further improving performance.
The proposed method achieves state of the art results on TFCL benchmarks,
2
Related works
Continual learning defines a learning paradigm which aims to learn a sequence of tasks without forgetting. Catastrophic forgetting a major challenge in continual learning. One of the most popular approaches to relieve forgetting is by imposing regularization loss within the optimization procedure
36), where the network's 's parameters deemed important to the past tasks are penalized when updating. Another kind of approaches for continual learning focuses on the memory system, which usually employs a small memory buffer [] 28 36
[1 13
16
19
25
26
31
34
35 40
44 59
for storing some past data or trains generator to provide the replay samples when learning new tasks 47 58 57 (2). However, these approaches usually rely on knowing the task information, which does not correspond to the context from the TFCL.
38 43 46
Task-free continual learning is special scenario in CL where model can only see one or very few samples in each training step/time without having any task labels. Using small memory buffer to store past samples has shown benefits for TFCL and was firstly investigated in 5 551. These memory replay approaches were then extended by employing Generative Replay Mechanisms (GRMs) for training both a Variational Autoencoder (VAEs) [5] and a classifier, where a new retrieving mechanism called the Maximal Interfered Retrieval (MIR) [1. is used to select specific data samples. The Gradient Sample Selection (GSS) K is another sample selection approach that treats sample selection as constrained optimization reduction. More recently, Learner- Evaluator

---

framework proposed for TFCL, called the Continual Prototype Evolution (CoPE) [ stores the same number of samples for each class in the memory in order to ensure the balance replay. Another direction for the memory-based approaches, called the Gradient based Memory EDiting (GMED) [2], consists in editing the stored samples aiming increase the loss in the upcoming model updates. This approach can be employed in existing CL. models to further enhance their performance.
Dynamic expansion models aim to automatically increase the model's capacity to adapt to new tasks by adding new hidden layers and units. The Continual Unsupervised Representation Learning (CURL) [38. dynamically builds new inference models whenever meeting the expansion criterion. However, since CURL still requires a GRM to relieve forgetting, it would lead to negative knowledge transfer when updating the network's 's parameters adapt new task. This issue can be addressed by using Dirichlet processes by adding new components while freezing all previously learnt umits, in a model called the Continual Neural Dirichlet Process Mixture (CN-DPM), (4).. However, the expansion criterion used by these approaches relies on changing the loss when training each time, which does not have any theoretical guarantees.
Theoretical analysis of TFCL
In this section, we firstly introduce the framework of learning settings and notations, and we analyze the forgetting behaviour for single model, and then extend this for dynamic expansion model by deriving their Generalization Bounds (GBs).
3.1 Preliminaries
Let the input space and y represent the output space which is {-1,1} for binary classi- N?
) fication and (1, n',,, n'' 2 for multi-class classification. Let DJ {x] v/ fj=1 and
" " 3 E y' are the image and the associated ground truth label. N and N represent the number of sam- ples for DJ and D respectively. In this paper, we mainly focus on the task-free caas-incrrmmenal learning, as described in the following.
D. = {x}, My Ju= represent the training and testing sets for the i-th dataset where x; E X and
Definition 1. (Data stream.) For given 1-th training dataset DV with Cf data categories, let us
consider data stream S consisting of samples D from each category, expressed as S = - Di, Let Dj represent the set of samples drawn from the j-th category of DJ'. Let IP,,, and P., represent the distributions for D and D respectively. Let TT. represent the marginal distribution over X. f.j
Definition (Learning setting.) Let Tú represent the i-th training step. We assume that there are a total of training steps for learning S, where during each training step T; is learnt small batch of paired samples (X., Y drawn from S. We have S = Uh_, (X. Y!), (X), Y n (X). Y Ø, where i # j. and a model (classifier) can only access (X., ; Y at T, while all previous batches are not available. After finishing all training steps, we evaluate the classification accuracy of the model on the testing set D.".
Definition 3. (Model and memory.) Let consider model h implemented by a classifier, and H = [h] h: X' - }} the space of classifiers. Let us consider a memory buffer M,, defined at Tr which continually adds new data to its memory at each training step and P., the probabilistic representation of M,, We also randomly remove samples from the memory buffer M,, in order to keep fixed its cardinality (number of samples), MMs..
3.2 Measuring the distribution shift
In TFCL, the probabilistic distance between the target domain (testing set) and the source domain (memory buffer) would be dynamically changed during each training step. We can use the discrepancy distance [29 to measure this gap through analyzing the model's 's risk.
Definition (The risk.) For given distribution PS the risk of model h is defined as R(h, P.,)) A
E (x.x)>=p_ (C(y))(x)) where L: y x y + [0, 1] is the loss function.
3

---

Definition 5. (Discrepancy distance.) For two given distributions PT, and P the discrepancy dis- tance Ca defined on two marginals as:
La (P P.S.))
sup (EL
E Bxwezyy [[(())))))) - Bx [=(4((), (
(I)
where {h, E H. In practice, the discrepancy distance Ca(+;-) -) can be estimated as the upper bound considering the Rademacher complexity which is used in the domain adaptation theory as measure of richness for particular hypothesis space [30] [60].
Corollary 1. For two given domains P2 and p... let Up and Up represent sample sets of sizes ".
mp and my, drawn independently from P and P.S.a' Let fr.x and f.S.* represent the empirical distributions for Up and Up. Let [(x', x) = xx' - x|2 be loss function (L1-Norm), satisfying v(x,x') E X, x'x,x') >
M, where M O. Then, with the probability
we have, 22
(V log (2) 2mp
log (1) 2mp
La(PI) P.S. 1J , +J
Ca(PTT) Pi + +3M
(2)
where -== 4g (Reup (H) + Rev, (H)) and Rey. (H) is the Rademacher complexity (Appendix-
from Supplemental Material (SM)). Let CF Tx PS5") represent the Right-Hand Side (RHS) of Eq. (2) We also assume that L: x y + [0, 1] is symmetric and bounded loss function v(y. v') )2 L(y.v)))) s U", and ((.,-) obeys the triangle inequality, where U" is positive number.
3.3 GB for single model
Based the definitions from Section 3.2 we firstly derive the GB for the learning process of single model under TFCL.
Theorem 1.
Let P. represent the distribution of all visited training samples (including all previous batches) drawn from S, at Te. Let hi, arg m R(h, P.) and h.M arg minkeR Rh, TP M,.)) represent the ideal classifiers for P; and PMM respectively. We derive the GB between P. and IP IPM., based on the results from Corollary
Rh, P.) < R(,, hm,, P...) + Ca(P,' ,PPP,PPM) PM))
(P.,
where (P,,, M.)) is the optimal combined error R(hp,, P.) + R(hp; hip, P.) where R.hp,, h,,, P. P...) the risk, expressed by Ex-PM, [C(hp, (x), hM, (x)) and hip the true label- ing function for P..
The proof is provided in Appendix-A from the SM. Compared the GB used in the domain adaptation [29]. Theorem I provides an explicit way to measure the gap between the model's predictions and the true labels each training step (Tt). During the initial training stages (i is very small), the memory M₁ can store all previous samples and GB is tight. However, as the number of training steps increases, the discrepancy distance C (P. P. ) would increase because M, loses the knowledge corresponding to previously learnt data. This can lead to degenerated performance on Pr, corresponding the forgetting process. Next we extend Theorem to analyze the generalization performance on testing sets.
Theorem 2. For a given target domain PL we derive the GB for a model the training step
R R(h, ITT))) S R(h, h,, PM) + CI IPX)) + (PP.),,,M.))
The proof is similar to that for Theorem 1. We can observe that the generalization performance target domain PT,,- by model is relying mainly the discrepancy distance between IT and P...- In practice, usually measure the generalization performance of h several data categories where each category is represented by a different underlying distribution. In the following, we extend Theorem for multiple target distributions.
Lemma For a given data stream = - D. consisting of samples from DJ,, let D. be the corresponding testing set and 187,, represent the distribution of samples for the j-th category from

---

DJ, we derive the GB for multiple target domains as:
c (R(,(R()PPT)) S E (R PM.) + Ca(P.)) ,PX) + (P(p (5)
where CT represents the number of testing data streams.
Remarks. Observations for Lemma 1 : 1) The optimal performance of the model h on the testing set can be achieved by minimizing the discrepancy distance between each target domain ITT.x' and the distribution F,, P., at the training step Te. 2) The study from [7]] employs the set theory to theoretically demonstrate that a perfect memory is crucial for CL. In contrast, evaluate the memory quality using the discrepancy distance in Eq. E which provides a practical way to investigate the relationship between the memory and forgetting behaviour of existing approaches [ [ at each training step without requiring any task information (see the Appendix-D from SM). 3) The study from [51] introduces a similar risk bound for forgetting analysis which requires the task information. In contrast, the proposed GB can be used in more realistic CL scenario and we also provide the theoretical analysis for component diversity (Appendix-C from SM), which is missing in (5l]
3.4 GB for the dynamic expansion mechanism
As discussed in Lemma 1, memory of fixed capacity would lead to degenerated performance on all target domains. The other problem for a single memory system is the negative backward transfer [27 in which the performance of the model is decreased due to samples being drawn from entirely different distributions [1]. A Dynamic Expansion Model (DEM) can address these limitations from two aspects :1) DEM relieves the negative transfer by preserving the previously learnt knowledge into frozen components which are make up mixture system; 2) DEM can achieve a better generalization performance under TFCL by allowing each component to model one or only a few similar underlying data distributions. We derive GB for DEM and show the advantages of DEM for TFCL.
Definition (Dynamic expansion mechanism.) Let G represent dynamic expansion model, with
Gy the j-th component in G, implemented by single classifier. During the initial training phase, G starts with training its first component, and then adds new components during the following training steps. In order to overcome forgetting, only the newly created component is updated each time, while all previously trained components have their parameters frozen.
Theorem 3. For a given data stream S = [X X let P(a)) represent the distribution of the j-th training batch X drawn from S (visited), at T. We assume that G (Gi, , Go} trained
components at Tí. Let T = (Tk,,"" Th. be set of training steps, where G, was frozen at Tr. We derive the GB for 9 at T, as:
Σ (R(L, s Σ (Fs(Pu.))))).
(6)
where Fs(...) is the selection function, defined as
Es (P(())) 2 min v=l.-..c
{R(h, h.me, .....) + La PP
Py PM... + (T(P,.))
(7)
where PP Mr, represents the memory distribution at Tk. The proof is provided in Appendix-B from SM. Notice that F -s(::)) can be used for arbitrary distributions. We assume an ideal model selection in Eq. 77 where always the component with the minimal risk chosen. DEM can achieve the minimal upper bound to the risk (Left Hand Side (LHS) of Eq. (6) when comparing with single model (Theorem 3). Then, in the following we derive the GB for analyzing the generalization performance of G on multiple target distributions.
Lemma 2. For a given data stream S S= U D consisting of samples from D,,, we have a set of target sets (DDT,,... DT L ). where each DE contains Ch batches of samples. Let TPT, represent the distribution of the d-th batch of samples in DTT We assume that consists of components trained on samples from S at Tú. We derive the GB for multiple target domains as
( d R(,,
E
(Fs(,,
(8)
Remark. We have several observations from Lemma 1) The generalization performance of G is relying the discrepancy distance between each target distribution 1o7" and the memory distribution
5

---

PM,e of the selected component (more details in Appendix-C from SM). 2) Eq. 8 provides the analysis of the trade-off between the model's complexity and generalization for DEM adding new components, G would capture additional information of each target distribution and thus improve its performance. On the other hand, the selection process ensures a probabilistic diversity for the stored information, aiming capture more knowledge with minimal number of components.
24 38]. By
In practice, we usually perform the model selection for g by using a certain criterion that only accesses the testing samples without task labels. Therefore, we introduce a selection criterion F(;,)) implemented by comparing the sample log-likelihood:
F(PT,, ((PT5 (d), 9) arg max {Bx=eT, (a)) (xx, GR,)l}:
w=1,"
where f(.,)) ) is pre-defined sample-log likelihood function. Then Eq. (9) is used for model selection: (s( 9) = {R PP.)) + Ca(Pe)) (d), PX) + n (d)
s (d))
(10)
We rewrite Eq. N by using Eq. (10). resulting in:
((M -
>Enn 1-1 {E E (Fs(PT,()))))
(11)
Eq.
when compared with ideal solution (Eq. (8)
) would involve the extra emror terms due to Ch.y
the selection process (Eq. (T0). expressed as Σ
Ed= (TT, 9) (4)))))))
In Section we introduce a new CL framework according to the theoretical analysis.
3.5 Parallels with other studies defining lifelong learning bounds
In this section, we discuss the differences between the results in this paper and those from other studies proposing lifelong learning bounds. The bound from Theorem 1 in (30 assumes that the model is trained on all previous samples, which is not practical under TFCL setting. In contrast, in Theorem from this paper, the model is trained on a memory buffer, which can used in the context of TFCL. In addition, the bounds from Theorem I 30 mainly provide the theoretical guarantees for the performance when learning a new data distribution, which does not represent an analysis of the model's forgetting. In contrast, this paper studies the forgetting analysis of our model and its theoretical developments can be easily extended for analyzing the forgetting behaviour of different continual learning methods, while this cannot be said about the study from [0 (See Appendix-D from supplemental material).
When comparing with [33] our theoretical analysis does not rely on explicit task boundaries, which a more realistic assumption for TFCL. In addition. (33) employs the KL divergence to measure the distance between two distributions, which would require knowing the explicit distribution form and thus hard to evaluate practice during the learning. However, our theory employs the discrepancy distance, which can be reliably estimated. Moreover, the study from (33) does not develop a practical algorithm to be used according to the theoretical analysis. In contrast, our theory provides guidelines for the algorithm design under the TFCL assumption. Finally, inspired by the proposed theoretical analysis, this paper develops a successful continual learning approach for TFCL.
Theorem in 4. similarly to the study from 30 assumes that the model is trained on uncertain data sets over time, which would not be suitable for TFCL since the model can access previously learnt samples under TFCL. In contrast, Theorem in our theory represents a realistic TFCL. scenario in which the model trained on fixed-length memory buffer and is evaluated on all previously seen samples. Therefore. our analysis for the forgetting behaviour of the model under TFCL relies on Theorem and its consequences. In addition, the study from 4. similarly to 30 33), only provides theoretical guarantee for a single model. In contrast, we extend our theoretical analysis to the dynamic expansion model, which motivates us to develop novel continual learning approach for TFCL. Moreover, this paper is the first work to provide the theoretical analysis for the diversity of knowledge recorded by different components (See Appendix-C from supplemental material). This analysis indicates that by maintaining the knowledge diversity among different components we ensure a good of between the model's 's complexity and the generalization performance, thus providing invaluable insights into algorithm design for TFCL.

---

4 Methodology
4.1 Network architecture
Each component G5 9 consists of a classifier h implemented by a neural network fsy (x) with trainable parameters Sj, and a variational autoencoder (VAE) model implemented by an encoding network ene.,, well as the corresponding decoding network doca,s with trainable parameters {w,, } Due to its robust generation and inference mechanisms, VAEs have been widely used in many applications [48 49 50 51] 54, 61]. This paper employs a VAE for discrepancy estimation and component selection. The loss function for the j-th component at T, is defined as:
I.M
Lclass (Gass M.))
{Lie (hy (x.))))))))
(12)
MM
(V) _ - -Ba (x|xx [log Pe, (xe zz] + DKL [9w, (z xx) |[z)]]
(13)
where (x,, v13 Mi, where [.M.] is the memory buffer size, and (.e(.) is the cross-entropy loss. (VAE(::)) is the VAE loss [15), Pa, (x, z) and quy (z x.) are the encoding and decoding distributions, implemented by encw, and deca,. respectively. We also implement f(-,-) in Eq. ( by -LvvE for the component selection at the testing phase. The training algorithm for the proposed ODDL consists of three stages (See more information in Appendix-E from SM). In the initial training stage, we aim to learn the initial knowledge of the data stream and preserve it into component G1 with frozen parameters, which can provide information for the dynamic expansion and can be sampled for the selection evaluation in the subsequent learning. During the evaluator training stage we train the current component as the evaluator that estimates the discrepancy distance between each previously learnt component and the memory buffer, deciding the model expansion. In the sample selection stage, we train the current component with new data, while we aim to promote knowledge diversity among mixture's components.
In the following, we firstly propose new dynamic expansion mechanism based on the discrepancy criterion. Then we introduce new sample selection approach for the memory buffer further improving the performance of the model. Finally, we provide the detailed algorithm implementation.
4.2 Discrepancy based mixture model expansion
From Lemma 2, we observe that the probabilistic diversity of the trained components in G can ensure a compact network architecture while maintaining a good generalization performance (See Theorem 4 in Appendix-CC from SM). In order to achieve this, we maximize the discrepancy between the current memory buffer and the probabilistic representation of each trained component of G, during the training, by using the discrepancy distance (notations are defined in Theorem 3), expressed as
M = arg max Σ, (C P.Y }
(14)
M
where i = Ric + 1,,.. n. represents the index of the training steps and k. is the e-th component trained at Tx. M²* is the closest learnt distribution of one of the components to that of the memory buffer. Eq. (14) can be seen components (c is increased) during the training. We derive an expansion criterion, based on the discrepancy distance from Eq. (4)) while balancing the model's complexity and performance.
recursive optimization problem when & dynamically adds new
min (Ca(PMM,, Px)) 22,
(15)
where À E [0, 4] is an architecture expansion threshold chosen empirically. If the current memory distribution P.M. is sufficiently different from each component (satisfies Eq. (15), 9 will add a new component to preserve the knowledge of the current memory M,,, while also encouraging the probabilistic diversity among the trained components.
In the following, we describe the implementation. We start by training the first component G1 which consists of classifier lú and a VAE model f1, which used for model selection at the testing phase. We also train an additional component called the evaluator G. (he, v. at the initial training stage, which aims to capture the future information about the data stream. Once the memory Mj reaches its maximum size at T/ we freeze the weights of the first component to preserve the previously learnt
7

---

knowledge while continually training the evaluator during the following training steps. Since the evaluator continually captures the knowledge from the current memory M,, we check the model expansion using Eq. 15 at Ti
(16)
Notice that we can not access the previously learnt memory distributions P. "My, and we approximate them by using the auxiliary distributions I formed by samples vf drawn from Gj, where the superscript j represents the first component finishing the training at T5. La(:)) is the estimator of the discrepancy distance, achieved by hy and hf:
of
C2 PM.) Ex-px (x),
ExWPM (((
(17)
If Eq. (T6) is fulfilled, then we add G into the model G, while building a new evaluator (G++1
Gz+") at 1 otherwise, we train G - Gi+ at the next training step, T-+1- Furthermore, for satisfying the diversity of knowledge in the components from the mixture model (See Lemma 2), we clear the memory M; when performing the expansion in order to ensure that we would learn non-overlapping distributions during the following training steps. As illustrated in Fig. [ we can also extend Eq. (16) to the expansion criterion for G [G1, G.-1} after creating and freezing (s - 1) components
min
(T P 2 A
(18)
where P. is the distribution of the generated samples by " and we denote Sj C2 P.M.) vy
by
4.3 Sample selection
Espand aztwork Arthatectare
According to Lemma 2, the probabilistic diver- sity of the knowledge accumulated in the com- ponents is crucial for the performance. In the following, also introduce novel sample selection approach for the memory buffer that further encourages the diversity of the learnt information. The primary motivation behind the proposed sample selection approach is that we desire to store in the memory buffer those samples that are completely different from the data used for training existing components. This mechanism enables the newly created compo- nent to capture different underlying data distri- bution. To implement this goal, we estimate the discrepancy distance pair of samples as
f2
V
Dhamppuryy
S2
Calcabte
Yes
hy
Memory
Cakesaat
Genentr
Ay-3
V
Tiscreprccy
S
Perform the sclestion
Calebire
Figure 1: We generate the known information by using the VAE (V,) of each previous component 1, used to evaluate the discrep- ancy distance S, 2 (Pet, IP, IM... at T. (Eq. (77) between Gj and the memory buffer. Then we use these discrepancy scores (S).... .S........ .S,-13 to check the model expansion (Eq. 18)
G,
1 & L(N (x.), - L(hh (x N (x) )))
E
C9 xx)
(19)
where x, is the e-th sample from M, while k: is drawn from G h (-) is the classifier of G from G. Eq. (19) evaluates the average discrepancy distance between the knowledge generated by each trained component and the stored samples, guiding the sample selection at the training step (T₂)
- --MM-- Mi-D MCD.
(20)
5-1
where M₁ is the sorted memory that satisfies the condition C M[()) > (XC for a < Mi represents the j-th stored sample and d = 10 is the batch size used in the experiments. We name the Online Discrepancy Distance Learning with sample Selection as ODDL-S.
8

---

Tabk
Eatan Sourt
34
33
03-
CIOLE DOSTO FIL
Randin (RONTE
13
30
(a) Risk on the training set.
(b) Risk on all target sets.
(c) Results Split MNIST.
Figure 2: The forgetting analysis of single model and DEM in (a) and (b) where the batch size is 64 the data stream. Data distribution shift and increasing the number of components ODDL-S during the training in (c).
4.4 Algorithm implementation
The algorithm has three main stages (pseudocode is provided in Algorithm 1, Appendix-E from SM)
(Initial training). We start by building two components [G1. G2} where we only add G1 in 9 and consider G. = G2 the Evaluator. G1 and G2 are trained jointly using Eq. (13) and Eq. (12)
during the initial training stage until the memory Mj reaches its maximum size [M maa at certain training step Tj- Then we freeze the first component G1.
(Evaluator training). In this stage, we only update the Evaluator using Eq. 13) and Eq. 12 on
My+1 at 71+1- If [M [M|mar then we evaluate the discrepancy distance using Eq.
to check the expansion (Eq. (18). the expansion criterion is satisfied we add Ge to 9 and build perform the sample selection.
Evaluator while clearing up the memory Mj+1 otherwise,
(Sample selection). We evaluate the diversity score for each stored sample in M using Eq. (19)
and perform the sample selection for MMj+ using Eq. (20). Then we return to the second stage.
5 Experiments
We perform the experiments to address the following research questions: 1) What factors would cause the model's forgetting, and how to explain such behaviour? 2) How efficient is the proposed ODDL-S under TFCL benchmarks? 3) How important is each module in OODL-S?
In this experiment, we adapt the TFCL setting from X which employs several datasets including
Split MNIST 22 53), Split CIFARIO 18 53 and Split CIFAR 100 [I8]. The detailed informa-
tion for datasets, hyperparameters and network architectures is provided in Appendix-F from the supplementary material. The code available at hhttttt:/ /ttpsz:/dddzii1e11 get.ggithuuu. io/ODDL/
5.1 Empirical results for the forgetting analysis
In this section, we investigate the forgetting be- haviour of the proposed model according to the theoretical framework. Firstly, we train a single classifier à on the Split MNIST database, baseline. We consider a memory buffer of max- imum size of 2000, with the batch size of 10; the extra stored samples are randomly removed when the memory full. We evaluate the source and target risks for the proposed ODDL under Split MNIST and plot the results in Fig. 2 We estimate the target risk on all visited training sam- ples (h, P.) and the source risk on the memory R.,h, h.. P..)) We plot the results in Fig. 2 where "Random (Source risk)" represents the source risk of a single classifier, considered as a baseline. At the same time, the baseline tends to increase the target risk on P; with the training steps, as shown in Fig. a. This happens because the memory loses the samples associated with the previous tasks, theoretically explained by Theorem
Table 1: The accuracy of various continual learn- ing models for five independent runs.
Methods
Split MNIST
Split CIFARIO
Split CIFARIOO
finctune
19.75 + 0.05
18:55 0.34
3.53 + 0.04
GEM
93.25 1 0.36
24.13 2.46
11.12 1: 2.48
ICARL*
83,95 + 0.21
37.32 266
10.80 0.37
reservoir
92.16:10075 0.75
42.48 3.04
19.57 1.79
MIR*
93.20 0.36
42.80 2.22
20.00 0.57
92.47 92.47
1.41
13.10
CoPF-CT*
0.877
2.66
CoPE
93.94
21 0.69
GMED
ER, GMED
1.50
CURL*
CNDPM
Dynamic-OCM
94.02 +: 0.23
49, 1.52
21.79 0.68
ODDL
94.85 + 0.02
51.48
0.12
26.20 + 0.72
ODDL-S
95.75 +: 0.05
52.69 0.11
27.21 +: 0.87

---

1. We also evaluate the risk of the baseline on all testing sets (target risk) and plot the results in Fig. b,, where we can observe that the baseline invariably leads large target risks even when the number of training steps increases. Meanwhile, the performance of ODDL on the distribution P. (target risk) does not degenerate during the whole training phase. These results show that ODDL can relieve forgetting and achieve better generalization than the baseline on all target sets, while expanding its architecture to learn novel knowledge.
5.2 Results on TFCL benchmark
We provide the results in Tab. [ where and it denote the results cited from 8 and [12, respectively, We compare with several baselines including: finetune that directly trains classifier on the data stream, GSS 3. Dynamic-OCM [53] MIR [1. Gradient Episodic Memory (GEM) (27). Incremental Classifier and Representation Learning (iCARL) [39,] Reservoir (45] CURL, CNDPM, CoPE, ER + GMED and ER, + GMED [12] where ER is the Experience Replay 42 and ER, is ER with data augmentation. The number of components in ODDL-S and ODDL for Split MNIST, Split CIFARIO and Split CIFAR 100 is 7, 9, 7, respectively. From Tab, the proposed ODDL-S outperforms CNDPM, which uses more parameters, on all datasets and achieves state-of-the-art performance.
In the following, we also evaluate the performance of the proposed approach on the large-scale dataset (MINI- ImageNet (2I]). and Permuted MNIST 19. We split MINI- ImageNet into 20 tasks (See details Appendix-F1 from SM), namely Split MlmageNet. We follow the setting from 1 where the maximum memory size is 10K, and smaller version of ResNet-18 18 [10] is used as the classi- fier. The hyperparameter À used Eq. (15) for expanding ODDL when learning Split MINI-ImageNet and Permuted MNIST (where pixels are randomly premuted is equal to 1.2 and 1.5, respectively. In Tab, we compare with several state-of-the-at methods under Split MIma- geNet, where the results of other baselines are from [12]. From these results, ODDL-S outperforms different base- lines under the challenging dataset.
Table 2: Classification accuracy for 20 runs when testing various mod- els on Split MImageNet and Permuted MNIST.
Methods
Split MImageNett
Permited MNIST
ER.
25.92
1.2
78.11 0.7
ER + GMED
27.27 1.8
78.86 1 0.7
MIR+GMED
26.50
79.25
images)
MIR
22
79.13 0.7
CNDPM
21.12
80.68 +:
ODDL
29
82.33 333::.0.6
ODDL-S
5.3 Ablation study
We investigate the expansion of the ODDL-S according to the proposed discrepancy- criterion. In Fig.2Ic we show the number of components in cach training step for ODDL-S, when training on Split MNIST, where the original MNIST database is learnt segment by segment. In Fig. 2 "Task" represents the number of tasks in each training step. We observe that the proposed discrepancy-basee criterion can detect the data distribution shift accurately, allowing ODDL-S to expand the network architecture each time when detecting the data distribution shift. This also encourages the proposed ODDL-S to use a minimal number of components while achieving optimal performance, as discussed in Lemma 2. We also provide the analysis of how maximize the memory bound in Appendix-F2.2,, while more ablation study results are provided in Appendix-F.2 from supplementary material.
6 Conclusion
In this paper, we develop novel theoretical framework for Task-Free Continual Learning (TFCL), by defining a statistical discrepancy distance between given data distributions and those learnt by the lifelong learning model. Inspired by the theoretical analysis, we propose the Online Discrepancy Distance Learning enabled by selective memory buffer sampling approach (ODDL-S) model, which trades off between the model's complexity and performance. The memory buffer sampling mechanism ensures the information diversity learning. The proposed theoretical analysis provides new insights into the model's forgetting behaviour during each training step of TFCL. Experimental results on several TFCL benchmarks show that the proposed ODDL-S achieves state-of-the-a performance.
10

---

References
[1] Rahaf Aljundi, Fugene Belilovsky, Tinne Tuytelaars, Laurent Charlin, Massimo Caccia, Min Lin, and Lucas Page-Caccia. Online continual learning with maximal interfered retrieval. Advances Neural Information Processing Systems (NewrIPS), pages 11872-11883, 2019.
[2] Rahaf Aljundi, Klaas Kelchtermans, and Tinne Tuytelaars. Tree-free continual learning. In Proc. of IEEE/CVF Conf. Computer Vision and Pattern Recognition, pages 1 11254 1263, 2019.
[3] Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient based sample selection for online continual learning. Advances in Neural Information Processing Systems (NeurIPS), pages 11817-11826. 2019.
[4] Verónica Álvarez, Santiago Mazuelas, and José Antonio Lozano. Minimax classification under concept drift with multidimensional adaptation and performance guarantees. In Proc. International Conference on Machine Learning (ICML), vol. PMLR 162, pages 486-499, 2022.
[5] Jihwan Bang. Heesu Kim, YoungJoon Yoo, Jung-Wor Woo Ha, and Jonghyun Choi. Rainbow memory: Continual learning with a memory of diverse samples. In Proc. of TEEE/CV Conf. on Computer Vision and Pottern Recognition (CVPR), pages 8218 9218-8227, 2021.
[6] A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P. Dokania, P. H. S. Torr, and M.A. Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.100 10486, 2019.
[7] Dai, Q. Yang. R. Xue, and Y. Yu. Boosting for transfer learning. In Proc. Int Conf. on Machine Learning (ICMI.). pages 193-200, 2007.
[8] Matthias De Lange and Tinne Tuytelaars. Continual prototype evolution: Learning online from non- stationary data streams. In Proc. of the IEEE/CVF Int. Conf. on Computer Vision (ICCV), pages 2250-8259. 2021.
[9] Ian J Goodfellow, Mchdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. An empirical investigation of catastrophic forgetting in gradient- neural networks. In arXiv preprint arXiv: 1312,6211 2014.
[10] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proc. of JEEE Conf. on Computer Vision and Puttern Recognition (CVPR). pages 770-778, 2016.
[11] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in neural network. In Proc. NIPS Deep Learning Workshop, arXiv preprint 1rXiv: 1503.02531 2014.
[12] Xisen Jin, Arka Sadhu, Junyi Du, and Xiang Ren. Gradient-based editing memory examples for online task-free continual learning. In Advances in Neural Information Processing Systems (NeurIPS), arXiv preprint arXiv:2006. 15294, 2021.
[13] H. Jung, J. Ju. M. Jung, and J. Kim. Less-forgetiigg learning in deep neural networks. In Proc. AAAI Conf. Artificial Intelligence, volume 32, pages 3358-3365, 2018
[14] Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu, and Lei Shu. Achieving forgetting prevention and knowledge transfer in continual learning. In Advances in Neural Information Processing System, arXiv preprint arXiv:21 volume 34, 2021.
[15] D. P. Kingma and M. Welling, Auto-encoding variational Bayes. arXiv preprint arXiv:1312.6114 2013.
[16] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinske D. Hassabis, C. Clopath, Kumaran, and Hadsell. Overcom- ing catastrophic forgetting in neural networks. Proc. of the National Academy of Sciences (PNAS), 114(13):351-3356, 2017.
[17] Jeremias Knoblauch, Hisham Husain, and Tom Diethe. Optimal continual learning has perfect memory and is Narhard. In Proc. Int. Conf. on Machine Learning (ICML), vol PMLR 119, pages 5327-5337, 2020.
[18] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Univ. Toronto, 2009.
[19] Richard Kurle, Botond Cseke, Alexej Klushyn, Patrick van der Smagt, and Stephan Günnemann. Continual learning with Bayesian neural networks for non-stationary data. In Int. Conf. on Learning Representations (ICLR), 2020.
11

---

[20] Sampo Kuutti, Richard Bowden, Yaochu Jin, Phil Barber, and Saber Fallah. A survey of deep learning applications to autonomous vehicle control. IEEE Transactions on Intelligent Transporiation Systems. 22(2):712-733, 2020.
[21] Ya Le and Xuan Yang. Tiny imageNet visual recognition challenge. Technical report, Univ. of Stanford, 2015.
[22] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient bbsdd learning applied to document recognition. Proc. the IEEE, 86(11):2278 2324, 1998.
[23] Sebastian Lee, Sebastian Goldt, and Andrew Saxe. Continual learning in the teacher-student setup: Impact of lask similarity. In Proc. Int. Conf. on Machine Learning (ICML), vol. PMLR 139, pages 6109-6119, 19, 2021.
(24] Soochan Lcc, Junsoo Ha, Dongsu Zhang, and Gunhee Kim. A neural Dirichlet process mixture model for task free continual learning. In Int. Conf. on Learning Representations (ICLR), arXiv preprint arXiv:2001.00M0 000689, 2020.
[25] Z. Li and D. Hoiem. Learning without forgetting. IEEE Trans. on Pattern Analysis and Machine Intelligence, 90(12):2935-2944 2017.
[26] Sen Lin, Li Yang. Deliang Fan, and Junshan Zhang. TRGP: Trust region gradient projection for continual learning. In Int. Conf. Learning Representations (ICLR), arXiv preprint 2rXiv2220..02931, 2022.
[27] David Lopez-Paz and Marc' Aurelio Ranzato. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pages 6477-6476, 2017.
[28] Divyam Madaan, Jachong Yoon, Yuanchun Li, Yunxin Liu, and Sung Hwang. Representational continuity for unsupervised continual learning. In Int. Conf. on Learning Representations (ICLR), arXiv preprint (e.Xx::110006977 2022.
[29] Yishay Mansour, Mehryar Mohri, and Afshin Rostamizadeh. Domain adaptation: Learning bounds and algorithms. In Proc. Conf. on Learning Theory (COLT), vol. LNCS 7568, 2009.
[30] Mehryar Mohri and Andres Munoz Medina. New analysis and algorithm for learning with drifting distributions. In International Conference on Algorithmic Learning Theory, pages 124-138. Springer, 2012.
[31] Cuong V Nguyen, Yingzhen Li, Thang D Bui, and Richard Turner. Variational continual learning. In Int. Conf. on Learning Representations (ICLR), arXiv preprini arXiv:1710.10625 2018.
[32] G. Parisi, R. Kemker, J. L. Part, C. Kanan, and Wermter. Continual lifelong learning with neural networks: A review. Neural Networks, 113:54-71, 2019.
[33] Anastasia Pentina and Christoph H Lampert. Lifelong learning with non-iid tasks. In Proc. Advances in Neural Information Processing Systems (NIPS), pages 1540 1540-154..77 2015.
[34] R. Polikar, L. Upda, S. S. Upda, and Vasant Honavar. Learn++:
An incremental learning algorithm for supervised neural networks. IEEE Trans. on Systems Man and Cybernetics, Part C, 31(4):497-50 2001.
[35] Mozhgan PourKeshavarzi, Guoying Zhao, and Mohammad Sabokrou. Looking back on learned experiences for class/task incremental learning. In Ini. Conf. on Learning Representations (ICLR). 2022.
[36] Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. GDumb: A simple approach that questions our progress in continual learning. In Proc. European Conference on Computer Vision (ECCV), vol. LNCS 12347, pages 524-540, 2020.
[37] Krishman Raghavan and Prasanna Balaprakash. Formalizing the generalization-foreeeee trade-off in continual learning. Advances in Neural Information Processing Systems, 34, 2021.
[38] Dushyant Rao, Francesco Visin, Andrei A. Rusu, Yee Whye Teh, Razvan Pascanu, and Raia Hadsell. Continual unsupervised representation learning. In Proc. Neural Inf. Proc. Systems (NIPS), pages 7645- 7655, 2019.
[39] Sylvestre-Alve Alvise Rehuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. iCaRL.: Incre- mental classifier and representation learning. In Proc. of the IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pages 2001-2010. 2017.
[40] B. Ren, H. Wang, J. Li, and H. Gao. Life-long learning based on dynamic combination model. Applied Sofi Computing, 56:398 404, 2017.
12

---

[41] Hippolyt Ritter, Aleksandar Botev, and David Barber. Online structured Laplace approximations for overcoming catastrophic forgetting. In Advances in Neural Information Processing Systems (NeurIPS), volume 31, pages 3742-3752, 2018.
[42] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy P. Lillicrap, and Gregory Wayne. Experience replay for continual learning. Advances in Neural Information Processing Systems 34 (NeurlPS), pages 148-358, 2019.
[43] H. Shin, J. K. Lee, J. Kim, and J. Kim. Continual learning with deep generative replay. In Advances in Neural Inf. Proc. Systems (NIPS), pages 2990-2290990 2999, 2017.
[44] Shengyang Sun, Daniele Calandriello, Huiyi Hu, Ang Li, and Michalis Titsias. coormation-theoreeie online memory selection for continual learning. In Int. Conf. on Learning Representations (ICLR), arXiv preprini arXiv:2204, 04763, 2022.
[45] Jeffrey S Vitter. Random sampling with reservoir. ACM Transactions Mathematical Software (TOMS), 11(1):37-57, 1985.
[46] Fei Ye and Adrian G. Bors. Learning latent representations across multiple data domains using lifelong VAEGAN. In Proc. European Conf. on Computer Vision (ECCV), vol. LNCS /2365, pages 777-795, 2020.
[47] Fei Ye and Adrian G. Bors. Lifelong learning of interpretable image representations. In Proc. Int. Conf. on Image Processing Theory, Tools and Applications (IPTA). pages 1-6, 2020.
[48] Fei Ye and Adrian G Bors. Mixtures of variational autoencoders. In Proc. Int. Conf. on Image Processing Theory, Tools and Applications (IPTA), pages 1-6, 2020.
[49] Fei Ye and Adrian G. Bors. InfoVAEAA:: Learning joint interpretable representations by information maximization and maximum likelihood. In Proc. IEEE Int. Conf. on Image Processing (ICIP), pages 749-753, 2021.
[50] Fei Ye and Adrian G Bors. Learning joint latent representations based on information maximization. Information Sciences, 567:216-236, 2021.
[51] Fei Ye and Adrian G. Bors. Lifelong infinite mixture model based on knowledge-driven Dirichlet process. In Proc. of the IEEE Int. Conf. Computer Vision (ICCV), pages 10695-10704, 2021.
[52] Fei Ye and Adrian G. Bors. Lifelong twin generative adversarial networks. In Proc. IEEE Int. Conf. on Image Processing (ICIP), pages 1289- 1293, 2021.
[53] Fei Ye and Adrian G Bors. Continual variational autoencoder learning via online cooperative memorization. In Proc. European Conference on Computer Vision (ECCV), vol. LNCS 13681, pages 531-549, 2022.
[54] Fei Ye and Adrian G. Bors. Deep mixture generative autoencoders. IEEE Transactions on Neural Networks and Learning Systems, 33810)55789-5803, 2022.
[55] Fei Ye and Adrian G Bors. Learning an evolved mixture model for task-free continual learning. In Proc. IEEE Int. Conf. on Image Processing (ICIP), 2022.
[56] Fei Ye and Adrian G. Bors. Lifelong generative modelling using dynamic expansion graph model. In Proc. of the AAAI Conference on Artificial Intelligence, volume 36, pages 8857-8865, 2022.
[57] Fei Ye and Adrian G. Bors. Lifelong teacher-student network learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(10))6280 (2200666688-66,,,,,, 2022.
[58] Fei Ye and Adrian G. Bors. Lifelong mixture of variational autnencoders. IEFE Transactions on Neural Networks and Learning Systems, 34(1):661-4744 2023.
[59] Jachong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online coresct selection for rehearsal based continual learning. In Int. Conf. on Learning Representations (ICLR), arXiv preprint arXiv:2106.01083 2022.
[60] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In Proc. International Conference Machine Learning (ICML), vol. PMLR 97, pages 7404-7413, 2019.
[61] Shengjia Zhao, Jiaming Song, and Stefano Ermon. VAFE: Balancing learning and inference in variational autoencoders. In Proc. AAAI Conf. on Artif. Intel., volume 33, pages 58825 2019.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "l3O5YUjSmZV", "reviewer": "Reviewer_VJEH", "summary": "- The authors derive the first generalisation bound for Task Free Continual Learning. The analysis is inspired by domain adaptation theory.\n- The authors leverage the discrepancy measure to derive a continual learning algorithm which balances between model complexity, generalisation and catastrophic forgetting.\n- In order to do so, the model is designed as a mixture of smaller classifiers. To each classifier is attached a VAE. The VAE is used to flush the memory and model selection. At test time, the generated samples by the VAE are used to compute the discrepancy with the test samples, in order to identify which model to use for inference. ", "questions": "- In the appendix, Table 3, it looks like the average accuracy gain with the method decreases with the complexity or scale of the dataset. Do you have any intuition why the improvement on Split MImageNet is much smaller than for CIFAR-10 and Split-MNIST ? \n- Do you have any indications for practitioners on how to set the threshold lambda ? Specifically in the case of very long streams, where the optimal lambda may vary over time.\n\nA few typos : \n- Replace \"CIFRA\" by \"CIFAR\"\n- Replace \"Memroy\" by \"Memory\" in Figure 2", "limitations": "The limitations and potential negative societal impact are discussed adequately in the section G of the Appendix.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Originality\n### 1- Are the tasks or methods new?\nThe following elements are new : \n- The generalisation bound for Task-Free Continual Learning.\n- Leveraging the tools of domain adaptation theory for continual learning.\n- The final algorithm ODDL with the model expansion, model selection, sample selection and discrepancy as a primitive.\n\n### 2- Is the work a novel combination of well-known techniques?\n- The work builds upon the out of distribution theory, and expands it in meaningful ways to derive an algorithm for Task-free Continual Learning. \n\n### 3- Is it clear how this work differs from previous contributions?\n- The way the contributions of this work stand out is clear.\n\n### 4- Is related work adequately cited ?\n- To my knowledge, the related work is adequately cited.\n\n## Quality\n### 1- Is the submission technically sound ?\n- The submission is technically sound.\n\n### 2- Are claims well supported (e.g., by theoretical analysis or experimental results) ?\n- The theoretical results are supported by proofs.\n- \n\n### 3- Are the methods used appropriate ?\n- Using the VAE to \"summarise\" the distribution of a subcategory looks relevant.\n- Also, the way discrepancy distance is used in the algorithm is theoretically motivated.\n\n### 4- Is this a complete piece of work or work in progress ?\n- It is a complete piece of work.\n\n### 5- Are the authors careful and honest about evaluating both the strengths and weaknesses of their work ?\n- The authors benchmarked their method ODDL against several baselines in Table 1.\n- The linear growth of the number of models in case of a very large number of tasks, was briefly discussed in the appendix, section G.. \n\n## Clarity:\n### 1- Is the submission clearly written?\n- I read the submission a few times, in order to understand the high level picture of the algorithm, and how its components fit.\n- One way to improve clarity is maybe by improving the diagram in Figure 2, by making it more verbose ?\n\n### 2- Is it well organised? (If not, please make constructive suggestions for improving its clarity.)\n- I suggest presenting a big picture of the algorithm first with the main building blocks at a high level, then presenting each element in detail. While reading through section 4.3, I realised I didn't read section 4.1 and 4.2 through the right angle, so I had to go back and forth between both sections a few times to get the right picture.\n\n### 3- Does it adequately inform the reader? (Note that a superbly written paper provides enough information for an expert reader to reproduce its results.)\n- The hyperparameters, code and required resources are shared by the authors.\n\n## Significance:\n### 1- Are the results important? \n- The results are important in the following ways : \n   - A theoretically grounded approach for Task-Free Continual Learning.\n   - An improvement of 7% of average accuracy against the top baseline, on CIFAR-10 and CIFAR-100.\n   - A modular algorithm, which is easy to interpret and debug.\n\n### 2- Are others (researchers or practitioners) likely to use the ideas or build on them?\n- Given the improvements over the baselines in the experiments (Table 1), it is likely that practitioners will benchmark this method. However, given that it requires a non-negligible amount of engineering and hyperparameter tuning, it is likely that practitioners will first benchmark simpler to implement methods.\n- Given that the way the approach is designed is clearly explained and grounded on theory, it make it more likely that researchers will build upon it.  \n\n### 3- Does it advance the state of the art in a demonstrable way? \n- The results advance the state of the art by 7% of average accuracy over the top baseline for CIFAR-10 and CIFAR-100. (Table 1)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_5VsMTWb9Er", "reviewer": "Reviewer_aR21", "summary": "The paper proposes a new theoretical framework to study the task-free continual learning problem. The contribution is twofold:\n* An extension of the domain adaptation theory allowing for multiple time steps and yielding time-dependent generalization bounds: This analysis explains the forgetting behavior with the discrepancy distance between the memory (source) and targets domains. \n* An algorithm inspired by the theoretical results: The discrepancy distance is used to first design a dynamic expansion mechanism. This distance provides a criterion to detect when to add new components to the model.  It is also used to propose a sample selection procedure that insures probabilistic diversity of the information stored in the memory. \nThis algorithm is then tested on different benchmarks and compared to prior task-free continual learning approaches, showing a substantial improvement. ", "questions": "* While the theory has been developed for the task-free case, can it be adapted to also account for the case with known task boundaries? \n* It seems in the initial definitions that it is assumed that the data is presented to the model ordered by category. Is this the case? If yes: While this is common in the literature, how important is this assumption to the analysis? Would it be possible to extend the analysis to also hold for the case where the data distribution is shifting in a smoother way (model exposed to a slowly changing mixture of categories for example)? \n* Can the authors comment on how he model deals with a case where knowledge previously seen is encountered again after initiating a new model? Given that the memory is erased when a new component is initiated, wouldn't this lead to some redundancy in the learnt models? Would it be possible to add a phase to recombine the knowledge encoded in the different components of the model? \n* Can the authors comment of the computational cost of their method, and how the model grows over time, as compared to the prior works? \n* The field has long focused on catastrophic forgetting. The analysis in this paper is also built around fighting this behavior. When training on a stream of data, however, one could argue that the present and future performance should have a higher weight. What do the authors think about this? Can their analysis and proposed method be adapted to look into forward transfer rather than preventing the model from forgetting? ", "limitations": "The authors do not discuss the limitations in the paper. As mentioned above, it would be interesting to consider analysing the computational cost of the proposed method. \n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n* The paper is generally clear and well structured. \n* The take of the authors on the problem is novel to the best of my knowledge. \n* The theoretical analysis proposed in this paper is sound, well motivated and well structured. It is of significance to the community and can shed the light on novel insights for the challenging problem it is focusing on. \n* The algorithm derived from the theory makes sense, and the experiments seem to validate its efficiency. \n\nWeaknesses: \n* Minor: There is an overload of abbreviation in the paper (e.g. SM for supplementary materials, GB for generalization bounds seems non-necessary), and some inconsistency in the notations (e.g.: The indices in the preliminary (3.1) seem wrong at some places, tasks are sometime indexed by t and sometimes by i. This can make reading the paper quite hard. \n* While the training phases of the expansion model are clear, it is quite unclear how the model is used at inference time. It would be great if this can be made clearer. It seems that the VAE is used to decide which trained component to use. Is this the case? \n* While the experiments are diverse and show an improvement over prior works, it is unclear how the methods compare from the computational (and less importantly the memory) cost point of view. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Zcf36qay0eY", "reviewer": "Reviewer_6NW9", "summary": "The paper proposes a theoretical analysis framework for task free continual learning. The presented framework  gives generalization bounds in terms on discrepancies between distributions. Then, the authors propose an algorithm for task free continual learning that maintains samples in a memory buffer and selects the samples measuring the discrepancy between distributions. ", "questions": "In numerical experiments, the proposed algorithms is compared with continual learning methods, is it fair the comparison with methods that are not designed for task free continual learning?\n\nYou evaluate the method on datasets where the distribution changes in a different manner. For example, in the Split CIFAR10 dataset each task contains new classes, while in the Permuted MNIST dataset each task assigns a random permutation. For which scenarios would it be more appropriate to use your algorithm? For example, for those where the distribution changes smoothly, scenarios where only the joint distribution of labels and instances changes, scenarios where the conditional distribution changes?\n\nIt is common in continual learning to assume that tasks are iid or have a common representation, would it make sense to compare your algorithm with methods designed for supervised classification under concept drift? As well as using datasets with concept drift?\n\nIncreasing the complexity of the model would lead to tighter bounds, but what is the model's complexity?\n\nCan the bounds presented be compared with those described in papers designed for scenarios with changing distribution? For instance, “New analysis and algorithm for learning with drifting  distributions.”, “Lifelong learning with non-i.i.d. tasks”, or “Minimax Classification under Concept Drift with Multidimensional Adaptation and Performance Guarantees”.", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written. There are points that should be included in the paper. For example, explain the relationship of the presented bounds with existing bounds for scenarios with changing distribution over time, compare with more methods designed for task free continual learning, and describe the complexity of the method. I include the questions in the following section.\n\nThere are some typos in the paper: \n\t- The authors use the acronym for the supplementary material (SM) in line 147 and define the acronym in line 160.\n\t- Figure 1 appears far from the text.\n\t- There is a repeated \"the\" in line 320\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept