Deep Class Conditional Gaussians for Continual Learning
Authors: Thomas L Lee
Amos Storkey
Venue: Submitted to ICLR 2023 (2023)
Keywords: Contiual Learning
Lifelong Learning
Bayesian
Emprical Bayes
Probabilistic Machine Learning

Abstract: The current state of the art for continual learning with frozen, pre-trained embedding networks are simple probabilistic models defined over the embedding space, for example class conditional Gaussians. However, as of yet, in the task-incremental online setting, it has been an open question how to extend these methods to when the embedding function has to be learned from scratch. In this paper, we propose an empirical Bayesian framework that works by storing a fixed number of examples in memory which are used to calculate the posterior of the probabilistic model and a conditional marginal likelihood term used to fit the embedding function. The learning of the embedding function can be interpreted as using a variant of experience replay, which is a highly performative method for continual learning. As part of our framework, we decide which examples to store by selecting the subset that minimises the KL divergence between the true posterior and the posterior induced by the subset, which is shown to be necessary to achieve good performance. We demonstrate the performance of our method on a range of task-incremental online settings, including those with overlapping tasks which thus far have been under-explored. Our method outperforms all other methods, including several other replay-based methods, evidencing the potential of our approach.

Under review as a conference paper at ICLR 2023
DEEP CLASS CONDITIONAL GAUSSIANS FOR CONTINUAL LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
The current state of the art for continual learning with frozen, pre-trained em- bedding networks are simple probabilistic models defined over the embedding space, for example class conditional Gaussians. However, as of yet, in the task- incremental online setting, has been an open question how to extend these meth- ods to when the embedding function has to be learned from scratch. In this pa- per, we propose an empirical Bayesian framework that works by storing a fixed number of examples in memory which are used to calculate the posterior of the probabilistic model and a conditional marginal likelihood term used to fit the em- bedding function. The learning of the embedding function can be interpreted using a variant of experience replay, which is a highly performative method for continual learning. As part of our framework, by selecting the subset that minimises the KL divergence between the true pos- terior and the posterior induced by the subset, which is shown to necessary achieve good performance. We demonstrate the performance of our method on a range task-incremental online settings, including those with overlapping tasks which thus far have been under-explored. Our method outperforms all other meth- ods, including several other replay-basee methods, evidencing the potential our approach.
decide which examples store
INTRODUCTION
Real world use of deep learning methods can often necessitate dynamic updating of solutions on non-stationary data streams (Farquhar Gal 2018 Antoniou et al. 2020p. This is one of the main problems studied in continual learning and as a result, continual learning has become of increasing interest to the machine learning community, with many proposed approaches (Parisi et al. 2019) and settings (Hsu et al. 2018: Antoniou et al. 2020 Delange al. 2021). Currently, the two biggest challenges in continual learning are catastrophic forgetting and positive transfer. Catastrophic for- getting describes the common occurrence in learning where unconstrained deep models easily forget information derived from previous data after updating on other data. Positive transfer is the ability of model, given the current data, improve its understanding of previous data and of what future data might imply. While there have been significant steps taken to solve these problems (Delange) al. 2021 Mai al. 2021), in many settings there are still gains be made (Farquhar Gal 2018
In common with many works in continual learning, the setting considered here is task-incremental online learning where data stream is split into sequential set of tasks and methods are given information about what the current task is (yan de Ven & Tolias 2019 Prabhu al. 2020). Each task is encapsulated by representative dataset (considered to be sampled i.i.d. from a task distribution) which is given to method batch by batch. Different tasks will generally be associated with different distributions, as well as different target problems. The target problems summarised in task objective function. classes being considered vary between tasks. The overall objective of method, after seeing all of the tasks, is to perform well on all them, given constraints on the amount of memory used by the method.
our case the task objective is classification (Hsu et al., 20188), where the
Currently one of the best ways to approach continual learning is to use a frozen pretrained embedding function and define simple probabilistic model on top to classify the data (Ostapenko al. 2022

---

Under review as a conference paper at ICLR 2023
Hayes & Kanan 2020). However, in some real-world settings it is necessary to learn the embedding function online, for example it might be impossible to pretrain the embedding function due to a lack of data or due to distribution shifts frozen pretrained encoder may become outdated and so perform badly (Ostapenko et al. 2022). Therefore, it is an interesting question to see how these methods can be adapted to settings when the embedding function must be continually learnt. We explore this question, reformulating it into an empirical Bayesian framework and propose a general approach: learning the parameters of the probabilistic model in Bayesian manner while giving a method to learn the embedding function using a conditional marginal log-likelihood loss. The approach also stores a small number of previous examples, used to calculate the posterior of the probabilistic model and the conditional marginal log-likelihood loss. The learning of the embedding function can be seen as a variant of experience replay, this is beneficial as experience replay is one of the best performing continual learning methods (van de Ven & Tolias 2019: WW et al.) 2022] Mirzadeh et al., 2020). Another key part of the approach is a method to select examples to store in memory, which is achieved by minimising the KL-divergence between the true posterior and the one induced by the subset of data to be stored in memory, which is shown to be necessary for the method to achieve good performance (see Section 6.3)
We also present a specific instantiation of our general approach, DeepCCG, where we use a class conditional Gaussian model with unknown mean and fixed variance on top of the neural network embedding function. We chose to explore this particular instantiation because its posterior is easy to compute and the class conditional Gaussian model has been shown to have sfate-of-the-art perfor- mance when using a frozen pretrained embedding function (Ostapenko et al. 2022 Hayes & Kanan, 2020). Additionally, in this case, our method for selecting what samples to store in memory reduces to selecting examples that preserve the means of the per-class clusters formed by the embedded data.
In our experiments we look two specific settings, the commonly used disjoint tasks setting, where each task contains different classes to any other (Delange et al.) (02T) and an underexplored shifting window setting, where there is an overlap between what classes are in each task. The reason we look at a setting with an overlap between tasks is to explore a methods ability for positive transfer, which is more rewarded in this setting as there is greater shared information (Bang et al., (02I). The results of our experiments show that DeepCCG performed the best out of all methods tested, gaining an average performance improvement of 2. 145% in terms of mean average accuracy, showing the potential of our approach.
2 RELATED WORK
When using frozen pretrained embedding functions in continual learning, simple metric-based prob- abilistic models have been shown to have state-of-the-an art performance (Ostapenko et al.] [022] Hayes & Kanan 2020 Pelosin 2022). For example, Hayes & Kanan (2020) show that online linear discriminant anal (LDA) on top of a frozen pretrained backbone is the best performing method in their experiments and Ostapenko et al. 2022) show that class-conditional Gaussian models perform the best in certain settings. These probabilistic models have the advantage that it is possible to learn the same estimate of parameters given any ordering of the data, including i.i.d orderings. However, in many cases frozen pretrained embedding functions cannot be used, which is the case we look at in this work, due to a lack of relevant data for pretraining or due to distribution shift pretrained embedding function will perform badly (Ostapenko et al. 2022).
When the embedding function has be learnt, there are three main paradigms for solving continual learning problems: regularisation, purameter-isolation and replay (Delange et al.) 202T) Regulari- sation methods aim to prevent catastrophic forgetting by adding terms to the loss which try to prevent the network from erasing the information of previous tasks (Kirkpatrick et al. [017]] Huszár 2018)) Purameter-Isolattee methods look at controlling what parameters of a neural network are used for what tasks (Mallya & Lazebnik] 2018). which can often be seen hard version of regularisation methods. Finally, Replay methods aim to solve continual learning problems by storing a subset of previously seen samples, which are then trained alongside new incoming data. This is the approach our method is part of. Replay methods have been shown to have competitive if not the best performance across many settings (van de Ven & Tolias 2019; Wu et al. 2022; Mirzadeh al. 2020). The standard replay method is experience replay (ER) (Chaudhry et al. 2019bb] Aljundi et al. 2019a), which in the setting explored in this work where
not store all the current task's data,
2

---

Under review as a conference paper at ICLR 2023
selects samples to store using reservoir sampling (Vitter 1985), we call this variant ER-reservoit One of the main questions to be answered by a replay- based approach is how to select what samples to store in memory. While reservoir sampling has been shown to be very effective (Wiewel & Yang, 2021) there have been other methods proposed for sample selection, for example ones which use information-theoretie criteria (Wiewel & Yang. 202T) and others maximising the diversity of the gradients of stored examples (Aljundi et al., 90).,). There has also been a Bayesian method pro- posed to select samples called InfoGS (Sun et al., 2022), which is somewhat similar to our method but only uses the probability model to select samples, not for prediction or training, and unlike our work looks at class- incremental learning, where methods do not have access to task identifiers.
There has been considerable work on using Bayesian methods in continual learning (Nguyen et al. 2018 Ebrahimi et al. 2020; Kurle et al. 2020), perhaps inspired by the fact that true online Bayesian inference cannot suffer from catastrophic forgetting (Nguyen et al. 2018). Bayesian perspectives have mainly been used for regularisation based methods, where a popular approach to use varia- tional inference (Nguyen et al. 2018: Farquhar & Gal 2019 Zeno al. 2018). These variational inference methods focus on the offline continual learning setting where a learner has access to all of the data of task at the same time and, in previous work,
often limited to being used in conjunction with small neural networks, mainly due to the need to sample multiple networks when not
calculating the loss (Henning et al., 2021 Nguyen al. 20118. Therefore, these methods suited to the settings we consider in this paper. Bayesian methods have also been used in generative replay based approaches. where instead of storing and replaying real samples they use generated pseudo-samples (Rao et al. (211)). However, when it comes to replay, with real examples, there has been relatively little work on using Bayesian methods, specifically for empirical Bayesian methods where there is requirement to prevent forgetting when fitting the embedding function, which we aim to help to fill in by proposing our method.
In this work we look at two types of settings, one where the sets of class labels are disjoint between tasks and another where there an overlap been the classes seen in each task. Disjoint tasks is a commonly explored setting (Delange et al. (02T) while having some class overlap between tasks is not often studied, but arguably is more realistic. Slightly tangential to the settings considered here, there has been some work looking at class overlap settings where blurry tasks are considered (Prabhu et al. 2020: Bang et al. 2021: Aljunndi et al.) (2019b), that is when a disjoint task setting is modified by adding to each task a small number of samples sampled from any class, often around 10% of the original task's 's size.
3 TASK-INCREMENTAL ONLINE CONTINUAL LEARNING
online continual learning is a setting where the learner given a sequence of
batches of data one by one, along with task identifier for each batch. Different batches may be drawn from different tasks according to some unknown non-stationary process. The goal is for the learner to be able to perform well on all tasks at the end of training (Hsu et al. 2018 Prabhu et al.) 2020).
Specifically, we consider classification tasks. Let X denote a shared data space, and C' denotes the set of all classes being considered. Let l E T denote particular task, which generates data S, = {(x;; vi = 1, .,...,, containing data instances x; E X and labels yı E Cr. where C, C C are the subset of classes relevant for this task. For simplicity, we follow previous work (Delange et al. 2021) and assume the number of classes in each task is the same, i.c. |Cc] = k, Vt.
The learner is provided with a temporal sequence (of length N) of data batches and task identifiers ((D),t,))5 )15 = 1, 2, .., .N), where indexes batches and tD denotes the task identifier for batch Dy- At each time step j. the batch Dj is a sample from a single task and the task identifier tp, determines this task, indicating the classes Can, which are relevant for that task. We use the notation
Dzj to denote D2........ ). the data from all batches seen prior to the time step j.
During training, the learner receives each batch in sequential order; after training on batch the leamerr must discard all the data it does not store in its memory buffer, which is of a fixed size (10 samples per-class in our experiments). At test time, the learnt model is given previously test set from all the classes seen during training. This setting is commonly used in the continual learning literature (Chaddhry et al. 2019a Prabhu al. 2020
3

---

Under review as a conference paper at ICLR 2023
It is often assumed that the set of classes of each task are disjoint (Bang et al. 2021 Prabhu et al. 2020 in our experiments we look at both the disjoint setting and an under-explored shifting window setting. We define the shifting window setting as follows. Let C' {c₂ f==0 and construct task t E (0. [o..... - + 1] by creating a dataset Se which consists of samples which belong to classes Ct.-..; where is the window length and where each sample can only belong one task. In other words, a task & contains samples from all the classes seen in the previous task except for the the class in the previous task with the smallest index and contains samples from a new class Ct+11 The reason we use windowing scheme in one of our settings is to test a method's performance when there is an increased overlap between tasks and where similar tasks are located temporally near to each other, which is a common real world property. We want to explore having large overlap between tasks so to allow the methods to more fully see what across-task information is useful to remember and therefore hopefully reward methods which allow for better forward and backward transfer. This is unlike the disjoint setting, where tasks can have little overlap and so discourages the use of methods which try to allow for a large amount of backwards transfer.
4 CONTINUAL LEARNING OF EMBEDDING FUNCTIONS AND PROBABILISTIC MODELS
To solve the problem of exploiting simple probabilistic models, like LDA, when the embedding
function has to be learnt online, we propose an empirical Bayesian method (shown in Algorithm 1
Let Z denote an embedding space. We assume that we have some embedding function fo X Z
and a fixed memory of size m, which counts the number of items (each an element of X x C x 7)
that can fit in a memory buffer. The memory buffer stores as one element sample (x, and its
associated task index ((a.y) in common with previous work (Chaudhry et al. 2019a). We propose to use Bayesian probabilistic model with parameters 0, such that
plylæ, ((x,))) / p(ylfo(x), O, (O))))
and assume the labels are conditionally independent given the task, 0 and the data instances (each an element of X). Therefore if fg is fixed, we can update our beliefs about the parameters on seeing batch D, and its task identifier tD, by simply using Bayes rule, calculating the posterior P( Dzj). However, in our setting, need to update fs continually, which would change the value of D Dzj). We cannot recompute p() Dzş)) as we are unable to store all the previous samples in memory. Therefore, we propose store in the buffer set of representative samples with which we can compute an approximate posterior after a change in fa. To achieve this goal. after every batch, we select a subset of data points, from the previous memory and current batch, to store in memory. We choose the subset which minimises the KL divergence between the posterior distribution induced by the subset and the posterior distribution induced by the whole data, (L tD,, M,)ll((((M')).
M Mj+1=
arg min
M'C(D, x (tD, J)MM, M'
where Mj is the memory which was carried forward when receiving the jth batch of data.
To update the embedding function, for each arriving batch Dj. task identifier tp, and corresponding memory M,, the method randomly subsamples Rí C Mj of given size r. Then we update/ fo using
Oj+1 02 + V D) R2 ID/ ,I, R2. Staaks; Myi
(3)
D2). DY (vl(,, v) E Dj} and 12? and R2 are defined likewise. This is a gradient step on a conditional marginal log-likelihood (i.e. we compute the marginal log-likelihood on some of the data having conditioned on the rest). Conditioning on part of the data helps to reduce the sensitivity of learning in respect to a potentially uninformative prior and has been shown in previous work to increase generalisation performance (Loti et al. 2022
where Stasks (((.,))(,,)) E R2). 1/ is a learning rate, D3 = (fo())((x,)))
'The marginal likelihood is the average likelihood over models generated from the prior, while the condi- tional likelihood is the average likelihood over models from the posterior induced by M,/R,. R,. Averaging over an uninformative prior will often result in a flat marginal likelihood which is not informative of what embedding function fits the data well. Whereas. conditioning the marginal likelihood on M, /R, the posterior we average over is more informative, having some belief of where the position of embedded data instances should be, and so provides a better signal fit the embedding function.
4

---

Under review as a conference paper at ICLR 2023
Algorithm General approach, learning step at time j
1: input Dj (eurrent batch), Ln, (current task identifier), Mr (current memory), Φ, (current embedding func- tion)
2:
3: Update embedding function
4: R UniformSample(MM,
5: Øj+1 Φ5 + V log PDD) Ry |D .tys 125. Staska, M Rí;
6:
7: Update memory
8:
M
arg min
(,, M,)
M'g(P, x(1, M
D
fo
8j
Lo lo ((N(.)y)//))
Mj/R,
Mj
Figure 1: Diagram of DeepCCG's training routine. At time the learner is given sample of data D, and has memory set stored datapoints My. The memory is split into the replay data R2 and the rest M,/Rj. Learning happens by taking gradient step the parameters of the embedding function o using a conditional marginal likelihood function over D, and Ry, where M is used to induce posterior over the means of the per-class Gaussians and so define the conditional marginal likelihood function used. Therefore, training aims to move the data points into per-class clusters, by drawing the embedded examples of Dj and R, towards their class means and away from the other class means, as shown in the embedding space diagram in the figure.
One way to view our update procedure for the embedding function fo is that it is a variant of experience replay, with different loss function. Experience replay is highly performing method to prevent forgetting (yan de Ven & Tolias 2019 Wu et al., 2022 Mirzadeh et al. 2020). so therefore method to learn the embedding function should be resistant to forgetting. Additionally, that their exists methods to select what examples to replay (Aljundi et al.) 2019a Shim et al. 202T) which are complemeentaryyorttggnnn to this work, and could be combined with our approach. We further note that while our general formulation requires the continual learning setting to provide task identifiers, if p(y) zx, 0, ((r,))) pzy|z, iz, 0). where the distribution of x can still shift between tasks, our general approach could be used in task-agnostic settings as well, which we see as a potential direction for future work.
note
5 DEEP CLASS CONDITIONAL GAUSSIANS
Having described the general method, we explore a specific instance of it in this paper, which we name DeepCCG. We use a neural network for the embedding function fe. We define class condi- tional Gaussian model in the embedding space Z and use the inverse model of that class conditional Gaussian to define p(y|z, f, 0), where the parameters = {u ly E C). that is the parameters of this model are the means of the per-class Gaussians. The reason chose use class conditional Gaussian model is due to having state-of the art performance when using a frozen pretrained em- bedding function (Ostapenko al. 2022 Hayes & Kanan 2020) and that by using its associated conjugate prior, the posterior distribution for # is tractable and easy to compute. Likewise, the marginal likelihood given by the inverse model is fully tractable. More formally, for each task
5

---

Under review as a conference paper at ICLR 2023
define
p(zly. My My)p(t)t)
p [molc E EC)) = Ecec plzle, Mo)p(ctt)
(4)
where
y/t Cat(Cr, (1/1C11)1)
(5)
zly Nayy
(6)
By NO0 Vo).
(7)
In this paper, we choose Vo = aI, where in practice we take - oo, and Cat(C,, (1/(C()) is the uniform categorical distribution over the classes of task
Learning the embedding To update the embedding function fo. in response to the arrival of batch D,. with task identifier tp,, and having received the memory buffer Mj, we take the following steps (shown in Figure I.. First, we select set R3 C My. of size r, from the memory buffer. Then we condition on the rest of the examples, M,/ / Rj. obtain the posterior predictive distributions p(yl (ar), (x.)): MM/RR) R2) for each sample (I,y) E Dj U R2. where ((x,y) is the task identifier for the sample. The posterior predictive distributions can be shown be (see Appendix for the derivation):
N (2 ( R2
M
plul= = Maw)) M,/R,) EoECe N M M Ry (1 )-
where M33, = (f M, y) are the embeddings of samples in the memory buffer with class y. R. is defined likewise and use the notation S to denote the mean of the elements of a set S. As in meta-learning (Garnelo et al.] [0188] Hospedales et al. (202I), the posterior predictive distributions used fixed likelihood term for each individual unseen data item. This leads to a log-marginal likelihood as the objective:
6j+1 = Φ3 + V
Σ (x.))ED,UR,
log(p(y|= =
where ((2,)) is the task identifier associated with sample (x,y).
Sample selection The second key component DeepCCG is the selection of samples for the memory buffer. Because all the information available to inform the value of the parameters so far is encapsulated in the full posterior over the parameters, we target choosing a set of memory samples that best recreate that posterior, preventing as much parameter information loss as possible. Hence, to perform sample selection we minimise the KL divergence between two posterior distributions: the posterior over parameters induced by the new memory being optimized, and the posterior induced by the current batch and the old memory. We keep the number of samples in memory for each class balanced, therefore minimising the KL divergence is equivalent to minimising the per-class KL divergence which reduces to the squared Euclidean distance between the means of a class's embedded data. Formally,
arg min Ms+1.y MiS(Ds, (to, I)MM 1M'
(KL(p(H,)D),)) M,,.D,,,.MMM,MMll.. Mywl)ll(
(10)
arg min MgC(P,, x (tn, 1,'
(1)
j.y U 122113)
(11)
where My++.y y the new memory to be selected for class y. M is the is the set of samples of class in memory when receiving the current batch of data and DJ.y is the set of samples of class in the current batch.
Performing the minimization in Eq. is computationally hard, so we utilise a relaxation of the problem using lasso (Hastie et al. 2009), whereby our method selects the new memory by assigning to each embedded input 2; a zero-to-one weight B; and performing gradient decent on the loss
C(3; Daa, Mia) = D U M
B + ||
(12)
|

---

Under review as a conference paper at ICLR 2023
Then, after termination, our method selects the samples with the largest weights to be the samples stored in memory, where they are stored along with their task identifiers.
6 EXPERIMENTS
6.1 EXPERIMENTAL. SETUP
Benchmarks In our experiments we look at two different settings: disjoint tasks and a shifting window setting. Furthermore. in these settings we utilize three datasets CIFAR-100 (Krizhevsky) 2009). MinilmageNet (Vinyals et al. 2016) and CIFAR- 10 (Krizhevsky, 2009p. where both CIFAR- 100 and MinilmageNet contain 100 classes, while CIFAR-10 contains 10 classes. For disjoint tasks, we split the datasets using the classes of the samples, splitting evenly the classes across a certain number of tasks and assigning all samples with that class to that task-this is often called the split tasks setting in previous work (Delange et al., 2021 Chaudhry et al., 2019a). We split CIFAR- 10 into 5 tasks where there are 2 classes per task and for CIFAR- 100 and MinilmageNNet we split the dataset into 20 tasks with 5 classes per task. While in the shifting window setting we split the datasets up into tasks by fixing ordering of the classes -1:-.., Ct and construct the /th task by selecting a set of samples from classes Ct,..., Ct+I. where is the window length, no two task datasets contain the same example and each task has the same number of samples per-class. For CIFAR-10 10 we use a window length of 2 and for CIFAR-100 and MinilmageNet we use a window length of 5. Additionally, for all experiments we train with 500 samples per-class.
Methods We compare DeepCCG to representative methods of the main paradigms of continual learning. For regularisation methods we compare against a fixed memory variant of EWC (Huszár) 2018 Kirkpatrick et al. 2017) and for parameter-isolation methods we compare to PackNet (Mallya & Lazebnik| (018). For replay methods, which includes DeepCCG, we compare against ER- Reservoir (Aljundi et al., 2019a). A-GEM (Chaudhry al.] 2019a), EntropySS (Wiewel & Yang 2021). GSS (Aljundi et al. 20019b;; where, EntropySS and GSS are memory sample selection strate- gies for ER. All methods are given task identifiers which state what classes are in task and so what logits to mask for predicting samples of that task, which is identical to using a separate head per task in the disjoint task setting. We also compare against two baselines: SGD which is when learning is performed using SGD with no modification and Multi-Task which is rough upper bound and is when we learn the base neural network offline, training on the same number of samples per batch as the replay methods tested, which is 20 (10 new samples and 10 replayed), with the same number of batches, to more fairly upper bound their performance. All methods use the same underlying embedding network for all experiments which is modified ResNet18 8 with six times fewer filters across all layers and Instance Normalisation (Ulyanov et al. 2016) instead of Batch Normalisation layers (Toffe & Szegedy 2013), which is similar to other work (Mirzadeh et al., 2020 Farajtabar let al. 2019).. Additionally, all methods use a batch size of 10, with all replay methods having the same sample replay size of 10 and memory size of 10 samples per-class for all experiments.
Metrics We evaluate the methods using a standard metric for continual learning, average accu- vacy(CCaudhry et al. (2019)). The average accuracy of a method is the mean accuracy on a reserved set of test data across all tasks after training on all tasks. More formally, the average accuracy is A = + Ever at, where a, is the accuracy of a method on the test data of task t after training on all of the tasks.
6.2 RESULTS
In the shifting window setting, we see from Table that DeepCCG performs the best out of all the methods compared. For CIFAR- 100, MinilmageNet and CIFAR-10, DeepCCG gets mean average accuracies of 56.62%, 42.18% and 74. 13%, respectively, which is 2.57%, 1.14% and 3.1% better than any other method, respectively. ER-reservoir is the next best performing method for C1F and MinilmageNet, showing that the memory selection strategies GSS and EntropySS hurt performance for this setting in these datasets. While for CIFAR-10, GSS outperforms ER- reservoir and achieves the second highest accuracy of 71.5%% indicating along with DeepCCG that for this dataset there are better ways of memory selection than uniformly sampling across the stream. It is important note however that while DeepCCG performs better than the others it is still far
7

---

Under review as a conference paper at ICLR 2023
Table I: Results of experiments on CIFAR-100 and MinilmageNet, where report mean average accuracy with their standard errors across three independent runs.
CIFAR-100
ManilmageNet
Method
Shifting Window
Disjoint Tasks
Shifting Window
Disjoint Tasks
EWC
36.65 1.073
42.39 + 0.787
32.42 + 1.129
29.57 + 0.689
PackNet
40.21 + 0.961
50.28 + 0.578
34.15 + 1.280
37.86 + 1.710
ER-reservoir
54.05 0.626
58.31 1.084
41.04 + 1.539
40.29 + 1.077
A-GEM
29.01 + 1.449
39.00 + 0.745
26.97 + 1.264
30.08 = 1.855
EntropySS
51.80 0.700
56.75 1: 0.806
40.03 + 0.609
41.12 = 0.513
GSS
48.20 + 0.332
49.92 - 0.496
37.91 + 0.493
38.77 = 0.980
DeepCCG (ours)
56.62 = 0.288
60.46 = 0.243
42.18 + 0.449
43.04 = 0.638
SGD
35.63 + 1.271
42.50 + 1.403
33.23 + 0.674
31.61 + 0.696
Multi-Task (UB)
90.42 + 1.866
59.76 0.487
89.81 + 0.250
48.46 + 1.211
Table 2: Results of experiments CIFAR-10, 10, where report mean average accuracy with their standard errors across three independent runs.
Method
Shifting Window
Disjoint Tasks
EWC
58.68 1.883
63.33 + 0.871 66.97 + 1.471
PackNet
69.29 + 2.272
ER-reservoir
70.44 + 0.806
66.71 + 0.896
A-GEM
57.63 + 2.163
57.28 + 2.606
EntropySS
67.93 = 0.625
64.96 0.911
GSS
71.55 = 1.45
67.30 + 1.268
DeepCCG (ours)
74.65 + 2.00
69.29 + 0.889
SGD
63.21 + 2.088
63.58 + 0.306
Multi-Task (UB)
96.20 + 0.690
73.37 + 1.820
below the multi-task performance of 90.42%, 89.81% and 96.20% for CIFAR-100, MinilmageNet and CIFAR-10. 10, respectively, unlike in the disjoint tasks setting. This shows that in the shifting window setting where there more overlap between tasks and hence across-task knowledge transfer is more rewarded there is still much work to be done continual learning methods.
For the disjoint tasks setting DeepCCG performs well, outperforming all the other methods tested. For CIFAR-100 100, DeepCCG achieves a mean average accuracy of 60.46%, which is 2.15% higher than any other method: for MinilmageNet, our DeepCCG approach achieves mean average ac- curacy of 43.04%, which is 1.92% higher than any other method and for CIFAR-10, 10, DeepCCG achieves a mean average accuracy of 69. 29% which is 1.99% better than any other method (see Table 2.. In this setting ER-reservoir is the second highest performing method for CIFAR-100 with GSS performing better than it and achieving the second highest performance for the CIFAR- 10 dataset, obtaining mean average accuracy of 67.30% compared to ER-reservoir's 66.71%. As noted above, for the shifting window setting GSS also performed better than ER-reservoir for CIFAR-10 showing that for this dataset its memory selection method is effective. For Minilma- geNet the second highest achieving method was EntropySS, which got mean average accuracy of 41. 11,,,, which indicates along with its performance for the shifting window setting this dataset, that its memory selection method is better than GSS's ' for MinilmageNet.
6.3 ABLATION STUDY
One way to view DeepCCG is as a relative of ER-reservoir, where the methods differ in the sam- ple section mechanism used and the probabilistic model defined on top of the embedding function. Therefore, to analyse where the performance of DeepCCG comes from, we ablate our method by separately changing these two components to what they are in ER-reservoir. The results are pre-
8

---

Under review as a conference paper at ICLR 2023
Table 3: Results of performing ablation DeepCCG, where report results Deep- CCG using reservoir sampling instead of its own sample selection method (DeepCCG-reservoin) when we replace the class-conditional Gaussian model with a standard classifier head (DeepCCG- standardHead) and when we do both modifications, in which case our method reduces to the ER- reservoir method. We report the mean average accuracy with their standard errors across three independent runs on the CIFAR-100 shifting window setting.
Method
Average accuracy
ER-reservoir
54.05 + 0.626 49.95 + 0.743 44.05 + 0.264
DeepCCG-reservoir
DeepCCG-standardHHer
DeepCCG
56.62 0.288
sented in Table 3 and show that both components their do not improve performance over ER ER-reservoir but together they do. We go into detail of the ablation of both components, the sample selection mechanism and probabilistic model, in the next two paragraphs. Additionally, we perform an ablation on the memory size used for the replay methods tested in Appendix D where we show that DeepCCG performs the best over all the memory sizes tested on.
Sample selection To explore the performance of our memory selection mechanism, we compare our full method to when we select samples using reservoir sampling (DeepCCG-reesrvoirr showing that the full method performs the best in the setting tested on. DeepCCG achieves a mean average accuracy 56.62% whereas DeepCCG-reservonl obtains lower mean average accuracy of 49.95% (see Table 3). This shows that our memory selection mechanism is necessary for DeepCCG to obtain good performance. Additionally,
predict that our memory selection mechanism will also perform well in settings with noisy, OOD and/or imbalanced data, unlike reservoir sampling (Sun)
al. 2022
Yoon et al. 2022 Aljundi et al. 2019b), which we leave future work validate.
Class conditional Gaussian model To understand the performance of using a class conditional Gaussian model instead of the standard discriminative softmax model, we look at the performance of DeepCCG without using the class conditional Gaussian model. That is, we evaluate the perfor- mance of DeepCCG when we replace the class conditional Gaussian model with a standard output head (a fully connected layer, then a softmax), while still using our sample selection method- The results show that DeepCCG-standarde achieving an average accuracy of 44.05% (see Table 3). which is significantly worse than when us- ing reservoir sampling, in Ee-reeevoir,, which gets mean average accuracy of 54.05%. This is to be expected as our sample selection strategy designed with the assumption that our embedding function will cluster the data into per-class clusters which may not be true when using standard multi-head output layer, while when using a class conditional Gaussian model the embedding func- tion will be encouraged to create per-class clusters.
calling it CeepCCGG-standardHH
performs poorly,
7 CONCLUSIONS
In this work we have demonstrated that using replay based empirical Bayesian procedure for con- tinual learning is a promising direction and can expand the use of simple probabilistic models from when the embedding function is frozen to when it is learnt online. We have proposed a general ap- proach to learn online both the embedding function and the probabilistic model defined on top of it. As part of the method we proposed a way of selecting samples to store in memory by approximately selecting the best subset that minimises the KI. divergence between the posterior induced by all the currently accessible data and the posterior induced by the subset. We also presented specific in- stantiation of our approach using a class conditional Gaussian model, DeepCCG. The results show that DeepCCG performs well for task-incremental online learning, outperforming all other methods in the commonly used disjoint tasks setting (Hsu et al. 2018) as well as the underexplored shifting window setting, where there is an increased overlap between the tasks. For future work, potential directions would be to look at adapting this approach to class-ineremental learning and in settings with imbalanced and/or noisy data (Sun al. 2022 Yoon al. 2022 Aljundi et al. 2019b).

---

Under review as a conference paper at ICLR 2023
8 REPRODUCIBILITY STATEMENT
To aid in the reproduction of our experiments, we provide a detailed explanation in Section 6. of the experimental setup and provide additional experimental details in Appendix C the code used for the experiments in the supplementary material.
Also, we provide
ETHICS STATEMENT
Continual Learning is a growing field and has the potential to have a large societal impact in the future. For example, a potential application of continual learning is for privacy-awars machine learning on mobile devices, where resource constraints mean that often continual learning solution is necessary. Another example of the potential use of continual leaning is for the adaptation of large models, as re-training large models from scratch when new data becomes available expensive and environmentally costly, therefore continual learning methods may provide way of updating these large models with minimal resource costs. So, this means that research in continual learning could have positive impact on society, however it is also important realise that without conscious ef- fort continual learning may have negative impacts as well. For instance, when deploying continual learning in the real world, mitigating social bias will be problem and importantly solutions pro- posed when using static datasets-for example, modifying the dataset in some way-are often not straightforwardly applicable
continual learning. Therefore, before continual learning methods can be used fully in the real world many problems, known and potentially unknown, need to be addressed but if through future research they are addressed, continual learning can have positive societal impact.
REFERENCES
Rahaf Aljundi, Lucas Caccia, Eugene Belilovsky, Massimo Caccia, Min Lin, Laurent Charlin, and Tinne Tuytelaars. Online Continual Learning with Maximal Interfered Retrieval. In Proceedings of the 33rd Conference on the Advances in Neural Information Processing Systems, pp. 11849- 11860, 2019a.
Rahaf Aljundi, Min Lin, Baptiste Goujaud, and Yoshua Bengio. Gradient Based Sample Selection for Online Continual Learning. In Proceedings of the 33rd Conference on the Advances in Neural Information Processing Systems. pp. 118 11816-11825, 2019b.
Antreas Antoniou, Massimiliano Patacchiola, Mateusz Ochal, and Amos Storkey. Defining Bench- marks for Continual Few-shot Learning. arXiv preprint arXiv:2004.11967 2020.
Jihwan Bang. Heesu Kim, YoungJoon Yoo, Jung-Wop Woo Ha, and Jonghyun Choi. Rainbow Memory: Continual Learning with a Memory of Diverse Samples. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8218-8227, 2021.
Arslan Chaudhry, Mare' Aurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient Lifelong I Learning with A-GEM. In Proceedings the 7th International Conference on Learning Representations, 2019a.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Marc' Aurelio Ranzato. On Tiny Episodic Memories in Contin- Learning. arXiv preprint arXiv: 10486, 2019b.
Matthias Delange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Ales Leonardis, Greg Slabaugh, and Tinne Tuytelaars. A Continual Learning Survey: Defying Forgetting in Classi- fication Tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021. doi: N0.1109/TPAMM.2221.33355
Sayna Ebrahimi, Mohamed Elhoseiny, Trevor Darrell, and Marcus Rohrbach. Uncertainty-guidar Continual Learning with Bayesian Neural Networks. In Proceedings of the 8th International Conference on Learning Representations, 2020.
Mehrdad Farajtabar, Navid Azizan, Alex Mott, and Ang Li. Orthogonal Gradient Descent for Con- tinual Learning. arXiv preprint arXiv: 19100070 2019.
10

---

Under review as a conference paper at ICLR 2023
Sebastian Farquhar and Yarin Gal. Towards Robust Evaluations of Continual Learning. arXiv preprint arXiv:180S.0973: 097333 2018.
Sebastian Farquhar and Yarin Gal. A Unifying Bayesian View of Continual Learning. arXiv preprini arXiv:1902.06494, 2019.
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional Neural Processes. In Proceedings of the 35th International Conference on Machine Learning. volume 80 of Pro- ceedings of Machine Learning Research, pp. 1704-1713, 2018.
Trevor Hastie, Robert Tibshirani, and Jerome H Friedman. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer, 2nd edition, 2009.
Tyler I. Hayes and Christopher Kanan. Lifelong machine learning with deep streaming linear dis- criminant analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. pp. 220-221, 2020.
Christian Henning, Maria Cervera, Francesco D' Angelo, Johannes von Oswald, Regina Traber, Benjamin Ehret, Seijin Kobayashi, Benjamin F. Grewe, and João Sacramento. Posterior Meta- Replay for Continual Learning. In Proceedings of the 35th conference on the Advances in Neural Information Processing Systems, pp. 14135-14149, 2021.
Timothy M Hospedales, Antreas Antoniou, Paul Micaelli, and Amos J Storkey. Meta-Learning in Neural Networks: A Survey. IEFE Transactions on Pattern Analysis and Machine Intelligence, 2021. doi:
Yen-Chang Hsu, Yen-Cheng Liu, Anita Ramasamy, and 7solt Kira. Re-evaluating Continual Learn- ing Scenarios: Categorization and Case for Strong Baselines. In Proceedings of the 3rd Con- tinual Learning Workshop, at the 32nd Conference on the Advances in Neural Information Pro- cessing Systems, 2018
Ferene Huszár. Note on the Quadratic Penalties in Elastic Weight Consolidation. Proceedings of the National Academy of Sciences,
2018. doi: 17.1073/pnas.77110411111
Sergey Ioffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pp. 448-456, 2015.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barwinska,, al. Over- coming Catastrophic Forgetting in Neural Networks. Proceedings of the national academy of sciences, (1((33):32113352 2017.
Alex Krizhevsky. Learning Multiple Layers of Features from Tiny Images. Preprint, 2009.
Richard Kurle, Botond Cseke, Alexej Klushyn, Patrick van der Smagt, and Stephan Günnemann. Continual Learning with Bayesian Neural Networks for Non-Stationary Data. In Proceedings of the 8th International Conference on Learning Representations, 2020.
Sanae Lotfi, Pavel Izmailov, Gregory Benton, Micah Goldblum, and Andrew Gordon Wil- son. Bayesian Model Selection, the Marginal Likelihood, and Generalization. arXiv preprint arXiv:2202.11675 2022.
Zheda Mai, Ruiwen Li, Jihwan Jeong, David Quispe, Hyunwoo Kim, and Scott Sanner. Online Con- tinual Learning in Image Classification: An Empirical Survey. arXiv preprint 1xX.::2001.1022) 2021.
Arun Mallya and Svetlana Lazebnik. PackNet: Adding Multiple Tasks Single Network by Iterative Pruning. In Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition, pp. 7765-7773, 2018
11

---

Under review as a conference paper at ICLR 2023
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Razvan Pascanu, and Hassan Ghasemzadeh. Under- standing the Role of Training Regimes in Continual Learning. In Proceedings of the 33rd confer- ence on the Advances in Neural Information Processing Systems, pp. 7308-73221 2020.
Cuong V. Nguyen, Yingzhen Li, Thang D. Bui, and Richard E. Turner. Variational Continual Learn- ing. In International Conference on Learning Representations, 2018.
Oleksiy Ostapenko, Timothee Lesort, Pau Rodríguez, Md Rifat Arefin, Arthur Douillard, Irina Rish, and Laurent Charlin. Foundational models for continual learning: An empirical study of latent replay. arXiv preprini arXiv:2205.00322) 2022.
German I. Parisi, Ronald Kemker, Jose L.. Part, Christopher Kanan, and Stefan Wermter. Continual Lifelong Learning with Neural Networks: A review. Neural Networks, 113:54 71, 2019. ISSN 0893-6080.
Francesco Pelosin. Simpler better: off-the-shelf continual learning through pretrained backbones. arXiv preprint vrXiv:2205.01586 2022.
Ameya Prabhu, Philip HS Torr, and Puneet K Dokania. GDumb: A Simple Approach that Questions our Progress Continual Learning. Procceding the 16th European Conference on Computer Vision, pp. 524-540, 2020.
Dushyant Rao, Francesco Visin, Andrei Rusu, Razvan Pascanu, Yee Whye Teh, and Raia Hadsell. Continual Unsupervised Representation Learning. In Proceedings of the 33rd Conference on the Advances in Neural Information Processing Systems, 2019.
Dongsub Shim, Zheda Mai, Jihwan Jeong, Scott Sanner, Hyunwoo Kim, and Jongseong Jang. On- line Class-Increman the 35th AAAI Conference on Artificial Intelligence, 2021.
Continual Learning with Adversarial Shapley Value. In Proceedings of
Shengyang Sun, Daniele Calandriello, Huiyi Hu, Ang Li, and Michalis Titsias. Information- Theoretic Online Memory Selection for Continual Learning. In Proceedings of the 10th Inter- national Conference on Learning Representations, 2022.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance Normalization: the Missing In- gredient for Fast Stylization. arXiv preprint 1rXiv:1607.0880 2016
Gido M van de Ven and Andreas Tolias. Three Scenarios for Continual Learning. arXiv preprint 1eXiv:/904.0770 2019.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, and Daan Wierstra. Matching Networks for One Shot Learning. Proceedings of the 30th Conference on the Advances in neural information processing systems, 2016.
Jeffrey S Vitter. Random Sampling with Reservoir. ACM Transactions on Mathematical Software, 11(1):37-57, 1985.
Felix Wiewel and Bin Yang. Entropy Entropy-Based Based Sample Selection for Online Continual Learning. In Proceedings of the 28th European Signal Processing Conference, pp. 1477-1481, 2021. doi: 10.239 7 202.6698222222888466
Tongtong Wu, Massimo Caccia, Zhuang Li, Yuan-Fang Li, Guilin Qi, and Gholamreza Haffari. Pretrained Language Models in Continual Learning: A Comparative Study. Proceedings of the 10th International Conference on Learning Representations, 2022.
Jachong Yoon, Divyam Madaan, Eunho Yang, and Sung Ju Hwang. Online Coreset Selection for Rehearsal- based Continual Learning. In Proceedings of the 10th International Conference on Learning Representations, 2022.
Chen Zeno, Itay Golan, Elad Hoffer, and Daniel Soudry. Task Agnostic Continual Learning Using Online Variational Bayes. arXiv preprint arXiv:1803.10122 2018.
12

---

Under review as a conference paper at ICLR 2023
A ALGORITHM FOR DEEPCCG
Algorithm DeepCCG, learning step time j
I: input Dj (currett batch), Ln, (current task identifier), M, (current memory), Φ (current embedding func- tion)
2:
3: Update embedding function
4: Rj = UniformSample(Mj
5: Ø5+1 < 0j + V LiaweDD,R log(p(yl= fo(x), Marwi, Mz)
6:
7: Update memory
8: for each class in M, do
initialise
10:
to do
11: B < 5 v (33 Dj.c: Mz.c)
12: end for
13: Set M be the of samples, with their task identifiers, with the largest & values
end for
B DETAILS OF LEARNING THE EMBEDDING FUNCTION
For our deep class conditional Gaussian method (DeepCCG), to compute the update to the embed- ding function fe. when given batch D, and corresponding memory Mj, proceed using the following steps. Firstly, we randomly sample R, C M, of size r from memory. Then using only the other samples in memory, Mj/ Rý. we compute the posterior density for each class mean- Pec with c E C:
p(polMy/Rg) pluclM5.o//
(13)
=N (1n,lMM
(14)
m
where Mj,e (f Ay c} are the embeddings of points in the memory buffer with class y. R2 is defined likewise and we use the notation to denote the mean of the elements of S. Then we compute the posterior distribution of the embedded inputs E Dj U R2 for each
class e E C' utilizing
M,,R,) = p(z|c, Mj,o/
(15)
f p(alc,, MMj.o/ R3 dde
(16)
=N (
1 -r)
(17)
Next, we compute the posterior predictive for each sample (x.v) E D, U R with a task identifier t(m.))) which known for samples in the current batch and stored by our method for samples stored in memory, and where fo (x) using
p(zly. M / R)))((l(()))))
p(yl,,4a11 R,,
(18)
E.so p = c, M,jjel R3) pp)p( c(tmm)
p(:ly. M j.y R2)
(19)
EcECo (x.)))
p = c, M j.cl R3)
Finally, we update the embedding function by performing gradient step using the formula
0++1 = Φ + V
Σ
log(p(y) 2 = fx( (Z.w), M₁/ R2)), M//Rg)),
(20)
(E.M)D,,UR,
which can be seen a per-sample conditional marginal lieelhodl.....
13

---

Under review as a conference paper at ICLR 2023
By using Eq. 17 and 19 the closed form of the posterior predictive distribution for sample (z. y) with task identifier & ((2.)) and where 2 = fa(z) is
p(zly, M Ri))
plulz, (a. M₁) - EeeOr p== = C, M.je//P.
(21)
N [ j.yl RBj, m-r)ll]
(22)
Eneorawn N (z)Mjeol Riy, m
C ADDITIONAL EXPERIMENTAL DETAILS
In addition to the experimental details mentioned in the main body of the paper, there are a few more details to mention. Firstly, for methods with hyperparameters same experimental set up as the real experiments and 10% of the training data as validation data. This led the selection of the hyperparameters of EWC and PackNet, the methods with free hyperparameters, shown in Table Secondly, we use the same learning rate 0.1 and the standard gradient decent optimiser for all methods. These were chosen by looking the commonly selected values in previous work (Mirzadeh et al., (020) and were shown to be performative for all methods tested. Finally, all experiments were run on laptop with a single NVIDIA GeForee GTX 1050 GPU.
performed a grid search using the
Table Values selected for the hyperparameters of EWC and PackNet, which are the regularisation coefficient and the percentage of available filters to be used per task, respectively. SW stands for shifting window and DT stands for disjoint tasks. MinilmageNet
CIFAR-100
CIFAR-10
Method
SW
DT
SW
DT
SW
DT
EWC
6
9
6
PackNet
0.1
0.1
0.05
0.2
0.3
0.3
D RESULTS ON USING DIFFERENT MEMORY SIZES
Table 5: Results of experiments looking the effect of memory buffer size for the replay methods tested, using the shifting window setting on CIFAR-100. We report mean average accuracy with their standard errors across three independent runs.
Method
m=750
m=1000
m=1250
ER-reservoir
53.17 + 0.656
54.05 + 0.626
55.22 + 0.592
A-GEM
27.18 + 1.091
29.01 1.449
27.87 + 0.172
EntropySS
+ 5 0.725
51.80 = 0.700
53.34 + 0.372
GSS
47.15 + 0.766
48.20 + 0.332
46.18 + 0.341
DeepCCG (ours)
53.42 + 0.460
56.62 0.288
57.98 + 0.514
One additional useful experiment is looking at the relationship between performance and the size of memory used for DeepCCG. Therefore. in Table we show the performance of DeepCCG and the other replay methods compared against with varying memory size. Table 5 shows that when the memory size is increased to m = 1250, DeepCCG has improvement in average accuracy relative to other methods, as it achieves 2.76% better than any other method for m 1250, while for 1000 it achieves 2.57% better than any other method, 0.19% improvement. When the memory size is decreased to m = 750, we see that DeepCCG's performance drops more than other methods it is only 0.25% better than other methods in this case. Therefore, experiments show that compared to other replay methods DeepCCG's performance increases the most when m increases and that for small buffer sizes DeepCCG performs less well, potentially due the fact that
14

---

Under review as a conference paper at ICLR 2023
the examples in the memory buffer used to infer the posterior over the means of the per-class Gaussians and so the method needs given amount of examples to specify the means well. We also note that in our experiments the performance ranking of the methods does not change with
15


Reviews:
[{"review_id": "yfCQuA0ogyM", "reviewer": "Reviewer_juyJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors proposed an approach for learning embedding function and a probabilistic model on top of it for task-incremental online learning. The method has two update steps, first updating the model parameters based on the log-marginal likelihood of the probabilistic model and then update a memory bank of class representative examples by optimizing the kl divergence between a current posterior and a candidate posterior which considers data from the current task. The authors presented an instance of this algorithm with Gaussian distributions. The authors presented results of their method on CIFAR-10/100 and miniImageNet datasets in two settings, disjoint tasks (in terms of classes), and a shifting window.", "summary_of_the_review": "I think that the paper is nice, but I am not sure that in terms of novelty, the approach, and the compared baselines it is good enough to be accepted. I am willing to change my mind in light of new information from the authors.", "strength_and_weaknesses": "Strengths:\n* A simple method that seems to work well compared to the baselines.\n* The method was evaluated in two settings, disjoint tasks and a shifting window which shows that it is relatively robust.\n* The paper is easy to follow and understand.\n* Code was provided, which is noteworthy.\n\nWeaknesses/Questions:\n* Although I am not an expert in the field, it seems that the true novel part is the KL divergence loss for selecting the memory bank. I would imagine that there were other strategies for making this selection in the continual, life-long, or incremental learning setups, so why this strategy is better than others? \n* Following the last question, Given that the authors chose a Gaussian class conditional distribution with unit covariance, I can imagine that the resulting selection strategy appeared in other papers as a frequentist loss function. Have the authors checked that?\n* Lastly on the same subject, why did the authors assume a unit covariance? is it a realistic assumption? I would imagine that even an isotropic covariance matrix that requires inference on an additional parameter per class will be more suitable.\n* I wonder what is the computational overhead of the two-step learning algorithm (i.e., learning the embedding and learning the probabilistic model on of them at each iteration), can you quantify that using wall-clock time or FLOPS?\n* To me it seems that most baselines are outdated, how does the method compare to newer ones?\n* It seems that the authors missed an important related work [1, 2] that share some of the assumptions of this work (for instance, [1] also assume that classes are grouped in a Gaussian ball in the feature space). In general, I wonder how this method or alternative methods that use GPs will work on the tasks presented in this paper. This seems like an important baseline.\n* Which of the baselines learn the embedding function and which isn't? It seems like important information and I didn't find an answer to that.\n\n[1] Achituve, I., Navon, A., Yemini, Y., Chechik, G., & Fetaya, E. (2021, July). GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning. In International Conference on Machine Learning (pp. 54-65). PMLR.\n\n[2] Titsias, M. K., Schwarz, J., Matthews, A. G. D. G., Pascanu, R., & Teh, Y. W. (2019, September). Functional Regularisation for Continual Learning with Gaussian Processes. In International Conference on Learning Representations.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "tHDfRO6ltT", "reviewer": "Reviewer_e2L5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the problem of lifelong learning, specifically without a fixed backbone by using a Bayesian approach. The approach shows good results in a challenging setting.  ", "summary_of_the_review": "The paper addresses an important challenge and gets improved results compared to other approaches", "strength_and_weaknesses": "Strengths:\n- The paper is well written and easy to follow\n- The approach is relatively novel. The main focus in continual learning is on fixed backbone, also the shifted setting got relatively little attention.\n- The results seem promising compared to other baselines.\n\nWeaknesses:\n- Not enough comparisons, cifar10 with only 10 classes is not a good benchmark for this method and cifar100 is a relatively easy benchmark. Would be better to compare on other benchmarks like CUB. Also use only a scaled-down version of resnet18\n- Overall results aren't that good, maybe because the difficult setting but also could be due to the network used. Would be more convincing if it would work with a larger network", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "FPVwRI07-Z", "reviewer": "Reviewer_U88A", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a deep class conditional model for the online task's incremental learning. The primary motivation of the paper is to learn the feature embedder since a fixed feature extractor is not useful in the dynamic environment. In this work, the model leverages the replay buffer to overcome the catastrophic forgetting and the buffer samples are selected by maximizing the posterior between the subset of samples and task samples. They used jointly the buffer and batch samples to update the embedder network parameter. Finally, the samples are classified by learning the prototype/mean of the class.", "summary_of_the_review": "The paper uses the simplest setting of CL and without replay recent model shows the result close to the upper-bound.\n\nRecent comparison with the similar and recent work is missing \n\nPaper has some novelty, but that is not significant for the publication. ", "strength_and_weaknesses": "Pors:\n\n1: The model is simple and requires a small memory buffer to overcome the catastrophic forgetting. \n\n2: Easy to follow and ablation for the sample selection strategy w.r.t. reservoir shows the improvement. Sample selection method is intuitive and it may be useful in the other replay based approach. However, practical time complexity may be a bottleneck.\n\n3: For the reproducibility, they provided the code.\n\nCons:\n\n1: The paper handles the easiest setting of the continual learning (CL) where task id are provided during inference, which is not practically useful. In this setting, a simple expansion based approach [a,b,c,d] without any replay, model achieves the performance near the upper-bound without using the pretrained backbone. In this paper, none of these methods are compared. Replay is required for the comparatively much complex setting, like class incremental learning or task-free continual learning.\n\n2: The model is not end-to-end, it learns the embedder and classifier separately. The embedder learns the class specific cluster representation and class conditional Gaussian learns the prototype classifier. Also, learning only the mean is too simple and it may not handle the complex scenarios. \n\n3: The sample selection strategy requires finding the subset of samples to maximize the posterior. Is there any optimization based model to select this subset of samples? If not, it’s like a brute-force model and sample selection is too costly.\n\n4: Novelty is limited conditional marginal likelihood was explored in earlier work, sample selection approach seems novel but the complexity of it may be a bottleneck.\n\n5: The baselines are weak and no recent model are included. How the author created the baseline for the previous work and upper-bound? The implementation details of the baseline are missing, do the author did the proper hyperparameter search for the baseline? \n\n[a] Calibrating CNNs for Lifelong Learning, NeurIPS-20\n\n[b] Ternary Feature Masks: zero-forgetting for task-incremental learning, CVPRW-21\n\n[c] Supermasks in Superposition, NeurIPS-20\n\n[d] Continual Learning using a Bayesian Nonparametric Dictionary of Weight Factors, AISTATS-21\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "1gJEwS41Y6", "reviewer": "Reviewer_rHsD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes using an empirical Bayesian framework to simultaneously learn an embedding model ($f_\\phi$) and a probabilistic model (parameterized w/ $\\theta$) for the continual learning setting. To that end, $f_\\phi$ is updated using a variant of experience replay through a running memory of samples, where the memory samples are updated with each task such that the KL divergence between the true posterior for $\\theta$ and the posterior with the new memory is minimized. While updating the embedding network, the authors use marginal log-likelihood wrt $y$ of a subset of data conditioned on the rest. The paper presents one instance of the proposed framework referred to as DeepCCG which uses conditional Gaussian model as the probabilistic model. Several experiments show the DeepCCG performs better than a few selected baselines in two different (disjoint and shifting window) task incremental settings. ", "summary_of_the_review": "Based on the above weaknesses and concerns, my recommendation is to reject the paper in its current state. While the idea of using DCCG w/ experience replay is interesting, there are some major concerns in terms of clarity and presentation. The baselines and related work can also be improved significantly.", "strength_and_weaknesses": "Strengths\n\n- The proposed DeepCCG seems to do reasonably well in the task-incremental setting compared to some of the considered baselines.\n- The experiment design of using a moving window to simulate task overlap has not been explored before.\n\nWeakness\n\n- *Missing related work*: There is an entire class of methods that do expansion-based [1-5] continual learning that have not been discussed in the related work. [1-4] also use a Bayesian framework (though for expansion) for continual learning. These methods also learn the entire model without using any pre-trained/frozen embedding network. This class of methods should be discussed in the related work.\n- *Replay Baseline*: Replay based on generative models [5-6] has not been considered as a baseline.\n- The writing and the presentation in the paper can be improved significantly. In terms of presentation, section 4 seems like an over-generalization of the methodology presented in section 5. Since the paper only discusses a simple variant of the general framework, which uses a class conditioned Gaussian to define the probabilistic model, section 4 seems a bit redundant. \n- Some of the claims over-state the contribution, for e.g. Pg 5 section 4 (last para): \"our general approach can be used in task-agnostic settings as well\" -- The task-agnostic continual learning is a much more challenging setting than task incremental. Without any empirical evidence, this claim is not well supported. \n- The sample selection strategy based on the KL divergence of the two posteriors in equation 11 assumes that the training data in each task is same across the tasks? From equation 11, it seems that given a new task with large number of samples, the new memory would be biased towards the new task, which could easily cause catastrophic forgetting of previous tasks. However, the paper only assumed settings with equal number of samples across all tasks.\n- The paper only considers the task-incremental setting, which is known to be a relatively easier setup than other challenging continual learning settings, including class-incremental and task-free continual learning. \n\n\nOther Minor concerns:\n- The notation in the paper can be improved:\n  - In equation 1, the marginalized likelihood is a function of the embedding network. Making this precise in the notation (e.g. $p_\\phi (y|x,t_{x,y})$) would make the discussion that follows in section 4 clearer.\n- In section 6.1 under benchmarks: \"in shifting windows ... there is no overlap between any two of the task\" -- Is this a typo? There is overlap with shifting windows, right? \n\n\n[1] A Neural Dirichlet Process Mixture Model for Task-Free Continual Learning, (Lee et al., (2020))\n[2] Continual Learning using a Bayesian Nonparametric Dictionary of Weight Factors (Mehta et al., (2021))\n[3] Bayesian structure adaptation for continual learning, (Kumar et al. (2020))\n[4] Hierarchical Indian Buffet Neural Networks for Bayesian Continual Learning, (Kessler et al, (2020))\n[5] Efficient Feature Transformations for Discriminative and Generative Continual Learning, (Kumar et al., (2021))\n[6] Deep Generative Replay, (Shin et al. (2017))\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject