Year: 2019

Abstract:
We introduce a unified probabilistic approach for deep continual learning with open set recognition, based on variational Bayesian inference. Our model combines a joint probabilistic encoder with a generative model and a linear classifier that get shared across tasks. The proposed open set recognition bounds the approximate posterior by fitting regions of high density on the basis of correctly classified data points and balances open set detection with recognition errors. Catastrophic forgetting is significantly alleviated through generative replay, where the open set recognition is used to reject statistical outliers from low density areas of the class specific posterior. Our approach naturally allows for forward and backward transfer while maintaining past knowledge without the necessity of storing old data, regularization or inferring task labels. We demonstrate compelling results in the challenging continual learning scenario of incrementally expanding the singlehead classifier for both class incremental visual and audio classification tasks, as well as incremental learning of datasets across modalities. We further quantitatively demonstrate our models ability to successfully distinguish unseen unknown data from trained known tasks and subsequently prevent misclassification.

heading: 1 INTRODUCTION
text: Most machine learning systems make the closed world assumption and are predominantly trained according to the isolated learning paradigm, where data is available at all times and is independently and identically distributed. However, in the context of continual learning, where tasks and data arrive in sequence, neither of these two principles is desirable. A neural network that is trained exclusively on a new task’s data forgets past knowledge and suffers from an early identified phenomenon commonly referred to as catastrophic forgetting (McCloskey & Cohen, 1989). Moreover, to overcome the closed world assumption, inclusion of a ”background” class is veritably insufficient as it is impossible to include all unseen concepts and classes explicitly in the loss function beforehand. Likewise, commonly applied thresholding of prediction values doesn’t prevent resulting large confidences for unseen classes if the data is far away from any known data (Matan et al., 1990).
Most of the existing continual learning literature concentrates efforts on either alleviating catastrophic forgetting, maximizing knowledge transfer or addressing ways in which to efficiently store subsets of past data. These works have identified weight regularization (McCloskey & Cohen, 1989; Zenke et al., 2017; Kirkpatrick et al., 2017; Li & Hoiem, 2016; Nguyen et al., 2018) and rehearsal techniques (Ratcliff, 1990; Lopez-Paz & Ranzato, 2017; Rebuffi et al., 2017; Bachem et al., 2015) or have postulated methods based on complementary learning systems theory (O’Reilly & Norman, 2003) through dual-model with generative memory approaches (Gepperth & Karaoguz, 2016; Shin et al., 2017; Wu et al., 2018; Farquhar & Gal, 2018; Achille et al., 2018) as mechanisms against catastrophic inference. On the one hand, regularization techniques can work well in principle, but come with the caveat of relying on a new task’s proximity to previous knowledge. On the other hand, training and storing separate models, including generative models for generative rehearsal, comes at increased memory cost and doesn’t allow for full knowledge sharing, particularly to already stored models. Specifically, the transfer of already attained knowledge to benefit new tasks, known as forward transfer, as well as the potential positive impact of learning new concepts to aid in existing tasks, known as backward transfer, are crucial to any continual learning system. Generally speak-
ing, most current approaches include a set of simplifications, such as considering separate classifiers for each new task, referred to as multi-head classifiers. This scenario prevents ”cross-talk” between classifier units by not sharing them, which would otherwise rapidly decay the accuracy (Zenke et al., 2017; Kirkpatrick et al., 2017; Rusu et al., 2016; Shin et al., 2017; Gepperth & Karaoguz, 2016; Rebuffi et al., 2017; Achille et al., 2018; Nguyen et al., 2018) as newly introduced classes directly impact and confuse existing concepts. In the multi-head scenario task ids thus need to be encoded or are often assumed to be given by humans in order to know which classifier to use for prediction. Correspondingly, in generative replay, generative and discriminative models are taken to be separate models (Shin et al., 2017; Farquhar & Gal, 2018; Nguyen et al., 2018). Similar to regularization of a classifier, a generative model can suffer from the learned approximate posterior distribution deviating further from the true posterior with each further task increment. In order to avoid catastrophic forgetting induced by learning to generate on previously generated data, previous works even store a separate generative model per task (Farquhar & Gal, 2018), in analogy to the multi-head classifier. An extended review of recent continual learning methods is provided by Parisi et al. (2019).
A parallel thread pursues a complementary component of identifying out-of-distribution and open set examples. While current continual learning approaches typically do not include this thread yet, it can be considered crucial to any system and a necessity in order to avoid encoding task labels and distinguishing seen from unknown data. Here, multiple methods rely on using confidence values as means of rejection through calibration (Liang et al., 2018; Lee et al., 2018b;a). Arguably this also includes Bayesian approaches using variational methods (Farquhar & Gal, 2018; Achille et al., 2018) or Monte-Carlo dropout (Gal & Ghahramani, 2015) to estimate uncertainties. Since the closed world assumption also holds true for Bayesian methods as the approximated posterior probability cannot be computed for unknown classes, misclassification still occurs, as the open space risk is unbounded (Boult et al., 2019). Recently Thomas et al. (2014); Bendale & Boult (2016); Dhamija et al. (2018) have proposed extreme value theory (EVT) based open set recognition to bound the open-space risk and balance it with recognition errors in deep neural networks.
In this work we propose a probabilistic approach to unify open set recognition with continual learning in a single deep model in order to remove or alleviate above mentioned common simplifications. Specifically, our contributions are:
• We introduce a single model for continual learning that combines a joint probabilistic encoder with a generative model and a linear classifier. Inspired by EVT based open set recognition (Bendale & Boult, 2016) for Softmax prediction layers, this model architecture gives rise to a natural way of open set recognition with statistical outlier rejection on the basis of the approximate posterior in Bayesian inference. The latter requires no upfront knowledge of open set data or corresponding modifications to loss or training procedure and can successfully prevent nonsensical predictions for unseen unknown data, a robustness feature that is currently not present in closed world continual learning systems.
• We show how this EVT bound to the posterior can be used for both identification and rejection of statistically outlying unseen unknown data instances, as well as exclusion of generated samples from areas of low probability density. When used in generative replay this leads to significantly reduced catastrophic forgetting without storing real data.
• We share our model across tasks and automatically expand a single linear classifier head with units for new classes, thus not requiring explicit task labels during inference.
• We demonstrate that our approach can incrementally learn the classes of two image and one audio dataset, as well as cross-dataset scenarios across modalities, while allowing for forward and backward transfer due to weight-sharing. When presented with novel data our model is able to distinguish between unseen data from various datasets and data belonging to known tasks. We further show that our approach readily profits from recent model advances such as variational lossy auto-encoders (Gulrajani et al., 2017; Chen et al., 2017).

heading: 2 UNIFYING CONTINUAL LEARNING WITH OPEN SET RECOGNITION
text: In isolated supervised machine learning the core assumption is the presence of i.i.d. data at all times and training is conducted using a dataset D ≡ {( x(n), y(n) )}N n=1
, consisting of N pairs of data instances x(n) and their corresponding labels y(n) ∈ {1 . . . C} for C classes. In contrast, in
continual learning task dataDt ≡ {( x (n) t , y (n) t )}Nt n=1
with t = 1, . . . , T arrives sequentially for T disjoint datasets, each with number of classes Ct. In our work, we consider this class incremental scenario from a perspective of variational Bayesian inference in deep neural networks (Kingma & Welling, 2013) that consist of a shared encoder with variational parameters θ, decoder and linear classifier with respective parameters φ and ξ. The joint probabilistic encoder learns an encoding to a latent variable z, over which a prior is placed, say a unit Gaussian. Using variational inference, its purpose is to approximate the true posterior to both pφ(x, z) and pξ(y,z). The probabilistic decoder pφ(x|z) and probabilistic linear classifier pξ(y|z) then return the conditional probability density of the input x and target y under the respective generative model given a sample z from the approximate posterior qθ(z|x). This yields a generative model p(x,y, z), for which we assume a factorization and generative process of the form p(x,y, z) = p(x|z)p(y|z)p(z). For variational inference with our model the following continual learning loss function thus needs to be optimized:
LUBt (θ,φ, ξ) = t∑
τ=1 Nτ∑ n=1 [E qθ,t(z|x(n)τ ) [log pφ,t(x (n) τ |z) + log pξ,t(y(n)τ |z)]
− βKL(qθ,t(z|x(n)τ ) || p(z))]
(1)
Here, we have added a weight term β to the KL divergence to balance the individual loss terms, similar to the work of Zhou et al. (2012) and Higgins et al. (2017). This factor regulates the tradeoff between the additional constraint imposed by the classifier, needing to be able to linearly separate the classes given z, and the quality of the approximation to the training data. To balance this tradeoff irrespective of input and latent dimensionality and number of classes, the losses are normalized according to dimensions. Note that in practice this changes the relative scale of the losses and thus the interpretation of specific β values with respect to the original authors Higgins et al. (2017). We provide a more detailed discussion with empirical examples for the role of β in the supplementary section. However, equation 1 requires the presence of all data for all tasks and is thus generally not feasible for continual learning where only the most recent task’s data is assumed to be available. In context of variational inference, two potential approaches offer solutions to this challenge: a prior-based approach using the former approximate posterior qθ,t−1 as the new task’s prior (Nguyen et al., 2018) or estimating the likelihood of former data through generative replay or other forms of rehearsal (Farquhar & Gal, 2018; Achille et al., 2018). For our proposed model, we follow the latter line of work and let the prior remain the same at all times. Making use of the generative nature of our model, we let the above upper-bound to task incremental continual learning become: Lt (θ,φ, ξ) = Ñt∑ n=1 [E qθ,t(z|x̃(n)t ) [log pφ,t(x̃ (n) t |z) + log pξ,t(ỹ (n) t |z)]− βKL(qθ,t(z|x̃ (n) t ) || p(z))]
+ Nt∑ n=1 [E qθ,t(z|x(n)t ) [log pφ,t(x (n) t |z) + log pξ,t(y (n) t |z)]− βKL(qθ,t(z|x (n) t ) || p(z))] (2)
Here, x̃t ∼ pφ,t−1(x|z) and ỹt ∼ pξ,t−1(y|z) with z ∼ p(z) is a sample from the generative model with the corresponding label obtained from the classifier. Ñt is the number of total data instances of all previously seen tasks or alternatively a hyper-parameter. This way the expectation of the log-likelihood for all previously seen tasks is estimated and the dataset at any point in time D̃t ≡ (xt ∪ x̃t, yt ∪ ỹt) is a combination of generations from seen past data distributions and the current task’s real data. For each newly arriving task with novel labels, the classifier is expanded with newly initialized units. We note that whereas the loss function with generative replay in equation 2 is used for continual training, equation 1 and thus real data is always used for testing.
The model is further trained in a denoising fashion, where noise is added to each input x to avoid over-fitting. This is preferable to weight regularization as it doesn’t entail unrecoverable units that are needed to encode later stage concepts. We have accordingly coined our model Classifying Denoising Variational Auto-Encoder (CDVAE). We optionally enhance the probabilistic decoder with an autoregressive variant, where generation of a pixel’s value is spatially conditioned on previous pixels (van den Oord et al., 2016; Gulrajani et al., 2017; Chen et al., 2017). Here, the denoising plays an additional crucial role of de-quantization.
Nonetheless, similar to existing dual-model approaches (Shin et al., 2017; Wu et al., 2018; Farquhar & Gal, 2018), by itself both CDVAE and PixCDVAE models accumulate errors as with each iteration
of generative replay deviations of the approximate from the true posterior get amplified. However, in our joint model, the linear classifier directly affects the partitioning of the latent space by influencing the joint probabilistic encoder’s weights, resulting in class specific areas of large probability density. This is particularly noticeable for lossy VAEs (Gulrajani et al., 2017; Chen et al., 2017) that leave the encoding of local structure to autoregressive layers and hence in our case attribute more influence on the latent space to the classifier. We note that these class specific areas in latent space are not necessarily encouraged for deeper classifiers, however would argue that with a sufficiently expressive probabilistic encoder such a classifier is not necessary. For visualization purposes, we have trained a CDVAE following the details of section 3 with a two-dimensional latent space on the class-incremental MNIST (LeCun et al., 1998) upper-bound and show the latent space embedding for the validation dataset at the end of continual learning in figure 1b. Corresponding intermediate visualizations for each task increment and PixCDVAE can be found in the supplementary material. We take advantage of the classifier’s impact on the latent space as the foundation for posterior based open set recognition and complementary generative replay with statistical outlier rejection. We refer to this extended model as Open-set Classifying Denoising Variational Auto-Encoder (OCDVAE) and PixOCDVAE respectively. An illustration of our joint probabilistic model is shown in figure 1a.

heading: 2.1 OPEN SET RECOGNITION WITH BOUNDS TO THE CLASS SPECIFIC POSTERIOR
text: We leverage the single-headed linear classifier’s presence and the resulting formation of class specific high density regions in latent space as the basis for open set recognition. Specifically, we draw inspiration from the EVT based OpenMax approach (Bendale & Boult, 2016), that uses knowledge about extreme distance values to modify a Softmax prediction’s confidence, but propose to instead
Algorithm 1 Open set recognition calibration for deep variational neural networks. At the end of task t, a Weibull model fit of tail-size η is conducted to bound the per class approximate posterior. Per class c Weibull models ρc,t with their respective shift τc,t, shape κc,t and scale λc,t parameters are returned. The CDVAE model can now be referred to as OCDVAE. Require: CDVAE with probabilistic encoder qθ,t(z|x) and probabilistic classifier pξ,t(y|z) Require: Classifier probabilities pξ,t(y|z) and samples from the approximate posterior z(x(i)) ∼
qθ,t(z|x(i)) for each training dataset example x(i) in dataset D̃t Require: For each class c, let S(i)c = z(x′(i)c ) for each correctly classified training example x′(i)c
1: for c = 1 . . . C do 2: Compute per class latent mean S̄c,t = mean(S (i) c ) 3: Weibull model ρc,t = (τc,t, κc,t, λc,t) = Fit Weibull ( ||Sc − S̄c,t||, η ) 4: Return means S̄t and Weibull models ρt
Algorithm 2 Open set probability estimation for unknown and uncertain inputs. At the end of any task t, novel data points are considered statistical outliers if a Weibull model’s cumulative distribution function’s (CDF) outlier probability value exceeds a prior Ωt.
Require: OCDVAE with probabilistic encoder qθ,t(z|x) Require: Per class latent mean S̄c,t and Weibull model ρc,t, each with parameters (τc,t, κc,t, λc,t)
For a novel input example x̂ sample z ∼ qθ,t(z|x̂) 2: Compute distances to S̄c,t: dc,t = ||S̄c,t − z||
for c = 1 . . . C do 4: Compute Weibull CDF ωc,t(dc,t) = 1− exp ( − ||dc,t−τc,t||λc,t )κc,t Reject input if ωc,t(dc,t) > Ωt for any class c.
bound the open-space risk by employing statistical outlier rejection on the basis of the approximate posterior in Bayesian inference. Considering a trained model at the end of task t, the EVT based open set recognition fits a Weibull distribution on the distances of each correctly classified training example’s sample from the approximate posterior z(x) ∼ qθ,t(z|x) to the respective per class sample mean. In other words, regions of high density of the approximate posterior for each class are identified for the subset of correctly identified data points, with the tail of the Weibull distribution bounding the open-space as well as regions of low-density. This procedure is described in algorithm 1. Once these bounds are established, for any novel input, the Weibull models’ cumulative distribution function can be used to estimate the statistical outlier probability, based on the unknown example’s sample(s) from the posterior and their distance to the class’ region of highest density. If the outlier probability is larger than a prior rejection probability, the input can be considered as unknown and a false overconfident classifier prediction avoided, or conversely it is classified into the already existing classes across all known tasks as detailed in algorithm 2.

heading: 2.2 GENERATIVE REPLAY WITH STATISTICAL OUTLIER REJECTION
text: As the obtained open set recognition models provide bounds between the posterior’s regions of high and low density, we can extend the use from rejection of statistical outliers for novel input examples to rejection of samples drawn directly from the prior for the purpose of generative replay. Consider generation of a data point x ∼ pφ,t(x|z). It is common practice to assume that the approximated posterior is close to the true posterior. If a sample from the prior z ∼ p(z) stems from an area of low density, one further inherently relies on the generative model’s capability for interpolation. In periodic generative rehearsal, these factors can entail accumulation of errors through increasing deviations between approximated and true posterior, as well as classifier confusion due to ambiguous examples. To inhibit the latter and as a result implicitly the former, our obtained bounds can be
Algorithm 3 Generative replay with outlier rejection. For generative replay after training task t, samples z ∼ p(z) are rejected if the Weibull CDF’s probability value exceeds the prior Ωt. Require: OCDVAE with probabilistic encoder qθ,t(z|x) and probabilistic classifier pξ,t(y|z) Require: Per class latent mean S̄c,t and Weibull model ρc,t, each with parameters (τc,t, κc,t, λc,t) Require: Number of samples per class Mc ∀ c = 1, . . . , C̃t with C̃t seen classes up to task t
Initialize: mc ← 0∀ c = 1, . . . , C̃, X̃t = ∅ and Ỹt = ∅ while ∑C̃ c=1mc < ∑C̃ c=1Mc do . in parallel
3: Sample from prior z ∼ p(z) Compute label ĉ = argmax (log pξ,t(y|z)) Calculate distance dĉ,t = ‖S̄ĉ,t − z‖
6: Compute Weibull CDF ωĉ,t(dĉ,t) = 1− exp ( − ||dĉ,t−τĉ,t||λĉ,t )κzĉ,t if ωĉ,t < Ωt and mĉ < Mĉ then Calculate decoder x̃ ∼ pφ,t(x|z) 9: Append to dataset X̃t ← X̃t ∪ x̃ and Ỹt ← Ỹt ∪ ĉ and mĉ ← mĉ + 1
else reject
exploited by rejecting samples from low density regions and replacing them with statistically inlying samples. Hence, we extend generative replay for the OCDVAE with such a rejection mechanism. We now first sample from the prior until a desired amount of statistical inliers per class is reached, whereas the label is obtained using the linear classifier and is accepted if it is in correspondence with the respective class’ Weibull model. We then proceed to generate the dataset with the probabilistic decoder. This bounded version of generative replay with statistical outlier rejection is detailed in algorithm 3. An example of MNIST images with outlier probabilities based on their sample from the prior are shown in figure 1c to illustrate the rejection of ambiguous and misclassified instances, with additional images in the supplementary material. The reason we use sampling with rejection is because our Weibull models are based on scalar distances and thus samples from the Weibull distributions cannot be inverted to high-dimensional z vectors. While this may sound detrimental to our method, we argue that both sampling from the prior z ∼ p(z) in large parallelized batches and likewise computation of a single layer classifier, even in high dimensions, is computationally negligible in contrast to the much more computationally heavy deep probabilistic decoder. The latter further only needs to be processed for accepted samples and thus does not add any computational complexity with respect to conventional generative replay. Note that the amount of samples that has to be drawn before a sample is accepted scales with the dimensionality of the latent space and could in principle be regarded as a limitation for very high dimensional latent spaces. However, practical VAEs typically do not tend to profit from very high dimensional latent spaces.

heading: 3 EXPERIMENTS
text: Similar to recent literature (Zenke et al., 2017; Kirkpatrick et al., 2017; Farquhar & Gal, 2018; Shin et al., 2017; Parisi et al., 2019), we consider the incremental MNIST (LeCun et al., 1998) dataset, where classes arrive in groups of two, and corresponding versions of the FashionMNIST (Xiao et al., 2017) and AudioMNIST dataset (Becker et al., 2018). For the latter we follow the authors’ procedure of converting the audio recordings into spectrograms, resized to 32 × 32. In addition to this class incremental setting, we evaluate cross-dataset scenarios with all inputs resized to 32 × 32, where dataset are sequentially added with all of their classes and the model has to learn across modalities.
We compare our proposed OCDVAE model with its counterpart CDVAE to highlight the improvement induced by algorithm 3. We further contrast these improvements with the dual model variant, consisting of a VAE for generative replay and a separate deep model for classification (Shin et al., 2017). Further, autoregressive pixel model variants are reported to demonstrate how approaches benefit from more recent advances in model architecture. We evaluate elastic weight consolidation (EWC) (Kirkpatrick et al., 2017) on the classification task without a decoder to show that approaches based on regularization fail at maintaining previous knowledge in a single-head classifier scenario. Although the latter has already been shown in a recent review by Parisi et al. (2019) and even for multi-head classifier scenarios (Kemker et al., 2018), we nevertheless provide these results for emphasis. We do not report episodic memory approaches like coresets (Bachem et al., 2015) that explicitly store real data or additional regularization as suggested by Nguyen et al. (2018). These methods are complementary to our proposed approach, reporting them separately might mislead the reader and an evaluation that additionally includes these methods on top is left for future work. To provide a frame of reference for achievable performance, we further consider upper- and lowerbounds for our joint model. The CDVAE lower-bound is obtained when only the current task’s data is available and provides the worst case performance where absolute catastrophic forgetting occurs. Conversely, the upper-bound is obtained with equation 1 when a task’s data is added to all previous tasks’ real data and yields a model’s maximum achievable performance if trained in an incremental fashion. The isolated learning baseline corresponds to the typical machine learning practice outside of continual learning where all tasks’ data is always present. All models can be trained on a single GTX 1080 GPU and we will make our code publicly available.

heading: 3.1 METRICS
text: Our metrics are inspired by previously proposed continual learning classification measures (LopezPaz & Ranzato, 2017; Kemker et al., 2018). In addition to overall accuracy, these metrics monitor forgetting by computing a base accuracy on the initial task, while also gauging the amount of new knowledge that can be encoded by monitoring the accuracy for the most recent increment. In the
multi-head classification scenario, both the overall and base accuracy is then divided with an ideal accuracy. As our single-head classifier scenario implies a natural decay of a task’s base accuracy with increasing amount of classes, we instead report the raw base and new accuracies and compare them with the upper-bound and isolated performance. This is a particularly important distinction to the multi-head scenario reported in many previous works, where the amount of forgetting can easily be determined by the amount that the accuracy of each binary classifier decays over time as, introduction of new classes does not directly affect the other tasks’ weights. We extend these concepts to the probabilistic decoder’s reconstruction loss. Our metrics are thus:
• Classification accuracy: base accuracy αt,base of initial task at increment t. New accuracy αt,new for the freshly added task. Accuracy αt,all over all classes of all tasks seen so far.
• Reconstruction negative-log-likelihood (NLL): base NLL γt,base of initial task at task increment t. New NLL γt,new for the added task. NLL γt,all for all tasks seen so far.
• Kullback-Leibler Divergence: between the approximate posterior qθ,t(z|x) and the prior p(z) distribution and thus always evaluated for all data up to and including task t.

heading: 3.2 TRAINING HYPER-PARAMETERS
text: We base our encoder and decoder architecture on 14-layer wide residual networks (He et al., 2016; Zagoruyko & Komodakis, 2016) as used in lossy auto-encoders (Gulrajani et al., 2017; Chen et al., 2017), with a latent dimensionality of 60 to demonstrate scalability to high-dimensions and deep networks. Our classifier always consists of a single linear layer. The optional autoregressive decoder adds three additional layers. For a common frame of reference, all methods’ share the same WRN architecture. We use hyper-parameters consistent with the literature (Gulrajani et al., 2017; Chen et al., 2017). Accordingly, all models are optimized using stochastic gradient descent with a minibatch size of 128 and Adam (Kingma & Ba, 2015) with a learning rate of 0.001, batch-normalization (Ioffe & Szegedy, 2015) in all hidden layers with a value of 10−5 and ReLU activations. We add noise sampled from N (0, 0.25) to the input to avoid over-fitting. Due to the inevitable data augmentation effect, we train all approaches in this denoising fashion. No further data augmentation or preprocessing is applied. We initialize all weights according to He et al. (2015). For our singlehead expanding classifier this ensures that all units are always initialized from the same distribution when they get added at each task increment, as the initialization depends only on the layer’s input dimensionality. This is not necessarily the case for other weight initialization techniques where a dependency on the amount of classifier units exists. We train all class incremental models for 120 epochs per task on MNIST and FashionMNIST and 150 epochs on AudioMNIST. Complementary incremental cross-dataset models are trained for 200 epochs per task. While our proposed model exhibits forward transfer due to weight sharing and need not necessarily be trained for the entire amount of epochs for each subsequent task, this guarantees convergence and a fair comparison of results. Isolated models are trained for 200 and 300 epochs until convergence respectively. For the generative replay with statistical outlier rejection, we use an aggressive rejection rate of Ωt = 0.01 (with analogous results with 0.05) and dynamically set tail-sizes to 5% of seen examples per class. The used open set distance measure is the cosine distance. We provide a detailed description of architectures and EWC hyper-parameters in the appendix. Results are averaged over five experimental repetitions, apart from the isolated, lower- and upper-bound that show negligible deviations.

heading: 3.3 INCREMENTAL CONTINUAL LEARNING RESULTS AND DISCUSSION
text: Results for the class incremental scenarios for all models, the upper- and lower-bounds and the isolated setting are shown in table 1. Corresponding results for the two directions of incremental cross-dataset experiments are summarized in table 2. In general the upper-bound values are almost identical to isolated learning. Similarly, the new task’s metrics are negligibly close, as the WRN architecture ensures enough capacity to encode new knowledge. In contrast to EWC that is universally unable to maintain its old knowledge, CDVAE and PixCDVAE are able to partially retain previous information. Yet they accumulate errors due to samples generated from low density regions. While the dual model approaches do not exhibit this behavior for MNIST, they displays similar forgetting for other experiments, particularly for Audio data. However, our proposed OCDVAE and PixOCDVAE generative replay overcomes this issue to a considerable degree. For the class incremental scenarios the best models feature less than 10% drop in accuracy on all datasets even with repeated
generative replay. Even stronger results can be observed for the cross-dataset scenarios, where forgetting is alleviated to the extent that final accuracy values are close to the upper bound. Likewise improvements are noticeable in the reconstruction NLL and KL divergences. The OCDVAE models
can consequently achieve reconstruction likelihoods akin to a dual model’s separate VAE, while fully sharing encoded knowledge and maintaining a classifier. As a result of OCDVAE’s shared weights, we observe backward transfer for some experiments. This is particularly apparent for AudioMNIST, where addition of the second increment first decays and inclusion of later tasks improves the second task’s accuracy. Due to space constraints, we provide a detailed account of all intermediate results and examples of generated images for all increments in the supplementary material. We note that the pixel decoders are trained for classification and reported NLL values are obtained through sampling from the multinomial distribution. Original losses are provided in the supplementary material.

heading: 3.4 OUT OF DISTRIBUTION DETECTION RESULTS AND DISCUSSION
text: In addition to previous results with focus on continual learning accuracy, where the open set recognition approach has been used for generative replay, we proceed to quantitatively analyze the models’ ability to distinguish unknown tasks’ data from data belonging to known tasks. Here, the challenge is to consider all unseen test data of already trained tasks as inlying, while successfully recognizing 100 % of unknown datasets as outliers. For this purpose, trained models on each of the three datasets are evaluated on their test set, the remaining datasets and additionally the KMNIST (Clanuwat et al., 2018), SVHN (Netzer et al., 2011), CIFAR10 and CIFAR100 (Krizhevsky, 2009) datasets.
We compare and contrast three criteria that could be used for open set recognition: classifier predictive entropy, reconstruction loss and our proposed latent space based EVT approach. Naively one might expect the Bayesian approach to yield distributions that are sufficiently different for unknown data. We thus evaluate the respective expectation for each criterion with 100 variational samples from the approximate posterior per data point. Figure 2 shows the three criteria and respective percentage of the total dataset being considered as outlying for the OCDVAE model trained on FashionMNIST. In consistence with recent literature (Nalisnick et al., 2019), we can observe that use of reconstruction loss can sometimes distinguish between the known tasks’ test data and unknown datasets, but results in failure for others. In the case of classifier predictive entropy, depending on the exact choice of entropy threshold, generally only a partial separation can be achieved. Furthermore, both of these criteria pose the additional challenge of results being highly dependent on the choice of the precise cut-off value. While drawing z samples multiple times, as well as repeatedly calculating the classifier, is computationally feasible, we further note that sampling the entire decoder is computationally prohibitively expensive in practice. In contrast to the other criteria, the test data from the known tasks is regarded as inlying across a wide range of rejection priors Ωt and the majority of other datasets is consistently regarded as outlying by our proposed open set mechanism.
As our latent space based EVT approach for open set recognition and respective algorithms 1 and 2 do not technically require a decoder, we can evaluate similar outlier detection for the variational dual model approach. While we also evaluate this scenario, note that the distinction the generative model makes need not necessarily apply to a separate classifier and vice versa. We report the outlier detection accuracy in table 3. Here, a 5% validation split is used to determine the respective value
at which 95% of the validation data is considered as inlying before using these priors to determine outlier counts for the known tasks’ test set as well as other datasets. While MNIST seems to be a distinct and easy enough dataset for all approaches, we can make two further major observations:
1. The latent based EVT approach outperforms the other criteria, particularly for the OCDVAE where a near perfect open set detection can be achieved.
2. Even though we can apply our proposed open set approach to just a classifier, the joint model with shared latent space consistently exhibits higher outlier detection values. While future investigation into this aspect is needed, we hypothesize that this is due to the joint model also optimizing a variational lower bound to the data distribution p(x).
We provide figures analogous to figure 2 for all models reported in table 3 in the supplementary material. We emphasize that our open set detection does not rely on knowing any open set data examples beforehand as we do not need modification to the loss function or other forms of explicit training. As there exists a body of complementary work (Liang et al., 2018; Lee et al., 2018b; Dhamija et al., 2018) that could readily be integrated, we leave such analysis for future work.

heading: 4 CONCLUSION AND OUTLOOK
text: We have proposed a unified probabilistic approach to deep continual learning. At the heart lies Bayesian inference with a model combining a shared probabilistic encoder with a generative model and a expanding linear classifier. Weight sharing across tasks allows for forward and backward transfer, while generative replay alleviates catastrophic forgetting. We have then introduced EVT based bounds to the approximate posterior enabled through class specific latent space partitioning induced by the classifier. Derived open set recognition and corresponding generative replay with statistical outlier rejection have been shown to achieve compelling results in both task incremental as well as cross-dataset continual learning across image and audio modalities, while being able to distinguish seen from unseen data. We have demonstrated that our approach readily benefits from recent model advances such as autoregressive models (Gulrajani et al., 2017; Chen et al., 2017) and therefore expect to apply our approach to more complicated data such as larger scale color images with further improvements in generative models. As our approach is extendible, we envision future work to encompass dynamical neural network expansion to increase representation capacity when task complexity increases (Yoon et al., 2018; Rusu et al., 2016; Zhou et al., 2012), combination with soft-targets (Hinton et al., 2014; Li & Hoiem, 2016) or transfer to entirely unsupervised scenarios where the classifier learns only dataset ids instead of distinct classes (Achille et al., 2018).

heading: A CONTINUAL LEARNING 2-D LATENT SPACE VISUALIZATION
text: A natural consequence of our joint model with a shared probabilistic encoder is that the classifier encourages the formation of class-specific regions of high density in the latent space. During continual incremental learning, these regions keep shifting with every task increment while maintaining their class-specificity. New regions of high density emerge for newly added classes. As can be observed in figure 3, at the end of the first task two regions have been formed around the mean of the N (0, 1) prior when training our CDVAE model on the MNIST (LeCun et al., 1998) dataset in a class-incremental upper-bound fashion. With every addition of the next classes, the latent embedding shifts around the mean of the prior to accommodate the new classes with distinct classes separated by regions of low density. Furthermore, it can also be seen in figures 3e and 3f that the shape and the location of the high density regions in the latent embedding are model dependent.

heading: B BALANCING CLASSIFICATION ACCURACY AND KULLBACK LEIBLER
text: DIVERGENCE - THE ROLE OF β
In equations 1 and 2, we have added a weight term β to the KL divergence to balance the individual loss terms, similar to the work of Zhou et al. (2012) and Higgins et al. (2017). In a standard βVAE Higgins et al. (2017), β can be seen as a hyper-parameter that balances the reconstruction accuracy with the distribution approximation quality in the latent space. As mentioned in the main body, in our work, this factor also regulates the trade-off between the additional constraint imposed by the classifier, needing to be able to linearly separate the classes given z, and the quality of the approximation to the training data. As reconstruction losses are typically dependent on the inputs’ spatial dimensionality and the classification loss on the respective number of classes, we normalize the loss accordingly. This yields loss values whose scale is comparable and in practice changes the relative scale of the losses to each other. Whereas typically the reconstruction loss heavily outweighs the Kullback Leibler divergence and classification in non-normalized training for a β value of 1.0, this is not the case for our approach. Although the benefit is a more stable training irrespective of chosen dimensionalities, note that this however changes the role of specific β values with respect to the original authors’ Higgins et al. (2017) interpretation and the contribution of regularization.
To illustrate this we have trained our WRN model on MNIST with 2-D and 60-D latent spaces for multiple values of β in tables 4 and 5 respectively. In these tables we report both the normalized
obtained losses during training, as well as the equivalent computed un-normalized losses as typically reported in the literature. While we have reported the latter for comparison purposes in our main body, the former are used in practice. We can observe that decreasing the value of beta leads to improvement of the classifier and reconstruction accuracy at the expense of KL divergence quality. In both 2-D and 60-D cases, a large beta value leads to significant underfit of the classifier. In order to enable adequate training of the classifier, a corresponding value of β thus has to be chosen. Empirically we have found that a β of 0.1 works well universally. A nuance of the normalized loss is that β values cannot be compared to ones in the un-normalized scenario. To pick a concrete example, the ratio between KLD and reconstruction loss with a β value of 1.0 for a 60-D latent space in table 5 is roughly 1 : 22 without normalization, but approximately 1 : 1.6 in our case. In practice our choice of β would translate to much larger values in a conventional β-VAE.
However, the presence of the classifier, irrespective of the value of β induces low density regions in latent space. For the 2-D latent space on a trained model for MNIST, we can observe that these low density regions become more apparent with smaller values of β, but manifest even for large values at the center due to the classifier’s need to disentangle the class representations.

heading: C GENERATIVE REPLAY EXAMPLES WITH CDVAE AND OCDVAE
text: As stated in section 2 of the main body, as well as exemplified in the previous section, the jointly optimized linear classifier directly affects the emergence of class specific areas of large probability density in the latent space. The effect of sampling from the prior without statistical outlier rejection for low density regions is shown in figure 5 for the MNIST dataset. For CDVAE/PixCDVAE we observe classifier confusion due to class interpolated examples, mentioned in section 2.2. As the generative model needs to learn how to replay old tasks’ data based on its own former generations, this confusion and interpolations accumulate rapidly. This is not the case for OCDVAE/PixOCDVAE, where misclassifications are scarce and the generative model is capable of maintaining high visual
fidelity throughout continual training. As the OCDVAE constrains the sampling to regions of high density, in principle the generative replay could reproduce solely the original data without any interpolation akin to an over-fit. However, for the purpose of generative replay and estimating the log-likelihood of former seen data distributions, this can be desirable in the continual learning scenario as long as variety is ensured. Both our continual learning results presented in the main paper, as well as the visual examples of this section’s figures indicate that this is the case. Similar tendencies can be observed for the other two datasets - FashionMNIST Xiao et al. (2017) (figure 6) and AudioMNIST (Becker et al., 2018) (figure 7). We note that we show AudioMNIST for the purpose of completeness as generated examples are difficult to interpret visually.
D ILLUSTRATION OF GENERATIVE REPLAY WITH STATISTICAL OUTLIER REJECTION
In figure 8 we show generated images x ∼ pφ,t(x|z) with z ∼ p(z) and their corresponding class c obtained from the classifier pξ,t(y|z) for an OCDVAE model trained on the class incremental MNIST, after the last task increment t = T . Based on its sample from the prior, for each image we have further noted the open set statistical outlier probability ωc,t from the respective class’ Weibull model. Images are depicted in rows, whereas each row corresponds to a distinct class label. The figure exemplifies the sampling process for a mini-batch of size 128, where the amount of class samples within the mini-batch is not necessarily balanced. We observe how generated images that feature blurring and ambiguity are considered as strong statistical outliers, as well as examples with class interpolation and therefore hold a misclassified label. Using the latter examples to create a dataset for continual learning with generative replay hence entails accumulation of errors. In contrast to the conventional version with unconstrained sampling, our generative replay with statistical outlier rejection algorithm shown in algorithm 3 of the main body rejects these examples and prevents such errors to a large degree.

heading: E FULL CLASS INCREMENTAL RESULTS
text: In addition to the comparative analysis provided in section 3.4 of the main body, we provide the class-incremental results for each of the three datasets at the end of every task increment, averaged over 5 experimental repetitions in tables 6, 7 and 8 respectively. These tables aid in making some additional observations about the behavior of the different continual learning algorithms across consecutive task increments.
We once again observe the increased effect of error accumulation due to unconstrained generative sampling from the prior in the CDVAE and the PixCDVAE models in comparison to their open set counterparts. The statistical deviations across experiment repetitions in the base and the overall classification accuracies are higher and are generally decreased by the open set models. For example, in table 6 the MNIST base and overall accuracy deviations of CDVAE are higher than the respective values for OCDVAE starting from the second task increment. Correspondingly, the accuracy values themselves experience larger decline for CDVAE than for OCDVAE with progressive increments. This difference is not as pronounced at the end of the first task increment because the models haven’t been trained on any of their own generated data yet. Successful literature approaches such as the variational generative replay proposed by the authors of Farquhar & Gal (2018) thus avoid repeated learning based on previous generated examples and simply store and retain a separate generative model for each task. The strength of our model is that, instead of storing a trained model for each task increment, we are able to continually keep training our joint model with data generated for all previously seen tasks by filtering out ambiguous samples through statistical outlier rejection. Similar trends can also be observed for the respective pixel models.
E.1 BACKWARD TRANSFER
The weight sharing and the presence of a generative expanding single-headed classifier open up the scope for both forward and backward transfer of knowledge in the continual learning context. Figure 9 shows an interesting case of the latter for class-incremental learning with our OCDVAE model on the AudioMNIST dataset. The addition of two new classes (four and five) at the end of the second increment leads to an improvement in the classification performance on class two, as indicated by the confusion matrices.
E.2 PIXEL MODEL BITS PER DIMENSION CLASSIFICATION LOSSES
Although the main body reports PixelVAE reconstruction log-likelihoods in nats, these models are practically formulated as a classification problem with a 256-way Softmax. The corresponding loss is in bits per dimension. We have converted these values to have a better comparison, but in order to do so we need to sample from the pixel decoder’s multinomial distribution to calculate a binary cross-entropy on reconstructed images. The bits per dimension classification loss values for our PixelVAE based experiments in the main body are provided for reference here. The PixCDVAE and PixOCDVAE achieve final losses on all tasks of 1.019 ± 0.014 and 1.047 ± 0.010 for MNIST,

heading: 3 18.84 19.34 11.34 ± 0.057 7.811 ± 0.799 21.25 ± 0.872 18.26 ± 0.818 17.35 ± 0.307 15.36 ± 0.530

heading: 2 16.54 17.47 10.65 ± 0.101 6.247 ± 0.710 17.60 ± 0.755 13.79 ± 0.282 15.56 ± 0.696 12.23 ± 0.287

heading: 1 12.17 12.20 9.710 ± 0.345 3.610 ± 0.856 13.21 ± 0.635 7.164 ± 0.759 13.28 ± 0.644 7.809 ± 1.255

heading: 3 220.7 246.1 221.9 ± 0.648 285.7 ± 0.510 227.2 ± 0.606 261.5 ± 2.970 224.9 ± 0.642 259.1 ± 0.929

heading: 2 224.2 240.4 223.8 ± 0.402 293.8 ± 0.349 226.6 ± 2.31 254.3 ± 1.513 226.9 ± 0.918 255.8 ± 0.436

heading: 1 209.7 209.8 207.7 ± 1.558 267.8 ± 1.246 208.9 ± 1.213 230.8 ± 3.024 209.7 ± 3.655 232.0 ± 2.159

heading: 3 213.6 211.8 211.6 ± 0.543 269.1 ± 0.616 215.4 ± 0.501 268.3 ± 3.852 213.0 ± 0.635 262.9 ± 1.893

heading: 2 241.1 240.2 238.7 ± 0.081 313.4 ± 1.006 241.8 ± 0.502 275.8 ± 1.888 241.9 ± 0.960 275.3 ± 1.473

heading: 1 209.7 209.8 207.7 ± 1.558 267.8 ± 1.246 208.9 ± 1.213 230.8 ± 3.024 209.7 ± 3.655 232.0 ± 2.159

heading: 3 207.6 258.7 213.0 ± 1.854 274.0 ± 0.552 219.5 ± 1.376 235.6 ± 2.784 216.9 ± 1.208 231.6 ± 0.832

heading: 2 207.4 240.7 209.0 ± 0.731 273.6 ± 0.631 212.7 ± 0.579 232.5 ± 1.582 212.1 ± 0.937 231.8 ± 0.416

heading: 1 209.7 209.8 207.7 ± 1.558 267.8 ± 1.246 208.9 ± 1.213 230.8 ± 3.024 209.7 ± 3.655 232.0 ± 2.159

heading: 3 93.02 33.33 34.34 ± 0.009 79.98 ± 0.634 76.77 ± 4.378 83.35 ± 1.597 84.07 ± 1.069 86.93 ± 0.870 87.30 ± 0.322

heading: 2 95.75 48.97 49.28 ± 0.242 91.91 ± 0.043 86.22 ± 3.704 91.83 ± 0.730 92.93 ± 0.160 92.31 ± 1.163 92.17 ± 1.425

heading: 1 99.65 99.60 99.17 ± 0.037 99.58 ± 0.062 99.57 ± 0.091 99.55 ± 0.035 99.58 ± 0.076 99.59 ± 0.082 99.54 ± 0.079

heading: 3 93.35 99.95 99.92 ± 0.012 86.06 ± 2.801 99.09 ± 0.367 90.26 ± 1.435 97.33 ± 0.725 83.40 ± 3.089 96.88 ± 1.156

heading: 2 95.55 97.95 96.09 ± 0.260 89.31 ± 0.311 97.73 ± 1.113 90.98 ± 0.626 96.47 ± 0.596 92.64 ± 2.302 97.31 ± 0.475

heading: 1 99.65 99.60 99.17 ± 0.037 99.58 ± 0.062 99.57 ± 0.091 99.55 ± 0.035 99.58 ± 0.076 99.59 ± 0.082 99.54 ± 0.079

heading: 3 95.95 00.00 01.63 ± 0.032 94.88 ± 0.432 78.55 ± 3.964 79.26 ± 4.170 83.70 ± 3.571 83.90 ± 2.310 87.66 ± 0.375

heading: 2 96.70 00.00 02.40 ± 0.122 94.50 ± 0.389 82.40 ± 6.688 92.02 ± 1.175 90.06 ± 1.782 92.36 ± 2.092 88.60 ± 1.998

heading: 1 99.65 99.60 99.17 ± 0.037 99.58 ± 0.062 99.57 ± 0.091 99.55 ± 0.035 99.58 ± 0.076 99.59 ± 0.082 99.54 ± 0.079

heading: 3 20.16 24.28 16.46 ± 0.122 4.923 ± 1.085 24.24 ± 1.974 12.13 ± 0.977 20.02 ± 0.161 10.17 ± 1.528

heading: 2 18.50 25.84 16.15 ± 0.149 3.177 ± 0.702 20.20 ± 1.188 9.238 ± 0.674 18.01 ± 0.154 7.495 ± 0.738

heading: 1 12.55 13.08 11.81 ± 0.123 1.410 ± 0.181 13.00 ± 0.897 5.629 ± 3.749 13.68 ± 0.785 5.635 ± 3.739

heading: 3 79.58 172.3 81.24 ± 0.262 104.8 ± 1.114 89.88 ± 3.172 114.9 ± 4.590 82.95 ± 1.878 114.6 ± 4.788

heading: 2 75.97 107.3 75.64 ± 0.600 102.9 ± 0.408 82.02 ± 5.488 111.9 ± 2.627 76.62 ± 1.695 112.7 ± 3.300

heading: 1 63.18 62.08 62.17 ± 0.979 90.52 ± 0.263 64.34 ± 2.054 100.0 ± 1.572 62.53 ± 1.166 99.77 ± 2.768

heading: 3 82.53 87.22 83.46 ± 0.992 107.7 ± 0.600 87.65 ± 0.530 118.3 ± 3.523 85.37 ± 1.725 116.5 ± 2.219

heading: 2 88.75 87.93 88.03 ± 0.664 115.8 ± 0.805 89.91 ± 0.107 125.7 ± 2.413 89.64 ± 3.709 124.6 ± 3.822

heading: 1 63.18 62.08 62.17 ± 0.979 90.52 ± 0.263 64.34 ± 2.054 100.0 ± 1.572 62.53 ± 1.166 99.77 ± 2.768

heading: 3 63.36 160.4 67.34 ± 0.445 91.92 ± 0.991 81.89 ± 10.09 100.3 ± 4.562 69.29 ± 1.541 101.1 ± 4.014

heading: 2 62.85 126.8 63.69 ± 0.576 91.27 ± 0.789 74.41 ± 10.89 100.4 ± 1.964 65.68 ± 1.166 101.2 ± 3.601

heading: 1 63.18 62.08 62.17 ± 0.979 90.52 ± 0.263 64.34 ± 2.054 100.0 ± 1.572 62.53 ± 1.166 99.77 ± 2.768

heading: 3 99.72 31.35 33.42 ± 0.027 99.32 ± 0.057 98.93 ± 0.291 95.01 ± 3.162 96.14 ± 1.836 98.46 ± 0.903 99.20 ± 0.057

heading: 2 99.81 49.92 50.16 ± 0.029 99.79 ± 0.049 99.60 ± 0.142 98.54 ± 1.638 98.37 ± 1.448 99.55 ± 0.036 99.69 ± 0.051

heading: 1 100.0 100.0 99.88 ± 0.010 99.98 ± 0.023 99.97 ± 0.002 99.97 ± 0.029 99.97 ± 0.026 99.98 ± 0.018 99.86 ± 0.084

heading: 3 99.67 99.94 99.94 ± 0.002 99.48 ± 0.294 99.41 ± 0.084 99.63 ± 0.172 99.22 ± 0.082 99.61 ± 0.055 99.56 ± 0.092

heading: 2 99.80 99.85 99.70 ± 0.013 99.81 ± 0.062 99.71 ± 0.122 99.75 ± 0.127 99.74 ± 0.052 99.80 ± 0.126 99.82 ± 0.027

heading: 1 100.0 100.0 99.88 ± 0.010 99.98 ± 0.023 99.97 ± 0.002 99.97 ± 0.029 99.97 ± 0.026 99.98 ± 0.018 99.86 ± 0.084

heading: 3 99.80 00.00 00.17 ± 0.045 99.51 ± 0.094 99.16 ± 0.611 87.66 ± 8.765 90.12 ± 5.846 96.69 ± 2.173 98.88 ± 0.491

heading: 2 99.82 00.00 00.61 ± 0.057 99.77 ± 0.032 99.54 ± 0.285 97.28 ± 3.184 96.90 ± 2.907 99.30 ± 0.100 99.64 ± 0.095

heading: 1 100.0 100.0 99.88 ± 0.010 99.98 ± 0.023 99.97 ± 0.002 99.97 ± 0.029 99.97 ± 0.026 99.98 ± 0.018 99.86 ± 0.084

heading: 1 99.99 100.0 100.0 ± 0.000 100.0 ± 0.000 100.0 ± 0.000 99.21 ± 0.568 99.71 ± 0.218 99.95 ± 0.035 99.27 ± 0.410

heading: 2 99.92 00.00 00.16 ± 0.040 93.08 ± 5.854 99.52 ± 0.273 98.98 ± 0.766 97.86 ± 0.799 98.61 ± 0.490 97.88 ± 2.478

heading: 3 100.0 00.00 00.29 ± 0.029 83.25 ± 6.844 93.15 ± 3.062 92.44 ± 1.306 81.38 ± 5.433 95.12 ± 2.248 95.82 ± 3.602

heading: 1 99.99 100.0 100.0 ± 0.000 100.0 ± 0.000 100.0 ± 0.000 99.21 ± 0.568 99.71 ± 0.218 99.95 ± 0.035 99.27 ± 0.410

heading: 2 99.75 100.0 99.78 ± 0.019 86.25 ± 8.956 99.71 ± 0.043 91.82 ± 4.577 99.78 ± 0.128 89.23 ± 7.384 99.81 ± 0.189

heading: 3 98.92 99.58 99.25 ± 0.054 95.16 ± 1.490 98.23 ± 1.092 95.20 ± 1.495 98.41 ± 0.507 94.43 ± 3.030 99.30 ± 0.550

heading: 1 99.99 100.0 100.0 ± 0.000 100.0 ± 0.000 100.0 ± 0.000 99.21 ± 0.568 99.71 ± 0.218 99.95 ± 0.035 99.27 ± 0.410

heading: 2 99.83 50.00 50.16 ± 0.119 89.67 ± 1.763 99.50 ± 0.157 93.84 ± 2.558 98.64 ± 0.875 93.93 ± 3.756 99.67 ± 0.033

heading: 3 99.56 33.19 33.28 ± 0.022 78.24 ± 3.315 95.37 ± 1.750 94.26 ± 1.669 90.10 ± 1.431 95.70 ± 1.524 97.77 ± 1.017

heading: 1 433.7 423.2 422.3 ± 0.573 434.2 ± 1.068 435.2 ± 15.69 432.6 ± 0.321 424.2 ± 2.511 433.8 ± 0.370

heading: 2 422.5 439.4 426.6 ± 2.840 434.4 ± 1.082 423.9 ± 0.517 432.5 ± 0.551 425.2 ± 1.402 433.5 ± 1.464

heading: 3 420.7 429.2 425.0 ± 0.339 434.6 ± 0.785 422.7 ± 0.690 432.9 ± 0.723 423.8 ± 1.148 433.1 ± 1.269

heading: 1 433.7 423.2 422.3 ± 0.573 434.2 ± 1.068 435.2 ± 15.69 432.6 ± 0.321 424.2 ± 2.511 433.8 ± 0.370

heading: 2 381.2 384.1 381.3 ± 2.039 390.4 ± 0.694 382.5 ± 1.355 389.4 ± 0.208 385.3 ± 12.56 389.4 ± 1.304

heading: 3 435.9 436.7 436.8 ± 0.188 444.7 ± 0.545 436.3 ± 0.639 442.7 ± 0.513 436.9 ± 0.688 442.4 ± 0.275

heading: 1 433.7 423.2 422.3 ± 0.573 434.2 ± 1.068 435.2 ± 15.69 432.6 ± 0.321 424.2 ± 2.511 433.8 ± 0.370

heading: 2 401.9 411.8 404.0 ± 2.407 412.4 ± 0.871 403.2 ± 0.831 410.9 ± 0.351 403.5 ± 1.274 411.5 ± 1.406

heading: 3 412.1 418.9 414.4 ± 0.385 423.3 ± 0.618 413.6 ± 0.410 421.0 ± 1.026 413.8 ± 0.573 421.9 ± 0.661

heading: 1 11.65 11.20 4.639 ± 0.107 4.361 ± 0.671 11.78 ± 1.478 9.293 ± 0.943 11.16 ± 0.713 11.87 ± 1.504

heading: 2 11.78 13.61 5.135 ± 0.127 5.130 ± 0.636 15.13 ± 1.128 14.00 ± 0.748 14.06 ± 1.140 12.40 ± 0.719

heading: 3 13.40 17.09 5.427 ± 0.105 5.399 ± 0.724 18.18 ± 1.140 20.28 ± 0.774 13.61 ± 0.901 14.41 ± 0.461
text: 2.851± 0.0026 and 2.852± 0.0047 for FashionMNIST, 4.425± 0.0010 and 4.451± 0.0198 for AudioMNIST. For cross-dataset experiments starting with FashionMNIST first, the corresponding loss values in bits per dimension for PixCDVAE are 2.260 ± 0.0078 and 2.238 ± 0.0021 for PixOCDVAE. In the reverse direction the values are 2.232 ± 0.0177 and 2.218 ± 0.0014 respectively. In the dual model scenario, the separate PixVAE achieves final losses on all tasks of 1.040 ± 0.0103 for MNIST, 2.242 ± 0.0124 for FashionMNIST and 4.406 ± 0.0024 for AudioMNIST. Loss values for the cross dataset setting are 2.253 ± 0.0047 when FashionMNIST gets trained first and 2.279 ± 0.0104 for the reverse direction starting with AudioMNIST.

heading: F ADDITIONAL OPEN SET RECOGNITION VISUALIZATION
text: As we point out in section 3 of the main paper, our posterior based open set recognition considers almost all of the unknown datasets as statistical outliers, while at the same time regarding unseen test data from the originally trained tasks as distribution inliers across a wide range of rejection priors. In addition to the outlier rejection curves for FashionMNIST and the quantitative results presented in the main body, we also show the full outlier rejection curves for the remaining datasets, as well as all dual model approaches in figures 10, 11 and 12. These figures visually support the quantitative findings described in the main body and respective conclusions. In summary, the joint OCDVAE performs better at open set recognition in direct comparison to the dual model setting, particularly when using the EVT based criterion. Apart from the MNIST dataset, where reconstruction loss can be a sufficient metric for open set detection, the latent based approach also exhibits less dependency on the outlier rejection prior and consistently improves the ability to discern unknown data.

heading: G ARCHITECTURE DEFINITIONS AND ADDITIONAL HYPER-PARAMETERS
text: Our previous description of the training hyperparameters in the main text is extended here by specifying the exact encoder and decoder architecture, and the additional hyperparameters for the Adam (Kingma & Ba, 2015) optimizer used for training in each of our evaluated methods. We also provide the hyperparameter values necessary for evaluating EWC in the class-incremental learning and cross-dataset scenarios.
We point the reader to tables 9 and 10 for detailed encoder and decoder configurations. For the autoregressive addition to our joint model, we set the number of output channels of the decoder to 60 and append 3 pixel decoder layers, each with a kernel size of 7 × 7 and 60 channels. The hyperparameters for Adam optimization include a β1 of 0.9, β2 of 0.999 and of 10−8.
For the EWC experiments, the number of Fisher samples is fixed to the total number of data points from all the previously seen tasks. A suitable Fisher multiplier (λ) value has been determined by conducting a grid search over a set of five values: 50, 100, 500, 1000 and 5000 on held-out validation data for the first two tasks in sequence. We observe exploding gradients if λ is too high. However, a very small λ leads to excessive drift in the network weight distribution across subsequent tasks that further results in catastrophic inference. A balance between these two phenomena is achieved for a λ value of 500 in the class-incremental scenario and 1000 in the cross-dataset setting.

Review summaries:
This paper tackles the problem of catastrophic forgetting when data is organized in a large number of batches of data (tasks) that are sequentially made available. To avoid catastrophic forgetting, the authors learn a VAE that generates the training data (both inputs and labels) and retrain it using samples from the new task combined with samples generated from the VAE trained in the previous tasks (generative replay). In this way, there's no need to store all past data and even the first learned batch keeps being refreshed and should not be forgotten.

This paper combines replay and openMax approach to help continual learning.  The results shows robustness on different dataset include image and audio in the continual learning condition, where the new come data has a different distribution but the model still able to maintain reasonable quality for the previously and newly come examples. To my understanding, this approach was not ground-breaking but seems a reasonable combinations.