Title: UNIFIED PROBABILISTIC DEEP CONTINUAL LEARN-ING THROUGH GENERATIVE REPLAY AND OPEN SET RECOGNITION

Abstract: We introduce a unified probabilistic approach for deep continual learning with open set recognition, based on variational Bayesian inference. Our model combines a joint probabilistic encoder with a generative model and a linear classifier that get shared across tasks. The proposed open set recognition bounds the approximate posterior by fitting regions of high density on the basis of correctly classified data points and balances open set detection with recognition errors. Catastrophic forgetting is significantly alleviated through generative replay, where the open set recognition is used to reject statistical outliers from low density areas of the class specific posterior. Our approach naturally allows for forward and backward transfer while maintaining past knowledge without the necessity of storing old data, regularization or inferring task labels. We demonstrate compelling results in the challenging continual learning scenario of incrementally expanding the singlehead classifier for both class incremental visual and audio classification tasks, as well as incremental learning of datasets across modalities. We further quantitatively demonstrate our models ability to successfully distinguish unseen unknown data from trained known tasks and subsequently prevent misclassification.

1. INTRODUCTION:

Most machine learning systems make the closed world assumption and are predominantly trained according to the isolated learning paradigm, where data is available at all times and is independently and identically distributed. However, in the context of continual learning, where tasks and data arrive in sequence, neither of these two principles is desirable. A neural network that is trained exclusively on a new task's data forgets past knowledge and suffers from an early identified phenomenon commonly referred to as catastrophic forgetting (McCloskey & Cohen, 1989) . Moreover, to overcome the closed world assumption, inclusion of a "background" class is veritably insufficient as it is impossible to include all unseen concepts and classes explicitly in the loss function beforehand. Likewise, commonly applied thresholding of prediction values doesn't prevent resulting large confidences for unseen classes if the data is far away from any known data (Matan et al., 1990) .

Most of the existing continual learning literature concentrates efforts on either alleviating catastrophic forgetting, maximizing knowledge transfer or addressing ways in which to efficiently store subsets of past data. These works have identified weight regularization (McCloskey & Cohen, 1989; Zenke et al., 2017; Kirkpatrick et al., 2017; Li & Hoiem, 2016; Nguyen et al., 2018) and rehearsal techniques (Ratcliff, 1990; Lopez-Paz & Ranzato, 2017; Rebuffi et al., 2017; Bachem et al., 2015) or have postulated methods based on complementary learning systems theory (O'Reilly & Norman, 2003) through dual-model with generative memory approaches (Gepperth & Karaoguz, 2016; Shin et al., 2017; Wu et al., 2018; Farquhar & Gal, 2018; Achille et al., 2018) as mechanisms against catastrophic inference. On the one hand, regularization techniques can work well in principle, but come with the caveat of relying on a new task's proximity to previous knowledge. On the other hand, training and storing separate models, including generative models for generative rehearsal, comes at increased memory cost and doesn't allow for full knowledge sharing, particularly to already stored models. Specifically, the transfer of already attained knowledge to benefit new tasks, known as forward transfer, as well as the potential positive impact of learning new concepts to aid in existing tasks, known as backward transfer, are crucial to any continual learning system. Generally speak-ing, most current approaches include a set of simplifications, such as considering separate classifiers for each new task, referred to as multi-head classifiers. This scenario prevents "cross-talk" between classifier units by not sharing them, which would otherwise rapidly decay the accuracy (Zenke et al., 2017; Kirkpatrick et al., 2017; Rusu et al., 2016; Shin et al., 2017; Gepperth & Karaoguz, 2016; Rebuffi et al., 2017; Achille et al., 2018; Nguyen et al., 2018) as newly introduced classes directly impact and confuse existing concepts. In the multi-head scenario task ids thus need to be encoded or are often assumed to be given by humans in order to know which classifier to use for prediction. Correspondingly, in generative replay, generative and discriminative models are taken to be separate models (Shin et al., 2017; Farquhar & Gal, 2018; Nguyen et al., 2018) . Similar to regularization of a classifier, a generative model can suffer from the learned approximate posterior distribution deviating further from the true posterior with each further task increment. In order to avoid catastrophic forgetting induced by learning to generate on previously generated data, previous works even store a separate generative model per task (Farquhar & Gal, 2018) , in analogy to the multi-head classifier. An extended review of recent continual learning methods is provided by Parisi et al. (2019) .

A parallel thread pursues a complementary component of identifying out-of-distribution and open set examples. While current continual learning approaches typically do not include this thread yet, it can be considered crucial to any system and a necessity in order to avoid encoding task labels and distinguishing seen from unknown data. Here, multiple methods rely on using confidence values as means of rejection through calibration (Liang et al., 2018; Lee et al., 2018b; a) . Arguably this also includes Bayesian approaches using variational methods (Farquhar & Gal, 2018; Achille et al., 2018) or Monte-Carlo dropout (Gal & Ghahramani, 2015) to estimate uncertainties. Since the closed world assumption also holds true for Bayesian methods as the approximated posterior probability cannot be computed for unknown classes, misclassification still occurs, as the open space risk is unbounded (Boult et al., 2019) . Recently Thomas et al. (2014) ; Bendale & Boult (2016) ; Dhamija et al. (2018) In this work we propose a probabilistic approach to unify open set recognition with continual learning in a single deep model in order to remove or alleviate above mentioned common simplifications. Specifically, our contributions are:

• We introduce a single model for continual learning that combines a joint probabilistic encoder with a generative model and a linear classifier. Inspired by EVT based open set recognition (Bendale & Boult, 2016) for Softmax prediction layers, this model architecture gives rise to a natural way of open set recognition with statistical outlier rejection on the basis of the approximate posterior in Bayesian inference. The latter requires no upfront knowledge of open set data or corresponding modifications to loss or training procedure and can successfully prevent nonsensical predictions for unseen unknown data, a robustness feature that is currently not present in closed world continual learning systems. • We show how this EVT bound to the posterior can be used for both identification and rejection of statistically outlying unseen unknown data instances, as well as exclusion of generated samples from areas of low probability density. When used in generative replay this leads to significantly reduced catastrophic forgetting without storing real data. • We share our model across tasks and automatically expand a single linear classifier head with units for new classes, thus not requiring explicit task labels during inference. • We demonstrate that our approach can incrementally learn the classes of two image and one audio dataset, as well as cross-dataset scenarios across modalities, while allowing for forward and backward transfer due to weight-sharing. When presented with novel data our model is able to distinguish between unseen data from various datasets and data belonging to known tasks. We further show that our approach readily profits from recent model advances such as variational lossy auto-encoders (Gulrajani et al., 2017; Chen et al., 2017) .

2. UNIFYING CONTINUAL LEARNING WITH OPEN SET RECOGNITION:

In isolated supervised machine learning the core assumption is the presence of i.i.d. data at all times and training is conducted using a dataset D ≡ x (n) , y (n) N n=1 , consisting of N pairs of data instances x (n) and their corresponding labels y (n) ∈ {1 . . . C} for C classes. In contrast, in continual learning task data

D t ≡ x (n) t , y (n) t Nt n=1

with t = 1, . . . , T arrives sequentially for T disjoint datasets, each with number of classes C t . In our work, we consider this class incremental scenario from a perspective of variational Bayesian inference in deep neural networks (Kingma & Welling, 2013 ) that consist of a shared encoder with variational parameters θ, decoder and linear classifier with respective parameters φ and ξ. The joint probabilistic encoder learns an encoding to a latent variable z, over which a prior is placed, say a unit Gaussian. Using variational inference, its purpose is to approximate the true posterior to both p φ (x, z) and p ξ (y, z). The probabilistic decoder p φ (x|z) and probabilistic linear classifier p ξ (y|z) then return the conditional probability density of the input x and target y under the respective generative model given a sample z from the approximate posterior q θ (z|x). This yields a generative model p(x, y, z), for which we assume a factorization and generative process of the form p(x, y, z) = p(x|z)p(y|z)p(z). For variational inference with our model the following continual learning loss function thus needs to be optimized:

EQUATION

Here, we have added a weight term β to the KL divergence to balance the individual loss terms, similar to the work of Zhou et al. (2012) and Higgins et al. (2017) . This factor regulates the tradeoff between the additional constraint imposed by the classifier, needing to be able to linearly separate the classes given z, and the quality of the approximation to the training data. To balance this tradeoff irrespective of input and latent dimensionality and number of classes, the losses are normalized according to dimensions. Note that in practice this changes the relative scale of the losses and thus the interpretation of specific β values with respect to the original authors Higgins et al. (2017) . We provide a more detailed discussion with empirical examples for the role of β in the supplementary section. However, equation 1 requires the presence of all data for all tasks and is thus generally not feasible for continual learning where only the most recent task's data is assumed to be available.

In context of variational inference, two potential approaches offer solutions to this challenge: a prior-based approach using the former approximate posterior q θ,t-1 as the new task's prior (Nguyen et al., 2018) or estimating the likelihood of former data through generative replay or other forms of rehearsal (Farquhar & Gal, 2018; Achille et al., 2018) . For our proposed model, we follow the latter line of work and let the prior remain the same at all times. Making use of the generative nature of our model, we let the above upper-bound to task incremental continual learning become:

EQUATION

Here, xt ∼ p φ,t-1 (x|z) and ỹt ∼ p ξ,t-1 (y|z) with z ∼ p(z) is a sample from the generative model with the corresponding label obtained from the classifier. Ñt is the number of total data instances of all previously seen tasks or alternatively a hyper-parameter. This way the expectation of the log-likelihood for all previously seen tasks is estimated and the dataset at any point in time Dt ≡ (x t ∪ xt , y t ∪ ỹt ) is a combination of generations from seen past data distributions and the current task's real data. For each newly arriving task with novel labels, the classifier is expanded with newly initialized units. We note that whereas the loss function with generative replay in equation 2 is used for continual training, equation 1 and thus real data is always used for testing.

The model is further trained in a denoising fashion, where noise is added to each input x to avoid over-fitting. This is preferable to weight regularization as it doesn't entail unrecoverable units that are needed to encode later stage concepts. We have accordingly coined our model Classifying Denoising Variational Auto-Encoder (CDVAE). We optionally enhance the probabilistic decoder with an autoregressive variant, where generation of a pixel's value is spatially conditioned on previous pixels (van den Oord et al., 2016; Gulrajani et al., 2017; Chen et al., 2017) . Here, the denoising plays an additional crucial role of de-quantization.

Nonetheless, similar to existing dual-model approaches (Shin et al., 2017; Wu et al., 2018; Farquhar & Gal, 2018) , by itself both CDVAE and PixCDVAE models accumulate errors as with each iteration of generative replay deviations of the approximate from the true posterior get amplified. However, in our joint model, the linear classifier directly affects the partitioning of the latent space by influencing the joint probabilistic encoder's weights, resulting in class specific areas of large probability density. This is particularly noticeable for lossy VAEs (Gulrajani et al., 2017; Chen et al., 2017) that leave the encoding of local structure to autoregressive layers and hence in our case attribute more influence on the latent space to the classifier. We note that these class specific areas in latent space are not necessarily encouraged for deeper classifiers, however would argue that with a sufficiently expressive probabilistic encoder such a classifier is not necessary. For visualization purposes, we have trained a CDVAE following the details of section 3 with a two-dimensional latent space on the class-incremental MNIST (LeCun et al., 1998) Require: CDVAE with probabilistic encoder q θ,t (z|x) and probabilistic classifier p ξ,t (y|z) Require: Classifier probabilities p ξ,t (y|z) and samples from the approximate posterior z(x (i) ) ∼ q θ,t (z|x (i) ) for each training dataset example x (i) in dataset Dt Require: For each class c, let S Compute per class latent mean Sc,t = mean(S Append to dataset Xt ← Xt ∪ x and Ỹt ← Ỹt ∪ ĉ and m ĉ ← m ĉ + 1 else reject exploited by rejecting samples from low density regions and replacing them with statistically inlying samples. Hence, we extend generative replay for the OCDVAE with such a rejection mechanism. We now first sample from the prior until a desired amount of statistical inliers per class is reached, whereas the label is obtained using the linear classifier and is accepted if it is in correspondence with the respective class' Weibull model. We then proceed to generate the dataset with the probabilistic decoder. This bounded version of generative replay with statistical outlier rejection is detailed in algorithm 3. An example of MNIST images with outlier probabilities based on their sample from the prior are shown in figure 1c to illustrate the rejection of ambiguous and misclassified instances, with additional images in the supplementary material. The reason we use sampling with rejection is because our Weibull models are based on scalar distances and thus samples from the Weibull distributions cannot be inverted to high-dimensional z vectors. While this may sound detrimental to our method, we argue that both sampling from the prior z ∼ p(z) in large parallelized batches and likewise computation of a single layer classifier, even in high dimensions, is computationally negligible in contrast to the much more computationally heavy deep probabilistic decoder. The latter further only needs to be processed for accepted samples and thus does not add any computational complexity with respect to conventional generative replay. Note that the amount of samples that has to be drawn before a sample is accepted scales with the dimensionality of the latent space and could in principle be regarded as a limitation for very high dimensional latent spaces. However, practical VAEs typically do not tend to profit from very high dimensional latent spaces.

(i) c ) 3: Weibull model ρ c,t = (τ c,t , κ c,t , λ c,t ) = Fit Weibull ||S c -Sc,

3. EXPERIMENTS:

Similar to recent literature (Zenke et al., 2017; Kirkpatrick et al., 2017; Farquhar & Gal, 2018; Shin et al., 2017; Parisi et al., 2019) , we consider the incremental MNIST (LeCun et al., 1998) dataset, where classes arrive in groups of two, and corresponding versions of the FashionMNIST (Xiao et al., 2017) and AudioMNIST dataset (Becker et al., 2018) . For the latter we follow the authors' procedure of converting the audio recordings into spectrograms, resized to 32 × 32. In addition to this class incremental setting, we evaluate cross-dataset scenarios with all inputs resized to 32 × 32, where dataset are sequentially added with all of their classes and the model has to learn across modalities.

We compare our proposed OCDVAE model with its counterpart CDVAE to highlight the improvement induced by algorithm 3. We further contrast these improvements with the dual model variant, consisting of a VAE for generative replay and a separate deep model for classification (Shin et al., 2017) . Further, autoregressive pixel model variants are reported to demonstrate how approaches benefit from more recent advances in model architecture. We evaluate elastic weight consolidation (EWC) (Kirkpatrick et al., 2017) on the classification task without a decoder to show that approaches based on regularization fail at maintaining previous knowledge in a single-head classifier scenario. Although the latter has already been shown in a recent review by Parisi et al. (2019) and even for multi-head classifier scenarios (Kemker et al., 2018) , we nevertheless provide these results for emphasis. We do not report episodic memory approaches like coresets (Bachem et al., 2015) that explicitly store real data or additional regularization as suggested by Nguyen et al. (2018) . These methods are complementary to our proposed approach, reporting them separately might mislead the reader and an evaluation that additionally includes these methods on top is left for future work. To provide a frame of reference for achievable performance, we further consider upper-and lowerbounds for our joint model. The CDVAE lower-bound is obtained when only the current task's data is available and provides the worst case performance where absolute catastrophic forgetting occurs. Conversely, the upper-bound is obtained with equation 1 when a task's data is added to all previous tasks' real data and yields a model's maximum achievable performance if trained in an incremental fashion. The isolated learning baseline corresponds to the typical machine learning practice outside of continual learning where all tasks' data is always present. All models can be trained on a single GTX 1080 GPU and we will make our code publicly available.

3.1. METRICS:

Our metrics are inspired by previously proposed continual learning classification measures (Lopez-Paz & Ranzato, 2017; Kemker et al., 2018) . In addition to overall accuracy, these metrics monitor forgetting by computing a base accuracy on the initial task, while also gauging the amount of new knowledge that can be encoded by monitoring the accuracy for the most recent increment. In the multi-head classification scenario, both the overall and base accuracy is then divided with an ideal accuracy. As our single-head classifier scenario implies a natural decay of a task's base accuracy with increasing amount of classes, we instead report the raw base and new accuracies and compare them with the upper-bound and isolated performance. This is a particularly important distinction to the multi-head scenario reported in many previous works, where the amount of forgetting can easily be determined by the amount that the accuracy of each binary classifier decays over time as, introduction of new classes does not directly affect the other tasks' weights. We extend these concepts to the probabilistic decoder's reconstruction loss. Our metrics are thus:

• Classification accuracy: base accuracy α t,base of initial task at increment t. New accuracy α t,new for the freshly added task. Accuracy α t,all over all classes of all tasks seen so far.

• Reconstruction negative-log-likelihood (NLL): base NLL γ t,base of initial task at task increment t. New NLL γ t,new for the added task. NLL γ t,all for all tasks seen so far.

• Kullback-Leibler Divergence: between the approximate posterior q θ,t (z|x) and the prior p(z) distribution and thus always evaluated for all data up to and including task t.

3.2. TRAINING HYPER-PARAMETERS:

We base our encoder and decoder architecture on 14-layer wide residual networks (He et al., 2016; Zagoruyko & Komodakis, 2016) as used in lossy auto-encoders (Gulrajani et al., 2017; Chen et al., 2017) , with a latent dimensionality of 60 to demonstrate scalability to high-dimensions and deep networks. Our classifier always consists of a single linear layer. The optional autoregressive decoder adds three additional layers. For a common frame of reference, all methods' share the same WRN architecture. We use hyper-parameters consistent with the literature (Gulrajani et al., 2017; Chen et al., 2017) . Accordingly, all models are optimized using stochastic gradient descent with a minibatch size of 128 and Adam (Kingma & Ba, 2015) with a learning rate of 0.001, batch-normalization (Ioffe & Szegedy, 2015) in all hidden layers with a value of 10 -5 and ReLU activations. We add noise sampled from N (0, 0.25) to the input to avoid over-fitting. Due to the inevitable data augmentation effect, we train all approaches in this denoising fashion. No further data augmentation or preprocessing is applied. We initialize all weights according to He et al. (2015) . For our singlehead expanding classifier this ensures that all units are always initialized from the same distribution when they get added at each task increment, as the initialization depends only on the layer's input dimensionality. This is not necessarily the case for other weight initialization techniques where a dependency on the amount of classifier units exists. We train all class incremental models for 120 epochs per task on MNIST and FashionMNIST and 150 epochs on AudioMNIST. Complementary incremental cross-dataset models are trained for 200 epochs per task. While our proposed model exhibits forward transfer due to weight sharing and need not necessarily be trained for the entire amount of epochs for each subsequent task, this guarantees convergence and a fair comparison of results. Isolated models are trained for 200 and 300 epochs until convergence respectively. For the generative replay with statistical outlier rejection, we use an aggressive rejection rate of Ω t = 0.01 (with analogous results with 0.05) and dynamically set tail-sizes to 5% of seen examples per class.

The used open set distance measure is the cosine distance. We provide a detailed description of architectures and EWC hyper-parameters in the appendix. Results are averaged over five experimental repetitions, apart from the isolated, lower-and upper-bound that show negligible deviations.

3.3. INCREMENTAL CONTINUAL LEARNING RESULTS AND DISCUSSION:

Results for the class incremental scenarios for all models, the upper-and lower-bounds and the isolated setting are shown in and examples of generated images for all increments in the supplementary material. We note that the pixel decoders are trained for classification and reported NLL values are obtained through sampling from the multinomial distribution. Original losses are provided in the supplementary material.

3.4. OUT OF DISTRIBUTION DETECTION RESULTS AND DISCUSSION:

In addition to previous results with focus on continual learning accuracy, where the open set recognition approach has been used for generative replay, we proceed to quantitatively analyze the models' ability to distinguish unknown tasks' data from data belonging to known tasks. Here, the challenge is to consider all unseen test data of already trained tasks as inlying, while successfully recognizing 100 % of unknown datasets as outliers. For this purpose, trained models on each of the three datasets are evaluated on their test set, the remaining datasets and additionally the KMNIST (Clanuwat et al., 2018) , SVHN (Netzer et al., 2011) , CIFAR10 and CIFAR100 (Krizhevsky, 2009) datasets.

We compare and contrast three criteria that could be used for open set recognition: classifier predictive entropy, reconstruction loss and our proposed latent space based EVT approach. Naively one might expect the Bayesian approach to yield distributions that are sufficiently different for unknown data. We thus evaluate the respective expectation for each criterion with 100 variational samples from the approximate posterior per data point. Figure 2 shows the three criteria and respective percentage of the total dataset being considered as outlying for the OCDVAE model trained on FashionMNIST. In consistence with recent literature (Nalisnick et al., 2019) , we can observe that use of reconstruction loss can sometimes distinguish between the known tasks' test data and unknown datasets, but results in failure for others. In the case of classifier predictive entropy, depending on the exact choice of entropy threshold, generally only a partial separation can be achieved. Furthermore, both of these criteria pose the additional challenge of results being highly dependent on the choice of the precise cut-off value. While drawing z samples multiple times, as well as repeatedly calculating the classifier, is computationally feasible, we further note that sampling the entire decoder is computationally prohibitively expensive in practice. In contrast to the other criteria, the test data from the known tasks is regarded as inlying across a wide range of rejection priors Ω t and the majority of other datasets is consistently regarded as outlying by our proposed open set mechanism.

As our latent space based EVT approach for open set recognition and respective algorithms 1 and 2 do not technically require a decoder, we can evaluate similar outlier detection for the variational dual model approach. While we also evaluate this scenario, note that the distinction the generative model makes need not necessarily apply to a separate classifier and vice versa. We report the outlier detection accuracy in table 3 . Here, a 5% validation split is used to determine the respective value at which 95% of the validation data is considered as inlying before using these priors to determine outlier counts for the known tasks' test set as well as other datasets. While MNIST seems to be a distinct and easy enough dataset for all approaches, we can make two further major observations:

1. The latent based EVT approach outperforms the other criteria, particularly for the OCD-VAE where a near perfect open set detection can be achieved. 2. Even though we can apply our proposed open set approach to just a classifier, the joint model with shared latent space consistently exhibits higher outlier detection values. While future investigation into this aspect is needed, we hypothesize that this is due to the joint model also optimizing a variational lower bound to the data distribution p(x).

We provide figures analogous to figure 2 for all models reported in table 3 in the supplementary material. We emphasize that our open set detection does not rely on knowing any open set data examples beforehand as we do not need modification to the loss function or other forms of explicit training. As there exists a body of complementary work (Liang et al., 2018; Lee et al., 2018b; Dhamija et al., 2018 ) that could readily be integrated, we leave such analysis for future work.

4. CONCLUSION AND OUTLOOK:

We have proposed a unified probabilistic approach to deep continual learning. At the heart lies Bayesian inference with a model combining a shared probabilistic encoder with a generative model and a expanding linear classifier. Weight sharing across tasks allows for forward and backward transfer, while generative replay alleviates catastrophic forgetting. We have then introduced EVT based bounds to the approximate posterior enabled through class specific latent space partitioning induced by the classifier. Derived open set recognition and corresponding generative replay with statistical outlier rejection have been shown to achieve compelling results in both task incremental as well as cross-dataset continual learning across image and audio modalities, while being able to distinguish seen from unseen data. We have demonstrated that our approach readily benefits from recent model advances such as autoregressive models (Gulrajani et al., 2017; Chen et al., 2017) and therefore expect to apply our approach to more complicated data such as larger scale color images with further improvements in generative models. As our approach is extendible, we envision future work to encompass dynamical neural network expansion to increase representation capacity when task complexity increases (Yoon et al., 2018; Rusu et al., 2016; Zhou et al., 2012) , combination with soft-targets (Hinton et al., 2014; Li & Hoiem, 2016) or transfer to entirely unsupervised scenarios where the classifier learns only dataset ids instead of distinct classes (Achille et al., 2018) .

A CONTINUAL LEARNING 2-D LATENT SPACE VISUALIZATION:

A natural consequence of our joint model with a shared probabilistic encoder is that the classifier encourages the formation of class-specific regions of high density in the latent space. During continual incremental learning, these regions keep shifting with every task increment while maintaining their class-specificity. New regions of high density emerge for newly added classes. As can be observed in figure 3 , at the end of the first task two regions have been formed around the mean of the N (0, 1) prior when training our CDVAE model on the MNIST (LeCun et al., 1998) dataset in a class-incremental upper-bound fashion. With every addition of the next classes, the latent embedding shifts around the mean of the prior to accommodate the new classes with distinct classes separated by regions of low density. Furthermore, it can also be seen in figures 3e and 3f that the shape and the location of the high density regions in the latent embedding are model dependent. 

B BALANCING CLASSIFICATION ACCURACY AND KULLBACK LEIBLER DIVERGENCE -THE ROLE OF β:

In equations 1 and 2, we have added a weight term β to the KL divergence to balance the individual loss terms, similar to the work of Zhou et al. (2012) and Higgins et al. (2017) . In a standard β-VAE Higgins et al. ( 2017), β can be seen as a hyper-parameter that balances the reconstruction accuracy with the distribution approximation quality in the latent space. As mentioned in the main body, in our work, this factor also regulates the trade-off between the additional constraint imposed by the classifier, needing to be able to linearly separate the classes given z, and the quality of the approximation to the training data. As reconstruction losses are typically dependent on the inputs' spatial dimensionality and the classification loss on the respective number of classes, we normalize the loss accordingly. This yields loss values whose scale is comparable and in practice changes the relative scale of the losses to each other. Whereas typically the reconstruction loss heavily outweighs the Kullback Leibler divergence and classification in non-normalized training for a β value of 1.0, this is not the case for our approach. Although the benefit is a more stable training irrespective of chosen dimensionalities, note that this however changes the role of specific β values with respect to the original authors ' Higgins et al. (2017) interpretation and the contribution of regularization.

To illustrate this we have trained our WRN model on MNIST with 2-D and 60-D latent spaces for multiple values of β in tables 4 and 5 respectively. In these tables we report both the normalized obtained losses during training, as well as the equivalent computed un-normalized losses as typically reported in the literature. While we have reported the latter for comparison purposes in our main body, the former are used in practice. We can observe that decreasing the value of beta leads to improvement of the classifier and reconstruction accuracy at the expense of KL divergence quality.

In both 2-D and 60-D cases, a large beta value leads to significant underfit of the classifier. In order to enable adequate training of the classifier, a corresponding value of β thus has to be chosen. Empirically we have found that a β of 0.1 works well universally. A nuance of the normalized loss is that β values cannot be compared to ones in the un-normalized scenario. To pick a concrete example, the ratio between KLD and reconstruction loss with a β value of 1.0 for a 60-D latent space in table 5 is roughly 1 : 22 without normalization, but approximately 1 : 1.6 in our case. In practice our choice of β would translate to much larger values in a conventional β-VAE.

However, the presence of the classifier, irrespective of the value of β induces low density regions in latent space. For the 2-D latent space on a trained model for MNIST, we can observe that these low density regions become more apparent with smaller values of β, but manifest even for large values at the center due to the classifier's need to disentangle the class representations. 

C GENERATIVE REPLAY EXAMPLES WITH CDVAE AND OCDVAE:

As stated in section 2 of the main body, as well as exemplified in the previous section, the jointly optimized linear classifier directly affects the emergence of class specific areas of large probability density in the latent space. The effect of sampling from the prior without statistical outlier rejection for low density regions is shown in figure 5 for the MNIST dataset. For CDVAE/PixCDVAE we observe classifier confusion due to class interpolated examples, mentioned in section 2.2. As the generative model needs to learn how to replay old tasks' data based on its own former generations, this confusion and interpolations accumulate rapidly. This is not the case for OCDVAE/PixOCDVAE, where misclassifications are scarce and the generative model is capable of maintaining high visual fidelity throughout continual training. As the OCDVAE constrains the sampling to regions of high density, in principle the generative replay could reproduce solely the original data without any interpolation akin to an over-fit. However, for the purpose of generative replay and estimating the log-likelihood of former seen data distributions, this can be desirable in the continual learning scenario as long as variety is ensured. Both our continual learning results presented in the main paper, as well as the visual examples of this section's figures indicate that this is the case. Similar tendencies can be observed for the other two datasets -FashionMNIST Xiao et al. ( 2017) (figure 6 ) and AudioMNIST (Becker et al., 2018 ) (figure 7 ). We note that we show AudioMNIST for the purpose of completeness as generated examples are difficult to interpret visually. 

E FULL CLASS INCREMENTAL RESULTS:

In addition to the comparative analysis provided in section 3.4 of the main body, we provide the class-incremental results for each of the three datasets at the end of every task increment, averaged over 5 experimental repetitions in tables 6, 7 and 8 respectively. These tables aid in making some additional observations about the behavior of the different continual learning algorithms across consecutive task increments.

We once again observe the increased effect of error accumulation due to unconstrained generative sampling from the prior in the CDVAE and the PixCDVAE models in comparison to their open set counterparts. The statistical deviations across experiment repetitions in the base and the overall classification accuracies are higher and are generally decreased by the open set models. For example, in table 6 the MNIST base and overall accuracy deviations of CDVAE are higher than the respective values for OCDVAE starting from the second task increment. Correspondingly, the accuracy values themselves experience larger decline for CDVAE than for OCDVAE with progressive increments. This difference is not as pronounced at the end of the first task increment because the models haven't been trained on any of their own generated data yet. Successful literature approaches such as the variational generative replay proposed by the authors of Farquhar & Gal (2018) thus avoid repeated learning based on previous generated examples and simply store and retain a separate generative model for each task. The strength of our model is that, instead of storing a trained model for each task increment, we are able to continually keep training our joint model with data generated for all previously seen tasks by filtering out ambiguous samples through statistical outlier rejection. Similar trends can also be observed for the respective pixel models.

E.1 BACKWARD TRANSFER:

The weight sharing and the presence of a generative expanding single-headed classifier open up the scope for both forward and backward transfer of knowledge in the continual learning context. Figure 9 shows an interesting case of the latter for class-incremental learning with our OCDVAE model on the AudioMNIST dataset. The addition of two new classes (four and five) at the end of the second increment leads to an improvement in the classification performance on class two, as indicated by the confusion matrices. 

E.2 PIXEL MODEL BITS PER DIMENSION CLASSIFICATION LOSSES:

Although the main body reports PixelVAE reconstruction log-likelihoods in nats, these models are practically formulated as a classification problem with a 256-way Softmax. The corresponding loss is in bits per dimension. We have converted these values to have a better comparison, but in order to do so we need to sample from the pixel decoder's multinomial distribution to calculate a binary cross-entropy on reconstructed images. The bits per dimension classification loss values for our PixelVAE based experiments in the main body are provided for reference here. The PixCDVAE and PixOCDVAE achieve final losses on all tasks of 1.019 ± 0.014 and 1.047 ± 0.010 for MNIST, 

F ADDITIONAL OPEN SET RECOGNITION VISUALIZATION:

As we point out in section 3 of the main paper, our posterior based open set recognition considers almost all of the unknown datasets as statistical outliers, while at the same time regarding unseen test data from the originally trained tasks as distribution inliers across a wide range of rejection priors.

In addition to the outlier rejection curves for FashionMNIST and the quantitative results presented in the main body, we also show the full outlier rejection curves for the remaining datasets, as well as all dual model approaches in figures 10, 11 and 12. These figures visually support the quantitative findings described in the main body and respective conclusions. In summary, the joint OCDVAE performs better at open set recognition in direct comparison to the dual model setting, particularly when using the EVT based criterion. Apart from the MNIST dataset, where reconstruction loss can be a sufficient metric for open set detection, the latent based approach also exhibits less dependency on the outlier rejection prior and consistently improves the ability to discern unknown data. 

G ARCHITECTURE DEFINITIONS AND ADDITIONAL HYPER-PARAMETERS:

Our previous description of the training hyperparameters in the main text is extended here by specifying the exact encoder and decoder architecture, and the additional hyperparameters for the Adam (Kingma & Ba, 2015) optimizer used for training in each of our evaluated methods. We also provide the hyperparameter values necessary for evaluating EWC in the class-incremental learning and cross-dataset scenarios.

We point the reader to tables 9 and 10 for detailed encoder and decoder configurations. For the autoregressive addition to our joint model, we set the number of output channels of the decoder to 60 and append 3 pixel decoder layers, each with a kernel size of 7 × 7 and 60 channels. The hyperparameters for Adam optimization include a β 1 of 0.9, β 2 of 0.999 and of 10 -8 .

For the EWC experiments, the number of Fisher samples is fixed to the total number of data points from all the previously seen tasks. A suitable Fisher multiplier (λ) value has been determined by conducting a grid search over a set of five values: 50, 100, 500, 1000 and 5000 on held-out validation data for the first two tasks in sequence. We observe exploding gradients if λ is too high. However, a very small λ leads to excessive drift in the network weight distribution across subsequent tasks that further results in catastrophic inference. A balance between these two phenomena is achieved for a λ value of 500 in the class-incremental scenario and 1000 in the cross-dataset setting.

Table 9 : 14-layer WRN encoder with a widen factor of 10. Convolutional layers (conv) are parametrized by a quadratic filter size followed by the amount of filters. p and s represent padding and stride respectively. If no padding or stride is specified then p = 0 and s = 1. Skip connections are an additional operation at a layer, with the layer to be skipped specified in brackets. Every convolutional layer is followed by batch-normalization and a ReLU. Table 10: 15-layered WRN decoder with a widen factor of 10. P refers to the quadratic input's spatial dimension. Convolutional (conv) and transposed convolutional (conv t) layers are parametrized by a quadratic filter size followed by the amount of filters. p and s represent padding and stride respectively. If no padding or stride is specified then p = 0 and s = 1. Skip connections are an additional operation at a layer, with the layer to be skipped specified in brackets. Every convolutional and fully-connected (FC) layer are followed by batch-normalization and a ReLU. The model ends on a linear transformation with a Sigmoid function. 

:



Figures and Tables:

have proposed extreme value theory (EVT) based open set recognition to bound the open-space risk and balance it with recognition errors in deep neural networks.

Figure 1: (a) Joint continual learning model consisting of a shared probabilistic encoder with variational approximation q θ (z|x), probabilistic decoder p φ (x, z) and probabilistic classifier p ξ (y, z). The dashed (purple) line denotes an optional pixel decoder with parameters φ p . For open set recognition and generative replay with outlier rejection, EVT based bounds for the variational approximation are established. (b) 2-D latent space visualization for continually learned incremental MNIST. (c) Generated MNIST images x ∼ p φ,t (x|z) with z ∼ p(z) and their corresponding class c obtained from the classifier p ξ,t (y|z) for c = 5, together with their open set outlier probability ω c,t .

c ) for each correctly classified training example x (i) c 1: for c = 1 . . . C do 2:

Figure 2: Trained FashionMNIST OCDVAE evaluated on unknown datasets. All metrics are averaged over 100 approximate posterior samples per data point. (Left) Classifier entropy values are insufficient to separate most of unknown from the known task's test data. (Center) Reconstruction loss allows for a partial distinction. (Right) Our posterior-based open set recognition considers the large majority of unknown data as statistical outliers across a wide range of rejection priors Ω t .

Figure 3: 2-D latent space visualization for continually learned incremental upper-bound MNIST at the end of every task increment for CDVAE (a-e) and at the end of training for all task increments for PixCDVAE (f).

Figure 4: 2-D MNIST latent space visualization with different β values for the used WRN architecture.

Figure 5: Generated images for continually learned incremental MNIST at the end of task increments for CDVAE (a-d), OCDVAE (e-h), PixCDVAE (i-l) and PixOCDVAE (m-p).

Figure 6: Generated images for continually learned incremental FashionMNIST at the end of task increments for CDVAE (a-d), OCDVAE (e-h), PixCDVAE (i-l) and PixOCDVAE (m-p).

Figure 7: Generated images for continually learned incremental AudioMNIST at the end of task increments for CDVAE (a-d), OCDVAE (e-h), PixCDVAE (i-l) and PixOCDVAE (m-p).

Figure 8: Illustration of generated images x ∼ p φ,t (x|z) with z ∼ p(z) and their corresponding class c obtained from the classifier p ξ,t (y|z), together with their open set outlier probability ω c,t , for an OCDVAE model trained on incremental MNIST, after the last task increment t = T . From top to bottom the identified classes are: 0, 3, 4, 5, 6 and 9. It is observable how the statistical outlier probability is proportional to the degree of interpolation between classes, blur and thus ambiguity.

Figure 9: AudioMNIST confusion matrices for incrementally learned classes of the OCDVAE model. When adding classes two and three the model experiences difficulty in classification, however is able to overcome this challenge by exhibiting backward transfer when later learning classes four and five. It is also observable how forgetting of the initial classes is limited.

Figure 10: Trained FashionMNIST OCDVAE evaluated on unseen datasets. All metrics are reported as the mean over 100 approximate posterior samples per data point. (a) The classifier entropy values by itself are insufficient to separate most of unknown from the known task's test data. (b) Reconstruction loss allows for a partial distinction. (c) Our posterior-based open set recognition considers the large majority of unknown data as statistical outliers across a wide range of rejection priors Ω t . (d-f) Corresponding outlier detection using the two separate models of the dual-model approach.

Figure 11: Trained MNIST OCDVAE evaluated on unseen datasets. All metrics are reported as the mean over 100 approximate posterior samples per data point. (a) The classifier entropy values by itself are insufficient to separate most of unknown from the known task's test data. (b) Reconstruction loss allows for distinction if the cut-off is chosen correctly. (c) Our posterior-based open set recognition considers the large majority of unknown data as statistical outliers across a wide range of rejection priors Ω t . (d-f) Corresponding outlier detection using the two separate models of the dual-model approach.

3 -160, p = 1; conv 1 × 1 -160 (skip next layer) conv 3 × 3 -160, p = 1 conv 3 × 3 -160, p = 1; shortcut (skip next layer)conv 3 × 3 -160, p = 1 Block 2 conv 3 × 3 -320, s = 2, p = 1; conv 1 × 1 -320, s = 2 (skip next layer) conv 3 × 3 -320, p = 1 conv 3 × 3 -320, p = 1; shortcut (skip next layer) conv 3 × 3 -320, p = 1 Block 3 conv 3 × 3 -640, s = 2, p = 1; conv 1 × 1 -640, s = 2 (skip next layer) conv 3 × 3 -640, p = 1 conv 3 × 3 -640, p = 1;shortcut (skip next layer) conv 3 × 3 -640, p = 1

× 3 -320, p = 1; conv t 1 × 1 -320 (skip next layer) conv 3 × 3 -320, p = 1 conv 3 × 3 -320, p = 1; shortcut (skip next layer) conv 3 × 3 -320, p = 1 upsample × 2 Block 2 conv t 3 × 3 -160, p = 1; conv t 1 × 1 -160 (skip next layer) conv 3 × 3 -160, p = 1 conv 3 × 3 -160, p = 1; shortcut (skip next layer) conv 3 × 3 -160, p = 1 upsample × 2 Block 3 conv t 3 × 3 -48, p = 1; conv t 1 × 1 -48 (skip next layer) conv 3 × 3 -48, p = 1 conv 3 × 3 -48, p = 1;shortcut (skip next layer) conv 3 × 3 -48, p = 1 Layer 2 conv 3 × 3 -3, p = 1

t ||, η 4: Return means St and Weibull models ρ t Algorithm 2 Open set probability estimation for unknown and uncertain inputs. At the end of any task t, novel data points are considered statistical outliers if a Weibull model's cumulative distribution function's (CDF) outlier probability value exceeds a prior Ω t . Reject input if ω c,t (d c,t ) > Ω t for any class c. Per class latent mean Sc,t and Weibull model ρ c,t , each with parameters (τ c,t , κ c,t , λ c,t ) Require: Number of samples per class M c ∀ c = 1, . . . , Ct with Ct seen classes up to task t

Table content: <table><tr><td colspan="2">Require: OCDVAE with probabilistic encoder q θ,t (z|x)</td><td/><td/></tr><tr><td colspan="4">Require: Per class latent mean Sc,t and Weibull model ρ c,t , each with parameters (τ c,t , κ c,t , λ c,t )</td></tr><tr><td/><td>For a novel input example x sample z ∼ q θ,t (z|x)</td><td/><td/></tr><tr><td colspan="2">2: Compute distances to Sc,t : d c,t = || Sc,t -z||</td><td/><td/></tr><tr><td>4:</td><td>for c = 1 . . . C do Compute Weibull CDF ω c,t (d c,t ) = 1 -exp -</td><td>λc,t ||dc,t-τc,t||</td><td>κc,t</td></tr><tr><td colspan="4">bound the open-space risk by employing statistical outlier rejection on the basis of the approximate</td></tr><tr><td colspan="4">posterior in Bayesian inference. Considering a trained model at the end of task t, the EVT based</td></tr><tr><td colspan="4">open set recognition fits a Weibull distribution on the distances of each correctly classified training</td></tr><tr><td colspan="4">example's sample from the approximate posterior z(x) ∼ q θ,t (z|x) to the respective per class sam-</td></tr><tr><td colspan="4">ple mean. In other words, regions of high density of the approximate posterior for each class are</td></tr><tr><td colspan="4">identified for the subset of correctly identified data points, with the tail of the Weibull distribution</td></tr><tr><td colspan="4">bounding the open-space as well as regions of low-density. This procedure is described in algo-</td></tr><tr><td colspan="4">rithm 1. Once these bounds are established, for any novel input, the Weibull models' cumulative</td></tr><tr><td colspan="4">distribution function can be used to estimate the statistical outlier probability, based on the unknown</td></tr><tr><td colspan="4">example's sample(s) from the posterior and their distance to the class' region of highest density.</td></tr><tr><td colspan="4">If the outlier probability is larger than a prior rejection probability, the input can be considered as</td></tr><tr><td colspan="4">unknown and a false overconfident classifier prediction avoided, or conversely it is classified into</td></tr><tr><td colspan="4">the already existing classes across all known tasks as detailed in algorithm 2.</td></tr><tr><td colspan="4">2.2 GENERATIVE REPLAY WITH STATISTICAL OUTLIER REJECTION</td></tr><tr><td colspan="4">As the obtained open set recognition models provide bounds between the posterior's regions of high</td></tr><tr><td colspan="4">and low density, we can extend the use from rejection of statistical outliers for novel input examples</td></tr><tr><td colspan="4">to rejection of samples drawn directly from the prior for the purpose of generative replay. Consider</td></tr><tr><td colspan="4">generation of a data point x ∼ p φ,t (x|z). It is common practice to assume that the approximated</td></tr><tr><td colspan="4">posterior is close to the true posterior. If a sample from the prior z ∼ p(z) stems from an area</td></tr><tr><td colspan="4">of low density, one further inherently relies on the generative model's capability for interpolation.</td></tr><tr><td colspan="4">In periodic generative rehearsal, these factors can entail accumulation of errors through increasing</td></tr><tr><td colspan="4">deviations between approximated and true posterior, as well as classifier confusion due to ambiguous</td></tr><tr><td colspan="4">examples. To inhibit the latter and as a result implicitly the former, our obtained bounds can be</td></tr><tr><td colspan="4">Algorithm 3 Generative replay with outlier rejection. For generative replay after training task t,</td></tr><tr><td colspan="4">samples z ∼ p(z) are rejected if the Weibull CDF's probability value exceeds the prior Ω t .</td></tr><tr><td colspan="4">Require: OCDVAE with probabilistic encoder q θ,t (z|x) and probabilistic classifier p ξ,t (y|z)</td></tr><tr><td colspan="3">Require: Initialize: m c ← 0 ∀ c = 1, . . . , C, Xt = ∅ and Ỹt = ∅</td><td/></tr><tr><td>3:</td><td>while C c=1 m c &lt; Sample from prior z ∼ p(z) C c=1 M c do</td><td/><td>in parallel</td></tr><tr><td/><td>Compute label ĉ = argmax (log p ξ,t (y|z))</td><td/><td/></tr><tr><td>6:</td><td>Calculate distance d ĉ,t = Sĉ,t -z Compute Weibull CDF ω ĉ,t (d ĉ,t ) = 1 -exp -</td><td>λ ĉ,t ||d ĉ,t -τ ĉ,t ||</td><td>κz ĉ,t</td></tr><tr><td/><td>if ω ĉ,t &lt; Ω t and m ĉ &lt; M ĉ then</td><td/><td/></tr><tr><td/><td>Calculate decoder x ∼ p φ,t (x|z)</td><td/><td/></tr></table>

Corresponding results for the two directions of incremental cross-dataset experiments are summarized in table 2. In general the upper-bound values are almost identical to isolated learning. Similarly, the new task's metrics are negligibly close, as the WRN architecture ensures enough capacity to encode new knowledge. In contrast to EWC that is universally

Table content: <table/>

Results for class incremental continual learning approaches averaged over 5 runs, baselines and the reference isolated learning scenario for the three datasets. α T and γ T indicate the respective accuracy and NLL reconstruction metrics at the end of the last task increment T = 5. KL T denotes the corresponding KL divergence. Arrows indicate whether lower or larger values are better. ± 0.489 98.59 ± 0.106 96.64 ± 0.079 75.08 ± 0.623 89.32 ± 0.626 88.29 ± 0.363 16.13 ± 0.225 Dual Pix Model 98.04 ± 1.397 97.31 ± 0.575 96.52 ± 0.658 92.05 ± 1.212 113.4 ± 0.820 106.1 ± 0.868 9.296 ± 1.346 CDVAE 19.86 ± 7.396 99.00 ± 0.100 64.34 ± 4.903 101.6 ± 8.347 93.55 ± 0.391 107.6 ± 1.724 30.61 ± 1.240 PixCDVAE 56.53 ± 4.032 96.77 ± 0.337 83.61 ± 0.927 102.4 ± 6.195 118.2 ± 1.572 118.7 ± 5.320 16.37 ± 0.970 OCDVAE 92.35 ± 4.485 99.06 ± 0.171 93.24 ± 3.742 77.16 ± 1.104 89.68 ± 0.618 92.92 ± 2.283 21.02 ± 0.717 PixOCDVAE 97.44 ± 0.785 98.63 ± 0.430 96.84 ± 0.346 100.5 ± 4.942 113.3 ± 0.755 111.9 ± 2.663 12.49 ± 0.551 ± 0.076 99.60 ± 0.023 20.06 ± 0.059 Dual Model 94.26 ± 0.192 93.55 ± 0.708 63.21 ± 1.957 217.7 ± 1.510 242.8 ± 0.898 230.5 ± 1.543 11.45 ± 0.228 Dual Pix Model 60.04 ± 5.151 98.85 ± 0.141 72.41 ± 2.941 274.1 ± 0.349 305.8 ± 0.286 289.5 ± 0.396 9.781 ± 1.068 CDVAE 39.51 ± 7.173 96.92 ± 0.774 58.82 ± 2.521 232.8 ± 5.048 248.8 ± 0.398 242.2 ± 0.754 26.68 ± 0.859 PixCDVAE 47.83 ± 13.41 97.91 ± 0.596 63.05 ± 1.826 241.1 ± 1.747 283.2 ± 2.150 271.7 ± 2.117 22.14 ± 0.377 OCDVAE 60.63 ± 12.16 96.51 ± 0.707 69.88 ± 1.712 222.8 ± 1.632 244.0 ± 0.646 234.6 ± 0.823 20.47 ± 0.742 PixOCDVAE 74.45 ± 2.889 98.63 ± 0.176 80.85 ± 0.721 234.1 ± 1.498 283.5 ± 2.458 267.2 ± 0.586 17.93 ± 0.360 ± 0.007 99.41 ± 0.207 19.98 ± 0.032 Dual Model 61.58 ± 0.747 89.41 ± 0.691 47.42 ± 1.447 425.2 ± 0.244 422.7 ± 0.784 432.7 ± 0.385 5.470 ± 0.055 Dual Pix Model 64.60 ± 8.739 98.18 ± 0.885 75.50 ± 3.032 435.1 ± 1.915 431.9 ± 1.032 440.3 ± 1.297 6.031 ± 0.832 CDVAE 59.36 ± 7.147 84.93 ± 6.297 81.49 ± 1.944 422.7 ± 0.182 423.9 ± 0.681 431.4 ± 0.255 22.96 ± 0.912 PixCDVAE 29.94 ± 18.47 97.00 ± 0.520 63.44 ± 5.252 431.4 ± 0.666 428.0 ± 0.851 436.9 ± 0.751 27.14 ± 1.139 OCDVAE 79.73 ± 4.070 89.52 ± 6.586 87.72 ± 1.594 423.5 ± 0.586 422.9 ± 0.537 430.9 ± 0.541 18.52 ± 1.131 PixOCDVAE 75.25 ± 10.18 99.43 ± 0.495 90.23 ± 1.139 432.3 ± 0.189 429.7 ± 1.223 437.7 ± 0.432 17.45 ± 0.835

Table content: <table><tr><td>Class-incremental</td><td/><td>α T (%) ↑</td><td/><td/><td>γ T (nats) ↓</td><td/><td>KL T (nats) ↓</td></tr><tr><td/><td>base</td><td>new</td><td>all</td><td>base</td><td>new</td><td>all</td><td>all</td></tr><tr><td>CDVAE ISO</td><td/><td/><td>99.45</td><td/><td/><td>78.12</td><td>22.12</td></tr><tr><td>CDVAE UB</td><td>99.57</td><td>99.10</td><td>99.29</td><td>64.99</td><td>85.88</td><td>81.97</td><td>21.01</td></tr><tr><td>CDVAE LB</td><td>00.00</td><td>99.85</td><td>20.16</td><td>123.2</td><td>92.00</td><td>163.7</td><td>24.87</td></tr><tr><td colspan="4">EWC Dual Model CDVAE ISO CDVAE UB CDVAE LB 97.31 FashionMNIST 00.45 ± 0.059 99.58 ± 0.052 20.26 ± 0.027 MNIST 89.54 92.20 97.50 89.24 00.00 99.80 19.97 EWC CDVAE ISO 97.75 CDVAE UB 98.42 98.67 97.87 CDVAE LB 00.00 100.0 20.02 00.17 AudioMNIST EWC 00.11</td><td>208.4 306.5 418.4 432.9</td><td>246.2 242.0 421.3 425.2</td><td>224.8 226.2 275.1 429.7 427.2 440.4</td><td>23.27 20.27 21.61 17.89 15.15 14.52</td></tr></table>

± 0.104 96.78 ± 0.067 91.75 ± 0.064 320.0 ± 1.275 431.1 ± 1.474 273.7 ± 1.174 12.80 ± 0.060 Dual Pix Model 82.88 ± 0.116 97.23 ± 0.212 92.16 ± 0.061 288.5 ± 0.723 437.7 ± 0.404 251.6 ± 0.231 9.025 ± 1.378 CDVAE 57.70 ± 4.480 96.73 ± 0.235 81.10 ± 1.769 360.9 ± 20.15 432.1 ± 0.231 296.4 ± 7.966 44.29 ± 4.047 PixCDVAE 56.44 ± 1.831 97.50 ± 0.184 80.76 ± 0.842 289.8 ± 1.283 438.1 ± 0.990 252.6 ± 1.424 29.99 ± 0.629 OCDVAE 80.11 ± 2.922 97.63 ± 0.042 91.13 ± 1.045 345.1 ± 7.446 430.7 ± 0.600 280.2 ± 1.069 25.42 ± 1.876 PixOCDVAE 81.84 ± 0.212 97.75 ± 0.169 91.76 ± 0.212 288.8 ± 0.141 437.1 ± 0.725 251.8 ± 0.636 21.07 ± 0.248 reconstruction likelihoods akin to a dual model's separate VAE, while fully sharing encoded knowledge and maintaining a classifier. As a result of OCDVAE's shared weights, we observe backward transfer for some experiments. This is particularly apparent for AudioMNIST, where addition of the second increment first decays and inclusion of later tasks improves the second task's accuracy. Due to space constraints, we provide a detailed account of all intermediate results

Table content: <table><tr><td>Cross-dataset</td><td>α T (%) ↑</td><td/><td/><td>γ T (nats) ↓</td><td/><td>KL T (nats) ↓</td></tr><tr><td>base</td><td>new</td><td>all</td><td>base</td><td>new</td><td>all</td><td>all</td></tr><tr><td colspan="7">CDVAE ISO CDVAE UB CDVAE LB EWC Dual Model CDVAE ISO CDVAE UB CDVAE LB EWC Dual Model Dual Pix Model 71.58 ± 2.536 88.76 ± 0.255 88.61 ± 0.547 445.8 ± 1.601 290.4 ± 0.603 255.0 ± 0.533 9.164 ± 1.312 94.95 269.6 24.97 89.10 97.88 95.00 311.2 434.3 269.7 25.20 00.00 98.12 22.70 689.7 341.0 511.7 98.74 22.85 ± 0.294 93.31 ± 0.138 43.42 ± 0.063 94.95 269.6 24.97 97.17 89.16 94.91 428.8 311.9 268.2 23.91 00.00 89.72 34.51 506.6 311.0 351.1 34.13 3.420 ± 0.026 87.54 ± 0.214 45.42 ± 0.731 66.82 ± 0.337 89.15 ± 0.050 87.70 ± 0.102 447.3 ± 6.700 308.5 ± 0.599 270.9 ± 1.299 12.89 ± 0.109 CDVAE 79.74 ± 2.431 88.50 ± 0.126 89.46 ± 0.600 448.6 ± 5.187 315.1 ± 1.305 281.6 ± 3.205 33.38 ± 0.898 PixCDVAE 49.38 ± 2.256 88.54 ± 0.042 82.18 ± 0.672 441.4 ± 0.495 287.0 ± 0.212 252.5 ± 0.201 30.60 ± 1.556 81.89 Audio-MNIST-Fashion Fashion-MNIST-Audio OCDVAE 94.53</td></tr></table>

Test accuracies and outlier detection values of the joint OCDVAE and dual model approaches when considering 95 % of known tasks' validation data is inlying. Percentage of detected outliers is reported based on classifier predictive entropy, reconstruction loss and our posterior based EVT approach, averaged over 100 z ∼ q θ (z|x) samples per data-point respectively.

Table content: <table><tr><td colspan="10">Outlier detection at 95% validation inliers (%) MNIST Fashion Audio KMNIST CIFAR10 CIFAR100 SVHN</td></tr><tr><td colspan="2">Trained Model</td><td colspan="2">Test acc. Criterion</td><td/><td/><td/><td/><td/></tr><tr><td/><td>Dual,</td><td>99.40</td><td>Class entropy</td><td>4.160</td><td>90.43</td><td>97.53 95.29</td><td>98.54</td><td>98.63</td><td>95.51</td></tr><tr><td>MNIST</td><td>CL + VAE Joint,</td><td>99.53</td><td colspan="2">Reconstruction 5.522 Latent EVT 4.362 Class entropy 3.948</td><td>99.98 99.41 95.15</td><td>99.97 99.98 99.80 99.86 98.55 95.49</td><td>99.99 99.95 99.47</td><td>99.96 99.97 99.34</td><td>99.98 99.52 97.98</td></tr><tr><td/><td>OCDVAE</td><td/><td colspan="2">Reconstruction 5.083</td><td>99.50</td><td>99.98 99.91</td><td>99.97</td><td>99.99</td><td>99.98</td></tr><tr><td/><td/><td/><td>Latent EVT</td><td>4.361</td><td>99.78</td><td>99.67 99.73</td><td>99.96</td><td>99.93</td><td>99.70</td></tr><tr><td>FashionMNIST</td><td>Dual, CL + VAE Joint, OCDVAE</td><td>90.48 90.92</td><td colspan="2">Class entropy Recon Latent EVT Class Entropy Reconstruction 0.601 74.71 5.535 96.22 66.91 Latent EVT 96.23</td><td>5.461 5.340 5.138 5.145 5.483 5.216</td><td>69.65 77.85 64.10 31.33 93.00 91.51 61.86 56.14 63.00 28.69 94.76 96.07</td><td>24.91 99.50 71.82 43.98 99.67 96.15</td><td>28.76 98.41 72.08 46.59 98.91 95.94</td><td>36.64 97.24 73.85 37.85 98.56 96.84</td></tr><tr><td>AudioMNIST</td><td>Dual, CL + VAE Joint, OCDVAE</td><td>98.53 98.57</td><td colspan="2">Class entropy Reconstruction 6.235 97.63 Latent EVT 99.82 Class entropy 99.23 Reconstruction 0.614 Latent EVT 99.91</td><td>57.64 46.32 78.74 89.33 38.50 99.53</td><td>5.066 95.53 4.433 98.73 5.038 99.47 5.731 99.15 3.966 36.05 5.089 99.81</td><td>66.49 98.63 93.44 92.31 98.62 100.0</td><td>65.25 98.63 92.76 91.06 98.54 99.99</td><td>54.91 97.45 88.73 85.77 96.99 99.98</td></tr></table>

Losses for the used WRN architecture with 2-D latent space and different β values for MNIST. As stated in the main body, losses are normalized per dimension for training. Unnormalized values in nats are reported in brackets for reference purposes.

Table content: <table><tr><td/><td/><td colspan="3">In nats per dimension (nats in brackets)</td></tr><tr><td colspan="2">2-D latent Beta</td><td>KLD</td><td>Recon loss</td><td>Class Loss</td><td>Accuracy [%]</td></tr><tr><td>train</td><td colspan="4">1.0 1.039 (2.078) 0.237 (185.8) 0.539 (5.39)</td><td>79.87</td></tr><tr><td>test</td><td/><td colspan="3">1.030 (2.060) 0.235 (184.3) 0.596 (5.96)</td><td>78.30</td></tr><tr><td>train</td><td colspan="4">0.5 1.406 (2.812) 0.230 (180.4) 0.221 (2.21)</td><td>93.88</td></tr><tr><td>test</td><td/><td colspan="3">1.382 (2.764) 0.228 (178.8) 0.305 (3.05)</td><td>92.07</td></tr><tr><td>train</td><td colspan="4">0.1 2.055 (4.110) 0.214 (167.8) 0.042 (0.42)</td><td>99.68</td></tr><tr><td>test</td><td/><td colspan="3">2.071 (4.142) 0.212 (166.3) 0.116 (1.16)</td><td>98.73</td></tr><tr><td>train</td><td colspan="4">0.05 2.395 (4.790) 0.208 (163.1) 0.025 (0.25)</td><td>99.83</td></tr><tr><td>test</td><td/><td colspan="3">2.382 (4.764) 0.206 (161.6) 0.159 (1.59)</td><td>98.79</td></tr></table>

Losses for the used WRN architecture with 60-D latent space and different β values for MNIST. As stated in the main body, losses are normalized per dimension for training. Unnormalized values in nats are reported in brackets for reference purposes.

Table content: <table><tr><td/><td/><td colspan="3">In nats per dimension (nats in brackets)</td></tr><tr><td colspan="2">60-D latent Beta</td><td>KLD</td><td>Recon loss</td><td>Class Loss</td><td>Accuracy [%]</td></tr><tr><td>train</td><td colspan="4">1.0 0.108 (6.480) 0.184 (144.3) 0.0110 (0.110)</td><td>99.71</td></tr><tr><td>test</td><td/><td colspan="3">0.110 (6.600) 0.181 (142.0) 0.0457 (0.457)</td><td>99.03</td></tr><tr><td>train</td><td colspan="4">0.5 0.151 (9.060) 0.162 (127.1) 0.0052 (0.052)</td><td>99.87</td></tr><tr><td>test</td><td/><td colspan="3">0.156 (9.360) 0.159 (124.7) 0.0451 (0.451)</td><td>99.14</td></tr><tr><td>train</td><td colspan="4">0.1 0.346 (20.76) 0.124 (97.22) 0.0022 (0.022)</td><td>99.95</td></tr><tr><td>test</td><td/><td colspan="3">0.342 (20.52) 0.126 (98.79) 0.0286 (0.286)</td><td>99.38</td></tr><tr><td>train</td><td colspan="4">0.05 0.476 (28.56) 0.115 (90.16) 0.0018 (0.018)</td><td>99.95</td></tr><tr><td>test</td><td/><td colspan="3">0.471 (28.26) 0.118 (92.53) 0.0311 (0.311)</td><td>99.34</td></tr></table>

Results for class incremental continual learning approaches averaged over 5 runs, baselines and the reference isolated learning scenario for MNIST at the end of every task increment. α t and γ t indicate the respective accuracy and NLL reconstruction metrics at the end of every task increment t. KL t denotes the corresponding KL divergence. ± 1.724 118.7 ± 5.320 92.92 ± 2.283 111.9 ± 2.663

Table content: <table><tr><td>MNIST</td><td colspan="4">t CDVAE ISO CDVAE UB CDVAE LB EWC</td><td colspan="3">Dual Model Dual Pix Model CDVAE</td><td>PixCDVAE OCDVAE</td><td>PixOCDVAE</td></tr><tr><td/><td>1</td><td>100.0</td><td>100.0</td><td colspan="2">99.88 ± 0.010 99.98 ± 0.023</td><td>99.97 ± 0.002</td><td>99.97 ± 0.029 99.97 ± 0.026 99.98 ± 0.018 99.86 ± 0.084</td></tr><tr><td>αbase,t</td><td>2 3</td><td>99.82 99.80</td><td>00.00 00.00</td><td colspan="2">00.61 ± 0.057 99.77 ± 0.032 00.17 ± 0.045 99.51 ± 0.094</td><td>99.54 ± 0.285 99.16 ± 0.611</td><td>97.28 ± 3.184 96.90 ± 2.907 99.30 ± 0.100 99.64 ± 0.095 87.66 ± 8.765 90.12 ± 5.846 96.69 ± 2.173 98.88 ± 0.491</td></tr><tr><td>(%)</td><td>4</td><td>99.85</td><td>00.00</td><td colspan="2">00.49 ± 0.017 98.90 ± 0.207</td><td>98.33 ± 1.119</td><td>54.70 ± 22.84 76.84 ± 9.095 94.71 ± 1.792 98.11 ± 0.797</td></tr><tr><td/><td>5</td><td>99.57</td><td>00.00</td><td colspan="2">00.45 ± 0.059 97.31 ± 0.489</td><td>98.04 ± 1.397</td><td>19.86 ± 7.396 56.53 ± 4.032 92.53 ± 4.485 97.44 ± 0.785</td></tr><tr><td/><td>1</td><td>100.0</td><td>100.0</td><td colspan="2">99.88 ± 0.010 99.98 ± 0.023</td><td>99.97 ± 0.002</td><td>99.97 ± 0.029 99.97 ± 0.026 99.98 ± 0.018 99.86 ± 0.084</td></tr><tr><td>αnew,t</td><td>2 3</td><td>99.80 99.67</td><td>99.85 99.94</td><td colspan="2">99.70 ± 0.013 99.81 ± 0.062 99.94 ± 0.002 99.48 ± 0.294</td><td>99.71 ± 0.122 99.41 ± 0.084</td><td>99.75 ± 0.127 99.74 ± 0.052 99.80 ± 0.126 99.82 ± 0.027 99.63 ± 0.172 99.22 ± 0.082 99.61 ± 0.055 99.56 ± 0.092</td></tr><tr><td>(%)</td><td>4</td><td>99.49</td><td>100.0</td><td colspan="2">99.87 ± 0.015 99.46 ± 0.315</td><td>98.61 ± 0.312</td><td>99.05 ± 0.470 97.84 ± 0.180 99.15 ± 0.032 98.80 ± 0.292</td></tr><tr><td/><td>5</td><td>99.10</td><td>99.86</td><td colspan="2">99.58 ± 0.052 98.59 ± 0.106</td><td>97.31 ± 0.575</td><td>99.00 ± 0.100 96.77 ± 0.337 99.06 ± 0.171 98.63 ± 0.430</td></tr><tr><td/><td>1</td><td>100.0</td><td>100.0</td><td colspan="2">99.88 ± 0.010 99.98 ± 0.023</td><td>99.97 ± 0.002</td><td>99.97 ± 0.029 99.97 ± 0.026 99.98 ± 0.018 99.86 ± 0.084</td></tr><tr><td>αall,t</td><td>2 3</td><td>99.81 99.72</td><td>49.92 31.35</td><td colspan="2">50.16 ± 0.029 99.79 ± 0.049 33.42 ± 0.027 99.32 ± 0.057</td><td>99.60 ± 0.142 98.93 ± 0.291</td><td>98.54 ± 1.638 98.37 ± 1.448 99.55 ± 0.036 99.69 ± 0.051 95.01 ± 3.162 96.14 ± 1.836 98.46 ± 0.903 99.20 ± 0.057</td></tr><tr><td>(%)</td><td>4</td><td>99.50</td><td>24.82</td><td colspan="2">25.36 ± 0.025 98.56 ± 0.021</td><td>98.22 ± 0.560</td><td>81.50 ± 9.369 91.25 ± 0.992 97.06 ± 1.069 98.13 ± 0.281</td></tr><tr><td/><td>5 99.45</td><td>99.29</td><td>20.16</td><td colspan="2">20.26 ± 0.027 96.64 ± 0.079</td><td>96.52 ± 0.658</td><td>64.34 ± 4.903 83.61 ± 0.927 93.24 ± 3.742 96.84 ± 0.346</td></tr><tr><td/><td>1</td><td>63.18</td><td>62.08</td><td/><td>62.17 ± 0.979</td><td>90.52 ± 0.263</td><td>64.34 ± 2.054 100.0 ± 1.572 62.53 ± 1.166 99.77 ± 2.768</td></tr><tr><td>γbase,t</td><td>2 3</td><td>62.85 63.36</td><td>126.8 160.4</td><td/><td>63.69 ± 0.576 67.34 ± 0.445</td><td>91.27 ± 0.789 91.92 ± 0.991</td><td>74.41 ± 10.89 100.4 ± 1.964 65.68 ± 1.166 101.2 ± 3.601 81.89 ± 10.09 100.3 ± 4.562 69.29 ± 1.541 101.1 ± 4.014</td></tr><tr><td>(nats) or</td><td>4</td><td>64.25</td><td>126.9</td><td/><td>70.41 ± 0.436</td><td>91.75 ± 1.136</td><td>90.62 ± 10.08 102.7 ± 7.134 71.69 ± 1.379 101.0 ± 4.573</td></tr><tr><td colspan="2">(bits/dim) 5</td><td>64.99</td><td>123.2</td><td/><td>75.08 ± 0.623</td><td>92.05 ± 1.212</td><td>101.6 ± 8.347 102.4 ± 6.195 77.16 ± 1.104 100.5 ± 4.942</td></tr><tr><td/><td>1</td><td>63.18</td><td>62.08</td><td/><td>62.17 ± 0.979</td><td>90.52 ± 0.263</td><td>64.34 ± 2.054 100.0 ± 1.572 62.53 ± 1.166 99.77 ± 2.768</td></tr><tr><td>γnew,t</td><td>2 3</td><td>88.75 82.53</td><td>87.93 87.22</td><td/><td>88.03 ± 0.664 83.46 ± 0.992</td><td>115.8 ± 0.805 107.7 ± 0.600</td><td>89.91 ± 0.107 125.7 ± 2.413 89.64 ± 3.709 124.6 ± 3.822 87.65 ± 0.530 118.3 ± 3.523 85.37 ± 1.725 116.5 ± 2.219</td></tr><tr><td>(nats) or</td><td>4</td><td>72.68</td><td>74.61</td><td/><td>73.23 ± 0.280</td><td>100.9 ± 0.659</td><td>79.49 ± 0.489 107.1 ± 5.316 74.75 ± 0.777 102.3 ± 1.844</td></tr><tr><td colspan="2">(bits/dim) 5</td><td>85.88</td><td>92.00</td><td/><td>89.32 ± 0.626</td><td>113.4 ± 0.820</td><td>93.55 ± 0.391 118.2 ± 1.572 89.68 ± 0.618 113.3 ± 0.755</td></tr><tr><td/><td>1</td><td>63.18</td><td>62.08</td><td/><td>62.17 ± 0.979</td><td>90.52 ± 0.263</td><td>64.34 ± 2.054 100.0 ± 1.572 62.53 ± 1.166 99.77 ± 2.768</td></tr><tr><td>γall,t</td><td>2 3</td><td>75.97 79.58</td><td>107.3 172.3</td><td/><td>75.64 ± 0.600 81.24 ± 0.262</td><td>102.9 ± 0.408 104.8 ± 1.114</td><td>82.02 ± 5.488 111.9 ± 2.627 76.62 ± 1.695 112.7 ± 3.300 89.88 ± 3.172 114.9 ± 4.590 82.95 ± 1.878 114.6 ± 4.788</td></tr><tr><td>(nats) or</td><td>4</td><td>79.72</td><td>203.1</td><td/><td>82.92 ± 0.489</td><td>103.9 ± 0.759</td><td>95.83 ± 2.747 114.3 ± 3.963 85.30 ± 1.524 112.1 ± 2.150</td></tr><tr><td colspan="8">(bits/dim) 5 78.12 1 2 107.6 KLall,t 81.97 163.7 88.29 ± 0.363 106.1 ± 0.868 12.55 13.08 11.81 ± 0.123 1.410 ± 0.181 13.00 ± 0.897 5.629 ± 3.749 13.68 ± 0.785 5.635 ± 3.739 18.50 25.84 16.15 ± 0.149 3.177 ± 0.702 20.20 ± 1.188 9.238 ± 0.674 18.01 ± 0.154 7.495 ± 0.738 3 20.16 24.28 16.46 ± 0.122 4.923 ± 1.085 24.24 ± 1.974 12.13 ± 0.977 20.02 ± 0.161 10.17 ± 1.528</td></tr><tr><td>(nats)</td><td>4</td><td>20.48</td><td>26.32</td><td/><td>16.09 ± 0.177</td><td>5.603 ± 1.250</td><td>27.01 ± 1.851 14.32 ± 1.040 20.26 ± 0.186 11.66 ± 1.004</td></tr><tr><td/><td>5 22.12</td><td>21.02</td><td>24.87</td><td/><td>16.13 ± 0.225</td><td>9.296 ± 1.346</td><td>30.61 ± 1.240 16.37 ± 0.970 21.02 ± 0.717 12.49 ± 0.551</td></tr></table>

Results for class incremental continual learning approaches averaged over 5 runs, baselines and the reference isolated learning scenario for FashionMNIST at the end of every task increment. α t and γ t indicate the respective accuracy and NLL reconstruction metrics at the end of every task increment t. KL t denotes the corresponding KL divergence.

Table content: <table><tr><td>Fashion</td><td colspan="4">t CDVAE ISO CDVAE UB CDVAE LB EWC</td><td colspan="3">Dual Model Dual Pix Model CDVAE</td><td>PixCDVAE OCDVAE</td><td>PixOCDVAE</td></tr><tr><td/><td>1</td><td>99.65</td><td>99.60</td><td colspan="2">99.17 ± 0.037 99.58 ± 0.062</td><td>99.57 ± 0.091</td><td>99.55 ± 0.035 99.58 ± 0.076 99.59 ± 0.082 99.54 ± 0.079</td></tr><tr><td>αbase,t</td><td>2 3</td><td>96.70 95.95</td><td>00.00 00.00</td><td colspan="2">02.40 ± 0.122 94.50 ± 0.389 01.63 ± 0.032 94.88 ± 0.432</td><td>82.40 ± 6.688 78.55 ± 3.964</td><td>92.02 ± 1.175 90.06 ± 1.782 92.36 ± 2.092 88.60 ± 1.998 79.26 ± 4.170 83.70 ± 3.571 83.90 ± 2.310 87.66 ± 0.375</td></tr><tr><td>(%)</td><td>4</td><td>91.35</td><td>00.00</td><td colspan="2">00.33 ± 0.097 82.25 ± 4.782</td><td>54.69 ± 3.853</td><td>50.16 ± 6.658 50.23 ± 7.004 64.70 ± 2.580 68.31 ± 3.308</td></tr><tr><td/><td>5</td><td>92.20</td><td>00.00</td><td colspan="2">00.17 ± 0.076 94.26 ± 0.192</td><td>60.04 ± 5.151</td><td>39.51 ± 7.173 47.83 ± 13.41 60.63 ± 12.16 74.45 ± 2.889</td></tr><tr><td/><td>1</td><td>99.65</td><td>99.60</td><td colspan="2">99.17 ± 0.037 99.58 ± 0.062</td><td>99.57 ± 0.091</td><td>99.55 ± 0.035 99.58 ± 0.076 99.59 ± 0.082 99.54 ± 0.079</td></tr><tr><td>αnew,t</td><td>2 3</td><td>95.55 93.35</td><td>97.95 99.95</td><td colspan="2">96.09 ± 0.260 89.31 ± 0.311 99.92 ± 0.012 86.06 ± 2.801</td><td>97.73 ± 1.113 99.09 ± 0.367</td><td>90.98 ± 0.626 96.47 ± 0.596 92.64 ± 2.302 97.31 ± 0.475 90.26 ± 1.435 97.33 ± 0.725 83.40 ± 3.089 96.88 ± 1.156</td></tr><tr><td>(%)</td><td>4</td><td>84.75</td><td>99.90</td><td colspan="2">99.95 ± 0.060 73.63 ± 3.861</td><td>97.55 ± 0.588</td><td>85.65 ± 2.127 96.12 ± 0.675 84.18 ± 2.715 95.47 ± 1.332</td></tr><tr><td/><td>5</td><td>97.50</td><td>99.80</td><td colspan="2">99.60 ± 0.023 93.55 ± 0.708</td><td>98.85 ± 0.141</td><td>96.92 ± 0.774 97.91 ± 0.596 96.51 ± 0.707 98.63 ± 0.176</td></tr><tr><td/><td>1</td><td>99.65</td><td>99.60</td><td colspan="2">99.17 ± 0.037 99.58 ± 0.062</td><td>99.57 ± 0.091</td><td>99.55 ± 0.035 99.58 ± 0.076 99.59 ± 0.082 99.54 ± 0.079</td></tr><tr><td>αall,t</td><td>2 3</td><td>95.75 93.02</td><td>48.97 33.33</td><td colspan="2">49.28 ± 0.242 91.91 ± 0.043 34.34 ± 0.009 79.98 ± 0.634</td><td>86.22 ± 3.704 76.77 ± 4.378</td><td>91.83 ± 0.730 92.93 ± 0.160 92.31 ± 1.163 92.17 ± 1.425 83.35 ± 1.597 84.07 ± 1.069 86.93 ± 0.870 87.30 ± 0.322</td></tr><tr><td>(%)</td><td>4</td><td>87.51</td><td>25.00</td><td colspan="2">25.21 ± 0.100 64.37 ± 0.707</td><td>62.93 ± 3.738</td><td>64.66 ± 3.204 64.42 ± 1.837 76.05 ± 1.391 76.36 ± 1.267</td></tr><tr><td/><td>5 89.54</td><td>89.24</td><td>19.97</td><td colspan="2">20.06 ± 0.059 63.21 ± 1.957</td><td>72.41 ± 2.941</td><td>58.82 ± 2.521 63.05 ± 1.826 69.88 ± 1.712 80.85 ± 0.721</td></tr><tr><td/><td>1</td><td>209.7</td><td>209.8</td><td/><td>207.7 ± 1.558</td><td>267.8 ± 1.246</td><td>208.9 ± 1.213 230.8 ± 3.024 209.7 ± 3.655 232.0 ± 2.159</td></tr><tr><td>γbase,t</td><td>2 3</td><td>207.4 207.6</td><td>240.7 258.7</td><td/><td>209.0 ± 0.731 213.0 ± 1.854</td><td>273.6 ± 0.631 274.0 ± 0.552</td><td>212.7 ± 0.579 232.5 ± 1.582 212.1 ± 0.937 231.8 ± 0.416 219.5 ± 1.376 235.6 ± 2.784 216.9 ± 1.208 231.6 ± 0.832</td></tr><tr><td>(nats) or</td><td>4</td><td>207.7</td><td>243.6</td><td/><td>213.6 ± 0.509</td><td>273.7 ± 0.504</td><td>223.8 ± 0.837 236.4 ± 3.157 217.1 ± 0.979 231.4 ± 2.550</td></tr><tr><td colspan="2">(bits/dim) 5</td><td>208.4</td><td>306.5</td><td/><td>217.7 ± 1.510</td><td>274.1 ± 0.349</td><td>232.8 ± 5.048 241.1 ± 1.747 222.8 ± 1.632 234.1 ± 1.498</td></tr><tr><td/><td>1</td><td>209.7</td><td>209.8</td><td/><td>207.7 ± 1.558</td><td>267.8 ± 1.246</td><td>208.9 ± 1.213 230.8 ± 3.024 209.7 ± 3.655 232.0 ± 2.159</td></tr><tr><td>γnew,t</td><td>2 3</td><td>241.1 213.6</td><td>240.2 211.8</td><td/><td>238.7 ± 0.081 211.6 ± 0.543</td><td>313.4 ± 1.006 269.1 ± 0.616</td><td>241.8 ± 0.502 275.8 ± 1.888 241.9 ± 0.960 275.3 ± 1.473 215.4 ± 0.501 268.3 ± 3.852 213.0 ± 0.635 262.9 ± 1.893</td></tr><tr><td>(nats) or</td><td>4</td><td>220.5</td><td>219.7</td><td/><td>219.5 ± 0.216</td><td>282.4 ± 0.321</td><td>223.6 ± 0.381 259.1 ± 1.305 220.9 ± 0.522 259.6 ± 2.050</td></tr><tr><td colspan="2">(bits/dim) 5</td><td>246.2</td><td>242.0</td><td/><td>242.8 ± 0.898</td><td>305.8 ± 0.286</td><td>248.8 ± 0.398 283.2 ± 2.150 244.0 ± 0.646 283.5 ± 2.458</td></tr><tr><td/><td>1</td><td>209.7</td><td>209.8</td><td/><td>207.7 ± 1.558</td><td>267.8 ± 1.246</td><td>208.9 ± 1.213 230.8 ± 3.024 209.7 ± 3.655 232.0 ± 2.159</td></tr><tr><td>γall,t</td><td>2 3</td><td>224.2 220.7</td><td>240.4 246.1</td><td/><td>223.8 ± 0.402 221.9 ± 0.648</td><td>293.8 ± 0.349 285.7 ± 0.510</td><td>226.6 ± 2.31 227.2 ± 0.606 261.5 ± 2.970 224.9 ± 0.642 259.1 ± 0.929 254.3 ± 1.513 226.9 ± 0.918 255.8 ± 0.436</td></tr><tr><td>(nats) or</td><td>4</td><td>220.4</td><td>238.7</td><td/><td>225.1 ± 3.629</td><td>284.9 ± 0.703</td><td>230.4 ± 0.524 263.2 ± 2.259 226.1 ± 0.560 259.5 ± 3.218</td></tr><tr><td colspan="2">(bits/dim) 5 224.8</td><td>226.2</td><td>275.1</td><td/><td>230.5 ± 1.543</td><td>289.5 ± 0.396</td><td>242.2 ± 0.754 271.7 ± 2.117 234.6 ± 0.823 267.2 ± 0.586</td></tr><tr><td/><td>1</td><td>12.17</td><td>12.20</td><td/><td>9.710 ± 0.345</td><td>3.610 ± 0.856</td><td>13.21 ± 0.635 7.164 ± 0.759 13.28 ± 0.644 7.809 ± 1.255</td></tr><tr><td>KLall,t</td><td>2 3</td><td>16.54 18.84</td><td>17.47 19.34</td><td/><td>10.65 ± 0.101 11.34 ± 0.057</td><td>6.247 ± 0.710 7.811 ± 0.799</td><td>17.60 ± 0.755 13.79 ± 0.282 15.56 ± 0.696 12.23 ± 0.287 21.25 ± 0.872 18.26 ± 0.818 17.35 ± 0.307 15.36 ± 0.530</td></tr><tr><td>(nats)</td><td>4</td><td>20.06</td><td>17.31</td><td/><td>10.96 ± 0.106</td><td>8.982 ± 0.812</td><td>25.21 ± 0.929 21.75 ± 0.561 19.81 ± 0.462 18.31 ± 0.333</td></tr><tr><td/><td>5 23.27</td><td>20.27</td><td>21.61</td><td/><td>11.45 ± 0.228</td><td>9.781 ± 1.068</td><td>26.68 ± 0.859 22.14 ± 0.377 20.47 ± 0.742 17.93 ± 0.360</td></tr></table>

Results for class incremental continual learning approaches averaged over 5 runs, baselines and the reference isolated learning scenario for AudioMNIST at the end of every task increment. α t and γ t indicate the respective accuracy and NLL reconstruction metrics at the end of every task increment t. KL t denotes the corresponding KL divergence.

Table content: <table><tr><td>Audio</td><td colspan="4">t CDVAE ISO CDVAE UB CDVAE LB EWC</td><td colspan="3">Dual Model Dual Pix Model CDVAE</td><td>PixCDVAE OCDVAE</td><td>PixOCDVAE</td></tr><tr><td/><td>1</td><td>99.99</td><td>100.0</td><td colspan="2">100.0 ± 0.000 100.0 ± 0.000</td><td>100.0 ± 0.000</td><td colspan="2">99.21 ± 0.568 99.71 ± 0.218 99.95 ± 0.035 99.27 ± 0.410</td></tr><tr><td>αbase,t</td><td>2 3</td><td>99.92 100.0</td><td>00.00 00.00</td><td colspan="2">00.16 ± 0.040 93.08 ± 5.854 00.29 ± 0.029 83.25 ± 6.844</td><td>99.52 ± 0.273 93.15 ± 3.062</td><td colspan="2">98.98 ± 0.766 97.86 ± 0.799 98.61 ± 0.490 97.88 ± 2.478 92.44 ± 1.306 81.38 ± 5.433 95.12 ± 2.248 95.82 ± 3.602</td></tr><tr><td>(%)</td><td>4</td><td>99.92</td><td>00.00</td><td colspan="2">00.31 ± 0.015 72.02 ± 0.677</td><td>81.55 ± 8.468</td><td colspan="2">76.43 ± 4.715 50.58 ± 14.60 86.37 ± 5.63</td><td>91.56 ± 5.640</td></tr><tr><td/><td>5</td><td>98.42</td><td>00.00</td><td colspan="2">00.11 ± 0.007 61.57 ± 0.747</td><td>64.60 ± 8.739</td><td colspan="2">59.36 ± 7.147 29.94 ± 18.47 79.73 ± 4.070 75.25 ± 10.18</td></tr><tr><td/><td>1</td><td>99.99</td><td>100.0</td><td colspan="2">100.0 ± 0.000 100.0 ± 0.000</td><td>100.0 ± 0.000</td><td colspan="2">99.21 ± 0.568 99.71 ± 0.218 99.95 ± 0.035 99.27 ± 0.410</td></tr><tr><td>αnew,t</td><td>2 3</td><td>99.75 98.92</td><td>100.0 99.58</td><td colspan="2">99.78 ± 0.019 86.25 ± 8.956 99.25 ± 0.054 95.16 ± 1.490</td><td>99.71 ± 0.043 98.23 ± 1.092</td><td colspan="2">91.82 ± 4.577 99.78 ± 0.128 89.23 ± 7.384 99.81 ± 0.189 95.20 ± 1.495 98.41 ± 0.507 94.43 ± 3.030 99.30 ± 0.550</td></tr><tr><td>(%)</td><td>4</td><td>97.33</td><td>98.67</td><td colspan="2">97.03 ± 0.019 62.52 ± 4.022</td><td>95.31 ± 0.868</td><td colspan="2">53.02 ± 6.132 94.30 ± 0.914 72.22 ± 8.493 97.87 ± 0.293</td></tr><tr><td/><td>5</td><td>98.67</td><td>100.0</td><td colspan="2">99.41 ± 0.207 89.41 ± 0.691</td><td>98.18 ± 0.885</td><td colspan="2">84.93 ± 6.297 97.00 ± 0.520 89.52 ± 6.586 99.43 ± 0.495</td></tr><tr><td/><td>1</td><td>99.99</td><td>100.0</td><td colspan="2">100.0 ± 0.000 100.0 ± 0.000</td><td>100.0 ± 0.000</td><td colspan="2">99.21 ± 0.568 99.71 ± 0.218 99.95 ± 0.035 99.27 ± 0.410</td></tr><tr><td>αall,t</td><td>2 3</td><td>99.83 99.56</td><td>50.00 33.19</td><td colspan="2">50.16 ± 0.119 89.67 ± 1.763 33.28 ± 0.022 78.24 ± 3.315</td><td>99.50 ± 0.157 95.37 ± 1.750</td><td colspan="2">93.84 ± 2.558 98.64 ± 0.875 93.93 ± 3.756 99.67 ± 0.033 94.26 ± 1.669 90.10 ± 1.431 95.70 ± 1.524 97.77 ± 1.017</td></tr><tr><td>(%)</td><td>4</td><td>98.60</td><td>24.58</td><td colspan="2">24.50 ± 0.017 60.43 ± 4.209</td><td>86.97 ± 2.797</td><td colspan="2">77.90 ± 4.210 75.55 ± 3.891 85.59 ± 3.930 95.41 ± 1.345</td></tr><tr><td/><td>5 97.75</td><td>97.87</td><td>20.02</td><td colspan="2">19.98 ± 0.032 47.42 ± 1.447</td><td>75.50 ± 3.032</td><td colspan="2">81.49 ± 1.944 63.44 ± 5.252 87.72 ± 1.594 90.23 ± 1.139</td></tr><tr><td/><td>1</td><td>433.7</td><td>423.2</td><td/><td>422.3 ± 0.573</td><td>434.2 ± 1.068</td><td colspan="2">435.2 ± 15.69 432.6 ± 0.321 424.2 ± 2.511 433.8 ± 0.370</td></tr><tr><td>γbase,t</td><td>2 3</td><td>422.5 420.7</td><td>439.4 429.2</td><td/><td>426.6 ± 2.840 425.0 ± 0.339</td><td>434.4 ± 1.082 434.6 ± 0.785</td><td colspan="2">423.9 ± 0.517 432.5 ± 0.551 425.2 ± 1.402 433.5 ± 1.464 422.7 ± 0.690 432.9 ± 0.723 423.8 ± 1.148 433.1 ± 1.269</td></tr><tr><td>(nats) or</td><td>4</td><td>419.9</td><td>428.5</td><td/><td>425.4 ± 0.081</td><td>434.2 ± 1.209</td><td colspan="2">422.8 ± 0.367 433.0 ± 0.781 423.5 ± 0.937 433.0 ± 1.283</td></tr><tr><td colspan="2">(bits/dim) 5</td><td>418.4</td><td>432.9</td><td/><td>425.2 ± 0.244</td><td>435.1 ± 1.915</td><td colspan="2">422.7 ± 0.182 431.4 ± 0.666 423.5 ± 0.586 432.3 ± 0.189</td></tr><tr><td/><td>1</td><td>433.7</td><td>423.2</td><td/><td>422.3 ± 0.573</td><td>434.2 ± 1.068</td><td colspan="2">435.2 ± 15.69 432.6 ± 0.321 424.2 ± 2.511 433.8 ± 0.370</td></tr><tr><td>γnew,t</td><td>2 3</td><td>381.2 435.9</td><td>384.1 436.7</td><td/><td>381.3 ± 2.039 436.8 ± 0.188</td><td>390.4 ± 0.694 444.7 ± 0.545</td><td colspan="2">382.5 ± 1.355 389.4 ± 0.208 385.3 ± 12.56 389.4 ± 1.304 436.3 ± 0.639 442.7 ± 0.513 436.9 ± 0.688 442.4 ± 0.275</td></tr><tr><td>(nats) or</td><td>4</td><td>485.9</td><td>487.1</td><td/><td>486.5 ± 0.432</td><td>497.4 ± 0.740</td><td colspan="2">486.7 ± 0.385 494.4 ± 0.700 486.5 ± 0.701 494.8 ± 0.386</td></tr><tr><td colspan="2">(bits/dim) 5</td><td>421.3</td><td>425.2</td><td/><td>422.4 ± 0.784</td><td>431.9 ± 1.032</td><td colspan="2">423.9 ± 0.681 428.0 ± 0.851 422.9 ± 0.537 429.7 ± 1.223</td></tr><tr><td/><td>1</td><td>433.7</td><td>423.2</td><td/><td>422.3 ± 0.573</td><td>434.2 ± 1.068</td><td colspan="2">435.2 ± 15.69 432.6 ± 0.321 424.2 ± 2.511 433.8 ± 0.370</td></tr><tr><td>γall,t</td><td>2 3</td><td>401.9 412.1</td><td>411.8 418.9</td><td/><td>404.0 ± 2.407 414.4 ± 0.385</td><td>412.4 ± 0.871 423.3 ± 0.618</td><td colspan="2">403.2 ± 0.831 410.9 ± 0.351 403.5 ± 1.274 411.5 ± 1.406 413.6 ± 0.410 421.0 ± 1.026 413.8 ± 0.573 421.9 ± 0.661</td></tr><tr><td>(nats) or</td><td>4</td><td>430.3</td><td>438.4</td><td/><td>433.9 ± 0.374</td><td>441.6 ± 0.420</td><td colspan="2">432.4 ± 0.436 439.8 ± 0.833 432.6 ± 0.862 439.8 ± 0.718</td></tr><tr><td colspan="2">(bits/dim) 5 429.7</td><td>427.2</td><td>440.4</td><td/><td>432.7 ± 0.385</td><td>440.3 ± 1.297</td><td colspan="2">431.4 ± 0.255 436.9 ± 0.751 430.9 ± 0.541 437.7 ± 0.432</td></tr><tr><td/><td>1</td><td>11.65</td><td>11.20</td><td/><td>4.639 ± 0.107</td><td>4.361 ± 0.671</td><td colspan="2">11.78 ± 1.478 9.293 ± 0.943 11.16 ± 0.713 11.87 ± 1.504</td></tr><tr><td>KLall,t</td><td>2 3</td><td>11.78 13.40</td><td>13.61 17.09</td><td/><td>5.135 ± 0.127 5.427 ± 0.105</td><td>5.130 ± 0.636 5.399 ± 0.724</td><td colspan="2">15.13 ± 1.128 14.00 ± 0.748 14.06 ± 1.140 12.40 ± 0.719 18.18 ± 1.140 20.28 ± 0.774 13.61 ± 0.901 14.41 ± 0.461</td></tr><tr><td>(nats)</td><td>4</td><td>13.61</td><td>14.41</td><td/><td>5.243 ± 0.135</td><td>5.817 ± 1.038</td><td colspan="2">22.93 ± 1.134 24.91 ± 0.845 17.58 ± 1.102 16.00 ± 0.505</td></tr><tr><td/><td>5 17.89</td><td>15.15</td><td>14.52</td><td/><td>5.470 ± 0.055</td><td>6.031 ± 0.832</td><td colspan="2">22.96 ± 0.912 27.14 ± 1.139 18.52 ± 1.131 17.45 ± 0.835</td></tr><tr><td colspan="9">2.851 ± 0.0026 and 2.852 ± 0.0047 for FashionMNIST, 4.425 ± 0.0010 and 4.451 ± 0.0198 for Au-</td></tr><tr><td colspan="9">dioMNIST. For cross-dataset experiments starting with FashionMNIST first, the corresponding loss</td></tr><tr><td colspan="9">values in bits per dimension for PixCDVAE are 2.260 ± 0.0078 and 2.238 ± 0.0021 for PixOCD-</td></tr><tr><td colspan="9">VAE. In the reverse direction the values are 2.232 ± 0.0177 and 2.218 ± 0.0014 respectively. In</td></tr><tr><td colspan="9">the dual model scenario, the separate PixVAE achieves final losses on all tasks of 1.040 ± 0.0103</td></tr><tr><td colspan="9">for MNIST, 2.242 ± 0.0124 for FashionMNIST and 4.406 ± 0.0024 for AudioMNIST. Loss values</td></tr><tr><td colspan="9">for the cross dataset setting are 2.253 ± 0.0047 when FashionMNIST gets trained first and 2.279 ±</td></tr><tr><td colspan="7">0.0104 for the reverse direction starting with AudioMNIST.</td><td/></tr></table>

Peer review (iclr2020):
This paper tackles the problem of catastrophic forgetting when data is organized in a large number of batches of data (tasks) that are sequentially made available. To avoid catastrophic forgetting, the authors learn a VAE that generates the training data (both inputs and labels) and retrain it using samples from the new task combined with samples generated from the VAE trained in the previous tasks (generative replay). In this way, there*s no need to store all past data and even the first learned batch keeps being refreshed and should not be forgotten. I like that this paper uses a single global probabilistic model instead of separate discriminative and generative ones. Unfortunately, there are several things that left me unconvinced about this paper: 1) Presentation of the paper - Variables x, y, z are introduced and talked about without explanation. The graphical model or factorization assumptions are not even mentioned until after the loss has been defined. A normal flow is to first describe the model and what the involved variables mean, and then talk about what the loss for learning it should be, not the other way around. - Text contradicting the equation: *In order to balance the individual loss terms, we normalize according to dimensions and weight the KL divergence with a constant of 0.1*. But equation (2) shows a loss with no weighting. I*m assuming the text is correct, but then a beta should be added to the equation in front of the KL divergence. - Tables and figures are inconveniently far from where they are referenced in the text. 2) Theoretical inconsistencies Although the system might work overall, two things seem to be technically incorrect: - The decoder and classifier are expected to approximate the distribution of training data according to the authors (for valid generative replay). This is not true in a beta-VAE. The weighting of the KL that the authors introduce is going to bias the learned generator towards the high probability regions. This is not a sound mechanism to achieve an as-faithful-as-possible (limited by the expressiveness of the encoder-decoder architectures) approximation to the training data. - A Weibull distribution is used to model the same data, again, in a different way. I.e., there are two different probabilistic models modeling the same data in inconsistent ways and one or the other is used depending on the part of the system. (As an example, q(z) could be arbitrarily multimodal as far as the encoder is concerned, but the Weibull seems to force one mode per class. But regardless of this, both models are inconsistent.) - Similarly, the proposed rejection sampling scheme of OCDVAE is not consistent with the theory of VAEs and it*s a post-hoc tweak that is not theoretically expected to provide a pdf of data with lower KL divergence to the true data pdf. 3) Experiments Finally, the experimental results do not look very compelling, it seems to be overall worse than the baselines in the two image datasets and slightly better in the audio dataset, so it*s unclear that this approach is superior.