Discriminative Variational Autoencoder for Continual Learning with Generative Replay

1 INTRODUCTION . Even though deep learning models such as Szegedy et al . ( 2015 ) ; He et al . ( 2016 ) ; Xie et al . ( 2017 ) have been showing remarkable performance on many image recognition tasks exceeding human accuracy , they suffer from the catastrophic forgetting problem . The catastrophic forgetting problem occurs when we want to learn several tasks sequentially without any continual learning ( CL ) technique ( Ratcliff ( 1990 ) ; French ( 1999 ) ; Goodfellow et al . ( 2013 ) ) . Theoretically , the catastrophic forgetting problem can be completely avoided if we store all of the past data , but it is sometimes difficult to store all of the previous data due to the limitation on memory capacity . To handle the forgetting problem in deep learning , some studies suggested weight regularization ( WR ) techniques such as EWC ( Kirkpatrick et al . ( 2017 ) ) , SI ( Zenke et al . ( 2017 ) ) , and IMM ( Lee et al . ( 2017 ) ) . In these approaches , each of them has a regularization term to constrain the update of some parameters that are important for previous tasks . Other researches such as GEM ( Lopez-Paz et al . ( 2017 ) ) , VCL ( Nguyen et al . ( 2018 ) ) introduced an additional memory called coreset which consists of the small number of real images of past tasks to further improve WR algorithm . We will call this approaches as experience replay ( ER ) -based methods in this paper . Specifically , ER-based methods are useful when we can access previous task data though the amount of past data is limited . Recently , generative replay ( GR ) -based approaches such as DGR ( Shin et al . ( 2017 ) ) and VGR ( Farquhar & Gal ( 2018 ) ) are also proposed to replay previous tasks with a generative model instead of storing the small number of images into coreset . They showed that if the generative model successfully approximates data distribution , this approach can be a strong way to alleviate the catastrophic forgetting problem . In this paper , we propose a new efficient and effective model called discriminative variational autoencoder ( DiVA ) ( Fig . 1 ) to address the GR-based CL problem . GR-based approaches can address a continual learning setting where a stream of real samples is seen only once , which can not be considered in ER-based CL algorithms . However , GR-based approaches have two shortcomings . First , they need an additional module , a generative model , in addition to a classifier . Second , distribution discrepancy between real and generated samples makes GR-based methods hard to address complex natural images such as CIFAR ( Krizhevsky & Hinton ( 2009 ) ) or ImageNet ( Deng et al . ( 2009 ) ) . In this point of views , we try to handle the shortcomings . Firstly , distributions of latent variables of DiVA are clustered class-wise discriminatively on the latent space by introducing mutual information maximization between latent variable z and class condition c. Thus , DiVA is directly applicable to both classification and class conditional sample generation with one integrated model . Also , we apply a simple but effective domain translation ( DT ) trick to mitigate the distribution discrepancy and thus DiVA can achieve better classification accuracy on past tasks . To the best of our knowledge , we firstly introduce the DT trick to GR-based CL problem . In summary , we describe three main contributions of our research as follows : • We suggest a new type of conditional VAE in terms of mutual information maximization , which can be expressed as integrated classifier and learnable prior network . It makes discriminative class-wise distributions for each class on latent space . • We propose a simple but effective DT trick , which significantly improve CL performance of GR-based approaches on a natural image dataset such as CIFAR10 . • By combining DiVA and DT , we achieved state-of-the-art accuracy on the class incremental learning settings . 2 RELATED WORKS . VAE with GMM prior VAE ( Kingma & Welling ( 2013 ) ) has been widely studied due to its concrete theoretical background and interesting latent representations . Similar to our mixture-like prior distribution , some researchers proposed learnable Gaussian mixture prior ( Dilokthanakul et al . ( 2016 ) ; Nalisnick et al . ( 2016 ) ) or hierarchical prior distributions ( Tomczak & Welling ( 2018 ) ) because the unit Gaussian assumption as prior in VAE is a too strong constraint . They mainly address an unsupervised learning task while we consider explicit supervision tasks . Wang et al . ( 2017 ) also proposed GMM VAE with class-specific and randomly fixed priors for a supervised task . In this case , the random initialization can map similar patterns to far regions on latent space , which could not be suitable for the main desiderata , smoothness , of representation learning ( Bengio et al . ( 2013 ) ) . On the other hand , since we consider learnable priors , DiVA satisfies the smoothness condition ( Fig . 2 ( d ) ) . Conditional VAE There are several ways to implement conditional VAE ( CVAE ) . The easiest and intuitive way is to concatenate one-hot vectored class conditions onto latent variables which are outputs of an encoder network ( Doersch ( 2016 ) ) . This method sets prior distributions for latent variables of all classes as a unit Gaussian . Since the CVAE always needs data-label pairs for concatenation , other researches propose to infer the class label with an independent heavy classifier ( Kingma et al . ( 2014 ) ; Sohn et al . ( 2015 ) ) . As a result , they can make their conditional generative model work with unlabeled data . However , concatenation makes feature representation unnecessarily sparse and requires additional weight parameters for feeding class conditions . In contrast , we do not need the heavy classifier and redundant weights as we model class-wise discriminative latent representations . Generative replay Recently , studies which use GR have been actively proposed to mitigate catastrophic forgetting in the field of CL . This approach generates previous task data when a new task data is coming , then optimizes them concurrently . In the DGR and VGR , they use a GAN-based generator to generate previous task data unconditionally . Since DGR and VGR consider a generator and a classifier as separate modules , they alternately optimize them . Also , other researchers propose GR-based algorithms using VAE such as RtF ( van de Ven & Tolias ( 2018 ) ) , CDVAE ( Mundt et al . ( 2019 ) ) , and DGDMN ( Kamra et al . ( 2017 ) ) . RtF and CDVAE are closely related to our work in that they consider classifier integrated VAE and address CL problem . However , they both assume the prior distribution as unit Gaussian , which contradicts discriminative loss . Also , recently , Hu et al . ( 2019 ) ( PGMA ) proposes a parameter generation and model adaptation trick to mitigate the catastrophic forgetting problem . 3 PRELIMINARY . Variational Autoencoder Considering we have observations X = { x1 , x2 , ... , xN } where xi ∈ Rd , the goal of the VAE is to maximize marginal log likelihoods of each observation log p ( x1 , x2 , ... , xN ) = ∑N i=1 log p ( xi ) with respect to both variational parameters and generative parameters . However , this includes an intractable term when we use a complex model . Because the intractable term is always positive , we can optimize the variational lower bound instead of the marginal log likelihood as follows : log p ( xi ) =DKL [ qθ ( z|xi ) ||pθ′ ( z|xi ) ] + L ( θ , θ′ ; xi ) ≥L ( θ , θ′ ; xi ) = Eqθ ( z|xi ) [ log pθ′ ( xi|z ) ] −DKL [ qθ ( z|xi ) ||p ( z ) ] , where qθ ( z|xi ) is the variational posterior ( the encoder ) and pθ′ ( xi|z ) is the decoder . In general , both the encoder and the decoder can be modeled with neural networks . The L ( θ , θ′ ) is called as the evidence lower bound ( ELBO ) . Mutual Information In information theory , mutual information of two random variables , X and Y , is a measure of the mutual dependence between the two variables . The mutual information can be decomposed into marginal entropy and conditional entropy as follows : I ( X ; Y ) = H ( X ) −H ( X|Y ) = H ( Y ) −H ( Y |X ) . The meaning of this definition is intuitive : if we observe Y then how much uncertainty of X changes . If X and Y are completely independent , i.e . H ( X|Y ) = H ( X ) , then the I ( X ; Y ) = 0 . On the contrary , if X is determined by observed Y , then the I ( X ; Y ) is larger than zero . 4 DISCRIMINATIVE VARIATIONAL AUTOENCODER . In Fig 2 , unconditional generative models such as standard VAE and CDVAE generate some ambiguous samples . Thus , we should consider a conditional generative model for sampling class-wise explicit images to handle the incremental learning problem using a GR-based approach ( further details are discussed at Sec.6.1. ) . Here , we consider a classifier integrated VAE . If latent variables for input images are located class-wise discriminatively on the latent space , we can conduct both class prediction and conditional generation . Fortunately , we can directly model the relationship between latent variables z and class condition c by maximizing mutual information between them . The optimization of DiVA is as follows : max θ , θ′ , φ , φ′ Eqθ ( z|x ) [ log pθ′ ( x|z ) ] −DKL [ qθ ( z|x ) ||p ( z ) ] + λI ( c ; z ) ( 1 ) This can be equivalently rewritten as : max θ , θ′ , φ , φ′ Eqθ ( z|x ) [ log pθ′ ( x|z ) ] −DKL [ qθ ( z|x ) ||q̂φ ( z|c ) ] + λEqθ ( z|x ) [ log p̂φ′ ( c|z ) ] ( 2 ) where , λ is a weighting parameter of the mutual information loss and Eqθ ( z|x ) [ log p̂φ′ ( c|z ) ] is a cross-entropy for multivariate Bernoullis or L2 loss for continuous-valued conditions . Also , θ , θ′ , φ , and φ′ mean network parameters of the encoder , the decoder , the prior network , and the classifier , respectively . When the encoder and decoder networks are sufficiently complex , it is enough to implement each the prior and classification network as one fully-connected layer . Detailed derivations are described at Sec.4.1 . 4.1 MUTUAL INFORMATION MAXIMIZATION . In this section , we describe how maximizing I ( z ; c ) with ELBO makes our DiVA . Vincent et al . ( 2010 ) ; Chen et al . ( 2016 ) showed that minimization of reconstruction error is related to maximization of mutual information between two variables . Similarly , we maximize mutual information between class condition c and latent variable z by minimizing a reconstruction loss . Here , we will start from I ( c ; z ) = H ( z ) −H ( z|c ) . Then , we will optimize1 : max I ( c ; z ) = maxH ( z ) −H ( z|c ) = maxH ( z ) + Eq ( c , z ) [ log q̂ ( z|c ) ] . ( 3 ) Now , we derive the conditional entropy H ( z|c ) as follows : H ( z|c ) = − ∑ c , z q ( c , z ) log q̂ ( z|c ) , = − ∑ c q ( c ) ∑ z q̂ ( z|c ) log q̂ ( z|c ) , ( 4 ) If we introduce p̂ ( c|z ) , then Eq . ( 4 ) can be rewritten as , H ( z|c ) = − ∑ c q ( c ) ∑ z q̂ ( z|c ) ( log p̂ ( c|z ) + log q̂ ( z|c ) p̂ ( c|z ) ) = − ∑ c q ( c ) ( Eq̂ ( z|c ) [ log p̂ ( c|z ) ] +DKL [ q̂ ( z|c ) || p̂ ( c|z ) ] ) ( 5 ) Since DKL [ q̂ ( z|c ) || p̂ ( c|z ) ] is always positive and q ( c ) is a constant in our dataset , we can get a lower bound of the mutual information as follows : I ( c ; z ) ≥ H ( z ) + Eq̂ ( z|c ) [ log p̂ ( c|z ) ] ( 6 ) where , Eq̂ ( z|c ) [ log p̂ ( c|z ) ] can be viewed as class reconstruction in autoencoding manner ; minimizing L2 or cross-entropy . In CDVAE , the authors directly optimize Eq ( z|x ) [ log p̂ ( c|z ) ] to impose class information on z so that a classifier can be integrated with VAE as follows : max θ , θ′ , φ′ Eqθ ( z|x ) [ log pθ′ ( x|z ) ] −DKL [ qθ ( z|x ) ||p ( z ) ] + λEqθ ( z|x ) [ logp̂φ′ ( c|z ) ] ( 7 ) 1For continuous variables , we can simply change the ∑ to ∫ . Algorithm 1 Train DiVA with domain translation procedure Train ( DiVA , CyGAN , dt ) procedure Train_CyGAN ( CyGAN , DiVA , Xi ) Input : T = { T_i } Ni=1 , where T_i = { Xi.Yi } Domain_A← Xi // training set X , real images // Learning all tasks for i = 0 , ... , until given epoch do : for all i = 0 , ... , N do : // for all tasks Domain_B← sample from DiVA // ∼ Taski // Training DiVA optimize CyGAN ( Domain_A↔ Domain_B ) for all ( x , y ) inXi , Yi , until convergence do : end for sampling ŷ // randInt for past task labels end procedure x̂← sampling from DiVAprev ( ŷ ) x̂← CyGAN : s2r ( x̂ ) if dt == s2r else x̂ procedure Test ( DiVA , CyGAN , i , dt ) minimize LDiVA ( [ x , x̂ ] , [ y , ŷ ] ) using Eq . 2 for j = 0 , ... , i do : //until current task end for for all x in Xj do : Test ( DiVA , CyGAN , i , dt ) if j 6= i : // for past tasks Train_CyGAN ( CyGAN , DiVA , Xi ) x← CyGAN : r2s ( x ) if dt == r2s else x DiVAprev ← DiVA prediction← DiVA ( x ) // classification end for end for return DiVA , CyGAN end for end procedure end procedure However , since they assume unit Gaussian p ( z ) as a prior for the variational posterior q ( z|x ) , relation between q̂ ( z|c ) and q ( z|x ) is not considered . Thus , latent space has low density regions because z becomes discriminative within unit Gaussian , which causes ambiguous sample generation ( Fig . 2 ( c ) ) . In contrast , we explicitly consider the relation between q̂ ( z|c ) and q ( z|x ) as follows . Firstly , since c corresponds to a class among a finite label set , we can simply assume q̂ ( z|c ) as a class-conditional Gaussian distribution , i.e . q̂ ∼ N ( z|µc , σ2c ) . Then , we can replace the unit Gaussian prior in Eq . ( 1 ) with the class-conditional Gaussian q̂ ( z|c ) , which corresponds to the second term in Eq ( 2 ) . Now , since we minimize DKL [ q ( z|x ) ||q̂ ( z|c ) ] in our loss , we can optimize Eq ( z|x ) [ log p̂ ( c|z ) ] instead of Eq̂ ( z|c ) [ logp̂ ( c|z ) ] . As a result , latent variable z of our DiVA holds information of not only input x but also class c without low density regions : each class condition has its own one mode compact Gaussian distribution . Furthermore , H ( z ) in Eq . ( 6 ) can be indirectly maximized to hold information of x by maximizing the ELBO in Eq . ( 2 ) . To summarize , our modeling for the probability of the latent variable given its class has three virtues : • Class is well-separable in the latent space . • The distribution of the latent variable given its class is simply modeled . • The latent variable has enough information to reconstruct the image . The first two virtues are related to the class condition ; classification network and prior network , respectively . The last virtue is achieved by the standard VAE .

Summary: In this paper, the authors focus on alleviating the catastrophic forgetting problem in continual learning.  The authors propose a discriminative variational autoencoder (DiVA) to solve this problem under the generative replay framework. DiVA modifies the objective function of VAE by introducing an additional term that maximizes the mutual information between the latent variables and the class labels.