Continual Density Ratio Estimation (CDRE): A new method for evaluating generative models in continual learning

1 INTRODUCTION . Density Ratio Estimation ( DRE ) ( Sugiyama et al. , 2012 ) is a methodology for estimating the density ratio between two probability distributions and it can be applied to two-sample tests in which only samples of the two distributions are available . It has a wide range of applications in machine learning , such as distribution comparison , mutual information estimation , outliers detection , etc .. However , under certain restrictive conditions in the real world , i.e . one distribution is changing over time and the samples of the other distribution are not available after some time point , the existing methods of DRE are not applicable any more which leads to the unavailability of those applications of DRE as well . In order to enable DRE under such restrictive conditions , we propose a novel method Continual Density Ratio Estimation ( CDRE ) which can estimate the density ratio between a target distribution and a model distribution without storing any samples from target distribution while the model distribution is changing over time . There can be diverse reasons in the real world of limiting access to the raw data after a model is trained on it . For example , researchers of a hospital may have trained a model for one type of disease and the raw data of patients can not be shared with other institutions , if they want to collaborate with another institution to enable the model capable of a similar type of disease as well , the model can only be incrementally trained on new data without sharing the previous data . Besides the privacy issue , limited cost budget can be another cause of such a problem , such as the data storage cost is quite high , or the data is not available for free after its copyright has expired . The restrictive conditions described above perfectly match the problem setting of continual learning in which a single model evolves over time by learning new tasks sequentially with none or very limited access to the data of old tasks and yet is able to perform on all seen tasks . The most crucial obstacle of continual learning is that a model tends to forget previous tasks while learning a new task , which is a phenomenon called catastrophic forgetting ( Kirkpatrick et al. , 2017 ) . Generative models play an important role in continual learning because they can be employed to help other models with keeping memories by generating samples of previous tasks ( Shin et al. , 2017 ; Wu et al. , 2018 ) , a method known as “ generative replay ” . A simplified scenario for generative models in continual learning is depicted in Fig . 1 , where the goal is to learn a generative model for one category ( digit ) per task , but still be able to generate samples of all previous categories . The training dataset of task i consists of real samples of category i and samples of task 1 · · · i−1 generated by the previous model . Despite the importance of generative models in continual learning , there is no effective way to evaluate them under the restrictions of continual learning . The related works ( Wu et al. , 2018 ; Shin et al. , 2017 ; Lesort et al. , 2018 ) either evaluate them by classification performance , or just displaying some model samples to let readers judge them visually , or applying usual measures of generative models in static learning by comparing with real samples from the target distribution . However , the performance of generative models is not always tied to the performance of a classification task , it may be decided by the fidelity of model samples as in many applications of generative models ( Brock et al. , 2019 ; Karras et al. , 2019 ; van den Oord et al. , 2017 ) . Moreover , it is questionable how many real samples we can obtain to evaluate generative models in continual learning . A small sample size ( less than a few hundreds ) as commonly used in episodic memories ( Lopez-Paz et al. , 2017 ; Chaudhry et al. , 2019 ) or coresets ( Nguyen et al. , 2018 ) can not guarantee the accuracy of measures for evaluating generative models . The principal idea of evaluating generative models is estimating the difference between the target distribution and the model distribution , for example , Fréchet Inception Distance ( FID ) ( Heusel et al. , 2017 ) and Kernel Inception Distance ( KID ) ( Bińkowski et al. , 2018 ) estimate Wasserstein-2 distance and Maximum Mean Discrepancy ( MMD ) , respectively . Likewise , f -divergences is a well-studied family of divergences that are commonly used to measure differences between two distributions , more importantly , one can estimate f -divergences by estimating density ratios . We show that our new method CDRE can effectively estimate f -divergences under the setting of continual learning . Consequently , we can evaluate generative models in continual learning using f -divergences estimated by CDRE , which is supported by experiment results in comparison with commonly used FID , KID . In the absence of prior work , we also provide empirical analysis of differences between FID , KID and f -divergences in terms of evaluating generative models . 2 PRELIMINARIES . In this section we introduce the basic formulation of Density Ratio Estimation ( DRE ) as it is the foundation of CDRE , and we discuss estimating f -divergences by DRE as well . 2.1 DENSITY RATIO ESTIMATION ( DRE ) .. There are two principal methods for DRE introduced in Sugiyama et al . ( 2012 ) : Kullback-Leibler Importance Estimation Procedure ( KLIEP ) and Least-Squares Importance Fitting ( LSIF ) , which are based on Kullback-Leibler ( KL ) divergence and Pearson ( χ2 ) divergence , respectively . Here we review the formulation of KLIEP , which will form a building block of our method . Let r∗ ( x ) = p ( x ) q ( x ) be the ( unknown ) true density ratio , then p ( x ) can be estimated by p̃ ( x ) = r ( x ) q ( x ) , where r∗ ( x ) is modeled by r ( x ) . Hence , we can optimize r ( x ) by minimizing the KL-divergence between p ( x ) and p̃ ( x ) with respect to r : DKL ( p ( x ) ||p̃ ( x ) ) = ∫ p ( x ) log p ( x ) p̃ ( x ) dx = ∫ p ( x ) logr∗ ( x ) dx− ∫ p ( x ) logr ( x ) dx ( 1 ) where r ( x ) satisfies r ( x ) > 0 and ∫ r ( x ) q ( x ) dx = ∫ p̃ ( x ) dx = 1 . The first term of the right side in Eq . ( 1 ) is a constant w.r.t . r ( x ) , then the objective of optimizing r ( x ) is as below : Jr = max r 1 N N ∑ i=1 logr ( xi ) , xi ∼ p ( x ) , s.t . 1 M M ∑ j=1 r ( x j ) = 1 , x j ∼ q ( x ) , and r ( x ) ≥ 0 , ∀x . ( 2 ) One convenient way of parameterizing r ( x ) is by using a log-linear model with normalization , which then automatically satisfies the constraints in Eq . ( 2 ) : r ( x ; β ) = exp ( ψβ ( x ) ) 1 M ∑ M j=1 exp ( ψβ ( x j ) ) , x j ∼ q ( x ) , ψβ : RD→ R , ( 3 ) where ψβ can be any deterministic function and we use a neural network as ψβ in our implementations , β representing parameters of the neural network . 2.2 ESTIMATING f -DIVERGENCES BY DENSITY RATIO ESTIMATION The definition of f -divergences is given in Eq . ( 4 ) , where f is a convex function and satisfies f ( 1 ) = 0 . Given a specified f , we can estimate the empirical divergence using density ratios without knowing the density functions . For example , taking f ( r ) = r logr recovers the KL divergence . D f ( p ( x ) ||q ( x ) ) = Eq ( x ) [ f ( p ( x ) q ( x ) ) ] ≈ 1 N N ∑ i=1 f ( r ( xi ) ) , xi ∼ q ( x ) , r ( xi ) = p ( xi ) q ( xi ) ( 4 ) Estimating f -divergences by density ratios has been well studied ( Kanamori et al. , 2011 ; Nguyen et al. , 2010 ) . Although it can also be estimated by the density functions of distributions , that is beyond the scope of this paper and density functions are often not available . One concern of estimating f -divergences by DRE is from high dimensional data which might require large sample size to achieve convergence ( Rubenstein et al. , 2019 ) . One way to mitigate the problem is to perform dimensionality reduction in combination with DRE , for which several methods have been introduced in Sugiyama et al . ( 2012 ) . A fundamental assumption of these methods is that the difference between two distributions can be confined in a subspace , which means p ( z ) /q ( z ) = p ( x ) /q ( x ) where z is a lower-dimensional representation of x . This is aiming for exact density ratio estimation , in another word , if the difference is large in the original space , it is still large in the subspace . Besides the expensive cost of searching such a subspace , significant differences between distributions still cause unrealistic convergence rate of DRE methods . For example , the convergence analysis of KLIEP is described in Sugiyama et al . ( 2008 ) , as in short , the convergence rate depends on the bounds of estimated ratios , as well as the bounds of the entropy of estimated ratios . Hence , it is desirable to scale down the difference between high dimensional distributions and make ratios well bounded for a easier convergence by dimensionality reduction , whilst ensuring that the estimated divergence is still a faithful reflection of the true divergence . On the other hand , DRE can work with high-dimensional data when the two distributions are close to each other . We demonstrate this by experiments with high-fidelity model samples in a high-dimensional space ( Appx . B.2 ) . According to the information monotonicity of f -divergences ( Amari , 2009 ) , we can estimate a lower bound of the f -divergence on an arbitrary surrogate feature space : D f ( q ( z ) ||p ( z ) ) ≤D f ( q ( x ) ||p ( x ) ) , where p ( z ) = ∫ p ( x ) p ( z|x ) dx , q ( z ) = ∫ q ( x ) p ( z|x ) dx and p ( z|x ) is an arbitrary transition probability . Using a surrogate feature space is a widely applied technique in measurements of generative models . For instance , the inception feature defined for Inception Score ( IS ) ( Salimans et al. , 2016 ) can be viewed as from a surrogate feature space and it is widely applied in other measures of generative models ( such as FID , KID , Precision and Recall for Distributions ( PRD ) ) as well . Analogously , we introduce Continual Variational Auto Encoder ( CVAE ) as a solution of generating lower dimensional features for CDRE in the later section .

Summary: In this paper, authors propose a continual learning for density-ratio estimation. The formulation of the CDRE Eq.(5)is quite intuitive, and it makes sense. Then, a log-linear model is employed for a density-ratio model, and it is estimated by using a density-ratio estimation algorithm. Through experiments, the authors show that the proposed algorithm outperforms existing methods.