Continual Density Ratio Estimation (CDRE): A new method for evaluating generative models in continual learning
Authors: Yu Chen
Song Liu
Tom Diethe
Peter Flach
Venue:  (2020)
Keywords: density ratio estimation
continual learning
evaluation
generative model
f divergence

Abstract: We propose a new method Continual Density Ratio Estimation (CDRE), which can estimate density ratios between a target distribution of real samples and a distribution of samples generated by a model while the model is changing over time and the data of the target distribution is not available after a certain time point. This method perfectly fits the setting of continual learning, in which one model is supposed to learn different tasks sequentially and the most crucial restriction is that model has none or very limited access to the data of all learned tasks. Through CDRE, we can evaluate generative models in continual learning using f-divergences. To the best of our knowledge, there is no existing method that can evaluate generative models under the setting of continual learning without storing real samples from the target distribution.

Under review as a conference paper at ICLR 2020
CONTINUAL DENSITY RATIO ESTIMATION (CDRE): A NEW METHOD FOR EVALUATING GENERATIVE MOD- ELS IN CONTINUAL LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We propose a new method Continual Density Ratio Estimation (CDRE), which can estimate density ratios between target distribution of real samples and a distribution of samples generated by a model while the model is changing over time and the data of the target distribution not available after certain time point. This method perfectly fits the setting of continual learning, in which one model is supposed learn different tasks sequentially and the most crucial restriction is that model has none or very limited access to the data of all learned tasks. Through CDRE, we can evaluate generative models in continual learning using f-divergences. To the best of our knowledge, there is no existing method that can evaluate generative models under the setting of continual learning without storing real samples from the target distribution.
INTRODUCTION
Density Ratio Estimation (DRE) (Sugiyama et al., 2012) is a methodology for estimating the den- sity ratio between two probability distributions and it can be applied to two-sample tests in which only samples of the two distributions are available. It has a wide range of applications in machine learning such as distribution comparison, mutual information estimation, outliers detection, etc.. However, under certain restrictive conditions in the real world, i.e. one distribution is changing over time and the samples of the other distribution are not available after some time point, the existing methods of DRE are not applicable any more which leads to the unavailability of those applica- tions of DRE as well. order to enable DRE under such restrictive conditions, we propose a novel method Continual Density Ratio Estimation (CDRE) which can estimate the density ratio between a target distribution and a model distribution without storing any samples from target distribution while the model distribution is changing over time.
There can be diverse reasons in the real world of limiting access to the raw data after a model is trained on it. For example, researchers of hospital may have trained model for one type of disease and the raw data of patients cannot be shared with other institutions, if they want to collaborate with another institution enable the model capable of a similar type of discase as well, the model can only be incrementally trained on new data without sharing the previous data. Besides the privacy issue, limited cost budget can be another cause of such a problem, such as the data storage cost is quite high, or the data is not available for free after its copyright has expired.
The restrictive conditions described above perfectly match the problem setting of continual learning in which a single model evolves over time by learning new tasks sequentially with none or very limited access to the data of old tasks and yet is able to perform on all seen tasks. The most crucial obstacle of continual learning is that model tends to forget previous tasks while learning a new task, which is a phenomenon called catastrophic forrgtttppi (Kirkpatrick et al., 2017). Generative models play important role in continual learning because they can be employed to help other models with keeping memories by generating samples of previous tasks (Shin et al., 2017; Wu et al., 2018), method known "generative replay". A simplified scenario for generative models in continual learning is depicted in Fig. 1, where the goal is to learn generative model for one category (digit) per task, but still be able generate samples of all previous categories. The training dataset of task i

---

Under review as a conference paper at ICLR 2020
Gencration lask
0
Go
Figure I: Demonstration of generative models in continual learning, task i the training set consists of samples category and samples generated by the model at the previous task, and the task is to generate samples from all previously
categories (figure reproduced from Lesort et al. (2018)).
consists of real samples of category i and samples of task ....i-
- generated by the previous model.
Despite the importance of generative models in continual learning, there is no effective way to evaluate them under the restrictions continual learning. The related works (Wu al., 2018; Shin et al., 2017; Lesort et al., 201 either evaluate them by classification performance, or just displaying some model samples to let readers judge them visually, or applying usual measures of generative models in static learning by comparing with real samples from the target distribution. However, the performance of generative models is not always tied to the performance of a classification task, it may be decided by the fidelity of model samples in many applications of generative models (Brock et al., 2019; Karras et al., 2019; van den Oord et al., 2017). Moreover, it is questionable how many real samples we can obtain evaluate generative models in continual learning. A small sample size (less than few hundreds) as commonly used in episodic memories (Lopez-Paz et al., 2017: Chaudhry et al., 2019) or coresets (Nguyen al., 2018) cannot guarantee the accuracy of measures for evaluating generative models.
The principal idea of evaluating generative models is estimating the difference between the target distribution and the model distribution, for example, Fréchet Inception Distance (FID) (Heusel et al., 2017) and Kernel Inception Distance (KID) (Binkowski et al., 2018) estimate Wasserstein-2 distance and Maximum Mean Discrepancy (MMD), respectively. Likewise, f-divergences is a well-studied family of divergences that are commonly used to measure differences between two distributions, more importantly, one can estimate f-divergences by estimating density ratios.
We show that our new method CDRE can effectively estimate f-divergences under the setting of continual learning. Consequently, f-divergences estimated by CDRE, which is supported by experiment results in comparison with commonly used FID, KID. In the absence of prior work, also provide empirical analysis of differences between FID, KID and f-divergences in terms of evaluating generative models.
evaluate generative models in continual learning using
PRELIMINARIES
In this section we introduce the basic formulation Density Ratio Estimation (DRE) as it is the foundation of CDRE, and we discuss estimating f-divergences by DRE as well.
2.1 DENSITY RATIO ESTIMATION (DRE).
There are two principal methods for DRE introduced in Sugiyama et al. (2012): Kullback-eeibe Importance Estimation Procedure (KLIEP) and Least-Squares Importance Fitting (LSIF)), which are
based on Kullback-Leibler (KL) divergence and Pearson (x²) divergence, respectively. Here we
review the formulation of KLIEP, which will form a building block of our method. Let r"()) p(x)
q(x)
be the (unknown) true density ratio, then p(x) can be estimated by p(x) r(x)q(x), where r" (x) is
modeled by r(x). Hence, we can optimize r(x) by minimizing the KL-divergence between p(x) and p(x) with respect to
p(x) dx / p(x))ogr'(x)dd p(x)1d)) p(x)
DEL (p(x)))(())) = p(x)log
(1)
2

---

Under review as a conference paper at ICLR 2020
where r(x) satisfies r(x) > 0 and [((x)g(x)dxx f 1. The first term of the right side in Eq. (1) is a constant w.r.t. r(x), then the objective of optimizing r(x) is as below:
N
1 M
= max E logr(x)), X p(x),
M E -(xx) xj xj ((), and r(x) 2 0, Vx.
(2)
One convenient way of parameterizing r(x) is by using log-linear model with normalization, which then automatically satisfies the constraints in Eq. (2):
ex)(Vx(x))
(x:;)
xj qx), VB R + IR,
where WB can be any deterministic function and we use neural network Vp in our implementa- tions, B representing parameters of the neural network.
2.2 ESTIMATING f f-DIVERGENCES BY DENSITY RATIO ESTIMATION
The definition of f-divergences is given in Eq. (4), where ʃ is a convex function and satisfies ((1) =0 O. Given specified f. estimate the empirical divergence using density ratios without knowing the density functions. For example, taking f(r) = rlogr recovers the KI. divergence.
[1( p(x)
p(xi)
2)(p(x)ll((s)) BBgt))
f(((x)),
x (x),;
r(x)
(4)
((x,)
Estimating f f-divergences by density ratios has been well studied (Kanamori et al., 2011; Nguyen et al., 2010). Although it can also be estimated by the density functions of distributions, that is beyond the scope of this paper and density functions are often not available.
One concern of estimating f-divergences by DRE is from high dimensional data which might require large sample size to achieve convergence (Rubenstein et al., 2019). One way to mitigate the problem to perform dimensionality reduction in combination with DRE, for which several methods have been introduced in Sugiyama et al. (2012). A fundamental assumption of these methods is that the difference between two distributions can be confined in subspace, which means p(z)/q(z) (xx) where is lower-dimensional representation of x. This is aiming for exact density ratio estimation, in another word, if the difference is large in the original space, it is still large in the subspace. Besides the expensive cost of searching such subspace, significant differences between distributions still cause unrealistic convergence rate DRE methods. For example, the convergence analysis of KLIEP is described in Sugiyama et al. (2008), as in short, the convergence rate depends on the bounds of estimated ratios, as well as the bounds of the entropy estimated ratios. Hence, desirable to scale down the difference between high dimensional distributions and make ratios well bounded for a casier convergence by dimensionality reduction, whilst ensuring that the estimated divergence is still faithful reflection of the true divergence. On the other hand, DRE can work with high-dimensions data when the two distributions are close to each other. We demonstrate this by experiments with high-fidelity model samples in nigh-dimensiooal space (Appx. B.2).
According to the information monotonicity of fdiivergences (Amari, 2009), we can estimate a lower bound of the f-divergence an arbitrary surrogate feature space: (9((2)) [p(z)) (p((()))l(( where p(z) = p(x) (zz) =I (((x))((()dd and p(z[x) is an arbitrary transition probability. Using surrogate feature space is widely applied technique in measurements of generative models. For instance, the inception feature defined for Inception Score (IS) (Salimans et al., 2016) can be viewed as from surrogate feature space and it is widely applied other measures of generative models (such as FID, KID, Precision and Recall for Distributions (PRD)) as well. Analogously, we introduce Continual Variational Auto Encoder (CVAE) as a solution of generating lower dimensional features for CDRE in the later section.
3 CONTINUAL DENSITY RATIO ESTIMATION
In this section We first introduce the general formulation of CDRE by only considering one target distribution through out the whole time. After that. we introduce the formulation under the full set- ting of continual learning in which case the model learns multiple target distributions in a sequential manner.
3

---

Under review as a conference paper at ICLR 2020
Let X denote the data from the target distribution and p(x) denote its density function. Note that p(x) will not change over time but will be unavailable when r > 1, where represents the index of current time steps. G, denotes a generative model at time 1, X, denotes samples generated by G, and (x) denotes the density function of &, which are all changing while r changing. The goal of CDRE is to estimate the density ratio p(x)/q (x), Vt, which can be decomposed as follows:
(x) = p(x) -x-((x) p(x)
= rs, (x),,-(xx).
(5)
(x) (x) q
where r., q(x))=q-1s))) (x) represents the empirical density ratio of model samples obtained at two adjacent time steps. This decomposition gives a method to estimate p(x)/q,xx) iteratively, without the needs of keeping raw data samples from p(x) as increases. The key point is that we can optimize rx( by optimizing Ts, (x) when (x) is known. Existing methods for DRE, such as KLIEP introduced in Sec. 2, can be applied to estimating ri (x) and rw (x) as the basic ratio estimator of CDRE. Furthermore, only one extra constraint is required:
f rx (x)
(x)
rx) (x)dx=1
For instance, when (x) is defined by using the log-linear form as in Eq. (3), ru, can be expressed as follows, where represents parameters of (x):
n(xx) Fs 1-1((x)
expl VB (x-
explV8 (x) VA_
Li=1 expl V/, (xr,))
Na. q, (x), X--1, x--1,- (x
When rs, satisfies the constraint in Eq. (6), we have following equality by replacing Eq. (7)
into
Eq. (6) which we approximate the expectation using Monte Carlo integration:
exp( VR (x.,))
N
N
expl VB (Xr.)) VB (x.i))
(8)
exp( VA (x))
rs can then be rewritten as:
explvA (x) v (x)}
exp( V/ (Ar.i) VA (Ari)}
In this manner, r, is in the same lon-linear form of Eq. (3). We can directly apply KLIEP to optimize
B. which gives the following objective:
1 N,
max L((x:B.) max A
Σ logra, (x- + Ao (X))
1)2,
&
(10)
N expl (pp (xr)), (x)
M Σ exp wp (xx,) VB (x,:)
where vX((,)
where Br can be viewed as constant since it has been learned at task -- The equality constraint of Eq. (8) has been transformed and put into the objective using hyperparameter Â We observe there is trade-off between bias and variance controlled by Ag. Larger value of A results in smaller bias but larger variance. Relevant experimental results are demonstrated in Fig. 2. The density ratio estimator of KLIEP is unbiased estimator when the constraint is satisfied. However, the hard constraint by a soft constraint, the larger 2 makes the estimator with soft constraint closer to the unbiased one, which leads smaller bias. The bias is getting less when increasing the lambda, and as a trade off, the variance starts to increase.
replace
It has been discussed in Mohamed & Lakshminarayanal (2016) that the discriminator of some type of Generative Adversarial Networks (GANs) can be viewed as a density ratio estimator. We have also applied the formulation of discriminators of f-GAN (Nowozin et al., 2016) to the basic ratio estimator in CDRE. Nonetheless, we found is less robust than KLIEP and may not satisfy the constraint of density ratios as it is not defined for the purpose of estimating ratios. For example, the
4

---

Under review as a conference paper at ICLR 2020
ratio can be negative by the formulation of d di-veeegeeecc in f-GAN. Therefore, stick to KLIEP as the basic ratio estimator of CDRE in our experiments. However, estimating some divergences which are not based on log-ratios it may be better try some other form of ratio estimators. For instance, it may be preferable to use LSIF when estimating Pearson x² divergence, since a small result in large differences. Also, since LSIF itself is based Pearson x² divergence, it appears to be a more natural choice.
deviation in log-ratio
CDRE in continual learning. Now we consider the full setting in continual learning, in which the model needs to learn denote the raw data from task t, which not available at task when / > t, and let p(x[t) denote the density function of X,. Similarly, let &r, denote samples of task generated by the model at time (G,) and q: (x/t) denote the density function &z,. Now we have
distribution at each time step and refer to it as task Let X,
(x = p(x[t) 9x-1(x/T) p(x[t)
rs (x)(), (x = (x
(11)
qr (x] (x(T) qr- (x
q: (x(r))
We optimize the estimator at time / by an average objective
max L((B.) max B
Σ [(x)
(12)
B
where =x)(x/t:B) is the same (x:BB) in Eq. (10) for given and rs (x/t) is defined by the same form Eq. (9) except V/ (x) has been replaced by Vp. (x, 7). By concatenating the task index to each data sample the input of ratio estimator, we can avoid learning Y' separate ratio estimators for 7' tasks. addition, we set the output of V/ (-) as a t-dimensional vector (oi,...,o, where O₂ corresponds to the output of Vy (x, t). We found that this improves the accuracy of the ratio estimation when increases because the model capacity increases well.
Estimating f-divergences by CDRE We evaluate generative model in continual learning by estimating the averaged f-divergences over all learned tasks:
2. Σ 2((e(s)))ll((441t
(13)
is noteworthy that estimation error obtained at each task will accumulate CDRE as shown below, where r", denotes the true ratio and Ars; denotes the estimation error:
(x| = rx) (x|T)), (x/z) = (x π r, (x|x) 1=++1
H e (x|T) (x|T),
(14)
where T ESt. r, (x]x)
Ts (x/t)) = rxx]tt)
We observe that larger errors often lead to larger variance of the estimated Saciaatedddddee f-divergences and larger errors are often caused by greater differences between two distributions. We give more analysis about the variance of the estimated f-divergences in Sec. 4 along with the experiment results.
As discussed in Sec. 2.2, we perform dimensionality reduction as preprocessing of CDRE for extracting features from high dimensional data. A pre-trained classifier is often used to generate surrogate features for image data (e.g. inception features (Salimans et al., 2016)). However, we consider using Variational Auto Encoder (VAB) without pre-training in our experiments for two reasons: (a) it may be difficult to obtain an homogeneous dataset for pre-train- ing; (b) it may not be able to train a classifier when there are no labels available. In order to cope with the setting of continual learning, we introduce Continual Variational Auto Encoder (CVAE) for feature generation in a pipeline with CDRE. The loss function of CVAE accordingly adjusted by the principle of Variational Continual Learning (VCL) (Nguyen et al., 2018):
Feature generation for CDRE
Li 0,) = NLL + (xlla
q (z) = N (Ne, (x), 0g (x)).
(15)
NLL = -
Σ
Eg (xte) B mn flo (xxz 8, ]]]] p p( 8
5

---

Under review as a conference paper at ICLR 2020
where 0 and 0 denote parameters of the encoder and decoder task respectively. It is different with VAEs of continual learning described in Nguyen et al. (2018) because the posteriors of adjacent tasks in the KL divergence is w.r.t latent codes rather than parameters 0 of the VAE. In Nguyen et al. (2018) the encoder is task-specific which is computational costly and when the encoder is shared across tasks we found that latent codes cannot preserve much differences between different tasks. In our case, we just expect the encoder gives similar z for a similar x at different time 1, which can guarantee the consistency of inputs for the previous estimator (i.e. VB_, (xx,) in Eq. (10)). This adjusted objective can achieve the goal with cheaper cost in our experiments. We are aiming to provide general solution of feature generation in practical situations, nevertheless, other commonly used methods (i.e. pre-trained classifiers) are also applicable to CDRE.
EXPERIMENTS
In the absence of prior work on evaluating generative models by f-divergences, we provide ex- perimental results in Appx. B on comparing f-divergences with FID (Heusel et al., 2017), KID (Binkowski et al., 2018) and PRD (Sajaadi et al., 2018) in a few toy experiments and high- dimensional dataset of the real world. Through these experiments, we show that f-divergences can be alternative measures of generative models and one may obtain richer criteria by f-divergences.
In this section, we first show that CDRE can effectively estimate f-divergences in the setting of con- tinual learning by experiments on synthetic data. We then evaluate WGAN (Arjovsky et al., 2017), WGAN-GP (Gulrajani et al., 2017) and several members of f-GAN (Nowozin et al., 2016) on two bench-man datasets in continual learning: Fashion-MNISS
(Xiao et al., 2017) and MNIST (LeCun al., 2010). All GANs tested in continual learning are conditional GANs (Mirza & Osindero, 2014) with task indices as conditioners, and one task includes a single class the dataset.
We have deployed two feature generators for experiments with MNIST and Reshion-MNISTY 1) A classifier which is a Convolutional Neural Network (CNN) trained on real samples of all classes, the extracted features are the activations of the last hidden layer (a similar setting is suggested in Binkowski et al. (2018) for testing KID on MNIST); 2) A CVAE trained incrementally in the procedure of continual learning, and the features are the output of the encoder. The dimension of features are 64 for both classifier and CVAE. More details of the experimental settings are described in Appx. A. We use the classifier as the feature generator for FID,KID and PRD and use the CVAE as the feature generator for CDRE in all experiments except specified explicitly.
4.1 EXPERIMENTS WITH SYNTHETIC DATA
Feiti 1 Serthiand
Eyeloaal
(a) single task (2-dim)
(b) single task (100-dim)
(c) tasks (2-dim) 1.
(d) t tasks (100-dim)
2- 0.1
A-- 100
A 200
Figure 2: Synthetic experiments of estimating KL-divergence by CDRE (error bars from 5 runs). The x-axis the index of time steps. Figs. 2a and 2b compare the true KL-divergence with estimated KL-divergence for single task, Figs. 2c and 2d compare the average KL divergence of r tasks at time t. KL" and KLadr denote the true value and estimated value of KL-divergence, respectively.
In the experiments with synthetic data, we first simulate a single Gaussian distribution drifting over time and estimate the KL-divergence between the model distribution (x) at time r, Vt 2 I and the target distribution p(x) seen at time = 1. The distribution q, (x) = M (H, a²1) where μ4 = No + Au x G; = Go + Ao x and p(x) = =. (Mo, l We set Ho = 0. 6o = 1, Au = - Ao = 0.05 for 2- 2-dimensional data and Au = -o 0.02 for 100-dimensional data. We set Ao to be negative, simulating underestimated variance as it is a common issue in generative models. The sample size of each distribution is 10000. The results are shown in Figs. 2a and 2b. We then simulate the scenario

---

Under review as a conference paper at ICLR 2020
target distribution at each time step learning new task. In this case, real samples of task are drawn from a Gaussian distribution p(x| = A (Mz, o. and the model samples are drawn from gr (x(x)) = . (N, + Au x k, (a, + Ao x 1)) where μrr = 2t, o = - + 1, T Figs. 2c and 2d display the estimated average KL-divergence over all learned tasks in comparison with the true value of averaged KL-divergence. We see that the estimated value close to the true value of KL-divergence in all cases. The model capacity of the estimator may affect the performance as increases; however, note that with CDRE we have the flexibility to extend the model architecture since the latest estimator only needs the output of the previous one.
of continual learning, adding
We can also see smaller be leading to smaller variance but larger bias for which we have explained in Sec. 3. Besides the effect of the hyperparameter Ae. there estimated f-divergences: 1). The ratio estimator is a neural network which has no assumption about data distributions and trained by stochastic gradient descent, thus different initializations generated by different random seeds may cause larger variance of the results comparing with FID and KID (FID assumes data distributions are Gaussian distributions and KID fits the first three moments by polynomial kernel). 2). In the formulation of the ratio estimator, we use finite samples to estimate the expectations, which may bring estimation errors and become another source of variance, especially when the overlapping mass of the two distribution is sparse. This explains why the variance increases while the model distribution getting further to the raw data distribution. In this sense, the variance can also be criterion of evaluating generative models.
major sources of the variance of the
WGAN
114
WGAN-G? 1-GAN
WGAN
FCAN nKL
WGAN-GP
AGANLES
FGAN-AO
50
(a) FID
(b) KID
(c) KLqp
WCAN
WGAN
WGAN
WGAN-GP
WGAN-GP GP
WGGN-GP
S-AAL
E-GAN-
1-AAN
34
47
(d) KLPq
(c) Jensen-Shannon
(I) Hellinger
Figure 3: Evaluating GANs in continual learning on Fashion-MNIST, features for FID and KID are extracted from the classifier, features for f-divergences are generated by the CVAE. The dimension of generated features 64. The sample size 6000 for each class. The shaded area are plotted by standard deviation of 10 runs. The y-axis in the right side of Fig. 3b is the y-axis of the purple line (J-GAN-rvKI.), which is in a much larger scale than others.
4.2 EXPERIMENTS WITH IMAGE DATA
Fig. 3 compares several GANs trained Mushion-MNISST in continual learning using FID, KID and few members of f-divergences. We display randomly chosen model samples of those GANs in Fig. The experiment results MNIST are shown the Appx.
In general, these measures have a consensus that f-GAN-rvKL gives the worst performance during the whole process. They also agree that f-GAN-JS is the second worst before task and WGAN-GP is the second worst after task 7. And not surprisingly, there are several disagreements between these measures. Regarding f-GAN-JS, FID and KID are decreasing from task 3 to 10 whereas members of f-divergences that there is no notable improvement observed from task to 10, which indicates the decreasing trend of FID and KID is doubtful. Moreover, KID of WGAN and WGAN-GP fluctuate around an
like a plateau from task 4 to 10. According Fig. 4c, would argue

---

Under review as a conference paper at ICLR 2020
WGAN
WGAN-GP
(c) f-GAN-JS
(d) F-GAN-
Figure 4: Fashion-MNIST samples generated by GANs in continual learning. In each figure, each row displays images generated by the model at each task,the order is from the top to bottom (task to 10). The displayed samples randomly chosen from generated samples of each class.
approximately horizontal line whilst other measures show an increasing trend. Visually, samples from WGAN and WGAN-GP (Figs. 4a and 4b) are obviously losing fidelity while learning more tasks, which matches the increasing trend in measures except KID. Another disagreement that WGAN and f-GAN-JS are considered as almost equally well from task to 10 by the evaluation from f-divergences whereas FID and KID prefer f f-GAN-JS more than WGAN. In Figs. 4a and 4c, We observe that f-GAN-JS generates more images with darker color and lower fidelity than WGAN, however, images with brighter color generated by show higher fidelity than those samples from WGAN. This ambiguity may result in the disagreement as these measures may focus different parts of the density mass, which is analogous to the second toy experiment (Fig. 5) in Appx. B. All in all, the experiment results show that f-divergences estimated by CDRE do provide meaningful evaluations of generative models continual learning.
DISCUSSION
We show that CDRE capable of estimating of f-divergences in the setting of continual learning- provides alternative approach for model selection when other measures not possible in continual learning. The results also demonstrate CDRE can work with a simple CVAE for feature generation when a pre-trained classifier is not available. Moreover, our experiments show that CDRE can work well when the differences between model samples and real samples are significant, which a rather difficult situation for estimating density ratios. We consider more sophisticated method of dimensionality reduction for CDRE as future work, making it work more stably with high- dimensional data.
It is also possible to estimate the Bregman divergence by ratio estimation (Uehara et al., 2016), giving even more options of divergences for evaluating generative models. DRE also has many other applications other than estimating f-divergences, such change detection (Liu et al., 2013), mutual information estimation (Sugiyama et al., 2012), etc.. Likewise, CDRE may be useful for more applications in continual learning which we would like to explore in the future.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shiens, Benoit Steiner, llya Sutskever, Kunal Talwar, Paul Tucker, Vin- cent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Watten- berg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL BtLhttp:// http:::///eeorffoooo org/ Software available from tensorflow.org.
Shun-Ichi Amari. Divergence function, information monotonicity and information geometry. In Workshop on Information Theoretic Methods in Science and Engineering (WITMSE). Citeseer, 2009.
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International Conference on Machine Learning, pp. 214-223, 2017.
Mikolaj Binkowski, Dougal J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying MMD GANs. In International Conference on Learning Representutions (ICLR), 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL https://cpenreviie net/ ftro/oiiddideseeeer 9Fm.
Arslan Chaudhry, MarcAurelio Ranzato, Marcus Rohrbach, and Mohamed Elhoseiny. Efficient lifelong learning with a-GEM. In International Conference Learning Representations, 2019.
URL httph:tt//// /ttp:::///peeeevviiii net http:arrrreeeeeeeeoooommmeddddd. http:/orrmmiiiidff_______e_
Ishaan Guirajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Im- proved training of Wasserstein GANs. In Advances in Neural Information Processing Systems, pp. 5-67-57777 2017.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Takafumi Kanamori, Taiji Suzuki, and Masashi Sugiyama. f f-divergence estimation and two-sample homogeneity test under semiparametric density-ratio models. IEEE Transactions on Information Theory, 58(2):708-720, 2011.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEÉE Conference Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988, 2014.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska- Barskkskka al. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, pp. 201611835, 2017.
Yann LeCun, Corinna Cortes, and CJ Burges, MNIST handwritten digit database. AT&T Labs [Online]. Available: http://yann. lecun. com/exdb/mnistt, 2:18, 2010.
Timothée Lesort, Hugo Caselles-Dupré, Michael Garcia-Ortiz. Andrei Stoian, and David Filliat.
Generative models from the perspective of continual learning. arXiv preprint arXiv: 1812.891111111 2018.
Song Liu, Makoto Yamada. Nigel Collier, and Masashi Sugiyama. Change-pott point detection in time- series data by relative density-ratio estimation. Neural Networks, 43:72-83,, 2013.

---

Under review as a conference paper at ICLR 2020
David Lopez-Paz et al. Gradient episodic memory for continual learning. In Advances in Neural Information Processing Systems, pp. 6467-6476, 2017.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint
arXiv: 14f1:11411.1888 2014.
Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. arXiv preprint arXiv:1610.03480 2016.
Cuong V Nguyen, Yingzhen Li. Thang D Bui, and Richard E Turner. Variational continual learning. In International Conference on Learning Representations, 2018.
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions Information Theory, 56(11):5847-5811, 2010.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-GAN: plers using variational divergence minimization. In Advances in Neural Information Processing Systems, pp. 271 271-279, 2016.
Training generative neural sam-
Paul K Rubenstein, Olivier Bousquet, Josip Djolonga, Carlos Riquelme, and Ilya Tolstikhin. Practi- cal and consistent estimation of f-divergences. arXiv preprint arXiv:1905.11112 2019. 2019.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assess- ing generative models via precision and recall. In Advances Neural Information Processing Systems, pp. 5228-5237, 2018.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training GANs. In Advances in neural information processing systems, pp. 2234-2242, 2016.
Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In Advances in Neural Information Processing Systems, pp. 2990-2999, 2017.
Bharath K Sriperumbudur, Kenji Fukumizu, Arthur Gretton, Bernhard Schölkopf, Gert RG Lanck- riet, et al. On the empirical estimation of integral probability metrics. Electronic Journal of Statistics, 6: 6:1550-1599, 2012.
Masashi Sugiyama, Taiji Suzuki, Shinichi Nakajima, Hisashi Kashima, Paul von Bünau, and Mo- toaki Kawanabe. Direct importance estimation for covariate shift adaptation. Annals of the Insti- sute of Statistical Mathematics, 60(4):699-746, 2008.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012.
Masatoshi Uehara, Issei Sato, Masahiro Suzuki, Kotaro Nakayama, and Yutaka Matsuo. Generative adversarial nets from density ratio estimation perspective. arXiv preprint arXiv:1610.02920, 2016.
Aaron van den Oord, Oriol Vinyals, al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6306-6315, 2017.
Chenshen Wu, Luis Herranz, Xialei Liu, yaxing wang, Joost van de Weijer, and Bogdan Raducanu. categories without forgetting. In Advances in Neural Information Processing Systems 31, pp. 5962-5972. 2018.
Memory replay GANs: Learning to generate
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: novel image dataset for bench- marking machine learning algorithms, 2017.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv:1505.0088 2015.
10

---

Under review as a conference paper at ICLR 2020
A EXPERIMENTAL SETTINGS
We elaborate the experimental settings of our experiments here and all implementations are based on Python 3 and TensorFlow 1.10 (Abadi al., 2015).
A.1 CONFIGURATION OF FEATURE GENERATORS
The classifier used extract features both datasets has two convolutional layers with filter shape [4,4,1,64], [4,4,64, 1266,1228 respectively, strides are all [1,2,2,1], and two dense layers with hidden units [6272, 64]. Batch normalization is performed the second conv layer and first dense layer.
The encoder of the VAE has two dense layers with hidden units [512, 256], output dimension is 64, and decoder has two dense layers with hidden units [256, 256], output dimension is 784. The variance of noise is set to 0.1. It's trained with L2 regularization and the Lagrange multiplier is 0.001.
Both the classifier and VAE are trained with batch size = 100, learning rate = 0.001 and 100 epochs. Activation function of all hidden layers is ReLU. The optimizer is Adam (Kingma & Ba, 2014) for all training runs.
A.2 CONFIGURATION OF RATIO ESTIMATORS
The ratio estimator we used in all experiments is the log-linear model as defined in Equation 3 and Equation 12, and (r( is neural network with two dense layers, each having 256 hidden units. is trained by Adam optimizer and batch size = 2000. On toy data in continual learning, learning rate 2c-5, 7 are shown in Figure 4, sample size is 10000 per task. On MNIST and Fashion-MNIST in continual learning, 2.- 1., learning rate le-5, sample size of each class is 1000, validation sample size of each class also 1000. Maximum number of epochs is 1000 for DRE and 2000 for CDRE each task, the training process could be early stopped when validation loss increasing. de increases linearly with the number tasks in continual learning, which means at task 1, Aeg = 0o.0 X
A.3 CONFIGURATION OF GANS
All GANs are trained with a discriminator having the same architecture as the classifier described above, except the last dense layer has 1024 hidden unites; a generator having two convolutional layers with filter shape [4,4,64, 128,,,44,,44 respectively and two dense layers with hidden units [1024,6272), applied batch normalization on two dense layers. Activation function is leaky ReLu (Xu et al., 2015) for all hidden layers. The optimizers are Adam, batch size is 64 and learning rate is 0.0002 and 0.001 for discriminator and generator respectively. wfveellffGGA. f-GAA-rvKKL is trained by 6 epochs as longer training makes it worse, all others trained by 15 epochs. The random input of generators has dimension 64 for both MNIST and Fashion-MNIST.
B EVALUATING GENERATIVE MODELS USING F-DIVERGENCES
We provide experimental results of comparing f f-divergences with FID (Heusel et al., 2017), KID (Binkowski et al., 2018) and PRD (Sajjadi et al., 2018) in a few toy experiments and high- dimensional dataset of the real world. Through these experiments, we show that f-divergences can be alternative measures of generative models and one may obtain richer criteria by f-divergences.
B. TOY EXPERIMENTS WITH MNIST
We present toy experiments to show the differences between f-divergences and FID, KID in terms of evaluating generative models. We demonstrate the experiment results through two most popular members of f-divergences: KL.-divergence and reverse Kl.-divergence.
In the first experiment, We have two cases: (i) the target distribution contains half of the classes of MNIST, and the evaluated distribution Q includes all classes of MNIST; (ii) the reverse of (i).
Source code be found ww thub. htw.b.com/reeeee leda fterreview

---

Under review as a conference paper at ICLR 2020
We obtain density ratios by KLIEP and then estimate f-divergences by density ratios. The results are shown in Tab. 1, with PRD curves displayed beside the table. Since the KLIEP objective is not symmetric (Eq. (2)), the estimated KL divergences
not symmetric when switching the two sets of samples. As expected, ())(P prefers Q with larger recall and vice versa. Neither FID nor KID are able to discriminate between these two scenarios.
1.0
Table 1: Results of the toy experiment. P
21/2 implies case (i), P1/2 implies case (ii). Standard deviations are from 5 runs.
0.8
0.6
0.4
FID
KID
(xI()))))
Dx (Q))P)
0.-
P = 11/2 Q = Pi/2
50.39 = 0.00
2.04 + 0.01
0.67 + 0.00
3.78 + 1.22
P=Oie
50.39 0.00
2.03 + 0.02
2.49 0.30
2.38 1.78
0.00
0.25
0.50
0.75
1.00
Recall
In the second experiment, we show that f-divergences may provide different opinions with FID and KID in cer- tain circumstances because FID and KID are based on Integral Probability Metrics (IPM) (Sriperumbudur et al., 2012) which focus on parts of the distribution with most based density ratios which may give more attention on parts with less mass (due the ratio of two small values can be very large). To show this, we simulate two sets of model samples by injecting different types of noise into MNIST data and evaluate them on the original feature space (which 784 dimensions). Regarding the first type of noise, we randomly choose 50% samples and dimension to be cor- rupted (set the pixel value to 0.5); for the second one, we randomly choose 10% samples and 50 dimensions to be corrupted. The results shown in Fig. 5, in which KL-divergence and reverse KL-divergence disagree with FID and KID regarding which set of samples is better than the other.
2.-
2.--
1.-
whereas f-divergences
5.0
72.51
Figure 5: Toy experiments with differ- ent types of noise injected into MNIST (error bars from 5 runs).
B.2 EXPERIMENTS WITH HIGH-DIMENSIONAL FEATURES ON FFHQ
In order to show that DRE can work with high dimen- sional data with high fidelity, We also conducted an ex- periment with samples generated by StyleGAN on FFHQ dataset (Karras et al., 2019). We obtain model samples by the pre-trained StyleGAN? and estimate f-divergences the inception feature space with 2048 dimensions. The sample size is 50000 and compare the model sam- ples with real samples. We see that the f-divergences estimated by DRE giving reasonable results with small variance (Tab. 2), which indicates it can be an alternative measure for those state-of-the- of-thh-art generative models.
Table 2: Evaluating StyleGAN using f- divergences estimated by DRE
StyleGAN
Real samples
KL
2.47 0.002
0.02 ee-
rv KI.
3.29 3.294-0.18
0.02 9.3e
IS
+0.01
0.01
+ 4.55e
Hellinger
1.04
0.01 + 4.6e
C EXPERIMENT RESULTS ON MNIST
We show experiment results MNIST in Figs. 6 and the settings of these experiments the same as experiments with Fashion-MNIST.
12

---

Under review as a conference paper at ICLR 2020
0.5
WSAN
WGAN-GP
WGAN
AGAN-J8
WGAN-GP
FGAN-IKL
FGAN-WKL
(a) FID
(b) KID
WGAN
WGAN
WGAN-GP
WGAN-GP
E-GAN-JS
2.
GAN--J
F.GAN-rKL
f-GAN-KKL
14
000
(c) KLpq
(d) KLqp
WGAN
WGAN
WGAN-GP
WGAN-GP
F-GAN-JS
F-
L-GN--rKil
E-GAN-rvKL
00
(e) JS
(1) Hellinger
Figure 6: Evaluating GANs in continual learning on MNIST, features for FID and KID are extracted from the classifier, features for f-divergences are generated by CVAE. The shaded area are plotted by standard deviation of 10 runs.
13

---

Under review as a conference paper at ICLR 2020
Ó
2
2
/
>
2
2
/
/
2
2
3
3
/
6
2
3
3
/
/
3
2
3
3
4
4
7
2
3
3
7
4
C
J
/
2
3
3
1
3
2
3
4
5
2
3
X
4
4
6
J
3
4
5
3
9
5
6
7
]
C/
5
6
7
7
7
3
4/
5
6
7
&
7
3
3
6
8
2
1
7
9
3
4
WGAN
WGAN GP
Q
O
0
0
0
1
O
O
O
/
O
O
/
2
2
O
/
/
A
2
3
0 C
/
2
3
3
I
/
2
3
3
O ] /
2
2
3
3
4
4
/
I
2
3
3
O /
/
2
2
3
3
4
6
/
3
5
/
/
2
3
5
6
J
2
3
1/
5
O
Q
/
3
4
5
6
7
3
4
5
6
7
0 /
2
3
4
5
6
7
8
0
0
2
4
6
7
8
2
3
4
6
7
9
S
5
7
9
(c) f-GAN-S
(d) -GLN
Figure 7: MNIST samples generated by evaluated GANs in continual learning. In each figure, each row displays figures generated each task, the order from the top bottom.
14


Reviews:
[{"review_id": "rJevdma6Fr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, authors propose a continual learning for density-ratio estimation. The formulation of the CDRE Eq.(5)is quite intuitive, and it makes sense. Then, a log-linear model is employed for a density-ratio model, and it is estimated by using a density-ratio estimation algorithm. Through experiments, the authors show that the proposed algorithm outperforms existing methods.\n\nThe paper is clearly written and easy to read. The density-ratio estimation algorithm for continual learning is new and interesting.\n\nDetailed comments:\n1. I am pretty new to the continual learning. The formulation of CDRE is interesting. However, I am still not that certain whether the setup is realistic. In the introduction, authors describe that we cannot obtain data points due to privacy or limited cost budget. More specifically, if it is a privacy issue, we may not be able to use the model trained by the private data as well. Also, could you give me a couple of examples of the limited cost budget case?\n\n2. In this paper, authors employed the log-linear model. If we use another model, performance can be changed?\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Skl_FzIaYS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "######### Updated Review ###########\n\nI'd like to thank the author(s) for their rebuttal. However, I am still on the same boat with R1 and recommend rejection for this submission. \n\n\n################################\n\n\nThis submission seeks to evaluate generative models in a continual learning setup without storing real samples from the target distribution.  The main technique the author(s) have used is the likelihood-ratio trick. I question the scope of this paper, as this is not a topic of general interest to the community. Additionally, the density ratio estimation technique is fairly standard. I vote to reject this submission for the lack of highlights and relevant potential applications. \n\nMy main argument for rejection. \nWhile continual learning is a trendy topic in the AI community, it's less well-received in the context of generative modeling, probably for the lack of real applications. Such works, including this one, fail to address any real challenge, as the hypothesized scenario is unrealistic. For example, I am not convinced of the significance of using f-div to evaluate model performance. And since importance sampling is notorious for its variance issues (the essential mathematical tool used in this model), the estimate is not expected to be reliable, say subsequent tasks q_t and q_{t-1} differ somehow. This submission feels more like playing a game with the rules defined by the author(s), not driven by practical considerations. \n\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Bye8oOS6Yr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission proposes a continual density ratio estimation method and suggested its application to evaluate a continually learned generative model without requiring the previous training data. The basis of the continual density estimation is based on a recursive relationship between the density ratio at step t and that at t - 1. Continually estimating the density ratio without storing the raw data is an interesting topic and could be useful for continual learning. To my knowledge, I have not seen this in earlier publications.\n\nHowever, I give reject to this paper because of the following reason: \n\nThe writing of this paper is not easy to follow. \n\n- The beginning of section 3 (CDRE in continual learning), I found it difficult to understand why the model q needs to be updated (indexed by t) while p(x) is not dependent on t. As far as I know, under the continual learning setting the data distribution p(x) is also conditioned on t. I interpret it as a general introduction on how density ratio could be estimated continually.\n- The Lagrange multiplier and the bias / variance statements need elaboration, I don't understand how it is affecting the bias and variance.\n- In the second part of section 3, the continual learning setting is introduced (in equation 11), however, it is no longer reasonable to use the symbol r_t in equation 12 which was initially defined in equation 5. \n- A loss for continual VAE is proposed in seciton Feature generation for CDRE, however, the p(x) is again independent of t. And I'm also suspicious that equation 13 is the correct way of adjusting VAE's objective with VCL. In VCL, the KL divergence is on the parameter distribution, which could help prevent forgetting, however, here the KL is between VAE's approximate posteriors, which alone is not sufficient for keeping the information of previous tasks.\n- There's lack of analysis / interpretation of results for section 4.1, e.g. what is the motivation of the experiments and what is the conclusion.\n- Through out section 4.2 - 4.3, it is not explained what is the source of variance in the experiment results.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper seems technically correct and has some novelty, but the relevance of the paper is questionable. Considering the selectiveness of ICLR, I cannot recommend the paper for acceptance at this point. 

In more detail: the authors propose a technique for estimating density rations between a target distribution of real samples and a distribution of samples generated by the model, without storing samples. The method seems to be technically well executed and verified. However, there was major concerns among multiple reviewers that the addressed problem does not seem relevant to the ICLR community. The question addressed seemed artificial, and it was not considered realistic (by R2 and also by R1 in the confidential discussion). R3 also expressed doubts at the usefulness of the method. 

Furthermore, some doubts were expressed regarding clarity (although opinions were mixed on that) and on the justification of the modification of the VAE objective to the continual setting. 
Decision: Reject