Implicit λ-Jeffreys Autoencoders: Taking the Best of Both Worlds
Authors: Aibek Alanov
Max Kochurov
Artem Sobolev
Daniil Yashkov
Dmitry Vetrov
Venue:  (2020)
Keywords: Variational Inference
Generative Adversarial Networks

Abstract: We propose a new form of an autoencoding model which incorporates the best properties of variational autoencoders (VAE) and generative adversarial networks (GAN). It is known that GAN can produce very realistic samples while VAE does not suffer from mode collapsing problem. Our model optimizes λ-Jeffreys divergence between the model distribution and the true data distribution. We show that it takes the best properties of VAE and GAN objectives. It consists of two parts. One of these parts can be optimized by using the standard adversarial training, and the second one is the very objective of the VAE model. However, the straightforward way of substituting the VAE loss does not work well if we use an explicit likelihood such as Gaussian or Laplace which have limited flexibility in high dimensions and are unnatural for modelling images in the space of pixels. To tackle this problem we propose a novel approach to train the VAE model with an implicit likelihood by an adversarially trained discriminator. In an extensive set of experiments on CIFAR-10 and TinyImagent datasets, we show that our model achieves the state-of-the-art generation and reconstruction quality and demonstrate how we can balance between mode-seeking and mode-covering behaviour of our model by adjusting the weight λ in our objective. 

Under review as a conference paper at ICLR 2020
IMPLICIT X-JEFFREYS AUTOENCODERS: TAKING THE BEST OF BOTH WORLDS
Anonymous authors Paper under double-blind review
ABSTRACT
We propose new form of an autoencoding model which incorporates the best properties of variational autoencoders (VAE) and generative adversarial networks (GAN). It is known that GAN can produce very realistic samples while VAE does not suffer from mode collapsing problem. Our model optimizes A-Jeffreys divergence between the model distribution and the true data distribution. We show that it takes the best properties of VAE and GAN objectives. It consists of parts. One of these parts can be optimized by using the standard adversarial training, and the second one is the very objective of the VAE model. However, the straightforward way of substituting the VAE loss does not work well if we use an explicit likelihood such as Gaussian or Laplace which have limited flexibility in high dimensions and are unnatural for modelling images in the space of pixels. To tackle this problem we propose a novel approach to train the VAE model with an implicit likelihood by an adversarially trained discriminator. In an extensive set of experiments on CIFAR-10 10 and Tinylmagent datasets, we show that our model achieves the state-of-the-art trade-off between generation and reconstruction quality and demonstrate how we can balance between mode-seeking and mass-covering behaviour of our model by adjusting the weight À in our objective.
INTRODUCTION
Variational autoencoder (VAE) (Kingma al., 2014 Rezende et al., 2014 Titsias & -Greill 2014) is one of the most popular approaches for modeling complex nigh-dimensional distributions. has been applied successfully to many practical problems. has several nice properties such as learning low-dimensional representations for the objects and ability to conditional generation. Due to an explicit reconstruction term in its objective, one may ensure that VAE can generate all objects from the training set. These advantages, however, come at price. It is known fact that VAE tends to generate unrealistic objects, e.g., blurred images. Such behaviour can be explained by the properties of maximum likelihood estimation (MLE) which is used to fit restricted VAE model (1() in data that comes from a complex distribution p" (z:). We can equivalently reformulate this MLE objective
minimization of the forward KI. divergence (KI(p" (x: (n(x)) which does not explicitly penalize the model (e(z) for generating unrealistic objects (Arjovsky & Bottou (017)). As result, many regions with the low value p" (x) may have high value of po(x)) in the case when the model Po (z) has the limited capacity (see Figure [a)..
Another popular generative model is generative adversarial network (GAN) (Goodfellow al. 2014) which is known for its ability to sample realistic objects. However, it suffers from an inability to cover the whole distribution p" (x) that leads mode-seeking behavior, also known as mode-collapse
(Salimans et al.) 2016 Metz et al. 2016 Goodfellow (016)). The main reason of such behaviour is properties the reverse KI. divergence DKL. (po(x) [Ip (z)) (or the -ensennShanon divergence
JSD(pa (z) [z)]pp (>p))ppp))))))))))))))))))))(( (x))) that is minimized during GAN training. These divergences DKL (pe (z))lpp" (x))
and JSD(pa (x) Uppp (z)) do not penalize the model pe (z) for ignoring some high value regions of p=(x) (Arjovsky & Bottou 2017). As result, the most probability mass of the restricted model
(a() can be concentrated in a small number of modes of p" (æ) (see Figure [b).
In this paper, we propose
form of an autoencoding model which is based on incorporating of VAE and GAN objectives. We consider A-Jeffreys divergence J (po())))pp (x)) which a weighted
sum of forward and reverse KI. divergences: JA (p(o(<)) (=)) (Dki(p" (z))))((z)) (1

---

Under review as a conference paper at ICLR 2020
13
3
13a
Data Forward KL
Data
Data Reverse
23
J5D
jelfreys
Jeffreys
Jeffreys
1311
jefffrrys Forward KL
jeftreys vsJSD
3 Joftres vs Reverse KL
d) Learned densitties
Figure I: Comparison of X-Jeffreys for X = 0.5 (red), Reverse KL (green), Forward KL (blue) and ISD (orange) divergences on the task of approximating a mixture of 4 Gaussians (black dashed line) with an equiprobable mixture of two Gaussians with learnable location and scale. Plots a)-c) show pairwise comparisons of optimal log-densities, the plot d) compares optimal densities themselves.
DKL (po (x) Ilp" (z)). This way, we encourage our model to be mode-secking while still having relatively high values of pa (z) on all objects from a training set, thus preventing the mode-collapse. We note that Jx (p (x)]lp" (x)) is not symmetric with respect to pe (z) and p" p²(x) and by the weight À we can balance between mode-seeking and mass-covering behaviour.
However, the straightforward way of substituting each KL term with GAN and VAE losses does not work well in practice we use an explicit likelihood for object reconstruction in VAE objective. Such simple distributions Gaussian or Laplace that are usually used in VAE have limited flexibility and are unnatural for modelling images in the space of pixels. To tackle this problem we propose a novel approach to train the VAE model in an adversarial manner. We show how we can estimate the implicit likelihood in our loss function by an adversarially trained discriminator.
We theoretically analyze the introduced loss function and show that under assumptions of optimal discriminators, our model minimizes the A-Jeffreys divergence J (Po(z) (r))lp' "ll''"""""" (x)) and we call our method as Implicit X-Jeffreys Autoencoder (A-IJAE). In an extensive set of experiments, we evaluate the generation and reconstruction ability of our model on CIFAR10 (Krizhevsky et al. (2099) and Tinylmagene datasets. It shows the state-of-the-ar trade-off between generation and reconstruction quality. We demonstrate how we can balance between the ability of generating realistic images and the reconstruction ability by changing the weight À in our objective. Based on our experimental study we derive a default choice for À that establishes reasonable compromise between mode-seeking and mass-covering behaviour of our model and this choice is consistent over these two datasets.
2 RELATED WORK
Relation to forward KL-based methods. We can say that all VAE-based models minimize the upper bound on the forward KL. DKL, (p"(())))p)) (Pp)) In recent years there have been many extensions and improvements of the standard VAE. One direction of research is to inroduce the discriminator as
a part of data likelihood (Larsen et al. 2015 Brock et al., 2017) to leverage its intermediate layers for measuring similarity between objects. However, these models do not have sound theoretical justification about what distance between pa (z) and p" (z) they optimize. The other way is to consider more complex variational distribution q (=(z)) One can either use better variational bounds (Agakov & Barber 2004; Maalge et al. 2016)] |Ranganath et al.) [0066] MMlchanov et al., 2018 Sobolev & Vetrov 2019) or apply the adversarial training to match (2 and the prior distribution p(=) (Mescheder et al. 2017) or to match the marginals (z) and p(z) (Makhzani et al. 2016). Although
these methods improve approximate inference in VAE model, they remain in the scope of MLE
framework. As we discussed above within this framework the model with limited capacity is going have the mass-covering behaviour.
Relation to reverse KL-based methods. The vanilla GAN framework is equivalent under the as- sumption of optimal discriminator to minimization of Sensen-Shanon divergence JSD (xx (x)) (Goodfellow et al.) 2014). With minor modification of a generator loss we can obtain the equiv- alence to minimization of the reverse KL DKL (x) (x)) (Huszar 2016 Arjovsky & Bottou 2017). There have been proposed many autoencoder models which utilize one of these two diver- One approach is to minimize the divergence between joint distributions p" 0z(zz) (z)g(22 a)) and (2(=22)p(2 (2) in a GAN framework (Donahue et al. 2017 Dumoulin et al.) 2017). ALICE model (Li et al., 2017) introduces an additional entropy loss for
gences JSD(p" (x) IIpe (z)) and DKL (pe (p(()))l>*(>))
dealing with the non-identifiabilit issues in previous works. Other methods (Chen et al. 2018
2

---

Under review as a conference paper at ICLR 2020
Pu et al.
2017a Rosca et al. 2017 Ulyanov al. 2018 Zhu et al. 2017)
the reverse KL as an additional term to encourage mode seeking behaviour.
D DKL (p x (=)
Relation to Jeffreys divergence- methods. To the best of our knowledge, there only two other autoencoder models which minimize X-Jeffreys divergence for À 0.5. is an important case when X-Jeffreys divergence equals symmetric KL divergence. These methods are AS-VAE (Pu et al. 2017a) and SVAE (Chen et al., 2018) and they are most closely related to our work. AS-VAE is special case of SVAE method therefore further we will consider only SVAE. There are two most crucial differences between SVAE and X-JJAE models. The first one is that SVAE minimizes (x (z) (z)q(((()) (pp (=)))((() =))())) between joint distributions p" q(( 2/2r) and pe (a: 2))p()) for x= = 0.5 while A-IJAE minimizes Jx(p" (x)) between marginal distributions p" (z) and po(r) for arbitrary X. The second difference is that the SVAE's loss Jx(p" (=))(=/=))l (a) (z)))
solely did not give good reconstructions in experiments. Therefore, authors introduced additional data-fit terms (x))(())) log Pa (z:12) + (x)s)(((( log (2(zz) where pa (zr 2) and qg (2|ar are
explicit densities. In contrast, A-IJAE model achieves good generation and reconstruction quality as it is and allows training implicit pa (zr(²) and (2(r)) distributions. These two differences make SVAE and X-IJAE models significantly distinet, and we observe it in practice.
BACKGROUND
Consider samples X from the true data distribution p" (z). The aim of generative models is to fit model distribution (x) to pr()). Most popular models are GAN and VAF. In practice, we observe that they have significantly different properties. VAE tends to cover all modes of p (æ) the cost of capturing low probability regions well. As result, it often generates unspecific and/or blurry images. On the other hand, GAN is highly mode-seeking, i.c. it tends to concentrate most of its probability mass in a small number of modes of p" (z)). Therefore it may not cover significant part of p" (z)) which is also known as a mode collapse problem. Such radical contrast between VAE and GAN can be explained by the fact that they optimize different divergences between pe(x) and p" (z).
Variational Inference. VAE is trained by MLE: maX Epp (ax) log Po(z). The distribution (0(x) is defined as an integral over a latent variable 2: po (xx = pa ar pp (2 )dz, and in practice it is typically intractable. Variational inference (Hinton & Van Camp 1993) sidesteps this issue by introducing an encoder model (also known as a variational distribution) qo (z|ær) and replacing the intractable log po (I)) with tractable evidence lower bound (ELBO):
(p) Po() Bn log po(x|z)p(z) (=(x(x)
= Ep) (=(x) log (x(2)) K [=(g(() COLBO s))
(1)
Then we maximize ELBO (O equivalent to optimizing the forward KI. (xI.(y" l|po))
(0, sp) with respect to 0 and One can easily derive that MLE is
p" (x)
= arg max [Ep-(n) (a) log (o = arg max
- Ep (x) log
= arg min DKT. (p" (pp)) (2)
Po (z)
0
Adversarial Training, The adversarial framework is based on game between a generator Go (2( and discriminator De (x) which classifies objects from p" (a)) versus ones from po
min max (Ep- (u) log (x) + Epo(z) log(1 Dx(x))]
(3)
8
Goodfellow et al. (0014) showed that the loss the generator
3 equivalent to the Jensen-Shanon (x)))
divergence JSD [pn) given an optimal discriminator D (z) (=)))pp((a)"
Ep (a) log D (x) + (x)(2) log(1 - D (x)) DKI. (p"
p" tre
+ DKI. (po
log 4
It is easy to recognize this as an instance of classification- (Sugiyama et al. 2012). Following this framework, one can consider different generator's 's objectives
Density Ratio Estimation (DRF)
De- (x)
while keeping the same objective (3) for the discriminator. DRE relies on the fact that T-D,-
3

---

Under review as a conference paper at ICLR 2020
(=) pe(x))
By this approach can obtain fikelihood-free estimator for the reverse DKL DRL(pell>) (pellp")
(Huszar 2016):
(2-(22) 1 - Da- =po((s) log po(x) px (z) (KL(Pollp")
E po(()) log
(Un)Biased Gradients in Adversarial Training, Since in practice the discriminator D& (z) is only trained to work for one particular set of generator parameters 0, we need to be cautious regarding validity of gradients obtained by DRE approach. For example, consider the forward KL DKL. Ipo))
If we apply DRE, we will arrive (x)) log D2(2) I-Dx(a)' However, we can notice that in practice this
Py(a) expression does not depend on 0 in any way, i.c. Vollip (a) log 1- = O. This is because the forward KL depends on 0 only through the ratio of densities, which is replaced by point estimate using a discriminator which has no idea regarding pa 's local behaviour.
This shows we need to be careful when designing adversarial learning objective as to ensure unbiased gradients. Luckily, (SD(pollp") and DKL (po lIp") are not affected by this problem:
2017) Let (x-(x) p=()))po(z)) for any Then Proposition 1. Meeccheeer al.
(x)
Poo log I-D (x)
Da-la)
VoDKL (po (lr)))))))))) and Volpo(z) log(1 De (x-(x)) = VoJSD(polll")
even if assume that VoD (x)
Proof. Given in Appendix, section
A
4 IMPLICIT X-JEFFREYS AUTOENCODER
VAE provide a theoretically sound way to learn generative models with a natural and coherent encoder. However, they are known to generate blurry and unspecific samples that have inferior perceptual quality compared to generative models based on adversarial learning. The main cause for that is that the root principle VAEs are built upon MLE framework is equivalent to minimization of the forward KL DKI.(V" (.po). While (x (IPo) recovers the true data-generating process p²(x) if the model (=)) has enough capacity, in more realistic case of an insufficiently expressive model pe (ar) is known to be mass-covering. As a result, the model is forced to cover all modes of p (xr) even at the cost of covering bow-poobayility regions well. This in turn might lead blurry samples as the model does not have the capacity to concentrate inside the modes. On the other hand, the reverse KL (KI(po pp)) has mode-seeking behavior that penalizes covering low-probability regions and thus the model Po (z) tends to cover only few of the modes of p" (z)).
Following this reasoning, we propose a more balanced divergence one that seeks modes, but still does decent job covering all modes of p*(x) to prevent mode collapse. We chose X-Jeffreys divergence (Jeffreys) 1998) between po(x) and p² (x)): (x(po())llp" (x)) ADKi.p" (p=((()) (x)]lpo(xr) +
- AA)LL (po (z)).
We illustrate the advantage of X-Jeffreys divergence for 0.5 over Forward KL, Reverse KL and JSD divergences in the case of model with limited capacity in Figure [ In this figure we compared divergences in a simple task (see Appendix, section B) of approximating a mixture of Gaussians with mixture of just two: both Reverse KL and JSD exhibit mode-seeking behavior, completely dropping side modes, whereas the Forward KL assigns much more probability to tails and does poor job capturing the central modes. On contrast, X-Jeffreys divergence capture the most probable mode, and the other to ensure mass-covering.
mixture component to
The optimization of X-Jeffreys divergence consists of two parts. The first one is the minimization of the reverse KL DKL (pe( [1))((")((( [[lp(())) which can be implemented as a standard GAN optimization as we discussed in Section 3 The second part the optimization of the forward KL DKL. (p" (a)llpo(z)
and we tackle it by maximization of the ELBO CELBO (f, yp) as in VAE. So, we obtain an upper bound on X-Jeffreys divergence by incorporating GAN and VAE objectives:
(O, up) = (1 A) Dz) (x)) AL 4) > (x(po())lle"(n)) (x))
The ELBO term S0, up) can be decomposed into two parts: (1) a reconstruction term Epp (x) E (ala) log (x/2)) (ii) KI. term (x)( (T(o((()))))))) While both terms easy
4

---

Under review as a conference paper at ICLR 2020
to deal with in cases of explicit p(z|z) and q(2[2)), implicit formulation poses some challenges. In the next two sections we address them.
Typically to optimize the reconstruction term (z]2) the conditional likelihood po(rlz) is defined explicitly as a fully factorized Gaussian or Laplace distribution (Kingma al. 2014 Rezende et al. 2014 Titsias &
Implicit Conditional Likelihood.
Bg log Po
Lázaro-Gredilla 2014 Pu et al., 2017b] [hen et al.J 2018: Rosca et al.) 2017] Mescheder et al.)
2017). While convenient, such choice might limit the expressivity of the generator Gø(z). As
discussed previously, optimization of the forward K L(p") p leads to a mass mass-covering behavior. The undesired properties of this behavior such as sampling unrealistic and/or blurry images can be more significant if a capacity of our model (e( is limited. Therefore we propose a technique which allows to extend the class of possible likelihoods for po po(x]2) implicit ones.
We note that typically in VAE the decoder (r())) first maps the latent code 2 to the space X, which then used to parametrize the distribution of 2's decodings 2/2. For example, this is the case for N (z) o)) Laplace(<: (x))a a/). We also use the output of the generator Gø(z) E X to parametrize an implicit likelihood. In particular, we assume pa (x/2z) = r=))) for some symmetric likelihood r=zly):
Definition 1. A density r(-l-) & x X R4 is symmetric likelihood if
(i) r(x = aly = b) = r(r = bly = a) V Va,b
(ii) r(x = aly b) has a mode at a b.
While the Gaussian and Laplace likelihoods are symmetric and explicit, in general we do not require r(xly) to be explicit, only being able to generate samples from r(xly) is required.
The idea introduce a discriminator (, z, v. which classifies two types of triplets:
real class: (z, 2,8) p" (1))
fake class: (x,, v) (x) ( (1lerr
We note that r(ylær) and r' (y)))) can be different and we will utilize this possibility in practice. Then train the discriminator (, z, y) using the standard binary cross-entropy objective:
(x)( (Er(lle) log D (xx, z, v) (y(G()))))) log(1 D, 2, max
(4)
we apply the straightforward way to obtain an objective for the generator Ga(z) we will derive D.(2,2.)
that we should minimize Ep (e)2) log 1- I-D, Indeed, given an optimal discriminator for
44 D, (2,
r(v/w)
obtain:
r(ula)+ +r') (r
Dy (x, 2,
E (=)()) log 1 D, (2)
Ep (s)g log
(x())o((2)
= EEpg
+ Const
So, we see that minimizing Eep (=Ix)) log I-D, (a.s.rr given the optimal D,- (æ, 2, v) is equivalent
12, (2.2.z)
to maximizing the reconstruction term with pa (z|2) r' (z)) However, face in practice the same issue as we discussed Section that (e)gp((*)), (=lar log (=.207) D= .) does not depend on 0 explicitly even for optimal 7 7".
(2.2.w)
0 because
We can overcome this issue by exploiting the properties of symmetric likelihoods we minimize
a slightly different loss for the generator Go(z): - - (x)qu(=la) log [e,, (o)) The following
(r.z,s,Gaaz))
theorem guarantees the gradients will be unbiased in the optimal discriminator case:
Theorem 1. Let Dy- (1, 2, be the optimal solution for the objective
4 and r(yla)
and r' (y)zz) symmetric likelihoods. Then (x) log I- I-D. (Z.2. VolEEp (ux) (y) log Gg(z)).
D,- Go(z)
Proof. Given in Appendix, section A
5

---

Under review as a conference paper at ICLR 2020
So. we obtain that we can maximize the reconstruction term (s)) logr(z)Go(2)) by (o()))
minimizing - -xn (ar) log 1- 1. (r,,, and optimize it using gradient based methods. We note again that we do not require an access to an analytic form of (ry))G(())
It is an open question what is the best choice for the r'y Gg(z)). Our expectations from r(y] (=)) that should encourage realistic reconstructions and highly penalize for visually distorted images. In experiments, as r(ylz) we use a distribution over cyclic shifts in all directions of an image This distribution is symmetric with respect all directions and has a mode z, therefore the symmetric likelihood (see Definition I for details).
Although practice we use r(ylæ) which has an explicit form due to non-optimality of D=(x, 2, v) (that is always the case when training on finite datasets) the ratio log Pe(a,a.Goo2))
(-D-(I.,,Ga(E)) sets implicit likelihood of reconstructions. We can think of the non-optimality of (x, 2. form of regularization that allows us to convert explicit function r(y(z) into implicit likelihood that has desirable properties, i.e. encourages realistic reconstructions of and penalizes unrealistic ones.
Implicit Encoder. The KI term from LELBO (p) can be computed either analytically, using the Monte Carlo estimation or by the adversarial manner. We chose the latter approach proposed by Mescheder et al. (2017) because it enables implicit variational distribution qu (2(2) defined by a neural sampler (encoder) Ex <) where & ( 1). For this purpose we should train discriminator Dę(2, 2) which tries to distinguish pairs (I, 2) from p" (z)4p(2(2z) versus the ones from p" (x)p
The training objective of Dç( 2)
(x))(t) log (x 2) + Ep (=))) log(1 (1 - (ç(2, 2)) + max
(5)
(2))) reverse KL with respect to parameters yo, therefore we can substitute by D.(x.s)
the expression - -B (xlx) log 1-22( (see Section 3).
Final Objectives. Putting it all together arrive the following objective:
Lx (0, sp) =(1 - DxL (po()))))l (x:) - )LELnO(O,
Da: (x)
= - (1 A) log
-DD (x)
Dy, (x, 2, (z()))
(x,
(x) (x) (zlx) log 1 D, (x, 2, Go(z)) + log T- (x-(x,
min 0.p
In practice, discriminators are not optimal therefore we train our model by alternating gradients. We maximize objectives (3). 4b. 5 for (z), (2(x, 2, v), De(x, z) respectively and minimize (o)
for the generator Go(z). E ( for the encoder (x, {) where (o()), LE(P) are:
(r, (((2))
Co( = (x) log 1 Dx(x) (1 A)E (1) (2) log 1- 1-1, D2(x, 2, Go(z))
min
(6)
D+(r, Go(z)) + log
De(2, 2)
CE(()) (s)
log 1 D2 (z,
min (7)
1 De
EXPERIMENTS
In experiments, we evaluate generation and reconstruction ability of our model on datasets CIFAR-10 and TinylmageNet. We used a standard ResNet architecture (Gulrajani et al. 2017) for the encoder (z(2, E). the generator G&(2) and for all three discriminators De D, (z, 2, y). (z(z, 2). The complete architecture description for all networks and hyperparameters used in X-JAAE can be found in Appendix, section[D [ To compare our method to other autoencoding methods in the best way, we also used official and publicly available code for baselines. For AGE we use a pretrained model. For SVAE TwoStage-VAE (2S VAB we report metrics reproduced using officially provided code and hyperparameters. For o-GAN also use public implementation with same architecture in
3
X-IJAE.
AGE github
SVAE SVthuu
TwoStageVAP. github
o-GAN github

---

Under review as a conference paper at ICLR 2020
CIFAR 10
Tinylmagenet
0.8
0.8
LPIPS
0.4
0.6
0.4
0.6
0.2
0.4
0.2
15
Figure 2: Comparison between X-IJEE and X-IJAE-11, X-IJAE-L2 models. We see that X-IJA results form pareto frontier with respect to IS and I.PIPS for different choice of
In experiments, for symmetric likelihoods r()[2)) and r'(yaz) we use the following: r(ylx) is continuous distribution over cyclic shifts in directions of an image x. In practice, we discretize this distribution. To sample from it: (i) we sample one of four directions (top, bottom, right, left) equally probable; (ii) then sample the size of shift (maximum size S pixels) from O to
with probabilities 1/ (11
(/(110+11)
1/ 1/(2 1/(++1))
1/ 1)
(iii) as result, we shift an image æ the selected direction in size which sampled. For Forr'(ylz) in practice we observe that the best choice is when ('(xx) is close to delta function 5,, Therefore, we use r' ('(y)x) Nyyl ol) which is clearly symmetric likelihood. We
10 For
r(ylæ:) as implicit likelihood we also studied distribution over small rotations of x, however, we observed that cyclic shifts achieve better results.
Evaluation. We evaluate our model on both generation and reconstruction tasks. The quality of the former assessed using Inception Score (IS) (Salimans et al.) 2016). To calculate these metrics we used the official implementation provided in tensorff foow 1. 13 (Abadi et al. 2015). The reconstruction quality evaluated using LPIPS, proposed by (Zhang et al. 2018). LPIPS compares images based on high-level features obtained by the pre-trat pre-trained network. It was show by /Zhang et al. (2018) that LPIPS is a good metric which captures perceptual similarity between images. We use the official implementation (LPIPS github) to compute LPIPS.
Ablation Study. To show the importance of the implicit conditional likelihood r(y(ar) we compare X-IJAE with its modification which has instead of implicit r(yl=)) standard Gaussian Laplace distribution. We call such models X-IJAE-L2 and X-IJAE-1 respectively. In Figure 2 we compare X-IAA with X-IJAE-L2 and A-IJAE-L terms of IS (generation quality) and LPIPS (reconstruction quality). We see that A-IJ significantly outperforms these baselines and allows to achieve pareto- optimal results for different choice of X.
Comparison with Baselines. We assess generation and reconstruction quality of X-IAA on CIFAR- 10 and TinyImageNe datasets. We compare the results to closest baselines with publicly available code. We provide visual results in Appendix, section C Quantitative results are given in Figure 3 and in Table Figure 3 we compare the methods with respect IS and LPIPS. Considering both metrics X-IJAE achieves better trade-off between reconstruction and generation quality within these
Tinylmagenet
0.20
1.0
1.0
ACC
0.8
0.15
503e
LPIPS
LPIPS
0.2
0.10
0.44
0.4
0.2
0.1
0.05
0.2
4
5
6
4.5
5.0
5.5
6.0
6.5
7.0
Figure 3: Evaluation of X-IAAE on CIFAR and TinylmageNet compared to baselines. Two metries, LPIPS and IS metrics considered to access reconstruction and sampling quality. Considering both metrics X-JAE achieves a better trade-off between reconstruction and sampling quality within datasets.
7

---

Under review as a conference paper at ICLR 2020
Table 1: Reconstruction and generation quality on CIFARIO and TinyImagenet for models that allow reconstructions, Baseline models were trained using publicly available code, if possible, to fill reconstruction quality metries. lower better, f higher is better, best is marked with bold.
Generation Quality
Reconstruction Quality LPIPS
Method
IS t
CIFAR 10
WAF Tolstikhin al.
2017
4.18 + 0.04
ALI (Dumoulan et al.
2017
5.34 0.04
ALICE (Li et al. 2017
6.02 = 0.03
AS- -S-AE (Pu et al. 201
6.3
VAE (resnet)
3,45 = 0.02
0.09 + 0.03
2Stage- VAE Dai & Wipf 2019
3.85 = 0.03
0.06 + 0.03
a-GAN (Rosca et al. 2017
5.20 0.08
0.04 0.02
AGE Ulyanov et al 2018)
5.90 = 0.04
0.06 + 0.02
SVAE CCenn et al.
2018
6.56 0.07
0.19 + 0.08
X-JAE (A 0.3)
6.98 = 0.1
0.07 + 0.03
Tiny Imagenet
AGE Ulyanov al. 2018
6.75 + 0.09
0.27 + 0.09
SVAB (Cenn et al. 2018)
5.09 0.05
0.28 + 0.08
2Stage VAB DDai & Wipf 2019
4.22 0.05
0.09 + 0.05
A-IJ (A 0.3)
6.87 = 0.09
0.09 + 0.03
datasets. We that small values of À give good IS score while remain the decent reconstruction quality in terms LPIPS. However, decrease À further LPIPS will start to degrade. Therefore, we chose the À 0.3 reasonable trade-off between generation and reconstruction ability of X-IJAE. For this choice of X we compute the results for Table I From these Table [ we see that X-IJAE achieves the sate-of-the-an trade-off between generation and reconstruction quality. confirms our justification about X-Jeffrees divergence that it takes the best properties of both KL divergences.
CONCLUSIONS
In the paper, we considered fusion of VAE and GAN models that takes the best of two worlds: has sharp and coherent samples and encode observations into -iwddiensiooal representations. We provide theoretical analysis of our objective and show that it is equivalent to the Jeffreys divergence. In experiments, we demonstrate that our model achieves good balance between generation and reconstruction quality. It confirms our assumption that the Jeffreys divergence is the right choice for learning complex nigh-dimensional distributions in the case of the limited capacity of the model.
REFERENCES
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, lan Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlow: Large-scale machine learning on heterogeneous systems, 2015. URL https: /twp:ttensor..... tensor low. org/ Software available from tensorflow.org.
Felix V Agakov and David Barber. An auxiliary variational method. In International Conference on Neural Information Processing, pp. 561-566. Springer, 2004.
8

---

Under review as a conference paper at ICLR 2020
Martín Arjovsky and Léon Bottou. Towards principled methods for training generative adversarial networks. International Conference on Learning Representations, 2017.
Andrew Brock, Theodore Lim, James M Ritchie, and Nick Weston. Neural photo editing with introspective adversarial networks. International Conference on Learning Representations, 2017.
Liqun Chen, Shuyang Dai, Yunchen Pu, Erjin Zhou, Chunyuan Li, Qinliang Su, Changyou Chen, and Lawrence Carin. Symmetric variational autoencoder and connections to adversarial learning. In Amos Storkey and Fernando Perez-Cruz (eds.), Proceedings the Twenty-First International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pp. 661-669, Playa Blanca, Lanzarote, Canary Islands. 09-11 Apr 2018. PMLR. URL http:://proceeiings mlr. wwwps//w/ssseeppepepepppp chen 8b html
Bin Dai and David Wipf. Diagnosing and enhancing vae models. In International Conference on Learning Representations, May 2019.
Jeff Donahue, Philipp Krähenbühl, and Trevor Darrell. Adversarial feature learning. International Conference on Learning Representations, 2017.
Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, and Aaron Courville. Adversarially learned inference. International Conference Learning Representations, 2017.
Ian Goodfellow. Nips 2016 tutorial: Generative adversarial networks.
arXiv preprint
arXiv:1701.00160, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural informa- fion processing systems, pp. 2672-2680, 2014.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron Courville, Improved training of wasserstein gans. Mar 2017. UtLttt:://arxxx org www./////... http://.b//74400.......
Geoffrey Hinton and Drew Van Camp. Keeping neural networks simple by minimizing the description length of the weights. In in Proc. of the 6th Ann, ACM Conf. on Computational Learning Theory. Citeseer, 1993.
Ferenc Huszar. sarial
An
alternative
update URL
rule for generative adver- /c/
networks,
2016.
an- brw.:paaaatteeeaaa...eeeeeeereea
Harold Jeffreys. The theory of probability. OUP Oxford, 1998.
Diederik P Kingma, Shakir Mohamed, Danilo Jimenez Rezende, and Max Welling. Semi-supervised learning with deep generative models. In Advances in Neural Information Processing Systems, pp. 3581-3589, 2014.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. Cifar-10 (canadian institute for advanced research). 2009. URL http::/www. cs httt:://ww..ss..oootttteeee///te Lto.onto..eeu/--rri/cc...e
html
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using learned similarity metrie. CoRR, 2015.
Chunyuan Li, Hao Liu, Changyou Chen, Yuchen Pu, Liqun Chen, Ricardo Henao, and Lawrence Carin. Alice: Towards understanding adversarial learning for joint distribution matching. In Advances in Neural Information Processing Systems, pp. 5495-5503, 2017.
Lars Maaløe, Casper Kaae Sønderby, Søren Kaae Sønderby, and Ole Winther. Auxiliary deep generative models. arXiv preprint arXiv:1602.085 1602.05473, 2016.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. International Conference Learning Representations, 2016.

---

Under review as a conference paper at ICLR 2020
Lars Mescheder, Sebastian Nowozin, and Andreas Geiger. Adversarial variational bayes: Unifying variational autoencoders and generative adversarial networks. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 2391 -2400, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL http:://pocceeeee ngs. ml press/
#70/mescheder17a.ccc
Luke Metz, Ben Poole, David Pfau, and Jascha Sahh-Dickstein. Unrolled generative adversarial networks. arXiv preprint arXiv: 11/.:///11222 1.02/63, 2016.
Dmitry Molchanov, Valery Kharitonov, Artem Sobolev, and Dmitry Vetrov. Doubly semi-implicit variational inference. arXiv preprint arXh::1810.02770 2018.
Yuchen Pu, Weiyao Wang, Ricardo Henao, Liqun Chen, Zhe Gan, Chunyuan Li, and Lawrence Carin. Adversarial symmetric variational autoencoder. In Advances in Neural Information Processing Systems, pp. 4330-4333, 2017a.
Yuchen Pu, Weiyao Wang. Ricardo Henao, Liqun Chen, Zhe Gan, Chunyuan Li, and Lawrence Carin. Adversarial symmetric variational autoencoder. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 4339-4339. Curran Associates, Inc., 2017b. URL http:://ppppeeeeee nips wew.lc-vvriiiionn cc/paper / 7020 www.aavvvvaarrae -aabeeerr---aaaeeraailllll---- itw.:aa....aaaaaatttaaaaaaaaaaaa
aut pdf
Rajesh Ranganath, Dustin Tran, and David Blei. Hierarchical variational models. In International Conference on Machine Learning, pp. 324-333, 2016.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. JCML, 2014.
Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, and Shakir Mohamed. Variational approaches for auto-encoding generative adversarial networks. arXiv preprint arXiv: 049977 2017.
Tim Salimans, Ian Goodfellow, Wojeiech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in Neural Information Processing Systems, pp. 2234-2242, 2016.
Artem Sobolev and Dmitry Vetrov. Importance weighted hierarchical variational inference. arXiv preprint 1rXiv:1905.0333 03290, 2019.
M. Sugiyama, T. Suzuki, and T. Kanamori. Density Ratio Estimation in Machine Learning. Cambridge books online. Cambridge University Press, 2012. ISBN 9780521 190176. htttst::: //books google. rw/books ? http:/oookk2..xxooolkkkk-.o
Michalis Titsias and Miguel Lázaro-Gredilla. Doubly stochastic variational bayes for non-conjugate
inference. In International conference on machine learning, pp. 1971-1979, 2014.
Ilya Tolstikhin, Olivier Bousquet, Sylvain Gelly, and Bernhard Schoelkopf. Wasserstein auto-encoders. Nov 2017. URL/tttp://axxiv org.
01558v3
Dmitry Ulyanov, Andrea Vedaldi, and Victor S. Lempitsky. It takes (only) two: Adversarial generator- encoder networks. In AAAI. AAAI Press, 2018
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired amage-to-image translation using cycle-consistent adversarial networks. In Computer Vision (ICCV), 2017 IEEE International Conference on, 2017.
10

---

Under review as a conference paper at ICLR 2020
Appendix
APPENDIX A PROOFS
Proposition 1. Mescheder et al. 2017) Let Dv-(x)
((( for any I. Then
((())+po(3)
-Volppo(u) log I-Dw- VaDKI Ip") and VolL log(1 De- (x)())) VoJSD(pollp")
(y-(w)
even if we assume that VoD (z)
Proof. The core of the proof the fact that for any distribution (o we have Bgg(() Ve log (x()) 0. We can write down the derivative for the divergence as a of two terms: the first quantifies dependence the divergence parameters 0 through samples æ and the second one quantifies dependence through log-densities. The first one be captured by optimal discriminator if use pathwise gradients (aka reparametrization trick), while the second one is more challenging.
(a( (z)
f po(c)) log dz
(o(x)
VoDKL (pell"") Volpo(z) log
Ve
p
p" (x
/ Vepo(z) log
(x)
po dr (x)
dx +
Po(x) log
pe
D (1)
(opo(x) log
dx po(x)) Vo log
D (x)
1 - (0)((1))
po((x) Vopa(a) de po(x)
Volpo(e) log
D (Ir)
1 - --D (1)
1 1 --Da (x) De- (x)
VoPpo(z) log
+ Vo po(x)de= = Vol log
(x-
The same can be shown for the -haenn-n divergence:
VeJSD(pellp") voDKL
(Poll po + VoDKL (poll 10
Vo / po((t) log
po(
de + Vo / (0(())log
p'(x)
dr + log 4
p" p0())
p" po po()))
(x(x)) po (x) dr / Vo log
po(x)
po(x)
Vopo(z) log
dx
p" (0 + po(x)
p (x) log(p" (x) + pa dda + 1og
/ Vopo(T) - D (x) - / po(x) (x) + pe(x))
+ / po(o)VologPo(t)dd / (= (x) + (o())) + log4
VoEpo(a) log(1 - Dos (x:) -
po(x))x' (2) Vo log po(x)+p" (w) dr + log
2
BoBpolo) log(1 De (z:) log
Theorem 1. Let (z,(2,2,)) be the optimal solution for the objective and r(y|æ)
and ('(y)z) are symmetric likelihoods. Then Vell, (x) E (=)1) log T D. (Z(2. VolEpp (as()))) Go(=)..
D Gaf..))))
11

---

Under review as a conference paper at ICLR 2020
Proof.
D. 2, Go(z))
E Da (x))) Vo log r(Ga(z)(a)
(x) log - D,- 2,, Gg(z])
r((a(())aa -=))
= B V (G (z)(z) Epp (a) (elm)) [Vo log (G)((
Now we will show the second term equal to zero given our assumptions:
['Go((2)a))aa voGo(=)
(x) (ala) Vo log (Go))
PBp (x) B (a/we
G(G)())Goo())
Where have used the (1) and (2) properties of the likelihoods r(vly) (Definition
(d) =0
APPENDIX B FIGURE SETUP
To generate the plot I we considered the following setup: target distribution mixture: (=(x) (z1 8, 0.23) + (zr - 3, 0.8²) + 0.3N 1) + (x(8,
While the model as an equiprobable mixture of two learnable Gaussians:
(a(x)= = (xx,, + 0.5.((z (=,,())))
The optimal / was found by making 10,000 stochastic gradient descent iterations on Monte Carlo estimations of the corresponding divergences with a batch size of 1000. We did 50 independent runs for each method explore different local optima and chose the best one based on a divergence estimate with 100,000 samples Monte Carlo samples.
APPENDIX C IMAGES
12

---

Under review as a conference paper at ICLR 2020
(a) Real Data
(b) 0.2-IAA
a-GAN
(d) AGE
(e) TwoStage-VAAs VAE
SVAE
Figure 4: Samples from models trained on CIFARIO dataset. Images for baselines were obtained running publicly available code.
Original
IJAE
a-GAN
AGE
2SVAE
VAE
SVAE
Figure 5: Reconstructions on the CIFARIO dataset for IJA model and closest baselines. Reconstruc- tions for baselines were obtained running publicly available code.
13

---

Under review as a conference paper at ICLR 2020
(a) Real Data
(b) IJAE
(c) AGE
(d) TwoStage-VAn VAE
(c) SVAE
Figure 6: Samples from models trained on Imnylmagenet dataset. Images for baselines were obtained running publicly available code.
Original
IJAE
AGE
2SVAE
SVAE
Figure 7: Reconstructions on the Tinylmagenet dataset for 1JAE model and closest baselines. Recon- structions for baselines were obtained running publicly available code.
14

---

Under review as a conference paper at ICLR 2020
APPENDIX D
NETWORK ARCHITECTURES AND HYPERPARAMETERS
Table Hyperparameters used experiments
Parameter
Value
Generator, Encoder Optimizer betas (0.5, 0.9) learning rate weight decay
Adam
0.0001
0
latent dimension
128
Discriminators Optimizer
Adam
betas = (0.5, 0.9)
learning rate weight decay
0.0001
Table 3: Encoder Network Architecture
Shortcuts
Basic Res Block
same as original Resnet
Inject Noise
Linear(128, num 1hammes3)))rmmml((s)211131)) + image
Layer
Parameters
Basic Res Block
in channels out channels 128, stride 2
Inject Noise
Basic Block
in channels 128, out channels 128, stride
Inject Noise
Basic Block
channels 128, out channels 128, stride
Basic Block
in channels = 128, out channels 128, stride = 2
ReLU
Inject Noise
Flatten
Linear
in features 512, out features 128
#Weights 6436224
Table 4: Generator Network Architecture
Shortcuts
Upsample Basic Res Block
same as in original Resnet with upsample instead of strided convolution
Layer
Parameters
Linear
in features 128, out features 2048
Reshape to Image
shape (128, 4, 4)
Upsample Basic Res Block
in channels 128, out channels 128, scale factor 2
Upsample Basic Res Block
in channels 128, out channels = 128, scale factor = 2
Upsample Basic Res Block
in channels 128, out channels 128, scale factor 2
Batch Norm
ReLU
Conv3x3
stride I, padding
Tanh
#1e4ghts=1154179
15

---

Under review as a conference paper at ICLR 2020
Table Single Image Discriminator Network Architecture
Shortcuts
Basic Res Block
same as in original Resnet Linear(122) num chamnelsynnormallsii--11111 + image
Inject Noise
Layer
Parameters
Basic Res Block
in channels 3, out channels 128, stride 2
Inject Noise
Basic Block
in channels 128, out channels 128, stride 2
Inject Noise
Basic Block
in channels 128, out channels 128, stride
Basic Block
channels 128, out channels 128, stride
ReLU
Global Average Pooling
Linear
in features 128, out features
#Weights 1053825
Table 6: Pair Image + Latent Discriminator Network Architecture
Shortcuts
Basic Res Block
same in original Resnet
Inject Noise
Linear(128 (128, num
+ image
Inputs:
left image, right image, latent code
Layer
Parameters
Stack Images
orient horizontal
Basic Res Block
in channels = 3, out channels 128, stride 2
Inject Noise
Basic Block
in channels 128, out channels 128, stride 2
Inject Noise
Basic Block
in channels 128, out channels 128, stride
Basic Block
channels 128, out channels 128, stride
ReLU
Global Average Pooling
Concat Latent Code
Linear
features 256, out features =
#Weights 1054081
Table 7: Image + Latent Discriminator Network Architecture
Layer
Parameters
Basic Block LeakyRel.U(0.2) Basic Block
kernel size 4 in channels 3, channels 64, stride 2. padding
kernel size = 4 in channels = 64, out channels = 128, stride = 2, padding =
LcakyReLU(O.22) Flatten
Linear
in features 8320, out features = 512
LeakyReL.U(O.2) Linear (ea.)Rel.U(0.2) Linear
in features 512, out features 256
in features 512, out features 256
#Weights 4526273
16


Reviews:
[{"review_id": "ryeRzOnX9S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new training objective for generative models that combines the objectives of VAEs and GANs. The objective is equivalent to minimizing the Jeffreys divergence (a type of f-divergence) between the true probability of the data and its probability under the model.  Furthermore, the objective comes with a knob to tradeoff the relative importance of each of the two terms.  In addition, the authors develop a implicit likelihood formulation which they claim and show empirically to outperform typical explicit formulations typically used in VAEs. \n\nOverall, it is an interesting paper that reuses a few good ideas to develop a novel training objective. The results show that using an implicit likelihood helps (Figure 2) and that it does relatively better than either GAN or VAE approaches. I have detailed comments below about the organization of the paper, some of the experimental claims as well as a few other works which may be good to cite. \n\n\n- Paper organization: I would suggest moving the related work to after the background.\n\n- GANs and VAEs are not models per se but rather training frameworks for generative models.\n\n- While VAEs and GANs can work on many types of data (at the very least continuous), your model seems to be developed for images. Could you make it clear what changes would be needed to apply it to non-image data?\n\n- There are many minor grammatical errors throughout the text.\n\n- It would be useful to provide the full algorithm somewhere (e.g., using an algorithm \"box\")\n\n- Possible related work. It may be worth citing these two paper:\n  - f-GAN: Training Generative Neural Samplers using Variational Divergence Minimization, NIPS'16\n  - Deep Generative Learning via Variational Gradient Flow, ICML'19\n \n- It would be useful to mention early that for IS higher is better and LPIPS lower is better.\n\n- Even though Figures 2 and 3 (to a certain extent) seem to show that results are somewhat robust to the exact value \\lambda how would you propose to set it in practice?\n\n- Figure 3 Left (CIFAR 10), it's not absolutely clear to me that alpha-GAN and perhaps AGE isn't at least as good as your approach. The meaning of the units of the axes is a bit unclear. Do you have a particular reason to prefer your method over these in this case?\n\n  Related: in Table 1, why are there no bolded results for CIFAR + Reconstruction?\n\n- In Figures 5 and 7 the reconstruction of IJAE sometimes seems to be pretty far from the original image (i.e., it's not that it's blurry as for VAEs, it's that the model seems to be reconstructing a completely different image). How do you explain these results?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SygnRBCatS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a model named lambda-IJAE, which combines the VAE and GAN training schemes to train a generative model achieving competitive performance. The combination of VAE and GAN is justified by its theoretical interpretation as a an optimization of the lambda-Jeffreys divergence between the real data distribution and the generation distribution. This work also introduces a reformulation of the reconstruction term of the VAE loss, allowing it to be estimated implicitly using an adversarial mechanism. Finally, the latent space of the VAE is also modelled implicitly using an adversarial mechanism, following (Mescheder et al. 2017).\n\nI am ambivalent about this paper. The proposed implicit likelihood mechanism is very interesting, but the paper contains several weaknesses that together make me unwilling to accept it.\n\nFirst of all, the paper presents itself as centered on the notion of optimizing the lambda-Jeyffreys distribution, while the main contribution is actually clearly the formulation of the implicit likelihood. The use of a weighted sum of the forward & KL divergences to train a generative model is hardly new, and has already been presented a few times (Larsen et al. 2015, Dosovitskiy & Brox 2016).\n\nIn this context the paper does not present the impact of its main contribution alone. How would behave a VAE trained solely with this implicit likelihood, but a regular Gaussian latent space and without the GAN loss? This ought to be part of the ablation study in my opinion.\n\nSecondly, the paper discusses the issue of VAE generating unrealistic samples. This is indeed a very real issue of the VAE linked to it being trained by maximum-likelihood. However illustrating it by \"blurry images\" (like is done several times in the paper) is a common misconception, as while this is a very classical issue with VAEs, it is mostly unrelated to the MLE estimation.\n\nIt is rather a simple consequence of the fact that using an unweighted squared error loss to model the reconstruction of the VAE is almost always a poor model. It is equivalent to modelling the observation with a Gaussian noise of variance 1/2, which is a huge noise when considering data normalized in [0;1] or [-1;1] like is traditional to do with images. Reducing this variance to a more sensible value (like a std of 0.1 for example) or allowing the model to learn it reveals the real failure mode of the VAE generating unrealistic images, which can hardly be described as \"blurry\".\n\nSimilarly, the ablation study evaluates the use of L1 or L2 noise instead of the cyclic shift likelihood, but does not say what variance has been used for these, which would (as explained above) be an important parameter to take into account. If a variance of 1 was used, then the results of figure 2 are unsurprising and not insightful, as the discriminator would have merely learned to differentiate between images containing a visible Gaussian noise from images that do not.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "SJgy_5i_OH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes to replace the KL-divergence in VAE training with the lambda-jeffreys divergence of which the symmetric KL-divergence is a special case. The paper proposes a pure implicit likelihood approach that uses three discriminator models to estimate the KL-divergences. Experiments are conducted on CIFAR-10 and TinyImageNet and several scores are reported to show that the proposed method performs as good if not better than current approaches.\n---------\nI think the paper tries to achieve too much in too little space and foregoes scientific exactness for the sake of claiming SOTA. Since there is a difference between claiming SOTA on a task and validating a new method, the small amount of space makes it difficult to substantiate both claims at the same time. In the rest of the review i will try to substantiate the claim:\n\n1. The paper claims on page 2: \"These models do not have a sound theoretical justification about what distance [...] they optimize\". While the paper tries to substantiate its claims by showing theoretically that it does the right thing using the optimal discriminator, it leaves the question open what happens with any other discriminator. The theory does not justify non-optimal solutions. It is argued on page 6 that non-optimality of the discriminator serves as some form of regularization, but  this requires some justification.\nMoreover, the paper uses LPIPS to measure reconstruction quality - but this measure is a deep neural network. So if those measures are good enough to compare solutions with and the theoretical justification of the proposed method is shaky in practice - why not use LPIPS for training?\n\n2. The paper proposes the discriminator in order to allow for an implicit likelihood. However, the r-function used in the experiments does not fulfill the property of a well defined likelihood, and Theorem 1 does not hold, since technically the KL-divergence is infinity. If we ignore this by adding a small amount of Gaussian noise around the sampled cyclical shifts - like the r' used in the experiments, we can easily write down the explicit likelihood function since:\n\nr(y|x)=\\sum_i w_i N(y|Shift_i(x), \\sigma)\n\nwhere Shift_i is the i-th shift in the set described in the paper and w_i its probability  p(y|q). So the explicit solution of theorem 1 can be written down and another ablation study would be training the method with the explicit formulation for this KL-term(i.e. only training two discriminator models). If the results are not equivalent, this implies that the discriminator does not reach the optimum. The implications of that should be discussed regarding 1. \n\n3.  Existing ablation studies are a bit of a straw-man: the paper compares changing r(y|x) by standard Gaussian or Laplace. However, we know that a large variance does not make any sense and almost all papers use tiny variances (e.g. in beta-VAE the beta-values tend to be very small, which is equivalent to small variances here). \n\n\n---------------------------\nSmaller things\n- Are the experimental results all with the same architecture for encoder/generator for all results you compared to? if not, the effect of that should also be tested.\n\n- my personal biased view on the generated images is: it looks worse than alpha-GAN. Every reconstructed image has a grey tone and the generated images also offer a strong grey palette. The details don't look better as well.\n\n- typo inroduce->introduce\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received Weak Reject scores from all three reviewers. The AC has read the reviews and lengthy discussions and examined the paper. AC feels that there is a consensus that the paper does not quite meet the acceptance threshold and thus cannot be accepted. Hopefully the authors can use the feedback to improve their paper and resubmit to another venue.
Decision: Reject