Title: Learning in Implicit Generative Models
Authors: Shakir Mohamed
Balaji Lakshminarayanan
Venue: ICLR 2017
Keywords: Unsupervised Learning

Abstract: Generative adversarial networks (GANs) provide an algorithmic framework for constructing generative models with several appealing properties: they do not require a likelihood function to be specified, only a generating procedure; they provide samples that are sharp and compelling; and they allow us to harness our knowledge of building highly accurate neural network classifiers. Here, we develop  our understanding of GANs with the aim of forming a rich view of this growing area of machine learning---to build connections to the diverse set of statistical thinking on this topic, of which much can be gained by a mutual exchange of ideas. We frame GANs within the wider landscape of algorithms for learning in implicit generative models---models that only specify a stochastic procedure with which to generate data---and relate these ideas to modelling problems in related fields, such as econometrics and approximate Bayesian computation. We develop likelihood-free inference methods and highlight hypothesis testing as a principle for learning in implicit generative models, using which we are able to derive the objective function used by GANs, and many other related objectives. The testing viewpoint directs our focus to the general problem of density ratio estimation. There are four approaches for density ratio estimation, one of which is a solution using classifiers to distinguish real from generated data. Other approaches such as divergence minimisation and moment matching have also been explored in the GAN literature, and we synthesise these views to form an understanding  in terms of the relationships between them and the wider literature, highlighting avenues for future exploration and cross-pollination.

TL;DR: Showing connections between GANs, ABC, ratio estimation and other approaches for learning in deep generative models.

Review discussion:
Thank you for an interesting read.
I guess all the theory presented are already around but I still like the presentation which walked through each route in a very clear way.
The only question from a practitioner perspective is: which method should I choose when I want to learn an implicit model? Or can you comment a bit on the difficulty of training for different objectives?
“Which method should I choose when I want to learn an implicit model? Or can you comment a bit on the difficulty of training for different objectives?”
This question is similar to the question of “which approximate inference method should I use for my model?”. And the answer is,  (perhaps not surprisingly) it really depends on the problem. We don’t yet have a clear understanding of if and when one method is necessarily theoretically superior to another for all possible models (and the family of implicit models is huge!). There are some practical differences, e.g. a classifier as a discriminator is more amenable to minibatch training as opposed to moment matching which could affected by minibatch noise;
hence, if the dataset is large, classifier approach might be preferable to moment matching. One could also think of combining multiple discriminator losses; this could help avoid the generator overfitting to any particular discriminator.
Thank you for an interesting read.
Given the huge interest in generative modelling nowadays, this paper is very timely and does provide very clear connections between methods that don't use maximum likelihood for training. It made a very useful observation that the generative and the discriminative loss do **not** need to be coupled with each other. I think this paper in summary provides some very useful insights to the practitioners on how to select the objective function to train the implicit generative model.
The only reason that I decided to hold back my strong acceptance recommendation is that I don't understand the acceptance criteria of ICLR.
First this paper has the style very similar to the Sugiyama et al. papers that are cited (e.g. presenting in different perspectives that were all covered in those papers but in a different context), making me unsure about how to evaluate the novelty.
Second this paper has no experiment nor mathematical theorem, and I'm not exactly sure what kinds of contributions the ICLR committee is looking for.
Thank you for your positive comments. Our aim is to provide clarity, share a different perspective, and to highlight all the related work addressing the same problem that is not otherwise cited and used. We believe there is both novelty in our presentation, and is a good fit and of relevance to ICLR. Please see the more detailed response we provide in general rebuttal to the paper, since this was highlighted by other reviewers as well.
The work of Sugiyama et al. is especially important and one that has been a great inspiration to us. It is important to keep in mind that estimating the density ratio is only part of the problem when learning in GANs and implicit generative models. Showing how this can be used to form both the ratio and generative losses is an important aspect of this paper.
The authors present a great and - as far as I can tell - accurate and honest overview of the emerging theory about GANs from a likelihood ratio estimation/divergence minimisation perspective. It is well written and a good read, and one I would recommend to people who would like to get involved in GANs.
My main problem with this submission is that it is unclear what precisely the novelty is - beyond perhaps articulating these views better than other papers have done in the past. A sentence from the paper "But it has left us unsatisfied since we have not gained the insight needed to choose between them.” summarises my feeling about this paper: this is a nice 'unifying review’ type paper that - for me - lacks a novel insight.
So my assessment is mixed: I think this is a great paper, I enjoyed reading it. I was left a bit disappointed by the lack of novel insight, and this is why I’m not highly confident this has to be a conference paper. I am open to be convinced either way.
Detailed comments:
I think the authors should probably discuss the connection of Eq. (13) to KLIEP: Kullback-Leibler Importance Estimation by Shugiyama and colleagues.
I don’t quite see how the part with equation (13) and (14) fit into the flow of the paper. By this point the authors have established the view that GANs are about estimating likelihood ratios - and then using these likelihood ratios to improve the generator. These paragraphs read like: we also tried to derive another particular formulation for doing this but we failed to do it in a practical way.
Typo in Csiszar divergence
Equation (15) is known (to me) as Least Squares Importance Estimation by Kanamori et al (2009).
I have stumbled upon the work of Sugiyama and collaborators on direct density ratio estimation before, and I found that work very insightful. Generally, while some of this work is cited in this paper, I felt that the authors could do more to highlight the great work of this group, who have made highly significant contributions to density ratio estimation, albeit with a different goal in mind.
I think the hypothesis testing angle is a bit oversold in the paper.  I’m not sure what additional insight is gained by mixing in some hypothesis testing terminology.
Other than using quantities that appear in hypothesis testing as tests statistics, his work does not really talk about hypothesis testing, nor does it use any tools from the hypothesis testing literature. In this sense, this paper is in contrast with Sutherland et al (in review for ICLR) who do borrow concepts from two-sample testing to optimise hyperparameters of the divergence used.
The paper provides an exposition of multiple ways of learning in implicit generative models, of which generative adversarial networks are an example. The paper is very clear, the exposition is insightful, and the presented material is clearly important.
It is hard to assess "novelty" of this work, as the individual pieces are not novel, and yet the exposition of all of them in the same space with clear outline of the connections between them is novel.
I believe this work is significant - it provides a bridge for language and methods used in multiple parts of statistics and machine learning. This has the potential to accelerate progress.
I recommend publishing this paper at ICLR, even though it is not the "typical" paper that get published at this conference (in that it doesn't offer empirical validation, nor makes a particular claim about relative merits of different methods).
Thank you for your positive and encouraging feedback!
I just noticed I submitted my review as a pre-review question - sorry about this. Here it is again, with a few more thoughts added...
The authors present a great and - as far as I can tell - accurate and honest overview of the emerging theory about GANs from a likelihood ratio estimation/divergence minimisation perspective. It is well written and a good read, and one I would recommend to people who would like to get involved in GANs.
My main problem with this submission is that it is hard as a reviewer to pin down what precisely the novelty is - beyond perhaps articulating these views better than other papers have done in the past. A sentence from the paper "But it has left us unsatisfied since we have not gained the insight needed to choose between them.” summarises my feeling about this paper: this is a nice 'unifying review’ type paper that - for me - lacks a novel insight.
In summary, my assessment is mixed: I think this is a great paper, I enjoyed reading it. I was left a bit disappointed by the lack of novel insight, or a singular key new idea which you often expect in conference presentations, and this is why I’m not highly confident about this as a conference submission (and hence my low score) I am open to be convinced either way.
Detailed comments:
I think the authors should probably discuss the connection of Eq. (13) to KLIEP: Kullback-Leibler Importance Estimation by Shugiyama and colleagues.
I don’t quite see how the part with equation (13) and (14) fit into the flow of the paper. By this point the authors have established the view that GANs are about estimating likelihood ratios - and then using these likelihood ratios to improve the generator. These paragraphs read like: we also tried to derive another particular formulation for doing this but we failed to do it in a practical way.
There is a typo in spelling Csiszar divergence
Equation (15) is known (to me) as Least Squares Importance Estimation by Kanamori et al (2009). A variant of least-squares likelihood estimation uses the kernel trick, and finds a function from an RKHS that best represents the likelihood ratio between the two distributions in a least squares sense. I think it would be interesting to think about how this function is related to the witness function commonly used in MMD and what the properties of this function are compared to the witness function - perhaps showing the two things for simple distributions.
I have stumbled upon the work of Sugiyama and collaborators on direct density ratio estimation before, and I found that work very insightful. Generally, while some of this work is cited in this paper, I felt that the authors could do more to highlight the great work of this group, who have made highly significant contributions to density ratio estimation, albeit with a different goal in mind.
On likelihood ratio estimation: some methods approximate the likelihood ratio directly (such as least-squares importance estimation), some can be thought of more as approximating the log of this quantity (logistic regression, denoising autoencoders). An unbiased estimate of the ratio will provide a biased estimate of the logarithm and vice versa. To me it feels like estimating the log of the ratio directly is more useful, and in more generality estimating the convex function of the ratio which is used to define the f-divergence seems like a good approach. Could the authors comment on this?
I think the hypothesis testing angle is oversold in the paper.  I’m not sure what additional insight is gained by mixing in some hypothesis testing terminology.
Other than using quantities that appear in hypothesis testing as tests statistics, his work does not really talk about hypothesis testing, nor does it use any tools from the hypothesis testing literature. In this sense, this paper is in contrast with Sutherland et al (in review for ICLR) who do borrow concepts from two-sample testing to optimise hyperparameters of the divergence used.
Thank you for your positive comments. Our aim is to provide clarity, share a different perspective, and to highlight all the related work addressing the same problem that is not otherwise cited and used. We believe there is both novelty in our presentation, and is a good fit and of relevance to ICLR. Please see the more detailed response we provide in general rebuttal to the paper, since this was highlighted by other reviewers as well.
The work of Sugiyama et al., is especially important and one that has been a great inspiration to us. We aim to cite it as much as possible, citing the book as our key source (rather than individual papers and specific names such as KLIEP and LSIE). It is important to keep in mind that estimating the density ratio is only part of the problem when learning in GANs and implicit generative models, and why our discussion has taken the form it has.
“I think it would be interesting to think about how this function is related to the witness function commonly used in MMD and what the properties of this function are compared to the witness function - perhaps showing the two things for simple distributions.”
Sriperumbudur et al showed in the paper “On the empirical estimation of integral probability metrics” 2012 that f-divergences and integral probability metrics (MMD, Wasserstein) intersect only at the total variation distance. See also Arthur Gretton’s slides at the NIPS adversarial training workshop:  http://www.gatsby.ucl.ac.uk/~gretton/papers/testing_workshop.pdf
For likelihood ratio estimation, we do not want to estimate the convex function of the ratio directly, since this will not allow us to exploit the duality properties of the divergence.
The importance of hypothesis testing is perhaps a philosophical one. Recognising the role of hypothesis testing (to test if two distributions are equal) helps to ask the question of why we should compute the density ratio or density difference, which is otherwise vague and unclear. Once this is recognised, the path to using the myriad of methods for comparing two hypotheses such as two-sample tests, density ratios, density-differences, and classical tests such as Wald and likelihood ratio test, are all valid tools for learning implicit models. This view has been recognised by other authors as well, e.g., Lopez-Paz and Oquab (2016).
Thanks for your detailed response and the pointers to the slides. I want to reiterate both to authors and to fellow reviewers/area chairs, that I do like this review and I think it is valuable to have it. I agree with other reviewers in that it is hard to evauate this paper as a conference submission, which also explains my relatively positive review with a not so good score.
I would argue for accepting this paper, unless it would be at the cost of rejecting excellent papers that present more original ideas.
The reviewers have two common concerns (1) relevance of this paper to ICLR, and (2) its novelty. We address the common concerns here and address other questions individually.
The aim of our paper is to review different approaches for learning in implicit generative models; GANs are a special case of implicit generative models and our work helps understand connections between GAN variants as well as understand how GANs are related to the wider statistical literature. We are not aware of any other work in the GAN literature which discusses the connections to approximate Bayesian computation, moment-matching and two sample-testing, unsupervised-as-supervised learning, optimal transport, likelihood free inference, or non-maximum likelihood estimation, even though these methods are clearly related. We believe these connections would be interesting to ICLR community for at least two reasons: (1) we can borrow tools from the related literature to improve optimisation and analyse convergence and (2) we can use GAN-like techniques to train differentiable simulators in other application domains (e.
g high energy physics, economics, ecology), thereby opening up exciting research directions. AnonReviewer4 writes “I believe this work is significant - it provides a bridge for language and methods used in multiple parts of statistics and machine learning. This has the potential to accelerate progress. ” And AnonReviewer1 writes “It is well written and a good read, and one I would recommend to people who would like to get involved in GANs. ” This was our aim, and why this paper is of relevance to ICLR.
Novelty: The view of hypothesis testing gives an insight about the learning principle used in likelihood free / implicit models. This view also helps understand connections between variants of GANs;
for instance, methods that use the density ratio (e.g. the original GAN paper, f-GAN and b-GAN) or the density difference (e.g. generative moment matching networks). To the best of our knowledge, this unifying view through hypothesis testing is novel (see also the concurrent submissions by Lopez-paz and Oquab (2016) and Uehara et al. (2016)). To quote AnonReviewer4, “the individual pieces are not novel, and yet the exposition of all of them in the same space with clear outline of the connections between them is novel.”
The unifying view opens up lots of interesting research directions, e.g., since our initial arxiv submission, Kareletsos (2016) has published “Adversarial Message Passing For Graphical Models” https://arxiv.org/pdf/1612.05048.pdf where he proposes GAN-like techniques for minimising divergence for message passing in graphical models.
Hello Authors,
Congratulations on the acceptance of the paper.
I've just reread parts of the revised paper and noticed a few things that you might want to consider and change before the camera-ready deadline.
* You now include a reference to KLIEP after Eqn. (16), but this procedure is in fact known as least-squares importance estimation.
* in turn, Eqn. (14) is actually more akin to KLIEP, the main difference being the use of the unnormalised form of the KL-divergence.
So I think you meant to put the KLIEP reference here.
Further comment on making Eqn. (14) practical:
If you read the KLIEP paper, they formulate the procedure as a constrained optimisation problem:
maximise
E_{p^*} log r_\phi(x)
subject to the constraint that
E_{q_\theta} r_\phi(x) = 1
Compare this constrained optimisation to your solution, it is easy to make the connection: if you introduce a Lagrange multiplier to handle the constraint, one obtains the following unconstrained optimisation problem:
seek stationary points of
\ell(\phi, \lambda) = E_{p^*} log r_\phi(x) - \lambda E_{q_\theta} (r_\phi(x) - 1)
I do think that solving this unconstrained optimisation problem is actually possible, you can do that via stochastic gradient descent, and it does not include your nasty cross-entropy term.
What am I missing?
Thanks,
Rev1
Hi,
The references are indeed mixed up, we will fix them.
Not sure if we understand your question correct, but I assume you are referring to the paragraph below (14)?
Eq (14) gives us a ratio loss (similar to KLIEP). There is no issue optimizing this with respect to r. The issue is the optimization with respect to q --- this is not possible due to the log q term which is not available for implicit models.
Balaji
PS: Our paper wasn't accepted to the conference btw.
Hi,
Apologies for both mistakes.
Regarding Eqn. (14): indeed, I have misread what you wrote in the paper, my bad. I remember going down exactly the same rabbit hole before...
Re: acceptance, I got confused as I've only read the subtitle of the PC's comment "ICLR 2017 conference paper187 acceptance", not the actual decision.

Meta-review:
role: PC
data: comment: This paper provides a unifying review of various forms of generative model. The paper offers some neat perspectives that could encourage links between areas of machine learning and statistics. However, there aren't specific new proposals, and so there is no empirical evaluation either. The PCs thus believe this contribution is more appropriate for the Workshop Track.

Decision: Invite to Workshop Track