Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: The format of this paper is not ICLR format.

The spherical interpolation makes an implicit assumption that all data points lie on a sphere, which maybe a better assumption than linear space for Gaussian distributed data but not for uniform distributions.  Also the theta parameter in spherical interpolation looks arbitrary, I wonder how is it chosen?
review_title: formatting and others

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a variety of techniques for visualizing learned generative models, focussing specifically on VAE and GAN models. This paper is somewhat challenging to assess since it doesn't propose a new algorithm, model, application etc. On the one hand these techniques will be highly relevant to the generative modeling community and I think this paper deserves a wide audience. The techniques proposed are simple, well explained, and of immediate use to those working on generative models. However, I'm not sure the paper is appropriate for an ICLR conference track as it doesn't provide any greater theoretical insights into sampling generative models and there are no comparisons / quantitative evaluations of the techniques proposed. Overall, I'm very much on the fence since I think the techniques are useful and this paper should be read by those interested in generating modeling. I would be willing to increase my core if the author could present a case for why ICLR is an appropriate venue for this work.
review_title: review

final_score: rating: null
review_content: Could you please explain why the spherical interpolation is better when the prior is uniform?
review_title: pre-review question

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposed a set of different things under the name of "sampling generative models", focusing on analyzing the learned latent space and synthesizing desirable output images with certain properties for GANs.  This paper does not have one single clear message or idea, but rather proposed a set of techniques that seem to produce visually good looking results.  While this paper has some interesting ideas, it also has a number of problems.

The spherical interpolation idea is interesting, but after a second thought this does not make much sense.  The proposed slerp interpolation equation (page 2) implicitly assumes that the two points q1 and q2 lie on the same sphere, in which case the parameter theta is the angle corresponding to the great arc connecting the two points on the sphere.  However, the latent space of a GAN, no matter trained with a uniform distribution or a Gaussian distribution, is not a distribution on a sphere, and many points have different distances to the origin.  The author's justification for this comes from the well known fact that in high dimensional space, even with a uniform distribution most points lie on a thin shell in the unit cube.  This is true because in high-dimensional space, the outer shell takes up most of the volume in space, and the inner part takes only a very small fraction of the space, in terms of volume.  This does not mean the density of data in the outer shell is greater than the inner part, though.  In a uniform distribution, the data density should be equal everywhere, a point on the outer shell is not more likely than a point in the inner part.  Under a Gaussian model, the data density is on the other hand higher in the center and much lower on the out side.  If we have a good model of data, then sampling the most likely points from the model should give us plausible looking samples.  In this sense, spherical interpolation should do no better than the normally used linear interpolation.  From the questions and answers it seems that the author does not recognize this distinction.  The results shown in this paper seem to indicate that spherical interpolation is better visually, but it is rather hard to make any concrete conclusions from three pairs of examples.  If this is really the case then there must be something else wrong about our understanding of the learned model.

Aside from these, the J-diagram and the nearest neighbor latent space traversal both seems to be good ways to explore the latent space of a learned model.  The attribute vector section on transforming images to new ones with desired attributes is also interesting, and it provides a few new ways to make the GAN latent space more interpretable.

Overall I feel most of the techniques proposed in this paper are nice visualization tools.  The contributions however, are mostly on the design of the visualizations, and not much on the technical and model side.  The spherical interpolation provides the only mathematical equation in the paper, yet the correctness of the technique is arguable.  For the visualization tools, there are also no quantitative evaluation, maybe these results are more art than science.
review_title: A mixture of many things

final_score: rating: 6: Marginally above acceptance threshold
review_content: In this paper the authors propose various techniques to sample visualizations from generative models with high dimensional latent spaces like VAEs and GANs. For example, the authors highlight the well known but often not sufficiently appreciated fact that the probability mass of high dimensional Gaussian distributions concentrates near a thin hyper-shell with a certain radius. They therefore propose to use spherical interpolations (great arcs) instead of the commonly used linear interpolations. In a similar spirit they propose a visualisation for analogies and techniques to reinforce structure in VAE latent spaces.

I find it hard to give clear recommendation for this paper: On the one hand I enjoyed reading it and I might want use some of the proposals (e.g. spherical interpolations; J-diagrams) in future work of mine. On the other hand, it’s obvious that this paper is not a typical machine learning paper; it does not propose a new model, or training method, or provide (theoretical/empirical) insight and it does not have the scientific quality and depth I’ve seen in many other ICLR submissions. But it does more than just describing useful “tricks”. And all things considered I think this paper deserves a wider audience (but  I'm not convinced that ICLR is the right venue)

review_title: null

final_score: rating: null
review_content: I (obviously) agree that we expect a majority of the probability mass to be concentrated near a shallow shell in the high dimensional latent space and that this fact merits discussion because it is often overlooked and not taken into consideration.

But did you actually test whether this assumption indeed holds for the (average) approximate posterior q(h|x), and not just for the prior p(h)? This could otherwise indicate an interesting mismatch between the prior and the model. A simple histogram over the L2 norms of samples from the approx. posterior might be sufficient here.
review_title: Shallow-shell assumption for the approx. posterior

Meta-review: This paper proposes some interesting ideas about visualizing latent-variable models. The paper is nicely written and presented, but the originality and importance of the work isn't enough. Also, neither the reviewers nor I were convinced that spherical interpolation makes more sense than linear interpolation.
Decision: Reject