Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Authors note that models may be trained for a certain distribution (e.g. uniform or Gaussian) but then "used" by interpolating or jittering known examples, which has a different distribution.  While the authors are clear about the fact that this is a mismatch, I did not find it well-motivated why it was  "the right thing to do" to match the training prior, given that the training prior is potentially not at all representative or relevant. The fact that a Gaussian/prior distribution is used in the first place seems like a matter of convenience rather than it being the "right" distribution for the problem goals, and that makes it less clear that it's important to match this "convenience" distribution. The key issue I had throughout is "what is the real-world problem metric or evaluation criteria and how does this proposal directly help"?

For example, authors cover the usual story that random Gaussian examples  lie on a thin sphere shell in high-d space, and thus interpolation of those examples will like on a thin shell of slightly less radius.  In contrast, the Uniform distribution on a hypercube [-1,1]^D in D dimensions "looks" like a sharp-pointy star with 2^D sharp points and all the mass in those 2^D corners. But the key question is, what are these examples being used for, and what are the trade-offs between interpolation (which tends to be fairly safe) and extrapolation of the given examples?

This is echoed in the experiments, which I found unsatsifactory for the same key issue: "What is the criteria for “higher-quality interpolated samples”? in the examples they give, it seems to be the sharpness of the images. Is that realistic/relevant? These are pretty images, but the evaluation criteria is unclear.



review_title: Evaluation Metric and Actual Problem Being Solved Unclear

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors demonstrate experimentally a problem with the way common latent space operations such as linear interpolation are performed for GANs and VAEs. They propose a solution based on matching distributions using optimal transport. Quite heavy machinery to solve a fairly simple problem, but their approach is practical and effective experimentally (though the gain over the simple SLERP heuristic is often marginal). The problem they describe (and so the solution) deserves to be more widely known.

Major comments:

The paper is quite verbose, probably unnecessarily so. Firstly, the authors devote over 2 pages to examples that distribution mismatches can arise in synthetic cases (section 2). This point is well made by a single example (e.g. section 2.2) and the interesting part is that this is also an issue in practice (experimental section). Secondly, the authors spend a lot of space on the precise derivation of the optimal transport map for the uniform distribution. The fact that the optimal transport computation decomposes across dimensions for pointwise operations is very relevant, and the matching of CDFs, but I think a lot of the mathematical detail could be relegated to an appendix, especially the detailed derivation of the particular CDFs.

Minor comments:

It seems worth highlighting that in practice, for the common case of a Gaussian, the proposed method for linear interpolation is just a very simple procedure that might be called "projected linear interpolation", where the generated vector is multiplied by a constant. All the optimal transport theory is nice, but it's helpful to know that this is simple to apply in practice.

Might I suggest a very simple approach to fixing the distribution mismatch issue? Train with a spherical uniform prior. When interpolating, project the linear interpolation back to the sphere. This matches distribution, and has the attractive property that the entire geodesic between two points lies in a region with typical probability density. This would also work for vicinity sampling.

In section 1, overfitting concerns seem like a strange way to motivate the desire for smoothness. Overfitting is relatively easy to compensate for, and investigating the latent space is interesting regardless.

When discussing sampling from VAEs as opposed to GANs, it would be good to mention that one has to sample from p(x | z) not just p(z).

Lots of math typos such as t - 1 should be 1 - t in (2), "V times a times r" instead of "Var" in (3) and "s times i times n" instead of "sin", etc, sqrt(1) * 2 instead of sqrt(12), inconsistent bolding of vectors. Also strange use of blackboard bold Z to mean a vector of random variables instead of the integers.

Could cite an existing source for the fact that most mass for a Gaussian is concentrated on a thin shell (section 2.2), e.g. David MacKay Information Theory, Inference and Learning Algorithms.

At the end of section 2.4, a plot of the final 1D-to-1D optimal transport function (for a few different values of t) for the uniform case would be incredibly helpful.

Section 3 should be a subsection of section 2.

For both SLERP and the proposed method, there's quite a sudden change around the midpoint of the interpolation in Figure 2. It would be interesting to plot more points around the midpoint to see the transition in more detail. (A small inkling that samples from the proposed approach might change fastest qualitatively near the midpoint of the interpolation perhaps maybe be seen in Figure 1, since the angle is changing fastest there??)


review_title: A clear and detailed explanation of a problem which arises when manipulating latent space samples for GANs and VAEs, and a novel solution using heavy machinery but which is simple to apply in practice.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper is concerned with the mismatch between the input distribution used for training and interpolated input. It extends the discussion on this phenomenon and the correction method proposed by White (2016), and proposes an optimal transport-based approach, which essentially makes use of the trick of change of variables. The discussion of the phenomenon is interesting, and the proposed method seems well motivated and useful. There are a number of errors or inconsistencies in the paper, and the experiments results, compared to those given by SLERP, see rather weak. My big concern about the paper is that it seems to be written in a rush and needs a lot of improvement before being published. Below please see more detailed comments.

- In Introduction, the authors claim that "This is problematic, since the generator G was trained on a fixed prior and expects to see inputs with statistics consistent with that distribution." Here the learned generative network might still apply even if the input distribution changes (e.g., see the covariate shift setting); should one claim that the support of the test input distribution may not be contained in the support of the input distribution for training? Is there any previous result supporting this? 
- Moreover, I am wondering whether Sections 2.2 and 2.3 can be simplified or improved--the underlying idea seems intuitive, but some of the statements seem somewhat confusing. For instance, what does equation (6) mean?
- Note that a parenthesis is missing in line 3 below (4). In (6), the dot should follow the equation.
- Line 1 of page 7: here it would be nice to make it clear what p_{y|x} means. How did you obtain values of f(x) from this conditional distribution?
- Theorem 2: here does one assume that F_Y is invertible? (Maybe this is not necessary according to the definition of F_Y^{[-1]}...)
- Line 4 above Section 4.2: the sentence is not complete.
- Section 4.2: It seems that Figure 3 appears in the main text earlier than Figure 2. Please pay attention to the organization.
- Line 3, page 10: "slightly different, however..."
- Line 3 below Figure 2: I failed to see "a slight loss in detain for the SLERP version." Perhaps the authors could elaborate on it?
- The paragraph above Figure 3 is not complete.
review_title: An interesting paper that seems to be written in a rush

Meta-review: This paper exposes a simple recipe to manipulate the latent space of generative models in such a way to minimize the mismatch between the prior distribution and that of the manipulated latent space. Manipulations such as linear interpolation are commonplace in the literature, and this work will be helpful to improve assessment on that front.

Reviewers found this paper interesting, yet unpolished and incomplete. In subsequent iterations, the paper has significantly improved on those fronts, however the AC believes an extra iteration will make this work even more solid. Thus, unfortunately this paper cannot be accepted at this time. 
Decision: Reject