Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper concerns a potentially serious issue with current GAN based approaches. Complex data distributions, such as natural images, likely lie upon many disconnected manifolds. However standard GANs use continuous noise and generators and must therefore output a connected distribution over inputs. This constraint results in the generator outputting what the paper terms “tunnels” regions of output which connect these actually disconnected manifolds but do not correspond to actual samples from valid manifolds. 

This is an important observation. The paper makes a variety of sensible claims - attributing incoherent samples to these tunnels and stating that complex datasets such as Imagenet are more likely to suffer from this problem. This behavior can indeed be observed during training on toy examples such as a 2d mixture of gaussians. However it is an open question how important this issue is in practice and the paper does not clearly separate this issue from the issue of properly modeling the complicated manifolds themselves. It is admittedly difficult to perform quantitative evaluations on generative models but much more work could be done to demonstrate and characterize the problem in practice.

The tunnel problem motivates the authors proposed approach to introducing discontinuities into the generator. Specifically the paper proposes training N different generators composed of N different linear projections of the noise distribution while sharing all further layers. A projection is chosen uniformly at random during training/sampling. An additional extension adds a loss term for the discriminator/generator to encourage predictability and thus diversity of the projection layers and improves results significantly. 

The only experimental results presented are qualitative analysis of samples by the authors. This is a very weak form of evidence suffering from bias as the evaluations are not performed blinded and are of a subjective nature. If the paper intends to present experimental results solely on sample quality then, blinded and aggregated human judgments should be expected. As a reader, I do agree that qualitatively the proposed approach produces higher quality samples than the baseline on CelebRoom but I struggle to see any significant difference on celebA itself. I am uncomfortable with this state of affairs and feel the claims of improvements on this task are unsubstantiated.

While discussion is motivated by known difficulties of GANs on highly varied datasets such as Imagenet, experiments are conducted on both MNIST and celebA datasets which are already well handled by current GANs. The proposed CelebRoom dataset (a 50/50 mixture of celebA and LSUN bedrooms) is a good dataset to validate the problem on but it is disappointing that the authors do not actually scale their method to their motivating example. Additionally, utilizing Imagenet would lend itself well to a more quantitative measure of sample quality such as inception score.

On the flip side, a toy experiment with known disconnected manifolds, while admittedly toy could increase confidence since it lends itself to more thorough quantitative analysis. For instance, a mixture of disconnected 2d gaussians where samples can be measured to be on or off manifold could be included.
 
At a high level I am not as sure as the authors on the nature of disconnected manifolds and the issue of tunnels. Any natural image has a large variety of transformations that can be applied to it that still correspond to valid natural images. Lighting transformations such as brightening or darkening of the image corresponds to a valid image transformations which allows for a “lighting tunnel” to connect all supposedly disjoint image manifolds through very dark/bright images. While this is definitely not the optimal way to approach the problem it is meant as a comment on the non-intuitive and poorly characterized properties of complex high dimensional data manifolds. 

The motivating observation is an important one and the proposed solution appears to be a reasonable avenue to tackle the problem. However the paper lacks quantitative evidence for both the importance of the problem and demonstrating the proposed solution.
review_title: Identifies and explores an important issue but lacks in quantitative analysis

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose to train multiple generators (with same set of parameters), each of which with a different linear mapping in the first layer. The idea is that the final output of the generator should be a distribution whose support are disconnected. The idea does look interesting. But a lot of details is missing and needs clarification.

1) A lot of technical details are missing. The main formula is given in page 6 (Sec. 4.2), without much explanation. It is also not clear how different generators are combined as a final generator to feed into the discriminator. Also how are the diversity enforced?

2) The experiments are not convincing. It is stated that the new method produces results that are visually better than existing ones. But there is no evidence that this is actually due to the proposed idea. I would have liked to see some demonstration of how the different modes look like, how they are disconnected and collaborate to form a stronger generator. Even some synthetic examples could be helpful.
review_title: Important details are missing

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

The paper studies the problem of learning distributions with disconnected support. The paper is very well written, and the analysis is mostly correct, with some important exceptions. However, there are a number of claims that are unverified, and very important baselines are missing. I suggest improving the paper taking into account the following remarks and I will strongly consider improving the score.

Detailed comments:

- The paper is very well written, which is a big plus.

- There are a number of claims in the paper that are not supported by experiments, citations, or a theorem.

- Sections 3.1 - 3.3 can be summarized to "Connected prior + continuous generator => connected support". Thus, to allow for disconnected support, the authors propose to have a discontinuous generator. However to me it seems that a trivial and important attack to this problem is to allow a simple disconnected prior, such as a mixture between uniforms, or at least an approximately disconnected (given the superexponential decay of the gaussian pdf) of a mixture of gaussians, which is very common. The authors fail to mention this obvious alternative, or explore it further, which I think weakens the paper.

- Another standard approach to attacking diverse datasets such as imagenet is adding noise in the intermediate layers of the generator (this was done by EBGAN and the Improved GAN paper by Salimans et al.). It seems to me that this baseline is missing.

- Section 3.4, paragraph 3, "the outputs corresponding to vectors linearly interpolated from z_1 to z_2 show a smooth". Actually, this is known to not perform very well often, indeed the interpolations are done through great circles in z_1 and z_2. See https://www.youtube.com/watch?v=myGAju4L7O8 for example.

- Lemma 1 is correct, but the analysis on the paragraph following is flat out wrong. The fact that a certain z has high density doesn't imply that the sample g_\theta(z) has high density! You're missing the Jacobian term appearing in the change of variables. Indeed, it's common to see neural nets spreading appart regions of high probability to the extent that each individual output point has low density (this is due in its totallity to the fact that ||\nabla_x g_\theta(z)|| can be big.

- Borrowing from the previous comment, the evidence to support result 5 is insufficient. I think the authors have the right intuition, but no evidence or citation is presented to motivate result 5. Indeed, DCGANs are known to have extremely sharp interpolations, suggesting that small jumps in z lead to large jumps in images, thus having the potential to assign low probability to tunnels.

- A citation, experiment or a theorem is missing showing that the K of a generator is small enough in an experiment with separated manifolds. Until that evidence is presented, section 3.5 is anecdotal.

- The second paragraph of section 3.6 is a very astute observation, but again it is necessary to show some evidence to verify this intuition.

- The authors then propose to partition the prior space by training separate first layers for the generator in a maximally discriminative way, and then at inference time just sampling which layer to use uniformly. It's important to note that this has a problem when the underlying separated manifolds in the data are not equiprobable. For example, if we use N = 2 in CelebRoom but we use 30% faces and 70% bedrooms, I would still expect tunneling due to the fact that one of the linear layers has to cover both faces and bedrooms.

- MNIST is known to be a very poor benchmark for image generation, and it should be avoided.

- I fail to see an improvement in quality in CelebA. It's nice to see some minor form on clustering when using generator's prediction, but this has been seen in many other algorithms (e.g. ALI) with much better results long before. I have to say also the official baseline for 64x64 images in wgangp (that I've used several times) gives much better results than the ones presented in this paper https://github.com/igul222/improved_wgan_training/blob/master/gan_64x64.py .

- The experiments in celebRoom are quite nice, and a good result, but we are still missing a detailed analysis for most of the assumptions and improvements claimed in the paper. It's very hard to make very precise claims about the improvements of this algorithm in such a complex setting without having even studied the standard baselines (e.g. noise at every layer of the generator, which has very public and well established code https://github.com/openai/improved-gan/blob/master/imagenet/generator.py).

- I would like to point a lot of tunneling issues can be seen and studied in toy datasets. The authors may want to consider doing targeted experiments to evaluate their assumptions.

=====================

After the rebuttal I've increased my score. The authors did a great job at addressing some of the concerns. I still think there is more room to be done as to justifying the approach, dealing properly with tunneling when we're not in the somewhat artificial case of equiprobable partitions, and primarily at understanding the extent to which tunneling is a problem in current methods. The revision is a step forward in this direction, but still a lot remains to be done. I would like to see simple targeted experiments aimed at testing how much and in what way tunneling is a problem in current methods before I see high dimensional non quantitative experiments.

In the case where the paper gets rejected I would highly recommend the acceptance at the workshop due to the paper raising interesting questions and hinting to a partial solution, even though the paper may not be at a state to be published at a conference venue like ICLR.
review_title: Review for "Towards Effective GANs for Data Distributions with Diverse Modes"

Meta-review: The paper presents a really interesting take on the mode collapse problem and argue that the issue arises because of the current GAN models try to model distributions with disconnected support using continuous noise and generators. The authors try to fix this issue by training multiple generators with shared parameters except for the last layer.

The paper is well written and authors did a good job in addressing some of the reviewer concerns and improving the paper.

Even though arguments presented are novel and interesting, reviewers agree that the paper lacks sufficient theoretical or experimental analysis to substantiate the claims/arguments made in the paper. Limited quantitative and subjective results are not always in favor of the proposed algorithm. More controlled toy experiments and results on larger datasets are needed. The central argument about "tunneling" is interesting and needs deeper investigation. Overall the committee recommends this paper for workshop.
Decision: Invite to Workshop Track