Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes to improve GAN by learning a function that splits the latent space in several parts and then feed each part to a different generator, this enables GAN to model distribution with disconnected support. They try the proposed approach on several toy examples where the support of the distribution are disconnected, the data is imbalanced or the support of the distribution intersect,  they in all this cases that the proposed approach improve performance over baseline, in particular the proposed method doesn't produce outliers. They also show how the method is robust to the choice of number of generators used. Finally they show improved performance on more challenging dataset in particular on the CelebA+Photo dataset as measured by FID score.

I'm slightly in favour to accept the paper. I think the idea is well motivated and shows real advantage over other methods on the toy experiments. The major downside of the paper is that the proposed method doesn't seem to improve that much in more realistic setting.

Main Argument:
+ The idea is well motivated and the paper precisely explain that they try to address the problem of modelling data when the manifold is disconnected and the class are imbalanced. They illustrate how the proposed approach is able to address this problem on some toy example

+ The paper also show how the method is robust to the choice of number of generators

+ Figure 7 and A1 are quite interesting showing how the different generators can learn different part of the data like different class.

- The main counterpoint is that the method doesn't seem to improve the performance that much in more realistic settings. The author point to the fact that this might be due to the fact that the metrics we used are not sensible to outliers. I wish the author had used another metric to show and confirm this hypothesis.

- I found the explanation of the proposed algorithm a bit confusing, it would be nicer if the final loss was clearly defined in the paper and the derivation of the loss explained. In particular in Algorithm 1 I don't understand why $L_{GAN} = D_{real} + (1-D_{fake})$ shouldn't there be some logarithm somewhere ?

- The author propose some measure of standard deviation for the different models but this computed on the three last model checkpoints. It would be much more valuable to compute the standard deviation and the mean with different seeds.

Minor comments:
- Conflicting notation: In the related work you use $G_i$ to denote a random Gaussian, the same notation is used to denote the generator. 

- The part about Machine Teaching and knowledge distillation seem a bit irrelevant to me. I don't really understand what they bring to the paper. Also this would save some space and enable to put the CIFAR10 results in the paper

- In the toy experiments I would be curious to see the results when the different modes have the same probability.

- It would be interesting to have the influence of the number of generators on the FID.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: Comments: 

-As I understand it this technique is relatively simple.  There are K distinct generators with different parameters.  There is a network NP(z) which maps from an input z to a weighting over the generators.  All of these generators are used during training and the loss on the discriminator is reweighted using NP(z).  

-This doesn't involve any explicit discrete decisions so the entire thing along with NP(z) can be trained end-to-end using the usual backprop.  

-It would be interesting to consider simply doing importance sampling after sampling NP(z) to select the most relevant generators to train on, as NP(z) is presumably much cheaper than G(z).  

-One thing that I'd like to see is a variant where the "different" generators share almost all of their parameters, but have different batch-norm "mean/sigma" parameters.  This would be the same as conditional batch norm.  I think it would be interesting if this could have some of the strengths of this method while still allowing the generators to share most parameters.  

Review: 

This paper presents a simple yet well motivated new method for helping GANs to model disconnected manifolds.  There are a few more explorations that I'd like to see (discussed in comments), but I still appreciate this paper for directly addressing an important challenge.  The 2D toy experiments do a good job of illustrating why the method helps and also the improvements on the "disconnected face/landscape" dataset are quite good.  It is a bit disappointing that it doesn't help on CIFAR10, although I think it's reasonable to leave this for future work.  
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Overall:
IIUC, the main contribution of this paper is to take previous work on training GANs with
multiple generators and add a learned assignment of z -> G_i, so that we have
G(z) = \sum_i NP(G_i | z) * G_i(z)
where NP is this learned assignment.

I think that technically this idea is new,
but there is important related work [1] that (IMO)
  - does essentially the same thing
  - better experimentally validates the thing they do
  - was published in last ICLR.

I also have a number of issues with the experimental design: see my detailed comments below.
Finally, the technique seems like it can't easily scale up to larger GANs or data sets 
because it requires instantiating many copies of the generator in memory?

For these reasons, I lean somewhat strongly toward rejection.


Detailed comments on draft:

It's worth noting that conditional GANs also sample from multiple disconnected `manifolds'.
I guess the value of adding your technique is that it can work without labels?

> Our approach differs significantly from those previously
strange sentence.

>  By dividing up the space into four slices...
I don't get this part. 
If you fed all the slices to a single generator you could
generate four `disconnected manifolds' with just 1 generator, no?

> , multiple-discriminator mixture in (Arora et al., 2017)
Nit: you don't need the parens there, IMO.

> our model learns to model the data over the full collection of generators in each minibatch
Given that modern GAN techniques (e.g. bigGAN), this is going to have some pretty unpleasant
performance characteristics, right?
Each generator has to be instantiated in memory all the time.

> LGAN
Nit: surely, given the state of the GAN literature, this name has been used before.


> We compare our NPGAN’s ability to model this distribution to
The following baseline seems like it would be much simpler and get the job done:
Train a GAN on a prior that's a mixture of like 100 gaussians or whatever. 
Then it seems like it could learn to assign 70, 50, and 30 of those modes
to each one of the gaussians in your underlying data?
I guess I haven't tried this myself, but it seems like a more fair comparison.
The way the current experiment is set up, you know a priori that your model
is the only one that can work.

Overall, I don't really understand why it's necessary to have multiple generators.
If I want the prior my generator `sees' to be disconnected, can't I just
pass samples from the prior through a standard relu network?
Surely a relu network can learn to separate one mode into two, and so forth.
But once you do that, you've essentially got [1].

> We next demonstrate
Would be nice to have a new subesection here.

I think your single-generator baseline in Fig 2 and 3 is unrealistically bad.
See fig 3 of [2], in which it looks bad but not nearly as bad as you've shown it.
I think this points to another issue with the experimental design.
If the generator is a big enough relu network, it ought to be able to automatically
generate `disconnected bits', but it has to be big enough, and you've compared a
generator with N parameters to 2 and 3 generators with N parameters each.
Moreover, your learned noise prior has some extra number of parameters (I don't see
where you described these but I may be missing it).

> A single generator is unable to model this non-disconnected data.
Again, I have a feeling that this is because of the small generator you use.
I'm pretty sure that a motivated person could get a normal GAN to model the
distribution in fig 5 reasonably well.

> The other models can only work if the precise optimal number of generators is known a
priori.
Isn't this not true for the method that throws out redundant generators?

The experiments described in 4.3 and 4.4 are pretty contrived.
It's ok to have some contrived experiments, but it seems like all experiments
on which you were able to provide evidence that your technique was helpful are contrived.
It also seems like the baselines you used are not obviously the right choice for these
experiments?

Re: your CIFAR experiments:
I think this supports my earlier claim that the extra parameters and bells and whistles
in modern GAN techniques *already implicitly do what your method is proposing to do*.
I think a lot of people miss this point when writing papers about GANs.
When people request comparisons against more modern techniques,
they're not (merely) being difficult: 
they want to know if the technique you propose "stacks" with other techniques,
in the sense that your method is doing something that wasn't already being done implicitly
by the old methods.

References:
[1] On Self Modulation for Generative Adversarial Networks (https://arxiv.org/abs/1810.01365)
[2] Discriminator Rejection Sampling (https://arxiv.org/pdf/1810.06758.pdf)

review_title: Official Blind Review #3

Meta-review: This paper introduces a modified GAN architecture that looks a lot like a mixture of experts, to address the problem of learning multiple disconnected manifolds.  They show this method helps on 2D toy experiments, and artificial tasks where different datasets are combined, but not on CIFAR.  They also introduced a new variant of FID that they claim is more sensitive to the improvements made by their model.

R2 didn't seem to think too hard about the paper, and R3 seemed a bit dismissive.

Overall the idea seems sensible but the particulars of this approach aren't all that well-motivated in my opinion, especially since the cost of the generator is increased.  Why not just use a mixture of Gaussians in the original untransformed space?

I also found the toy experiments unconvincing, particularly the claim that a standard GAN couldn't learn a mixture of 3 Gaussians.  Learning a mixture of 8 Gaussians was one of the results in the unrolled GAN paper, for instance.

The results on the mixed datasets experiments seem encouraging, but I'm afraid that proposing a new GAN architecture in 2019 requires even more baselines than the authors compared against, and the fact that the task was artificially constructed undercuts its importance.
Decision: Reject