Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper is tackling the problem of training generative adversarial networks with incomplete data points. The problem appears to be important for semi-supervised training of image to image translation models, where we may have a lot of observations in both domains, but a little annotated correspondences between the domains.

The solution proposed by the authors involves an observation that discriminator in GANs is estimating the density ratio between real and fake distributions. This ratio can then be decomposed into a product of marginal density ratios, with two additional multipliers, corresponding to density ratios between a joint real/fake distribution and a product of its marginals. The authors then use discriminators to approximate all the ratios, which allows them to facilitate semi-supervised training.

My decision is "weak accept".

It is not clear to me to what extent does the proposed model outperform the regular CycleGAN on a large amount of paired training samples due to architectural changes (including spectral normalization).

Also, it would be nice if the comparison was carried out with a newer, possibly SotA models for unpaired image-to-image translation (MUNIT, FUNIT, BicycleGAN).

Moreover, there are some simple modifications that can be made to a standard CycleGAN/Pix2pix training pipeline that would facilitate the small number of annotations (for example, see "Learning image-to-image translation using paired and unpaired training samples").

It is hard to evaluate the comparative performance of the method without the comparisons mentioned above.
review_title: Official Blind Review #4

final_score: rating: 8: Accept
review_content: I found this paper very easy and clear to follow - the authors present, what I believe to be an elegant, approach to training a GAN in the presence of missing data or where many marginal samples might be available but very few complete (e.g. paired) samples. The approach proceeds by identifying that the joint distributions (true and approximate) can be factored so as to yield a number of different density ratios which can then be estimated by specific discriminators; in particular, these include the appropriate marginal density ratios and then corresponding overall correction factors. As a caveat to the review I should point out that while I am familiar with GANs, they are not my main area of expertise so this should be taken into consideration - apologies if there is literature I have missed.


Experiments: The authors provide a number of illustrative experiments that demonstrate the efficacy of the approach across a number of tasks. There are many differing GAN models but due to the nature of the problem I don't have a big issue with the majority of the comparisons being against a standard GAN since the tasks are suitably designed. For the paired MNIST experiment I found it hard to assess the qualitative results visually and am always concerned about the ad-hoc nature of Inception Distances - I find it difficult to attribute weight to them quantitatively since they are usually being used to assess things where they might suffer from a common error (e.g. they are both based on NNs). Also, I'm not fully on board with the dependency metric in (5) but then the authors also point out the same concerns. The other experiments I found more convincing.

I appreciated having error bars on some of the plots to help assess significance - would it not be possible to put error bars on all plots?

I found the additional extensions presented in the appendix to be interesting ideas as well and would be interested to see how the approach works with other GAN objectives as mentioned for future work.

I am mostly very positive about this work - my main concern is really common to most GANs - all the analysis relies on the premise that the discriminators can be setup as good estimators for the density ratios. We know that this is not always the case since everything comes from samples and if the capacities of each of the discriminators are not set appropriately then I would expect problems to occur - has this been explored by the authors? It would be no detriment to the work to include failure examples where the authors purposefully make use of inappropriate architectures for some of the discriminators to check for this? For example, there will be large imbalances in the number of training samples used for the different discriminators - how does this affect stability?


Other notes:

- Whilst I understand the point about independent marginals in 2.4 I'm not sure I see the motivation as clearly since it seems that the model is much more useful when there is dependent information but maybe there's a use-case I'm not thinking of?
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The authors present FactorGANs, which handle missing data scenarios by constructing conditional marginal estimates from ratios of joint and marginal distributions, estimated with GANs. FactorGANs are applied to the problem of semi-supervised (paired+unpaired) translation and demonstrate good performance.

Strengths:
-Nice formulation, which I believe is novel. Well written, good initial results.

Limitations:
-The most serious limitation of the paper is that the technique is not compared with any other semi-supervised methods, such as the Augmented CycleGAN. Because of this it is not clear how the technique compares with SOTA, and so the significance of the paper is not clear.
-The approach scales linearly with the number of marginals, which may limit its applicability to more general imputation tasks.
-The title is the same as an the arxiv paper title, and so the double-blind requirement is trivially violated.

Overall:

A nice formulation, but weak experimental investigations (no comparisons to SOTA semi-supervised translation) make the significance of the paper unclear. This makes it a borderline paper. I strongly encourage the authors to update their experiments accordingly.

Post Response:

Thank you to the authors for the detailed response and additional experimentation. I have updated my rating. It is a nice formulation, and the experimental validation of the technique has been strengthened. The additional experiments (i.e. comparing to the augmented cyclegan) that the authors are following through on will further improve the paper, making it a clear accept.

review_title: Official Blind Review #2

Meta-review: All three reviewers appreciate the new method (FactorGAN) for training generative networks from incomplete observations. At the same time, the quality of the experimental results can still be improved. On balance, the paper will make a good poster.
Decision: Accept (Poster)