Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: Disentangled GANs for controllable generation of High-Resolution Images introduces two new high resolution synthetic scene datasets for studying disentanglement in generative models and benchmarks two Style-GAN based architectures for controllable generation on these datasets. The datasets, though still synthetic, provide a significant quality boost over some of the simpler toy datasets previously studied in the disentanglement literature. A variety of experiments are conducted looking at 3 metrics (FID, MIG, and latent reconstruction) as well as qualitative analysis of samples. The paper studies how the performance of these architectures varies with hyperparameter and design decisions. The authors demonstrate that AC-StyleGAN performs well in fully supervised settings achieving the desired conditional generation, but, when controlling only a subset of factors, does not correctly disentangle. To address this, the author’s other architecture modification, FC-StyleGAN an image to image model is demonstrated to improve performance in this setting. The paper provides the reader with a useful overview of the behavior of the two models on these two datasets.

My rating is weak reject. While the paper has a variety of contributions (most notably introducing two new datasets and modifications of StyleGAN) and interesting results (such as relatively high performance with only 5% labeled data and the disentanglement issues when controlling a subset of factors), the core contributions of new datasets and architectures are not validated or analyzed rigorously enough.

1) No prior work is used as baselines in order to compare / validate the newly introduced architectures. The authors dismiss prior work in the introduction but do not provide any direct evidence that prior work is unable to handle the datasets introduced in the paper. In order to consider acceptance based on the value of the architectures proposed in the paper, there should be some direct evidence that the proposed Style-GAN modifications AC/FC, outperform prior architectures. One particular choice which is unclear to the reviewer is why AC GAN was chosen over the Projection Discriminator of Miyato and Koyama 2018 which demonstrated significantly better results than AC GAN and was adopted by major followup work such as BigGAN.

2) Given the core contributions of new architectures for disentanglement, the lack of any results on real, non synthetic, datasets in order to validate these (for instance, CelebA) which significant prior work on disentanglement has been evaluated on is an unfortunate omission. It is unclear how the introduced architectures handle complex natural image distributions with the much more diverse sources of natural variation they contain.

3) The authors claim their method is effective in the semi-supervised setting and demonstrate this by showing relatively strong performance in a low data regime. This demonstrates the importance of the third term in the loss function, but as other work for semi-supervised learning has shown (Oliver et al 2018) care must be take to properly attribute the performance of a semi-supervised algorithm to the actual semi-supervised components and purely supervised baselines are often quite competitive. The addition of an ablation / analysis demonstrating the contribution of the second term in settings where labels are also available would strengthen the author's claims and fully demonstrate the model is an effective semi-supervised learner.

Additional comments:

The datasets introduced in the paper do seem like potentially valuable contributions to the community - more discussion on the motivations behind their creation, the differences with prior work, the open difficulties / challenges, as well as the recommended evaluation protocols could add to their value.

The authors could more clearly motivate the work / applications the paper is interested in and how each contribution / experiment fits in with this. Without a clear sense of the authors mission / goals with the work, it feels a bit difficult to interpret the results. 


review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: === A. Summary ===

This paper proposes to train a new conditional GAN model that allows for controllable image generation by changing the input factors of variations (e.g. object color).
The supervised labels for the controllable attributes are obtained from a 3D renderer.
That is, the work combines the recent StyleGAN (that learns to generate images with disentangle latent vectors in an unsupervised manner) with AC-GAN (a clas-conditional GAN but here class information is replaced by the attribute information that we want to control).
The resultant AC-StyleGAN has essentially two latent vectors, one trained unsupervised and one trained with supervised labels.

The proposed GANs were thoroughly tested with different factors of variations (lighting, camera, objects) and on two different datasets self-contructed via 3D renderer.
The work is a solid demonstration that GANs can be used to synthesize images with fine-grained and coarse controllability if we have supervision signals!
The authors also released the anonymous code (which is a plus!).


=== B. Decision ===

Weak Reject.

I voted for Weak Reject because this paper presents a fairly incremental advance over what has been done (e.g. HoloGAN, StyleGAN, AC-GAN).
The impact of the unsupervised work (e.g. HoloGAN or StyleGAN) is much higher since they are generally applicable to real data without labels.
Here, although reasonable, the demonstration was done on a relatively small-scale 3D synthetic data (where there is only one scene and one object being manipulated).
Therefore, the claim that only 5% of supervised labels is required may not carry over to larger-scaled datasets with larger scene variability.
Plus, I don't see any baseline whatsoever being compared with the proposed methods here. 


=== C. Suggestions for Improvement ===

I have no problems with the novelty or idea of the work.
For me, the key problem with this work is the low impact or significance.

Some suggestions for showing the impact of this work:
- Show how your pre-trained GANs can be fine-tuned or transferred to the real data where we don't have labels.
- You could also plug in your pre-trained GANs to a separate synthetic-to-real image translation model to show that we could indeed learn to control these factors of variations of the real images. Hopefully, would the above setups yield better results than HoloGAN or StyleGAN?
- Clarify the main focus points of this paper and try to substantiate the result. The author wrote "Our work extends the above works by scaling up the disentanglement learning to high- resolution images, and emphasizing the importance of supervision in controllable generation." <---- but (1) high-res images here are synthetic and limited in scene variability; (2) the second part is expected given previous work.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary:

The paper is reasonably clear (though see some of my detailed comments on writing below).
The idea seems well-motivated and somewhat new though not revolutionary,
and the new data sets are nice (though see comments below about how I'm not qualified to evaluate them),
but I don't understand why the proposed algorithm wasn't evaluated on existing data sets as well,
and I don't understand why it wasn't compared against other algorithms that purport to do the same thing.
Barring certain exceptions (again see detailed comments), it seems like those two things are things
we ask of essentially all machine learning paper submissions, and I don't see why this paper is different?
I am open to being persuaded, but as of now I can't recommend acceptance.

Detailed Comments:

> More importantly, only using 5% of the labelled data significantly improves the disentanglement quality.
Hard to parse.
Does this mean using only 5% of data is better for disentangling than using 100%?


>  Generative adversarial networks (GANs) (Goodfellow et al., 2014) have achieved great success at generating realistic images, such as StyleGAN
Strange sentence - styleGAN is not itself a realistic image.


>  the controllable generation of high-resolution images is possible
Nit: i would cut the 'the'


> which characterizes how significant that the model can change a factor.
What does this mean?


> present a state-ofthe-art challenge
What does it really mean for a challenge to be state of the art?

> that enables conditional generation of high-fidelity images
Pretty minor nit, but I don't think it makes sense to refer to samples themselves as high fidelity.
High fidelity to what?
It makes a little more sense to refer to a trained generator as having high fidelity to the training data set,
but TBH I still don't even like the phrase in that context. 

Fig 1 is good. 
I felt like I could understand what's going on mostly from looking at the figure, which is nice.

The section surrounding eqs 1 and 2 is a little hard to read for me. 
Lots of single letter variable names and it's hard to keep them all in my head when I read the equation.
I'd replace e.g. c_r with \text{code}_r and so forth.

> Note that AC-StyleGAN reduces to an InfoGAN variant in the special case
This is helpful.

> AC-StyelGAN

>  FCStyleAGN

> by symmetry
I don't follow this part.


> while its high-resolution blocks accounts for fine styles
I feel like it's worth making a distinction between high frequency bits of an image and non-essential (or nuisance variables or whatever) bits.
They often are the same, but not always (it really just depends how close up your photo is, right?) but this technique is really separating high-freq from low-freq, IIUC.

>  handling complex high-fidelity images
Again, I just don't feel like this phrase makes any sense.

Re eq 3:
I guess you can call this the 'interpolation variance'
if you want, but it's not really a new thing.
You're just getting discretized measurements of the Jacobian of the mapping from code to predicted code, and I don't really think in a way that fits with the
intuition you describe.
There are a few things I can think of that are weird about this measurement, but here's just one:
I might have a dimension of the code that causes one really peaky change at one point, and then the change quickly reverts.
If i measure the variance of the output, I won't see much (in fact, depending on how you discretize, maybe I'll miss it altogether),
even though the fact that the change is peaky doesn't say anything about the 'importance' of the change.
I think you want something more like a line integral.
It's possible I'm misunderstanding something about the description of this, however.

Re: the experiments:
I like the new data sets, although I'm not familiar enough w/ the robotics literature to know whether they add something really new.
What I don't understand is why there's not really any attempt to do either of:
a) compare the models from the paper to existing models 
b) evaluate the models from the paper on existing data-sets?

I understand that sometimes one comes up with a thing that does something totally new,
and then reviewers will (maybe unjustifiably) still insist that the some kind of comparison be made even though it doesn't make sense,
but that doesn't seem to be what has happened here?
This paper proposes a new algorithm for synthesizing images and controlling various attributes of the images, but this has certainly been done in the past,
so why can't your technique be compared to those prior techniques?

Similarly, there are lots of data sets that people are already familiar with that you could have evaluated these models on, right?

review_title: Official Blind Review #2

Meta-review: The paper presents a model combining AC-GAN and StyleGAN for semi-supervised learning of disentangled generative adversarial networks. It also proposes new datasets of 3d images as benchmarks. The main claim is that the proposed model can achieve strong disentanglement property by using 1-5% of the annotations on the factors of variation. The technical contribution is moderate but the architecture itself is not highly novel. While the proposed method seems to work for controlled/synthetic datasets, overall technical contribution seems incremental and it's unclear whether it can perform well on larger-scale, real datasets. The experimental results on CelebA don't look convincing enough.

Decision: Reject