Venue: ICLR 2017 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper proposes a model for image generation where the back-ground is generated first and then the foreground is pasted in by generating first a foregound mask and corresponding appearance, curving the appearance image using the mask and transforming the mask using predicted affine transform to paste it on top of the image. Using AMTurkers the authors verify their generated images are selected 68% of the time as being more naturally looking than corresponding images from a DC-GAN model that does not use a figure-ground aware image generator.

The segmentations masks learn to depict objects in very constrained datasets (birds) only, thus the method appears limited for general shape datasets, as the authors also argue in the paper. Yet, the architectural contributions have potential merit.

It would be nice to see if multiple layers of foreground (occluding foregrounds) are ever generated with this layered model or it is just figure-ground aware.
review_title: a figure-ground shape aware GAN mode for image generation

final_score: rating: null
review_content: Thank you for your submission.
Would you like to visualize for comparison the segmentation masks inferred from the non category specific models in CIFAR, when the images are not classified according to different categorical labels? It is interesting to see under which conditions the inferred segmentation masks obtain reasonable shapes.
A question is: even in this complex real dataset cases, where there are many object categories, if the masks do not have reasonable shape, does the resulting visual quality also degrade? 

Typos: Figure 13 (set of 6 columns instead of 5) also, the 7th column is just training images? You could add this in the caption for clarity.

“Learning contextual dependencies between the image layers:” In Figure 13 many backgrounds have bird holes which are not covered after the foreground pasting, the recurrent generation should be able to handle this, that is, to enlarge the foreground accordingly, isn’t this true?

“To the best of our knowledge, ours is the first work to use a recursive layered image generation model (especially in the context of GANs).” It seems the proposed model is similar to Huang and Murphy which also use layering and affine transformations for pasting image parts but it is trained under a GAN loss instead of VAE and scaled up to more realistic datasets. Thus, the sentence is not entirely accurate.
review_title: category specific and category agnostic models, clarity

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents an interesting framework for image generation, which stitches the foreground and background to form an image. This is obviously a reasonable approach there is clearly a foreground object. However, real world images are often quite complicated, which may contain multiple layers of composition, instead of a simple foreground-background layer. How would the proposed method deal with such situations?

Overall, this is a reasonable work that approaches an important problem from a new angle. Yet, I think sizable efforts remain needed to make it a generic methodology. 
review_title: review

final_score: rating: 7: Good paper, accept
review_content: The authors propose a method that generates naturally looking images by first generating the background and then conditioned on the previous layer one or multiple foreground objects. Additionally they add a image transformer layer that allows the model to more easily model different appearances.

I would like to see some discussion about the choice of foreground+mask rather than just predicting foreground directly. For MNIST, for example the foreground seems completely irrelevant. For CUB and CIFAR of course the fg adds the texture and color while the masks ensures a crisp boundary. 
- Is the mask a binary mask or a alpha blending mask?
- I find the fact that the model learns to decompose images this nicely and learns to produce crisp foreground masks w/o too much spurious elements (though there are some in CIFAR) pretty fascinating.

The proposed evaluation metric makes sense and seems reasonable. However, AFAICT, theoretically it would be possible to get a high score even though the GAN produces images not recognizable to humans, but only to the classifier network that produces P_g. E.g. if the Generator encodes the class in some subtle way (though this shouldn't happen given the training with an adversarial network).

Fig 3 shows indeed nicely that the decomposition is much nicer when spatial transformers are used. However, it also seems to indicate that the foreground prediction and the foreground mask are largely redundant. For the final results the "niceness" of the decomposition appears to be largely irrelevant.

Furthermore, the transformation layer seems to have a small effect, judging from the transformed masked foreground objects. They are mainly scaled down.

- What is the 3rd & 6th column in Fig 9? It is not clear if the final composed images are really as bad as "advertised".

Regarding the eval experiment using AMT it is not clear why it is better to provide the users with L2 minimized NN matches rather than random pairs.

I assume that Tab 1 Adversarial Divergence for Real images was not actually evaluated? It would be interesting to see how close to 0 multiple differently initialized networks actually are. Also please mention how the confidences/std where generated, i.e. different training sets, initialisations, eval sets, and how many runs.

review_title: Layerwise image generation.

Meta-review: The paper proposes a layered approach to image generation, ie starting by generating the background first, followed by generating the foreground objects. All three reviewers are positive, although not enthusiastic. The idea is nice, and the results are reasonable. Accept as poster. For the camera ready, the AC suggests making the generated images in the results larger, to allow the readers to fully appreciate their quality.
Decision: Accept (Poster)