Title: LR-GAN: Layered Recursive Generative Adversarial Networks for Image Generation
Authors: Jianwei Yang
Anitha Kannan
Dhruv Batra
Devi Parikh
Venue: ICLR 2017
Keywords: Computer vision
Deep learning
Unsupervised Learning

Abstract: We present LR-GAN: an adversarial image generation model which takes scene structure and context into account. Unlike previous generative adversarial networks (GANs), the proposed GAN learns to generate image background and foregrounds separately and recursively, and stitch the foregrounds on the background in a contextually relevant manner to produce a complete natural image. For each foreground, the model learns to generate its appearance, shape and pose. The whole model is unsupervised, and is trained in an end-to-end manner with conventional gradient descent methods. The experiments demonstrate that LR-GAN can generate more natural images with objects that are more human recognizable than baseline GANs.

TL;DR: A layered recursive GAN for image generation, which considers the structure in images and can disentangle the foreground objects from background well in unsupervised manner.

Review discussion:
Thank you for your submission.
Would you like to visualize for comparison the segmentation masks inferred from the non category specific models in CIFAR, when the images are not classified according to different categorical labels? It is interesting to see under which conditions the inferred segmentation masks obtain reasonable shapes.
A question is: even in this complex real dataset cases, where there are many object categories, if the masks do not have reasonable shape, does the resulting visual quality also degrade?
Typos: Figure 13 (set of 6 columns instead of 5) also, the 7th column is just training images? You could add this in the caption for clarity.
“Learning contextual dependencies between the image layers:” In Figure 13 many backgrounds have bird holes which are not covered after the foreground pasting, the recurrent generation should be able to handle this, that is, to enlarge the foreground accordingly, isn’t this true?
“To the best of our knowledge, ours is the first work to use a recursive layered image generation model (especially in the context of GANs).” It seems the proposed model is similar to Huang and Murphy which also use layering and affine transformations for pasting image parts but it is trained under a GAN loss instead of VAE and scaled up to more realistic datasets.
Thus, the sentence is not entirely accurate.
Thanks for the comments! They are really helpful. We have added more experimental results and updated the sentences in our latest submission. We answer the questions one by one as follows.
Question1: Would you like to visualize for comparison the segmentation masks inferred from the non category specific models in CIFAR, when the images are not classified according to different categorical labels? It is interesting to see under which conditions the inferred segmentation masks obtain reasonable shapes.
A question is: even in this complex real dataset cases, where there are many object categories, if the masks do not have reasonable shape, does the resulting visual quality also degrade?
Answer1: For limited space, we showed the results obtained by the non category specific model in Figure 15 in the Appendix. For comparison, the category specific generation results are shown in Figure 17. As we can see from these two figures, the generated masks are more reasonable in the category specific case, and thus the resulting generated images have higher quality. Since an object is mainly characterized by its appearance, shape and pose, a good shape (i.e. mask) contributes a good amount to the final visual quality. It can be seen in Figure 15, in the third column, our model generated some horse-like, car-like shapes, and the corresponding images are more recognizable. It is also visually demonstrated in Figure 13 and Figure 14. The generated images with more bird-like shapes are more recognizable as well.
So qualitatively, the resulting visual quality degrades if the model does not generate reasonable shape.
Question2: Typos: Figure 13 (set of 6 columns instead of 5) also, the 7th column is just training images? You could add this in the caption for clarity.
Answer2: Thanks for pointing this out. By "set of 5 columns", we were referring to the 5 columns within each of the 7 columns. Yes, the 7th column shows training images that are nearest neighbors to the generated images. We have updated the caption for clarity.
Question3: “Learning contextual dependencies between the image layers:” In Figure 13 many backgrounds have bird holes which are not covered after the foreground pasting, the recurrent generation should be able to handle this, that is, to enlarge the foreground accordingly, isn’t this true?
Answer3: Yes, that's true. Ideally, the foreground object should be enlarged accordingly so that the resulting images look more natural. We suspect that some holes are not completely covered because in our experiments, we empirically set the minimum scaling factor in the transformation matrix to be 1.2 across all datasets except for MNIST-TWO.
However, there are a number of birds with even larger scales.
As a result, the generated foreground could not cover the hole completely.
Though it is not able to cover the whole hole in some cases, we can see that the birds after transformation are always placed at the holes to cover as much of the hole region as possible. To verify our intuition, we set the minimum scaling factor to 1.1, and re-ran our model on CUB-200. The results are shown in Figure 14. As we can see, the holes in the background do disappear.
Question4: “To the best of our knowledge, ours is the first work to use a recursive layered image generation model (especially in the context of GANs).” It seems the proposed model is similar to Huang and Murphy which also use layering and affine transformations for pasting image parts but it is trained under a GAN loss instead of VAE and scaled up to more realistic datasets.
Thus, the sentence is not entirely accurate.
Answer4: While we cite Huang and Murphy 2015 on these two premises,  we will  expand further on a few more  differences from their work: First, their generative model expressed each layer by its appearance and transformation (pose) only, without any explicit shape model. Due to the lack of shape masks, the inference requires simultaneous unrolling of encoder and decoder so that the encoder can identify the next layer based on the residual.
Also, there no results are shown in that paper for cases when the background has texture (natural scenes);
in fact, it is not conclusive if their proposed inference can separate foreground/background from a single image of a natural scene.
In contrast, LR-GAN explicitly models object appearance, shape and transformation, and the generative model dictates a layered composition that combines all these factors. The use of shape masks guide the model to associate pixels to the different object layers, making the composition realistic. Training is performed in the same way as in GANs, with the backward pass propagating gradients to every layer based on the discriminative loss of whether the generated image is real or fake. Our results on single image generation through layered composition shows the promise of our work in learning object appearances and shapes.
We have also updated the sentences in Section 3.2  to better reflect the differences in the layered (de)composition methods used for image generation.
Thanks again for the comments!
The authors propose a method that generates naturally looking images by first generating the background and then conditioned on the previous layer one or multiple foreground objects.
Additionally they add a image transformer layer that allows the model to more easily model different appearances.
I would like to see some discussion about the choice of foreground+mask rather than just predicting foreground directly. For MNIST, for example the foreground seems completely irrelevant. For CUB and CIFAR of course the fg adds the texture and color while the masks ensures a crisp boundary.
- Is the mask a binary mask or a alpha blending mask?
- I find the fact that the model learns to decompose images this nicely and learns to produce crisp foreground masks w/o too much spurious elements (though there are some in CIFAR) pretty fascinating.
The proposed evaluation metric makes sense and seems reasonable.
However, AFAICT, theoretically it would be possible to get a high score even though the GAN produces images not recognizable to humans, but only to the classifier network that produces P_g. E.g. if the Generator encodes the class in some subtle way (though this shouldn't happen given the training with an adversarial network).
Fig 3 shows indeed nicely that the decomposition is much nicer when spatial transformers are used.
However, it also seems to indicate that the foreground prediction and the foreground mask are largely redundant. For the final results the "niceness" of the decomposition appears to be largely irrelevant.
Furthermore, the transformation layer seems to have a small effect, judging from the transformed masked foreground objects. They are mainly scaled down.
- What is the 3rd & 6th column in Fig 9? It is not clear if the final composed images are really as bad as "advertised".
Regarding the eval experiment using AMT it is not clear why it is better to provide the users with L2 minimized NN matches rather than random pairs.
I assume that Tab 1 Adversarial Divergence for Real images was not actually evaluated? It would be interesting to see how close to 0 multiple differently initialized networks actually are.
Also please mention how the confidences/std where generated, i.e. different training sets, initialisations, eval sets, and how many runs.
Dear Reviewer,
Thanks for the review. Below, we respond to the points you made:
**Question**:
I would like to see some discussion about the choice of foreground+mask rather than just predicting foreground directly. For MNIST, for example the foreground seems completely irrelevant. For CUB-200 and CIFAR-10 of course the fg adds the texture and color while the masks ensures a crisp boundary.
**Response**:
From presentation and implementation perspectives, it is simpler and “cleaner” to assume that the generated background and foreground images are the same size as the final image so that the intermediate images can be seamlessly combined and updated.
However, clearly the foreground objects in real images vary in their shapes and sizes. The masks provide a natural way to model the shape of the foreground objects so that we can effectively carve/segment out the foreground appearance to be composed with the image generated so far. The idea of using foreground appearance and masks is widely used in the context of modeling videos (for instance, Wang and Adelson, 1994).
Moreover, in an ablated model without the mask, we found that the training process could not converge to any sensible separation of foreground from background, for both CUB-200 and CIFAR-10 (Section 6.8).
**Question**:
Is the mask a binary mask or a alpha blending mask? - I find the fact that the model learns to decompose images this nicely and learns to produce crisp foreground masks w/o too much spurious elements (though there are some in CIFAR) pretty fascinating.
**Response**:
We used a softmax layer on top of the mask generator.
Hence, the mask is an alpha blending mask, whose values range in (0, 1). In the initial iterations of training, the masks tend to be gray-scale. As the training proceeds, the mask becomes more and more sharp/near-binary.
**Question**:
The proposed evaluation metric makes sense and seems reasonable.
However, AFAICT, theoretically it would be possible to get a high score even though the GAN produces images not recognizable to humans, but only to the classifier network that produces P_g. E.g. if the Generator encodes the class in some subtle way (though this shouldn't happen given the training with an adversarial network).
**Response**:
Agreed, theoretically this is possible, but is unlikely to happen because (a) as you pointed out, of the adversarial training and (b) we do not optimize the generator parameters to maximize the proposed metrics. Designing perceptually faithful metrics (so that gaming is less of a concern) is still an open research problem.
**Question**:
Fig 3 shows indeed nicely that the decomposition is much nicer when spatial transformers are used.
However, it also seems to indicate that the foreground prediction and the foreground mask are largely redundant. For the final results the "niceness" of the decomposition appears to be largely irrelevant.
**Response**:
In Fig 3, it is indeed the case that the final results both look good, though the decomposition is totally different.
But these results are on MNIST-ONE that has white foreground on uniform gray background, which is easy to model.
However, on more challenging datasets, e.g., CUB-200, CIFAR-10, we can indeed find differences between images generated by a model that includes the transformer and one that does not, which we show in more detail in Fig 20 (which in hindsight, we should have included in the main paper to begin with). We find that the model without the transformer produces “degenerate” results, in the sense that it is unable to break the symmetry between foreground and background layers,  often generating object appearances in the model's background layer and vice versa (see Section 6.7 for more discussions).  Beyond this advantage, generating foreground shape also facilitates the unsupervised scene parsing demonstrated in Fig 22 and Fig 23.
**Question**:
Furthermore, the transformation layer seems to have a small effect, judging from the transformed masked foreground objects. They are mainly scaled down.
**Response**:
While it may seem that explicit transformations may not be needed for these datasets, we validate their importance in two ways:  We have now added a section 6.9 in the Appendix along with Fig 22 that shows the histograms of transformation parameters. Clearly, these parameters are used by the model, albeit in different ways on the different datasets (for instance, for CUB-200, translation on x-coordinate is more pronounced while for CIFAR-10, there are more diverse configurations).
We also provide qualitative results showing the importance of transformations using samples generated from our model and an ablated model that does not have transformations: Please see Section 6.7 and Fig 20, which  further emphasizes that we can  learn better image composition structures by explicitly accounting for transformation, and enabling  appearance and masks generator to focus on just doing that, and not spend its capacity on learning transformed appearances of these objects. This will become even more important as we learn models at higher resolutions and with more complex scenes.
**Question**:
What is the 3rd & 6th column in Fig 9? It is not clear if the final composed images are really as bad as "advertised".
**Response**:
The 3rd and 6th columns in Fig 9 (now Fig 20) are generated masks for CUB-200 and cifar-10.
As stated above, in Fig 20, we have added the final generated images for these two datasets from an ablated model where the transformations are not considered. As we can see, the images generated without the transformation are worse. Please see Section 6.7 for more discussions.
**Question**:
Regarding the eval experiment using AMT it is not clear why it is better to provide the users with L2 minimized NN matches rather than random pairs.
**Response**:
We think it helps the subjects make their assessments more easily if the two birds being compared are similar. That way they can focus on which of the two images is more realistic, rather than be distracted by other differences between the two birds.
**Question**:
I assume that Tab 1 Adversarial Divergence for Real images was not actually evaluated? It would be interesting to see how close to 0 multiple differently initialized networks actually are.
**Response**:
For evaluations, we have three models, which are trained on the real images (model A), generated images based on DCGAN (model B) and generated images based on LR-GAN (model C), respectively.
Then we compute Adversarial Divergence between the outputs of model B and model A, and also between the outputs of model C and model A. The Adversarial Divergence between the outputs of model A with itself is exactly equal to zero.
Following the reviewer's suggestion, we computed the Adversarial Divergence on real images in two more ways: one is training networks with different initialization on the same training set, and the other one is training two models based on two different training sets. For the first way, we trained 4 different models, then computed the Adversarial Divergence for each pair of models. The mean is 0.0025, and std is 0.0004. We can see it is fairly low among different initializations. For the second way, we trained models on the first 10,000 images in the training set and test set, respectively.
Then we compute the Adversarial Divergence between the outputs of these two models given the remaining 40,000 training images. The mean score is 0.822 with std of 0.017. Note that this is significantly smaller than the Adversarial Divergence scores of generated images (Table 1).
**Question**:
Also please mention how the confidences/std where generated, i.e. different training sets, initialisations, eval sets, and how many runs.
**Response**:
For clarity, we describe the calculation of the three metrics separately. For both MNIST-ONE and CIFAR-10, we have a training set and validation set of real images.
1) Inception Score: We first trained a model based on the training set. We chose the checkpoint with highest accuracy on the validation set. This model is used to obtain P(y|x) on the training set, generated images based on DCGAN, and generated images based on LR-GAN. All these three sets have the same number of images.
Then, as used in the original Inception Score code, we split each of three sets into 10 sub-folds, and then compute the mean/std on these 10 sub-folds.
2) Adversarial Accuracy: We trained three models based on the training set, generated images based on DCGAN, and generated images based on LR-GAN, respectively. After that, we applied these models to the validation set of real images to obtain the recognition accuracy. We trained the model with three different initializations, and then calculated the mean/std of the accuracies.
3) Adversarial Divergence: we used the same models as used to compute the Adversarial Accuracy. As mentioned above, we directly apply these models to the training set.
Similarly, we split the training set into 10 sub-folds, and then compute the mean/std.
The paper proposes a model for image generation where the back-ground is generated first and then the foreground is pasted in by generating first a foregound mask and corresponding appearance, curving the appearance image using the mask and transforming the mask using predicted affine transform to paste it on top of the image. Using AMTurkers the authors verify their generated images are selected 68% of the time as being more naturally looking than corresponding images from a DC-GAN model that does not use a figure-ground aware image generator.
The segmentations masks learn to depict objects in very constrained datasets (birds) only, thus the method appears limited for general shape datasets, as the authors also argue in the paper.
Yet, the architectural contributions have potential merit.
It would be nice to see if multiple layers of foreground (occluding foregrounds) are ever generated with this layered model or it is just figure-ground aware.
Dear Reviewer,
Thanks for the review.
Regarding the points made:
1) Our model has an explicit assumption that the object in each foreground layer undergoes an affine transformation.
When this assumption is largely respected (e.g. rigid objects), our model is able to successfully recover the object appearance and shapes, as evidenced by CUB-200, LFW datasets and categories in CIFAR-10 such as horse, boats, cars, etc.  For categories such as cats and dogs where there is richer local deformation, our modeling assumption is violated and the model does not perform as well. As future work, one can model richer local deformations (in addition to global transformations) to handle this.
2) Results on the MNIST-TWO dataset show that our model is indeed capable of generating multiple foregrounds. To understand how our model performs with occluding objects, we extended the MNIST-TWO dataset (that had no occlusions) to compose two digits with occlusions. Interestingly, we find that the trained model has two modes of generation: when it wants to generate an image in which the digits are non-occluding, it generates the two digits in successive steps.
However, when  generating an image with severe occlusion, it generates a collage of two digits in a single time step. We conjecture that since the model does not include priors on what 'normal' shapes look like, it has the freedom to choose what it believes is the parsimonious way to generate an image.  In general, unsupervised learning of image segmentation and object appearances and shape is a hard problem and our model takes preliminary steps in this direction for natural image generation. Future efforts to cope with occlusion may involve observing the participating objects under a wide variety of transformations and modeling the generator at a higher resolution. At 32x32, occluding objects/digits are often hard to segment out even for humans.
The paper presents an interesting framework for image generation, which stitches the foreground and background to form an image. This is obviously a reasonable approach there is clearly a foreground object.
However, real world images are often quite complicated, which may contain multiple layers of composition, instead of a simple foreground-background layer. How would the proposed method deal with such situations?
Overall, this is a reasonable work that approaches an important problem from a new angle.
Yet, I think sizable efforts remain needed to make it a generic methodology.
Dear Reviewer,
Thanks for the review.
We would like first to stress that we proposed a generic model which is recurrent and can naturally handle pasting multiple foregrounds. To demonstrate that, we performed experiments on the MNIST-TWO dataset, which contains two digits (foregrounds). We have shown that our model was able to generate background and two foreground digits separately (See Fig. 4 in Section 5.3). As we responded to AnonReviewer3, generating natural complex scenes remains an open interesting problem. Our approach proposes a detailed generative model for GANs. There are many natural extensions to explore in our setting: adding discriminators for intermediate outputs (such as for object shapes), improvements on the discriminators, and enabling the discriminator to share more than a gradient from binary classification loss. Exploring these is part of future work.
We thank the reviewers for their insightful comments. We have conducted supplementary experiments and incorporated comments into our updated submission. The major changes are:
1. Additional qualitative evidence for the role of affine transformation (Section 6.7): Our submission already contained full generation results on MNIST-ONE, and intermediate decomposition results on CUB-200 and CIFAR-10 to compare our model with and without the transformation. We have now added full generation results on CUB-200 and CIFAR-10, as well as results on LFW. We have placed images generated by the model without transformation next to the those generated by the model with transformation for easier inspection (Fig 20)
2. Quantitative evidence for the role of affine transformation (Section 6.9): We also have quantitative analysis of the learned transformation parameters that confirms that the model does in fact rely on the transformation (Fig 22)
3. Importance of modeling shapes (Section 6.8): We establish importance of the mask generator through an ablation of our model without it (Fig 21).
We also conducted several additional experiments and analyses that are described in our responses to reviewers below.
Please note that the sections in Appendix have been reordered to increase readability.
Dear Reviewers,
Thanks again for your feedback. We posted responses and updated our paper last week.
It would be great if you can let us know your updated thoughts or if you have any additional questions we can help address.
As a recap, among other things, we have included additional quantitative and qualitative results to further demonstrate the role of the learnt masks and transformations (including experiments on an additional dataset).
Thanks,
Authors

Meta-review:
role: PC
data: comment: The paper proposes a layered approach to image generation, ie starting by generating the background first, followed by generating the foreground objects. All three reviewers are positive, although not enthusiastic. The idea is nice, and the results are reasonable. Accept as poster. For the camera ready, the AC suggests making the generated images in the results larger, to allow the readers to fully appreciate their quality.

Decision: Accept (Poster)