Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper proposes a novel method for compositional generation for 2D segment-like elements.
It decomposes the task into two stages: a sketching stage for soft layout generation and a painting stage for refining the layout and generating photo-realistic images conditioned on the soft layout. The paper uses different modified versions of GANsformer in each stage.  Experiments are carried out on multiple datasets with better results compared with prior works. The proposed framework is capable of separating style from structure and decomposition of multiple individual object properties.
main_review: I recommend accepting the paper. Overall, this paper is fairly well written and organized. The proposed SceneGAN is definitely interesting and addressing a very important problem - compositional generation of 2D segment-like elements. It undoubtedly reaches the new State-of-The-Art in this area. I summarize the strengths and weaknesses as follows:

Strengths:

- It successfully fulfills the goal of separating structure from style with careful design of the two careful-designed different-functioning stages and the losses of each stage.
- Soundly evidenced by ablation studies and extensive experiments in the paper, the proposed novel semantic consistency objective functions particularly hit the nail on the head, ensuring semantic consistency while not hinder diversity.
- This paper marches a powerful step towards explainable and controllable generative scene decomposition.
- Results are very promising in terms of diversity, fidelity and decomposition; the efficacy of the methods is clearly demonstrated through experiments carried on multiple datasets.

Weakness:

- A few typos
    - line 64 (through multiple)
    - line 72 (fir->for)
    - line 193 ($d(x,y)_i$ -> $d_i(x,y)$)
    - line 300 (poentially -> potentially)
    - line 301 (content diversity)

Questions:
- Since the transformers are already capable of reasoning contextual information, why do the transformers in the sketching stage still need to be recurrent? I'm aware that Fig.6 shows the empirical proof. Is there any theoretical or intuitive evidence?
- How does the segmentation network in the semantic matching loss work with the blurry and meaningless generated images of the early/initial training stage? Will this not introduce misleading gradients?
limitations_and_societal_impact: The authors realize the potential negative social impact of fake or unauthentic content synthesis. However, this is one of the most common stumbling blocks in this area and we believe more and more anti-fake technologies will help us embrace more positive opportunities.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes SceneGAN that uses a two-stage strategy to perform image generation. The SceneGAN is based on a previous transformer-based GAN model, it firstly uses iterative generation to generate scene layout then based on the layouts to perform image generation. The authors demonstrate their model's performance on clevr and real images and show some good results.
main_review: originality: low

There is no new techniques introduced in the proposed model. The SceneGAN simply adopts a transformer gan as its architecture.

quality:

1. The authors used an iterative method to generate layouts, are there any experiments showing it's better than a single-pass generation?
2. It seems that the authors didn't benchmark their numbers with available settings where other papers also used this two stage generation pipeline.

missing citation:
[1] Compositional Video Synthesis with Action Graphs -> (compositional generation, CLEVR)
[2] Probabilistic neural programmed networks for scene generation -> (compositional generation, CLEVR)


clarity: The writing is pretty clear to me

significance: As mentioned, there's no significantly new techniques proposed or any new insights that can be drew from this paper.
limitations_and_societal_impact: No related concern
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces an object-oriented GAN model, which generates an image of a scene in a two stage process. First, it generates a layout, where different masks show what object should be painted in which part of the image. Second, this layout is refined by a second network that converts it into a real-world-looking image. Both stages are based on modified GANformer networks. Finally, the paper introduces two novel losses. These make sure that the generated image reflects the generated layout, and that each portion of the image (as given by the layout) looks realistically. The method is fairly well evaluated, but the paper lacks comparison with two key baselines, that is BlockGAN (Nguyen-Phuoc et al) and GIRAFFE (Niemeyer & Geiger), which are not cited. Modelling decisions are evaluated in an ablation study.  The introduced method is supervised and requires paired (image, segmentation masks; this is in contrast to what the authors claim, see my comments below) inputs.


# UPDATE
The authors addressed my questions about unpaired training and using CNNs instead of Transformers. I'm still waiting for details of the CNN experiment, but I am already happy to increase my score to 7.
main_review: Originality:
- The task of object-oriented scene generation is well established. This work seems to achieve new SOTA results on a number of datasets.
- The majority of this work is a combination of well-known techniques (layout-conditioned image generation, GANformer), but it also introduces two novel and conceptually sound losses that significantly improve the results.
- AFAIK, this is the first application of transformer to object-centric scene generation.
- The related work seems to be adequately cited, but it misses two key approaches that should be compared against, that is BlockGAN (Nguyen-Phuoc et al) and GIRAFFE (Niemeyer & Geiger).
- The paper makes many references to "sketching": I think you should cite SPIRAL (Ganin et al) to say that sketching was considered as a stand-alone task in a GAN.,
- The author claim that their method produces unsupervised depth estimates. While it very much look like it does, it is very likely that some of the previous object-centric generative model (Monet, Iodine, Genesis+v2) should provide those estimates as well---it is just that they were never analyzed from this point of view. Moreover, BlocKGAN, GIRAFEE and Obsurf (Stelzner et. al.) do provide depth estimates. So this point is not novel, and is hardly differentiation from prior art.

Quality: Is the submission technically sound? Are claims well supported (e.g., by theoretical analysis or experimental results)? Are the methods used appropriate? Is this a complete piece of work or work in progress? Are the authors careful and honest about evaluating both the strengths and weaknesses of their work?

Good:
- The submission is technically sound. The overall approach makes sense, and there are ablation studies that validate modelling decisions.
- The numerical results are SOTA (though it would be easier to interpret PR AUC instead of separate precision and recall scores), and the qualitative results look superb.
- The authors acknowledge the limitation that segmentation masks are required.

Bad:
- In L46-47 you write: ", existing approaches fail to provide controllability at the level of the individual object or local region". This is very much not true. Models like Monet, Iodine, Genesis (+v2), Obsurf and many others, not to mention HoloGAN and GIRAFFE do provide that controllability. Please remove that claim.
- it seems that one of the key components of the method is using GANformer. I wonder how important it is, i.e. what are the advantages, both theoretical and empirical, over using a CNN-based generator? Please provide ablations where you change the decoder to a convolutional one and discuss.
- The authors claim that paired inputs are not necessary to train the model. I **strongly disagree**. Currently, the "layout" stage is pre-trained with externally-provided semantic segmentation. I believe that this is vital. The generated layouts are used as targets in an auxiliary loss for the discriminator. Therefore, if the "layout" stage is not pre-trained, it is almost a given that the generated layout would converged to empty images, which is a degenerate solution that is trivially matched by the discriminator.

Clarity: 
- It is first unclear that the paper is about supervised learning. I only realized it by noticing "unsupervised depth ordering" in L54. Please make it explicit very early on.
- The term "panoptic segmentation" is defined only in L284-285. Please move it to the very beginning of the paper as some readers (myself included) in this community are unaware of this term.
- 
- L1: delete "explored".
- L6: unclear what is "honed and refined".
- L9: I am not sure how "efficiency" is encouraged by this approach.
- L26: "imply semantic information"---I disagree; this is dependent on prior knowledge about the world which is absent here.
- L68: what is "amodal completion" please define before using.
- L72: :s/fir/for/
- L136 seems to suggest that both the latents and the image have the same dimensionality "d". Is that true? If so, this is non-standard; please make it more explicit.
- L158: the 2nd part of (1) does not make sense given the "spatial distribution" assumption, please rephrase.
- L162: The way you use depth in L193 in the softmax, and this relation, suggests that d(x, y) is negative depth. Please explain that in the paper.
- L172-173: how do you choose the number of segments to generate per iteration?
- L179-189 and L220-226: you only mention changing the key function, but this seems insufficient. You need to also modify the value function if you want to use the auxiliary inputs in the attention. Also, since you want to augment these auxiliary inputs as a by-product, you probably also change the query function? In total, it seems to me that you are using normal self-attention but concatenate the previous latents to the current inputs. Please clarify that.
- Tab 1. "over 10 random", you meant to add "runs"?

Significance: 
- Object-centric image (or scene) generation is currently very fashionable. This paper introduces a major improvement into current modelling framework. I would say that the most important pieces are the auxiliary losses introduced in the paper. To know how significant the current work really is, we would need to see comparisons with GIRAFFE and BlockGAN. But as of now, I think it is a solid piece of work that will benefit the community a lot.

I will increase my score if comparisons with GIRAFFE and BlockGAN are favourable and if you remove some of the claims (as discussed above).
limitations_and_societal_impact: Adequately discussed.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: - This paper introduces a novel GANsformer-based architecture for two-staged, compositional image generation. The first stage samples a panoptic-segmentation-like instance mask of the scene which is re-painted in the second stage to yield a high-fidelity image consisting of multiple objects.
- The explicit architectural biases w.r.t. scene layout, object instances, depth ordering and appearance modelling enable explainable and controllable image generation.
- The paper presents a thorough quantitative analysis of its proposed approach conducted on five suitable and challenging datasets and compared to seven adequate baselines models.
main_review: ## Updated Review (24/08/21)

After considering the additional experiments provided substantiating the claims w.r.t. to disentanglement and the additional baselines provided for related models (BlockGAN, GIRAFFE), my main concern about the weak corroboration of the paper's claims has been mitigated substantially. I appreciate the authors' effort of providing these additional experiments and thank them for their clarifying statements in the comments. I expect them to transfer the clarifications provided during the review process into the final manuscript - especially the disentanglement and controllability studies as well as the additional baseline results. In the light of the reviews and comments, I have increased my initial rating of 4 to 6.


## Original Review

TL;DR: The paper proposes a reasonable, two-stage image generation process using a compositional scene layout as intermediate representation and employing GANsformers for the modality translation tasks. While the paper presents a comprehensive experimental evaluation in terms of datasets and baselines, the lack of clarity in the presentation of the model's details and the weak corroboration of the claims about compositionality and controllability prevent it from meeting the bar for acceptance in its current form.

### Pros and Cons
- Pros
  - The paper presents a reasonable coarse-to-fine drawing approach for image generation utilising a GANsformer for modality translation.
  - The iterative layout generation of the model affords a compositional and interpretable image formation process.
  - The authors perform a comprehensive evaluation across multiple datasets and against several suitable GAN-based baselines for image generation.
- Cons
  - The model architecture and new loss functions which are main contributions of the paper are only superficially described in prose and are never specified with equations. The way the model is presented makes it almost impossible, even for an expert in this domain, to faithfully re-create the network and replicate the results.
  - The experiments section is unfortunately riddled with quality problems: The results tables (tab. 1, 2) feature unexplained metrics like P(recision), R(ecall), mIoU or pAcc which cannot be directly linked to any experiment or claim of the paper. Furthermore, a split of the COCO dataset is introduced for no apparent reason - why is this necessary?
  - Claims about controllability and disentanglement properties of the model are only weakly corroborated by qualitative experiments in fig. 7 and in supplementary fig. 3 despite being among the main selling points for the compositional image generation model.
  - The manuscript also lacks clarity of presentation in the captions which makes many figures very hard to interpret (e.g. fig. 4, 6, 7).

### Originality
- Framing image generation as a coarse-to-fine drawing process is reasonable and effective, although not novel; earlier methods (e.g. DRAW) have followed a similar analogy.
- The paper presents a novel aspect in the two-stage "translation" process: (1) noise -> instance mask; (2) instance mask -> image. While each stage in itself is not entirely novel, the composition in this pipeline using GANsformers for the individual translation processes, is an original idea.

### Quality
- Fig. 6 is missing units on the x-axis.
- In l. 321 fig. 5 is referenced for the visualization of a latent interpolation experiment where color, material, shape and position of an object are supposedly controlled independently. However, fig. 5 depicts the translation of segmentation maps into colored images. I suspect, fig. 7 was supposed to be referenced. However, fig. 7 is also very hard to interpret as its caption is not explaining what is truly happening in the picture strips. Fig. 3 of the supplementary material does a slightly better job in that regard.
- incorrect statements:
  - ll. 46-47: "existing approaches fail to provide controllability at the level of the individual object or local region" -> This is incorrect. For instance, [1, 16] allow to control the location and appearance of individual objects via designated elements of the latent code.
  - ll. 95-96: "the need for independence assumptions made by earlier works [16, 55, 68]" -> This is also incorrect as [16, 68] explicitly model inter-object relationships in latent space to overcome independence assumptions of earlier works.
  ll. 296-298: "1.01 FID score when conditioned on a ground-truth layout. This result serves as an indication for the particular aptitude of the model for compositional scenes." -> If the ground-truth layout (i.e. instance segmentation map) is provided, the model is relieved from the hard task of composing a plausible scene layout and is only tasked with re-painting the semantic areas with suitable textures. It is well known that GANs excel at such a task (e.g. GauGAN). Consequently, this result does not corroborate the claim of the model's aptitude for compositionality.
- typos, formatting errors:
  - l. 56: ~~the~~ it is ready,
  - l. 64: through **_** multiple
  - l. 72: noise scheme **for** synthesizing
  - ll. 142-143: the equations are missing equation numbers
  - l. 146: ~~to~~ shape and impact
  - l. 238: and **assess** the fidelity
  - l. 259: that SceneGAN **possesses**
  - l. 264: in modelling **compositional** scenes
  - ll. 300-301: content **_** diversity
  - ll. 322-323: the fact **that** it allocates
- The manuscript occasionally uses embellished, slightly unsuitable choices of words for a scientific conference paper, e.g. "celebrated StyleGAN2" (l. 267), "great Pix2PixHD repository", "the wonderful 1.01 FID score" (ll. 296-297)

### Clarity
- Fig. 1 reads more like a system diagram than an architecture overview. The important details about model architecture, losses and training procedure are missing.
- The manuscript would greatly benefit from a more explicit and self-contained presentation of the model using a network architecture diagram and equations for the newly introduced loss terms.
- In tab. 1 most results seem to coincide with the numbers reported at https://github.com/dorarad/gansformer#evaluation, but some numbers differ -> Have the experiments been re-implemented or re-run or have the numbers just been copied?
- In tab. 1, the P(recision)/R(ecall) evaluation appears out of nowhere with no justification or explanation in the text. What is the purpose of this evaluation and how does it support any of the paper's claims?
- The paper features many very sparse figure captions to the point that the figures becomes uninterpretable, e.g. fig. 4, fig. 6, fig. 7.
- The model in its current form is slightly limited by the need of paired training data featuring an RGB image and a segmentation mask. While it is hinted at the fact that an unpaired training setup could work as well (ll. 249-250), this is never explained in detail.

### Significance
- Compositional image generation is an important research problem in Computer Vision and a model which yields high-fidelity images while also providing scene layout masks and relative depth maps could certainly be useful for image generation and editing tools.

limitations_and_societal_impact: - The authors have appropriately addressed one of the model's limitations in its training setup and the consequences of potential applications in section 5.
review_title: null

Meta-review: - The proposed method tackles an important problem, takes a reasonable and novel approach (the loss function and recurrent generation), and the experiments are well planned, executed, and show SOTA results. 
- The rebuttal addressed the major concerns from the reviewers well and thus some reviews raised the score.
- The clarity is good enough but some figures and the details of the architecture need to be clarified further. 

Decision: Accept (Poster)