Generating unseen complex scenes: are we there yet?
Authors: Arantxa Casanova
Michal Drozdzal
Adriana Romero
Venue:  (2021)
Keywords: generative adversarial networks
conditional scene generation
zero-shot generalization
out of distribution

Abstract: Although recent complex scene conditional generation models generate increasingly appealing scenes, it is very hard to assess which models perform better and why. This is often due to models being trained to fit different data splits, and defining their own experimental setups. In this paper, we propose a methodology to compare complex scene conditional generation models, and provide an in-depth analysis that assesses the ability of each model to (1) fit the training distribution and hence perform well on seen conditionings, (2) to generalize to unseen conditionings composed of seen object combinations, and (3) generalize to unseen conditionings composed of unseen object combinations. As a result, we observe that recent methods are able to generate recognizable scenes given seen conditionings, and exploit compositionality to generalize to unseen conditionings with seen object combinations. However, all methods suffer from noticeable image quality degradation when asked to generate images from conditionings composed of unseen object combinations. Moreover, through our analysis, we identify the advantages of different pipeline components, and find that (1) encouraging compositionality through instance-wise spatial conditioning normalizations increases robustness to both types of unseen conditionings, (2) using semantically aware losses such as the scene-graph perceptual similarity helps improve some dimensions of the generation process, and (3) enhancing the quality of generated masks and the quality of the individual objects are crucial steps to improve robustness to both types of unseen conditionings.

Under review as a conference paper at ICLR 2021
GENERATING UNSEEN COMPLEX SCENES: ARE WE THERE YET?
Anonymous authors Paper under double-blind review
ABSTRACT
Although recent complex scene conditional generation models generate increas- ingly appealing scenes, it is very hard to assess which models perform better and why. This is often due to models being trained to fit different data splits, and defining their own experimental setups. In this paper, we propose methodology to compare complex scene conditional generation models, and provide an in-depth analysis that assesses the ability of each model to (1) fit the training distribution and hence perform well on seen conditionings, (2) to generalize to wnseen con- ditionings composed of seen object combinations, and (3) generalize unseen conditionings composed of unseen object combinations. As a result, we observe that recent methods are able to generate recognizable scenes given condi- tionings, and exploit compositionality to generalize to unseen conditionings with seen object combinations. However, all methods suffer from noticeable image quality degradation when asked to generate images from conditionings composed of unseen object combinations. Moreover, through our analysis, we identify the advantages of different pipeline components, and find that (1) encouraging com- positionality through rssance-wise spatial conditioning normalizations increases robustness to both types of unseen conditionings, (2) using semantically aware losses such as the scene-graph perceptual similarity helps improve some dimen- sions of the generation process, and (3) enhancing the quality of generated masks and the quality of the individual objects are crucial steps to improve robustness to both types of unseen conditionings.
INTRODUCTION
The recent years have witnessed significant advances in generative models (Goodfellow et al. 2014
Kingma & Welling] 2014 van den Oord et al., 2016a MMyato Koyama] 2018] MMyato et al,
2018 Brock et al. 2019), enabling their increasingly widespread use in many application domains van den Oord et al. 20166] Vondrick et al.) 2016, Zhang et al.) 2018a Hong et al.) 2018 Sun & Wu 2020). Among the most promising approaches, Generative Adversarial Networks (GANs) [Goodfellow et al.] 2014) have achieved remarkable results, generating high quality, high resolu- tion samples in the context of single class conditional image generation (Brock et al. 2019). This outstanding progress has paved the road towards tackling more challenging tasks such as the one of complex scene conditional generation, where the goal to generate high quality images with multiple objects and their interactions from a given conditioning (e.g. bounding box layout, seg- mentation mask, or scene graph). Given the exploding number of possible object combinations and their layouts, the requested conditionings oftentimes require zero-shot generalization. Therefore, successfully generating high quality, high resolution, diverse samples from complex scene datasets such COCO-Stuff (Caesar et al. 2018) remains stretch goal.
Despite recent efforts producing increasingly appealing complex scene samples (Hong et al. 2018
Hinz et al., 2019) Park et al.) 2019) Ashual & Wolf 2019 [Snn & Wu 2020 [Sylvain et al. 2020),
and as previously noted in the unconditional GAN literature (Lucic et al. [2018] Kurach et al. 2018
it is unfortunately very hard to assess which models perform better, and perhaps more importantly
why. In the case of conditional complex scene generation, this is often due to models being trained to fit different data splits, using different conditioning modalities and levels of supervision bound- ing box layouts, segmentation masks, scene graphs -, and reporting inconsistent quantitative metrics (e.g. repeatedly computing previous methods' results using different reference distributions, and/or

---

Under review as a conference paper at ICLR 2021
using different image compression algorithms to store generated images), among other uncontrolled sources of variation. Moreover, these methods disregard the challenges that emerge from their ex- pected generalization to unseen conditionings. This lack of rigour leads to conclusion replication failure, and hinders the identification of the most promising directions to advance the field.
Therefore, in this paper aim to provide an in-depth analysis and propose methodology to com- pare current conditional complex scene generation methods. We argue that such analysis has the potential to deepen the understanding of such approaches and contribute their progress. The pro- posed methodology addresses the following questions: (1) How well does each method perform on seen conditionings (training conditionings)?: (2) How well does each method generalize to unseen conditionings composed of seen object combinations?? and (3) How well does each method gener- alize to unseen conditionings composed of unseen object combinations? We investigate the answers to the previous questions both from a icene-wise and an object-wise standpoints. As result, we observe that: (1) recent methods are capable of generating identifiable scenes from seen condition- ings; (2) they exhibit some generalization capabilities when using unseen conditionings composed of seen object combinations, exploiting compositionality to generate scenes with different object arrangements; (3) they suffer from poorer generated image quality when asked for unseen object combinations, especially the method presented in (Sylvain et al.] (020). However, in all cases, the quality of the generated objects generally suffers from missing high frequency details, especially for those classes in the long tail of the dataset distribution. Finally, through an extensive ablation, we are able identify the strengths and weaknesses of different pipeline components. In particular, we note that endowing the generator with an instance-wist normalization module (Sun & Wu 2019) results in increased individual object quality and better robustness both types of unseen condi- tionings, whereas exploiting the normalization module of [Sylvain et al. (2020) results in improved scene quality, suggesting that exploiting scene compositionality in the generator helps improve gen- eralization. Moreover, we find that including a scene-graph similarity (Sylvain et al. (020) while training complex scene conditional generation models leads to better conditional consistency, es- pecially for unseen object combinations, emphasizing the promise of moving towards semantically aware training losses to improve generalization. We also identify the improvement of generated segmentation masks as one promising avenue to promote generalization to unseen conditionings. Finally, by leveraging these findings, we are able to compose a pipeline which obtains state-of-thr art results in metrics such as FID, while maintaining competitive results in almost all other studied metrics.
RELATED WORK
Evaluation metrics for GANs. The most widely used metrics are the Inception Score (IS) Sali- mans et al. (216)) and the Fréchet Inception Distance (FID) (Heusel et al. (2017), which aim to capture visual sample quality and sample diversity. On the one hand, IS was designed to evaluate single-object image generation on problems where the expected marginal distribution over classes is uniform, an unrealistic expectation in many real-world datasets with multi-object images. On the other hand, FID was introduced for unconditional image generation and yields a single distribution- specific score, which hinders the analysis of individual failure cases. To overcome the former, researchers have attempted to extend FID to the conditional generation case (DeVries et al., 2019) Benny et al.) 20200. To overcome the latter, researchers have proposed disentangle visual sam- ple quality and diversity into two different metrics (Shmelkoy et al.] 2018 Sajjadi et al. 2018 Kynkäänniemi et al. 2019 Ravuri & Vinyals 2019) by modilying the definition of precision-recal in different ways. Finally, the diversity of generated samples has also been quantified by measuring the perceptual similarity between generated image patches, as proposed by /Zhang et al. (0186))
Complex scene conditional generation. The conditional generation literature encompasses va- riety of input conditioning modalities. Several existing works focus on generating peato-realistic scenes from detailed semantic segmentation masks (Chen & Koltun 2017] Qi et al., 2018 Park cet al., 2019: Wang et al.) 2018 TTang et al. 2020). Although semantic segmentation masks are in- formation rich, they may be difficult to obtain with enough fine-graines details. This is especially relevant in applications where a user is expected to specify the input conditioning. As such, other commonly used input conditioning modalities include text descriptions (Reed et al. 2016 Hong jet al., 2018) Tan et al. 2 scene graphs @Johnson et al.) 2018 Ashual & Woll|2019 and bound- ing box layouts specifying the position and scale of the objects in the scene (Zhao et al. 2019 Sun &
2

---

Under review as a conference paper at ICLR 2021
Wu 2019 2020: Sylvain et al. 2020). In addition to those, recent works have focused generating bounding box layouts from a set of classes (Jyothi et al. 2019) and designing remarkably flexible pipelines to enable user interactions (Ashual & Wolf 2019). From a model perspective. complex scene conditional generation has been improved through the development of tailored class-aware (Park al.] (219),, instance-aware (Sun & Wu 2019: (020), and slass-and-instance-aaa izations (Sylvain et al., (020)) incorporated in the GAN generator. The task has also benefited from the introduction of refinement pipelines (Sun & Wu 2020), as well as the design of a multi-modal perceptual loss to drive scene graph embeddings and generated image embeddings close to each other (Sylvain et al., 2020). All aforementioned models are often trained on different number of training samples, use different backbone architectures, and report metrics on different splits and/or using different pre-trained models. This results in repeated reevaluations of baselines to fit each methods' setup, making it hard to pinpoint the most promising research directions.
normal-
3 METHODOLOGY
We start by defining three sets of data, composed of triplets of images and their respective finegrained and coarse conditionings. The finegrained conditionings represent detailed scene layouts by means of scene graphs, bounding boxes or segmentation masks. The coarse conditionings are an abstraction of the finegrained ones into a set of object labels present in the scene. For example, given a scene layout depicting two glasses on table, the corresponding set of labels is {glass, table).
Let D, be a set containing triplets of real images and their corresponding conditionings. We use D, to train the complex scene conditional generation models, and hereinafter refer to the conditionings used to train the models as seen conditionings. Let D,, be a set containing triplets of real images and their corresponding conditionings, composed of seen coarse conditionings but unseen finegrained conditionings. Note that we use the term unseen to refer to conditionings that are not present in D,- Similarly, let D be a set containing triplets of real images and their corresponding conditionings, composed of unseen coarse conditionings and unseen finegrained conditionings. We use Du and the finegrained conditionings
Duz only for evaluation purposes. After training the models, in D., Du, and D,= to generate three sets of model samples: S,, S, and S.a.
The goal of our analysis is three-fold: (1) assess how well each model fits the training data, (2) to assess each models' generalization to unseen finegrained conditionings, and (3) to assess each models' generalization to unseen coarse conditionings. To do so, we measure discrepancies between the distributions defined by the images in D., Du and D,a and by the images in S., S, and S,a, respectively. In particular, we evaluate the quality of the generation processes from both scene-wise and object-wise standpoints. Note that the scene-wiss evaluation considers full real images and full generated images, whereas the object-wise wise evaluation considers crops of individual objects from real and generated images. Throughout our analysis, we consider the following metrics to assess scene-wise and object-wise generation processes: precision, recall, consistency, FID, LPIPS-based diversity score (DS), and object classification accuracy or image(scene)-to- prediction Fl-score. In the remainder of this section, we will use image to refer to either full images or cropped objects.
We follow the definition of precision and recall introduced by Kynkäänniemi al. 2019). Intu- itively, precision captures the sample quality, whereas recall captures the coverage of the generated distribution. In partieular, precision verifies whether generated images lie within reference mani- fold of real images, and recall verifies whether real images lie within a reference manifold of gener- ated images. The real and generated manifolds are estimated in the feature space, which in our case is defined by a ResNext101 (Xie et al. 2017). and are obtained surrounding each data point with a hyper-sphere that reaches its k-th sitive to the number of samples used to estimate the reference manifolds, resulting in overestimated manifold volumes when the sample size is small. Hence, when [Dal and [Duae] are small, we esti- mate the manifold's 's hyper-sphere radii using the images in D, UUDUUUD Analogously, we estimate the manifold of generated images from S, U.S, U S.. Although successful conditional generation models should respect their conditionings and generate high quality samples which are consistent with those, the adopted definition of precision does not guarantee that the reference real sample and the generated sample found in its hyper-sphere will have the same coarse conditionings. Therefore, we complement precision with a metric to measure the consistency of the generated samples that fall within the estimated real manifold. More precisely, we compute the intersection over union
5) nearest neighbor. Therefore, precision and recall are sen-
3

---

Under review as a conference paper at ICLR 2021
1.0
70
1D5
60
50
PInjC F1 DS
140
0.6
100
0.6
40
0.5
2.6
F.88
100
70
0.2
0.2
0.0
C
F1
RD
05
0.0
AI
DS
(a) Scene-wise S.
(b) Scene-wise S.,
(c) Scene-wise Su
0.8
70
0.8
90
60
110
80
P. C./.cc
0.6
50
P/R/C/AS /AC.
0.6
10a
C Acc.
0.6
40
*
0.4
30
0.4
90
0.4
*
60
80
301
20
0.0
Azc.
FID
8D
0.0
FID
(d) Object-wise S,
(c) Object-wise S..
(f) Object-wise S.z
G2im
G2im-GT*
LostGANv1
LostGANv2
*
OC-GAN
Comparison of state-of-the-arr methods in terms of precision (P), recall (R), consistency (C), Fl-score (F1), object accuracy (Acc.), FID and DS for (a-d) seen conditionings (S.). (b-e) unseen finegrained conditionings (Sw). (c-f) unseen coarse conditionings (S,z). Plots show mean and std over 5 generation processes. *Trained using the open-sourced code. Best viewed in color.
Figure 1:
(loU) between the classes present the reference sample and those present in the conditioning of the generated sample. The consistency is averaged over all generated images, assigning consistency to samples lying outside the estimated manifold.
ANALYSIS
We perform our analysis using five recently introduced complex scene generation pipelines: (1)
G2im (Ashual & Wolf 2019); (2) G2im-GT (Ashual & Wolf] 2019) which is variant of G2im, where the instance segmentation masks and instance bounding box layouts are not generated, but directly taken from the ground-truth data; (3) LostGANv1 (Sun & Wul 2019); (4) LostGANv2 (Sun) & Wu (2020) which is an improved version of LostGANvI; and (5) OC-GAN (Sylvain et al. 2020).
Detailed description of these approaches can be found in the Appendix C
4.1 DATASET
We perform our analysis on the challenging COCO-Stuff (Caesar et al. 2018 given its ubiquitous use conditional image generation. The training and evaluation splits consist of 75k and 3k images respectively, following Sun & 2u19))19). We use the whole training set as D,, and divide the vali- dation set into three sets, namely D ... D,a and a set used for ayper-paramete search and validation D Note that DD is defined and used as in (Ashual & Wolf 2019). We use 128x 128 resolution worth noting that the proportions
images, as all studied methods support this resolution. Finally, of loig-tail classe?7]in the previously defined D., D., and D,, are 47%, 37% and 50%, respectively, resulting in more well represented classes in D., than D.,.
'D. included when computing reference manifold, and corresponding generated images S, are included when computing the manifold of generated images. We consider long-tail classes the top-25 frequent classes in the COCO-Stuff dataset.
4

---

Under review as a conference paper at ICLR 2021
1.0
0.8
0.0
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.0
0.0
0.0
R
(a) Scene Su (color), (whitet)
(b) Object (coooo))) ((""(white) (c) Object (color) St (white)
G2im*
G2im-GT
LostGANv1
LostGANv2
*
OC-GAN
Comparison of waae-offtheear methods in terms of precision (P), recall (R), and consis- tency (C), and matching class distributions for (a-b) seen and unseen finegrained conditionings, (c)
Figure 2:
seen and unseen coarse conditionings, *Trained using the open-sourced code. Best viewed in color.
4.2 FITTING SEEN CONDITIONINGS
We start by analyzing the quality of the generation processes of the methods from a scene-wise perspective All methods have been trained for 125 epochs to give them equal opportunity to fit the training data. Figure [aldepicts the precision (P), recall (R), consistency (C), Fl-score (F1), FID and DS achieved by the methods under study. We observe that all methods have rather high precision scores (~~ 90%). Despite its remarkable precision, G2Im and G2Im-GT suffer from poor recall and consistency. At the same time, LostGANvl and LostGANv2 exhibit at least 10% higher recall and consistency than G2Im and/or G2lm-GT, with LostGANv2 being slightly better than LostGANy], hence suggesting better generated distribution coverage and conditional consistency, which could be attributed to its mask refinement process and/or its larger number of parameters. Despite the superiority of LostGANs, their achieved consistency is only around 50%, indicating that image conditionings are often violated. Additionally, their recall is slightly below 60%, which implies that about 40% of the real images fall outside of the manifold of generated images. Moreover, when compared to Grid2Im-Grs LostGANs' superiority cannot be attributed having access to improved layouts (Grid2Im-GT uses ground truth layouts), but rather to their modeling choices. It is worth noting that although G2lm has higher precision than G2lm-GT, G2lm-GT obtains significantly better recall. This could suggest that Grid2Im is only able to model some modes of the layout distribution. If we consider Fl-score and FID, we see that the trend preserved. with LostGANs being the top performers. When it comes to DS, all methods behave similarly. Although recall can also be thought of as a diversity metric, it is concerned with diversity the whole image space, whereas DS tries to capture the intra-conditioning diversity. The absence of a clear leading method in terms of DS raises the question whether there could be any major improvement given the nature of the dataset, which contains single instantiation of each finegrained conditioning.
Figure Id shows similar trend for object-wise metries, with LostGANv2 scoring higher recall but similar precision and consistency than LostGANvI, suggesting that refined masks improve more scene than object quality. is worth noting that object-wise metrics exhibit lower scores than scene- wise metrics, suggesting that the generation processes might lead appealing generated scenes, which however suffer from high frequency details.
Finally, Figure 3 depicts two examples of images resulting from conditional generation processes from seen conditionings, showing in most cases recognizable objects in the scene, which could however be significantly improved, and emphasizing the quantitative superiority of LostGANy2.
5

---

Under review as a conference paper at ICLR 2021
dludd
g7355
mountain
know
layout
Ground-truth
G2im
G2im-3T
LostGANv1
Los1GAN22
ISLA+SG
ISLA+M+SG
(a) Generated images for seen conditionings
layout
Ground-truth
G2im
G2im-GT*
LosIGANv1
LosIGAN22
OC-GAN
ISLA+SG
ISLA+M+SG
(b) Generated images for unseen finegrained conditionings
pirplane
Such
88 layout
Ground-truth
G2im
G2in-CT'
LostGANv1
LostGAN:2
OC-GAN
ISLA+SG
ISLA+M+SG
(c) Generated images unseen coarse conditionings
Figure 3: Qualitative results. Generated images with resolution 128 x 128 for several methods.
4.3 GENERALIZATION TO UNSEEN FINEGRAINED CONDITIONINGS
Here we compare the quality of the generation processes all methods, when generating images from unseen finegrained conditionings (S.).. We start by assessing the methods' performances from both scene-wise and object-wise standpoints, see Figures [bb and Ic] In this case, we consistently that OC-GAN and LosIGANvI/v2 show the most competitive results across most of the met- rics. In particular, OC-GAN performs on par with the best version of LostGAN for both scene and object-wise precision, i.e. it matches the top scene precision performer and the top object precision performer at the same time, suggesting that this method does not trade scene precision for object precision. Moreover, OC-GAN consistently exhibits improved performance in terms of scene/object consistency, as well as scene F1-score and object accuracy, which highlights that its high scene/object precision also respects the (coarse) conditionings. When it comes FID, Lost- GANv2 leads both the scene-wise and object-wise rankings, reiterating their ability to generate high quality images from unseen finegrained conditionings. However, OC-G- OC-GAN's object-wise FID ranks the model as 3rd, together with G2Im-GT, whereas G2lm-GT achieves worse FID than OC-GAN scene-wise. This suggests that OC-GAN may suffer from poorer object diversity, resulting in lower FID, and that having access to high quality (GT) layouts may lead to better object quality. When comes to DS, OC-GAN has by far the lowest result, suggesting method trades image quality for intra-conditioning diversity. G2lm and G2lm- 2TIm-GTTe the weakest performers, with G2lm-GT achieving more competitive results.
TOC'sAN's code is not publicly available, nor are its pretrained models. The authors kindly shared their evaluation images S., and S. through personal communication. Those were obtained after a 170 epochs training process. Unfortunately, do not consider this method evaluate S, do compute its recall in the next sections.
could not get any generated images from the training conditionings, and so

---

Under review as a conference paper at ICLR 2021
1SP
1SR
1SC
YOP
'OR
1OC
[F1
Acc.
1DS
JSFID
JOFID
unseen finegranne conditionings S
ISLA
89.7
65.2
63.6
73.1
28.4
45.9
76.7
43.1
0,41
78.5
66.1
SPADE(oc)
90.9
57.0
65.7
70.2
26.6
41.5
75.9
39.3
0.17
78.7
67.6
ISLA M
91.6
57.7
68.1
73.8
31.6
49.3
77.0
45.4
0.12
75.2
61.7
ISLA SG
64.9
66.1
70.5
32.2
45.3
78.9
0,41
78.4
62.6
ISLA + M + SG
92.1
58.8
72.5
72.6
28.7
50.6
80.8
49.3
0.18
72.6
59.2
unseen coarse conditionings Sy
ISLA
88.4
58.5
45.9
74.7
24.9
40.5
59.6
30.7
0.44
53.5
38.2
SPADE(oc)
90.1
53.2
47.6
73.6
24.5
35.7
59.2
26.4
0.20
55.6
40.2
ISLA + M
91.7
55.3
53.1
77.7
28.1
43.4
61.1
29.0
0.11
52.3
36.0
ISLA + SG
88.4
59.1
47.2
74.2
28.3
42.7
62.2
31.4
0.44
52.1
34.2
ISLA M + SG
91.8
54.8
55.9
74.3
25.0
43.7
64.3
34.5
0.19
50.2
33.2
Table I: Ablation results. We report average precision, recall and consistency for scenes (SP, SR, SC) and objects (OP, OR, OC), Fl-score, Accuracy (Acc.), DS, scene and object FIDs (SFID and OFID). Results show the mean over 5 random seeds at test time. Extended table in Appendix
When analyzing the generalization gap, for both the seen conditionings (S.) and the unseen finegrained ones (S.), whereas object-wise pre- cision is slightly better for S, than for S,,. However, recall and consistency show consistently better scores for S., than for S. (see Figures Ta
that scene-wise precision achieves the same score
Tel. We hypothesize that the lower presence of long-tail classes in S,, results better metrics for this split. To confirm the hypothesis, we sub- sample D. to match the same class distribution as in D., creating the split Db=u, and generate the corresponding set of model samples S Results are presented in Figure 2 and show that for Sh=" the recall performance gap with S, is roughly closed, whereas the consistency performance gap becomes smaller. Overall, the results observed for S, indicate good generalization to unseen finegrained conditionings for all methods. This is further supported by the visualizations in Fig- 3 where the quality of generated images for unseen finegrained conditionings equates the one of generated images for seen conditionings, suggesting that state-of-the oftthe-art models properly exploit the compositionality the scenes, and as such are able to generate images with recognizable objects from unseen finegrained conditionings.
1b
4.4 GENERALIZATION TO UNSEEN COARSE CONDITIONINGS
Here we compare the quality of the generation processes for all methods, when generating images from unseen coarse conditionings (S,²). We start by assessing the methods' performances from both scene-wise (see Figure [c] and object-wise (see Figure If standpoints. Similarly to previous experiments, we observe that all methods achieve high precision values, especially scene-wise, and relatively low recall and consistency values. G2Im exhibits competitive results in terms of scene average precision but lags behind in the vast majority of the metrics. Interestingly, the high scene precision of G2Im does not translate to its object precision, suggesting that the method does not capture enough high frequency detail. The overall quality of the scene generation process of OC- GAN is mostly on par with the one of LostGANv 1v2, achieving similar precision and consistency and slightly lower scene FID score. However, OC-GAN results in lower FID scores object-wise, suggesting that OC-GAN suffers from object diversity, as in S,, Once again, OC-GAN shines when comes preserving conditional consistency (see consistency, FI, object accuracy metrics) and but appears to be the weakest in terms of DS.
Interestingly, when analyzing the generalization gap, we see that scene- and object-wise pre- cision and recall achieve roughly the same scores for the seen conditionings (S,) and the unseen coarse (S,²) ones. However, scene and object consistencies are more accurate for S, than for S,2. If we compare the generalization to S,, against the generalization to S,e, we observe that for scenes and objects, S,z presents lower recall and consistency for most of the methods. Further, object-wise precision is also affected for S,a, altering the methods' ranking. We note that OC-GAN loses its 1st ranked position (shared with the LostGANs) in S, to become 3rd (together with G2Im-Gi) GT) in S,,, and suffers important loss in terms of object FID, suggesting that OC-GAN may suffer par- ticularly from unseen coarse conditionings. For completeness, and to assess whether the marginally lower recall and consistency of S,2 come from the heavier long tail of S,2, subsample D. to

---

Under review as a conference paper at ICLR 2021
match the same class distribution in Dus. In this case. this experiment can only be performed object-wise, as none of the class combinations in Dua appears in D.. The results presented in Fig- ure 2 show that precision, recall and consistency of both splits are very similar.
Finally, looking at Figure g we observe that images generated from unseen coarse conditionings present lower quality than images generated from seen unseen finegrained conditionings for all methods. For instance, if we consider the conditioning in Figure Bb. which depicts a person ski- ing with dog, we observe that the overall quality of the generated images is rather low, with oversmoothed or implausible looking persons and, in some cases, invisible dogs. Similarly, if we consider the conditioning of an airplane on a paper, most of the resulting generated images also exhibit rather low quality. The most compelling image is the one generated by G2lm-GT GT,, which di- rectly leverages a high quality (GT) layout and can focus on filling in the object textures. Additional qualitative results can be found in Appendix F
4.5 ABLATION OF PIPELINE COMPONENTS
In this subsection, we switch the gears of the analysis and design an ablation study to better under- stand how elements of complex scene generation pipelines affect their performance. In particular, we inspect components that were either presented as a novelty in the methods under study or that are an important defining factor of these approaches. More precisely, we analyze the following components: (1) the spatial conditioning normalization module; (2) the effect of imperfect instance segmentation mask prediction on the image quality; and (3) the influence of scene graph loss in- troduced in OC-GAN. We carry our analysis on models derived from LosIGANv2. Note that the multi-stage generation process introduced in (Sun & Wu 2020) constitutes the fundamental dif- ference between LostGANvI and LostGANv2 and, such, is already under study in Subsections We use hyperbands (Li et al. 2017) to find the best hyperparameters for each model.
Spatial conditioning normalization modules. We compare LostGANv2's ISLA-norm the vari- ant of SPADE introduced in OC-GAN, hereinafter called SPADE(oe). The results of these com- parisons are presented in Table T both for unseen finegrained conditionings and for unseen coarse conditionings. When comparing ISLA to SPADE(oc) in the unseen finegrained conditionings sce- nario, we observe that ISLA improves upon diversity metrics such as scene recall, object recall and DS on average. The high scene variation achieved by ISLA could possibly be explained given it design, which leverages latent vectors to compute instance-wist normalizing parameters. Interest- ingly, ISLA also favors object-wise metrics, by improving object consistency and accuracy. The same trend can be observed in the case of unseen coarse conditionings. This might be due to the fact that ISLA considers individual objects information through its input. However, SPADE(oc) dominates the scene-wise metrics, leading to better scene precision and scene consistency on av- erage. This might be due to the module's design, which directly exploits bounding box semantic segmentation masks. These results suggest that exploiting scene compositionality in the generator's spatial conditionings (as in ISLA) helps improve generalization. Additional ablations with variants of SPADE can be found in Appendix
Ground-truth masks (M). As expected, using GT masks in ISLA+M greatly improves over ISLA. In particular, for S., using ground truth masks improves upon most of the ISLA results on aver- age, at the expense of scene recall and DS decrease. We observe the same trend for S,2, where improvements are mainly observed in terms of object and scene precision and consistency metrics. Again, using GT masks reduces the scene layout variability and translates into lower diversity- related scores. Moreover, using unseen GT masks forces the generator to hallucinate objects with unseen shapes. This may in some cases lead to undesirable outcomes, with worse generated image quality than when using predicted masks. However, generating good quality masks remains one of the most promising directions to improve generalization.
Scene-graph similarity module (SG). When endowing ISLA with SG (ISLA+SG), we observe notable boosts in scene-wise conditional consistency metries in S, (see scene consistency and FI). This is not surprising given that the module's goal is to match information from scene graphs and generated images to encode the location of specific objects in the scene. It is worth noting that adding the SG module does not come at the expense of intra-conditionin diversity but harms scene recall on average. In S,,,, we observe a marked increase in object recall, object consistency, F1, and both FIDs, while noticing slight decrease in terms of average precision, emphasizing the general
8

---

Under review as a conference paper at ICLR 2021
benefits of the SG module for unseen coarse conditionings. When using both GT masks and the SG module (ISLA+M+SG), we observe further improvements. For both S, and S,,, ISLA+M+SG achieves the highest results in terms of average scene precision, consistency, F1, as well as FID, and improves upon object consistency, accuracy and FID, while harming diversity metrics. Note that ISLA+M+SG slightly mitigates the decrease in DS/recall experienced by ISLA+M. The compelling improvements of ISLA+M+SG suggest that the quality of the generated masks plays a crucial role in improving generalization
unseen conditionings. Figure shows generated images for both variants of the model with the SG module. We can see that the generated images display high frequency details for the scenarios of seen and unseen finegrained conditionings. For unseen coarse conditionings, we can appreciate better results for ISLA+SG that could be combining both instance- wise compositionality from ISLA and the consistency boost SG module. Moreover, ISLA+M+SC leverages GT masks, showing further quality improvement. The advantages of using SG suggest that semantically aware training losses promote generalization and opens up future research avenues in this direction.
Finally, as a byproduct our ablation, we were able to come up with model modifications that lead to superior generalization, achieving state-of-the-an results with ISLA+SG. In particular, outperform LostGANv2 denoted as ISLA in Table - which was the best method in terms of FID the date. More precisely, we achieve lower scores for both scene and object FID and for both generalization scenarios (S,, and S,,), while matching or surpassing for the other reported metrics. For more detailed comparison with state-of-the-ar methods, refer to Appendix H
5 CONCLUSIONS
We proposed methodology to analyze current complex scene conditional generation methods. Us- ing this methodology, we evaluated recent complex scene generation pipelines from the points of view of training data fitting as well as their ability to generalize unseen scenes. Through an exten- sive evaluation, we observed that current methods: (I) fit the training distribution with a moderate success, generating recognizable scenes which can however still be notably improved enhance the overall quality of the generation process; (2) display decent generalization to unseen finegrained conditionings; (3) have significant space for improvement when it comes to generating images from unseen coarse conditionings; and (4) the individual quality of the generated objects generally suf- fers from missing high frequency details, especially for objects belonging to the long tail classes of the dataset. Thus, complex scene generation is still a stretch goal, especially for unseen condi- tionings, where the methods' performances are still in their infancy. To provide some pointers on how to move towards high quality complex scenes generation for unseen scenarios, we performed an ablation study that highlights the importance of model components when it comes to general- ization, and show that by exploiting our findings we are able achieve new state-of-the-art in most reported metrics. We note that promising avenues to improve generalization in complex scene conditional generation include: (1) incorporating compositional priors in the generator; (2) moving towards semantically aware training losses; and (3) focusing on improving mask generation as way to improve scene generation quality.
REFERENCES
Oron Ashual and Lior Wolf. Specifying object attributes and relations in interactive scene genera- tion. In Proceedings of the IEEE International Conference on Computer Vision, pp. 4561-4569, 2019.
Yaniv Benny, Tomer Galanti, Sagie Benaim, and Lior Wolf. Evaluation metrics for conditional image generation. arXiv preprint arXiv:2004 /2361, 2020.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference on Learning Representations, 2019. URL
htttps://pperrvviee wet.iiewwweeeeeeeeeeeeeeeeeeeeee for.mtidd http:///geeeeeeeeeee.eeeeocottts
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Computer vision and pattern recognition (CVPR), 2018 IEEE conference on. IEFE, 2018.

---

Under review as a conference paper at ICLR 2021
Qifeng Chen and Vladlen Koltun. Photographic image synthesis with cascaded refinement networks.
In Proceedings of the IEEE international conference on computer vision, pp. 1511-1520, 2017.
Terrance DeVries, Adriana Romero, Luis Pineda, Graham W Taylor, and Michal Drozdzal. On the evaluation of conditional gans. arXiv preprint www.iv:107.08111 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances neural infor- mation processing systems, pp. 2672-2680, 2014.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In Advances in neural information processing systems, pp. 6626-6637, 2017.
Tobias Hinz, Stefan Heinrich, and Stefan Wermter. Generating multiple objects at spatially distinct locations. In International Conference on Learning Representations, 2019. URL https://
openreview beeerreviwwnnt//fee itt
Seunghoon Hong, Dingdong Yang, Jongwook Choi, and Honglak Lee. Inferring semantic layout for hierarchical text-to-image synthesis. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1219-1228, 2018.
Akash Abdu Jyothi, Thibaut Durand, Jiawei He, Leonid Sigal, and Greg Mori. Layoutvae: Stochas- tic scene layout generation from a label set. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9895-9904, 2019.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. ICLR, 2014.
Karol Kurach, Mario Lucic, Xiaohua Zhai, Marcin Michalski, and Sylvain Gelly. The gan landscape: Losses, architectures, regularization, and normalization. arXiv preprint 18Xiv:1807 (rXX:::807.04720, 2018.
Tuomas Kynkäänniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Improved precision and recall metric for assessing generative models. In Advances in Neural Information Processing Systems. pp. 3929-3938, 2019.
Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar. Hyperband: A novel bandit- based approach to hyperparameter optimization. The Journal of Machine Learning Research, 18(1):6765- 6816, 2017.
Mario Lucic, Karol Kurach. Marein Michalski, Sylvain Gelly, and Olivier Bousquet. Are gans created equal? a large-scale study. In Advances in neural information processing systems, pp- 700-709, 2018.
Takeru Miyato and Masanori Koyama. cGANs with projection discriminator. In International Conference on Learning Represemtations, 2018. URU/ttps://ppeneeiiee http://pppwnnei....eeeeeooooo//a
Id-ByS1VpgRZ
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018.
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Luis Pineda, Amaia Salvador, Michal Drozdzal, and Adriana Romero. Elucidating image-to-set prediction; An analysis of models, losses and datasets. CoRR, abs/1904.05709, 2019. URL http:/ / org. abs/ 904 111./1909400090999
Xiaojuan Qi, Qifeng Chen, Jiaya Jia, and Vladlen Koltun. Semi Semi-parametrie image synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8808- 8816, 2018
10

---

Under review as a conference paper at ICLR 2021
Suman Ravuri and Oriol Vinyals. Classification accuracy score for conditional generative models. In H. Wallach, H. Larochelle, A. Beygelzimer, F. Alché-Buc, E. Fox, and R. Gamett (eds.), Advances in Neural Information Processing Systems 32, pp. 12268 12279. Curran Associates, Inc., 2019. URL. http:::/pppprrr.nnns..coo. pdf
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. arXiv preprint arXiv: 2016.
Mehdi SM Sajjadi, Olivier Bachem, Mario Lucie, Olivier Bousquet, and Sylvain Gelly. Assess- ing generative models via precision and recall. Advances Neural Information Processing Systems, pp. 5228-5237, 2018.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016.
K. Shmelkov, C. Schmid, and K. Alahari. How good is my GAN? In Proceedings of European Conference on Computer Vision, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1409, 1556, 2014.
Wei Sun and Tianfu Wu. Image synthesis from reconfigurable layout and style. In Proceedings of the IEEE International Conference on Computer Vision, pp. 10531-10540, 2019.
Wei Sun and Tianfu Wu. Learning layout and style reconfigurable gans for controllable image synthesis. arXiv preprint arXiv:2003.11571 2020.
Tristan Sylvain, Pengchuan Zhang, Yoshua Bengio, R Devon Hjelm, and Shikhar Sharma. Object- centric image generation from layouts. arXiv preprint arXiv:2003.07449 2020.
Fuwen Tan, Song Feng, and Vicente Ordonez. Text2scene: Generating compositional scenes from textual descriptions. CVPR, 2019.
Hao Tang, Dan Xu, Yan Yan, Philip HS Torr, and Nicu Sebe. Local class-specific and global image- level generative adversarial networks for semantic-guided scene generation. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7870-7879, 2020.
Aäron van den Oord, Nal Kalchbrenner, and Koray Kavukcuoglu. Pixel recurrent neural networks. In Proceedings of the 33rd International Conference on International Conference Machine Learning Volume 48, ICML' 16, pp. 1747-1756. JMLR.org, 2016a. URL http:://d acm.
org/ org/citation.c.. cf 300t:mndd=33433300 00 3045575
Aäron van den Oord, Sander Dicleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alexander Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In Arxiv, 2016b. URL
03499
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 613 621. Curran Associates, Inc., 2016. URL ltttp://paper: nips cttpppec/p//////c/ 61 1194-generat ng- www.wwidwwwwie..
D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and
axcene-dyname cs.
Ting- Chun Wang, Ming- Yu Liu, Jun Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. High- resolution image synthesis and semantic manipulation with conditional gans. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8798-8807, 2018.
S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5987-5995, 2017.
11

---

Under review as a conference paper at ICLR 2021
Qiantong Xu, Gao Huang, Yang Yuan, Chuan Guo, Yu Sun, Felix Wu, and Kilian Weinberger. An empirical study on evaluation metrics of generative adversarial networks. arXiv preprint (ttl:8800.0775.55 2018.
Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang. Xiaogang Wang, Xiaolei Huang, and Dim- itris N. Metaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial networks. IEEE Transactions on Puttern Analysis and Machine Intelligence, July 2018a.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features as a perceptual metric. In CVPR, 2018b.
Bo Zhao, Lili Meng, Weidong Yin, and Leonid Sigal. Image generation from layout. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2019.
A EXTENDED EXPERIMENTAL SETUP
Dataset details. We use COCO-Stuff (Caesar et al. 2018)), with training and evaluation splits consisting of 75k and 3k images respectively, following SSnn & Wu (2019). We use the whole training set as DD. and divide the validation set into three sets, namely D.. Duz and a set used for hyper-paramette search and validation D,.. This results in [D,] 75k, [Dwl 675, [Pua| and [D,| = 1024 triplets to evaluate the quality of the generation processes. We use the above described splits to compute the scene-wise metrics, and crop individual objects from the scenes to compute object-wise metrics. In particular, we consider objects from things categories exclusively, given that crops of stuff categories most often also contain foreground objects from things categories. This results in 145k objects crops extracted from D., 1163 from D.., 2706 from D, and 2062 from D.. In both cases and only for evaluation purposes, we preprocessed the data sets by merging redundant and prone-to-confusion classes (see Appendix B for a detailed explanation of the dataset preprocessing).
1375
Evaluation networks. Our object classifier has ResNext101 backbone and is trained indi- vidual objects cropped from the images in D,. Similarly, the imagc(scene)-to- predictor has a ResNext101 backbone and is trained on D.. In both cases, the architecture and optimization hyper- parameters are chosen independently on Dr and by means of the Hyperband algorithm (Li et al.] 2017), following (Pineda al. 2019). Both evaluation networks are trained on the merged COCO- Stuff categories.
Evaluation details. We report all results with 128x128 images, as all studied methods support this resolution size. The image features to compute precision, recall and consistency metrics are extracted from the evaluation networks (image-to-set or object classifier). In both cases, we use the features from the last fully connected layer before the classifier. For each finegrained conditioning, we generate images with each method from 5 random seeds, which change the input noise fed to the generators, and so allow for some image variation. The metrics are computed and averaged across the 5 seeds for S,, S, and Sy².
B DATASET DETAILS AND CATEGORIZATION CLEANING
In COCO-Stufi the distinctions between stuff categories and some thing categories is difficult to grasp even for humans. Analyzing a confusion matrix extracted for feature extractor on object crops and some designed rules, we cleaned the COCO-Stuff category list minimize mistakes in evaluation due to poor dataset categorization. Note that is is only done for evaluation, since all studied generative models have been trained with the original COCO-Stuff categories.
We extracted VGG16 Simonyan & Zisserman (014) features for all cropped training objects. We computed pair-wise Euclidean distances between all feature crops, and assigned as the predicted class the one corresponding to the closest crop. Using these class assignations, we obtain confusion matrix, that illustrates which are the most confused class and by what other classes they are confused. To decide which classes to merge, we applied the following steps:
For each target class class, we took the 5-top most confused classes that have, at least, the same probability the target class.
12

---

Under review as a conference paper at ICLR 2021
We discarded the class 'person' from any possible confusion, as this miss-classification can come from the bounding box crop.
We also do not consider some confused classes where the confusion can come from the bounding box: for instance, confusing 'building' crops for "tree" or "sky" class.
We did not mix stuff classes that contain the suffix "-other', as they usually contain a high variety of classes by themselves that are difficult to categorize. Exception: when the entire super-class (grouping of COCO-Stuff classes) can be mixed into one, including classes with suffix '-other'.
Only merge classes that by individual inspection, are semantically close.
C METHODS UNDER STUDY
G2im (Ashual& Wolf, 2019) uses a scene graph as input conditioning, where nodes represent ob- jects in the scene with their class annotation, and edges represent the relative positions among those objects. The scene graph used to generate instance segmentation masks and instance bounding box layouts, which are fused to obtain a semantic segmentation mask of the scene. The result- ing mask then combined with object features, obtained separately by feeding object crops from training images to an auto-encoder, to create the final generated image.
G2im-GT (Ashual & Wolf 2019) is a variant of G2im, where the instance segmentation masks and instance bounding box layouts are not generated, but directly taken from the ground-truth data.
LostGANv1 (Sun & Wu 2019) uses instance bounding box layouts with their corresponding class annotations input conditioning. This input is used to compute object embeddings, which are subsequently used to generate instance segmentation masks. The instance segmentation masks are then fed to scene generator which uses a spatial conditioning module, called ISLA-norm, that learns independent normalization parameters per object instance. These normalization parameters are applied through all stages of the scene generation process.
LostGANv2 (Sun & Wu 2020) takes the same input conditioning, and computes the same object embeddings LosIGANvI. In this case, both the instance segmentation mask generator and the scene generator are extended to follow a multi-stage generation process. In particular, the instance segmentation masks are refined using the mask from the previous stage and some intermediate image features from the scene generator. In turn, each time the mask is refined, it is used to update the normalization parameters used in the scene generator, therefore using stage-wiss wise ISLA-norm
OC-GAN (Sylvain et al. 2020) takes the same input conditioning as the LostGANs. In contrast to LostGANv ISLA- OC-GAN uses variant of SPADE's normalization module (Park al.] 2019), which spatially conditions the scene generation on instance segmentation masks, per instance bounding box boundaries, as well as semantic bounding box layout. Moreover, OC-GAN uses a scene-graph similarity module, called SGSM, that attends scene visual features with graph features, and matches global and local information from both the scene graph and the generated image to encode the location of specific objects in the scene. This scene graph similarity module is used at training time but discarded at test time.
D METRIC TABLES
Metries reported in the analysis are detailed as follows: Table presents the Fl-score and accuracy, Table 3 the diversity score, and Table the FID values for both scenes and objects.
E EXTENDED ABLATION METRICS
The variants of SPADE that are ablated are the following: (1) SPADE uses a global semantic seg- mentation mask and object boundaries in (Park et al.) 2019) (2) SPADE()) that uses object em- beddings and instance masks, (3) SPADE(s) that uses a global semantic segmentation mask obtained from fusing instance masks, and (4) SPADE(OC) that uses object embeddings, instance segmentation masks, object boundaries and a bounding box semantic layout, as proposed by /Sylvann et al. (2020)
The results of these comparisons presented in detail in Tables
13

---

Under review as a conference paper at ICLR 2021
Mean Fl-score
Mean accuracy
S
S
S
S
Su
S
GT
75.2 + 0.0
64.7
= 0.0
61.0 0.0
70.1 0.0
54.8 0.0
G2im*
41.2
0.5
38.1
12.6
19.7
9.7
G2im-GT*
5.1
5.7
0.5
24.2
312
19.
LostGANvI
6.17
7..8
57.7 0.4
30.9 0.1
41.0
26.8
LostGANv2
64.8 0.1
0.1
58.6 0.3
30.8 0.1
40.6
26.3 = 0.5
OC-GAN
80.0 - 0.1
63.8 = 0.3
49.6 0.9
36.7 - 0.5
Table 2: Fi-score and accuracy averaged across examples in splits S,, S, and S, This table presents the mean and standard deviation of different random seeds for the generative models. Trained using the open-sourced code.
t DS
S.
S,
S
GT
G2im
0.56 + 0.06
0.55 0.07
0.56 0.06
G2im-GT*
0.48 + 0.09
0.47 0.11
0.48 + 0.09
LostGANv1
0.47 + 0.09
0.46 0.09
0.49 + 0.08
LosIGANv2
0.45 + 0.09
0.43 0.09
0.46 - 0.08
OC-GAN
0.14 0.07
0.13 + 0.06
Table 3: Diversity score using LPIPS metric in all evaluation splits S,, Sus S.z. Trained using the open-sourced code.
ADDITIONAL QUALITATIVE RESULTS
Figure shows qualitative results, for state-of-the-art methods and two of the best ablated pipelines.
G ADDITIONAL METRICS PER CLASS
Figure 5 shows the class-wise precision, recall and consistency for the top 10 most represented foreground classes in the training set. Comparing the metries S,, and Sya to the ones S., observe that: G2im exhibits lower recall for giraffe and zebra in S., while all other class-wise metrics are similar and equally low across splits, suggesting that the method generates poor quality objects for all splits, consistency for car class, with only lower precision for caf and lower recall for zebra and giraffe; however, we cannot observe lower metrics for the S,, split. These observations could be explained by, on one hand, the fact that metrics are still fairly low compared to LostGANs and OC-GAN to observe significant differences between splits; on the other hand, using ground-truth masks could positively impact generalization for unseen splits as seen in Section 4.5 LostGANy1 generally achieves higher scores than G2im and G2im-GT, but shows some signs of lower generalization for some classes in both splits. In particular, in S,, classes giraffe and zebra exhibit lower recall, while Su², cat, dog and truck show lower recall; as an outlier, giraffe scores higher recall. Finally, for LostGANy2,
in Sections 4.2
4.3 and 4.4 G2im-GT exhibits higher precision and
can observe decrease in S., in terms of precision for the classes and dog. while giraffe and airplane exhibit lower recall. In S,2, the classes cat and zebra exhibit lower precision, while person, cat, dog and train present lower recall; finally, zebra and airplane score lower consistency. This suggests that more classes suffer from lower metric scores in S, than in S, (in line with Section 4.4).
for well represented classes such the top 10 studied classes.
Note that we cannot compare to metrics in S, for OC-GAN, as the code is not publicly available nor its pretrained models. The authors kindly shared their evaluation images in S,, and S,2.
14

---

Under review as a conference paper at ICLR 2021
FID
Object FID
S.
S.,
S
S+
S.
S
GT
0.0
0.0
0.0
0.0
0.0
0.0
G2im+
62.3 0.1
137.6 1.6
99.9
1.0
5..2
112.3 1.5
80.8 01.::0.6
G2im-GT*
40.7 +: 0.1
112.9
82.1 =
31.0
0.5
85.2
0.9
55.8
LostGANv1
15.2 0.1
84.2 1.0
57.2 0.4
20.3 = 0.2
71.9 0.5
45.2 0.2
LostGANv2
12.8
80.0
= 0.4
55.2 0.5
15.5 15.5 0.6
67.1 0.7
40.9 - 0.3
OC-GAN
85.8 - 0.5
60.1 = 0.2
85.8 0.6
59.4 + 0.3
Table 4: FID for each data split and model. As a reference, ground-truth images from each of the D., D,, and Dua splits are used accordingly.BB Bothh splits have different number of points, so FID not comparable across splits. This table presents the mean and standard deviation of 5 different random seeds for the generative models." Trained using the open-sourced code.
Scene Precision
Scene Recall
Scene Consistency
S,
S,²
S.
S.2
S. S,²
LostGANv2
89.1 + 1.4
87.5 + 0.3
63.9 + 2.0
58.0 + 0.9
61.1 + 0,9
43.5 + 0.4
OC-GAN
89.2 1 0.9
86.9 1 0.6
63.6 1 1.0
45.8 1 0.5
ISLA
9.9
88.4 0.5
65.2 + 1.1
58.5 + 0.7
63.6 1.2
45.9 0.8
SPADE
0.4
$ 0.5
49.2 0.5
61.5 0.6
06.0
SPADE())
0.7
85.9 0.5
53.0 + 0.5
46.3 0.8
49.5 0.4
37.7 0.7
SPADE(s)
86.3 9.0
+ 1.2
44.0 + 0.7
53.1 0.4
38.4
SPADE(oc)
0.5
90.1 0.6
57.0 + 0.5
53.2 + 0.7
65.7 + 0.7
47.6 - 0.3
ISLA+M
91.7 91.6+
57.7 +
55.3 + 0.6
68.1
53.1 +
ISLA+SG
+
0.7
64.9 + 0.7
59.1 + 0.4
66.1 1.1
47.2 + 0.5
ISLA+M+SG
92.1 0.5
91.8 0.4
58.8
+
1.4
54.8
+ 0.6
72.5 + 0.7
55.9 + 0.6
Table 5: Ablation results for ablated pipelines, scene-wise precision, recall and coverage for the two splits S, and S,2. This table presents the mean and standard deviation of 5 random seeds that control the input noise.
All in all, class-wise metrics follow trend similar to the class averaged object-wise metrics. More specifically, we observe a metric degradation in Su, which further accentuated in S,2, while results in S, present better metrics, can be seen for LostGANv2. However, there is no strong correlation between a specific class and the generalization capabilities of each model. Trends are better observed in Figures ld Te and where the metrics are averaged across all foreground classes in the dataset.
1 Object Precision
Object Recall
1 Object Consistency
S.,
S,²
S.
S,
S.
S,²
LostGANv2
69.6 + 0.3
73.2 0.5
30.2 + 1.2
25.8 + 0.5
40.9 0.7
36.3 + 0.6
OC-GAN
70.4 1 0.7
70.9
0.8
48,2
42.1 +
ISLA
73.1
74.7 0.6
28.4 0.8
24.9 0.7
45,9 0.9
40.5 =
SPADE
0.6
0.5
24.5 + 0.6
20.4 - 0.4
38.1 0.9
32.5 0.5
SPADE()
0.8
21.1
19.5 1 0.6
1
32.4 0.4
SPADE(S)
1.2
72.0
21.7
18.9 0.6
35.0
31.8
SPADE(Oc)
7.92
7 0.4
26.6 1.1
24.5 0.5
41.5 0.7
3
ISLA+M
73.8
77.7 + 0.3
31.6 0.2
28.1 + 0.8
49.3 0.6
43.4 + 0.9
ISLA+SG
+ 1.4
74.2 1.0
32.2 + 0.8
28.3 + 0.5
45.3 + 1.3
42.7 + 1.1
ISLA+M+SG
72.6
74.3 0.7
28.7 1.2
1.2
25.0 + 0.4
50.6 + 1.2
43.7 0.7
Table 6: Ablation results for all ablated pipelines. Metrics reported object-wise precision, recall and coverage for the two splits S, and S,2.This table presents the mean and standard deviation 5 random seeds that control the input noise.
15

---

Under review as a conference paper at ICLR 2021
Mean F1-score
Mean accuracy
t DS
S.=
Su
S.a
S.
S,2
ISLA
76.7 + 0.3
59.6 0.2
43.1
0.7
30.7 + 0.3
0.41 + 0.10
0.44 + 0.09
SPADE
74.1
37.6
39.8 + 0.6
25.0 0.3
0.01 0.01
0.01 0.01
SPADE())
52.6 0.5
32.1 0.6
23.8
0.26
0.26
SPADE(S)
6.49
53.2 0.3
0.7
0.5
0.15
0.17
SPADE(oc)
0.5
59.2 = 0.3
0.9
26.4 1.0
0.17
0,20 0.07
ISLA+M
0.3
= 0.2
45.4
29.0
0.12
0.11 = 0.06
ISLA+SG
78.9 28.9
62.2 = 0.1
42.1 1 0.6
31.4 0.8
0.41 0.10
0.44 = 0.08
ISLA+M+SG
80.8 - 0.4
64.3 = 0.4
49.3 + 1.0
34.5 0.3
0.18 = 0.07
0.19 = 0.08
Table Ablation results for all ablated pipelines. Fc-sco accuracy and DS for the two splits S. and table presents the mean and standard deviation of 5 random seeds that control the input noise.
# Parameters
4 FID
Object FID
Generator
Da
D.a
D.
D.a
SLA
40.5M
78.5 + 0.3
0.5 53.5 0.5
66.1
38.2 = 0,3
SPADE
40.8M
83.8 8.3
62.2
0.1
44,9 0.9
SPADE()
40.8M
9.4
0.4
76.3
44.5
SPADE(S)
40.8M
9.74
0.3
44.1 7.3
SPADE(oc)
41.7M
78.7 0.5
0.2
67.6 0.5
40.2
ISLA+M
35.3M
75.2 +
0.1
61.7 0.1
36.0
ISLA+SG
47.5M
78.4 + 0.6
52.1 0.2
62.6 -
34.2
ISLA+M+SG
42.3M
72.6 + 0.5
50.2
- 0.2
59.2 + 0.2
33.2 - 0.1
Table Ablation results for all ablated pipelines. Number of parameters per generator for each of the pipelines, scene FID and object FID for the two splits S, and S,e.This table presents the mean and standard deviation of random seeds that control the input noise.
16

---

Under review as a conference paper at ICLR 2021
sky-other
iifboard
pea
Tence
persorpe senoo
playingfield
BB layout
Ground-truth
G2im
G2im-GT*
LostGANv1
LostGANv2
ISLA+SG
ISLA+M+SG
(a) Generated images for seen conditionings
blyingfe
BB layout
Ground-truth
G2im
G2im-GT
LostGANv1
LostGANv2
OC-GAN
ISLA+SG
ISLA+M+SG
(b) Generated images unseen finegrained conditionings
Baadd
hathrrather
Tumbure OFFICE
d0th
sensan
BB layout Ground-Iruth
G2im
G2im-GT
LostGANv1
LosIGANv2
3C-GAN
ISLA+SG
ISLA+M+SG
(c) Generated images for unseen coarse conditionings
Figure 4: Extended qualitative results. Generated images with resolution 128 x 128 for several
methods. * Trained using the open-sourced code.
H STATE-OF-THE-ART RESULTS FOR OUR BEST ABLATED METHOD
We consider best model to be ISLA+SG, stated in Section 4.5 avoid leveraging ground- truth masks as ISLA+SG+M and allow a fair comparison with LostGANy2 and OC-GAN, the two best existing methods as shown in Sections 4.2 4.3 and 4.4
In Table 9 we provide a direct comparison between LostGANv2, OC-GAN and our ISLA+SG in terms of: average scene precision, recall and consistency (also found with standard deviation and
17

---

Under review as a conference paper at ICLR 2021
1.0
1.0
1.0
0.0
0.8
0.8
O
0.6
Precision
0.6
Precision
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
person
car
intor
deg
bua
nippase
piratic
bia
treck
niplaae
parson
Gat
car
giatte
matra
dng
tha
macl
nepiane
train
(a) Class-wise in
(b) Class wise
(c) Class wise P Pin S.²
0.5
0.5
0.5
0.3
0.3
Recall
0.2
0.2
0.0
0.0
0.0
Rerson
Car
giraffe
retira
dagg
Nose
track
airllans
pesson
car
weaffe
zetrr
track
airptome Fran
Rerssos
car
car
piatfe
deg
toxt
treef
airiore
train
(d) Class-wise in
(e) Class-wise in S.
Class-wise R in S,
0.8
0.8
0.6
0.6
0.6
Consistency
Consistency
0.4
0.4
0.2
0.2
0.0
0.0
0.0
person
cat
car
pirafte
rntora
dagg
bis
trick
airplane
person
eat
piratte
zetrra
das
bist
triek
airitane
Persson
giaate
radra
dig
bist
track
airpione
trai?
(g) Class-wise C in S
(h) Class-wise C in S
Class-wise C in S
G2im*
G2im-GT
LostGANv1
LostGANv2
* OC-GAN
Figure 5: Class-wise comparison of state-of-the-arr methods in terms of object-wise precision (P), recall (R) and consistency (C) (a.d.g) for seen conditionings (S,), (b,c.b) unseen finegrained condi- tionings (8.). (c.f.i) unseen coarse conditionings (S,²). Plots show mean and std over 5 generation processes. "Trained using the open-sourced code. Best viewed in color.
for other ablated methods in Table average object precision, recall and consistency (more details in Table 6). Fl-score, accuracy and diversity scores (found Tables and 2 and finally, scene and object FID (in Tables[aand]] 4).
As shown in the table, the method exploiting the findings of analysis, namely ISLA+SG, achieves comparable or state-of-the-art results in the vast majority of image quality and diversity metrics, at the expense exhibiting slightly worse performance in terms of consistency: Fl-score and accuracy in both S, and S,a, and object consistency in S.,
I DETAILED INCONSISTENCIES BETWEEN SETUPS USED IN STUDIED METHODS
In this section we detail the main inconsistencies between the setups used to train and evaluate the studied methods in their original papers. The main inconsistencies are:
18

---

Under review as a conference paper at ICLR 2021
TSP
1SR
|SC
1OP
1OR
10C
1F1
1 Acc.
1DS
LSFID
JOFID
fonegrained conditioninss S.
Los1GANv2 (Snn & Wu. 2020
89.1
63.9
61.1
69.6
30.2
40.9
75.9
40.6
0,43
80.0
67.1
OC-GAN (Slvain et al.
2020
89.2
63.6
70.4
48.2
80.0
49.6
0.14
85.8
85.8
ISLA
89.7
65.2
63.6
73.1
28.4
45.9
76.7
43.1
0.41
78.5
66.1
ISLA + SG
90.5
64.9
66.1
70.5
32.2
45.3
78.9
42.1
0.41
78.4
62.6
unseen coarse conditionings S,a
LostGANy2 (Sun & Wu 2020
87.5
58.0
43.5
73.2
25.8
36.3
58.6
26.3
0,46
55.2
40.9
OC-GAN (Sylvam et al. 2020
86.9
45.8
70.9
42.1
63.8
36.7
0.13
60.1
59.4
ISLA
88.4
58.5
45.9
74.7
24.9
40.5
59.6
30.7
0.44
53.5
38.2
ISLA + SG
88.4
59.1
47.2
74.2
28.3
42.7
62.2
31.4
0.44
52.1
34.2
Table 9: Direct comparison with the best offte-of-thh-ar methods. We report average precision, recall and consistency for scenes (SP, SR, SC) and objects (OP, OR, OC), Fi-sore, Accuracy (Acc.), DS, scene and object FIDs (SFID and OFID). Results show the mean over 5 random seeds at test time. Note that ISLA is the same architecture as Los/GANv2, but trained with the experimental setup proposed in the ablation, that leads to generally better results than the ones reported in the original paper (Sun Wu 2019), Bold numbers represent the best average metric across methods.
Different training sets. Grid2im uses a training set of 25k images, while LostGANs uses 74777 images and OC-GAN authors). The discrepancy between LostGANv2 and OC-GAN comes from the way the data filtered. All images are filtered have between and 8 objects, but LostGAN only considers objects that are segmented with polygons and not uncompressed RLE. This filtering, aside from altering the number of images used for training, can also alter which images are used and which objects are considered valid.
74121 images (according to communications with the
Different validation sets. In Grid2im, they use only 2k images for validation, while Lost- GANs and OC-GAN use 3k images for reporting results.
Different reference distribution for FID. When computing FID, LostGANs generated
samples per conditioning, ending with 15k generated samples, against 3k real samples. In Grid2im and OC-GAN, they only generate sample per conditioning, ending up with 3k generated samples. Note that changing the number of samples to compute FID, especially when using less than 10k samples, can significantly change FID scores (Kynkäänniemi et al. 2019 Xu et al. 2018), hindering the possibility of drawing robust conclusions when comparing methods.
Different image formats. While Grid2im reported results computed generated images saved in PNG format, LostGANs and OC-GAN do so on JPEG saved images. While this might seem a harmless difference, can drastically affect the FID: computing FID on 2k validation coco-stuff images 1 128x using ground-truth PNG and JPEG images results in 20 points difference of FID score.
Training details. Even though the studied methods do use early stopping, they train for different number of epochs. In the original papers, Grid2im and Grid2im GT have been trained for 82 epochs (on much smaller dataset), while LostGANs have been trained for 125 epochs and OC-GAN for 170 epochs (on much larger dataset). This gives more opportunity OC-GAN to fit the data and potentially achieve better results. Note that in Section 4.5 training LostGANv2 and other ablation models for longer and using early stopping on FID improved results.
Additionally, each method uses different types of losses (one or many perceptual losses, using pixel- wise loss or not, type of adversarial loss, etc), and it is unclear whether any benefit comes from the loss choices, evaluation choices or model's contribution. Moreover, generator and discriminator choices affecting capacity and connectivity between modules change across papers.
19


Reviews:
[{"review_id": "x1FMGfK703G", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Problem: There has been a plethora of work on image synthesis from a given layout of objects or label maps. However, it is not clear what has led to those results because there are no fixed backbone, optimization, training data, and evaluation protocol in each of them. This paper introduces a methodology to study three approaches (G2im, LostGAN, OC-GAN) that input a layout of objects to synthesize a new image.\n\nWhat does the study include?: The study primarily assesses the ability of each model to (1) learn an effective mapping on the given data distribution; (2) generalize to unseen conditioning to seen object combinations; and (3) generalize to unseen conditioning to unseen object combinations. \n\nConclusions of the study: (1) The current approaches are able to nicely learn an effective mapping on the given data distribution and are able to generalize to unseen conditioning of seen object configurations. However, they struggle for unseen conditioning of unseen object configurations. (2) The authors further found three essential components that help in getting good performance. These are: (a) instance-wise spatial conditional normalization layer that increases the robustness of the model to unseen conditioning; (b)  scene-graph perceptual similarly helps improve scene generation; (c) improving the quality of label map leads to better results. \n\n\nSetup of the study: \n\n1. Data: There are three parts: (a) $D_{s}$: seen data on which the model is trained on; (b) $D_{u}$: unseen data in the validation set that has similar object combinations like $D_{s}$; and (c) $D_{u^2}$: unseen data in the validation set that does not have similar object combination like $D_{s}$. The authors use COCO-stuff dataset in the evaluation.\n\n2. Evaluation Protocol: Following metrics are used in the study: (a) Precision; (b) Recall; (c) Consistency; (d) FID; (e) LPIPS-based Diversity Score (DS); and (f) object classification accuracy or image-to-set prediction F1 score. \n\nOnce you understand the setup, the analysis in Section-4 can be quickly understood. All the approaches follow a similar trend.\n\n\nMajor concerns with the study: \n\n1. In my understanding (developed using the three-axis of study), one of the underlying goals is to contrast between memorization and generalization. *This study is incomplete without an extensive comparison with simple nearest neighbors*. The current model fairs well on $D_{s}$ that is a good sign about memorization. However,  $D_{u}$ is not too different from  $D_{s}$ in this case, especially when considering the COCO-stuff dataset. The examples shown in Figure-3 and Figure-4 can be very easily seen in the training set. \n\n2. It is even more alarming to see that the approaches under study are not able to generalize to $D_{u^2}$. This is the scenario when we only change the object combinations. \n\n3. Proper setup of nearest neighbors in both the input space and output space needs consideration. Without this, I find it hard to accept any claims about the different approaches.\n\n4. The evaluation and the related work should also include Tan et al., CVPR 2019 (Text2Scene: Generating Compositional Scenes from Textual Descriptions).\n\nMinor Point: \n\n5. Section-4.5 (ground-truth masks): Though no example is provided in this paper, I looked through the generated masks in the original paper and it seems to be another issue that may be orthogonal to this study. I am not really sure how this point fits well to this study.", "extra_scores": {}, "extra_text": {}}, {"review_id": "LyPVBdpBD2B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper discusses the achievements and limitations of existing conditional generative models. As expected, while existing methods can generalize reasonably well to unseen layouts of seen object combinations, none of the methods can work well to unseen object combinations.\n\nStrengths:\nThe paper provides extensive experiments to verify the performances of different methods under various conditions. The results should be valuable to audiences interested in future developments in the area.\n\nWeaknesses:\nThere’s only one dataset that is being evaluated. Given that this paper is trying to test performances under various situations, it would be better to evaluate on more datasets to verify the results are not biased.\n\nThe conclusion is rather not surprising. I feel the paper would be stronger if it could provide more insight to the problems, especially on how to improve the generalization problem of existing methods. That is the interesting part and would really become a big contribution to the society. In the current form, it feels more like this paper only points out a problem that is already expected.\n\nWhile the authors have identified some components that seem particularly useful in improving performances (Sec 4.5), I feel it is still not enough. In particular, most of the paragraphs are still simply spelling out the comparison results, without careful explanation of the insights. The authors should design more experiments to verify the hypotheses, explain the observed phenomena, and give details about the learned lessons. How these components are identified in the first place is worth mentioning too. Could there be other components that the authors are missing that could potentially be useful? Did the authors experiment with something else that turned out to be rather incremental? I think these are the questions that readers would want to know. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HPW5cMU7_YI", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary ####\nThis paper studies the problem of scene conditional image generation with a focus on the evaluation of existing works towards unseen complex scene generation on the COCO-Stuff dataset. Specifically, it evaluates the model performances from three aspects, namely, image generation from seen conditionings, unseen fine-grained conditionings, and unseen coarse conditionings. For each evaluation, it computes the precision, recall, conditional consistency, F1-score, object accuracy, FID and diversity score for both object-wise and scene-wise measures. \n\n#### Comments ####\nThis paper studies an important problem in scene conditional image generation. Reviewer appreciates the experimental efforts and reasonably thorough evaluations of existing methods. However, reviewer feels this is an okay submission but not good enough.\n\nW1: Most of the “findings” described in the evaluations are either known or somewhat expected. Reviewer fails to obtain in-depth understandings and insights through reading the paper. Based on the evaluations, the proposed ablations studies (see Table 1) also failed to advance this field further. Though the paper contains several tables with ablation studies or side-by-side comparisons, it is hard to learn more from just the numbers in the table. Reviewer would highly suggest to either provide in-depth analysis (e.g., at category level, or at bounding box level) if you choose to go this route (see my points in W2) or propose a novel method that generalizes much better than previous work.\n\nW2: The current analysis of scene conditional generation is very preliminary and limited. Reviewer would like to see the analysis per category or per bounding box (currently the performance evaluation is mixed). In addition, it would be good to visualize the learned feature maps of the existing work to see whether the model has achieved high-level understanding of the task or not, similar to what has been done before [Ref1-4]. This way, the quality and impact of the paper can be greatly improved, as readers will gain much better understanding from reading the paper than the current form.\n\n[Ref1] GAN Dissection: Visualizing and Understanding Generative Adversarial Networks, Bau et al. In ICLR 2019. \n\n[Ref2] Seeing What a GAN Cannot Generate, Bau et al. In ICCV 2019.\n\n[Ref3] Understanding Neural Networks Through Deep Visualization, Yosinski et al. In ICML 2015 Deep Learning Workshop.\n\n[Ref4] Visualizing and Understanding Convolutional Networks, Zeiler and Fergus. In ECCV 2014.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "26xhwWFEMq5", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: \n\nThe paper provides a set of comparisons among different scene generation methods. It assesses ability of the models to fit the training set (seen conditionings), generalize to unseen conditionings of seen object combinations, and generalize to unseen conditionings composed of unseen object combinations. It finds that these models fit the training distribution with a moderate success, display decent generalization to unseen fine-grained conditionings, and have significant space for improvement when it comes to generating images from unseen coarse\nconditionings. \n\n############################################################\n\nStrengths:\n\nThe authors provide a comprehensive set of experiments to compare performance of different scene generation methods using several metrics. This can be helpful for researchers to assess strengths and weaknesses of each model and its components, and helps them to gain insights into which aspect of models need to be improved.  \n\n###########################################################\n\nWeaknesses:\n\n1. While the comparisons among different scene generation methods are valuable, there are concerns about novelty of the paper especially since similar works have been published considering other computer vision tasks (e.g. [A, B]). The authors assess existing models in different settings and report their findings. \n2. There are some concerns regarding the overall setup for the experiments. Out of the three cases considered, the ability of a model to fit its training set (case 1) is not very interesting practically as we are more interested in the model’s generalization. Case 3, generalizing to unseen conditionings composed of unseen object combinations, is also not expected as the models are not particularly trained to be generalizable to unseen coarse conditionings. If one is seeking models with generalization ability to unseen coarse conditionings, he/she needs to incorporate a form of transfer/meta-learning and train the models differently. Generalization to novel categories is also an issue in object-level GANs.      \n3. The paper claims that it is very hard to assess which models perform better due to “models being trained to fit different data splits, using different conditioning modalities and levels of supervision, and reporting inconsistent quantitative metrics (e.g. repeatedly computing previous methods’ results using different reference distributions, and/or using different image compression algorithms to store generated images), among other uncontrolled sources of variation”. However, I do not see a clear evidence supporting this. Each of the other papers (LostGAN, OC-GAN, etc.) provides a set of comparisons with other methods. The authors need to note specifically which setups are inconsistent among different papers. They report that LostGAN-v2 outperforms other models in most tasks. This is consistent with results reported in the LostGAN-v2 paper (although they evaluate their method on a smaller number of metrics).  \n\n##############################################################\n\nReason for rating: \n\nWhile the paper provides a systematic evaluation of scene generation methods, there are some concerns regarding novelty and the proposed setup. I hope the authors clarify these in the rebuttal. \n\n##############################################################\n\nAdditional comments:\n\nThere are some minor grammatical errors in the paper, and it needs further proofreading. \n\n##############################################################\n\nReferences:\n\n[A] Are GANs Created Equal? A Large-Scale Study; Lucic et al.; NeurIPS 2018 \n\n[B] A metric learning reality check, Musgrave et al., ECCV 2020\n\n\n################################################################\n\nAfter author response: The authors have addressed my comment about inconsistent evaluation setups among different papers. However, I sill think novelty of the paper is limited as it is a conditional counterpart of [A]. As mentioned by other reviewers, findings of the paper are quite incremental and are in line with LostGAN-v2 although the authors use a more consistent evaluation setup. \nOverall, I keep my current rating. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received mixed reviews that overall lean negative. 

The main concern shared by reviewers is the novelty of the findings. Although the paper presents a systematic study that certainly has value, reviewers do not find sufficient insights from the analysis. The ACs agree with the reviewers that the paper is below the bar for acceptance. 
Decision: Reject