Generating unseen complex scenes: are we there yet?

1 INTRODUCTION . The recent years have witnessed significant advances in generative models ( Goodfellow et al. , 2014 ; Kingma & Welling , 2014 ; van den Oord et al. , 2016a ; Miyato & Koyama , 2018 ; Miyato et al. , 2018 ; Brock et al. , 2019 ) , enabling their increasingly widespread use in many application domains ( van den Oord et al. , 2016b ; Vondrick et al. , 2016 ; Zhang et al. , 2018a ; Hong et al. , 2018 ; Sun & Wu , 2020 ) . Among the most promising approaches , Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) have achieved remarkable results , generating high quality , high resolution samples in the context of single class conditional image generation ( Brock et al. , 2019 ) . This outstanding progress has paved the road towards tackling more challenging tasks such as the one of complex scene conditional generation , where the goal is to generate high quality images with multiple objects and their interactions from a given conditioning ( e.g . bounding box layout , segmentation mask , or scene graph ) . Given the exploding number of possible object combinations and their layouts , the requested conditionings oftentimes require zero-shot generalization . Therefore , successfully generating high quality , high resolution , diverse samples from complex scene datasets such as COCO-Stuff ( Caesar et al. , 2018 ) remains a stretch goal . Despite recent efforts producing increasingly appealing complex scene samples ( Hong et al. , 2018 ; Hinz et al. , 2019 ; Park et al. , 2019 ; Ashual & Wolf , 2019 ; Sun & Wu , 2020 ; Sylvain et al. , 2020 ) , and as previously noted in the unconditional GAN literature ( Lucic et al. , 2018 ; Kurach et al. , 2018 ) , it is unfortunately very hard to assess which models perform better , and perhaps more importantly why . In the case of conditional complex scene generation , this is often due to models being trained to fit different data splits , using different conditioning modalities and levels of supervision – bounding box layouts , segmentation masks , scene graphs – , and reporting inconsistent quantitative metrics ( e.g . repeatedly computing previous methods ’ results using different reference distributions , and/or using different image compression algorithms to store generated images ) , among other uncontrolled sources of variation . Moreover , these methods disregard the challenges that emerge from their expected generalization to unseen conditionings . This lack of rigour leads to conclusion replication failure , and hinders the identification of the most promising directions to advance the field . Therefore , in this paper we aim to provide an in-depth analysis and propose a methodology to compare current conditional complex scene generation methods . We argue that such analysis has the potential to deepen the understanding of such approaches and contribute to their progress . The proposed methodology addresses the following questions : ( 1 ) How well does each method perform on seen conditionings ( training conditionings ) ? ; ( 2 ) How well does each method generalize to unseen conditionings composed of seen object combinations ? ; and ( 3 ) How well does each method generalize to unseen conditionings composed of unseen object combinations ? We investigate the answers to the previous questions both from a scene-wise and an object-wise standpoints . As a result , we observe that : ( 1 ) recent methods are capable of generating identifiable scenes from seen conditionings ; ( 2 ) they exhibit some generalization capabilities when using unseen conditionings composed of seen object combinations , exploiting compositionality to generate scenes with different object arrangements ; ( 3 ) they suffer from poorer generated image quality when asked for unseen object combinations , especially the method presented in ( Sylvain et al. , 2020 ) . However , in all cases , the quality of the generated objects generally suffers from missing high frequency details , especially for those classes in the long tail of the dataset distribution . Finally , through an extensive ablation , we are able to identify the strengths and weaknesses of different pipeline components . In particular , we note that endowing the generator with an instance-wise normalization module ( Sun & Wu , 2019 ) results in increased individual object quality and better robustness to both types of unseen conditionings , whereas exploiting the normalization module of Sylvain et al . ( 2020 ) results in improved scene quality , suggesting that exploiting scene compositionality in the generator helps improve generalization . Moreover , we find that including a scene-graph similarity ( Sylvain et al. , 2020 ) while training complex scene conditional generation models leads to better conditional consistency , especially for unseen object combinations , emphasizing the promise of moving towards semantically aware training losses to improve generalization . We also identify the improvement of generated segmentation masks as one promising avenue to promote generalization to unseen conditionings . Finally , by leveraging these findings , we are able to compose a pipeline which obtains state-of-theart results in metrics such as FID , while maintaining competitive results in almost all other studied metrics . 2 RELATED WORK . Evaluation metrics for GANs . The most widely used metrics are the Inception Score ( IS ) ( Salimans et al . ( 2016 ) ) and the Fréchet Inception Distance ( FID ) ( Heusel et al . ( 2017 ) ) , which aim to capture visual sample quality and sample diversity . On the one hand , IS was designed to evaluate single-object image generation on problems where the expected marginal distribution over classes is uniform , an unrealistic expectation in many real-world datasets with multi-object images . On the other hand , FID was introduced for unconditional image generation and yields a single distributionspecific score , which hinders the analysis of individual failure cases . To overcome the former , researchers have attempted to extend FID to the conditional generation case ( DeVries et al. , 2019 ; Benny et al. , 2020 ) . To overcome the latter , researchers have proposed to disentangle visual sample quality and diversity into two different metrics ( Shmelkov et al. , 2018 ; Sajjadi et al. , 2018 ; Kynkäänniemi et al. , 2019 ; Ravuri & Vinyals , 2019 ) by modifying the definition of precision-recall in different ways . Finally , the diversity of generated samples has also been quantified by measuring the perceptual similarity between generated image patches , as proposed by Zhang et al . ( 2018b ) . Complex scene conditional generation . The conditional generation literature encompasses a variety of input conditioning modalities . Several existing works focus on generating photo-realistic scenes from detailed semantic segmentation masks ( Chen & Koltun , 2017 ; Qi et al. , 2018 ; Park et al. , 2019 ; Wang et al. , 2018 ; Tang et al. , 2020 ) . Although semantic segmentation masks are information rich , they may be difficult to obtain with enough fine-grained details . This is especially relevant in applications where a user is expected to specify the input conditioning . As such , other commonly used input conditioning modalities include text descriptions ( Reed et al. , 2016 ; Hong et al. , 2018 ; Tan et al. , 2019 ) , scene graphs ( Johnson et al. , 2018 ; Ashual & Wolf , 2019 ) , and bounding box layouts specifying the position and scale of the objects in the scene ( Zhao et al. , 2019 ; Sun & Wu , 2019 ; 2020 ; Sylvain et al. , 2020 ) . In addition to those , recent works have focused on generating bounding box layouts from a set of classes ( Jyothi et al. , 2019 ) and designing remarkably flexible pipelines to enable user interactions ( Ashual & Wolf , 2019 ) . From a model perspective , complex scene conditional generation has been improved through the development of tailored class-aware ( Park et al. , 2019 ) , instance-aware ( Sun & Wu , 2019 ; 2020 ) , and class-and-instance-aware normalizations ( Sylvain et al. , 2020 ) incorporated in the GAN generator . The task has also benefited from the introduction of refinement pipelines ( Sun & Wu , 2020 ) , as well as the design of a multi-modal perceptual loss to drive scene graph embeddings and generated image embeddings close to each other ( Sylvain et al. , 2020 ) . All aforementioned models are often trained on a different number of training samples , use different backbone architectures , and report metrics on different splits and/or using different pre-trained models . This results in repeated reevaluations of baselines to fit each methods ’ setup , making it hard to pinpoint the most promising research directions . 3 METHODOLOGY . We start by defining three sets of data , composed of triplets of images and their respective finegrained and coarse conditionings . The finegrained conditionings represent detailed scene layouts by means of scene graphs , bounding boxes or segmentation masks . The coarse conditionings are an abstraction of the finegrained ones into a set of object labels present in the scene . For example , given a scene layout depicting two glasses on a table , the corresponding set of labels is { glass , table } . Let Ds be a set containing triplets of real images and their corresponding conditionings . We use Ds to train the complex scene conditional generation models , and hereinafter refer to the conditionings used to train the models as seen conditionings . LetDu be a set containing triplets of real images and their corresponding conditionings , composed of seen coarse conditionings but unseen finegrained conditionings . Note that we use the term unseen to refer to conditionings that are not present in Ds . Similarly , let Du2 be a set containing triplets of real images and their corresponding conditionings , composed of unseen coarse conditionings and unseen finegrained conditionings . We use Du and Du2 only for evaluation purposes . After training the models , we use the finegrained conditionings in Ds , Du and Du2 to generate three sets of model samples : Ss , Su and Su2 . The goal of our analysis is three-fold : ( 1 ) to assess how well each model fits the training data , ( 2 ) to assess each models ’ generalization to unseen finegrained conditionings , and ( 3 ) to assess each models ’ generalization to unseen coarse conditionings . To do so , we measure discrepancies between the distributions defined by the images in Ds , Du and Du2 and by the images in Ss , Su and Su2 , respectively . In particular , we evaluate the quality of the generation processes from both scene-wise and object-wise standpoints . Note that the scene-wise evaluation considers full real images and full generated images , whereas the object-wise evaluation considers crops of individual objects from real and generated images . Throughout our analysis , we consider the following metrics to assess scene-wise and object-wise generation processes : precision , recall , consistency , FID , LPIPS-based diversity score ( DS ) , and object classification accuracy or image ( scene ) -to-set prediction F1-score . In the remainder of this section , we will use image to refer to either full images or cropped objects . We follow the definition of precision and recall introduced by Kynkäänniemi et al . ( 2019 ) . Intuitively , precision captures the sample quality , whereas recall captures the coverage of the generated distribution . In particular , precision verifies whether generated images lie within a reference manifold of real images , and recall verifies whether real images lie within a reference manifold of generated images . The real and generated manifolds are estimated in the feature space , which in our case is defined by a ResNext101 ( Xie et al. , 2017 ) , and are obtained surrounding each data point with a hyper-sphere that reaches its k-th ( k = 5 ) nearest neighbor . Therefore , precision and recall are sensitive to the number of samples used to estimate the reference manifolds , resulting in overestimated manifold volumes when the sample size is small . Hence , when |Du| and |Du2 | are small , we estimate the manifold ’ s hyper-sphere radii using the images inDs∪Du∪D2u . Analogously , we estimate the manifold of generated images from Ss ∪ Su ∪ S2u . Although successful conditional generation models should respect their conditionings and generate high quality samples which are consistent with those , the adopted definition of precision does not guarantee that the reference real sample and the generated sample found in its hyper-sphere will have the same coarse conditionings . Therefore , we complement precision with a metric to measure the consistency of the generated samples that fall within the estimated real manifold . More precisely , we compute the intersection over union ( IoU ) between the classes present in the reference sample and those present in the conditioning of the generated sample . The consistency is averaged over all generated images , assigning consistency 0 to samples lying outside the estimated manifold .

Summary: The paper provides a set of comparisons among different scene generation methods. It assesses ability of the models to fit the training set (seen conditionings), generalize to unseen conditionings of seen object combinations, and generalize to unseen conditionings composed of unseen object combinations. It finds that these models fit the training distribution with a moderate success, display decent generalization to unseen fine-grained conditionings, and have significant space for improvement when it comes to generating images from unseen coarse