Generating unseen complex scenes: are we there yet?

1 INTRODUCTION . The recent years have witnessed significant advances in generative models ( Goodfellow et al. , 2014 ; Kingma & Welling , 2014 ; van den Oord et al. , 2016a ; Miyato & Koyama , 2018 ; Miyato et al. , 2018 ; Brock et al. , 2019 ) , enabling their increasingly widespread use in many application domains ( van den Oord et al. , 2016b ; Vondrick et al. , 2016 ; Zhang et al. , 2018a ; Hong et al. , 2018 ; Sun & Wu , 2020 ) . Among the most promising approaches , Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) have achieved remarkable results , generating high quality , high resolution samples in the context of single class conditional image generation ( Brock et al. , 2019 ) . This outstanding progress has paved the road towards tackling more challenging tasks such as the one of complex scene conditional generation , where the goal is to generate high quality images with multiple objects and their interactions from a given conditioning ( e.g . bounding box layout , segmentation mask , or scene graph ) . Given the exploding number of possible object combinations and their layouts , the requested conditionings oftentimes require zero-shot generalization . Therefore , successfully generating high quality , high resolution , diverse samples from complex scene datasets such as COCO-Stuff ( Caesar et al. , 2018 ) remains a stretch goal . Despite recent efforts producing increasingly appealing complex scene samples ( Hong et al. , 2018 ; Hinz et al. , 2019 ; Park et al. , 2019 ; Ashual & Wolf , 2019 ; Sun & Wu , 2020 ; Sylvain et al. , 2020 ) , and as previously noted in the unconditional GAN literature ( Lucic et al. , 2018 ; Kurach et al. , 2018 ) , it is unfortunately very hard to assess which models perform better , and perhaps more importantly why . In the case of conditional complex scene generation , this is often due to models being trained to fit different data splits , using different conditioning modalities and levels of supervision – bounding box layouts , segmentation masks , scene graphs – , and reporting inconsistent quantitative metrics ( e.g . repeatedly computing previous methods ’ results using different reference distributions , and/or using different image compression algorithms to store generated images ) , among other uncontrolled sources of variation . Moreover , these methods disregard the challenges that emerge from their expected generalization to unseen conditionings . This lack of rigour leads to conclusion replication failure , and hinders the identification of the most promising directions to advance the field . Therefore , in this paper we aim to provide an in-depth analysis and propose a methodology to compare current conditional complex scene generation methods . We argue that such analysis has the potential to deepen the understanding of such approaches and contribute to their progress . The proposed methodology addresses the following questions : ( 1 ) How well does each method perform on seen conditionings ( training conditionings ) ? ; ( 2 ) How well does each method generalize to unseen conditionings composed of seen object combinations ? ; and ( 3 ) How well does each method generalize to unseen conditionings composed of unseen object combinations ? We investigate the answers to the previous questions both from a scene-wise and an object-wise standpoints . As a result , we observe that : ( 1 ) recent methods are capable of generating identifiable scenes from seen conditionings ; ( 2 ) they exhibit some generalization capabilities when using unseen conditionings composed of seen object combinations , exploiting compositionality to generate scenes with different object arrangements ; ( 3 ) they suffer from poorer generated image quality when asked for unseen object combinations , especially the method presented in ( Sylvain et al. , 2020 ) . However , in all cases , the quality of the generated objects generally suffers from missing high frequency details , especially for those classes in the long tail of the dataset distribution . Finally , through an extensive ablation , we are able to identify the strengths and weaknesses of different pipeline components . In particular , we note that endowing the generator with an instance-wise normalization module ( Sun & Wu , 2019 ) results in increased individual object quality and better robustness to both types of unseen conditionings , whereas exploiting the normalization module of Sylvain et al . ( 2020 ) results in improved scene quality , suggesting that exploiting scene compositionality in the generator helps improve generalization . Moreover , we find that including a scene-graph similarity ( Sylvain et al. , 2020 ) while training complex scene conditional generation models leads to better conditional consistency , especially for unseen object combinations , emphasizing the promise of moving towards semantically aware training losses to improve generalization . We also identify the improvement of generated segmentation masks as one promising avenue to promote generalization to unseen conditionings . Finally , by leveraging these findings , we are able to compose a pipeline which obtains state-of-theart results in metrics such as FID , while maintaining competitive results in almost all other studied metrics . 2 RELATED WORK . Evaluation metrics for GANs . The most widely used metrics are the Inception Score ( IS ) ( Salimans et al . ( 2016 ) ) and the Fréchet Inception Distance ( FID ) ( Heusel et al . ( 2017 ) ) , which aim to capture visual sample quality and sample diversity . On the one hand , IS was designed to evaluate single-object image generation on problems where the expected marginal distribution over classes is uniform , an unrealistic expectation in many real-world datasets with multi-object images . On the other hand , FID was introduced for unconditional image generation and yields a single distributionspecific score , which hinders the analysis of individual failure cases . To overcome the former , researchers have attempted to extend FID to the conditional generation case ( DeVries et al. , 2019 ; Benny et al. , 2020 ) . To overcome the latter , researchers have proposed to disentangle visual sample quality and diversity into two different metrics ( Shmelkov et al. , 2018 ; Sajjadi et al. , 2018 ; Kynkäänniemi et al. , 2019 ; Ravuri & Vinyals , 2019 ) by modifying the definition of precision-recall in different ways . Finally , the diversity of generated samples has also been quantified by measuring the perceptual similarity between generated image patches , as proposed by Zhang et al . ( 2018b ) . Complex scene conditional generation . The conditional generation literature encompasses a variety of input conditioning modalities . Several existing works focus on generating photo-realistic scenes from detailed semantic segmentation masks ( Chen & Koltun , 2017 ; Qi et al. , 2018 ; Park et al. , 2019 ; Wang et al. , 2018 ; Tang et al. , 2020 ) . Although semantic segmentation masks are information rich , they may be difficult to obtain with enough fine-grained details . This is especially relevant in applications where a user is expected to specify the input conditioning . As such , other commonly used input conditioning modalities include text descriptions ( Reed et al. , 2016 ; Hong et al. , 2018 ; Tan et al. , 2019 ) , scene graphs ( Johnson et al. , 2018 ; Ashual & Wolf , 2019 ) , and bounding box layouts specifying the position and scale of the objects in the scene ( Zhao et al. , 2019 ; Sun & Wu , 2019 ; 2020 ; Sylvain et al. , 2020 ) . In addition to those , recent works have focused on generating bounding box layouts from a set of classes ( Jyothi et al. , 2019 ) and designing remarkably flexible pipelines to enable user interactions ( Ashual & Wolf , 2019 ) . From a model perspective , complex scene conditional generation has been improved through the development of tailored class-aware ( Park et al. , 2019 ) , instance-aware ( Sun & Wu , 2019 ; 2020 ) , and class-and-instance-aware normalizations ( Sylvain et al. , 2020 ) incorporated in the GAN generator . The task has also benefited from the introduction of refinement pipelines ( Sun & Wu , 2020 ) , as well as the design of a multi-modal perceptual loss to drive scene graph embeddings and generated image embeddings close to each other ( Sylvain et al. , 2020 ) . All aforementioned models are often trained on a different number of training samples , use different backbone architectures , and report metrics on different splits and/or using different pre-trained models . This results in repeated reevaluations of baselines to fit each methods ’ setup , making it hard to pinpoint the most promising research directions . 3 METHODOLOGY . We start by defining three sets of data , composed of triplets of images and their respective finegrained and coarse conditionings . The finegrained conditionings represent detailed scene layouts by means of scene graphs , bounding boxes or segmentation masks . The coarse conditionings are an abstraction of the finegrained ones into a set of object labels present in the scene . For example , given a scene layout depicting two glasses on a table , the corresponding set of labels is { glass , table } . Let Ds be a set containing triplets of real images and their corresponding conditionings . We use Ds to train the complex scene conditional generation models , and hereinafter refer to the conditionings used to train the models as seen conditionings . LetDu be a set containing triplets of real images and their corresponding conditionings , composed of seen coarse conditionings but unseen finegrained conditionings . Note that we use the term unseen to refer to conditionings that are not present in Ds . Similarly , let Du2 be a set containing triplets of real images and their corresponding conditionings , composed of unseen coarse conditionings and unseen finegrained conditionings . We use Du and Du2 only for evaluation purposes . After training the models , we use the finegrained conditionings in Ds , Du and Du2 to generate three sets of model samples : Ss , Su and Su2 . The goal of our analysis is three-fold : ( 1 ) to assess how well each model fits the training data , ( 2 ) to assess each models ’ generalization to unseen finegrained conditionings , and ( 3 ) to assess each models ’ generalization to unseen coarse conditionings . To do so , we measure discrepancies between the distributions defined by the images in Ds , Du and Du2 and by the images in Ss , Su and Su2 , respectively . In particular , we evaluate the quality of the generation processes from both scene-wise and object-wise standpoints . Note that the scene-wise evaluation considers full real images and full generated images , whereas the object-wise evaluation considers crops of individual objects from real and generated images . Throughout our analysis , we consider the following metrics to assess scene-wise and object-wise generation processes : precision , recall , consistency , FID , LPIPS-based diversity score ( DS ) , and object classification accuracy or image ( scene ) -to-set prediction F1-score . In the remainder of this section , we will use image to refer to either full images or cropped objects . We follow the definition of precision and recall introduced by Kynkäänniemi et al . ( 2019 ) . Intuitively , precision captures the sample quality , whereas recall captures the coverage of the generated distribution . In particular , precision verifies whether generated images lie within a reference manifold of real images , and recall verifies whether real images lie within a reference manifold of generated images . The real and generated manifolds are estimated in the feature space , which in our case is defined by a ResNext101 ( Xie et al. , 2017 ) , and are obtained surrounding each data point with a hyper-sphere that reaches its k-th ( k = 5 ) nearest neighbor . Therefore , precision and recall are sensitive to the number of samples used to estimate the reference manifolds , resulting in overestimated manifold volumes when the sample size is small . Hence , when |Du| and |Du2 | are small , we estimate the manifold ’ s hyper-sphere radii using the images inDs∪Du∪D2u . Analogously , we estimate the manifold of generated images from Ss ∪ Su ∪ S2u . Although successful conditional generation models should respect their conditionings and generate high quality samples which are consistent with those , the adopted definition of precision does not guarantee that the reference real sample and the generated sample found in its hyper-sphere will have the same coarse conditionings . Therefore , we complement precision with a metric to measure the consistency of the generated samples that fall within the estimated real manifold . More precisely , we compute the intersection over union ( IoU ) between the classes present in the reference sample and those present in the conditioning of the generated sample . The consistency is averaged over all generated images , assigning consistency 0 to samples lying outside the estimated manifold .

Summary: This paper studies the problem of scene conditional image generation with a focus on the evaluation of existing works towards unseen complex scene generation on the COCO-Stuff dataset. Specifically, it evaluates the model performances from three aspects, namely, image generation from seen conditionings, unseen fine-grained conditionings, and unseen coarse conditionings. For each evaluation, it computes the precision, recall, conditional consistency, F1-score, object accuracy, FID and diversity score for both object-wise and scene-wise measures.