Sim2SG: Sim-to-Real Scene Graph Generation for Transfer Learning

1 INTRODUCTION . Scene Graphs ( SGs ) in both computer vision and computer graphics are an interpretable and structural representation of scenes . A scene graph summarizes entities in the scene and plausible relationships among them . SGs ( Dai et al. , 2017 ; Herzig et al. , 2018 ; Li et al. , 2017 ; Newell & Deng , 2017 ; Xu et al. , 2017 ; Yang et al. , 2018 ; Zellers et al. , 2018 ) are a manifestation of vision as inverse graphics . They have found a variety of applications such as image captioning , visual question answering , high level reasoning tasks , image retrieval , image generation , etc . However , most prior work on SG generation relies on the availability of expensive and limited number of labeled datasets such as Visual Genome ( Krishna et al. , 2017 ) and Visual Relationship Dataset ( VRD ) ( Lu et al. , 2016 ) . One of the main limitations in machine learning applications is the general lack of sufficient labeled data for supervised learning tasks . Synthetic data is a viable alternative to this problem since annotations are essentially free . Synthetic data has been used for a variety of tasks such as image classification , object detection , semantic segmentation , optical flow modeling , 3D keypoint extraction , object pose estimation , 3D reconstruction , etc . ( Borrego et al. , 2018 ; Butler et al. , 2012 ; Dosovitskiy et al. , 2015 ; McCormac et al. , 2016 ; Mueller et al. , 2017 ; Richter et al. , 2016 ; Ros et al. , 2016 ; Suwajanakorn et al. , 2018 ; Tremblay et al. , 2018 ; Tsirikoglou et al. , 2017 ) . It has also been shown to be effective in initializing task networks ( Prakash et al. , 2019 ) and for data augmentation . However , the use of synthetic data for SG generation and visual relationships is yet to be explored . One crucial issue with training on a labeled source domain ( synthetic data ) and evaluating on an unlabeled target domain ( real data ) is the performance gap known as domain gap ( Torralba & Efros , 2011 ) . This gap is due to the difference of data distribution between the source and target domains . Kar et al . ( 2019 ) argue that domain gap can be divided into appearance and content gap . The appearance gap can be addressed by making scenes photo-realistic ( McCormac et al. , 2016 ; Wrenninge & Unger , 2018 ) , by using image translations ( Hoffman et al. , 2018 ; Huang et al. , 2018 ; Zhu et al. , 2017 ) , by feature alignment ( Chang et al. , 2019 ; Chen et al. , 2018 ; Li et al. , 2019 ; Luo et al. , 2019 ; Saito et al. , 2019 ; Sun et al. , 2019 ) , or by learning robust representations based on domain randomization ( Prakash et al. , 2019 ; Tobin et al. , 2017 ) . There are also studies that address the content gap for image classification ( Azizzadenesheli et al. , 2019 ; Lipton et al. , 2018 ; Tan et al. , 2019 ) . We present a thorough investigation of the domain gap between source and target domains . We assume a gap in both appearance and content , expand those gaps into different sub-components and provide a way to address them . We primarily apply our method to reduce the domain gap for SG generation . Nonetheless , our techniques can also be applied to other vision tasks such as image classification , image segmentation and object detection among others . We propose Sim2SG ( Simulation to Scene Graph ) ; a model that learns sim-to-real scene graph generation leveraging labeled synthetic data and unlabeled real data . Extending the formulation in ( Wu et al. , 2019 ) , Sim2SG addresses the domain gap by bounding the task error ( where the task is scene graph generation ) on real data through appearance , prediction , label ( ground truth ) discrepancies between the two domains and task error on synthetic data . Our work differs from ( Wu et al. , 2019 ) as they do not provide a way to address the content gap , and their risk discrepancy is intractable . To the best of our knowledge , Sim2SG is the first work to introduce a tractable error bound on the content component of the domain gap . We minimize the appearance and prediction discrepancies by aligning the corresponding latent and output distributions via Gradient Reversal Layers ( Ganin et al. , 2017 ) . We address discrepancy in the label using principles of self-learning ( Zou et al. , 2018 ) . However , self-learning based on pseudo labels often suffer from the inaccurately generated labels ( e.g . predicted bounding boxes are ill-categorized or imprecise , hence , the model will regress on the wrong objects ) ( Zheng & Yang , 2020 ; Kim et al. , 2019 ) . Therefore , we instead propose to collect a higher level statistic ( e.g . list of objects and their type , position and relationships for placement ) , that we call pseudo-statistics , from target data and leverage the synthetic data generator to produce valid objects with their precise labels ( e.g . bounding boxes ) . We experimentally demonstrate our method in three distinct environments– all synthetic CLEVR ( Johnson et al. , 2017 ) , more realistic Dining-Sim and Drive-Sim with a driving simulator evaluated on KITTI ( Geiger et al. , 2012 ) . We almost close the domain gap in the Clevr environment and we show significant improvements over respective baselines in Dining-Sim and Drive-Sim . Through ablations , we validate our assumptions about appearance and content gaps . Sim2SG differs from other unsupervised domain adaptation methods ( Chen et al. , 2018 ; Xu et al. , 2020 ; Li et al. , 2020 ) as it can modify the source distribution ( via self-learning based on pseudostatistics to align with the target distribution ) with access to a synthetic data generator . We also outperform these domain adaptation baselines ( Chen et al. , 2018 ; Xu et al. , 2020 ; Li et al. , 2020 ) as shown in Section 4.3 . Contributions : Our contributions are three-fold : In terms of methodology , to the best of our knowledge , ( 1 ) We are the first to propose sim-to-real transfer learning for scene graph generation . We do not require costly supervision from the target real-world dataset . ( 2 ) We study domain gap from synthetic to real data in detail , provide a tractable error bound on the content component of the gap and propose a novel pipeline including pseudo statistics to fully handle the gap . Experimentally , ( 3 ) we show that Sim2SG can learn SG generation and obtains significant improvements over baselines in all three scenarios - Clevr , Dining-Sim and Drive-Sim . We also present ablations to illustrate the effectiveness of our technique . 2 PROPOSED METHOD : SIM2SG . Our proposed Sim2SG pipeline is illustrated in Figure 1 . We first describe how we generate scene graphs in Section 2.1 . When we naïvely train on a source distribution ( synthetic data ) and evaluate on a target distribution ( real data ) , we have a domain gap ( Torralba & Efros , 2011 ) . We study it in more detail in Section 2.2 and propose methods to address it . 2.1 SCENE GRAPHS . This section describes scene graphs ( SGs ) and how we train the SG predictor module using labels from the source domain . Notation : We represent a scene graph of a given image I as a graph G with nodes o and edges r. Each node is a tuple oi = 〈bi , ci〉 of bounding box bi = { xmini , ymini , wi , hi } and category ci . Relationships r are a triplet of 〈oi , p , oj〉 where p is a predicate . SG prediction has two key components : feature extractor φ and graph predictor h. φ maps input space x to a latent space z and h maps from latent space z to output space y . The predicted SG is G = h ( φ ( x ) ) . We use Resnet 101 ( He et al. , 2016 ) to implement φ and GraphRCNN ( Yang et al. , 2018 ) architecture to implement h. We train the networks φ and h using the following task loss ( Yang et al. , 2018 ) : cross entropy loss for object classification & relationship classification and ` 1 loss for bounding boxes . Our framework Sim2SG is illustrated in Figure 1 . It is worth noting that we predict the relationships among objects in the scene , but not their attributes like ( Yang et al. , 2018 ) . 2.2 DOMAIN GAP . We now study the domain gap between source and target domains and formulate the SG generation task error on real domain as a function of the task error on synthetic domain and appearance & content gap between the domains . We then propose methods to address each subcomponent of the gap . Notation and assumptions : We have two domains : synthetic 〈xs , ys〉 ∼ p ( x , y ) and real domain 〈xr , yr〉 ∼ q ( x , y ) . Note that xs or xr is the input ( image ) and ys or yr is the output ( SG ) as introduced in Section 2.1 It is also worth noting that yr is not known . We assume that synthetic and real domains contain the same categories of objects . Both domains also share similar scenarios ( e.g . both have driving scenes ) . However , joint distributions of scenes ( images ) are different in both domains ( i.e . p ( x ) 6= q ( x ) ) . The label ( ground truth ) distributions are also different in the two domains ( i.e . p ( y ) 6= q ( y ) ) . Using the formulation in ( Wu et al. , 2019 ) , the task ( SG generation ) error ( i.e . risk ) on synthetic domain as a function of the latent space z is given by : s ( φ , h ) = ∫ p ( z ) esdz ( 1 ) where es is the risk defined as es = |p ( y|z ) − ys| . p ( z ) is the distribution of features , p ( y ) is the distribution of labels and p ( y|z ) is the output distribution . The bound on task error in real domain r ( φ , h ) from Wu et al . ( 2019 ) is a function of three terms which are : ( 1 ) task error on the synthetic domain s ( φ , h ) , ( 2 ) risk discrepancy between the domains c ( φ , h ) and ( 3 ) feature discrepancy between the two domains a ( φ , h ) . Our goal is to minimize the task error on the real domain : r ( φ , h ) = ∫ q ( z ) erdz = ∫ q ( z ) erdz + ∫ p ( z ) esdz − ∫ p ( z ) esdz + ∫ q ( z ) esdz − ∫ q ( z ) esdz = ∫ p ( z ) esdz + ∫ q ( z ) ( er − es ) dz ∫ ( q ( z ) − p ( z ) ) esdz = s ( φ , h ) + c ( φ , h ) + a ( φ , h ) ( 2 ) We drop the terms φ , h from now on . We would need to minimize s , a and c to reduce the task error on the real domain r ≥ 0 . If the error r reduces to zero on the target domain , we have closed the domain gap . Since we have access to the label ys , we can minimize the task error on synthetic domain s as described in Section 2.1 . However , we do not have access to the label distribution q ( y ) for the real domain . This makes the risk discrepancy c intractable . We split this discrepancy into tractable components and address them in Section 2.2.1 . We call the discrepancy content gap based on our Algorithm 1 Pseudocode for Sim2SG training 1 : Given : φθ , hθ . Encoder , Scene Graph predictor 2 : Given : Xs , Ys , R , Xr . Synthetic Images , Synthetic Labels , Data Generator , Real Images 3 : Hyperparameters : E , I . Epochs , Iterations 4 : for e ∈ E do . Training 5 : loss = 0 ; 6 : for i ∈ Im do 7 : loss += a ( φ ( xs ) , φ ( xr ) ) . Appearance gap 8 : loss += c , pred ( h ( φ ( xs ) ) , h ( φ ( xr ) ) ) . Content prediction gap 9 : loss += task ( xs , ys ) . Scene Graph generation task loss 10 : φθ , hθ = optimize ( φθ , hθ , loss ) ; . Network parameters θ ; SGD step 11 : end for 12 : ps = { h ( φ ( xr ) ) : xr ∈ Xr } . extracting pseudo statistics from predictions 13 : Xs , Ys = R ( ps ) . generating synthetic data with labels aligned to real data 14 : end for ablations in Section 4.1 . Recent work shows that features frequently account for appearance ( Geirhos et al. , 2019 ) , and therefore we call the feature discrepancy a as appearance gap and empirically show through ablations in Section 4.1 to be the case . We address this discrepancy in Section 2.2.2 . The pseudo-code of our proposed method is illustrated in Algorithm 1 . 2.2.1 CONTENT GAP : c Content gap refers to discrepancies between the two domains including the difference in distribution of the number of objects and their class , placement , pose and scale . This affects the position , dimension and the type of the labels leading to the problem of label shift ( i.e . discrepancy in the ground truth distribution ) . However , minimizing c in the current form is not tractable . Since the task error ( risk ) r is non-negative and both r and er are lower bounded by zero , we also assume a lower bound of zero for c. Hence , we approximate the ( er - es ) as ( ( q ( y|z ) − p ( y|z ) ) + ( ys − yr ) ) . We show this in equation 3 . We assume ys ≡ yr and q ( y|z ) ≡ p ( y|z ) as a sufficient condition for ( ( q ( y|z ) − p ( y|z ) ) + ( ys − yr ) ) to be zero and to avoid any degenerate solutions . Based on these assumptions , we also empirically demonstrate in Section 4 that the label discrepancy c , label and the prediction discrepancy c , pred help reduce the domain gap . argmin θ c = argmin θ ∫ q ( z ) ( er − es ) dz ' argmin θ ∫ q ( z ) ( ys − yr ) dz + ∫ q ( z ) ( q ( y|z ) − p ( y|z ) ) dz = argmin θ c , label + c , pred ( 3 ) Label discrepancy and Pseudo-Statistics : Minimizing c , label is challenging because we do not have access to the label ( yr ) of the target ( real ) domain . We then propose to get an estimate of yr through principles of self-learning ( Zou et al. , 2018 ) based on minimum reliable statistic ( pseudostatistic ) of target data and generate aligned synthetic data ( ys ) to narrow the gap . We generate SGs for all input images ( xr ) of the target domain and derive the pseudo statistic ps from each SG by retaining the minimum information needed for 3D representation ( e.g . we discard the entire bounding box and keep the centroid position ) . The statistic ps is a list of objects with each object ’ s type , centroid location and relationship with others . Using either known or assumed camera intrinsic ( e.g . car dash cam ) , we map ps to a full 3D scene . Some unknown parameters ( e.g . texture or pose ) and context ( ground , sky , light ) are randomized as done in ( Prakash et al. , 2019 ) . We use a synthetic data generator to render those scenes as shown in Figure 11 . More details on scene generation for different environments are in Sections A.2.1 , A.2.2 and A.2.3 of Appendix . This is also analogous to an Expectation Maximization algorithm where we compute pseudo-statistics and generate aligned synthetic data ( E-step ) and then use it to train the Sim2SG model ( M-step ) . Contrarily , self-learning based on pseudo labels ( Zou et al. , 2018 ) generated from and trained on real data often suffer from poorly generated labels ( Zheng & Yang , 2020 ; Kim et al. , 2019 ) . For instance , training SG generation model on ill-categorized or imprecise bounding boxes can lead to poor detection results . Pseudo-Statistics ( described previously ) integrated with a synthetic data generator ( renderer ) will still produce a valid scene for those ill-detected objects ( false positives ) with precise labels ( bounding boxes ) . We show through quantitative experiments that our method performs better than pseudo label based self-learning in Section 4.3 . Please note that our method is orthogonal to pseudo labels based self-learning and the latter can potentially be applied in conjunction . Prediction discrepancy : The output of the scene graph generation model should be the same for same categories in different domains . To address the prediction discrepancy c , pred , we propose to align output distribution for latent code belonging to the same visual category . To this end , we align the output distributions p ( y|z ) and q ( y|z ) using a GRL based technique ( Ganin et al. , 2017 ) that we will discuss in Section 2.2.2 . 2.2.2 APPEARANCE GAP : a Appearance gap is the discrepancy in the appearance of the two domains . This includes differences in texture , color , light , reflectance , etc . of objects in the scene . To address the appearance gap , we want to avoid photo-realism in synthetic data as it requires high quality assets and a tremendous amount of effort from artists ( McCormac et al. , 2016 ; Wrenninge & Unger , 2018 ) . Therefore , we propose to learn an appearance invariant representation z so we can avoid the appearance bias in the model . However , the features z : φ ( x ) have both content and appearance components and aligning them may be detrimental ( Saito et al. , 2019 ; Wu et al. , 2019 ) as shown in the ablation in Section 4.3 . To overcome this challenge , we align the appearance gap a only after we align the content gap c of the two domains using the method described in Section 2.2.1 . We minimize this appearance gap a by aligning the feature distributions p ( z ) and q ( z ) as follows : θ∗ = argmin θ ∫ ( q ( z ) − p ( z ) ) esdz ( 4 ) where we exploit the fact that p ( z ) ≡ q ( z ) is a sufficient condition for a to be zero ( Wu et al. , 2019 ) . We use Gradient Reversal Layer ( GRL ) ( Ganin et al. , 2017 ) to align the distributions p ( z ) and q ( z ) along with a domain classifier D to classify them . We minimize the D ’ s loss w.r.t . its own parameters while maximizing w.r.t . the network parameters of φ ( θ ) . We do so through GRL that acts as an identity function during forward propagation and flips the sign of the gradients during back propagation from D to φ . We provide details of GRL layers in Section A.1

Summary: The paper addresses the problem of learning scene graphs from synthetic data and unlabeled real data while performing well on real data by narrowing the content and appearance gap between the two domains when training on synthetic data. Scene graphs are extracted in a two-step process, mapping input to an intermediate latent space and generating the final prediction from the latent space. The authors decompose the content gap into two components: (a) label discrepancy, i.e. how much do the label distributions between the two domains differ, and (b) prediction discrepancy, i.e. the difference in distributions of outputs predicted from the latent space for the two domains. They further model the appearance gap by aligning the latent representation for both domains after accounting for the content gap (to avoid spurious influence of differing content distributions as the latent space is expected to comprise content and appearance). Most of these components are intractable and the paper provides approximations. Empirical investigation on two entirely synthetic and one real/synthetic data set provide evidence for the benefit of the method in closing the posed domain gaps as well as the quality of chosen approximations, the influence of the individual content and appearance gap terms, and the effectiveness of the optimization procedure.