Sim2SG: Sim-to-Real Scene Graph Generation for Transfer Learning
Authors: Aayush Prakash
Shoubhik Debnath
Jean Francois Lafleche
Eric Cameracci
Gavriel State
Marc T Law
Venue:  (2021)
Keywords: computer vision
application
sim2real
scene graph
object detection
simulation in machine learning
transfer learning
synthetic data
driving simulation.

Abstract: Scene graph (SG) generation has been gaining a lot of traction recently. Current SG generation techniques, however, rely on the availability of expensive and limited number of labeled datasets. Synthetic data offers a viable alternative as labels are essentially free. However, neural network models trained on synthetic data, do not perform well on real data because of the domain gap.
To overcome this challenge, we propose Sim2SG, a scalable technique for sim-to-real transfer for scene graph generation. Sim2SG addresses the domain gap by decomposing it into appearance, label and prediction discrepancies between the two domains. We handle these discrepancies by introducing pseudo statistic based self-learning and adversarial techniques. Sim2SG does not require costly supervision from the real-world dataset.
Our experiments demonstrate significant improvements over baselines in reducing the domain gap both qualitatively and quantitatively.
We validate our approach on toy simulators, as well as realistic simulators evaluated on real-world data.

Under review as a conference paper at ICLR 2021
SIM2SG: SIM-TO-REAL SCENE GRAPH GENERATION FOR TRANSFER LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Scene graph (SG) generation has been gaining lot of traction recently. Current SG generation techniques, however, rely on the availability of expensive and limited number of labeled datasets. Synthetic data offers viable alternative as labels are essentially free. However, neural network models trained on synthetic data, do not perform well on real data because of the domain gap. To overcome this challenge, we propose Sim2SG, a scalable technique for sim-to-real transfer for scene graph generation. Sim2SG addresses the domain gap by decomposing it into appearance, label and prediction discrepancies between the two domains. We handle these discrepancies by introducing pscudo statistic based self-learning and adversarial techniques. Sim2SG does not require costly supervision from the real-world dataset. Our experiments demonstrate significant improvements over baselines in reducing the domain gap both qualitatively and quantitatively. We validate our approach on toy simulators, as well as realistic simulators evaluated on real-world data.
INTRODUCTION
Scene Graphs (SGs) in both computer vision and computer graphics are an interpretable and structural representation of scenes. A scene graph summarizes entities in the scene and plausible relationships among them. SGs (Dai et al., 2017; Herzig et al., 2018; Li et al., 2017; Newell & Deng, 2017; Xu al., 2017; Yang et al., 2018; Zellers et al., 2018) are manifestation of vision as inverse graphics. They have found a variety of applications such as image captioning, visual question answering, high level reasoning tasks, image retrieval, image generation, etc. However, most prior work on SG generation relies on the availability of expensive and limited number of labeled datasets such as Visual Genome (Krishna et al., 2017) and Visual Relationship Dataset (VRD) (Iuu et al., 2016).
One of the main limitations in machine learning applications is the general lack of sufficient labeled data for supervised learning tasks. Synthetic data viable alternative to this problem since annotations are essentially free. Synthetic data has been used for variety of tasks such as image classification, object detection, semantic segmentation, optical flow modeling, 3D keypoint extraction, object pose estimation, 3D reconstruction, etc. (Borrego et al., 2018; Butler al., 2012; Dosovitskiy et al., 2015; McCormac et al., 2016; Mueller et al., 2017; Richter et al., 2016; Ros et al., 2016; Suwajanakorn al., 2018; Tremblay al., 2018; Tsirikoglou al., 2017). It has also been shown to be effective in initializing task networks (Prakash et al., 2019) and for data augmentation. However, the use of synthetic data for SG generation and visual relationships yet be explored.
One crucial issue with training on a labeled source domain (synthetic data) and evaluating on an unlabeled target domain (real data) is the performance gap known as domain gap (Torralba & Efros, 10. This gap is due to the difference of data distribution between the source and target domains. Kar et al. (2019) argue that domain gap can be divided into appearance and content gap. The appearance gap can be addressed by making scenes photo-realistic (McCormac et al., 2016; Wrenninge & Unger, 2018), by using image translations (Hoffman et al., 2018; Huang al., 2018; Zhu et al., 2017), by feature alignment (Chang et al., 2019; Chen et al., 2018; Li et al., 2019; Luo et al., 2019; Saito et al., 2019; Sun et al., 2019), or by learning robust representations based on domain randomization (Prakash et al., 2019; Tobin et al., 2017). There are also studies that address the content gap for image classification (Azizzadenesheli et al., 2019; Lipton et al., 2018; Tan et al., 2019). We present a thorough investigation of the domain gap between source and target domains. We assume a gap in both appearance and content, expand those gaps into different sub- aub-components and provide way to address them. We primarily apply our method to reduce the domain gap for

---

Under review as a conference paper at ICLR 2021
SG generation. Nonetheless, techniques can also be applied to other vision tasks such image classification, image segmentation and object detection among others.
We propose Sim2SG (Simulation to Scene Graph); model that learns sim-to-real scene graph generation leveraging labeled synthetic data and unlabeled real data. Extending the formulation in (Wu et al., 2019), Sim2SG addresses the domain gap by bounding the task error (where the task is scene graph generation) on real data through appearance, prediction, label (ground truth) discrepancies between the two domains and task error on synthetic data. Our work differs from (Wu et al., 2019) as they do not provide a way to address the content gap, and their risk discrepancy is intractable. To the best of our knowledge, Sim2SG is the first work to introduce a tractable error bound on the content component of the domain gap.
We minimize the appearance and prediction discrepancies by aligning the corresponding latent and output distributions via Gradient Reversal Layers (Ganin et al., 2017). We address discrepancy in the label using principles of self-learning (Zou al., 2018). However, self-learning based on pseudo labels often suffer from the inaccurately generated labels (e.g. predicted bounding boxes are ill-categorized or imprecise, hence, the model will regress on the wrong objects) (Zheng & Yang. 2020; Kim et al., 2019). Therefore, we instead propose to collect a higher level statistic (e.g. list of objects and their type, position and relationships for placement). that we call eveudo-statistice from target data and leverage the synthetic data generator produce valid objects with their precise labels (e.g. bounding boxes). We experimentally demonstrate our method in three distinct environments- all synthetic CLEVR (Johnson al., 2017), more realistic Dining-Sim and Drive-Sim with driving simulator evaluated on KITTI (Geiger et al., 2012). We almost close the domain gap in the Clevr environment and show significant improvements over respective baselines in Dining-Sim and Drive-Sim. Through ablations, we validate our assumptions about appearance and content gaps. Sim2SG differs from other unsupervised domain adaptation methods (Chen et al., 2018; Xu et al., 2020; Li et al., 2020) as it can modify the source distribution (via self-learning based on pseudo- statisties to align with the target distribution) with access to a synthetic data generator. We also outperform these domain adaptation baselines (Chen et al., 2018; 8; Xu et al., 2020; Li et al., 2020) as shown in Section 4.3.
Contributions: Our contributions are three-fold: In terms of methodology, to the best of our knowledge, (1) We are the first to propose sim-to-real transfer learning for scene graph generation. We do not require costly supervision from the target real-world dataset. (2) We study domain gap from synthetic to real data in detail, provide a tractable error bound on the content component of the gap and propose novel pipeline including pseudo statistics fully handle the gap. Experimentallyy, (3) we show that Sim2SG can learn SG generation and obtains significant improvements over baselines in all three scenarios- Clevr, Dining-Sim and Drive-Sim. We also present ablations to illustrate the effectiveness of our technique.
2 PROPOSED METHOD: SIM2SG
Our proposed Sim2SG pipeline illustrated in Figure 1. We first describe how we generate scene graphs in Section 2.1. When we naively train source distribution (synthetic data) and evaluate on a target distribution (real data), we have a domain gap (Torralba & Efros, 201 1). We study it in more detail in Section 2.2 and propose methods address it.
2.1 SCENE GRAPHS
This section describes scene graphs (SGs) and how we train the SG predictor module using labels from the source domain.
Notation: We represent a scene graph of a given image 7 as a graph G with nodes o and edges Each node is a tuple o, = (b,, ea) of bounding box 6: = {rming ymin, Wi, ht) and category C,. Relationships r are a triplet of (o,, p, oj) where p is a predicate. SG prediction has two key components: feature extractor and graph predictor h. ó maps input space to a latent space and h maps from latent space to output space y. The predicted SG G = h (d(x)).
We use Resnet 101 (He et al., 2016) to implement & and GraphRCNN (Yang et al., 2018) architecture to implement h. We train the networks o and h using the following task loss (Yang et al., 2018): cross entropy loss for object classification & relationship classification and & loss for bounding boxes.
2

---

Under review as a conference paper at ICLR 2021
Sell-Leannngg
Rendewer
Real
3D Scenies
Labeled Sim
Predactor
Unlabeled Real
Kncoder
GRL
Prediction falrnd
Sim Task F,
Figure Overview of Sim2SG. We first map the labeled synthetic and unlabeled real data shared representation Z using encoder ø. Then we train scene graph prediction network h on Z using synthetic data. We align label discrepancies (content gap) between the two domains using pseudo statistics based self-learning. We further bridge the domain gap by aligning features in the representation space Z (appearance gap) and output space (content gap) using Gradient Reversal Layer (GRL) and domain discriminator (Ganin al., 2017). Blue, green and red arrows indicate the flow of synthetic data, real data and back propagation, respectively.
Our framework Sim2SG is illustrated in Figure 1. It is worth noting that we predict the relationships among objects in the scene, but not their attributes like (Yang et al., 2018).
2.2 DOMAIN GAP
We now study the domain gap between source and target domains and formulate the SG generation task error real domain as function of the task error synthetic domain and appearance & content gap between the domains. We then propose methods to address each subcomponent of the gap.
Notation and assumptions: We have two domains: synthetic (r., ys) p(r,, y) and real domain (Ir, Wr) ~ q(r, y). Note that Ss or Ey is the input (image) and ys or yr is the output (SG) as introduced in Section 2.1 It is also worth noting that yr not known. We assume that synthetic and real domains contain the same categories of objects. Both domains also share similar scenarios (e.g. both have driving scenes). However, joint distributions of scenes (images) are different in both domains (i.e. p(z) a q(z)). The label (ground truth) distributions are also different in the two domains (i.e. p(y) * {g))). Using the formulation in (Wu et al., 2019), the task (SG generation) error (i.e. risk) on synthetic domain function of the latent space is given by:
c,(0, = p(z)e,dz
where 6, is the risk defined as e, (p((l2) - yol- p(2) is the distribution of features, p(y) is the distribution of labels and p(y]z) is the output distribution. The bound task error in real domain (, h) from Wu et al. (2019) is a function of three terms which are: (I) task error on the synthetic domain (d, h), (2) risk discrepancy between the domains ( h) and (3) feature discrepancy between the two domains e (Ø, h). Our goal is to minimize the task error on the real domain:
f q())e,dz= = ((=)e,dz + / p(z)),dz= -
Soodds--sse p()))cde + g(())edz - ff / p(z)e,dz / -c))dz ((((t) (ole/(o()--(()))) -((2))edd
Go( h) =
= (1)
( + (b.
drop the terms Φ, h from now on. We would need to minimize e., e and e to reduce the task error on the real domain e, 0. If the error e, reduces to zero on the target domain, we have closed the domain gap. Since we have access to the label ye. we can minimize the task eтror on synthetic domain 6, described in Section 2.1. However,
do not have access to the label distribution g(v) for the real domain. This makes the risk discrepancy e" intractable. We split this discrepancy into tractable components and address them in Section 2.2.1. We call the diserepancy content gap based
3

---

Under review as a conference paper at ICLR 2021
Algorithm Pscudocode for Sim2SG training
1: Given: de, heg
Encoder, Scene Graph: predictor
2: Given: Ya,Ys R, X
Syntheric Images, Synthetic Labels, Generator, Real Images
3: Hyperparameters: E,,
Fpochs, Iterations b Training
4: for E E do
5: loss =0;
6: for E Im do
loss += ( o(Er))
Appearance gap
8:
loss += (h ((((())))))))))))))))))))))))( h(o(x,))
Content prediction gap
loss += tz= Ua)
D Scene Graph generation
10:
do, hig optimize(@ei..lle Ie. loss);
Network parameters 0:; SGD slep
11:
end for
12:
ps = (h (d((z))
X.)
D extracting preudo statisties from predictions generating synthetic with labels aligned
13:
X., Y. R(ps)
14: end for
ablations in Section 4.1. Recent work shows that features frequently account for appearance (Geirhos et al., 2019), and therefore we call the feature discrepancy e as appearance gap and empirically show through ablations in Section 4.1 to be the case. We address this discrepancy in Section 2.2.2. The pseudo-code of our proposed method is illustrated in Algorithm 1.
2.2.1 CONTENT GAP:
Content gap refers to discrepancies between the two domains including the difference in distribution of the number of objects and their class, placement, pose and scale. This affects the position, dimension and the type of the labels leading to the problem of label shift (i.e. discrepancy in the ground truth distribution). However, minimizing ef in the current form is not tractable. Since the task error (risk) is non-negative and both and e,, are lower bounded by zero, we also assume lower bound of zero for e Hence, we approximate the (e, - e.) as ((q(ylz) =) - p(y]=)) ) (y. y-)). We show this in equation 3. We assume y/ Y, and ((vl=) = p(y|=) =) as sufficient condition for ((q(viz) - p(y[=)) (ye y-)) to be zero and to avoid any degenerate solutions. Based on these assumptions, also empirically demonstrate in Section that the label discrepancy e.,label and the prediction discrepancy ereeed help reduce the domain gap.
arg min e" arg min / q(=)(e, - e2)dz
arg min / q(=)(v= yr al=)(alul))- p(()l)))))) p(yl=))dz
arg min
cdabbel
C-pred
Label discrepancy and "Beudo-Statisticss Minimizing ed.label challenging because we not have access to the label (y,)) of the target (real) domain. We then propose to get estimate of y through principles of self-learning (Zou et al., 2018) based on minimum reliable statistic (pscudo- statistic) of target data and generate aligned synthetic data (ys) narrow the gap. We generate SGs for all input images (I,) of the target domain and derive the pseudo statistic ps from each SG by retaining the minimum information needed for 3D representation (e.g. discard the entire bounding box and keep the centroid position). The statistic ps is a list of objects with each object's type, centroid location and relationship with others. Using either known or assumed camera intrinsic (e.g. car dash cam), we map ps to a full 3D scene. Some unknown parameters (e.g. texture or pose) and context (ground, sky, light) are randomized as done in (Prakash al., 2019). We use a synthetic data generator to render those scenes as shown in Figure 1.. More details scene generation for different environments are in Sections A.2.1, A.2.2 and A.2.3 of Appendix. This also analogous to an Expectation Maximization algorithm where we compute pseudo-statisties and generate aligned synthetic data (E-step) and then use it to train the Sim2SG model (M-step).
Contrarily, self-learning based on pseudo labels (Zou et al., 2018) generated from and trained on real data often suffer from poorly generated labels (Zheng & Yang, 2020; Kim et al., 2019). For instance, training SG generation model on ill ill-cateorrized imprecise bounding boxes lead to poor detection results. Pseudo-Statisticn (described previously) integrated with a synthetic data generator (renderer) will still produce valid scene for those ill-detected objects (false positives) with
4

---

Under review as a conference paper at ICLR 2021
precise labels (bounding boxes). We show through quantitative experiments that our method performs better than pseudo label based self-learning in Section 4.3. Please note that our method is orthogonal to pseudo labels based self-learning and the latter can potentially be applied in conjunction.
Prediction discrepancy:
The output of the scene graph generation model should be the same for same categories in different domains. To address the prediction discrepancy cl-reed we propose to align output distribution for latent code belonging to the same visual category. To this end, we align the output distributions p(y]2) and g(y]=) using a GRL based technique (Ganin et al., 2017) that we will discuss in Section 2.2.2.
2.2.2 APPEARANCE GAP: ex
Appearance gap is the discrepancy in the appearance of the two domains. This includes differences in texture, color, light, reflectance, etc. of objects in the scene. To address the appearance gap, we want to avoid photo-realism in synthetic data as it requires high quality assets and tremendous amount of effort from artists (McCormac et al., 2016; Wrenninge & Unger, 2018, 8). Therefore, we propose to learn appearance invariant representation so we can avoid the appearance bias in the model. However, the features 2 ((x) have both content and appearance components and aligning them may be detrimental (Saito et al., 2019; Wu et al., 2019) as shown in the ablation in Section 4.3. To overcome this challenge, we align the appearance gap c* only after we align the content gap e of the two domains using the method described in Section 1.2.I. We minimize this appearance gap ca by aligning the feature distributions p(z) and (22) as follows:
0- = arg min / (a(2) (((2)-p(23) p(=))e,d=
(4)
0
where we exploit the fact that p(z) = g(2) is a sufficient condition for e" to be zero (Wu et al., 2019). We use Gradient Reversal Layer (GRL) (Ganin et al., 2017) to align the distributions p(z) and (=)) along with domain classifier D to classify them. We minimize the D's loss w.t. its own parameters while maximizing w.c.t. the network parameters of 6 (0). We do so through GRL that acts as identity function during forward propagation and flips the sign of the gradients during back propagation from D to é. We provide details of GRL layers in Section A.1
3 RELATED WORK
Scene Graphs: There has been a plethora of work on Visual Relationships and Scene Graph (SG) generation (Dai et al., 2017; Herzig et al., 2018; Li et al., 2017; Newell & Deng, 2017; Xu et al., 2017; Yang al., 2018; Zellers et al., 2018). While all this work is fully supervised and some use priors, they still depend on availability of expensive, large scale data (Krishna et al., 2017: Lu et al., 2016). Krishna et al. (2019) propose a way to generate SGs using limited labels and Dornadula et al. (2019) learn to generalize SG prediction to rare categories in few shot setting. However, they still use either existing object detection models trained on large data or ground truth object labels. We, on the other hand, require no labels whatsoever in the target domain. Alternatively, there are also works on scene generation from SG (Ma et al., 2018; Herzig et al., 2020)
Synthetic Data has been used for many tasks including, but not limited to, object detection (Kar et al., 2019; Prakash et al., 2019), semantic segmentation (Richter et al., 2016; Ros et al., 2016; Tsirikoglou et al., 2017), optical flow modeling (Butler et al., 2012: Dosovitskiy al., 2015), scene flow (Mayer et al., 2016), classification (Borrego et al., 2018), stereo (Qiu & Yuille, 2016; Zhang al., 2026 6). 3D keypoint extraction (Suwajanakorn al., 2018), object pose estimation (Mueller et al., 2017: Tremblay et al.. 2018) and 3D reconstruction (McCormac et al., 2016). However, to the best of our knowledge, synthetic data has not been applied to scene graph generation.
Domain Gap is the performance gap when the network is trained on synthetic domain and evaluated on real data. Kar et al. (2019) argue that domain gap has two components which are appearance and content. We work with the same assumptions. Most prior work addresses the appearance gap by image translations (Chen et al., 2019; French et al., 2018; Hoffman et al., 2018; Huang et al., 2018; Li et al., 2018; Zhu et al., 2017), clever feature alignment (Chen et al., 2018; Li et al., 2019; Luo et al., 2019; Saito et al., 2019; Xu et al., 2020; Li et al., 2020) and domain randomization (Prakash et al., 2019; Tobin et al., 2017). There are few works which handle the content gap (Azizzadenesheli et al., 2019; Lipton et al., 2018; Tan et al., 2019; Zhao et al., 2019). More specifically. they address the label shift between the two domains. However, they do not exploit the unlabeled images from
5

---

Under review as a conference paper at ICLR 2021
Table 1: Quantitative results of Sim2SG when evaluated on the target domain in CLEVR environment.
Trained on
Cube AP
Cylinder AP
Sphere AP
mAP @0.5 ToU
Recall @ 20
Source only
0.479 10.130
0.810 +0.048
0.881 +0.024
0.723 +0.053
0.356 +0.047
Source + e-labbl
0,805 10.081
0.779 #0.093
0.913 #0.027
0.832 10.046
0.493 10.064
Source + c
0.831 10.
0.723 +0.176
0.908 +0.002
0.821 +0.048
0.815 +0.026
Source + c² c.abbll
0.903 #0.009
0.827 +0.098
0.944 +0.034
0.892 #0.024
0.888 +0.018
the target domain. We, on the other hand, leverage the images from the target domain to reduce the domain gap further. We are also interested in a scene graph generation task which is more complex than classification. The idea of self-training with pseudo labels (Zou et al., 2018) is used in (Li et al., 2019; Tan et al., 2019) to learn models from target distribution. However, the labels predicted by the model on the target are often inaccurate because of domain gap (Zheng & Yang, 2020). We instead rely on pseudo statistics and use a synthetic data generator to produce accurate labels. Similar to us, Chang et al. (2019); Sun et al. (2019) also train their task model on top of domain invariant features for image classification and image segmentation.
4 EXPERIMENTS
We evaluate Sim2SG in three different environments with increasing complexity. In each environment we have a fully labeled source domain and unlabeled target domain with labeled test data. We present a simple environment using CLEVR (Johnson et al., 2017) as described in Section 4. 1.1 and a more realistic synthetic environment in Dining-Sim using ShapeNet (Chang et al., 2015) in Section 4.2. We then use an Unreal Engine 4 based driving simulator similar to (Prakash et al., 2019) as the source domain, and real images from KITTI (Geiger et al., 2012) the target domain in Section 4.3. Using these experiments, we show that Sim2SG learns sim-to-real scene graph (SG) generation, it reduces the domain gap, and our label alignment (Eq. (3)), prediction alignment (Eq. (3)) and appearance aligmment (Eq. (4)), work as intended. We compare Sim2SG to several baselines in DriveSim environment (Section 4.3). Our quantitative evaluation metric includes detection mAp (mean average precision) @ 0.5 IoU (Intersection over Union) and relationship triplet recall @20 or @50 (Krishna al., 2017). Note that relationship triplet recall implicitly includes object detection recall as well (see Appendix). All the mcan and standard deviations are based on five runs.
4.1 TOY EXPERIMENTS ON THE CLEVR DATASET
The goal of the experiments on the CLEVR environment (Johnson et al., 2017) is to show that Sim2SG can learn sim-to-real scene graph generation on an unlabeled domain while addressing the domain gap between source and target domains. Additionally, we show that appearance alignment e and label alignment co.bell work as intended through ablations. For our experiments, we have classes of objects: cube, sphere and cylinder and 4 kinds of relationships: front, behind, left and right. Our source domain mimics the synthetic environment by having simple texture, different number of objects (4) and closer objects. Nonetheless, objects placed at random locations in the scene as shown in the top row of Figure 4. The target domain is representative of real data with a different set of colors, number of objects (2 3) and objects placed with larger margin. Regardless, the objects can be placed anywhere and have more complex texture by applying a style transfer network to the generated scenes (second row of Figure 4). We use 1000 labeled images of source, 1000 unlabeled images of target for training and 200 labeled images of source and target for evaluation. Details of the environment, training and typer-parameters are in Section 1 of Appendix.
Results: Quantitative evaluation of Sim2SG is reported in Table 1. When we train on the source domain and evaluate on a test set from the source domain itself, we achieve 1.0 mAP @0.5 IoU for detection 0.986 recall @ 20 for relationship triplets. Hence, the first row shows that there is a domain gap from source to target as seen from the lower mAP and recall values. Second and third rows show how label aligmment e.label and appearance alignment c², drastically reduce the domain gap compared to baseline (source only). We see that e.label improves detection performance as is possible that the detection recall is more sensitive to content gap. Nonetheless, e is more effective in improving the relationship triplet recall as we have found it to reduce false positive detections effectively (Figure 6). The domain gap reduces further by combining both ca.aaell and e" terms (fourth row). Please note that we report saturation performance (details in Appendix). Qualitative improvements of scene graph recall over baseline are shown in first row of Figure 2.

---

Under review as a conference paper at ICLR 2021
Table Left (resp. right): Source and target domains have different (resp. similar) appearance but
similar (resp, different) content distribution. All the evaluations are the target domain.
Trained on
mAP @0.5 IoU 2ecall@20
Trained on
mAP @0.5 IaU
Recall @220
Source only
0.675
0.339
Source only
0.76
Source + etabell
0,923
0,646
Source e
0.97
0.722
Source c
0.938
0.938
Source e c.labes
1.0
0.996
2
chair.D
Q1
right
01
Figure 2: Qualitative results of Sim2SG on the target domain for CLEVR (first row), Dining Sim (second row) and Drive-Sim environments (last three rows). First column shows that the source only baseline fails to either detect objects or have high number of false positives (mislabels) leading to poor scene graph. Our method detects objects better, has way fewer false positives and ultimately generates more accurate scene graphs as shown in second and third column respectively. Objects are color coded. For better visibility, only show partial scene graph for Drive-Sim.
Ablations: We conduct two sets of experiments on the CLEVR dataset and quantitative results are shown in Table 2. The first experiment studies appearance gap: source and target have the same number of objects and range of margin, but they use different color and texture. Additionally, we transform the target by using the style transfer network. We observe that style alignment e reduces the domain gap with significant improvement over baseline. However, the label alignment cllabel fails to have significant improvement on relationship triplet recall. Similarly, the second experiment studies content gap only (i.e. source and target use same color and texture but different number of objects and margin). We observe that label alignment e.label closes the domain gap completely, However e leads to performance degradation. These experiments show our label alignment cllabel reduces content gap and appearance alignment e" addresses appearance gap.
4.2 EXPERIMENTS ON DINING-SII
The goal of our experiments on Dining-Sim created from ShapeNet objects (Chang et al., 2015) is to show that Sim2SG works as intended on more complex dataset where target domain is representative of real data. This dataset has 3 classes of objects chair, table and laptop. There are 5 kinds of relationships front, behind, left, right, and The source and target domains are illustrated in the top two rows of Figure 7 and details are in Appendix. The results agree with the findings of Section 4.1. We also see domain gap in this environment evaluation on source domain (1.000
7

---

Under review as a conference paper at ICLR 2021
Table 3: Evaluation on KITTI hard when training Drive-Sim synthetic environment. The class specific AP and mAP are reported at 0.5 IOU.
Trained
Car
Podestrian
Fouse
Veg.
mAP
R 50
Source only (Prakesh et al. 2019)
0.382 2:0 029
0.168 2:0.01
0.211 +:0023
0.174=00.010
0.234 1:0.00000000 1-0.006
0.070 200.007
DA-FasteRCNN (Ches etal. 2018)
0.424 4:0 002
0.170 0010:00.22
0.200
0.169 4:0014
0.241 200014
0.074 4000015
Meto Sum (Kar al. 2019)
0.413 +0.-99
0.197
0236
0.164 +00023
0253
0,055 400000
Seif-lennnng (7on et al. 2018)
0.312 1312:20000 .066
0.167 20.015
0.191 20 003
0.363 4: 006
0.233 2:0 004
0.62 20.003
GPA (Nu al., 2020)
0.174 2:0 040
0011 +:0016
0.106 +0.031
0.059 +00027
0.087 +00020
0.155 +0.005
SAPNet (Tá 2020)
0:362 054
0.085 4:0
0.116 1:00220022
0.067 40.022
0.157 10.024
Ours (e (eabb))
0:410 1110:1:0000 009
0.262 1:0 025
02400 1:0 0.0.01
0:229 1:0.036
0.285 1:0 ::0.003 003
0.101 100.00
(g lubel
0.453 0.00
0.252 40.014
0,247 4-0.012
0.253 4-0.200
0,311 +0.311
0.127 +0.004
Outs (e faabbe c eoppee
0,501 100000
0.241 1:0.018
0.254 1:0.010
0.269 1:0.014
0.316 1:0.004
0.139 1:0.004
mAP @0.5 IoU & 0.995 50) is higher than on target domain (0.584 mAP @0.5 loU & 0.331 recall@ 50). The oracle performance on target domain stands at 0.904 mAP@0.5 loU & 0.846 recall @ 50, Label alignment eatbael drastically improves performance on target domain (0.713 mAP @0.5 IoU & 0.501 recall 50). e" reduces false positives (Figure 10). We achieve the best scene graph recall @50 using a combination of label alignment eo.label, appearance alignment ea and prediction alignment esereed (0.729 mAP @0.5 loU & 0.547 recall@50). Complete results that validate our approach are in Table 4 in Appendix. Qualitative improvements of scene graph generation over baseline are illustrated in the second row of Figure 2.
4.3 REAL-WORLD EXPERIMENTS ON DRIVE-SIN
In this section, we validate our approach on a real-world dataset. For synthetic data, we use a simulator similar to (Prakash et al., 2019) with minor simplifications (fixing camera and road spline parameters, exclusion of some objects, see details in Appendix) to make our generation easier. The number of lanes, sidewalk. cars, vegetation, houses, pedestrians; their positions, pose, color, texture; light settings are randomly picked from a set of realistic values (details in Appendix.) akin to (Prakash al., 2019) as shown in Figure 13. We use four classes: car, pedestrian, vegetation, house and four types of relationships: front, left, right, behind. All our relationships have the car as the subject. For example, 'car behind car', 'vegetation left car', etc. Although we show the 'on' relationship to work in Dining-Sim environment, we found these relationships were always trivial to predict because they are always true: e.g. cars are always 'on' road, pedestrian 'on' sidewalk, etc. Therefore, we did not include them in the experiments. We use KITTI (Geiger et al., 2012) as the target domain. We need a small amount of labels on KITTI for evaluation only. Hence, while keeping the existing annotations for cars and pedestrians, we add annotations for vegetation and houses along with relationships among them. We will release these annotations to the community. We use 6000 labeled synthetic images, 6000 unlabeled KITTI images for training, and 1000 labeled synthetic and 550 labeled KITTI images for evaluation. See Appendix for details on KITTI annotation schema, training and
We compare Sim2SG to the randomization based method (Prakash et al., 2019), the method addressing content gap (Kar et al., 2019), self-learning based on pseudo labels (Zou et al., 2018) and domain adaptation methods for object detection (Chen et al., 2018; Xu et al., 2020; Li et al., 2020).
Baselines:
Prakash et al. (2019) interest(carr. Kar et al. (2019) learn the parameters of a renderer to match the target distribution to address the content gap. Unsupervised domain adaptation methods (Chen et al., 2018; Xu al., 2020; Li et al., 2020) align the features from source and target domain. We discuss self-learning based on pseudo labels (Zou et al., 2018) in Section 1.2. We train our baselines on 6000 images, using the hyper parameters provided by the authors. We adapt the baselines to our framework with Resnet101 as backbone. The details can be found in Section A.2.3.
context based randomization of pose, position and texture of objects of
We evaluate scene graph generation on three KITTI evaluation modes: easy, moderate and hard based on object size, occlusion and truncation (Appendix). In all three settings, we improve over our baselines. We report the results for KITTI hard in Table 3, the detailed results for KITTI easy and moderate can be found Table 6. We perform better than Meta-Sim as it cannot align the structure of the scenes (e.g. number of object in a scene) to match the target distribution. Our predicted scene graphs also differ from the intermediate scene graphs in Meta-Sim which lacks the notion of relationships. We outperform other domain adaptation methods (adapted to our framework) for object detection (Chen et al., 2018; Xu et al., 2020; Li et al., 2020) because we believe that feature alignment without good label/content alignment may not be effective. We discussed this
Results:
8

---

Under review as a conference paper at ICLR 2021
Initialization
First epoch
Third epoch
Target
Figure Synthetic data in Drive-Sim environment changing through the training using Label alignment cd.label From left to right: synthetic data at initialization (different number, placement of objects w.r. KITTI,, after first epoch (alignment of number and placement of objects, but with noise), after third epoch (better alignment of number and placement of objects w.r.t KITTI) and corresponding KITTI samples)
briefly in Section 2.2.2 and further show an ablation (next paragraph) that Sim2SG also suffers from the same issue. We believe that our label alignment e.label can be used in conjunction with other domain adaptation methods. The reason we improve over self-learning based on pseudo labels also discussed in Section 2.2.1. The last three rows Table show that most improvements come from label alignment e ellabel and appearance alignment e" and the combination of ec,label e and en-pred achieves the best relationship triplet recall. We notice that the AP of the pedestrian category does not improve with c and e'-pred The reason might be that pedestrians are inder-represented. small and hard to detect class KITTI. Sim2SG can align the label distribution but cannot address the class imbalance in the target domain. The qualitative results are shown in last three rows of Figure 2. We see that Sim2SG significantly improves on both false positives and recall of objects. As result, generates more accurate scene graphs. This is because label alignment clabbll generates nicely aligned data (Fig. 11) and appcarance alignment e" reduces false positives (see Fig. 2). Figure qualitatively shows how synthetic data is adjusted over the duration of training towards more label alignment w.r. KITTI using daaabll
Ablations: As briefly discussed in Section 2.2.2, run the label alignment cotobel before appearance aligmment e" and prediction alignment ed-pred to address the fact that feature alignment can be detrimental if the content of both domains drops significantly when we train Sim2SG without ecpred and evaluate in the same setting as Table 3. Sim2SG with ex e-ered gives 0.246 mAP @0.5 IoU for detection & 0.076 recall @ 50 for relationship triplets while simply adding erer to it, we get 0.316 mAP @0.5 IoU for detection & 0.139 recall @ 50 for relationship triplets (KITTI Hard). This shows the effectiveness of daabel and importance of the entire Sim2SG framework.
aligned. We indeed found that our performance
5 CONCLUSION
In this work, we propose Sim2SG, model that achieves sim-to-real transfer learning for scene graph generation on unlabeled real-world datasets. We decompose the domain gap into label, prediction and appearance discrepancies between synthetic and real domains. We propose methods to address these discrepancies and achieve significant improvements over baselines in all three environments Clevr, Dining-Sin and Drive-Sim. We do require access to simulator and 3D assets. However, this limitation is mitigated with the availability of open source simulators (Dosovitskiy al., 2017; Deitke et al., 2020; Kolve et al., 2017: To et al., 2018; Crespi et al., 2020; Denninger et al., 2019; Xiang al., 2020) and exciting work around object mesh generation (Mescheder et al., 2019; Xu et al., 2019; Gkioxari et al., 2020; Wang et al., 2018). For future work, we plan to learn more complex relationships and explore generative modeling for learning our representation.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Kamyar Azizzadcnesheli, Anqi Liu, Fanny Yang, and Animashree Anandkumar. Regularized learning for domain adaptation under label shifts. In International Conference on Learning Representations, 2019, URL net/ forum?id Dorum?id=::10rRR9P
J. Borrego, Atahak Dehhan, Rui Figueiredo, Plinio Moreno, Alexandre Bernardino, and J. Santos- Applying domain randomization to synthetic data for object category detection. arXiv:/807.000 2018.
Daniel J. Butler, Jonas Wulff, Garrett B. Stanley, and Michael J. Black. A naturalistic open source movie for optical flow evaluation. In A. Fitzgibbon et al. (Eds.) (cd.), ECCV, Part IV, LNCS 7577, pp. 611-625. Springer-Verkse 2012.
Angel X. Chang. Thomas Funkhouser, Leonidas Guihas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich 3d model repository, 2015.
Wei-Lun Chang, Hui-Po Wang. Wen-Hsiao Peng, and Wei-Chen Chiu. All about structure: Adapting structural information across domains for boosting semantic segmentation. 2019 IEEE/CVF Conference on Computer Vision and Pattemn Recognition (CVPR), Jun 2019, doi: 10.1 10.1109/vpr.2219..00222 URL http:/ /ttp////xxx......./ org/10. 1109/ /wt.9///////2// 2019. 00200.
Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Domain adaptive faster r-cnn for object detection in the wild. 2018 TEEE/CVF Conference on Computer Vision and Pattern Recognation, Jun 2018. doi: 10. 10.11 00352................ URL http: / dx doi. org/10/ggo/oooooooooo 1t///10110000 1tt.1119//CC0 /CV//. 2018 00352.
Yun-Chun Chen, Yen-Ye Yu Lin, Hing-Hsuan Yang. and Jia-Bin Huang. Crdoco: Pixel-level domain transfer with cross domain consistency. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2019. doi: 10.1109/cxp:2019.000E URL http: http://xx..oooooo www.aaa.....oooooooooooooooooooe 10. 1109/CVER.
2019.001890
Blender Online Community, Blender a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL http: /www. blender. org.
Adam Crespi, Cesar Romero, Srinivas Annambhotla, Jonathan Hogins, and Alex Thaman. Unity perception. 2020. https:/ //bl:////ooo.so/ y com/20200 /020/06/0///
Bo Dai, Yugi Zhang, and Dahua Lin. Detecting visual relationships with deep relational networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jul 2017. doi: 10.12.99 wecii.org/in wrw.10................oo wwt..1199/c......
URL /xx dei.
352.
Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozheh Mottaghi, Jordi Salvador. Dustin Schwenk, Eli VanderBilt, Matthew Wallingford. Luca Weihs, Mark Yatskar, and Ali Farhadi. RoboTHOR: An Open Simulation-tor Ro-l Embodied AI Platform. In CVPR, 2020.
Maximilian Denninger, Martin Sundermeyer, Dominik Winkelbauer, Youssef Zidan, Dmitry Olefir, Mohamad Elbadrawy, Ahsan Lodhi, and Harinandan Katam. Blenderproc. arXiv preprint
2019.
Apoorva Dornadula, Austin Narcomey, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Learning predicates as functions to enable few-shot scene graph prediction, 2019.
A. Dosovitskiy, P. Fischer, E. Ilg, P. Häusser, C. Hazıraa, V. Golkov, P. Smagt, D. Cremers, and T. Brox. FlowNet:
Learning optical flow with convolutional networks. In IEEE International Conference on Computer Vision (ICCV), 2015.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. In CORL, pp. 1-16, 2017.
Geoffrey French, Michal Mackiewicz, and Mark Fisher. Self-ensembling for visual domain adaptation. In ICLR, 2018.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Paseal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. Advances in Computer Vision and Pattern Recognition, URL http::/tt./...c dci. org 10. 1007000000000 8- 3- 58347-13471711
209- 2017. ISSN 2191-6594 doi: 10. 00007/97833319-584-77
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for Autonomous Driving? The KITTI Vision Benchmark Suite. In CVPR, 2012.
10

---

Under review as a conference paper at ICLR 2021
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trainee CNNs are biased towards texture, increasing shape bias improves accuracy and robustness. In International Conference on Learning Representations, 2019, URL https: http:::///opcccvillc net/ forum? 1du JOKKX.
Georgia Gkioxari, Jitendra Malik, and Justin Johnson. Mesh r-cnn, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference Computer Vision and Pattern Recognition (CVPR), Jun 2016. doi: 10. 10.1009/pr:2006..000 URL http: / /xxx dei. orgg 2/gg/.0...0 .. 99/ cvpr 2016 90.
Roei Herzig. Moshiko Raboh, Gal Chechik, Jonathan Berant, and Amir Globerson. Mapping images to scene structured prediction. In Advances in Neural Information Processing
graphs with permutation-invan Systems, pp. 7211 1 122211 2018.
Roei Herzig, Amir Bar. Huijuan Xu, Gal Chechik, Trevor Darrell, and Amir Globerson. Learning canonical representations for graph image generation. In European Conference on Computer Vision, 2020.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun-Yan Zhu, Phillip Isol, Kate Saenko Alexei A. Efros, and Trevor Darrell. Cycada: Cynss-con adversarial domain adaptation. In ICML, 2018.
Xun Huang, Ming-Yo Yu Liu, Serge Belongie, and Jan Kautz. Multimodal unsupervised image-to-image translation. In ECCV, 2018.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
Amlan Kar, Aayush Prakash, Ming- Yu Liu, Eric Cameracci, Justin Yuan, Matt Rusiniak, David Acuna, Antonio Torralba, and Sanja Fidler. Meta-sim: Learning to generate synthctic datasets. 2019 IEEE/CVE International Conference on Computer Vision (ICCV), Oct 2019. doi: 10. 1.100//cc..2011.0446. URL http: org/10. 11..0.1109//111 ICCV. 2019.000000000 00465.
Seunghyeon Kim, Jachoon Choi, Tackyung Kim, and Changick Kim. Self-training and adversarial background regularization for unsupervised domain adaptive one-stage object detection. 2019 IEEE/CVP International Conference Computer Vision (ICCV), Oct 2019. doi: 10. 00.1099/cev.2219.00611 URL htthttp: /di. org/10. 11w111119911//1 2019. 00619.
Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An interactive 3D environment for visual Al. arXiv preprint 1rXiv:1712.05474, 2017.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, and et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-33, Feb 2017. ISSN 1573-1405. s11263- 1111263006--80
doi: 10.10077s11223-016-699 0981-77. 0- 0981-7.
URL http://dei /ttp://xxx.d doi. org/ /10 1007/
Ranjay Krishna, Vincent Chen, Paroma Varma, Michael Bernstein, Christopher Re, and Li Fei-Fei. Scene graph prediction with limited labels. 2019 IEEE/CVF International Conference Computer Vision (ICCV), Oct 2019. doi: 10.1109/ice.2011900000
URL http: / /xx dol www./rlg....... http//...111../. ICCV, 2019 00267.
Congcong Li, Dawei Du, Libo Zhang, Longyin Wen, Tiejian Luo, Yanjun Wu, and Pengfei 7hu. Spatial attention pyramid network for unsupervised domain adaptation, 2020.
Peilun Li, Xiaodan Liang, Daoyuan Jia, and Eric P. Xing. Semantic-aware grad-gan for virtual-to-real urban scene adaption. In BMVC, 2018.
Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang, and Xiangang Wang. Scene graph generation from objects, phrases and region captions. 20/7 IEFE International Conference on Computer Vision (ICCV), Oct 2017. doi: 10.1 1109/ccv.2017.145
URL http: doidoi....oo.oooo.ooo.....o..... www.doo.....o..oo.oo..ooo..oo... .org/ 10. 1099 1CCV 2017 142.
Yunsheng Li, Lu Yuan, and Nuno Vasconcelos. Bidirectional learning for domain adaptation of semantic segmentation. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Jun 2019. doi: 10. 1099cvpp.001..007100 URL http:/ //ap::/dxx..e www.dooooooooooooooo oogg/ 1109//VVRR 2019 00710.
7achary C. Lipton, Yu-Xiang Wang, and Alex Smola. Detecting and correcting for label shift with black box predictors, 2018.
11

---

Under review as a conference paper at ICLR 2021
Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. In European Conference on Computer Vision, 2016.
Yawei Luo, Liang Zheng. Tao Guan, Junqing Yu, and Yi Yang. Taking closer look domain shift: Category- level adversaries for semanties consistent domain adaptation. 2019 IEEE/CVF Conference on Computer Vision Puttem Recogmition (CVPR), Jun 2019, doi: 10.1 10.11099/yp:22199.0226. URL http://t///// ptttt://ddxx /txp//xcc//// doi. org/ 1 10. 1109/ CVVRR 2019.0000 00261.
Rui Ma, Akshay Gadi Patil, Matthew Fisher, Manyi Li, Sören Pirk, Binh-Son Hua, Sai-Kit Yeung, Xin Tong, Leonidas Guibas, and Hao Zhang. Language-driven synthesis of 3d scenes from scene databases. In SIGGRAPH Asia 2018 Technical Papers, pp. 212. ACM, 2018.
Nikolaus Mayer, Eddy 1lg, Philip Hausser, Philipp Fischer, Daniel Cremers, Alexey Dosovitskiy, and Thomas Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In CVPR, 2016.
John McCormac, Ankur Handa, Stefan Leutenegger, and Andrew J Davison. Scenenet rgb-d: 5m photorealistic images of synthetic indoor trajectories with ground truth. arXiv preprint arXiv:1612.05079 2016.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 4460-4470, 2019.
Matthias Mueller, Vincent Casser, Jean Lahoud, Neil Smith, and Bernard Ghanem. Sim4CV: A photo-realistic simulator for computer vision applications. In arXiv:1708.05866 2017.
Alejandro Newell and Jia Deng. Pixels graphs by associative embedding. In Advances in neural information processing systems, pp. 2171-2180, 2017.
Aayush Prakash, Shaad Boochoon, Mark Brophy, David Acuna, Eric Cameracci, Gavriel State, Omer Shapira, and Stan Birchfield. Structured domain randomization: Bridging the reality gap by context-aware synthetic data. 2019 International Conference on Robotics and Automation (ICRA), pp. 7249-7255 7255. IEEE, 2019.
Weichao Qiu and Alan Yuille. Unrealev: Connecting computer vision unreal engine. In European Conference on Computer Vision, pp. 909-916. Springer. 2016.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems, pp. 91-99, 2015.
Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In ECCV, 2016.
German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio Lopez. The SYNTHIA Dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In CVPR, 2016.
Kuniaki Saito. Yoshitaka Ushiku, Tatsuya Harada, and Kate Saenko. Strong-west weak distribution alignment for adaptive object detection. 2019 JEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2019. doi: 10.1 10.11099vpp.2001.0000 00.001...... 0019.00712.
URL http:// .oi wooiiiorg/111. 10. //VPR.
Yu Sun, Eric Tzeng, Trevor Darrell, and Alexei A. Efros. Unsupervised domain adaptation through self- supervision, 2019.
Supasorn Suwajanakorn, Noah Snavely, Jonathan Tompson, and Mohammad Norouzi. Discovery of latent 3D keypoints via end-to-end geometric reasoning. NIPS, 2018.
Shuhan Tan, Xingchao Peng, and Kate Saenko. Generalized domain adaptation with covariate and label shift co-aligmment, 2019.
Thang To, Jonathan Tremblay, Duncan McKay, Yukie Yamaguchi, Kirby Leung, Adrian Balanon, Jia Cheng, and Stan Birchfield. NDDS: NVIDIA deep learning dataset synthesizer, 2018. aattps://gat NVIDIA/ Daatt/tt.atat......aaaaaaaaaaee. Datasst__Satthesieeee
com/
Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider. Wojciech Zaremba, and Picter Abboel. Domain random- ization for transferring deep neural networks from simulation to the real world. In IROS, 2017.
A. Torallb and A. A. Efros. Unbiased look at dataset bias. In CVPR 2011, pp. 1521-1528, 2011.
Jonathan Tremblay, Thang To, Balakumar Sundaralingam, Yu Xiang, Dieter Fox, and Stan Birchfield. Deep object pose estimation for semantic robotic grasping of household objects, 2018.
12

---

Under review as a conference paper at ICLR 2021
Apostolia Tsirikoglou, Joel Kronander, Magnus Wrenninge, and Jonas Unger. Procedural modeling and physically based rendering for synthetic data generation automotive applications. arXiv:1710.06270, 2017.
Nanyang Wang. Yinda Zhang, Zhuwen Li. Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d mesh models from single rgb images. 2018.
Magnus Wrenninge and Jonas Unger. Synscapes: photorealistic synthetic dataset for street scene parsing. In arXiv:1810008705, 2018.
Yifan Wu, Ezra Winston, Divyansh Kaushik, and Zachary Lipton. Domain adaptation with asymmetrically- relaxed distribution alignment, 2019.
Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Sapien: A simulated part-based interactive environment. Proceedings of
Yifu Yuan, He Wang, JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11097-11107, 2020.
Danfei Xu, Yuke Zhu, Christopher Choy, and Fei-Fei. Scene graph generation by iterative message passing. 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), Jul 2017. doi: 10.1 10.1109/evp:2011.3333. URL ht //xx doi. wwwoooo//.............oooo.....o 110./CVPRC.CC 2017.330.
Minghao Xu, Hang Wang, Bingbing Ni, Qi Tian, and Wenjun Zhang. Cross-domain detection via graph-induced prototype alignment. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Jun 2020. doi: 10. 10.1109/ccr4222000222000..333 URL http://ddk doi org/10 L109/CVPR42200. 2020. 01237.
Qiangeng Xu, Weiyue Wang. Duygu Ceylan, Radomir Mech, and Ulrich Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction, 2019.
Jianwei Yang. Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation. Lecture Notes Computer Science, pp. 190-706, 2018. ISSN 1611-3349 doi: 00000/977-3300-1122300 41. URL http: / dx. doi. org/10..//.oooo 1007/100000 9988//98--3311 0-000 01246-5_41.
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context, 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Jun 2018, doi: 10.1 20111000ev::2118.0000 8.00611. URL http: http//xx.doodoooooo////ooo http:/...111/1111111 1ww1////9...../ 2018 006111.
Zhang, W. Qiu, Q Chen, X. Hu, and A. Yuille. UnrealSteren: synthetic dataset for analyzing stereo vision. In 1612v::112.0466 2016.
Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey J. Gordon. On lcarning invariant representation for domain adaptation, 2019.
Zhedong Zheng and Yi Yang. Rectifying pscudo label learning via uncertainty estimation for domain adaptive semantic segmentation, 2020.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexci A Efros. Unpaired image-to-image translation using sycle-consisten adversarial networkss. In ICCV, 2017.
Yang Zou, Zhiding Yu, B. K. Vijaya Kumar, and Jinsong Wang. Domain adaptation for semantic segmentation via bals-ba self-trainin training.... In ECCV, 2018.
13

---

Under review as a conference paper at ICLR 2021
A
APPENDIX
A.1 ARCHITECTURE DETAILS
Encoder á and SG Predictor h We use Resnet 101 (He et al., 2016) with imagenet pretraining as the backbone or encoder neural network. We use the RaNNer-RCNN (Ren et al., 2015) and Graph Convolution Network based architecture from GraphRCNN (Yang et al., 201 8) to implement the SG Predictor
For appearance alignment e", we use 2 layer 2D convolution neural network based discrimi- nator with Relu activation. For prediction alignment en-pred we use 2 fully connected neural network based discriminator. We also scale the gradients to the encoder network ø from the discriminator by a factor of 4 in above cases.
GRL
A.2
EXPERIMENTS
A.2.1 CLEVR
The source and target domains of the CLEVR (Johnson al., 2017) environment leverage Blender (Community, 2018) to render 320x240 images and corresponding ground truth scene graphs. Details of the two domains are available Section 4.1. We use colors (blue, green, magenta, yellow) and material (metal) for source domain and different colors (pink, brown, white) and material (rubber) for target domain. Additionally, we transform the target by using style transfer network For both domains, sample each class and their size(small, medium & large) with equal probability. The environment has three lights and a fixed camera. We add a small random jitter to their initial positions during the rendering process. Some samples of source and target domain are shown in Figure 4.
Setup
Details of Generation using Pseudo-Statistics how we generate scenes from pseudo statistics. We assume access to camera parameters.
The Label Discrepancy in Section 2.2.1 describes
Training Details We our experiments in stages. In the first stage, train with appearance alignment ea for 70k iterations. In the second stage, we continue training the model using pseudo statistic based self-learning (label alignment cddabel) for 3 epochs each with 10k iterations.
We optimize the model using a SDG optimizer with learning rate of 0.0001 and momentum of 0.9. We train our model using batch size 4 on NVIDIA DGX workstations. We report saturation peak performance in all our tables. We give equal regularization weights to source task loss appearance alignment e and label alignment clabel
Results More qualitative results of Sim2SG evaluated on the target domain for CLEVR are shown in Figure 5. We see better recall and fewer false positive object detections leading to more accurate scene graphs. Label alignment eaalabll improves object recall, but occasionally introduces some false positive detections. Our appearance alignment e helps in reducing such false positives shown in Figure 6.
A.2.2 DINING-SIM
Setup The Dining-Sim environment is written using Pixar's USD API and rendered with propri- etary renderer. The source domain is rendered with 2 spp (samples per pixels) followed by denoiser. We select chair (cantilever chair), table (workshop table) and 1 laptop (PC). We randomly place chair and table on the floor and laptop the floor as well on the table with random orientation. The asset for each subcategory is randomly chosen from a list of subcategory specific ShapeNet (Chang al., 2015) assets. We also ensure that objects do not overlap by applying collision avoidance with simple box collision volumes. A subset of to 5 simple materials that vary only in diffuse colour is created for each of the walls, floor, chair and table. Laptops use the original asset texture
httpa: wtt.ggghhhh.oooo..o. www.://.wwwoopt..oooooeoooo
14

---

Under review as a conference paper at ICLR 2021
Source
Target
Source
Target
Source
Target
Figure 4: Samples from source and target distributions from Clevr environment. Row 1-2: Source and Target differ in both appearance and content. Row 3-4: Source and Target differ in content but have same appearance. Row 5-6: Source and Target differ in appearance but have same content.
15

---

Under review as a conference paper at ICLR 2021
16+
3phe
Figure 5: Qualitative results of Sim2SG on the target domain for CLEVR. First column shows that the baseline fails to either detect objects or have high number of false positives (mislabels) leading to poor scene graph. Our method detects objects better, has way fewer false positives and ultimately generates more accurate scene graphs shown in second and third column respectively. Objects are color coded.
Figure 6: Appearance alignment e reducing false positive. Top row: source + esilabel bottom row:
source + e c,label + e
16

---

Under review as a conference paper at ICLR 2021
The target domain is rendered using path tracing with 20 spp (samples per pixels) followed by denoiser. We use 4 chairs (Windsor chair), table (kitchen table) and 2 laptops (MacBook). We first place the table with random orientation and position the floor. We then place the four chairs at each side of the table, oriented towards the table centre. Two laptops are then placed randomly on the table surface with a random rotation. The asset for each subeategory is randomly chosen from list of subcategory specific ShapeNet (Chang et al., 2015) assets. For materials, we use subset of 4 to 6 physically based, highly detailed materials for each of the walls, floor, chair and table. Laptops use the original asset texture.
Both domains share room parameters: a fixed camera (60 degree field of view, positioned at far side of the room) and 3 fixed spherical lights. Samples from the source and the target domains are shown in Figure 7. There are five kinds of relationships front, behind, left, right, and on with table as subject. We use 5000 labeled images from source, 5000 unlabeled images from target for training and 1000 labeled images from both source & target domains for evaluation. We use 1024 x 768 image resolution for training and evaluation.
Details of Generation using Pseudo-Statistic Section 2 describes how we generate synthetic data using pseudo-statistics. We assume access to camera parameters. Pseudo statistic derived from a SG contains list of objects, their type, 2D centroid position and relationship with others. We filter the objects and relationships among them using an adaptive threshold (details in the next paragraph) for the generation. Using camera parameters, we place each object in the 3D scene by picking random 3D asset according its type(class) and assigning random pose in the range 09-360* We assume context like ground, wall as described in the previous paragraph. We refine the 3D scene further according to the predicted relationships among objects. For example, we use "on" relationship to refine object placements by adjusting the object (laptop or chair) elevation to match the table top.
Training Details We optimize the model using a SGD optimizer with learning rate of 0.0001 and momentum of 0.9. We train our model using a batch size 2 on NVIDIA DGX workstations. We report saturation peak performance in all our tables. We give equal weights to source task loss E,, appearance alignment c², prediction alignment eepee and label alignment ddlabel
We conduct training in three stages. In the first stage, we train for 40,000 iterations on the source domain. Second stage is training the model using pseudo statistic based self-learming (label alignment e e.labell for 6 epochs each with 10k iterations and score threshold of 0.5. We use the aligned synthetic data from this stage (generated with score threshold of 0.9) to train the next stage. In the third stage, we add appearance alignment e" and prediction alignment e--eeed and train for an additional 20,000 iterations. It takes 24 hours for full training including rendering time.
The purpose of the Dining-Sing Sim environment in more complex setting that is similar to a real-world application. We present the full quantitative results in the table 4 and qualitative results in the Figure 8. We observe that the combination of all alignment terms e.,aaell E² and es-ree gives the best relationship triplet recall of 0.547 @ 50. In order to keep our approach as general as possible, we do not enforce strict rules on object placements and prefer to randomize parameters that are not predicted such as orientation as evident in the qualitative results of label aligmment elolabel in Figure 9. When target domain assets are too dissimilar from the assets in the source domain, it often results in incorrect reconstructions as shown in Figure 9 (last column). We also observe that after label alignment e e.label the model occasionally has false positive detections, particularly in areas of the floor that have intricate patterns. We qualitatively show that these false positives disappear with the addition of appearance alignment e" term (Figure 10).
Results
show that the Sim2SG works as intended
Ablations We conduct two sets of experiments on the Dining- Sim environment. The first experi- ment studies appearance gap: source domain has different appearance but similar content from the target domain. The source domain is generated using the target generation scheme but using source dataset materials as shown in Figure 7. We observe that the appearance alignment e" helps reduce the appearance gap, increasing relationship triplet recall from 0.625 @ 50 to 0.821 @ 50. Similarly, the second experiment studies content gap where the source and target use the same materials but have different assets, object positions and number of objects. We accomplish this by modifying the source generation scheme to select materials from the target dataset. Samples of source and target are shown in rows 3, 4 of Figure 7. We observe that the label alignment e colabel term aids in reducing the
17

---

Under review as a conference paper at ICLR 2021
Source
Target
Source
Target
Source
Target
Figure 7: Samples from source and target distributions for Dining-So Sim. Row 1-2: Source and Target domains differ in both appearance and content. Row 3-4: Source and Target differ in content but have same appearance. Row 5-6: Source and Target differ in appearance but have same content.
18

---

Under review as a conference paper at ICLR 2021
daak
laptop_3
duil.
on
tati__3
Char
dsir2
lynw.
thek-
chir22
left
Fextt
Iebist
1able
Figure Qualitative results of Sim2SG on the target domain for Dining-Sim. First column shows that the baseline fails to either detect objects have high number of false positives (mislabels) leading poor scene graph. Our method detects objects better, has way fewer false positives and ultimately generates accurate scene graphs shown in second and third column respectively, Objects color coded.
Figure 9: Source reconstructions for target samples in Dining-Sim environment using Label alignment
g.label Target samples (bottom) and corresponding Source samples (top).
19

---

Under review as a conference paper at ICLR 2021
Figure 10: Appearance alignment e reducing false positive. Top row: source + C.lobel
bottom row:
source ecllabel
Table 4: Quantitative results of Sim2SG on a target domain in Dining- Sim environment.
Trained
Chair AP
Tihle AP
Taptop AP
mAP 80.5 Boll
Recall
Source only
0,842 1:0.379
0.519 1:0
1:0.0881 0881
0.392 20.0511
0,581 10.0486
0.331 1:0.6337
Jubely
+0000044
4.000400
Ours (c S:labei t eppedd
0,770 1:00020
0.757 1:0.0368
0.689 0.00051
0,729 10.0147
0,547 1:000488
Table 5: Dining-Sim ablations. Left (resp. right): Source and target domains have different (resp. similar) appearance but similar (resp. different) content distribution. All the evaluations the target domain.
Trained
mAP 60.5 loU
Recalle:
Trained on
mAP 80,55 loU
Recall
Source only
0.722 4-0.0432
36000663
Source only
0.66 +0.0112
0.468 +60.0663
Source
Source + cilobei
0.737 1.0.243
0.539 1:0.0000000 00.00881
content gap, increasing relationship triplet recall from 0.468 @ to 0.539 The relatively modest improvement makes sense as the two domains still differ in content (source and target domain assets differ).
A.2.3 DRIVE-SIM
Setup As mentioned Section 4.3, we use an Unreal Engine 42 based driving simulator akin to (Prakash et al., 2019) to generate synthetic data. We have cars(1-2 per lane), trees(1-3), houses/buildings((11
pedestrians(O-2), sidewalk(2), roads(2-6). We do not have poles, street signs any other objects. We have straight roads. We use realistic random placements, e.g. only be placed on a lane, pedestrians on sidewalk, houses on ground and trees on both sidewalk and ground. We randomize the time of the day, cloud density and use directional light. We assume real world scale. We place our camera at car height on random right lane with fixed camera parameters (0 yaw, pitch, 90 fov). We add realistic texture and color each object similar (Prakash et al., 2019). We use 1242 x 375 image resolution for training and evaluation.
Details on Synthetic Data generation using Pseudo-Statistics Section 2.2.1 describes how we generate synthetic data using pseudo-statisties We do not have access to KITTI camera parameters and we use the camera parameters described in the previous paragraph. Pseudo statistic derived from SG contains list of objects, their type, 2D centroid position and relationship with others. We filter the objects and relationships among them using a confidence threshold of 0.2. Using camera parameters, place each object in the 3D scene by picking a random 3D asset according its type(class) and assigning random pose in the range -3-360" (except cars that are aligned to the lane). We assume context like road, ground, sky, sidewalk described in the previous paragraph. We refine the 3D scene further according to the predicted relationships among objects. We also assume consistent lane width, and number of roads are determined by positions of the detected vehicles in the scene. We place multiple Trees if the projected 3D volume permits.
Pnttpp: w/wwwe/ww////// ur leng! comcom/
20

---

Under review as a conference paper at ICLR 2021
Table 6: Evaluation three modes of KITTI: easy (top), moderate (middle), hard (bottom) when
training Drive-Sim synthetic environment. The class specific and mAP are reported 0.5 loU.
Trained on
Car
Pedes.
Hloae
Veg.
mAP
Source oully
2019)
+0 OUT
+:0 025
+0. 022
0.177 +0.010
+0.005
0.112 +00 009
DM FasterkCNN (Chen et al. 2018)
0.23 +40 U3B
0.209 4-0 Use
0.003 +40 UST
0.111 +0.012
0271 +0.00 U1.
0.119 4:0 022
Meta-Sim (Kar sl, 2019)
0.555 +0 008
0.227 +0.024
02522 +00 008
0.174 400033
0.307 +0.003
0.143 +0.000
Satf-kamnig (Zon
2018)
0.466 2:0 .09
02115 20.016
0.189 4:0
0.365 40.008
0.244 4:0
0.129 4:0
GHA al., 2020)
00.022
SAPNet (La et al., 2020)
0.420 20 052
0.124 20.035
0018 20 004
0.042 20.010
0.151 20 010.000
(e (a5hae)
0.566
10.0000000
4:0000
10.011
Ours (s c.label 4
0.006 +00021
0.309 +:0003
0.272 +00008
0.250 #:0001
0.362 +00007
0.220 +00010
Ours (e S.lobet, e-preed
1: 033
1: 010.018
1:0 007
1:00015
1:0 005
0.240 1:0 003
1:0
1:0
1:0
1:0
RCNN
0.472 2:0 022
0.181 + 5000031
0.203 +:0 041
0.168 +:0 01.3
0.256 +:0 012
0.091 +:0 019
Meta Sim Self Jearning
0.455 +:0
0.203 +-0.026
0.242 +-0. 009
0.176 +-0.029
0.269 +00.007 007
0.093 +00.005
0.377 1:0.007
0.174 1:0.014
0.197 1:0 0.0002
0.263 1:0.008
0253 1:0 10.004
0.077 1:0 1.0005
SAPNet
0.419 4:0 068
0,98 4:0 4:0.028
0.017 4:0 005
0,088 4:0. 4:0.008 008
0.143 10.007
(e (allbbl,,
20.033
0.273 00.027
0.244 +0.010
+0.036
+0.006
0.128 +: 000
Ours tabel +
0.511 +:0.002 0.000222
0.266 4:0.014
0251 +:0.013
0.256 4:0. 1.0.021
0.221 +:0 0..004
0.155 +00.005
Ours (e c.lasel
sipredd
40.017
::0.015
+00007
+:0 0004
+-0.017
+-00033
1-0.006
DA-FusterCCN
0.424 1:0 028
0.170 1:0 029
0,200 1:0 041
0.169 11: 0144
0.41 1:0 014
0.074 1:0 015
Meta-Sim
0.413 -20 000
0.197 1:0 027
0.236 1:0 009
0.164 1:0 023
0.253 1:0 003
0.075 1:0 005
Self leamning
0312 1:0 :00.008
0.167 4:0.015
4-0.003
0.263 4.0.068
0233 +:0 +.0.004
0.022 1:0
0.174::00
SAPNe
0.362 1:0 054
0.085 1:0 051
0.116 1:0 021
0.067 1:0 1:0.22
0.157 1: 024
c.label
#0.009
0.262
0,240 10.010
0.229 20.036
0,285 -
0.104 + 006
Ours (e Gabbel +
0.493 2: 004
0,252 20.014
0,447 1: 012
0.253 1:0 020
0.311 2: 11
0.127 :: 004
Ours S.label
sipred)
-0 006
20.018
+0.010
+0.014
+0.004
+000000000 0.0004
Figure 11: Synthetic reconstructions for target KITTI samples in Drive-Sim environment using Label alignment Gllabel KITTI samples (right) and corresponding synthetic samples (left)
21

---

Under review as a conference paper at ICLR 2021
50R
Qualitative results of Sim2SG on the target domain for Drive Sim. First column shows that the source only baseline (Prakash et al., 2019) fails to either detect objects or have high number of false positives (mislabels) leading to poor scene graph. Our method detects objects better, has way fewer false positives as shown in second column. Objects are color coded.
Figure 12:
Training Details We optimize the model using SGD optimizer with learning rate of 0.0001 and momentum of 0.9. We train our model using a batch size on NVIDIA DGX workstations. We report saturation peak performance in all our tables. We give equal regularization weights to source task loss Cs. appearance alignment c", prediction alignment co,peed and label alignment co.labei
We experiments in two stages. First stage is training the model using pseudo statistic based self-learning (label alignment c.tobel) for 3 epochs each with 10k iterations. We use the aligned synthetic data from this stage to train the next stage. During second stage we train the model with appearance alignment e and prediction alignment e expred for an additional 60,000 iterations. This makes sense e" works better when content/labels are aligned between the two domains. The total training takes 24 hours including the rendering time.
Baselines: We adapt domain adaptation baselines (Chen et al., 2018; Xu et al., 2020) our framework by using the same backbone (Resnet 101) and SG Predictor (GraphRCNN (Yang et al., 2018)) network as Sim2SG, but their loss function. We do not adapt SAPNet (Li et al., 2020). We train these baselines on 6000 images from the source domain (Prakash et al., 2019) using the same optimizer and learning rate as Sim2SG for 60k iterations. We found GPA (Xu et al., 2020) and SAPNet (Li et al., 2020) detection performance to be lower than that reported in their work especially for pedestrian, vegetation and house classes. is worth noting that their reported class-wise performance numbers only overlap with some of the classes in our work.
We train (Kar et al., 2019) for 40 epochs with a batch size of 16 and learning rate 0.001 as per the authors. We then obtain 6000 images and train it Sim2SG framework (Resnet 101 backbone and GraphRCNN SG predictor) for 60k iterations using the same optimizer and learning rate as Sim2SG. For self-learning based on pseduo labels (Zou al., 2018), we obtain the pseudo labels KITTI images using the most confident predictions by synthetic pretrained GraphRCNN network (as per the authors). We then train these labeled KITTI images on Sim2SG framework for 60k iterations using the same optimizer and learning rate as Sim2SG.
KITTI Annotation We use the existing bounding box annotations of Vehicle and Pedestrians. We annotate Trees and Houses/Buildings of all sizes, occlusion and truncation in KITTI. We use the available camera parameters to project the 2D bounding box into 3D space to help us annotate spatial relationships-front behind, left and right.
Results Full quantitative evaluations results are in Table 6 on all KITTI (Geiger et al., 2012) evalu- ation criteria- easy, moderate and hard. In all three criteria, Sim2SG is able to achieve significantly better results (higher detection mAP @0.5 IoU and relationship triplet recall @ 50) than source only baseline (Prakash al., 2019). More qualitative results of label alignment laabel is in Figure 11. We show qualitative improvements (better object recall and fewer false positive detections) over source only (Prakash et al., 2019) baseline in Figure 12 and the corresponding accurate and full scene graphs in Figure 14.
22

---

Under review as a conference paper at ICLR 2021
Target Source
Figure 13: Samples from source and target distributions of Drive-Sim environment including real images from KITTI
3
0
2
PINUSSLL
KONU
iegrationn
Inpriaionn
Whblle. U
Seet
while
Figure 14: Qualitative results of Sim2SG on the target domain for Drive Sim. Sim2SG generates accurate scene graphs.
23


Reviews:
[{"review_id": "xSZ77KjExK", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper addresses the problem of learning scene graphs from synthetic data and unlabeled real data while performing well on real data by narrowing the content and appearance gap between the two domains when training on synthetic data. Scene graphs are extracted in a two-step process, mapping input to an intermediate latent space and generating the final prediction from the latent space. The authors decompose the content gap into two components: (a) label discrepancy, i.e. how much do the label distributions between the two domains differ, and (b) prediction discrepancy, i.e. the difference in distributions of outputs predicted from the latent space for the two domains. They further model the appearance gap by aligning the latent representation for both domains after accounting for the content gap (to avoid spurious influence of differing content distributions as the latent space is expected to comprise content and appearance). Most of these components are intractable and the paper provides approximations. Empirical investigation on two entirely synthetic and one real/synthetic data set provide evidence for the benefit of the method in closing the posed domain gaps as well as the quality of chosen approximations, the influence of the individual content and appearance gap terms, and the effectiveness of the optimization procedure.\n\n### Strengths:\n[S1] The paper addresses a relevant problem: Learning from unlabeled data can be an important component in scaling vision systems to the real-world.\n\n[S2] Claims, contributions and motivations are laid out clearly, evidenced empirically (see W2 below, however) and the writeup is well organized.\n\n[S3] The presented empirical results demonstrate a significant improvement over the state of the art on a challenging problem.\n\n[S4] The method is complex and there are several approximations, but they are largely motivated well and their impact is empirically investigated.\n\n### Weaknesses:\n[W1] Although a good amount of detail is provided, due to the complexity of the optimization process, reproducibility of the method seems challenging. I consider this a minor weakness due to the complexity of the task itself.\n\n[W2] Overall, the presented empirical evidence is somewhat limited in complexity: (a) type and number of modeled relations in scene graph, e.g. \"car\" is always subject for KITTI, (b) number of nodes types: car, pedestrian, vegetation, house. The authors do acknowledge this and put it in context. Nonetheless, it would strengthen the case for the paper if the authors could elaborate more on their expectations when encountering more challenging visual relationship data, either with more relations, more complex relations or more object types. What would be necessary to make the method work there? What part of the method would need to be changed? Would I need to add something to the model?\n\n[W3] I would like to see more discussion on the limits of the proposed approach: What are the expectations on fundamental limitations, e.g. computationally or in terms of representation?\n\n### Further comments\n[C1] I may have overlooked this: In table 6 I find it curious that the appearance and prediction discrepancy do not improve results for pedestrians, but only for the other three classes. Is this coincidence or are there speculations on why this is?\n\n### Summary\nI believe that methods for efficient transfer learning from synthetic data can have significant impact in a variety of domains in computer vision. As a main weakness of the submission, the presented evidence is largely on toy data or very domain specific (\"autonomous driving\"), however I do feel that there is sufficient contribution and insight. I suggest to the authors to enable reproducibility of the method by the community to allow more experimentation and further improvements.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "oawATQTXnm3", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The proposed paper introduces a novel approach for scene graph generation with a focus on bridging the domain gap between synthetic and real data. The proposed model learns the sim-to-real scene graph generation based on labeled synthetic data and unlabeled real data. The method is evaluated on multiple datasets.\n\nOverall, this is an interesting paper that has everything going for it. The topic this work addresses is important from a computer vision, robotics, and computer graphics perspective, the method is technically sound, the results are promising, and the method is evaluated well. Moreover, the paper is well-written and easy to follow. Therefore, I support accepting this work to ICLR. \n\nComments: \n\n- In Section 2.2 it would help to more carefully define what x and y are. \n- In Section 2.2.1 it is not clear why it comes to label shift. It would help to add another sentences describing it. \n- I enjoyed reading the discussion provided in Sections 2.2.1 and 2.2.2. \n- Currently, the work does not discuss any limitations. As is, it is not clear in which situations the method fails. I would encourage the authors to add a discussion on failure cases to the final version of the paper. \n- I would suggest to tone-down the statements on claiming that scene graph labels in sim are mostly free. This claim does not hold as generating scene graphs for applications, such as games, is often a complex endeavor. Once the scene graphs are defined, they can then be used for automatic label generation, but these labels do not come for free. This should be discussed appropriately.   \n- For completeness sake the related work could also mentioned the following recent paper: \n\nR. Ma, A. Gadi Patil, M. Fisher, M. Li, S. Pirk, B.-S. Hua, S.-K. Yeung, X. Tong, L. Guibas, H. Zhang, Language-Driven Synthesis of 3D Scenes from Scene Databases, ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia), 2018", "extra_scores": {}, "extra_text": {}}, {"review_id": "CYjHG8PYp0P", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThe paper tackles the problem of sim2real transfer for scene graph inference. It proposes an approach for closing the gap between simulated training data and real test data, to allow models trained purely on simulated data to be deployed on real images. The approach is tested on multiple environments, including transfer from a driving scene simulator to real KITTI scenes.\n\n## Strengths\n- the problem of sim2real transfer is important, especially for training scene graph inference models, for which it is very expensive to obtain ground-truth scene graph supervision\n- the paper covers a wide range of related works on scene graph inference and domain adaptation\n- the approach is compared to a range of baseline approaches and improves performance on transfer to KITTI images\n- the paper ablates several components of the proposed approach and shows qualitative prediction results\n\n## Weaknesses\n- **unclear writing**: the writing of introduction and approach section does not clearly express the novel idea presented in the paper. After reading over the paper multiple times, I now understand that the main novelty lies in the usage of pseudo-statistics of the target domain to adjust the source domain data distribution, which is possible since the paper is looking at *sim2real transfer* as a special case of domain transfer in which we have full control of the source domain. In contrast, most of the derivation in section 2.2 was already presented in Wu et al. 2019, except that they assumed to have no control over the label distribution (since they assumed the general domain transfer setting where the source data distribution is fixed too). Based on this, I think the main contribution of the paper should be explained much more clearly (see suggestions below).\n- **unclear focus**: the paper claims two seemingly orthogonal contributions: (1) applying sim2real transfer to the problem of scene graph inference and, (2) introducing an approach for modifying the training data distribution to better match the \"real\" test data distribution specifically for sim2real domain adaptation scenarios. Both problems are valid on their own (even though (1) might not be a sufficient contribution by itself), but the paper fails to make a coherent argument why both should be jointly investigated. Focussing on one of the problems can improve the clarity of the paper.\n- **training data alignment requires domain-specific knowledge**: the core novelty proposed in the paper is the usage of pseudo-statistics to adjust the source data distribution. While the main paper remains unclear *how exactly* the pseudo-statistics are translated to a new training dataset, the appendix has more details that describe how the process is specifically adjusted to every domain. This domain-specific adjustment (like randomly placing trees in the \"permittable\" region in the KITTI training scenes) represents additional domain knowledge which conventional sim2real approaches do not require\n- **baselines not explained in detail**: the experimental section does not properly explain the features of the baselines that the approach is compared to. Without a more detailed explanation it is hard to follow the discussion in the \"Results\" paragraph of section 4.3 that discusses the worse baseline performance. Further, the mentioned dataset sizes in the experimental section are confusing since the proposed method is continually re-generating its synthetic training data which makes it unclear whether the comparison to baselines operating on a fixed dataset is fair.\n- **no analysis experiments**: the experimental section lacks experiments/visualizations that give a more detailed analysis of the novel part of the proposed algorithm. Specifically, it would be nice to see how the approaches changes the training data distribution over the course of training or to analyze the stability of the iterated procedure of training --> data adjustment --> training...\n- **details about derivation unclear**: it seems that crucial parts of the derivation of equation (2) were moved to the appendix without referring to them in the text, which makes it very hard to understand section 2.2, which is the core section for the proposed approach\n\n## Questions\n- the reduction of the prediction discrepancy aims to reduce the gap between the predicted label distributions, but the approach is manually aligning these distributions (by computing the pseudo-statistics), so training the model to accurately fit the training data label distribution should already minimize this discrepancy, so why do we need an additional loss for that? (it anyways seems to be only used in one out of the three experiments, why only there?)\n- the experimental section mentions fixed sizes for the used synthetic datasets, however, the proposed method is iteratively generating new data every epoch -- how is that accounted for in the dataset size? do the baselines get access to comparable amounts of synthetic data?\n\n## Suggestions to improve the paper\n- determine a clear focus for the paper: the most coherent story would be to focus on domain adaptation for sim2real transfer using pseudo-label source content alignment, but then additional experiments on non-scene-graph-inference problems would be required to show that the method is generally applicable.\n- improve writing clarity by clearly mentioning how the investigated problem differentiates from the more general domain adaptation problem, then show (best experimentally) how usual domain adaptation methods struggle with the content gap between source and target environment and how the ability to modify the training distribution in the sim2real setting leads to the proposed method\n- expand on the explanation of the baselines (+test them in all environments) to make it clearer why they fail (particularly since there is no reference work for the sim2real scene graph inference problem)\n- add experiments that analyze the technically novel part of your paper, show how the training distribution changes over time to approach the target distribution\n- if possible, add a baseline that was trained with target, real image labels, particularly in KITTI environment, to show the performance gap of sim2real methods\n- the proposed approach seems like an EM-like procedure: compute pseudo-statistics with the current model + adjust synthetic training data distribution (E-step), then train model on the new data (M-step). Presenting it in this way could make the procedure clearer. It would also be good to discuss potential tradeoffs with prior work (eg is the EM-like procedure prone to instability, do we need any additional assumptions like which quantities to compute pseudo-statistics over and which to ignore?)\n- the caption of table 2 does not mention for which dataset the evaluation is performed (it is mentioned in the text, but it would be good to add it to the caption too)\n\n## Overall Recommendation\nOverall, the lack of clarity in the paper's writing makes it hard to clearly grasp the core problem and delta to prior work, but I believe that some major restructuring and a few additional analysis experiments can substantially improve the clarity and focus of the paper. In its current form it is not ready for acceptance.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "6j7F3EyObRU", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "##########################################################################\n\nSummary: \n\nThis paper introduces a framework to utilize the synthetic data as augmentations in the scene graph generation task, which is able to narrow the domain gap by decomposing it into several discrepancies between the two domains. They are the first to propose the synthetic-to-real transfer learning for SGG. The experimental results show the Sim2SG can improve the baseline models in three different scenarios: CLEVR, Dining-Sim, and Drive-Sim.\n\n##########################################################################\n\nPros:\n\n+ The limitations (long-tailed, noisy, and ambiguous) of the SG dataset have long been noticed in the SGG field. Yet, due to the high costs of collecting the SG dataset, it's hard to provide an ideal dataset for large-scale SGG. This paper introduces a novel idea to augment the SG dataset through synthetic data and transfer learning, which may inspire the later researchers.\n+ The proposed framework successfully combines the pseudo-statistic-based self-learning with the Gradient Reversal Layer to solve the intractable label discrepancy and appearance discrepancy.\n+ It's able to improve the baseline in different scenarios: CLEVR, Dining-Sim, and Drive-Sim.\n\n##########################################################################\n\nConcerns:\n\n-  (Correct me if I'm wrong) The proposed Sim2SG requires a specifically designed synthetic data generator for each different scenario, e.g., if we want to apply it to the general scene graph dataset like VisualGenome, the generator is also required to be capable of generating all kinds of real-life scenarios. It looks like the proposed method can only be applied to the dataset with limited scenarios. \n\n- There are too many approximations in the implementations of the proposed discrepancies. It's okay but lots of details are missing, which makes the audiences hard to follow, e.g., the details of GRL used in appearance discrepancy and prediction discrepancy.\n\n- The three datasets used in this paper only contains several simple relationships: right, left, front, e.t.c. Does it mean the proposed Sim2SG only works on spatial relationships?\n\n\n##########################################################################\n\nReasons for scores:\n\nThis paper introduces an interesting direction to augment the scene graph learning, but the proposed method looks like it only works well on a few limited simple scenarios. If I was wrong, please kindly address my concerns above.\n\n##########################################################################", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Paper addresses the problem of sim2real (training with synthetic data and then applying the  learned model on real data) in the context of scene graph generation.  The paper was reviewed by four expert reviewers  who identified the following pros and cons of  the method. 

> Pros:
- Paper addresses relevant and important problem [R1, R2, R4]
- Paper containing compelling results with respect to a number of baselines [R2, R4]

> Cons:
- Lack of clear motivation, focus, and explanation of novelty [R4]
- Missing details, which makes paper hard to follow [R3, R4]
- Lack of explanations for baselines [R4]
- Lack of focused analysis of specific contributions [R4]
- Lack of discussion on the limitations of the approach [R1, R2]
- Presented evidence is largely on toy data or very domain specific [R2]

A number of the shortcoming were addressed by the authors during the rebuttal through revisions.  However, opinion of reviewers on the paper remained split, with paper receiving  following scores:

- 5: Marginally below acceptance threshold
- 6: Marginally above acceptance threshold
- 7: Good paper, accept
- 5: Marginally below acceptance threshold

Overall, all reviewers and AC agree that the paper addresses an important and interesting problem. At the same time  AC agrees with R2 and others that point out that there are significant limitation in terms of applicability of the approach in more complex scenarios, where readily available simulator may not exist. On balance, and considering the large number of high quality submissions to ICLR this year, the paper was deemed marginally below the acceptance threshold. 

Decision: Reject