Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: ## Summary
The paper tackles the problem of sim2real transfer for scene graph inference. It proposes an approach for closing the gap between simulated training data and real test data, to allow models trained purely on simulated data to be deployed on real images. The approach is tested on multiple environments, including transfer from a driving scene simulator to real KITTI scenes.

## Strengths
- the problem of sim2real transfer is important, especially for training scene graph inference models, for which it is very expensive to obtain ground-truth scene graph supervision
- the paper covers a wide range of related works on scene graph inference and domain adaptation
- the approach is compared to a range of baseline approaches and improves performance on transfer to KITTI images
- the paper ablates several components of the proposed approach and shows qualitative prediction results

## Weaknesses
- **unclear writing**: the writing of introduction and approach section does not clearly express the novel idea presented in the paper. After reading over the paper multiple times, I now understand that the main novelty lies in the usage of pseudo-statistics of the target domain to adjust the source domain data distribution, which is possible since the paper is looking at *sim2real transfer* as a special case of domain transfer in which we have full control of the source domain. In contrast, most of the derivation in section 2.2 was already presented in Wu et al. 2019, except that they assumed to have no control over the label distribution (since they assumed the general domain transfer setting where the source data distribution is fixed too). Based on this, I think the main contribution of the paper should be explained much more clearly (see suggestions below).
- **unclear focus**: the paper claims two seemingly orthogonal contributions: (1) applying sim2real transfer to the problem of scene graph inference and, (2) introducing an approach for modifying the training data distribution to better match the "real" test data distribution specifically for sim2real domain adaptation scenarios. Both problems are valid on their own (even though (1) might not be a sufficient contribution by itself), but the paper fails to make a coherent argument why both should be jointly investigated. Focussing on one of the problems can improve the clarity of the paper.
- **training data alignment requires domain-specific knowledge**: the core novelty proposed in the paper is the usage of pseudo-statistics to adjust the source data distribution. While the main paper remains unclear *how exactly* the pseudo-statistics are translated to a new training dataset, the appendix has more details that describe how the process is specifically adjusted to every domain. This domain-specific adjustment (like randomly placing trees in the "permittable" region in the KITTI training scenes) represents additional domain knowledge which conventional sim2real approaches do not require
- **baselines not explained in detail**: the experimental section does not properly explain the features of the baselines that the approach is compared to. Without a more detailed explanation it is hard to follow the discussion in the "Results" paragraph of section 4.3 that discusses the worse baseline performance. Further, the mentioned dataset sizes in the experimental section are confusing since the proposed method is continually re-generating its synthetic training data which makes it unclear whether the comparison to baselines operating on a fixed dataset is fair.
- **no analysis experiments**: the experimental section lacks experiments/visualizations that give a more detailed analysis of the novel part of the proposed algorithm. Specifically, it would be nice to see how the approaches changes the training data distribution over the course of training or to analyze the stability of the iterated procedure of training --> data adjustment --> training...
- **details about derivation unclear**: it seems that crucial parts of the derivation of equation (2) were moved to the appendix without referring to them in the text, which makes it very hard to understand section 2.2, which is the core section for the proposed approach

## Questions
- the reduction of the prediction discrepancy aims to reduce the gap between the predicted label distributions, but the approach is manually aligning these distributions (by computing the pseudo-statistics), so training the model to accurately fit the training data label distribution should already minimize this discrepancy, so why do we need an additional loss for that? (it anyways seems to be only used in one out of the three experiments, why only there?)
- the experimental section mentions fixed sizes for the used synthetic datasets, however, the proposed method is iteratively generating new data every epoch -- how is that accounted for in the dataset size? do the baselines get access to comparable amounts of synthetic data?

## Suggestions to improve the paper
- determine a clear focus for the paper: the most coherent story would be to focus on domain adaptation for sim2real transfer using pseudo-label source content alignment, but then additional experiments on non-scene-graph-inference problems would be required to show that the method is generally applicable.
- improve writing clarity by clearly mentioning how the investigated problem differentiates from the more general domain adaptation problem, then show (best experimentally) how usual domain adaptation methods struggle with the content gap between source and target environment and how the ability to modify the training distribution in the sim2real setting leads to the proposed method
- expand on the explanation of the baselines (+test them in all environments) to make it clearer why they fail (particularly since there is no reference work for the sim2real scene graph inference problem)
- add experiments that analyze the technically novel part of your paper, show how the training distribution changes over time to approach the target distribution
- if possible, add a baseline that was trained with target, real image labels, particularly in KITTI environment, to show the performance gap of sim2real methods
- the proposed approach seems like an EM-like procedure: compute pseudo-statistics with the current model + adjust synthetic training data distribution (E-step), then train model on the new data (M-step). Presenting it in this way could make the procedure clearer. It would also be good to discuss potential tradeoffs with prior work (eg is the EM-like procedure prone to instability, do we need any additional assumptions like which quantities to compute pseudo-statistics over and which to ignore?)
- the caption of table 2 does not mention for which dataset the evaluation is performed (it is mentioned in the text, but it would be good to add it to the caption too)

## Overall Recommendation
Overall, the lack of clarity in the paper's writing makes it hard to clearly grasp the core problem and delta to prior work, but I believe that some major restructuring and a few additional analysis experiments can substantially improve the clarity and focus of the paper. In its current form it is not ready for acceptance.

review_title: Review

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper addresses the problem of learning scene graphs from synthetic data and unlabeled real data while performing well on real data by narrowing the content and appearance gap between the two domains when training on synthetic data. Scene graphs are extracted in a two-step process, mapping input to an intermediate latent space and generating the final prediction from the latent space. The authors decompose the content gap into two components: (a) label discrepancy, i.e. how much do the label distributions between the two domains differ, and (b) prediction discrepancy, i.e. the difference in distributions of outputs predicted from the latent space for the two domains. They further model the appearance gap by aligning the latent representation for both domains after accounting for the content gap (to avoid spurious influence of differing content distributions as the latent space is expected to comprise content and appearance). Most of these components are intractable and the paper provides approximations. Empirical investigation on two entirely synthetic and one real/synthetic data set provide evidence for the benefit of the method in closing the posed domain gaps as well as the quality of chosen approximations, the influence of the individual content and appearance gap terms, and the effectiveness of the optimization procedure.

### Strengths:
[S1] The paper addresses a relevant problem: Learning from unlabeled data can be an important component in scaling vision systems to the real-world.

[S2] Claims, contributions and motivations are laid out clearly, evidenced empirically (see W2 below, however) and the writeup is well organized.

[S3] The presented empirical results demonstrate a significant improvement over the state of the art on a challenging problem.

[S4] The method is complex and there are several approximations, but they are largely motivated well and their impact is empirically investigated.

### Weaknesses:
[W1] Although a good amount of detail is provided, due to the complexity of the optimization process, reproducibility of the method seems challenging. I consider this a minor weakness due to the complexity of the task itself.

[W2] Overall, the presented empirical evidence is somewhat limited in complexity: (a) type and number of modeled relations in scene graph, e.g. "car" is always subject for KITTI, (b) number of nodes types: car, pedestrian, vegetation, house. The authors do acknowledge this and put it in context. Nonetheless, it would strengthen the case for the paper if the authors could elaborate more on their expectations when encountering more challenging visual relationship data, either with more relations, more complex relations or more object types. What would be necessary to make the method work there? What part of the method would need to be changed? Would I need to add something to the model?

[W3] I would like to see more discussion on the limits of the proposed approach: What are the expectations on fundamental limitations, e.g. computationally or in terms of representation?

### Further comments
[C1] I may have overlooked this: In table 6 I find it curious that the appearance and prediction discrepancy do not improve results for pedestrians, but only for the other three classes. Is this coincidence or are there speculations on why this is?

### Summary
I believe that methods for efficient transfer learning from synthetic data can have significant impact in a variety of domains in computer vision. As a main weakness of the submission, the presented evidence is largely on toy data or very domain specific ("autonomous driving"), however I do feel that there is sufficient contribution and insight. I suggest to the authors to enable reproducibility of the method by the community to allow more experimentation and further improvements.

review_title: Complex method, addresses important problem, some concerns on limited empirical evaluation

final_score: rating: 7: Good paper, accept
review_content: The proposed paper introduces a novel approach for scene graph generation with a focus on bridging the domain gap between synthetic and real data. The proposed model learns the sim-to-real scene graph generation based on labeled synthetic data and unlabeled real data. The method is evaluated on multiple datasets.

Overall, this is an interesting paper that has everything going for it. The topic this work addresses is important from a computer vision, robotics, and computer graphics perspective, the method is technically sound, the results are promising, and the method is evaluated well. Moreover, the paper is well-written and easy to follow. Therefore, I support accepting this work to ICLR. 

Comments: 

- In Section 2.2 it would help to more carefully define what x and y are. 
- In Section 2.2.1 it is not clear why it comes to label shift. It would help to add another sentences describing it. 
- I enjoyed reading the discussion provided in Sections 2.2.1 and 2.2.2. 
- Currently, the work does not discuss any limitations. As is, it is not clear in which situations the method fails. I would encourage the authors to add a discussion on failure cases to the final version of the paper. 
- I would suggest to tone-down the statements on claiming that scene graph labels in sim are mostly free. This claim does not hold as generating scene graphs for applications, such as games, is often a complex endeavor. Once the scene graphs are defined, they can then be used for automatic label generation, but these labels do not come for free. This should be discussed appropriately.   
- For completeness sake the related work could also mentioned the following recent paper: 

R. Ma, A. Gadi Patil, M. Fisher, M. Li, S. Pirk, B.-S. Hua, S.-K. Yeung, X. Tong, L. Guibas, H. Zhang, Language-Driven Synthesis of 3D Scenes from Scene Databases, ACM Transactions on Graphics (Proceedings of SIGGRAPH Asia), 2018
review_title: Interesting topic, method technically sound, easy to follow 

final_score: rating: 5: Marginally below acceptance threshold
review_content: ##########################################################################

Summary: 

This paper introduces a framework to utilize the synthetic data as augmentations in the scene graph generation task, which is able to narrow the domain gap by decomposing it into several discrepancies between the two domains. They are the first to propose the synthetic-to-real transfer learning for SGG. The experimental results show the Sim2SG can improve the baseline models in three different scenarios: CLEVR, Dining-Sim, and Drive-Sim.

##########################################################################

Pros:

+ The limitations (long-tailed, noisy, and ambiguous) of the SG dataset have long been noticed in the SGG field. Yet, due to the high costs of collecting the SG dataset, it's hard to provide an ideal dataset for large-scale SGG. This paper introduces a novel idea to augment the SG dataset through synthetic data and transfer learning, which may inspire the later researchers.
+ The proposed framework successfully combines the pseudo-statistic-based self-learning with the Gradient Reversal Layer to solve the intractable label discrepancy and appearance discrepancy.
+ It's able to improve the baseline in different scenarios: CLEVR, Dining-Sim, and Drive-Sim.

##########################################################################

Concerns:

-  (Correct me if I'm wrong) The proposed Sim2SG requires a specifically designed synthetic data generator for each different scenario, e.g., if we want to apply it to the general scene graph dataset like VisualGenome, the generator is also required to be capable of generating all kinds of real-life scenarios. It looks like the proposed method can only be applied to the dataset with limited scenarios. 

- There are too many approximations in the implementations of the proposed discrepancies. It's okay but lots of details are missing, which makes the audiences hard to follow, e.g., the details of GRL used in appearance discrepancy and prediction discrepancy.

- The three datasets used in this paper only contains several simple relationships: right, left, front, e.t.c. Does it mean the proposed Sim2SG only works on spatial relationships?


##########################################################################

Reasons for scores:

This paper introduces an interesting direction to augment the scene graph learning, but the proposed method looks like it only works well on a few limited simple scenarios. If I was wrong, please kindly address my concerns above.

##########################################################################
review_title: This is an interesting work yet (seems to be) have lots of limitations.

Meta-review: Paper addresses the problem of sim2real (training with synthetic data and then applying the  learned model on real data) in the context of scene graph generation.  The paper was reviewed by four expert reviewers  who identified the following pros and cons of  the method. 

> Pros:
- Paper addresses relevant and important problem [R1, R2, R4]
- Paper containing compelling results with respect to a number of baselines [R2, R4]

> Cons:
- Lack of clear motivation, focus, and explanation of novelty [R4]
- Missing details, which makes paper hard to follow [R3, R4]
- Lack of explanations for baselines [R4]
- Lack of focused analysis of specific contributions [R4]
- Lack of discussion on the limitations of the approach [R1, R2]
- Presented evidence is largely on toy data or very domain specific [R2]

A number of the shortcoming were addressed by the authors during the rebuttal through revisions.  However, opinion of reviewers on the paper remained split, with paper receiving  following scores:

- 5: Marginally below acceptance threshold
- 6: Marginally above acceptance threshold
- 7: Good paper, accept
- 5: Marginally below acceptance threshold

Overall, all reviewers and AC agree that the paper addresses an important and interesting problem. At the same time  AC agrees with R2 and others that point out that there are significant limitation in terms of applicability of the approach in more complex scenarios, where readily available simulator may not exist. On balance, and considering the large number of high quality submissions to ICLR this year, the paper was deemed marginally below the acceptance threshold. 

Decision: Reject