Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a conditional random field (CRF) based joint modeling of objects and relations for the task of scene graph generation. Unlike previous relation discovery methods, this paper employs embedding-based relation feature representation, and thus enables an unified modeling of the unitary and clique potential functions. Using MCMC sampler and mean-field variational inference, the proposed JM-SGG can effectively update the relation triplets utilizing the contextual label dependency. The experiments show that JM-SGG has good performances on the relationship retrieval task, even under the zero-shot setting.
main_review: Overall, this study is helpful to researchers in the fields of visual understanding, especially scene graph generation. It is happy to see how the probabilistic graph model is successfully integrated into scene graph generation. This paper is well written, clearly illustrated, and appropriately structured. The experimental results also validate the claims.

But in my opinion, conditioned random field and mean-field variational inference are not at the first time applied into scene graph generation. For example, Cong et al. 2018, Scene Graph Generation via Conditional Random Fields has introduced some important techniques used in this paper. Even though this paper was just `published' in arxiv, it would be appreciated to discuss their technical connections, and how the proposed method outperforms.

Another concern lies in the experiments. At first, why not in addition report the results by non-graph constraint? Each ordered object pair should contain more than one relation if the relation label cannot convincingly exclude each other. For example, <person, has, jean> is not wrong, even though <person, wearing, jean> may be a more precise description. It would be nice if the proposed JM-SGG can capture the most possible relations for a particular object pair.
limitations_and_societal_impact: Yes, the authors have discussed the limitations and potential negative societal impact of this work in the section of the conclusion, and in the supplementary material.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a joint prediction of the entire scene graph that fully captures the dependency among different objects and relations using a unified conditional random field.  The proposed model can be summarized as starting from joint modeling of the object and relation component label and visual features. Joint modeling uses CRF to model comprehensive dependency of the object using unnormalized likelihood and partition function. In addition, the potential function used for object and relation components computed affinity through a distance-based learnable prototype. Finally, the knowledge graph embedding techniques for projection of different embedding to the same embedding space (e.g. context space to relation space, object space to relation space) have been used.

All the learnable parameters are trained with maximum log-likelihood function and the partition function uses a mean-field variational inference for efficient sampling. Mean-field inference initialized with independent relation labels for triplet component factor update.

Authors have conducted extensive experiments on the Visual Genome dataset and a brief ablation study that provide a good insight into the method. 
main_review: The main idea of the paper for joint modeling of object and relation using CRF and mean-field variance algorithm is novel and significant in the SGG task. The paper is well written, although the introduction and description of the long list of components hindrance its readability. The combination of CRF and knowledge graph technique will lead to exploration of commonality between scene and knowledge graph, thus making it a relevant work in this area.


limitations_and_societal_impact: There are a few limitations of the paper : 
1. Authors should conduct more experiments on relatable datasets like GQA or Open Image.
2. Ablation focuses on the joint modeling, factor update, initialization, etc, where the paper assumes  CRM and prototypical embedding helped the network. Ablation should provide the impact of these components.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a new method for supervised scene graph generation from images. The new method models dependencies between objects and their relations as a conditional random field (CRF), conditioned on the output of an object detector. To facilitate inference in the GRF, the deterministic prediction by the object detector is used to initialize a mean field approximation to the posterior. This is then updated iteratively via message passing. The method is shown to yield improved performance in relationship retrieval on the Visual Genome dataset.
main_review: 
### Method
The intuition behind the proposed method is clear, and the design choices made in its implementation are plausible. The theoretical analysis is correct as far as I can tell. The use of the mean field approximation is a limitation, but one that is reasonable and also discussed in the paper.

### Originality
The introduction and related work section claims that the proposed method is the first to model dependencies across the entire scene graph. I don't believe that this is the case: At the least, Suhail et al. [33] propose an energy based model that uses a graph neural network to assign energy values to scene graphs, which will be able to model arbitrary dependencies between the nodes. They also use a similar inference procedure, by first obtaining an initial scene graph via a deterministic predictor, and then iteratively updating it to maximize the likelihood (energy) under the probabilistic model. While the proposed method may be the first to model dependencies via a traditional graphical model like a CRF, and there may be some benefits to that, such a claim would need to be discussed and supported by evidence.

### Empirical Evaluation
The model is evaluated in the standard setting of relationship retrieval on the Visual Genome dataset, with Table 1 indicating that the proposed methods performs almost universally better than previous methods. However, given that the authors report baseline numbers for VCTree, VCTree-TDE, and VCTree with the EBM loss from [33], I am very confused that they chose to not also report the results for VCTree-TDE with EBM loss [33]. This appears to be the current state of the art on this benchmark, and also beats the proposed method on most of the reported metrics. Am I missing any reason why this wouldn't be a fair comparison?

More generally, the robustness of the evaluation could be improved by including another dataset, such as GQA.


### Clarity
Overall, the proposed method is explained in a clear and reproducible way.
The paper could benefit from another editing pass, e.g., the caption of Fig. 2(c) is broken.
The readability of some of the math, especially formulas (12) and (13) could be improved, e.g., by avoiding nested subscripts such as $y_{o_h}$, and
merging summation signs.

### Summary
Overall, the paper proposes a plausible method, but the discussion of related work and the empirical evaluation raise questions. I am open to increasing my score if these can be addressed during the response phase.

### Update
In their response, the authors have agreed to remove the claim that they are the first to jointly model scene graphs. They have also demonstrated that their contribution is orthogonal to the unbiased prediction techniques employed in VCTree-TDE, and that the two can be combined to achieve even better performance. This addresses my two main concerns. I have therefore raised my score from 5 to 6.
limitations_and_societal_impact: Limitations and societal impact are adequately addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: * The paper proposes a method for generating scene graphs of images by modeling the objects and relationships with a Conditional Random Field. The paper claims that other existing methods do not model the dependency between all the objects and relations in an image. The proposed method is meant for modeling all the relations jointly which could lead to better performance in generating scene graphs.
* The potential function for object components is defined using the representation obtained from an object detector network. The potential function for relationship components is defined using the knowledge graph embedding method which represents both the objects and relationships in the same embedding space.
* A mean-field approximation of the probability distribution of the CRF is obtained by treating the labels for objects and relations as independent of each other. A message passing algorithm is used to iteratively update the factors and perform inference with the mean-field approximation. This is also helpful for sampling the scene graphs during the learning phase.
* The experiments are performed on the VisualGenome dataset and results are shown for a few different tasks like Relationship Predicate Classification, Scene Graph Classification and Generation. The paper also includes results on the task of zero shot relationship retrieval that evaluates the model’s ability to identify head-relation-tail combinations not observed during training. 

main_review: I have a few concerns regarding the Originality, Quality and Clarity. The concerns are marked with the [-ve] prefix.

Originality
* [-ve] The claim that other works don’t model all the relationships in an image is not true. The idea of using factors for relationships and aggregating information over all the relationships using an iterative message passing algorithm has been used by W.Cong et al. [4]. The Algorithm 1 in [4] has a message passing step that gathers information from the neighbors of a node in a graph, thus aggregating information from the entire graph over the iterations. The aggregation method is not just special in [4], it is also a pretty standard way to do inference over graphs.
* [-ve] The paper claims to propose a novel inference algorithm. However, it is not clear what aspect of the inference algorithm is novel when compared to that of existing mean-field approximation for CRF models. This inference algorithm has been used for CRFs in many prior computer vision works. Ex: Krahenbuhl and Kolton, NeurIPS 2011.

Quality
* [-ve] The results are better than some of the state of the art methods. While the paper claims that this is due to joint scene graph modeling (line 337), I think a more thorough analysis would be required to justify the claim. The effect of the proposed relationship potential function that is different from other works is not fully measured.
* The definitions of the potential functions seem to be the main distinguishing contribution when compared to works like Cong et al. [4]. The work in [4] uses TransE (Bordes et al. 2013) whereas this paper uses TransR [19]. Could this be the one of the reasons for the improved results? This could be verified with an ablation experiment where TransR is substituted with TransE. I am suggesting this experiment because the results from the ablation settings which don’t perform full scene graph modeling (triplet-only and without factor update) are already better in some settings when compared to the other methods.
* The results from the model without factor update is better than the triplet only model which does one step of factor update. This is an interesting result which needs some explanation. Does this imply that the results can be noisy if we don’t perform full scene graph modeling? It will be good to mention this explicitly which adds to the motivation of doing full scene graph modeling.
* The experiments section provides a good sensitivity analysis showing how the results converge with increasing number of factor update iterations.
* The code is provided for reproducing the results

Clarity
* [-ve] The paper is written well to provide a clear explanation of the method along with the proofs of the mathematical lemmas. However, the related work section is lacking since it doesn’t mention how the proposed method is different and/or better compared to existing works.

Significance
* The results are better than the state of the art methods in many settings.
* The work is useful for the community as an example for an alternative method to perform scene generation. While the state of the art methods like Dhingra et al., CVPR 2021 show much better performance on the same tasks using Transformers, the paper proposed a viable method that uses CRFs. It is beneficial to the community since it answers the question “How would CRF models perform on the task of scene generation?”

Other suggestions
* The experiments use VGG16 as the backbone for comparison with other methods. However, it will be good to include additional results with Resnet as the backbone since the latest methods have shown improvements when switching from VGG16 to Resnet. Ex: Dhingra et al., CVPR 2021.

limitations_and_societal_impact: Yes, the authors have provided a short discussion on the limitations and any societal impact the work might have.
review_title: null

Meta-review: Thank you for submitting your work to NeurIPS. The paper introduces a supervised approach to scene graph generation from images: objects and relations among them are captured using a CRF conditioned on a (deep) object detector and using e.g. distance-based learnable prototype. Overall, the rolling discussion helped to clarify many of issues raised by the reviewers. This is a solid but also somewhat incremental paper that, and this is a big advantage of the paper, shows the value of hybrid methods for challenging AI tasks. In any case, please incorporate your feedback from the rolling discussion into the final version. 
Decision: Accept (Poster)