Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper propose an end-to-end scene graph generation framework, in which design iterative refinement manner to gradually optimize the predictions. Besides, it also introduce the reweighting loss to tackle the long tail problem in this task. Author verified this method in two popular dataset and achieve the superior performance. But this manuscript is not well-written and some expressions need to be improved.
strengths_and_weaknesses: #Strengths:
This paper reformulate the task of scene graph generation into an iterative optimization process. Based on transformer architecture, author expend the message passing within a Markov Random Field and design iterative refinement procedure. The performance in VG and AG dataset have achieve the state-of-the-art level.
#Weaknesses:
Although an end-to-end framework for generating scene graph is a good research direction, scene graph also involves object prediction. I think the pretrained object detector in two-stage methods will provide more better prediction, which authors did not given relevant introduction. The qualitative refinement analysis is not enough to support the effectiveness of the proposed method.
questions: 1. I have a question about the assumption in line 157.  The [1] had pointed out that the direction of the edge is important property in relationship prediction, but you assume a fixed information flow from subject to object and finally reach predicate. Please give the explaination of eliminating graph property in your assumption.
2. The proposed iterative refinement manner for SGG should involves the variation for both object and relationship, but figure 3 only present the estimation of different refinement steps for relationship.
3. The scaling parameters for reweighting loss is hard to set and your experiment in Table 1 also attempt various combinations, which present a relatively large impact. So, is there a more effective way to set these two parameters? 
4. Author adopt the MOTIF [2] as baseline. As far as I know, this method use the prior information of dataset to bias the final predicted distribution. I would like to know that in your end-to-end framework, this kind of bias is still necessary for the your prediction? 
[1]GPS-Net: Graph Property Sensing Network for Scene Graph Generation. In CVPR 2020.
[2]Neural Motifs: Scene Graph Parsing with Global Context. In CVPR 2018.
limitations: Scene graph is first proposed to use for image retrieval and it is also the more direct application compared with autonomous driving. So author should focus on some simple but effective applications for scene graph.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes an end-to-end paradigm to predict scene graphs from image inputs. The key observation of this paper is that assuming a fixed factorization of subject-object-predicate of predicting relationships can be detrimental, as errors can accumulate in this on direction flow of this information. To alleviate this problem, the authors propose an iterative refinement process, where, although the factorization is still the same within a refinement step, information from previous steps can be utilized by later steps, thus allows information to flow in all directions. A joint matching loss is proposed, using the same matching across all steps, to stabilize this refinement procedure. Comprehensive evaluations show that the proposed method achieves good performances and outperforms many prior works.
strengths_and_weaknesses: ### Strengths
+ The idea of using an iterative refinement procedure to alleviate the issue with a fixed factorization is novel, makes sense intuitively, and seems to work on well in practice.
+ Careful design choices for the aforementioned refinement procedure, including appropriate modification to the inputs of the architecture used (transformers), and a matching loss utilizing the same matching across all steps to ensure the stability of the iterative refinement process.
+ Comprehensive evaluation and good empirical results, with good discussion of some of the nuances over prior works.

### Weaknesses
- Overall, I think there lacks enough evidence supporting the central claim of the paper: that this iterative refinement process is important and avoids the problem with a fixed factorization:
    * It is unclear how the qualitative examples are chosen, and overall too little qualitative examples are shown to support claims that this process fixes the problem with a fixed factorization. More, randomly drawn, examples shown the same qualitative behavior would greatly increase my confidence in the usefulness of this process. I would also expect some results to demonstrate how the proposed method solves issues discussed in the paper e.g. "where an object entailed in interaction is small and not discernible on its own" (L7).
    * Quantitative analysis over the output of each of the refinement step would help to show how the refinement process is actually improving the results. Ablations over the number of steps would also help here. I get that this is somewhat done in Supplementary Table A1, but that can also be due to the models having more parameters overall. I am looking more more fine grained analysis on what specific do the refinement process help the most in improving e.g. if the issue quoted above (L7) is addressed by the proposed method, that can probably be revealed by analysis showing what categories are improved the most over the steps. This currently does not appear to be the case to me since both recall and mean recall follows the same trend in table A1. 
    * The fact that removing both CAS and CWS results in minimal performance drop seems to indicate that there isn't much actual ``refinement" going on.
 - The claim that the proposed method can be used to improve other methods is not that well supported: although the performance is marginally better, it is unclear whether it's simply due to the modified model having more parameters or not.
- Originality of the paper is pretty limited apartment from this (still questionable to me) procedure of refinement. The core of the method is still a decoder in the style if [2] and this has been attempted already in earlier literatures e.g. [30].
- In general, I am not a fan of going into very fine details to show that a method outperforms prior/concurrent works e.g. 14-16, 21-22 in Table 1. Showing that the relative advantages of methods have to come down to such details, to me, is rather unnecessary. A good idea should be able to show its value with more specific evaluation methods / examples in addition to a holistic number about overall performance (see points above).

###Post-rebuttal Comment

I thank the authors for the clarifications regarding my concerns. It seems that I initially missed some evaluations, which did provide decent amount of evidence that there is actually a "refinement" process, and gives some level of intuition behind how this process works. Subsequently, I am raising my score from a 4 to a 6. I would encourage the authors to add more discussions around these points in the main paper, as well as providing more qualitative examples in the supplementary.
questions: As mentioned earlier, my main issue is with the (relatively) lack of analysis of the concrete benefits of the core idea of the paper. More evidence, in addition to "better numbers overall", that it is solving the issues with fixed factorizations will sway me towards accepting this paper.

Some additional comments:
- Might help to highlight the best/top performing entries in the larger tables. Very hard to parse the table as for now.
- Would be interesting to visualize the qualitative behavior of the model without the joint matching loss. I suppose the model is not really doing refinement without that?
- Following previous point: ablations of model with CAS and CWS, but without JL, would be nice to have.
limitations: Limitations are discussed but those are not limitations specific to the central idea of this paper. More discussions on how to improvement the iterative refinement procedure would be more desirable here.

The authors are right in pointing out that the proposed work takes the right step towards alleviating some of the negative social impacts associated with the problem studied here.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposes dynamic conditioning using Markov Random Filed for the Scene Graph generation. It implemented transformer based iterative refinement procedure . Conditional quaries has been used during the refinement and to reduce the exponential search space. Three separate multi layer decoder has been used for relation triplet and perform joint reasoning in a single stage architecture. It achieved sota performance on VG and AG dataset. 
strengths_and_weaknesses: Strength:
1. The paper proposes a single stage Scene Graph generation approach, with novel transformer based formulation which include separate decoder for relational triplet, and conditional query.
2. The paper show that their results improved VG and AG sota.
3.  Tested the approach on both video and images.

Weakness : 
Depite their strength this paper lack few fundamental point :
1. IMHO this paper lacks a strong motivation, author should clearly mention on the introduction why the conditional query is needed and how it can be incorporated to architecture and why they need thee decoder in a one stage..the story should be connected ...
2. What is the initial query, is it same like DETR or how do u process or the initial query are only subject queries ? (sec 4.2)
3. It has three separate layer of decoder and we know that scene graph has exponential complexity...author should analyiz dat

Acknowledgement :
It would be better if you acknowledge  other single stage paper, like Relationformer - Shit at el, RelTr - Cong at el.. and other earlier  transformer based paper like Seq2Seq- Lu at el, Relation Transformer Network - Koner at el

Suggention :
The paper has technical novelty, please write in a way such that it would be easy to follow and understand your contribution

questions: 1. How you have incorporated the method in video, using some tracking? please mention
2. Please provide clear motivation for three decoders and why it is necessary despite their complexity 


limitations: yes.
review_title: null

Meta-review: metareview: The authors propose a new approach for end-to-end training of predicting scene graphs from images (different from the traditional two-stage approach.) The key observation that the fixed factorization approach can be suboptimal due to error compounding is reasonable and is supported by the experiment results. The proposed solution with iterative refinement is reasonable and the design choices of the method are sound. Evaluation is comprehensive overall and the result is convincing. 

Most of the key concerns raised by reviewers Vh2W and 5s37 who gave low scores (3 & 4) seem to be well addressed by the authors but the reviewers were not responsive and engaged in the follow-up discussion, so also no score update as well. However, the other reviewer h8U6 expressed that the authors addressed the concerns well and after reading the paper I agree on this based on my personal assessment. Therefore, even if two reviewers gave a rather low score 3 and 4, I cannot weigh those review scores much and rather rely more on the other reviewers giving 6 and my own assessment. So, I recommend accepting this paper even if the average score is rather lower than usual.
confidence: Certain
Decision: Accept