Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper analyses design choices of one-stage architectures for the task of Scene Graph Generation (SGG) - specifically, the degrees of feature entanglement for predicting predicates (relations) and entities (objects). An observation is made in Fig 1 and Tab 1, that complete coupling of features for predicate and entity classification leads to lower performance, while on the other hand complete decoupling gives better performance but it’s undesirable since it leads to overhead in the form of disjoint queries. The authors propose a mechanism to condition queries for predicting entities with the predicates, which leads to a desired trade-off between completely decoupled and coupled approaches. The authors show improved performance over its counterparts.
strengths_and_weaknesses: ## Strengths
- The approach is well-motivated and is communicated clearly

## Weaknesses

- Table 1
    - I couldn’t find the result of TracCQ  here. Is it the same as the last row of Table 2? 
    - If yes
        - then is it fair to say that DDTR already outperforms the best performing single-stage approach in Table 2?
        - Also what changes resulted in better performance of DDTR baseline compared to the prior single-stage approaches?
    - Line 294-295: “… It also exhibits a much less drastic drop in the mean recall values when we go from 100 to 20 for mR@K …”
        - In fact, mR@20 scores in Table 1 for SD, DD and DDTR are either comparable or better than the best-performing single-stage approaches. Do the authors have an opinion about this? Are there any specific biases in the paper’s approach that accounts for this behavior?
- Table 3: Correction v/s no-correction
    - The authors mentioned in line 183 that the predicate decoder ($\mathcal{H}$) “… is trained so as to not be penalized for inexact bounding of the subject and object entities. …”
    - But the loss ($\mathcal{L}_p$) in equation 12 includes the term for box loss with no modification for any leeway for box predictions or be penalized lower than what is suggested
    - Moreover, the ablation in Table 3 seems to suggest that when the refinement of boxes is turned off (row 1), the performance doesn’t get affected by a lot and in fact shows better results for mR @ 20. Additionally this also shows up in the visualization of refined v/s original boxes in Supplementary Fig 2, which seems to suggest that no too much correction is happening
    - Does this suggest that the predicate decoder is also predicting boxes as accurately as the final model? Can the authors quantify the correction?
    - Do the authors have an ablation for a variant of the model where the entity refinement decoder only predicts the class of the objects, to really see whether the box refinement is crucial?


### Overall comment
- I like the idea of reducing the search space of entity refinement with the help of predicted predicates, and the fact that preliminary investigation in the form of Table 1 supports the intuition that the authors build on, but the experimental results and ablation could be made stronger and discussed more to substantiate the authors’ claim (given that most of the main paper prioritizes discussing the approach)
questions: See "Weaknesses" above
limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper presents a single-stage visual relationship detection method based on a new design of CNN-Transformer network architecture. The main idea is to first compute a shared representation based on the CNN-Transformer encoder, and then to decouple the visual relationship detection into two sequential subtasks: 1) Prediction of the entity-pair location and the predicate class based on a predicate decoder;  and 2) Refinement of the entity locations and prediction their classes given the predicate prediction based on a conditional entity refinement decoder.  A multi-task loss based on Hungarian matching is designed to train two decoders separately, each focusing on its corresponding subtask.  The proposed method is evaluated on the VG150 benchmark with comparisons to the prior methods. 
strengths_and_weaknesses: Strengths:
+ The paper presents an efficient single-stage visual relationship detection strategy based on a novel decomposition of the triplet detection task.  The resulting two sequential subtasks have a smaller search space, which simplifies the overall detection task. 
+ The proposed method achieves strong results on the VG dataset, outperforming the prior SOTA with more compact network architecture. 

Weaknesses:
1) Potential inconsistency in scene graph generation. While the proposed method can produce a set of visual relationship predictions, they are generated conditioned on the predicate estimation. In particular, the locations of entities in each visual relation are refined independently. As a result, two neighboring visual relations (sharing one entity), or two visual relations with the same sub/obj entities can produce two different localizations for the shared entity.
2) Some of the motivation lacks clarity.  
+ What is the feature $f_i$ in Line 44? How do "the entity-scene interactions characteristic of each predicate" instantiate in the models described in this paper? 
+ Why the matching in the DDTR is computationally expensive? (Line 143) 
+ Why the task $Pr(\mathcal{P}|\mathcal{I})$ is a simpler subtask? (Line 241) The variation of the predicates seems to be much larger than entities and they also have more severe long-tail problems. 
3) The experimental evaluation is a bit lacking in the following aspects:
+ The paper adopts the PnP DETR as the transformer architecture. It is unclear if this backbone introduces additional benefits in performance, such as entity detection quality. 
+ The paper lacks a comparison in time complexity for model inference. 
+ In the ablation on the entity refinement module, why use the outputs from the predicate decoder? The details of the evaluation on the task PredDet seem to be missing.  
   
questions: 1. How does the method cope with the potential inconsistency in the SSG prediction?
2. What is the contribution of PnP DETR to the performance improvement? 
3. What is the time complexity of the proposed method in comparison to the other baselines?
 
limitations: The discussion on the limitations is included in the appendix.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The authors tackle the task of scene graph generation by proposing a new formulation which models the problem as a product of two conditionals - the first of which generates a bounding box estimate and a predicate label, and the second takes the aforementioned estimates to get subject and object class labels and refined bounding boxes. The two distributions are realized via a DETR style encoder-decoder transformer architecture. The experimental analysis shows improved performance compared to the baselines on the Visual Genome dataset. 
strengths_and_weaknesses: $$\large{\textbf{Strengths}}$$

1. The paper is well written and easy to follow. The motivation is well explained and Figure 2 provides a clear overview of the approach. 
2. The factorization assumed by the paper (Equation 14) is novel compared to existing work which often condition the predicate prediction on the subject/objects (Equation 8). This formulation provides interesting opportunities for future research.

$$\large{\textbf{Weaknesses}}$$

1. The claims learning in the non-combinatorial predicate space is easier than the $O(\varepsilon \times \varepsilon)$ entity pair space due to the former being smaller. This is supported by analysis in Section 4 (Comparision to Previous Approaches subsection). Firstly, the analysis makes the assumption that labels $\mathcal{Y}, \mathcal{P}$ are independent of bounding boxes $\mathcal{B}$ to simplify the factorization in Equation 14. This assumption clearly doesn't hold in the context of scene graph generation as bounding boxes are crucial to localizing where in the image a particular interaction is happening, and between what objects the aforementioned interaction occurs. Therefore, the simplification of the factorization in Equation 14 under this assumption is misleading. Secondly, looking at Equation 14 without the simplification, it is unclear why the term $\Pr(\mathcal{Y}, \mathcal{B} | \mathcal{P}, \hat{\mathcal{B}}, \mathcal{I})$ should be better to learn from. Even though it is conditioned on a smaller space as compared to the term $\Pr(\mathcal{P}| \mathcal{B}, \mathcal{Y}, \mathcal{I})$ in Equation 8, the proposed formulation is still predicting in a much larger space consisting of entities labels $\mathcal{Y}$ and bounding boxes $ \mathcal{B}$  (which are continuous in nature). 

2. The analysis in Table 1 shows the model performance to improve when increasing the amount of disentanglement. However, this performance improvement could be a direct result of increasing the model capacity. For example, going from SD to DD would entail an additional decoder (which I'm assuming is 6 layers deep based on the experiments), which could provide the observed gains as the model has a greater capacity to fit the training data. As the authors haven't discussed the hyperparameters for the three models shown, disentanglement cannot solely be considered as the cause for the observed improvements.

3. Building on the previous point, Table 2 in the supplementary (ablation on the hyperparameter k) shows a considerable improvement in performance when going from k=1 to k=5, as the number of predictions generated by the model increasing from 200 to 1000. The reported performance gains of the proposed TraCQ to the baselines in Table 1 of the supplementary, and to other single stage transformer models (like [25]) in Table 2 of the main paper are misleading due to the increased number of predictions made by the model. This is majorly due to the recall metrics not checking for prediction precision. The authors have not mentioned the number of queries used by the models in Table 1. If, for example, it is 200, then the proposed TraCQ model would be worse than the DDTR model on mR@20 assuming the same number of predictions (200).

4. From the ablation in Table 3, the refinement module $\mathcal{C}$ doesn't impact the overall model performance as much. Looking at the numbers, it reduces performance on mR@20, which is a stricter and more desirable metric compared to mR@50/100. The visualizations in Figure 2 of the supplementary does not show the change in the bounding box before and after refinement, making it difficult to analyze the importance of the proposed module $\mathcal{C}$. As the module $\mathcal{H}$ already predicts the predicate class and provides a reasonable estimate of the object bounding boxes, one could easily add another couple detection heads to $\mathcal{H}$ to predict the entity class labels as well. This would do away with the entity refinement module (which has almost the same number of parameters as $\mathcal{H}$). The authors should provide an ablation / address why this simpler architecture is worse than the proposed TraCQ. 

5. The authors mention that the initialize the network with parameters pre-trained on VG object detection task. Doing so already biases the predicate decoder $\mathcal{H}$ to be aware of the entity labels (as the object detector was trained to predict both bounding boxes and labels). It is unclear what effect this initialization has on the overall performance, and whether learning $\mathcal{H}$ would be much harder if the weights were initialized randomly.

6. There are some missing ablations with regards to the what effect do the parameters $N_{ce}, N_p$ have on the performance. Additionally, the authors should also report the zero-shot recall performance (introduced in [a]) that computes the recall@K for triplets not present in the training data. It would highlight the generalizability of the proposed approach. 

[a] Lu, Cewu, et al. "Visual relationship detection with language priors." European conference on computer vision. Springer, Cham, 2016.
questions: Although I like the novel formulation resulting in a unique factorization of the SGG problem (Equation 14), there are certain issues with experiments and the described analysis that makes it harder for me to evaluate the proposed approach. 

1. It would be ideal if the authors could further clarify the analysis in Table 1. More specifically, what are the hyperparameters used by the different baseline models and whether they have the same number of parameters. If they don't, why is the performance improvement from SD to DD to DDTR not a direct consequence of improved model capacity. Additionally, when comparing TraCQ to the baselines, the performance under the same number of predictions should be shown. (See points 2 and 3 in weaknesses)

2. The authors should clarify the importance of the refinement module $\mathcal{C}$. Why is the refinement module $\mathcal{C}$ necessary? Why is the predicate decoder $\mathcal{H}$ unable to predict entity labels, when it already predicts a reasonable estimate of the entity bounding boxes? (See point 4 in weaknesses)

3. Using pretrained weights adds certain biases to the predicate decoder $\mathcal{H}$. Would initializing the weights randomly make the learning much harder ? (See point 5 in weaknesses)

4. The authors should report ablations on parameters $N_{ce}, N_p$ and also report the zero shot recall performance. Additionally, they should also report the per class predicate performance to help understand what predicate classes the model does better on (See point 6 in weaknesses)
limitations: The authors have addressed certain limitations and potential negative societal impact of their work. Other limitations could arise from the choice of DETR architecture. For example, the vanilla DETR model ([3]) has poorer performance on detecting smaller objects (see [b]). 

[b] Zhu et al. "Deformable DETR: Deformable Transformers for End-to-End Object Detection." ICLR 2021.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes an end-to-end trainable framework for SGG using a Transformer architecture that models entity and predicate distributions with conditional queries simultaneously, and then weakly couples predicate and entity detection through the refinement of entity locations & the prediction of labels. It achieves a decent performance on the Visual Genome benchmark.
strengths_and_weaknesses: Strengths:
This paper is well organized, clearly written, and easy to follow. It has a clear explanation of the motivation based on the observation of the preliminary works. Besides, I like the design that decouples entity and predicate distributions with conditional queries and couples them with another instance refinement decoder. The final experiment results also support their claim and show superior performance in the widely-used datasets.

Weaknesses:
1. Missing literature: The authors do not include some recent works, like papers on SGG [1] and the similar human-object interaction detection task [2].
2. I'm not sure whether this work really achieves SOTA as it claims as I find the results in [1] are higher (but they use RN101, maybe an unfair comparison).
3. Typo: Line 135, two 'use of'

[1] Li et.al. SGTR: End-to-end Scene Graph Generation with Transformer
[2] Kim et.al. Hotr: End-to-end human-object interaction detection with transformers
questions: Overall this paper gives a clear introduction to their motivation and solutions. One minor question is the preliminaries. I'm not sure how Dual decoders actually work, and the detailed experimental setup of SS & DD.
limitations: The author already mentioned the limitation of their work in Suppl, which is their method doesn't take into account long-tail distribution into consideration. I expect to see further improvement in the future to address these issues.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a single-stage end-to-end trainable DETR-based architecture named TraCQ. It uses a separate encoder-decoder architecture that disentangles entity and predicate space. Unline earlier architecture they demonstrated strong coupling between entity space and predicate space hurts the performance. Hence it proposes conditional learning of predicate couple learning of entity scene and predicate prediction and establishes a weak coupling by loosely estimating bounding boxes of subject and object. This conditional and couple learning also reduces the parameter significantly. They conducted their experiment on Visual Genome Dataset and achieved a new benchmark on mean SGDet and SGDet recall.
strengths_and_weaknesses: Strength :
The idea has potential, disentanglement of predicate and entity space seems beneficial for predicate prediction. But there are some drawbacks which are mentioned later.


Weakness :
There are a few weaknesses in the paper:
1. It uses pnp DETR, but never mentioned its object detection performance, whereas SGDet is highly reliant on Object Detection performance. Despite the good performance on SGDet Table 3 shows the model is underperforming on Predicate Detection. It may possibly the performance may come from superior object detection performance. Authors should provide additional experiments.
2. What is the object-scene interaction feature $f_{i}$? its better to introduce or explain a bit.
3. Visual Genome has high biases, for a strong result section authors should test TraCQ on multiple datasets.

questions: 1. Have you conducted any experiments from class to predicate? as this is the normal order
2. Can you scale up for a large number of objects as the complexity is high?
Some of the suggestions:
Please add the object detection score and FPS in Tab.2 for completeness. 


Good to have but doesn't impact the rating : 
The presentation and writing can also be improved. 
It is better to include some of the early transformer-based two-stage networks like : 
1.Relation Transformer Network, Koner at el,
2. Context-aware Scene Graph Generation with Seq2Seq Transformers , Lu at el
limitations: The authors should provide some of the limitations of the method like:  is it applicable to other relation detection metrics (e.g. SGCls)
review_title: null

Meta-review: metareview: Paper was reviewed by five reviewers and received: 3 x Borderline Accept, 1 x Borderline Reject and 1 x Weak Accept. Generally, the reviewers thought that the paper was interesting and had merits. Raised issues revolved around (1) lack of clarity in certain parts of exposition; (2) evaluations and ablations that could have been made stronger, and (3) the role of PnP DETR, which isn't a contribution, towards improved performance. Additional reservations dealt with (4)  claims that learning in the non-combinatorial predicate space is easier than the entity pair space, and (5) fairness of comparisons with respect to model capacity and other factors. Authors have provided a compelling rebuttal and this has alleviated many of the reviewer concerns at lest to an extent. Post rebuttal, [ZNFu] remains concerned that generating more predictions may be what is causing improved performance and points out that certain ablations are still missing and can improve the paper. At the same time, [ZNFu], while remains at Borderline Reject, acknowledges that rebuttal has resolved some of the issues in the original review. 

AC has carefully considered the reviews, the rebuttal, and the paper itself. This appears to be a rather borderline case, however, considering that the overall sentiment of reviewers is positive and that rebuttal has convincingly addressed an important fraction of concerns raised by [ZNFu] and others, even if not all, it is AC's decision that acceptance of the paper is warranted. Authors are very strongly encouraged to add the ablations, as well as make corrections, suggest by reviewers, for the camera ready. 
confidence: Less certain
Decision: Accept