Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generation

Authors: Minghan Chen, Guikun Chen, Wenguan Wang, Yi Yang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Scene Graph Generation, Visual Relation Detection

Abstract:
DETR introduces a simplified one-stage framework for scene graph generation (SGG) but faces challenges of sparse supervision and false negative samples. The former occurs because each image typically contains fewer than 10 relation annotations, while DETR-based SGG models employ over 100 relation queries. Each ground truth relation is assigned to only one query during training. The latter arises when one ground truth relation may have multiple queries with similar matching scores, leading to suboptimally matched queries being treated as negative samples. To address these, we propose Hydra-SGG, a one-stage SGG method featuring a Hybrid Relation Assignment. This approach combines a One-to-One Relation Assignment with an IoU-based One-to-Many Relation Assignment, increasing positive training samples and mitigating sparse supervision. In addition, we empirically demonstrate that removing self-attention between relation queries leads to duplicate predictions, which actually benefits the proposed One-to-Many Relation Assignment. With this insight, we introduce Hydra Branch, an auxiliary decoder without self-attention layers, to further enhance One-to-Many Relation Assignment by promoting different queries to make the same relation prediction. Hydra-SGG achieves state-of-the-art performance on multiple datasets, including VG150 (16.0 mR@50), Open Images V6 (50.1 weighted score), and GQA (12.7 mR@50). Our code and pre-trained models will be released on Hydra-SGG.

Decision: accept (poster)
Reviewer scores: 5, 6, 8

Reviews and discussion:
<thread>
creation date:
1730437654084

thread_type:
official_review

<thread_object>
creator: reviewer_arzw

<content>
summary:
This paper proposes a new method called Hydra-SGG for Scene Graph Generation (SGG), which addresses the challenges of sparse supervision and false negatives in one-stage SGG. To this end, it employs a one-to-many assignment strategy and a transformer decoder without self-attention called a hydra-branch. Through comprehensive experiments across various datasets and detailed ablation studies, the paper confirms the efficiency and effectiveness of this method.

strengths:
1. This paper pinpoints the critical problems in one-stage SGG, which are sparse supervision and false negatives. By addressing this problem, the proposed method is superior on efficiency and effectiveness.
2. The author conducts extensive experiments, including effectiveness on diverse datasets and ablation studies, thereby validating the superiority of the proposed method. Especially, it significantly reduces the training epoch compared to the baselines.
3. The proposed method is simple yet effective.

weaknesses:
The paper lacks discussion on its limitations and future works.

There are no critical points for rejection.

questions:
1. While the One-to-Many alignment approach helps mitigate sparse supervision, there is a concern that it may introduce significant noise, potentially leading to an increase in false positives. What are some strategies to manage this noise effectively?
2. How would performances be impacted if the weights for One-to-One and One-to-Many alignment losses were changed? For example, I'm curious how performance would change if the weight of One-to-Many alignment loss were increased or decreased.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730598338803

thread_type:
official_review

<thread_object>
creator: reviewer_kzea

<content>
summary:
This paper addresses key challenges in the field of one-stage Scene Graph Generation (SGG), particularly focusing on sparse supervision and false negatives samples.To tackle these challenges, the authors propose two main contributions:
(1) Hybrid Relation Assignment: This method combines One-to-One and One-to-Many Relation Assignment strategies, increasing the number of positive samples during training and thus enhancing supervision.
(2) Hydra Branch: An auxiliary decoder branch without self-attention layers, designed to further support the One-to-Many Relation Assignment by promoting consistent relation predictions across queries.
Above methods jointly improve the training efficacy and performance of DETR-based SGG models, leading to state-of-the-art results on multiple benchmarks with significantly fewer training epochs. Overall, this paper’s motivation is clear, and the proposed method iseffective.

strengths:
(1) Innovative Method Design: The proposed Hybrid Relation Assignment effectively combines One-to-One and One-to-Many matching strategies. This design helps mitigate the sparse supervision issue in existing methods and improves model convergence speed and learning efficiency.
(2) Efficient Auxiliary Branch: The Hydra Branch supports One-to-Many assignment by promoting consistent relation predictions across queries, all without adding inference-time computational costs. 
(3) Good Experimental Results: The experimental results show that Hydra-SGG achieves state-of-the-art performance across multiple datasets (VG150, Open Images V6, and GQA), reaching these results in significantly fewer training epochs.

weaknesses:
(1) Lacking theory/analysis, technical solutions appear to be intuitive and empirical. Faster convergence may come from more diverse samples or more training data within an epoch, because the proposed method is likely to give 16 images, but will generate more relation training pairs for training.

(2) Limited Exploration. Why removing Self-Attention benefits the One-to-Many Relation Assignment. The paper introduces the Hydra Branch without self-attention to support the One-to-Many relation assignment. However, the explanation of why removing self-attention aids this assignment is somewhat surface-level, lacking an in-depth analysis of the interaction between self-attention and One-to-Many assignment.

questions:
Does faster convergence come from more diverse samples or more training data within an epoch, because the proposed method is likely to give 16 images, but will generate more relation training pairs for training?

It would be better to compare the methods with unbiased SGG methods or adopted unbiased strategies, considering the higher performance of mean Recall.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730857758775

thread_type:
official_review

<thread_object>
creator: reviewer_o4ve

<content>
summary:
The manuscript introduces Hydra-SGG, a novel one-stage Scene Graph Generation (SGG) method designed to address the challenges of sparse supervision and false negative samples inherent in DETR-based SGG models. The authors propose a Hybrid Relation Assignment strategy that combines One-to-One and IoU-based One-to-Many Relation Assignments to increase positive training samples and mitigate sparse supervision issues. Additionally, the paper introduces Hydra Branch, an auxiliary decoder without self-attention layers, to enhance the One-to-Many Relation Assignment by promoting duplicate relation predictions among different queries. The model achieves state-of-the-art performance on multiple datasets, including VG150, Open Images V6, and GQA, with a significant reduction in training epochs compared to previous methods.

strengths:
**1.Clear Explanation of Methodology:** The paper does an excellent job of explaining the methodology in a step-by-step fashion. The One-to-One Assignment, One-to-Many Assignment, and HydraBranch are each discussed with sufficient detail, making it easier for readers to follow the technical contributions. Furthermore, the inclusion of pseudocode and diagrams helps clarify how each component interacts within the framework.

**2.Comprehensive Experimental Validation:** The paper provides extensive experiments on two widely-used datasets, Visual Genome, OpenImage, GQA, and demonstrates that Hydra-SGG, consistently outperforms existing baselines. The authors also conduct detailed ablation studies, examining the impact of various components (Threshold T, Training Epoch, Number of Relation Queries). The results substantiate the claim that the framework is a substantial improvement over current model.

weaknesses:
**1.Lack of Novelty:** The motivation and methodology of this work closely resemble those presented in [3], raising questions about its originality and distinct contributions. In particular, [3] introduced the one-to-many relation query matching technique, known as Quality-Aware Multi-Assignment, which enhances training efficiency by allowing a single ground truth (GT) to correspond to multiple sufficiently close predictions or queries. While this work makes a slight modification by removing self-attention among relation queries, this adjustment alone does not constitute a significant advancement in one-stage SGG research. As a result, it is difficult to regard this work as a notably innovative contribution. Further discussion is warranted to clarify how this work differentiates itself from existing approaches.
**2.Perverse Experimental Results:** The rationale for assigning ground truth (GT) to multiple sub-optimal queries is based on label softening and knowledge distillation. However, applying this approach to scene graph datasets with long-tailed distributions (e.g., Visual Genome, OpenImage, PSG) introduces significant challenges. This method tends to overemphasize common relationships (e.g., on, of, has) in the annotations, which can lead to model overfitting on these frequent relationships. Consequently, this emphasis on common relationships undermines the model’s ability to accurately capture rare relationships (e.g., standing on, walking on, lying on), ultimately lowering average recall performance. Given these limitations, a decline in mRecall performance would be expected for such datasets. Surprisingly, however, the experimental results in Table 1 show the opposite. Since this study does not incorporate any specific solutions to address data imbalance, the observed improvement in mRecall is unexpected. This outcome appears to contradict the foundational assumptions of the proposed Hydra-SGG approach, indicating a need for further analysis.

**3.Inadequate Model Performance:** Despite showing a substantially lower R@100 score than the DSGG method [6], the Hydra-SGG model demonstrates a corresponding improvement in the mR@100 metric. Based on prior research [5], incorporating the F-Recall metric would provide a more comprehensive assessment of Hydra-SGG’s effectiveness relative to state-of-the-art approaches. Additionally, the paper lacks a thorough comparison with other relevant studies [2].

**Ref:**

[1] Pair then Relation: Pair-Net for Panoptic Scene Graph Generation. Arxiv 2023

[2] Structured Sparse R-CNN for Direct Scene Graph Generation. CVPR 2022

[3] Groupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detection. CVPR 2024

[4] Not All Relations are Equal:Mining Informative Labels for Scene Graph Generation. CVPR 2021

[5] Fine-Grained Scene Graph Generation with Data Transfer. ECCV 2022

[6] DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation. CVPR 2024

questions:
please refer to the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732329674427

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
## Summary of Revisions

To all reviewers:
We sincerely appreciate your time and effort in reviewing our manuscript. Below, we outline the major changes made in response to your feedback:

- We clarify the novelty of Hydra-SGG compared to SpeaQ and other existing one-stage DETR-based SGG models, addressing Reviewer o4Ve's concerns.
- We offer more detailed discussions regarding why mRecall performance is high, according to Reviewer o4Ve's comments.
- We provide comprehensive F-Recall results and comparisons with state-of-the-art methods, including SSR-CNN, per Reviewer o4Ve's request.
- We clarify the analyses conducted to propose Hydra-SGG, addressing Reviewer KZeA's concerns.
- We add detailed statistical analysis on query embeddings to demonstrate how HydraBranch enhances One-to-Many Assignment, responding to Reviewer KZeA's feedback.
- We include additional comparisons with unbiased SGG methods (TDE, NICE, VETO, etc.) showing Hydra-SGG's competitive performance, as requested by Reviewer KZeA.
- We conduct an ablation study on different loss weights between One-to-One and One-to-Many assignments, addressing Reviewer ARZw's question.
- We expand the discussion on limitations and future work, including potential improvements to One-to-Many Relation Assignment strategies, per Reviewer ARZw's suggestion.
- We add additional analyses, ablation studies, and comparisons of rebuttal in the revised manuscript.

Sincerely yours,

Authors.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces Hydra-SGG, a novel method for scene graph generation that addresses sparse supervision and false negatives by employing a hybrid relation assignment strategy and a Hydra branch decoder. These innovations contribute to improved training and competitive results across several datasets, achieved with relatively few training epochs.

The methodology is acknowledged for its clarity and detail. The effectiveness and efficiency of the proposed approach in tackling critical issues in one-stage scene graph generation, such as sparse supervision and false negatives, are also well-appreciated. The experimental results are robust. However, some concerns and questions were raised during the initial round of feedback, including queries about the originality of the approach compared to existing work, unexpected results in the mRecall metric, insufficient performance metrics, and a lack of thorough analysis and exploration.

The authors have provided detailed clarifications, analyses, and additional experimental results focusing on the F-Recall metric and comparisons with additional baselines. After careful review of the paper, its revisions, and the discussion, I believe the key concerns including model performance and unexpected performance explanation have been adequately addressed. Therefore, I recommend the acceptance of this paper.