Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generation
Authors: Minghan Chen
Guikun Chen
Wenguan Wang
Yi Yang
Venue: ICLR 2025 Poster (2025)
Keywords: Scene Graph Generation
Visual Relation Detection
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: DETR introduces a simplified one-stage framework for scene graph generation (SGG) but faces challenges of sparse supervision and false negative samples. The former occurs because each image typically contains fewer than 10 relation annotations, while DETR-based SGG models employ over 100 relation queries. Each ground truth relation is assigned to only one query during training. The latter arises when one ground truth relation may have multiple queries with similar matching scores, leading to suboptimally matched queries being treated as negative samples. To address these, we propose Hydra-SGG, a one-stage SGG method featuring a Hybrid Relation Assignment. This approach combines a One-to-One Relation Assignment with an IoU-based One-to-Many Relation Assignment, increasing positive training samples and mitigating sparse supervision. In addition, we empirically demonstrate that removing self-attention between relation queries leads to duplicate predictions, which actually benefits the proposed One-to-Many Relation Assignment. With this insight, we introduce Hydra Branch, an auxiliary decoder without self-attention layers, to further enhance One-to-Many Relation Assignment by promoting different queries to make the same relation prediction. Hydra-SGG achieves state-of-the-art performance on multiple datasets, including VG150 (16.0 mR@50), Open Images V6 (50.1 weighted score), and GQA (12.7 mR@50). Our code and pre-trained models will be released on Hydra-SGG.

Published as a conference paper at ICLR 2025
HYDRA-SGG: HYBRID RELATION ASSIGNMENT FOR ONE-STAGE SCENE GRAPH GENERATION
Minghan Chen'
Guikun Chen' Wenguan Wang
Yi Yang**
RReLER Lab, AAII, University of Technology Sydney
RReLER Lab, CCAI, Zhejiang University
ABSTRACT
DETR introduces a simplified one-stage framework for scene graph generation (SGG) but faces challenges of sparse supervision and false negative samples. The former occurs because each image typically contains fewer than 10 relation amno- tations, while DETR-based SGG models employ over 100 relation queries. Each ground truth relation is assigned to only one query during training. The latter arises when ground truth relation may have multiple queries with similar matching scores, leading to suboptimally matched queries being treated as nega- tive samples. To address these, we propose Hydra-SGG, one-stage SGG method featuring a Hybrid Relation Assignment. This approach combines a One-to-One Relation Assignment with an IoU-based One-to-Man Relation Assignment, in- creasing positive training samples and mitigating sparse supervision. In addition, we empirically demonstrate that removing self-attention between relation queries leads to duplicate predictions, which actually benefits the proposed One-to-Many Relation Assignment. With this insight, we introduce Hydra Branch, an auxiliary decoder without self-attention layers, to further enhance One-to-Man Relation Assignment by promoting different queries to make the same relation prediction. Hydra-SGG achieves state-of-o the-art art performance VG150 (16.0 mR (0), Open Images V6 (50.1 weighted score), and GQA (12.7 mR 50). Our code and pre-trained models will be released Hydra-SGG.
multiple datasets, including
INTRODUCTION
Ours
A scene graph is data structure that describes the entities (objects) in a scene and the relations between these objects [3]. Scene graph gener- ation (SGG) has attracted significant research
Hydm-SGG
method (0:50
Twa-stage
Hydra-SGG
16.0
DSGG [13]
13.0
14
PE-Net [91]
12.4
attention [20, 50, 3, 43, 71] due to its ability to enhance machines' semantic comprehension of visual content. has been widely adopted in various downstream applications, such as robotic vision and interaction [47, 60, 80, 48], image synthesis and manipulation [79, 10, 22]. visual question answering [21, 66, 41], and video understanding [78, 68, 76, 81].
PE-Net
DSGG
SGTR [40]
12.0
mR
SpeaQ [26]
11.8
12
SpeaQ
SGTR
ReITR
10.8
GGT[299
9.1
ReTTR
10
VETO [61]
8.1
EGTR [18]
7.9
IS-GGT
[24]
7.4
VETO
BGTR
Mainstream SGG methods [85, 64, 50, 91, 61, 29, 35] work in a two-stage fashion. First, off-the-shelf object detector extracts all entities within image. Then, the extracted entities are permuted, yielding (NN -1) entity pairs for N detected entities. These entity pairs used to predict the relationships between the corresponding entities. However, the two-stage meth- ods face a critical limitation: they predict relations for all entity pairs, even though many pairs do
0
50
100
150
200
250
Training Epochs
Figure Comparison with other SGG methods mR @ and training epochs on VG150 [75].
"Coresponniigg author

---

Published as a conference paper at ICLR 2025
(a)
One-to-One Relation Assignment
(b)
Hyybid Relation Assignment (Ours)
GT Triplets:
ride c
wait
GT Triplets:
ride
M
wait B
Relation
Relation
Queries:
Queries:
One-to-One Relation
One-to-One Relation
One-to-Many Relation Queries
Queries
Queries
Figure 2: (a) Previous DETR-bases SGG methods such as RelTR [7] and SGTR [40] match each GT relation with only one query. (b) Our Hybrid Relation Assignment utilizes both One-to-One and One-to-Mant assign- ments, generating more positive samples and thus accelerating training.
not participate in any relations. This incurs heavy computational overhead and time-consuming inference, especially in complex scenes with numerous entities and intricate interactions.
Recently, the simplicity of DETR-based SGG methods [7.40] 40J has led to an ongoing paradigm shift apart from the two-stage framework. Specifically, sequence of visual tokens, mapped from the input image, interacts with a predefined set of relation queries in a Transformer decoder for simul- tancous object detection and relation prediction. Then, One-to-One set matching (i.e., Hungarian matching [28]) is used to assign ground truth labels to the predictions (Fig. 2a). This set-prediction framework allows DETR-based SGG models eliminate hand-designed components, such Non- Maximum Suppression (NMS), which are commonly used in traditional two-stage methods.
Unfortunately, one significant drawback of DETR-based SGG models is their slow convergence. For instance, one-stage ReLTR [7] requires 150 training epochs to converge, while a recent two- stage method, PE-Net [91], only needs 32 epochs. This drawback can be attributed to the sparse relation supervision induced by Hungarian matching, which assigns each ground truth to only one relation query. The sparsity of relation annotations per image further exacerbates this issue. For instance, VG150 [75] train averages only 5.5 ground truth relation triplets per image [43]. This means that each image provides mere 5.5 positive relation queries optimize the loss functions. In the case of ReITR [7]. which has 200 predefined relation queries, only 7.7%% of these queries are positive samples per optimization step. Consequently, relation queries require more optimization steps to learn due to the limited number of positive samples for training. Furthermore, approximately 50% [26] of the plausible but suboptimally matched queries (i.e., queries with correct subject-object classification and IoU >> 0.6 for both boxes) simply treated as no-relate on due to the One- to-One constraint of Hungarian matching. This constraint discards valuable supervisory signals by treating suboptimal yet informative queries negative samples. These queries, while not the best matches for ground truth labels, may capture informative relational cues that could contribute to model learning. Simply assigning these queries as negative samples may introduce false negatives, which in turn leads to label noise and performance degradation [45].
To accelerate the training of DETR- SGG models, we introduce Hydra-SGG', an efficient framework that addresses the slow convergence problem in one-stage DETR-based SGG models. The cores Hydra-SGG are Hybrid Relation Assignment and Hydra Branch. Specifically, Hybrid Relation Assignment synergizes One-to-One and One-to-Many Relation Assignment strategies, pro- viding over 50% more positive queries per training step than previous arts such as RelTR [7]. To further enhance this assignment strategy, introduce an auxiliary branch called Hydra Branch. This branch is specifically designed to encourage different queries to predict duplicated relations by removing self-attention in the decoder, creating synergistic effect with our Hybrid Relation Assignment. The branch shares all other parameters with the original decoder, and this intentional design choice leads improved supervision signals during training ($3.3).
Hydra-SGG makes three main contributions to the field of SGG: First, we propose an efficient framework that effectively addresses the slow convergence problem in one-stage DETR-basee SGG models through a hybrid query-label assignment and synergistic architectural design. Second, our Hybrid Relation Assignment strategy significantly increases relation supervision signals by min-
Hyydra is an abhreviation for "Hybrid Relation Assignment". The name is chosen hecause it reflects the multi-branch structure of our model, which resembles the multiple heads of the Hydra in Greek mythology.
2

---

Published as a conference paper at ICLR 2025
ing false negative samples to increase positive samples during training, providing over 50% more positive samples per training step. Third, the proposed Hydra Branch complements our assign- ment strategy by encouraging duplicate relation predictions during training. while maintaining in- ference efficiency as it shares parameters with the original decoder and is only used during training. Hydra-SGG achieves state-of-the-art performance with remarkable training efficiency, converging 10x faster than existing one-stage SGG counterparts [7, 40, 18].
Extensive experiments on three challenging SGG benchmarks, VG150 [75], Open Images V6 [30], and GQA [16], demonstrate the effectiveness of our Hydra-SGG. It achieves 16.0 mR @ @50 on VG150 test only 12 epochs (Fig. 1), surpassing the previous state-of-the-an the one-stage method SGTR [40] by +4.0 and two-stage method PE-Net [91] by +3.6.
RELATED WORK
Two-stage SGG. Lu et al. [50] first proposed SGG task and designed a two-stage pipeline. Later, many SGG models have been built on it, using various techniques such as recurrent neural networks or its variants [85, 64, 75, 70, 58), visual translation embedding [88, 17], graph neural networks [83, 62, 74, 83, 77]. external or internal knowledge integration [6, 12, 84, 1, 42, 82, 35, 4, 72], attention mechanisms [23, 53, 9], and Transformer based models [51, 7, 40, 61, 8, 59].
Despite the promising performance, two-stage SGG methods rely heavily on manually designed modules, such as NMS, anchor generation, and entity pairing generation modules [85, 64, 9]. In addition, their designs (ypically involve separate stages for detection, pairing, and relation classi- fication, resulting in a complicated pipeline that cannot be trained in a fully end-to-end manner. Furthermore, these methods predict dense entity pairs in inference, leading to high time complexity and computational burden. In contrast, this paper proposes Hydra-SGG, a one-stage SGG method that offers significant advantages over two-stage methods. Hydra-SGG enables end-to-end training and surpasses the performance of two-stage methods.
One-stage SGG. Oteestage methods have gained increasing attention for their simplicity and end- to-end training ability. Early works in this line adopt CNN-based one-stage detectors [46] or query- based sparse R-CNN [67] for direct relationship prediction. Recently, the DETR [2] framework significantly advances SGG and fluman-Objec' Interaction [54, 92, 7, 40, 63, 95, 44, 25, 86, 26, 18, 38, 37, 731. This framework enables end-to-end training by associating ground truth labels with output queries. Existing DETR-based SGG methods focus on different aspects: SGTR [40] and DSGG [13] explore query designs for relation feature extraction, SpeaQ [26] investigates relation- specific query grouping strategies to improve the specialization and discrimination of queries, while other works focus on architectural innovations [7] and lightweight frameworks [18].
However, DETR- based SGG models, while eliminating NMS, face critical challenge of slow convergence due to sparse relation supervision, requiring significantly more training epochs than two-stage counterparts (e.g., 150 for RelTR [7] vs. 32 for PE-Net [91]). Our proposed Hydra-SGo specifically addresses this limitation through query-label assignment and architectural design while maintaining the advantages of DETR-based approaches. Hydra-SGG achieves remarkably fast con- vergence with only 12 epochs, surpassing both one-stage counterparts and two-stage methods to achieve of-t--of-the-rr performance across multiple SGG datasets (16.0 mR@50 on VG150 [75], 50.1 weighted score on Open Images V6 [30], and 12.7 mR @ 50 on GQA [16]).
DETR for Object Detection. The introduction of DETR by Carion et al. [2] marks significant shift away from traditional CNN-based object detection models [57, 55, 56, 45, 14], adopting an end-to-end trainable approach with a novel application of the Transformer architecture and bipartite matching. While DETR streamlines the detection process, it requires 500 epochs to converge [93]. This spurs wave of research that focuses on improving low training efficacy, including enhancing training signal [5, 15, 19, 94], the adoption of anchor boxes [49, 52], and the leverage of efficient attention mechanism [93, 11]. For instance, Co-DETR [94] introduces extra groups of object queries and DN-DETR [31] utilizes noisy queries to increase positive samples.
In this paper, we present Hydra-SGG, a framework that addresses the sparse relation supervision challenge inherent in DETR- SGG models. Our method introduces Hybrid Relation Assign- ment, which significantly increases the number of positive samples per training iteration, effectively mitigating the sparse relation supervision issue and accelerating the model's convergence.
3

---

Published as a conference paper at ICLR 2025
3 HYDRA-SGG
$3.1 introduces our baseline model, which employs One-to-One Relation Assignment. Next, $3.2 describes Vanilla Hydra-SGG, which enhances the baseline by introducing a Hybrid Relation As- signment strategy. Finally, $3.3 details the complete Hydra-SGG model, which includes Hydra Branch (HydraBranchi) an auxiliary decoder that promotes One-to-Many Relation Assignment. $3.4 performs a statistical analysis to validate our design of Hydra Branch, demonstrating its effec- tiveness in enhancing the One-to-Mann Mo-May assignment strategy.
3.1 ONE-TO-ONE RELATION ASSIGNMENT BASELINE
Problem Formulation. Given an input image, SGG models aim to generate scene graph in the form of relation triplet: (Csab- P. Coty). where each entity Gsub. Cobj E & is represented by a category label E C (e.g., cat, people, car) and a bounding box b, and p P is a specific relation type (e.g., on, have, ride). In this paper, we distinguish between the terms "relation" and "relation triplet". A relation refers the interaction relationship between entities, while relation triplet represents the complete structure containing the subject 6sub, object Cobj. and their relation p.
Baseline Architecture. Our baseline model is composed of Backbone model, Encoder, and Re DDecoder (Relation Decoder). The relation queries interact with image tokens extracted from the input image by Ba Backbone and Encoder. The updated queries are processed by box regression, entity classification, and relation classification heads to generate the final predictions.
Backbone maps an input image into a feature map that has a H x W x C dimen- sion, where И and W denote the spatial size of the feature map, while C' is the channel dimension (e.g., 2048 or 1024).
Backbone:
Encoder: Encoder captures comprehensive spatial and contextual information across the im- age. Before feeding the feature map into Encoder, 1 xx] conv layer is employed to reduce the dimension of the feature map. The enhanced image tokens are denoted as F E HW 256
RelDecoder: In RelDecoder, we introduce relation queries Qrel Nx512
which are formed by concatenating subject queries Qub E IR-N x226 and object queries Qabj RN x256 along the channel dimension (i.e., 256). Here N denotes the number of queries. Re RelDecoder(Quu Qobj. F)) works as follows:
Qaa Qoij = SA( =SA((==AAA)) sub: Qoti)) E 2226
Q sub CA(F, Q subb) E IR Nx256
Q obj CA Qay)) ) E IR Nx256
Specifically, the subject queries Qab and object queries Qobi first interact in self-attention layer (SA) to obtain updated queries Qub and Qoy Subsequently, these updated queries interact with image tokens in cross-altention layers (CA). The output subject and object queries are fed into independent box regression heads and classification heads, producing box predictions b dotj E N and entity class predictions Psub: Pory E NxCC respectively. The relation predictions obtained by a relation prediction head. For example, with 300 relation queries (i.e., N = -300) and 50 relation classes (i.e., |P| 50). the output dimension of frel would be 300 x 50. Finally, the outputs of RelDecoder are combined to generate the predicted relation triplets R= {((Psub) bsub). Pret, (Pony, bos))),, J Note that the i-th subject query is concatenated only with the i-th object query. and there no permutation process.
Pret NxPP)
One-to-One Relation Assignment Loss. In One-to-On Relation Assignment Loss Lozo [7, 40], Hungarian matching (HM) is employed to find the optimal correspondence between the predicted relation triplets R and the ground truth triplets R. The loss function can be formulated
C.. = CR, R).
(2)
The classification losses (including entity and relation) and box regression losses computed between matched predictions and ground truth labels, which is identical with the previous mod- els [7, 40]. The Hungarian matching hyperparameters the same RelTR [7]. A detailed

---

Published as a conference paper at ICLR 2025
Qui
Re.Decader
One-to-One Assignmmnt
It
Y on RB no-rel
Seft
heads
A' ride c
Lazo (Eq.
Backbore
wait
Encoder
512
LHydra
One- to-Many Assignment
(Eq. 7)
Rean
I
heads
on
A.
Y on f%
Loom (Bg. 3)
rides=
ride d=b
x 512
HydraBranch
Figure 3: Overall pipeline of Hydra-SGG: For simplicity, FFN inside the Transformer layer omitted. Hydra- SGG incorporates two Transformer decoders: HydraBranch and Re Decoder. anchhhh shares its parameters with Re Recodrr r but removes self-attention layers. Hydra-SGG combines One-to-One and One- to-Many assignments in synergy, generating more supervision signals.
implementation of the loss function, training strategy, and model architecture provided in Ap- pendix. The evaluation of our baseline model is given in $4.3.
3.2 VANILLA HYBRID RELATION ASSIGNMENT
VG150 [75] contains average of only 5.5 ground truth relation triplets per image in train. Our baseline adopts a One-to-One Relation Assignment and only assigns approximately relation queries match the ground truth relation triplets for each image (i.e., N 300). This severe sparse supervision reduces the training efficacy, as only about 2% of queries each training step match with ground truth labels for learning, while the remaining 98% are treated as negative samples. Consequently, the model requires a significantly higher number of training steps to learn effectively, leading to slow convergence.
out of 300
To enrich the relation supervision signals, we propose a Hybrid Relation Assignment. Specif- ically, we first devise a One-to-Many Relation Assignment (o2m) and then embed into the baseline to cooperate with One-to-One Relation Assignment. Given a ground truth triplet r ([Crab: baub), Crel, (Conj: labi)) E R and predicted triplet = ((Psub; baah), Pret, (Ponj: bobs)) R. the One-to-Many Assignment score So2m is calculated by:
Sozm(T,F)
Psub
(conl [cai] IoU buab) LoU(babj, botj).
+ Pobj
(3)
Sozn combines the predicted class probabilities and IoU scores for the subject and object bounding
boxes. The notations Sual and leatl represent the probabilities of the ground truth classes for the subject and object, respectively. For example, if the ground truth subject class Csub is we would extract the probability corresponding to class 3 from Psub-
Given M ground truth relation triplets and predicted relation triplets, the final One-to-Many Relation Assignment score matrix, with dimensions M x N. is computed by applying Eq. 3 to each pair of ground truth and predicted relation triplets. We keep the results with scores greater than a threshold T and select the top queries for each ground truth (see details in 84.3).
The proposed Vanilla Hydra-SGG applies both One-to-One and One-to-Many Relation Assignment strategies to the same set of predicted relation triplets R. These triplets obtained from the updated queries Qnd after passing through the model. By combining these two assignment strategies, we can formulate the vanilla version loss function
Lvanilla Luzo R.) + Luzm (R,R) R))
Our Vanilla Hydra-SGG provides richer relation supervision signals compared to the baseline by harmonizing the supervision of two assignment strategies. Specifically, it increases the number of positive samples by 65.5% in VG150 [75] train and 58.7% in val (Fig. 4a, b), Vanilla Hyd significantly improves training efficacy and performance ($4.3).
5

---

Published as a conference paper at ICLR 2025
(8)
One-to Oieeto--One Assignment
(b)
Relative
(c) ADS
(d)
Hybrid Relation Assignment
Increase
VG150
10-
30
PPostivve Quirtes.
Relative Incresse Ratio
8
Pooiiie Samples
80-
8
ADS
40-
ID
Selaatt
Q
Selfatm
above
20-
Canerg above
52
Comeng
191 near
above
67
has
has
Train
Val
Train
Val
SA off SA on
209
above 126
on
has
Figure 4: (a) The average number of positive samples of VG150 train and val for One-to One and Hybrid Relation Assignment. (b) The percentage increase positive samples achieved by Hybrid Relation Assignment compared to the One-to-Ons baseline. (c) ADS on VG150 val. (d)-(c) The visualizations show that for the same group of queries that previously predicted different relations, removing the self-attention layers causes them to make identical predictions. The ID column represents the IDD of each relation query.
3.3 COMPLETE HYDRA-SGG
Each Re Decoder layer contains self- and cross-attention layers, and point-wise feed-forwar network (FFN). The self-attention layer enables inter-query interactions [69], while cross-attention and FFN do not explicitly support query interactions. Previous studies indicate that self-attention helps inhibit duplicate predictions [2, 7]. In this work, we further find that the self-attention layer in RelDecoder is crucial reduce duplicated relation predictions.
We introduce a Diversity Score (DS) to quantify the impact of the self-attention layer on the diversity of relation predictions. Specifically, DS defined as the number of distinct relation categories predicted by the model for a single image. For example, if the model predicts relations for queries (sit: sit, t, on, on, has), DS would be 3, there 3 distinct relation categories (s i on, has). We calculate DS using the trained Vanilla Hydra-SGG model, with and without self-attention in the Re Decoder. Let ( and D represent the DS for the k-th image with self-attention enabled and disabled, respectively. The average DS (ADS) across the dataset is then computed as:
ADS.on
Σ
K
DS
ADS
K
DS R
(5)
K
where K is the total number of images. A higher ADS.o compared to ADS would indicate that self-attention promotes diverse relation predictions. As shown in Fig. 4c, ADS and ADS of VG150 [75] val 4.6 and 6.6, respectively. Fig. 4d, illustrate how the self-attention layer enables diverse relation predictions. Without self-attention, queries converge to the same relation. These examples demonstrate self-attention's role in reducing duplicate relations.
The above findings reveal critical interactions between self-attention and Many Relation As- signment: i) Conflict with Self-Attention: Applying One-to-Many Relation Assignment strategy to aatttentionnudaattee
relation queries in Re DDeceer of Vanilla Hydra-SGG causes mismatch in optimization objectives, self-attention promotes diversity while One-to-Many Relation Assign- ment assigns one ground truth to multiple queries. ii) Benetit without Self-Attention: Conversely, removing self-attention leads more duplicated relation predictions, potentially synergizing with our One-to-Many Relation Assignment.
To address this potential conflict and fully harness the benefits of both the self-attention layers and the One-to-Many Relation Assignment strategy, we propose HydraBranch (Hydra Branch, Fig. 3), an auxiliary decoder that shares parameters with Re Decoder but removes the self- attention layers. This multi-branch architecture decouples the learning objectives: the self-attention layers the main RelDecd can focus on promoting diversity in relation predictions, while HydraBranch facilitates One-to-Many Relation Assignment. This separation enables each branch to optimize its specific function without compromising the other.
HydraBranch operates with the main RelDecoder in parallel, processing the same input but without the influence of self-attention layers. Specifically, the initial One-to-Many relation queries Q02m are set to Q (i.e., Q02m Qrei) before being sent into HydraBranch. The

---

Published as a conference paper at ICLR 2025
process of HydraBranch( Qai FF) is follows:
Q e
CA(
E TR. Nx256
Q CA Qu6y) E N x256
(6)
The One-to-Many predicted relation triplets Ro2m ((riur F22m) Polm (Pob) bog )) /n=1 are
derived in the same manner as R and share prediction heads with R. Compared with Eq. 1, subject and object queries do interact in a self-attention layer before they are sent into subsequent cross- attention layers. We apply One-to-Many Relation Assignment described in Eq. 3 to Ro2m and R. Hybrid Relation Assignment Loss is then given by:
Hydra Lo20
(R, R) + Lozml R RR)
By incorporating HydraBranch, the complete version of Hydra-SGG achieves 10.6 and 16.0 on mR @20 and mR 50, respectively, in just 12 training epochs, further boosting the performance compared to the vanilla version ($4.3). Note that HydraBranch is used only in training and discarded in inference, thus bringing no extra parameters or delay.
3.4 STATISTICAL ANALYSIS OF HYDRA BRANCH
To quantify how Hydra Branch enhances the One-to-Many assignment strategy, we analyze the Euclidean distances between query embedding pairs. Using 5,000 images from the VG150 val set, we compute pairwise distances for all 300 queries per image (totaling (22) 44, 850 pairs per image) two conditions: with and without self-attention layers. The removal of self-attention reduces the mean query distance from 7.55 to 7.23 (5% decrease). A paired t-test confirms statistical significance (t = 54.502, p < 0.001), with a Cohen's d effect size of 0.771 approaching the threshold for large effect (d 0.8). These results demonstrate that Hydra Branch promotes query similarity through self- attentieno removal, thereby facilitating One-to-Many assignment.
EXPERIMENT
4.1 EXPERIMENTAL SETUP
Datasets. We conduct experiments on three datasets:
Visual Genome (VG150) [27, 75] contains 150 entity and 50 relation categories. It is split into 57,723 training, 5,000 validation, and 26,446 testing images.
Open Images V6 [30] features 288 entity and 30 relation categories, including 126,368 training. 1,813 validation, and 5,322 testing images with relation annotations.
GQA [16] encompasses 200 entity and 100 relation types, with split of 52,623 training, 5,000 validation, and 8,209 testing images annotated for SGG tasks.
Evaluation Metrics. We focus on the scene graph detection (SGDet) setting on VG150 [27], GQA [16] and Open Images V6 [30] datasets. For VG150 and GQA, report Recall (R @ k), mean Recall@k [6, 64] (mR@k), and F-Recall [87] performance. mR @K calculates R@K for each predicate individually, then averages these values. It is important note that recall met- rics are more influenced by dataset bias [3], whereas mean recall provides a more holistic evalu- ation of the model's performance. F-Recall the harmonic average of Recall and mean Recall. For Open Images V6, we follow evaluation protocols [30]: Recall@5) 50, the weighted mean Av- erage Precision (wmAP) for relationship detection (wmAPrel), and phrase detection (wmAPphr). The overall score, denoted as scorewid is calculated as weighted average of these metrics: 0.2 R 50 + 0.4 x wmAP,ed + 0.4 wmAP phrr.
Competitors. Hydra-SGG is compared with methods from two categories: (1) Two-stage SGG
methods, including MOTIFS [85], VCTree-TDE [65], BGNN [39], PF-Net [91], IS-GGT [29], VETO [61], UniVRD 190], DRM [32]. CFA [34], and SHA [9]; (2) One-stage methods includ-
ing SGTR [40], SSR-CNN [67]. ISG [24], ReIR [7], DSGG [13], SpeaQ [26], and EGTR [18].

---

Published as a conference paper at ICLR 2025
Table 1: SGDet evaluation on VG150 [27] test ($4.2). *: detector pre- pre-trained on VG150. FPS (Frames Second) indicates inference speed. F-Recall of Hydra-SGG is calculated based on the best results.
Method
Backbone
Epoch FPS
Param
R 20/50/100
mR 20/50/100
2020050/1000
Two--tage methods
MOTIFS [85]
ResNeXt101-FRR
369.9M
/5.1//2.1/00
4.1/5.5/68
7.1/92//117
VCT [65] CEVEENN
ResNeXt101-
361. .M
19.6/
6.3/93/11.
88/1224/-
BONN [39] [CVROS211
ResNeX(101-FF
341.9M
23.3/310/3588
7.5/ /127
11.3/15.5/19
Net.Ne [91] [eVRRSIN
ResNeXt101
32"
-/30.7/35.2
-/124/145
1/17.7/21.
IS-GGT [eVRROG2)
ResNet101
-1-1-
-19.//133
-/-1-
VETO [6]] [ncevooa1)
ResNeXI101-FPN
33
//2.5/3155
-/8.1/9.5
1/1.5/14.6
UniVRD [90] pocviion)
CLIP VIT.B
-1-1-
-/9.6/121 12.1
-/-1-
DRM [32] [CVR00G
ReaNeXt101-FPN
//34.0/38.9
-/9.0/122 11.
/14.2/17744
One-stage methods
SGTR [40] (1990221
ResNet101
117.IM
-/25.1 26.66
1/..0/14.6
//12//
SSR CNN [67] JEVRRSIN
ResNet101
274.3M
25.8/ /36.9
6.1/8.4/10
9./1344/15.
ISG (Nwi1100011
ResNet101
52
93.5M
/9.55//32.1
-/7.4/8.4
1/118 13.3
ReTTR [7] m1M03125
ResNet50
150
6,5
63.7M
21.2/2775/0.77
10.3 /5.5/
DSGG [13]
60
-/32./38.55
-/13.0 17.3
-I // 23,9
SpeaQ [26]
coveesi
ResNet101
//32./366 /3.00
-/118/10 14.1
1/1..4 20.3
EGTR [18]
ResNet50
275*
7.7
42.5M
23.5/302//3.3
5.5/99/101
89/125//56
Ours
21./2286/354
10.3/1599 15.9 19.4
Hydra-SGG (Ctлaао1))
ResNet50
12
5.3
67.6M
14.0 20.5 24.7
#0.1 H011/42//00
-0.2/ +0./ +0.
Table 2: Evaluation on Open Images V6 [30] test ($4.2).
detector pre-trained on Open Images V6.
Backhone
Epoch
Param
R@50
wmAP..
wmAP,a.
scorcua
Two-stage methods
Motifts [85] (CVPR20US)
ResNcXtt01-FPN
369.00 9M
71.6
29.9
31.6
38.9
BGNN [39] (CVPESMM)
ResNeX:101-FPN
341.9M
75.0
35.5
34.2
42.1
PE-Net [91] (CV9R2023)
ResNcXt101-FPN
76.5
35.4
34.9
44.9
One-stage methods
SGTR [40] (CVPR2022)
ResNer101
123
117.IM
59.9
37.0
38.7
42,3
ReITR (TBAM2222))
ResNet50
150
63.7M
71.7
34.2
37.5
43.0
BGTR [18] (CVPR2004)
ResNet50
275
42.5M
75.0
42.0
41.9
48.6
Oury
Hydra SGG (ouzu3111
ResNet50
7
67.6M
76.01oa
42.8
44.1 1o.2
50.0
4.2 QUANTITATIVE COMPARISON RESULT
VG150 [75] test. Table I reports the comparison results on VG150 test. Hydra-SGG demon- strates outstanding performance on challenging SGDet, achieving @0@20 and mR @ 50 scores of 10.6 and 16.0, respectively, setting a new state-of -ffe-offmit ofte-offthe-att the-art. Hydra-SGe achieves this performance with significantly shorter training time, requiring only 12 epochs. This training time is substantially shorter compared to SpeaQ [26]. EGTR [18], and DSGG [13], with reductions of 40, 263, and 48 epochs, respectively. Despite the shorter training, Hydra-SGG outperforms these methods by +4.2, +8.1, and +3.0 on the mR 50. Furthermore, it even outperforms the state-of-the-art two-stage model. PE-Net [91], by a margin 3.6 on the mR 50 metric. We speculate that the significant improvement in mR can be attributed to our One-to-Many Assignment strategy, which ensures bal- anced supervision across both rare and common relations by allocating a fixed number of six queries per relation category 3.2). For instance, in image containing ten "on" relations and two "sit" relations, while One-to-One Assignment would allocate ten queries to "on" and only two to "sit", our approach assigns six queries each, resulting in 60% increase for common relations and substantial 300% increase for rare relations.
Open Images V6 [30] test. As shown in Table Hydra-SGG achieves SOTA in only epochs while SGTR [40], ReITR [7] and EGTR [18] require 119, 150, and 275 epochs. Although Open Images V6 contains more than 120,000 images, its scenes are not as complex those VG150 [75].
Training Hydra-SGG for 7 epochs is sufficient to achieve good performance.
GQA [16] test. Our experimental re- sults demonstrate that Hydra-SGG achieves
Table 3: Evaluation on GQA [16] test (84.2).
Method
Backbone
5
500
SOTA mean Recall performance on GQA. As shown in Table 3, Hydra-SGG achieves 12.7 and 15.9 for 550 and mR @ 100 respectively. surpassing previous best sults. Our model employs the relatively lightweight ResNet50 as its backbone, in contrast to the heavier ResNeXt101 used by other methods.
SHA [9 [eveeu
ResNeXt101-Fa FPN
25.5729.1
6.677.8
VETO [ocev2e2)
ResNeXt1O1-FPP
26.1/29.0
7.0
CFA [34paovai
ResNeXt101-FPP
10.8 / 12.6
Owrs
23.1/26.8 12.5 15.6
Hydra-SGG (CCR0225)
ResNet50
10.3
40.2/ 40.
8

---

Published as a conference paper at ICLR 2025
Table 4: A of ablative experiments about on VG150 [27] test (84.3). The adopted hyperpa- rameters are marked in red.
Method
m0@20
50
Lpoch
Method
Total training time
mR @:20
mR@
Baseline
8.7
12.9
50
RelTR [7]
50.0h (150 epochs)
6.8
10.8
Vanilla Hydra-SGG
(+12)
(+2.0)
12
Baseline
16.7h (50 epochs)
8.7
12.9
Hydra-SGG
10.6 (+1.9)
16.0 (+3.1)
12
Hydra-SGG
12.0h (12 epochs)
10.6
16.0
(a) Key Components
(b) Training Time
T
mR
20
mR@50
mR 100
# Epoch
mR 20
mR 50
mR @ 100
# Query
mR
mR m0
mR 100
0.3
10.4
14.9
18.9
10
10.0
13.4
18.6
100
8.8
14.1
17.8
0.4
10.6
16.0
19.7
12
10.6
16.0
19.7
200
9.9
15.3
18.4
0.5
9.4
15.5
19.6
11.6
16.1
19.7
300
10.6
16.0
19.7
0.6
10.3
14.9
19.1
10.4
15.2
19,9
400
10.6
15.5
18.9
(c) Threshold T
(d) Training Epoch
(e) Number of Relation Queries
Loss Ratio (1-to-1 I-to-M)
mR 50
mR 100
Method
mR 50 mR 100
1:0.5
14.6
18.9
TDE [65] CVVR22020)
9.2
11.1
1:0.8
15.6
19.4
NICE [33]
10.4
12.7
1:
16.0
19.7
IETrans [87] BCCV2022)
12.5
15.0
1:5
15.4
18.7
CFA [34] (CCV2021)
12.3
14.6
1:2
15.6
19.5
VETO [61] (CCV20233
10.6
13.8
1:3
15.4
19.3
NICEST [36] (PAM22004)
10.4
12.4
(f) Loss Weights Between One-to-One (1-to-1) and One-to-Many (I-to-M) Losses.
Hydra-SGG (CLR2025)
16.0
19.7
(g) Comparisons with Unbiasing Methods.
Comparisons with Unbiasing Methods. As shown in Table 4g. Hydra-SGG achieves very compet- itive performance on VG150 compared these specialized debiasing methods, including TDE [65], NICE [33], IETrans [87], CFA [34], VETO [61], and NICEST [36].
4.3 DIAGNOSTIC EXPERIMENT
Key Components. We first investigate the effectiveness of our core Hybrid Relation Assignment ($3.2) and Hydra Branch ($3.3) Table 4a. The first row gives the score of our baseline model ($3.1). The second row corresponds to the result of Vanilla Hydra-SGG, which directly applies Hybrid Relation Assignment into RelDecoder. The third row lists the results of the complete Hydra-SGG model. Our results show that taking the proposed Hybrid Relation Assignment in the baseline model improves the mean recall and achieves 9.9/14.9 mR 20/mR @0. Furthermore, the integration of Hydra Branch with Hybrid Relation Assignment yields synergistic effect, boost- ing performance substantially. Specifically, Hydra Branch increases mR @ 20 from 9.9 to 10.6 and mR 550 from 14.9 to 16.0, highlighting the complementary nature of these strategies.
Training Efficacy. In Table 4b, we compare the training time costs on VG150 [75]. All models are trained on eight NVIDIA RTX 4090 GPUs with ResNet50 backbone. Our model achieves 16.0 mR 50 in 12 hours, whereas ReITR takes 50 hours to achieve 6.8 mR 50. This demonstrates the superior training efficacy of Hydra-SGG.
Threshold T. We next study the influence of the threshold T in One-to-Many Relation Assignment Table 4c. The threshold T' controls the quality and quantity of positive relation queries. A higher value of 7' keeps only the high-quality positive relation queries but at the cost of reducing their number. Conversely, lowering T' yields more positive samples, but their quality may decrease. This trade-off between quality and quantity directly affects the final performance of the model. Experimental results show optimal performance of 16.0. As we increase T to 0.5 and 0.6, the performance gradually decreases. This suggests that while maintaining high-quality positive samples important, having sufficient number of them also crucial for the model to learn effectively.
0.4, yielding m of 10.6 and mR 50
Training Epochs. We further investigate the influence of training epochs on the performance. As presented in Table 4d, Hydra-SGG achieves state-of-the-art mean Recalls in just 12 epochs, demon- strating its fast convergence. We observe further improvements when increasing the epochs to 21, i.e., 10.6 mR @ 20 and 16.0 mR @ 50 respectively. However, the performance decreases at epoch 24, with mR @ 20 and mR @ 50 dropping to 10.4 and 15.2, respectively. This reduction in performance can be attributed to overfitting.

---

Published as a conference paper at ICLR 2025
(+) Image
(b) Ground truth
(c) Hydra-SGG
(d) ReITR [7]
biul
Hebadd
Usplash Image?
Usplash Image
Usplash Image
Figure 5: Qualitative results $4.4. (a)-(d) compare Hydra-SGG and RelTR [7] VG150 [27] image. We use the same color for each entity category, and the color predicate matches that of its subject. Differences are highlighted with red dashed rectangles [.). (e)-(g) show scene graphs generated by Hydra-SGG from images sourced from Unsplash, a platform for freely-usable images. These images real real-world, "in the wild" scenarios, demonstrating our model's capability to handle diverse and unseen visual content.
Number of Relation Queries. Lastly, we studied the effect of the number of relation queries on model performance, as shown in Table 4e, The performance with 200 queries and 400 queries is inferior to that with 300 queries. The model achieved mR @ 20/50 of 9.9/15.3 with 200 queries, and 10.6/15.5 with 400 queries. contrast, with 300 queries, the model reached its highest performance, with 10.6 mR @ 20 and 16.0 mR@ @:50 50. We hypothesize that using 200 queries may be insufficient to capture the diversity of relationships within the data. On the other hand, using 400 queries could introduce excessive noise and false positives, thereby reducing the overall precision of the model.
Impact of Loss Weights. As shown in Table 4f, we conducted an ablation study to investigate the effect of different loss weights between One-to-One and One-to-Many losses. With 1:1 loss ratio, the model achieves the best results (mR@ @ 50: 16.0, mR @ 100: 19.7). When the One-to- to-Many loss weight is reduced (e.g., 1:0.5), the model fails to fully utilize the additional positive samples avail- able. Conversely, increasing this weight excessively (e.g.. 1:3) leads to an overemphasis on One- to-Many triplets, which typically have lower quality compared One-to-One triplets. This quality difference arises from the fundamental nature the assignments: One-to-One assignment matches each relation label with its optimal query exclusively, while One-to-Many assignment matches relation label with multiple queries meeting certain criteria.
4.4 QUALITATIVE COMPARISON RESULT
In Fig. 5a-d, visualize the generated scene graph results the image of VG150 [27] val. Both Hydra- Sadra-SGGaa and ReITR [7] detect plausible relations, but these relations were not annotated, so these reasonable predictions become false negatives. Our Hydra-SGG detects more fine fmnegrained relations such as rock-on-mountal ain and 1 logo-on-shi rt. while RelTR fails to detect such relations. In Fig. Se-g, visualize generated scene graph results unseen Unsplash images.
CONCLUSION
This paper introduces Hydra-SGG, one-stage DETR- scene graph generation (SGG) model that addresses the slow convergence issue existing DETR-based SGG models. The key contri- butions of our work are as follows: i) We propose a Hybrid Relation Assignment, which combines One-to-One and One-to-Many Relation Assignment strategies to increase relation supervision sig- nals. ii) We introduce Hydra Branch, auxiliary decoder that encourages relation queries to pre- dict duplicate relations, further enhancing the proposed One-to-Man Relation Assignment. These innovations work in synergy significantly accelerate the learning process. Consequently, Hydra- SGG achieves of results on VG150 [75], GQA [16], and Open Images V6 [30] just 12, 12, and training epochs, demonstrating remarkable performance and efficiency.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
[1] Stephan Baier, Yunpu Ma, and Volker Tresp. Improving visual relationship detection using semantic modeling of scene descriptions. In ECCV, pp. 53-68, 2017.
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pp. 213-229, 2020.
[3] Xiaojun Chang, Pengzhen Ren, Pengfei Xu, Zhihui Li, Xiaojiang Chen, and Alex Hauptmann. comprehensive survey of scene graphs: Generation and application. TPAMI, 45(1):1-26, 2021.
[4] Guikun Chen, Jin Li, and Wenguan Wang. Scene graph generation with role- vole-playing large language models. NeurlPS, 2024.
[5] Qiang Chen, Xiaokang Chen, Jian Wang, Shan Zhang, Kun Yao, Haocheng Feng, Junyu Han, Errui Ding, Gang Zeng, and Jingdong Wang. Group detr: Fast detr training with group-wise one-to-many assignment. In ICCV, pp. 6633-6642, 2023.
[6] Tianshui Chen, Weihao Yu, Riquan Chen, and Liang Lin. Knowledge-embedded routing net- work for scene graph generation. In CVPR, pp. 6163-6171, 2019.
[7] Yuren Cong, Michael Ying Yang, and Bodo Rosenhahn. Reltr: Relation transformer for scene graph generation. TPAMI, 2023.
[8] Qi Dong, Zhuowen Tu, Haofu Liao, Yuting Zhang, Vijay Mahadevan, and Stefano Soatto. Vi- sual relationship detection using part-and-sum transformers with composite queries. In ICCV, pp. 3550-3559, 2021.
[9] Xingning Dong, Tian Gan, Xuemeng Song, Jianlong Wu, Yuan Cheng, and Liqiang Nic. Stacked hybrid-attention and group collaborative learning for unbiased scene graph genera- tion. In CVPR, pp. 19427-19436, 2022.
[10] Azade Farshad, Yousef Yeganch, Yu Chi, Chengzhi Shen, Böjm Ommer, and Nassir Navab. Scenegenie: Scene graph guided diffusion models for image synthesis. In ICCV, pp. 88-98, 2023.
[11] Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, and Hongsheng Li. Fast convergence of detr with spatially modulated co-attention. In ICCV, pp. 3621-3630, 2021.
[12] Jiuxiang Gu. Handong Zhao, Zhe Lin, Sheng Li, Jianfei Cai, and Mingyang Ling. Scene graph generation with external knowledge and image reconstruction. In CVPR, pp. 1969-1978, 2019.
[13] Zeeshan Hayder and Xuming He. Dsgg: Dense relation transformer for end-to-end scene graph generation. CVPR, pp. 28317-28326, 2024.
[14] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, pp- 2961-220 29699 2017.
[15] Zhengdong Hu, Yifan Sun, Jingdong Wang, and Yi Yang. Dac-detr: Divide the attention layers and conquer. In NeurlPS, 2024.
[16] Drew Hudson and Christopher Manning. Gqa:
new dataset for real-world visual rea- soning and compositional question answering. In CVPR, pp. 6700-6709, 2019.
[17] Zih-Siou Hung, Arun Mallya, and Svetlana Lazebnik. Contextual translation embedding for visual relationship detection and scene graph generation. TPAMI, 43(11):3820-3832 2020.
[18] Jinhae Im, GeongYeon Nam, Nokyung Park, Hyungmin Lee, and Seunghyun Park. Egtr: Ex- tracting graph from transformer for scene graph generation. CVPR, pp. 34229-242333 2024.
[19] Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. In CVPR, pp. 19702-19712, 2023.
11

---

Published as a conference paper at ICLR 2025
[20] Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bersstein, and Li Fei-Fei. Image retrieval using scene graphs. In CVPR, pp. 3668-3678, 2015.
[21] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual rea- soning. In ICCV, pp. 2989-2998, 2017.
[22] Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In CVPR, pp. 1219-1228, 2018.
[23] Deunsol Jung, Sanghyun Kim, Won Hwa Kim, and Minsu Cho. Devil's the edges: Selective quad attention for scene graph generation. In CVPR, pp. 18664-18674, 2023.
[24] Siddhesh Khandelwal and Leonid Sigal. Iterative scene graph generation. In NeurIPS, 2022.
[25] Burssoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo Kim. Hotr: End-to- end human-object interaction detection with transformers. In CVPR. pp. 74-83, 2021.
[26] Jongha Kim, Jihwan Park, Jinyoung Park. Jinyoung Kim, Schyung Kim, and Hyunwoo Kim. Groupwise query specialization and quality-aware multi-assignmae for transformer- based vi- sual relationship detection. CVPR, 2024.
[27] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Con- necting language and vision using crowdsourced dense image annotations. IJCV, 123:32-73, 2017.
[28] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2(1-2):83-97) 97,88-9-7111 1955.
[29] Sanjoy Kundu and Sathyanarayanan N Aakur. Is-ggt: Iterative scene graph generation with generative transformers. In CVPR, pp. 6292-6301 2023.
[30] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection scale. IJC,, 12867):1956-1981 2020.
[31] Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In CVPR. pp. 13619-13627. 2022.
[32] Jiankai Li, Yunhong Wang, Xiefan Guo, Ruijie Yang, and Weixin Li. Leveraging predicate and triplet learming for scene graph generation. In CVPR, 2024.
[33] Lin Li, Long Chen, Yifeng Huang, Zhimeng Zhang, Songyang Zhang, and Jun Xiao. The devil is in the labels: Noisy label correction for robust scene graph generation. In CVPR, pp- 18869-18878, 2022.
[34] Lin Li, Guikun Chen, Jun Xiao, Yi Yang, Chunping Wang, and Long Chen. Compositional feature augmentation for unbiased scene graph generation. In ICCV, pp. 21685-21695, 2023.
[35] Lin Li, Jun Xiao, Guikun Chen, Jian Shao, Yueting Zhuang, and Long Chen. Zero-shot visual relation detection via composite visual cues from large language models. In NeurIPS, 2023.
[36] Lin Li, Jun Xiao, Hanrong Shi, Hanwang Zhang, Yi Yang, Wei Liu, and Long Chen. Nicest: Noisy label correction and training for robust scene graph generation. TPAMI, 2024.
[37] Liulei Li, Wenguan Wang, and Yi Yang. Human-object interaction detection collaborated with large relation-driven diffusion models. In NeurIPS, 2024.
[38] Liulei Li, Jianan Wei, Wenguan Wang, and Yi Yang. Neural-logic human-object interaction detection. In NewrlPS, 2024.
[39] Rongjie Li, Songyang Zhang, Bo Wan, and Xuming He. Bipartite graph network with adaptive message passing for unbiased scene graph generation. In CVPR, pp. 11109-11119, 2021.
12

---

Published as a conference paper at ICLR 2025
[40] Rongjie Li, Songyang Zhang, and Xuming He. Sgtr: End-to-en scene graph generation with transformer. CVPR, pp. 19486-19496, 2022.
[41] Yicong Li, Xiang Wang, Junbin Xiao, Wei Ji, and Tat-Seng Chua. Invariant grounding for video question answering. CVPR, pp. 2928 2937,,997722 2022.
[42] Yikang Li, Wanli Ouyang, Xiaogang Wang, and Xiao' ou Tang. Vip-enn: Visual phrase guided convolutional neural network. In CVPR, pp. 1347-1356, 2017.
[43] Yuanzhi Liang, Yalong Bai, Wei Zhang. Xueming Qian, Li Zhu, and Tao Mei. Vrr-vg: Refo- cusing visually-relevant relationships. In ICCV, pp. 10403-10412, 2019.
[44] Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu. Gen-vikt: Simplify association and enhance interaction understanding for hoi detection. In CVPR, pp. 20123- 20132, 2022.
[45] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, pp. 2980-2988, 2017.
[46] Hengyue Liu, Ning Yan, Masood Mortazavi, and Bir Bhanu. Fully convolutional scene graph generation. In CVPR, pp. 11546-11556, 2021.
[47] Rui Liu, Xiaohan Wang, Wenguan Wang, and Yi Yang. Bird's-eye-view scene graph for vision- language navigation. In ICCV, pp. 10968-10980, 2023.
[48] Rui Liu, Wenguan Wang, and Yi Yang. Vision-language navigation with energy-based policy. In NewrlPS, 2024.
[49] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang. Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. DAB-DETR: Dynamic anchor boxes are better queries for DETR. In ICLR, 2022.
[50] Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. In ECCV, pp. 552-869, 2016.
[51] Yichao Lu, Himanshu Rai. Jason Chang, Boris Knyazev, Guangwei Yu, Shashank Shekhar, Graham W Taylor, and Maksims Volkovs. Context-aware scene graph generation with seq2seq transformers. In ICCV, pp. 15931-15941, 2021.
[52] Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional detr for fast training convergence. In ICCV, pp. 3651-3660, 2021.
[53] Mengshi Qi, Weijian Li, Zhengyuan Yang, Yunhong Wang, and Jiebo Luo. Attentive relational networks for mapping images to scene graphs. In CVPR, pp. 3957-3966, 2019.
[54] Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu. Learning human-object interactions by graph parsing neural networks. In ECCV, pp. 401-417, 2018
[55] Joseph Redmon and Ali Farhadi. Yolo9000: better, faster, stronger. In CVPR, pp. 7263-7271, 2017.
[56] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint 1rXivv1804..2777 2018.
[57] Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
[58] Hanrong Shi. Lin Li, Jun Xiao, Yueting Zhuang, and Long Chen. From easy to hard: Learning curricular shape-aware features for robust panoptic scene graph generation. IJCV, pp. 1-20, 2024.
[59] Suprosanna Shit. Rajat Koner, Bastian Wittmann, Johannes Paetzold, Ivan Ezhov, Hongwei Li, Jiazhen Pan, Sahand Sharifzadeh, Georgios Kaissis, Volker Tresp, et al. Relationformer: A unified framework for image-to-graph generation. In ECCV, pp. 422-439. 2022.
13

---

Published as a conference paper at ICLR 2025
[60] Kunal Pratap Singh, Jordi Salvador, Luca Weihs, and Aniruddha Kembhavi. Scene graph contrastive learning for embodied navigation. ICCV, pp. 10884-10894, 2023.
[61] Gopika Sudhakaran, Devendra Singh Dhami, Kristian Kersting, and Stefan Roth. Vision rela- tion transformer for unbiased scene graph generation. In ICCV, pp. 21 21882-21893, 2023,
[62] Mohammed Suhail, Abhay Mittal, Behjat Siddiquie, Chris Broaddus, Jayan Eledath, Gerard Medioni, and Leonid Sigal. Energy-based learning for scene graph generation. CVPR, pp- 19936-13945, 2021.
[63] Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpic: Query- pairwise human- object interaction detection with image-wide contextual information. In CVPR, pp. 10410- 10419, 2021.
[64] Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to compose dynamic tree structures for visual contexts. In CVPR. pp.
2019.
[65] Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In CVPR, pp. 3716-3725, 2020.
[66] Damien Teney, Lingqiao Liu, and Anton van Den Hengel. Graph-structured representations for visual question answering. In CVPR, pp. 2017.
[67] Yao Teng and Limin Wang. Structured sparse r-cnn for direct scene graph generation. In CVPR, pp. 19437-19446, 2022.
[68] Yao Teng, Limin Wang, Zhifeng Li, and Gangshan Wu, Target adaptive context aggregation for video scene graph generation. In ICCV, pp. 13688-13697, 2021.
[69] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurlPS, 2017.
[70] Wenbin Wang, Ruiping Wang, Shiguang Shan, and Xilin Chen. Exploring context and visual pattern of relationship for scene graph generation. In CVPR, pp. 3188-8197, 2019.
[71] Wenguan Wang, Yi Yang, and Yunhe Pan. Visual knowledge in the big model era: Retrospect and prospect. FITEE, 2024.
[72] Wenguan Wang, Yi Yang, and Fei Wu. Towards data-and knowledge-driven ai: A survey neuro-symbolic computing. TPAMI, 47(2) 2025.
[73] Jianan Wei, Tianfei Zhou, Yi Yang, and Wenguan Wang. Nonverbal interaction detection. In ECCV, pp. 277-295, 2024.
[74] Sanghyun Woo, Dahun Kim, Donghyeon Cho, and In So Kweon. Linknet: Relational embed- ding for scene graph. In NeurlPS, 2018
[75] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In CVPR, pp. 5410-5419, 2017.
[76] Li Xu, Haoxuan Qu, Jason Kuen, Jiuxiang Gu, and Jun Liu. Meta apatio-tempores debiasing for video scene graph generation. In ECCV, pp. 374-390, 2022.
[77] Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation. In ECCV, pp. 670-685, 2018.
[78] Jingkang Yang, Wenxuan Peng, Xiangtai Li, Zujin Guo, Liangyu Chen, Bo Li, Zheng Ma, Kaiyang Zhou, Wayne Zhang, Chen Change Loy, al. Panoptic video scene graph generation. In CVPR, pp. 18675-18685, 2023.
[79] Ling Yang. Zhilin Huang, Yang Song, Shenda Hong. Guohao Li. Wentao Zhang, Bin Cui, Bernard Ghanem, and Ming-Hsuan Yang. Diffusion-bused scene graph to image generation with masked contrastive pre-training. arXiv preprint arXiv:2211.11138. 2022.
14

---

Published as a conference paper at ICLR 2025
[80] Wei Yang, Xiaolong Wang. Ali Farhadi, Abhinav Gupta, and Roozbeh Mottaghi. Visual mantic navigation using scene priors. arXiv preprint arxiv:1810.06543, 2018.
[81] Zongxin Yang, Guikun Chen, Xiaodi Li, Wenguan Wang, and Yi Yang. Doraemongpt: Toward understanding dynamic scenes with large language models (exemplified video agent). In ICML, 2024.
[82] Yuan Yao, Ao Zhang, Xu Han, Mengdi Li, Cornelius Weber, Zhiyuan Liu, Stefan Wermter, and Maosong Sun. Visual distant supervision for scene graph generation. In ICCV, pp. 15816- 15826, 2021.
[83] Guojun Yin, Lu Sheng, Bin Liu, Nenghai Yu, Xiaogang Wang, Jing Shao, and Chen Change Loy. Zoom-net: Mining deep feature interactions for visual relationship recognition. In ECCV, pp. 522-338, 2018.
[84] Ruichi Yu, Ang Li, Vlad Morariu, and Larry Davis. Visual relationship detection with internal and external linguistic knowledge distillation. In ICCV, pp. 1974-1982, 2017.
[85] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In CVPR, pp. 5831-5840, 2018.
[86] Aixi Zhang, Yue Liao, Si Liu, Miao Lu, Yongliang Wang, Chen Gao, and Xiaobo Li, Mining the benefits of two-stage and one-stage hoi detection. In NeurlPS, 2021.
[87] Ao Zhang, Yuan Yao, Qianyu Chen, Wei Ji, Zhiyuan Liu, Maosong Sun, and Tat-Seng Chua. Fine-grained scene graph generation with data transfer. In ECCV, pp. 409-424, 2022.
[88] Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, and Tat-Seng Chua. Visual translation em- bedding network for visual relation detection. In CVPR, pp. 5532-5540, 2017.
[89] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel Ni, and Heung- Yeung Shum. DINO: DETR with improved denoising anchor boxes for end-to-end object detection. In ICLR, 2023.
[90] Long Zhao, Liangzhe Yuan, Boqing Gong, Yin Cui, Florian Schroff, Ming Asugn Yang, Hartwig Adam, and Ting Liu. Unified visual relationship detection with vision and language models. In ICCV, pp. 6962-6973, 2023.
[91] Chaofan Zheng, Xinyu Lyu, Lianli Gao, Bo Dai, and Jingkuan Song. Prototype-based embed- ding network for scene graph generation. In CVPR, pp. 22783-22792, 2023.
[92] Tianfei Zhou, Wenguan Wang. Siyuan Qi, Haibin Ling, and Jianbing Shen. Cascaded human- object interaction recognition. In CVPR, pp. 4263 42633-422,211 2020.
[93] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR, 2021.
[94] Zhuofan Zong, Guanglu Song, and Yu Liu. Detrs with collaborative hybrid assignments train- ing. In ICCV, pp. 6748-6758, 2023.
[95] Cheng Zou, Bohan Wang. Yue Hu, Junqi Liu, Qian Wu, Yu Zhao, Boxun Li, Chenguang Zhang, Chi Zhang, Yichen Wei, al. End-to-end human object interaction detection with hoi transformer. In CVPR, pp. 11825-11834 2021.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
For a better understanding the main paper, we provide additional details in this supplementary
material, which is organized as follows:
A details the implementation details.
B provides the pseudo code of Hydra-SGG.
SC shows failure cases of Hydra-SGG in VG150 [75].
$D discusses our limitations, societal impact, and directions future work.
A IMPLEMENTATION
Inspired by ReITR [7], also relation queries be composed of subject and object queries. We adopt the same training techniques and attention mechanisms as previous works [31, 89, 7] in Hydra-SGG. In particular, we use anchor boxes embeddings to accelerate training and employ deformable attention [93] respectively. We input noised box-label pairs to generate both entity and relation denoising queries. To prevent information leakage, we employ attention masks. Given the absence of an attention map in the deformable decoder, we simplify the model by omitting the convolutional mask head in RelTR [7]. In particular, we concatenate the output subject and object queries and input them into an MLP. This simplification not only reduces computational complexity but also maintains performance.
Both One-to-One and One-to-Many losses composed of box regression loss, entity classification loss, and relation classification loss as previous works [2. 40, 7. 18, 63]. Specifically, we use Focal loss [45] for both relation and detection, in previous works [49, 31, 89]. In the post-processing stage, the predictions are first ranked by the relation probability. Then we use the relation index to find corresponding subject and object predictions to compose the prediction triplets. Finally, follow the same process as ReITR [7] that removes predictions where the subject and object are the same, such data does not exist VG150 [75], but do not use this process in Open Images V6 [30].
For training, we adopt the same data augmentation techniques as RelTR [7] but discard random cropping since some triplcts could be incomplete. The default training epochs are 12 for VG150 [75] and for Open Images V6 [30]. For VG150, the learning rate is scaled by 0.1 epoch 11, while for
Open Images V6, it is scaled at epoch 6. Open Images V6 [30] contains more than 1 120,000 training images, but most scenes in the dataset are simpler than those VG150 [75], therefore training for 7 epochs is enough to achieve state-of-the-arr performance. Extended training on Open Images V6 may improve the performance, but considering the dataset size, the marginal benefits are limited.
Due to the inherent bias in SGG datasets, numerous unbiasing algorithms have been developed [65, 34, 33, 61, 91]. While adopting these unbiasing techniques typically leads to significant performance improvements, opt for fair comparison by evaluating Hydra-SGG against methods that do not employ such techniques.
B PSEUDO CODE
The pseudo-code of Hydra-SGG is given in Algorithm S1. Our code and pre-trained models will be made publicly available.
C FAILURE CASES
In this section, we present failure cases of scene graphs generated by Hydra-SGG on VG150 [75] val. In the first row (Fig. Sla-c), Hydra-SGG successfully recognizes the airplane the image, but the predicted label is 'plane'. Although "plane" is a plausible label, the ground truth label is "air- plane", which leads to all predicted relationships being incorrect since they don't 't match the ground truth. However, from semantic perspective, the model correctly identifies these relationships and even recognizes more fine-grained relationships that are not annotated in the ground truth. For in- stance, the ground truth only marks the right engine of the airplane, but the model correctly identifies
16

---

Published as a conference paper at ICLR 2025
Algorithm Hydra-SGG: PyTorch-like Pseudo-code
Re RDecooerr shares parameters with Hydra Branch memory: tokens
D)
Query_rel: rolation queries of RelDasoder (N x D)
Query_rel_o2m; relation cueries of HydraBranch, (N DJ
less_oRa; apply regression entity,
relation class faasifiaation lesses wetn
matched queries and GIs by Hungarian matching. loss_o2m: apply regression, entity, relation classification lcssea betweea matched querles and GIs by One-to-Many Re Relation Asaignment.
Hydra SGG forward
def forward target):
the input memory Encoder (Backbone (sample)) (b x DJ
excract enhance features
init relation queries for RelDecuder and Hydra Branch Query_rel Query_rel_o2z self seif..nit.queries]
Query_rel_baz - (Queryy memory) (N x D) (ydraBranchhuueryyrel....
Quezy_rel_o2m__
memory) (N
1osa_1 loss_ Loss_o2m
(Query targets)
targets)
l05s lass_hydra less_ loss_2
loss www.hyydr...aakaaadd.
ReIDecader
def RelDecoder (Query_rei, memoryl:
Query_rel Self_atth iquery_rel) IN D)
Query_ Cross_attnn (Query_rel, memory) D)
Quezy_rel - TFN (Query return Query_rel
4 Hydra Branch
HydraBranch (Query_ra) _e___,_ memoryl:
query_rel_te Cross_attnn (t (Query_rel_ooee r2m_ menory) (N x DJ
Query_rel_oze FFW (Query_rel_com return sucry_roi_olo
Algorithm S1: Hydra-SGG core implementation.
Togo
engine
hirplane
plane
engiin
Sngine
Image
Ground
(c) Hydra-SGG
shirt
shirt
above
wearing
letter
WOTTED
ebori
short
Dound
WERTT
Aoove
shose
weering
beach
(d) Image?
(c) Ground truth 2
(1) Hydra-SGGi 2
Figure S1: Failure cases of Hydra-SGG on VG150 [75] val.
both engines. In addition, the model correctly identifies the logo and windows on the airplane, which reasonable and correct predictions but are considered incorrect because they not annotated in the ground truth. In the second row (Fig. SId-f), the ground truth annotation uses a single box to label two women, which is unreasonable annotation. The model correctly predicts women
17

---

Published as a conference paper at ICLR 2025
and identifies that both women are wearing shirts and shorts, which annotations absent in the ground truth but are semantically accurate.
D DISCUSSION
Limitation Analysis. A significant limitation of the current Hydra-SGa is its inability to predict object and relation categories beyond a predefined closed set. The algorithm can only make predic- tions for object and relation classes that have been explicitly defined and included in the training dataset. This constraint means that the model lacks the capability to recognize infer novel object types or relationship categories that were not present during the training phase. Such a limitation restricts the model's applicability in real-world scenarios, where encountering previously unseen objects or relations is common. In addition, since the current One-to anny Relation Assignment rules are manually designed, they may lack flexibility and potentially introduce some noise the training process.
Societal Impact. Hydra-SGG has the potential to significantly enhance autonomous driving sys- tems, thereby improving road safety and efficiency. For instance, when integrated into self-driving vehicles, Hydra-SGG can provide deeper understanding of complex traffic scenarios. Beyond merely identifying vehicles, pedestrians, and road signs, it can comprehend the spatial relationships and potential interactions between these elements. This advanced scene understanding could enable autonomous vehicles to more accurately predict the behavior of other road users, leading to safer and more efficient navigation in diverse traffic conditions.
Future Work. A promising direction for future rescarch is to extend the current model towards open vocabulary SGG. This advancement would allow Hydra-SGG to recognize and predict objects and relationships beyond the predefined closed set of categories used in training.
As mentioned in the limitations, since the current One-to-Many Many Relation Assignment rules are man- ually designed, they may introduce noise during training that affects learning. Therefore, another direction for future work could focus on designing better one-to-many strategies to further improve performance. For instance, could develop feature composition module to generate new relation triplets, or we can incorporate a relationship refine module to transfer coarse coarse-grained relations into fine-grained relations.
18


Reviews:
[{"review_id": "z0bBc1OgI3", "reviewer": "Reviewer_o4Ve", "summary": "The manuscript introduces Hydra-SGG, a novel one-stage Scene Graph Generation (SGG) method designed to address the challenges of sparse supervision and false negative samples inherent in DETR-based SGG models. The authors propose a Hybrid Relation Assignment strategy that combines One-to-One and IoU-based One-to-Many Relation Assignments to increase positive training samples and mitigate sparse supervision issues. Additionally, the paper introduces Hydra Branch, an auxiliary decoder without self-attention layers, to enhance the One-to-Many Relation Assignment by promoting duplicate relation predictions among different queries. The model achieves state-of-the-art performance on multiple datasets, including VG150, Open Images V6, and GQA, with a significant reduction in training epochs compared to previous methods.", "questions": "please refer to the weakness part.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "**1.Clear Explanation of Methodology:** The paper does an excellent job of explaining the methodology in a step-by-step fashion. The One-to-One Assignment, One-to-Many Assignment, and HydraBranch are each discussed with sufficient detail, making it easier for readers to follow the technical contributions. Furthermore, the inclusion of pseudocode and diagrams helps clarify how each component interacts within the framework.\n\n**2.Comprehensive Experimental Validation:** The paper provides extensive experiments on two widely-used datasets, Visual Genome, OpenImage, GQA, and demonstrates that Hydra-SGG, consistently outperforms existing baselines. The authors also conduct detailed ablation studies, examining the impact of various components (Threshold T, Training Epoch, Number of Relation Queries). The results substantiate the claim that the framework is a substantial improvement over current model.", "weaknesses": "**1.Lack of Novelty:** The motivation and methodology of this work closely resemble those presented in [3], raising questions about its originality and distinct contributions. In particular, [3] introduced the one-to-many relation query matching technique, known as Quality-Aware Multi-Assignment, which enhances training efficiency by allowing a single ground truth (GT) to correspond to multiple sufficiently close predictions or queries. While this work makes a slight modification by removing self-attention among relation queries, this adjustment alone does not constitute a significant advancement in one-stage SGG research. As a result, it is difficult to regard this work as a notably innovative contribution. Further discussion is warranted to clarify how this work differentiates itself from existing approaches.\n**2.Perverse Experimental Results:** The rationale for assigning ground truth (GT) to multiple sub-optimal queries is based on label softening and knowledge distillation. However, applying this approach to scene graph datasets with long-tailed distributions (e.g., Visual Genome, OpenImage, PSG) introduces significant challenges. This method tends to overemphasize common relationships (e.g., on, of, has) in the annotations, which can lead to model overfitting on these frequent relationships. Consequently, this emphasis on common relationships undermines the model’s ability to accurately capture rare relationships (e.g., standing on, walking on, lying on), ultimately lowering average recall performance. Given these limitations, a decline in mRecall performance would be expected for such datasets. Surprisingly, however, the experimental results in Table 1 show the opposite. Since this study does not incorporate any specific solutions to address data imbalance, the observed improvement in mRecall is unexpected. This outcome appears to contradict the foundational assumptions of the proposed Hydra-SGG approach, indicating a need for further analysis.\n\n**3.Inadequate Model Performance:** Despite showing a substantially lower R@100 score than the DSGG method [6], the Hydra-SGG model demonstrates a corresponding improvement in the mR@100 metric. Based on prior research [5], incorporating the F-Recall metric would provide a more comprehensive assessment of Hydra-SGG’s effectiveness relative to state-of-the-art approaches. Additionally, the paper lacks a thorough comparison with other relevant studies [2].\n\n**Ref:**\n\n[1] Pair then Relation: Pair-Net for Panoptic Scene Graph Generation. Arxiv 2023\n\n[2] Structured Sparse R-CNN for Direct Scene Graph Generation. CVPR 2022\n\n[3] Groupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detection. CVPR 2024\n\n[4] Not All Relations are Equal:Mining Informative Labels for Scene Graph Generation. CVPR 2021\n\n[5] Fine-Grained Scene Graph Generation with Data Transfer. ECCV 2022\n\n[6] DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation. CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CywOB1ihww", "reviewer": "Reviewer_KZeA", "summary": "This paper addresses key challenges in the field of one-stage Scene Graph Generation (SGG), particularly focusing on sparse supervision and false negatives samples.To tackle these challenges, the authors propose two main contributions:\n(1) Hybrid Relation Assignment: This method combines One-to-One and One-to-Many Relation Assignment strategies, increasing the number of positive samples during training and thus enhancing supervision.\n(2) Hydra Branch: An auxiliary decoder branch without self-attention layers, designed to further support the One-to-Many Relation Assignment by promoting consistent relation predictions across queries.\nAbove methods jointly improve the training efficacy and performance of DETR-based SGG models, leading to state-of-the-art results on multiple benchmarks with significantly fewer training epochs. Overall, this paper’s motivation is clear, and the proposed method iseffective.", "questions": "Does faster convergence come from more diverse samples or more training data within an epoch, because the proposed method is likely to give 16 images, but will generate more relation training pairs for training?\n\nIt would be better to compare the methods with unbiased SGG methods or adopted unbiased strategies, considering the higher performance of mean Recall.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) Innovative Method Design: The proposed Hybrid Relation Assignment effectively combines One-to-One and One-to-Many matching strategies. This design helps mitigate the sparse supervision issue in existing methods and improves model convergence speed and learning efficiency.\n(2) Efficient Auxiliary Branch: The Hydra Branch supports One-to-Many assignment by promoting consistent relation predictions across queries, all without adding inference-time computational costs. \n(3) Good Experimental Results: The experimental results show that Hydra-SGG achieves state-of-the-art performance across multiple datasets (VG150, Open Images V6, and GQA), reaching these results in significantly fewer training epochs.", "weaknesses": "(1) Lacking theory/analysis, technical solutions appear to be intuitive and empirical. Faster convergence may come from more diverse samples or more training data within an epoch, because the proposed method is likely to give 16 images, but will generate more relation training pairs for training.\n\n(2) Limited Exploration. Why removing Self-Attention benefits the One-to-Many Relation Assignment. The paper introduces the Hydra Branch without self-attention to support the One-to-Many relation assignment. However, the explanation of why removing self-attention aids this assignment is somewhat surface-level, lacking an in-depth analysis of the interaction between self-attention and One-to-Many assignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5neFPnahLl", "reviewer": "Reviewer_ARZw", "summary": "This paper proposes a new method called Hydra-SGG for Scene Graph Generation (SGG), which addresses the challenges of sparse supervision and false negatives in one-stage SGG. To this end, it employs a one-to-many assignment strategy and a transformer decoder without self-attention called a hydra-branch. Through comprehensive experiments across various datasets and detailed ablation studies, the paper confirms the efficiency and effectiveness of this method.", "questions": "1. While the One-to-Many alignment approach helps mitigate sparse supervision, there is a concern that it may introduce significant noise, potentially leading to an increase in false positives. What are some strategies to manage this noise effectively?\n2. How would performances be impacted if the weights for One-to-One and One-to-Many alignment losses were changed? For example, I'm curious how performance would change if the weight of One-to-Many alignment loss were increased or decreased.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. This paper pinpoints the critical problems in one-stage SGG, which are sparse supervision and false negatives. By addressing this problem, the proposed method is superior on efficiency and effectiveness.\n2. The author conducts extensive experiments, including effectiveness on diverse datasets and ablation studies, thereby validating the superiority of the proposed method. Especially, it significantly reduces the training epoch compared to the baselines.\n3. The proposed method is simple yet effective.", "weaknesses": "The paper lacks discussion on its limitations and future works.\n\nThere are no critical points for rejection.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)