Single-Stage Visual Relationship Learning using Conditional Queries
Authors: Alakh Desai
Tz-Ying Wu
Subarna Tripathi
Nuno Vasconcelos
Venue: NeurIPS 2022 Accept (2022)
Keywords: Scene graph generation
DETR
Transformer

Abstract: Research in scene graph generation (SGG) usually considers two-stage models, that is, detecting a set of entities, followed by combining them and labeling all possible relationships. While showing promising results, the pipeline structure induces large parameter and computation overhead, and typically hinders end-to-end optimizations. To address this, recent research attempts to train single-stage models that are more computationally efficient. With the advent of DETR, a set-based detection model, one-stage models attempt to predict a set of subject-predicate-object triplets directly in a single shot. However, SGG is inherently a multi-task learning problem that requires modeling entity and predicate distributions simultaneously. In this paper, we propose Transformers with conditional queries for SGG, namely, TraCQ with a new formulation for SGG that avoids the multi-task learning problem and the combinatorial entity pair distribution. We employ a DETR-based encoder-decoder design and leverage conditional queries to significantly reduce the entity label space as well, which leads to 20% fewer parameters compared to state-of-the-art one-stage models. Experimental results show that TraCQ not only outperforms existing single-stage scene graph generation methods, it also beats state-of-the-art two-stage methods on the Visual Genome dataset, yet is capable of end-to-end training and faster inference. 

Single-Stage Visual Relationship Learning using Conditional Queries
Alakh Desai!, Tz- Ying Subarna Tripathi', Nuno Vasconcelos University of California San Diego, USA Intel Labs, USA
Abstract
Research in scene graph generation (SGG) usually considers two-stage models, that is, detecting a set of entities, followed by combining them and labelling all possible relationships. While showing promising results, the pipeline structure induces large parameter and computation overhead, and typically hinders end-to-end optimiza- tions. To address this, recent research attempts to train single-stage models that are computationally efficient. With the advent of DETR 31, a set based detection triplets
model, one-stage models attempt to predict a set of subject-prediiate--be directly in single shot. However, SGG is inherently a multi-task learning problem that requires modeling entity and predicate distributions simultaneously. In this paper, we propose Transformers with conditional queries for SGG, namely, TraCQ with a new formulation for SGG that avoids the multi-task learning problem and the combinatorial entity pair distribution. We employ DETR- -based mncoder-decoden design and leverage conditional queries to significantly reduce the entity label space as well, which leads to 20% less parameters compared to saate-of-hhe-ar stage models. Experimental results show that TraCQ not only outperforms existing single-stage scene graph generation methods, it also beats many state-of- the-art two-stage methods on Visual Genome dataset, yet is capable of end-to-end training and faster inference.
Introduction
Scene graph generation (SGG) provides structured representation of visual relations in a scene. A scene graph set of subject-predicate-obiiee
triplets, where subjects and objects are entity nodes in the graph and predicates are edges representing the relationship between pairs of entities. Due to its compact structure, SGG has been adopted as a foundational step for several high-level machine
cognition tasks, including caption generation 48
47
29]. visual question answering [13] 30 image
However, SGG far from a trivial problem, due to the complexity of detecting and pairing subject-object entities and inferring the predicate between each subject-object pair, requiring the model to excel at both entity and relationship detection. Hence, most research in the problem decomposes into two separate sub-tasks, entity detection and predicate detection, which are modeled sequentially. This leverages success in object detection
retrieval [4 38 and image generation 15
22
36
31 and
leads to the predominance of two-stage networks in the literature 4
52
35
34
21
45 25)
where an object detector such as the faster-RCNN [31] solves the first sub-task, reducing the SGG modeling to the second stage that addresses the pairing of subject-object entities and classifying of predicate categories. The predicate detection part is typically achieved by ranking the (N2) triplet proposals generated by the exhaustive pairing of N entity predictions. While these models have shown promising results, the pipeline structure usually comes with significant parameter and computation overhead, which hinders end-to-end SGG optimization under memory constraints, and leads to sub-optimal solutions.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

To enable end-to-end optimization, some recent works introduce one-stage models based on point- based detection [26] and transformer-baset detectors 33] 20). Since SGG is inherently multi-task learning problem, the main idea is to model the entity detection and predicate detection in parallel, which can be done in multiple ways. These models, however, face the problem of having to learn joint feature space for two very different tasks. To circumvent the difficulty of this problem, they disentangle entity detection and predicate classification by training separate decoder branches for the two tasks. However, we hypothesize that this form of disentangiement is too strong for SGG, since capturing the interactions in a scene requires three kinds of features: fe tuned for entity detection (object detection), fp tuned for predicate classification, and fs tuned for capturing the relationships between entities and their surroundings. While fe can be learned well under a complete disentanglement of the two tasks, fo is hard to capture without some amount of feature tuning for entity detection. In general, the task of predicate classification is what drives the learning of the interaction features. Therefore, fp and fa rely heavily each other and benefit from some amount of coupled learning.
With this in mind, we propose Transformers with conditional queries (TraCQ), with new formulation for scene graph generation. Instead of trying to learn fe and fi together the entity detection branch, TraCQ learns fp and fi together in predicate detection branch H. which then conditions the learning of fe by a separate entity refinement module C. A weak coupling between predicate and entity detection is then ensured by forcing H. to learn f. through the prediction of loose estimates of subject and object bounding boxes. major effect of this formulation is that the distribution that H learns is conditioned the non-combinatorial predicate space, while typical SGG models disentangle the learning of fp from (fi, fe) where the entity decoder is expected to learn a distribution based on the combinatorial (E E) space. Given that we have limited features and (fi, fp) are closely related, it easier to learn from the non-combinatorial predicate space than the OE x E) entity pair space. This reduction in distribution space implies that simpler and smaller model can be used for SGG,
Overall, this paper makes the following contributions. First, we introduce a new formulation of the single-stage SGG task that avoids the multi-task learning problem and the combinatorial entity pair space. Second, we propose a novel architecture, Transformers with conditional queries (TraCQ), wherein conditional queries are leveraged to significantly reduce the inference time and the distribution space to learn, which leads to 20% less parameters compared to state-of-the-an single-stage models. Finally, we show that TraCQ achieves improved SGG performance than otate-of-the-ar methods on Visual Genome dataset, yet is capable of end-to-end training and faster inference.
2 Preliminaries
Transformers [37] powerful models for sequence modeling, based an encoder-decoder architecture. Both the encoder and the decoder modules stack multiple attention based blocks, differing on the type of attention mechanisms employed. Encoding blocks perform self-attention across input tokens, while decoding blocks perform cross-attention between encoder output and predictions. The attention operation is defined over a set of queries Q Rnxd, keys K R and values V E Ruxds, where dg = dR, according to
QK Vdk
Attention( Q K, V) Softmax
V.
Information from different representation subspaces is learned at different positions with resort to multihead attention based on different attention heads,
MultiHead Q K, V) = Concat(head), .... ,.....,hh,a,,,.. head,,) ) WO head; Attention QW? KW VW!),
where wo E 1gdo
(2)
where ? W. K W are the parameters of the tt head. This is followed by a normalization layer 2 (LN) with residual connection in each block. We denote the entire module the attention stack, Attr (Q, K, V), of queries Q, keys K and values V, moving forward.
DETR 3 is a Transformer-based object detector. A CNN backbone first generates fea- ture tensor F5 E H xd for an image I. An encoder then learns context features F, Attn (Qg. KE, VE) E _xdd where L H x W, QE Kg Vg (latten(Fs)) + Ep.

---

Single Decoder (SD)
Dual Decoders (DD)
Dual DETR (DDTR)
-FNN (8,
FFN
I
Enc
FFN (D
FEN
Decp
FFN
Encp
Decp
FFN
(((*))
FEN F
SPO
I
Enc
Q
SPO
Qp
SPO
Q+ +
Dec
FFN
Dece
FEN
Ence
Dece
FEN [g)mo))
FFN
+FFN (pra1)
FFN
(1)
Qe
Figure Baselines. Enc, Dec, and M denote the encoder, decoder, and Hungarian matching respectively. (-)p represents that the instance is for predicate detection, and (-)e is for entity detection. SPO denotes triplets of <subjjec..rediiaat..ojjee
and Epp E R.od is a fixed positional encoding. The decoder transforms a set of entity queries Qentity R.Nxdde into entity representations
Zg Atth Qentitys Qentity: F.) R RNxx
(4)
Separate feed forward networks (FFNs) are finally used predict each entity è = (6, é), composed by class label and a bounding box 6, from Zg.
DETR adopts a set prediction loss for entity detection, which performs bipartite matching between the ground truth set Y and the predicted set Y with the Hungarian matching algorithm [9] i.c.
N. Σ
Ô = arg min OCGNe
Cmatos (E)()) eq),
(5)
where ON. denotes the set of permutations of No predicted entities, E; - ((i, b:) E Y, c, indicate target class and box coordinates respectively, (a( E E.,
Cmaach (ē, -1 (c + { Lebor (b,
and p is the logit for class e of entity Note that Y is padded with background tokens ø. Finally, the set prediction loss is formulated as,
N. (os))(0()) + [ec Co (ba(s) b:)l,
LDERR
where Cets denotes the cross-entropy loss for label classification, Loow consists of L1 and generalized IoU loss [32] for box coordinates regression.
In this work, we adopt modified attention mechanism, namely Poll and Pool attention from PnP. (ETR[39), which abstracts the image feature map into fine foreground object feature vectors and small number of coarse background contextual feature vectors. We use PnP-DETR for faster convergence and computation efficiency.
3 Toward Single-Stage End-to-End Scene Graph Generation
3.1 Two-stage models
Given an image I, scene graph generation produces scene graph G to describe visual relations in scene. consists of a set entity vertices E, typically objects, and set of directed edges P representing predicates, typically relationships between objects such as "to the left of." Each vertex E is a tuple consisting each edge in P denotes the predicate label between a pair of entities. Each edge and its connected vertices form a relation tuple < (bawh, =dd) - p (bob), nbi) >, which is alternatively called subjject-rreiccttt-hheee (SPO) triplet, e.g. man in region Jaub, to the left of a car in region 3ob3 This is complex task involving entity and predicate detection, which requires to model both entity and predicate distributions. Prior research [52 6 usually models the problem as
the bounding box b and the class label y of an entity instance, while
Pr Pr(y, B (r
(8)

---

Table I: Preliminary experiments. The architectures of these baselines are presented in Figure
mean Recall (1)
#params ())
Inference time (4)
Model
@20
@50
@100
(M)
(sec)
SD
5.7
6.2
6.3
41.7M
0.063
DD
6,5
6.9
7.0
51.1M
0.068
DDTR
9.2
11.8
13.0
82.9M
0.220
where y'= [ and B -- A state-of-the-art object detector 7 31 is then used to handle the first component, reducing the SGG problem to the second. While this two-stage formulation has produced good results, it leads to quite slow inference, since it requires an exhaustive search through the space of ((N2). In addition, two-stage models usually have large parameter and computation overhead, which inhibits end-to-end SGG optimization, and makes the overall solution sub-optimal.
3.2 Learning scene graph generation with set predictions
With the advent of DETR 3 there has been shift single-stage end-to-end SGG learning, by adoption of a Transformer-based encoder-decoder architecture for SPO triplet set predictions. Several models have been proposed [3] 33 20]. While efficient, these methods tend have a weaker performance than two-stage models. We hypothesize that this is due to the entanglement between the feature spaces used to represent entities and predicates. This entanglement is unavoidable, since the detection of predicate always requires some knowledge of the associated subject and object. However, a very strong entanglement is undesirable, because predicates and entities have very different distributions. Since the individual distributions of predicates and entities are long-tailed, the joint distribution of their pairs is extremely long-tailed. Hence, most pairs are very poorly represented in highly entangled feature space.
We next test this hypothesis by performing some preliminary experiments on a of baseline models of varying levels of disentanglemer between the entity and predicate feature spaces.
Single decoder (SD) uses pair of encoder and decoder modules, followed by five FFNs to decode each element < (baub, yzub) p - (beb), yd > of the SPO tuple. The encoder learns a context feature tensor from the input image I, and the decoder takes these features and random queries Q to generate the FFN input. Since it generates the whole SPO triplet at once, this model learns a feature space where entities and predicates highly entangled.
Dual decoders (DD) consists of shared encoder and dual decoders for predicate and entity detection respectively, where the former decodes < boub -p- 1obj > tuples and the latter decodes < you yo > pairs. By feeding common set of random queries Q to the two decoders, we ensure that no matching between the two output sets is needed. This is important to keep complexity competitive with that of SD. While introducing some additional cost, the use of two separate decoders encourages the disentanglement of the predicate and entity feature spaces. However, two architecture components still encourage entanglement: 1) the use of a common encoder and 2) the sharing of queries by the two decoders.
Dual DETRs (DDTR) avoids these problems by introducing separate DETR models, each with its own encoder, decoder and random queries, for detecting predicates and entities respectively. Similar to DD, one model decodes < jeab p - jobj >, and the other < su yob >. Since each DETR has an individual set of random queries, brute-forced matching between the two sets of outputs is needed to generate the detected graph. Assuming M predicates and entity predictions, there are M x P.N SPO tuples. Generating the cost matrix between the two sets is computationally expensive. However, this model also has the weakest entanglement between feature spaces.
In table we compare the performance of these three baselines, in terms of mean Recall (see Section 5.1) on the Visual Genome (VG) [18] dataset. The number of parameters and the inference time are also provided. It is clear that disentangling predicate and entity representations is good strategy for solving SGG. However, the computational cost of disjoint queries quite high, including more parameters and lower inference speed. The question is whether the DD architecture can be improved achieve performance closer to DDTR, while maintaining low parameter and computation overhead. To accomplish this, next propose model, TraCQ, based on the DD architecture. TraCQ

---

Zp
Detection Heads
FD
CNN
Feature Encoder f
Predicate Decoder H
V2p 2p
P FFN sub
FFN
E Epos
p
pp
FFN bobj
(
Zee
Vz oe
p
E mb
Entity Refinement C
FFN
5e
5e
Emba
FFN
be
T addition
Ve
(sub, abj3
QCE
Figure 2: Model architecture of TraCQ.
reduces entanglement by replacing the shared queries of DD with conditional queries and leveraging decoupled training for predicates and entities.
4 Transformers with Conditional Queries
In this section, we introduce the proposed architecture, Transformers with conditional queries (TraCQ).
Architecture As shown in Figure the TraCQ model employs the encoder-decoder Transformer design, composed of a feature extractor F, predicate decoder H. and conditional entity refinement decoder C. The predicate decoder H predicts set Spred of < b p Jobs > tuples. The refinement decoder C predicts entity class labels and refines the bounding boxes detected by H. to predict- < (boub, y= - (bobi, زوو) > pairs. The idea that, the use of C to refine the entity bounding boxes, frees H from solving this task with high accuracy. Hence, the feature space of H has to less representative entities, enhancing its disentanglement from the space of predicates. The details of each component described below.
Feature Extractor The feature encoder F is transformer encoder that takes feature tensor F;. extracted from the image by the CNN backbone and a fixed positional encoding EPos and implements the self-atten self-attention stack to produce feature tensor F, that encodes the global image context. To ensure that Fp emphasizes attention on scene relations, F is trained jointly with H. and then frozen for the training of C. This results in features that optimal for predicate detection. These features are richer than those for entity detection, since predicate detection involves an understanding of the constituent entities in order to detect the relationship between them and even captures the interplay between the entities in the scene which difficult to learn if F was trained with C.
The feature F, is used as the K and V values for both decoders H and C. This module, therefore, acts as the common encoder for both decoders.
Predicate Detection The predicate detection module H lies at the heart of TraCQ. This module comprises of decoder which complements the encoder from F. Similar to entity detector, transforms a set of predicate queries Op E N. into the predicate representations Zp Attn Qp. Qp: Fp) E Xd Hence, the composition F and H is equivalent to that of DETR. However, Zp is fed to three distinct multi-layer FFNs that learn to predict the bounding boxes joub and jobj and the predicate class p.
In this way, learns to predict not only the predicate label but also a pair of bounding boxes for the subject and the object. This requires H to both learn to localize related entities and understand the relationship between them. However, predicting two bounding boxes along with the predicate label is far from trivial, due to the entanglement problems discussed above. To reduce entanglement, H learns to only roughly localize the subject and object, leaving the exact bounding box localization to C. Since the key goal of the DETR- like combination of F and H. is to predict predicate categories

---

accurately, it learns a visually rich feature space for the predicate distribution. Overall, H. predicts a set Spred of Np 3-tuples < boub P. Jobs >, which is fed into the entity refinement module C.
Conditioned Entity Refinement C While not accurate, the bounding boxes {b-ub, Babjj received from H. provide an initial estimate of entity locations. The entity refinement model, C, leverages these estimates to propose set of Nee refined bounding boxes {heus, _ conditioned on the predicate label estimate p. This is implemented with a second decoder that shares the features Fp produced by the feature extractor and a set of queries designed to induce the conditional operation.
For each bounding box estimate in Spred: C predicts Noe refined entity bounding boxes. The queries used to generate these refinements are
Q Eb) + (p) + Q6" where i {1. ..., Noe (mbs(.) and (m are two embeddings and Qu is a DETR-like randomly generated set of Nee queries, introduced to guarantee that the queries Qfe are distinct. From the conditional queries Que- C produces Nee distinct corrections by computing representations Zee = Attn (Que: Que: Fp) E Reuxd xd_ _ It follows from Ieub and jobi This constraint encourages to predict boxes in a substantially smaller region, thereby limiting the box search space. Furthermore, because attention is modulated by the predicted predicate label p. this search is selective for boxes that comply with the former, which encourages agreement between entities and predicates, thus limiting the entity label search space.
(9)
that the queries Que bias the attention of C' to the bounding boxes
For each bounding box in Spred: this process produces Nee entity refinements, for a total of N ce 5-tuples. ls 5-tuples are chosen per predicate in Spred be part of the predicted scene graph G, which contains & x Np scene graph nodes.
Training TraCQ is trained with set prediction loss for triplet detection that generalizes the entity detection set prediction loss of 7 Denote by du =< baub, D, jobj > the prediction of H. by VN =< baud p, jobj > its groundtruth, by üic =< ;r, i >> the entity correction of C and by Vc =< y², be > its groundtruth. TraCQ predicts N₂ relationships, where Np is larger than the number of relations in any given image. Similarly to DETR, this is handled by padding the ground truth set of relations with no-relation tokens ø. This circumvents a difficulty of single-stage models, which must account for the fact that relations of the types < valid subj- rel valid object> are different from those of the type < no subj - no rel - no object >, or other combinations. The predicate detection matching cost Cmaach (fH, V/u) between predicted and ground truth triplet generalizes the matching cost function of (6), considering both the predicate class prediction and the similarity of predicted b and ground truth subject/object boxes, according to
Cmatch (gH. yx) - (r (pt0) [Crom (6 bouh) + Looz (6oh), Bohi)]
(10)
Similarly, the entity refinement matching cost Cmater (úe, Nc) uses both entity boxes and labels to calculate similarity, according to
Crarchh (fc, Mc) - (v PV (vr Stoz (i=, (b)
(11)
where the groundtruth entity e is either the subject (sub) or object (obj) entity of the corresponding prediction by H. For both subject and object, the box alignment cost is
(bo b) =
b) + XL, LL, (b.6)
Given the triplet cost matrix Cmatch, the Hungarian a algorithm 199 is executed for the bipartite matching and each ground truth triplet i is assigned to a predicted triplet o Let the ōy be the matching of H. and de that of C. Two losses are then defined as
N₂
Lp =
Auna (P5.P)) + I (prtt)) (Cron (bgub 1ouh) + Looo Lig (
1)=0x(0) (12)
No
Noc Σ
Le =
Σ (po
Frow Cate vf) + (ug Crow (bg, b9) Ik=
(13)
where Lots is the cross-entropy loss. F and H are trained using C,. whereas C is trained with L.. This encourages the decoupling of feature spaces, encouraging the features of and to specialize in predicate prediction and those of C on predicate-conditionedd entity detection.

---

Inference
The inference stage involves combining the outputs of entity refinement FFNs of C and the predicate detection FFNs of H. to form final triplets. Due to the conditional structure of the decoder, the predicate detection outputs and the set of k entity refinement outputs have a one-to-one correspondence. Therefore, k Np SPO-tuples < (beub, yy pp (bol), yood > are automatically generated.
An SGG triplet score Striplet is computed per SPO-tuple with Striplet Spred8 Sob where Saub and Sobj are the scores of subject and object classification from C, respectively, and Spred is the predicate classification score from H. Finally, the predictions are sorted by descending Striplet and the top < kNp predictions selected as nodes of the predicted scene graph G.
Comparison to previous approaches the model S by
When compared to two-stage SGG models, TraCQ replaces
Pr(g = E PPPOPBBPPPOOBBBPP..
(14)
where B is the set of bounding box estimates computed by H and the marginalization over B is performed by the conditioning of the queries of C on the predictions of H. Assume, for simplicity, that the labels y. P are independent of the bounding boxes B. Then, (8) reduces to Pr(g
Pr()]I))r((C Pr Pr(BBI) Pr())) and its modeling complexity is dominated by the term Pr(P))', I),
which a distribution conditioned the large combinatorial space of O x E) entity pairs. On and its complexity is
the other hand, (1)) reduces to Pr(GIZ) determined by PrOj|,, I). which a distribution conditioned on the non non-combinatorial space of predicates. Hence, TraCQ has an innate advantage over the two-stage models, the modeling of probabilities conditioned on much smaller spaces, which allows it to perform inference much more efficiently. Furthermore, the conditional queries Q enforce something akin to ROI pooling, highly constraining the refinements made by C. First, these are constrained to the region of the bounding boxes predicted by H. Second, the queries are generated with the predicate label well and therefore must learn only those entities that are consistent with this label.
Pr
When compared single-stage models, TraCQ has the advantage of a feature representation where entities and predicates are less entangled, due to the training of the two decoders with different loss functions, and the use of decoder C to predict the entity labels and refine the bounding boxes predicted by H. This places less stress on the training of F and H to produce high quality features for entity localization and encourages C to produce such features.
In other one-stage models, which predict combinations of entities and decouple just the predicate classification from entity classification, observe that the features are not fully disentangled leading to drop in performance. The disentanglement induced by the predicate detection task reduces the feature representation space needed to detect the predicates and allows the use of lightweight model to learn it.
5 Experiments
In this section, we present the results of TraCQ and ablates its components.
5.1 Settings
Dataset. Visual Genome (VG) [8] is a popular benchmark for SGG. While it is a large dataset containing 75k object categories and 37k predicate categories, both object and predicate distributions are highly long-tailed, where most of the categories only have few instances. Hence, popular subset VG150 of VG is proposed by 4 which contains the most frequent 150 object classes and 50 predicate classes. We follow the setting of prior works, adopting VG150 in all the experiments.
Metrics. Since the annotations for SGG datasets are usually incomplete, early works in SGG are mostly evaluated with Recall (R @ K). However, since SGG a highly long-tailed problem, most recent research 4 34 also present mean Recall@K (R @ K), K {20, 50, 100} as the metric.
7

---

Table 2: Quantitative results of TraCQ in comparison with stae-off-the-an methods the VG dataset. TraCQ achieves new tate-of-the-a results outperforming existing one-stage and two-stage models in most metrics, while reducing model complexity without the need for any extra features (e.g.. glove vector, knowledge graph, etc.). Note that indicates that the corresponding results are unavailable.
Extra Features
mean Recall (T)
Recall (f)
#params ())
Backbone
@20
(M)
MOTIFS 52
X-101FPN
4.2
6.6
21.4
27.2
30.5
240.7
Two- Stage
KERNE 4
VGG16
6.4
7.3
22.3
27.1
405.2
GPS 25
VGG16
BGNN 2
X-101FPN
7.5
10.7
13.6
23.3
31.0
34.6
341.9
VCTree- TDE
X-101FPN
11.1
19.6
23.2
360.8
IMP- (43
VGG16
2.9
3.8
4.8
14.6
20.7
24.5
203.8
G-RCNN 45
X
VGG16
11.4
13.7
FCSGG 26
X
HRNerW48 55 FPN
2.7
3.6
4.2
16.1
21.3
25.1
87.1
One Stage
Re]TR[5
ResNet-50
25.2
Relationformer
ResNet-50
4.6
9.3
10.7
22.2
28.4
31.3
92.9
TraCQ (ours)
ResNet-50
12.0
13.8
14.6
19.7
28.3
35.7
51.2
5.2
Implementation details
The PnP-DETR (39] is set with sample ratio a of 0.33 and pool samples
We adopt ResNet- 50 [10] with 6-layer transformer encoder the visual feature extractor F. Both decoders H. and have a 6-layer transformer, with and attention heads respectively. The number of queries Np is set to 200 and No. is set to 10 with & 5. The FFNs for bounding box prediction have linear layers with ReLU, while the FFNs for predicting object and relation labels have one linear layer.
We initialize the network with the parameters of PnP-DETR trained on VG for object detection, and adopt the default weight coefficients λµΙ- AcTOU and A from [39. The network is optimized with AdamW W [27 where the weight decay is 10 The learning rates for training the backbone and the rest modules are 10-3 and 10- respectively. All experiments are conducted on 8 NVIDIA TITAN X GPUs, with total batch size of 32.
5.3 Comparisons to baselines
The baselines in Table shoow the importance of disentanglement, where the performance increase from SD to DDTR. However, disjoint queries lead to large computational overhead due to the post-processing matching step and there is no way for further joint optimization. Learning from these baselines, TraCQ is DD-type model leveraging decoupled training disentangle the two distributions representation learning while maintaining certain degree of dependence between the two tasks. As result, TraCQ outperforms all these baselines with large margin (2.8 point gain over DDTR mR 20), while maintaining similar number of parameters as DD and achieving 1.5x faster inference than DDTR. This highlights that complete disentanglement not possible, since the detection of a predicate always requires some knowledge of the associated subject and object.
5.4 Comparisons to prior methods
We compare scores of @@K mR @ and number of parameters of TraCQ with several two-stage and one-stage models in Table The usage of extra features such as semantic and statistic information by the model is indicated to distinguish those models from visual appearance based models. TraCQ beats all models in terms of mR K while having 20% fewer parameters than [5], reportedly having the smallest model size. It also exhibits a much less drastic drop in the mean recall values when we go from 100 to 20 for mR @ K. This shows that the model is more confident about it's predictions than others. In @@20 and R @5, TraCQ lags behind only those two-stage models, that are heavier and/or use extra features in their pipeline. TracQ is the first one-stage model that beats two-stage models in terms of performance with much smaller model size.
5.5 Ablation studies
We perform the following ablations TraCQ to understand the influence of each component.

---

Table 3: Ablations on the orders of entity detection and predicate detection.
Table 4: Ablation the hyper-parameter k. SGG is evaluated with mR 20 and @@20
mcan Recall (f)
#Predictions (1)
m))
R (f)
Model
@0
@5
@ 100
200
8.9
18.4
1000
12.0
19.7
Entity-first
11.2
12.3
12.7
2000
12.1
19.7
Predicate-first (ours)
12.0
13.8
14.6
15
3000
12.1
19.7
Table 5: Ablation on bounding box refinements.
Table Ablation on conditioned queries.
Entity box prediction e
mR@ @ 20 / 100
Conditioned queries
mR@ @ 30/50/ 100
From H (no correction)
17.2 18.0/ 18.1
w/o Embp (P) Qce (ours)
10.8 / 12.4 13.1
From C (ours)
16.9 18.6/ 19.2
12.0 / 13.8 14.6
Ablations on the formulation To validate the effectiveness of the proposed formulation, also conduct the experiment of the entity-first variant. Different from TraCQ that predicts entity labels based predicate predictions, this variant performs entity detection first and conditions predicate classification on it. We maintain similar setup as TraCQ on other components to ensure fair comparison. Table 3 shows that this variant underperforms the proposed formulation. This supports our idea of avoiding the combinatorial entity label space.
Ablations the hyperparameter k The hyperparameter k controls the number of candidate entity bounding boxes. We evaluate TraCQ with different values of k and present the mR @ 20 and R 20 in Table We that the performance saturates quickly as the number of predictions (denoted as #Predictions) increases. Since the inference time is proportional to k, we pick k = 5.
Ablations on bounding box refinements The entity refinement module. C. performs entity detec- tion using the global features and the conditioned queries entity bounding boxes and predicates. To study how much box refinement was happening from H to C, directly take the predicted bounding box of subject and object, i.e. jaub and Johj from the predicate decoder 76 as the final prediction, but not correct the entity bounding boxes with C. Since H does not produce class labels for subjects and objects, evaluate the network in the task of Predicate Detection (PredDet), where ground-truth entity labels are given and the task is to predict < joub results compared to TraCQ under the mR @ metric, which shows that the model performance drops when removing the entity refinement modulc.
jobs >> tuples. Table presents such
Ablations conditioned queries We also ablate the impact of using the predicate labels f for generating the predicate conditioned queries Qce for C. We retrain conditioned only on the bounding boxes from H. namely, removing Emb,((p from 9 The bounding box conditioning cannot be removed, since that reduces C to a general entity set detection model. Table presents the effect of removing the predicate conditioning for C. The performance of the overall model drops by points at mR @20. This shows that the explicit condition forced onto C by the predicate conditioned queries does fact reduce the searching space for inferring entity labels.
6
Related work
Scene graph generation (SGG) has long been studied the literature due to its wide applicability
48
47
3
14
38
15
22
36 15 22 36). Since SGG a highly complex problem, pioneer works [18
43
leverage marginalization to decompose the problem into predicate classification, entity classification and entity localization. This formulation largely impacts the latter research. Most of the works follow the same pipeline structure which first detects set of entity proposals with of-e-oo--he-art object detectors [I] and then classify all the pairs of entity proposals into predicate class at the second stage. While object detection is considered as solved, most research
52
focuses on learning better mapping between entity pairs and predicate classes
54
24
40
42
This can be achieved either by learning better representations for contextual
reasoning
52
49
17 28, leveraging external knowledge 9 51
50 or unbiased
learning strategy
34
25
Among these, Relation Transformer Network [7 and Lu et

---

al 28 are pioneering works adopting Transformer encoder-decoder pair for learning SGG, while both of them rely on bounding box predictions from a CNN-based object detector. These research has provided insightful observations and promising results. However, the two-stage formulation has fundamental problem, the searching space for relation detection always (N2)), which can retard the inference speed. In addition, building another computation module top of a large object detector is inefficient in terms of parameters and computations, which prevents end-to-end optimizations for the SGG task.
To address this, 26 points and relationships as vector fields. Although it is lightweight and fast, it has a significant performance gap compared other two-stage methods. Others adopt DETR [3] -l-based architectures. RelTR[5] combines subject and object queries to decode the triplet predictions, while [33] produces a set of object tokens and relation token, and decodes them respectively. (0) is a concurrent work that requires bipartite graph assembling similar to DDTR. While these works provide inspirational ideas and allows end-to-end optimizations, they still perform poor than two-stage methods. We hypothesize that the reason is because of the entangled feature space of entities and predicates. This paper focuses on a single-stage model with less entanglement between the two distributions.
20] propose one-stage models for SGG. 26 encodes objects as center
Transformer-basee Detection has become popular for many computer vision tasks recently. The pioneering work, DETR [3) introduces the set prediction with Transformer incoder-decoder architecture, where the encoder learns contextual features and the decoder takes object queries to generate object proposals. This drastically changes the way object detection is viewed, and inspires several later works on other tasks, such as instance segmentation [12], visual grounding [[]], multi- object tracking (53 and HOI detection
[16]. HOI detection most similar to our task, SGG, that it localizes and recognizes the relationships of each human-object pair. However, there are inherent differences between these two tasks. The HOI task does not need to deal with the O x E) space since the subject of each interaction tuple is always the human in the scene. Hence, most of the HOI models cannot directly transferred to tackle the complexity of the SGG task.
7 Conclusions
Scene graph generation (SGG) deals with two different distributions, namely entities and predicates. Most methods rely on detecting all entities, followed by combining pairs of entites. Unlike such two-stage methods, we propose end-to-end trainable framework for SGG using Transformer architecture which decouples these two distributions effectively with conditional queries (TraCQ), leading to small model size and an efficient inference mechanism. TraCQ significantly outperforms existing single-stage methods, even beating the performance of many state-of-the-e two-stage methods on Visual Genome benchmark.
8 Societal Impacts
SGG models have wide applicability, e.g. caption generation 48 47 visual question answering 13] The model itself is harmless and helpful for providing a compact description of the scene. However, there may be some privacy concerns when the model utilized for monitoring people and their interactions. Note that explicit recognition is involved in this framework.
Limitations
In this paper, we focus on developing a single-stage detector for visual relations. We extensively study different formulations of SGG models with different degrees of entanglement between the predicate and entity detection task. However, we didn't deliberately aimed at addressing the long-tail nature of the two distributions in this work. The unbiased training techniques like resampling or reweighting can be considered the future work.
Acknowledgments
This work was partially funded by NSF awards IIS1924937 and IIS-2041009, a gift from Amazon, gift from Qualcomm, and NVIDIA GPU donations. We also acknowledge and thank the use of the Nautilus platform for some of the experiments discussed above.
10

---

References
[1] Sherif Abdelkarim, Aniket Agarwal, Panos Achlioptas. Jun Chen, Jiaji Huang. Boyang Li, Kenneth Church, and Mohamed Elhoseiny. Exploring long tail visual relationship recognition with large vocabulary. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15921-15930, 2021.
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagonuyko. End-to-end object detection with transformers. In Computer Vision - ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part page 213-229, Berlin, Heeidelberg, 2020. Springer-Verles Verlag.
Tianshui Chen, Weihao Yu, Riquan Chen, and Liang Lin. Knowledge-embecded routing network for graph generation. Conference on Computer Vision and Pattern Recognition, 2019.
[5] Yuren Cong, Michael Ying Yang. and Bodo Rosenhahn. Reltr: Relation transformer for scene graph generation, 2022.
[6] Alakh Desai, Tz- Ying Wu, Subarna Tripathi, and Nuno Vasconcelos. Learning of visual relations: The devil is in the tails, 2021.
[7] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hicrarchies for accurate object detection and semantic segmentation, 2013.
[8] Nikolaos Gkanatsios, Vassilis Pitsikalis, Petros Koutras, and Petros Maragos. network for scalable scene graph generation. In Proceedings the IEEE/CVF International Conference on Computer Vision Workshops, pages 0-0, 2019.
[9] Juuxiang Gu, Handong Zhao, Zhe Lin, Sheng Li, Jianfei Cai, and Mingyang Ling. Scene graph generation with external knowledge and image reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1969-19788 2019.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEFP Conference Computer Vision and Pattern Recognition (CVPR), 2016.
[11] Chih-Hui Ho, Srikar Appalaraju. Bhavan Jasani, R Manmatha, and Nuno Vasconcelos. Yoro-lightweight end to end visual grounding. In ECCV 2022 Workshop on International Challenge on Compositional and Multimodal Perception, 2022.
[12] Hu, Liujuan Cao, Yao ShengChuan Zhang, Yan Wang, Li, Feiyue Huang, Ling Shao, and Rongrong Ji. Istr: End-to-end instance segmentation with transformers. arXiv preprint arXiv:2105.00637, 2021.
[13] Drew A. Hudson and Christopher D. Manning, GQA: A new dataset for real- worldworld visual reasoning and compositional question answering. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
[14] J. Johnson, R. Krishna, M. Stark, Li, D. Shamma, M. S. Bemstein, and Fei-Fei. Image retrieval using scene graphs. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). volume 00, pages 3668-3678, June 2015.
[15] Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[16] Bumsoo Kim, Junhyun Lec, Jacwoo Kang, Eun-Sol Kim, and Hyunwoo J. Kim. Hotr: End-to-end human-object interaction detection with transformers. In CVPR. IEÉE, 2021.
[17] Rajat Koner, Suprosanna Shit, and Volker Tresp. mmXiv:2004.06195 2020.
Relation transformer network. arXiv preprint
[18] Ranjay Krishna, Yuke Zhu. Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael Bernstein, and Fei-Fei Li. Visual genome: Connecting language and vision using crowdsourced dense image annotations. CoRR, abs/1602.07332. 2016.
[19] Harold W. Kuhn. The hungarian method for the assignment problem. In 50 Years of Integer Progranming, 2010.

---

[20] Rongjie Li, Songyang Zhang, and Xuming He. Sgtr: End-to-end scene graph generation with transformer. 2021.
[21] Rongjie Li, Songyang Zhang, Bo Wan, and Xuming Hc. Bipartite graph network with adaptive message passing for unbiased scene graph generation. In CVPR, 2021.
[22] Yikang Li, Tao Ma, Yeqi Bai, Nan Duan, Sining Wei, and Xiaogang Wang. Pastegan: A semi- semi-paraettric method to generate image from scene graph. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NewrIPS 2019, 8-14, December 2019. Vancouver. BC. Canada, pages 3950-3960, 2019.
[23] Yikang Li, Wanli Ouyang, Bolei Zhou, Jianping Shi, Chao Zhang, and Xiaogang Wang. Factorizable net: an efficient subgraph- baaeepphbaaeeddde based framework for scene graph generation. In Proceedings of the European Conference on Computer Vision (ECCV). pages 335 355-351,220 2018.
[24] Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang. and Xiaogang Wang. Scene graph generation from objects. phrases and region captions. In Proceedings of the IEEE international conference on computer vision, pages 1261 1270, 2017.
[25] Xin Lin, Changxing Ding, Jinquan Zeng, and Dacheng Tao. Gps-net: Graph property sensing network for scene graph generation, 2020.
[26] Hengyue Liu, Ning Yan, Masood Mortazavi, and Bir Bhanu. Fully convolutional scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11546-1156, June 2021.
[27] Ily Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2017.
[28] Yichao Lu, Himanshu Rai, Jason Chang, Boris Knyazev, Guangwei Yu, Shashank Shekhar, Graham W Taylor, and Maksims Volkovs. Context-aware scene graph generation with seq2seq transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15931-159441 2021.
[29] Kien Nguyen, Subarna Tripathi, Bang Du, Tanaya Guha, and Truong Q. Nguyen. In defense of scene graphs for image captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1407-1416, October 2021.
[30] Tianwen Qian, Jingjing Chen. Shaoxiang Chen, Bo Wu, and Yu-Gang Jiang. Scene graph refinement network for visual question answering. IEEE Transactions on Multimedia, pages 1-1, 2022.
[31] Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. In C. Cortes, N. Lawrence. D. Lee. M. Sugiyama. and R. Garnett, editors. Advances in Neural Information Processing Systems, volume 28, pages 91-99. Curran Associates, Inc..
2015.
[32] Hamid Rezatofighi, Nathan Tsoi, JunYounng Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection union: A metric and loss for bounding box regression, 2019.
[33] Suprosanna Shit, Rajat Koner, Bastian Wittmann, Johannes Paetzold, Ivan Ezhov, Hongwei Li, Jiazhen Pan, Sahand Sharifzadeh, Georgios Kaissis, Volker Tresp, and Bjoern Menze. Relationformer: A unified framework for image-to to-phaphhhh-hhooo generation, 2022.
[34] Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang, Unbiased scene graph genera- tion from biased training. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle WA. USA, June 13-19, 2020, pages 3713-3722 IEEE, 2020.
[35] Kaihua Tang, Hanwang Zhang. Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to compose dynamic tree structures for visual contexts. In The IEEE Conference on Computer Vision and Puttern Recognition (CVPR), June 2019.
[36] Hung-Yu Yu Tseng, Hsin ying Lee, Lu Jiang. Ming-Hsuan Yang, and Weilong Yang. RetrieveGAN: Image synthesis via differentiable patch retrieval. In ECCV, 2020.
[37] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In L. Guyon, U. Von Luxburg. S. Bengio. H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, in Neural Information Processing Sysiems. volume 30. Curran Associates, Inc., 2017.
[38] Sijin Wang, Ruiping Wang, Ziwei Yao. Shiguang Shan, and Xilin Chen. Cross-modal seene graph matching for relationship-aweer image-text retrieval. In IEEE Winter Conference on Applications of Computer Vision, WACV 2020, Snowmass Village, CO. USA. March 1-5, 2020, pages 1497-1506. IEEE, 2020.
12

---

[39] Tao Wang, Li Yuan, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan. Pup-detr: Towards efficient visual analysis with transformers, In Proceedings the IEEE/CVE International Conference on Computer Vision, pages
2021.
[40] Tzu-Jui Julius Wang. Selen Pehlivan, and Jorma Laaksonen. Tackling the unannotated: Scene graph generation with bias-reduced models. arXiv preprint arXiv:2008.07822 2020.
[41] Wenbin Wang. Ruiping Wang, Shiguang Shan, and Xilin Chen. Exploring context and visual pattern of relationship for scene graph generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8188-8197, 2019.
[42] Sanghyun Woo, Dahun Kim, Donghyeon Cho, and In So Kweon. Linknet: Relational embedding for scene graph. Advances in Neural Information Procexsing Systems, 31, 2018.
[43] Danfei Xu, Yuke 7hu, Christopher Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. Computer Vision and Pattern Recognition (CVPR). 2017.
[44] Shaotian Yan, Chen Shen, Zhongming Jin, Jianqiang Huang, Rongxin Jiang, Yaowu Chen, and Xian-Sheng Hua. PCPL: predicate-correlation perception learning for unbiased scene graph generation. MM '20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020, pages 265 273, 2020.
[45] Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation, 2018.
[46] Jianwei Yang, Jiasen La,, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation. In Proceedings of the European conference on computer vision (ECCV), pages 670-685, 2018.
[47] Yang, Kaihua Tang, Hanwang Zhang. and Jianfei Cai. Auto-encoding scene graphs for image captioning. In The IEEE Conference on Computer Vision and Pottern Recognition (CVPR), June 2019.
[48] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Exploring visual relationship for image captioning. In The European Conference on Compuder Vision (ECCV), September 2018.
[49] Guojun Yin. Lu Sheng. Bin Liu, Nenghai Yu. Xiaogang Wang. Jing Shao, and Chen Change Loy. Zoom-net Mining deep feature interactions for visual relationship recognition. In Proceedings of the European Conference on Computer Vision (ECCV), pages 322-338, 2018.
[50] Ruichi Yu, Ang Li, Vlad Morariu, and Larry S Davis. Visual relationship detection with internal and external linguistic knowledge distillation. In Proceedings of the IEEE international conference on computer vision, pages 1974-1982, 2017.
[51] Alireza Zareian, Svebor Karaman, and Shih-Fu Chang. Bridging knowledge graphs to generate scene graphs. In European Conference on Computer Vision, pages 606-623. Springer, 2020.
[52] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. CoRR, abs/17111.0644 2017.
[53] Fangao Zeng, Bin Dong. Yuang Zhang, Tiancai Wang. Xiangyu Zhang, and Yichen Wei. Motr: End-to-end multiple-ojject tracking with transformer. arXiv preprint arXiv:2105.03244 2021.
[54] Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, and Tat-Seng Chua. Visual translation embodding network for visual relation detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5532-5540, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(C) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
13

---

(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proof' of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [No] The code will be released.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (c.g., code, data, models) or curating/relcasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals. if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "yNzLD5X8S8w", "reviewer": "Reviewer_ZNFu", "summary": "The authors tackle the task of scene graph generation by proposing a new formulation which models the problem as a product of two conditionals - the first of which generates a bounding box estimate and a predicate label, and the second takes the aforementioned estimates to get subject and object class labels and refined bounding boxes. The two distributions are realized via a DETR style encoder-decoder transformer architecture. The experimental analysis shows improved performance compared to the baselines on the Visual Genome dataset. ", "questions": "Although I like the novel formulation resulting in a unique factorization of the SGG problem (Equation 14), there are certain issues with experiments and the described analysis that makes it harder for me to evaluate the proposed approach. \n\n1. It would be ideal if the authors could further clarify the analysis in Table 1. More specifically, what are the hyperparameters used by the different baseline models and whether they have the same number of parameters. If they don't, why is the performance improvement from SD to DD to DDTR not a direct consequence of improved model capacity. Additionally, when comparing TraCQ to the baselines, the performance under the same number of predictions should be shown. (See points 2 and 3 in weaknesses)\n\n2. The authors should clarify the importance of the refinement module $\\mathcal{C}$. Why is the refinement module $\\mathcal{C}$ necessary? Why is the predicate decoder $\\mathcal{H}$ unable to predict entity labels, when it already predicts a reasonable estimate of the entity bounding boxes? (See point 4 in weaknesses)\n\n3. Using pretrained weights adds certain biases to the predicate decoder $\\mathcal{H}$. Would initializing the weights randomly make the learning much harder ? (See point 5 in weaknesses)\n\n4. The authors should report ablations on parameters $N_{ce}, N_p$ and also report the zero shot recall performance. Additionally, they should also report the per class predicate performance to help understand what predicate classes the model does better on (See point 6 in weaknesses)", "limitations": "The authors have addressed certain limitations and potential negative societal impact of their work. Other limitations could arise from the choice of DETR architecture. For example, the vanilla DETR model ([3]) has poorer performance on detecting smaller objects (see [b]). \n\n[b] Zhu et al. \"Deformable DETR: Deformable Transformers for End-to-End Object Detection.\" ICLR 2021.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "$$\\large{\\textbf{Strengths}}$$\n\n1. The paper is well written and easy to follow. The motivation is well explained and Figure 2 provides a clear overview of the approach. \n2. The factorization assumed by the paper (Equation 14) is novel compared to existing work which often condition the predicate prediction on the subject/objects (Equation 8). This formulation provides interesting opportunities for future research.\n\n$$\\large{\\textbf{Weaknesses}}$$\n\n1. The claims learning in the non-combinatorial predicate space is easier than the $O(\\varepsilon \\times \\varepsilon)$ entity pair space due to the former being smaller. This is supported by analysis in Section 4 (Comparision to Previous Approaches subsection). Firstly, the analysis makes the assumption that labels $\\mathcal{Y}, \\mathcal{P}$ are independent of bounding boxes $\\mathcal{B}$ to simplify the factorization in Equation 14. This assumption clearly doesn't hold in the context of scene graph generation as bounding boxes are crucial to localizing where in the image a particular interaction is happening, and between what objects the aforementioned interaction occurs. Therefore, the simplification of the factorization in Equation 14 under this assumption is misleading. Secondly, looking at Equation 14 without the simplification, it is unclear why the term $\\Pr(\\mathcal{Y}, \\mathcal{B} | \\mathcal{P}, \\hat{\\mathcal{B}}, \\mathcal{I})$ should be better to learn from. Even though it is conditioned on a smaller space as compared to the term $\\Pr(\\mathcal{P}| \\mathcal{B}, \\mathcal{Y}, \\mathcal{I})$ in Equation 8, the proposed formulation is still predicting in a much larger space consisting of entities labels $\\mathcal{Y}$ and bounding boxes $ \\mathcal{B}$  (which are continuous in nature). \n\n2. The analysis in Table 1 shows the model performance to improve when increasing the amount of disentanglement. However, this performance improvement could be a direct result of increasing the model capacity. For example, going from SD to DD would entail an additional decoder (which I'm assuming is 6 layers deep based on the experiments), which could provide the observed gains as the model has a greater capacity to fit the training data. As the authors haven't discussed the hyperparameters for the three models shown, disentanglement cannot solely be considered as the cause for the observed improvements.\n\n3. Building on the previous point, Table 2 in the supplementary (ablation on the hyperparameter k) shows a considerable improvement in performance when going from k=1 to k=5, as the number of predictions generated by the model increasing from 200 to 1000. The reported performance gains of the proposed TraCQ to the baselines in Table 1 of the supplementary, and to other single stage transformer models (like [25]) in Table 2 of the main paper are misleading due to the increased number of predictions made by the model. This is majorly due to the recall metrics not checking for prediction precision. The authors have not mentioned the number of queries used by the models in Table 1. If, for example, it is 200, then the proposed TraCQ model would be worse than the DDTR model on mR@20 assuming the same number of predictions (200).\n\n4. From the ablation in Table 3, the refinement module $\\mathcal{C}$ doesn't impact the overall model performance as much. Looking at the numbers, it reduces performance on mR@20, which is a stricter and more desirable metric compared to mR@50/100. The visualizations in Figure 2 of the supplementary does not show the change in the bounding box before and after refinement, making it difficult to analyze the importance of the proposed module $\\mathcal{C}$. As the module $\\mathcal{H}$ already predicts the predicate class and provides a reasonable estimate of the object bounding boxes, one could easily add another couple detection heads to $\\mathcal{H}$ to predict the entity class labels as well. This would do away with the entity refinement module (which has almost the same number of parameters as $\\mathcal{H}$). The authors should provide an ablation / address why this simpler architecture is worse than the proposed TraCQ. \n\n5. The authors mention that the initialize the network with parameters pre-trained on VG object detection task. Doing so already biases the predicate decoder $\\mathcal{H}$ to be aware of the entity labels (as the object detector was trained to predict both bounding boxes and labels). It is unclear what effect this initialization has on the overall performance, and whether learning $\\mathcal{H}$ would be much harder if the weights were initialized randomly.\n\n6. There are some missing ablations with regards to the what effect do the parameters $N_{ce}, N_p$ have on the performance. Additionally, the authors should also report the zero-shot recall performance (introduced in [a]) that computes the recall@K for triplets not present in the training data. It would highlight the generalizability of the proposed approach. \n\n[a] Lu, Cewu, et al. \"Visual relationship detection with language priors.\" European conference on computer vision. Springer, Cham, 2016.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NgqglTE4tyv", "reviewer": "Reviewer_P8oQ", "summary": "The paper presents a single-stage visual relationship detection method based on a new design of CNN-Transformer network architecture. The main idea is to first compute a shared representation based on the CNN-Transformer encoder, and then to decouple the visual relationship detection into two sequential subtasks: 1) Prediction of the entity-pair location and the predicate class based on a predicate decoder;  and 2) Refinement of the entity locations and prediction their classes given the predicate prediction based on a conditional entity refinement decoder.  A multi-task loss based on Hungarian matching is designed to train two decoders separately, each focusing on its corresponding subtask.  The proposed method is evaluated on the VG150 benchmark with comparisons to the prior methods. ", "questions": "1. How does the method cope with the potential inconsistency in the SSG prediction?\n2. What is the contribution of PnP DETR to the performance improvement? \n3. What is the time complexity of the proposed method in comparison to the other baselines?\n ", "limitations": "The discussion on the limitations is included in the appendix. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ The paper presents an efficient single-stage visual relationship detection strategy based on a novel decomposition of the triplet detection task.  The resulting two sequential subtasks have a smaller search space, which simplifies the overall detection task. \n+ The proposed method achieves strong results on the VG dataset, outperforming the prior SOTA with more compact network architecture. \n\nWeaknesses:\n1) Potential inconsistency in scene graph generation. While the proposed method can produce a set of visual relationship predictions, they are generated conditioned on the predicate estimation. In particular, the locations of entities in each visual relation are refined independently. As a result, two neighboring visual relations (sharing one entity), or two visual relations with the same sub/obj entities can produce two different localizations for the shared entity.\n2) Some of the motivation lacks clarity.  \n+ What is the feature $f_i$ in Line 44? How do \"the entity-scene interactions characteristic of each predicate\" instantiate in the models described in this paper? \n+ Why the matching in the DDTR is computationally expensive? (Line 143) \n+ Why the task $Pr(\\mathcal{P}|\\mathcal{I})$ is a simpler subtask? (Line 241) The variation of the predicates seems to be much larger than entities and they also have more severe long-tail problems. \n3) The experimental evaluation is a bit lacking in the following aspects:\n+ The paper adopts the PnP DETR as the transformer architecture. It is unclear if this backbone introduces additional benefits in performance, such as entity detection quality. \n+ The paper lacks a comparison in time complexity for model inference. \n+ In the ablation on the entity refinement module, why use the outputs from the predicate decoder? The details of the evaluation on the task PredDet seem to be missing.  \n   ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "LtUfZuOVZs", "reviewer": "Reviewer_zNec", "summary": "This work proposes an end-to-end trainable framework for SGG using a Transformer architecture that models entity and predicate distributions with conditional queries simultaneously, and then weakly couples predicate and entity detection through the refinement of entity locations & the prediction of labels. It achieves a decent performance on the Visual Genome benchmark.", "questions": "Overall this paper gives a clear introduction to their motivation and solutions. One minor question is the preliminaries. I'm not sure how Dual decoders actually work, and the detailed experimental setup of SS & DD.", "limitations": "The author already mentioned the limitation of their work in Suppl, which is their method doesn't take into account long-tail distribution into consideration. I expect to see further improvement in the future to address these issues.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThis paper is well organized, clearly written, and easy to follow. It has a clear explanation of the motivation based on the observation of the preliminary works. Besides, I like the design that decouples entity and predicate distributions with conditional queries and couples them with another instance refinement decoder. The final experiment results also support their claim and show superior performance in the widely-used datasets.\n\nWeaknesses:\n1. Missing literature: The authors do not include some recent works, like papers on SGG [1] and the similar human-object interaction detection task [2].\n2. I'm not sure whether this work really achieves SOTA as it claims as I find the results in [1] are higher (but they use RN101, maybe an unfair comparison).\n3. Typo: Line 135, two 'use of'\n\n[1] Li et.al. SGTR: End-to-end Scene Graph Generation with Transformer\n[2] Kim et.al. Hotr: End-to-end human-object interaction detection with transformers", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0ZLljybolJ", "reviewer": "Reviewer_zyLi", "summary": "The paper proposes a single-stage end-to-end trainable DETR-based architecture named TraCQ. It uses a separate encoder-decoder architecture that disentangles entity and predicate space. Unline earlier architecture they demonstrated strong coupling between entity space and predicate space hurts the performance. Hence it proposes conditional learning of predicate couple learning of entity scene and predicate prediction and establishes a weak coupling by loosely estimating bounding boxes of subject and object. This conditional and couple learning also reduces the parameter significantly. They conducted their experiment on Visual Genome Dataset and achieved a new benchmark on mean SGDet and SGDet recall.", "questions": "1. Have you conducted any experiments from class to predicate? as this is the normal order\n2. Can you scale up for a large number of objects as the complexity is high?\nSome of the suggestions:\nPlease add the object detection score and FPS in Tab.2 for completeness. \n\n\nGood to have but doesn't impact the rating : \nThe presentation and writing can also be improved. \nIt is better to include some of the early transformer-based two-stage networks like : \n1.Relation Transformer Network, Koner at el,\n2. Context-aware Scene Graph Generation with Seq2Seq Transformers , Lu at el", "limitations": "The authors should provide some of the limitations of the method like:  is it applicable to other relation detection metrics (e.g. SGCls)", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength :\nThe idea has potential, disentanglement of predicate and entity space seems beneficial for predicate prediction. But there are some drawbacks which are mentioned later.\n\n\nWeakness :\nThere are a few weaknesses in the paper:\n1. It uses pnp DETR, but never mentioned its object detection performance, whereas SGDet is highly reliant on Object Detection performance. Despite the good performance on SGDet Table 3 shows the model is underperforming on Predicate Detection. It may possibly the performance may come from superior object detection performance. Authors should provide additional experiments.\n2. What is the object-scene interaction feature $f_{i}$? its better to introduce or explain a bit.\n3. Visual Genome has high biases, for a strong result section authors should test TraCQ on multiple datasets.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "-wV-nd5n_-v", "reviewer": "Reviewer_2zbH", "summary": "The paper analyses design choices of one-stage architectures for the task of Scene Graph Generation (SGG) - specifically, the degrees of feature entanglement for predicting predicates (relations) and entities (objects). An observation is made in Fig 1 and Tab 1, that complete coupling of features for predicate and entity classification leads to lower performance, while on the other hand complete decoupling gives better performance but it’s undesirable since it leads to overhead in the form of disjoint queries. The authors propose a mechanism to condition queries for predicting entities with the predicates, which leads to a desired trade-off between completely decoupled and coupled approaches. The authors show improved performance over its counterparts.", "questions": "See \"Weaknesses\" above", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- The approach is well-motivated and is communicated clearly\n\n## Weaknesses\n\n- Table 1\n    - I couldn’t find the result of TracCQ  here. Is it the same as the last row of Table 2? \n    - If yes\n        - then is it fair to say that DDTR already outperforms the best performing single-stage approach in Table 2?\n        - Also what changes resulted in better performance of DDTR baseline compared to the prior single-stage approaches?\n    - Line 294-295: “… It also exhibits a much less drastic drop in the mean recall values when we go from 100 to 20 for mR@K …”\n        - In fact, mR@20 scores in Table 1 for SD, DD and DDTR are either comparable or better than the best-performing single-stage approaches. Do the authors have an opinion about this? Are there any specific biases in the paper’s approach that accounts for this behavior?\n- Table 3: Correction v/s no-correction\n    - The authors mentioned in line 183 that the predicate decoder ($\\mathcal{H}$) “… is trained so as to not be penalized for inexact bounding of the subject and object entities. …”\n    - But the loss ($\\mathcal{L}_p$) in equation 12 includes the term for box loss with no modification for any leeway for box predictions or be penalized lower than what is suggested\n    - Moreover, the ablation in Table 3 seems to suggest that when the refinement of boxes is turned off (row 1), the performance doesn’t get affected by a lot and in fact shows better results for mR @ 20. Additionally this also shows up in the visualization of refined v/s original boxes in Supplementary Fig 2, which seems to suggest that no too much correction is happening\n    - Does this suggest that the predicate decoder is also predicting boxes as accurately as the final model? Can the authors quantify the correction?\n    - Do the authors have an ablation for a variant of the model where the entity refinement decoder only predicts the class of the objects, to really see whether the box refinement is crucial?\n\n\n### Overall comment\n- I like the idea of reducing the search space of entity refinement with the help of predicted predicates, and the fact that preliminary investigation in the form of Table 1 supports the intuition that the authors build on, but the experimental results and ablation could be made stronger and discussed more to substantiate the authors’ claim (given that most of the main paper prioritizes discussing the approach)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept