Iterative Scene Graph Generation
Authors: Siddhesh Khandelwal
Leonid Sigal
Venue: NeurIPS 2022 Accept (2022)
Keywords: Scene Graphs
Object Detection
Transformer Networks
Image Understanding

Abstract: The task of scene graph generation entails identifying object entities and their corresponding interaction predicates in a given image (or video). Due to the combinatorially large solution space, existing approaches to scene graph generation assume certain factorization of the joint distribution to make the estimation feasible (e.g., assuming that objects are conditionally independent of predicate predictions). However, this fixed factorization is not ideal under all scenarios (e.g., for images where an object entailed in interaction is small and not discernible on its own). In this work, we propose a novel framework for scene graph generation that addresses this limitation, as well as introduces dynamic conditioning on the image, using message passing in a Markov Random Field. This is implemented as an iterative refinement procedure wherein each modification is conditioned on the graph generated in the previous iteration. This conditioning across refinement steps allows joint reasoning over entities and relations. This framework is realized via a novel and end-to-end trainable transformer-based architecture. In addition, the proposed framework can improve existing approach performance. Through extensive experiments on Visual Genome and Action Genome benchmark datasets we show improved performance on the scene graph generation.

Iterative Scene Graph Generation
Siddhesh Khandelwal" 1,2 and Leonid Sigal 1,2,3
Deparrment of Computer Science, University of British Columbia Vector Institute for Al *CIAAR AI Chair
(skhandel, lsigal ubc.ca
Abstract
The task of scene graph generation entails identifying object entities and their corresponding interaction predicates in given image (or video). Due the combinatorially large solution space, existing approaches to scene graph generation assume certain factorization of the joint distribution to make the estimation feasible (e.g., assuming that objects are conditionally independent of predicate predictions). However, this fixed factorization is not ideal under all scenarios (e.g., for images where an object entailed in interaction is small and not discernibe on its own). In this work, we propose novel framework for scene graph generation that addresses this limitation, as well as introduces dynamic conditioning on the image, using message passing Markov Random Field. This is implemented as an iterative refinement procedure wherein each modification is conditioned on the graph generated the previous iteration. This conditioning across refinement steps allows joint reasoning over entities and relations. This framework is realized via novel and end-to-end trainable transformer-baat the proposed framework can improve existing approach performance. Through extensive experiments on Visual Genome [30] and Action Genome [25] benchmark datasets we show improved performance on the scene graph generation task. The code is available at github com/ uuc- waab-vvssonnnteat wwwiiinnnnneaiiiSS.S.
architecture. addition,
Introduction
Scene graphs allow for structured understanding objects and their interactions within scene. scene graph is a graph where nodes represent objects within the scene, each detailed by the class label and spatial location, and the edges capture the relationships between object pairs. These relationships are usually represented by a <subject, predi cate, object> triple. Effectively generating such graphs. from either images or videos, has emerged as a core problem computer vision [14, 35, 39, 46, 50, 52, 54]. Scene graph representations can be leveraged to improve performance on variety of complex high-level tasks like VQA [24, 47], Image Captioning [17, 53], and Image Generation [26].
The task of scene graph generation involves estimating the conditional distribution of the relation- ship triplets given an image. Naively modelling this distribution is often infeasible the space of possible relationship triplets is considerably larger than the space of possible subjects, objects, and predicates. To circumvent this issue, existing methods factorize the aforementioned distribution into casy-to-estimate conditionals. For example, swo-stage approaches, like [11, 47, 55], follow the graphical model image [subject object] + predi independently obtained via a pre-trained detector like Faster RCNN [41]. These are then consumed by downstream network to estimate predicates. Any such factorization induces conditional de- pendencies (and independencies) that heavily influence model characteristics. For example, in the aforesaid graphical model, errors made during the estimation of the subjects and objects are naturally propagated towards the predicate distribution, which makes the estimation of predicates involving
wherein the subjects and objects are
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

classes with poor detectors challenging. Furthermore, the assumed fixed factorization might not always be optimal. Having information about predicates an image can help narrow down the space of possible subjects/objects, e.g. predicate wearing makes it likely that the subject is person.
Additionally, due the two-stage nature of most existing scene graph approaches (barring very recent methods like [14, 31, 35]), the image feature representations obtained from a pre-trained task-oblivious detector might not be optimally catered towards the scene graph generation problem. Intuitively, one can imagine that the information required to accurately localize an object might not necessarily be sufficient for predicate prediction, and by extension, accurate scene graph generation. Morever, two-stage approaches often suffer with efficiency issues as the detected objects are required to be paired up before predicate assignment. Doing so naively, by pairing all possible objects [50], results in quadratic number of pairs that need to be considered. Traditional approaches deal with this using heuristics, such as loU-based threshold- -asddd pairing [47, 55].
In this work, we aim to alleviate the previously men- tioned issues arising from a fixed factorization and potentially limited object-centric feature representa- tions by proposing a general framework wherein the subject, objects, and predicates can be inferred jointly (i.e.. depend on one another), while simultaneously avoiding the complexity of exponential search space of relational triplets. This is achieved by performing mes- sage passing within a Markov Random Field (MRF) defined by components of a relational triplet (Figure (a)). Unrolling this message passing is equivalent to an iterative refinement procedure (Figure 1(b)), where each message passing stage takes in the estimate from the previous step. Our proposed framework models this iterative refinement strategy by first producing a scene graph estimate using a traditional factorization, and then systematically improving it over multiple refinement steps, wherein each refinement is condi- tioned on the graph generated in the previous iteration. This conditioning across refinement steps compensates for the conditional independence assumptions and lets framework jointly reason over the subjects (s), objects (o), and predicates (p).
000
P
Pt
Pt
G
G2
(a)
(b)
Figure 1: Iterative Refinement for Scene Graph Generation. By unrolling the mes- sage passing in a Markov Random Field (a), our proposed approach essentially models an iterative refinement process (b) wherein each modification step is conditioned on the previously generated graph estimate.
Contributions. To realize the aforementioned iterative framework, we propose a novel and intuitive transformer [48] based architecture. On technical level, our model defines three separate multi-layer multi-output synchronized decoders, wherein each decoder layer is tasked with modeling either the subject, object, or predicate components of relationship triplets. Therefore, the combined outputs from each layer of the three decoders generates a scene graph estimate. The inputs to each decoder layer are conditioned to enable joint reasoning across decoders and effective refinement of previous layer estimates. This conditioning is achieved implicitly via a novel joint loss, and explicitly via cross- decoder and layer-wise attention. Additionally, each decoder layer is also conditioned on the image features, which are provided by a shared encoder. As our proposed model is end-to-end trainable, it addresses the limitation of two-stage approaches, allowing image features to directly adopt to the scene graph generation task. Finally, to tackle the long-tail nature of the scene graph predicate classes [11], we employ a Joss weighting strategy enable flexible trade-off between dominant (head) and underrepresented (tail) predicate classes in the long-tail distribution. In contrast to data sampling strategies [11, 32], this has benefit of not requiring additional fine-tuning of models with sampled data post training. We illustrate that our proposed architecture achieves statc-of-ther of-tte-rrtt-tttto performance on two benchmark datasets- Visual Genome [30] and Action Genome [25]; and thoroughly analyze effectiveness of the approach function of the refinement steps, design choices employed and as a generic add-on to an existing, MOTIF [55], architecture.
2 Related Work
Scene Graph Generation. Scene graph generation has emerged as popular research area in the vision community [11, 28, 31, 35, 39, 40, 46, 47, 50, 52, 54, 55J. Existing scene graph generation methods can be broadly categorized as either one-stage or two-stage approaches. The first step of the predominant approach the two-stage methods involves pre-training strong object detector

---

for all object classes in the dataset, usually using detector like Faster-RCNN [41]. The graph generation network is then built on top of the object information (bounding boxes and corresponding features) obtained from the detector. This second step entails freezing the detector parameters and solely training the graph generation network. The graph generation network is realized via different architectures such as Bi-directional LSTMs [55], Tree-LSTMM [47], Graph Neural Networks [52], and other message passing algorithms [40, 50J. The limitation of all two-stage approaches is apparent - the graph generation network has no influence over the detector and object features. One stage approaches overcome this obstacle by employing architectures that allow end-to-end training, such fully convolutional networks [35] and transformer based models [7, 14, 31, 43]. The joint optimization enables interaction between the detection and the graph generation modules, leading to better scene graphs. However, all the aforementioned methods operate under the assumption of a fixed factorized model, inducing conditional independencies that might not be ideal under all scenarios.
Long-Tail Recognition. The task of scene graph generation suffers from the challenge of long- tail recognition due to the combinatorial nature of relational triplets (both object and predicate distributions are skewed). Long-tail recognition, in general, is a well studied problem in literature. Data sampling is popular strategy [3, 20, 21, 27, 37, 42, 58] wherein the training data is modified to either over-represent tail classes (oversampling) or under-represent the head classes (undersampling). Specific to the task of scene graph generation, [11, 32] have explored the use of data sampling strategies to improve performance on tail predicate classes. On the contrary. Loss re-weighting
strategies [1, 9, 13, 27, 34] assign higher weights impose larger decision boundaries for tail classes. [51] recently adopted this paradigm for the task of scene graph generation by proposing re-weighting strategy based on correlations between predicate classes. Conceptually similar to [51], we propose a novel re-weighting strategy and illustrate its effectiveness both on its own, and in combination with data resampling.
Transformer Models. [48], authors introduced a new attention-based architecture called trans- formers for the task of machine translation, doing away with recurrent or convolutional architectures. Transformers have since widely been adopted for a variety of tasks, such as object detection [2, 57], image captioning [8, 23], and image generation [12]. More recently, scene graph generation methods have also adopted transformer architectures [7, 14, 31, 43], owing to their end-to-end trainable nature and parallelism. Perhaps conceptually closest, [14], which focuses visual relation detection (VRD) and not scene graph prediction, leverages the composite nature of relationships to simultaneously de- code an entire relationship triplet alongside its constituent subject, object, and predicate components. Unlike [14], we do not explicitly model prrjecctprediccae-oojeeet "sum" composites, resulting in a simpler architecture, and leverage iterative refinement procedure that relies on different factorized attention mechanism. In the latest, concurrent work, [31] generates a set of entities and predicates separately, and then utilizes graph assembly procedure to match the predicates to a pair of entities. In contrast to our formulation, this precludes conditioning of entities on predicate estimates, therefore relying on accuracy of less contextualized predictions for subsequent pairing.
3 Formulation
For a given image I, scene graph G can be represented as a set of triplets G (ri)isn {(s;, Pi, o;) where r; denotes the i-th triplet (s., Pi, 0;), and n denotes the total number of triplets. The subject s; denotes a tuple (St,e) Sq,b), where S,,c E IF" is the one-hot class label, and SLL E RR is the corresponding bounding box coordinates. 17 is the total number of possible entity classes in the dataset. The object o; and predicate p. can similarly be represented as tuples (or.e: or.a) and (Pi.c: pi,t) respectively. Note that, pi. corresponds to the box formed by the centers' of $;,b and o, as the diagonally opposite coordinates. Additionally, Pie E R" represents the corresponding one-hot predicate label between the pair (s,, o:), where is the total number of possible predicate classes in the dataset. Then the task of scene graph generation can be thought of as modelling the conditional distribution Pr(G] I). This distribution can be expressed as a product of conditionals,
Pr (G I) = Pr ((s.) I) Pr ((o.) [s:), Pr ((p.) (ss). (o:).I)
where (-) denotes a set. For brevity, we omit explicitly mentioning the total number of set elements throughout the paper. Existing approaches model this product of conditionals by making some underlying assumptions. For example, [11, 47, 55] assume conditional independence between s; and o they rely on heuristics to obtain the entity pairs. However, modelling the conditional in Equation
3

---

Po
Pp
Alln
Alla
Aln
Alla
Allo
Webrrng
Encuder
Subject
Object
Predicate
fence
Backhone
Decoders
Figure Transformer Architecture for Iterative Refinement. For a given image, the model extracts features via a convolutional backbone and transformer encoder. The individual components of relationship triplet are generated using separate subject, object, and predicate multi-layer decoders. The inputs to each layer of the decoder is appropriately conditioned. For example, for the predicate decoder, the positional embeddings are conditioned on the outputs generated by the subject and object decoders (blue Attn module) and the queries are conditioned on the previously generated graph estimate (orange Attn module). The model is additionally implicitly conditioned and trained in an end-to-end fashion using joint matching loss.
1 (or any other equivalent factorization) in such a "one-shot" manner makes certain assumptions on the flow of information, which, in this case, is from s; o, P,. Therefore, any errors made during the estimation of are naturally propagated towards the estimation of o, and p,. Additionally, the subject (or object) estimation procedure Pr ([s;) I) is completely oblivious the estimated predicate p;. Having access to such information can help the subject (or object) predictor update its beliefs and significantly narrow down the space of feasible entity pairs.
Contrary to existing works, our proposed formulation moves away from the "one-shot" generation ideology described Equation 1. We instead argue for modelling the task of scene graph generation as an iterative refinement procedure, wherein the scene graph estimate at step t, G', is dependent on the previous estimates (G' free. Formally, our aim is to model the conditional distribution Pr (G (G Jea,, I). Assuming Markov property holds, this can be conveniently factorized as,
Subject Predictor
Object Predictor
Pr (r G G = =Pr ((s( G G Pr (fo)) (s)) I
Pr ((p)) (ss) (o
Predicate Prodictor
Note that though assume the flow of information to be from S - o - for G'. condition ing on the previous graph estimate G allows the subject, object, and predicate predictors to jointly reason and update their beliefs, leading to better predictions. Additionally, the framework described in Equation 2 is model agnostic and can be implemented using any of the existing architectures.
4 Transformer Based Iterative Generation
As described in Section 3, our proposed iterative scene graph generation formulation is agnostic to the architectural choices used to realise the subject, object, and predicate predictors. this section provide a realization of the proposed formulation in Equation using Transformer networks [48]. The choice of using transformer networks is motivated by their natural tendency to model iterative refinement behaviour under the Markov property, wherein each layer of the transformer decoder takes as input the output the previous layer. Our novel end-to-end trainable transformer based iterative generation architecture builds top of the DETR [2] framework, which is shown to be effective for the task object detection. Our proposed model architecture is shown Figure 2.
Given an image, our proposed approach first obtains corresponding image features using combina- tion of convolutional backbone and multi-layer transformer encoder, akin DETR [2]. These image features used as inputs to the subject, object, and predicate predictors, each implemented as

---

a multi-layer Transformer decoder. To generate scene graph estimate G' at step in accordance with Equation 2. the queries used in each predictor decoder are appropriately conditioned. For example, in the case of the predicate decoder, its input queries are conditioned on the subject ((S?)) and object ((oi)) estimates at step Additionally, the input to each predictor decoder is infused with all decoder estimates {($} p - -) from the previous step t- via structured attentional mechanism. The entire model is trained end-to-end, with a novel joint loss applied at each step i to ensure the generation of a valid scene graph at every level. This section describes these components in detail.
4.1 Image Encoder
Similar to DETR [2], for each image I, our proposed architecture uses a deep convolutional network (like ResNet [22]) obtain image level spatial map I E Roxwx,, where is the number of channels, and w. h correspond to the spatial dimensions. A multi-layer encoder fe then transforms into a position-awes aware flattened image feature representation Z Rdxmh, where d < c.
4.2 Predictor Decoders
Our approach models each of the subject, object, and predicate predictors using a multi-layer transformer decoder [2], which are denoted by fa. fo. and fp respectively. The t-th layer of each decoder, denoted as f x {s, o, p). is tasked with generating the step scene graph G Therefore, at each step t, the decoders output a set of n feature representations ( E (s, o, p)), which are transformed into set of triplet estimates
[()) f ) via fully-connected feed forward layers.
Specifically, for a decoder x e and set of learnable positional encodings (Px,). where 9 Px, TR.. The output representations {qk,)) are then obtained via a combination of sellf-ttention between the input queries, and encoder- decoder attention across the encoder output Z. These attention modules allow the decoder to jointly reason across all queries, while simultaneously incorporating context from the input image.
an arbitrary layer & takes as input a set of queries (gk,))
At a given step f, naively using the inputs {9 and {px.a} J to generate G' forgoes leveraging the compositional property of relations. formulation entails two types of conditioning for better scene graph estimation. The first involves conditioning decoders on the step outputs, specifically the object decoder fo on the subject decoder fa and the predicate decoder fo on both f and fo- The second requires all three decoder layers at step be conditioned on the outputs generated step design, we modify the inputs to each decoder layer f*. Specifically, the positional encodings {Px,i} are modified to condition them on step & outputs, and the queries g are updated to incorporate information from the previous step t - 1. Modifying the positional encoding and queries separately allows the model to easily disentangle and differentiate between the two conditioning types.
described Equation 2, for any arbitrary step t. our proposed
To effectively implement this
Conditional Positional Encodings. At particular step the conditional positional encodings for the three decoders are obtained as.
(P (Paa); (P6,) (Po..) + FFN (MultiHead ((Po.), (4,)). (4).))
(3)
(P).) 3 + FFN [MultiHead ((Pp.): (7h, + " (q',, + 1))
where MultiHead (Q, K, V) is the Multi-Head Attention module introduced in [48], FFN is fully-connected feed forward network, and is the concatenation operation. Additionally, 9 t + p x E (s, o. p} is the position-aware query.
Conditional Queries. Similarly, for a step f, conditional queries for the subject decoder are defined,
(G = (q + FFN (MultiHead ((q6,"), }, (9 1 + d.. + 9p..1 ). (q + g + o
(4)
/ and G are defined identically. For decoder layer f E {s, p} we use the conditioned positional encodings (R(,) and queries (a(_1) } as input.
4.3 End-To-End Learning
Our proposed transformer based refinement architecture can be trained in an end-to-end end fashion. To ensure that valid scene graph is generated at every level, we propose novel joint loss that applied at each step t. Therefore, the combined loss L can be expressed as L Σ, L =. C. + C% + C..
5

---

where x {s, o. p} represents the loss applied to the t-th layer of the decoder fx. Our approach generates a fixed-size set of triplet estimates {f {($; 0()] at each step t, where n is larger than the number of ground truth relations for given image. Therefore, in order to effectively optimize the proposed model, we obtain an optimal bipartite matching between the predicted and ground truth triplets. Note that, contrary to the matching algorithm in [2]. our proposed matching is defined over triplets rather than individual entities. Additionally, instead of independently computing the loss over each decoder layer as in [2], our loss computes joint matching across all refinement layers.
Let G (r.} {s., Pi, 0:]} denote the ground truth scene graph for an image I. Note that, as the number of ground truth relations is less than n, we convert G to n-sized set by padding it with Ø (no relation). The goal then is find a bipartite matching between the ground truth graph G and the set of all graph estimates { that minimizes the joint matching cost. Specifically, assuming o to be a valid permutation of elements,
á arg min ΣΣ frel (ri, F
(5)
where the pair-wise relation matching cost Liel is defined as,
(1).c * Si,e - Lbux (5t (6),),6
Lxel (bo) - (1:
Oa(i), -ss,c Loox 66' (11)), 05,6)
(6)
+ Pa(t). PPic Lbox (a(()),>> (r(i),,PPP)) Pi,b)
where is the vector dot product, and Loox a combination of the I.-1 and generalized loU losses. Please refer to Section 3 for clarification on the notations. The optimal permutation ē can then be computed using the Hungarian algorithm. The loss Cg is then defined as,
Σ log (S0,) si.c) + Il (1)+2)) Soox (8 (810)))
C
C. and C are defined identically. Note that as we use the same permutation for all refinement layers I, it induces strong implicit dependencies between the subject, object, and predicate decoders. The potency the aforementioned implicit conditioning is highlighted in the experiment section.
4.4 Loss Re- Reighigg
Due to the inherent long-tail nature of the scene graph generation task, using an unbiased loss often leads to the model prioritizing the most common ka.k.a,, head) predicate classes like has and on, which have abundant training examples. To afford our proposed model the flexibility to achieve the trade-off between head/tail classes, we integrate loss-reweighing scheme into the model training procedure. Note that, contrary to existing methods that do this post-hoc via finetuning the final layer of the trained network (see Section 2), instead train the model with this weighting allow the internal feature representations to reflect the desired trade-off. Note, to the best of our knowledge, our paper is the first illustrate effectiveness of such strategy for the task of scene graph generation. For
particular predicate class c [1, v). we define the class weight w. as max (1/1.)) .1.0). where
fe is the frequency of the predicate class e in the training set, and {ar, (3 are scaling parameters. Note that this weighting scheme is similar to the data sampling strategy described in [19, 32]. However, instead of modifying the training set, we scale each class weight by the factor we when computing the predicate classification loss L Therefore, C,p can be defined similarly to Equation 7,
C
Σ
we log (Pá().c Pi..) + (ert)))ooo (Poc)...P..))
(8)
5 Experiments
We demonstrate the effectiveness of our proposed approach on two datasets,
Visual Genome [30]. This is benchmark for scene graph generation. We use the common processed
subset from [50], which contains 108k images, with 150 object and 50 predicate categories.
Action Genome [25]. This dataset provides scene graph annotations over videos in the Charades dataset [44] for the task of bujen-object interaction. It contains 9, 848 videos across 35 object

---

Table 1: Scene Graph Generation on Visual Genome. Mean Recall (mR@K) Recall (R (R@K), K). and Harmonic Recall (hR @K) shown for baselines and our approach. B-Backbone, D=Dettetor. Baseline results are borrowed from [31]. M indicates the number of top-k links used by the baseline [31]. Note that our approach implicitly assumes M 1. For the approaches that use the ResNet-
backbone [22] (akin to our method), the best result is highlighted in red, second best in blue.
D #
Method
MRR@50/100
50/10000
hR 55/100
Head Body Tail
RelDN [56]
6.0 7.3
31.4
35.9
10.1
12.1
MOTIF [55]
5.5 6.8
32.1
36.9
9.4
11.5
VCTree [47]
6.6 7.7
31.8
36.1
10.9
12.7
-FN
Faster RCNN
BGNN [32]
10.7 12.6
31.0
35.8
15.9
18.6
34.0
12.9
6.0
X101
VCTree-TDE [46]
9.3 11.1
19.4
23.2
12.6 15.0
VCTree-DL.FE. [6]
11.8 13.8
22.7
26.3
15.5
18.1
VCTree-EBM [45]
9.7 11.6
20.5
24.7
13.2
15.8
CCTTree-BPLSAA [18]
13.5 15.7
21.7
25.5
16.6
19.4
DT2-ACBS [11]
22.0
24.4
15.0
/ 16.3
17.8
19.5
BGNN [32, 31]
8.6 10.3
28.2
/ 33.8
13.2
15.8
29.1
12.6
2.2
ResNet- 101
RelDN [56, 31]
4.4550 5.4
30.3
34.8
7.7
7.3
9.3
31.3
2.3
0.0
AS-Net [4]
6.1 7.2
18.7
21.1
9.2
10.7
19.6
7.7
2.7
HOTR [29]
9.4
12.0
23.5
27.7
13.4
16.7
26.1
16.2
3.4
DETR
Concurrent
Work
SGTR - [31]
12.0 14.6
25.1 / 26.6
16.2
18.8
27.1
17.2
6.9
SGTR M-3 [31]
12.0 15.2
24.6
28.4
16.1
19.8
28.2
18.6
7.1
M33 (2) [31]
15.8 / 20.1
20.6
25.0
17.9
22.3
21.7
21.6
17.1
Ours (a=000=)))
8.0 8.8
29.7
32.1
12.6
13.8
13.8
31.7
9.0
1.4
Ours 330.1.0001)))
14.4/ 16.4
27.9
30.4
19.0
21.3
21.3
30.0
17.3
11.2
Ours 7.
15.7 17.8
27.2
29.8
19.9
22.3
28.5
18.8
13.3
Ours (1.44.0004
15.8 18.2
26.1
/ 28.7
19.7
22.3
28.2
19.4
13.8
Ours 0.15 [32]
17.1 19.2
22.9
/ 25.7
19.6
22.0
24.4
20.2
16.4
(a= (3=0011.000777))MMM
19.5
23.4
30.8
35.6
23.9
28.2
32.9
28.1
15.8
(excluding class person) and 25 relation categories. As not all video frames are annotated, consistent the annotated frames provides by [25]. Additionally, in [16],
with prior work [16], remove frames without any person or object annotations as they do not provide usable scene graphs.
Implementation Details (Iransformer-bused approach). We use ResNet-101 [22] as the backbone network for image feature extraction. Each of the subject, object, and predicate decoders have layers, with feature size of 256. The decoders use 300 queries. For training we use batch size of 12 and initial learning rate of 10 which gradually decayed. Note that although our model predicts individual relation triplets, graph is obtained by applying a non-maximum suppression (NMS) strategy [15, 41] to group the predicted subjects and objects into entity instances.
Implementation Details (MOTIF), For the re-implementation of the MOTIF [55] baseline and the subsequent augmentation of our proposed framework, we follow the same training procedure as [55]. Specifically, we assume the Faster-RCNN detector [41] with the RtsNeXt- 101---111FFPN [49] backbone. The detector is first pre-trained on the Visual Genome dataset [30]. As the case with two-stage approaches, when learning the scene graph generator, the detector parameters are freezed. The specifics on how our approach is augmented to MOTIF is described in the supplementary (Sec. A).
Evaluation Metrics. To measure performance, we report results using standard scene graph eval- uation metries, namely Recall (R@K) [50] and Mean Recall (mR K) [S, 47]. While recall is class agnostic, mean recall averages the recalls computed for each predicate category independently. Usually, higher R @@K indicative of better performance on dominant (head) classes, where as higher mR @ K suggests better tail class performance. Recent methods [11, 32] have argued for the use of mean recall as reduces influence dominant relationships such as on and has on the metric. However, even for the same model architecture, one can trade-off @ K for mR @ K by using long tail recognition techniques described in Section 2. This trade-off is seldom analyzed or reported, making it difficult to compare performance across methods. Therefore, inspired by the generalized zero-shot

---

Table 2:
Ablation of Model Components. Mean Recall (mR@K) and Recall (R@K) reported on the Visual Genome dataset. CAS-Conditioning Across Steps (Eq. 4), CWS-Conditionigg Within Step (Eq.
Table 3: Augmentation to MOTIE Mean Recall (mR @K) and Recall (R@K) is re- ported on the Visual Genome dataset for the baseline and the variant with our it- erative formulation. t denotes our re- implementation of the method.
3), JL=Joint Loss (Sec. 4.3).
CAS CWS JL mR @ 20/50 R 20/50
Model
t mR@ 20/50
R@ 20/50
11.8/
15.8
21.0/
/ 26.1
MOTIF! [55]
6.0 / 8.0
23.6 / 30.4
1.7
/ 1.9
2.7
/ 4.3
11.2
14.7
20.1/
/ 25.4
MOTIF
1
6.0 / 8.1
23.7
/ 30.6
/ /
11.5
15.3
20.9
26.1
+
2
6.2 / 8.4
23.9
30.7
11.7
/ 15.4
20.9
/ 25.9
Ours
3 6.4 / 8.5
24.0
30.8
Table 4: Ablation of Loss Re-weighting Parameters. We vary , (Sec. 4.4) and report recall, mean recall, and harmonic recall on the Visual Genome test set.
Table 5: Ablation of Model Parameters.
4 Our proposed transformer approach with 6 decoder layers. 2 = transformer
with 6 decoder layers devoid of our pro- posed refinement. 3 transformer with 1 decoder layer devoid of our proposed refine- ment.
8
mR @ 20/50
hR @ 20/50
0.0
5.8 / 8.0
24.2
29.7
9.4
12.6
0.07 0.75
11.2 15.7
21.8
27.2
14.8/ / 19.9
0.14 0.75
11.8
15.8
21.0/
26.1
15.1
19.7
Model Size
0.21 0.75
11.3/ / /1.55
20.0/
/
24.8
14.4 19.1
# mR @ 20/50
R @@ 20/50
0.14 0.5
10.3 / 14.4
22.6
27.9
14.2/
19.0
11.8 / 15.8
21.0/ 26.1
1x
0.14 0.625
11.0 / 15.1
21.4
/
26.4
14.5 / 19.2
0.14 0.75
11.8
15.8
21.0
26.1
15.1
19.7
1.7
1.9
2.7 / 4.3
0.9x
0.14 0.875
11.2
15.9
19.3
24.5
14.2
19.3
9.9
13.1
17.5/ / 21.8
1.1x
learning harmonic average metric [10], we propose a new metric harmonic recall (hR @ K), defined as the harmonic mean of mR @ K and R K. Such a metric encourages methods to have healthy balance between the head and tail class performance. Additionally, we report the performance of our model for different values of the loss weighing parameters described in Section 4.4. Furthermore, we also report the mR @ 100 on each long-tail category subset, namely head, body, and tail, as in [32].
5.1 Ablation Study
All the subsequent ablation studies are done the Visual Genome dataset [30].
Model Components. We analyze the importance of each of our model components, namely the
conditioning within particular step t (CWS; Equation 3), the conditioning across steps (CAS; Equation 4), and the joint loss (JL; Section 4.3) in Table 2. For a fair comparison, all models are trained using the same loss weighing parameters, proposed novel joint loss provides significant improvement in performance (4) when compared to the ablated model that uses separate losses (as in [2]: A This is largely due to the joint loss being able to induce strong implicit dependencies between the three decoders. As consequence, even without CAS and CWS, the proposed joint loss by itself enables refinement. The two forms of conditioning (+))) provide a structured pathway to incorporating knowledge from previously generated estimates or other triplet components an explicil fashion, leading to improved scene graphs. CAS (4) builds upon the joint loss, integrating information that the aforementioned implicit conditioning is unable to capture. CWS (4) is complementary to the refinement process, and allows for more consistent graph generation within a step. As these two types of conditionings capture complementary information, using them together (<) further improves on performance.
0, = 0.75. It can be seen that our
Refinement Across Steps. To further analyze the effectiveness of our refinement procedure, we contrast the quality of scene graphs generated at each refinement step for a particular trained model the supplementary (Table A1). Visually,
(a = 0.14, 3 = 0.75). We provide a detailed analysis Figure 3 highlights steady improvements in the graph quality over refinement steps. Owing to our structured conditioning, the model is able to improve on both predicate detection and entity localization over refinement iterations.
8

---

Unionelliyy
Timbrelia
Limbrellat
Bench
Beath
Dench
Linieella
People
Linlrelle
People
Limbeelln
Eoople
Figure 3: Qualitative Refinement Analysis. Graph estimates for different refinement steps shown. Colours red and green indicate incorrect and correct predictions respectively. Additional visualizations are shown in the supplementary (Section C.3).
Loss Re- Weightigg,, As mentioned in Section 4.4, we employ a simple loss re-weighting scheme, allowing for an easy trade-off between recall for mean recall. Some results with different weightings
are reported in Table ( (2)) To investigate this further, we train our proposed model with different values for the parameters and B3 and report the performance in Table 4. We fix / in the top half of the table (first 4 rows). and a in the bottom half (last rows). It can be seen that increasing a and / generally biases the model more towards the tail classes (higher mean recall), whereas lower values tend to favour the head classes (higher recall). As a consequence, our proposed approach is flexibly able to function in wide range of recall and mean recall values. However, as is the case with every long-tail recognition approach, choosing values that are too high (e.g. a = 0.21, 8 = 0.75) lead to poor performance due the model being extremely biased and therefore unable to learn accurate feature representations on the popular classes.
Model Parameters. We additionally argue that the improvements obtained by our proposed re- finement procedure are not a direct consequence of having an increased number of parameters. To corroborate this, we contrast our proposed model with alternatives with similar number of parameters but devoid of refinement in Table 5. ① corresponds to our transformer-basee approach with 6 decoder layers that uses the proposed joint loss and both forms of conditioning. ④ is similar model with 6 decoder layers but bereft of the aforementioned proposed implicit and explicit conditionings. ( on the other hand, is single decoder layer transformer model without refinement, wherein the number of parameters are increased to be comparable to our proposed approach. The inferior performance of ② and 2 highlight that increasing model capacity either via making each layer larger or via adding more layers (increasing model depth) does not emulate refinement. Our proposed refinement procedure, realized by the joint loss and explicit conditionings, allows for better learning, and the performance gains observed cannot be attributed to having additional parameters.
5.2 Comparison to Existing Methods
Visual Genome. We report the mean recall, recall, and harmonic recall values contrasting our proposed model with existing scene graph methods, including the concurrent work in [31], in Table Contrary to existing approaches, depending on the loss weighting parameters used, our proposed approach is able to easily operate on wide spectrum of recall and mean recall values ( More concretely, compared to the most competitive bascline in SGTR [3] ( and other one stage methods in AS-Net [4] (L) and HOTR [29] (()), approach is able to achieve a considerably better recall on the head classes ( 3.5 mR @ 100 better on head classes compared to [31]). Additionally, when comparing a variant of approach that has similar R@ 100 value to SGTR ( and ()), we do significantly better on mean recall (3.8 higher mR 50) highlighting the proficiency of our method to generalize tail classes wherein the training data is limited. Furthermore, our proposed model in (2) achieves the best performance across all models (including two-stage methods that use a superior backbone [49]) on hR 50/100, underlining the capability of our method to perform well on both the head and tail classes simultaneously. SGTR [31] additionally reports numbers by utilizing a data sampling approach BGNN [32] to bias the model towards the tail classes (1O). We highlight that, compared to (i)) our model in 20) performs similarly on mR @ 50 but much better R 50/100 (5.5 @ 50 higher). The poorer performance on mR @ 100 is largely due SGTR [31] selecting top-3 subjects and objects for each predicate, leading to more relationship triplets being generated. On the contrary, our approach only uses predicate per subject-objer pair. For a fairer comparison, we evaluate our model in using similar top-k strategy. wherein we select the top-3 predicates for each subject-object pair. The resulting model (3) outperforms the closest baseline by a significant margin (3.3 higher mR @ 100, 10.6 higher R @@ 100 compared to (I)) Additionally, we show that our loss re-weighting strategy is complementary to existing data sampling approaches by fine-tuning our trained model in 2 using BGNN [32] (c)). For further analysis, supplementary Section C.2.

---

Table 6: Scene Graph Prediction on Action Genome. Mean Recall (mR @ K). Recall (R@K). and Harmonic Recall (hR (K) shown for baselines and variants of our approach with different loss weighting parameters. B=Backbone, D-Detector. Baseline results are taken from [16].
B D #
|ethhdd
R @ 20 50
mR @ 20/ 50
hR @ 20/50
VRD [36]
10.3 / 10.9
Faster RONN
Freq Prior [55]
24.0 24.9
X101 FP-N
IMP [50]
23.9 25.5
MSDN [33]
24.0 25.6
Graph R-CNN [52]
24.1 25.8
ReIDN [56]
25.0 26.2
SimpleBase [16]
27.9
30.4
8.3 / 9.1
12.8 / 14.0
R-101 DETR
(arS(a
31.0 37.5
20.9
/ 25.3
24.9
30.2
Ours 00050000.07,0000700
30.1
36.5
36.9
/ 42.8
33.1/
/ 39.4
(00$(0=00.0..800075)
29.2
35.3
37.9
44.0
32.9
39.2
Action Genome. We additionally report recall, mean recall, and harmonic recall values on the Action Genome dataset [25] in Table 6. Similar to the observations on Visual Genome, we observe 3.1 higher @@20 and 12.6 higher mR @ 20 (c) when compared the closest baseline in [16] (() despite using an inferior backbonn [49]. Furthermore, by effectively biasing our model towards the tail classes, we able to achieve 29.6 better mR @ 20 while having better R@ 20/50 values (w)) compared to O
We provide per class relation recall and object AP for both datasets the supplementary (Section C).
5.3 Generality of Proposed Formulation
Although our transformer model effectively generates better scene graphs, the proposed iterative refinement formulation (Equation 2) can be applied to any existing method. To demonstrate this, we augment this framework to the simple two-stage MOTIF [55] architecture. The details on how this achieved is deferred to the supplementary (Section A). We contrast the performance of the baseline and our augmented approach on the Visual Genome dataset [30] in Table It can be seen that with each refinement step, the model is able to generate better scene graphs (~6% higher on mR @ 20 and mR @ 50 after 3 steps). Note that, owing to the two-stage nature of MOTIF [55], the image features used by refinement framework are fixed and oblivious the task of scene graph generation (as the detector parameters are frozen). Contrary to the proposed transformer model that can query the image each layer, the performance gains when using the augmented MOTIF variant are largely limited due to the image features being the same across the refinement steps.
Conclusion, Limitations, and Societal Impact
In this work we propose a novel and general framework for iterative refinement that overcomes limitations arising from a fixed factorization assumption in existing methods. We demonstrate its effectiveness through transformer based end-to-end trainable architecture.
Limitations. Limitations of the proposed approach include using a shared image encoder, which improves efficiency, but may limit diversity of representations available for different decoder layers; and limited ability to model small objects inherited from the DETR object detection design [31].
Societal Impact. While our model does not have any direct serious societal implications, its impact could be consequential if is used in specific critical applications (e.g.. autonomous driving). such cases both the overall performance and biases in object / predicate predictions would require careful analysis and further calibration. Our loss re- re-weighting strategy is a step in that direction.
A Acknowledgments and Disclosure of Funding
This work funded, in part, by the Vector Institute for AI, Canada CIFAR AI Chair, NSERC CRC and an NSERC DG and Accelerator Grants. Hardware resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute!. Additional support was provided by JELF CFI grant and Compute Canada under the RAC award. Finally, we sincerely thank Bicheng Xu and Muchen Li for their valuable feedback on the paper draft.
www www.weettrinstiiuueee a1
wwt.uueteee..eeeeeeeeeeee s1/#partners
10

---

References
[1] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with fabel-distriution--warr margin loss. Advances in neural information processing systems, 32, 2019.
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[3] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
[4] Mingfei Chen, Yue Liao, Si Liu, Zhiyuan Chen, Fei Wang, and Chen Qian. Reformulating hoi detection as adaptive set prediction. In Proceedings of the IEEF//VF Conference on Computer Vision and Pattern Recognition, pages 9004-9013, 2021.
[5] Tianshui Chen, Weihao Yu. Riquan Chen, and Liang Lin. Knowledge-embeädedd routing network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition, pages 1163-6171, 2019.
[6] Meng-Jiun Chiou, Henghui Ding. Hanshu Yan, Changhu Wang. Roger Zimmermann, and Jiashi Feng. Recovering the unbiased scene graphs from the biased ones. In Proceedings of the 29th ACM International
Conference on Multimedia, pages 1581-1590, 2021.
[7] Yuren Cong, Michael Ying Yang. and Bodo Rosenhahn. Reltr: Relation transformer for scene graph generation, arXiv preprint arXiv:2201.1 / 1460, 2022.
[8] Marcella Cornaa, Matteo Stefanini, Lorenzo Baraldi, and Rita Cucchiara. Meshed-memory transformer for image captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10578- 105877 2020.
[9] Yin Cui, Menglin Jia, Tsung-Yi Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective numher of samples. In Proceedings of the IEEF/CVF conference on computer vision and pottern recognition, pages 9268-9277 2019.
[10] Shabnam Daghaghi, Tharun Medini, and Anshumali Shrivastava. Sdm-net: a simple and effective model for generalized zero-shot learning. In Uncertainty in Artificial Intelligence, pages 2103 201-2213. PMLR,
2021.
[11] Alakh Desai, Tz- Ying Wu, Subarna Tripathi, and Nuno Vasconcelos. Learning of visual relations: The devil is in the tails. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
35404-15415 2021.
[12] Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang I Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering text-to-image generation via transformers. Advances in Neural Information Processing Systems, 34. 2021.
[13] Qi Dong, Shaogang Gong, and Xiatian Zhu, Class rectification hard mining for imbalanced deep learning, In Proceedings of the IEEE International Conference on Computer Vision, pages 1851-1860) 2017.
[14] Qi Dong, Zhuowen Tu, Haofu Liao, Yuting Zhang, Vijay Mahadevan, and Stefano Soatto. Visual relationship detection using part-and-sum transformers with composite queries. In Proceedings of the
IEEE/CVF International Conference on Computer Vision, pages 3550-3559,200 2021.
[15] Ross Girshick, Jeff Donahuc, Trevor Darrell, and Jitendra Malik. Rich feature hicrarchies for accurate object detection and semantic segmentation. In 2014 IEEE Conference on Computer Vision and Pattern Recognition, pages 580-587, 2014.
[16] Raghav Goyal and Leonid Sigal. A simple baseline for -upervssuee human-centric relation detection.
In British Machine Vision Conference, 2021.
[17] Jiuxiang Gu, Shafqq Joty, Jianfei Cai,, Handong 7hao, Xu Yang, and Gang Wang. Unpaired image captioning via scene graph alignments. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 10323-10332. 2019.
[18] Yuyu Guo, Lianti Gao, Xuanhan Wang. Yuxuan Hu, Xing Xu, Xu Lu, Heng Tao Shen, and Jingkuan Song. From general specific: Informative scene graph generation via balance adjustment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16383-16392, 2021.
[19] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation.
In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pages 5356-5364,
2019.
[20] Hui Han, Wen-Yuan Wang, and Bing-Huan Mao. Borderline-smote: a new over-sampling method in imbalanced data sets learning. In International conference intelligent computing, pages 878-887.
Springer. 2005.
[21] H He. Y Bai, EA Garcia, and S ADASYN Li. adaptive synthetic sampling approach for imbalanced learning. iece international joint conference on neural networks. In 2008 (IEEE World Congress On Computational Intelligence), 2008.
[22] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016. [23] Sen Hc, Wentong Liao, Hamed R Tavakoli, Michael Yang. Bodo Rosenhahn, and Nicolas Pugcault. Image captioning through image transformer. In Proceedings of the Asian Conference on Computer Vision, 2020.
[24] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6 6709, 2019.

---

[25] Jingwei Ji, Ranjay Krishna, Li Fei-Fei. and Juan Carlos Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10236-10247. 2020.
pages [26] Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In Proceedings of the
IEEE conference on computer vision and pattern recognition, pages 1219-1228, 2018
[27] Bingyi Kang. Saining Xie. Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis, Decoupling representation and classifier for long-tailed recognition. In International Conference
on Learning Representationn 2020.
[28] Siddhesh Khandelwal, Mohammed Suhail, and Leonid Sigal. Segmentation Segmentation-groundee grounded scene graph genera- tion. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 15879-15889, 2021.
[29] Bumsso Kim, Junhyun Lee, Jaewoo Kang. Eun-Sol Kim, and Hyunwoo J Kim. Hotr: End-to-end human- object interaction detection with transformers. In Proceedings of the IEER/CVF Conference on Computer Vision and Pattern Recognition, pages 74-83, 2021.
[30] Ranjay Krishna, Yuke Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 333(1))3-773, 2017.
[31] Rongjie Li, Songyang Zhang, and Xuming He. Sgtr: End-to-end scene graph generation with transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19486-
19496, 2022.
[32] Rongjie Li, Songyang Zhang, Bo Wan, and Xuming He. Bipartite graph network with adaptive message passing for unbiased seene graph generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patiern Recognition, pages 1119-11119. 2021.
[33] Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang. and Xiaogang Wang. Scene graph generation from objects. phrases and region captions. In Proceedings of the IEEE international conference on computer
vision, pages 1261- 1270, 2017.
[34] Tsung- Yi Lin, Priya Goyal, Ross Girshick. Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988.
2017.
[35] Hengyue Liu, Ning Yan, Masood Mortazavi, and Bir Bhanu. Fully convolutional scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages I 11546 11556, 2021.
[36] Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language
priors. In European conference on computer vision, pages 852- 0228669.68 Springer, 2016.
[37] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe. and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pages 181-196, 2018.
[38] Depu Meng. Xiaokang Chen, Zejia Fan, Gang Zeng. Houqiang Li, Yuhui Yuan, Lei Sun, and Jingdong Wang. Conditional detr for fast training convergence. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3651-3660, 2021.
[39] Alejandro Newell and Jia Deng. Pixels to graphs by associative embedding. Advances in neural information
processing systems, 30, 2017.
[40] Mengshi Qi, Weijian Li, Zhengyuan Yang, Yunhong Wang, and Jiebo Luo. Attentive relational networks for mapping images to scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3957-3966, 2019.
[41] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[42] Li Shen. Zhouchen Lin. and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In European conference on computer vision, pages 467-482. Springer, 2016.
[43] Suprosanna Shit, Rajat Koner, Bastian Wittmann, Johannes Pactzold, Ivan Ezhov, Hongwei Li, Jiazhen Pan, Sahand Sharifzadeh, Georgios Kaissis, Volker Tresp, et al. Relationformer: A unified framework for image-to-graph generation. arXiv preprini arXiv:2203. J0202, 2022.
[44] Gunnar A Sigurdsson, Gül Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsourcing data collection for activity understanding. In European Conference on Computer Vision, pages 510-526. Springer, 2016.
[45] Mohammed Subail, Abhay Mittal, Behjat Siddiquie, Chris Broaddus. Jayan Eledath, Gerard Medioni, and Leonid Sigal. Energy-based learning for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13936-13945 2021.
[46] Kaihua Tang. Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang Unbiased scene graph generation from biased training. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 37 571653725, 2020.
[47] Kaihua Tang, Hanwang Zhang. Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to compose dynamic tree structures for visual contexts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6619-6628, 2019.
12

---

[48] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[49] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma- tions for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[50] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the JEEE conference on computer vision and pattern recognition, pages 5410-5419, 2017.
[51] Shaotian Yan, Chen Shen, Zhongming Jin, Jianqiang Huang, Rongxin Jiang, Yaowu Chen, and Xian-Sheng Hua. Pepl: Predicate-correlation perception learning for unbiased scene graph generation. In Proceedings of the 28th ACM International Conference on Multimedia, pages 265 273, 2020.
[52] Jianwei Yang, Jiasen La, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-enn for scene graph generation.
In Proceedings of the European conference on computer vision (ECCV), pages 670-685, 2018. [53] Yang, Kaihua Tang, Hanwang Zhang, and Jianfei Cai. Auto-encoding scene graphs for image captioning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10685-10694, 2019.
2019.
[54] Alireza Zareian, Svebor Karaman, and Shih-Fu Chang. Bridging knowledge graphs to generate scene graphs. In European Conference on Computer Vision, pages 606-623. Springer. 2020.
[55] Rowan Zellers. Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5831-5840, 2018.
[56] Ji Zhang, Kevin Shih, Ahmed Elgammal, Andrew Tao, and Bryan Catanzaro. Graphical contrastive losses scene graph parsing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11535-1 11:5--15444 11543, 2019.
[57] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiangang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04152 2020.
[58] Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV), pages 399-305,100 2018.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] see Section 6.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] see Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] All the data we use in our experiments publicly available. The code for our approach is available at github. conn/ubcvvisie
IterativeSG.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] We use widely popular data splits for our experiments. We briefly describe this in Section The hyperparameters and additional data details are also mentioned in the supplementary (Section B).
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] As scene graph models are computationally expensive to train, none of the existing literature report error bars. However, do report the perfor- mance of our method on different hyperparameter values to enable better understanding of the performance improvements.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] This is described in the supplementary (Section B).
4. If you using existing assets (e.g., code, data, models) or curating/releusing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] All the datasets we use are publicly available. We cite the creators of Visual Genome (Krishna et al. [30]) and Action Genome (Ji ef al. [25]) throughout the paper.
(b) Did you mention the license of the assets? [Yes] Visual Genome [30] is licensed under the Creative Commons Attribution 4.0 International License. Action Genome [25] is licensed under the MIT license.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A] We don't 't introduce any new assets in the paper.
(d) Did you discuss whether and how consent obtained from people whose data you'l 're using/curating? [N/A] Both Visual Genome [30] and Action Genome [25] are publicly available.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "cixcvBtLei", "reviewer": "Reviewer_Vh2W", "summary": "This paper propose an end-to-end scene graph generation framework, in which design iterative refinement manner to gradually optimize the predictions. Besides, it also introduce the reweighting loss to tackle the long tail problem in this task. Author verified this method in two popular dataset and achieve the superior performance. But this manuscript is not well-written and some expressions need to be improved.", "questions": "1. I have a question about the assumption in line 157.  The [1] had pointed out that the direction of the edge is important property in relationship prediction, but you assume a fixed information flow from subject to object and finally reach predicate. Please give the explaination of eliminating graph property in your assumption.\n2. The proposed iterative refinement manner for SGG should involves the variation for both object and relationship, but figure 3 only present the estimation of different refinement steps for relationship.\n3. The scaling parameters for reweighting loss is hard to set and your experiment in Table 1 also attempt various combinations, which present a relatively large impact. So, is there a more effective way to set these two parameters? \n4. Author adopt the MOTIF [2] as baseline. As far as I know, this method use the prior information of dataset to bias the final predicted distribution. I would like to know that in your end-to-end framework, this kind of bias is still necessary for the your prediction? \n[1]GPS-Net: Graph Property Sensing Network for Scene Graph Generation. In CVPR 2020.\n[2]Neural Motifs: Scene Graph Parsing with Global Context. In CVPR 2018.", "limitations": "Scene graph is first proposed to use for image retrieval and it is also the more direct application compared with autonomous driving. So author should focus on some simple but effective applications for scene graph.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#Strengths:\nThis paper reformulate the task of scene graph generation into an iterative optimization process. Based on transformer architecture, author expend the message passing within a Markov Random Field and design iterative refinement procedure. The performance in VG and AG dataset have achieve the state-of-the-art level.\n#Weaknesses:\nAlthough an end-to-end framework for generating scene graph is a good research direction, scene graph also involves object prediction. I think the pretrained object detector in two-stage methods will provide more better prediction, which authors did not given relevant introduction. The qualitative refinement analysis is not enough to support the effectiveness of the proposed method.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "MhCB75CxxnW", "reviewer": "Reviewer_5s37", "summary": "The paper proposes dynamic conditioning using Markov Random Filed for the Scene Graph generation. It implemented transformer based iterative refinement procedure . Conditional quaries has been used during the refinement and to reduce the exponential search space. Three separate multi layer decoder has been used for relation triplet and perform joint reasoning in a single stage architecture. It achieved sota performance on VG and AG dataset. ", "questions": "1. How you have incorporated the method in video, using some tracking? please mention\n2. Please provide clear motivation for three decoders and why it is necessary despite their complexity \n\n", "limitations": "yes.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The paper proposes a single stage Scene Graph generation approach, with novel transformer based formulation which include separate decoder for relational triplet, and conditional query.\n2. The paper show that their results improved VG and AG sota.\n3.  Tested the approach on both video and images.\n\nWeakness : \nDepite their strength this paper lack few fundamental point :\n1. IMHO this paper lacks a strong motivation, author should clearly mention on the introduction why the conditional query is needed and how it can be incorporated to architecture and why they need thee decoder in a one stage..the story should be connected ...\n2. What is the initial query, is it same like DETR or how do u process or the initial query are only subject queries ? (sec 4.2)\n3. It has three separate layer of decoder and we know that scene graph has exponential complexity...author should analyiz dat\n\nAcknowledgement :\nIt would be better if you acknowledge  other single stage paper, like Relationformer - Shit at el, RelTr - Cong at el.. and other earlier  transformer based paper like Seq2Seq- Lu at el, Relation Transformer Network - Koner at el\n\nSuggention :\nThe paper has technical novelty, please write in a way such that it would be easy to follow and understand your contribution\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ITGhrDfGp27", "reviewer": "Reviewer_h8U6", "summary": "This paper proposes an end-to-end paradigm to predict scene graphs from image inputs. The key observation of this paper is that assuming a fixed factorization of subject-object-predicate of predicting relationships can be detrimental, as errors can accumulate in this on direction flow of this information. To alleviate this problem, the authors propose an iterative refinement process, where, although the factorization is still the same within a refinement step, information from previous steps can be utilized by later steps, thus allows information to flow in all directions. A joint matching loss is proposed, using the same matching across all steps, to stabilize this refinement procedure. Comprehensive evaluations show that the proposed method achieves good performances and outperforms many prior works.", "questions": "As mentioned earlier, my main issue is with the (relatively) lack of analysis of the concrete benefits of the core idea of the paper. More evidence, in addition to \"better numbers overall\", that it is solving the issues with fixed factorizations will sway me towards accepting this paper.\n\nSome additional comments:\n- Might help to highlight the best/top performing entries in the larger tables. Very hard to parse the table as for now.\n- Would be interesting to visualize the qualitative behavior of the model without the joint matching loss. I suppose the model is not really doing refinement without that?\n- Following previous point: ablations of model with CAS and CWS, but without JL, would be nice to have.", "limitations": "Limitations are discussed but those are not limitations specific to the central idea of this paper. More discussions on how to improvement the iterative refinement procedure would be more desirable here.\n\nThe authors are right in pointing out that the proposed work takes the right step towards alleviating some of the negative social impacts associated with the problem studied here.\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n+ The idea of using an iterative refinement procedure to alleviate the issue with a fixed factorization is novel, makes sense intuitively, and seems to work on well in practice.\n+ Careful design choices for the aforementioned refinement procedure, including appropriate modification to the inputs of the architecture used (transformers), and a matching loss utilizing the same matching across all steps to ensure the stability of the iterative refinement process.\n+ Comprehensive evaluation and good empirical results, with good discussion of some of the nuances over prior works.\n\n### Weaknesses\n- Overall, I think there lacks enough evidence supporting the central claim of the paper: that this iterative refinement process is important and avoids the problem with a fixed factorization:\n    * It is unclear how the qualitative examples are chosen, and overall too little qualitative examples are shown to support claims that this process fixes the problem with a fixed factorization. More, randomly drawn, examples shown the same qualitative behavior would greatly increase my confidence in the usefulness of this process. I would also expect some results to demonstrate how the proposed method solves issues discussed in the paper e.g. \"where an object entailed in interaction is small and not discernible on its own\" (L7).\n    * Quantitative analysis over the output of each of the refinement step would help to show how the refinement process is actually improving the results. Ablations over the number of steps would also help here. I get that this is somewhat done in Supplementary Table A1, but that can also be due to the models having more parameters overall. I am looking more more fine grained analysis on what specific do the refinement process help the most in improving e.g. if the issue quoted above (L7) is addressed by the proposed method, that can probably be revealed by analysis showing what categories are improved the most over the steps. This currently does not appear to be the case to me since both recall and mean recall follows the same trend in table A1. \n    * The fact that removing both CAS and CWS results in minimal performance drop seems to indicate that there isn't much actual ``refinement\" going on.\n - The claim that the proposed method can be used to improve other methods is not that well supported: although the performance is marginally better, it is unclear whether it's simply due to the modified model having more parameters or not.\n- Originality of the paper is pretty limited apartment from this (still questionable to me) procedure of refinement. The core of the method is still a decoder in the style if [2] and this has been attempted already in earlier literatures e.g. [30].\n- In general, I am not a fan of going into very fine details to show that a method outperforms prior/concurrent works e.g. 14-16, 21-22 in Table 1. Showing that the relative advantages of methods have to come down to such details, to me, is rather unnecessary. A good idea should be able to show its value with more specific evaluation methods / examples in addition to a holistic number about overall performance (see points above).\n\n###Post-rebuttal Comment\n\nI thank the authors for the clarifications regarding my concerns. It seems that I initially missed some evaluations, which did provide decent amount of evidence that there is actually a \"refinement\" process, and gives some level of intuition behind how this process works. Subsequently, I am raising my score from a 4 to a 6. I would encourage the authors to add more discussions around these points in the main paper, as well as providing more qualitative examples in the supplementary.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept