Multi-Modality Alone is Not Enough: Generating Scene Graphs using Cross-Relation-Modality Tokens
Authors: Gopika Sudhakaran
Devendra Singh Dhami
Stefan Roth
Kristian Kersting
Venue: Submitted to ICLR 2023 (2023)
Keywords: scene graphs
transformers
fusion strategies
multi-modal

Abstract: Recent years have seen a growing interest in Scene Graph Generation (SGG), a comprehensive visual scene understanding task that aims to predict the relationships between objects detected in a scene. One of its key challenges is the strong bias of the visual world around us toward a few frequently occurring relationships, leaving a long tail of under-represented classes. Although infusing additional modalities is one prominent way to improve SGG performance on under-represented classes, we argue that using additional modalities alone is not enough. We propose to inject entity relation information (Cross-Relation) and modality dependencies (Cross-Modality) into each embedding token of a transformer which we term primal fusion. The resulting Cross-RElAtion-Modality (CREAM) token acts as a strong inductive bias for the SGG framework. Our experimental results on the Visual Genome dataset demonstrate that our CREAM model outperforms state-of-the-art SGG models by around 20% while being simpler and requiring substantially less computation. Additionally, to analyse the generalisability of the CREAM model we also evaluate it on the Open Images dataset. Finally, we examine the impact of the depth-map quality on SGG performance and empirically show the superiority of our model over the prior state of the art by better capturing the depth data, boosting the performance by a margin of around 25%.

Under review as a conference paper at ICLR 2023
MULTI-MODALITY ALONE IS NOT ENOUGH: GENERATING SCENE GRAPHS USING
CROSS-
RELATION-MODALITY TOKENS
Anonymous authors Paper under double-bling brindd review
ABSTRACT
Recent years have seen growing interest in Scene Graph Generation (SGG), a comprehensive visual scene understanding task that aims to predict the relation- ships between objects detected in a scene. One of its key challenges is the strong bias of the visual world around us toward a few frequently occurring relation- ships, leaving long tail of under-represented classes. Although infusing addi- tional modalities is one prominent way to improve SGG performance on under- represented classes, we argue that using additional modalities alone is not enough. We propose to inject entity relation information (Cross-Relatioes and modality de- pendencies (Cross-Modality) into each embedding token of transformer which we term primal fusion. The resulting acts as strong inductive bias for the SGG framework. Our experimental results on the Visual Genome dataset demonstrate that our CREAM model outperforms state-of-the-art SGG models by around 20% while being simpler and requiring substantially less computation. Additionally, to analyse the generalisability of the CREAM model we also evaluate it on the Open Images dataset. Finally, we ex- amine the impact of the depth-map quality on SGG performance and empirically show the superiority of our model over the prior state of the art by better capturing the depth data, boosting the performance by margin of around 25%.
(CREAM) token
INTRODUCTION
Visual scene understanding has evolved in recent years from mere object detection and recognition tasks to more complex problems such as Visual Question Answering (VQA) (Antol et al. 2015) and Image Captioning (IC) (Hossain et al., 2019). One prominent tool for scene understanding scene graph generation (SGG) (Lu et al. 2016): Given any two entities in scene, the task of SGG is to detect any existing relationships between them. While standard SGG uses entity features from the RGB images to detect relations, to move towards the goal of generating scene graphs that adequately typify our visual world, we need additional clues to effectively capture under-represented classes in SGG. In this regard, researchers have taken multiple directions such as infusing complementary modalities (Zareian et af.] 2020; (Sharifzadeh et al., (02T) or conditioning using additional image context (Lu al.) (021). Among SGG methods based on further modalities, Zareian al. (2020) exploit external knowledge by using a late fusion mechanism in which the scene graphs generated from the RGB features are refined in multiple iterations per relation detection iteration using knowl- edge graphs, resulting in very high computational cost. On the other hand, SGG with additional depth data (Sharifzadeh et al. (02T) has used an early fusion mechanism. Although [Sharifzadeh] et al. (2021) requires less computation, it fails to effectively fuse the modalities and uses depth maps of limited quality (Fig. 7
To effectively fuse different modalities, transformer models can be beneficial, with their usage ex- panding from text (Vaswani al. 2017) to other modalities such as images (Dosovitskiy et al. 2021) and speech (Lin and Wang 2020). Recently, transformers have also been used in SGG, mostly to capture dependencies across time in video-based SGG (Cong et al. (021). In the case of still im- ages, transformers were used to extract object dependencies (Dhingra et al. 2021) and context de- pendencies (Lu et al. 2021). Capturing known dependencies explicitly can boost the performance under-represented classes (Lu et al. 2021). However, using transformers for multi-modal fusion

---

Under review as a conference paper at ICLR 2023
Token Types:
Layer
Class
CREAM
Subj-Depth
Obj-Depth
Subj- RGB
Obj-RGB
Input tokens
Primal fusion
Early fusion
Figure/ 1: Primal vs Early Fusion. In the case of primal fusion the modalities and the subject-ohject features are fused explicitly form the transformer input tokens. Contrastingly, for early fusion there no interaction between the modalities and subject-object features prior to entering the transformer.
can be challenging because of the high computational cost and model complexity (Nagrani et al, 2021]. The major reason for the increased expense in multi-modal transformers stems from the fusion strategies used. Although there is lot of research into multi-modal fusion, it has primarily focused on fusion strategies inside the transformers, which results in an increased sequence length thus increasing the model complexity. Moreover, explicitly modeling the known inductive bias is challenging for fusion strategies that start inside the transformer. The limitations of the existing methods raise the following open questions: (Q1) How can we effectively combine different modal- ities in SGG? (Q2) What is more important for improving SGG performance, better quality depth maps or a better fusion strategy? (Q3) Will having multiple modalities alone be enough to boost the coverage of under-represented classes in SGG?
To address the above questions, we propose simple yet effective token generation strategy that we call Cross-RElAtionn MMddality (CREAM) tokens, by strategically combining entity relations (be- tween the features of the subject and the object entity) with modality dependencies (i.e. between RGB and the corresponding depth modality). As depicted in Fig. [ we explicitly combine the modalities and subject-object features prior entering the transformer unlike early fusion in which we rely on implicit fusion inside the transformer. We call our fusion strategy primal fusion. Sur- prisingly, our primal fused CREAM tokens are able capture the inherent inductive bias in SGG using a single encoder-only transformer without using any complex cross-attention (Chen et al.) 202T), fusion (Prakash al. 202T). or encoder-decoder (Lu et al., 2021) architectures. In partic- ular, we are using our system to study the effect of the depth-map quality in scene graph genera- tion using an improved depth map. VG-Depth. v2, generated for the Visual Genome (VG) dataset (Krishna et al. 2017) using the monocular depth estimator of Yin et al. (202T) and compare to VG-Depth.vl maps (Sharifzadeh al., (202T) (Fig. [ This study is crucial for two reasons: (/) to evaluate the relationship between architectural choice and depth-map quality in SGG perfor- mance and (2) for the efficacy of SGG in real-world visual scene understanding scenarios such as automated driving, where instantaneous 3D reconstruction is far-fetched but using monocu- lar depth estimators to generate depth maps is feasible. We make our code publicly available at www.en.peeenc.....oo...ee.e wwt.seicieeeeneeeeeaa.eeeeaeesee Model-113E
Precisely, we make the following contributions: (CI) We propose novel token generation strategy (CREAM tokens) for transformer based multi-modal SGG. Our CREAM tokens explicitly force the culti--haa--sll-aattenii
(MSA) component of the transformer to learn enriched representation by focusing on different subspace. Our proposal can significantly boost the performance while also reducing the computational cost. (C2) By primaly fusing CREAM tokens, we outperform state- of of-the-art models on the mRecall metric despite not using any additional context. (C3) We conduct extensive dapth-data analysis and ablation studies to show the significance of our proposed approach.
RELATED WORK
SGG. Scene graphs have been receiving increased attention from the research community due to their potential usability in assisting downstream visual reasoning tasks (Shi et al.) 2019 Wang al.) 2019 Krishna et al. 2018). The scene graph generation (SGG) task was first introduced by Lu al. 2016). The bias problem in SGG was first brought into focus by Zellers et al. (2018) and an
unbiased evaluation metric, meanRecall (mRecall), was proposed by Chen et al. 2019a) and Tang al. 2019). Various debiasing techniques were proposed subsequently such using unbiased loss
2

---

Under review as a conference paper at ICLR 2023
functions including the class-balanced loss (Zareian et al. 2020), employing reinforcement learning (Lu et al. 2021), and disentangling biased representations from the unbiased (Misra et al. 2016 Cadene et al. 2019 Tang et al., 2020h.
Privileged Scene Graph Generators (PSGGs). Recent progress in SGG pursues additional in- formation sources to generate scene graphs beyond the standard subject-object features, labels, and bounding boxes. We call these models privileged SGG (PSGG) models. One such direction in-
volves utilizing context information (Lu al., 2021; Wang et al. 2019 Hung et al., 2020 Chen et al. 2019b). Lu et al. (202T) introduce a transformer- based SGG approach that propagates global context by making sequential prediction on the decoder by conditioning on the earlier decoded pre- dictions. Through experiments, we show that CREAM model that uses depth as additional informa- tion instead of context, outperforms Lu et al. (202T), Another set of PSGGs use external knowledge graphs (KG) as additional information (Zareian et al., 2020 Chen et al. 2019a Gu et al.) 2019). KG-based SGG generally uses Graph Neural Networks for KG-SGG information propagation. Due to the complex architecture required for KG-based SGG, these models are highly computationally expensive. We show that our proposed approach outperforms KG-based SGG by a significant mar- gin. (Sharifzadeh et al. (202T) and (Yang et al.) (2018) have explored using depth maps as additional information source for SGG. However, the depth-maps used are of limited quality and the effective fusion of the depth modality with the RGB data has not been explored in detail. To address this, we analyse the impact of the dapth-map quality on SGG performance and propose an effective way to combine the modalities and the entity features.
Two stage and one stage SGGs. The above works use two stage SGG pipeline in which the ob- ject proposals and relation detection are carried out at two separate stages using pretrained object detectors. On the other hand, one stage SGG (Li et al.] 2022 Zou et al.] (02T) generates the ob- ject proposals and entity relations in parallel. Though one stage SGG can produce sparse proposal set and faster models they are less generalisable. Thus, considering the generalisability and the availability of off-the-shell object detectors we use the two stage SGG pipeline in our work.
3. PRIMAL FUSION: EARLY FUSION OF INDUCTIVE BIASES
To introduce primal fusion and to illustrate it using CREAM, let us briefly review the the core idea of the transformers exploited in our work and the significance of the proposed fusion strategy.
Huutt-Head-SSlf-AAteee Self-Attention (MSA): The core idea of our work is to effectively ex- ploit the MSA component of the transformer (Lu et al.) 2021]. MSA K)
Concat (SA1, (AA,) oo with SA, = softmax Q.K.
Val V.. where SA refers to self- attention, Q, K, and V refer to query, key, and value, and Wo are the trainable parameters. The input token to the transformer is split into multiple heads to be attended to in parallel by the MSA.
The key to successfully capture the relationship between subject-object pairs in SGG set-up is to effectively express the subject-object joint features that accounts for their intricate relationship. The MSA component of the transformer can jointly attend to diverse features by enabling each head to focus on different subspace with different semantic or syntactic meanings. Hence, we propose to use the MSA component to capture the inductive bias in the scene graph generation task. By enriching each token with both the modalities and subject-object features, we are exploiting the multi-head--slf-attenne
component of the transformer effectively by feeding it with different sub- space representations to leam from. As a result, the MSA with multiple heads can jointly attend to the diverse features which enables the transformer to learn richer representation.
Multi-Model Fusion: The question of where to fuse the modalities in a multi-modas setup is a pressing issue. Various fusion methods such as early, mid, late and bottleneck fusion were introduced (Nagrani et (002). Early and mid fusion passes the token sequences of all the modalities as input to the transformer and the cross-modal information is exchanged implicitly starting from the initial or later layers in the model. In the case of late fusion, no cross-modal information is exchanged. The bottleneck fusion mechanism restrict the pairwise self-attention across specific units. Deviating from the above, our primal fusion explicitly fuse each embedding token with known dependencies, which acts as a strong inductive bias for SGG. In addition to performance gain the overall sequence length of the input tokens is reduced to single modality sequence while having enriched tokens.
3

---

Under review as a conference paper at ICLR 2023
Token Types
CREAM
Leereblee
Cless, Location
channel, wi height
Clocsleer
SGG
Transfocunt
Learbable enkeddings
Primel Fusion
Cruss Modality
prro
pro
Cross Relation Pusion
Cross Relation Posion
2 Pa
Fiee
RPN
Depth Feoture
7% avg pool
pool
d. BPg pool
t.d. lang pool
Extrrcttrr
Extruster
Depth Netwoek
RGB
deyth-mag map
Fox Mr.h
Mr.
Figure 2: The CREAM architecture. An object detector, operating cach RGB image, first yields en- tity proposals and entity features v. Moreover, depth map is estimated from the RGB input, which is also passed through the feature extractor obtain depth features Then, for each subject-object pair, sequence of wweeaeiio--ooaaliiii tokens are generated through Cross-Relation subject-object features. The resultant tokens are passed through the transformer to yield relation prediction.
Cross- Modalityy fusion
3.1 THE CREAM ARCHITECTURE
Before describing the CREAM architecture let briefly review the standard two-stage SGG pipeline that serves as the basis of our SGG approach. The first stage of a standard SGG pipeline (Lu et al., 2016) starts with image feature extraction followed by detection and classification of the entities in the image. The major components for relation detection between entities form part of the second stage. For any given pair of entities detected in an image, the second stage of a vanilla SGG model extracts the RGB features of the entities' bounding box regions (location). and projects these RGB features along with their location and class information. The final layers of the second stage then contain multi-layer perceptron followed by relation classification head.
Our CREAM architecture also comes under the two stage SGG pipeline. Fig. 2 shows the overall architecture of our proposed estimate its depth map using a monocular depth estimator. We use Faster R-CNN (Ren et al, 2015) to generate object proposals the RGB features extracted from /, restricted to the proposal region v. The same proposal information is used to extract the corresponding depth features d. The object proposal information constitutes the bounding boxes, also called the location information and the class probability distribution c. Followed by feature extraction, the input tokens for the transformer encoder are generated such that each token sequence contains information of a single subject-object pair. Each token contains primal fused information from a part of the subject-object pair from both RGB and depth modalities. The resultant tokens are passed through the transformer encoder-only model to predict the relations for the subject-object pair.
transformer. For a given image T, we first
3.2 GENERATING CREAM TOKENS
Next, we describe the CREAM token generation strategy for given pair of subject proposal and object proposal o. Let co, w. h and ca, w, h be the channel, width, and height dimensions for v and d, respectively. Further, let v, and c, be the RGB features of the subject and object region proposals, and let d, and d, be the corresponding depth features of the subject and object region proposals. Our primal fusion constitutes two stage fusion. In the first stage average pool U., Vo. d,, and d, along and w to summarize the average presence of the features that are crucial for relation prediction. The resultant pooled features have height and width which we fuse channel-wise for each modality. Thus the first stage fusion comprise of Uso = fusion, (v., v,) and d = fusion, (dg, d,), where fusion (.) denotes the pooled features that are combined along the channel dimension. Then split
4

---

Under review as a conference paper at ICLR 2023
Uso channel-wise into sequential patches such that if Co. P. p. and lo are the channel, width, height, and patch size of Uso: we obtain (p/k)² patches for Usa- Similar Usa. the fused depth features dso are also split into sequential patches such that we obtain (p/k)² patches for dæo with channel size Cd. Each individual patch comprise of the summary of object features in specific region, now by combining the subject and object patches together, we are making explicit to the transformer that there are some feature dependencies between the subject and object patches that need to be exploited for the relation prediction. Hence, the first stage fusion corresponds to the Cross-Relation part of the token generation.
Note that this differs from Lu et al. (021)) who use transformers to capture relations between entities and treat each entity the same way text token in natural language processing. For SGG setup, this assumption is limited because for effective relation detection we should capture dependencies between localised parts within and between entities rather than considering the entity as a whole. While splitting a word would make it meaningless, splitting an object would aid capturing its part compositions. Our Cross-Relation component overcomes this limitation by capturing the feature dependencies more effectively by facilitating the model to attend to crucial dependent features easily when passed through the transformer to predict the relation.
Followed by the Cross-Relation component creation, the second stage fusion starts where we lin- early project the patches of both the modalities such that the depth modality has higher projected dimensionality than the RGB data, owing to the need to represent 3D structure: proj_ _ = I (#so: p'), proj d' = f (dao- p²) where f(-) denotes the linear projection, p" and pa the projec- tion dimensionality of Ueo and deo. and denotes the patch number ranging from 1 to (p/k)², Note that the sensitivity of our model to the exact proportion of depth and RGB dimensions in CREAM tokens is negligible in practice (Fig After linear projection of all the patches, the corresponding patches of both the modalities are fused together. This reduces the total number of tokens from ((p/k)² 2) to (p/k)². The second stage of fusion combines Cross-Relation patches of both the modalities, resulting in Pross-Relation-oo Relation-Modallty token generation: z = fusiony (vi., d(,o) where fusion2(.) denotes the modality specific projection f(-) followed by
fusion of de- pendant features. The modality dependency is based on the fact that every feature patch in the RGB modality has a corresponding feature patch in the depth modality. Thus the fusiong reduces the length of the token sequence while ensuring that dependent modality information is closely-knit to be effectively exploited by the Multi-head Self Attention (MSA) mechanism.
Additionally, the class and the location information obtained from the region proposal network for the subject-ohject pair are combined and linearly projected to replicate the token dimension p" + p². This results in two additional tokens: 2(r/k)² +1 = h (f (Iso, p" + p²)) and 20//))+2 +2 (f (c20.p" + )) where f(-) is the linear projection and h(-) the non-linear function and lao- Cso denotes the location and class information of the subject-object pair. As we -Moaaity token generation strategy, our encoder- will see in Sec. 4 by using this tht-hhe-ttttt art models despite not using any cross-attentin complex only model outperforms prior state-of-ff oft------hhhhhee fusion mechanism, or global context propagation.
3.3 TRANSFORMER ENCODER
We use standard vision transformer encoder architecture (Dosovitskiy et al. 2021 We prepend the CREAM tokens with an additional learnable token 2g/ass and learnable ID positional embed- ding 2008 is added to each token to preserve the positional information. The resultant sequence is passed as input to the encoder. The transformer consists of L encoder layers, each with MSA and MLP blocks. Each encoder layer also contains LayerNorm (LN) before every block and residual connections after every block:
= Lglass, =b::..
(p/k))
(0//k)) (p/R)++1 (p/k))+2] ;20
+ 2008 200 E (((0//))+3)x(((
2o
-00 3.40
4 MSA (LN (22-1))
22-1,
1.....L
(2)
28 = MIP (LN (z4) + 2/2
1.1...
(3)
y LN LN (=7)
(4)
Finally, y is linearly projected to total number of predicate relations and forms our relation classifi- cation head.
5

---

Under review as a conference paper at ICLR 2023
Table I: mRecall results on the Visual Genome (VG) dataset (the higher, the better). Context models have additional privilege to mitigate errors propagated from classification (SGCLS) and detection (SGDET) unlike the knowledge graph (KG) and depth-privileen models. Hence, our major baselines for SGCLS and SGDET tasks are based on KG and depth. The best results overall on the SGCLS and SGDET tasks are denoted by the superscript *
PRDCLS
SGCLS mRecall @
SGDET
Model
Privilege
mReecll @
mRecall
20
50
100
20
50
100
20
50
100
MOTIFS Zellers et al.
2018
context
10.8
14.0
15.3
6.3
7.7
8.2
4.2
5.7
6.6
FREQ
Tang ct al. 2019
context
8.3
13.0
16.0
5.1
7.2
8.5
4.5
6.1
7.1
RTN
Koner ct al.
2020
context
20.3
12.6"
RU-NeI
Lin et al. 2022
context
24.7
13.9
10.1"
VCTree Tang et al 2019
context
14.0
17.9
19.4
8.2"
10.1"
10.8
5.2"
6.9"
8.0
KERN CChen ct al.
2013a
KG
17.7
19.2
9.4
10.0
6,4
7.3
GB-NET (Zarcian ei
2020
KG
19.3
20.9
9.6
10.2
6.1
8.3
Depth- VRD DeDth-VRDD (Shani zadeh el al. 1
depth
16.4
20.7
22.7
CREAM (ours)
depth
20.3
25.8
28.1
10.5
5.2
6.5
Training. For RGB and depth feature extraction, we follow the procedure of our baseline mod- els (Zellers et al. 2018 Sharifzadeh et al.] (02T). We train our model using the Adam optimizer (Kingma and Ba 2015) with batch size of 4 images and learning rate of 1e 3. For our trans- former model, we use patch size of 2. pooled entity resolution of 8, leading to a total of 19 input tokens ((p/k)² +3) where p = 8, k = 2) with embedding size of 576; we employ attention heads and 6 encoder layers. The model is trained for 45 epochs. We cross-entropy as the loss function C. To compare our model against previous dehiased SGG models, we use a modified cross-entropy loss that
importance weighting for each class as proposed by Cui et al. (2019), which was also used in Zareian et al. (2020). Specifically, we use the following class balanced cross-entropy loss: LCB [p. v) T- L(P, v). where p denotes the predicted relation probabilities, y/ denotes the ground-truth relation, ny the number of samples in the ground-truth relation y. and , denotes the hyperparameter. / would result in the standard cross-entropy loss, whereas increasing / strenghtens the debiasing. We used the setting from Zareian et al. (2020) for our class-balanced loss.
I-
4 EXPERIMENTAL EVALUATION
We use the SGG benchmark of the Visual Genome (VG) dataset (Krishna et al. 2017) for our evaluation and the split proposed by Xu et al. (2017). We conducted our experiments on 3 tasks: (1) predicate classification (PRDCLS), (2) scene graph classification (SGCLS), and (3) scene graph detection (SGDET). To ensure fair comparison against leading baselines, we created two variants of our models: (I) CREAM and (2) CREAM (8-loss). While both the models are compared against leading PSGGs in general, the CREAM (8-loss) uses a balanced cross-entropy loss (B-loss) as a debasing strategy yield fairer comparison against leading baselines that also uses additional debiasing strategies. We compare against PSGGs with additional context, knowledge graphs and further modalities. We trained each of our models 5 times with different random seeds. The observed variation in performance across training was very small with a variance of 0.20.
Results and analysis. mRecall results without using any debiasing strategy are shown in Table 1 For a fair comparison, all the baselines that are chosen for this comparison also do not use any de- biasing strategy. We notice that our approach, denoted as CREAM, improves over the prior state of the art by more than 20% on the predicate classification task. For the scene graph classification and detection tasks, our model outperforms both the depth and knowledge graph-based SGG approaches. It can also be noticed that the models with additional context information show slightly better per- formance on the SGCLS snd SGDET tasks. This is expected because image context can provide additional clues to the model as it is independent of the classification score and the region proposals from the pre-trained Faster R-CNN (Ren et al.) 2015) for any given subject-object pair. On the other the same region proposals from Faster-RCNY for
hand, the extracted depth features are based given subject-object pair, and hence the errors get propagated. One interesting future work direction to enhance performance further would be to bring in context into our CREAM tokens.

---

Under review as a conference paper at ICLR 2023
Table 2: mRecall results for debiased models on Visual Genome (VG) (the higher, the better). Context models have additional privilege to mitigate errors propagated from classification (SGCLS) and detection (SGDET) unlike knowledge graph (KG) and depth- pritt-privileeed models. Hence, our major baseline for SGCLS and SGDET tasks is based on KG. The best results overall on the SGCLS and SGDET tasks are denoted by the superscript *
PRDCLS
SGCLS
SGDET
Model
Privilege
mRecall@
mRRecall
mRRecall@
20
50
100
20
50
100
20
50
100
MOTIFS Tang et
2020
comtext
18.5
24.9
28.3
11.1
13.9
15.2
6.6
8.5
9.9
VTrasFE-TEEGATT Tang 8020
context
18.9
25.3
28.4
9.8
13.1
14.7
6.3
8.5
10.2
VCTre+TDE::SMM Tlang el a.
1020)
conlext
18.4
25.4
28.7
8.9
12.2
14.0
6.9
9.3
11.1
Seq2Seq Rossoss [Lu
context
21.3
30.5
11.9"
BGNN: multitask loss (L 2021
context, resampling
30.4
32.9
14.3
16.5'
10.7'
12.6'
GB N.N::E---os loss Zareian al. 20200
KG
22.1
24.0
12.7
13.4
7.1
8.5
CREAM:3-loss (purs)
depih
25.1
31.6
34.8
10.4
12.8
14.0
5.8
7.9
9.1
Table 3: mRecall results for PRDCLS task on Open Images v4 and v6 (the higher, the better). Baseline results from official code are denoted by the superscript 9.
PRDCLS (v4)
PRDCLS (v6)
Model
Privilege
mRecall@
mRecull @
20
50
100
20
50
100
MOTIFS' (Zellers et al 2018 Depth-VRD (hhrrlzadeh ef al. 2021
context
83.6
84.2
84.4
70.3
70.5
70.5
depth
80.2
80.4
80.4
69.8
70.0
70.0
CREAM(ourS)
depth
92.2
92.4
92.4
76.2
76.3
To compare against models that additionally use unbiasing strategies, have created variant of our model as shown in Table 2 While there are various debiasing strategies (Zareian et al., 2020)
TTag et al. 2020 Lu al., 2021), our motivation is to show the effectiveness of our proposed architecture in effectively utilising relation and modality dependencies. Hence, we chose a simple debiasing technique that uses a class-balanced cross entropy loss (B-loss). More complex debiasing strategies may result in further improvement in performance; we leave that for potential future study. As discussed earlier, our biggest improvements are expected for the predicate classification task. Table 2 confirms once more that also our debiased model outperforms prior state-of state-of-the-arr
debiased models. It should also be noticed that our model outperforms BGNN (Li et al., 202I) that additionally does bi-level data re-sampling in addition to debiasing and using context.
Fig. 3 depicts the leaes recall performance for our CREAM model in comparison to the Depth- VRD (Sharifzadeh et al.] 2021). Both the models are trained using the VG-Depth.v2 dataset (see below). The yellow line denotes the long-tail of under-represented classes in the VG dataset (Krishna ct al. 2017). It can be noticed that when we get closer to the tail, our CREAM model has higher recall values per class than Depth- VRD, showing how our CREAM tokens more effectively exploit the modality dependencies compared prior depth based multi-modal SGG approaches. The trend in Fig.B is also confirmed by the qualitative samples in Fig. While Depth-VRD captures widely occurring relations such as 'wearing' and 'on', it fails capture rare spatial relations in the data such as 'sitting on', 'covered in', and the co-occurring triplet 'Table and chair'. Our CREAM model's s success at capturing both the rare spatial and co-occurring relations (highlighted in green) shows the effectiveness of our Cross- Relation and Cross-Modality combined CREAM tokens.
To investigate on the generalisability of our CREAM model we further evaluate on Open Images v4 and v6 dataset (Kuznetsova et al., 2020). We use the same data split and backbone used by Li al. (202T) and compare the mkccall performance on the PRDCLS task. As depcited in Table CREAM model consistently outperforms the baseline models once again proving its effectiveness.
Depth-map analysis. We analysed the significance of the depth-map quality on the SGG perfor- mance and the importance of careful architectural design to make use of the depth map efficiently. As depicted in Table[4al the performance improvement for our model on VG-Depth. v2 generated us- ing the monocular depth estimator of Yin et al. (2021) over VG-Depth. v1 (Sharifzadeh et al.) 2021) is more than 20% for all the mRecall thresholds. To the contrary. the model of Sharifzadeh et al. (222)) shows noticeable improvement on the high quality VG-Depth.v2 dataset. This reveals that the improved performance of our model is not just due to the high-quality depth maps (VG- Depth.v2) but due to CREAM architecture, which allows to take advantage of the improved
7

---

Under review as a conference paper at ICLR 2023
Depá:
40%
0.9
35%
0.8
25%
0.5
20%
15%
10%
uo Bemte
paPAR Poran
B040
un Dampppo
Goan
Prodicato Clooses
Figure 3: Class-level recall 100 for PRDCLS. The yellow line depicts the data distribution per predicate class. Our CREAM model, denoted using blue bars, has higher recall than Depth-VRI Sharifzadch et al. 2021 for under-represented classes.
CREAM
CREAM
Man sitting bench Man wearing shirt
Bear has Bear on bed Book on bed Bottle on bed
Man wearing pant Man holding umbrella
Mountain covered in tree Building behind people
Bear holding bottle
on book
Man at table
Table and chair
Man at table
Curtain behind table
Depth-V RRD
Depth-V
Man holding umbrella
Rear has
Man wearing shirt
Bear on bed
wearing pant
Book on bed
Man at table
Bottle on bed
Man at table
Paw on
Figure 4: Qualitative results for PRDCLS. The triplets correctly predicted by CREAM while missed by Depth- (Sharifzadeh et al., 2021) are highlighted in green. The CREAM model is successful at predicting both spatially-aware scenarios such as 'sitting on', "covered in' (left) and relation-aware scenarios 'and" (right) missed by Depth-VRD.
depth maps in effective manner. Also, our model's results on VG-Depth.vl reveal the quality limitations of VG-Depth.v v1, as despite the improved modeling power of our CREAM architecture, there is no significant mRecall improvement on VG-Depth.vl.
Computational cost. We compare the computational cost of our model against leading baselines in Table 4b For fair comparison, all the models timed on single NVIDIA A100 GPU with 40 gigabytes of memory. The most crucial aspect is the inference time of the model when it is deployed in real-time scenarios. Our model is 20 times faster than GB-NET (Zareian al. 2020) while being significantly better at capturing under-represented classes. also has around 35% and 3% fewer parameters than GB-NET and Depth-VRI. respectively.
Table Impact of depth-map quality and computational cost of CREAM vs. the state the art.
Mitil
(ours)
VG-Dspiks
VC-Dcpli
Improsennnt
Model
Tramning Time (minkkpoch)
Inference Time
(is/inngge)
AParameters
(million)
GB-NET Depch-VPR VRD
500
25
298
Depeh-VRR
CREAM (ours)
26
25
288
(a) mReecal performance improvement Visual Genome (VG) PREDCLS (higher, the better).
(b) Computational cost comparison to the state of the art.
8

---

Under review as a conference paper at ICLR 2023
Table 5: Ablation study. mRecall results on Visual Genome (VG) for β-loss model variants (the higher, the better). The token design of the model variants is considerd v: visual, d: depth, subject, and o: object information; '.' denotes separate tokens.
PRDCLS
Model Variants
Model Description
mRecall @
20
50
100
VSO vso-dso vds-vdo ys-vo-ds-de
Cross-Relation (no depth)
15.7
19.43
21.0
Cross-Relation
21.2
25.5
27.5
Etoss-Modalit,
17.1
21.4
22.0
no cross tokens
16.7
20.0
21.4
CREAM (vdso)
LuassRRelation-Moddaa
25.1
31.6
34.8
Ablation study. To evaluate the importance of our CREAM token design, we conducted an ex- tensive ablation study as shown in Table Here, vso represents model variant without using the depth modality such that every token in the sequence contains only RGB-based information about the subject-object pair. vso-dso denotes the model that uses both depth and RGB modalities but each token in the sequence contains either depth or RGB subject-object pair information and not both the modalities. vel-vdoo denotes model in which each token contains information about both depth and RGB modalities but either only subject or only object information. Finally, vs-vo-ds-oo denotes the model in which the modalities and relations are all split into separate tokens. As Table 5) shows, the removal of depth features results in the highest performance drop. Moreover, Cross- Relation seems be the most crucial fusion combination that outperforms other model variants. The effectiveness of Cross-Relation fusion strengthens our claim that for effective relation detection is crucial capture the dependencies between localised parts within and between entities. We can also notice that the standard transformer token variant that does early fusion (us-vo-ds-do) improves only slightly over the no-depth model vso. Moreover, we find that Cross- Mooss-Modality fusion (vds-vdo) results in an improvement over no-cross tokens model. Overall, our CREAM model that exploits Cross-Relation Relation and Cross-Modalle fusion significantly outperforms all other model variants.
Now are ready to answer the open questions raised in the Introduction] T
Q1. Effective way to combine different modalities in SGG: Through strong quantitative and quali- tative results of our CREAM model discussed above, we prove that explicitly capturing the known inductive biases is one way to effectively combine different modalities in SGG.
Q2. Better quality depth maps better fusion strategy: From the depth data analysis in Table 4a can conclude that it is crucial to have both better depth maps and better fusion strategy to improve the SGG performance.
Q3. Boosting the coverage of under-represented classes in SGG using multi-modality alone: Our ablation study results in Table 5 proves that multi-modality alone is not enough, effectively combin- ing the Cross-Relan Relation and Cross-Modality information crucial at boosting SGGs performance on under-represennted classes.
CONCLUSION
In this paper, we show that simply relying on multiple modalities is not enough cope with under- represented classes in SGG. Careful attention must be paid to how the modalities and features are processed. We propose an effective way of moving towards multi-modalii in SGG by introducing CREAM tokens, which enables the model to learn richer representation by explicitly capturing the inductive bias in an SGG set-up. We outperform leading SGG baselines by feeding CREAM tokens to the transformer using our primal fusion technique. Through an extensive ablation study and depth data analysis, we show the significance our proposed approach. Though our model shows significant performance improvements on the PREDCLS task, to further mitigate the classification and detection errors in SGCLS and SGDET tasks we will have to bring in additional image context. One interesting avenue for future work would be enrich CREAM tokens with context. Another promising research direction would be to expand our CREAM architecture to an end-to-end relation system that unifies Multi-Modality and Cross-Modality correlations.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zit- nick, and Devi Parikh. VQA: Visual Question Answering. In Proceedings of the IEEE Interna- tional Conference on Computer Vision, pages 2425-2433, 2015.
Remi Cadene, Corentin Dancette, Matthieu Cord, Devi Parikh, et al. RUBi: Reducing Unimodal Biases for Visual Question Answering. In Advances in Neural Information Processing Systems, volume 32, 2019.
Chun-Fu Richard Chen, Quanfu Fan, and Rameswar Panda. CrossVIT: Cross-Attention Multi-Scale Vision Transformer for Image Classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 357-366, 2021.
Tianshui Chen, Weihao Yu, Riquan Chen, and Liang Lin. Knowledge- Embedded Routing Network for Scene Graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6163-6171, 2019a.
Yunian Chen, Yanjie Wang, Yang Zhang, and Yanwen Guo. PANet: A Context Based Predicate Association Network for Scene Graph Generation. In Proceedings of the IEEE International Conference on Multimedia and Expo, pages 508-513, 2019b.
Yuren Cong, Wentong Liao, Hanno Ackermann, Bodo Rosenhahn, and Michael Ying Yang. Spatial- Temporal Transformer for Dynamic Scene Graph Generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16372-16382, 2021.
Yin Cui, Menglin Jia, Tsung-Yi Yi Lin, Yang Song, and Serge Belongie. Hass-Balanced Loss Based on Effective Number of Samples. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2219.
Naina Dhingra, Florian Ritter, and Andreas Kunz. BGT-Net: Bidirectional GRU Transformer Net- work for Scene Graph Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2150-2159, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In Proceedings of the International Conference on Learning Representations, 2021.
Jiuxiang Gu, Handong Zhao, Zhe Lin, Sheng Li, Jianfei Cai, and Mingyang Ling. Scene Graph Generation with External Knowledge and Image Reconstruction. In Proceedings the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1969-1978, 2019.
MD Zakir Hossain, Ferdous Sohel, Mohd Fairuz Shiratuddin, and Hamid Laga. A Comprehensive Survey of Deep Learning for Image Captioning. ACM Computing Surveys, 51(6):1-36, 2019.
Zih-Siou Hung, Arun Mallya, and Svetlana Lazebnik. Contextual Translation Embedding for Visual Relationship Detection and Scene Graph Generation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(1 1)33200-3822, 2020.
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proceedings of the International Conference on Learning Representations, 2015.
Rajat Koner, Suprosanna Shit, and Volker Tresp. Relation transformer network. arXiv preprint arXiv:2004.066 00040006933. 19,, 2020.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual Genome: Connecting Lan- guage and Vision Using Crowdsourced Dense Image Annotations. International Journal of Com- puter Vision, 123(1)322-73, 2017.
Ranjay Krishna, Ines Chami, Michael Bernstein, and Li Fei-Fei. Referring Relationships. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6867-6876, 2018.
10

---

Under review as a conference paper at ICLR 2023
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesmikov, et al. The open images dataset v4. International Journal of Computer Vision, 128(7):1956-195) 1981, 2020.
Rongjie Li, Songyang Zhang, Bo Wan, and Xuming He. Bipartite graph network with adaptive mes- sage passing for unbiased scene graph generation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2021.
Rongjie Li, Songyang Zhang, and Xuming He. Sgtr: End-to-end scene graph generation with trans- former. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, 2022.
Xin Lin, Changxing Ding, Jing Zhang, Yihing Zhan, and Dacheng Tao. Ru-net: Regularized un- rolling network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Yan-Bo Lin and Yu-Chiang Frank Wang- Audiovisual Transformer with Instance Attention for Audio Visual Event Localization. In Proceedings of the Asian Conference on Computer Vision, 2020.
Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual Relationship Detection with Language Priors. In Proceedings of the European Conference on Computer Vision, pages 852- 869, 2016.
Yichao Lu, Himanshu Rai, Jason Chang, Boris Knyazev, Guangwei Yu, Shashank Shekhar, Gra- ham W Taylor, and Maksims Volkovs. Context-Aware Scene Graph Generation with Seq2Seq Transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15931-1594 2021.
Ishan Misra, C Lawrence Zitnick, Margaret Mitchell, and Ross Girshick. Seeing Through the Human Reporting Bias: Visual Classifiers from Noisy Comaan-Centic Labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2930-2933 2016.
Arsha Nagrani, Shan Yang, Anurag Arnb,, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention Bottlenecks for Multimodal Fusion. In Advances Neural Information Processing Systems, volume 34, 2021.
Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-Modal Fusion Transformer for End-to- End Autonomous Driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7077-7087, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. In Advances Neural Information Processing Systems, volume 28, 2015.
Sahand Sharifzadeh, Sina Moayed Baharlou, Max Berrendorf, Rajat Koner, and Volker Tresp. Im- proving Visual Relation Detection using Depth Maps. In Proceedings of the 25th International Conference Pattern Recognition. pages 3597-3604, 2021.
Jiaxin Shi, Hanwang Zhang, and Juanzi Li. Explainable and Explicit Visual Reasoning over Scene Graphs. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pages 8376-8384, 2019.
Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to Compose Dynamic Tree Structures for Visual Contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern recognition, pages
2019.
Kaihua Tang, Yulei Niu, Jianqiang Huang. Jiaxin Shi, and Hanwang Zhang, Unbiased Scene Graph Generation from Biased Training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3716-3725 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Advances in Neural In- formation Processing Systems, volume 30, 2017.

---

Under review as a conference paper at ICLR 2023
Wenbin Wang, Ruiping Wang, Shiguang Shan, and Xilin Chen. Exploring Context and Visual Pattern of Relationship for Scene Graph Generation. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition, pages 8188-8888888888 819-81977780 2019.
Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene Graph Generation by Itera- tive Message Passing. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5410-5419, 2017.
Hsuan- Kung Yang. An-Chieh Cheng, Kuan- Wei Ho, Tsu-Jui Fu, and Chun- Yi Lee. Visual Relation- ship Prediction via Label Clustering and Incorporation of Depth Information. In Proceedings of the European Conference on Computer Vision Workshops, 2018.
Wei Yin, Jianming Zhang, Oliver Wang, Simon Niklaus, Long Mai, Simon Chen, and Chunhua Shen. Learning to Recover 3D Scene Shape from Single Image. In Proceedings ofe IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Alireza Zareian, Svebor Karaman, and Shih-Fu Chang. Bridging Knowledge Graphs to Generate Scene Graphs. In Proceedings of the European Conference on Computer Vision, pages 606-623, 2020.
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motif's: Scene Graph Parsing with Global Context. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5831-5840, 2018.
Cheng Zou, Bohan Wang, Yue Hu, Junqi Liu, Qian Wu, Yu Zhao, Boxun Li, Chenguang Zhang, Chi Zhang, Yichen Wei, et al. End-to-end human object interaction detection with hoi transformer. In Proceedings of the IEEE/CVF conference on Computer Vision and Pattern Recognition, 2021.
12

---

Under review as a conference paper at ICLR 2023
A APPENDIX
This section contains additional results and analysis that were excluded from the main document due to space constraints. At first carry a sensitivity analysis of the modality proportion in the CREAM token. Then we discuss the mRecall results across model runs to show the stability of our model. We conclude by providing the Recall results and additional qualitative samples.
A.1 SENSITIVITY ANALYSIS
We conducted modality proportion sensitivity analysis depicted in Fig. 5 varying the number of projection dimensions of RGB vam and depth dam- which in total sum to a dimensionality of 576.
Starting with fdim 64 and dam 512, the relative proportion is varied across runs by increasing dim by 64 and decreasing ddim by incrementally. The seed of all the model variants fixed to O and the resulting mRecall @ 50, and 100 summed get the best overall results. In Fig. [5] we observe that increasing Udim and reducing daim from left to right shows an overall decreasing trend. This trend could be attributed to the fact that the depth modality requires higher number of projected dimensions to represent the high-quality 3D information. Note that this hypothesis is consistent with the increased file size of the depth maps in comparison the RGB data. Moreover, we observe that our CREAM architecture seems not to be particularly sensitive to the modality proportion. We have used Udim = and ddim = 512 in our CREAM model variants.
A.2 MODEL STABILITY
In the main document have reported the average of five model runs obtained by using random seeds O to 4. Fig. 6 depicts the mRecall results obtained for each of those five model runs. Our proposed CREAM model is highly stable with mRReclll@20 20, 50, and 100 the set ranging from 24.3 to 25.8, 31.2 to 32.2, and 34.2 to 35.3, respectively. The mRecall results thus have low variances of 0.27, 0.18, and 0.17, respectively.
A.3 RECALL RESULTS
Though the major focus of our work is to improve the coverage of under-represeented classes by using mRecall as the evaluation metric we additionally analysed our models performance on Recall metric. Similar to the observation of previous works Lu et al. (2021); [Tang et al. (2020); Zareian eet al. (2020) we also noticed that optimising for Recall results in degradation of mRecall and vice versa. While the optimal mRecall results obtained the learning rate of 1e 1--3. 3, optimal Recall results were obtained at the learning rate le - 4. Table 6 shows that our Recall results obtained
learning rate of le are also competitive with the state the art for the PREDCLS task. At
le - 3 learning rate we noticed around 1% drop in Recall results reported in Table 6
A.4 VG-DEPTH.VI VS VG-DEPTH.v2
Figure shows "noisy" depth-map samples reported in Sharifzadeh et al. (202I). Comparing the noisy depth maps from Sharifzadeh et al. (2021) (VG-Depth) from the second row to the corre- sponding high-quality depth maps (VG-Depth.v2) in the bottom row.
A.5 QUALITATIVE SAMPLES
To show the effectiveness of our method qualitatively, Fig. 8 and 9 compare scene graphs generated by our method to those generated by the Depth- VRD Sharifzadeh et al. (202T) baseline model. For fairness, both the models use our improved VG-Depth.v2 depth maps. Though both the models uses the same high quality depth maps, our model can effectively capture under-represented classes (classes represented less than 5% of samples in the dataset) such as 'over', 'sitting on', 'laying on', "behind" (Fig. () and 'covering', 'standing on', "walking in' (Fig. 9.. An additional description can be found in the respective figure caption.
13

---

Under review as a conference paper at ICLR 2023
Sensitivity Analysis
90.
38.7
80:1
Pos. dsi2
Viza 4440
Viozu d
Pass. 4200
Vazn- daza
Pap4.
Pees: 4128
P/a2::
Projection dimersion
Figure 5: Sensitivity analysis by varying the modality proportion within the CREAM tokens. From to right Vam is incremented by 64 while damm is decremented by 64, We plot the sum of mRecall @20, 50, and 100.
20
50
40
35
34.9
35.3
35.2
31.22
32.2
31.2
31.4
24-7
25.8
24.3
25.4
20
15
Se600
seedl
seed2
seed3
seed!
mRRec ell acTORS TIRS
Figure 6: CREAM(;-lOss): Variation across training runs. mRecall 20, and 100 the set plotted for all the model variants obtained by varying the seed from 0
Table Visual Genome (VG) recall results (the higher, the better).
PRDCLS
SGCLS
SGDET
Recall @
20
50
100
20
50
100
20
50
100
MOTIFS Zellers et al. (2018)
58.5
65.2
67.1
32.9
35.8
36.5
21.4
27.2
30.3
Frequency+Overian Zellers et al. 2018
53.6
60.6
62.2
29.3
32.3
32.9
20.1
26.2
30.1
Message Passing+ Zellers et al. 2018
52.7
59.3
61.3
31.7
34.6
35.4
14.6
20.7
24.5
VCTree Tang al. 2019
60.1
66.4
68.1
35.2
38.1
38.8
22.0
27.9
31.3
GB-NET Zareian 2020
66.6
38.0
38.8
26.4
30.0
Depth-vrd Shantzadeh et al. 2021
59.4
66.2
68.0
CREAM
59.3
66.3
68.1
31.3
34.5
35.3
17.0
21.9
25.0
Figure 7: Failure cases reported by Sharifzadeh et al. 2021). The second row shows the noisy depth
maps VG-Depth.vl from Sharifzadeh et al. (2021). The bottom row represents the improved depth map VG-Depth.v2.
14

---

Under review as a conference paper at ICLR 2023
CREAM
Pole umbrella Person holding
surthoard
person
Umbrella over man Man sitting on seat.
Depth VRD
DepthV/RD
umbrella
Figure 8: Qualitative results for PRDCLS. The triplets correctly predicted by CREAM while missed by Depth- VRD Sharifzadeh et al. (2021) are highlighted in green. The CREAM model successfully detects the under--epresented classes "over", 'sitting on' (left) as well as 'laying on', 'behind' (right) missed by the Depth- VRD model.
Man wonring part
Person wonting jacket Person walkinng in Person standing on snow
Person bolding pole
holding pole
Snow covecing mountain
Porson on skn
Roof huilding
covering
Depth VRD
Depth-
Man wearing pant
Person wearing jociott
Person holding pole
wearing Pecson in scow
Siter holding pole
Person on
Roof building
Figure 9: Qualitative results for PRDCLS. The triplets correctly predicted by CREAM while missed by Depth- VRD Sharifzadeh al. (021) are highlighted in green. Our proposed CREAM model successfully detects the under-represented class "covering" (left) missed by the Depth- VRD model. The figures on the right show that model can capture the descriptive under-represented classes 'walking in' 'standing on' (highlighted in green). while Depth-VRD could only relate the pair using the bighel-representedd. less descriptive relation 'in' (highlighted red).
15


Reviews:
[{"review_id": "mu-_wlUf_7r", "reviewer": "Reviewer_adsx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles the Scene Graph Generation (SGG) problem, with a focus on resolving biased and under-represented relationship classes. The proposed method is built upon a multi-modality SGG model where a Transformer is used to model the relationship information and modality dependencies. The proposed method is tested on the Visual Genome and Open Images datasets and improve over the state-of-the-art baselines.", "summary_of_the_review": "As addressed above, the novelty of the paper is not clear unless the discussion with the related work is explained. Also, the motivation behind not using the image context, and using multi-modality instead is not motivated well enough.", "strength_and_weaknesses": "Strength\n+ An extensive ablation study and analysis is provided\n+ it achieves the state-of-the-art performance on the PREDCLS task.\n+ The methods does not rely on the image context, while performing in a reasonable level.\n\nWeakness\n- The use of self attention to capture the relationship between subject-object pairs, which is argued as the core idea of this paper, has been proposed in the previous scene graph generation methods. One of the references is “LinkNet: Relational Embedding for Scene Graph, NeurIPS 2018”. There is no discussion and citation in the paper regarding these previous SGG methods based on the self attention mechanism.\n- The proposed method shows worse SGDET score compared to the KERN and GB-Net methods. SGDET is the ultimate form of the Scene Graph Generation, which I believe is the most important metric among the PREDCLS, SGCLS, and SGDET.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aRLwQWeBqT", "reviewer": "Reviewer_XdXf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on Scene Graph Generation (SGG), a task that aims to predict the relationships between objects detected in a scene. In this paper, they propose \"primal fusion\" which tries to inject entity relation information (between the features of the subject and object entity) and modality dependencies (between RGB and the corresponding depth modality) into each embedding token of the Transformer. The authors argue that this Cross-RElAtion-Modality (CREAM) token acts as a strong inductive bias for the SGG framework. Results on two benchmarks (Visual Genome and Open Image) are reported to show the effectiveness of the proposed method.", "summary_of_the_review": "Generally speaking, the basic idea of designing better tokens for Transformer-based SGG models is interesting. However, the whole motivation or philosophy of each step is not clear, ie, the writing quality needs to be further improved. Meanwhile, the comparison between the state-of-the-art methods is not convincing, which has missed lots of stronger baselines.", "strength_and_weaknesses": "## Strengths\n+ The direction of designing a better tokenization method for Transformer-based SGG model is interesting.\n\n## Weaknesses\n+ The contributions are incremental. Based on my understanding, the only technique contribution is the new token generation strategy for the Transformer structure.\n\n+ The motivations or logicality of the CREAM tokens designs are not clear. Specifically, in Sec.3.2, although the paper detailedly introduces the procedures to generate the CREAM tokens (cf. Figure 2), it is still unclear the advantages of this design. In another word, why this design \"explicitly fuse each embedding token with known dependencies, which acts as a strong inductive bias for SGG\".\n\n+ The comparison between the state-of-the-art methods (Table 1 and Table 2) is unfair and incomplete. Firstly, it is unfair to directly compare these SGG methods with different privileged information. Secondly, compared with debiased methods separately in Table 2 is unreasonable. In contrast, it would be more complete to compare all SGG methods (in both Table 1 and Table 2) with both mRecall@K and Recall@K metrics. Thirdly, many strong baselines are missing.\n\n+ Based on Figure 1, the difference between primal fusion and early fusion is not obviously. For example, in the early fusion case, start from the second layer, all the tokens features are fused from both modalities and subject-object features.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AQH1INsBt6", "reviewer": "Reviewer_JG3W", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The core idea of the paper is to explore how to combine different modalities such as RGB and depth to improve scene graph generation tasks, specially to improve the performance on data-scare classes. The authors introduced the idea of combining subject-object relations with modality dependencies. To that end, they came up with a token generation strategy, termed as CREAM. They show the SGG task can utilize depth as another modality instead of utilizing context, or external knowledge graphs. Essentially the method is a two-stage one.   \nThe authors claim that their method achieve state-of-the-art performance on Visual Genome for mean Recall metrics.  ", "summary_of_the_review": "The paper hasn't done the prior art survey on this very topic. In my opinion, the paper missed a crucial point by ignoring the recent and relevant literature. The claim about outperforming existing methods maynot be a correct one.   ", "strength_and_weaknesses": "SGG is essentially a long-tailed and multi-task problem as it deals with two long-tailed distributions, namely entities and predicates. The evaluation metrics should not only emphasize on the head classes, but also on middle or tail classes. Thus, mean Recall should be more highlighted than recall. The paper follows the right directions from that perspective.\n\nAlthough depth can provide complementary information, but given the combinatorial nature of SGG, the context shouldn't be ignored. \n\nMore importantly, the paper hasn't done the prior art survey on this very topic. Plenty of methods already exist that perform significantly better on SGG tasks as measured by the mean Recall metric. In fact, two popular methods that specifically emphasized the importance of mean Recall  are TDE (CVPR 20) and KERN (CVPR 19). Although these two papers were referred here but not compared in the main table. References such as PCPL (ACM MM 2020), DT2 (ICCV 2021), which notably improved on mean Recall metrics on Visual Genome, are missing. \n\nThe comparative performance as shown in Table 3 is for PredCls, where the set of all objects with their corresponding classes and bounding boxes are given. I'd rather like to look at such metrics for SGCls or SGDet. \n\n\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject