Can Transformers Capture Spatial Relations between Objects?
Authors: Chuan Wen
Dinesh Jayaraman
Yang Gao
Venue: ICLR 2024 poster (2024)
Keywords: Spatial relation
Transformer
Spatial predicates
Visual recognition
Neural network architecture
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We tackle a seemingly fundamental yet surprisingly difficult spatial relation prediction visual task through a systematic investigation of the design space of vision transformers, advancing the state of the art.

Abstract: Spatial relationships between objects represent key scene information for humans to understand and interact with the world. To study the capability of current computer vision systems to recognize physically grounded spatial relations, we start by proposing precise relation definitions that permit consistently annotating a benchmark dataset. Despite the apparent simplicity of this task relative to others in the recognition literature, we observe that existing approaches perform poorly on this benchmark. We propose new approaches exploiting the long-range attention capabilities of transformers for this task, and evaluating key design principles. We identify a simple ``RelatiViT'' architecture and demonstrate that it outperforms all current approaches. To our knowledge, this is the first method to convincingly outperform naive baselines on spatial relation prediction in in-the-wild settings. The code and datasets are available in \url{https://sites.google.com/view/spatial-relation}.

Published as a conference paper at ICLR 2024
CAN TRANSFORMERS CAPTURE SPATIAL RELATIONS BETWEEN OBJECTS?
Chuan Wen Dinesh Jayaraman', Yang .3.1. "Istitttee for Interdisciplinary Information Sciences, Tsinghua University University Pennsylvania
Shanghai Artificial Intelligence Laboratory
"Shnnghai Qi Zhi Institute
ABSTRACT
Spatial relationships between objects represent key scene information for hu- mans to understand and interact with the world. To study the capability of cur- rent computer vision systems recognize physically grounded spatial relations, we start by proposing precise relation definitions that permit consistently anno- tating benchmark dataset. Despite the apparent simplicity of this task rela- tive to others in the recognition literature, we observe that existing approaches perform poorly this benchmark. We propose new approaches exploiting the long-range attention capabilities of transformers for this task, and evaluat- ing key design principles. We identify simple "RelatiViT" architecture and demonstrate that it outperforms all current approaches. To our knowledge, this is the first method to convincingly outperform naive baselines spatial rela- tion prediction in in-the-win wild settings. The code and datasets are available attps: /ttes://sitees gwogle.ooooooooooooooooooooo. com wwtpa//wwwwwww....eeeae
big /
covering MuakkXstatne statue front building/ /poper right pen poontowell paper v
Figure 1: Illustrations of spatial relation prediction (SRP) task on Rel3D and SpatialSense+ datasets.
INTRODUCTION
The spatial relationships between objects are crucial for visual scene understanding. For example, humans understand that "the saucer is under the coffee cup", so we can make reasonable manip- ulation plans: move the cup before picking up the saucer. How well can current computer vision systems complete such a spatial relation recognition task? Given image and bounding boxes of two queried instances (subject and object), the task to recognize the spatial relationships. Modern computer vision systems have been widely benchmarked to demonstrate progress along various axes of the visual understanding problem such as scene and object category recognition, object segmen- tation, etc., but to lesser extent for understanding object relationships, particularly in the precise physically grounded sense outlined above.
Many prior works have studied semantic object relationships (Lu et al. 2016 Zellers et al. 2018 Tang et al., 2019 Li et al., (2011)), including those tied to the spatial orentations of objects. How- ever, such re relations are not always physically grounded. For example, in this semantic sense, a cat that walking on top
picture that is only attached horizontally to wall is "on wall", but of a wall. These relationships reflect linguistic conventions rather than precise physically grounded spatial relationships between objects such as the cup atop the saucer above. These examples demon- strate that such semantic relationships can ambiguous and not necessarily capture the physical
Corresponding author.

---

Published as a conference paper at ICLR 2024
properties of the objects involved. Indeed, this shortcoming has been noticed in prior works, where it has been shown that just names of the categories of the objects involved, with no image information at all, already predict such relationships well (Liang et al. 2019 Yang et al. 2019).
Detecting such semantic relationships does not suffice to inform robot aiming to pick up the saucer mentioned in the opening paragraph. That requires more precise and physically grounded notion of spatial relationships. To study such relations in-the-wild, we first set up new benchmark. Two existing datasets Re13D (Goyal et al. (2200) and SpatialSense (Yang et al. (019)) also target spatial relation prediction. However, as we will show in Section 3.2 Re13D primarily handles synthetic scenarios with clean backgrounds and textures, and SpatialSense suffers from annotation inconsis- tencies caused by relation definition ambiguity and language bias, etc. To fix this, we first propose precise, unambiguous, definitions for various commonly used spatial relations in a manner that per- mits consistent assignments of target labels in complex real-world scenarios. We then relabel the SpatialSense dataset to be consistent with our definitions. Figure shows some samples from this new dataset, SpatialSense+. We use these two datasets, Re13D (synthetic) and SpatialSense+ (real- istic) to comprehensively benchmark spatial relation prediction approaches.
How should we address the spatial relation prediction (SRP) task? This task may appear sim- ple at first glance relative to more complex visual recognition tasks (He al.] 2016, 2017 Long et al. 2 that have already been shown to be achievable with modern computer vision systems. Indeed, what makes this task particularly interesting to study is that despite its apparent simplic- ity, prior works have consistently reported that no existing computer vision approaches are able to outperform naive "bbox-only" baseline making predictions only based on the bounding box co- ordinates (Goyal et al. 2020, Yang et al. 2019). The existing methods are in very similar paradigms: extracting local features of the two objects from frozen CNN backbone as input to an MLP rela- tion classifier. Note these architectural choices; while standard for many vision tasks, they make it difficult explicitly model long-range interactions between objects, as might be necessary for SRP
If this architecture mismatch is indeed the root cause of the difficulty of SRP with modern com- puter vision approaches, how might we overcome it? Transformer architectures (Vaswani et al. 2017) have recently achieved great success across various fields, c.g., NLP (Devlin et al. 2019) and computer vision (Dosovitskiy et al., 2021). Unlike CNNs with first-order convolutional operations between images and weights, the Vision Transformer (ViT) (Dosovitskiy et al. 202I) takes advan- tage of the attention mechanism (Vaswani et al. (017) to fuse features among image patches. This explicitly introduces the inductive bias about pair-wish relationships and comparison between the image patches, suggesting that transformer-ben based solutions may be particularly well-suited for SRP.
In this work, we systematically study several carefully designed transformer- architectures on our precise and physically grounded spatial relation prediction benchmark, comparing them with the best existing CNN-based methods, and identifying clear winning design, which we call "Rel- ati Vii" Our experimental results demonstrate that Relati VTT significantly outperforms all the ex- visual information to improve performance on
isting methods and is the first to convincingly this task beyond just relying on the 2D spatial coordinates of the objects. Our experiments further show that even the most advanced large-scale Vision Language Models, such GPT-4V, Gemini, LLa and MiniGPT-4, fail to achieve satisfactory results on our benchmark, highlighting that the spatial relation prediction is an essential and challenging task for visual reasoning.
RELATED WORK
Spatial relationship. Recognizing the relations between the objects in an image is a challeng- ing problem beyond object recognition and detection in computer vision. Visual relation detection is a well-known task to predict the semantic relations between objects and generate a scene graph from an image. The commonly used benchmarks VRD (Lu et al. (016)). Visual Genome (Krishna al. (017) and Open Images (Kuznetsova et al.) 2020) are reported suffer from severe language bias (Tang et al. 2020b] Zellers et al. 2018) Liang et al.) 2019) and ill-defined evaluation met- rics (Yang et al., 2019). As result, the existing visual relation detection methods mainly focus
on improving the information aggregation on the graph (Xu et al. 2017: Zellers et al.) [018 Tang
2019 Lin al. 2020 Li et al. 2021a Lin et al. 2022). data debiasing (Tang al. 2020b Chen et al. 2019 Li et al. 2022), (,22),00 and utilizing commonsense (Zareian et al. 2020) or language pri- ors(Lu et al. 2016] Yao al.] 2021). Spatial relationship is also leveraged some recent robotics
al.
works for task planning, but they either only work on limited domains (Bobu et al.) 2022 Wang
et al. 2022
Ku et al., 2023) or require access point clouds (Liu et al.] 2023b). indicating the
2

---

Published as a conference paper at ICLR 2024
difficulty of recognizing spatial relations from 2D vision representations. Two recent works propose the Rel3D (Goyal et al. 2020) and SpatialSense (Yang et al. 2019) datasets to place attention on the physically grounded spatial relation prediction task from 2D images.
Moving to methods for SRP, the above works employ Rol Align to extract the visual features of objects from CNN encoder, and then feed them into a prediction head (based on message pass- ing (Xu et al. 2017) or LSTM (Zellers al., 2018; Tang et al. 2019)) to classify the relationship between them. SorNet (Yuan et al. (022) assumes that object canonical views (pictures taken with the objects facing the camera) are available as queries, relying on them localize the objects and extract object embeddings before predicting relations. This assumption is impractical in in-the-wild settings with unknown objects. Moreover, their datasets explore only simple cuboid-like geome- tries of "objects", , uncluttered scenes with limited occlusions, fixed tabletop scene layouts, etc. We study a more challenging and realistic setting with real images, complex scene layouts and arbitrary camera pose.
Relation extraction. Relation extraction is widely studied task in other fields of deep learning. In graph learning, the relation extraction task is to predict the existence of edges between nodes (Zhang)
Chenn 2018) or the attributes of the edges (Onuki et 2019 Cui et al. 2021). In natural language
processing. there are many existing works about sentence- level (Zhang et al. 2017c Peng et al.
2020) and document-level relation extraction (Tang et al., 2020a; Wang et al. 2019 Zhou et al,,
2021 Tan et al., 2022). Here, Wang al.) (2019): Zhou et al. (2021): Tan et al. (2022) show that
extracting relation features from a transformer pretrained by BERT (Devlin et al.) 2019) is more
effective than the graph-based methods (Tang et al. 22200a) Nan al. 2020). Similarly, we propose
to extract the relation features from a pretrained VIT for Visual relation prediction tasks.
Vision Transformer. Building on the success of transformers in NLP (Vaswani al. 2017 Devlin
let al. 2019). Vision Transformer ( (ViT) is proposed to handle image data (Dosovitskiy et al. 2021 Taking advantage of long horizon memory properties of the attention mechanism, ViT has larger perceptual field and thus excels across various computer vision tasks, e.g., object recognition (Tou- vron et al. 202 detection (Li al. b02 and semantic segmentation (Strudel al. 202 Xu ct al. 2022). ). etc. VIT utilizes the attention mechanism to explicitly model pair-wise relationships be- tween image patches, which inspires us to explore object spatial relation prediction from pretrained
|
ViTs (Caron et al. 2021 He et al. 2022 Chen et al. 2021 Zhou et al. 2022: Radford et al. 202I).
3
PRELIMINARIES
3.1 PROBLEM FORMULATION
Consider the example "statue in front of building" in Figure [ the spatial relation prediction (SRP) task expects the model to recognize that the predicate "in front of"" is true because the depth of the subject "statue" is smaller than the object "building"" Because there might be multiple relations holding simultaneously between the subject and object (multi-label property). task as a K-way binary classification: given an image 7, two bounding box coordinates of "subject" and "'ojject" b., bo E TR4, corresponding object categories Cs, Co E C, and relation predicate r E (0...... K- - 1}. the r-th prediction head f of the model is required to predict whether the relation is valid between subject and object, i.e., Vrls. E {0, 1}, where @ is the parameters of neural pa networks, K is the number of relation classes, and and R are pre-defined object and relationship category sets. The objective is:
formalize the SRP
min - -Ely log(D))o.o) + (1 - v.)) log(1 Fyao)), where Ghan = S6 (T,
B by Cr c.).
3.2 REFINED SPATIAL RELATION BENCHMARK
We aim to benchmark various approaches for precise and physically grounded SRP. The two ex- isting benchmarks most closely aligned to our problem are SpatialSense (Yang et al. 2019) and Rel3D (Goyal et al. (2200. SpatialSense is a realistic dataset annotated from Flickr (Plummer et al., 2017) and NYU (Nathan Silberman & Fergus 2012) images, with various scenes includ- ing indoor, street and wilderness and many different objects such as humans, animals, plants, household items, etc. But because the precise definition of each relation category is not pro- vided during the annotation process, still inherits the linguistic biases of crowd annotators.
3

---

Published as a conference paper at ICLR 2024
In Figure showcase some key issues in the original SpatialSense dataset: (1) mix- ing object-centric and viewer-centric relations together, which might have totally opposite meanings, e.g., from the perspective of the man in Figure the bass is on his left, but from the camera's ss view, the bass is to his right; (2) ambiguity caused by polysemous words, e.g., the "on"s s in "books on shelf" and "cup on ta- ble" are different relationships; (3) language bias introduced by some idiomatic expressions, e.g., "boy snow" but not "boy on snow". Re13D aims to fix these issues but does so by rendering synthetic dataset completely in Blender (Community 2018). Though Re13D has diverse spatial layouts of objects, is not photo-realistie - the background and object textures are simple, shown in Figure II While easy comparison with prior works, we seek a more thorough study of real-world in-the-wild images with diverse objects and scenes, closer to the domain of SpatialSense.
bass to the left of man/ books on shelfV (a)
boy in snow v (c)
(b)
Figure 2: Three issues of original SpatialSense: a) mixing the object-centric and viewer-centric annotations; b) ambiguity caused by polysemous words; c) language bias.
evaluate our approaches on Rel3D as of our two benchmarks for
Therefore, we propose to construct a precise and physically grounded real-world dataset by relabeling SpatialSense. We propose precise, unambiguous, physically grounded meanings for various commonly used spatial relations for SpatialSense. For example, we restrict the meaning of "in" be: "The subject is inside
Table I: Statistics of Rel3D and SpatialSense+ in our benchmark.
Dataset
#Predicate
#Train
#Validation
#Tost
Ret3D
30
20454
2138
4744
SpatialSenset
5346
808
the object, and the object must at least semi-enclose the subject." " So, the "boy in snow" sample in Figure (c) must be annotated as False. The full definitions of all the relation categories shown in Appendix A. We randomly sample a subset of SpatialSense (amounting to 7254 relation triplets and 4418 images) and strictly relabel it by ourselves according to the new definitions. Eventually, get a smaller but much cleaner version "SpatialSense*" that fixes the above-noted inconsistencies, including all the examples in Figure 2
Some basic statistics are shown in Table D
4 IN SEARCH OF AN ARCHITECTURE FOR SPATIAL RELATION PREDICTION
Recently, transformer architectures have achieved great success in various fields such as NLP (De- vlin et al., 2019) and computer vision (Dosovitskiy al. 2021), J. by taking advantage of the attention mechanism (Vaswani et al.) 2017). Besides the advantages such as long horizon memory, attention in transformers explicitly models the pair-wise relationship between tokens, which is conceptually in line with our spatial relation prediction (SRP) task. Motivated by this, we aim to systemati- cally study how utilize the transformer architecture extract spatial relation information from the images. We first introduce the basic components to solve this task and the architecture design principles for these steps. Based on these principles, we design multiple architectures for SRP. We introduce and compare the four most representative and natural designs in the main paper, while the remaining ones are presented in Appendix B.I
4.1 DESIGN AXES
As introduced in Section 3.1 in the SRP task, the inputs are an image and two instance queries (subject and object) specified with their bounding boxes 6,, b, and object categories Ca, cg. Because focus extracting relation features from visual representations, we omit the object categories from the inputs. In general, predicting the spatial relation between two objects from an image is composed of the following four components.
Learning semantic representations from high-dimensional images is the first step of spatial relation prediction. In general. we produce visual features by backbone such as CNNs (e.g.. ResNet (He et al. (016) or VGG (Simonyan & Zisserman 2015) or ViT (Dosovitskiy et al. 2021) which is representative of all the various transformer architectures in computer
Feature extraction:
vision (Liu al. 2021 Touvron et al. 2021b Hassani et al. 2021 Chu et al. 202T).
Query localization: Given two bounding box queries, how to localize the queries and get their local features? One option is use Rol Align, popular in object detection tasks, to localize the bounding
4

---

Published as a conference paper at ICLR 2024
MLP
MUP
MLP
8.
Transfertner Eneoter
pooling
pooling
poeling
pooling
L1M,
16,
J.
VIT Decoddr
RolAlgg
RatAligo
VIT Eacoder
ViT Enceder
ViT Enredar
VET Encoder
VIT
CNN
Parch Kmieedling
Patch Embedding
Imagge
Image
Image
subject query I, (c) CrossAttVVIT
object query
Image
sutject query
object
I
(a) RegionVIT
(b) CNNTransformer
(d) RelatiViT
Figure 3: Four different architecture designs for SRP.
boxes in the low-resolution feature space (He et al., 2017). Another option is localize the regions in the image space according to the bounding box coordinates, then mask out other regions, and get the local features by feeding the masked image through the feature extraction backbone.
Next, how should we represent and use context information around the two query objects? For example, information about the obstacles between them is helpful determine whether the relation is "next to". And the shadows might indicate some depth and contact infor- mation. Existing spatial relation prediction methods extract visual features of the union region of subject and object by Rol Align as context information (Li et al. 2017; Zhang et al.] 2017b] Zellers et al. 2018). For transformer, the attention operation is natural aggregate contextual features, De- pending on where input tokens masked images e,,, ET,, the context aggregation in transformer can be divided into two types: I) self attention: feed queries together with image patches as one sequence into the VIT encoder and then aggregate the context information in original image patches by self-attention layers; 2) cross attention: use queries the input, then image features from the encoder as keys and values, and then use cross-attention layers to fuse the context information.
Context aggregation:
Pair interaction:
Different from object recognition, our task is to recognize pair-wise relation- ship between subject and object, which cannot be learned by these two features separately. There- fore, modeling the interaction between two features is also critical for our task. The casiest way to combine the pair of features is concatenating them and feeding them into MLP. Or in more transformer-centricc design, we could explicitly model such pairwise interactions through attention.
4.2 ARCHITECTURE DESIGNS
Based on the potential design options of each component discussed in Section we make different combinations of these options and obtain various transformer- architectures to extract spatial relationships. As shown in Figure 3 we present the four most representative designs here: (a)
Region (b) CNNTransformer, (c) VrossAttVVi., (d) RelatiViT. Table outlines their different choices for each design axis. And we compare more different designs in Appendix B.
Region VIT: a duraight-forware architecture to encode the visual features by a VIT. utilize Rol Align to localize subject and object embeddings c,, eo efficiently from the low-resolution feature map eд. To get the contextual information, we again use Rol Align to extract the feature e, in the union region of two bounding boxes, which is very common in the visual relation detection literature et al.) 2017;; Zhang et al. 2017bb Zellers et al. 2018 Tang et al. 2019). And the pair-wise interaction simply concatenating the features of subject and object. This architecture aims to study whether such commonly used ferw-forward style is suitable for VIT and the SRP task.
Motivated by the object detection models (He et al. 2017 Li al. 2021b). design
CNNTransformer: In Table (b), different from Region VIT. we move the transformer from the ViT, feature extraction module to prediction head, i.e.. we obtain the global visual embedding e, through a conventional CNN (e.g., ResNet (He et al. (2016) or VGG (Simonyan & Zisserman) 2015)), local- ize Rol embeddings ex; 6, by Rol Align, tokenize e,, Es, and then input them into transformer to aggregate contextual and pair-wise information simultaneously. The purpose of this design is to investigate capability of transformers capture relations top of semantic representations.
5

---

Published as a conference paper at ICLR 2024
Table 2: The choices of each component in different architectures.
Feature
Query
Context
Pair
Architecture
extraction
localization
aggregation
interaction
RegionVir
VIT
Rol Align
Rol Align on union
Concatenation
CNNTansformer
Align
Self attention
Self attention
CrosAAmviT
VIT
Masked image
Cross attention
Self attention
RelatiVIT
VIT
Masked image
Self attention
Self attention
ViT:: Motivated by the original transformer encoder-decoder architecture for machine translation tasks /Vawwani et al. (017), we propose to attach a cross-attention decoder after ViT, which is shown in FigureB][c) In this pipeline, we first obtain the subject, and object query images Io by masking out the regions other than RoI, which completes the query localization step. We tokenize the original, subject and object query images by patch embedding layer (Dosovitskiy al. 202I). Patch embeddings of these three images are fed into a shared ViT encoder extract the visual embeddings. Then the encoded embeddings of the original image and two masked images put into the VIT decoder. In each layer of the decoder, there self-attention layer (Vaswani et al. 2017). The cross-attention layer takes the embeddings of two masked images as queries, and the ones of the context image é keys and values, which aggregates contextual information. And the self-attention layer enables the message passing between subject and object. Finally, we take the output of ViT decoder é,,, ér, and get the relation-grounded embed- ding éx2 & through a pooling layer. This architecture to study whether such an encoder-decoder structure commonly used for transformer-base models can effectively extract relation information.
rross-attention layer and one
Besides modular architectures mentioned above, we propose an end-to-end archi- tecture VeT, shown in Figure 3 (d). We utilize the same strategy as CrossAttn to localize the subject and object queries by image masking. After getting the embeddings e,, c from the patch embedding layer. we concatenate them as long sequence and then feed them into the VIT encoder. During the forward process of ViT encoder, the self-attention mechanism completes context aggregation and pair interaction simultancously. For example, subject tokens e,, are able to attend context features ez and object features er, because they are in the same sequence. After aggregating the contextual and pair-wise features in ViT encoder, we get ěJ, We downsample these embeddings by a pooling layer and get final oolatin-groundeee embeddings ca, Co for the subject and object respectively.
RelatiViT:
5 EXPERIMENTS
We evaluate the architectures proposed in Section 4.2 identify useful design principles. Then, we compare our best model with the existing methods. We also conduct& series of analytical experiments probe our methods' performance.
benchmark (Section 3.2). aiming to
EXPERIMENTAL SETUP
We train all the models on Re13D and SpatialSense+ datasets. The input of each model includes an RGB image, bounding boxes and categories of the subject and object, and a spatial relation predicate between them, where all images 224 224 and object categories
encoded into 300-dimensional embeddings by Glove (Pennington et al. 2014). Note that some methods may part of the inputs, c.g., our methods proposed in Section 4.2 do not use the object categories. For fair comparisons, we adopt ViT-Base and ResNet101 as our ViT and CNN backbones which have close numbers of parameters. And we load the IBOT pretrained model (Zhou al. (0022) for ViT-Base and ImageNet supervised pretrained model for ResNet101. During training, we use data augmentations RGB images, e.g., random crop resize and color jitter. More training details are shown in Appendix A.2
We report two evaluation metrics: 1) %Avg. Acc.: the average accuracy over relation classes, which avoids the overall accuracy being dominated by the relation classes with more samples; 2) %F1: the commonly-used F1 score in binary classification. We tune the hyperparameters on the validation set and the details about hyperparameters are listed in Appendix A..3 The checkpoints with the highest Avg. Acc. on the validation set are taken for testing. We repeat the experiments 5 times with different seeds, and report the average values and standard deviations in Table 65
5.2 BASELINES
Following Goyal et al. 20200; Yang al. 2019), we use naive baselines: bbox-only and bbox-language to demonstrate whether other methods successfully utilize the visual information. The bbox-only method only takes bounding box coordinates and relation predicates
Naive baselines:

---

Published as a conference paper at ICLR 2024
Table 3: Comparison between different designs. RelatiViT shows superior performance over Re- gion CNNTransformer, and CrossAttn ViT across two datasets.
Ret3D
SpatialSenset
mothod
%Avg. Ace.
%FI
%vvg. Acc.
%F1
RegionViT
72.43 + 0.55
74.64:
76.20 + 1.04
75.15 4: 0.96
CNNTransformer
71.13 + 0.23
73.85 0.29
68.21 + 0.92
06.83 - 0.72
CrossAmuVi
76.55 + 0.28
78.76
- 0.35
73.41 2.00
71. 4: 3.40
RelatVVT
80.09 1 0.33
82.05
1 0.49
80.02 0.73
78.32 1: 1.01
as imput, while bbox-language takes ground truth object categories in addition. Both two models are simple MLPs. Note that these are notoriously hard baselines to beat in the most relevant prior works (Yang et al. 2019 Goyal et al. 2020).
Existing methods. Because there are still no good solutions for SRP, we follow Goyal et al. (2020); Yang et al. (2019) to adapt five visual relation detection models (DRNet (Dai et al. (017)) ). Vip-
|01 201 Motifs- Net (Zellers et al., 2018), and RUNet (Lin et al. (022 to our task. Specifically, we replace the object detection module with the ground truth bounding box coordinates and object classes, and then we modify the model output from K-way classification probabilities into a K-way binary clas- sification probability conditioned on the input relation predicate. These baselines are divided into two groups based on the information used in the final MI.P classifier: 1) vision: Vip-CNN and PPR- FCN, only taking vision features to make predictions; 2) vision+bhox+class: DRNet, VTransE, MotifsNet and RUNet, using visual features, bounding box and object class embeddings.
CNN (Li et al., [2017). VTransE (Zhang et al.] (017a). PPR-FCN (Zhang et al.
Vision Language Models. Beyond these spatial-relation-speciife baselines, we also compare with four Vision Language Models (VLMs): MiniGPT-4 (Chen et al., (023). LLaVA (Liu et al. 2023a), Gemini (Team et al. 2023) and GPT-4V (Achiam et 2023) to demonstrate the capability of these SOTA large models to capture spatial relations. We only evaluate these VLMs on the realistic dataset SpatialSense+. The specific version and input prompts are shown in Appendix.
5.3 COMPARISON BETWEEN DIFFERENT DESIGNS
As shown in Table 3, we compare the selected four architectures introduced in Section 4.2 on Rel3D and SpatialSense+. More fine-grained comparisons between more architectures by changing design options one by one are shown in Appendix B. By comparing their performance, we can obtain some insights about how to design transformer-based models for relation-based tasks.
CNNTransformer performs worse on both Re13D and SpatialSense+ than Region This indicates that ViT stronger backbone for extracting visual relation information compared CNN. An external transformer head is insufficient to compensate for the weak feature extraction performed by CNN. The better performance of "ViT RoiAlign Self-attention" than CNNTransformer in Appendix B.I further verifies this argument by only replacing CNN with VIT.
Considering the results of both Rel3D and SpatialSense+, CrossAttn VIT performs similarly Re- ViT. We further ablate the effect of "RoiAlign v.s. MaskImage" and "Self-attention v.s. Cross- attention" based on CrossAttn VIT in Appendix [B.]] but their performance also does not have sig- nificant differences. The commonly-use do not benefit SRP performance of these cascading architectures, indicating that these cascading architectures are not effective in extracting relational information from the pretrained VIT.
Rol Align technique and advanced transformer decoder
Relati ViT performs significantly better than the other three architectures. In contrast to the three cascading models, RiT feeds the patch embeddings ez, Gr, Cr, as unified sequence into the ViT encoder. Consequently, the long-horizon memory of the transformer allows the queries ez, e1, refer one another and attend the contextual information in e, automatically, enabling si- multaneous completion of query localization, context aggregation, and pair interaction. Relati ViT overcomes the limitations of other designs: 1) it employs stronger ViT backbone than CNN, facil- itating the extraction of relation-related features; 2) it does not introduce extra parameter overheads, mitigating optimization challenges; 3) the end-to-end modeling enables the extraction of relation- grounded information across all ViT layers.
5.4 COMPARISON WITH BASELINES
As shown in Table 4 to verify that transformer is conceptually suitable for spatial relation prediction tasks, compare our best transformer-based model Relati VIT with the existing methods.
7

---

Published as a conference paper at ICLR 2024
Table Comparison with baselines. RelatiViT significandy outperforms all the existing methods.
ReI3D
SpatialSence+
Method
Signal
%bvg.
%F1
StAvg. Acc.
boox-only
74.35 + 0.77
77.81 0.44
77.35 0.78
75.30 + 1.35
bbox-language
bbox+class
73.11 = 0.29
76.63
0.49
73.88 + 1.39
71.81 = 1.79
DRNet Dai et
2017
72.63 0.36
75.28
0.49
76.96 1.53
75.74
1.44
VTransE:
Zhang ai
201
7a
bisionsb
72.50 1: 0.50
75.54
0.48
77.10 1. 0.84
75.58 1: 1.13
MotifaNet
Zallers
201X
vision-
Hhoion--hox-ellss
72.42 + 1.34
74.67 +
75.58
74.42
1.05
RUNet
Lin ét al
75.03 = 0.48
77.83 0
76.81 0.90
74.20 + 0.73
Vip-CNN
LL al 2417
vision
69.72 = 0.67
73.07
0.81
1.33 70.68
69.71
67.73 + 1.66
PCR
Zhang
7b
vision
71.20 1.42
1.28 72.95
0.97
69.88 1: 1.36
RelatiViT' (ours)
80.09 0.33
82
0.49
78.
MiniGPT-4 Chen al
2023
vision-bbox-ellas
33.33
LLaVA Liu
vision-bbox+cllas
51.20
53.66
Gemin Team of of 2476
vision-bbox+ellas
59. 17
59.55
GPT-4V Achram
vision-bbox+cllass
68.26
66.51
With access to object classes, bbox-language cannot beat bbox-only, indicating that the object cate- gory information does not benefit this not obtain more useful information than bbox-only, but instead causes optimization difficulty due to the high-dimensional word embedding input, resulting higher variance and worse performance.
spatial relation task. Bbox-language does
We can see that none of the existing methods performs better than bbox-only on both datasets. Espe- cially, the methods "vision+bbox+clas"" group (DRNet, VtransE. Motifsnet and RUNet) perform slightly worse than bbox-only. We hypothesize that this is caused by shortcut learning (Geirhos) al., 22200 Wen et al. 2022)- deep neural networks prefer solutions with simpler decision rules rather than the complex intended ones. They are prone to simply predict the relationships based on the bounding box coordinates rather than learn the intended solutions by referring to the RGB input,
which further investigated in Appendix [8.5] As for the methods in "vision" group (Vip-CNN and PPR-FCN), they perform poorly because it is difficult for CNN backbones extract relation- grounded representations.
Furthermore, we extend our analysis to incorporate comparisons with advanced Vision Language Models (VLMs), showing the trend: GPT-4V > Gemini > LLaVA MiniGPT-4. Despite their re- markable performance in VQA tasks, these large-scale pretrained models exhibit poor performance on the SpatialSense+ dataset. This discrepancy underscores the significance of our spatial relation prediction task as pivotal benchmark for visual reasoning, essential for evaluating the capabilities of scrge-scale mudti models in the future.
Our best model RelatiViT outperforms all the baselines. With ViT's powerful representation ca- pabilities and our end-to-end modeling of context aggregation and pair-wise interaction, RelatiViT successfully extracts the spatial relations from images and becomes the first one so far to clearly out- perform the naive bbox-only baseline. For prediction visualizations, please refer to Appendix B.6
Table 5: Ablation studies on the design components.
CONTEXT
FEATURE
REI.3D
SPATIAL-SNNSE
INTERACTION
AGGREGATION
EXTRACTION
AvG Acc.
RF1
%AvG.
%FI
80.09 + 0.33
82.05 + 0.19
80.02
0.73
78.32 + 1.01
X
79.18 + 0.61
81.30 + 0.57
76.37 0.86
75.01 + 0.87
X
70.08 + 0.41
72.70
# 0.32
74.04 0.75
71.25
+ 0,98
X
X
55.51 + 0.48
57.59 :: 1.07
60.80 0.61
59.34 1 0.58
5.5 ANALYSIS ON RELATIVIT
We now study the winning RelatiViT architecture more closely through careful analyses.
Ablation on design axes. We perform ablation studies on the three components of Relati feature extraction, context aggregation, and pair interaction. Specifically, we (1) remove the VIT encoder, (2) add attention masks to context image tokens, and (3) corresponding query tokens, respectively. It is worth noting that we exclude query localization because it is an integral component. In Table all three components are essential for the performance of RelatiViT. Feature extraction serves as the foundational module for extracting features from raw pixels, and its absence results in the most significant performance decline. Context aggregation plays a significant role in the spatial relation prediction task since the model relies on context information to deduce the relationship between the subject and object. The removal of pair interaction has relatively smaller impact on performance.
8

---

Published as a conference paper at ICLR 2024
Table 6: Accuracy of each class on Sense..
Method
above
behind
in
right
under
next
bront
left
Mox-cnly
81.8 +:
71.3 1.7
71.0
03.5
74.5 = 2.8
76.5 1.8
65.8
71.0 1.5
91.6
0.9
DRNet
74.2
65.9 4
3.0
76.3
93.4
71.5
7.2
73.6
69.7125
75.7 2.47
92.0
RebsVVT
80.2
2.3
03.5 3.8
78.3
64.5229
92.4
Performance on each class. As shown in Table 66 we report the accuracy of each class on Spa- tialSense+. We compare three models, bbox-only, DRNet and Relati the most representative ones in naive baseline, existing methods and our architectures respectively. We can see that our method outperforms the bbox-only baseline on "above", "behind", "in", "on", "under" and "in front of",, which requires visual information about depth, object poses and shapes, etc. Bbox-only per- forms better on "to the right of"" than RelatiViT because according the relation definition proposed in Section 3.2, this relation can indeed be predicted only by bounding box coordinates. The results on Re13D are shown in Appendix B.2 and the conclusion is consistent. In summary, our RelatiViT successfully extracts effective visual representations for SRP.
Relat/VIT
CrossAtrnViT
Image
context suhjectquery object query
Image
context
subject query object query
subject attention
object attention
egg label: False Prediction: False
lomato egg label: False Prediction: True
Figure 4: Comparison of attention maps. In each subfigure, the 1st and 2nd rows visualize the attention matrices averaged from the subject and object query embeddings respectively. The 1st column is the original image with the bounding box; the 2nd one is the attention map reflecting how much the query attends to the context image (context aggregation); the 3rd and 4th ones show the attention between queries (pair-wise interaction).
Attention map visualization. To study how our ap- proach refers to visual information, we visualize the at- tention maps of RelatiViT and CrossAttn ViT, which re- flect how much and in which area each query attends to
75
the context image (context aggregation), itself and other query (pair-wise interaction). As shown in Figure 4 the model is required to recognize whether the tomato is next to the egg. We can see that besides focusing on the queries' regions, both subject and object queries of Rela- tiViT also attend to the lemon in between. Consequently, RelatiViT knows there is an obstacle between the tomato and egg, so it predicts "False". On the contrary, atten- tion maps of CrossAAttn ViT are messy and hard to explain. This illustrates RelatiViT outperforms other methods by correctly attending to the critical regions of the context image and two masked query images. More visualizations are shown in Appendix B.7
770
45
hoos-only
FRR-FCN
Vitans
MositNett
DaNel
RUv
ViCcNN
ReaVV
0.1
0.2
0.4
0.5
07
088
09
Figure 5: The performance on different sizes of the Rel3D training set.
the the F1 scores of each model with training set size varying in 13%, 25%, 40%, 90%]. The results show that RelatiViT outperforms all the baselines all the training set sizes. Moreover, its performance on the 25% dataset is comparable with the best result of baselines on 90% training data. This verifies that our RelatiViT is more sample efficient by taking advantage of the inductive bias pair-wise relation and prior knowledge pretrained models.
Effect of dataset size. In Figure we plot
CONCLUSION
In this paper, we benchmark the various transformer-baat based models to investigate the optimal way to predict precise and physically grounded relationships. basic vision task that remains surpris- ingly challenging even with modern approaches, and which is essential for robotics manipulation and general scene understanding. Our experiments yield key insights about design principles for Transformer-based systems for this task, and identify clear winner ReT This is the first system thus far to surpass naive baselines for this basic visual capability. And it will serve as an important starting point and baseline for future efforts in this direction.

---

Published as a conference paper at ICLR 2024
7 REPRODUCIBILITY STATEMENT
The main implementations of our proposed models are in Section 4.2 The settings of the exper- iments and training details, e.g., data preprocessing and hyper-parameters, in Section 5.1 Ap- pendix A.2 and Appendix A.3
8 ACKNOWLEDGEMENT
We are grateful to Chen Wang, Kaihua Tang, Long Chen, and Jiacheng You for fruitful discussions and comments.
This work is supported by the Ministry of Science and Technology of the People's Republic of China, the 2030 Innovation Megaprojects "Program New Generation Artificial Intelligence" (Grant No. 2021AAA0150000), and the National Key R&D Program of China (2022ZD0161700), This work is also supported by NSF CAREER Award 2239301.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXivv2303.0827 08774, 2023.
Andreca Bobu, Chris Paxton, Wei Yang. Balakumar Sundaralingam, Yu-Wei Chao, Maya Cakmak, and Dieter Fox. Learning perceptual concepts by bootstrapping from human queries. 1EEE Robotics and Automation Letters, 7(4):1 2022.
Jake Bruce, Michael Dennis, Ashley Edwards, Jack Parker-Holded Yuge Shi, Edward Hughes, Matthew Lai, Aditi Mavalankar, Richie Steigerwald. Chris Apps, et al. Genie: Generative teractive environments. arXiv preprint arXiv:2402.15330 2024.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9650-9660, 2021.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechu Liu, Pengchuan Zhang, Raghuraman Kr- ishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny. Minigpt-v2: large language model as unified interface for vision-language multi-task learning. arXiv preprint arXiv:2310.09478, 2023.
Long Chen, Hanwang Zhang, Jun Xiao, Xiangnan He, Shiliang Pu, and Shih-Fu Chang. Counter- factual critic multi-agent training for scene graph generation. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp.
2019.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training ceff-superviee vision transformers. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp- 9640-9649, 2021.
Xiangxiang Chu, Zhi Tian, Yuging Wang, Bo Zhang, Haibing Ren, Xiaolin Wei, Huaxia Xia, and Chunhua Shen. Twins: Revisiting the design of spatial attention in vision transformers. Advances in Neural Information Processing Systems, 34:9355-9366 9366, 2021.
Chia-Chi Chuang, Donglin Yang. Chuan Wen, and Yang Gao. Resolving copycat problems in visual imitation learning via residual action prediction. In European Conference on Computer Vision, pp. 392-409. Springer, 2022.
Blender Online Community. Blender a 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL (tRt:/tt::///at //wt:: b Lender. org.
Zijun Cui, Pavan Kapanipathi, Kartik Talamadupula, Tian Gao, and Qiang Ji. Type-augmented relation prediction in knowledge graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 7151-7159, 2021.
10

---

Published as a conference paper at ICLR 2024
Bo Dai, Yuqi Zhang, and Dahua Lin. Detecting visual relationships with deep relational networks. In Proceedings of the IEEE conference on computer vision and Puttern recognition, pp. 3076-3086, 2017.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference Learning Representations, 2021.
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 11:66)6667333 673, 2020.
Ankit Goyal, Kaiyu Yang, Dawei Yang, and Jia Deng. Rel3d: A minimally contrastive benchmark for grounding spatial relations in 3d. Advances Neural Information Processing Systems, 33: 10514-10525, 2020.
Ali Hassani, Steven Walton, Nikhil Shah, Abulikemu Abuduweili, Jiachen Li, and Humphrey Shi. Escaping the big data paradigm with compact transformers. arXiv preprint arXiv:2104.05704, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Kaiming Hc, Xinlci Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Comnecting lan- guage and vision using crowdsoureed dense image annotations. International jourrna of computer vision, 123(1):32-3,, 2017.
Chahyon Ku, Carl Winge, Ryan Diaz, Wentao Yuan, and Karthik Desingh. Evaluating robustness of visual representations for object assembly task requiring spatio natti-geommttrcal reasoning. arXiv preprint (tti::2310009944: 2023.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision, 128(7):19561999 1981, 2020.
Lin Li, Long Chen, Hanrong Shi, Wenxiao Wang, Jian Shao, Yi Yang, and Jun Xiao. Label semantic knowledge distillation for unbiased scene graph generation. arXiv preprint arXiv:2208.037633 2022.
Rongjie Li, Songyang Zhang, Bo Wan, and Xuming He. Bipartite graph network with adaptive mes- sage passing for unbiased scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 11109-11119, 2021a.
Yanghao Li, Saining Xie, Xinlei Chen, Piotr Dollar, Kaiming He, and Ross Girshick. Benchmarking detection transfer learning with vision transformers. arXiv preprint arXiv:2111.11429, 2021b.
Yikang Li, Wanli Ouyang, Xiaogang Wang, and Xiao' ou Tang. Vip-enn: Visual phrase guided convolutional neural network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1347-1356, 2017.
11

---

Published as a conference paper at ICLR 2024
Yuanzhi Liang, Yalong Bai, Wei Zhang, Xueming Qian, Li Zhu, and Tao Mei. Vn-vg: Refocusing visually-relevant relationships. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10403-10412, 2019.
Xin Lin, Changxing Ding, Jinquan Zeng, and Dacheng Tao. Gps-net: Graph property sensing network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3746-3753, 2020.
Xin Lin, Changxing Ding, Jing Zhang, Yibing Zhan, and Dacheng Tao. Ru-net: regularized un- rolling network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19457-19466, 2022.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023a.
Rosanne Liu, Joel Lehman, Piero Molino, Felipe Petroski Such, Eric Frank, Alex Sergeev, and Jason Yosinski. An intriguing failing of convolutional neural networks and the coordconv solution. Advances in neural information processing systems, 31, 2018
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hicrarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 10012-10022, 2021.
Zeyi Liu, Arpit Bahety, and Shuran Song. Reflect: Summarizing robot experiences for failure explanation and correction. In Conference Robot Learning, pp.
PMLR, 2023b.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3131-334400 2015.
Cewu Lu, Ranjay Krishna, Michael Bersstein, and Li Fei-Fei. Visual relationship detection with language priors. In European conference on computer vision, pp. 852-869. Springer, 2016.
Guoshun Nan, Zhijiang Guo, Ivan Sekulic, and Wei Lu. Reasoning with latent structure refine- ment for document-level relation extraction. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 1546-1557, Online, July 2020. Association for Computational Linguistics.
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. Indoor segmentation and support inference from rgbd images. ECCV, 2012.
Yohei Onuki, Tsuyoshi Murata, Shun Nukui, Seiya Inagi, Xule Qiu, Masao Watanabe, and Hiroshi Okamoto. Relation prediction in knowledge graph by multi-label deep neural network. Applied Network Science, 4:1-17, 2019.
Hao Peng, Tianyu Gao, Xu Han, Yankai Lin, Peng Li, Zhiyuan Liu, Maosong Sun, and Jie Zhou. Learning from Context or Names? An Empirical Study on Neural Relation Extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3661-3672, Online, November 2020. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
Bryan A. Plummer, Liwei Wang, Christopher M. Cervantes, Juan C. Caicedo, Julia Hockenmaier. and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. IJCV, 123(1):74-93, 2017.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2024
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. Yoshua Bengio and Yann LeCun (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Truck Proceed- ings, 2015.
Robin Strudel, Ricardo Garcia, Ivan Laptev, and Cordelia Schmid. Segmenter: Transformer for semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7262-7272, 2021.
Qingyu Tan, Ruidan He, Lidong Bing. and Hwee Tou Ng. Document-level relation extraction with adaptive focal loss and knowledge distillation. In Findings of the Association for Computational Linguistics: ACL 2022, pp. 1672-1681, 2022.
Hengzhu Tang, Yanan Cao, Zhenyu Zhang, Jiangxia Cao. Fang Fang, Shi Wang, and Pengfei Yin. Hin: Hierarchical inference network for decument-level relation extraction. Pucific-Asia Con- ference Knowledge Discovery and Data Mining, pp. 197-209. Springer, 2020a.
Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to compose dynamic tree structures for visual contexts. In Proceedings the puter vision and pattern recognition, pp. 661 9-6628, 2019.
the IEEE/CVF conference on com-
Kaihua Tang. Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. I Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 3716-3725, 2020b.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 2023.
Hugo Touvron, Matthieu Cord, Matthij Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve. and Hervé Jégou. Going deeper with image transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 32-42, 2021b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Chen Wang, Danfei Xu, and Li Fei-Fei. Generalizable task planning through representation pre- training. IEEE Robotics and Automation Letters, 7:8299-8306 2022. doi: 10.1109/LRA.2022. 3186635.
Haoyu Wang, Ming Tan, Mo Yu, Shiyu Chang, Dakuo Wang, Kun Xu, Xiaoxiao Guo, and Saloni Potdar. Extracting multiple-relations in one-pass with pre-trained transformers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1371-1377, Florence, Italy, July 2019. Association for Computational Linguistics.
Chuan Wen, Jianing Qian, Jierui Lin, Jiaye Teng, Dinesh Jayaraman, and Yang Gao. Fighting fire with fire: avoiding dnn shorteuts through priming. In International Conference Machine Learning, pp. 23723-23750. PMIR, 2022.
Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative mes- sage passing. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5410-5419, 2017.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18134 1814, 2022.
Kaiyu Yang, Olga Russakovsky, and Jia Deng. Spatialsense: An adversarially crowdsourced bench- mark for spatial relation recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2051-2066 2019.
13

---

Published as a conference paper at ICLR 2024
Yuan Yao, Ao Zhang, Xu Han, Mengdi Li, Coreeliss Weber, Zhiyuan Liu, Stefan Wermter, and Maosong Sun. Visual distant supervision for scene graph generation. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 15816-15826, 2021.
Wentao Yuan, Chris Paxton, Karthik Desingh, and Dieter Fox. Sornet: Spatial object-centric repre- sentations for sequential manipulation. In Conference on Robot Learning, pp. 148-157. PMLR, 2022.
Alireza Zareian, Zhecan Wang, Haoxuan You, and Shih-Fu Chang, Learning visual commonsense for robust scene graph generation. European Conference on Computer Vision, pp. 642-657. Springer, 2020.
Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5831-5840, 2018
Hanwang Zhang, Zawlin Kyaw, Shih-Fu Chang, and Tat-Seng Chua. Visual translation embedding network for visual relation detection. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 5532-5540, 2017a.
Hanwang Zhang, Zawlin Kyaw, Jinyang Yu, and Shih-Fu Chang. Ppr-fen: Weakly supervised visual relation detection via parallel pairwise r-fen. Proceedings of the IEEE international conference on computer vision, pp. 4233 4241, 2017b.
Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems, 31, 2018.
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher D. Manning. Position- aware attention and supervised data improve slot filling. In Proceedings of the 2017 Conference on Empirical Methods Natural Language Processing, pp. 45-45, Copenhagen, Denmark, Septem- ber 7c. Association for Computational Linguistics.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. Image BERT pre-training with online tokenizer. In International Conference on Learning Representa- tions, 2022.
Wenxuan Zhou, Kevin Huang, Tengyu Ma, and Jing Huang. Document-level relation extraction with adaptive thresholding and localized context pooling. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 14612-14444 14620, 2021.
14

---

Published as a conference paper at ICLR 2024
A MORE DETAILS OF DATASETS AND EXPERIMENT SETUP
A.1 FULL DEFINITION OF THE RELATION CLASSES
Table 7: The precise defnitton of each relation category in SpatialSense+.
Relation
Definition
From the direction of gravity, the position of the subject is higher than object, the two are not in contact, and the two overlap in the direction of gravity.
above
The depth of subject is larger than the object.
behind
If two objects in very different directions, then skip this sample.
in
The subject is inside the object, and the object must at least semi-enclose the subject.
The depth of the subject is smaller than the object. If two objects are in very different directions, skip this sample.
in front of next to
Subject is near to object, and there is no other obstacle between them.
The subject is on top of object and the two are in contact. If the subject is attached to the side of the object. it is not considered as "on". If the subject is semi-enclosed by the object, then it is not considered as "on".
on
The subject is to the left of object from the labeler's view.
to the left of the right of
If the difference in depth height between the two is too large, skip the sample.
The subject is to the right of the object from the labeler's view.
If the difference depth or height between the two is too large, skip the sample.
under
From the direction of gravity, the position of the subject is higher than the object.
Table The hyperparameters of Relati
papee-prrameters
Re13D
SpatialSensc+
epochs early stop optimizer learning rate Ir schedule Ir warm-up epoch weight decay layer decay query embedding pooling
200
100
20
20
adamw
adamw
le-5
le-5
cosine
cosine
5
le-4
le-3
0.75
0.75
max
max
A.2 TRAINING DETAILS
Input preprocessing: The images of ReI3D are center cropped to 224 x 224 and the ones of Spa- tialSense+ resized to 224 x 224. We perform RandomCropResize and ColorJitter augmentations on the input images to make the training samples more diverse. At the same time, the bounding box coordinates changed accordingly. Then normalize the images and feed them into the models. As for the object class inputs, we use Glove (Pennington et al., 2014 to encode the words into 300-dimensional embeddings. The bounding box input is 4-dimensional vector representing the coordinates of the upper left and lower right corners.
Training objective: We use binary cross-entrops loss to train all the models. Following the settings in Goyal et al. (2020) and Yang et al. (2019), we use the weighted loss for Re13D and not for Spnteallense+, where the weights are computed according to the frequency of the relation categories.
For the CNN models, we use ResNet101 as the feature extraction backbone, and load the ImageNet supervised pretrained weights. For ViTs, we take ViT-Base as the backbone, and load the IBOT [hhou et al. 2022) pretrained model by default. In Appendix B.3 we also
Model details:
15

---

Published as a conference paper at ICLR 2024
try other pretrained models, such as DEIT Touvron al. 2021a). MOCO-v3 (Chen et al. 2021
DINO (Caron et al., 2021), MAE (He al. 2022) and CLIP (Radford al. 202
A.3 HYPERPARAMETERS
The hyperparameters of our best model ViT are shown in Table 8 The hyperparameters of other baselines follows the setting in Goyal et al. (0200) and Yang al. (2019).
A.4 DETAILS OF VISION LANGUAGE MODELS
The specific version of each vision language model shown in Table The input prompts are shown below, where (} denote the placeholders for the names of the subject, object, and predicate, and the corresponding bounding box coordinates:
"Here are the definitions of relation predicates. Above:
From the direction of gravity, the position of the subject is higher than object, the two are not in contact, and the two overlap in the direction of gravity.
Behind: The depth of subject is larger than the object.
In: The subject is inside the object, and the object must at least semi-enclose the subject. In front of: The depth of the subject smaller than the object.
Next to: Subject is near to object, and there is no other obstacle between them. On: The subject is top of object and the
in contact.
To the left of: The subject is to the left of object from the labeler's view. To the right of: The subject Under: From the direction gravity, the position of the subject is lower than the object.
the right of the object from the labeler's view.
According to the definitions, tell me whether the statement that the O in blue bounding box (whose upper left corner coordinate is [x.y]=[] and lower right corner coordinate is (x,y)=()) is O the O the red bounding box (whose upper left corner coordinate [x.y]=(] and lower right corner coordinate is (x,y)=[) is True. Just tell me True or False."
Table 9: Version of each Vision Language Models.
Model
Version
MiniGPT-4 LLaVA
V2
LLaVA-1.5 13B
Gemini
Gemini-Pro- Vision
GPT-4V
GPT-4V Azure API
B MORE EXPERIMENTAL RESULTS
B.1 MORE ARCHITECTURE DESIGNS AND COMPARISON RESULTS.
In Table 10, change the design options of the four components introduced in Section 4. one by one in order to deliberately show the effect of each ax.
We can that "VIT RoiAlign + sclf-attention" is better than CNNTransformer, indicating that VIT features are more suitable for the spatial relation prediction task than CNN.
The performance of "VIT MaskImage self-attention and "VIT RoiAlign + self-attention" is similar, illustrating that the commonly-use RoiAlign technique (He et al. 2017) does bring more benefits beyond the simple MaskImage strategy. But the latter is more suitable for end-to-end transformer modeling because of its simplicity.
Comparing "ViT + MaskImage self-attention" with CrossAutnVi,, we find the self-attention rela- tion decoder performs similarly to the cross-attention. We hypothesize the reason that different
16

---

Published as a conference paper at ICLR 2024
from the machine translation tasks where the cross-attention decoder is commonly used, the self- attention and cross-attention decoder are appropriately equivalent because the queries and keys are in the same domain in our spatial relation prediction task.
Finally, we compare the cascading architecture "VIT+ Maskimage + self-attention" with Relati VIT, and the superior performance of RelatiViT illustrates the effectiveness of the non-cascading design.
Table 10: Fine- sine-grained comparison between more designs.
RellD
SpatialSense+
Model
S:Avg. Acc.
RFF
%.Avg. Avc.
SFF1
CNN RoiAlign Self-atiention (CNNTranstormer)
72.09 = 0.52
74.86
+ 0.35
68.21 + 0.92
66.83 + 0.72
RosAlign + Self-attention
79.10
0.57
VIT + MaskImage + Self- attention
74.37
+ 1.25
77.01
+: 1.23
74.40
+: 2.14
72.59
+: 2.33
VIT + MaskImage + Cross-attention (CrossArttVii))
76.55
0.28
78.76
0.55
73.41
+ 2.00
71.77
3.40
VIT MaskImage Self-attention End-to-Ead (RelatiViT)
$0.09
0.33
82.05
+ 0.19
80.02
+ 0.73
78.32
1.01
B.2 ACCURACY ON EACH CLASS ON REL3D
The detailed results of accuracy on each relation class on Re13D are shown Table We can see that RelatiVIT outperforms the bbox-only and DRNet most of the relation classes. Especially classes where visual information is critical such as "lean against", "inside", "aligned to", "touching" and "on top of"" etc., our method is significantly better than the baselines, indicating that our method successfully extracts the visual representations containing spatial relation information.
Table Aceuracy of each class on Rel3D with average values and standard deviations.
Method
behind you)
to the side of
the side of you)
bhox..only only
1.15
66.95
82.93
DRNet
76.65 + 1.09
95.78 + 1.06 94.00 + 1.17
64.25 + 3.52 72 82 1.80
$1.20
1.48
85.28 + 1.67
RelauiV/T(ous)
84.68
1.68
0.98 87.
95.50 near
1.36
methodl
to the right of
in front of
points towards
behind
bbox-only
54.17 + 2.17
62.03 + 4.35
51.30 0.96
59.09 2.54
91.60 + 0.27
DRNet
49.50 3.52
58.28 1: 2.78
50.43 1.98
57.27 3.51
84.79 1: 2.17
RelatVViT(uur))
1.62 63.17
71.56 +: 1.36
72.61 1.06
62.73 3.66
91.85 1.21
method
points away
aligned to
Jaces towards
leaning against
Faces away
bbox-only
50.32 + 2.49
61.21 + 2.31
53.50 0.62
70.00 2.16
55.84 + 1.16
DRNet
52.70 2.49
64.14
2.70
55.50
RelasiViT(ours)
57.46 + 2.11
72.59 + 2:21
64.00 + 2.26
80.81 + 2.32
56.10 + 1.86
(wri you)
on lup
louching
bbox-only
80.07 2.96
95.00 + 1.11
87.66 1. 1.08
86,22 0.54
67.27 + 3.75
DRNet
74.41
4.42
93.06 + 3.04
73.62 1.56
82.84 1.10
67.12 :: 2.07
RelatiViT((uus)
85.72
1.23
96.39 1 1.42
2.77 80.85
91.76 0.79
75.45 1 1.70
method
covering
to the left of (wit you)
to the right (wit
the of
below
bhox-only
86.94 0.81
94.60 + 0.64
94.52 0.44
63.33 7.
81.49 + 2.65
DRNet
81.01 + 1.26
92.06 + 0.56
91.70 1.27
62.00
7.77
78.21 = 1.88
RelatiViT(ous)
88.03 0.74
95.16 + 0.58
0.52 97.22
4.40 around
method
Under
passing through
inside
outside
bbox-only
83,83 + 1.70
73.58 + 1.19
71.33 1.35
79.76 0.71
63,33 = 4.86
DRNet
83.19 1.04
72.08 + 4.68
71.50 2.76
82, 2.02
68.33 + 3.33
RelatiViT(ous))
90.21 0.80
72.08 1 0.96
80.17 1.43
85.98 1.02
72.50
5.65
B.3 ANALYSIS PRETRAINED MODELS
Effect of pretrained representations. We investigate the effect of pretraining methods for the spa- tial relation prediction task. We compare the performance of ReTatiViT with different pretrained models: random initialization (scratch), DEIT (Touvron et al.) 2021a), MOCO-v3 (Chen et al,
2021), CLIP (Radford al. 202T), DINO (Caron al. 2021) IBOT (Zhou al. 2022 and
17

---

Published as a conference paper at ICLR 2024
MAE (He et al. 2022) We evaluate these pretrained models both Re13D and SpatialSenset. Looking at the average performance over two settings (the third cluster of bars) in Figure 6 we can see that most of the self-supervised methods are better than the supervised one (DEIT) and training from scratch, except MOCO-v3. Among all the self-supervised learning on ImageNet, the self- distillation methods (IBOT and DINO) performs best. And IBOT performs better than DINO be- cause the patch-wise pretraining IBOT implicitly learns the relation information. Besides, CLIP performs well (only worse than IBOT) because it takes advantage of the multi-modal contrastive learning and the larger-scale pretraining dataset.
80
Average Accuracy
70
60
50
40
30
Rel3D
SpatialSense+
average over settings
supervised
self-distillation
seratch
DEIT
MOCO-v3 CLIP
DINO IBOT MAE
contrastive
mask image modeling
Figure Effectiveness of different pretraining methods.
B.4 COMPARISON WITH CNN MODELS WITH POSITIONAL EMBEDDINGS
There are two main differences between transformers and CNN: attention mechanism and positional embeddings. To further strengthen our argument that transformers leverage the attention mecha- nism to capture the relation information, we compare our RelatiViT to CNN-based models with positional embeddings implemented by CoordConv Liu et al. (2018). Specifically, we incorporate positional embeddings into the input tensor of the first layer of ResNet in DRNet, RUNet, and CN- NTransformer. As shown in Table 12] the performance these CNN-based models remains inferior compared to VeT indicating that they do not significantly benefit from positional embeddings. This outcome reinforces enhanced ability to capture spatial relationships through attention mechanisms, as opposed to the sliding convolution in traditional CNNs.
hypothesis that RelatiViT's superior performance is largely due to its
Table 12: Comparison with CNN-based models with CoordConv.
SpatialSensee
Model
%.Agg. Ave.
DRNet CoordCony
76.86 0.96
73.44 + 0.90
RUNet + ConedConv
75.91 0.73
74.68
1: 1.26
CNNTransfomer CoordCony
68.69 1.23
66.47
1: 1.81
RelatiVIT
80.02 + 0.73
78.32 + 1.01
B.5 EFFECTS OF VISUAL FEATURES ON DIFFERENT MODELS
To quantitatively verify statement that the existing methods tend make predictions mainly according to bounding box coordinates rather than image content, we conduct an intervention exper- iment by replacing the input images with random ones while keeping the original bounding boxes
18

---

Published as a conference paper at ICLR 2024
during evaluation. This is common technique to diagnose the causal effect of each input sig- (Wen al., 2022 Chuang et al. 2022, Bruce et al. 2024). As shown in Table 13 we report the prediction pipping ratio, which measures the frequency of predictions changing from True to False, or vice versa, across all samples. higher flipping ratio suggests greater reliance on image content, while lower ratio indicates tendency to base predictions on bounding box coordinates. evident that the prediction flipping ratios for CNNTransformer and RelatiViT are comparable to each other, and are notably higher than those for RUNet and DRNet. This result suggests two key insights:
1) Our transformer-centrie design principles (including feature extraction, query localization, con- text aggregation, and pair interaction), which do not incorporate bounding box coordinates into the input unlike previous methods, are indeed more inclined to utilize visual information for predicting spatial relations. This finding supports the rationale behind our design choices, especially in the context of spatial relation prediction tasks.
2) Despite similar emphasis visual information, RilatiViT significantly outperforms CN- NTransformer. This indicates the superior capability of the vision transformer backbone in ex- tracting spatial relation information compared to CNNs.
Table 13: Intervention results on SpatialSense+. The piiping ratio denotes the percentage of samples whose prediction flips after we intervene their input images.
Model
Preediction llipping ratio
DRNet
2.33 2.11
RUNet
6.99 4: 2.26
CNNTranffrmee
30.18 +: 1.69
RelatiVTT
29.47 = 1.88
B.6 VISUALIZZATIONS OF PREDICTIONS
We show some examples of the output predictions of the baselines and RelatiViT in Figure Our Relati performs significantly better and more reasonably than all the baselines. The performance on these hard samples that cannot be handled by the naive methods such as bbox-only illustrates that ReltiViT predicts the spatial relations based on the visual information, e.g., object contact, occlusion, depth, background context, etc.
B.7 MORE VISUALIZATIONS OF ATTENTION MAPS.
Table shows more visualization results of the attention maps of Relati and VIT cross-attention decoder. We can see that the attention maps of RelatiViT are clearer, and the subject and object queries attend to the contextual information between the two, in addition to referring to their own regions, which is conceptually reasonable for predicting spatial relations.
B.8 RESULTS ON ORIGINAL SPATIALSENSE
We also conduct experiments on the original SpatialSense dataset and the results are shown in Ta- ble 14 Because of the imprecise definition of the relation classes, the original dataset still contains language biases during annotating. As result, insufficient to benchmark the authors' intended objective- predicting spatial relations based on visual information. Consequently, our method Rel- ViT performs similarly the baselines. This illustrates the necessity to define the precise, unam- biguous and physically grounded relation categories and relabel the SpatialSense dataset.
19

---

Published as a conference paper at ICLR 2024
food bowl ? GT: True
giraffe front of gruss ? GT: True
bbox-only: False
bbox-only: False
DRNet: False
DRNet: True
Vtransi:: False
Viranst:: False
Motif&Net: False
Motif&Net: False
VIP-CNN: True
VIP-CNN: False
PPR-FCN: False
PPR-FCN: False
(elrt)VTT(ouss): True
(olr:))TT(our):: True
door behind person? ? GT: True
lamp to the right of bed ? GT: True
bbox-only: True
bbox-only: True
DRNet: True
DRNet: False
VtransE:: True
VtrunsE; False
MotifiNet: True
MotifiNet: False
VIP-CNN: False
VIP-CNN: False
False
True
(elatiViTTours): True
(eladiVTT(ours): True
bottle next man GT: False
chair on table? GT: False
bbox-only: True
bbox-only: True
DRNet: True
DRNet: True
Viransl::
Viransl:: True
MotifiNet: True
MotifNNet: Fulse
VIP-CNN: True
VIP-CNN: True
PPR-FCN: True
PPR-FCN: True
(olut)) False
(elus)VTTouus)): False
tuble the left frame? GT: Fulse
mountains above truck? GT: False
bbox-only: True
bbox-only: True
DRNet: False
NRNet: True
VurasE: True
VtransE: True
MotifiNet: True
MotifNNet: True
VIP-CNN: False
VIP-CNN: True
PPR-FCN: False
PPR-FCN: True
RelatiViT(ours): False
RelatiVTT(ours); False
Figure The prediction examples of the models, where GT denotes "ground truth".
Table 14: The results on the original SpatialSense.
original SpatialSense
method
&Avg. Acc.
%Fl
bbox-only bbox-language
64.91 = 0.71
66.06 + 1.29
61.88 0.66
63.01 1: 1.05
DRNet DDai al. 2017
66.96 = 0.98
69.32 + 1.25
VTransE: (Zhang
22077
64.96 0.96
66.42 1.27
MotifsNet (Zellers et al 20088
65.21 1.04
67.88 + 1.17
Vip-CNN (Li et al. 2017
63.29 + 0.47
64.80 + 0.65
PPR-FCN (Zhang et al
2017b
59.44 0.59
62.33 + 0.56
RiTatiViT (ours)
65.79 + 1.12
68.55 + 1.63
20

---

Published as a conference paper at ICLR 2024
VIT Cross attentittnionn Decoder
image
contest
subject query object query
image
context subject query object query
subjoet
attention
artention
eake
gt:
cake frunt of buy
gt: True
pred:Fabee
subject
artenrion
object attention
book on shelf gf False pred:: False
book shelf
pt: pt:Fulse
pred:True
subject
attention
object attention
lamp mext
gt:Tiree pred:True
lamp prediFulse
subjert
attention
object attention
Indder the side frame
gt:Fabbe
Pred:False
ladder the side frame
gt:False
pred:True
subject attention
object
attention
shoe in front of (wrt you) controller pt:True pred: True shoe front of (wrt you) controller gi=True pred:False
subject
attention
object
attention
passing through pred:True
passing through
gi:True pred:False
Figure 8: More attention map visualizations of End-to-end ViT and VIT cross-attention decoder Re13D and SpatialSense+, where "gt" denotes the ground truth label and "pred" is the prediction of the model.
21


Reviews:
[{"review_id": "dev9JIBRAa", "reviewer": "Reviewer_Jenr", "summary": "This paper investigates the task of spatial relation classification on a single 2D image. Given two objects bounding boxes, the authors uses the nine predicates proposed in the SpatialSense dataset and correct some of its ambiguous labels. To solve the task, the authors present different architectures and promote a simple transformer based model. Experiments are also conducted on REL3D dataset with ablation studies.", "questions": "in section 5.4, the author wrote \"bbox-language cannot beat bbox-only, indicating that we have successfully removed the language bias in our benchmark.\"\n\nHowever, in Table 11 where there is an evaluation on the original SpatialSense, where the bbox-only is also superior to bbox-language. Then, how the author can conclude to the relations between these two baselines and the language bias ?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The authors propose a more accurate definition of the problem of spatial relation, although the limitations of this definition is not completely covered.\n\nThe motivation to use transformers to model object relation is sensible, as relation modelling this is one of the main motivations over CNNs baselines.\n\nThe proposed architecture is validated with different baselines and ablation studies. In particular, I appreciate that the author also report the results of naive baselines. Experiments confirmed the usefulness of the proposed transformer mechanism to model relations.", "weaknesses": "The current superiority of backbone transformers over CNNs and the ability of transformers to model object relations are commonly admitted. So in this aspect, this downgrade the insights brought by the results of the paper. \n\nThe claim of the authors to provide \"a precise and unambiguous definition\" seems a bit exaggerated. There are natural ambiguities in the task of spatial relationships. For instance, even for humans, it is difficult to judge on same natural images the relations \"above\" if the objects are very distant and that it is not easy to judge if they are on the same plane. For instance, between a car and a mountain in the background (this is one of the examples provided in the supplementary material).\n\nThe definition provided by the authors include subjective (which seems naturally inevitable for this task) appreciations in the supplementary material. For instance, the relation \"behind\" is ignored if the objects are \"in different directions\" and the relation \"in front of\" is ignored \"if the distance is too large\". This definition is not provided by the author, and in practice, it might even depend on the application and the image context. In the case of robotic application, which is the motivation suggested in the introduction of the paper, the definition of \"next\" could applied to all objects belonging to the same group. \n\nLastly, the author reuse the same predicates as the previous dataset. This limits their contribution to making corrections on an existing dataset.\n\nIn general, the paper does not bring a lot of new insights about the method as the modelling power of transformer is well known. This work do bring an improvement to an existing dataset, enabling researchers to better evaluate their method in the task of spatial relations, but I am not sure whether this in itself would be of interest to the ICLR community. Maybe this paper would be a better fit for a more vision or robotic dedicated conference.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aWPMo135xx", "reviewer": "Reviewer_XTRZ", "summary": "The paper studies how to understand spatial relationship between objects within a single image. GIven that the current benchmarks are synthetic (Rel3D) or ambiguous (SpatialSense), it starts with a re-annotation of SpatialSense with precise relation definitions and calls the new dataset SpatialSense+. Since existing state-of-the-art methods perform poorly on this benchmark, the paper proposes a RelatiViT, which is a ViT based network to resolve the problem. This is the first approach which outperforms naive baseline significantly on multiple benchmarks.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Overall, I think this is a solid paper, and recommend accepting the paper.\n\n- The paper discusses limitations of current spatial relationship benchmarks. That makes a lot of sense to me.\n- The paper did an extensive study of different network architectures to capture the spatial relationship between objects.\n- Experiments suggest the proposed RelatiViT has strong performance on multiple benchmarks.\n- The writing is clear and easy to follow.", "weaknesses": "- The paper mainly compared RelatiViT with ConvNet and ViT based networks. However, some of current multimodal Large language models (e.g. LLaVA [1], or even GPT-4V) seem be to capable of handling some spatial relationships. But I think it's acceptable to ignore these literature as they are very recent works and are not specially designed for this task.\n- The annotation of SpatialSense+ can be very hard to scale, since it requires annotators to understand precise definition of spatial relationships.\n\n\n[1] Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee. Visual Instruction Tuning. NeurIPS 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YOZlyPAqjB", "reviewer": "Reviewer_MUvq", "summary": "This paper sets out to study transformers' capability of understanding spatial relations in a scene. The authors strictly define spatial relations and refine previous datasets accordingly. The authors propose several transformer-based models to tackle spatial relation prediction task.", "questions": "Please see weaknesses above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- It is interesting to delve into whether transformers could understand spatial relations in a scene instead of simply exploiting spatial biases.\n\n- The linguistic biases during annotation brought up by this paper are reasonable.\n\n- The proposed models are reported to be effective on refined datasets. The discussion on design axes is informative.\n\n- Attention map visualization is insightful as it shows that the proposed model captures meaningful contextual information when making a prediction.", "weaknesses": "- Although resolving linguistic biases in annotations is a well-motivated aspect of this work, I am not sure if these issues are nontrivial. It would be great if there were a quantitative bias analysis of previous datasets.\n- Besides linguistic biases during the annotation process, the statistics of spatial relations are biased, which could lead spatial relation detection to simply object classification. Rel3D addressed it by employing minimally contrastive construction, I would like to see authors' discussion on SpatialSense+. \n- Despite being titled as \"Can Transformers Capture Spatial Relations between Objects\", all prior baselines are CNN-based methods, it would be great to see authors include more advanced transformer-based models [1].\n\n[1] SGTR: End-to-end Scene Graph Generation with Transformer, Li et al, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ElMRUlhHrK", "reviewer": "Reviewer_hAXs", "summary": "The paper investigates the ability of current computer vision systems to recognize physically grounded spatial relations between objects. The authors propose precise relation definitions that allow for consistent annotation of a benchmark dataset and find that existing approaches perform poorly on this task. They propose new approaches that exploit the long-range attention capabilities of transformers and evaluate key design principles. The authors identify a simple \"RelatiViT\" architecture that outperforms all current approaches and demonstrate that it is the first method to convincingly outperform naive baselines on spatial relation prediction in in-the-wild settings. The findings have important implications for the development of computer vision systems and their ability to recognize spatial relations.", "questions": "Please refer to the weakness part.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper proposes precise relation definitions that permit consistently annotating a benchmark dataset. \n2. the paper identifies a simple \"RelatiViT\" architecture that outperforms all current approaches and convincingly outperforms naive baselines on spatial relation prediction in in-the-wild settings.\n3. The paper evaluates key design principles and provides insights into the effectiveness of different approaches for recognizing physically grounded spatial relations between objects.", "weaknesses": "1. The spatial relationships examined in this paper are rather straightforward, as they do not encompass interactions between humans and objects or comparative relationships among multiple objects. This limitation might confine the applicability of the proposed method.\n2. This paper would benefit from a more extensive examination of the reasons behind the superior performance of \"RelatiViT\" compared to the \"bbox-only\" baseline. Current methods utilizing image features do not match the simplicity and effectiveness of the \"bbox-only\" baseline. What sets \"RelatiViT\" apart?\n3. As illustrated in Table 6, \"RelatiViT\" does not consistently outperform the \"bbox-only\" baseline across various categories of spatial relations. The potential advantages of \"RelatiViT\" might not be evident when taking into account the higher computational demands it imposes for image feature processing.\n4. The paper fails to elucidate the reasons why \"RelatiViT\" exhibits superior performance in the \"left\" category as opposed to the \"bbox-only\" baseline, while displaying subpar performance in the \"right\" category.\n5. Although it is good to unveil that the current methods and datasets have limitations on this new task, it is still imperative to do a thorough study on each proposed component to show why such a design is essential to the performance improvements. It is hard to be convinced about the significance of the contribution based on the current form.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LzHjpDBO0W", "reviewer": "Reviewer_ubwR", "summary": "This paper studies the task of spatial relation prediction. The paper observes that prior works have focused on semantic relationships that may not be physically grounded and which can be ambiguous. To alleviate this issue, they define more precise and physically grounded definitions for relationships and reannotate an existing dataset (SpatialSense) to provide an improved test bed. Furthermore, the propose a new architectural choice that performs well on both existing synthetic datasets as well as the newly annotated benchmark.", "questions": "- Are the newer annotations simplifying the task or providing a stronger evaluation? I would appreciate it if the authors commented on the points raised in the weaknesses and provided some quantitative or qualitative results supporting it. I want to note that I think the paper is still valuable even if it was simplifying the annotation, it is just a matter of better explaining the contributions and limitations of the proposed dataset as an evaluation for this task. \n- How are the ablations conducted? I would appreciate it if the authors could explain what the model looks like under each ablations. As noted above, I think it would be stronger if the ablations are conducted independantly. If this is not possible (due to some reliance on each other), it would be good to explain that.\n- Do positional embeddings explain the difference between CNN and Transformers in Table 3?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is very well motivated with a clear explanation of how the proposed analysis and method fits within the larger scope of the field.\n- The paper identifies a very interesting and nuanced ambiguity in existing annotations and provides clear examples of such issues. \n- The paper provides a good explanation for the different subtasks that a model needs to do and the corresponding architectural components that do it in Sec 4.1. \n- I appreciated that the authors included the very strong performing naive baselines and elaborated on why they perform well. \n- The paper has a nice flow where the proposed architecture follows logically from the analysis and design considerations presented in the work.", "weaknesses": "I found three weaknesses in the paper: (1) it is unclear if the newer annotation provide a better evaluation or a simpler task; (2) ablations are not very clear; (3) several statements are not supported by the results. I have listed the major concerns below. I also included some minor concerns that should be dealt with as extra suggestions, and do not need to be addressed in the rebuttal.\n\n\n- The paper very nicely motivates language ambiguity as an issue for obtaining spatial relations. However, it is unclear if the newer annotation scheme results in more accurate annotation or simply annotation that is better correlated with a relative location prior. \n    - Comparing Tables 4 and 11, one finds that all methods achieve a higher absolute score on the newer dataset, while the bbox baselines and the proposed method seeing stronger gains. Furthermore, Table 7 shows that the new annotation seems more specific regarding annotations being done in the labelers point of view or gravity to minimize ambiguity. As a result, while it is possible that the new definitions are more precise and a better evaluation, it is also possible that they are more strongly correlated with the relative locations in the image, which would introduce a strong prior. \n    - Some of the definitions in Table 7 support the point above. For example, \"A in front of B\" is defined as A is closer to the camera than B. While this is a less ambiguious definition, it is a much easier one than the typical usage of \"in front of\" which often also requires reasoning about the subject and object's relative orientation. For example, if two people are facing away from the camera, the further away person would be in front of the person closer to the camera. This is still unambiguous, but the prediction would be more difficult as it requires the model to both reason about relative depth as well as orientation of each person. \n    - Said differently, the newer annotation scheme reduces the ambiguity but it also makes the relationships less dependant on context for cases like front/behind where the RelatiViT shows greatest performance improvements on the baseline (Table 6). This can be shown by the great improvements in performance of the bbox-only baseline. It would be nice to provide some qualitative analysis or quantitative comparisons that address this. \n\n- The ablations are not very clear and the conclusions drawn do not seem supported by the results. \n    - It is unclear how the method works without Pair Interaction, Context Aggregation, or Feature Extraction. For example, pair interaction is stated as the MLP that combines the two features, how is the prediction made without it? Additionally, how does one make predictions without feature extraction? is it based on pixel values, patch embeddings, something else? I think it is very important to explain exactly what each ablated model looks like. \n    - The ablations are additive, so that the feature extraction ablation is applied to an already ablated model. While this is valid experiemntal choice, it makes it difficult to understand the most critical modules as they are not ablated independatly of each other. This is more important in this problem where we know that feature extract is not actually very important since bbox baselines perform very well, as shown in the paper. \n\n- Several statements or claims are not fully supported/substantiated.\n    - The statement `it is difficult for CNN backbones to extract relation-grounded representations` (sec 5.4) is strongly confounded by the use of positional embeddings. Specifically, based on the description, it seems that all methods (ViTs) have access to positional embeddings, while the CNN-Transformer baseline doesn't. While I understand that this is a common choice for CNNs, it seems like a strong confounder here due to the strong performance of bbox-only baseline. It is possible that ViTs outperform CNNs due to the explicit inclusion of positional embeddings rather than due to attention explicitly modeling pairwise relationships as suggested in Sec 4. While I think that attention is playing a strong role here, I think providing the CNN model with some positional embeddings, eg, through CoordConv (Liu et al, NeurIPS 2018), would help clairfy this point and explain whether CNNs are still poor feature extractors even if another model performs context aggregation afterwards (following the design axes in sec 4.1). \n    - The paper states that `[prior methods] are prone to simply predict the relationships based on the bounding box coordinates rather than learn the intended solutions by referring to the RGB input.` (Sec 5.4) but that statement is not supported by the results. Most methods are consistently outperformed by the bbox-only baseline sugguestion that they are probably doing something other than predicting relationships baseline on bounding box coordinates. While I agree that those models are likely learning shortcuts, it seems odd that this shortcut would be using the bounding box coordinates if they are all underperforming the bounding-box baseline. Without conducting some further analysis, it is difficult to make such statements about what those models are doing. \n    - The description under Table 6 doesn't match the results. The paper states `Bbox-only performs better on “to the right of” than RelatiViT because according to the relation definition proposed in Section 3.2` which is not true. All methods achieve the same mean performance on the `right` relationship, while bbox outperforms RelatiViT on `Next` and `Above`.\n\n- I believe that there is a typo in the equation in Sec 3.1. I think you want to minimize the negative log likelihood, while the current term minimizes the log likelihood, as for $y{=}1$ the term would be minimized by estimating $\\hat{y}$ to be 0. If I am wrong, could you please clarify why this is the desired objective?\n\n\n\n\n------------\n**Minor concerns and suggestions:** Those are just things that I thought would improve the paper or minor issues I noticed. You do not need to address any of them and I did not factor them into any ratings. \n\n- While the authors do not have to fully explain every prior method used as a baseline, I think it would have been good to explain DRNet as it is used as the other baselines in the detailed comparisons presented in Table 6. \n- I think the conclusion of `In summary, RelatiViT successfully extracts effective visual representations for SRP` (sec 5.5) doesn't seem well supported. While it is great that a method can outperform such baselines and I appreciate the authors making such baselines very prominent in the paper, the results further showcase that we still do not have any methods that perform well on this task as their performance is still very close to models that only get bounding boxes. I think the paper is a good step towards better modeling and evaluation, but it is unclear if we have any methods that extract good feature for SRP.\n- Table 4: I think that adding some columns to clarify the difference between the methods would be useful. Speficially, Sec 4.1 did a nice job at clarifying the main modules for this task, and I thought that clarifying them in Table 4 would improve it.\n- Figure 4: A little bit of white space or gap between the plots would make the figure a lot more easier to parse especially since the attention maps meld into each other.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)