Multiview Scene Graph
Authors: Juexiao Zhang
Gao Zhu
Sihang Li
Xinhao Liu
Haorui Song
Xinran Tang
Chen Feng
Venue: NeurIPS 2024 poster (2024)
Keywords: scene representation; spatial understanding; place recognition; object correspondence; scene graph
Primary area: machine_vision
TL;DR: building a place+object multiview scene graph from unposed images as a topological scene representation

Abstract: A proper scene representation is central to the pursuit of spatial intelligence where agents can robustly reconstruct and efficiently understand 3D scenes. A scene representation is either metric, such as landmark maps in 3D reconstruction, 3D bounding boxes in object detection, or voxel grids in occupancy prediction, or topological, such as pose graphs with loop closures in SLAM or visibility graphs in SfM.
  In this work, we propose to build Multiview Scene Graphs (MSG) from unposed images, representing a scene topologically with interconnected place and object nodes. 
  The task of building MSG is challenging for existing representation learning methods since it needs to jointly address both visual place recognition, object detection, and object association from images with limited fields of view and potentially large viewpoint changes.
  To evaluate any method tackling this task, we developed an MSG dataset and annotation based on a public 3D dataset.
  We also propose an evaluation metric based on the intersection-over-union score of MSG edges. 
  Moreover, we develop a novel baseline method built on mainstream pretrained vision models, combining visual place recognition and object association into one Transformer decoder architecture. 
  Experiments demonstrate that our method has superior performance compared to existing relevant baselines.

Multiview Scene Graph
Juexiao Zhang
Gao Zhu
Sihang Li Xinhao Liu
Haorui Song
Xinran Tang Chen Feng"
New York University {juexiao zhang, cfeng)@nyu. edu
Abstract
A proper scene representation is central to the pursuit of spatial intelligence where agents can robustly reconstruct and efficiently understand 3D scenes. A scene representation either metric, such as landmark maps in 3D reconstruction, 3D bounding boxes in object detection, or voxel grids in occupancy prediction, topological, such as pose graphs with loop closures SLAM or visibility graphs in SfM. In this work, we propose build Multiview Scene Graphs (MSG) from unposed images, representing scene topologically with interconnected place and object nodes. The task of building MSG challenging for existing representation learning methods since it needs to jointly address both visual place recognition, object detection, and object association from images with limited fields of view and potentially large viewpoint changes. To evaluate any method tackling this task, we developed an MSG dataset based on a public 3D dataset. We also propose evaluation metric based on the intersectionnovve--uion score of MSG edges. Moreover, we develop novel baseline method built on mainstream pretrained vision models, combining visual place recognition and object association into one Transformer decoder architecture, Experiments demonstrate that our method has superior performance compared to existing relevant baselines. All codes and resources are open-source at https:/aaiec..ihhbbiMMMMMM
Introduction
The ability to understand 3D space and the spatial relationships among 2D observations plays a central role in mobile agents interacting with the physical real world. Humans obtain such spatial intelligence largely from our visual intelligence [26, 45]. When humans are situated in an unseen environment and try understand the spatial structure from visual observations, don't perceive and memorize the scene by exact meters and degrees. Instead, we build cognitive maps topologically based visual observations and commonsense [27, 48). Given imagery observations, are able to associate the images taken at the same place by finding overlapping visual clues and identifying the same or different objects from various viewpoints. This ability establish correspondence from visual perception constitutes the foundation of our spatial memory and cognitive representation of the world. Can we equip Al models with similar spatial intelligence?
Motivated by this question, we propose the task of building a Multiview Scene Graph (MSG) to explicitly evaluate representation learning model's 's capability of understanding spatial correspon- dences. Specifically, as illustrated in Figure 1, given a set of unposed RGB images taken from the same scene, this task requires building place+object graph consisting of images and object nodes, where images taken at nearby locations are connected, and the appearances of the same object across different views should be associated together one object node.
Corresponding author. The work was supported in part through NSF grants 2238968 and 2322242, and the NYU IT High Performance Computing resources, services, and staff expertise.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Multiview Scene Graph
Multiview Scene Graph Network
Television TV Stand
Oven Kitchen hood
Chairs
Painting
MA
Unposed images
Scene
Figure 1: Multiview Scene Graph (MSG). The task of MSG takes unposed RGB images as input and outputs place+object graph. The graph contains place-place edges and place-object edges. Connected place nodes represent images taken the same place. The same object recognized from different views is associated and merged as one node and connected to the corresponding place nodes.
We position the proposed Multiview Scene Graph general topological scene representation. It bridges the place recognition from robotics literature [3, 4, 36] and the object tracking and semantic correspondence tasks from computer vision literature [20, 23, 64]. Different from previous work in topological mapping that evaluates a method's performance on downstream tasks such as navigation, propose directly evaluate the quality of the multiview scene graph, which explicitly demonstrates a model's 's spatial understanding with correct visual correspondence of both objects and places across multiple views. Moreover, the MSG does not require any metric map, depth, pose information, making it adaptable to the vast data of everyday images and videos. This also differentiates MSG from the previous work in 2D and 3D scene graphs [5, 32, 35, 69], where they emphasize objects' semantic relationships or require different levels of 3D and metric information.
To facilitate the research of MSG, curated a dataset from publicly available 3D scene-level dataset ARKitScenes [8] and designed a set of evaluation metrics based on the intersection-over union of the graph adjacency matrix. The detailed definition of the MSG generation task and the evaluation metrics are discussed in Section 3.1. Mcanwhile, since this task mainly involves solving place recognition and object association, benchmarked popular baseline methods respectively in place recognition and object tracking, as well as some mainstream pretrained vision foundation Transformer-basee architecture as method, Attention Association
models. We also designed MSG, dubbed AoMSG, which learns place and object embeddings jointly in a single Transformer decoder and builds the MSG based on the distances in the learned embedding space. Our experiments demonstrate the superiority of our new model compared with the baselines by a great margin, yet still reveal strong needs for future advances in research for spatial intelligence.
In summary, our contributions are two-fold:
We propose the Multiview Scene Graph (MSG) generation new task for evaluating spatial intelligence. We curated a dataset from a publicly available 3D scene dataset and designed evaluation metrics facilitate the task.
We design novel Transformer decoder architecture for the MSG task. It jointly learns embeddings for places and objects and determines the graph according to the embedding distance. Experiments demonstrate the effectiveness of the model over existing basclines.
2
Related work
Scene Graph Scene graphs [35, 69] are originally proposed to represent the spatial and semantic relationships between objects an image. The generated scene graph can be used for image captioning [47] and image retrieval [35]. Although they provide a structured spatial representation, remains the 2D image level. 3D scene graphs 32, 32, 33, 66, 71] extend this concept into 3D, representing scene topological graph with objects, rooms, and camera positions as their nodes.

---

These graphs are typically built by abstracting from 3D meshes, point clouds, or directly from RGB-D images. [67] proposes incrementally building 3D scene graphs from RGB sequences, describing semantic relationships between objects. As a new type of scene graph, MSG is built from unposed images without sequential order, emphasizing the understanding of relationships between objects and places via multiview visual correspondences. MSG complements existing scene graphs, as their object-object relationship edges can be a seamless add-on to extend MSG with more semantic information. Therefore, we believe MSG provides meaningful contribution to the scene graph community by enhancing its representational depth and flexibility.
Scene Mapping Simultaneous localization and mapping (SLAM) [17, 46, 57, 59] is a classic way of creating maps of an environment from observations. The metric maps built from SLAM are subsequently utilized as the spatial representation for the robots to perform tasks such as navigation. In contrast to metric maps, topological mapping [56] is inspired by landmark based memory in animals, and follows a more natural and human-likk understanding of the environment to better support navigation tasks. The quality of the topologiaal maps is evaluated mostly through navigation tasks [1 1, 15]. Another line of scene mapping work harnesses object or semantic information to build more robust maps [25, 54, 68], with TSGM [37] being the most relevant our work. Differently, our proposed MSG serves as a general-purpose scene representation and can be directly evaluated using our proposed metrics. The quality of MSG that model can build explicitly evaluates its capability of understanding spatial correspondences.
Visual Place Recognition Visual Place Recognition (VPR) is often formed as an image retrieval problem. This involves extracting image features and retrieving the closest neighbor from an image database. Traditional approaches rely on handcrafted features [9, 41]. NetVLAD and its variants[4, 16, 44] use deep-learned image features improve recall performance and robustness. The emergence of self-supervised foundation models, such as DINOv2 [49], enables universal image representations, offering significant progress [34, 36] across many VPR tasks. However, VPR is framed as an image retrieval problem, whose output- the image features does not directly equal a graph. Although graph can be built by proximity search in the VPR feature space, the widely used recall metric in VPR does not directly reflect how good the graph is, i.e. how many pairs of connected images in this graph are truly at the same place. Instead, our proposed task and evaluation metric focus only on the graph generated from the model. The metric straightforwardly reflects the quality of the scene representation.
Object Association Traditionally, object association is approached by matching keypoint features across image pairs [41, 55). Recently, CSR [24] learns feature encodings of object detections and measures the cosine similarity between the learned features to determine object matching. ROM [23] on the other hand follows SuperGlue [55] and uses attentional GNN and Sinkhorn distance [58] for relational object matching. Our method draws inspiration from this previous work but adopts Transformer decoder architecture and learns object instance embeddings jointly in unified model with place recognition. Literature in multi-object tracking [50, 64, 65, 72] and video object segmentation [19, 20, 31, 62] also handles object association. They mostly leverage temporal dependencies or memories such as by propagating detection bounding boxes or segments through time. Therefore, these models may lack sense of space and suffer when objects reappear from very different viewpoint or after a longer period. Interestingly. a recent study Probe3d [22] reveals that even though the pretrained vision foundation models have undergone tremendous progress in the recent years [14, 21, 30, 38, 49], they still struggle with associating spatial correspondences of objects from large viewpoint change. Our method learns scene representation with spatial correspondence, where multiple views of the same places or the same objects are close in the embedding space.
3 Multiview scene graph
3.1 Problem definition
Multiview Scene Graph Given set of unposed images of a scene X = (a:)i=o....,, we represent
a Multiview Scene Graph as a place+object graph:
G (P,O,E E
(I)
3

---

where P and O respectively refer to the sets of place and object nodes. The set of object nodes O contains all the objects detected from X. The same object detected from different images across different viewpoints should always be considered as one object node. For the definition of places, we follow the definition in the VPR literature and set P X. This means each image corresponds to a node for a place, and if two images are taken within only small translation and rotation distance, they are considered as taken in the same place and are connected with an edge in EPP. Consequently, the E is the set of place-place edges which refers to the edges that connect the images regarded as in the same place, and the EPO represents the set of place-object edges, referring to the edges that connect the places and the objects that appear in those places. Therefore, object can be seen in multiple images and thus connected to more than just one place node. These images can be close by or from distance. Naturally, place node can connect to more than one object node, since image can contain multiple objects' appearances.
MSG generation task As illustrated in Figure 1, the MSG generation task requires building an estimated place+object graph & from the unposed RGB image set. The graph is further represented as poacc+object adjacency matrix Å of size (IP) + (ől) x (1P) + (O]) while the groundtruth G is represented by A of size (I + (O1) x (IPI+ + 101). Note that the object set Ô may differ from O. The quality of & evaluated by measuring A against the groundtruth A. According to our definition, the adjacency matrix can be further decomposed into the following block matrix:
APP
APO
A
AOP
40o
(2)
where APP = AICISIPIASSIII and APO ALS:SIIIIPHSSSSIPPHII
The same decomposition applies to A. Since the MSG contains only the place-place edges and the place-object edges, A is left blank, Meanwhile, AP is symmetric to AOP, So our evaluation will focus on APP and A APO.
3.2 Evaluation metric
Given that the two adjacency matrices A and Å are binary, we evaluate their intersection over union (IoU) to measure how much the two graphs align. As aforementioned, an adjacency matrix A essentially consists of two parts: the place-place part A²" and the place-object part A So we evaluate them respectively as PP loU and PO IoU and combine them to get the whole graph IoU, We provide a precise mathematical definition of the IoU calculation for any two binary adjacency matrices in Appendix B. and we denote this function by IoU(.:-) the following for simplicity.
PP IoU For the PP IoU, the calculation is relatively straightforward since the number of images is deterministic and the one-to-one correspondence between the groundtruth AP and the prediction APP is fixed. As a result, the PP loU is simply:
PP loU = IoP AAP)).
Additionally, we also report the Recall@ score alongside PP IoU since it is the standard evaluation metric for visual place recognition.
PO IoU However, it less straightforward for the PO IoU. The number of objects in the predicted set Ô may differ from O, and their correspondence cannot be determined directly from the adjacency matrix. For a fair evaluation, we need to align Ô with as much as possible. In other words, before computing IoU, we need to find the best matching object for each groundtruth object. This truth-to-result matching is also an important issue in multi-object tracking [53]. To do so, we also record the object bounding boxes in each image and calculate the generalized IoU score (GIoU) of the bounding boxes following
[52]. Then we compute one-to-one matching between O and Ô based on the accumulated GloU score across all the images. Details of the score computation are included in the Appendix B.2. According to the matching, we can reorder Ô to best align with the objects in O. This can be mathematically expressed as permutation matrix S E IR []] x|O] to permute the columns of A. Formally, the PO IoU expressed as the following:
PO loU IoU AA S).
(4)

---

Image tokens Exxx
Transformer
Image encoder
decoder
reshupe
Gurry
Unposed images
Bounding boxes
Place head
Objjett head
Detection results
Object
Ayptace
detector
Parametess
frozen
Whole image
Pammeters
Attention Association
Fealure map
Object feature
Rounding embedding
bounding box
leamable
MSG Model
The AoMSG model. Places and objects queries are obtained by cropping the image feature map using corresponding bounding boxes. The queries are then fed into the Transformer decoder to obtain the final places and objects embeddings. Bounding boxes are in different colors for clarity, The parameters in the Transformer decoder and the linear projector heads are trained with supervised contrastive learning. Image encoder and object detector are pretrained and frozen.
Figure 2:
Our Baseline: Attention Association MSG Generation
When developing new model for the MSG generation task, we adhere to two core principles: Firstly, the model should capitalize on the strengths of pretrained vision models. These pretrained models offer robust initialization for subsequent vision tasks, as their output features encapsulate rich semantic information, forming solid foundation for tasks like ours. Secondly, both place recognition and object association fundamentally address the problem of visual correspondence and can mutually reinforce each other through contextual information. Thus, our model is designed to integrate both tasks within a unified framework. With these guiding principles, we propose the Attention Association MSG (AoMSG) model, depicted in Figure 2.
Place and object encodings Given a batch of unposed RGB images as input, the AoMSG model first employs pretrained vision encoders and detectors to derive image tokens and object detection bounding boxes from each image. We utilize the Vision Transformer baaedd pretrained model DINOv2 [49] our encoder, though our design adaptable any Transformer- based CNN-based encoder that produces a sequence of tokens reshape the output token sequences into a feature map, which is then aligned to the object bounding boxes, aggregating an encoding feature for each detected object. To integrate place recognition and object association within a unified framework, we obtain the place encoding feature by treating it as large object with bounding box that encompasses the entire image, aggregating features as if detected object. The obtained place feature is then positioned alongside the object features, serving as queries for the Transformer decoder, as detailed in the subsequent sections.
feature map. In the case of the DINOv2 encoder, we
AoMSG decoder We follow DETR-like structure [13] to design our AoMSG decoder. Specifically, the derived place feature and object features are stacked sequence queries for the Transformer decoder, while the preceding image tokens are used keys and values. As shown in Figure 2, we enhance the queries by incorporating positional encodings by normalizing and embedding the bounding box coordinates. For instance, for the place feature, the equivalent bounding box is the entire image as aforementioned, resulting in the normalized coordinates of [0, O. 1]. These coordinates are projected to match the dimensionality of the encoding and added elementwise to the place query. The outputs of the AoMSG Transformer decoder are the place and object embeddings that have aggregated context information from the image tokens. Then two linear projector heads are applied to each object and place embeddings respectively to obtain the final object and place embeddings, projecting them into the representation space for the task.
Losses and predictions For training, we compute supervised contrastive learning [51] respectively on the place and object embeddings from the same training batch multitasking fashion. For the object loss, simply use binary cross-entropy with higher positive weights. For the place loss,
5

---

Table Main results. Our method uses DINOv2[49] the backbone. GDino stands for the detector GroundingDINO(390 AoMSG-2 and AoMSG-4 represent AoMSG models with 2 and 4 layers of Transformer decoder respectively. The best results are underlined. * indicates trivial result since its input is given in temporal order, and consecutive frames are trivially recalled.
Method
Metric
PO IoU
Place
Object
Recall@1
PP IoU
w/ GT detection
w/ GDino [39]
Anyloo [36]
97.1
34.2
NetVlad [4]
96.6
35.5
Mickey [7]
100*
33.1
SALAD [34]
97.1
35.6
UniTrack [64]
17.4
13.0
DEVA [20]
16.2
16.6
SepMSG Direct
96.0
31.4
50.4
24.5
SepMSG Linear
96.9
34.9
59.3
24.6
SepMSG MLP
94.3
29.2
56.9
23.4
AoMSG-2
97.2
40.7
69.1
28.1
AoMSG-4
98.3
42.2
74.2
28.1
Table 2: Comparison of different projector dimensions in AoMSG and SepMSG models. Both are using DINOv2-base[49] the backbone. Results are evaluated at 30 epochs.
AoMSG-4
SepMSG-Linear
Projector dimension
Recall
PP IoU
PO IoU
Recall@1
PP IoU
PO IoU
512
97.7
41.3
72.9
93.2
20.3
59.2
1024
98.3
42.2
74.2
96.9
34.9
59.3
2048
97.9
41.8
72.4
96.5
35.0
58.9
the mean square error is minimized for their cosine distances, which gives better empirical results. During inference, simply compute the cosine similarity among the place embeddings and apply threshold to obtain the place-place predictions in Å. For the objects, track their appearances and maintain a memory bank of the existing objects for each scene, updating their embeddings or registering new objects based cosine similarity and thresholding. The results consequently converted the place-object part in Á. Notably, there could be many possible choices to compute the contrastive losses and determine the predictions, we keep our choices simple as we empirically find the standard losses and the simple cosine thresholding can already produce decent results while keeping the embedding spaces straightforwardly meaningful. We discuss the results in detail in Section 5.
5 Experiment
5.1
Data
The MSG models can be trained with any dataset that provides camera poses and object instance labels. We utilized the publicly available 3D indoor scene dataset ARKitScenes [8] construct our dataset. ARKitScenes contains point clouds and 3D object bounding boxes of the scenes, well as the calibrated camera poses obtained from an iPad Pro. We transform the point clouds the 3D bounding boxes with respect to the camera poses to obtain the 2D bounding boxes in each frame. The resolution of each frame 192 x 256. 4492 scenes are used for training and 200 scenes are used for testing. Note that none of the two scenes share the same objects. We leverage the camera poses to obtain the place annotations. Translation threshold and rotation threshold are set meter and radian respectively, images taken within both thresholds considered as capturing the same place.

---

75
PO loU direct
63
PO JaU AoMSG
PP ToU direct
60
1oU
50
45
40
30
25
ConvNeXt
ViT
DINOY2-small
DINOv
DINOv2-tare
Figure 3: Performance of different encoder back- bones. We report results from the base models for both ConvNext [40] and ViT [21].
Figure 4: Visualization of the same objects and the same places. Objects are annotated with their predicted IDs.
5.2 Baselines
VPR We adopt protocols in the previous VPR benchmark literature as outlined in [10, 36]. In our off-the-shelf baselines, we evaluated VPR using DINOv2 [49] either as the global descriptor or followed by a VLAD dictionary generated from large-scale indoor dataset following [36], as feature extraction backbone [34]. For the trained baseline, we conduct our experiments mainly with ResNet-50 50 [28] + NetVLAD used in [10]. Additionally, we also test recent pose estimation baseline [7] and use the poses to estimate the places according to the same thresholds as in the dataset.
Object association We adopt two popular baselines for object association, Unitrack [64] from multi-object tracking, and DEVA [20] from video object segmentation. The image sets are processed in temporal order just like tracking. Unitrack can take any detection backbones and associate object bounding boxes by comparing their features with an online updating memory bank. For a fair comparison, extend its memory buffer length to cover the whole set of images for every scene. DEVA leverages the Segment Anything model [38] to segment and track any object throughout a video without additional training. Their tracking results can be easily converted for evaluating object association based on the tracker IDs.
SepMSG
We also evaluate the pretrained vision models by first separately encoding images and object detections to features and directly evaluating MSG based on those features. This baseline is referred to as
-Direct, where Sep means separately handling places and objects. Then as common way of evaluating pretrained models [29, 30], we conduct probing [2] by further training linear or MLP classifier on those frozen features. These baselines are referred to as SepMSG-Linear and MepMSG-M The SepMSG baselines serve as ablation to validate our model against simply using features learned from the pretrained backbones.
5.3 Experimental setups
For AoMSG, experimented with different choices of backbones, sizes of the Transformer decoder, and dimensions the final linear projector heads. Their results are discussed in Section 5.4, All the models are trained single H100 or GTX 3090 graphics card for 30 epochs until convergence. We provide detailed hyperparameters in the appendix. During training, we randomly shuffle the scenes and mix data from multiple scenes in single batch that the model diversified negative samples at every epoch. Additionally, we monitor the total coding rate as in [60] to avoid the embeddings from collapsing.
To keep the evaluation focused on the quality the graph rather than the quality of object detection, choose not to train the detector together with the MSG objectives. Instead, we use the groundtruth
7

---

detection bounding boxes and popular open-vocabulary object detector GroundingDINO [39]. Results on both configurations are listed in Table 1 and discussed in the following.
5.4 Results
Main results Table shows comparison of our results and baselines. We find that for the place Recall @1 I and PP IoU, the baselines have competitive performance. While the results from the SepMSG baselines are comparable, AoMSG outperforms them all and produces the best results in both metrics. We also notice that all the models produce high Recall @ I, but their PP IoU scores are varied and less than 50. This suggests that having high recall is not enough to guarantee a good graph. For PO IoU, AoMSG models outperform all the baselines by big margins. Both Unitrack and DEVA perform poorly as they struggle when objects reappear after large viewpoint changes or long periods of time. We note that all the MSG methods produce relatively worse results when using GroundingDINO as the detector rather than the ground truth detection. This indicates the performance gap caused by inaccurate object detection. Nevertheless, their performances are still consistent and AoMSG still performs the best. This suggests a better detector will likely give better results for the MSG task. To conclude, AoMSG gives the best performance for all the metrics.
Projector dimensions As listed in Table we compared the impact of different projector dimen- sions as it is reportedly important to performance in the literature of self-superviser representation learning [6. 12, 18]. We find the empirical results are comparable in our experiments.
Choices of backbones Figure shows the performance of difference choices of pretrained back- bones. We experimented with state-of-the-art CNN-based model ConvNext [40], Vision Transformer (ViT) [21] and DINOv2 [49]. We find DINOv2 performs the best, consistent with the observations made in [22]. We use DINOv2 as our default encoder. Interestingly, performance saturates with the size of DINOv2. We suspect it will still increase we could further scale the size the data.
Qualitative In Figure we visualized the learned object embeddings on 6 scenes by AoMSG, the SepMSG-Linear baseline, and SepMSG-Direct that directly uses the output features from the pretrained DINOv2 encoder for the task. The visualization aims to qualitatively assess the learned object embeddings to how separated different objects are in the embedding space. We can see the pretrained embeddings already provide some decent separations. SepMSG-Linear only tunes a linear probing classifier on top so the separation is slightly improved. For example, see the first and second scenes to the left. Compared with them, AoMSG gives the most significant separations, with appearances of the same objects pushed closely and different objects pulled far away. Additionally, Figure visualizes results on some places and objects, and we provide more in Appendix D.
Diseussion
6.1
Application
Given the recent advances in novel view synthesis, 3D reconstruction, and metric mappings, one might wonder whether the proposed MSG is still useful. Here we provide some justifications and showcase application. Echoing literature in 3D scene graphs [32, 66], we believe the MSG can be versatile mental model for embodied Al agents and robots. At global level, keeps a lightweight topological memory of the scene from purely 2D RGB inputs, which serves as basis for robot navigation [15, 37]. At a finer level, it can scamlessly couple MSG with the 3D reconstruction methods, to estimate depth and poses and build local reconstructions. Therefore, robot traverse the environment, localize itself referring to the MSG, and build local reconstruction when needed for tasks that require metric information such as the manipulation tasks.
As a showcase application, we provide two local 3D reconstruction cases illustrated in Figure using the most recent to a dense image set greatly consumes GPU memory, which may be infeasible for mobile robots. Whereas a random subsample does not guarantee the reconstruction quality. Instead, with MSG, we can provide the Dust3r with locally interconnected subgraphs for fast and reliable local reconstruction. The subgraphs and local reconstructions can be object-centric thanks to the pojcetobject nature of
3D reconstruction model Dust3r [63]. Directly applying Dustår
8

---

Direet
Linear
AoMSG
Figure 5: Object embedding visualization using t-SNE [61]. SepMSG-Direct, SepMSG--inear, and AoMSG-2 are shown in each row respectively, Results from the scene are aligned vertically. Colors indicate different objects. Each point is an appearance of an object. It is best viewed in color.
Reconsmution
Topological
Mesh
Scone
Scene
Figure 6: Local 3D reconstruction from 2D MSG using off-the-shelf model Dust3r [63]. The 3D meshes of scenes are shown side by side, with 3 subgraphs circled in gray and reconstructed the top of each scene.
MSG. Moreover, the local reconstructions are topologically connected by MSG. This suggests MSG can provide a flexible scene representation balancing 2D and 3D, abstractions and details.
6.2 Limitation
The current work still has many limitations. Firstly, we only conducted experiments one dataset. Although the dataset contains around 5k scenes, which is sufficient to obtain convincing results, would still be great to see if training on more diversified data collections can produce better models and stronger generalization work. Secondly, scenes in the current dataset contain only static objects, extending to dynamic objects direction worth exploring.
observed in [63], especially for larger models. We leave this to future
Additionally, given the scope of the work propose MSG as new vision task promoting spatial intelligence, focus explicitly evaluating the quality of the graph. Therefore, did not

---

investigate the object detection quality, nor did we deploy the MSG to downstream tasks such as navigation. Note that detection quality does affect the MSG performance though we find it to be consistent across different detection modes, i.e. the groundtruth and the GroundingDINO. Training detectors together with the MSG model and applying MSG to downstream tasks will be our next step to make the work more complete system.
7 Conclusion
This work proposes building the Multiview Scene Graph (MSG) as new vision task for evaluating spatial intelligence. The task gives unposed RGB images as input and requires model to build place+object graph that connects images taken at the same place and associates the object recogni- tions from different viewpoints, forming a topological scene representation. To evaluate the MSG generation task, we designed evaluation metrics, curated dataset, and proposed a new model that jointly learns place and object embeddings and builds the graph based on embedding distances. The model outperforms existing baselines that handle place recognition and object association separately. Lastly, we discussed the possible applications of MSG and the current limitations. We hope this work can stimulate future research on advancing spatial intelligence and scene representations.
Acknowledgement. sions and suggestions.
The authors thank Yiming Li and Shengbang Tong for their valuable discus-
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 17
[2] Guillaume Alain and Yoshua Bengio. Understanding intermediate layers using linear classifier probes. arXiv preprini 160iv: 644, 2016.
[3] Amar Ali-Bey, Brahim Chaib-Draa, and Philippe Giguere. Mixvpr: Feature mixing for visual place recognition. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2998-3009 2023. 2
[4] Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivie. Netvlad: Con architecture for weakly supervised place recognition. In Proceedings the IEFE conference on computer vision and pattern recognition, pp. 5297-5307, 2016. 2, 3,6
[5] Iro Armeni, Zhi- Maii He, JunYoung Gwak, Amir Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5664-5673,
2019. 2
[6] Adrien Bardes, Jean Ponce, and Yann LeCun. Viereg: Varianc--ivaranc--vvaiinc regular- ization for self-supervised learning. arXiv preprint arXiv:2105.04900 2021. 8
[7] Axel Barroso- Barross-Laguna, Sowmya Munukutla, Victor Prisacariu, and Eric Brachmann. Matching 2d images in 3d: Metric relative pose from metric correspondences. In CVPR, 2024. 6,7
[8] Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman. ARKitscenes diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data. In Thirty-fftth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021, URL https: /ttpe/////peeeeeeeeeeeeeeeeeeeee net/ fforummdd=tjjjv _aC_EC___g_____________eee qh_CE. 2, 6. 27
[9] Herbert Bay, Tinne Tuytelaars, and Lue Van Gool. Surf: Speeded up robust features. In Aleš Leonardis, Horst Bischof, and Axel Pinz (eds.), Computer Vision - ECCV 2006, pp. 404417, Berlin, Heidelberg. 2006. Springer Berlin Heidelberg. ISBN 978-3-540-33833-8. 3
[10] Gabriele Berton, Riccardo Mereu, Gabriele Trivigno, Carlo Masone, Gabriela Csurka, Torsten Sattler, and Barbara Caputo. Deep visual geo-localization benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5396-5407, 2022. 7
10

---

[11] Fabian Blochliger, Marius Fehr, Marcin Dymezyk. Thomas Schneider, and Rol Siegwart. Topomap: Topological mapping and navigation based on visual slam maps. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 3818-3825. IEEE, 2018. 3
[12] Florian Bordes, Randall Balestriero, and Pascal Vincent. Towards democratizing joint- embedding self-supervised learning. arXiv preprint arXiv:2303.01990 2023. 8
[13] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 229.229. Springer, 2020. 5
[14] Mathilde Caron, Hugo Touvron, Ishan Mísra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. Proceedings of the IEEE/CVF international conference computer vision, pp. 9650-9660, 2021. 3
[15] Devendra Singh Chaplot, Ruslan Salakhutdinov, Abhinav Gupta, and Saurabh Gupta. Neu- ral topological slam for visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12875-12884, 2020. 3, 8
[16] Chao Chen, Xinhao Liu, Xuchu Xu, Yiming Li, Li Ding, Ruoyu Wang, and Chen Feng. Self- supervised visual place recognition by mining temporal and feature neighborhoods, arXiv preprint arXiv:2208.00933 2022.
[17] Chao Chen, Xinhao Liu, Yiming Li, Li Ding, and Chen Feng. Deepmapping2: Self-supervised large-scale lidar map optimization. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 9306-9316, 2023. 3
[18] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
[19] Ho Kei Cheng and Alexander Schwing. XMem: Long-term video object segmentation with an atkinson-shiffrin memory model. In ECCV, 2022.
[20] Ho Kei Cheng, Scoung Wug Oh, Brian Price, Alexander Schwing, and Joon- Youn-Yung Lee. Tracking anything with decoupled video segmentation. In ICCV, 2023. 2, 6,
[21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020. 3, 7,8
[22] Mohamed EI Banani, Amit Raj, Kekis-Kokitsi Maninis, Abhishek Kar, Yuanzhen Li, Michael Rubinstein, Deqing Sun, Leonidas Guibas, Justin Johnson, and Varun Jampani. Probing the 3D Awareness of Visual Foundation Models. In CVPR, 2024. 3,8 8
[23] Cathrin Elich, Iro Armeni, Martin Oswald, Mare Pollefeys, and Joerg Stueckler. Learning- based relational object matching across views. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 5999-60055. IEEE, 2023. 2,3
[24] Samir Yitzhak Gadre, Kiana Ehsani, Shuran Song, and Roozbeh Mottaghi. Continuous scene representations for embodied ai. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
[25] Sourav Garg, Krishan Rana, Mehdi Hosseinzadeh, Lachlan Mares, Niko Sünderhauf, Feras Dayoub, and Ian Reid. Robohop: Segment-based topological map representation for open-world visual navigation. arXiv preprint arXiv:2405.08 057922 2024. 3
[26] Katalin M Gothard, William E Skaggs, Kevin M Moore, and Bruce L McNaughton. Binding of hippocampal cal neural activity to multiple reference frames in a landmark- -bsed navigation task. The Journal of neuroscience, 16(2):823, 1996.
[27] Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cogni- tive mapping and planning for visual navigation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2616-2625, 2017.

---

[28] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.7 7
[29] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 9729 97738, 2020. 7
[30] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 16000-16009 2022. 3,7
[31] Lingyi Hong, Wenchao Chen, Zhongying Liu, Wei Zhang, Pinxue Guo, Zhaoyu Chen, and Wenqiang Zhang. Lvos: A benchmark for long-term video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13480-13492, 2023. 3
[32] N. Hughes, Y. Chang, and L.. Carlone. Hydra: A real-time spatial perception system for 3D> scene graph construction and optimization. 2022.
[33] Nathan Hughes, Yun Chang, Siyi Hu, Rajat Talak, Rumaisa Abdulhai, Jared Strader, and Luca Carlone. Foundations of spatial perception for robotics: Hierarchical representations and real-time systems. 2023. 2
[34] Sergio Izquierdo and Javier Civera. Optimal transport aggregation for visual place recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024. 3, 6,
[35] Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3668-3678 2015. 2
[36] Nikhil Keetha, Avneesh Mishra, Jay Karhade, Krishna Murthy Jatavallabhula, Sebastian Scherer, Madhava Krishna, and Sourav Garg. Anyloc: Towards universal visual place recognition. JEEE Robotics and Automation Letters, 2023. 2, 3, 6,
[37] Nuri Kim, Obin Kwon, Hwiyeon Yoo, Yunho Choi, Jeongho Park, and Songhwai Oh. Topologi- cal semantic graph memory for image-goal navigation. In Conference on Robot Learning, pp. 393-402. PMLR, 2023. 3,
[38] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026,
2023. 3,7
[39] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint vrXiv22303.05990 2023. 6,
[40] Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer. Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19976-19886, 2022. 7,8
[41] David G Lowe. Object recognition from local scale-invariant features. In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pp. 1150-1157. leee, 1999.
[42] Stephanie Lowry, Niko Sünderhauf, Paul Newman, John J Leonard, David Cox, Peter Corke, and Michael Milford, Visual place recognition: A survey. ieee transactions on robotics, 32(1):
1-19, 2015. 16
[43] Zonglin Lyu, Juexiao Zhang, Mingxuan Lu, Yiming Li, and Chen Feng. Tell me where you are: Multimodal Ilms meet place recognition. arXiv preprint 175200,00..7 2024. 17
12

---

[44] Federico Magliani, Tomaso Fontanini, and Andrea Prati. A dense-depth representation for vlad descriptors in content-based image retrieval. In Advances in Visual Computing: 13th International Symposium, ISVC 2018, Las Vegas, NV, USA, November 19-21, 2018, Proceedings 13, pp. 662-671. Springer, 2018.
[45] Robert Muller and John Kubie. The effects of changes in the environment on the spatial firing of hippocampal complex-spike cells. Journal of Neuroscience, 7(7):1951-1968 1987.
[46] Raul Mur-Artal, Jose Maria Martinez. Montiel, and Juan D Tardos. Orb-slam: versatile and accurate monocular slam system. IEEE transactions on robotics, 31(5): 147--1163 2015. 3
[47] Kien Nguyen, Subarna Tripathi, Bang Du, Tanaya Guha, and Truong Q Nguyen. In defense of scene graphs for image captioning. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1407-1416, 2021. 2
[48] John O' O'keefe and Lynn Nadel. The hippocampus as a cognitive map. Oxford university press, 1978.
[49] Maxime Oquab, Timothée Dareet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 5, 6. 7, 8
[50] Zheng Qin, Sanping Zhou, Le Wang, Jinghai Duan, Gang Hua, and Wei Tang. Motiontrack: Learning robust short-term and long-term motions for multi-object tracking. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 17939-17948, 2023. 3
[51] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-878778 8763. PMLR, 2021. 5
[52] Hamid Rezatofighi, Nathan Tsoi, Jun Youno Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: metric and loss for bounding box regression. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 658-666, 2019, 4, 16
[53] Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi. Performance measures and a data set for multi-target, multi-camera tracking. In European conference on computer vision, pp. 17-35. Springer, 2016. 4
[54] Renato F. Salas-Moreno, Richard A. Newcombe, Hauke Strasdat, Paul H.J. Kelly, and Andrew J. Davison. Slam++: Simultaneous localisation and mapping at the level of objects. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2013. 3
[55] Paul-Edouard Sarlin, Daniel DeTone, Tomasz Malisiewicz, and Andrew Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4938-4947, 2020. 3
[56] Nikolay Savinov, Alexey Dosovitskiy, and Vladlen Koltun. Semi-parametric topological memory for navigation. arXiv preprini arXiv: 00653, 2018.
[57] Tixiao Shan and Brendan Englot. Lego-loam: Lightweight and ground-optimized lidar odometry and mapping on variable terrain. In 2018 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1758-4766 IEEE, 2018. 3
[58] Richard Sinkhorn and Paul Knopp. Concerning nonnegative matrices and doubly stochastic matrices. Pacific Journal of Mathematics, 21(2):343-325 348, 1967. 3
[59] Zachary Teed and Jia Deng. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances neural information processing systems,
2021. 3
[60] Shengbang Tong, Yubei Chen, Ma, and Yann Lecun. Emp-ssl: Towards self-supervised learning in training epoch. arXiv preprint 273..72304.03977. 2023,
13

---

[61] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[62] Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Chuanxin Tang, Xiyang Dai, Yucheng Zhao, Yujia Xie, Lu Yuan, and Yu-Gang Jiang. Look before you match: Instance understanding matters in video object segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2268-226222222 2278,, 2023. 3
[63] Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r:
Geometric 3d vision made easy. In CVPR, 2024. 8, 9
[64] Zhongdao Wang. Hengshuang Zhao, Ya-Li Li, Shengjin Wang, Philip Torr, and Luca Bertinetto. Do different tracking tasks require different appearance models? Advances in neural information processing systems, 34:726-736 2021. 2, 3, 7,
[65] Dongming Wu, Wencheng Han, Tiancai Wang, Xingping Dong, Xiangyu Zhang, and Jianbing Shen. Referring multi-object tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14633-14642, 2023. 3
[66] Shun-Cheng Wu, Johanna Wald, Keisuke Tateno, Nassir Navab, and Federico Tombari. Scene- graphfusion: Incremental 3d scene graph prediction from rgb-d sequences. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7515-7525, 2021.
2,8 8
[67] Chun-Cheng Wu, Keisuke Tateno, Nassir Navab, and Federico Tombari. Incremental 3d semantic scene graph prediction from rgb sequences. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5064-5074, 2023. 3
[68] Yanmin Wu, Yunzhou Zhang, Delong Zhu, Zhiqiang Deng, Wenkai Sun, Xin Chen, and Jian Zhang. An object slam framework for association, mapping, and high-level tasks. IFEE Transactions on Robotics, 2023. 3
[69] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5410-5419, 2017. 2
[70] Mubariz Zaffar, Sourav Garg, Michael Milford, Julian Kooij, David Flynn, Klaus McDonald- Maier, and Shoaib Ehsan. Vpr-bench: An open-source visual place recognition evaluation framework with quantifiable viewpoint and appearance change. International Journal of Computer Vision. 129(7):213--2177 2021. 16
[71] Chaoyi Zhang, Xitong Yang. Ji Hou, Kris Kitani, Weidong Cai, and Fu-Jen Chu. Egosg: Learning 3d scene graphs from egocentric rgb-d sequences. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2535-2545, 2024.
[72] Yuang Zhang, Tiancai Wang, and Xiangyu Zhang. Motrv2: Bootstrapping end-to-end multi- object tracking by pretrained object detectors. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattrrn Recognition, pp. 22056-2002 -2065,, 2023. 3
14

---

A Hyperparameter setting
Hyperparameter
Value/Range
Original image size
192 x 256
Input image size
224 x 224
Batch size
384
Scenes per training batch
6
Images per scene per batch
64
Learning rate
2e-5
Epochs
30
Optimizer
Adam W
Scheduler
None 0.01
Weight decay
AoMSG layers
2, 4
AoMSG patch size
14
AoMSG hidden dim
384
Projector head dim
512. 1024, 2048 MSE cosine
L ppiae Place Loss Function
Lobject Object Loss Function
BCE w/ positive weight=10
Loss ratio Lplace Lobject
1: 1
Place Similarity Threshold
0.3
Object Similarity Threshold
0.2
Table 3: Hyperparameters used in the AoMSG main experiments.
B Details of evaluation metrics
B.1 IoU between two adjacency matrices
Given two binary adjacency matrices {0, m)mAx and B (0, 1] Suppose the vertices in their corresponding graphs have been compared and best-matched, we can directly compute the IoU as the following:
m = m.n mB)
(5)
n² min n nB)
(6)
WA Σ
Aij
(7)
m <ismam <iSnA
WB Σ
Bij
(8)
<iS mssmann <jSng
Eisigm Aij A Bij
IAU Sisismussis Ai Bij + WA + wB
99)
The inclusion of and wy in the denominator implies that the IoU value decreases when the two graphs contain additional but not isolated vertices.
From the graph perspective, this loU evaluates binary edge prediction on augmented graphs. Specif- ically, given a groundtruth graph and predicted graph, after matching their vertices discussed in See 3.2, we can augment both graphs with unmatched vertices from the other. We name eq(9) as IoU because under this construction, the defined IoU aligns with the following definition in binary classification based on the standard True Positives (TP), False Negatives (FN), and False Positives (FP):
TP
IoU
(10)
FP+ FN Here, a "positive" prediction indicates the presence of an edge.
15

---

B.2
Object truth-to-result matching
Here we elaborate on the computation of the object matching score mentioned in Sec. 3.2.
Given any groundtruth object vyy E O and any predicted object Vr Ô, record and compare their detections across all the 7 frames. Denote D(y, 1) and [T,, t) as the groundtruth and predicted object detections in frame for y and respectively. Use the indicator functions I[:, to signal the existence of an object in frame &: if exists in 6, then [[(T, t) = 1, otherwise [l(T, t) = 0.
Therefore, the accumulated GloU of and 7 is:
C2yy7
Σ I( II(TT)) * GloU ((T,)), EET
(11)
which is the sum of the generalized bounding box loU [52] across all the frames that both objects are present. To obtain the final matching score between and Cy,r is further normalized by the sum of the following four terms:
The number of the matched frames, where both 7 and exist and their GloU is positive.
The number of the unmatched frames, where both y and T exist but their GloU is zero. The number of the "false positive" frames, where only T exists and doesn's 't.
The number of the "false negative" frames, where only y exists and doesn't. t.
The sum of these four terms is in fact equivalent to computing the union of the appearances of and across all the T frames:
+ I((,,t) - > I I(T,t).
(12)
KET
SET
TET
Consequently, the matching score between y and is computed as:
CT.T
Myy
(13)
In practice, we take my as the cost used in solving the one-to-one assignment problem via Hungarian Matching.
C Additional Analysis
C.1 Learned relative pose distributions
We set thresholds the dataset hyperparameters which is conventional setup in visual place recognition (VPR) tasks and datasets [42, 70]. Since the MSG task involves place recognition, we choose to adopt this convention. VPR tasks require model to classify whether not two images taken from the same place. The concept of "place" a discretization of the space that continuous by nature, necessitating the of thresholds in the VPR setup.
To give closer look the effect the threshold has on the model, in Figure 7 we report the relative pose distributions (orientation and translation) for the connected and non-connect nodes based on our model's prediction. The figures show that instead of collapsing to only represent the fixed thresholds, the pose distributions have clear yet smooth separation across the spatial thresholds.
C.2 Failure cases
We visualize some failure cases for place recognition in Figure 8. We observe that most failure cases can be attributed to either having very similar visual features with relatively large pose differences (false positives), such as observing room from two opposite sides, having few similarities in visual features with relatively smaller pose differences (false negatives). We note that the recall metric, conventional VPR, is straightforward and effective for image retrieval against database. However, it falls short in reflecting challenging false positives and negatives, especially when constructing topological graph like the MSG where the number of positives varies. This highlights the usefulness of our proposed IoU metric, which consistently evaluates the quality of the graph.
16

---

(a) Orientation difference.
(b) Translation difference.
Figure 7: Relative pose distribution in histograms the test set. Blue is for the connected and red for the not connected. The green dashed lines are the spatial thresholds.
Figure 8: Failure cases for place recognition. The top 2 rows are false positive and the bottom rows are false negative. Listed pairs.
C.3 Approaching MSG Generation with Multimodal Large Language Model
Multimodal Large Language Models (MLLMs) have exhibited strong emergent abilities for many tasks, and we are intrigued to try them out for the MSG generation task. However, querying MILMs with every image pair in an image set is huge amount of work and cost, and sending all images together poses a challenge to the context length limit while also hurting performance. Therefore, we conducted case study with
a pilot study.
Specifically, we sampled a scene with a relatively small number of images and further subsampled all the images containing annotated objects, resulting in 22 images in total. We then queried the GPT-4o [1] 231 times with each image pair annotated with object bounding boxes as visual prompts, the corresponding box coordinates, and the task prompt. By parsing the GPT-4o outputs, we obtained the results Table 4.
In Table 4, the model total represents the performance of our model the entire scene, and the model adjusted represents performance evaluated only on those 22 subsampled images for fair comparison. Besides the issues with computation cost and context limits, we note that common failure pattern of VLM the failure to maintain consistent object associations. The limitations of VLM in VPR are also discussed in the literature [43].
17

---

Table Pilot study for MLLM on MSG. For the MLLM, we use GPT4o. The "model adjusted" is evaluated on the same set of images as the VLM.
Metric
model total
model adjusted
VLM
PP IoU
59.3
63.0
30.3
PO loU
85.0
85.0
62.5
Nevertheless, this is only small-scale pilot study. It is well possible to have better VLMs in the future and come up with better prompts, and we are excited about the future possibilities of MLLM + MSG.
C.4 A qualitative real-world experiment
To examine how our method generalizes real-world environments, we conducted a qualitative experiment. Specifically, We have self-recorded an unposed video with an iPhone in household scenario and run our trained AoMSG model with a pretrained Grounding DINO detector on
In Figure 9, we show some resulting images with object instance ID labeled and a visualization the generated graph. Results suggest that model is able obtain sensible outputs arbitrary videos outside of the dataset.
18

---

1550f
pb 1-sofo
p8-1-s0fo
p3-1-1 1-tv--ttootooo
iv_moontorr
p5- 1- -obbinet
Current Node
Figure 9: Qualitative real-world experiment. Top: results visualization. "px" is the object instance label. Bottom: screenshot of the interactive graph visualization. Place nodes are in blue and object nodes are in orange.
D More visualizations
19

---

Figure 10: Visualization for the place nodes. Every 3 images shown side by side are those connected the MSG, meaning they are considered from the same place.
20

---

K2- roeeod
6TH Bo-
p4 O Orotee s-T-opber
Figure 11: Visualization for the object nodes. The same objects recognized across different views are grouped as one object node. Each object visualized with a colored bounding box with the annotation format: predicted ID groundtruth ID groundtruth category. For this visualization. we use the groundtruth detection bounding box in each frame as explained in the main paper. Note that images in some scenes are taken sideways, in the visualization choose to keep is.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The newly proposed MSG task rigorously defined in Section 3.1. We thor- oughly describe the curation of our dataset and lahel used for the experiment in Section 5.1. The evaluation metric for the MSG task is introduced in Section 3.2. The proposed pipeline is explained in Section 4 and experiment results reported in Section 5.4.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitation of the work is discussed in detail in Section 6.2
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
22

---

Answer: [NA]
Justification: This work focuses on deep learning tasks and architectures. Theoretical results are not in the scope of this work, but we provide detailed empirical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We fully disclose the information needed to reproduce our experiment results. We discuss the experimental details in Section 5, We further provide detailed hyperparame- ters in the supplementary. We fully open- open-sourced our data and code after acceptance.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
23

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We will submit the code for experiment together with the paper submission. We will open source our code and data upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
public/ cublic/guides// htte:/uuddde//coee
cemmiisssoiiiyyyoo for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. wwtppbccccc//cccccccccccccccccct gwwpc/gg//////eeeeceeeecccccicee Cttpa///CoSSSuuiieeeeeeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We discuss the experimental details and settings in Section 5. We also provide other more detailed information in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: We did not repeat experiments with multiple seeds since experiments for all comparisons are time-consuming and we did not find evidence of our model being particularly sensitive specific hyperparameters throughout the course of the project. Meanwhile, we fixed all the system seeds to 42 aforchand without seed-tuning.
Guidelines:
The answer NA means that the paper does not include experiments.

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details of the computing resource in Section 5.3.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics sttepe///eerrrp...c//bb//////// (eppublic//hhccuuuideeee.ee ines?
Answer: [Yes]
Justification: The authors have viewed the NeurIPS Code of Ethics and are sure that the research is compliant.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
25

---

Justification: Models trained for the MSG task can have real-world implications, especially their use in mobile agents. We believe is essential to assess potential unintended conse- quences when deploying such models to real-world robots, such as safety risks, algorithmic biases, or the impact on humans during social interaction. However, this beyond the focus of this work since the presented task and the models have not been deployed to the real-world.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (c.g., disinformation, generating fake profiles, surveillance), fairness considerations deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The model trained for the MSG task mainly performs spatial understanding and reasoning via visual perception. There is generative model scraped datasets used in this research. The dataset we use comes from a published dataset on which the original authors have made safeguards.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
26

---

Justification:
We credit the ARKitScenes [8] dataset in the paper. The dataset is released un- 4.0 International Public License.
der the Commons SorrrionnnooCCmmmrallllll-l--i- The license and terms are properly respected in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswithcode thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We consider the code, curated data labels, and trained models for the experi- ments this paper as new assets. These assets are well-documented and open- sourced after paper acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
27

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
28


Reviews:
[{"review_id": "o3w9yoQI9D", "reviewer": "Reviewer_qE49", "summary": "The paper introduces the novel task of Multiview Scene Graph generation from unposed RGB images, whereas this type of scene graph encodes the notion of 'places', i.e., images from spatially close locations, and detected objects as graph nodes. The motivation is to combine spatial reasoning of object association and place recognition into a joint representation. \n\nAn evaluation metric based on the edge adjacency is introduced along an initial baseline method (AoMSG, based on a Transformer architecture and off-the-shelf image encoder and object detector) for this new task, which is compared against sub-task specific methods on the ARKit scenes dataset.", "questions": "- Missing experimental details of Figure 3: How do the numbers provided in the graph relate to the numbers in Table 1, i.e., which model specifically was trained with the specific backbone?", "limitations": "The checklist is complete, including justifications for each item. Limitations of the work are discussed in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper presents and motivates a new task of building a topological scene graph combing place recognition and object association. An example application task (3D Reconstruction using Dust3r on sub-sets of input images retrieved from the MSG) which would benefit from the proposed MSG is described in the discussion section. \nThe proposed adjacency-IoU metric is simple, however the object-object \"alignment\" can play a dominant role.", "weaknesses": "- The current task and proposed baseline model does not seem to allow for an adjustable \"place\" definition beyond the train set constraints (place = 1m and 1 rad). \n- To give further insights about the implication of the proposed metric, it would be useful to provide failure case which results in low IoU scores, especially for PP IoU which is significantly lower than PO IoU.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Q1__: The current task and model do not seem to allow for an adjustable \"place\" definition beyond the train set constraints\n\n__A1__: Thank you for the question. The thresholds are dataset hyperparameters which is a conventional setup in visual place recognition (VPR) tasks and datasets [1, 2]. Since our MSG task involves place recognition, we choose to adopt this convention. VPR tasks require a model to classify whether two images are taken from the same place or not. The concept of “place” is a discretization of the space which is continuous by nature, which necessitates the use of thresholds in the VPR setup.\n\nThe proposed baseline model conducts supervised training, so it is possible to set different hyperparameters for different datasets and train the model on multiple datasets. To give a closer look at the effect the threshold has on the model, in the attached PDF file, we also included figures of relative pose distributions (orientation and translation) for the connected and non-connect nodes based on our model’s prediction. The figures show that instead of collapsing to only represent the fixed thresholds, the pose distributions have a clear yet smooth separation across the spatial thresholds. \n\n[1] Lowry, Stephanie, et al. \"Visual place recognition: A survey.\" ieee transactions on robotics 32.1 (2015): 1-19.\n\n[2] Zaffar, Mubariz, et al. \"Vpr-bench: An open-source visual place recognition evaluation framework with quantifiable viewpoint and appearance change.\" International Journal of Computer Vision 129.7 (2021): 2136-2174.\n\n__Q2__: Give further insights about the implication of the proposed metric … provide failure cases that result in low IoU scores.\n\n__A2__: Thank you for the suggestion. We have added visualizations of some failure cases in the attached PDF file. We observe that most failure cases can be attributed to having very similar visual features with relatively large pose differences (false positives), such as observing a room from two opposite sides, or having fewer similar visual features with relatively smaller pose differences (false negatives).\nWe note that the recall metric, conventional in VPR, is straightforward and effective for image retrieval against a database. However, it falls short in reflecting challenging false positives and negatives, especially when constructing a topological graph like MSG where the number of positives varies. This highlights the usefulness of our proposed IoU metric, which consistently evaluates the quality of the graph.\n\n\n\n__Q3__: Missing details in Figure 3.\n\n__A3__: Thank you for pointing it out. We made a mistake with the numbers and y-axis. We have uploaded a new figure with more information in the attached PDF file to replace the original one. The numbers marked as *direct* are obtained by directly evaluating the pretrained encoders corresponding to the SepMSG-direct baseline. Those marked as *AoMSG* are obtained by training the *AoMSG-4* model on top of the frozen encoders, following the same setup as the main experiments in Table 1 in the manuscript. We chose the DINOv2 base as the default encoder for our main experiments as it gives the best performance when evaluated directly. We will revise the manuscript with the updated Figure 3.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rYXrshHpmV", "reviewer": "Reviewer_AiA2", "summary": "The manuscript proposes the problem of inferring a scene graph from unposed images of a space. The key distinguishing factor to previous work is using multiple frames (as opposed to a single frame) and not requiring poses and depth (like for typical metric 3d scene graphs). The scenegraph is defined as a the set of images connected by edges implying topological connectedness and the set of objects. Objects are connected to place nodes based on where they are observed from. The manuscript also proposes a sensible baseline that outperforms more basic baselines for MVSG generation as well as methods that just work on place recognition or just on object tracking.", "questions": "- l 204 whats the \"benchmark VG pipeline\" ? \n- SepMSG: I assume you are also using the same 2d encoder as for AoMSG? Could you clarify.\n- The place localization is trained using relative pose thresholding to obtain positive samples. I am curious if that means the features just learn those spatial thresholds? During inference what is the relative pose distribution of the connected places? And the disconnected ones?", "limitations": "Limitations are addressed adequately in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Associating objects across frames and long stretches of time without observation is a challenging and important task that does not truly require 3D information (but is greatly helped by it). Introducing and studying the relaxed problem of multi-view unposed scenegraph generation has the potential to yield novel solutions that can be deployed on any video stream. And of course methods that utilize 3d information (say from running some deep slam on the video first) might yield the best performance. It puts the different approaches onto the same benchmark which is useful for the community.  \n\nGiven the 3D training data (to supervise associations correctly) the proposed model is conceptually and implementation wise simple and leverages state of the art vision backbones. Based on the evaluation this simple model is also effective in generating scene graphs from unposed images. The SepMSG baselines are useful in convincing about the usefulness of the AoMSG model. It is interesting to see that this baseline model outperforms existing tracking methods by a large margin.\n\nOverall the paper is well written and the illustrations are high quality to support the understanding of the method and ideas.", "weaknesses": "The constraint of not requiring poses and theoretically enables the method to be run on any video sequence. However there are no examples of even qualitative experiments on some arbitrary video sequences. I think this should be easy to add and would help support the idea that unposed-image posed scene graphs are useful. The other aspect that is not addressed in the manuscript is how to train the proposed system without access to poses and 3d surfaces  or 3d bounding boxes of objects in order to supervise the model. It is unclear if the place localizer just learns to replicate the pose thresholds used during training (see question).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Q1__: Example of qualitative experiments on some arbitrary video sequences.\n\n__A1__: Thank you for the great suggestion! We have self-recorded an unposed video with an iPhone in a household scenario and run our trained AoMSG model with a pretrained Grounding DINO detector on it. In the attached PDF file we show some resulting images with object instance id labeled. Results suggest that our model is able to obtain sensible outputs on arbitrary videos outside of the dataset. We have also made a simple frontend tool for interactive visualization with MSG, which we show a screenshot in the PDF. We will release the tool along with the full source code when camera-ready.\n\n__Q2__: How to train the model without poses or other 3D annotations.\n\n__A2__: Thank you for the question. Like many other new tasks, certain requirements of annotation exist in the current MSG task. We rely on explicit camera poses and 3D object annotations in the ARKitScene to generate our dataset since marking the same places and objects requires knowing camera poses and object instances. We note that any 3D dataset or environment providing such annotations can be leveraged for the MSG task and there are many large-scale 3D datasets available for use such as ScanNet [1], ScanNet++[2], and HM3D[3]. Nevertheless, we acknowledge that this is a limitation and in the future, we will try to explore training on a combined recipe of different datasets for strongly generalizable performance or explore training with less annotation. \n\n[1] Dai, Angela, et al. \"Scannet: Richly-annotated 3d reconstructions of indoor scenes.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.\n\n[2] Yeshwanth, Chandan, et al. \"Scannet++: A high-fidelity dataset of 3d indoor scenes.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[3] Ramakrishnan, Santhosh K., et al. \"Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai.\" arXiv preprint arXiv:2109.08238 (2021).\n\n\n__Q3__:  What is the benchmark VG Pipeline\n\n__A3__: We thank the reviewer and apologize for the confusion. VG stands for Visual Geo-localization[1] which refers to the same task of Visual Place Recognition (VPR). This pipeline is referenced as [8] in the manuscript. We will revise this line in the manuscript for better clarity.\n\n[1] Berton, Gabriele, et al. \"Deep visual geo-localization benchmark.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n__Q4__: Encoder of SepMSG\n\n__A4__: Thank you for the question. To clarify, SepMSG baselines use the same image encoder as the AoMSG methods. We use pretrained encoders and keep them frozen while training all the methods. Specifically, SepMSG-direct refers to directly evaluating the output feature vector of the encoders, SepMSG-linear/MLP trains linear probing or MLP on top of the encoders and AoMSG trains the Transformer-based decoder. We empirically find that the pretrained DINO-base model produces the best performance in direct evaluation, so all the experiments in Table 1 use it as the baseline. Below is a table for the detailed performance of different backbone choices when used in SepMSG-direct. We will revise the manuscript with better clarification and the results.\n\n| Encoder for SepMSG-direct | PP IoU | PO IoU |\n|---------------------------|-------|-------|\n| ResNet 50                 | 26.30 | 45.23 |\n| ConvNext base                 | 27.62 | 46.29 |\n| ViT base                      | 28.86 | 46.04 |\n| DINO small                | 29.54 | 50.37 |\n| DINO base                 | 30.94 | 54.78 |\n| DINO large                | 31.02 | 50.02 |\n\n\n__Q5__: if features just learn those spatial thresholds? Calculate the relative pose distribution\n\n__A5__: Thank you for the great suggestion. Annotating based on spatial thresholds is a conventional setup in visual place recognition (VPR) tasks and we chose to follow this convention.  While training the features, we adopt a metric learning approach instead of simple binary prediction. During training, embeddings positive pairs are drawn closer in terms of cosine similarity, while negative pairs are pushed further. We also monitor the training dynamic using the Total Coding Rate as described in [1] to alert possible feature collapse. Thus, the learned features are not confined to just those spatial thresholds.\n\nWe have also added 4 plots for relative pose distribution (both orientation and translation) in the attached PDF file. They are obtained by taking the model’s predictions on the test set and calculating the histograms of the relative orientation and translation differences of the predicted connected and not-connected places. From the plots, we can see that while the model makes some mistakes (currently the PP IoU is around 0.40), the distribution difference between the connected and the not-connected places is clear. There is a clear yet smooth separation at the spatial thresholds on all the plotted distributions.\n\n[1] Yu, Yaodong, et al. \"Learning diverse and discriminative representations via the principle of maximal coding rate reduction.\" Advances in neural information processing systems 33 (2020): 9422-9434.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1SOR3VfTDM", "reviewer": "Reviewer_9wC7", "summary": "The paper proposes a novel task: generating scene graphs from unposed RGB images as well as a new benchmark based on ARKitscenes. To achieve this, the paper proposes a novel approach that use off-shelf image encoder and detector that are frozen and only train a decoder that takes in the features learned from frozen image encoder and detector to generate scene graphs. The author ablates the pre-trained weights for backbones and show a promising results compared to baselines.", "questions": "1. do you need the unposed images must have overlap with each other?\n2. Have you tried SAM? and do you think DinoV2 performs best is because of the detector's feature is closed to image encoder, when using DINOv2 as image encoder?", "limitations": "it is not that hard to get poses from either sensors or visual cues nowadays.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The author proposes a new benchmark based on ARKitscenes, and a novel method trained on those, showing promising results.\n2. the authors also ablate the influence of different pre-trained backbones.", "weaknesses": "1. The task is not that novel, see EgoSG: Learning 3D Scene Graphs from Egocentric RGB-D Sequences, where a very similar task is propose, as in, generating scene graphs from unposed RGB-D images. And from RGB to RGB-D, a off-shelf depth detector can be leveraged.\n\n2. it is unclear, how was ViT pre-trained, it is using MAE?\n\n3. the authors use a self-made baseline, i am wondering if other baselines could be compared to, such as RGB + depth predictor to simulate a RGB-D methods, or using the visual clues for pose estimations and using existing methods as baselines.\n\n4. Lack of explanation of design of choices: why explicit leveraging detection predictions by cropping features? What if you just feed in the image features? How much does \"explicit leveraging detection results\" help?\n\n5. Seeing the gap between w/ GT detection w/ GDino, it seems better detector is more important than the proposed detector itself, have you tried  more powerful detectors?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Q1__:The task is not novel, see EgoSG.\n\n__A1__: Thank you for the question. We will cite EgoSG as a related work. However, while both carry “scene graph” in the names, the MSG task is completely different from the EgoSG and other existing 3D scene graphs.\n\nFirstly, the definition of the graph is different. The existing scene graph connects objects and describes their relationships with the graph edges. We propose MSG as place-object graphs, where edges between images reflect the topological connectivity, and edges between objects and images reflect the object locations after resolving the object association. As reviewer 2 (AiA2)\n acknowledges, associating objects across frames and long periods without observation is a challenging and important task that does not truly require 3D information.  We believe introducing and studying the multiview unposed scene graph will provide a meaningful topological scene representation beneficial to other downstream tasks in computer vision and robotics, such as loop closure, SfM[1], visual SLAM[2], visual navigation[3], and mobile manipulation.\n\nSecondly, the input data format is different. EgoSG and other 3D scene graphs require having 3D scene representations before obtaining the scene graphs, which is already a very good starting point for many vision tasks and is not always available. Our proposed MSG focuses on a different challenge. The input is purely unposed RGB images without any additional spatial knowledge. MSG exactly aims to extract topological spatial knowledge from these inputs. This allows it to work on arbitrary videos when depth information is not given or needs to be estimated and lays a good foundation for reconstructing 3D information from 2D inputs. While estimating depth and poses to convert the problem into RGB-D could improve performance, it does not solely solve our task as the estimation itself can introduce errors.  As suggested by the reviewer(discussed in the following response Q3A3), we have tried a recent strong pose estimation method [4] and found the performance less satisfactory due to the accumulated drifting pose errors. The loop closure provided by the MSG will in fact help better estimate poses.\n\nAdditionally, our MSG will be complementary to the existing scene graphs. Edges for object-object relationships can be a seamless add-on to extend the MSG with more semantic information. Therefore, we believe our work adds a meaningful contribution to the scene graph community.\n\nIn conclusion, we believe this is a novel task different from the previous scene graphs and also important for many downstream applications in computer vision and robotics. This paper contributes by introducing the MSG task, the set of new baselines, and benchmarks to stimulate future research.\n\n[1]Xiao, Jianxiong, Andrew Owens, and Antonio Torralba. \"Sun3d: A database of big spaces reconstructed using sfm and object labels.\" ICCV. 2013.\n\n[2] Salas-Moreno, Renato F., et al. \"Slam++: Simultaneous localisation and mapping at the level of objects.\" CVPR. 2013.\n\n[3] Chaplot, Devendra Singh, et al. \"Neural topological slam for visual navigation.\" CVPR. 2020.\n\n[4] Barroso-Laguna, Axel, et al. \"Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences.\" CVPR. 2024.\n\n__Q2__: how was ViT pre-trained:\n\n__A2__: Thank you for the question. We will update the manuscript with this information. For the ViT model, we are using the base model with the default weights pretrained on ImageNet-21K. It is not a masked auto-encoder (MAE).  MAE is also an important image pretraining model. We would like to note that any visual encoders that can produce feature maps or tokens are adaptable to our method. However, we did not sweep through more available choices as we found that DINOv2 gives significantly better performance. \n\n__Q3__: other baselines with existing methods.\n\n__A3__: Thank you for the suggestion. We agree adding more baselines linked to existing tasks and methods will benefit our work. Therefore, We have added a new baseline based on pose estimation. We are happy to explore other possible directions in the future. Specifically, We use a pretrained Mickey model [4] and provide the image data sequentially in temporal order and also the corresponding intrinsics ( not provided to MSG). We compute relative poses and convert them to absolute poses w.r.t the first frame. Then use the same threshold as the data set hyperparameter to obtain the p-p adjacency matrix and P-P IoU. The results are listed in the table below. We see that the performance is close to that of Anyloc. Recall@1 is easily 1.0 since the data is given in temporal order, and consecutive frames are trivially recalled. But its P-P IoU is inferior to Anyloc, suggesting the drifting issue of estimated poses and the need for loop closure, which is precisely MSG’s strength.\n\n| model | Recall@1 | PP IoU |\n|---------------------------|-------|-------|\n| Anyloc                 | 97.1 | 34.2 |\n| Mickey[1]              | 100 | 33.1 |\n\n[4] Barroso-Laguna, Axel, et al. \"Matching 2D Images in 3D: Metric Relative Pose from Metric Correspondences.\" CVPR. 2024.\n\n\n__Q4__: why explicit leveraging detection predictions by cropping features? What if you just feed in the image features? How much does \"explicit leveraging detection results\" help?\n\n__A4__: The task of MSG involves associating objects from images across views. Therefore, we leverage detection bounding boxes to crop features to obtain features of object appearances. If directly feeding the image features as a whole, it would be difficult and unnatural for object association since the model would have no prior knowledge of what are the interested objects. So explicitly leveraging detection results will help the model locate the object appearances in the input images and associate them to build the multiview scene graph.\n\n\n**Continue in the official comment.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RPc2SjsSga", "reviewer": "Reviewer_o1XP", "summary": "The paper introduces a new scene graph format, which regards objects and places as nodes. The edges are intuitive, basically saying which objects are in which places and which places are close to each other. The paper also provides two metrics on which the graph generation method can perform two tasks better than the chosen baselines. The authors also provide an application using the proposed graph to accelerate DUST3R's process.", "questions": "The same as above.", "limitations": "Limitations have been thoroughly discussed.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper introduces a new scene graph format, which regards objects and places as nodes. The paper also provides two metrics on which the graph generation method can perform two tasks better than the chosen baselines.", "weaknesses": "### Motivation\nEven though I understand what the authors are trying to express in the paper, the necessity of the proposed format of scene graph still raises my concern. \n1. Firstly, the current scene graph extracted from multiple views is a bit simple; there is no spatial or other high-level semantic relationship between objects. From my point of view, the graph only associates the adjacent views for scene nodes and the same objects in the scene as object nodes. Do authors think one can leverage a VLM to do a similar job? Just query the VLM to associate the objects that appear in the multiple frames and query to associate the same scene if adjacent frames are close. \n2. Secondly, do the authors think the proposed graph can somehow have a better ability to evaluate these tasks (VPR and object association)? If so, could you explain motivation more? What are the usages of edges between objects and places? Can existing scene graph formats, e.g., [1,2], do similar tasks in this paper?\n\n### Experimental Issue\nThe experiments were not sufficient enough. The baselines in the paper were from early ages, and I suggest more recent ones should be compared. For example, two papers are first online in Nov 2023 [3] and in Feb 2024 [4]. Authors are encouraged to choose other methods as long as they are sensible to be recent baselines.\n\n### Misc\n1. Do the authors want to modify the title a bit? the current one seems to be a bit broad.\n2. I may miss the related part, but what object detector did the authors use in Figure 2?\n3. In the Supplementary, it is good to see the authors provide the source code, but the README is a void file, which makes it hard to understand the details. \n4. The paper has several typos, e.g., \"objects nodes\" in Line 32 and \"Simoutenous\" in Line 72. Please do a careful review.\n\n[1] Armeni I, He Z Y, Gwak J Y, et al. 3d scene graph: A structure for unified semantics, 3d space, and camera. CVPR 2019.\n\n[2] Wu S C, Tateno K, Navab N, et al. Incremental 3d semantic scene graph prediction from rgb sequences. CVPR 2023.\n\n[3] Izquierdo S, Civera J. Optimal transport aggregation for visual place recognition. CVPR 2024.\n\n[4] Lu F, Lan X, Zhang L, et al. CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognition. CVPR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "__Q1__: the current scene graph extracted from multiple views is a bit simple\n\n__A1__: We acknowledge the fact that the proposed MSG has a rather simple format as it is free of spatial or high-level semantic relationships between objects. \nHowever, this does not mean this task is simple, insignificant, or can be readily solved. As acknowledged by reviewer 2 (AiA2) and demonstrated by the baseline performances in the paper, building topological connectivity between the unposed images and associating objects across frames and long stretches of time without observation is a challenging and important task. Constructing such multiview scene graphs from unposed images is fundamental to many tasks, such as 3D reconstruction[1], visual navigation[2], visual slam[3], etc. Having such a graph can be beneficial to all these applications.\n\nFurthermore, the proposed MSG is complementary to the existing scene graphs. Edges for object-object relationships can be a seamless add-on to extend the MSG with more semantic information. Therefore, we believe our work adds a meaningful contribution to the scene graph community.\n\nWe also demonstrate in the next response that current Vision-Language Models (VLMs) are not yet capable of solving the MSG task.\n\n\n[1]Xiao, Jianxiong, Andrew Owens, and Antonio Torralba. \"Sun3d: A database of big spaces reconstructed using sfm and object labels.\" Proceedings of the IEEE international conference on computer vision. 2013.\n\n[2] Chaplot, Devendra Singh, et al. \"Neural topological slam for visual navigation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.\n\n[3] Salas-Moreno, Renato F., et al. \"Slam++: Simultaneous localisation and mapping at the level of objects.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2013.\n\n\n\n\n__Q2__: Do authors think one can leverage a VLM to do a similar job?\n\n__A2__: We appreciate the suggestion. VLMs exhibit strong emergent abilities for many tasks, and we agree it would be interesting to try them out for MSG. However, querying VLMs with every image pair is a huge amount of work and cost, and sending all images together poses a challenge to the context length limit while also hurting performance. Therefore, we conducted a case study with one scene as a pilot study.\n\nSpecifically, we sampled a scene with a relatively small number of images and further subsampled all the images containing annotated objects, resulting in 22 images in total. We then queried the GPT-4o[1] 231 times with each image pair annotated with object bounding boxes as visual prompts, the corresponding box coordinates, and the task prompt. By parsing the GPT-4o outputs, we obtained the following results:\n\n| metric  | model total | model adjusted | VLM   |\n|---------|-------------|----------------|-------|\n| PP IoU  | 59.3        | 63.0           | 30.3  |\n| PO IoU  | 85.0        | 85.0           | 62.5  |\n\nThe *model total* represents the performance of our model on the entire scene, and *model adjusted* represents the performance evaluated only on those 22 subsampled images for a fair comparison. Besides the issues with computation cost and context limits, we note that a common failure pattern of VLM is the failure to maintain consistent object associations. The limitations of VLM in VPR are also discussed in the literature [2].\n\nNevertheless, we acknowledge that this is only a small-scale pilot study. It is well possible to have better VLMs in the future and come up with better prompts, and we are excited about the future possibilities of VLM + MSG.\n\n[1] Achiam, Josh, et al. \"Gpt-4 technical report.\" arXiv preprint arXiv:2303.08774 (2023).\n\n[2] Lyu, Zonglin, et al. \"Tell Me Where You Are: Multimodal LLMs Meet Place Recognition.\" arXiv preprint arXiv:2406.17520 (2024).\n\n\n\n__Q3__: do the authors think the proposed graph can somehow have a better ability to evaluate these tasks (VPR and object association)? … Motivation and usage. Can existing scene graph formats do similar tasks in this paper? \n\n__A3__: Thank you for the question. The proposed MSG encompasses both visual place recognition (VPR) and object association. Our proposed evaluation metrics, PP IoU and PO IoU, reflect the quality of place recognition and object association. Our proposed baseline combines the two tasks in a joint architecture. Therefore, we believe training and evaluating on the task of MSG will help both VPR and object association. \n\nWe are motivated by the fact that building spatial correspondence by associating images and objects is fundamental for scene understanding. The MSG takes unposed RGB image sets as input, allowing it to be applied to any arbitrary image sets and video streams. As discussed in Q1, obtaining MSG from the unposed RGB images will provide a foundation for many downstream tasks in computer vision and robotics, such as loop closure, SfM, visual SLAM, navigation, and mobile manipulation.\n\nExisting scene graphs focus on describing the spatial and semantic relationships between different objects, which is different from the MSG task. Meanwhile, existing 3D scene graphs require having 3D scene representations before obtaining the scene graphs. This, where object association has been given, is already a good starting point for many vision tasks. However, such 3D scenes are not always available or can be reliably obtained from unposed RGB observations. Therefore, we believe our MSG is a complementary contribution to the existing scene graph literature. \n\n**Continue in the official comment**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank our reviewers for their encouraging comments, helpful suggestions, and insightful questions. 

**Acknowledgements**. 
-  All reviewers acknowledge our novelty. 
   -  R1 qE49: “The paper introduces the novel task…”.
   -  R3 9wC7: “The paper proposes a novel task…” “proposes a new benchmark … a novel method … showing promising result”. 
   -  R4 o1XP: “The paper introduces a new scene graph format…”. 
   -  R2 AiA2 further acknowledges that our posed MSG “is a challenging and important task” and “is useful for the community”. 
- Several reviewers also think the paper is well-written with good presentation quality. 
   -  R2 AiA2: “Overall the paper is well written and the illustrations are high quality to support the understanding of the method and ideas.” 
   -  R3 9wC7 rates “Presentation: 4: excellent.” 
We thank all the reviewers for their appreciation of this work.

**Questions and suggestions**. 

Reviewers’ questions and suggestions are very constructive. We sincerely thank them for their advice in making the work of better quality.  
   -  R1 qE49 and R2 AiA2 suggest adding additional analysis of the relative pose distributions and failure cases. We added these results in the attached PDF file and provided our thoughts and analysis in the responses. 
   -  R3 9wC7 and R4 o1XP suggest additional baselines. We implement them and report the results and analysis in the PDF file as well as in the responses. 
   -  R3 9wC7 and R4 o1XP also raise questions regarding the similarities and differences between our proposed Multivew Scene Graph (MSG) and other existing formats of scene graphs, to which we explain the difference in the definition and format of the graph, the input data, and the downstream applications. We also note that the MSG is complementary to the existing scene graphs. We answer the question in more detail in the corresponding responses.

**Attached PDF file**. 

In total, we collected reviewers' constructive feedback and added the following 6 qualitative and quantitative experiments:
1. An analysis of relative pose distribution from the model’s prediction.
2. Provides failure cases which results in low IoU scores.
3. A qualitative real-world experiment on self-recorded videos using an iPhone.
4. Additional baseline using the most recent pose estimation method.
5. A pilot study of using VLM to try to solve the proposed MSG task.
6. Additional baseline using the most recent VPR method.

Please find the details in the PDF file and the responses. We will also revise the manuscript to reflect these experiments when the paper is camera-ready.

**Typos and revisions**. 

We thank all the reviewers for pointing out the typos and confusion. Per the rules of the rebuttal session, we have marked all the typos and revisions suggested by the reviewers and will revise the manuscript when camera-ready.

Meta-review: This paper proposes a methodology to build Multiview Scene Graphs from a set of unposed images. The paper initially received discordant ratings (2x BR, 2x WA). After rebuttal and discussion, the two positive reviewers confirmed their initial ratings (WA) and one of the two BR decided to raise their rating to BA. 
The 4th reviewer was unresponsive and did not participate to the discussion. Given their initial rating is borderline and we don't know how the rebuttal has met their initial concerns, their initial rating is not taken into account for the final decision. 
The final AC recommendation is thus to accept this work as poster.
Decision: Accept (poster)