Multiview Scene Graph

Authors: juexiao  zhang
gao  zhu
sihang  li
xinhao  liu
haorui  song
xinran  tang
chen  feng
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 17:49:13

Abstract: A proper scene representation is central to the pursuit of spatial intelligence where agents can robustly reconstruct and efficiently understand 3D scenes. A scene representation is either metric, such as landmark maps in 3D reconstruction, 3D bounding boxes in object detection, or voxel grids in occupancy prediction, or topological, such as pose graphs with loop closures in SLAM or visibility graphs in SfM.
  In this work, we propose to build Multiview Scene Graphs (MSG) from unposed images, representing a scene topologically with interconnected place and object nodes. 
  The task of building MSG is challenging for existing representation learning methods since it needs to jointly address both visual place recognition, object detection, and object association from images with limited fields of view and potentially large viewpoint changes.
  To evaluate any method tackling this task, we developed an MSG dataset and annotation based on a public 3D dataset.
  We also propose an evaluation metric based on the intersection-over-union score of MSG edges. 
  Moreover, we develop a novel baseline method built on mainstream pretrained vision models, combining visual place recognition and object association into one Transformer decoder architecture. 
  Experiments demonstrate that our method has superior performance compared to existing relevant baselines.

Main paper: 
 introduction: 
 The ability to understand 3D space and the spatial relationships among 2D observations plays a central role in mobile agents interacting with the physical real world. Humans obtain such spatial intelligence largely from our visual intelligence [26, 45] . When humans are situated in an unseen environment and try to understand the spatial structure from visual observations, we don't perceive and memorize the scene by exact meters and degrees. Instead, we build cognitive maps topologically based on visual observations and commonsense [27, 48] . Given imagery observations, we are able to associate the images taken at the same place by finding overlapping visual clues and identifying the same or different objects from various viewpoints. This ability to establish correspondence from visual perception constitutes the foundation of our spatial memory and cognitive representation of the world. Can we equip AI models with similar spatial intelligence? Motivated by this question, we propose the task of building a Multiview Scene Graph (MSG) to explicitly evaluate a representation learning model's capability of understanding spatial correspondences. Specifically, as illustrated in Figure 1 , given a set of unposed RGB images taken from the same scene, this task requires building a place+object graph consisting of images and object nodes, where images taken at nearby locations are connected, and the appearances of the same object across different views should be associated together as one object node. Connected place nodes represent images taken at the same place. The same object recognized from different views is associated and merged as one node and connected to the corresponding place nodes.We position the proposed Multiview Scene Graph as a general topological scene representation. It bridges the place recognition from robotics literature [3, 4, 36] and the object tracking and semantic correspondence tasks from computer vision literature [20, 23, 64] . Different from previous work in topological mapping that evaluates a method's performance on downstream tasks such as navigation, we propose to directly evaluate the quality of the multiview scene graph, which explicitly demonstrates a model's spatial understanding with correct visual correspondence of both objects and places across multiple views. Moreover, the MSG does not require any metric map, depth, or pose information, making it adaptable to the vast data of everyday images and videos. This also differentiates MSG from the previous work in 2D and 3D scene graphs [5, 32, 35, 69] , where they emphasize objects' semantic relationships or require different levels of 3D and metric information.To facilitate the research of MSG, we curated a dataset from a publicly available 3D scene-level dataset ARKitScenes [8] and designed a set of evaluation metrics based on the intersection-overunion of the graph adjacency matrix. The detailed definition of the MSG generation task and the evaluation metrics are discussed in Section 3.1. Meanwhile, since this task mainly involves solving place recognition and object association, we benchmarked popular baseline methods respectively in place recognition and object tracking, as well as some mainstream pretrained vision foundation models. We also designed a new Transformer-based architecture as our method, Attention Association MSG, dubbed AoMSG, which learns place and object embeddings jointly in a single Transformer decoder and builds the MSG based on the distances in the learned embedding space. Our experiments demonstrate the superiority of our new model compared with the baselines by a great margin, yet still reveal strong needs for future advances in research for spatial intelligence.In summary, our contributions are two-fold:• We propose the Multiview Scene Graph (MSG) generation as a new task for evaluating spatial intelligence. We curated a dataset from a publicly available 3D scene dataset and designed evaluation metrics to facilitate the task. • We design a novel Transformer decoder architecture for the MSG task. It jointly learns embeddings for places and objects and determines the graph according to the embedding distance. Experiments demonstrate the effectiveness of the model over existing baselines.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The newly proposed MSG task is rigorously defined in Section 3.1. We thoroughly describe the curation of our dataset and label used for the experiment in Section 5.1.The evaluation metric for the MSG task is introduced in Section 3.2. The proposed pipeline is explained in Section 4 and experiment results are reported in Section 5.4. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Scene Graph Scene graphs [35, 69] are originally proposed to represent the spatial and semantic relationships between objects in an image. The generated scene graph can be used for image captioning [47] and image retrieval [35] . Although they provide a structured spatial representation, it remains at the 2D image level. 3D scene graphs [5, 32, 33, 66, 71] extend this concept into 3D, representing a scene as a topological graph with objects, rooms, and camera positions as their nodes.These graphs are typically built by abstracting from 3D meshes, point clouds, or directly from RGB-D images. [67] proposes incrementally building 3D scene graphs from RGB sequences, describing semantic relationships between objects. As a new type of scene graph, MSG is built from unposed images without sequential order, emphasizing the understanding of relationships between objects and places via multiview visual correspondences. MSG complements existing scene graphs, as their object-object relationship edges can be a seamless add-on to extend MSG with more semantic information. Therefore, we believe MSG provides a meaningful contribution to the scene graph community by enhancing its representational depth and flexibility.Scene Mapping Simultaneous localization and mapping (SLAM) [17, 46, 57, 59 ] is a classic way of creating maps of an environment from observations. The metric maps built from SLAM are subsequently utilized as the spatial representation for the robots to perform tasks such as navigation.In contrast to metric maps, topological mapping [56] is inspired by landmark-based memory in animals, and follows a more natural and human-like understanding of the environment to better support navigation tasks. The quality of the topological maps is evaluated mostly through navigation tasks [11, 15] . Another line of scene mapping work harnesses object or semantic information to build more robust maps [25, 54, 68] , with TSGM [37] being the most relevant to our work. Differently, our proposed MSG serves as a general-purpose scene representation and can be directly evaluated using our proposed metrics. The quality of MSG that a model can build explicitly evaluates its capability of understanding spatial correspondences.Visual Place Recognition Visual Place Recognition (VPR) is often formed as an image retrieval problem. This involves extracting image features and retrieving the closest neighbor from an image database. Traditional approaches rely on handcrafted features [9, 41] . NetVLAD and its variants [4, 16, 44] use deep-learned image features to improve recall performance and robustness. The emergence of self-supervised foundation models, such as DINOv2 [49] , enables universal image representations, offering significant progress [34, 36] across many VPR tasks. However, VPR is framed as an image retrieval problem, whose output-the image features-does not directly equal a graph. Although a graph can be built by proximity search in the VPR feature space, the widely used recall metric in VPR does not directly reflect how good the graph is, i.e. how many pairs of connected images in this graph are truly at the same place. Instead, our proposed task and evaluation metric focus only on the graph generated from the model. The metric straightforwardly reflects the quality of the scene representation.Object Association Traditionally, object association is approached by matching keypoint features across image pairs [41, 55] . Recently, CSR [24] learns feature encodings of object detections and measures the cosine similarity between the learned features to determine object matching. ROM [23] on the other hand follows SuperGlue [55] and uses attentional GNN and Sinkhorn distance [58] for relational object matching. Our method draws inspiration from this previous work but adopts a Transformer decoder architecture and learns object instance embeddings jointly in a unified model with place recognition. Literature in multi-object tracking [50, 64, 65, 72] and video object segmentation [19, 20, 31, 62] also handles object association. They mostly leverage temporal dependencies or memories such as by propagating detection bounding boxes or segments through time. Therefore, these models may lack a sense of space and suffer when objects reappear from a very different viewpoint or after a longer period. Interestingly, a recent study Probe3d [22] reveals that even though the pretrained vision foundation models have undergone tremendous progress in the recent years [14, 21, 30, 38, 49] , they still struggle with associating spatial correspondences of objects from large viewpoint change. Our method learns scene representation with spatial correspondence, where multiple views of the same places or the same objects are close in the embedding space.3 Multiview scene graphQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The limitation of the work is discussed in detail in Section 6.2 Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
problem definition: 
 Multiview Scene Graph Given a set of unposed images of a scene X = {x i } i=0,...,T , we represent a Multiview Scene Graph as a place+object graph:EQUATIONwhere P and O respectively refer to the sets of place and object nodes. The set of object nodes O contains all the objects detected from X. The same object detected from different images across different viewpoints should always be considered as one object node. For the definition of places, we follow the definition in the VPR literature and set P = X. This means each image corresponds to a node for a place, and if two images are taken within only a small translation and rotation distance, they are considered as taken in the same place and are connected with an edge in E P P . Consequently, the E P P is the set of place-place edges which refers to the edges that connect the images regarded as in the same place, and the E P O represents the set of place-object edges, referring to the edges that connect the places and the objects that appear in those places. Therefore, an object can be seen in multiple images and thus connected to more than just one place node. These images can be close by or from a distance. Naturally, a place node can connect to more than one object node, since an image can contain multiple objects' appearances.MSG generation task As illustrated in Figure 1 The quality of Ĝ is evaluated by measuring Â against the groundtruth A. According to our definition, the adjacency matrix can be further decomposed into the following block matrix:EQUATIONwhereA P P = A 1≤i≤|P |,1≤j≤|P | and A P O = A 1≤i≤|P |,|P |+1≤j≤|P |+|O| .The same decomposition applies to Â. Since the MSG contains only the place-place edges and the place-object edges, A OO is left blank. Meanwhile, A P O is symmetric to A OP . So our evaluation will focus on A P P and A P O .Given that the two adjacency matrices A and Â are binary, we evaluate their intersection over union (IoU) to measure how much the two graphs align. As aforementioned, an adjacency matrix A essentially consists of two parts: the place-place part A P P and the place-object part A P O . So we evaluate them respectively as PP IoU and PO IoU and combine them to get the whole graph IoU.We provide a precise mathematical definition of the IoU calculation for any two binary adjacency matrices in Appendix B.1 and we denote this function by IoU(•, •) in the following for simplicity.PP IoU For the PP IoU, the calculation is relatively straightforward since the number of images is deterministic and the one-to-one correspondence between the groundtruth A P P and the prediction ÂP P is fixed. As a result, the PP IoU is simply:PP IoU = IoU(A P P , ÂP P ).Additionally, we also report the Recall@1 score alongside PP IoU since it is the standard evaluation metric for visual place recognition.PO IoU However, it is less straightforward for the PO IoU. The number of objects in the predicted set Ô may differ from O, and their correspondence cannot be determined directly from the adjacency matrix. For a fair evaluation, we need to align Ô with O as much as possible. In other words, before computing IoU, we need to find the best matching object for each groundtruth object. This truth-to-result matching is also an important issue in multi-object tracking [53] . To do so, we also record the object bounding boxes in each image and calculate the generalized IoU score (GIoU) of the bounding boxes following [52] . Then we compute a one-to-one matching between O and Ô based on the accumulated GIoU score across all the images. Details of the score computation are included in the Appendix B.2. According to the matching, we can reorder Ô to best align with the objects in O. This can be mathematically expressed as a permutation matrix S ∈ R | Ô|×| Ô| to permute the columns of Â. Formally, the PO IoU is expressed as the following: EQUATIONQuestion: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
our baseline: attention association msg generation: 
 When developing a new model for the MSG generation task, we adhere to two core principles: Firstly, the model should capitalize on the strengths of pretrained vision models. These pretrained models offer a robust initialization for subsequent vision tasks, as their output features encapsulate rich semantic information, forming a solid foundation for tasks like ours. Secondly, both place recognition and object association fundamentally address the problem of visual correspondence and can mutually reinforce each other through contextual information. Thus, our model is designed to integrate both tasks within a unified framework. With these guiding principles, we propose the Attention Association MSG (AoMSG) model, depicted in Figure 2 .Place and object encodings Given a batch of unposed RGB images as input, the AoMSG model first employs pretrained vision encoders and detectors to derive image tokens and object detection bounding boxes from each image. We utilize the Vision Transformer-based pretrained model DINOv2 [49] as our encoder, though our design is adaptable to any Transformer-based or CNN-based encoder that produces a sequence of tokens or a feature map. In the case of the DINOv2 encoder, we reshape the output token sequences into a feature map, which is then aligned to the object bounding boxes, aggregating an encoding feature for each detected object. To integrate place recognition and object association within a unified framework, we obtain the place encoding feature by treating it as a large object with a bounding box that encompasses the entire image, aggregating features as if a detected object. The obtained place feature is then positioned alongside the object features, serving as queries for the Transformer decoder, as detailed in the subsequent sections.AoMSG decoder We follow a DETR-like structure [13] to design our AoMSG decoder. Specifically, the derived place feature and object features are stacked as a sequence of queries for the Transformer decoder, while the preceding image tokens are used as keys and values. As shown in Figure 2 , we enhance the queries by incorporating positional encodings by normalizing and embedding the bounding box coordinates. For instance, for the place feature, the equivalent bounding box is the entire image as aforementioned, resulting in the normalized coordinates of [0, 0, 1, 1]. These coordinates are projected to match the dimensionality of the encoding and added elementwise to the place query. The outputs of the AoMSG Transformer decoder are the place and object embeddings that have aggregated context information from the image tokens. Then two linear projector heads are applied to each object and place embeddings respectively to obtain the final object and place embeddings, projecting them into the representation space for the task.For training, we compute supervised contrastive learning [51] respectively on the place and object embeddings from the same training batch in a multitasking fashion. For the object loss, we simply use binary cross-entropy with higher positive weights. For the place loss, the mean square error is minimized for their cosine distances, which gives better empirical results.During inference, we simply compute the cosine similarity among the place embeddings and apply a threshold to obtain the place-place predictions in Â. For the objects, we track their appearances and maintain a memory bank of the existing objects for each scene, updating their embeddings or registering new objects based on cosine similarity and thresholding. The results are consequently converted to the place-object part in Â. Notably, there could be many possible choices to compute the contrastive losses and determine the predictions, we keep our choices simple as we empirically find the standard losses and the simple cosine thresholding can already produce decent results while keeping the embedding spaces straightforwardly meaningful. We discuss the results in detail in Section 5. 
data: 
 The MSG models can be trained with any dataset that provides camera poses and object instance labels. We utilized the publicly available 3D indoor scene dataset ARKitScenes [8] to construct our dataset. ARKitScenes contains point clouds and 3D object bounding boxes of the scenes, as well as the calibrated camera poses obtained from an iPad Pro. We transform the point clouds in the 3D bounding boxes with respect to the camera poses to obtain the 2D bounding boxes in each frame. The resolution of each frame is 192 × 256. 4492 scenes are used for training and 200 scenes are used for testing. Note that none of the two scenes share the same objects. We leverage the camera poses to obtain the place annotations. Translation threshold and rotation threshold are set to 1 meter and 1 radian respectively, images taken within both thresholds are considered as capturing the same place. We report results from the base models for both ConvNext [40] and ViT [21] .Figure 4 : Visualization of the same objects and the same places. Objects are annotated with their predicted IDs.VPR We adopt protocols in the previous VPR benchmark literature as outlined in [10, 36] . In our off-the-shelf baselines, we evaluated VPR using DINOv2 [49] either as the global descriptor or followed by a VLAD dictionary generated from a large-scale indoor dataset following [36] , or as a feature extraction backbone [34] . For the trained baseline, we conduct our experiments mainly with ResNet-50 [28] + NetVLAD used in [10] . Additionally, we also test a recent pose estimation baseline [7] and use the poses to estimate the places according to the same thresholds as in the dataset.Object association We adopt two popular baselines for object association, Unitrack [64] from multi-object tracking, and DEVA [20] from video object segmentation. The image sets are processed in temporal order just like tracking. Unitrack can take any detection backbones and associate object bounding boxes by comparing their features with an online updating memory bank. For a fair comparison, we extend its memory buffer length to cover the whole set of images for every scene. DEVA leverages the Segment Anything model [38] to segment and track any object throughout a video without additional training. Their tracking results can be easily converted for evaluating object association based on the tracker IDs.SepMSG We also evaluate the pretrained vision models by first separately encoding images and object detections to features and directly evaluating MSG based on those features. This baseline is referred to as SepMSG-Direct, where Sep means separately handling places and objects. Then as a common way of evaluating pretrained models [29, 30] , we conduct probing [2] by further training a linear or MLP classifier on those frozen features. These baselines are referred to as SepMSG-Linear and SepMSG-MLP. The SepMSG baselines serve as ablation to validate our model against simply using features learned from the pretrained backbones.For AoMSG, we experimented with different choices of backbones, sizes of the Transformer decoder, and dimensions of the final linear projector heads. Their results are discussed in Section 5.4. All the models are trained on a single H100 or GTX 3090 graphics card for 30 epochs or until convergence. We provide detailed hyperparameters in the appendix. During training, we randomly shuffle the scenes and mix data from multiple scenes in a single batch so that the model sees diversified negative samples at every epoch. Additionally, we monitor the total coding rate as in [60] to avoid the embeddings from collapsing.To keep the evaluation focused on the quality of the graph rather than the quality of object detection, we choose not to train the detector together with the MSG objectives. Instead, we use the groundtruth detection bounding boxes and a popular open-vocabulary object detector GroundingDINO [39] .Results on both configurations are listed in Table 1 and discussed in the following.Table 1 shows comparison of our results and baselines. We find that for the place Recall@1 and PP IoU, the baselines have competitive performance. While the results from the SepMSG baselines are comparable, AoMSG outperforms them all and produces the best results in both metrics. We also notice that all the models produce high Recall@1, but their PP IoU scores are varied and less than 50. This suggests that having high recall is not enough to guarantee a good graph. For PO IoU, AoMSG models outperform all the baselines by big margins. Both Unitrack and DEVA perform poorly as they struggle when objects reappear after large viewpoint changes or long periods of time. We note that all the MSG methods produce relatively worse results when using GroundingDINO as the detector rather than the ground truth detection. This indicates the performance gap caused by inaccurate object detection. Nevertheless, their performances are still consistent and AoMSG still performs the best. This suggests a better detector will likely give better results for the MSG task. To conclude, AoMSG gives the best performance for all the metrics.Projector dimensions As listed in Table 2 , we compared the impact of different projector dimensions as it is reportedly important to performance in the literature of self-supervised representation learning [6, 12, 18] . We find the empirical results are comparable in our experiments.Choices of backbones Figure 3 shows the performance of difference choices of pretrained backbones. We experimented with state-of-the-art CNN-based model ConvNext [40] , Vision Transformer (ViT) [21] and DINOv2 [49] . We find DINOv2 performs the best, consistent with the observations made in [22] . We use DINOv2 as our default encoder. Interestingly, performance saturates with the size of DINOv2. We suspect it will still increase if we could further scale the size of the data.Qualitative In Figure 5 we visualized the learned object embeddings on 6 scenes by AoMSG, the SepMSG-Linear baseline, and SepMSG-Direct that directly uses the output features from the pretrained DINOv2 encoder for the task. The visualization aims to qualitatively assess the learned object embeddings as to how separated different objects are in the embedding space. We can see the pretrained embeddings already provide some decent separations. SepMSG-Linear only tunes a linear probing classifier on top so the separation is slightly improved. For example, see the first and second scenes to the left. Compared with them, AoMSG gives the most significant separations, with appearances of the same objects pushed closely and different objects pulled far away. Additionally, Figure 4 visualizes results on some places and objects, and we provide more in Appendix D. 
application: 
 Given the recent advances in novel view synthesis, 3D reconstruction, and metric mappings, one might wonder whether the proposed MSG is still useful. Here we provide some justifications and a showcase application. Echoing literature in 3D scene graphs [32, 66] , we believe the MSG can be a versatile mental model for embodied AI agents and robots. At a global level, it keeps a lightweight topological memory of the scene from purely 2D RGB inputs, which serves as a basis for robot navigation [15, 37] . At a finer level, it can seamlessly couple MSG with the 3D reconstruction methods, to estimate depth and poses and build local reconstructions. Therefore, a robot can traverse the environment, localize itself referring to the MSG, and build a local reconstruction when needed for tasks that require metric information such as the manipulation tasks.As a showcase application, we provide two local 3D reconstruction cases illustrated in Figure 6 using the most recent off-the-shelf 3D reconstruction model Dust3r [63] . Directly applying Dust3r to a dense image set greatly consumes GPU memory, which may be infeasible for mobile robots. Whereas a random subsample does not guarantee the reconstruction quality. Instead, with MSG, we can provide the Dust3r with locally interconnected subgraphs for fast and reliable local reconstruction.The subgraphs and local reconstructions can be object-centric thanks to the place+object nature of The current work still has many limitations. Firstly, we only conducted experiments in one dataset.Although the dataset contains around 5k scenes, which is sufficient to obtain convincing results, it would still be great to see if training on more diversified data collections can produce better models and stronger generalization as observed in [63] , especially for larger models. We leave this to future work. Secondly, scenes in the current dataset contain only static objects, extending to dynamic objects is a direction worth exploring.Additionally, given the scope of the work is to propose MSG as a new vision task promoting spatial intelligence, we focus on explicitly evaluating the quality of the graph. Therefore, we did not investigate the object detection quality, nor did we deploy the MSG to downstream tasks such as navigation. Note that detection quality does affect the MSG performance though we find it to be consistent across different detection modes, i.e. the groundtruth and the GroundingDINO. Training detectors together with the MSG model and applying MSG to downstream tasks will be our next step to make the work a more complete system. 
conclusion: 
 This  
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: Models trained for the MSG task can have real-world implications, especially their use in mobile agents. We believe it is essential to assess potential unintended consequences when deploying such models to real-world robots, such as safety risks, algorithmic biases, or the impact on humans during social interaction. However, this is beyond the focus of this work since the presented task and the models have not been deployed to the real-world. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The model trained for the MSG task mainly performs spatial understanding and reasoning via visual perception. There is no generative model or scraped datasets used in this research. The dataset we use comes from a published dataset on which the original authors have made safeguards. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? Answer: [NA] Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Given two binary adjacency matrices A ∈ {0, 1} m A ×n A and B ∈ {0, 1} m B ×n B . Suppose the vertices in their corresponding graphs have been compared and best-matched, we can directly compute the IoU as the following:EQUATIONEQUATIONEQUATION)The inclusion of w A and w B in the denominator implies that the IoU value decreases when the two graphs contain additional but not isolated vertices.From the graph perspective, this IoU evaluates binary edge prediction on augmented graphs. Specifically, given a groundtruth graph and a predicted graph, after matching their vertices as discussed in Sec 3.2, we can augment both graphs with unmatched vertices from the other. We name eq (9) as IoU because under this construction, the defined IoU aligns with the following definition in binary classification based on the standard True Positives (TP), False Negatives (FN), and False Positives (FP):EQUATION)Here, a "positive" prediction indicates the presence of an edge.Here we elaborate on the computation of the object matching score mentioned in Sec. 3.2.Given any groundtruth object ∀γ ∈ O and any predicted object ∀τ ∈ Ô, we record and compare their detections across all the T frames. Denote D(γ, t) and D(τ, t) as the groundtruth and predicted object detections in frame t for γ and τ respectively. Use the indicator functions I(•, t) to signal the existence of an object in frame t: if τ exists in t, then I(τ, t) = 1, otherwise I(τ, t) = 0.Therefore, the accumulated GIoU of γ and τ is:EQUATIONwhich is the sum of the generalized bounding box IoU [52] across all the frames that both objects are present. To obtain the final matching score between γ and τ , c γ,τ is further normalized by the sum of the following four terms:• The number of the matched frames, where both γ and τ exist and their GIoU is positive.• The number of the unmatched frames, where both γ and τ exist but their GIoU is zero.• The number of the "false positive" frames, where only τ exists and γ doesn't.• The number of the "false negative" frames, where only γ exists and τ doesn't.The sum of these four terms is in fact equivalent to computing the union of the appearances of γ and τ across all the T frames:EQUATIONConsequently, the matching score between γ and τ is computed as:EQUATIONIn practice, we take 1 -m γ,τ as the cost used in solving the one-to-one assignment problem via Hungarian Matching.We set thresholds as the dataset hyperparameters which is a conventional setup in visual place recognition (VPR) tasks and datasets [42, 70] . Since the MSG task involves place recognition, we choose to adopt this convention. VPR tasks require a model to classify whether or not two images are taken from the same place. The concept of "place" is a discretization of the space that is continuous by nature, necessitating the use of thresholds in the VPR setup.To give a closer look at the effect the threshold has on the model, in Figure 7 we report the relative pose distributions (orientation and translation) for the connected and non-connect nodes based on our model's prediction. The figures show that instead of collapsing to only represent the fixed thresholds, the pose distributions have a clear yet smooth separation across the spatial thresholds.We visualize some failure cases for place recognition in Figure 8 . We observe that most failure cases can be attributed to either having very similar visual features with relatively large pose differences (false positives), such as observing a room from two opposite sides, or having few similarities in visual features with relatively smaller pose differences (false negatives). We note that the recall metric, conventional in VPR, is straightforward and effective for image retrieval against a database. However, it falls short in reflecting challenging false positives and negatives, especially when constructing a topological graph like the MSG where the number of positives varies. This highlights the usefulness of our proposed IoU metric, which consistently evaluates the quality of the graph. Multimodal Large Language Models (MLLMs) have exhibited strong emergent abilities for many tasks, and we are intrigued to try them out for the MSG generation task. However, querying MLLMs with every image pair in an image set is a huge amount of work and cost, and sending all images together poses a challenge to the context length limit while also hurting performance. Therefore, we conducted a case study with one scene as a pilot study.Specifically, we sampled a scene with a relatively small number of images and further subsampled all the images containing annotated objects, resulting in 22 in total. We then queried the GPT-4o [1] 231 times with each image pair annotated with object bounding boxes as visual prompts, the corresponding box coordinates, and the task prompt. By parsing the GPT-4o outputs, we obtained the results in Table 4 .In Table 4 , the model total represents the performance of our model on the entire scene, and the model adjusted represents the performance evaluated only on those 22 subsampled images for a fair comparison. Besides the issues with computation cost and context limits, we note that a common failure pattern of VLM is the failure to maintain consistent object associations. The limitations of VLM in VPR are also discussed in the literature [43] . Nevertheless, this is only a small-scale pilot study. It is well possible to have better VLMs in the future and come up with better prompts, and we are excited about the future possibilities of MLLM + MSG.To examine how our method generalizes to real-world environments, we conducted a qualitative experiment. Specifically, We have self-recorded an unposed video with an iPhone in a household scenario and run our trained AoMSG model with a pretrained Grounding DINO detector on it.In Figure 9 , we show some resulting images with object instance ID labeled and a visualization of the generated graph. Results suggest that our model is able to obtain sensible outputs on arbitrary videos outside of the dataset. NeurIPS Paper Checklist 


Decision: This paper proposes a methodology to build Multiview Scene Graphs from a set of unposed images. The paper initially received discordant ratings (2x BR, 2x WA). After rebuttal and discussion, the two positive reviewers confirmed their initial ratings (WA) and one of the two BR decided to raise their rating to BA. 
The 4th reviewer was unresponsive and did not participate to the discussion. Given their initial rating is borderline and we don't know how the rebuttal has met their initial concerns, their initial rating is not taken into account for the final decision. 
The final AC recommendation is thus to accept this work as poster.

Reviews: review: paper_summary: paper_summary: The paper introduces the novel task of Multiview Scene Graph generation from unposed RGB images, whereas this type of scene graph encodes the notion of 'places', i.e., images from spatially close locations, and detected objects as graph nodes. The motivation is to combine spatial reasoning of object association and place recognition into a joint representation. 

An evaluation metric based on the edge adjacency is introduced along an initial baseline method (AoMSG, based on a Transformer architecture and off-the-shelf image encoder and object detector) for this new task, which is compared against sub-task specific methods on the ARKit scenes dataset.
strength_weakness: strength_weakness: - The current task and proposed baseline model does not seem to allow for an adjustable "place" definition beyond the train set constraints (place = 1m and 1 rad). 
- To give further insights about the implication of the proposed metric, it would be useful to provide failure case which results in low IoU scores, especially for PP IoU which is significantly lower than PO IoU.
questions: questions: - Missing experimental details of Figure 3: How do the numbers provided in the graph relate to the numbers in Table 1, i.e., which model specifically was trained with the specific backbone?
limitations: limitations: The checklist is complete, including justifications for each item. Limitations of the work are discussed in the paper.
paper_summary: paper_summary: The manuscript proposes the problem of inferring a scene graph from unposed images of a space. The key distinguishing factor to previous work is using multiple frames (as opposed to a single frame) and not requiring poses and depth (like for typical metric 3d scene graphs). The scenegraph is defined as a the set of images connected by edges implying topological connectedness and the set of objects. Objects are connected to place nodes based on where they are observed from. The manuscript also proposes a sensible baseline that outperforms more basic baselines for MVSG generation as well as methods that just work on place recognition or just on object tracking.
strength_weakness: strength_weakness: The constraint of not requiring poses and theoretically enables the method to be run on any video sequence. However there are no examples of even qualitative experiments on some arbitrary video sequences. I think this should be easy to add and would help support the idea that unposed-image posed scene graphs are useful. The other aspect that is not addressed in the manuscript is how to train the proposed system without access to poses and 3d surfaces  or 3d bounding boxes of objects in order to supervise the model. It is unclear if the place localizer just learns to replicate the pose thresholds used during training (see question).
questions: questions: - l 204 whats the "benchmark VG pipeline" ? 
- SepMSG: I assume you are also using the same 2d encoder as for AoMSG? Could you clarify.
- The place localization is trained using relative pose thresholding to obtain positive samples. I am curious if that means the features just learn those spatial thresholds? During inference what is the relative pose distribution of the connected places? And the disconnected ones?
limitations: limitations: Limitations are addressed adequately in the paper.
paper_summary: paper_summary: The paper proposes a novel task: generating scene graphs from unposed RGB images as well as a new benchmark based on ARKitscenes. To achieve this, the paper proposes a novel approach that use off-shelf image encoder and detector that are frozen and only train a decoder that takes in the features learned from frozen image encoder and detector to generate scene graphs. The author ablates the pre-trained weights for backbones and show a promising results compared to baselines.
strength_weakness: strength_weakness: 1. The task is not that novel, see EgoSG: Learning 3D Scene Graphs from Egocentric RGB-D Sequences, where a very similar task is propose, as in, generating scene graphs from unposed RGB-D images. And from RGB to RGB-D, a off-shelf depth detector can be leveraged.

2. it is unclear, how was ViT pre-trained, it is using MAE?

3. the authors use a self-made baseline, i am wondering if other baselines could be compared to, such as RGB + depth predictor to simulate a RGB-D methods, or using the visual clues for pose estimations and using existing methods as baselines.

4. Lack of explanation of design of choices: why explicit leveraging detection predictions by cropping features? What if you just feed in the image features? How much does "explicit leveraging detection results" help?

5. Seeing the gap between w/ GT detection w/ GDino, it seems better detector is more important than the proposed detector itself, have you tried  more powerful detectors?
questions: questions: 1. do you need the unposed images must have overlap with each other?
2. Have you tried SAM? and do you think DinoV2 performs best is because of the detector's feature is closed to image encoder, when using DINOv2 as image encoder?
limitations: limitations: it is not that hard to get poses from either sensors or visual cues nowadays.
paper_summary: paper_summary: The paper introduces a new scene graph format, which regards objects and places as nodes. The edges are intuitive, basically saying which objects are in which places and which places are close to each other. The paper also provides two metrics on which the graph generation method can perform two tasks better than the chosen baselines. The authors also provide an application using the proposed graph to accelerate DUST3R's process.
strength_weakness: strength_weakness: ### Motivation
Even though I understand what the authors are trying to express in the paper, the necessity of the proposed format of scene graph still raises my concern. 
1. Firstly, the current scene graph extracted from multiple views is a bit simple; there is no spatial or other high-level semantic relationship between objects. From my point of view, the graph only associates the adjacent views for scene nodes and the same objects in the scene as object nodes. Do authors think one can leverage a VLM to do a similar job? Just query the VLM to associate the objects that appear in the multiple frames and query to associate the same scene if adjacent frames are close. 
2. Secondly, do the authors think the proposed graph can somehow have a better ability to evaluate these tasks (VPR and object association)? If so, could you explain motivation more? What are the usages of edges between objects and places? Can existing scene graph formats, e.g., [1,2], do similar tasks in this paper?

### Experimental Issue
The experiments were not sufficient enough. The baselines in the paper were from early ages, and I suggest more recent ones should be compared. For example, two papers are first online in Nov 2023 [3] and in Feb 2024 [4]. Authors are encouraged to choose other methods as long as they are sensible to be recent baselines.

### Misc
1. Do the authors want to modify the title a bit? the current one seems to be a bit broad.
2. I may miss the related part, but what object detector did the authors use in Figure 2?
3. In the Supplementary, it is good to see the authors provide the source code, but the README is a void file, which makes it hard to understand the details. 
4. The paper has several typos, e.g., "objects nodes" in Line 32 and "Simoutenous" in Line 72. Please do a careful review.

[1] Armeni I, He Z Y, Gwak J Y, et al. 3d scene graph: A structure for unified semantics, 3d space, and camera. CVPR 2019.

[2] Wu S C, Tateno K, Navab N, et al. Incremental 3d semantic scene graph prediction from rgb sequences. CVPR 2023.

[3] Izquierdo S, Civera J. Optimal transport aggregation for visual place recognition. CVPR 2024.

[4] Lu F, Lan X, Zhang L, et al. CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognition. CVPR 2024.
questions: questions: The same as above.
limitations: limitations: Limitations have been thoroughly discussed.

Comments: title: Thank you
Thank for the rebuttal
Inquiry About Any Additional Concerns
Inquiry About Any Additional Concerns
Thank you
Continue to answer reviewer's questions
Continue to answer reviewer's question
comment: Thank you for your valuable reviews and we are very happy to see your positive decision. Yes, we appreciate your constructive advice and we will surely include these adjustments in the revised final paper.
Hi, I am happy to see your rebuttal. I hope you can include adjustment in your final paper. I would like to change the rate to BA.
Dear reviewer o1XP

Thanks for your reviews, which are valuable for improving the overall quality of this manuscript.

To address your concerns, we have added discussions and explanations of the definition and format of our proposed Multivew Scene Graph (MSG) and how it is different from other existing formats of scene graphs. We have also discussed the potential applications and benefits of the MSG. 

For the experiments, we appreciate your constructive suggestions and have conducted an additional baseline based on the recent VPR literature you recommended. We will also add these references to the revised version. We would also like to thank you for suggesting the idea of using VLM. We like this idea and conducted a pilot study. We have presented the results and discussion in the response above as well as in the attached PDF file.

We have also provided answers and clarifications to the other questions of your concerns. We thank you for pointing out the typos and we promise to fix them in the revised version. We have also explained our motivation for choosing the title and we would love to hear from you for any advice on this. 

Could we kindly ask if our responses have addressed your concerns and if you have any new questions? Thanks for your time and effort in reviewing our submission.

Authors of Submission ID 18923
Dear reviewer 9wC7

Thanks for your comments, which are valuable for improving the overall quality of this manuscript.

To address your concerns, we have provided a thorough discussion and explanation of similarities and differences between our proposed Multivew Scene Graph (MSG) and other existing formats of scene graphs such as EgoSG. We really appreciate your thoughts and will include this in our references as well as the discussion of it in the related works section in the revised version. We have also conducted an additional baseline based on pose estimation for the place recognition part of MSG, thanks to your suggestion.

We have also provided a thorough explanation to the other questions of your concerns, such as the use of the detectors in our proposed method, the design choices, encoder choices, and image choices.

Could we kindly ask if our responses have addressed your concerns and if you have any new questions? Thanks for your time and effort.

Authors of Submission ID 18923
Dear reviewer qE49,

Thank you for your valuable reviews and we are pleased to see your positive decision. We really appreciate your constructive suggestions and we will include them in the revised version.
Dear Authors,

thank you for providing the additional results regarding failure cases and their interpretation - this helps in understanding the metric and potential cause of errors in applying the method.

Following the initial question of fellow Reviewer o1XP and the comment  AiA2, I was also excited about the provided VLM comparison and the Author's effort in running such an experiment given the limitation (e.g., context length) of current public VLM services.

I'm happy to stick to my initial rating.
Dear reviewer AiA2,

Thank you for your response. We are glad that you appreciate our work and the additional experiments. For the qualitative experiments, we have generated a whole video with object bounding box visualizations that can show the tracking and re-association effects. But we could only show a few frames in the attached PDF due to space limit. We definitely agree that more frames will be a better visualization and We are releasing more visualization videos on a website along with the interactive visualization tool after the anonymous phase. 

As for showing which objects are identified, that is an essential feature and we currently support it in the front-end visualization tool in the following ways: 
 - a) the orange nodes are object nodes, and hovering the mouse over it will show the object category and an object ID corresponding to the per-frame visualization, which has an object category and an object id labeled to the bounding box. 
 - b) when clicking, some edges will be highlighted in blue (as you can see in the attached screenshot in the PDF). Those between the orange nodes and the blue nodes indicate which and where an object is identified. We are excited to release it for more interactive visualization and we are super happy if you could provide more suggestions so that we can make it better.

Please do not hesitate if you have more questions or suggestions. Thank you again for your time and efforts.
Thank you for addressing my questions and comments. I appreciate the histograms showing the learned relative pose separation based on semantic features. 

The qualitative experiments on iPhone video is good to see although the limited set of frames makes it hard to judge how well the model was able to re-associate and track objects. It seems there are 12 objects - it would be good to see which ones were identified? 

I also had a look at the other reviews; I do agree that EgoGS using RGBD is a related work worth citing. I do think that the work of the authors has a potential for more impact since it can be run on any video sequence without additional processing (as demonstrated by the iPhone video experiment). While we could run a monodepth network, it is unclear how the typical problem of inferring consistent scale will impact the performance of EgoGS with monodepth. 

I was also excited to see the VLM comparison. It supports the need for dedicated approaches (at least thus far) that can run on long video sequences. 

All in all I am happy to stick with my rating.
__Q4__: Experiment. The baselines in the paper were from early ages

__A4__: We thank the reviewer for this question. All the baselines are from recent years. The proposed MSG task involves two parts of baselines: visual place recognition (VPR) and object association. For the VPR part, our NetVlad baseline is adapted from the widely used baseline deep VG benchmark [1] (reference [8] in the manuscript), which was published in 2022. Anyloc was published in 2023. For the object association part, UniTrack is a widely used multi-object tracking method released in 2021. DEVA is from ICCV 2023. 

We appreciate the reviewer’s suggestion and have also evaluated optimal transport aggregation (Salad) [2] as an additional and more recent baseline for the VPR part. The results are reported below.  We find its performance comparable to Anyloc on the recall while slightly better on PP IoU. We note that both of the baselines are evaluated off-the-shelf.

| model | Recall@1 | PP IoU |
|---------------------------|-------|-------|
| Anyloc                 | 97.1 | 34.2 |
| Salad[2]               | 97.1 |  35.6 |
| AoMSG-4	    | 98.3 | 42.2 |

We would like to emphasize that the baseline methods we proposed (AoMSGs) are designed to be straightforward and easily extensible. Future work can incorporate more advanced techniques and insights from the VPR and object association fields, including but not limited to the Sinkhorn algorithm[3] used in the Salad paper. It would be exciting to explore more ways of combining the wisdom in both place and object association fields for MSG in future work.

[1] Berton, Gabriele, et al. "Deep visual geo-localization benchmark." CVPR. 2022.

[2] Izquierdo S, Civera J. Optimal transport aggregation for visual place recognition. CVPR 2024.

[3] Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26, 2013.


__Q5__: Do the authors want to modify the title a bit?

__A5__: Thank you for the question. We chose “Multiview Scene Graph” as the title because we believe it best describes the contribution of this work while maintaining conciseness. A multiview scene graph refers to our proposed task, where a place-object graph is inferred by associating unposed images across views, predicting their topological connectivity, and associating objects across multiple views simultaneously. So multiview association is the key characteristic and essence of the proposed scene graph and the task and we believe that naming the work “Multiview Scene Graph” highlights its core focus and makes it both memorable and easy to understand.

However, we are more than happy to consider any advice on better titles from the reviewer.


__Q6__: What Object Detector is used in figure 2?

__A6__: Any detector can be used as long as it proposes object bounding boxes. The detector is frozen and bounding boxes are the only thing our method needs from the detector. For training, we use groundtruth detection for efficient supervision. We also used Grounding DINO as our off-the-shelf object detector for open-set detection and it shows consistent performance order for all the methods.


__Q7__: The README is a void file in the supplementary materials.

__A7__: Thanks for pointing it out. We deleted README while trying to keep anonymity. We will upload the complete code base and model checkpoint when our paper is camera-ready.


__Q8__: typos.

__A8__: We thank the reviewer for pointing this out. We will fix all typos for the camera-ready version.
__Q5__: Seeing the gap between w/ GT detection w/ GDino, it seems a better detector is more important than the proposed detector itself, have you tried more powerful detectors?

__A5__: Our MSG task and model tackle object association rather than object detection. Association happens after object detection and refers to identifying whether the object detected in one image is the same object detected in another. While it is true that a better detector will enhance the overall performance of MSG, it is inaccurate to say that the detector is more important than MSG and object association itself, as they focus on different tasks.

The gap between ground truth detection and Grounding Dino (GDino) is expected since GDino is used off-the-shelf and in a zero-shot manner. Our aim is to demonstrate that although MSG uses ground truth detections during training, any available detector can be incorporated into the framework, and the performance order remains consistent. Thus, we anticipate that a more powerful detector than GDino or one fine-tuned on the same data should yield better results.

The task focuses on associating places and objects, and we believe having ground truth detections and consistent results with the commonly used GDino is sufficient to convey our contributions. We appreciate the reviewer’s suggestion and will consider training or benchmarking detectors in future explorations.


__Q6__: if overlapped unposed images are needed:

__A6__: No, we do not make such overlapping assumptions for the unposed images. The input of the MSG task is just a set of unposed RGB images. The model learns to figure out whether the images are taken from the closeby positions and associate the object appearances, i.e. whether they are the same object. The input of the entire process is simply unposed RGB images, with no overlapping required.


__Q7__: Have you tried SAM? do you think DinoV2 performs best because the detector's feature is close to the image encoder when using DINOv2 as the image encoder?

__A7__: Yes, our baseline DEVA uses SAM to obtain object segmentations and perform video segmentation by tracking these object segments across the frames. Detector’s features are not used in our model. The detector only provides bounding boxes. Grounding Dino is different from DINOv2 although bearing a similar name. The former is a Transformer-based detection model and the latter is a Transformer-based self-supervised vision pretraining model. DINOv2’s features have shown great performance in downstream tasks, such as in Anyloc. In this work, we have a similar observation that DINOv2 as the encoder backbone produces the best performance. We think the reason for its strong performance is possibly due to its vast and diverse pretraining data.