Rooms from Motion: Un-posed Indoor 3D Object Detection as Localization and Mapping
Authors: Justin Lazarow
Kai Kang
Afshin Dehghan
Venue: NeurIPS 2025 poster ()
Keywords: 3d
3d object detection
mapping
Primary area: applications
TL;DR: 3d object detection (with or without known camera pose) can be considered mapping and localization with objects as 3D oriented boxes.

Abstract: We revisit scene-level 3D object detection as the output of an object-centric framework capable of both localization and mapping using 3D oriented boxes as the underlying geometric primitive. While existing 3D object detection approaches operate globally and implicitly rely on the a priori existence of metric camera poses, our method, Rooms from Motion (RfM) operates on a collection of un-posed images. By replacing the standard 2D keypoint-based matcher of structure-from-motion with an object-centric matcher based on image-derived 3D boxes, we estimate metric camera poses, object tracks, and finally produce a global, semantic 3D object map. When a priori pose is available, we can significantly improve map quality through optimization of global 3D boxes against individual observations. RfM shows strong localization performance and subsequently produces maps of higher quality than leading point-based and multi-view 3D object detection methods on CA-1M and ScanNet++, despite these global methods relying on overparameterization through point clouds or dense volumes. Rooms from Motion achieves a general, object-centric representation which not only extends the work of Cubify Anything to full scenes but also allows for inherently sparse localization and parametric mapping proportional to the number of objects in a scene.

Rooms from Motion:
Un-posed Indoor 3D Object Detection as Localization and Mapping
Justin Lazarow
Kai Kang
Afshin Dehghan
Apple
{jlazarow, kai_kang, adehghan)@apple..coo com
https://ggithei hatp://////pp/aalllaaaaaaaaaaaaa
Rooms from Motion realizes centcc- framework for metric localization and semantic 3D object-level mapping from un-posed RGB images without the need explicit 2D keypoints or point clouds. Given an unordered collection of images, Rooms from Motion detects every object as metric box within each image, uses a learned object matcher to associate objects across frames, estimates relative poses using the 3D boxes of matched objects, and finally estimates absolute camera poses and forms global, semantic 3D object tracks (akin 3D object detection). Above: We visualize the semantics-aware map and camera localization of Rooms from Motion from un un-posed RGB images on two challenging ScanNet++ scenes: a large laboratory a rooms within a residential space. Below: We show class-agnostic results on an open space from the CA- A--MM dataset.
Abstract
We revisit scene-level 3D object detection as the output of an object-centric frame- work capable of both localization and mapping using 3D oriented boxes the underlying geometric primitive. While existing 3D object detection approaches op- erate globally and implicitly rely the priori existence of metric camera poses, our method, Rooms from Motion (RfM) operates on a collection of un-posed images. By replacing the standard 2D keypoint-based matcher of structure-from- motion with an object-centric matcher based on image-derived 3D boxes, we estimate metric camera poses, object tracks, and finally produce global, semantic 3D object map. When priori pose is available, we can significantly improve map quality through optimization of global 3D boxes against individual observations. RfM shows strong localization performance and subsequently produces maps of higher quality than leading point-bace based and multi-view 3D object detection methods
CA-IM and ScanNet++, despite these global methods relying overparame- terization through point clouds or dense volumes. Rooms from Motion achieves an object-centric representation which allows for inherently sparse localization and parametric mapping proportional to the number of objects a scene.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Can objects serve as the basis localization and mapping? Currently, points are the fundamental primitive for localization and mapping techniques. For localization, this usually means matching keypoints between images. For mapping, this means producing sparse point-based representations. Whether these representations are subsequently densified, meshed, or voxelized, they lack explicit semantic and object-level understanding of the mapped spaces. Rather, it has been the goal of additional machinery (i.e., entire areas of research) to subsequently extract meaningful semantics and objects from the estimated point representations. However, if localization and an object-level representation of the world is desired, we set out to answer whether object-level representation itself is sufficient for mapping and localization algorithms.
Can we effectively extract global 3D object-level structure without explicit point clouds or voxel grids? Methods for global 18 5 or multi-view scene-level 3D object detection [25] [9] require building explicit global 3D representation e.g., point cloud or voxel grid. Not only is this an overparameterizatiion of what we want to extract (3D boxes), but requires metric camera poses a priori. However, having access to camera poses implies that great deal of the 3D world has already been solved for. Therefore, we seek an alternative to merely creating more accurate 3D point- based representations from which objects may be extracted and instead seek to make the object the central representation of 3D.
Towards answering these questions, we present framework, Rooms from Motion, which allows for localization and mapping within indoor spaces (i.e., rooms) using 3D objects as the fundamental primitive without the need for boint-based representations, priori poses, or any metric quantities. We do this by constructing object-centric components analogous to existing point-based structure- from-motion principles. Given an unordered collection of RGB images, Rooms from Motion:
1. Independently detects every object within each image in metrically accurate way in the form of oriented 3D bounding box alongside an embedding vector 6..
2. Given two frames, each with set of detected 3D objects, a learned matching network determines object-level correspondences between the frames. Metric relative poses are derived using the set of matched 31) boxes.
3. From the set relative poses, traditional averaging components are used to form a global set of poses 14 15]
4. 3D object tracks are established using the set of all matched 3D object pairs and ultimately produce a global 3D object map.
5. Finally, global 3D objects can be refined using optimization to overcome inherent partial observations of objects.
Rooms from Motion achieves this using both an internal and output representation defined by oriented 3D boxes. No larger representation ever used. Therefore, unlike volume or point- boint-based methods which must explicitly build world-space 3D representation, Rooms from Motion naturally scales to large scenes with a representation that scales proportional to the number of objects in a scene versus the actual geometry of the scene. Since the number of objects in scene is dramatically smaller than any spatial quantity (e.g., voxels, distance fields, radiance fields) while still providing a continuous parametrization of each object, Rooms from Motion provides a compact, yet rich representation for downstream tasks.
2 Related Work
Object-centric Localization and Mapping Using objects as a basis for localization and mapping has appeared throughout the literature. SLAM++ 220 considered the concept of "object-oriented SLAM"" which operates on RGB-D measurements and localizes while mapping a set of pre-defined objects characterized by scanned meshes using a pose-graph optimization framework. Other representations like quadrics [13 27 have also been integrated and can used to optimize for 3D object hypotheses against multi-view consistency derived from 2D object detections. Rooms from Motion, however, is generalist framework capable of mapping any object directly at the 3D box level using class-agnostic 3D object detector and without the need for prior models of the objects to map.

---

Oégiet Dotectin
Ofject Conner Marching
Peliika Seteestion
Absolute Pose Eatimation
Global Obgect Map Extimation
yea
Figure 2: An overview of Rooms from Motion. Two-view geometry is considered first. Semantic objects as 3D boxes are independently detected in individual frames using CuTR (Section [.1]. Objects are subsequently matched at the 3D box level (Sec. 3.2) and at the implicit 3D box corner level (See. 3.3). Relative pose can be estimated using the matched objects and corners between the images as point sets (Sec. 3.4). Averaging operates on the estimated relative poses to produce global camera poses (Sec. 3.5). Given the set of all matched objects, we establish object tracks and use each observation to aid in estimating the global 3D box and semantics for each track. Finally, duplicate tracks are subsequently merged or suppressed. Bottom right: We show lifted 3D box observations belonging to each object track by color as well as the final representative 3D boxes (red) for each track.
3D Object Detection The final map of Rooms from Motion is equivalent to the output of a standard 3D object detector. However, most 3D object detection frameworks operate on acquired aggregated point clouds which are sparsely processed by point/voxel-bar based architectures [18 5 analogous to 2D detection architectures. These methods require access to both pose and depth. Multi-view methods relax the depth requirement but generally lose sparse computation by backprojecting rays into a dense 3D volume. As such, they are significantly limited by the memory required to produce the dense volume and scale poorly to small objects or large scenes, Rooms from Motion, however, computes and stores sparse object-centric features independent of image resolution which allows for limited memory footprint.
119 25
3 Rooms from Motion
Rooms from Motion (RfM) operates on an unordered collection of RGB or RGB-D images with or without a priori poses. We describe the components within our framework in analogous manner to a typical global SfM pipeline [14]: detection, matching, relative pose estimation, global pose estimation, track formation, and global optimization. We provide high-level overview of our method in Figure We refer readers to [6] for specifics on the Cubify Transformer architecture and 14 15 for details related to 4-DoF rotation and translation averaging.
3.1 Detecting 3D objects
Analogous to keypoint detectors 10
17 2] we rely on strong image-based 3D object detector
capable of predicting oriented 3D boxes. Unlike traditional keypoint-based methods we require 3D boxes metrically scaled. Therefore, we adopt Cubify Transformer (CuTR) (6) as our base
3D object detector. Given image I and optionally depth map, CuTR produces 3D object
3

---

detections in the form (,,)) which correspond respectively to 3D oriented boxes, per-object feature embeddings, classification labels, and classification scores, Since CuTR's output is resolution independent, we run CuTR on high resolution images (e.g., 876 x 584 for ScanNet++) which assists in detecting small and far objects.
3.2 Matching 3D objects
Analogous to 2D keypoint matchers 23
91. we design a 3D object matcher. We extend CuTR and apply the general design from LightGlue 9 to objects. Specifically, we consider two images I and Iz. CuTR produce a set of detections in the form of oriented 3D boxes B1 and B3 for each image alongside features Fi and F2 corresponding to the final activations of the query embeddings. We threshold these sets by (0.25 for CA-1M, 0.2 for ScanNet++) using the confidences in S₁ and S2.
21
Given the thresholded objects FT) and F5) we treat the features FI and J as the inputs to a sequence of self and bidirectional-attenn encoding derived from the respective 3D boxes Bi and By, Pairwise object matching scores and a partial object assignment P., are the output of the network. We define the set of matched objects by those matches in the object assignment whose scores are above a threshold (0.5 for CA- and 0.4 for Sca We generally use matching network of 3 blocks. We do not use pruning or early exiting since the number of objects in an image is relatively small. We supervise the matching using the ground-truth object IDs in each dataset. Additional details are provided in the supplementary.
layers from LightGlue [9]. We use positional
3.3 Matching 3D boxes
While it is straightforward to define a coarse object matcher as in Section 3.2 we do not immediately have a set of point correspondences which may be used to derive relative transformation between I, and Iz. While one may implicitly derive point (e.g., the center or corner) from each 3D box to use as correspondence, objects (unlike points) are often partially observed and these implicitly matched points would be unlikely to correspond to the same point in 3D. Therefore, we design an additional matching network the same design as in Section 3.2 but which operates top-down on 3D points of interest derived from the corners of each 3D box. Specifically, we use the network to match two point
sets derived by concatenating the eight box corners C. = [(x] y/, + ..., (x²; y/. ;=²))
} across all
objects in each image into global "box cloud" of 8 [B, queries alongside 3D positional encoding
derived from each corner's 's position and an interpolated 2D image feature at its 2D projection. We
emphasize that this differs from traditional 2D point-level matching [9] since each match in our framework is characterized by a 3D point. We ensure the point-level matches respect the object- level matches by only retaining point-level matches which correspond to the same matched object. Additional details are provided in the supplementary.
3.4 Relative pose estimation and geometric verification
Given the set of matched objects and corresponding matched 3D corners, we estimate relative pose His2 using Kabsch alignment [4] with the matched 3D corners between Iı and Iz serving as point sets. Since the 3D boxes/corners are expected to already be metrically accurate, the translation components of R122 are also expected to be metric. We treat this as a 4-DoF alignment (yaw and translation) and use circular regression [15] since CuTR already assumes the existence of gravity measurements that are readily available on commodity devices.
Some object and corner matches may be incorrect, therefore, estimate relative poses across the matched objects using a minimal set of 2 matched objects. We exhaustively test these samples since the number of matched objects between two frames is generally small. For each minimal set, we do not perform any sampling on the matched corners. For each matched object pair (B), m BB), we compute a box-based matching error as - LOU3D(B12 m. B²) where B32 corresponds to the reprojection of box B from image I to Iz under the estimated R22. We consider matched object to be an inlier match if the underlying box reprojection error is less than 0.75 (i.e., IoU3D 2 0.25). We consider a sample to be an inlier if each box in the sample is an inlier under R22 and the ratio of inliers when considering the full set of matched objects between /, and // under Ri, is at least 0.5. We take the inlier sample (if any) with lowest mean matching error to produce a verified relative pose R12 Additionally, we can compute a corner matching error by taking the 1,2 distance of the reprojection of the matched corners of each box. We define matched corner to be an inlier match if

---

Figure 3: Bundle adjustment can be extended to global 3D object track. Each observation of track characterized by an image and particular 3D box detection from CuTR. Since these individual 3D boxes (shown as thin, white boxes) are usually aligned well with the RGB image, they can be used enforce a reprojection cost function by comparing the projected corners of these detections to the projected corners of the object track's representative 3D box. The point tracks established in Section 3.3 (shown in 8 colors) allow us to map the corners of these individual 3D boxes to the corners of the track's s global 3D box. In the rightmost pane, we show an initial mismatch of the projected corners from the individual detection (brighter color) and the object track (darker color) that may be minimized by optimizing the track's 3D box parameters.
its reprojection under 12 is less than 10 cm from the matched corner in 1z. Corner inliers are used solely within global optimization (Section 3.6
3.5 Absolute pose, object track establishment, and merging
Averaging and filtering We run CuTR on every image and Cubify Match between each pair of images I; and to estimate relative poses Rij and produce view graph G consisting of images I; and geometrically verified R.j relative poses as edges. We use glomap [4] as a guide to perform rotation and translation averaging. Therefore, outlier relative pose estimates which have disagreement with the estimated relative rotations and the relative rotation after averaging by more than 3 degrees. For translation averaging, we fix the scale parameter since we desire metric scale. We run three rounds of translation averaging and remove outlier relative pose estimates whose relative translations disagree by more than 10 with the relative translations derived after averaging. After averaging and filtering, we have a global pose RT;; for each image I; within the largest connected component of G.
run three rounds of rotation averaging and eliminate
Object track establishment While a traditional SfM pipeline would form point tracks, we initially form object tracks. Each object track Oj is characterized by set of (I; B:) pairs which correspond to an observation of that object in image i as 3D box B,. Each object track is formed by taking the union-find across all inlier object matches remaining after the averaging process. Now, given an object track O, with a series of observations (L, B,), we can lift each B; into the global frame using RT; We then must define representative 3D box B, for the track. While many possible methods exist, we choose a simple initialization based on the box B, which has the highest geometric mean of average mutual 3D IoU with all other 3D boxes in the track and underlying detection score. For experiments where semantic classes are available (e.g., ScanNet++ (28)) ) we produce class distribution weighted by the individual classification scores of each observation and use the class with the maximum probability the representative semantic label for the track. Finally, we include details on track merging the supplementary.
3.6 (Partial) Bundle Adjustment: Object track optimization
Global optimization can further aid in the accuracy of global 3D boxes given the partial or occluded observation that naturally happens within images. We rely on global optimization to improve map quality in the scenario where we have access to pose but lack other metric information (i.e., we use monocular RGB images). Global optimization allows us to optimize the reprojection error of the representative 3D box B, to each observation (I, B.). Since optimizing the 3D box as a volume (rendering) expensive, we choose to use the 3D box corners to establish point tracks analogous to the explicit point tracks in traditional SIM. Using similar procedure to establishing object tracks,
5

---

we consider the union- mnoon-find across the 3D corner inlier matches from the representative box B, to 3D corners for all other box observations in the object track. This produces point tracks of the form (n) = {Ta,, B5, m) where < n. < 8 corresponding to which corner of the original image-based box B; corresponds to corner n of the representative box B,. We differentiably optimize the underlying 3D box through cost function that considers the projection of each corner's point track to each observation using PyCeres [.. We visualize an example of a single object track problem across three views in Figure 3 Additional details are provided in the supplementary.
4 Experiments
We evaluate Rooms from Motion on its mapping (i.c., 3D object detection) and metric localization capabilities. We conduct experiments on two large-scale datasets: CA-IM [6] and ScanNet++ [28). CA-IM, while class-agnostic, contains exhaustive labeling of objects as oriented 3D box across thousands of captures. This allows us to explore Rooms from Motion's 's ability to localize and map any object regardless of semantics. Additionally, CA- 1M has multiple captures of the same room, allowing us to study relocalization. We use ScanNet++ (IK) to study semantic classification using its 84-class instance taxonomy. Unlike CA-1M, 3D boxes are not explicitly annotated and axis- aligned boxes are derived from provided instance segmentation. Additionally, the DSLR captures of ScanNet++ allow us the opportunity to study the behavior of collections that are not captured from a continuous trajectory. Both datasets have access to highly accurate ground-truth poses and ground -tund-truth depth. Therefore, we perform experiments in a few settings:
1. Ground-truth depth and poses: This explores a pure mapping variant of Rooms from Motion where does not need to estimate camera poses and where Cubify Transformer has access to ground-truth depth from RGB-D images. This is directly comparable to the most common setting of 3D object detection where ground-truth point cloud is the input to a sparse 3D object detector [18 5
2. Ground-truth depth alone: This explores a localization and mapping variant where the metric pose must be estimated, however, Cubify Transformer has access to strong metric information through ground-truth depth maps. This explores the setting where depth may be available, however, running a distinct localization pipeline may be undesirable.
3. Monocular RGB and ground-truth pose: This also considers the pure mapping variant of Rooms from Motion where Cubify Transformer only has access to monocular RGB images (i.e., without scale information) and must recover scale information. This is directly comparable to multi-view 3D object detection methods [19 25
4. Monocular RGB alone: This is the most extreme setting where only RGB images are available. Rooms from Motion must estimate metrically accurate camera poses and 3D box predictions purely from RGB images.
Inference settings While point and volume- based methods are feedforward methods which run once on an aggregated point cloud, RfM runs Cubify Transformer [6] on the set of provided frames in batched manner and subsequently performs matching all captures into 100 frames. Rooms from Motion is oblivious to temporal aspects (e.g.. locality) of this sampling. For CA-IM, we evaluate on the unique set of 3D objects observed across the ground-truth in all sampled frames i.c., we do not consider objects in the scene that were not observed at least one frame. However, for ScanNet++, we always evaluate on the full set of 3D boxes since each capture corresponds to specifically labeled mesh. This advantages the point-based methods over RfM since there are no guarantees that the DSLR capture observes all such objects. We do not evaluate on walls, ceilings, or floor objects.
all pairs of frames. We uniformly sample
Runtime RfM takes approximately 65 ms to process each frame through Cubify Transformer at resolution of 640 x 480 resolution on a V100 GPU, although this part can be batched. For CA-1M, it takes average of 700 ms to insert a new frame (perform matching and compute relative pose estimation to all other frames) into the view graph for an input of 100 frames. Lastly, finalizing the graph (i.e.. performing averaging) takes approximately 10 seconds.
Un-posed alignment When evaluating camera poses or 3D object detection results, we align all un-posed results by pure rotation and translation (i.c., SE(()) using the traditional alignment of the

---

positions (or rotations) of each set of camera poses, No scaling is used except for when comparing to Colmap [22]. which produces results with scale ambiguity.
4.1 Mapping: 3D Object Detection
Table We evaluate the mapping ability of Rooms from Motion against state-of-the-art, point-based 3D object detection methods. All methods use RGB images and ground-truth depth and are limited 1000 detections. Only Rooms from Motion can produce 3D maps without priori poses.
CA-1M
ScanNet++
Method
AR15
AP25
AR25
AP15
ARI5
AP25
AR25
Global, salame-based methods
FCAF (Posed)
28.7
32.5
28.9
28.6
26.8
UniDet3D (Posed)
S
33.3
48.1
31.6
46.0
Sequentiol, image- bused methuds
Rooms from Motion (Un-posed)
39.6
47.2
30.8
38.7
33.4
42.9
29.6
38.0
Rooms from Motion (Posed)
47.4
53.8
38,9
45.6
38.8
35.6
45.0
Table 2:
We evaluate the mapping ability of Rooms from Motion against multi-view 3D object detection methods. All methods use only RGB images.
CA-IM
ScanNet++
Method
API5
ARIS
AP25
AR25
API5
ARIS
AP25
AR25
Global, volume-based methods
ImVoxelNet (Posed) 19
13.2
18.8
8.3
12.9
17.7
30.1
15.1
26.8
ImGeoNet (Posed)
18.6
23.1
13.7
17.8
18.8
31.3
16.9
27.8
Sequenrial, image-based methods
Rooms from Motion (Un- posed)
10.7
13.6
Rooms from Motion (Posed)
24.6
34.3
17.7
26.1
27.5
36.1
23.1
30.3
Reoms from Motion (Posed BA)
31.3
40.0
23.2
31.3
30.5
39.3
25.6
33.4
We present the main 3D object detection results in Tables I and 2 We use the standard average precision (AP) and average recall (AR) metrics at IoU 3D levels of 15% and 25%. For CA-IM, the metries are class-agnostic. For ScanNet+++, the metrics are averaged across the standard instance classes. Overall, our results support that Rooms from Motion is a strong 3D object detector in wide range of configurations:
Ground-truth depth and poses: In Table I we compare to point- based baselines FCAF 18) on both CA-IM and ScanNet++ where ground-trum depth and poses are used aggregate the requisite input point cloud. On ScanNet++, where meshes are available, we additionally compare to UniDet3D [5] which relies on superpoints. Rooms from Motion significantly outperforms FCAF in this setting despite it operating at 1 em voxel resolution. Even the unposed variant of RfM is capable of outperforming FCAF. Since CA-1M consists of many small objects and ground truth depth may not always be complete due to LiDAR scanner artifacts, we believe that the image-centric nature of RfM allows for significantly improved detection abilities. On ScanNet++, Rooms from Motions achieves significantly improved precision metrics over UniDet31 and FCAF, not only validating its detection capabilitics, but also validating its semantic classification quality. While the point- based methods show better recall, especially FCAF which computes 3D box for every position in the sparse voxel grid, this comes at a cost of decreased precision. Furthermore, we again see that even an unposed variant of RIM can outperform FCAF in average precision.
Monocular RGB and poses: In Table 2 we compare to multi-view baselines VovolelNet [9] and ImGeoNet (25). These methods only rely on RGB images and ground-truth pose. However, in the absence depth, they build dense feature volumes and run dense 3D convolutions in order to process the scene in single pass. Therefore, this also inherently limits them with respect to GPU memory and even on 32 GB V100, they can only use 45 linearly sampled views and voxel size of cm. This may, in part, explain the significant gaps in all metrics between them and Rooms from Motion. While
7

---

the unposed variant of RfM is competitive, the posed variant has greatly improved precision and recall across both CA- 1M and ScanNet++. Furthermore, optimizing the object tracks through partial bundle adjustment (BA) from Section 3.6 significantly boosts map quality and leads to an overall 60-70% improvement across all metries compared to ImGeoNet. Not only the advantage of using pixely rather than voxels evident from our results, but also compelling is the ability to optimize object tracks with respect to per-framn fermframe observations rather than relying on the aggregation of inputs (i.c., points) that has been historically done in the 3D object detection literature.
4.2 Localization: Camera Pose Estimation
While the previous section considered the mapping performance of Rooms from Motion, under- standing the camera pose estimation performance allows us to understand the suitability of box primitives for localization. At the same time, good localization is a prerequisite good mapping. We calibrate the performance of Rooms from Motion in both the RGB and RGB-D settings on CA-IM and S in Table 3 While CA-1M captures correspond to a smooth trajectory recorded handheld device, the DSLR captures from ScanNet++ may not be smooth and can be challenging for traditional SLAM systems. We emphasize large-scale evaluation (50 sequences for ScanNet++ and 107 for CA-IM) rather than individual sequences. For RGB-D, we compare against DROID-SLAN 24 and for RGB, we compare against CUT3R [26] (not to be confused with CuTR). CUT3R is particularly suitable since it maintains metric predictions using Mast3r 7 which is not always the case
For Rooms from Motion and CUT3R, we subsample uniformly to 100 frames. For DROID-SLAM, we consider denser subsampling at 250 and 500 frames for fairer comparison and this often exceeds the available frames in ScanNet++ DSLR captures. We find -LB- [[]] unable to provide reliable tracking and therefore do not compare to it. Since Rooms from Motion only registers the largest connected component of its view graph, we make note of how many frames were, on average, registered per sequence.
Table 3:
We evaluate the localization ability of Rooms from Motion. We report both the averaged median error and (averaged RMSE error) for rotation and translation as well as the percentage of images that are successfully registered in the global maps. Ground -ruth alignments are done with an S SF)) transformation (i.e., no scaling) except for Colmap. We note that Rooms from Motion has access to local camera pitch and roll through gravity measurements.
CA-IM
ScanNet++
Method
ARE (dog)
ATE (cm)
Registered
ARE (deg)
ATE (em)
Registered
DROID SLAM
21
(250 frames, w/ GT depth)
10.1 (14.5)
14.9 (22.5)
100%
62.8 (81.0)
110.1 (158.8)
1005
BLAI 24 (500 frames, w/ GT depth)
2.6 (6.1)
4.7 (7.8)
100%
58.2 (79.3)
104.7 (151.2)
100%
Rooms from Motion (w/ GT depth)
1.8 (4.33
4.0 (8.0)
96%
1.1 (2.1)
(6.7)
99.5%
CUT3R (RGB) 26
10.7 (12.6)
35,9 (45.6)
(11.)
27.2 (36.6)
100%
Mast3r (RGB)
24.5 (29.0)
100%
25.8 (31.7)
1005
Colmap (RGB, scaled)
18.5 (50.1)
38.5 (68.0)
100%
9.2 (33.3)
28.8 (61.0)
100%
Rooms from Motion (RGB)
2.5 (5.8)
12.7 (17.9)
93%
(5.I.
15.3 (21.7)
97%
CA-1M Both Rooms from Motion and DROID-SLAM perform well with handheld RGB-D captures on CM and are able to achieve similar median trajectory error of 4.0 vs. 4.7 cm. With respect to RGB only, CUT3R generally results in significantly worse rotation and translation errors than Rooms from Motion. Surprisingly, Rooms from Motion on RGB (i.e., no depth) is competitive with RGB-D DROID-SLAM running on 250 frames.
ScanNet++ The DSLR captures of ScanNet++ are sparse (usually a few hundred frames) and may not correspond to smooth trajectory. For SLAM- based methods like DROID-SLAM (24). we can observe degraded performance that likely can be attributed to this despite having access to GT depth. On the other hand, the RGB-D variant of Rooms from Motion is able to register nearly every frame while maintaining a median rotation error of nearly a single degree and 3.5 cm of median translation error. With respect to RGB only, Rooms from Motion continues to significantly outperform CUT3R.
8

---

Rooms from Motion is competent at metric camera localization. Both DROID-SLAM and CUT3R rely on point-based primitives for localization. Despite relying on 3D box primitives alone, Rooms from Motion registers nearly all frames and provides superior localization ability within the indoor setting alongside sparse captures. We believe this supports the suitability of 3D box primitives as an alternative to point-based approaches for localization.
4.3
Ablation: Do more objects help for localization?
The ability for Rooms from Motion to accurately estimate relative poses between image pairs is inherently limited by the matching capabilities of Cubify Match, which we expect to be highly dependent on number of objects that can be detected by Cubify Transformer. Therefore. in Table 4 we ablate the performance of Rooms from Motion with respect to completeness of the object labels from which Cubify Transformer is trained. For CA-1M, we consider subset of the object labels corresponding to: 18 types of objects corresponding to ARKitScenes classes versus the exhaustive CA-IM set of objects. For ScanNet++, we consider the standard 84-class instance taxonomy as well as expanded taxonomy where we add all remaining instances as "unlabeled".
We observe that coarse laxonomy like ARKitScenes taxonomy consisting of 18 large object classes) appears to be insufficient. Not only are the rotation and translation emrors significantly higher than when using CA-IM, but the registration rate significantly drops. When trained on the full set of objects included in CA-IM, more objects are detected per- per-frame and this likely increases the chances of matching more objects between frames - critical for accuracy and connectivity of the view graph.
On ScanNet++, for which the standard taxonomy is 84 types of objects, we observe reasonably good performance. However, observe improvements in the registration rate (from 93% 97%) when using all available objects. Notably, since these remaining objects are added as "unlabeled" rather than their specific labels, we see both with ScanNet++ and CA-IM that the localization within Rooms from Motion is not sensitive to having the actual class label of an object.
Table 4: Taxonomy sensitivity of Rooms from Motion's localization performance with respect to the taxonomy of objects detected by the underlying Cubify Transformer.
Taxonomy
ARE (deg)
ATE (em)
Registered
ScanNet++ (Standard)
2.5
15.3
93%
ScanNet++ (AII)
2.4
15.3
97%
ARKitScones
4.4
16.2
61%
CA-IN
2.5
12.7
93%
4.4 Qualitative Map Results
We present qualitative 3D object maps across all methods in Figure 4 The 3D object maps projected (not rendered) and overlaid onto a panorama with respect to a FARO stationary scanner position as provided by both CA-1M and ScanNet++. For CA-1M, this is an RGB panorama captured by the scanner software itself, whereas for ScanNet++, this is rendered from the provided mesh. For understanding spatial accuracy, which this projection does necessarily capture, we refer the reader to the quantitative results of Section I For brevity, we only present unposed results of Rooms from Motion whereas all other methods presented are posed.
Rooms from Motion appears to provide the most visually complete maps. Small, thin objects like ceiling lamps and objects on the walls are more faithfully captured within RfM maps. Despite being unposed and viewed from a "novel" position (the FARO scanner), we also observe satisfactory RGB alignment as the 3D boxes generally project to the expected object boundaries in the panorama. Additionally, we see that multi-view methods like ImGeoNet particularly struggle when it comes to detecting smaller objects and providing accurate 3D box orientation.

---

IRM De-pooee
FCA Roaid RGB-D
poscd.
Figure 4: Qualitative comparisons from CA-IM (top half) and ScanNet++ (bottom half) of resulting 3D object maps across different methods as projected the viewpoint FARO scanner. We note that RfM results here unposed (i.e., using aligned, estimated pose), while the 3D object detection-bast methods rely on priori pose.
5 Limitations and Discussion
We introduced framework which is capable of producing global 3D box detections (i.e.. maps) of indoor spaces while simultaneously able to estimate camera poses from unordered RGB(-D) collections. This allows new settings e.g., 3D object detection from un-posed RGB images that may enable more general 3D object mapping of the world.
Our method is currently limited to indoor scenes by a lack of training data of more general indoor and outdoor scenes. At the same time, we can only register images which have objects in them. Similarly, more diverse cameras would facilitate the generalization of CuTR and RiM.
Acknowledgements
We thank Shih- Yu Sun and Sam Parsa for motivating discussions on the intersection between 3D object detection and SLAM. We thank David Griffiths for feedback on the manuseript and helpful discussions. Selected visualizations produced with Rerun [16]
10

---

References
Agarwal, S., Mierle, K., Team, T.C.S.: Ceres Solver (10 2023), http::///ggttb...oo////eeee////e colve-
DeTone, D., Malisiewicz, T., Rabinovich, A.: Superpoint: Self-supervised interest point detection and description. Proceedings the IEEE conference on computer vision and pattern recognition workshops. pp. 224 236 (2018)
[3] Duisterhof, B., Zust, L., Weinzaepfel. P., Leroy, V., Cabon, Y., Revaud, J.: Mastôr-sfm; a iully-integrated solution for unconstrained cuucture-frommmotiie arXiv preprint arXiv:2409.1913. (2024)
[4] Kabsch, W.: A solution for the best rotation to relate two sets of vectors. Foundations of Crystallography 32(5), 922-923 (1976)
[5] Kolodiazhnyi, M.. Vorontsova, A., Skripkin, M., Rukhovich, D., Konushin, A.: Unidet3d: Multi-dataset indoor 3d object detection. arXiv preprint 2Xix:2409.042344 (2024)
[6] Lazarow, J., Griffiths, D., Kohavi, G., Crespo, F., Dehghan, A.: Cubify anything: Scaling indoor 3d ohject detection. arXiv preprint awxiv:2412.0445555 (2024)
[7] Leroy, V., Cabon, Y., Revaud, J.: Grounding image matching in 3d with mastär. In: European Conference Computer Vision. pp. 71 91. Springer (2024)
[8] Li, K., DeTone, D.. Chen, Y.F.S., Vo, M., Reid. I., Rezatofighi, H.. Sweeney, C., Straub, J., Newcombe,
R.: Odam: Object detection, association, and mapping using posed rgb video. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5998 0008-6008000 (2021)
[9] Lindenberger, P., Sarlin, P.F.,, Pollefeys, M.: Lightglue: Local feature matching at light speed. In: Praceed- ings the IERE/CVF International Conference on Computer Vision. pp. 17627-17638 (2023)
[10] Lowe, D.G.: Distinctive image features from scale-invariant kcypoints. International journal of computer vision 60, 91-110 (2004)
[11] Mur-Artal, R., Tardós, J.D.: Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE transactions on robotics 33(5), 1255-1262 (2017)
[12] Murai, R., Dexheimer, E., Davison, A.J.; Mast3r-slam: Real-time dense slam with reconstruction priors. arXiv preprint arXiv:2412. (2024)
[13] Nicholson, I.,, Milford, M., Sünderhauf, N.: Quadricslam: Dual quadries from object detections as
landmarks in object-oriented slam. IEEE Robotics and Automation Letters 4(1), 1-8 (2018)
[14] Pan, L., Baráth, D., Pollefeys, M., Schönberger, J.L.: Global structure- froo-mmotinn revisited. In: European Conference on Computer Vision. 58 77. Springer (2024)
[15] Pan, L., Pollefeys, M., Baráth, D.: Gravity-aligned rotation averaging with circular regression. In: European Conference Computer Vision. pp. 16. Springer (2024)
[16] Rerun Development Team: Rerun: A visualization sdk for multimodal data (2024), https: https://uww/ rerun io available from https//wwww.rerun.i/ in/ and https://gihuub.oo/rrrun--iie
[17] Rublee, E., Rabaud, V., Konolige, K., Bradski, G.: Orb: An efficient alternative to sift or surf. In: 201 Icce (2011)
International conference computer vision. pp.
[18] Rukhovich, D., Vorontsova, A., Konushin, A.: Feaf3d: Fully convolutional anchor-free 3d object detection. In: European Conference Computer Vision. pp. 477-493. Springer (2022)
[19] Rukhovich, D., Vorontsova, A., Konushin, A.: Imvoxelnet: Image voxels projection for monocular and
multi-view general-purpose 3d object detection. In: Proceedings of the IEEE/CVE Winter Conference Applications Computer Vision, pp. 2397-2406 (2022)
[20] Salas-Moreno, R.F., Newcombe, R.A., Strasdat, H., Kelly, P.H.,, Davison, A.J.: Slam++: Simultaneous localisation and mapping at the level of objects. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 1352-1359 (2013)
[21] Sarlin, P.E.,, DeTone, D., Malisiewicz, T., Rabinovich, A.: Superglue: Learning feature matching with graph neural networks. In: Proceedings of the IEEE/CVF conference computer vision and pattern recognition. pp. 4938 4947 (2020)

---

[22] Schönberger, J.L., Frahm, J.M.: Prructure-frommmone revisited. In: Conference on Computer Vision and Pattern Recognition (CVPR) (2016)
[23] Sun, Shen, Z., Wang, Y., Bao, H., Zhou, X.: Loftr: Detector-free local feature matching with transformers. In: Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 8922-8922 8931 (2021)
[24] Teed, Z., Deng. J.: Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems 34, 16558 16569 (2021)
[25] Tu, T., Chuang, S.P., Liu, Y..., Sun, C.,, Zhang. K., Roy, D.. Kuo, C.H., Sun, M.: Imgeonet: Image-induced geometry-awe aware voxel representation multi-view 3d object detection. Proceedings of IEEE/CVF International Conference on Computer Vision. pp. 6996-7007 (2023)
[26] Wang", Q. Zhang*, Y., Holynski, A., Efros, A.A., Kanazawa, Continuous 3d perception model with persistent state. In: CVPR (2025)
[27] Wu, Zhang, Y., Zhu, D.. Deng, Z., Sun, W., Chen, X., Zhang, J.: object slam framework for association, mapping, and high-level tasks. IEEE Transactions on Robotics 39(4), 2912-2932 (2023)
[28] Yeshwanth, Liu. Y.C., Nießner, M., Dai, A.: Scannet++: high-fidelity dataset of indoor scenes, In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 12-22 (2023)
12

---

A Appendix
A.1 Additional Experiments
A.1.1 Relocalization
We additionally examine the basic ability of Rooms from Motion relocalize given map. Using CA-IM, which includes 3 distinct captures of each room, we run Rooms from Motion on one of the captures (107 total), and store each registered camera pose alongside its corresponding CuTR detections. Given an additional capture of the same room (222 total), we attempt to relocalize each new image. We run Cubify Match on the new image against the previously registered images. If successful, we use the registered camera pose and estimated relative pose of the image with the most object inliers as the camera pose the new image. Only single registered image is selected no averaging across relative pose estimates is applied.
Table 5: Relocalization performance of Rooms from Motion on CA-IM. Given an initial map created from capture of room, we relocalize additional captures (i.e., different trajectories within the same rom) with respect to the initial map.
Input
ARE (deg)
ATE (cm)
Relocalized
RGB
3.1 (6.8)
15.9 (24.4)
98%
RGB-D
2.6 (4.8)
6.0 (71.2)
99%
We observe high recall with respect to relocalization registration and achieve similar accuracy in rotation and translation errors on par with our full localization experiments in Table 3 Further work remains to understand how to update the mapped set of objects given additional images.
A.1.2 Ablations
We additional ablate how field of view impacts the ability detect more objects and match more objects between frames.
Table 6: Field of view sensitivity of Rooms from Motion is examined using both the standard FoV images of each dataset versus the additional high FoV captures available in each dataset.
view
ARE (deg)
ATE (cm)
Registered
ScanNet++ (Standard)
6.1
31.0
87%
ScanNet++ (DSLR)
2.4
15.3
97%
CA-IM (Standard)
7.9
26.1
68%
CA-IM (Ultrawide)
2.5
12.7
93%
Does field of view matter? All experiments so far use images that correspond to the higher field of view (FoV) images present in CA-1M and ScanNet++. We validate the hypothesis that larger field view corresponds to better localization in Table 6 All localization metrics are significantly improved across each dataset alongside large increases in the registration success rate.
B Additional Model Details
We provide additional details on our models and setup.
B.1
General
All models use the ViT-B variant of Cubify Transformer. Cubify Match composed of two matching
networks of the LightGlue design [9] each consisting of 3 selfibi-directional attention layers using 256-dimensional embedding dimension. All models are trained in Py PyTorch using batch size of 64 over 4 x 32GB V100s for the standard schedule from Cubify Transformer [6].
13

---

B.2
Cubify Match
B.2.1
More details
While we generally follow the design of LightGlue and apply it to object (boxes) and points, we more explicitly describe the function.
Object Matching Network Given a pair of images, we run Cubify Transformer on each image independently. For CA-IM, we threshold detections by 0.25 (since CA-IM is class agnostic) and for ScanNet++, we threshold by 0.20 (since ScanNet++ is class specific). Each detection includes a 256 dimensional embedding corresponding to the (final) query embedding. Each detection's 's 3D box in camera space, we use a small MLP to produce a positional encoding to XYZ center and WILH dimensions (we do not encode the angle). The resulting sets (one for each image) of query embeddings and positional encodings are fed into 3 blocks consisting of self attention and bi-directional attention [9]. No early exiting/pruning is used, which keeps the network quite simple. Finally, match assignment must be produced using double softmax which corresponds to an (N + 1) x (M+1) + similarity matrix of scores where N is the number of thresholded detections in the first image, and M is the number in the second image. The additional row and column correspond to the matchability score which modulates the entire row/column. We use the same filtering code in order to produce the final matches (if any).
Box/Corner Matching Network Given the thresholded detections, we project the corners of each 3D box to its respective image to produce 8 (x. y) positions for each box. We bilinearly interpolate the output features of the VIT backbone from Cubify Transformer. We add this to the final query embedding of the corresponding object. We embed the 3D X positions using an MI.P to act positional embedding. The resulting set of flattened inputs (i.e., the inputs the embeedingss/oositinnal encodings of the flattened corners across all detections). This essentially gives two "box clouds" (i.e., N 8 points M x 8 points based the aggregate of the box corners) as inputs into the matching network. These are fed into 3 blocks consisting of self attention and bi-directional attention 99. A match assignment is produced similarly as for the object network using the same thresholds. Given the assignment of corners, we reject assignments of corners which do not correspond to the set of matches from the object matching network.
B.2.2 Track merging
Despite the track establishment process, we may find two object tracks which correspond to the same underlying object but are not connected during establishment when no inlier path between the observations can be established. However, given the existence of a global set of estimated poses, we can consider a merging and suppression procedure. We enumerate all pairs of established object tracks (O;, O,) and ignore pairs whose generalized IoU 3D is low (smaller than -0.6). Then, we compute the track affinity between the tracks by considering the average observation affinity between all pairs of observations between the tracks. We define the observation affinity as the product of the object matching score and the generalized IoU 3D of the corresponding representative 3D boxes (shifted by 1.0 to ensure non-negative values). We merge track when the track affinity exceeds 0.25. We suppress a track when the track does not meet the affinity threshold for merging, but the IoU 3D between the tracks is above 0.15.
B.2.3 (Partial) Bundle Adjustment
Bundle adjustment relies on building point tracks based on the 3D box corners of object tracks. Subsequently, we can define a simple cost function that rather than optimizing the 3D point positions independently, optimizes the underlying box parameters (center, dimensions, yaw) differentiably through the corners. This can be seen in the following code:
Listing I: Bundle adjustment cost
class ReprojectionnEroor (obtretrrobbect))) (ojject):
def unitt_(selff size, K, idx, xy): # Image size.
self size = torch. tensor (size) ). to (device=xy. device, dtype=torch float64) # Intrinsics.
14

---

self.K K = K
# 3D corner to be projected. self = idx # Expected 2D projection. sel xy
# Box parameters and camera pose.
def -_
center, dims, yaw, rq T):
R quaternion_ tottttotootttonotot matrix (rq [None ])[0]
box_xyz = box to corners (center, dims, yaw) [ self idx ] box_xy self K @ (R @ box_xyz _ox_xyz= T)
box_xy = box_xy [:2] box_xy[2:]
return (box_xy - self xy) / self size
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We believe all claims are supported by the paper. Our quantitative results for detection and localization, in particular, use standard measures of performance which support the claims made in the abstract and introduction.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the (known) limitations of our work with respect to setting (indoors) and the presence of objects in images. We perform relatively large evaluations for mapping and localization (over 150 sequences) which support the applicability across entire datasets rather than individual ones. We believe this meets the general standards of evaluation the relevant subfields.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our work is entirely empirical and make theoretical claims.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yes]
As much is possible, the combination of our main paper and supplementary include relevant details (especially hyperparameters) We reference the relevant components that make up individual choices. All data used is publicly available. Given that system relies on many complex components, we intend to release the code and models to ensure full verification of results by the community.
Justification:
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We link repository which contains our models and inference code.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training and test details are explained throughout the main paper and supple- mentary.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
16

---

We provide wide range of experiments (model variant, dataset) which we believe reduces the likelihood any results being due to particular initialization or factors. The total validation sets include 157 captures, each run on 100 images, which corresponds to nearly 15,700 images. We do not find the variance of our model is particularly significant across runs.
Justification:
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We include the compute resources used for all experiments in the supplemen-
tary.
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We believe our submission conforms the NeurlPS Code of Ethics. All data involved takes particular steps (e.g., anonymization) towards privacy and no human subjects were involved. The data used corresponds to actual captures done with mobilc/handheld devices by the respective dataset owners. Finally, we do not believe our work has a significant likelihood of negative societal impact.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We believe work will have neither positive negative societal impact. Our model has no generative abilities, relation to text, and is unlikely to be useful in surveillance given that it outputs (3D boxes) retain little information about the images they are derived from. Our work is also unlikely to be useful for military applications given these limitations.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our models are only capable of detecting inanimate objects as 3D boxes and believe they pose minimal risk to the community and world.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets (datasets our case) used are properly cited and are available for academic (research) use.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We do release any assets part of this submission.
14. Crowdsourcing and research with human subjects
17

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No crowdsourcing or research with human subjects was conducted as part of this submission.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Question:
Answer: [NA]
Justification: crowdsoureing or research with human subjects was conducted as part of this submission.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: Our work is neither related to LLMs nor does any aspect of it (e.g., writing) LLMs.
18


Reviews:
[{"review_id": "BRIpItwvOY", "reviewer": "Reviewer_n63D", "summary": "In this paper, authors proposed rooms from motion -- a mapping a localization framework that is grounded around objects instead of traditional points.  Authors uses off-the-shelf 3D bounding box detector, match it across the images, and then run \"object corner point\" based matching to get relative pose between two images. After that utilizing all structure-from-motion tricks, optimized poses and map can be retrieved. Unlike traditional point based map, this map is grounded in objects and is less heavy. Authors also show that their method can work in pose + RGB-D, RGB-D, pose + RGB, and RGB only settings. \nThrough experiments on CA-1M and ScanNet++ datasets authors show that their method beats other baselines in respective categories.", "questions": "- Table-1, FCAF is cited wrongly\n- Can authors report mAP, similar to other baseline methods? (e.g. UniDet3D report mAP)\n- Why do we only compare with CUT3R in RGB pose estimation baseline (Table 3)? This is the place where authors need to prove their localization can achieve better results than point based methods (COLMAP[1] , MASt3R-SLAM, VGGT)\n\n\n [1] Johannes L. Schönberger. “Robust Methods for Accurate and Efficient 3D Modeling from Unstructured Imagery.” ETH Zürich, 2018.\n[2] Murai R, Dexheimer E, Davison AJ. MASt3R-SLAM: Real-time dense SLAM with 3D reconstruction priors. InProceedings of the Computer Vision and Pattern Recognition Conference 2025 (pp. 16695-16705).\n[3] Wang J, Chen M, Karaev N, Vedaldi A, Rupprecht C, Novotny D. Vggt: Visual geometry grounded transformer. InProceedings of the Computer Vision and Pattern Recognition Conference 2025 (pp. 5294-5306).", "limitations": "Authors have discussed limitations of their work in sufficient detail.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "- Authors use an off-the-shelf 3D bounding box detector (Cubify Transformer - CuTR) and base their pipeline on top of this. While this is interesting, there's limited originality. Especially, authors have just replicated structure-from-motion pipeline for objects instead of points. Even if we consider this as innovation, the relative pose estimation still rely on point based method. Authors uses corners of detected 3D bounding boxes and project them to 2d, and then matches among these points. So, relative pose still comes from points and not objects as claimed in the paper.\n\n- Another aspect this paper is missing is that they are using off-the-shelf 3d object detector. This is fine, but we can also just generate dense scenes usign off-the-shelf 3D reconstruction methods such as VGGT and then run point cloud based 3D object detection. The presented method might perform better, but it needs to be compared with others. \n\n- I also have concerns about evaluation. The paper needs to compare with SOTA baselines in respective areas and not just ANY baseline. For example, comparison with point-based localization and mapping is completely missing. I know that DROID-SLAM is point based, but its SLAM, what authors are presenting is SfM. Also, in RGB only methods, it uses CUT3R as a baseline, while much better baselines do exist (e.g. MASt3R). \n\n- Regarding evaluation, I think authors should follow standards in respective eval. For example, for 3D objects mapping -- its common to report mAP at various threshold (not just AP and AR). Similarly, for pose estimation its common to report percentage of image localized within different error thresholds. I would highly encourage authors to read relevant baselines and follow their standards.\n\n- Overall, I believe this is an interesting application of structure-from-motion for objects, but there's little additional insights provided by authors that adds value to NeurIPS readers.", "was_revised": true, "final_justification": "I read authors rebuttal and other reviewers response. Based on all the discussion I agreed that paper does a bit more than just using objects instead of points. Authors have been responsive answering my questions. Based all all evidence I increased my rating from reject to weak accept.", "rebuttal": "**limited originality**: We believe that our work has significant originality and is more than \"replicated structure-from-motion pipeline for objects instead of points\". We believe this underestimates the difficulty of making such a framework work in a general and large-scale manner. As far as we know, no previous work has considered a learned _object centric_ matching as a basis for estimating relative pose. Similarly, no work proves that consistent 3D object tracks can be established _without a priori pose_, as we do in this work. Furthermore, we introduce early insights into the idea of global optimization (bundle adjustment) on 3D boxes, which has not been explored in previous work, as far as we know. Instead, previous work generally uses 2D keypoints to detect features of interest with little semantic value. We show that 3D objects (as detected from images) are themselves a sufficient primitive that can be matched across images. Notably, this matching uses _metric 3D points_ derived from the set of detected objects (using the corners in a metric 3D space). Furthermore, this matching is enforced to respect the object-level matching and has high semantic value. We emphasize that this matching is _not_ done using 2D pixels/points, instead, we only show the projections in Figure 2 as a means of visualization.\n\n**we can also just generate dense scenes using off-the-shelf 3D reconstruction methods such as VGGT**. While it is true that methods like VGG-T/Mast3r-SLAM/colmap are exciting for dense reconstruction, methods like VGG-T/Mast3r-SLAM/colmap output _normalized_ reconstructions (i.e., up to a scale factor). This is fundamentally different from our approach which outputs _metric_ camera poses and 3D boxes (expressed in real-world meters). This is a fundamental requirement for a 3D object detector --- it must produce distances and dimensions in a real-world scale in order to be useful. Additionally, one important aim of our work is to show the _plausibility_ of using objects (as 3D boxes) as the sole primitive for performing localization and mapping in a large scale setting. As we show in the the RGB-D/posed case, the methods we compare to (FCAF, UniDet3D) are, in fact, operating on the dense reconstruction as an aggregated point cloud or mesh. Nonetheless, we show that these methods can be outperformed by RfM which challenges the idea that a dense reconstruction is even the best representation for 3D object detection. Furthermore, in order to efficiently process a dense reconstruction, one needs additional machinery like sparse 3D convolutions which are quite costly and not easily applied in certain settings.\n\n**FCAF is cited wrongly**: Thank you, we will correct this.\n\n**mAP**: We clarify that we _do_ report mAP (AP averaged over all classes) in the exact manner as FCAF, UniDet3D, and others. We will update the notation to reflect this.\n\n**comparison with point-based localization and mapping is completely missing**: We emphasize that the main goal of this paper is un-posed 3D object detection. We primarily intend the localization evaluation and experiments to _calibrate_ our method and show that 3D object detection can be performed as a localization and mapping task using 3D objects as a primitive.\n\n**only compare with CUT3R in RGB**: We note that the comparisons in our paper are with respect to metric camera pose estimation and 3D object detection. The baselines mentioned (COLMAP, Mast3r-SLAM, VGG-T) all produce camera poses and reconstruction _up to an unknown scale_. The best suitable metric-capable baseline we have found is CUT3R, which acts similarly to Mast3r, and generally gives results comparable if not better than Mast3r-GA."}, {"review_id": "N7UrrptaA6", "reviewer": "Reviewer_uKXv", "summary": "This paper proposes Rooms from Motion (RfM), an object-centric framework for 3D object mapping and camera localization from unposed RGB(-D) images, aiming to solve the problem of indoor 3D scene understanding without relying on known metric camera poses.\n\nRfM replaces traditional keypoint-based structure-from-motion (SfM) with an object-level pipeline: it detects metrically scaled 3D bounding boxes using a pretrained detector (CuTR), matches objects and their 3D box corners across images, estimates relative and global camera poses, and constructs global 3D object tracks, which are further refined through bundle adjustment.\n\nExperiments on CA-1M and ScanNet++ demonstrate that RfM achieves competitive or superior performance in 3D object detection compared to point- and volume-based baselines, even without access to ground-truth poses or depth. In the localization task, RfM shows strong results, outperforming state-of-the-art methods such as DROID-SLAM and CUT3R in both RGB-D and RGB-only settings.\n\nOverall, the paper presents a pose-free, object-centric approach to 3D detection and localization that offers a novel alternative to conventional volume-based 3D detectors and SfM methods.", "questions": "1. **Training details**: Could the authors clarify the training procedure for the object- and corner-matching networks? Specifically, what datasets were used, and what architectures and hyperparameters were used?\n\n2. **Fairness of localization comparisons**: Were the RfM matchers trained on the same datasets (CA-1M or ScanNet++) used for evaluation in the localization experiments? If so, how do the authors justify the fairness of comparing to methods like DROID-SLAM and CUT3R, which were not trained or adapted to these domains? Have the authors considered evaluating RfM in cross-dataset settings to assess generalization?\n\n3. **Runtime and scalability**: Can the authors provide timing statistics for the major components of the pipeline (e.g., CuTR inference, object matching, pose averaging, track-level optimization)? This would help assess the method’s practicality in large-scale or time-sensitive applications.\n\n4. **Generalization to other domains**: Have the authors considered applying RfM to domains beyond indoor scenes, such as outdoor environments or videos with different camera characteristics (e.g., lower resolution, motion blur)? If not, can the authors comment on the expected limitations or required adaptations?\n\n5. **Robustness in sparse or low-overlap scenes**: How does RfM perform in cases where object density is low or where object overlap between frames is minimal? Does the view graph often become fragmented in such settings, and are there any mechanisms to recover from such cases? Quantitative or qualitative analysis of these failure modes would be helpful.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n1. The paper introduces a shift from traditional point-based SfM to an object-centric formulation. By treating 3D bounding boxes as the fundamental geometric primitive, the approach provides a sparse yet semantically meaningful representation for both mapping and localization.\n\n2. The method is evaluated thoroughly on two large-scale datasets (CA-1M and ScanNet++), across multiple settings (posed/unposed, RGB/RGB-D). Results show consistent improvements over both point-based and voxel-based baselines, in both 3D object detection and camera localization.\n\n**Weaknesses**\n1. The paper does not clearly specify the training procedure for key components, particularly the object- and corner-matching networks. It is strongly implied that they are trained using ground-truth object correspondences in CA-1M and possibly ScanNet++, but this is not stated explicitly, nor are hyperparameters, or architectures described.\n\n2.  In the camera pose estimation experiments, RfM appears to be evaluated on the same datasets it is (likely) trained on, whereas competing methods like DROID-SLAM and CUT3R are used off-the-shelf without domain adaptation. This setup favors RfM and may overstate its performance advantage. A fairer comparison would involve either evaluating all methods in cross-domain settings or including additional baselines with access to similar supervision.\n\n3. The paper does not report any timing statistics for detection, matching, pose estimation, or optimization. This makes it difficult to evaluate the method’s scalability or real-time potential, especially considering the potentially expensive steps like pairwise matching, graph-based pose averaging, and object-level bundle adjustment.\n\n4. All experiments are conducted on indoor RGB(-D) datasets (CA-1M and ScanNet++), both of which are highly curated and structurally similar. There is no discussion or empirical evidence regarding generalization to more diverse environments (e.g., outdoor scenes, cluttered motion blur, or different camera types). This raises questions about the robustness of the approach outside the training distribution.\n\n5. The framework relies on successful object detection and matching to drive both mapping and localization. It is unclear how the method performs in sparsely populated environments or in scenes with limited object overlap between views. In such cases, the view graph could become disconnected, but no quantitative analysis of failure modes is provided.", "was_revised": true, "final_justification": "Most concerns have been well addressed. However, I remain unconvinced about the fairness of the comparison to DROID-SLAM, as it was not finetuned on the ScanNet++ or ARKitScenes datasets. Finetuning DROID-SLAM on these datasets and then conducting the comparison would provide a more convincing evaluation for readers.\n\nWhile I suggest the authors include the finetuned DROID-SLAM results to strengthen the narrative, the experiments presented in the main paper are sufficient to demonstrate the effectiveness of the proposed method. Thus, I will maintain my rating as accept, especially since the direction on object-level localization and mapping offers an interesting perspective.", "rebuttal": "Thank you.\n\n**Regarding the cited _questions_**:\n\n**Training details:** We will clarify training procedures and hyperparameters in an update to part B of our appendix. Some aspects of note:\n\n1. We train models on the same datasets (CA-1M or ScanNet++) as used for evaluation and use the train and validation splits respectively.\n2. We generally adopt the same model/training hyperparameters as the original Cubify Transformer paper.\n3. The additional Cubify Match components are trained end-to-end using the double softmax loss from LightGlue.\n\n**Fairness of localization comparisons**: We note that CUT3R is trained on ScanNet++ and ARKitScenes (which shares the same underlying data as CA-1M). Therefore, we believe the comparison to CUT3R is fair --- they will have seen the same sets of images as our method and, in fact, many more since they train on much more than just these datasets. For DROID-SLAM, we note that its training is emphasized to be quite generalizable. Additionally, for DROID-SLAM, we believe the failures on ScanNet++ are less do to training datasets and more fundamental to SLAM systems which cannot easily handle non-smooth/sparser trajectories that are present in ScanNet++.\n\n**Runtime and scalability**: We provide some basic timings for a video (100 frames) running on a V100:\n\n1. Detection of objects per frame: ~65ms _per frame_ (640x480 image)\n2. Average time to \"add\" a frame to the view graph (perform matching, relative pose estimation to all other frames): 700ms\n3. ~10 seconds to \"finalize\" the graph (averaging/filtering)\n\nThe optional bundle adjustment runs on CPU using PyCeres and takes approximately 3-5 minutes (depending on the number of objects).\n\nFurther optimization could be applied to steps (1) and (2) to chunk/batch these operations which are highly parallel.\n\n**Generalization...**: Generalization to other domains (e.g., outdoors) is exciting further research and something we are actively investigating, however, we note that our method is only applied to Indoor 3D Object Detection. We do find our method has slightly degraded performance at lower resolutions (as the CuTR method slightly degrades) but can work successfully down to resolutions like 256 x 192 and is especially robust if depth is also provided at lower resolutions. Regarding motion blur, _most videos_ in CA-1M have a large amount of motion blur due to the scanning pattern speed and so we do find our method is capable of working with motion blur since it is trained on motion blurred data to start with.\n\n**Robustness in sparse...**:  Regarding sparse scenes, we believe the impact may been seen in the ablation (Table 4). Particularly, when we have less objects due to a smaller taxonomy (e.g., ARKitScenes), this will impact the ability for successful localization. This shows the importance of a detector like CuTR which is trained to detect nearly every object, albeit in a class-agnostic manner. Additonally, regarding overlap, FoV does matter as seen in Table 5 where a narrow FoV camera does lead to worsened results. We believe further research can help reduce this requirement by incorporating more priors/context (like dust3r/mast3r) into this type of framework."}, {"review_id": "m9YlQlFsxm", "reviewer": "Reviewer_p8G3", "summary": "The paper introduces Rooms-from-Motion (RfM), a method that treats the 3D-bounding boxes of objects in a scene as first-class citizens and generates both camera poses and a map of 3D bounding-box positions of a scene using unordered images. \n\nRfM uses a multistep pipeline. In a first step, it uses the prior work Cubify Transformer (CuTR) to detect 3D bounding boxes of objects in each image individually. RfM then matches both objects and object corners across image pairs, thereby generating point matches between images. Using some filtering heuristics to suppress wrong matches, these derived point matches then are used to estimate camera poses and global object poses. CuTR can process metric depth maps for improved performance, and metric depth as well as ground-truth camera poses can also be used in the last step of the pipeline to increase accuracy. \n\nRfM shows improved 3D object detection performance on both the CA-1M and the ScanNet++datasets. It also outperforms DROID-SLAM and the recent CUT3R on the task of camera pose estimation.", "questions": "- Could you add some additional strong baselines for the Camera Pose Estimation task? (see Weaknesses)\n\n- I would like to see a diagram of the box matching network and the box corner matching network, even if only in the appendix.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "+ **Strengths**\n    + new, and intriguing idea to use only object-bounding-boxes as scene representation (map)\n        + I can imagine that by focussing on objects as \"first-class citizens\", there might be a bit more robustness in the matching, though at the expense of fewer / less accurate individual point matches\n        + The main advantage of this approach I think is in using the object-centric scene representation / detected objects for downstream applications like scene understanding / robotic planning. From this perspective however, not modelling the background geometry (floor, walls, ceiling) seems like a limitation (though one that it should be possible to overcome)\n    + straightforward pipeline, well-explained\n    + strong results in a number of settings and different tasks\n\n- **Weaknesses**\n    - Limited Baselines for Camera Pose Estimation. \n        - would like to see something like COLMAP with strong pixel matchers, and perhaps VGG-SfM and/or VGG-T as baselines\n    - The bounding-box-focussed scene representations might be less appropriate in outdoor scenes / object-sparse scenes (beaches, highways) / non-human environments (e.g. forests)", "was_revised": true, "final_justification": "I agree with other reviewers that the camera localisation experiments experiments should compare against some stronger baselines, either ones that require ground-truth depth (which the authors use in some of their experiments already and can be cheaply obtained with phone-based LiDAR sensors) or using point-based methods that do output metric predictions, such as MASt3R. \n\nI suspect that such baselines will, perhaps significantly, outperform the present method in camera pose estimation, but my rating does not depend on the method's performance on a single application. \nI believe the presented approach is novel and promising and has the potential to spark future fruitful work in exploring the best domains and applications for it. I therefore maintain my rating.", "rebuttal": "Thank you.\n\n**Regarding the cited _weaknesess_:**\n\n**Limited Baselines for Camera Pose Estimation**: We would like to add additional RGB only baselines, as suggested, however this is actually quite difficult given that a fair comparison requires a method capable of outputting metric camera poses from only RGB images. Methods like colmap, VGG-T, Mast3r-SLAM do not output metric information (only up to scale) and require explicit scaling by some means to achieve this. We found CUT3R to be the only appropriate baseline here. As a result, we do emphasize in the paper that these results are meant to _calibrate_ and inspect our method and less so prove that it is a superior localization method. The main purpose of the paper is to perform un-posed 3D object detection, which has not generally been explored before. We emphasize that 3D object detection is inherently a _metric_ task which helps inform our approach.\n\n**diagram of**: We will add additional diagrams of the box matching and corner matching networks in a future revision (notably, due to the rebuttal format, we cannot upload them here)."}, {"review_id": "0N9MDJAX8K", "reviewer": "Reviewer_ZnXN", "summary": "The main idea is an object-based SfM, using object cuboids and their descriptors in an analogous way to points and their descriptors in point-based SfM. Cuboids are detected and feature descriptions extracted using CuTr; pairwise image matching is performed using an extension of SuperGlue; pairwise pose is solved; and global pose is solved using rotation/translation averaging.  The matching and pose estimation steps require some innovation to deal with the cuboid candidate matches, e.g. a second SuperGlue Model is trained to produce corner points that better correspond.\n\nExperiments on ScanNet++ and CA-1M demonstrate good results for multiview 3D object detection (compared to methods that rely on some form of point/dense fusion) and pose estimation (compared to Droid-SLAM and Cutt3r).", "questions": "**Comments** \n* For DroidSLAM would have been nice to compare with monocular depth\n* \"Rooms from Motion\" isn't the best title because it's not really about estimating rooms.  \"Object-based SfM\" or \"Objects from Motion\" may fit better\n* After reading the paper, I'm still not sure of the practical use, e.g. does this have potential to be part of a well-engineered object mapping system, would it be used in conjunction with points or as a post-process to SfM, etc.\n* It's not clear how the method is adjusted when you have ground truth depth or pose", "limitations": "OK", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The paper is creative and has plenty of little innovations, as well as a good proof-of-concept evaluation.  The main weakness is that it does a poor job explaining the relationship to object-based SLAM and SfM (particularly GlobalSfM).  Also, it would have been nice to compare the pose estimation to the Global SfM methods that are more analogous, such as GLOMap or 1d-sfm, and to incremental methods such as COLMAP.  Also, the speed and scalability of the method isn't analyzed.  It feels half-baked.\n\n**Strengths**\n* Revisiting SfM using class-agnostic object cuboids is an interesting idea\n* The paper offers many little innovations needed to make it work\n* Results in both 3D object detection and pose estimation are good compared to recent methods\n* The paper is mostly clear about the approach and results\n\n**Weaknesses**\n* The related work is much too cursory, basically saying that there is a bunch of stuff on object-centric SLAM and 3D object detection, without explaining or citing most of that work. The paper and figures generally feel unpolished, so it seems like this is due to running out of time, rather than well-considered space prioritization\n* Does not compare to SfM approaches (either incremental or global), while the proposed method is closer in approach to them than it is to SLAM\n* Does not provide any analysis of timings \n\n**Recommendation: Weak accept**\nThe paper is interesting, provides good proof-of-concept experiments, and could inspire future work.  However, the paper feels thrown together with its barely-there related work section and conclusion, small text in figures, and questionable choices of figure and text space allocation. It feels half-baked and, while worth reading in its current form, the authors should consider a revise-and-resubmit to maximize impact.", "was_revised": true, "final_justification": "I disagree with reviewer n63D's statement \"interesting application of structure-from-motion for objects, but there's little additional insights\".  \n\nThe paper does have significant deficiencies in position with respect to related work and clearly stating the goals and contributions.\n\nThat said, I stick with my original assessment that while the paper could do a better job of positioning itself, it's an interesting paper that is above bar in its current state.", "rebuttal": "Thank you. We will elaborate on the set of related work in a future revision. We agree \"Objects from Motion\" is an appropriate name and will work to clarify this.\n\n**Regarding the cited _weaknesses_:**\n\n**Does not compare to SfM**: We emphasize that the main goal of this paper is un-posed 3D object detection. We provide localization comparisons in order to calibrate its ability. Additionally, our method is not easily comparable to SfM, even if it is inspired by it. In particular, we do matching in a metric 3D space and expect to output metric camera poses and 3D objects/boxes. Traditional SfM techniques do not generally output metric quantities and must rely on some mechanism of external scaling.\n\n**Does not provide any analysis of timings**: We provide some basic timings for a video (100 frames) running on a V100:\n\n1. Detection of objects per frame: ~65ms _per frame_ (640x480 image)\n2. Average time to \"add\" a frame to the view graph (perform matching, relative pose estimation to all other frames): 700ms\n3. ~10 seconds to \"finalize\" the graph (averaging/filtering)\n\nThe optional bundle adjustment runs on CPU using PyCeres and takes approximately 3-5 minutes (depending on the number of objects).\n\nAs is the case for global SfM methods, our primary goal is to run offline and a streaming version (closer to SLAM) is an interesting area of future research. We will add these timings to an updated revision.\n\n**Regarding the provided _comments_:**\n\n**nice to compare with monocular depth**: We do not compare with DROID-SLAM for monocular depth since in that scenario, it will only produce camera poses _up to scale_ which is incompatible with our setting of _metric_ camera localization.\n\n**not sure of the practical use**: We believe our framework can immediately replace traditional 3D object detection systems which rely on point clouds and other machinery like 3D sparse convolution when depth is present. Furthermore, we believe that our 3D object detection and/or localization can be applied on systems which lack the ability to capture images at a rate necessary for traditional SLAM _or_ lack depth sensors. Generally, our framework will make 3D object detection more readily applied in a wider set of scenarios than previously thought capable.\n\n**how the method is adjusted**: We will clarify this in an updated revision. To incorporate depth, we adopt the RGB-D variant of CuTR which takes both an RGB image as input alongside a metric depth map. To incorporate pose, we use the ground-truth relative pose rather than the estimated relative pose. Additionally, we use the ground-truth absolute poses which is used to lift individual 3D object tracks into world space."}]

Author rebuttal:

Meta-review: This paper receives 2x borderline accepts and 2x accepts. The reviewers believe that the proposed approach is novel and promising. It also has the potential to inspire future works in this direction. The work  shows clear goal and contributions. However, there are still some remaining concerns on the comparison with stronger baselines. The authors are strongly encouraged to include comprison with fine-tuned DROID-SLAM in the final paper. Nonetheless, the reviewers think that this missing comparison does not affect the acceptance of the paper. The ACs follow the final ratings of the reviewers to accept the paper.
Decision: Accept (poster)