Object-X: Learning to Reconstruct Multi-Modal 3D Object Representations
Authors: Gaia Di Lorenzo
Federico Tombari
Marc Pollefeys
Daniel Barath
Venue: NeurIPS 2025 poster ()
Keywords: Object Reconstruction
Gaussian Splatting
Multi-Modal Representation Learning
Primary area: applications

Abstract: Learning effective multi-modal 3D representations of objects is essential for numerous applications, such as augmented reality and robotics. 
Existing methods often rely on task-specific embeddings that are tailored either for semantic understanding or geometric reconstruction. 
As a result, these embeddings typically cannot be decoded into explicit geometry and simultaneously reused across tasks.
In this paper, we propose Object-X, a versatile multi-modal object representation framework capable of encoding rich object embeddings (e.g., images, point cloud, text) and decoding them back into detailed geometric and visual reconstructions. 
Object-X operates by geometrically grounding the captured modalities in a 3D voxel grid and learning an unstructured embedding fusing the information from the voxels with the object attributes. 
The learned embedding enables 3D Gaussian Splatting-based object reconstruction, while also supporting a range of downstream tasks, including scene alignment, single-image 3D object reconstruction, and localization.
Evaluations on two challenging real-world datasets demonstrate that Object-X produces high-fidelity novel-view synthesis comparable to standard 3D Gaussian Splatting, while significantly improving geometric accuracy. 
Moreover, Object-X achieves competitive performance with specialized methods in scene alignment and localization.
Critically, our object-centric descriptors require 3-4 orders of magnitude less storage compared to traditional image- or point cloud-based approaches, establishing Object-X as a scalable and highly practical solution for multi-modal 3D scene representation.

Object-X: Learning to Reconstruct Multi-Modal 3D Object Representations
Gaia Di Lorenzo'
Federico Tombari?
Mare Pollefeys 1.3
Daniel Barath 1.2
'ETH Zurich
Google
Microsoft
gdilorenzo@ethz.co
Label Arnolakr
Viual Localtation
Scate Algemmen
Single Image Recormuction
Muttklew Imag
Full Scone Recontruction
7
Mulii Modal Object Reps
U-3DGS
3DG5
Figure 1: Object-X learns object-centric embeddings from an input object segmentation of a 3D scene reconstruction. The embeddings learned from multi-modal data (e.g., mesh, images, text descriptions) enable fast 3D Gaussian Splat reconstruction via a specifically trained decoder, and other downstream tasks operating directly in the latent space, such localization and scene alignment. Object-X allows for representing the scene as set of object descriptors without having to store garage-heavy representations like point clouds and image databases, while providing similar functionalities.
Abstract
Learning effective multi-modal 3D representations of objects is essential for nu- merous applications, such as augmented reality and robotics. Existing methods often rely on task- sescispecific embeddings that are tailored either for semantic un- derstanding geometric reconstruction. As a result, these embeddings typically cannot be decoded into explicit geometry and simultaneously reused across tasks. In this paper, we propose Object-X, a versatile multi-modal object representa- tion framework capable of encoding rich object embeddings (e.g., images, point cloud, text) and decoding them back into detailed geometric and visual reconstruc- tions. Object-X operates by geometrically grounding the captured modalities 3D voxel grid and learning unstructured embedding fusing the information from the voxels with the object attributes. The learned embedding enables 3D Gaussian Splatting-based object reconstruction, while also supporting range downstream tasks, including scene aligmment, annle-image 3D object recon- struction, and localization. Evaluations two challenging real-world datasets demonstrate that Object-X achieves high-fidelit novel-view synthesis compara- ble to standard 3D Gaussian Splatting, while significantly improving geometric accuracy. Moreover, Object-X achieves competitive performance with specialized methods in scene alignment and localization. Critically, our object-centric descrip- tors require 3-4 orders of magnitude less storage compared to traditional image- point cloud-based approaches, establishing Object-X as a scalable and highly practical solution for multi-modal 3D scene representation. The code is available at lhttps://githann com/ httmm//aaaailloeeee//bbeeett
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Robust 3D scene understanding, incorporating geometric, visual, and semantic information, forms cornerstone for advances in robotics, augmented reality (AR), and autonomous systems [27 (2.
A key goal is to develop 3D representations that are not only accurate but also compact, efficient, and flexible enough to integrate multiple sensor modalities and support diverse downstream tasks.
Traditional 3D representations, often relying on explicit geometry such as dense point clouds 4 20 or meshes (18), alongside collections of images, tend to incur prohibitive storage and computational costs. More recently, implicit neural and Gaussian representations, notably Neural Radiance Fields (NeRF) 16) and 3D Gaussian Splatting (3DGS) (II), have achieved state-of-the-arr results in synthesising novel views from images, jointly encoding geometry and appearance. However, these methods typically generate a monolithic, scene-level representation, primarily driven by visual input. As consequence, they inherently lack object-level modularity, making it difficult to reason about individual objects, efficiently incorporate other modalities (e.g., text, semantics), or casily use the representation for object-level tasks beyond rendering or 3D reconstruction.
To address the need for modularity, object-centric approaches, such as 3D scene graphs (, have gained traction. Such methods decompose a scene into collection of objects and their relationships, often associating a learned embedding with each object. Such embeddings have proven effective for abstract, object-level tasks, including cross-modal localization (15), scene retrieval (24), and 3D) scene alignment (23). However, a critical limitation persists: existing object embeddings are generally learned for specific tasks and cannot be decoded to reconstruct the explicit, high-fidelit appearance and geometry of the object they represent. This forces systems to retain the original, high-band bandddhh source data (images, point clouds, meshes) alongside the learned embeddings, undermining the goals of creating a compact, self-contained, and truly versatile object representation.
In this work, we bridge this crucial gap by introducing Object-X, framework for learning rich, multi-modal, object-centric embeddings that are simultaneously suitable for downstream tasks and decodable into explicit, high-quality 3D representations. Object-X geometrically grounds multiple input modalities pertaining to an object within a 3D voxel structure, fusing this with semantic attributes (like class labels or object descriptions) to learn compact, latent embedding. Crucially, we design a decoder that uses this embedding to predict the parameters of a set of 3D Gaussians, enabling high-fidelity. object-level rendering and geometry extraction via 3D Gaussian Splatting. The same learned embedding can be directly leveraged for diverse downstream tasks.
Our experiments on challenging, real-world datasets demonstrate that Object-X - supports high-fidelity novel-view synthesis comparable to, and geometric reconstructions superior to, standard 3DGS, while also achieving competitive performance on scene alignment and localization tasks. Critically, Object-X reduces storage requirements by 3-4 orders of magnitude compared to storing the underlying point clouds or images, while offering similar functionalities. Our main contributions are:
1. A novel framework for learning compact, multi-modal, object-centric embeddings that can be decoded into fighfiidlitty geometry and appearance, parameterized by 3D Gaussians.
2. The demonstration that a single, unified embedding supports both high-quality 3D reconstruc- tion (encompassing novel view synthesis and detailed geometry) and performs competitively on diverse downstream tasks, such as scene alignment and visual localization.
3. Significant storage reduction (3-4 orders of magnitude) compared to explicit representations as the decodability of our embeddings obviates the need to store raw images or point clouds.
2 Related Works
Understanding 3D scenes is a fundamental problem in computer vision, with applications spanning robotics, augmented reality, and 3D content creation. Numerous representations have been proposed to capture the complexities of 3D environments, each offering different trade-off's between accuracy, efficiency, storage, and ease of use. Our work, Object-X, builds on and connects several key areas by proposing a novel object-centric embedding strategy.
3D Representations. Traditional methods for representing environments include point clouds 4 meshes (2)), and voxel grids (18 38 34). Point clouds offer a direct representation of geometry but lack structure and demand significant storage for detailed scenes. Meshes provide structure by connecting points with polygons, facilitating efficient rendering and geometric operations, though

---

Project
Spane Enooder
Spane Decoder
Voxefoe
Segmented Marh
Seructured Represensation
Denarty
3D Unet
Spaniiyy
DINO VIE
Putch Encodar
Object
Pasch
X
Siilaiiyy
Structure Labent
U-30GS
Seructured
Figure 2: Overview of Object-X, learning object embeddings to reconstruct 3D Gaussians and support other tasks such as visual localization (15). (a) The method takes mesh or point cloud of an object along with posed images observing it. The canonical object space is voxelized based on object geometry, and DINOv2 features extracted from the images are assigned to each voxel. This produces 64 x 8 structured latent (SLat) representation (31). (b) The SLat is further compressed into a 16 x U-3DGS embedding using a 3D U-Net. The embedding is trained with a masked mean squared error loss to ensure accurate reconstruction of the SLat, which in turn enables decoding into 3D Gaussians using standard photometric losses. (c) Additional task-specific losses, such as those for visual localization (15), can be incorporated to optimize the embedding for multiple objectives.
the meshing process can lead detail loss, especially when optimizing for storage. Voxel grids discretize 3D space, but suffer from high memory requirements at resolutions needed for fine detail. These representations localization, which introduces the overhead of managing and storing large image collections. The pursuit compact and versatile representations motivates our work on learnable object embeddings.
often accompanied by posed image datasets for applications such visual
Recently, neural implicit representations have gained attention. Neural Radiance Fields (NeRF) (16) represent scenes with MLPs, enabling photorealistic novel view synthesis but be computationally intensive. 3D Gaussian Splatting (3DGS) (11) presents a compelling alternative, using a collection of 3D Gaussians for high-quality, real-time rendering. While methods like MV-Splat 33, Depth- Splat (32). NoPoSplat (35), and alternatives learn monolithic 3DGS model for an entire scene from object-centric paradigm. We learn compact, multi-modal embeddings for individual objects, which can then be decoded into object-level 3DGS parameters. Although recent works have explored 3DGS modifications for editing (37) or compression (25). they typically operate on or refine an existing 3DGS scene. In contrast, Object-X learns fundamental object embeddings that serve not only source for 3DGS reconstruction but also versatile descriptors for various other downstream tasks, offering a more holistic object representation.
images, focus
3D Scene Graphs. Scene graphs (10) provide structured representation by capturing objects, their attributes, and interrelations. 3D scene graphs (I) extend this concept to 3D, integrating semantics with spatial and camera information. They have proven useful for tasks like scene alignment (23), retrieval (15 24), and task planning
(5). The construction of 3D scene graphs has been streamlined by recent advances in object detection and relationship prediction, with tools such MAP-ADAPT 38
OpenMask3D (26). and ConceptGraphss (5). Existing scene graph methodologies often associate nodes with learned embeddings tailored for specific tasks (c.g., alignment, retrieval). However, key limitation is that these embeddings typically lack generative or reconstructive capability; they cannot be decoded back into explicit object geometry or appearance. This necessitates retaining the original sensor data (images, point clouds) alongside the graph, undermining compactness. Object-X addresses this gap by learning rich, multi-modal object embeddings that are explicitly designed to be decodable into high-fidelity 3DGS representations. Furthermore, these same embeddings retain strong descriptive power, enabling competitive performance on downstream tasks like localization and alignment without requiring task-specific modifications. This dual capability reconstructive and descriptive a core contribution of our work, offering pathway to leverage the semantic richness while also providing access to explicit 3D object representations.

---

3D Generative Methods for content creation have rapidly advanced, with 3DGS (T emerging as an expressive and efficient primitive. Several methods leverage 2D distillation from text or images to generate 3D 3DGS scenes 27 21). More recent works explore structured latent spaces for improved scalability and control in generation. For instance, Trellis (31) uses a unified Structured LATent (SLat) representation, decodable into various 3D forms including Gaussians, for large-scale object generation. L3DG (22) employs a latent diffusion framework with VAEs for efficient sampling and 3DGS rendering, while DifIGS (39) introduces a diffusion model for controlling Gaussian parameters.
These methods focus on de novo generation or generation from abstract imputs (e.g.. text prompts, style images), showcasing the potential of learned latent spaces for 3D content creation. Object-X shares the goal of leveraging learned latents but differs in its main objective. Instead of open-ended generation. our focus is on learning compact and versatile embeddings from observed multi-modal data. The key to create embeddings that capture an object geometry and appearance for reconstruction, while also being discriminative enough for tasks like localization and alignment. Thus, we emphasize robust representation learning from captured data for multi-task utility, rather than pure synthesis.
3 Learning Versatile Object Embeddings
We propose Object-X, taking a reconstructed scene with a 3D) object segmentation as input and learning compact and descriptive embedding for each object from their associated multi-modal data (e.g., images, point cloud). To achieve this, we process each 3D object through the following steps:
1) Extract Structured Latent Representation (SLat): We first process the input data for an object. We voxelize the object into a canonical 3D grid and aggregate local image features within these voxels via multi-view projection, inspired by (31). A 3D encoder then transforms these voxel-aligned features into a structured set of latent vectors (the SLat), which represents the object's initial rich encoding.
2) Project SLat to an Unstructured Embedding: We then learn to compress SLat into a fixed, significantly smaller dimension.
dense and unstructured latent representation Besides compression loss, other objectives are also considered during training to facilitate downstream tasks, e.g., object retrieval. This embedding, which we term the U-3DGS Embedding IT] becomes our core storage unit and descriptor for an object.
3) Decode the U-3DGS Embedding to 3DGS: One key application of the U-3DGS embedding is its decodability. For reconstruction, we map this dense embedding back to SLat representation and then decode this into a full 3D Gaussian Splat model for the object.
Below, detail the formation of the SLat representation. We then describe its compression into the versatile U-3DGS embedding. Following this, we explain the decoding mechanism that enables 3DGS-based reconstruction from this embedding. The utility of U-3DGS for other direct downstream applications will be demonstrated in subsequent sections. Fig. 2 summarizes the overall pipeline.
3.1 Structured Latents from Multi View Images
Let a set of object instances O be given, where each object o (P,I,,,, O is associated with mulli-modai input, such as multi-view images (I), instance masks (M), point clouds (P) and other attributes (A4). For each object o, we extract an object-specific latent embedding w.. The scene representation is defined as W = [woJoEo. enabling lightweight storage compared to dense point clouds or images. The goal to learn mapping O - W, where W allows decoding into 3D objects. Here, we focus on modalities P and I to learn the reconstructable embedding. Other modalities will be discussed when training also for downstream applications in the next section. We denote the resulting latent object embedding w which is the central representation used throughout the paper.
Voxelization and Feature Extraction. Given multi-view images of an object o, we first voxelize its canonical 3D space into an N x N x N grid (e.g., 649). For each voxel pæ that intersects the surface of the object, we project p; into each image to retrieve localized features. Similar (3I), we employ a pre-traiin pre-trained DINOv2 feature encoder (19) on masked object images. Averaging these image-level features yields a por-voxel feature f. E RD. Concatenating over all active voxels forms a sparse, voxel-aligned feature set as
& N3.
Named for its designed decodability into 3D Gaussian Splats, though it serves broader purposes.

---

Structured Latent Representation (SLat). We convert f into structured latent representation 2 {(Zi, p:)) via 3D encoder follows:
z = [=<(..)))) = 8U), 2, ERR Pt E NO
Each voxel p; is paired with a local latent 2; capturing shape and appearance. By preserving the sparse grid structure, 2 offers geometric grounding (through p:) and localized feature encoding (through 2;).
3.2 Compression to a Dense Embedding
Although is sparse relative to raw 3D data, it can still be large at high resolutions. Capturing fine details requires dense grids, which remain memory intensive. Moreover, downstream tasks favor compact embedding for each object, rather than collection of per-voxel vectors. Thus, learn a mapping to compress z into a fixed-size vector w E I, where d is small and independent of the size of the object at hand as = fU-DDGs (z), where Splatting (U-3DGS) embedding. We implement fD- using a 3D network (similar to U-Net) that first organizes into a dense N X N x N x C' tensor (with zero-filling for inactive voxels), then, downsamples and encodes it to W.
is Unstructured 3D Gaussian
Masked MSE for Compression Learning. We supervise SU.DDGS by requiring that the dense embedding w can be decoded back into the original structured representation z (or a close approxi- mation). Specifically, we introduce a decompression function faccomp that maps back to predicted 2 = {& pi J We then encourage & R 2 under a masked geea-sqqure-erre (MSE) LI loss that focuses on occupied voxels as = fuccomp (w), and the objective includes
Na Σ
Commpress
Ma 21 - 2 + 1-M,
2 - z4 [2]
10
where binary mask M, indicates voxel ps is occupied, and w is a down-weighting factor for non-occupied regions.
3.3 Decoding to 3D Gaussian Splats
In this section, we use the learned object embeddings w to reconstruct the object into 3DGS representation. As discussed the previous section, we can obtain the reconstructed structured latent representation & from by applying decompression network 2 S'tecomp (w).
Deterministic Autoencoder for 3DGS. We next train a decoder DGs that takes in the reconstructed structured latents & (obtained from w) and outputs set of 3D Gaussian Splat parameters as follows:
θ = (2s + = {(or Sis 4t ax ci)
M
i=1
Each Gaussian is specified by position Ti, scale 8;, rotation qi, opacity Or and color c, We train Dos by rendering these Gaussians from multiple viewpoints and minimizing image-space reconstruction losses (c.g.. LI, SSIM, and LPIPS) against ground-trutt images of the object using loss Crender ALL + (1 A) LSSIM] + LIPIPS. Since å can accurately encode fine object details, Dos learns to produce high-fidelity splats.
Voxel-Level Offsets. For spatial alignment, the center of each Gaussian is computed 2; P: + tanh(o:). where o, is an offset predicted by Dos and p; is the voxel location. This ensures positions remain near the coarse voxel layout, but can adjust locally for more precise fits.
Training. The proposed pipeline is end-to-end trainable, learning a compact embedding space in a single training pass. The optimization jointly minimizes the reconstruction loss, ensuring accurate recovery the SLat representation from the U-3DGS embedding, and photometric loss, learning the mapping from SLat features into 3DGS that can be used for NVS and surface reconstruction.
3.4 Learning Auxiliary Tasks
The U-3DGS embedding, primarily learned to capture object appearance and geometry for high- fidelity reconstruction, also serves as a potent foundation for various downstream auxiliary tasks such as visual localization (15) and 3D scene alignment 23 24 (see Fig. 3). To further enhance
performance on such tasks, we augment the pre-trained U-3DGS representation by integrating information from other relevant modalities. This augmentation involves incorporating features
5

---

Oojecs
Colecil
Tirget Scane tro
Coarse Visual Localization
Scene Aligmment
Scene Reconstruction
Figure 3: The proposed Object-X learns per-object embeddings that are beneficial for number of downstream tasks, besides object-wise 3DGS reconstruction, such as cross-modal visual localiza- tion (15) (via image-to-object matching). 3D scene alignment (23) (via object-to-object matching), and full-scene reconstruction by integrating per-object Gaussians primitives.
from auxiliary data sources like textual descriptions, object relationships, or broader scene context (which can be derived, for example, from a 3D scene graph structure (3)). Each auxiliary modality processed by its own dedicated encoder (e.g., a CLIP-based model with a projection head for text features), following (23). The resulting feature vectors from these auxiliary encoders are then concatenated with the original U-3DGS embedding to form a richer, multi-faccted representation for the object. The training strategy for these augmented representations, leveraging the pre- pre-trained U-3DGS encoder and decoder, proceeds in two main stages after the initial U-3DGS pre-training:
Auxiliary Encoder Training with Frozen Core: Initially, the pre-trained U-3DGS encoder and decoder (responsible for appearance and geometry) are kept frozen. The newly introduced encoders for the auxiliary modalities are trained. In this stage, learning is guided only by the task-specific loss Lisk (Weoneat), where Wconaat is the full concatenated embedding (U-3DGS + auxiliary features). For example, Lask may be a contrastive loss for localization. This ensures that the new auxiliary features are learned in way that remains compatible with, and does not corrupt, the reconstructive capabilities of the core U-3DGS representation.
Joint Fine-tuning: After the auxiliary encoders have been trained, all network components are unfrozen. The entire ensemble is then fine-tuned end-to-end using a combined objective:
Laux
Liask ( conem)) Arecon Frecom WD..DGS)
where Lreenn the reconstruction loss also used for pre-training the U-3DGS embeddings, applied using the U-3DGS decoder the corresponding WU-3DGS- Arcon balances these two objectives. This stage allows for mutual adaptation of all parts of the representation, further optimizing for both the specific auxiliary task and the foundational 3D reconstruction quality.
Through this process, Object-X learns effectively fuse intrinsic object properties (geometry, appearance via U-3DGS) with extrinsic or contextual information (text, relationships via auxiliary encoders). We will demonstrate this approach by training our model for visual localization (15). and subsequently evaluate its zero-shot or fine-tuned performance related tasks like 3D scene alignment, showcasing the versatility and robustness of the learned augmented embeddings.
4 Experiments
Next, we will provide experiments on various tasks benefiting from Object-X. Ablation studies, more visuals, and detailed descriptions of baselines are provided in the supplementary material.
Mesh extraction. To evaluate the geometric accuracy, we extract triangle mesh from the optimized 3D Gaussians, following the procedure from 2DGS (41). We first render depth maps from different viewpoints. These depths are fused using the Truncated Signed Distance Function (TSDF) integration in Open3D (40). Finally, a triangle mesh is extracted using Marching Cubes (14).
Implementation details. All experiments are conducted a machine with an A100 GPU with 80GB of RAM. During sparsification, a threshold of 0.5 is applied to the predicted occupancy. The mesh is constructed using voxel size of 0.015 and an SDF truncation value of 0.04.

---

3DGS
SDGS
2DGS (12V)
Object-X
Figure 4: Object reconstructions. Each row shows an input object (left) and its reconstruction obtained by, from left to right: (i) 3DGS (28) optimized on images, (ii) 3DGS (iii) 2DGS 44 using only 12 multi-view images, and (iv) Object-X. For each method, present a rendered image from the reconstructed 3D Gaussians and the corresponding mesh.
Table 1: 3DGS Object reconstruction photometric quality, geometric accuracy, runtime, and storage efficiency on 3RScan (9) and ScanNet ④ We compare Object-X - with baselines that store objects as set of 12 (3RScan) or 4 images (ScanNet) and reconstruct 3D Gaussians at test time using 3DGS (28), 2DGS (11, and DepthSplat (32). As a reference, we report the results of 3DGS, optimizing directly on all dataset images. We report NVS scores (SSIM, PSNR, LPIPS), geometric accuracy (Accuracy, Completion, and FI score at 0.05 m threshold), per-object run-time (secs), and storage (MB). We do not show geometric accuracy for DepthSplat as it failed mesh reconstruction.
Method
SSIM
PSNR
LPIPS
Acc. 00.05
Compl.600.05
F1
Time (x)
Storage (MB)
3DGS 2X
0.956
0.051
73.38
"DGGS (12V)
28
0.944
31.613
0.072
35.72
77.81
44.35
58.461
6.71
3RScan
2DGS (12V)
0.932
29.613
0.095
26.55
67.69
35.67
84.280
6.71
DepthSplat (12V)
21.669
Object-X
0.953
30.981
0,065
80,22
77,80
77,80
0,051
0,14
3DGS 2X
0.975
38.138
0.032
33.81
79.13
43,00
129.02
270.97
"DGS (4V)
28
(1969
307744
0.059
36.95
$2.37
15.13
129.022
55.26
ScanNet
2DGS (4V)
0.945
29.604
0.075
26.96
74.85
37.10
169.65
55.26
DepihSplat (4V)
0.832
26.152
0.134
0.138
55.26
Object-X
0.966
31.563
0.047
88.66
90.08
89.09
0.033
0.14
Datasets. The 3RScan dataset (9) consists of 1,335 annotated indoor scenes covering 432 distinct spaces, with 1,178 scenes (385 rooms) used for training and 157 scenes (47 rooms) reserved for validation and testing. The dataset provides semantically annotated 3D point clouds, with certain scenes captured over extended periods to reflect environmental changes. Scene graph annotations are available from (28). Since the test set lacks such annotations, we reorganized the original validation split, allocating 34 scenes (17 rooms) for validation and 123 scenes (30 rooms) for testing. Objects without available images were removed to ensure consistent evaluation.
test ScanNet (4) without training our model on it. Since ScanNet does not provide scene graph annotations, we apply SceneGraphFusion (30) on RGB-D sequences generate 3D instance segmentations and object relationships (used for auxiliary tasks). This allows us to assess robustness to errors in 3D) instance segmentation in the practical setting. Compared 3RScan, ScanNet captures RGB-D sequences at higher frame rate with minimal motion between consecutive frames. To ensure diverse viewpoints, we sample one image every 25 frames. We use 77 test scenes from the split defined in (15). Scenes, where SceneGraphFusion fails to generate annotations, are excluded. As in 3RScan, objects without associated images are discarded. The test split, along with its annotations, will be publicly released.
ScanNer. To evaluate generalization.
1. Object Reconstruction. First, we evaluate the Object-X decoder in terms of storage efficiency, geometric fidelity, and visual quality on the object reconstruction task.
Baselines. 3DGS (TT serves as a high-fidelin baseline, representing each object as a set of 3D Gaussians, While this approach captures fine details, it requires substantial storage, as every object is
7

---

represented by a set of Gaussians. We provide results for 3DGS (12 views) that stores each object as 12 images captured from different viewpoints. During reconstruction, 3DGS is applied recover the 3D Gaussians from these views. This reduces storage compared to full 3DGS but introduces a trade-off: reconstruction takes longer, and the quality may be slightly degraded. 2DGS (12 views) ( follows the same 1 12-image storage strategy but employs 2DGS 41 instead of 3DGS. Both 2DGS and 3DGS leverage the segmented mesh as initialization. Default parameters are used. DepthSplat (12 views) (32) also relies on 12 stored views but reconstructs objects using DepthSplat, a fast feed-forward network. We use the pre-trained model provided by the authors, which was not trained on 3RScan. Since train on 3RScan, comparisons on this dataset may be unfavorable to SppthSplat. However, we also evaluate on ScanNet, where neither SepthSplat nor our method has been trained.
For the baselines. we select k frames that maximize viewpoint diversity by applying k-means clustering to the positions of the cameras observing the object. In 3RScan, we use the maximum number of frames supported by SepthSplat (i.e., 12), while in ScanNet, where fewer frames are available, we limit the selection to four views per object. To ensure fairness, 3DGS/2DGS are also evaluated under this sparse- setting (< 12 views), matching the input regime of DepthSplat and reflecting the natural sparsity of 3RSean/ScanNet. DepthSplat itself is evaluated in its intended setup on unmasked scene-level images, with object masks applied only after reconstruction, ensuring consistent and fair comparison.
We present examples in Fig. 6 showing renderings and the reconstructed meshes. Object-X produces significantly smoother renderings and higher-quality meshes, whereas meshes reconstructed by baselines exhibit strong artifacts and fail to achieve accurate geometry.
Metrics. We evaluate method using the standard novel view synthesis scores: PSNR, SSIM, and LPIPS. Additionally, we report standard geometric metrics: accuracy, completeness, and Fl score.
Results on 3RScan. The top part of Table [ reports the results 3RScan. Object-X achieves comparable novel view synthesis quality to the baselines, with an SSIM score closest to 3DGS, the second-best PSNR score, and the best LPIPS score. Object-X substantially outperforms all baselines in geometric accuracy. Our method improves geometric accuracy by a large margin of 46 percentage points compared to all baselines while also exhibiting good completeness. This demonstrates that the proposed U-3DGS embeddings effectively capture object geometry, accurately recovered by the Object-X decoder. We omit geometric results for DepthSplat (32), which failed to produce reasonable geometry. We attribute the higher geometric accuracy to the voxel- roxel-grounded latent representation, which spatially constrains and aligns the decoded Gaussians to surfaces, leading to improved consistency compared to unstructured Gaussian sets.
Our runtime is three orders of magnitude faster than methods relying on optimization. Moreover, our approach requires an order-of- maggmaunntdd less storage, as we only store a single embedding per object instead of numerous images or 3D Gaussians.
Results on ScanNet. The bottom part of Table I reports the results on ScanNet. Note that we did not train our model on this dataset and used the model trained on 3RScan. Even without training. we achieve the highest novel view synthesis scores compared to the baselines, being the closest to the reference 3DGS reconstruction. Our geometric accuracy significantly outperforms all baselines. Because ScanNet provides higher-resolution images than 3RScan, optimization-bann based approaches are considerably slower, even when using only 4 views. We are four orders of magnitude faster than 3DGS (4V) and 2DGS (4V) requiring 3 ms to reconstruct an object on average.
2. Scene Reconstruction. Although we do not explicitly perform scene reconstruction, we evaluate scene composition by integrating object-level reconstructions. Object-X composes the full scene by decoding embeddings for each object and rendering their splats jointly. While this method is highly efficient, artifacts can emerge due to missing object segmentations. We also show results for Object-X + Opt, where the initial scene is constructed via Object-X, and optimized using 3DGS on the same image set used in 3DGS (12V). This setup isolates the benefit of Object-X as strong initialization while allowing to overcome the problems caused by missing objects.
Table reports results on the 3RScan dataset. While Object-X achieves lower SSIM and PSNR compared to 3DGS (12V), it significantly outperforms all methods in geometric accuracy. Also, it runs two orders of magnitude faster than the baselines. With refinement (Object-X + Opt), our method not only matches or exceeds the photometric quality of 3DGS (12V), but also achieves the
8

---

Method
LPIPS L
PSNR 1
SSIM 1
F1 1
Modalities
10 scenes
Method
RGGDD
3DGS
0.131
26.58
0.891
27.01
P
I
O
3DGS
R@1
R@3
R@S
Object-X
0.099
28.03
0.926
44.79
SGLoc [5
K
K
53.6
81.9
92.8
3DGS
0.129
26.72
0.896
8,70
CrossOver
24
46.0
77.9
90.5
Object-X
27.27
0.918
10.53
Objert-X
K
56.6
82.2
91.8
(a) Imagl Object Reconstruction on
(b) Coarse Visual Localization 3RScan (9).
RGB (depth predicted by (6) and RGB-D
Retrieval recall at various thresholds using vari- ous methods and input modalities.
frames from 3RScan (9). Photometric and geo-
metric accuracy of 3DGS vs. Object-X.
Figure 5: Comparison of (a) object reconstruction and (b) coarse localization performance using 3DGS and Object-X across tasks and input modalities.
highest geometric accuracy by large margin. Runtime remains comparable to the 12-view baselines, and significantly faster than full-scene 3DGS optimization.
While we note that full-scene reconstruction is not the primary focus of this work, we include these experiments to demonstrate the viability of composing object-level embeddings into larger-scale reconstructions, where Object-X X provides competitive quality, the highest geometrie accuracy, and serves as fast initializer for subsequent refinement.
3. Single-Imape Reconstruction. We further evaluate the flexibility of Object-X by reconstructing objects from a single RGB or RGB-D frame. Let us note that the Object-X was not explicitly trained for this task nor to infer unseen parts of an object. In the RGB-only scenario, we estimate monodepth using (6). Using the object mask and depth map, we lift the object pixels in 3D, obtaining a point cloud. We then voxelize it following the process described in Sec. 3.1 Object-X is then applied to obtain the object embedding from this input which then fed directly into our decoder. We compare our approach to 3DGS (28) which optimizes 3D Gaussian splats based on single masked image. Results in Table Ta show that Object-X outperforms 3DGS in all metrics on RGB-D and RGB inputs while also achieving significantly better Fl scores, indicating more precise and reliable reconstructions. We note that while there is still room for improvement in this task, achieving improved results demonstrate the versatility of the learned object embeddings. A promising direction, for example, is the principled integration of monocular 3D priors (33 36 29) geometric fidelity without sacrificing efficiency.
could further enhance
Additionally, we compare Object-X with the recent MIDI-3D K. a diffusion- based generative model for single-image 3D reconstruction. Unlike Object-X, which encodes metrically grounded geometry from observed data, MIDI a purely generative approach that learns to synthesize plausible 3D shapes directly from RGB inputs. The results are shown in Table While MIDI achieves impressive visual quality on in-distribution images similar to its training data, its performance drops significantly on -iitt--distribttion scenes, such those in 3RScan, where object shapes, scales, and textures differ from common internet imagery. these cases, MIDI often produces geometrically inconsistent outputs that remain visually plausible but lack realism.
In contrast, Object-X focuses on reconstruction rather than generation, leveraging voxel-grounded latent representations to maintain geometric consistency even under large appearance or domain shifts. As result, it generalizes better to real-world scenes without requiring instribution-specine fine-tuning, producing reconstructions that are both structurally coherent and metrically faithful.
Method
Accuracy@00 @0.05 f
Completion@ @0.05 t
@1@0.05
MIDI S
29.522
44.342
34.516
Object-X (RGB)
43.480
57.214
48.397
Object-X (RGB-D)
65.780
61.759
63.223
Table 2: Comparison with MIDI 8 on a subset of 3RScan. Metrics are computed at a em threshold. Methods marked with use automatic alignment without manual registration.
Coarse Visual Localization. We evaluate visual localization on 3RScan. Following the protocol from SceneGraphL.co (15), we evaluate on 123 scenes from 30 rooms in the test set. We select query images for each scene and match them against 10 candidate scenes (including the target) to determine if the correct scene can be identified. This process is repeated for every image in each room, resulting

---

Table 3: Full-scene composition 3RScan (9. We compare Object-X 3DGS (28) optimized on all unmasked images, and two 12- baselines: 3DGS (12V) and 2DGS (12V), which optimize scenes using subset of training images constructed by taking the union of the 12 best views selected per object. Object-X achieves the second highest geometric accuracy the fastest. When combined with refinement (Object-X +
Opt), it also achieves the best perceptual quality among all methods.
Method
SSIM +
PSNR 1
LPIPS L
Acc.@0.05 f
Compl. @0.05 t F1@0,05 1
Time (8) !
3DGS (28
0.855
24.404
0.417
16.48
31.99
21.40
260
3RSean
3DGS (12V)
28
0.767
18.806
0,317
16,81
32.63
21.88
150
2DGS (12V)
0.752
18,900
0,523
18.09
30.92
22.61
250
Object-X
0.677
15.488
0.526
46,22
54.98
49.99
Object-X+ + Opt
0.727
17.098
0.507
63.00
71.54
66.49
150
Table 4: 3D Scene Alignment on 3RScan ( by Object-X, SGAligner (23) and EVA 13). We report
Mean Reciprocal Rank and Hits@ K that denotes the proportion of correct matches appearing within the top K, based on cosine similarity. Evaluations are conducted using modalities: point cloud (P), others (O), and 3DGS. In constrast to the baselines, Object-X is used without training on this task.
Modalities
Method
Mean RR
Hits@1
Hits@21 f
Hits@3 f
H 1
Hits@5 1
P
3DGS
EVA 13
/
X
X
0.867
0.790
0.884
0.938
0.963
0.977
SGAligner
23
V
X
x
0.884
0.835
0.886
0.921
0.938
0.951
SGAligner
23
/
x 0,950
0.923
0.957
0.974
0,982
0,987
Object-X
x
/
0.910
0.864
0.917
0.948
0.965
0.975
in a total of 30,462 query images used for evaluation. Our method, along with SceneGraphLoc and the recent CrossOver (24), uses ViT to extract per-patch object embeddings from the query image. For each candidate scene, a similarity score is calculated by identifying the most similar object embedding in the scene for each patch in the image. Robust voting is then performed across all patch-object similarities to determine the scene where the query image was captured. We report the scene retrieval recall 1, 3, and 5, measuring the percentage of queries for which the correct scene is ranked within the top 1, 3, and retrieved scenes. The results are shown in T.. We indicate whether method uses point cloud (P), image (I), other modalities like object attribute and relationship (O), or the proposed U-3DGS embedding. Our method, leveraging 3DGS and other modalities, achieves the highest R @ I and R@3 scores.
5. 3D Scene Alignment. To further assess our generalization capabilities, we evaluate on a new task, Scene Alignment, 3RScan, following the protocol from SGAligner (23). This task involves matching objects across partially overlapping scans of the same scene by comparing their embeddings. Unlike SGAligner. explicitly trained for this task using point cloud and object-level modalities, method relies solely on the proposed Object-X embedding trained with reconstruction and localization losses, without finetuning for scene alignment. As shown in Table Object-X performs comparably in all metrics (achieving the second highest accuracy) to the baselines tailored for this task.
5 Conclusion
We introduced Object-X, a novel framework for learning compact, versatile, multi-modal object- centric embeddings. These unique embeddings are decodable into high-fidelity 3D Gaussian Splats for object reconstruction while also serving as potent descriptors for diverse downstream tasks, such as visual localization, single-image reconstruction, and 3D scene alignment, often without task-specific fine-tuning. Object-X achieves excellent geometric accuracy and novel-view synthesis, comparable or superior to specialized methods, while drastically reducing storage requirements by 3-4 orders of magnitude by obviating the need for raw sensor data. Our approach effectively bridges the gap between abstract learned object representations and detailed explicit 3D models, offering scalable and practical solution for advanced 3D understanding.
Limitations. Despite these advances, Object-X X has limitations. The high degree of compression can lead to a loss of the finest details in some reconstructions, particularly for large or complex objects. Furthermore, while promising in zero-shot scenarios for tasks like single-image object reconstruction. performance does not yet consistently match that of optimized task-specific methods.
Acknowledgements. This work was supported by an ETH Zurich Career Seed Award.
10

---

References
[1] Iro Armeni, Zhi- Yang He, Young Gwak, Amir R. Zamir, Martin Fischer, Jitendra Malik, and Silvio Savarese. 3d scene graph: A structure for unified semantics, 3d space, and camera, 2019.
[2] Chaochao Chen, Fei Zheng, Jamie Cui, Yuwei Cao. Guanfeng Liu, Jia Wu, and Jun Zhou. Survey and open problems in presecng knowledge graph: merging, query, representation, completion, and applications. International Journal of Machine Learning and Cybermetics, 15(8)3311-3332, 2024.
[3] Yuedong Chen, Haofei Xu, Chuanxia Zheng. Bohan Zhuang, Mare Pollefeys, Andreas Geiger, Tat-Jen Cham, and Jianfei Cai. Mysplat: Efficient 3d gaussian splatting from sparse multi-view images. In European Conference on Computer Vision. pages 370- 386. Springer, 2024.
[4] Angela Dai, Angel X. Chang. Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. 3d reconstructions of indoor scenes. Proc. Computer Vision and Pattern Recognition (CVPR), JEEE, 2017. Available for academic use under custom license. See wttp://www.scan net.org/ for details.
Scannet: Richly-annotater
[5] Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, al. Conceptgraphs:
3d scene graphs for perception and planning. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pages 5021-5028. IEEE, 2024.
[6] Mu Hu. Wei Yin. Chi Zhang, Zhipeng Cai. Xiaoxiao Long, Hao Chen, Kaixuan Wang. Gang Yu, Chunhua Shen, and Shaojie Shen. Metric3d v2: A versatile monocular geometric foundation model for zero- shot metric depth and surface normal estimation. JEFE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[7] Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. In Special Interest Group Computer Graphics and Interactive Techniques Conference Conference Papers '24, SIGGRAPH '24, page 1-11. ACM. July 2024.
[8] Zehuan Huang. Yuan-Cher Guo, Xingqiao An, Yunhan Yang. Yangguang Li, Zi-Xin Zou, Ding Liang, Xihui Liu, Yan-Pei Cao, and Lu Sheng. Midi; Multi-instance diffusion for single image to 3d scene generation, 2025.
[9] Nassir Navab Federico Tombari Matthias Niessner Johanna Wald, Armen Avetisyan. Rio: 3d object instance re-localization in changing indoor environments. 2019. Dataset licensed under CC BY-NC-SA
4.0: http://crcattvecoommmsss org/licensees/y- -awpensss//bnncss/...00
[10] Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David A Shamma, Li Fei-Fei, and Michael S Bernteinn. Image retrieval using scene graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3668 3678. 2015.
[11] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023.
[12] Micheal Lanham. Learn
of Google ARCore: Learn to build augmented reality apps for Android, Unity, and the web with Google ARCore 1.0. Packt Publishing Ltd, 2018
[13] Fangyu Liu, Muhao Chen, Dan Roth, and Nigel Collier. Visual pivoting for (unsupervised) entity alignment. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 4257-4266, 2021. [14] William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. pages 347-353, 1998.
[15] Yang Miao, Francis Engelmann, Olga Vysotska, Federico Tombari, Marc Pollefeys, and Dániel Béla Baráth. Scenegraphloc: Cross-modal coarse visual localization on 3d scene graphs, 2024.
[16] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1)-99-106, 2021.
[17] Raul Mur-Artal, Jose Maria Martinez Montiel, and Juan D Tardos. Orb-slam: A versatile and accurate
monocular slam system. IEEE transactions on robotics, 31(5):1147-116) 2015.
[18] Helen Oleynikova, Zachary Taylor, Marius Fehr, Roland Siegwart, and Juan Nieto. Voxblox: Incremental
3d euclidean signed distance fields for on-board mav planning. In 20/7 IEEE/RSJ International Conference on Intelligent Robots and Syxtems (IROS), pages 1366-1373. IEEE, 2017.
[19] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel Haziza. Francisco Massa, Alaaeldin El-Nouby. et al. Dinov2: Learning robust visual features without supervision, arXiv preprint ar Xxxxx23344007000 2023. Code available under Apache 2.0 License: https://ittub.com//fe com/ffcccoookresanchddnnoo
[20] Vojtech Panek. Zuzana Kukelova, and Torsten Sattler. Meshloc: Mesh-based visual localization. In
European Conference on Computer Vision, pages 589-609. Springer, 2022.
[21] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion,
2022.
[22] Barbara Roessle, Norman Müller, Lorenzo Porzi, Samuel Rota Buld, Peter Kontschieder, Angela Dai, and Matthias NieBner. L3dg: Latent 3d gaussian diffusion. In SIGGRAPH Asia 2024 Conference Pupers, SA
'24, page 1-11. ACM, Dec. 2024.
[23] Sayan Deb Sarkar, Ondrej Miksik, Mare Pollefeys, Daniel Barath, and Iro Armeni. Sgaligner: : 3d scene alignment with scene graphs. 2023.
[24] Sayan Deb Sarkar, Ondrej Miksik, Mare Pollefeys, Daniel Barath, and Iro Armeni. Crossover: 3d scene cross-modal alignment. Conference Computer Vision and Pattern Recognition, 2025.
11

---

[25] Yuang Shi, Simone Gasparini, Géraldine Morin, Chenggang Yang, and Wei Tsang Ooi. Sketch and patch: Efficient 3d gaussian representation for man-made scenes. arXiv preprint uuXiv:2501.1304 2025.
[26] Ayça Takmaz, Elisabetta Fedele, Robert W Sumner, Mare Pollefeys, Federico Tombari, and Francis Engelmann. Openmask3d: Open-vocabulary 3d instance segmentation. arXiv preprint arXiv:2306. 13631, 2023.
[27] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation, 2024.
[28]
Johanna Wald, Helisa Dhamo, Nassir Navab, and Federico Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. In Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[29] Jianyuan Wang. Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny,
Vegt: Visual geometry grounded transformer, 2025.
[30] Shun-Cheng Wu, Johanna Wald, Keisuke Tateno, Nassir Navah, and Federico Tombari, Scenegraphfusion:
Incremental 3d scene graph prediction from rgb-d sequences. 2021.
[31] Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured 3d latents for scalable and versatile 3d generation, 2024.
[32] Haofei Xu, Songyou Peng, Fangjinhua Wang, Hermann Blum, Daniel Barath, Andreas Geiger, and Marc Pollefeys. Depthsplat: Connecting gaussian splatting and depth. Computer Vision and Pattern Recognition, 2025.
[33] Lihe Yang. Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2, 2024.
[34] Miao Yang. Iro Armeni, Marc Pollefeys, and Daniel Barath. Volumetric semantically consistent panoptic mapping. In JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2024), Abu
Dhabi, United Arab Emirates, October 14-18, 2024, 2024.
[35] Botao Yc, Sifei Liu, Haofei Xu, Xueting Li, Mare Pollefeys, Ming-Hsuan Yang, and Songyou Peng. No
pose, problem: Surprisingly simple 3d gaussian splats from sparse unposed images. arXiv preprint
arXiv:2400.24200 2024.
[36] Chongjic Yc, Lingteng Qiu, Xiaodong Gu, Qi Zuo, Yushuang Wu, Zilong Dong, Liefeng Bo, Yuliang Xiu,
and Xiaoguang Han. Stablenormal: Reducing diffusion variance for stable and sharp normal, 2024.
[37] Qihang Zhang. Yinghao Xu, Chaoyang Wang. Hsin-Ying Lee, Gordon Wetzstein, Bolei Zhou, and Ceyuan
Yang. 3ditscene: Editing any scene via language-guided disentangled gaussian splatting. 2024.
[38] Jianhao Zheng, Daniel Barath, Marc Pollefeys, and Tro Armeni. Map-adapt: real-time quality-adaptive
semantic maps. European Conference on Computer Vision, pages 220-237. Springer, 2024.
[39] Junsheng Zhou, Weiqi Zhang, and Vu-Shen Liu. Diffgs: Functional gaussian splatting diffusiion, 2024.
[40]
Qian Yi Zhou, Jacsik Park, and Vladlen Koltun. Open3d: modern library for 3d data processing. In
Computer Vision and Pattern Recognition (CVPR). 2018.
[41] Zehao Zhu, Shaohui Ding, Tianhang Wu, Zhou, Ying Feng Yu, and Hao Wang. 2d gaussian splatting for geometrically accurate radiance fields. arXiv preprint arXiv:2312.00 02.11110.111111 2023.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction summarize the paper's contributions. Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings. It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors? Answer: [Yes]
Justification: Limitations are clearly discussed in the Conclusion section, highlighting challenges such as loss of fine detail in compressed representations and performance gaps in zero-shot tasks.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms
and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faireess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-
referenced.
13

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented
by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Justification:
The paper describes implementation details, model architecture, training setup, datasets used, and evaluation metrics in detail (see Section 4 and Implemen- tation Details), additional implementation details will be included in the supplementary material. Furthermore, the code will be made public. Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The code will be made public, but it is not yet available at submission time. Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis cc/
oliccy for more details.
14

---

reproducible, they While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https //nips. cc/ wptppp//guuiippppppppppppppppppp
CodeeSubiisee ss woooooiciioo for more details. The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments
should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Detailed experimental settings, including datasets (3RScan, ScanNet), evalua- tion protocols, hardware used (A100 100 GPU), and hyperparameters, are provided in Section 4 and Implementation Details, additional implementation details will be included in the supplementary material. Furthermore code will be made public for further understanding. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The paper does not report error bars, confidence intervals, or significance tests.
Results are shown as single point metrics (c.g., SSIM, PSNR, FI).
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula,
call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
15

---

For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 4 provides hardware details (A 100 GPU, 80GB RAM), per-object runtime, and comparisons to baseline methods in terms of compute efficiency. Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental rums well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http::/:/eaarrrs...cc/oo/lll/lla
Answer:
[Yes]
Justification: The paper does not violate NeurlPS Code of Ethics. Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid-
cration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper does not include Broader Impact section because focuses research in 3D object representation. The work does not involve user-facing systems, sensitive data, or applications with immediate societal or ethical implications. Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal
impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
16

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The model and data used do not pose high misuse risks and are standard in 3D vision research.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper uses publicly available datasets (3RScan, ScanNet) and pretrained models (e.g., DINOv2), all of which are properly cited. Licensing terms for these assets (e.g., academic non-commercial use for ScanNet) will be included in the supplementary materia
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paper sxperswite thooee com. dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
The paper introduces test splits and annotations for 3RScan and ScanNet, which are not included in the submission but will be publicly released with documentation upon publication.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
17

---

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (TRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required. Answer: [NA]
Question:
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non ota-standard components.
Please refer to our LLM policy (https: :tfp:::/neeune /ttp///eeiip.../ee wwt.co/eeeenccccc//////c/coccca/ /025/
LLM) for what should or should not described.
18

---

Supplementary Material
This supplementary material provides additional training details, experimental setups, visualizations, and ablation studies in support of the main paper. It is organized as follows:
1. Additional details on baseline comparisons and visualizations for object-level, image-based, and scene-level reconstruction (Section[A)
2. Training procedures for pretraining, compression, and downstream adaptation (Section B
3. Setup and evaluation details for visual localization using U-3DGS embeddings (Section
4. Scene alignment task setup, including sub-scene construction and evaluation metrics (Sec- tion D)
5. Single image reconstruction experiment (Section F
6. Ablation results on compression, occlusion robustness, and architectural variants (Section
A Baselines and Visualizations
This section details the implementation and evaluation protocols for baseline methods discussed in the main paper. We cover object-level, scene-level, and single-image reconstruction settings. To ensure a fair and rigorous comparison across all experiments, we maintain consistent supervision levels, initialization strategies, and evaluation metrics when assessing storage requirements, visual quality, and geometric fidelity.
A.1 Object Reconstruction Baselines
All optimization-basee methods, specifically 3D Gaussian Splatting (3DGS) ①I and 2DGS (4), operate on masked RGB-D input sequences. Our comparative analysis includes three primary optimization- based baselines:
1. 3DGS (Full Scene): Utilizes all available images from a given scene serve an upper- bound reference reconstruction.
2. 3DGS (A:V): Employs l pre-selected views that observe the target object.
3. 2DGS (&V); Also uses k pre-selected views observing the target object.
To ensure fair comparison by providing identical starting conditions for all methods, initialize Gaussian splats directly from ground-truth object meshes. The views for object-specific baselines selected using k-means clustering strategy (detailed below) to promote viewpoint diversity and ensure high object visibility. Crucially, evaluation is consistently performed on a disjoint test set of images that were not utilized during the training optimization phases of any method. For both 3DGS and 2DGS, we conduct optimization for 7,000 iterations using their default hyperparameter settings. For experiments typically offers fewer views per object instance, we restrict the number of selected views, k. to maximum of per object.
the 3RScan (9) dataset, we use 12 views and, ScanNet (4), which
Regarding DepthSplat (32), we evaluate using the publicly available pre-trained model. As this model was not trained on object reconstruction tasks, apply it to the unmasked image and, post-process its output by removing any splats that fall entirely outside the masked object region.
Visual comparisons are provided in Figure 6 reconstructions derived from the 3D Gaussians using the TSDF fusion technique, as proposed in (1I). These visualizations demonstrate that our proposed method, Object-X, achieves significantly smoother novel view syntheses and more geometrically accurate mesh reconstructions compared to the baselines.
Alongside rendered novel views, we present mesh
Frame Selection Protocol. To ensure consistent and representative view selection across all relevant experiments (object-level and scene-level k-view baselines), we employ a clustering- based strategy for choosing training/opiimization views. From the available set of frames for an object or scene, we first cluster their camera extrinsics (position and orientation) using k-means. Subsequently, we select one frame from each resulting cluster, prioritizing the frame that exhibits the fewest masked pixels (i.e., maximal object visibility within the frame). Any objects for which no valid test images remain after this selection process (e.g., due to insufficient visibility in all remaining frames not reserved for testing) are excluded from the evaluation set maintain fairness.
19

---

Ground Truth
3DGS
3DGS (12V)
2DGS (12V)
Object-X
Figure 6: Object reconstructions. Each row shows an input object (left) and its reconstruction obtained by, from left right: (i) 3DGS (28) optimized all images, (ii) 3DGS (iii) 2DGS 41 using only 12 multi-view images, and (iv) Object-X. For each method, we present a rendered image from the reconstructed 3D Gaussians and the corresponding mesh.
A.2 Scene-Level Reconstruction
For full-scene evaluation, we adapt 3DGS and 2DGS to operate jointly across all objects within a scene. This achieved by utilizing the union of the same views per object as the object- level experiments (specifically, 12 views for 3RScan and 4 for ScanNet), but here the RGB-D
20

---

inputs are unmasked. Our method, Object-X X, reconstructs the scene by independently decoding the learned U-3DGS embedding for each constituent object and then rendering their collective splats. This compositional approach requires no additional scene-level optimization. We also evaluate an augmented version, denoted as Object-X Opt. This variant leverages the compositional scene from Object-X as an initialization for a subsequent refinement stage. Specifically, it undergoes an additional 4,000 iterations of 3DGS optimization. To ensure stability during this fine-tuning process, all learning rates are reduced by a factor of 10x compared to the standard 3DGS settings.
Visualizations of scene-level reconstructions are presented in Figure X While Object-X generally produces significantly smoother results than the baseline methods, its performance can be affected by objects missing from the input segmentations (e.g., a poster on a wall, as shown in the first row of the figure, or the objects on the desk, as shown in the second row). Additionally. fine- tne-grained details might sometimes be diminished. However, applying 3DGS optimization as a post-processing step (Object-X + Opt) yields substantial improvements in accuracy, effectively recovering such lost details.
A.3 Single-Image Reconstruction
We extend our evaluation to a single-view reconstruction setting for all methods. In this scenario, 3DGS is optimized from scratch using single masked RGB-I) image (and its corresponding RGB image) for 3,000 iterations. To ensure a fair comparison, Object-X utilizes the same reference image. This image is selected based criteria that maximize unmasked object coverage while minimizing cropping along the image borders. We also test our method with only RGB input with depth predicted by Mctric3D (6) to generate the initial point cloud. Similarly the scene reconstruction case, we use Object-X X to provide an initial reconstruction which we further refine by applying an additional 1,000 iterations of 3DGS optimization, using learning rates reduced by a factor of 10x (consistent with the scene-level refinement). Visual results for this setting are presented in Figure 7
Notably, despite Object-X not being explicitly trained for single- reconstruction tasks, it frequently produces visually cleaner reconstructions than 3DGS when both methods are constrained to the same single input view and 3DGS is optimized from seratch under these conditions. The reconstructed meshes are also substantially more accurate than the ones from 3DGS.
A.4 Evaluation Summary
Across all experimental settings, methods are evaluated on fixed set of test views, distinct from traiiing/otimiizaioon views, on a per-object or per-scene basis as appropriate. We report standard quantitative metrics, Peak Signal-to-Noise Ratio (PSNR), and the perceptual metric LPIPS. Qualitative comparisons are provided in the relevant figures accompanying each experimental section ( (e.g.. Figure 6 for object-level, Figure X for scene-level, and Figure 7 for single-image results). Across all evaluated levels single-view, multi-view object reconstruction, and full-scene composition - Object-X X demonstrates strong performance, simultaneously offering significant advantages in terms of computational efficiency and flexibility initialization.
B Training Details
The training procedure encompasses three primary phases: sparse representation learning, compres- sion model training, and adaptation for downstream tasks. Each phase employs distinct optimization settings to ensure both stability and efficiency. For the sparse transformer- based encoder and decoder, we apply gradient clipping at threshold of 0.01. This is crucial for stabilizing the training process and preventing excessively large updates within the structured latent space. Optimization conducted using the AdamW W optimizer with a learning rate of x 10-4, This learning rate is selected to strike an effective balance between training stability and convergence speed. AdamW is chosen for its decoupled weight decay mechanism, which aids in regularizing the model without adversely affecting the gradient-based optimization updates. During the compression phase, a 3D U-Net architecture is trained to map the structured latent representation to a more compact form suitable for efficient storage or transmission. higher learning rate of 10-3 is utilized in this phase. This facilitates accelcrated convergence while preserving reconstruction quality. Explicit gradient clipping is not
21

---

Object-X (RGB-D)
3DGS (RGB-D)
Object-X (RGB)
3DGS (RGB)
Image
Figure 7: Qualitative comparison for image 3D. We compare the proposed Object-X standard 3DGS (28) on RGB and RGB-D imputs. For each method, we present the image from which the object (left column) is reconstructed and the rendered novel view together with the mesh reconstructed from the 3D Gaussians by: (2nd column) proposed Object-X with RGB-D input; (3rd) 3DGS with RGB-D; (4th) proposed Object-X with RGB input; (5th) 3DGS with RGB. The proposed method leads to significantly cleaner novel views and meshes than 3DGS applied to single image.
deemed necessary for the U-Net, as its inherent hierarchical structure and typical training dynamics provide sufficient stabilization.
For adaptation to downstream tasks, such as object localization or instance retrieval, training is performed using the AdamW optimizer with learning rate of 10-3 when the voxel-based latent representation is kept frozen. However, if the voxel representation is fine-tuned concurrently with the task- suse-specific modules, lower learning rate of 1 x 10-4 is adopted. This approach helps to mitigate the risk of catastrophic forgetting of the learned representations. Key regularization techniques employed include the aforementioned gradient clipping and structured weight decay (e.g.. as provided by the A optimizer).
C Supplementary: Coarse Visual Localization
We provide additional details for the visual localization experiment the 3RScan dataset. This ex- periment designed to evaluate the downstream utility of the U-3DGS embeddings when augmented with auxiliary modalities.
Training. The model utilized for localization is trained following the auxiliary learning setup described in the main paper. We freeze the pre-trained U-3DGS encoder and decoder. Auxiliary encoders are then trained on object-level inputs derived from 3D scene graphs, specifically object relationships, attributes, and structural context. Each RGB query image is processed through DINOv2 backbone followed by a patch-level encoder to generate patch-wise descriptors. A contrastive loss function aligns these image patches with the corresponding object embeddings within the scene. Concurrently, a compression loss ensures that the original U-3DGS component of the joint embedding remains accurately decodable. After this initial stage, all modules, including the U-3DGS components and auxiliary encoders, are jointly fine-tuned to enhance task-specific performance while preserving reconstruction fidelity.
Setup. Following evaluation protocol established by SceneGraphLoc (5), we sample 123 distinct scenes from 30 rooms within the 3RScan test split. For each query image, the objective is to identify correct scene from a candidate pool of 10 scenes, which includes the ground-truth scene. This experimental setup results in total of 462 query evaluations.
22

---

Ground Truth
3DGS
3DGS (12V)
2DGS (12V)
Object-X
Object-X Optim
Figure 8: Qualitative comparison for full-scene composition. We compare the proposed Object-X X standard 3DGS (28) optimized all unmasked scene images, and two 12-view baselines: 3DGS (12V) and 2DGS (12V), which optimize scenes using subset of training images constructed by taking the union of the 12 best views selected per object.
Evaluation. At test time, each posed RGB image is encoded into set patch-level embeddings. For every candidate scene in the pool, these image patch embeddings compared against all available object embeddings from that scene using cosine similarity. The final prediction for the scene is determined via robust voting mechanism that aggregates all patch-object similarity scores. We report @ecall@1 for K E (1,3,5) 5), which measures the frequency with which the correct scene appears among the top-K predicted scenes. This metric directly reflects the model's capability to localize images effectively using the learned, object-centric multimodal representation.
Results.
We compare our results with SceneGraphL.oc (15) and the recent CrossOver method (24). Our approach demonstrates competitive localization accuracy while crucially maintaining compat- ibility with 3D reconstruction and other downstream applications, benefit stemming from
23

---

jointly trained, modular representation. Detailed results are presented in Table In this table, we indicate whether given method utilizes point clouds (P), images (I), other modalities such as object attributes and relationships (O), or the proposed U-3DGS embedding. Our method, leveraging U-3DGS embeddings conjunction with other modalities, achieves the highest Recall and Recall @3 scores. This outcome suggests that the proposed U-3DGS embeddings furnish information comparable to, or even richer than, that provided by raw point clouds or images for the task of visual localization.
Furthermore, we present an ablation study for our method (indicated with an asterisk * in Table using only the U-3DGS embeddings, without any specific fine-tuning of our main encoder for this localization task. As anticipated, the auxiliary modalities (attributes, relationships, etc.) offer valuable complementary information, contributing significantly to the superior performance of the full model. Interestingly, even in this constrained setting (U-3DGS embeddings alone, without targeted training), our method performs comparably to the recent CrossOver approach (24). This highlights the inherent richness and suitability of our learned U-3DGS embeddings for visual localization tasks, even without explicit optimization for this specific application.
Modalities
10 scenes @ecall@3
Method
P
I
O
3DGS
Recall @
Recall @
SGLoc
15
/
X
X
53.6
81.9
92.8
CrossOver
244
/
46.0
77.9
90.5
Object-X
X
X
/
56.6
82.2
91.8
Object-X*
X
x
X
/
28.7
58.5
76.5
Object-X"
X
X
V
V
44.8
72.6
85.7
Table Coarse visual localization on the 3RScan dataset (9) using the proposed U-3DGS embed-
ding, compared to SceneGraphLLoc (15) and CrossOver (24). We report retrieval recall at 1, 3, and when selecting the correct scene from 10 candidates. Evaluations are conducted using different map modalities: point cloud (P), image (I), other modalities (O), and 3DGS. In the lower section (*), we also present results where the U-3DGS embedding used without task-specific training.
D Supplementary: Scene Alignment
We provide additional details the setup and evaluation procedure for the 3D Scene Alignment task on the 3RScan dataset (9) following the protocol established by SGAligner (23).
Setup. To construct the evaluation data, we generate sub-scenes by selecting fixed-length sequences of consecutive RGB-D frames from the 3RScan validation set. Each such sequence is then fused into a partial 3D reconstruction using volumetric integration. This process results in total of 848 sub-scenes, each representing distinct viewpoint or region within an original, larger scene. From these sub-scenes, we create 1,906 puirs by selecting pairs that originate from the same ground-truth scene. These pairs are deliberately constructed to span a wide range of spatial overlap percentages (from 10% to 90%) thereby ensuring coverage of both straightforward and challenging alignment scenarios.
Evaluation. We extract object embeddings independently from each sub-scene. These embeddings are produced by the same network architecture and weights trained for the scene localization task, as detailed in Section C During the evaluation phase, we compute the cosine similarity between every object embedding in one sub-scene and all object embeddings in its paired sub-scene, For each object in the first sub-scene, candidate objects from the second sub-scene are ranked based on this similarity score. We evaluate the quality of these rankings using standard retrieval metrics: Mean Reciprocal Rank (MRR) and Hits K, where K E (1,2,.... 5). The Hits K metric measures the proportion of queries for which a correct match appears within the top-K ranked results, while MRR quantifies the average inverse rank of the first correct match.
24

---

E
Supplementary: Single-Image Reconstruction
We provide qualitative results for the comparison of MIDI and Object X. The scenes on which the experiment is tested on are made available in the code.
Ground Truth
Object-X (RGB-D)
Object-X (RGB)
Image
MIDI
Qualitative comparison between MIDI and Object-X on 3RScan single-image inputs. MIDI often produces realistic but misaligned shapes, while Object-X yields accurate and structurally coherent reconstructions that preserve scale and geometry.
Figure 9:
F
Ablation Studies
This section analyzes the impact of key components and design choices in our proposed method. O results as function of the compression rate, which is defined by the resolution of the underlying voxel grid, where each voxel stores eight parameters. As reference, we also report results for standard 3D Gaussian Splatting (3DGS). In addition to proposed 3D U-Net architecture for compression, we evaluate a naive downsampling approach that applies max pooling followed by interpolation.
The results corresponding to a 64 voxel grid resolution effectively represent our Structured Latent (SLat) representation without any subsequent compression, as this directly matches the original voxel resolution described in the main paper. The ablation results demonstrate that employing naive downsampling leads to significant degradation in accuracy as the resolution decreases. In contrast, our proposed U-Net maintains high fidelity with only a marginal loss accuracy, while substantially reducing the number of parameters required per object from 64 2 097 152 to
25

---

mere 8 8 4 096. Based on this analysis, we adopt a resolution of 16 for the compressed representation all our main experiments.
Table evaluates the robustness of our method to varying degrees of occlusion by systematically removing parts of object before it is encoded. Occlusion is simulated selecting random point on the object's 's surface and removing all geometry within a sphere of diameter d. The diameter d is defined approximately 40% of the object's volume being removed. The results indicate that even under severe occlusion, our proposed method maintains high reconstruction accuracy, thereby demonstrating its resilience to incomplete or missing input data.
fraction of the object's characteristic size; for example. d 0.4 corresponds to
Resolution
Method
LPIPS (Mean # +
Median
PSNR (Mean +
Median t
3DGS
0.086 + 0.082
0.060
30.15 + 5.06
30.14
64 (SLat)
0.094 + 0.101
0.059
27.30 + 6.13
27.06
323
Naive
0.124 = 0.126
0.076
25.28 5.51
25.80
3D Unet
0.099 0.108
0.060
27.06 + 6.18
26.84
Naive
0.189 = 0.137
0.137
21.32 5.53
21.41
163
3D Unet
0.103 = 0.113
0.062
27.01 = 6.29
26.71
Naive
0.257 = 0.187
0.211
17.46 + 5.26
16.86
83
3D Unet
0.110 = 0.119
0.065
26.74 + 6.35
26.50
Table 6: Ablation study on latent dimensions. Mean and median LPIPS and PSNR on subset of scans from the test set. We compare the standard 3DGS (as a reference), the SLat embedding without dimensionality reduction, and U-3DGS with compressed representations at 32 16 and 83, Also, we evaluate naive downscaling approaches using max pooling and interpolation alongside the proposed 3D U-Net. The 16 resolution is selected for all other experiments significantly reduces storage while maintaining near-optimal reconstruction accuracy.
d
LPIPS (Mean o) |
Median
PSNR (Mean + σ) f
Median t
0.0
0.104 + 0.114
0.062
26.85 6.25
26.66
0.1
0.104 + 0.113
0.063
26.96 6.32
26.69
0.2
0.106 + 0.114
0.064
26.70 + 6.44
26.50
0.4
0.113 0.119
0.068
26.07 6.70
25.93
Table 7: Ablation study on occlusion. Before encoding an object, we randomly select a point on its surface and remove all parts within spherical region of diameter d. For example, d = 0.4 corresponds to removal region spanning 40% of the object's 's size. We report LPIPS and PSNR scores for different values of d to assess the impact of occlusion reconstruction quality.
26


Reviews:
[{"review_id": "awa8GOHt0D", "reviewer": "Reviewer_fTgB", "summary": "Conventional approaches in 3D reconstruction and rendering typically rely on either implicit neural representations like 3D Gaussian Splatting or object-centric representations such as 3D scene graphs, but rarely both. The former lacks object-level modularity, making it difficult to reason about individual objects or integrate other modalities (e.g., text), while object-centric embeddings struggle to reconstruct high-fidelity appearance and geometry, thereby requiring the storage of complementary original source data such as images, point clouds, and meshes. Object-X seeks to capture advantages of both methods by first learning a unified object embedding that fuses geometry and semantics, then decoding this embedding into a set of 3D Gaussians. The approach involves voxelizing objects into a 3D grid, aggregating local image features into a structured set of latent vectors, compressing them into an unstructured latent representation, and decoding to 3DGS for the object. Experiments show that Object-X achieves high geometric accuracy comparable to specialized 3DGS-based methods on tasks such as object, scene, and single-image reconstruction, while requiring much less storage and inference time. Qualitative results indicate that the model excels in single-image reconstruction and captures correct object semantics in scenes, but further refinement seems necessary for high-quality full-scene reconstruction.", "questions": "Please see the weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nNovel approach that combines implicit 3DGS representations and object-centric scene graph representations.\nEvaluation over diverse tasks: object, scene, and single-image 3D reconstruction.\nClear benefits in storage efficiency and inference time, which are important for scaling 3D object and scene generation.\nDemonstrates competitive downstream performance on tasks like localization and alignment, showing great utility of the unified embedding.\n\n\nWeaknesses:\nThe paper is missing ablations on removing the 3D scene graph representation, so it is not clear how critical 3DSG is to the overall method. For example, Table 3 shows that SGAligner achieves high performance without using 3DSG, raising questions about the specific contributions of the scene graph. Including an ablation or more analysis clarifying the necessity and impact of the scene graph within Object-X would strengthen the work.\nThere is a reliance on generating accurate scene graph annotations (e.g., SceneGraphFusion), which could become a bottleneck if the quality of the generated 3D scene graphs is poor. It would be beneficial for the authors to discuss the impact of noisy or erroneous scene graphs or include experiments that measure performance under imperfect annotations.\nWhile single-image object reconstruction is impressive and Object-X captures object semantics well, the full-scene composition appears significantly worse in quality when using the vanilla Object-X model. The need for additional refinement to achieve high scene quality potentially negates the storage and inference time benefits in practical full-scene settings. More discussion or experiments on this trade-off, and whether further improvements could address it, would be helpful.\nThe reason for the high geometric accuracy and F1 scores in Table 1 is not clearly explained, especially given the smaller relative improvements in other metrics like SSIM and PSNR. Providing more analysis, or qualitative examples explaining these discrepancies, would be helpful.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns about reliance on 3DSG, and the additional single-image results show the benefits of their multi-modal embedding. The quality of full-scene composition is sacrificed compared to the SoTA works, though authors justify this with their storage efficiency. The trade-offs should be adequately explained for Object-X to be effective. I maintain my rating.", "rebuttal": "We thank the reviewer for their feedback and for recognizing Object-X as a “compact latent representation” that decodes to 3D Gaussians with “similar photometric quality and better geometric accuracy,” and for appreciating its “voxel-grounded design” enabling downstream tasks like localization and scene alignment.\n\n**Misunderstanding regarding 3DSG / 3DGS.**  \nWe thank the reviewer for raising this point and apologize for the confusion caused by this *typographical* oversight. In several instances, \"3DSG\" (3D Scene Graph) was incorrectly used where we intended to write \"3DGS\" (3D Gaussian Splatting).  \n\nTo clarify: *the core Object-X pipeline does not require 3D scene graphs*. Object-X only needs object segmentations as input and learns a unified object embedding that can be decoded into 3DGS for geometry and rendering. Scene graphs (*3DSG*) are used *only as auxiliary inputs for specific downstream tasks* such as scene retrieval and localization (see Sec. 4.3), where relational cues between objects are beneficial. Thus, the strong results in object reconstruction (Tab. 1, main paper) and single-image reconstruction (Fig. 4, main paper) are obtained *without* any 3DSG input. We will fix this typo in the camera-ready version.\n\n**Results without 3D Scene Graphs (3DSG).**  \nWe provide ablation studies for visual localization in Table 1 of the supplementary material. Specifically, we compare the proposed Object-X without task-specific training (denoted as \\*), which uses only the proposed U-3DGS embedding, against a variant augmented with additional modalities derived from the 3DSG (i.e., object attributes and relational information). Object-X*, using only the 3DGS embedding (without 3DSG), achieves competitive performance. As expected, incorporating these additional modalities further improves results: Object-X with task-specific training and 3DSG augmentation achieves state-of-the-art Recall@1 and Recall@3, demonstrating that while 3DSG features are not required, they provide complementary benefits when available. We will clarify this in the final manuscript.\n\n**Impact of noisy or erroneous scene graphs (3DSG).**  \nAs noted above, concerns regarding noisy or erroneous scene graphs apply only to downstream auxiliary tasks (scene retrieval and localization). To evaluate robustness explicitly, we use two distinct scenarios:  \n\n- 3RScan, which provides high-quality, manually annotated 3D scene graphs (via SGAligner), serving as a clean, controlled benchmark; and  \n- ScanNet, which does not have annotated scene graphs, requiring us to automatically predict scene graphs using SceneGraphFusion.  \n\nThe latter introduces real-world noise, incompleteness, and segmentation artifacts. Object-X demonstrates strong performance on ScanNet despite *no retraining* (Table 1, main paper, bottom), outperforming baselines in both photometric and geometric metrics. This indicates our learned embeddings are resilient to annotation noise and generalize effectively, even with noisy or incomplete 3DSG inputs.\n\n**Full-scene composition quality.**  \nWe thank the reviewer for this insightful comment. As explicitly stated in Sec. 4.2 and the Limitations section, *full-scene composition is not the primary focus of this work*. Instead, our objective is to learn reconstructable, multi-modal object embeddings. The scene composition experiments (Table 2, main paper) demonstrate only the *potential viability* of Object-X for larger-scale reconstruction with further research.\n\nReduced photometric quality of the vanilla model primarily arises from two factors:  \n\n- the fixed-resolution voxel grid, which under-represents large surfaces (e.g., walls); and  \n- imperfect object segmentations that omit small or thin structures (e.g., posters, clutter).  \n\nDespite these limitations, Object-X achieves the highest geometric accuracy (F1@0.05 = 49.99%), running approximately 150× faster than the second-fastest baseline. Furthermore, after a lightweight refinement step (Object-X + Opt), run for the same time budget as the second-fastest method, Object-X surpasses 3DGS (12V) in photometric quality while preserving its geometric advantage. This underscores Object-X’s strength as a fast, high-quality initializer for scene-level refinement. We will further clarify this in the final version of the manuscript.\n\n**Reason for the high geometric accuracy.**  \nWe appreciate the reviewer’s observation. The higher geometric accuracy (F1 score) is due to the voxel-grounded nature of our latent representation. Unlike standard 3DGS, which uses an unstructured set of Gaussians, our method first embeds features within a structured voxel grid. This ensures the decoded Gaussians are spatially constrained and surface-aligned, improving geometric consistency and convergence, even from sparse inputs (e.g., single images). In contrast, photometric metrics such as SSIM and PSNR are more sensitive to fine-grained appearance details (e.g., textures), which are partially smoothed due to the fixed voxel resolution. This explains the larger relative improvement in geometric metrics compared to photometric ones. While we cannot include additional visuals in the rebuttal, we will expand this analysis and provide qualitative examples in the camera-ready version.\n\n**Additional single image reconstruction results.**  \nWe further compare Object-X to the very recent MIDI method [Huang et al., CVPR 2025] on a subset of the ScanNet dataset. Neither Object-X nor MIDI was trained on ScanNet. While MIDI is specifically designed for single-image 3D reconstruction, Object-X was not explicitly trained for this task but is designed as a versatile, multi-modal embedding for various downstream applications.  \n\nThe table below reports geometric accuracy for Object-X using only RGB input (directly comparable to MIDI), RGB-D input, and multiple images via the full pipeline (*). Despite its broader scope, Object-X significantly outperforms MIDI even in the RGB-only setting. We will include this result into the final manuscript.\n\n| **Method**           | **Accuracy @ 0.05** ↑ | **Completion @ 0.05** ↑ | **F1 @ 0.05** ↑ |\n|----------------------|----------------------|-------------------------|-----------------|\n| MIDI (CVPR 2025)     | 29.522               | 44.342                  | 34.516          |\n| Object-X (RGB)       | 43.480               | 57.214                  | 48.397          |\n| Object-X (RGB-D)     | 65.780               | 61.759                  | 63.223          |\n| Object-X*            | 79.289               | 89.911                  | 83.616          |"}, {"review_id": "Wa1TmRDE7T", "reviewer": "Reviewer_uwjw", "summary": "This paper proposes Object-X, a framework to learn latent object embeddings from multi-view images that can be decoded to 3DGS reconstructions and are 3-4 orders smaller in storage compared to raw sensor data. The embeddings are learned by first encoding multi-view RGB-D observations into a structured latent representation, then further encoding it to a fixed-length dense descriptor. The decoded 3DGS has comparable photometric quality to 3DGS/2DGS optimized with sparse views and better geometric accuracy. They also show the feasibility of augmenting the latent embedding with task-specific parts, such as ones useful for visual localization and 3D scene alignment.", "questions": "- How does the method compare to a baseline that directly fuses masked depth inputs with TSDF and run marching cubes for mesh extraction? \n- How is the storage computed for baselines and the proposed method? If the current storage is based on the size of the input images, what is the size of the Gaussian representation (using default setting and SH degree=0 settings)? (There are also many research works focusing on Gaussian compression that effectively reduce raw Gaussian sizes. I wonder how the proposed method’s storage compares with compressed Gaussians.)\n- While the experiment shows some benefit from the reconstruction-based latent code to the task of visual localization, whether this benefit 1) persists when multiple tasks are learned together, 2) is worthy of the more complicated training procedure should be further discussed.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- The proposed object latent representation is compact in storage size, and can be decoded into 3DGS reconstructions with similar photometric quality and better geometric accuracy compared to optimization-based 3DGS/2DGS with sparse inputs. The intermediate representation - structured latent representation grounded in voxels - contributes to the geometric accuracy.\n- The learned latent embeddings can be further augmented with task-specific portions and corresponding training objectives, such as the ones for visual localization and 3D scene alignment. The paper demonstrates the idea of representing the scene as object latents capable of both reconstruction and downstream tasks.\n\nWeaknesses\n- Reconstruction quality - the visualization in the supplementary materials looks quite smoothed out and lacks texture details. Despite achieving a high compression rate, the quality seems to be sacrificed. The better geometric accuracy can be attributed to the voxel-based representation. How does the method compare to a baseline that directly fuses masked depth inputs with TSDF and runs marching cubes for mesh extraction? \n- Comparison to baselines \n    - 3DGS/2DGS are constrained to a sparse view setting, which I’m not sure is a fair setting, given that they are optimization-based and have no learned prior. \n    - DepthSplat is trained on scene-level data, where input frames have overlaps. However, it is evaluated on object-level data with non-overlapping frames (L265: while in ScanNet, where fewer non-overlapping frames are available, we limit the selection to four frames per object.) Although the paper claims “neither methods are trained on ScanNet”, the train-test domain gap is still much smaller for the proposed method, putting depthsplat at an unfair disadvantage. \n    - Why are baselines taking 12 views as *input* considered as “(L253-254) *stores* each object as 12 images”? Are the storage in Table 1 computed using the size of these input images, instead of their output Gaussian representation? If so, why is the storage of the proposed method computed differently, based on the output latents? \n- Support for downstream tasks - The support for downstream tasks is achieved by attaching another latent code to the original one, and training them with two-stage joint training. This requires more training time compared to directly training a separate task-specific latent code for each task, especially when given multiple downstream tasks. While the experiment shows some benefit from the reconstruction-based latent code to the task of visual localization, whether this benefit 1) persists when multiple tasks are learned together, 2) is worthy of the more complicated training procedure, should be further discussed.\n- Confusing technical specifications \n    - In the “auxiliary encoder training with frozen core” stage, if both U-3DGS encoder and decoder are frozen, and the reconstruction loss is only applied using the U-3DGS decoder that operates on the U-3DGS portion of the latent code - which should come from the frozen U-3DGS encoder - everything seems frozen in the reconstruction loss part, and the loss is redundant. It would be less confusing to only keep the task loss in this stage and introduce the reconstruction loss in the following joint training stage.\n    - $L$ is first used to refer to the sparse voxels. If the compression loss also sums over $L$ sparse voxels, shouldn’t $M_i$ be all ones? It seems to make more sense if the loss sums over all dense voxels. $N$ is used to describe the size of the voxel grid, but later also used for the total number of Gaussians in (1). \n\n\nTypo - Fig 5. (b) and Table 3. 3DSG -> 3DGS", "was_revised": true, "final_justification": "I appreciate the authors' response, which answers many of my questions and partially addresses my concerns. I'm raising my rating to borderline accept with the condition that the following can be incorporated into the final manuscript.\n- Instead of claiming geometric accuracy, I think it would be more convincing to position the method's reconstruction ability as keeping Trellis' strengths while achieving significant compression and a fixed-length embedding more suitable for downstream tasks - as the authors summarized in their rebuttal. \n- It would also help to make clearer claims on multi-task versatility.", "rebuttal": "We thank the reviewer for their feedback and for recognizing Object-X as a “compact object latent representation” that decodes to 3D Gaussians with “similar photometric quality and better geometric accuracy”. We also appreciate their acknowledgment of our “voxel-grounded latent representation” and its ability to “augment embeddings with task-specific components”, enabling tasks like visual localization and scene alignment.\n\n**Inaccuracy in the Summary.**  \nWe thank the reviewer for summarizing our contributions, but would like to correct a misunderstanding: *Object-X does not take RGB-D observations as input.* As described in Sec. 3.1, our method operates on multi-view RGB images and a coarse geometric representation (mesh or point cloud), which can be obtained from standard multi-view stereo or similar sources. While such geometry may be derived from RGB-D sensors in practice, it is *not* a requirement of our pipeline. We will clarify this in the paper.\n\n**Quality and Texture Detail.**  \nWe appreciate the reviewer’s observation. The smoother appearance of our reconstruction stems from the fixed-resolution voxel grid used to ground the latent representation, which trades some high-frequency texture detail for compactness and geometric consistency. This design enables extreme compression (3–4 orders of magnitude reduction in storage) while maintaining strong geometric fidelity (highest F1@0.05 in Tab. 1).\n\nFor *object reconstruction*, this trade-off still results in *higher photometric accuracy than 2DGS, 3DGS, and DepthSplat*, while significantly improving geometric quality due to the voxelized representation.\n\nFor *scene reconstruction*, as discussed in Sec. 4.2 and the Limitations, reduced photometric quality in the vanilla model arises mainly from (i) fixed-resolution voxels under-representing large surfaces (e.g., walls), and (ii) missing segmentations for small or thin objects. A lightweight refinement step (Object-X+Opt), run for the same time budget as the second-fastest baseline, recovers photometric quality comparable to fully optimized 3DGS (12V) while preserving our geometric advantage and efficiency. This underscores Object-X’s strength as a fast, high-quality initializer for scene-level refinement. \n\nCrucially, Object-X achieves this while producing a *single compact embedding* usable across diverse downstream tasks (e.g., localization, scene alignment, single-image reconstruction), underscoring the broader utility of our representation. We will clarify this trade-off and add qualitative examples in the supplementary material.\n\n**Comparison to TSDF + Marching Cubes.**  \nWe thank the reviewer for this suggestion. We note that the ground-truth reconstructions in 3RScan are derived from fused RGB-D frames, whereas Object-X is not an RGB-D reconstruction method but, instead, learns compact object embeddings from multi-view RGB images and coarse geometric priors. While a TSDF + Marching Cubes baseline could provide additional context for pure geometry reconstruction, our goal is to deliver a versatile, multi-modal representation that supports _both_ geometric and photometric reconstruction as well as downstream tasks, rather than performing dense RGB-D fusion. We will clarify this distinction in the final manuscript.\n\n**Fairness in the 3DGS/2DGS Comparison.**  \nWe thank the reviewer for raising this concern. There is no standardized object-level reconstruction protocol for 3DGS. We chose a sparse-view setting of up to 12 views per object (Sec. 4.1) primarily to ensure a fair comparison with recent learned-prior methods like DepthSplat, which accepts a maximum of 12 views as input. This setting also reflects the natural data sparsity in datasets like 3RScan and ScanNet, where the average number of total views per object is 20 (with ~50% overlap), making our setup a realistic evaluation scenario. Importantly, the 3DGS/2DGS baselines were optimized directly on these views without compression constraints, whereas Object-X encodes them in a compact latent embedding. We will clarify this in the camera ready.\n\n**Unfair Comparison to DepthSplat (\"evaluated on object-level data with non-overlapping frames\").**  \nWe believe this misunderstanding stems from our incorrectly formalized sentence at L265, and we apologize for that. The claim that the selected frames are non-overlapping is incorrect: by construction, the selected views for each object do overlap (otherwise, the object would not be visible in all selected frames; ~50% overlap on average). DepthSplat was trained to handle even lower overlaps. This has now been clarified in the paper. \n\nTo ensure a fair comparison, we evaluated DepthSplat in the same configuration it was trained for: on unmasked, scene-level images. We only applied the object masks *after* reconstruction to remove background Gaussians. Thus, DepthSplat was used under its intended operating conditions, making the comparison to Object-X fair. We will revise the text to make this procedure clear.\n\n**Storage Computation for Baselines.**  \nWe thank the reviewer for raising this important point. For 3DGS/2DGS baselines, we reported storage based on the size of the 12 input images, as their fully optimized Gaussian representations require significantly more storage: on average, 28.41 MB per object, reduced to 7.79 MB (mean, up to 30 MB in some cases) even with aggressive compression (SH=0; leading to reduced accuracy). Object-X encodes objects into a fixed-size latent descriptor of only 4.2 KB, achieving a 3–4 order-of-magnitude storage reduction. Detailed numbers will be included in the supplementary material.\n\nAlthough advanced Gaussian compression methods exist, our approach uniquely provides:  \n(i) multi-modal embeddings supporting downstream tasks (localization, alignment),  \n(ii) extremely fast reconstruction (~150× faster than the next-best baseline), and  \n(iii) single-image reconstruction capabilities.\n\n**Increased Training Time with Downstream Tasks.**  \nWe thank the reviewer for this insightful comment. The additional training time introduced by our two-stage training procedure is a one-time cost, as the embedding is trained once and reused zero-shot on other tasks, making this overhead negligible in practice.\n\nRegarding the persistence of benefits when jointly training multiple tasks: as discussed in Sec. 4.3, Object-X jointly trains object reconstruction and localization, achieving strong performance across various tasks, including *zero-shot* scene alignment and single-image reconstruction. We will clarify this explicitly in the manuscript.\n\nOn the justification of training complexity: supplementary Table 1 clearly demonstrates substantial improvements from task-specific training - improving Recall@1, Recall@3, and Recall@5 by 11.8, 10.4, and 6.1 points, respectively. This shows that multi-task training on a shared reconstruction-informed embedding consistently boosts performance, fully justifying the additional complexity.\n\n**Confusing technical specifications.**  \nWe thank the reviewer for highlighting these points and will revise the manuscript accordingly. Specifically, we will clearly define the voxel occupancy term, restrict the auxiliary encoder’s reconstruction loss to the correct training stage, and resolve all notational inconsistencies (e.g., mislabeling of L). Typos such as \"3DSG → 3DGS\" in Fig. 5(b) and Table 3 will be corrected. We appreciate the reviewer’s detailed feedback and will incorporate these suggestions to enhance clarity.\n\n\n**Additional single image reconstruction results.**  \nWe further compare Object-X to the very recent MIDI method [Huang et al., CVPR 2025] on a subset of the ScanNet dataset. Neither Object-X nor MIDI was trained on ScanNet. While MIDI is specifically designed for single-image 3D reconstruction, Object-X was not explicitly trained for this task as it is designed as a versatile, multi-modal embedding for various downstream applications.  \n\nThe table below reports geometric accuracy for Object-X using only RGB input (directly comparable to MIDI), RGB-D input, and multiple images via the full pipeline (*). Despite its broader scope, Object-X significantly outperforms MIDI even in the RGB-only setting. We will include this result in the final manuscript.\n\n| **Method**           | **Accuracy @ 0.05** ↑ | **Completion @ 0.05** ↑ | **F1 @ 0.05** ↑ |\n|----------------------|----------------------|-------------------------|-----------------|\n| MIDI (CVPR 2025)     | 29.522               | 44.342                  | 34.516          |\n| Object-X (RGB)       | 43.480               | 57.214                  | 48.397          |\n| Object-X (RGB-D)     | 65.780               | 61.759                  | 63.223          |\n| Object-X*            | 79.289               | 89.911                  | 83.616          |"}, {"review_id": "TLjlaEhWOz", "reviewer": "Reviewer_J59e", "summary": "This paper proposes a framework for learning compact, multi-modal (e.g., images, point cloud, text), object-centric embeddings that can be decoded into 3D Gaussians.\nThis is in contrast to task-specific embeddings that cannot be decoded into explicit geometry and reused across tasks.  The learned embedding enables 3DGS-based object reconstruction. It also supports other downstream\ntasks, such as scene alignment, single-image 3D object reconstruction, and localization.", "questions": "See weaknesses above.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "A notable advantage of this approach is the significant storage reduction it achieves, while maintaining performance comparable to existing methods.\n\nEncouraging  results in multi task evaluation, object reconstruction, visual localisation and 3D scene alignment.\n\nThe paper suffers from clarity issues in both writing and presentation. The notations are inconsistent and often confusing. For instance, in the beginning of Section 3.1, the object latent embedding $\\mathbf{w}$ is introduced, but its meaning and role are not explained until Section 3.2. In Line 158, the 3D encoder is incorrectly denoted as $E$; it should be $\\mathcal{E}$. In Section 3.4, $L_{recon}$ is not clearly explained.\n \nFurthermore, Figure 2 lacks a clear explanation of the overall idea. The relationships among the featured voxel grid, the sparse encoder, the structured latent space, and the sparse decoder are not well articulated. Additionally, important notations such as $\\mathbf{z}$, $\\mathbf{f}$, $\\mathbf{w}$, $f_{\\text{U-3DGS}}$, and $f_{\\text{decomp}}$ are not shown in the figure, making it difficult for readers to follow the pipeline.\n \nAlthough the proposed compression from $\\mathbf{z}$ to $\\mathbf{w}$ contributes significantly to the storage savings, this idea is already present in previous works that decode Gaussians from latent space. The main difference might be this paper uses a fixed-size vector which is independent of the size of the object. Moreover, the paper omits important implementation details about how the pre-trained DepthSplat model is integrated into their method.\n\nIt indeed has practical value as it reduces the storage and the motivation is good. The overall framework is reasonable, although the core idea is not particularly novel. The writing is not clear, making it difficult to follow the implementation details. As a result, it becomes hard to assess the fairness and reproducibility of the experiments. Additionally, the evaluation is not sufficiently comprehensive, as it lacks comparisons with stronger state-of-the-art methods, such as NoPoSplat.", "was_revised": true, "final_justification": "My main concerns are about the clarity of the presentation which the authors acknowledge and promise to improve. \nThe authors did not compare to NoPoSplat on the basis of unfair comparison.  This however would have be unfair to NoPoSplat which does not require poses in contrast to the proposed approach, thus addressing more challenging task as agreed by the authors. As such the proposed method is inferior to NoPoSplat but it does introduce a useful idea for efficiency.\nThe rebuttal did not improve my initial impression and I'm on the fence with this paper.", "rebuttal": "We thank the reviewer for their constructive feedback and for recognizing our framework’s “significant storage reduction” while maintaining strong performance. We also appreciate their acknowledgment of the “encouraging results in multi-task evaluation” and the effectiveness of our method in “object reconstruction, visual localization, and 3D scene alignment”.\n\n**Clarity issues.**  \nWe thank the reviewer for identifying these issues. We acknowledge that certain notations and explanations in Sections 3.1–3.4 were unclear or inconsistently presented. Specifically, we will:  \n(i) introduce and clearly explain the object latent embedding in Section 3.1 rather than delaying it to Section 3.2;  \n(ii) correct the mislabeling of the 3D encoder at Line 158; and  \n(iii) provide a more explicit and clear description of the components in Section 3.4.\n\n**Figure 2 and Pipeline Explanation.**  \nWe appreciate the reviewer pointing out the lack of clarity in Figure 2. In the revision, we will redesign Figure 2 to clearly illustrate the relationships among the voxel grid, sparse encoder, structured latent space, and sparse decoder, ensuring all relevant notations (e.g., latent codes, variables) are explicitly included. Unfortunately, due to rebuttal constraints, we cannot present the updated figure here, but it will be included in the final camera-ready version.\n\n**Comparison to Previous Works Decoding Gaussians from Latent Space.**  \nWe thank the reviewer for this comment. While we would be better able to address this with specific citations, we recognize that some recent works (such as DepthSplat, included as a baseline in Table 1) predict Gaussians from embeddings derived mainly from image features. However, to our knowledge, no previous method employs our approach of compressing object observations into a *fixed-size structured latent vector independent of object size*, enabling a uniform, highly compact representation that simultaneously supports high-quality reconstruction and diverse downstream tasks. We will clarify this distinction in the revised manuscript.\n\n**Description of DepthSplat Integration.**  \nWe thank the reviewer for pointing out this omission and apologize for the oversight. In our experiments, DepthSplat is used in its standard pre-trained configuration, applied to the same unmasked scene-level images used by other baselines and our proposed Object-X. Only after reconstruction, object masks are applied to remove background Gaussians, ensuring DepthSplat operates under its intended conditions and aligns with our object-centric evaluation protocol. We will detail this integration in the final manuscript.\n\n**Evaluation Lacks Comparison with Stronger State-of-the-Art Methods, such as NoPoSplat.**  \nWe respectfully disagree with the reviewer on this point. We benchmark against DepthSplat (CVPR 2025), a recent (published <2 months ago) and relevant state-of-the-art method specifically designed for sparse-view Gaussian reconstruction with known camera poses, making it a robust baseline for our evaluation. NoPoSplat, while an excellent method, addresses the different and more challenging task of jointly estimating camera poses and geometry from sparse input. Our method assumes known poses, making a direct comparison non-trivial without substantial modifications. We will clarify this distinction and explicitly position our contributions in relation to these prior works in the revised manuscript.\n\n**Additional single image reconstruction results.**  \nWe further compare Object-X to the very recent MIDI method [Huang et al., CVPR 2025] on a subset of the ScanNet dataset. Neither Object-X nor MIDI was trained on ScanNet. While MIDI is specifically designed for single-image 3D reconstruction, Object-X was not explicitly trained for this task but is designed as a versatile, multi-modal embedding for various downstream applications.  \n\nThe table below reports geometric accuracy for Object-X using only RGB input (directly comparable to MIDI), RGB-D input, and multiple images via the full pipeline (*). Despite its broader scope, Object-X significantly outperforms MIDI even in the RGB-only setting. We will include this result into the final manuscript.\n\n| **Method**           | **Accuracy @ 0.05** ↑ | **Completion @ 0.05** ↑ | **F1 @ 0.05** ↑ |\n|----------------------|----------------------|-------------------------|-----------------|\n| MIDI (CVPR 2025)     | 29.522               | 44.342                  | 34.516          |\n| Object-X (RGB)       | 43.480               | 57.214                  | 48.397          |\n| Object-X (RGB-D)     | 65.780               | 61.759                  | 63.223          |\n| Object-X*            | 79.289               | 89.911                  | 83.616          |"}, {"review_id": "ft523o22Vw", "reviewer": "Reviewer_SdV9", "summary": "This paper proposed a pipeline that can generate object-centric embeddings from input object segmentation. To do this, they create sparse feature voxels from multiview images with DINO feature, and use 3D Unet to furthur compress the representation.  \n\nThe main benifit of this representation contains rich DINO information and can be used for tasks that can tolarate the loss of fine-grained details, like localization and scene matching.\n\nThe representation can be decoded back to 3DGS, but the reconstructed object is not complete (if input multiview has occlusion) and somehow lost of detail.\n\nThis paper has similar idea to https://arxiv.org/pdf/2412.01506, but add an additional encoder to compress the sparse voxel.", "questions": "1. line 151, \"Given multi-view images of an object o, we first voxelize its canonical 3D space into an N × N × N grid\". Which algorithm do you use?\n\n2. In Tab. 1, why not test Object-X on all images like 3DGS reference?\n\n3. In Figure 4, does \"graph2splat\" refer to Object-X?  \n\n4. In Figure 4 Object reconstruction, even 3DGS and 2DGS are trained on multiview, the object still looks incomplete. Is it because the narrow distribution of the camera?\n\n5. For scene reconstruction, how to identify the instance mask for each object? And how to handle the background?\n \n\n6. In line 313 Coarse Visual Localization, ViT is used to extract per-patch object embeddings from the query image and compare with Obj-X embeddings. But according to Figure 2, the embeddings space are constrained by DINOv2. Will these two have different distribution?\n\n7. Since https://arxiv.org/pdf/2412.01506 has similar pipeline with this paper and can also output sparse voxel from RGB image, it may be a good comparision to test its performance on the object reconstruction experiment.", "limitations": "Not a big limitation, but I think the authors can add more techincal details for the experiment and more visulization result.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. the proposed method can generate an object mebedding that handles multiple tasks, like visual localization, scene alignment, scene reconstruction, \n\n2. The model achieve good performance, as shown in main paper and supplementary.\n\n3. The model has the potential to intergrate other pretrained features, like CLIP, although it's not explored in the paper.\n\nWeakness:\n\n1. The proposed method only reconstruct partial 3D instead of full 3D from the observations, as shown in Figure 4. Some feed-forward method can also reconstruct full 3D even from single RGB, like [MIDI](https://huanngzh.github.io/MIDI-Page/). If only target to reconstruct the partial 3D, I think adding monocular prior for 2DGS and 3DGS baseline, like [DepthAnything-v2](https://github.com/DepthAnything/Depth-Anything-V2) / [Stable normal](https://arxiv.org/pdf/2406.16864) / [VGGT](https://vgg-t.github.io/), can improve the fitting result.\n\n2. In Full-scene composition (Supp Fig. 3), Object-X seems loss some fine-grained details after reconstruction.", "was_revised": true, "final_justification": "Thanks the authors to respond to my comments. This paper use a 3D-Unet transfer the sparse representation to a fix size representation, which can preserve the reconstruction quality while adapt to more downstream tasks. I think the paper provides a interesting insight to the community and a good exploration to latent representation to 3D object. But I agree with uwjw that the authors still need more experiments to clearly show their technical contribution and prove that it can be useful in more downstream tasks.", "rebuttal": "We thank the reviewer for their constructive feedback and for recognizing our method’s ability to “generate an object embedding that handles multiple tasks, like visual localization, scene alignment, and scene reconstruction”. We also appreciate their acknowledgment that it “achieves good performance” and their note on its potential to integrate additional pretrained features (e.g., CLIP).\n\n**Some feed-forward methods can also reconstruct full 3D even from single RGB, like MIDI.**\n\nWe thank the reviewer for this suggestion and for pointing us to this recent baseline. We conducted a new experiment comparing Object-X to MIDI on the task of single-image 3D reconstruction using the authors’ released code. Both methods were evaluated in a zero-shot setting on objects extracted from two randomly chosen scenes in the ScanNet dataset, which neither Object-X nor MIDI had seen during training. The quantitative results are summarized below:\n\n| **Method**            | **Accuracy @ 0.05** ↑ | **Completion @ 0.05** ↑ | **F1 @ 0.05** ↑ |\n|-----------------------|----------------------|------------------------|-----------------|\n| MIDI (CVPR 2025)      | 29.522               | 44.342                 | 34.516          |\n| Object-X (RGB)        | 43.480               | 57.214                 | 48.397          |\n| Object-X (RGB-D)      | 65.780               | 61.759                 | 63.223          |\n| Object-X*             | 79.289               | 89.911                 | 83.616          |\n\nAs shown in the table above, Object-X _significantly outperforms_ MIDI in geometric fidelity, even in the purely single-image setting (RGB only; same configuration as MIDI). Single-image reconstruction with Object-X from an RGB-D input or reconstruction with the entire pipeline (*) significantly improves geometric accuracy further.\n\nUpon visual inspection, MIDI often appears to mimic the closest shape from the synthetic dataset it was trained on, which rarely coincides with the true geometry of the target object. This aligns with the fact that the original MIDI paper provides *no quantitative* evaluation on real-world datasets, focusing instead on synthetic data. Combined with our results, this suggests that MIDI may have limited generalization to challenging real-world data such as ScanNet. We will add visualizations to the final manuscript.\n\nThus, while MIDI remains an excellent and influential contribution, our experiment indicates that Object-X offers a more robust solution for single-image reconstruction in realistic settings. Furthermore, single-image reconstruction is only one of several applications supported by our learned embedding. As demonstrated in the main paper, the core contribution of Object-X is its versatile, multi-modal representation that achieves strong performance across diverse downstream tasks such as visual localization and 3D scene alignment. We will include this new comparison in the final manuscript to further underscore the robustness and generality of our approach.\n\n**Adding monocular priors for 2DGS and 3DGS baselines can improve fitting.**\n\nWe appreciate the reviewer’s suggestion to incorporate monocular priors such as DepthAnything-v2, Stable Normal, or VGGT. \nWhile these priors could potentially improve geometric fidelity, they introduce trade-offs: increased storage requirements (e.g., storing floating-point depth maps per image instead of compact descriptors per object) or additional computation time to run a monocular depth network before reconstruction, adding seconds to the pipeline. In contrast, our method achieves state-of-the-art geometric accuracy while using 3–4 orders of magnitude less storage (4.2 KB per object) than RGB images and reconstructing in just milliseconds. That said, we agree that a principled and efficient integration of monocular 3D priors within our framework is a compelling direction for future work, and we will note this in the revised manuscript.\n\n**In Full-scene Composition, Object-X Seems to Lose Fine-grained Details.**\n\nWe thank the reviewer for this observation. As noted in Sec. 4.2 and the Limitations section, full-scene composition is not the primary focus of Object-X. Reduced fine-grained detail in large-scale reconstructions primarily results from the fixed-resolution voxel grid, which under-represents large surfaces (e.g., walls), and missing segmentations of small or thin structures. Despite this, Object-X achieves the highest geometric accuracy while being significantly faster and more storage-efficient than baselines.  \n\nFurthermore, applying a lightweight refinement step (Object-X+Opt), within the same time budget as the second-fastest baseline, improves photometric quality to levels comparable to fully optimized 3DGS, while preserving our geometric advantage. This underscores Object-X’s strength as a fast, high-quality initializer for scene-level refinement. We will clarify this trade-off and include additional qualitative examples in the supplementary material.\n\n**Clarifications to Reviewer Questions.**\n\n- **Voxelization algorithm (line 151):**  \n  We scale the object into a unit cube and mark voxels as occupied if they intersect the mesh, utilizing `open3d.geometry.VoxelGrid`.\n\n- **Why not test Object-X on all images (Table 1)?**  \n  We selected 4 (ScanNet) and 12 (3RScan) images per object due to (i) limited additional images observing each object in these datasets, and (ii) fairness in comparison with DepthSplat, which can only process up to 12 images. We will clarify this in the camera-ready version.\n\n- **Does “graph2splat” refer to Object-X (Figure 4)?**  \n  Yes, “Graph2Splat” was an earlier name for Object-X. We thank the reviewer for highlighting this and will correct it in the final version.\n\n- **Incomplete objects even with multi-view training (Figure 4):**  \n  The incompleteness is a direct consequence of the limited camera coverage in datasets like ScanNet and 3RScan, where parts of some objects are never observed. While optimizing on all available views cannot fully invent unseen geometry, our voxel-grounded representation provides a strong structural prior that encourages plausible shape completion in these sparsely viewed regions. We quantitatively demonstrate this robustness to occlusion in supplementary Table 3, which shows that Object-X can compensate for such partial observations to a degree.\n\n- **Scene reconstruction: instance masks and background handling:**  \n  Many methods are available for obtaining 3D object instance segmentation from scene reconstructions, e.g., SceneGraphFusion (CVPR'21), MAP-ADAPT (ECCV'24), Mask3D (ICRA'23), OpenMask3D (NeurIPS'23), and Search3D (RA-L'25). For our experiments, we used the provided annotations on 3RScan and SceneGraphFusion on ScanNet. Regarding background handling, indoor scenes inherently include objects (e.g., walls, furniture) in all pixels of all views. Extending our method to unbounded outdoor settings would require mask pre-filtering, which we consider as future work.\n\n- **Distribution differences between ViT and DINOv2 features (line 313):**  \n  The ViT model used for query embedding belongs to the same model family (DINOv2 with additional projection layers) as the one used for generating structured object features, ensuring compatibility.\n\n- **Comparison to Trellis (sparse voxel from RGB):**  \n  We implemented a variant inspired by this approach (64³ voxel encoder in supplementary Table 2). This method was memory-intensive (requiring 64³ floating-point numbers compared to our 16³) and generalized poorly to auxiliary tasks due to its sparse representation. Our compressed latent embedding offers significantly lower memory usage and improved task generalizability, while maintaining comparable reconstruction accuracy."}]

Author rebuttal:

Meta-review: The proposed Object-X aims to learn compact, multi-modal, object-centric embeddings that can be reconstructed into high-fidelity 3D geometry and appearance. It is designed to be descriptive for various downstream tasks while maintaining storage efficiency. Several new modules are introduced to achieve this goal, including Structured Latent Representation (SLat), Unstructured U-3DGS Embedding, and a 3D Gaussian Splatting Decoder.

This paper received final ratings of "borderline accept" from all reviewers. One reviewer increased the initially negative rating, while the others maintained their scores. The primary concerns raised by the reviewers included unclear technical contributions, fairness of evaluations, and comparisons with baseline methods. To address these issues, the authors conducted extensive experiments, including single-image object reconstruction and TSDF fusion, along with providing clarifications and explanations.

The reviewers acknowledged the effort made by the authors during the rebuttal process and the subsequent discussion. They also recognized the advantages of the proposed method, noting its ability to achieve significant compression and provide a fixed-length embedding that is more suitable for downstream tasks. Furthermore, regarding the claim of multi-task versatility, the reviewers suggested that the revision should include additional experiments to better illustrate the technical contributions and demonstrate the method's usefulness across a broader range of downstream tasks. 

Considering the reviewers' mostly positive feedback, the area chair recommends accepting this paper.
Decision: Accept (poster)