Title: Object-X: Learning to Reconstruct Multi-Modal 3D Object Representations
Venue: NeurIPS 2025
Accepted: yes
Keywords: Object Reconstruction

Gaussian Splatting

Multi-Modal Representation Learning
Abstract: Learning effective multi-modal 3D representations of objects is essential for numerous applications, such as augmented reality and robotics. 
Existing methods often rely on task-specific embeddings that are tailored either for semantic understanding or geometric reconstruction. 
As a result, these embeddings typically cannot be decoded into explicit geometry and simultaneously reused across tasks.
In this paper, we propose Object-X, a versatile multi-modal object representation framework capable of encoding rich object embeddings (e.g., images, point cloud, text) and decoding them back into detailed geometric and visual reconstructions. 
Object-X operates by geometrically grounding the captured modalities in a 3D voxel grid and learning an unstructured embedding fusing the information from the voxels with the object attributes. 
The learned embedding enables 3D Gaussian Splatting-based object reconstruction, while also supporting a range of downstream tasks, including scene alignment, single-image 3D object reconstruction, and localization.
Evaluations on two challenging real-world datasets demonstrate that Object-X produces high-fidelity novel-view synthesis comparable to standard 3D Gaussian Splatting, while significantly improving geometric accuracy. 
Moreover, Object-X achieves competitive performance with specialized methods in scene alignment and localization.
Critically, our object-centric descriptors require 3-4 orders of magnitude less storage compared to traditional image- or point cloud-based approaches, establishing Object-X as a scalable and highly practical solution for multi-modal 3D scene representation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a pipeline that can generate object-centric embeddings from input object segmentation. To do this, they create sparse feature voxels from multiview images with DINO feature, and use 3D Unet to furthur compress the representation.  \n\nThe main benifit of this representation contains rich DINO information and can be used for tasks that can tolarate the loss of fine-grained details, like localization and scene matching.\n\nThe representation can be decoded back to 3DGS, but the reconstructed object is not complete (if input multiview has occlusion) and somehow lost of detail.\n\nThis paper has similar idea to https://arxiv.org/pdf/2412.01506, but add an additional encoder to compress the sparse voxel.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. the proposed method can generate an object mebedding that handles multiple tasks, like visual localization, scene alignment, scene reconstruction, \n\n2. The model achieve good performance, as shown in main paper and supplementary.\n\n3. The model has the potential to intergrate other pretrained features, like CLIP, although it's not explored in the paper.\n\nWeakness:\n\n1. The proposed method only reconstruct partial 3D instead of full 3D from the observations, as shown in Figure 4. Some feed-forward method can also reconstruct full 3D even from single RGB, like [MIDI](https://huanngzh.github.io/MIDI-Page/). If only target to reconstruct the partial 3D, I think adding monocular prior for 2DGS and 3DGS baseline, like [DepthAnything-v2](https://github.com/DepthAnything/Depth-Anything-V2) / [Stable normal](https://arxiv.org/pdf/2406.16864) / [VGGT](https://vgg-t.github.io/), can improve the fitting result.\n\n2. In Full-scene composition (Supp Fig. 3), Object-X seems loss some fine-grained details after reconstruction.\n\nQUESTIONS:\n1. line 151, \"Given multi-view images of an object o, we first voxelize its canonical 3D space into an N × N × N grid\". Which algorithm do you use?\n\n2. In Tab. 1, why not test Object-X on all images like 3DGS reference?\n\n3. In Figure 4, does \"graph2splat\" refer to Object-X?  \n\n4. In Figure 4 Object reconstruction, even 3DGS and 2DGS are trained on multiview, the object still looks incomplete. Is it because the narrow distribution of the camera?\n\n5. For scene reconstruction, how to identify the instance mask for each object? And how to handle the background?\n \n\n6. In line 313 Coarse Visual Localization, ViT is used to extract per-patch object embeddings from the query image and compare with Obj-X embeddings. But according to Figure 2, the embeddings space are constrained by DINOv2. Will these two have different distribution?\n\n7. Since https://arxiv.org/pdf/2412.01506 has similar pipeline with this paper and can also output sparse voxel from RGB image, it may be a good comparision to test its performance on the object reconstruction experiment.\n\nLIMITATIONS:\nNot a big limitation, but I think the authors can add more techincal details for the experiment and more visulization result.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a framework for learning compact, multi-modal (e.g., images, point cloud, text), object-centric embeddings that can be decoded into 3D Gaussians.\nThis is in contrast to task-specific embeddings that cannot be decoded into explicit geometry and reused across tasks.  The learned embedding enables 3DGS-based object reconstruction. It also supports other downstream\ntasks, such as scene alignment, single-image 3D object reconstruction, and localization.\n\nSTRENGTHS AND WEAKNESSES:\nA notable advantage of this approach is the significant storage reduction it achieves, while maintaining performance comparable to existing methods.\n\nEncouraging  results in multi task evaluation, object reconstruction, visual localisation and 3D scene alignment.\n\nThe paper suffers from clarity issues in both writing and presentation. The notations are inconsistent and often confusing. For instance, in the beginning of Section 3.1, the object latent embedding $\\mathbf{w}$ is introduced, but its meaning and role are not explained until Section 3.2. In Line 158, the 3D encoder is incorrectly denoted as $E$; it should be $\\mathcal{E}$. In Section 3.4, $L_{recon}$ is not clearly explained.\n \nFurthermore, Figure 2 lacks a clear explanation of the overall idea. The relationships among the featured voxel grid, the sparse encoder, the structured latent space, and the sparse decoder are not well articulated. Additionally, important notations such as $\\mathbf{z}$, $\\mathbf{f}$, $\\mathbf{w}$, $f_{\\text{U-3DGS}}$, and $f_{\\text{decomp}}$ are not shown in the figure, making it difficult for readers to follow the pipeline.\n \nAlthough the proposed compression from $\\mathbf{z}$ to $\\mathbf{w}$ contributes significantly to the storage savings, this idea is already present in previous works that decode Gaussians from latent space. The main difference might be this paper uses a fixed-size vector which is independent of the size of the object. Moreover, the paper omits important implementation details about how the pre-trained DepthSplat model is integrated into their method.\n\nIt indeed has practical value as it reduces the storage and the motivation is good. The overall framework is reasonable, although the core idea is not particularly novel. The writing is not clear, making it difficult to follow the implementation details. As a result, it becomes hard to assess the fairness and reproducibility of the experiments. Additionally, the evaluation is not sufficiently comprehensive, as it lacks comparisons with stronger state-of-the-art methods, such as NoPoSplat.\n\nQUESTIONS:\nSee weaknesses above.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Object-X, a framework to learn latent object embeddings from multi-view images that can be decoded to 3DGS reconstructions and are 3-4 orders smaller in storage compared to raw sensor data. The embeddings are learned by first encoding multi-view RGB-D observations into a structured latent representation, then further encoding it to a fixed-length dense descriptor. The decoded 3DGS has comparable photometric quality to 3DGS/2DGS optimized with sparse views and better geometric accuracy. They also show the feasibility of augmenting the latent embedding with task-specific parts, such as ones useful for visual localization and 3D scene alignment.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The proposed object latent representation is compact in storage size, and can be decoded into 3DGS reconstructions with similar photometric quality and better geometric accuracy compared to optimization-based 3DGS/2DGS with sparse inputs. The intermediate representation - structured latent representation grounded in voxels - contributes to the geometric accuracy.\n- The learned latent embeddings can be further augmented with task-specific portions and corresponding training objectives, such as the ones for visual localization and 3D scene alignment. The paper demonstrates the idea of representing the scene as object latents capable of both reconstruction and downstream tasks.\n\nWeaknesses\n- Reconstruction quality - the visualization in the supplementary materials looks quite smoothed out and lacks texture details. Despite achieving a high compression rate, the quality seems to be sacrificed. The better geometric accuracy can be attributed to the voxel-based representation. How does the method compare to a baseline that directly fuses masked depth inputs with TSDF and runs marching cubes for mesh extraction? \n- Comparison to baselines \n    - 3DGS/2DGS are constrained to a sparse view setting, which I’m not sure is a fair setting, given that they are optimization-based and have no learned prior. \n    - DepthSplat is trained on scene-level data, where input frames have overlaps. However, it is evaluated on object-level data with non-overlapping frames (L265: while in ScanNet, where fewer non-overlapping frames are available, we limit the selection to four frames per object.) Although the paper claims “neither methods are trained on ScanNet”, the train-test domain gap is still much smaller for the proposed method, putting depthsplat at an unfair disadvantage. \n    - Why are baselines taking 12 views as *input* considered as “(L253-254) *stores* each object as 12 images”? Are the storage in Table 1 computed using the size of these input images, instead of their output Gaussian representation? If so, why is the storage of the proposed method computed differently, based on the output latents? \n- Support for downstream tasks - The support for downstream tasks is achieved by attaching another latent code to the original one, and training them with two-stage joint training. This requires more training time compared to directly training a separate task-specific latent code for each task, especially when given multiple downstream tasks. While the experiment shows some benefit from the reconstruction-based latent code to the task of visual localization, whether this benefit 1) persists when multiple tasks are learned together, 2) is worthy of the more complicated training procedure, should be further discussed.\n- Confusing technical specifications \n    - In the “auxiliary encoder training with frozen core” stage, if both U-3DGS encoder and decoder are frozen, and the reconstruction loss is only applied using the U-3DGS decoder that operates on the U-3DGS portion of the latent code - which should come from the frozen U-3DGS encoder - everything seems frozen in the reconstruction loss part, and the loss is redundant. It would be less confusing to only keep the task loss in this stage and introduce the reconstruction loss in the following joint training stage.\n    - $L$ is first used to refer to the sparse voxels. If the compression loss also sums over $L$ sparse voxels, shouldn’t $M_i$ be all ones? It seems to make more sense if the loss sums over all dense voxels. $N$ is used to describe the size of the voxel grid, but later also used for the total number of Gaussians in (1). \n\n\nTypo - Fig 5. (b) and Table 3. 3DSG -> 3DGS\n\nQUESTIONS:\n- How does the method compare to a baseline that directly fuses masked depth inputs with TSDF and run marching cubes for mesh extraction? \n- How is the storage computed for baselines and the proposed method? If the current storage is based on the size of the input images, what is the size of the Gaussian representation (using default setting and SH degree=0 settings)? (There are also many research works focusing on Gaussian compression that effectively reduce raw Gaussian sizes. I wonder how the proposed method’s storage compares with compressed Gaussians.)\n- While the experiment shows some benefit from the reconstruction-based latent code to the task of visual localization, whether this benefit 1) persists when multiple tasks are learned together, 2) is worthy of the more complicated training procedure should be further discussed.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nConventional approaches in 3D reconstruction and rendering typically rely on either implicit neural representations like 3D Gaussian Splatting or object-centric representations such as 3D scene graphs, but rarely both. The former lacks object-level modularity, making it difficult to reason about individual objects or integrate other modalities (e.g., text), while object-centric embeddings struggle to reconstruct high-fidelity appearance and geometry, thereby requiring the storage of complementary original source data such as images, point clouds, and meshes. Object-X seeks to capture advantages of both methods by first learning a unified object embedding that fuses geometry and semantics, then decoding this embedding into a set of 3D Gaussians. The approach involves voxelizing objects into a 3D grid, aggregating local image features into a structured set of latent vectors, compressing them into an unstructured latent representation, and decoding to 3DGS for the object. Experiments show that Object-X achieves high geometric accuracy comparable to specialized 3DGS-based methods on tasks such as object, scene, and single-image reconstruction, while requiring much less storage and inference time. Qualitative results indicate that the model excels in single-image reconstruction and captures correct object semantics in scenes, but further refinement seems necessary for high-quality full-scene reconstruction.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nNovel approach that combines implicit 3DGS representations and object-centric scene graph representations.\nEvaluation over diverse tasks: object, scene, and single-image 3D reconstruction.\nClear benefits in storage efficiency and inference time, which are important for scaling 3D object and scene generation.\nDemonstrates competitive downstream performance on tasks like localization and alignment, showing great utility of the unified embedding.\n\n\nWeaknesses:\nThe paper is missing ablations on removing the 3D scene graph representation, so it is not clear how critical 3DSG is to the overall method. For example, Table 3 shows that SGAligner achieves high performance without using 3DSG, raising questions about the specific contributions of the scene graph. Including an ablation or more analysis clarifying the necessity and impact of the scene graph within Object-X would strengthen the work.\nThere is a reliance on generating accurate scene graph annotations (e.g., SceneGraphFusion), which could become a bottleneck if the quality of the generated 3D scene graphs is poor. It would be beneficial for the authors to discuss the impact of noisy or erroneous scene graphs or include experiments that measure performance under imperfect annotations.\nWhile single-image object reconstruction is impressive and Object-X captures object semantics well, the full-scene composition appears significantly worse in quality when using the vanilla Object-X model. The need for additional refinement to achieve high scene quality potentially negates the storage and inference time benefits in practical full-scene settings. More discussion or experiments on this trade-off, and whether further improvements could address it, would be helpful.\nThe reason for the high geometric accuracy and F1 scores in Table 1 is not clearly explained, especially given the smaller relative improvements in other metrics like SSIM and PSNR. Providing more analysis, or qualitative examples explaining these discrepancies, would be helpful.\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-09"}