Retri3D: 3D Neural Graphics Representation Retrieval
Authors: Yushi Guan
Daniel Kwan
Jean Sebastien Dandurand
Xi Yan
Ruofan Liang
Yuxuan Zhang
Nilesh Jain
Nilesh Ahuja
Selvakumar Panneer
Nandita Vijaykumar
Venue: ICLR 2025 Spotlight (2025)
Keywords: Neural Graphics Representation; 3D Retrieval; Database
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A framework for the retrieval of neural graphics representation

Abstract: Learnable 3D Neural Graphics Representations (3DNGR) have emerged as promising 3D representations for reconstructing 3D scenes from 2D images. Numerous works, including Neural Radiance Fields (NeRF), 3D Gaussian Splatting (3DGS), and their variants, have significantly enhanced the quality of these representations. The ease of construction from 2D images, suitability for online viewing/sharing, and applications in game/art design downstream tasks make it a vital 3D representation, with potential creation of large numbers of such 3D models. This necessitates large data stores, local or online, to save 3D visual data in these formats. However, no existing framework enables accurate retrieval of stored 3DNGRs. In this work, we propose, Retri3D, a framework that enables accurate and efficient retrieval of 3D scenes represented as NGRs from large data stores using text queries. We introduce a novel Neural Field Artifact Analysis technique, combined with a Smart Camera Movement Module, to select clean views and navigate pre-trained 3DNGRs. These techniques enable accurate retrieval by selecting the best viewing directions in the 3D scene for high-quality visual feature embeddings. We demonstrate that Retri3D is compatible with any NGR representation. On the LERF and ScanNet++ datasets, we show significant improvement in retrieval accuracy compared to existing techniques, while being orders of magnitude faster and storage efficient.

Published as a conference paper at ICLR 2025
RETRI3D: 3D NEURAL GRAPHICS REPRESENTATION RETRIEVAL
Yushi Guan', Daniel Kwan', Jean Sebastien Dandurand', Xi Yan', Ruofan Liang', Yuxuan Zhang', Nilesh Jain?, Nilesh Ahuja', Selvakumar Panneer", Nandita Vijaykumar (guanyushi, akwaa, js dandurand, xyan, ruofan, yuxuan, nandita) lcs tcronto. {sel vellvakmarr panneor, nilesh. jain, nilesh. jaj @intel.com com
ABSTRACT
Learnable 3D Neural Graphies Representations (3DNGRs) have emerged as promis- ing 3D representations for reconstructing 3D scenes from 2D images. Numerous works, including Neural Radiance Fields (NeRF), 3D Gaussian Splatting (3DGS), and their variants, have significantly enhanced the quality of these representations. The ease of construction from 2D images, suitability for online viewing/sharing, and applications in game/art design downstream tasks make it a vital 3D representa- tion, with potential creation of large numbers of such 3D models. This necessitates large data stores, local online, save 3D visual data in these formats. However, no existing framework enables accurate retrieval of stored 3DNGRs. In this work, we propose, Retri3D, a framework that enables accurate and efficient retrivval of 3D scenes represented as NGRs from large data stores using text queries. We introduce a novel Neural Field Artifact Analysis technique, combined with a Smart Camera Movement Module, to select clean views and navigate pre-trained 3DNGRs. These techniques enable accurate retrieval by selecting the best viewing directions in the 3D scene for high-quality visual feature embeddings. We demonstrate that Retri3D is compatible with any NGR representation. On the LERF and Scan- Net++ datasets, we show significant improvement in retrieval accuracy compared to existing techniques, while being orders of magnitude faster and storage efficient.
INTRODUCTION
A radiance field maps 3D spatial locations and 2D viewing directions to radiance value, typically encoding the color and intensity (Pharr & Humphreys, 2016). It has recently been combined with learnable representations, such neural networks and trainable 3D Gaussians, revolutionizing novel view synthesis tasks (Mildenhall et al., 2020; Kerbl et al., 2023). We refer to these solutions, which learn radiance fields for novel view synthesis, 3D Neural Graphics Representations (3DNGRs). Traditional 3D formats, such as meshes, voxels, and point clouds, explicitly define the geometry of the 3D data. In contrast, NGRs differentiable rendering learn the 3D scenes from sparse sets of 2D images. This approach has many advantages including compact storage (Mildenhall et al., 2020), continuous and unbounded representation (Barron et al., 2021; Kerbl al., 2023), and high-quality novel view synthesis (Liang et al., 2023). Additionally, NGRs can be integrated with Al content generation pipelines (Poole al., 2022) with many applications in 3D art and game design (Saito et al., 2023; Gu et al., 2023; Unity Technologies. 2024; NVIDIA Corporation, 2024). The ease of constructing NGRs from casually captured 2D images has resulted in an abundance of diverse 3D scenes available for viewing, sharing, and downloading online (Polycam, 2024; Luma AI, 2024).
Despite the abundance of NGR assets, framework for storing and retrieving 3D scenes represented as NGRs from a data store has not yet been developed. Many existing works tackle the storage and retrieval of images and traditional 3D formats (meshes, point clouds, voxel grids) (Tan al., 2021; Lahay & Tal, 2020; Sanghi, 2020; Wang et al., 2018). Existing NeRF semantic understanding solutions are performed on scene-by-scene basis and cannot be easily scaled to hundreds of scenes as general retrieval solution (Kerr et al., 2023; Qin et al., 2023).
In this work, we identify three main challenges in enabling efficient retrieval of NGR assets in data store: First, the representation format is highly heterogeneous. A wide range of NGR formats exist, for example, NeRF, 3DGS, and their variants. In addition, new formats continuously being developed. It is critical for a retrieval system to be compatible with any pre-trained NGR, ensuring that existing 3D assets do not become obsolete and that new NGR formats can be seamlessly added to the

---

Published as a conference paper at ICLR 2025
(e) User Text Query
Flower
Visual Embeddings
VLM
Text Embedding
VLM
VLM
(d) Database of NGRs and Visual Embeddings
(c) VLM for Visual Embedding Generation
(a) Pre-trained
(b) Naise Analysis for
NGR Scenes
Clean View Rendering
from Clean Rendering
(f) Retrieved NGR & Image
Figure I: Overview of Retri3D: (a) Retri3D enables the storage and retrieval of any neural graphics representation (NGR). (b) We perform noise analysis and viewpoint selection select high -uallit artifact-free views (See Section 3.3 and 3.4). Videos of rendered trajectories are provided in the
supplementary material. (c) Visual embeddings for retrieval are generated using pre-trained Vision-
Language Model (VLM). (d) Both NGRs and visual embeddings are stored in the database. (c) Given user query, we use the same VLM to generate a corresponding text embedding. (1) Retrieval of the relevant scene is based on the highest cosine similarity between the text and visual embeddings, returning the NGR and optionally, a rendered image, to the user.
data store. Second, 3D scenes exhibit a high degree of diversity and complexity. Since NGRs allow 3D scenes to be easily constructed from casually captured 2D images, retrieval solution must be compatible with any semantic concepts or sentence-like descriptions. Unlike traditional 3D retrieval solutions that focus on isolated 3D objects with single concept, 3D scenes contain numerous potential views, each with multiple concepts. This necessitates a solution capable of extracting different visual features from different parts of the scene. Moreover, the solution should efficiently extract relatively compact embeddings to minimize storage and computational overhead. Third, the presence of noise and floater artifacts NGRs makes visual feature extraction from the scenes much more challenging. These artifacts are primarily due to incomplete scene coverage during capture but are also affected by factors such as image blurs, depth, and lighting ambiguities. For a 3D scene, artifacts can significantly affect rendered 2D images from randomly selected viewpoints, even with methods specifically designed to reduce floaters and artifacts in NGRs (Philip & Deschaintre, 2023). The pre- pre-trained NGRs typically stored by itself without the images used for training (Polycam, 2024), making the selection of good viewpoints for understanding the scene crucial. We demonstrate that these floaters severely impact visual feature extraction of the scenes, highlighting the need for solution that can efficiently detect and identify floater-free viewpoints in any pre-trained NGR.
In this work, we propose Retri3D, the first framework that addresses these challenges to enable accurate and efficient retrieval of NGRs of any format using text queries. To enable generality across various NGR formats, we propose to first render images from the 3DNGRs and then generate visual semantic embeddings from the rendered images. We demonstrate that selecting random viewpoints for rendering images from the 3D representations leads to artifacts, severely impacting the quality of visual embeddings and, thus, the accuracy of the retrieval. To address this, we introduce novel Neural Graphics Noise Analysis technique, which identify artifacts in an RGB image. We compare with state-of-the-ar NeRF uncertainty measurement work, and demonstrate that our solution identifies the RGB pixel error in the rendered images with higher accuracy (Goli al., 2023). Coupled with Smart Camera Movement Module, our system navigates the camera within the 3D scene viewpoints that ensure high-qualin rendering by avoiding floater artifacts. We show that our solution significantly improves the efficiency and accuracy of retrieval. Finally, we leverage recent advances pre-trained Vision Language Models (VLMs) to convert high-quality rendered images into compact visual embeddings in an efficient manner. Our system utilizes the zero-shot capabilities of pre-trained VLMs, allowing it to easily benefit from potential future improvements in VLMs.
2

---

Published as a conference paper at ICLR 2025
We demonstrate the effectivvnesss of Retri3D using the LERF and ScanNet++ datasets (Kerr et al., 2023; Yeshwanth et al., 2023). For the LERF dataset, we show that our system can extract visual features from highly varied 3D scenes with complex details. Compared with adapting existing NGR semantic analysis solutions for retrieval, we demonstrate our system has higher accuracy while being more general and orders of magnitude faster and storage-efficient. On the ScanNet++ dataset, we show that our solution scales to hundreds of scenes, performing a scene lookup in millisecond while only requiring under 20 seconds for visual feature extraction of the scene during insertion.
This work makes the following contributions: (i) We present the first end-to-end framework for accurate and efficient retrieval of 3D scenes represented NGRs using text queries. (ii) We propose novel technique to detect NGR noise through rendered RGB images and pre- trained VLMs, achieving superior RGB error measurement accuracy over existing methods. (iii) We develop solution for rapid camera movement to artifact-fres viewpoints for high-quality embedding extraction. (iv) Through LERF and + datasets, we show that Retri3D achieves efficient, scalable, and high-accuracy 3D scene retrieval.
RELATED WORK
To our knowledge, this is the first work that tackles the accurate and efficient retrieval of 3D scenes represented as NGRs. Our proposed work leverages advances in pre-trained (VLMs). It is inspired by research in related fields such as traditional 3D object retrieval and 3D) semantic understanding. We now describe the most relevant works in this section.
Models
2.1 LEARNABLE 3D NEURAL GRAPHICS REPRESENTATIONS
In recent years, neural graphics methods have demonstrated great advances in 3D reconstruction and novel view synthesis, with innovative methods, such as neural radiance fields (NeRFs) (Mildenhall et al., 2020) and 3D Gaussian splatting (3DGS) (Kerbl al., 2023), being proposed.
NeRF-based methods (Barron et al., 2021; Müller et al., 2022; Chen et al., 2022; Yu et al., 2021) represent the 3D scene as spatial volumes with neural parameters (e.g.. MLPs, feature grids, and hashtables, etc.), optimized through volumetric rendering. 3DGS (Kerbl et al., 2023) uses trainable 3D Gaussians with alpha-bleun blending rasterization to replace NeRF's volumetric rendering, achieving faster training and inference speed.
There are other types of 3D neural representations, such as neural signed distance fields (Wang et al., 2021; Yariv et al., 2021) and differentiable meshes (Shen et al., 2021b; Munkberg et al., 2021), also demonstrating great performance in 3D reconstruction tasks. Given different NGR formats, formulating a completely general solution for analyzing the learned parameters is extremely challenging. With Retri3D, propose use RGB image renderings for embedding generation to enable a fully general solution.
2.2 PRE-TR TRAINED VISION LANGUAGE MODELS
The introduction of large-scale text-image contrastive learning by CLIP has significantly improved the performance of pre-trained Vision Language Models (VLMs) (Radford et al., 2021). These large models are capable of understanding and processing a wide range of natural language inputs and images, making them highly versatile. Recent works apply similar large-scale pre-training approaches to tasks such as object detection and (semantic) segmentation with open-vocabulary (Kirillov et al., 2023; Liu al.. 2023b; Zhang et al., 2023a: Zou et al., 2022: 2023a). In this work, we leverage different pre-trained VLMs (XDecoder and OpenCLIP) to demonstrate our solution's 's applicability when leveraging different VLMS (Zou et al., 2022: Cherti et al., 2022). We use the VLMs in two ways: (i) we use their visual and textual embeddings for text-scene retrieval; (ii) we use their activations to determine if region contains NGR noise artifacts or not. To the best of our knowledge, we are the first to demonstrate noise analysis for NGR can be performed with pre-trained VLM activation features. Note that for all VLMs, We do not perform fine-tuning to demonstrate the noise analysis and retrieval performance achievable through zero-shot learning, thereby allowing to generalize the performance to unseen scenes and leverage future advancements in VLMs.
2.3 TRADITIONAL 3D DATA RETRIEVAL
There is extensive work on the retrieval of 3D models represented in traditional explicit formats such as point clouds (Sanghi, 2020; Qi et al., 2016; 2017: Qian et al., 2022; Zhao et al., 2022), voxel grids (Wang et al., 2017: 2018), and meshes (Hanocka et al., 2019; Mitchel et al., 2021; Lahay & Tal, 2020). Another popular method involves converting 3D models into multi-view images for
3

---

Published as a conference paper at ICLR 2025
retrieval (Jiang et al., 2019; Hamdi et al., 2020; Wei et al., 2020; He et al., 2018; Yavartanoo et al., 2018). One common property of these methods is their focus on the retrieval of isolated 3D objects, e.g., ModelNet (Wu et al., 2014) and ShapeNet (Chang et al., 2015). Prior works leverage deep learning methods to convert the 3D model into single embedding, with retrieval performed based on embedding similarity. In contrast, 3D scenes encompass multiple semantic concepts/ohjects and cannot be easily represented as a single embedding (Peng et al., 2022; Zhang et al., 2023b; Kerr et al., 2023; Qin al., 2023; Guan et al., 2025).
2.4 3D NEURAL GRAPHICS REPRESENTATIONS SEMANTIC UNDERSTANDING
Prior works integrate semantic analysis on the 2D images into NGR training. These methods often modify the NGR architecture to jointly train semantics with RGB components (Vora et al., 2021; Zhi et al., 2021; Xu et al., 2023; Liu et al., 2023a; Ye et al., 2022; Cheng et al., 2023; Fu et al., 2023;
Siddiqui et al., 2022; Bhalgat et al., 2023; Hu et al., 2023; Mazur et al., 2022: Shafiullah et al., 2022; Li et al., 2022: Liu et al., 2023c; Kerr et al., 2023; Qin et al., 2023), which results in non-general, architecture-specfii solutions and increased computational and storage overhead (Liu et al., 2023c; Kerr et al., 2023; Qin et al., 2023). Semantic understanding in NGRs is crucial for our work, it is necessary for scene retrieval. Existing works can be categorized into two groups: (i) NGR object annotation and detection, and (ii) visual feature embedding. The annotation and detection methods usually apply to a set of predefined object categories (Vora et al., 2021; Zhi et al., 2021; Xu et al., 2023; Liu et al., 2023a; Ye et al., 2022; Cheng al., 2023), which lacks rich semantic information and limits the retrieval to specific object sets.
Most relevant to our work are methods that embed visual feature embeddings into NGRs, allowing visual embeddings to be rendered and compared with any text embedding post-training. LERF (Kerr et al., 2023) and LangSplat (Qin et al., 2023) generate visual embeddings during NGR training, injecting them into NeRF and 3DGS, respectively. LERF uses multi-scale CL.PP embeddings, while LangSplat segments the scene into different objects and generates embeddings for each. At test time, these visual embeddings can be rendered through volumetric rendering (LERF) or rasterization (pangS similar to RGB rendering. Liu al. (Liu et al., 2023c) proposed a similar approach that is limited to forward-facing scenes. Rashid et al. (Rashid et al.) applied LERF to single changing scene. N2F2 (Bhalgat et al., 2024) follows LERF's methodology but explicitly encodes multi-scale visual embeddings. LEGaussians adopts the same visual embedding method as LERF but records embeddings using Gaussians instead of a NeRF field. TIGER (Xu et al., 2024) employs a visual embedding similar to LangSplat and further introduces scene editability. ConDense (Zhang et al., 2024) proposes training a 3D transformer to generate the visual embedding field, but it requires thousands of GPU hours to train 3D transformer that is limited to a specific NGR type. ConDense also uses SuperPoint (DeTone et al., 2017) to sparsify the dense visual feature rendered from the visual embedding field, but only applicable the visual features from training views. Other works embed visual embeddings into point clouds (Peng et al., 2022; Zhang et al., 2023b).
We adapt LERF and LangSplat methods for scene retrieval because they are open-sourced. have settings closest to our work, and differ in both visual embedding generation and encoding methods. Compared to our approach, LERF and LangSplat incur significant training and rendering overhead and generate orders of magnitude more embeddings, which negatively impact storage and retrieval speed. These limitations similarly apply to other works using similar methods. We discuss the implications of these characteristics for retrieval solutions in more detail in Appendix B.9.
METHOD
3.1 OVERVIEW
Retri3D retrieves a learned 3D radiance field given user text queries, offering compatibility with any pre-trained NGR format. The text query can be simple object labels or sentence descriptions. We visualize our solution in Fig 1. We assume the pre-trained scenes are shared as is, containing no information about the original training images or poses, which is typical for online sharing plat- forms (Polycam, 2024). In Section 3.2, we discuss how to generate diverse but compact embeddings from images. In Section 3.3, we propose new method for detecting NGR noise and artifacts based on images only, when combined with the Smart Camera Movement Module (Section 3.4), Retri3D selects high-quality renderings to improve retrieval accuracy.
4

---

Published as a conference paper at ICLR 2025
Clean
Nolsy
Doth
-30 -20 -10
20 -30 -20
-30 -20 -10
Figure 2: 2D t-SNE visualization of clean (O) and noisy (X) pixel from XDecoder. Different colors
indicate different LERF scenes. The t-SNE was computed jointly using clean and noise features. They are displayed separately for clarity. We make the following observations: (1) The noise features are much more tightly clustered compared to the clean features; (2) small amount of noise features intrude into the clean features, but they constitute small portion and have minimal impact the learned Gaussian; (3) The noise features from one scene are closer to the noise features from another scene, compared to clean features from their own scene.
3.2
VISUAL EMBEDDING GENERATION
Existing NGR semantic analysis methods require extensive training to integrate an NGR with a visual semantic field, typically represented as a NeRF 3D Gaussian blobs. This approach necessitates manual engineering of the visual semantic field for each NGR format. Additionally, rendering the visual semantic field introduces significant computational and storage overhead, making these methods unsuitable for general-purpose NGR databases.
To address this, we propose general solution compatible with any pre-trained NGRs capable of rendering RGB images. Our method leverages VLM to generate visual embeddings. summarizing the image segments of the image from NGR renderings. These embeddings database upon acquiring the NGRs as is. During retrieval, using the same VLM, we create a text embedding from user queries and retrieve the visual embedding (along with the associated scene and image) that has the highest cosine similarity. We evaluate our approach with both an older VLM (CLIP) and more recent VLM (XDecoder) to demonstrate its flexibility. For detailed description of the VLMs and database indexing, please refer to Appendix A.I.
stored in the
3.3 NEURAL GRAPHICS NOISE ANALYSIS
The existence noise and floater artifacts in NGRs has been a longstanding problem observed by many existing works in NGR (Mildenhall al., 2020; Warburg et al., 2023; Goli et al., 2023). In this section, we introduce Neural Graphics Noise Analysis, a method for distinguishing noise (and floater artifact) regions of NGR renderings from clean regions. The goal is to the detected noise to redirect cameras to cleaner viewpoints to generate high-quality visual embeddings for improved retrieval accuracy. We refer to noise and floater artifacts collectively as noise below.
We observe that the activation features of pre-trained VLM exhibit the ability to separate noise regions from clean regions. In Figure 2, show t-SNE plots of clean region features vs. noise region features generated from VLM. Figure 2 demonstrates that the pixel-wise' noise features from different NGR scenes are tightly clustered, and well separated from the clean features. To obtain these features, we pass rendered image I R3x H X through the visual encoder of pre-trained VLM. We obtain intermediate activation map A Φ' (I: d'). where &' is section of the visual encoder and A E c is the activation with features of length c. Our solution is compatible with different VLMMs and VLM layers, Appendix A.2.
To train multivariate Gaussian distribution to represent the noise, we collect K training (tr) activation features {an, K
== from random viewpoint renderings' activations to represent noise features. The renderings can be obtained from NGRs to be stored the database, or other separate NGR scenes. We compute the mean ş and covariance Σ of these features to fit multivariate Gaussian distribution:
By pixel-wise, we refer to the pixels of the activation map. This activation map can be downsampled from the original image in spatial resolution.
5

---

Published as a conference paper at ICLR 2025
Random
Multiple Neural Graphics
Viewpoint
VLM Vision
Rendering
3D Scenes
Encoder's
Activation
Legend:
Training
(a) Noise Renderings
(b) Noise Activations
Scene Analysis
VLM Vision Encoder's Activation
Mahalanobis
Distance
Rendering for Scene Analysls
(c) Multivariate Gaussian Distribution
(d) Analyzed
for Noise Representation
Noise (Dark), Clean (Bright)
Figure 3: Noise analysis training and inference process. (a) We generate noisy images, which can be done simply through rendering random viewpoints in pre-trained NGR scenes. Some content be noise-free, but they constitute only small portions of the images. (b) Using pre-trained VLM's vision encoder, generate the pixel-wise activation features. (c) We train Multivariate Gaussian distribution to represent the noise features. (d) During inference, we calculate Mahalanobis distance of RGB rendering' activations the trained noise Gaussian produce noise map.
K
K
== K-1
Σ t=1
an..
1 cI
K
1=1
where e small regularization constant to ensure E is invertible. For an activation feature aan. during the analysis of an NGR scene, we calculate the Mahalanobis distance dm to the Gaussian:
dm = (( (e) = - 11)
We then compute normalized clean score s using a sigmoid function o:
s= a (a (dm - c)), a (xx
(3)
I+e-r
where scaling factor and is a cutoff value. Pixels with larger Mahalanobis distances are considered cleaner (higher s), while those with smaller distances are considered noisier (lower s). Note that Existing NGR works' visualizations typically viewpoints close the training points, resulting in images that appear clean. We explain this further in Appendix C.5. Since noisy regions dominate random viewpoints, treat all regions from these renderings as "noise".
Our method offers several key advantages. First, demonstrates strong generalizability: noise features remain consistent across different scenes due to shared rendering artifacts, allowing our Gaussian model, trained on random renderings, to generalize effectively to unseen scenes. We also observe similarities in noise features across different models. We further discuss the similarity of noise across models, and its impact on noise analysis C.6. Second, it operates without requiring training image viewpoints clean pixel features, making the approach applicable even when such data unavailable. Third, it has low computational overhead, adding minimal processing time to the activation feature generation during VLM inference. Lastly, it is highly versatile and compatible with various pre-trained Visual Language Models (VLMs);; we showcase its performance using XDecoder and CLIP.
Depending on the scenario, our method can be applied in two main approaches. The first is self- supervised noise analysis, where the Gaussian distribution is trained on random viewpoint renderings from a set of scenes; and subsequently used to analyze new renderings from the same set scenes. We demonstrate this approach in our LERF dataset retrieval results. The second approach is cross- scene transferred noise analysis, where the Gaussian model is trained on pre-existing scenes and then applied to new, unseen scenes, leveraging the consistent nature of noise features across different environments. This method is demonstrated in our ScanNet++ retrieval results.

---

Published as a conference paper at ICLR 2025
RGB
Clean Score
Next Viewpoint Selection
Step 1
(a)
(b)
(c)
Step 2
(d)
(c)
(f)
Figure 4: Application of our noise analysis method. (a) Starting from a random viewpoint rendering: (b) Noise analysis creates a clean score map highlighting clean (bright) and noisy (dark) regions; (c) Based on the noise analysis, the best next viewpoint is selected (visualized rendering from the new viewpoint shows reduced noise. This process can be iterated multiple times.
green box); (d) The
We provide additional noise analysis experiments with different VLM layers in Appendix A.2. We demonstrate the superiority of our Noise Analysis method identifying RGB error rendered images compared to state-of-the-art NeRF uncertainty estimation methods in Appendix A.5. Another potential way to distinguish noise from clean pixels is to fine-tune a VLM to classify noise from non-noise. We found such method often treats the image content with high-frequency details as noise. We provide further analysis in Appendix A.6.
3.4 SMART CAMERA MOVEMENT MODULE
We propose the Smart Camera Movement Module (SCMM), which discovers clean viewpoints for rendering by iteratively redirecting the camera to less noisy views. The clean views enable higher- quality visual embeddings from VLMs to improve the retrieval performance. SCMM is achieved by moving the camera towards the less noisy pixels as predicted by our noise analysis algorithm. We provide an example of such camera movement in Figure 4.
The SCMM moves the camera to cleaner view by estimating the noise level beyond the current field of view. We apply an iterative "render, noise analysis, camera movement" process to refine the camera viewpoint. Given rendering, the clean score for the entire rendering is represented S, where the clean score per pixel is $3.s
h, We estimate the clean score around the current field of view by extending it beyond the current image border. We apply a linear-decay aage-padding to the current clean score:
S (
w)
S' {5 - و j' < 2h,
S 1' < 2w)
(5)
if 1 < h w < 0
(left pad)
Si
if h 50,15 /' < w
(top pad)
(6)
sin
if h
(top left pad)
10
(others)
where s is pixel of the Gaussian blurred version S* of the original clean score map S. Given blue kernel G with std a and radius k; the Gaussian blurred clean score map S* is given by:
k &
x² + y²
S Σ Σ
G(m, m-nn (S * G) G(z, v) = 2TO
exp
(7)
20
カー=-カ==-
The Gaussian blur is applied to reduce the effect of individual edge and corner pixels' clean score on the estimated clean score beyond the current view. Lastly, efficiently find the maximum sum submatrix (Figure ((c)(f), green box), representing the highest expected clean score. Using cumulative sum arrays, finding the submatrix only has time complexity of O(hw), linear w.n.t. the
7

---

Published as a conference paper at ICLR 2025
Object Label
LLaVA Caption Viewpoint
# Img.
Viewpoint
Training
SCMM
Random
Training
SCMM
Random
57.89
30.08 (-27.81)
17.29 (-40.60)
45.12
32.97 (-12.15)
15.28 (-29.84)
5
68.42
64.66 ( -3.76)
18.80 (-49.62)
55.07
57.17 (+2.10)
23.58 (-31.49)
10
78.95
67.67 (-11.28)
24.81 (-54.14)
67.07
61.23 (-5.84)
28.92 (-38.15)
20
83.46
73.68 ( -9.78)
34.59 (-48.87)
72.41
63.25 (-9.16)
33.01 (-39.40)
50
84.21
77.69 ( -6.52)
49.62 (-34.59)
71.80
65.82
(-5.98)
35,58 (-36.22)
100
84.95
80.02 ( -4.93)
57.89 (-27.06)
73.67
70.48 (-3.19)
43.04 (-30.63)
Table Retrieval Accuracy (P@) for Splatfacto Model on LERF Dataset
spatial resolution. The camera center is then rotated to point at the center of this new maximum sum submatrix.
We observe that renderings at lower resolution still be robust for noise analysis and the SCMM process. Therefore, to generate high-quality visual embeddings for retrieval, we propose to perform SCMM with low-resolution renderings discover the clean viewpoint first. Then, we perform high-resolution rendering and visual embedding generation at discovered clean viewpoint. See Appendix A.4 for details.
We discussed the application of SCMM for camera viewpoint rotation. However, by rendering sets of images that vary in depth (in the current viewpoint's 's coordinate), can find the next view with the highest expected clean score in 3D. This allows our camera to both translate and rotate in 3I) and move in a trajectory that iteratively points cleaner fields. We provide further discussion and visualization in Appendix A.3.
EXPERIMENTS AND RESULTS
We perform experiments with LERF (13 scenes) and ScanNet++ (280 scenes) datasets (Kerr et al., 2023; Yeshwanth et al., 2023). We use either the object labels that came with the dataset or queries generated by the LLaVA model (using the prompt: Describe the image in detail in one sentence.) (Liu et al., 2023b). A retrieval is successful if the retrieved scene contains the object, or is used to generate the caption. We report retrieval accuracy P@k=1,5,10, corresponding to whether the top k retrievals contain correct scene. Due to space limitations, focus results XDecoder + LERF dataset + Splatfacto model in the main paper. We provide more experiments details and results including the testing platform, different/mixxd NGR, and different VLMMs in Appendix B.
4.1 COMPARISON BETWEEN DIFFERENT RENDERINGS
In this section, we compare retrieval accuracy between different rendering viewpoints. As the retrieval accuracy significantly increases as more images are used, we report accuracy for different numbers of renderings. Training Viewpoint Render is the best scenario where there is knowledge of the training camera poses, creating the highest quality renderings. However, we do not assume the training poses available and use it best-case comparison. SCMM-chosen camera viewpoints utilizes novel solution SCMM detailed in Section 3.4 to choose high-quality viewpoints for clean renderings. Random Viewpoint renders from randomly selected viewpoint.
As shown in Table 1, the retrieval accuracy for SCMM is only few percent lower than the retrieval accuracy with Training Viewpoint. Compared to Random Viewpoint, our solution has more than 20% higher accuracy most cases. The accuracy differences hold for both object label text query and L.L. VA caption sentence query.
4.2 COMPARISON WITH BASELINES
In Table 2, we compare the retrieval accuracy of our method with the LangSplat and LERF methods. As LangSplat and LERF cannot select viewpoints smartly, render LERF and LangSplat at Training Camera Viewpoints. At 100 rendered images, comparing our solution with LangSplat, we achieve 5.12% higher object label retrieval accuracy if training pose is also available to us, or 0.19% if not. LangSplat encodes visual embeddings by first segmenting objects in the scene and then using the CLIP visual encoder to encode objects individually. As result, the embeddings are poor matches for image-level sentence descriptions. Our solution has about 20% higher accuracy than LangSplat. retrieval (explained further
The LERF model's embedding is particularly ineffective for in Appendix 10).). LERF, trained from multi-scale CLIP image embedding (without segmentation), very uniform and can not distinguish different objects effectively.
8

---

Published as a conference paper at ICLR 2025
Object Label
LLaVA Caption
Ours
Baselines
Ours
Baselines
# Img.
Training
SCMM
LangSplat
LERF
Training
SCMM
LangSplat
LERF
57.89
30.08
44.73
16.32
45.12
32.97
18.25
17.63
5
68.42
64.66
61.16
16.53
55.07
57.17
36.62
19.23
10
78.95
67.67
62.57
16.67
67.07
61.23
42.45
20.58
20
83.46
73.68
74.21
18.87
72.41
63.25
46.84
26.96
50
84.21
77.69
76.36
23.45
71.80
65.82
49.26
31.48
100
84.95
80.02
79.83
26.24
73.67
70.48
51.92
36.89
Table 2: Retrieval Accuracy (P@) for Different Models LERF Dataset
Ours
Baselines
Stage
Action Storage
SplatFacto
Nerfacto
LangSplat
LERF
NGR
Train Time (min)
6.82
8.24
90.5
40.1
Training
Model Size (MB)
478.49
176.02
958.67
1282.4
NGR
Generate
17.25
19.23
152.5
53.6
Analysis
Visual Emb. (s)
Database
Embedding Size
20.58MB
20.58MB
18. 18.78GB
225.4 2.B.4GB
& Retrieval
Retrieval Time (s)
5e
5e-5
le
17
Table 3: Speed and storage comparison with 50 rendered images for LERF scene
4.3 COMPUTATION AND STORAGE EFFICIENCY
In this section, compare the speed and storage for different stages of the retrieval process with LERF and LpngSplat. As we use pre-trained NGRs as is, there is little overhead for training or storing the NGRs. Our retrieval speed is achieved through using FAISS's implementation of inverted file index /VF/024 for efficient highest cosine similarity match (Douze et al., 2024). IVF1024 was also used for LangSplat and LERF. As shown in Table 3, LERF and LangSplat have orders of magnitude overhead in terms of speed and storage at multiple stages. This largely results from needing to train a NGR encode the visual embeddings, well the costly dense visual embedding rendering from the trained NGR. We explain this further in Appendix B.9.
4.4 RETRIEVAL FROM SCANNET++
Our solution easily scales to hundreds of pre-trained NGR scenes, demonstrate through testing on the ScanNet++ dataset. We use the 280 ScanNet++ scenes with semantic labels. As the many different ScanNet++ scenes contain similar objects (such as Sofa from different living rooms). We measure the retrieval accuracy based on whether the retrieved scene contains the object label. As demonstrated the table, ours still has a high degree of accuracy for retrieval even with hundreds of scenes. Using the /VF/024 /024 index, the measured retrieval speed per query is 6e second. We show more results with LLaVA caption and comparison with LangSplat in Appendix B.7.
4.5 SCENE COVERAGE METRIC AND RESULT
To evaluate 3D scene coverage using rendered images, we collect average coverage statistics for Nerfacto across 13 LERF scenes. Utilizing Nerfacto's spatially contracted space, we partition into 64 x 64 64 voxel grid and measure orientation based on the angle between camera rays and voxel faces. voxel is considered covered any rendering ray passes through it before reaching the estimated depth. Details provided in Appendix A.7.
The %Grid in Table shows the scene coverage (a voxel is also considered covered if the density is greater than 0.5). The &Train shows that, for voxels covered by Training, the percentage of those observed by SCMM or Random. We make two important observations. First, Training has low coverage of the scene, which explains the existence of significant noise in the NGRs. Such a low coverage is often due to the continuous video trajectory, which is how real-world NeRF datasets are usually captured. Second, SCMM provides both high coverage of the scene (% (Gid),, and high coverage of the scene section that was observed by the training images (Trii.).. This indicates that SCMM rendering tends to favor viewpoints similar to those of the training images, enhancing the capture semantic content present the original video and improving the rendering quality for better retrieval.

---

Published as a conference paper at ICLR 2025
Img.
Training
SCMM
Random
10
41.62
39.63 (-1.99)
18.39 (-23.23)
20
50.06
47.34 (-2.72)
23.58 (-26.48)
50
58.23
54.39 (-3.84)
27.93 (-30.30)
100
65.03
64.18 (-0.85)
29.04 (-35.99)
Table 4: Retrieval Accuracy @P@ 1) for Splatfacto Model with Object Labels for +
Training %Gri
SCMM
Random
Img.
%Grid
%Taii
%Gri
%Trii
xyz
w/t
xyz
w/f
xyz
w/0
xyz
w/0
xyz
w/O
9.9
9.9
9.9
9.9
0.0
0.0
9.9
9.9
0.0
0.0
13.0
11.4
15.3
12.7
11.3
16.2
14.3
12.7
14.3
12.2
32.8
16.5
42.7
23.6
35.6
19.6
29.5
14.4
10
17.2
13.9
44.2
24.9
65.0
46.0
53.7
32.4
49.4
24.5
20
20.6
15.8
56.4
29.8
83.3
58.1
61.7
41.5
64.7
39,2
50
41.6
17.5
71.1
46.9
87.5
69.3
78.2
54.7
73.7
65.0
46.7
30.7
79.3
53.3
87.9
79.4
83.6
61.0
85.6
78.6
Table 5: Scene Coverage statisties for Training, SCMM, and Random Renderings. Location-only (xyz); ircation-and-Orientann (w/f)
4.6 RETRIEVAL VISUALIZATION
In this section, we present the visualization results of our retrievals. Figure 5 showcases successful retrievals from various LERF scenes. As illustrated, the VLM used (XDecoder) accurately produces masks, and the associated visual embeddings closely align with the corresponding text embeddings, facilitating efficient matching. Importantly, these visual embeddings and masks are generated during the scene analysis phase, independent of any user text query. This feature ensures that the visual embeddings can be matched with any potential user query. We demonstrate additional failure cases in Figure 26.
CONCLUSION
We introduced the first end-to-end framework for retrieval of pre-trained neural graphies represen- tations. By performing visual analysis rendered RGB images, it is compatible with any form of pre-trained NGR. We propose novel techniques, Neural Graphics Noise Analysis and Smart Camera Movement Module, for detection of noise, navigation in NGR scenes, and selection of viewpoints for high-quality rendering. These high-quality renderings significantly improve retrieval accuracy. As result, solution achieves significantly higher retrieval accuracy than baselines such as LangSplat and LERF, even when training images are unavailable. Retri3D is thus general across NGRs, while being orders of magnitude more efficient in terms of speed and storage compared to existing techniques.
(a) big white crinkly flower
(b) checkerboard pattern
(c) espresso machine
(d) breakfast sandwich (Top 1)
(e) breakfast sandwich (Top 2)
(1) breakfast sandwich (Top 3)
Figure Successful Retrievals from LERF dataset with object label queries.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. 2022 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp.
2021.
Yash Bhalgat, Iro Laina, João F. Henriques, Andrew Zisserman, and Andrea Vedaldi. Contrastive lift:
3d object instance segmentation by slow-fast contrastive fusion. ArXiv, abs/2306.04633. 2023,
Yash Bhalgat, Iro Laina, Joao Henriques, Andrew Zisserman, and Andrea Vedaldi. N2f2: Hierar- chical scene understanding with nested neural feature fields. ArXiv, abs/2403.10997, 2024.
Wenjing Bian, Zirui Wang, Kejie Li, Jiawang Bian, and Victor Adrian Prisacariu. Nope-nerf: Optimising neural radiance field with no pose prior. 2023 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4160-4166 2022.
Angel X. Chang, Thomas A. Funkhouser, Leonidas J. Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, L. Yi, and Fisher Yu. Shapenet: An information-riee 3d model repository. ArXiv, abs/1512.03012, 2015.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields.
ArXiv, abs//203.00951,, 2022.
Hansheng Chen, Jiatao Gu, Anpei Chen, Wei Tian, Zhuowen Tu, Lingjie Liu, and Haoran Su. Single- stage diffusion nerf: A unified approach to 3d generation and reconstruction. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2416-2425, 2023.
Xinhua Cheng, Yanmin Wu, Mengxi Jia, Qian Wang, and Jian Zhang. Panoptie compositional feature field for editable scene rendering with network-inferred labels via metric learning. 2023 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4947-4957, 2023.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image -magg learning. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818 282899 2022.
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-am aanhoteannottted 3d reconstructions of indoor scenes. Proc. Computer Vision and Pattem Recognition (CVPR), IEEE, 2017.
Daniel DeTone, Tomasz Malisiewiez, and Andrew Rabinovich. Superpoint: Self-supervised interest point detection and description. 2018 JEFE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 337-33712, 2017.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng, Jeff Johnson, Gergely Szilvasy, Herre-Emmanuel
Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024.
Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Xiaowei Zhou, Andreas Geiger, and Yiyi Liao. Punopticnerf-360) Panoramic 3d-to-2d label transfer in urban scenes. ArXiv, abs/2309.10815. 2023.
Lily Goli, Cody Reading, Silvia Sellán, Alec Jacobson, and Andrea Tagliasacchi. Bayes' rays: Uncertainty quantification for neural radiance fields. ArXiv, hts/23309033888 2023.
Jiaming Gu, Minchao Jiang, Hongsheng Li, Xiaoyuan Lu, Guangming Zhu, Syed Afaq Ali Shah, Liang Zhang, and Bennamoun. Ue4-nerf: Neural radiance field for real-time rendering of large- scale scene. ArXiv, abs/2310.13263. 2023.
Yushi Guan, Daniel Kwan, Ruofan Liang, Selvakumar Panneer, Nilesh Jain, Nilesh Ahuja, and Nandita Vijaykumar. Inret: general framework for accurate retrieval inrs for shapes. 2025 International Conference 3D Vision (3DV), 2025.
Abdullah Hamdi, Silvio Giancola, Bing Li, Ali K. Thabet, and Bernard Ghanem. Mvtn: Multi-view transformation network for 3d shape recognition. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) pp. 1-11, 2020.
11

---

Published as a conference paper at ICLR 2025
Rana Hanocka, Amir Hertz, Noa Fish, Raja Giryes, Shachar Fleishman, and Daniel Cohen-Or, Meshenn: a network with an edge. ACM Transactions on Graphics (TOG), 38:1 12, 2019.
Xinwei He, Yang Zhou, Zhichao Zhou, Song Bai, and Xiang Bai. Triplet-center loss for multi-view 3d object retrieval. Proceedings of the IEEE conference on computer vision and puttern recognition, pp. 1945-1954, 2018.
Benran Hu, Junkai Huang, Yichen Liu, Yu-Wing Tai, and Chi-Keung Tang. Instance neural radiance field. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 787-796, 2023.
Jianwen Jiang, Di Bao, Ziqiang Chen, Xibin Zhao, and Yue Gao. Mlvenn: Multi-loop-vee view con- volutional neural network for 3d shape retrieval. In AAAI Conference on Artificial Intelligence, 2019.
Wen Jiang, Boshu Lei, and Kostas Daniilidis, Fisherrf: Active view selection and uncertainty quantification for radiance fields using fisher information. ArXiv, abs/2311.178777 2023.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuehler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (TOG), 42:1 14, 2023.
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 19672-19682, 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloc Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross B. Girshick. Segment anything. 2023 IEEE/CVF International Conference Computer Vision (ICCV), pp.
3992-4003, 2023.
Alon Lahav and Ayellet Tal. Meshwalker: Deep mesh understanding by random walks. ACM Trans. Graph., 39:263:11263:11 2020.
Yixing Lao, Xiaogang Xu, Zhipeng Cai, Xihui Liu, and Hengshuang Zhao. Corresnerf: Image
correspondence priors for neural radiance fields. ArXiv, abs/2312.06662, 2023.
Boyi Li, Kilian Q. Weinberger, Serge J. Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. ArXiv, abs/2201 .03546, 2022.
Ruofan Liang, Hui-Hsia Chen, Chunlin Li, Fan Chen, Selvakumar Panneer, and Nandita Vijayku- mar. Envidr: Implicit differentiable renderer with neural environment lighting. 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 79-89, 2023.
F. Liu, Chubing Zhang, Yu Zheng, and Yueqi Duan. Semantic ray: Learning a generalizable semantic field with cross-reprojection attention. 2023 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17386-17396, 2023a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ArXiv, abs/2304.08485, 2023b.
Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb El Saddik. Chris- tian Theobalt, Eric Xing, and Shijian Lu. Weakly supervised 3d voca-uocyy segmentation. In Neural Information Processing Systems, 2023c.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-33 shro one image to 3d object. 2023 IEEF/CVF International Conference on Computer Vision (ICCV), pp. 9264-9275, 2023d.
Luma AL. Luma ai, 2024. URL https: / luwallbb.bb.. .ai. Accessed: 2024-04-30.
Kirill Mazur, Edgar Sucar, and Andrew J. Davison. Feature-realistic neural fusion for real-time, open set scene understanding. 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 8201-8207, 2022.
12

---

Published as a conference paper at ICLR 2025
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf. Communiccations of the ACM, 65:99 106, 2020.
Thomas W. Mitchel, Vladimir G. Kim, and Michael M. Kazhdan. Field convolutions for surface cnns. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9981-9991, 2021.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (TOG), 41:1 13, 2022.
Jacob Munkberg, Jon Hasselgren, Tianchang Shen, Jun Gao, Wenzheng Chen, Alex Evans, Thomas Müller, and Sanja Fidler. Extracting triangular 3d models, materials, and lighting from images. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8270-8280, 2021.
NVIDIA Corporation. Instant nerf artists, 2024. URL https://:/ nvwdia.iii..... com/ en-m/en-u// research/ wewrrttgllllerynntttttee n Accessed: 2024-05-07.
Xuran Pan, Zihang Lai, Shiji Song, and Gao Huang. Activenerf: Learning where to with uncertainty estimation. In European Conference on Computer Vision, 2022.
Songyou Peng, Kyle Genova, ChiyuMaxJiang, Andrea Tagliasacchi, Marc Pollefeys, and Thomas A. Funkhouser. Openscene: 3d scene understanding with open vocabularies. 2023 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 815-824, 2022.
Matt Pharr and Greg Humphreys. Physically based rendering: From theory to implementation. 2016.
Julien Philip and Valentin Deschaintre. Floaters more: Radiance field gradient scaling for improved near-camera training. 2023.
Polycam. Polycam lidar & 3d scanner for iphone & android, 2024. URL https://poly http://olll........... Accessed: 2024-04-30.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. ArXiv, Ab982829..14988, 2022.
C. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 77-85, 2016.
C. Qi, L. Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. In NIPS, 2017.
Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Abed Al Kader Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. ArXiv, abs/2206.04670, 2022.
Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d language gaussian splatting. ArXiv, abs/2312. 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. 2021.
Adam Rashid, Chung Min Kim, Justin Kerr, Letian Fu, Kush Hari, Ayah Ahmad, Kai-Peng Chen, Huang Huang, Marcus Gualtieri, Michael Wang, Christian Juette, Tian Nan, Liu Ren, and Ken Goldberg, Cloud-enabled lifelong lerf for semantic inventory monitoring.
Shunsuke Saito, Gabriel Schwartz, Tomas Simon, Junxuan Li, and Giljoo Nam. Relightable gaussian codec avatars. ArXiv, abs/2312.037044, 2023.
Aditya Sanghi. Info3d: Representation learning on objects using mutual information maximization and contrastive learning. ArXiv, abs/2006.02598, 2020.
13

---

Published as a conference paper at ICLR 2025
Nur Muhammad (Mahi) Shafiullah, Chris Paxton, Lerrel Pinto, Soumith Chintala, and Arthur Szlam, Clip-fields: Weakly supervised semantic fields for robotic memory. ArXiv, abx/2210.05663, 2022.
Jianxiong Shen, Adria Ruiz, Antonio Agudo, and Francesc Moreno-Noguet Stochastic neural radi- ance fields: Quantifying uncertainty in implicit 3d representations. 2021 International Conference on 3D Vision (3DV), pp. 972-981] 2021a.
Jianxiong Shen, Antonio Agudo, Francese Moreno-Noe -oguer, and Adria Ruiz. Conditional nerf: Accurate 3d modelling with reliable uncertainty quantification. ArXiv, abs/2203. 10192, 2022.
Tianchang Shen, Jun Gao, Kangxue Yin, Ming- Yu Liu, and Sanja Fidler. Deep marching tetrahedra: hybrid representation for high-resolution 3d shape synthesis. Advances in Neural Information Processing Systems, 34:6087-6101, 2021b.
Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bul'o, Norman Muller, Matthias Nieβner, Angela Dai, and Peter Kontschieder. Panoptic lifting for 3d scene understanding with neural fields. 2023 IEEE/CVF Conference Computer Vision and Patern Recognition (CVPR), pp. 9043-9052, 2022.
Niko Sünderhauf, Jad Abou-Chaker and Dimity Miller. Density-aware nerf ensembles: Quantifying predictive uncertainty in neural radiance fields. 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 9370-9376, 2022.
Fuwen Tan, Jiangbo Yuan, and Vicente Ordonez. Instance-level image retrieval using reranking transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 12085- 12095, 2021.
Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong Li, Brent Yi. Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. In ACM SIGGRAPHI 2023 Conference Proceedings, SIGGRAPH '23, 2023.
Unity Technologies. Unity, 2024. URL httph:tt/////. /ttl::///untttt /tt////ntttttt..t.e/...//.ttt..t com/. Accessed: 2024-05-07.
Suhani Vora, Noha Radwan, Klaus Greff, Henning Meyer, Kyle Genova, Mehdi S. M. Sajjadi, Etienne Pot, Andrea Tagliasacchi, and Daniel Duckworth. Nesf: Neural semantic fields for generalizable semantic segmentation of 3d scenes, ArXiv, abs/2111. 113660, 2021.
Peng Wang. Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. ArXiv, (bbs2106.10689) 2021.
Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Yu Sun, and Xin Tong. O-enn. ACM Transactions on Graphics (TOG), 36:1 - 11, 2017.
Peng-Shuai Wang, Chun-Yu Sun, Yang Liu, and Xin Tong. Adaptive o-cnn: A patch-based deep representation of 3d shapes. arXiv: Computer Vision and Pattern Recognition, 2018.
Frederik Warburg, Ethan Weber, Matthew Tancik, Aleksander Holynski, and Angjoo Kanazawa. Nerf- busters: Removing ghostly artifacts from casually captured nerfs. 2023 IEEE/CVF International Conference Computer Vision (ICCV), pp. 18074-18084 2023.
Xin Wei, Ruixuan Yu, and Jian Sun. View-gen: View-based graph convolutional network for 3d shape analysis. 2020 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1847-1856, 2020.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1912-1920, 2014.
Teng Xu, Jiamin Chen, Peng Chen, Youjia Zhang, Junqing Yu, and Wei Yang. Tiger: Text-instructed 3d gaussian retrieval and coherent editing. ArXiv, abs/2405. 14455, 2024.
14

---

Published as a conference paper at ICLR 2025
Xiaomeng Xu, Yanchao Yang, Kaichun Mo, Boxiao Pan, L. Yi, and Leonidas J. Guibas. Jacobinerf: Nerf shaping with mutual information gradients. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16498-16507, 2023.
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-4815, 2021.
Mohsen Yavartanoo, Euyoung Kim, and Kyoung Mu Lee. Spnet: retrieval using stereographic projection. In Asian Conference on Computer Vision, 2018.
Deep 3d object classification and
Weicai Yc, Shuo Chen, Chong Bao, Hujun Bao, Marc Pollefeys, Zhaopeng Cui, and Guofeng Zhang. Intrinsicnerf: Learning intrinsic neural radiance fields for editable novel view synthesis. 2023
IEEE/CVF International Conference on Computer Vision (ICCV), pp. 339-151, 2022.
Chandan Yeshwanth, Yuch- Yueh-Cheng Liu, Matthias Nießnnrr, and Angela Dai. Scannet++: A high-fidelipg dataset of 3d indoor scenes, In Proceedings of the International Conference on Computer Vision (ICCV), 2023.
Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. 2022 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5491-5500, 2021.
Hao Zhang, Feng Li, Xueyan Zou, Siyi Liu, Chun yue Li, Jianfeng Gao, Jianwei Yang, and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. 2023 IEEF/CVF International Conference Computer Vision (ICCV), pp. 1020-1031, 2023a.
Junbo Zhang, Runpei Dong, and Kaisheng Ma. Clip-fo3d: Learning free open-world 3d scene representationn from 2d dense clip. 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pp. 2040-2051, 2023b.
Xiaoshuai Zhang, Zhicheng Wang, Howard Zhou, Soham Ghosh, Danushen Gnanapragasam, Varun Jampani, Hao Su, and Leonidas J. Guibas. Condense: Consistent 2d/3d pre-training for dense and sparse features from multi-view images. 2024.
Chen Zhao, Jiaqi Yang, Xin Xiong, Angfan Zhu, Zhiguo Cao, and Xin Li. Rotation invariant point cloud analysis: Where local geometry meets global topology. Pattern Recognit., 127: 127:108626, 2022.
Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew J. Davison. In-place scene labelling and understanding with implicit scene representation. 2021 IEEE/CVF International Conference Computer Vision (ICCV), pp. 15818-15827, 2021.
Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Singh Behl, Jianfeng Wang, Lu Yuan, Nanyun Peng. Lijuan Wang, Yong Jae Lee, and Jianfeng Gao. Generalized decoding for pixel, image, and language. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15116-15127, 2022.
Xueyan Zou, Jianwei Yang, Hao Zhang, Feng Li, Linjie Li, Jianfeng Gao, and Yong Jae Lee. Segment everything everywhere all at once. ArXiv, abs/2304.0n 03330406678.,.. 2023a.
Zi-Xin Zou, Weihao Cheng, Yan-Pei Cao, Shi-Sheng Huang, Ying Shan, and Songiie Zhang. Sparse3d: Distilling multiview-consisten consistent diffusion for object reconstruction from sparse views. ArXiv, 14078, 2023b.
15

---

Published as a conference paper at ICLR 2025
Notay
10
10
15
-00
-10.3R-20.10
_00100
Figure 6: 3D t-SNE for clean pixel feature (O) and noisy pixel feature (X) for XDecoder's Feature Pyramid Network (Level Activation Resolution: 512 64 88)
APPENDIX
A ADDITIONAL METHOD DETAILS
A.1 VISUAL EMBEDDING GENERATION
To enable zero-shot performance on unseen scenes, our system is compatible with pre-trained VLMs without fine-tuning. This design choice contrasts with other NGR semantic analysis methods, which have high training and engincering overhead for each NGR format. Given a visual encoder of a VLM & parameterized by o, we generate the visual embeddings Ev:
(8) where I E 3 is the rendered RGB image, and Eg are visual embeddings dimension d. Different visual encoders produce varying numbers of embeddings per image. For example, CLIP generates a single embedding (I. 1) for the entire image (Cherti al., 2022). More recent works, such as XDecoder produce multiple embeddings:
E, +(1: RLxdd
Ewwhoee Eseg: Maeg)= = (I:
(9)
where E is pre-trained XDecoder visual encoder parameterized by Φz. Ewhote E RIxd represents the whole image embedding. similar to a CLIP's embedding for the entire image. XDecoder also generates K image segment embeddings (Eeeg E æK*d) with masks Mseg E IR xxx indicating the correspondence between the embedding and segments the image. See Figure for a visualization of these segments and the associated concepts.
In our system, do not utilize all image segment emheddings from XDecoder, some masks may not have a corresponding segment in the image. Specifically, we ignore image embeddings with corresponding mask of less than 10 pixels they usually don't 'tn' correlate with any concepts in the image.
Our dataset then stores all the image embeddings generated via VLMs given all rendered images as inputs. During lookup. image embedding (and the associated scene) is retrieved based on the highest cosine similarity with the text embedding, generated from the user query g via the VLM's text encoder V, parameterized by vb. Given scenes indexed by n E N, rendered images of m E M per scene, the top scene/inage/embeeddng is given by:
arg max cossim(e.. "...d Ex) n.m.!
(10)
where en.mml E Rd is the ith image (segment) embedding for the mith image of the nith scene, and e, (g: Rd is the text embedding based on user query q.
Our system flexibly supports any VLM that employs image-text contrastive pre-training and is capable of independently generating visual and text embeddings with & and V. This adaptability allows the system to benefit not only from advancements in pre-trained VLMs but also from fine-tuning on domain-specific visual textual concepts when necessary. In this work, however, test without fine-tuning to highlight the zero-shot performance while ensuring compatibility across any NGR format capable of rendering RGB images.
16

---

Published as a conference paper at ICLR 2025
Clean
No
Both
(a) I-SNE for XDecoder's Feature Pyramid Network (Level 2, Activation Resolution: 512 x 32 X 44)
Clean
Nosy
Both
-00
20
-30
10
15
(b) I-SNE for XDecoder's Feature Pyramid Network (Level 3, Activation Resolution: 512 x 16 x 22)
Figure 7: t-SNE visualization of clean pixel features (O) and noisy pixel features (X) for different levels of XDecoder's is Feature Pyramid Network.
Figure 8: t-SNE for Nerfacto clean/noisy pixel feature (O/X) and Splatfacto clean/noisy pixel feature (O/+)
A.2 NGR NOISE ANALYSIS VIA PRE-TRAINED VLMS
We demonstrate that the ability to generate well-separated features of clean pixels and noisy pixels not a property specific to a certain layer of the XDecoder VLM, but applies to different layers of different VLMs. In addition, we also demonstrate that the noise features from NeRF (Nerfacto) and 3DGS (Splatfacto) are very close with each other, well-separated from the clean pixel features from the respective NGRs.
The t-SNE visualizations in Figure 2 and the rest in this section are generated via sampling activation features from the specified VLM layer given clean renderings and noise renderings. The clean renderings are generated via sampling 10 images (linspace) from the training image sequence of each scene; linspace is selected to ensure best diversification of the training clean images. The noise renderings are generated with 10 renderings at random viewpoints for each scene. We sample activations per rendering randomly generate the t-SNE plot. We use all 13 LERF scenes for the renderings. We select a small number of images and activations to avoid clustering and provide better visualization.
17

---

Published as a conference paper at ICLR 2025
For calculation of the actual multivariate Gaussian distribution representing the noise specified in Eq I, we use 50 renderings from random viewpoints for each scene, and 50 activations from each of the renderings. The random viewpoints' camera origins are sampled uniformly from the center 1 x x 1 box of the scene. The view direction components x, y, and are sampled uniformly from [-1, IJ. and the view direction vector is normalized. Note that do not require the clean training images for calculating the noise Multivariate Gaussian distribution. XDecoder's 's image backbone ends with Feature Pyramid Network. We use XDecoder's (variant: X-Decoder-oq2011 Feature Pyramid Network's 's highest resolution level activation (as viewed in Figure 2 and 6) for noise multivariate Gaussian distribution training unless otherwise specified.
First, we provide a 3D tSNE of the clean and noisy features in Figure 6. The randomly sampled features are the same as the one demonstrated in the 2D t-SNE plot in Figure 2. We visualize them in 3D here to demonstrate that the clean features occupy a much larger volume. This shows that, in high dimensions, the noise features are much more tightly clustered than the clean features.
In Figure 7, we demonstrate the t-SNE plot for activations from other layers of the XDecoder visual encoder. Despite the lower resolution of the activation map for these feature pyramid network levels, the noise features and clean features are still well-separated.
In Figure 8, we demonstrate the t-SNE plot of pixel features from Nerfacto renderings and Splatfacto renderings. As seen in the plots, observe that the noisy pixel features from the different NGRs are close to each other, separated from the clean pixel features from their respective NGRs. This means that using one type of NGR to train multivariate Gaussian distribution model the noisy regions from another NGR is possible. We will demonstrate the impact on retrieval accuracy with this cross-NGR noise Gaussian model training in Table 9.
Clean
No
Both
-10
-30
-3
Figure 9: t-SNE for clean pixel feature (0) and noisy pixel feature (X) for the open-elip model ViT-L-14-336 (second to last transformer layer, Activation Resolution: 1024 x 24 x 24)
Lastly, we visualize the t-SNE embeddings from the OpenCLIP model (variant: VTT-L-14-336) in Figure 9, which demonstrates a similar degree of separation between clean and noisy features. V4T-L-14-336 divides the image into 14 x 14 patches (tokens), and consists of 24 transformer layers. We utilize the activations from the penultimate (23rd) transformer layer to calculate the Gaussian distribution for noise. The final transformer layer is excluded as its patch-wise outputs are not leveraged during the contrastive training process, and thus lack sufficient semantic information. FLIP made similar observation, and suggested performing token-wise contrastive learning to ensure the final layer's ' activations carry meaningful semantic content (?). However, we found this alternative training scheme to be unnecessary for distinguishing between noise and clean features. The activations from the penultimate layer of a standard contrastively trained vision transformer are sufficient for this task.
In summary, we observed that the noise features can be separated from clean features given different kinds of VLMs, as well as different kinds of NGRs.
A.3 SMART CAMERA MOVEMENT IN 3D
In Section 3.4, we discussed how to rotate the camera to a viewpoint containing cleaner pixels given single rendering. Here, demonstrate that by rendering set of images from viewpoints that vary in depth (w.r.t. the center image in its local coordinate), we allow the camera to change viewpoints for both translation and rotation, in order to navigate in 3D.
18

---

Published as a conference paper at ICLR 2025
Front
Center
Back
Splatfacto
(a)
(b)
XDecoder- Splatfacto
CLIP-
Splatfacto
(g)
(h)
Nerfacto
(k)
XDecoder- Nerfacto
(0) Figure 10: Splatfacto and Nerfacto Rendering with Varying Depth in Camera's Coordinate; And the Corresponding Clean Score Map from Different VI.Ms
(m)
(n)
Figure 10 visualizes set of three images with identical rotation matrices but translated along the z-axis (depth) in the camera's 's local coordinates. Noise analysis is performed all three images, and the optimal next viewpoint is selected based on the noise analysis result, aiming for the highest estimated clean score.
The result of this algorithm is method for guiding the camera to move incrementally toward cleaner views of the scene, starting from random initialization point. While SCMM with depth variation could be integrated into our retrieval system to produce higher-quality renderings and improve retrieval accuracy,
opt not to this approach. The depth variation requires rendering set of images instead of just one, and the associated computational cost better allocated to selecting more initialization points to achieve greater scene coverage. However, how navigation can be achieved using only pre-trained NGR and VLM, which could benefit other works that interact with NGRs such as robot navigation. We provide example of SCMM recovering from an extremely occluded viewpoint below.
SCMM with depth demonstrates
In Figure 11, we show example where the "center" current viewpoint is under extreme occlusion by floater artifacts. The "back" viewpoint provides some hints regarding where should the camera viewpoint point for the next step. The SCMM algorithm leverages the analyzed noise to caleulate
19

---

Published as a conference paper at ICLR 2025
Front
Center
Back
Image
(a)
(b)
Clean Score
Extended Clean Score
(h)
Figure 11: Visualization of Images, Clean Scores, and Extended Clean Scores for LERF Dozer Scene Rendering Set (Front, Center, Back).
(a) location
(b) location
(c) location 3
(d) location
(c) location 5
(f) location 6
Figure 12: Dozer Scene Smart Camera Movement Recovering from Extremely Noisy Viewpoints via
Movement in Depth
the extended clean score map for translation and rotation. Figure 12 shows how SCMM recovers from the poor initialization seen in Figure 11, iteratively moving the camera toward cleaner views. While our algorithm cannot guarantee successful recovery in every case, it often requires only a small hint, such as this example, to complete the recovery process.
A.4 EFFICIENT NOISE ANALYSIS AND SMART CAMERA MOVEMENT VIA LOW-RESOLUTION RENDERING
In order to render high-resolution clean images efficiently, first render low-resolutio images to perform noise analysis. We visualize this process in Figure 13. Given the noise analysis result, we move the camera to select viewpoints for cleaner images. From all the low-resolution images,
20

---

Published as a conference paper at ICLR 2025
Neural Graphics
Scene
Smart Movement
Initial Camera
Low Resolution Rendering
Selected Top K Camera Viewpoint
Noise Analysis
Viewpoint Sampling
Legend:
Low Resolution Ops
Top K Unsimilar Clean Rendering
Clean & High Resolution Rendering
High Resolution Ops
Selection
Figure 13: Process for Rendering of Clean and High Resolution Image from Noise Analysis and Smart Camera Movement with Low Resolution Renderings
RGB
Clean Score
Clean Score Extended
Low Res
(b)
High Res
(d)
(c)
(f)
Figure 14:
Low and High Resolution Rendering and Noise Analysis Comparison for Ramen Scene.
select the top-k not-similar images (image embeddings with cosine similarity less than 0.9) to render in high resolution. Note that most of the steps can be completed in the low-resolution space.
We visualize the noise analysis result for the same viewpoint, rendered in high resolution and low resolution (1/16 of the high-resolution in pixel count) for comparison in Figure 14. We use the same multivariate Gaussian distribution (trained through activations from high-resolution renderings). We note that despite much lower resolution, the predicted noise remains mostly the same. The different predicted noises can lead to differences in terms of the exact camera movement, but both move the camera towards cleaner regions.
Such low fow-resolution rendering allows us to efficiently navigate within the NGR scene, adding only a few seconds overhead to the final high-resolution rendering and visual embedding generation. We provide guantitative results on the speed in Appendix B.8.
A.4.1 CAMERA INITIALIZATION AND CONVERGENCE
We initialize the camera in SCMM by randomly sampling the camera origin components y, and z independently from a Normal distribution with standard deviation of 0.5. The view direction components.. x, y, and z sampled uniformly from [-1, IJ, and the view direction vector is normalized. The same sampling method is applied to random viewpoint rendering. The Normal distribution sufficiently distributes the initial cameras, with a bias for more center views which more likely to have partially noise-free views. Combined with uniform sampling of the viewing direction, this approach is more likely to provide different angles of the objects the center while also capturing objects in the periphery. This approach provides good coverage of the scenes, while enabling the cameras to navigate to more noise-free views.
Our SCMM (rotation only, as used the retrieval process described the paper) is capable of recovering from poor initializations if the camera can be rotated better viewing direction without
21

---

Published as a conference paper at ICLR 2025
(a) Nerfacto's 's Depth
(b) GT Depth
(c) Depth Error
(d) Nerfacto's RGB
(e) GT RGB
(1) RGB Error
Figure 15: Depth and RGB Error for Image of ScanNet Scene 0001 (Red is higher error; Blue is lower error.)
(b) Ours Uncertainty Figure 16: Estimated Uncertainty for Bayes' Rays and Ours. (Red is higher uncertainty; Blue is lower uncertainty.)
(a) BayesRays' Uncertainty
moving. It converges to a local optimum as long as the estimation of the cleanliness level beyond the current field of view is reasonable, which we found be almost always the case in practice. To recover from poor initializations where translation required to achieve cleaner views (e.g., when the camera origin is inside artifact clouds objects), SCMM with rotation and translation must be used. However, we do not use SCMM with rotation and translation for the retrieval accuracy measurements, as we found that using rotation only results in shorter scene analysis time while maintaining the same level of retrieval accuracy.
Should the ability to recover from poor initializations become critical, SCMM with rotation and translation can be employed. It is important to note that both versions of SCMM may still result in individual bad renderings. Therefore, sampling multiple renderings and rejecting bad ones is necessary to improve the collective visual semantics quality for the scene.
A.5 COMPARISON OF NEURAL GRAPHICS NOISE ANALYSIS WITH NERF UNCERTAINTY ESTIMATION
Our predicted noise level through Neural Graphics Noise Analysis can be interpreted as measurement
of the uncertainty within the trained NGR scene. To evaluate this, we compare our noise (uncertainty)
Scene #
001
079
158
316
Depth Bayes"
0.28
0.35
0.20
0.29
Depth Ours
0.42
0.40
0.34
0.49
RGB Bayes
0.38
0.36
0.42
0.26
RGB Ours
0.18
0.23
0.21
0.25
Table AUSE for Bayes" Rays and Ours (Measured against Depth and RGB Error), Lower is Better
22

---

Published as a conference paper at ICLR 2025
#Degth Lipenaary (Bayskka)) 60eeth Licesterr sgars (Durg
(A6t Intomikiy-saa (Rageday)))
0908 by Unowtrrryy Saa% 10a99
100
0.0
0.8
0.6
0.6
0.4
0.4
0.-
32-
0.0
0.0
0.2
9.1
0.6
0.8 1.0
0.0
0.2
0.4
0.0
0.0
1.0
Robio Removed
Robio Romoved
(a) Depth Sparsification Error
(b) RGB Sparsification Error
Figure 17: Sparsification Error for Bayes' Rays (Purple) and Ours (Green), Measured Against Depth and RGB Error. (Solid line represents AUSE, Lower is Better)
prediction with Bayes' Rays, the state-of-the-ard method for uncertainty prediction in NeRF (Goli et al., 2023).
We note that other uncertainty estimation methods exist. For example, several works modify the NGR neural network to output uncertainty using Bayesian Neural Networks (Shen et al., 2021a; Pan et al., 2022: Shen et al., 2022). Niko al. proposed ensemble of NeRFs to estimate uncertainty, which is computationally expensive (Sünderhauf et al., 2022). FisherRF uses the Fisher information matrix between learned parameters of NGRs, but this is only applicable models like 3DGS and Plenoxels with sparse, uncorrelated parameters (Jiang et al.. 2023). Bayes' Rays perform post-hoc training on NeRFs, requiring both RGB and density information from the NeRF MLPs (Goli et al., 2023). Since Bayes' Rays achieved the state-of-the-arr performance, we choose to compare our solution with Bayes' Rays.
Assessing the quality of estimated uncertainty is challenging. Bayes' Rays proposed using the Area Under Sparsification Error (AUSE), measured against depth. The idea is that the pixels sparsified (gradually removed) in two ways: first by removing pixels based on depth error (highest depth uncertainty pixels are removed first), and second by removing pixels based on predicted uncertainty (highest uncertainty pixels are removed first). Ideally, higher uncertainty predictions should correspond to greater depth error, resulting in both sparsification processes retaining similar errors. The gap of remaining eтrors between these two sparsification processes constitutes the AUSE- Depth error. As shown in Figure 17)),, Bayes' Rays achieves a lower (better) AUSE-Depth than our method. This is expected since Bayes' Rays estimates spatial uncertainty via neural network perturbations within NeRF's MLP.
However, when we conduct the same AUSE measured against RGB error, our method achieves a lower (better) AUSE-RGB compared Bayes' Rays (Figure 17())). This indicates that our approach more effectively estimates uncertainty that aligns with RGB rendering errors, which is reasonable given that our uncertainty (noise) is derived from RGB renderings the first place. We show the quantitative comparison for the 4 scenes provided in Bayes' Rays dataset in Table 6. On all four scenes, we perform better for AUSE-RGB while Bayes' Rays perform better for AUSh-Depth.
To generate our uncertainty predictions, we simply use the multivariate Gaussian distribution trained with LERF scene noise renderings. Using this Gaussian distribution, estimate the noise level for the ScanNet scene's 's rendering from a Nerfacto model. Our multivariate Gaussian distribution did not renderings from ScanNet scenes during training. Unlike Bayes' Rays, our method does not require access to or post-processing of the NGR model. Moreover, our approach is applicable to both 3DGS and NeRF, while Bayes' Rays is limited to bese-based solutions.
While our method outperforms Bayes' Rays AUSE-RGB and Bayes' Rays excels in AUSE- Depth, we observe notable differences from all of the error by depth, error by RGB, and uncertainty predictions from both Bayes' Rays and our method. This highlights a need for further research into uncertainty estimation and the development of appropriate metrics to assess their effectiveness.
23

---

Published as a conference paper at ICLR 2025
(a) Correct Noise Identification
(b) Incorrect Hlgh-FFrquency Object Noise Figure 18: Noise Identification via Finetuning of XDecoder Segmentation Head
d (0, 1)
d (1, 0, 0)
d (0, 0, 11
(a) Location- based Coverage
(b) iinitaationbbased Coverage
Figure 19: Coverage Map Visualization
A.6 NOISE ANALYSIS THROUGH VLM CLASSIFICATION
In addition modeling noise with multivariate Gaussian distribution, another viable approach is to fine-tune a segmentation model to segment the noise regions from clean regions. In summary, we find that a fine-tuned XDecoder often treats high- freuurry objects as noise, as seen in Figure 18. This particularly detrimental for scene retrieval as it will direct cameras away from interesting regions containing high-freet sighffrey objects.
To test the idea, we fine-tune an XDecoder model's segmentation head to classify between clean and noisy. Specifically, we the same random viewpoint renderings that were used fit the multivariate Gaussian distribution (50 from each of Splatfacto NGRs representing the 13 LERF scene), coupled with 50 training viewpoint renderings from each scene. We modify the last layer of the segmentation head so it only produces two classes, trained to predict all training viewpoint renderings as clean, and all random viewpoint renderings as noise. We unfreeze the segmentation head parameter and train for 35 epochs with SGD (Ir=le-4, momentum=0.9, weight decay=le-4). We also tried unfreezing the Feature Pyramid Network but obtained similar results.
more ideal fine-tuning solution would involve using images with proper segmentation of noise and clean regions. However, found this be very ambiguous task not easily carried out by manual labor. As we have discussed in Appendix A.5, existing uncertainty estimation methods do generate high-quality segmentations (particularly when evaluated using RGB error) automatically either.
A.7 SCENE COVERAGE CALCULATION
We evaluate coverage for different rendering methods within the spatially contracted space of Nerfacto. Nerfacto uses spatial contraction technique modified from MipNeRF-360 (Barron et al., 2021), which warps infinite distance into box of size 4 x 4. We divide the box into a grid of
24

---

Published as a conference paper at ICLR 2025
64 x 64 x 64 voxels. We did not evaluate the coverage with Splatfacto as it is difficult to measure such a statistics in an infinite space, but we expect similar conclusions to hold for both NGRs. To update the coverage statistics within this grid, we perform ray traversal for each rendering. Rays are traced from the near plane of the rendering to the estimated depth predicted by Nerfacto, marking any voxel they traverse through covered. For the location+orientation statistics, we compute the dot product between the ray and the normal vector of the voxel's face. We visualize this in Figure 19. If the dot product is positive, the orientation is recorded as observed. Although this is relatively relaxed criterion, it is consistently applied across all rendering types.
Before rendering, we also sample the density at the center of each voxel. If a voxel's density exceeds 0.5, we mark the entire voxel, along with all its orientations, as covered.
For the TTaiin measurement, we track voxels (and faces) covered by rays from the training renderings, without pre-filling the grid with coverage information based on density. We then calculate the percentage of voxels (and faces) covered by the training rays that are also covered by other rendering types.
B ADDITIONAL RESULTS
B. EXPERIMENT SETTING
Dataset We utilize the LERF and ScanNet++ datasets, comprising 13 scenes and 280 scenes respectively (Kerr et al., 2023; Yeshwanth et al., 2023). Only ScanNet++ scenes with object labels are included.
Text Query We employ the original object labels from LERF and ScanNet++ text queries. Additionally, we generate image captions from the training images using the LLaVA (v1.5-13b) model with the prompt: Describe the image in detail one sentence. It is important to note that the VLM used in our retrieval (XDecoder) never observes the exact training images at any point. Furthermore, the LLaVA model developed independently of XDecoder (Liu et al., 2023b; Zou et al., 2022).
Neural Graphics Representations We use the nerfstudio implementations of 3D Gaussian Splat- ting and NeRF, specifically Splatfacto and Nerfacto (Tancik al., 2023). All models are trained using the default configuration for 30,000 epochs. For Nerfacto, we also enable the "use gradient-scant option to scale the gradient near the camera, reducing artifacts and creating a stronger baseline for rendering from random poses (Philip & Deschaintre, 2023). LERF employs the same Nerfacto as its RGB rendering component. LangSplat uses the original 3DGS implementation, while the nerfstudio Splatfacto aims to replicate 3DGS but may have minor implementation differences (Tancik et al., 2023; Qin et al., 2023; Kerbl et al., 2023).
Metries A retrieved scene is considered correct it matches the user query. For object labels, this means the scene contains the queried object. For LLaVA image captions, this means the scene was used to generate the caption. Note that LERF has minimal label overlap between scenes, whereas ScanNet++ has significant overlap. We measure precision P@k=1,5,10, 10, indicating whether the top-k retrieved scenes include the correct scene.
Experimental Platform Experiments are conducted on a desktop with an Intel 17-13700K CPU, Nvidia RTX 4090 GPU, and 64GB of RAM. We use PyTorch 2. 1.2 with CUDA 12.0 on Ubuntu 22.04 LTS.
B.2 DIFFERENT VLM
In addition to XDecoder, also evaluate our system's performance when other VLMs are used. In particular, we select open-clip (Cherti et al., 2022) to generate the visual embeddings and text embeddings. Compared to XDecoder, open-clip does not perform segmentation or generate multiple embeddings (corresponding to different segments) per image. It involves a simple process of generating one image embedding per image. We use this image embedding that describes the entire image for retrieval in our system.
In Table 7, we compare the retrieval performance of our system when the XDecoder CLIP model is used. Regardless of the viewpoint choice, our system achieves higher performance when XDecoder used. This is understandable XDecoder is a more recent solution, and it also captures sub-image contents in addition to the whole image embedding. We note that the Random viewpoint rendering is particularly difficult for CLIP to generate any meaningful embedding. Even with 100 images being
25

---

Published as a conference paper at ICLR 2025
Decoder Viewpoint
OpenCLIP Viewpoint
# Img.
Training
SCMM
Random
Training
SCMM
Random
57.89
30.08 (-27.81)
17.29 (-40.60)
41.35
20.59 (-20.76)
11.63 (-29.72)
5
68.42
64,66 (-3.76)
18.80 (-49.62)
60.90
46.83 (-14.07)
13.45 (-47.45)
78.95
67.67 (-11.28)
24.81 (-54.14)
61.65
47.24 (-14.41)
16.43 (-45.22)
20
83.46
73.68 (-9.78)
34.59 (-48.87)
63.16
50.47 (-12.75)
18.24 (-44.92)
50
84.21
77.69 (-6.52)
49.62 (-34.59)
64.62
53.79 (-10.83)
22.49 (-42.13)
100
84.95
80.02 (-4.93)
57.89 (-27.06)
65.29
57.68 (-7.61)
24.93 (-40.36)
Table 7: Retrieval Accuracy (P@ 1) for Splatfacto Model on LERF dataset. Comparison between XDecoder and OpenCLIP.
#
Viewpoint SCMM
Img.
Training
Random
1
57.89 81.95 197.74
30.08 / 80.28 95.96
17.29 / 54. 83.46
5
68.42 93.98 98.50
64.66 / 99.66 97.09
18.80 48.87 885.1
10
78.95 96.24 99.25
67.67 195.69 99.25
24.81 / 58.65 / 89.47
20
83.46 98.50/ 100.0
73.68 / 197822 99.68
34.59 / 69.17 /93.23
50
84.21 / 99.25 / 100.0
77.69/ 98.23 100.0
49.62 / 80.45 99.448
100
84.95 98.50/ 100.0
80.02/ 98.61 100.0
57.89 87.22 99.25
Table 8: Retrieval Accuracy (P@ 1/5/10) for Splatfacto on LERF Dataset with LERF Object Labels
rendered per scene, still has a significant gap with the Training viewpoint rendering. This is due to the noisy component in the images severely hindering CLIP's ability to generate quality image embedding. For XDecoder, even when large sections of the image are noise, it can still generate some image embeddings highly associated with the clean regions in image.
The CLIP model's 's performance demonstrates the importance of selecting noise-free high quality view when performing 3D NGR analysis with VLMs.
B.3 SPLATFACTO @@ K RETRIEVAL ACCURACY
In Table 8, we report top-k retrieval accuracy for the Splatfacto model. As shown the in table, the retrieval accuracy for equals 5 and 10 increases as the top I retrieval accuracy increases.
B.4 NERFACTO OBJECT LABEL AND LLAVA CAPTION RETRIEVAL
In Table 9, show the result for Nerfacto model on the LERF dataset. Compared to Splatfacto's result in Table 1, we see comparable results. The object label retrieval accuracy slightly decreases but the LLaVA caption accuracy slightly increases. We believe this is result of the Nerfacto model being worse at capturing small details of the objects but still having good overall image rendering quality.
In addition, we also demonstrate the retrieval accuracy when different sources are used for training the multivariate Gaussian distribution for noise. Specifically, we use either Gaussian distributions trained from Nerfacto noise renderings or Splatfacto noise renderings. We found the training from the same source the model (Nerfacto) to work slightly better, but the different in accuracy minimal.
B.5 MULTIPLE QUERIES RETRIEVAL
As the scenes are complex and may contain similar objects, users may want to retrieve a scene that satisfies several descriptions or contains several objects. We perform another experiment allowing the users to specify multiple queries at time.
For evaluation purposes, queries, we calculate the product of the exponential of the maximum cosine similarity for each scene with respect to each query (q). The top scene, indexed by n, identified follows:
combinations of 1, 2, 5 object labels from LERF To combine the
arg max II exp(1
co))im(en...G))
(E(1...,L)
where L is the total number of visual embeddings for each scene. A notable challenge arises when all embeddings are stored single database, requiring the calculation of cosine similarities for all stored embeddings to retrieve the top scene, which is highly inefficient. To address this, we can
26

---

Published as a conference paper at ICLR 2025
Object Label SCMM
LLaVA Caption SCMM
Training
Random
Training
Random
Noise Gaussian
Nerfacto
Splatfacto
Nerfacto
Splatfacto
52.79
24.24
23,89
14.86
55.37
23.03
21.46
17.6
63.60
61.45
59.36
19.37
64.08
67.64
66.73
21.54
10
67.21
65.74
64.68
19.82
69.30
71.52
70.25
29.64
20
76.76
65.52
63.60
34.23
71.42
73.84
72.01
40.76
50
75.86
72.43
71.92
40.09
75.09
75.18
74.55
49.73
100
78.50
72.23
72.50
44.59
75.51
75.63
74.15
55.86
Table 9: Retrieval Accuracy (P@ 1) for Nerfacto Model on LERF dataset; Noise Gaussian indicates the Source of Training for the Noise Multivariate Gaussian Distribution
#
Training
SCMM
Random
Img.
1
57.89 63.85 64.69
30.08 / 29.62 29.02
17.29 17.69 12.41
5
68.42 83.82 /91.08
64.66 73.85 89.86
18.80 18.85 12.59
10
78.95 91.15 /9..43
67.67 83.08 94.76
24.81 25.38 19.58
20
83.46 95.00 98.008
73.68 / 84.23 95.45
34.59 / 36.15 / 31.99
50
84.21 95.77 99.655
77.69 / 88.46 99.65
49.62 57.69 64.34
100
84.95 96.54 99.83
80.02 / 90.01 / 19.448
57.89 /7.15 87.76
Table 10: Retrieval Accuracy (P@ 1) for Splatfacto on LERF dataset with Multiple Labels (1/2/5)
store of embeddings per scene, making the calculation scale linearly with the number of scenes instead of the number of visual embeddings. If multiple query retrieval is unnecessary, storing all visual embeddings in single database remains option.
Overall, accuracy significantly improves as the number of labels increases, achieving over 99% accuracy when 5 labels are used.
B.6 MIXED NGR RETRIEVAL ACCURACY
In Table 11, we show the retrieval accuracy when mixture of Splatfacto and Nerfacto models are stored in the database. In our system, rendering, and then performing analysis from different models is very easy. As shown in the table, our system achieves a high level of accuracy even when the stored NGRs are different.
B.7 SCANNET+ RETRIEVAL ACCURACY COMPARED WITH LANGSPLAT
In Tables and 13, we present comparison between our method and LangSplat on the ScanNet++ dataset, using either object labels LLaVA caption sentences queries. To make the database for storing LangSplat features manageable, we only store 200 embeddings per image, randomly selected from all the embeddings. Our approach consistently achieves higher retrieval accuracy than LangSplat, particularly when training viewpoints are available. The performance gap is even more pronounced (almost double) when LLaVA captions used, as LangSplat struggles handle whole image descriptions effectively. Given that ScanNet++ contains highly similar scenes (offices, bedrooms, living rooms, etc.), we also adapt our multi-object-label approach (Appendix B.5), using on multiple sentences. This multi mentti-eenteneeeeee is unique to our system, as our efficient query process leverages the small number of embeddings generated during scene analysis. The rapid querying capability allows to perform multiple queries with additional sentences, leading to more robust collective decision.
B.8
VIEW SELECTION EFFICIENCY ANALYSIS
In Table 14, we perform a speed comparison between the low/high resolution rendering and noise analysis, well the total time for using and not using the hierarchical view selection process. As seen in the table, rendering 80 additional low-resolution images for each set of 20 high-resolution images adds less than 50% overhead. Since our algorithm is already very fast in analyzing a 3D scene, such overhead is relatively small, and the retrieval accuracy is much better compared to rendering more images from random viewpoints.
27

---

Published as a conference paper at ICLR 2025
Object Label
LA Caption
# Img.
Viewpoint
Viewpoint SCMM
Training
SCMM
Random
Training
Random
54.28
28.46
14.92
49.43
33.24
14.24
5
65.68
62.84
15.84
67.04
59.26
24.67
10
72.24
63.12
22.42
69.00
62.85
26.85
20
79.49
75.43
32.43
71.42
64.63
31.46
50
80.02
77.43
46.74
72.80
65.43
33.75
100
80.67
79.62
54.34
74.63
71.86
45.64
Table 11: Retrieval Accuracy @P@ 1) for 7-to-6 mix of Splatfacto Model and Nerfacto Model
Representing the LERF scenes
Object Label
Ours
LangSplat
Viewpoint
Viewpoint
Img.
Training
SCMM
Training
10
41.62
39.63
38.19
20
50.06
47.34
43.68
50
58.23
54.39
51.52
Table 12: Retrieval Accuracy (P@ 1) for Splatfacto Model with Object Labels for ScanNet++ Scenes.
Comparison between ours and LangSplat.
B.9 LERF AND LANGSPLAT EXPLAINATIONS
In this section, we discuss the implementation of LERF and LangSplat and address the issues in terms of computation and storage when adapting these methods retrieval system.
LERF LERF embeds multi-scale image embeddings from open-clip into "semantic radi- ance field" (Cherti et al., 2022). While the original RGB radiance field only outputs 4 values <dGiB+density), the semantic field outputs features of size 512 open-clip image features have size 512. To encode such large features effectively, significantly larger MLPs and hash-tables are used for the semantic field compared to the RGB field encoding. For LERF, overhead in computing and storage for both training and rendering stems from this excessively large semantic field.
LangSplat LangSplat uses the 3D Gaussians to encode the semantic features. They used SAM to segment the training images into different segments and used open-clip encode each segment into an embedding to train a "Semantic 3D Gaussian" (Kirillov et al., 2023). However, instead of encoding features of size 512, they encode features of size This is achieved via training a per-scene autoencoder that maps the feature of size 512 to feature of size 3 during training. The "Semantic 3D Gaussian" is relatively small since it only encodes feature size 3. However, such a feature is not comparable between scenes the autoencoders are trained on per-scene basis. To perform retrieval across different scenes, we decode the features to size of 512 using the trained autoencoders, and store them in the database. In summary, for training, the LangSplat overhead comes for open-clip embedding generation for a large number segments created by SAM and training the autoencoder. For rendering, the LangSplat overhead comes from converting pixel-wise size 3 feature size 512 feature with the trained autodecoder.
B.9.1 LERF AND LANGSPLAT SPEED AND STORAGE
For NGR Training, our methods uses the nerfstudio's 's Splatfacto and Nerfacto implementations as is. Our training of the RGB-only NGRs resulted in significantly simpler pipelines and lower overhead than LERF and LangSplat. Below, we will explain the LERF and LangSplat overhead a high level, and would refer readers to their paper for more details (Kerr et al., 2023; Qin et al., 2023).
LERF's training overhead comes from training the field to output visual embeddings alongside the RGB values. The visual embedding's size (512) is significantly larger than the RGB's size (3). As result, LERF chose to use large hash-table feature grid to store and learn the visual embeddings. In addition, they used a multi-scale visual embedding, which included 30 levels scales. If one renders all 30 levels of scales, the total embedding size for I image is 45GB (assuming resolution of 994 x 738 as the LERF data). In LERF, only one scale is selected during the interactive session for visualization purposes. However, such selection depends the text prompt. In retrieval setting like ours, this is impossible. Therefore, to make implementation feasible, we chose 3 scales instead of 30, which still leads to very large size but experiments could be conducted.
28

---

Published as a conference paper at ICLR 2025
LLaVA Caption
Ours
LangSplat
# Img.
Viewpoint
Viewpoint
Training Sent
SCMM
SCMM
SCMM
SCMM
Training
Sent
5 Sent
10 Sent
20 Sent
10
20.56
19.12
31.46
35.66
36.24
8.17
20
26.68
24.69
28.92
37.72
39.79
9.52
50
30.92
27.57
29.46
40.53
43.58
11.28
Table 13: Retrieval Accuracy (P@ 10) for Splatfacto Model with LLaVA Captions for ScanNet++
Scenes. We optionally use multiple sentences for retrieval. Comparison between ours and LangSplat.
Ix Low Res
Ix High Res
20x High Res
80x Low + 20 High
Rendering Time
0.0029
0.036
0.72
0.952
Noise Analysis Time
0.005
0.005
0.31
0.5
Total Time
0.0079
0,041
1.03
1,45
Table 14: Rendering Technique Speed Comparison
To further complicate things, LERF uses negative prompts to increase the relevancy of the positive prompt. To calculate the relevancy in consideration of negative prompts, one need simultancous access to the positive prompt embedding, LERF field encoded visual embedding and negative prompt embedding. From a retrieval perspective, this requires calculation between the positive query embedding with all visual embeddings (which is 4GB per image assuming 3 scales and 45GB per image assuming 30 scales). This is impossible to calculate directly. In our testing, we first select the top 10 images that have the highest cosine similarity match, and then calculated the relevancy score as specified in the LERF paper at 3 scales.
LpngSplat's s actual 3DGS process is relatively fast, after the main RGB 3DGS is trained, a secondary one that encodes the visual embeddings (in size 3) is trained on top of it. Its main overhead comes from the segmentation of the images (with SAM), and encoding each segmentation separately (CLIP), and training a per-scene autoencoder that maps the visual embedding from size 512 to 3. LangSplat is slow its rendered visual embedding (size 3) need to be decoded to size 512 with the autoencoders cross-scene comparison. This is expensive as one visual embedding is rendered for every pixels, and the autoencoders need be run many times.
10 LERF VISUALIZATIONS
As shown the left of Figure 20, the cosine similarity exhibits almost variation despite the query coffee mug should respond to the "coffee mug" in the top scene and "coffee cup" in the bottom scene. Variations in the cosine similarity only become visible in the middle column, where the min and max cosine similarity is used to create the color scale. On the right column, we demonstrate a relevancy map calculation based the 3 multi-scale features we can store on disk for lookup (instead of the 30 scales originally used in the LERF paper). As shown in the right column, the coffee mug incorrectly matched with the "coffee cup" in the incorrect scene. But most importantly, is not showing relevance with the coffee mug in the correct scene.
In summary, adapting solutions like LERF for the retrieval process is non-trivial, leading to significant storage and speed overhead while reaching low accuracy.
C ADDITIONAL VISUALIZATIONS
C.I ADDITIONAL SMART CAMERA MOVEMENT VISUALIZATION
In Figure 21, we visualize the smart camera movement allowing movement in depth for the camera origin.
In Figure 22, we visualize smart camera movement without movement depth. which what we use in the system for retrieval. As shown in the image, from viewpoint looking at the ceiling, which an uncovered area during training image sequence capture, our solution moves to a clear view of the scene in two steps.
In Figure 23, show another challenging case where the initialization point is mostly random noise. Our Smart Camera Movement efficiently moves the viewpoint to a location where the person can be clearly viewed.
29

---

Published as a conference paper at ICLR 2025
Figure 20: LERF cosine similarity and relevancy map given query coffee mug. Left: cosine similarity
visualized in scale [-1, 1]. Middle: cosine similarity scaled [min cos-sim, max cos-sim] of the individual image. Right: relevancy map calculated with the assist of negative prompts.
(a) location
(b) location
(c) location 3
(d) location
(c) location 5
location 6
(g) location
(h) location
(i) location
Figure 21:
Camera Movement Allowing Changes in Depth
C.2 SMART CAMERA MOVEMENT COMPARISON BETWEEN Low AND HIGH RESOLUTION
In Figure 24 and Figure 25, visualize the difference between low and high-resolution renderings in terms of noise analysis and smart camera movement. We observe that the contrast of clean score between clean and noisy regions is lower for the lower resolution, but there is still difference between the noisy and non non-n regions, resulting in the camera moving in the right direction. Therefore, always use low-resolution rendering during Smart Camera Movement iterations.
30

---

Published as a conference paper at ICLR 2025
(a) Step RGB
(b) Step I Clean Score
(c) Step Extended Clean Score
(d) Step 2 RGB
(e) Step 2 Clean Score
(I) Step 2 Extended Clean Score
(i) Step Extended Clean Score Figure 22: Smart Camera steps for the bouquet scene in different views: RGB, Clean Score, and Extended Clean Score.
(g) Step 3 RGB
(h) Step 3 Clean Score
C.3 FAILED RETRIEVAL VISUALIZATIONS
We demonstrated successful retrievals in the main paper. In this section, demonstrate additional retrieval failure cases. We observe that our framework generally avoids unreasonable matches. Most incorrect matches are due inherent difficulties and ambiguities in object labeling. Figure 26 visualizes these failure cases. In the first row, the prompt table from the bouquet scene matched with table in the donuts scene. This a common issue with objects that appear in multiple scenes but are labeled only once. In the second row, the lamp shade prompt retrieved the back of a camera flash, which somewhat resembles lamp shade. In the third row, the "incorrect" retrieval seems more reasonable the sheep in the groundtruth scene is very small. In the fourth row, the foam darts prompt was misunderstood by the VLM, retrieving a completely unrelated item. Even the top match from the groundtruth scene was incorrect, as the foam darts are small foam bullets for NeRF gun, as shown Figure 26(e). anticipate that our solution's ability to understand foreign objects will improve with future VLM advancements. However, additional research is needed to address ambiguities.
C.4 LLAVA CAPTION AND RETRIEVAL VISUALIZATIONS
In Figure 27, we visualize the retrieved scene (image) given LLaVA captions as queries. As shown in the figure, the retrieved image matches closely with the image used generate the caption for the success case. We visualize the failure cases in Figure 28, the failure cases show that the query is misinterpreted by the XDecoder model. Note that the LLaVA VA caption itself can also be inaccurate, for example, LLaVA referred to the donuts as orange.
C.5 RANDOM VIEWPOINT RENDERING VISUALIZATION
We visualize the random noise from pre-trained NGRs Figure 29. Notice that across very different scenes and datasets, the noise pattern is very similar (flat cloud region elongated Gaussian blobs).
31

---

Published as a conference paper at ICLR 2025
(a) Step RGB
(b) Step I Clean Score
(c) Step Extended Clean Score
(d) Step 2 RGB
(e) Step 2 Clean Score
(I) Step 2 Extended Clean Score
(i) Step Extended Clean Score Figure 23: Smart Camera steps for the handhand scene in different views: RGB, Clean Score, and Extended Clean Score.
(g) Step 3 RGB
(h) Step 3 Clean Score
Existing works evaluate novel view synthesis at viewpoints that are not far from the training image viewpoints. For example, previous studies, the quality of ScanNet novel view rendering is assessed by using every 8th image in the video sequence for evaluation, with all other images used for training. This approach results in evaluation images that are very close to the training images in terms of viewpoint (Dai et al., 2017; Lao et al., 2023; Bian et al., 2022). Nerfbusters identified this issue and proposed evaluating based on images captured from a second trajectory, different from training image trajectory. However, even with 3D diffusion modules designed to remove artifacts, rendering quality remains significantly lower than when rendering viewpoints along the training image trajectory (Warburg et al., 2023). Although the second trajectory in Nerfbusters captures different paths, still includes similar areas the scene. Our approach, using completely randomly chosen viewpoints, is markedly different from the training images.
Some NGR works utilize diffusion models for 3D object generation from text or images or to enhance the quality of 3D models given extremely sparse image captures (Liu et al., 2023d; Poole et al., 2022; Zou al., 2023b). We do not evaluate these methods, as the majority NGRs do not employ them, and these diffusion models tend to hallucinate the 3D scene rather than accurately capture it (Zou et al., 2023b; Chen et al., 2023). the datasets used by our work, each scene contains hundreds to thousands of training images. This distinguishes our training setup from those focusing sparsity, The noise artifacts we observe are due more to incomplete coverage than to training image sparsity, which we believe is an important issue for real-world use and understanding of NGRs not receiving sufficient attention (Warburg et al., 2023).
C.6
NOISE ANALYSIS AND CAMERA MOVEMENT IN SPLATFACTO AND NERFACTO COMPARISON
In Figure 30, we present additional visualizations of noise analysis and camera movement behavior for Splatfacto and Nerfacto using the noise distribution. Specifically, employ the noise Gaussian distribution trained on Splatfacto noise and evaluate it on renderings from the same viewpoints for both Splatfacto and Nerfacto.
32

---

Published as a conference paper at ICLR 2025
(a) Step
RGB
(b) Step Clean Score
(c) Step
Extended Clean Score
(d) Step 2 RGB
(c) Step 2 Clean Score
(f) Step Extended Clean Score
(i) Step Extended Clean Score Figure 24: Smart Camera steps low resolution: RGB, Clean Score, and Extended Clean Score.
(g) Step 3 RGB
(h) Step 3 Clean Score
We observe two key results. First, the noise distribution consistently assigns significantly higher clean scores to clean areas compared to noisy ones, irrespective of whether the noise originates from Splatfacto between Splatfacto noise and clean content than between Nerfacto noise and clean content. While this difference is observable, it is minor and has minimal impact the eventual camera movement.
Nerfacto. Second, the noise distribution distinguishes slightly more sharply
C.7 COMPLEX PROMPTS AND SCENE VISUALIZATION
In Figures 31, 32, 33, 34 and 35, we provide additional visualizations for different types of prompts. Specifically, Figure 31 focuses on complex prompts describing subtle relationships between objects (e.g.. "a group of." "near," " "attached to"). The query Figure (a) also incorporates an environmental style ("A garden"), ") which is distinct from the rest of the LERF scenes, highlighting the VLM's ability to generalize to diverse contexts. For the query Figure (d), we demonstrate the retrieval result when the object is partially occluded. In this case, the VLM successfully matches the text query with an unoccluded viewpoint of the object as the top choice, while an occluded viewpoint is matched as non-top choice. This demonstrates the VLM's ability to handle object occlusion some extent but also highlights the importance providing high-quality renderings from multiple viewpoints through SCMM.
Figure 32 presents visualizations for retrieving glass objects, which exhibit distinct lighting properties compared to most other materials. Figure 35 retrieves "bright sunlight", which is unique envi- ronmental style and lighting condition that applicable to outdoor scenes or scenes with windows. Figure 33 illustrates retrieval results for the prompt "A wood object," showcasing the system's specific material. Lastly, Figure 34 evaluates
capability to identify various objects composed system's 's performance with the prompt "A round object," focusing solely object shape. We note that not all retrieved results are correct- for example, some "glass-like like" objects, such as plastic cups, retrieved. These examples highlight challenges in resolving certain complex associations, which could benefit from further advancements VLMs. Retri3D, as a framework, is well well-positioned to powerful VLMS become available.
take advantage of such adyancements
33

---

Published as a conference paper at ICLR 2025
(a) Step RGB
(b) Step I Clean Score
(c) Step Extended Clean Score
(d) Step 2 RGB
(e) Step 2 Clean Score
(I) Step 2 Extended Clean Score
(i) Step Extended Clean Score Figure 25: Smart Camera steps for high resolution: RGB, Clean Score, and Extended Clean Score.
(g) Step 3 RGB
(h) Step 3 Clean Score
In summary, the VLM generally succeeds in associating complex queries with the objects described the images, effectively utilizing the renderings provided by Retri3D'ss SCMM module. However, certain cases reveal limitations, such as the misclassification visually similar objects. The resolution of these limitations could benefit from continued development in VLMs.
C.8 CLEAN SCENE SCMM AND RANDOM VIEWPOINT BEHAVIOR COMPARISON
In Figure 36, we present the blender lego scene as an example of a well-covered scene without significant floater noise artifacts. While floater artifacts are minimal due to the high-quality training image coverage, we demonstrate that SCMM still offers advantages over random viewpoint sampling. Specifically, random viewpoint sampling may occasionally generate views that are extremely close to the object or even inside Such views can cause the rendered object to appear "noise-like" because Gaussian blobs, which are typically trained external perspectives of objects, not optimized to represent these unconventional viewpoints.
SCMM mitigates this issue by guiding the camera to viewpoints that are more suitable for rendering and analysis, avoiding positions where Gaussian blobs fail to represent the scene effectively. This results in renderings that maintain higher visual quality and semantic consistency, even in scenes that are otherwise well-covered and free from floater artifacts. This example highlights how SCMM enhances the robustness of rendering and retrieval processes, particularly in challenging scenarios involving suboptimal camera positions.
34

---

Published as a conference paper at ICLR 2025
(a) table groundtruth scene
(b) table retrieved scene
(c) lamp shade groundtruth scene
(d) lamp shade retrieved scene
(e) sheep groundtruth scene
(f) sheep retrieved scene
(g) foam darts groundtruth scene
(b) foam darts retrieved scene
Figure 26: Failed retrievals from LERF dataset. Groundtruth scene containing the label Retrieved scene.
35

---

Published as a conference paper at ICLR 2025
(Success) Query: A kitchen with white piano, a table with of flowers, and chair.
(Success) Query: table with a sandwich and a cup of coffee on it, along with a box of donuts.
Figure 27: Successful retrievals with LLAVA. Left is the image that generated the LLaVA caption.
Right is the top match image from the retrieved scene. The blue mask over the entire image indicates the whole image embedding is matched.
36

---

Published as a conference paper at ICLR 2025
(Fail) Query: A room with glass door and desk with box of oranges it.
(Fail) Query: man walking through a large room with a table full of donuts and a laptop.
Figure 28: Failed retrievals with LLAVA. Left is the image that generated the LLaVA caption. Right the top match image from the retrieved scene. The blue mask over section of the image indicates the section image embedding is matched.
37

---

Published as a conference paper at ICLR 2025
Figure 29: Splatfacto Noise Visualization. One scene per row. From top to bottom: LERF scene bouquet, donuts, fruit aisle, sumnyside; ScanNet++ scene 036bce3393, 6cc223/b9c
38

---

Published as a conference paper at ICLR 2025
RGB
Clean Score
Clean Score Extended
Hand Hand GS
(a)
(b)
Hand Hand NeRF
Espresso GS
(g)
(h)
Espresso NeRF
Shoe Rack GS
(m)
(n)
Shoe Rack NeRF
(p)
(t)
Figure 30: Comparison of Noise Analysis and Camera Movement for Splatfacto and Nerfacto. We note that the location noise occurrence can be different due different models and initializations. We observe that the noise between Splatfacto and Nerfacto be similar, and more importantly, the noise Gaussian Distribution trained with Splatfacto noise can direct the camera to cleaner areas in both Splatfacto and Nerfacto.
39

---

Published as a conference paper at ICLR 2025
1st choice
2nd choice
(a) Prompt: garden with statue of bird rocks, a bamboo fence, and statue of Buddha.
1st choice
2nd choice
(b) Prompt: group of ducks sitting a pile of rocks near and a pink flower bush.
1st choice
2nd choice
(c) Prompt: metal railing attached to stone wall, and the railing covered in black paint
1st choice
5th choice
Prompt: A duck cube.
Figure 31: Visualization of complex object relations within more intricate scenes. Prompts (a. b, c) are generated by LLaVA, while prompt (d) is hand-designed. (a) Given a complex prompt describing multiple objects, the XDecoder VLM successfully identifies the associated scene. The top choice corresponds to the whole image embedding (hence the mask is associated with the entire image), while the second choice focuses on the bird object in the scene. (b) For a prompt involving complex object relations such as "group of" and "near," the VLM identifies multiple viewpoints associated with the prompt. Since the segmentation masks are created during scene analysis (prior to prompt availability), they may not perfectly align with the prompt's 's description. (c) Although the LLaVA caption's object association "attached" may not be the most suitable description, the VLM still identifies images closely related to the prompt. (d) The VLM successfully identifies the duck a cube from a cluster of objects. Its top choices highlight viewpoints where both the duck and the cube clearly visible. Additionally, we present lower-ranked choice (d, right), where the VLM identifies the combination of the duck and cube from occluded perspective. The term "cube" was dend-dei, as we found that XDecoder does not recognize "Rubik's 's Cube."
40

---

Published as a conference paper at ICLR 2025
(a) choice
(b) 2nd choice
(c) 3rd choice
(d) 4th choice
(c) 5th choice
(f) 6th choice
Figure 32: Top choices for the prompt "A glass mug, with repeated associations to the same object within the scene ignored. Strictly speaking, the 1st and 4th choices are glass mugs. However, objects similar to glass mugs, such as the glass cup in the 2nd choice and the (likely) plastic cup in the 3rd choice, are selected first as they are more easily identifiable.
(a) Bouquet
(b) Teatime
(c) "Waldo Kitchen"
(d) Figurines
(c) Sunnyside
(f) "Dozer Nerfgun Waldo"
Figure 33: Selected retrievals for the prompt A wood object" from different scenes. We observe that the VLM successfully identifies various wood objects as well as wood-like objects, such the table in "Dozer Nerfgun Waldo."
41

---

Published as a conference paper at ICLR 2025
(a) 1st choice
(b) 2nd choice
Pure
Twidders
(c) 3rd choice
(d) 4th choice
Figure 34: Retrievals in the scene "figurine" given the prompt "A round object." " The scene "figurine" is used to demonstrate the retrieval of a specific shape (round). While the scene contains multiple round objects, and the VLM successfully identifies most of them, the system also misses some round objects, such as the green apple.
(a) Sunnyside
(b) Figurines
(c) Teatime
Figure 35: The retrieval results for the prompt "Bright sunlight." " Retrieved images either contain
outdoor sky, or windowed view of outdoor sky.
42

---

Published as a conference paper at ICLR 2025
(a) Renderings blender lego object with SCMM (with translation and rotation)
(b) Renderings of blender lego object with random
Figure 36: Renderings from SCMM and random in an almost noise-free scene (nerf-synthetic blender lego). Due sufficient training image coverage, floater artifacts minimal in the lego scene. However, random viewpoint rendering may capture the object from undesirable angles, causing the object itself appear "noise-like."
43


Reviews:
[{"review_id": "EeScUGjUc4", "reviewer": "Reviewer_vhd3", "summary": "The paper introduces Retri3D, a novel framework for text-to-3D scene retrieval from repositories of neural graphics representations (NGRs). Retri3D leverages pretrained Vision-Language Models (VLMs), like CLIP, to generate embeddings of both text and rendered images, enabling efficient retrieval across a wide range of 3D scene representations, such as NeRF and 3D Gaussian Splats (3DGS). The core contributions include a Neural Graphics Noise Analysis (NGNA) and a Smart Camera Movement Module (SCMM), which collectively enhance the quality of view selection by detecting and avoiding artifacts, thus improving retrieval accuracy. The system demonstrates compatibility with multiple datasets (LERF, ScanNet++) and achieves superior retrieval speed and storage efficiency.", "questions": "1. Adding recent works as mentioned in W1 in the relevant section (related works, experiments, and/or discussions), especially [2] and [3].\n2. Adding a dedicated discussion/analysis on the model robustness as mentioned in W2 and W3.\n3. The paper presents a robust noise analysis technique, yet an interesting avenue might be replacing NGNA with NeRF uncertainty estimation methods. How would this alternative affect the accuracy, speed, and storage efficiency of Retri3D?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. Retri3D introduces a novel approach to 3D retrieval by using a two-pronged methodology: noise analysis via Neural Graphics Noise Analysis and selective viewpoint rendering through the Smart Camera Movement Module. These elements enable the retrieval of high-quality embeddings from rendered images given the typical noise in the NGRs.\n\n2. By leveraging pretrained VLMs, Retri3D achieves efficient and accurate retrieval without the need for extensive retraining or dataset-specific tuning, which is an improvement over previous methods. The noise analysis method proposed is demonstrated to outperform traditional NeRF uncertainty estimation techniques, emphasizing the robustness of the framework in achieving high-quality feature extraction under diverse conditions​.\n\n3. The paper is overall well-written and well-organized. The proposed pipeline is clean yet effective, with potential for a significant number of downstream applications.", "weaknesses": "I do not have a major concern over the paper. Addressing the following would further improve the quality of this paper:\n\n1. Comparative Analysis Limitations: While Retri3D shows strong results on LERF and ScanNet++ datasets, the paper lacks comparative results with more recent methods such as TIGER[1] or N2F2[2]. Including these would provide a broader context for Retri3D’s advancements and its relative strengths and weaknesses across a more diverse set of models and techniques​. The author should also include ConDense[3] in the related works and have a dedicated discussion/comparison, since it could also be applied to this specific task.\n\n2. Dependency on VLM Embeddings: Retri3D relies heavily on VLMs for generating text and visual embeddings, which inherently limits its performance to the capabilities of the underlying VLM model. This dependency means that Retri3D’s retrieval quality might suffer in cases where the VLM struggles with certain text-visual correlations, particularly in scenes with complex or subtle object relations.\n\n3. Potential Generalizability Issues with Scene Complexity: Although Retri3D demonstrates high performance on scenes with distinct and identifiable objects, its retrieval accuracy in complex scenes (compositional) with overlapping or occluded objects is less explored. A detailed discussion of how Retri3D would handle such scenarios or results comparing its retrieval accuracy in simple versus complex scenes would clarify its effectiveness across varying levels of scene complexity​.\n\nReferences: [1] Xu, Teng, et al. “TIGER: Text-Instructed 3D Gaussian Retrieval and Coherent Editing.” arXiv preprint arXiv:2405.14455 (2024). [2] Bhalgat, Yash, et al. “N2F2: Hierarchical Scene Understanding with Nested Neural Feature Fields.” arXiv preprint arXiv:2403.10997 (2024). [3] Zhang, Xiaoshuai, et al. \"ConDense: Consistent 2D/3D Pre-training for Dense and Sparse Features from Multi-View Images.\" ECCV 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ng4CITYhxo", "reviewer": "Reviewer_qmDY", "summary": "The authors propose a novel framework for text-based retrieval of NGR representations (NeRFs, Splats and derivatives).\nThey approach the problem by utilizing an off-the-shelf VLMs to extract feature embeddings from clean scene renders and match them with query text embeddings. Clean scene renders are obtained by iteratively applying Noise Analysis (using the same VLMs) to identify the direction towards clean scene and using smart camera movements module to converge to a cleaner render.", "questions": "1. (line 303) Major claim of the paper:\n > noise features remain consistent across different scenes and **models**\n\n  While it is clear and safe to assume noise features are of the same distribution within the same model family, it's not clear why the same holds for various models, e.g. are NeRFs and Splats features distributed the same? Maybe a mixture of gaussians suit better here? Please discuss.\n\n2. (Claim on line 288) Major claim of the paper:\n > [*...about sampling random view-points in pre-trained NGR scenes...*] Some content can be noise-free, but they constitute only small portions of the images.\n\n  I would expect a robust retrieval pipeline to work on both noisy and clean scenes. For example, how would it handle \"ground-truth\" scenes? With recent advancements in Splats, I would assume that the percentage of clean renders from random camera viewpoints could be significantly higher. As you train on increasingly clean datasets, SMCC’s performance may degrade. Please discuss how this issue could be mitigated.\n\n3. Could you clarify how cameras are initialized in the SMCC module? If they are initialized randomly, does SMCC consistently converge, or is there a risk of getting stuck in local minima?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* To my knowledge, the concept of iteratively refining camera positions to achieve cleaner renders appears to be novel.\n* The use of SMCC is well-supported by the evidence:\n  * The retrieval quality is demonstrated to be significantly higher than previous baselines or random views.\n  * Section 4.5 effectively highlights that the smart camera approach offers high coverage of the scene and the training portion of the scene\n  * Strong speed / memory benchmarks\n* It is well-proven that proposed solution is compatible with various VLMs\n* Overall, the paper is concise, and the conclusions are mainly well-supported.", "weaknesses": "1. Two major assumptions of the paper are not well-addressed (please refer to the Questions section), namely:\n* *noise features remain consistent across different scenes and models*\n* *Some content can be noise-free, but they constitute only small portions of the images.*", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "05FMTrGfUf", "reviewer": "Reviewer_eN88", "summary": "This paper aims to solve a 3D radiance field retrieval problem using text inputs and queries from the dataset constructed by multiview embedding a trained 3D radiance field. The key challenge is to obtain high-quality multiview image embeddings for the database. To solve this problem, the author proposes a noise analysis module to qualify the rendering quality of each rendering view and a camera moving module to guarantee viewpoints that render less noise images. With the powerful Visual-Language model, the method extracts rich information from a constructed database and text embedding for the retrieval task. The authors com", "questions": "1. It would be more helpful for the readers to understand why we need a retrieval task by providing detailed applications. \n2.  Why not compare with some retrieval method based on conversion 3D representation as listed in lines 45-46? It would also be possible to convert the radiance field representation like 3DGS to a simplified point cloud for this task.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The overall framework is well-motivated. The authors pay more attention to utilizing the multiview images as the data representation and propose several following designs to conduct the retrieval tasks, which is reasonable.\n2. The experiment results are extensive to cover different aspects of the proposed framework as in the main draft and supplementary.", "weaknesses": "1. The effectiveness of the smart camera moving module in Tab.1. As shown in Tab.1, using 20 training views almost outperformed every design. In some real applications, storing the radiance field with 20 training views for the following retrieval tasks would still be possible. It only introduces a tiny storage overhead while benefiting the tasks. This makes the setting only suitable if we don't have the training view. \n2. The authors propose to quantify the noise in Sec. 3.3 with a heuristic-designed viewpoint selection to determine the noise and clean features, which is counterintuitive for me. It would be better to include more details like how such random viewpoints are generated, and why the random viewpoints will contain valid noise for GMM training.\n3. The initial value of the smart camera movement module should also influence the quality. How does it initialize and if we initial from the training views, will the module converge to a bad choice in the test set?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JtmedrsEhr", "reviewer": "Reviewer_aXbV", "summary": "This paper presents Retri3D, a novel framework for retrieving 3D Neural Graphics Representations (3D NGRs) using text queries. The system leverages cosine similarity between embeddings generated by a pretrained Visual Language Model (VLM) from text queries and RGB renderings of 3D scenes. To enhance the quality of visual feature embeddings, Retri3D introduces two key techniques: Neural Field Artifact Analysis, which uses a multivariate Gaussian model to differentiate clean from noisy pixels using activation maps of VLM, and a Smart Camera Movement Module that iteratively samples new camera angles to reduce noise. Experiments show that Retri3D excels in accuracy, training time, embedding size, and retrieval performance on the LERF and ScanNet++ datasets, outperforming existing baselines.", "questions": "1. Do tightly clustered noisy features consistently affect all VLMs, and how might performance vary with different architectures?\n\n2. Are the results only applicable to indoor scenes, or does the framework also extend to complex outdoor environments?\n\n3. How are initial cameras set up for a given NGR, and what happens if the initial pose is poorly sampled?\n\n4. As the Smart Camera Movement Module seeks cleaner viewpoints, how does it ensure comprehensive coverage of the entire 3D scene? Will some important parts, such as a corner in the scene, be neglected in all the sampled viewpoints?\n\n5. What field of view (FOV) is used for rendering RGB images, and could wider FOV (such as a fisheye lens or a panorama image) improve feature capture in 3D scenes?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. This paper addresses a novel problem in retrieving 3D NGRs that previous works have not covered.\n\n2. Retri3D is the first framework capable of efficiently retrieving 3D scenes from large datasets using text queries without requiring training views or camera poses.\n\n3. The proposed Artifact Analysis effectively distinguishes clean and noisy regions in RGB renderings, while the Smart Camera Movement Module identifies cleaner viewpoints.\n\n4. Extensive experiments validate that Retri3D generates accurate embeddings and retrieves 3D scenes efficiently, utilizing moderate storage and training time compared to baseline methods.\n\nIn conclusion, the paper's contributions are significant, establishing a novel framework for 3D NGR retrieval. The introduction of two innovative modules enhances embedding quality, and the experiments comprehensively evaluate retrieval accuracy, computational efficiency, and scene coverage, clearly demonstrating advantages over baselines that integrate language features into scene representation.", "weaknesses": "1. The experiments focus on only two types of text queries (object labels and LLaVA-generated queries). Testing more complex queries—such as those describing object shapes, textures, environmental styles, materials, lighting, and specific object arrangements—could further reveal the retrieval limits.\n\n2. While the authors evaluate two NGRs (Splatfacto and Nerfacto), many other NGRs may present different artifact patterns, potentially impacting the neural graphics noise analysis. Additionally, noisy or blurred regions could be more pronounced with fewer training views, especially in few-shot reconstruction settings or if the model underfits, which may hinder the noise analysis module's effectiveness in new scenes.\n\n3. The framework's understanding of 3D scenes relies on 2D renderings, treating embeddings from different viewpoints independently. This approach may fail to answer queries about 3D-specific structures or details. Furthermore, since the current retrieval operates at a uniform resolution, querying localized small areas could result in inaccuracies due to the limitations of 2D rendering resolution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)