DistillNeRF: Perceiving 3D Scenes from Single-Glance Images by Distilling Neural Fields and Foundation Model Features

Authors: letian  wang
seung wook kim
jiawei  yang
cunjun  yu
boris  ivanovic
steven  waslander
yue  wang
sanja  fidler
marco  pavone
peter  karkus
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-10 21:09:37

Abstract: We propose DistillNeRF, a self-supervised learning framework addressing the challenge of understanding 3D environments from limited 2D observations in outdoor autonomous driving scenes. Our method is a generalizable feedforward model that predicts a rich neural scene representation from sparse, single-frame multi-view camera inputs with limited view overlap, and is trained self-supervised with differentiable rendering to reconstruct RGB, depth, or feature images. Our first insight is to exploit per-scene optimized Neural Radiance Fields (NeRFs) by generating dense depth and virtual camera targets from them, which helps our model to learn enhanced 3D geometry from sparse non-overlapping image inputs. Second, to learn a semantically rich 3D representation, we propose distilling features from pre-trained 2D foundation models, such as CLIP or DINOv2, thereby enabling various downstream tasks without the need for costly 3D human annotations. To leverage these two insights, we introduce a novel model architecture with a two-stage lift-splat-shoot encoder and a parameterized sparse hierarchical voxel representation. Experimental results on the NuScenes and Waymo NOTR datasets demonstrate that DistillNeRF significantly outperforms existing comparable state-of-the-art self-supervised methods for scene reconstruction, novel view synthesis, and depth estimation; and it allows for competitive zero-shot 3D semantic occupancy prediction, as well as open-world scene understanding through distilled foundation model features. Demos and code will be available at https://distillnerf.github.io/.

Main paper: 
 introduction: 
 Understanding and interpreting complex 3D environments from limited 2D observations is a fundamental challenge in autonomous driving and beyond. Many efforts have been made to tackle this challenge by learning from labor-intensive and costly 3D annotations, such as 3D bounding boxes [1, 2] and semantic occupancy labels [3, 4, 5, 6] . However, these approaches typically struggle with scalability due to their excessive reliance on expensive annotations.Neural scene representations, such as NeRFs [7, 8] and 3D Gaussian Splatting (3DGS) [9] , have recently emerged as a compelling paradigm for learning 3D representations from 2D signals in a selfsupervised manner. While these methods demonstrated strong capabilities in view synthesis for indoor scenes [10] , and more recently also for challenging dynamic outdoor scenes [11, 12, 13, 14] , they require delicately training a new representation for each new scene, leading to extensive computation budget and time needs, typically in the order of hours or minutes. This falls short of the real-time computational requirements for autonomous driving, which typically demand a processing speed of 2-10 Hz. Additionally, most works focus on view synthesis only [11, 12, 13] , resulting in learned 3D representations that lack semantics, sidestep downstream tasks, and do not fully exploit the potential 38th Conference on Neural Information Processing Systems (NeurIPS 2024). of neural scene representations to transform various sources of 2D information into 3D, such as features from 2D visual foundation models [15, 16] .To this end, we propose DistillNeRF (Fig. 1 ), a conceptually simple framework for learning generalizable neural scene representations in driving scenes, by distilling 1) offline optimized NeRFs for enhanced geometry and appearance, and 2) pre-trained 2D foundation models, DINOv2 [17, 16] and CLIP [15] , for enriched semantics. Generalizable scene representations with feed-forward models is an active area of research [18, 19, 20] , and the autonomous driving domain remains particularly challenging due to sparse camera views with little overlap: prior generalizable NeRFs for driving are shown to be struggling with view synthesis [21, 22, 23] . In contrast, we show that by distilling per-scene optimized NeRFs and visual foundation models, DistillNeRF allows predicting a 3D feature volume with strong geometry, appearance, and semantics, from sparse single-timestep images. The representation is capable of rendering tasks without per-scene optimization (e.g. scene reconstruction, novel view synthesis, foundation model feature prediction), and support downstream tasks, such as open-vocabulary text query and zero-shot 3D semantic occupancy prediction.DistillNeRF comprises two stages: offline per-scene NeRF training, and distillation into a generalizable model. The first stage trains a NeRF for each scene individually from each scene's driving log, exploiting all available multi-view, multi-timestep information. Specifically, we use EmerNerf [14] , a recent NeRF approach with decomposed static and dynamic fields. The second stage trains a generalizable encoder to directly lift multi-camera 2D images captured at a single timestep to a 3D continuous feature field, from which we render images, and supervise with dense depth and novel-view RGB targets generated from the per-scene optimized NeRFs, and foundation model features. Specifically, we propose a novel model architecture with 1) a two-stage Lift-Splat-Shoot encoder [24] to lift 2D observations into 3D; 2) a sparse hierarchical 3D voxel for efficient runtime and memory, parameterized to account for unbounded driving scenes; 3) feature image generation via differentiable volumetric rendering, decoded into appearance, and optionally, foundation model features.Extensive experiments on the NuScenes [25] and Waymo NOTR [26, 14] dataset demonstrate that Dis-tillNeRF allows for high-quality scene reconstruction and novel view synthesis in previously unseen environments without per-scene training, on par with test-time per-scene optimization approaches, and significantly outperforming previous generalizable approaches. We also show strong results for zero-shot 3D semantic occupancy prediction, and promising quantitative results for open-vocabulary scene understanding. 
related work: 
 Neural Scene Representations. Neural scene representations, like NeRFs [7, 27] and 3DGS [9, 28] , have brought unprecedented success in learning powerful representations of 3D scenes, and have also been successfully applied to challenging driving scenes populated with dynamic objects [29, 30, 12, 31, 32, 33, 34, 35] . However, these methods typically require expensive training for each scene, typically in the order of hours or minutes.Generalizable NeRFs. Generalizable Neural Radiance Fields, such as [18, 36, 37, 38, 39] , adapt the capabilities of conventional NeRFs for 3D scene reconstruction and novel view synthesis into a generalizable feedforward model. They replace the costly per-scene optimization with a single feedforward pass through the models. Recent works have extended such approaches to challenging driving scenes and demonstrated the potential in down-stream tasks [22, 5, 23, 40, 41] . However, due to the challenging sparse-view limited-overlapping camera settings on vehicles, these methods usually fail to show strong rendering performance. To the best of our knowledge, we are the very first work to achieve strong scene reconstruction and reasonable novel-view synthesis on par with offline per-scene optimized NeRFs in driving scenes.NeRFs with Feature Fields. Recent advancements extend NeRFs beyond novel view synthesis by integrating 2D features from foundation models into 3D space, equipping neural fields with semantic understanding [42] . Recent approaches also demonstrate similar capabilities in outdoor driving scenes [14] by distilling DINO features into the scene representation. However, these approaches suffer from prolonged optimization times when combined with feature distillation, and thus are impractical for online autonomous driving due to their costly per-scene optimization. Closely related to our work, FeatureNeRF [20] is a generalizable method that distills DINOv2 [17] features into 3D for keypoint transfer, but only investigates simple indoor object-level synthetic datasets like ShapeNet [43] , where a large number of overlapping camera images are placed around the object to scan it from various angles. In contrast, we address the more challenging outdoor scene-level settings of autonomous driving, with sparse limited-overlapping camera image inputs. Our method, DistillNeRF, infers 3D feature fields in a single forward pass, making real-time application possible.NeRFs in Driving. Most NeRF-related works in autonomous driving focus on 1) offline scene reconstruction or sensor simulation [11, 12, 14, 13] , that accurately reconstruct 3D or 4D scenes with detailed appearance and geometry; 2) exploring potential in downstream tasks [21, 23, 22, 40] , that uses volumetric rendering to learn 3D representations from sensor inputs to enlighten online driving.Our work takes the best of these two lines of research: we distill the precise geometry and novel views from offline per-scene optimized NeRFs and rich semantic features from foundation models into our online model. Consequently, our online model not only excels in scene reconstruction and novel view synthesis, but also shows competitive downstream performance, such as zero-shot semantic occupancy prediction, and open-vocabulary query. To the best of our knowledge, we are the first to do so.Distilling NeRFs. Model distillation is a well-established idea [44] . NeRFs have also been distilled into, e.g., Generative Adversarial Networks in [45] , and feed-forward models for temporal object shape prediction in [46] . However, prior work mainly focuses on static, object-centric, or indoor scenes. To the best of our knowledge, we are the first to propose distilling a per-scene optimized NeRFs with static-dynamic decomposition into a generalizable model for outdoor driving scenes. 
method: 
 DistillNeRF predicts a generalizable scene representation in the form of sparse hierarchical voxels from single-timestep multi-view RGB image inputs, and is self-supervised by natural sensor streams, through volumetric rendering to output RGB, depth, and feature images.The method is depicted in Fig. 1 , the detailed architecture in Fig. 2 , and key capabilities in Fig. 3 and Fig. 4 . Inputs are N posed RGB camera images {I i } N i=1 . We use a 2D backbone to extract N feature images {X i } N i=1 . We then lift the 2D features to a 3D voxel-based neural field V ∈ R H×W ×D×C , and apply sparse quantization and convolution to fuse features from multiple views. To account for unbounded scenes we use a parameterized neural field with fixed-scale inner voxels, and varyingscale outer voxels contracting the infinite range. Volumetric rendering is performed to supervise the reconstruction of the scene. For better guidance on scene geometry, we "distill" knowledge from offline optimized NeRFs, using rendered dense depth images from original camera views and virtual camera views. Foundation model features, from CLIP or DINOv2, are set as additional reconstruction objectives and thus are also "distilled" into our model to enrich scene semantics. We introduce design principals in the following section, and refer to Appendix A.7 for implementation details. Single-View Lifting. For each of the N camera image inputs, we follow a similar procedure as Lift-Splat-Shoot (LSS) [24] to lift the 2D image features to the 3D neural field. Unlike typical LSS and variants [24, 41, 47] that predict depth in one shot, we propose a two-stage, coarse-to-fine strategy with two jointly trained predictors to capture more nuanced depth. The first stage, following prior LSS works, predicts categorical depth and aggregates them into a single prediction with ray marching. The second stage then predicts a distribution over a fine-grained set of categorical depth values, which are centered around the first-stage predicted coarse depth. Specifically, in the first stage, we feed each image to a 2D backbone to generate a depth feature map of size H × W × D. The depth feature map is regarded as a discrete frustum where D denotes the number of pre-defined categorical depths. Inspired by the volume rendering equation [7] , each entry in the frustum is a density value. That is, the d'th channel of the frustum at pixel (h, w) represents the density value σ h,w,d of the frustum entry at (h, w, d). The occupancy weight of entry (h, w, d) is thenEQUATIONwhere δ d = t d+1 -t d is the distance between each pre-defined depth t in the frustum. Coarse depth for pixel (h, w) is obtained by aggregating with ray marching:EQUATIONIn the second stage, centered around the initial coarse depth prediction, we dynamically sample a set of D ′ fine-grained depth candidates. This involves uniform sampling, with the sampling range adaptively adjusted based on the coarse depth estimate. We then embed these fine-grained depth candidates, and combine their embeddings with the depth features from the first stage, and feed them to another network to generate the density of each fine-grained depth candidate. The occupancy weights O ′ of the fine-grained depth candidates are predicted similarly by Eq 1, which can also be regarded as probabilities of each fine-grained depth candidate.With the candidate depths associated with probabilities, we then lift 2D image features to 3D. Specifically, we use a 2D image backbone to get 2D image features ϕ, and assign the 2D image features to the 3D frustum according to each pixel's depth. That is, for pixel (h, w), its image feature ϕ h,w is distributed to each fine-grained depth candidatest ′ d by [O ′ h,w,d ϕ h,w , σ ′ h,w,d ],where we scale the pixel image feature ϕ h,w with occupancy O ′ h,w,d and concatenate it with density σ ′ h,w,d . Multi-View Fusion. After constructing the frustum for each view, we transform the frustums to the world coordinates using the camera poses, and fuse them into a shared 3D voxel-based neural field V, where each voxel represents a region in the world coordinates and carries both densities and features. When lifted frustum entries from different views lie in the same voxel, we fuse them with average pooling.Sparse Hierarchical Voxels. Unlike previous works [41, 48, 21] using dense voxels, which uniformly quantizes the neural field and potentially wastes computation and memory on large empty regions, we apply sparse quantization on the neural field. Specifically, we follow the octree representation [49] to recursively divide the neural field into specified levels, according to the 3D positions of the lifted 2D features. While an octree with many levels and thus smaller voxel sizes can capture more accurate 3D positions of lifted features, overly fine-grained octrees can lead to difficulty in querying features during rendering (e.g. missing far-away features due to large gaps between sampled rays). To this end, we generate two octrees with different quantization granularities, one fine octree with more quantization levels capturing details of the lifted features, and one coarse octree to represent general information of a larger range. Sparse convolutions [50] are then applied to both octrees to encode the relationships and interactions among voxels, during which the features in the fine octree are also downsampled and concatenated with the coarse octree to enhance details.Neural Field Parameterization. Unlike prior works that consider a neural field covering a fixed range [41, 21, 22] , our work aims at accounting for the unbounded-scene settings in the driving scenes by proposing a parameterized neural field. We want to keep the inner-range voxels at the real scale and high resolution due to their importance to various downstream tasks (e.g., occupancy prediction in 50 meters' range), while contracting the scene up to infinite distance in the outer range of the voxels at a lower resolution, so we can render with low memory and computation cost (e.g. sky, far-away buildings). Inspired by [8, 51] , we propose a transform function that maps a 3D point ). This quality can be further enhanced by applying simple color alterations to account for camera-specific coloring discrepancies (row 3). After fine-tuning (row 4), our model surpasses the offline per-scene optimized EmerNeRF, achieving higher PSNR (29.84 vs. 28.87) and SSIM (0.911 vs. 0.814). See Tab 3 for quantitative results. in the world coordinates p = (x, y, z) to the coordinates in the parameterized neural field:EQUATIONThe transformed coordinates f (p) will always be within [0, 1], where p inner sets the range of the inner voxel (region of interest) and varies in x, y, z directions, and α ∈ [0, 1] denotes the contraction ratio, namely the proportion of the inner range in the parameterized neural field. Consistent parameterizations are enforced for both the single-view lifting process (on the depth space) and the multi-view fusion process (on the 3D coordinate space).Volume Rendering from Sparse Hierarchical Voxels. Finally, we use differentiable volumetric rendering to project the 3D neural field onto 2D feature maps and render images. Specifically, for each pixel of each camera, we shoot a ray originating from the camera to the neural field according to the camera poses, and sample points along the ray. Feature Querying: For each sample point, we query both fine and coarse octree to get the density and features of the corresponding voxel that the sample point lies in. Further, to capture both high-level information and fine-grained details, the features from both octrees are concatenated as the final feature. Density Querying: Regarding the density, while the fine octree captures more accurate 3D positioning, since the fine octree voxel only covers a small region, the sample points could be easily within empty voxels and thus query no information, especially for faraway regions. To this end, for each sample point, we first query the fine octree to get the fine density. If the fine density is empty, we query the coarse octree to complement the density. Two-Stage Sampling: Regarding the sampling strategy, we follow [8] to sample points for each ray with two phases: first we sample a set of points uniformly, then we sample another set of points with importance sampling given densities for the first set of points, so to enhance surface details in the scene. With the densities and features of the sampled points we do volumetric rendering using Eq 2 to get the 2D feature map for each camera. Decoding: The rendered 2D feature maps are then fed into a CNN decoder to enhance high-frequency details, and upsample the final RGB image without increasing the rendering resolution/cost. Note that from the volume rendering process, we can also get the expected depth for each pixel [29] .Distillation from Offline NeRFs. While our model can be trained by simply reconstructing RGB images, it remains challenging to learn scene geometry from only single time-step camera image inputs. The challenge is especially pronounced with typical autonomous vehicle setups where mounted cameras are facing outwards and have limited view overlap, making multi-view recon-struction degrade to the monocular setting and aggravating depth ambiguity. A natural idea is to use images from multiple time steps to encourage view overlaps. However, driving scenes typically contain many dynamic objects that move between time steps, introducing noise to the reconstruction objective. Instead, we propose to leverage the high-quality geometry of per-scene optimized NeRFs that aggregate information from a full sensor stream. Specifically, we use EmerNeRF [14] , a recent NeRF approach that handles dynamic objects by decomposing the scene into static and dynamic fields in a self-supervised manner. We propose two different ways to distill knowledge from per-scene optimized NeRFs, which together construct L N eRF , a distillation loss from offline NeRFs:• Dense 2D depth. Depth supervision from LiDAR point clouds, L depth , is commonly used to facilitate 3D geometry learning, however, point clouds are typically sparse and only provide depth labels for a limited horizontal/vertical range. Thus we propose to use offline optimized NeRFs as a depth auto-labeling tool. Specifically, for each training target image we render a dense depth map from the offline NeRF, and use it as additional depth supervision, L depth ′ . • Virtual cameras. In addition to depth distillation from original camera views, we can leverage temporally decomposed NeRFs to render depth from "virtual cameras", i.e., novel views, while keeping the time dimension frozen. In this manner, the virtual depth and RGB images can be used as additional reconstruction targets, thus the number of target images and the view overlap between cameras can be artificially increased, encouraging consistent depth prediction and improving novel-view synthesis performance.Distillation from Foundation Models. In addition to RGB and depth prediction, can we learn 3D representations that contain rich semantics and enable a wider range of downstream tasks? Witnessing the rise of vision foundation models with generalized capabilities across various vision tasks, we propose to distill 2D foundation model features, such as CLIP [15] and DINOv2 [16] , into our 3D scene representation model. We achieve this by simply introducing an additional MLP to our rendered 2D feature images, and train the model to reconstruct the foundation model feature images with an L1 loss L f ound . We demonstrate early attempts at utilizing these foundation models on open-vocabulary text query tasks shown in Training objective. In summary, we train our model for a linear combination of loss terms:EQUATIONwhere L rgb and L depth are rendering losses for RGB and depth; L density is a density entropy loss encouraging clearer surfaces and structured density values as in [14] ; L N eRF and L f ound denote distillation losses from offline NeRFs and foundation models. Please refer to Appendix A.5 for more details on the losses. 
experiments: 
 We benchmark DistillNeRF against SOTA generalizable NeRFs, offline NeRFs, as well as comparable methods on the popular NuScenes dataset [52] . We first evaluate the rendering performance, i.e., scene reconstruction, novel view synthesis, and feature reconstruction (Table 1 ). We further evaluate the learned 3D geometry through depth estimation (Table 2 ), and 3D semantic occupancy prediction (Table 4 ). Ablations of DistillNeRF are also analyzed in each task and displayed in each Dataset. The nuScenes dataset [52] contains 1000 driving scenes from different geographic areas, each scene capturing approx. 20 seconds of driving, resulting in approx 40000 frames in total. Scenes are captured via six cameras mounted on the vehicle heading in different directions along with point clouds from LiDAR. We use the default data split, 700 scenes for training, 150 scenes for validation. We adopt the resolution of input RGB images, rendered RGB, and rendered depth are 114×228, 114×228, and 64×114 respectively. We also evaluate the generalizability of our method on the Waymo NOTR dataset [14] , a balanced and diverse benchmark derived from the Waymo Open Dataset [26] . NOTR features 120 unique, hand-picked driving sequences, split into 32 static, 32 dynamic, and 56 diverse scenes across seven challenging conditions. We adopt the resolution of input RGB images, rendered RGB, and rendered depth are 144×216, 144×216, and 72×108 respectively. Setup. We evaluate our model on previously unseen scenes from the validation set. For scene reconstruction, we compare the rendered images against GT images for the same time step. For novel-view synthesis, we render the novel-view image from the next timestep's camera pose, and compare it against the next timestep's GT image. We use standard metrics: peak signal-to-noise ratio (PSNR) and structural similarity index (SSIM). We compare with two SOTA generalizable NeRF methods in driving scenes, SelfOcc [22] and UniPAD [21] , which do not see the validation set during training, similar to ours. We also compare with the SOTA per-scene optimized method, EmerNeRFs [14] , which is trained on the validation set. Since EmerNerfs are trained on all timesteps in the scene, we cannot evaluate them for novel views. Instead, for novel-view evaluation we adapt Single-Frame EmerNeRFs, each of which is trained only on a single timestep and then evaluated for the next timestep. Due to the prohibitive training cost of Single-Frame EmerNerf on all timesteps, for all methods we report mean metrics over only the second frame of each scene.RGB Reconstruction and Novel-View Synthesis. In Table that are invisible in the original camera pose. Further, dynamic objects that move between frames act as noise in our novel-view targets. Qualitative results are in Fig. 6 . EmerNerf and our approach are close to the ground truth, while UniPAD generates blurry reconstructions with scan patterns, and SelfOcc generates grayish images and struggles to reconstruct the scene precisely. Setup. We evaluate depth up to 80m using common metrics (Abs Rel, Sq Rel, RMSE, RMSE log, and δ < t) [53, 54, 55] . We use two different depth targets: Sparse LiDAR GT, the common evaluation setting using LiDAR point cloud as ground truth, which is accurate but spare and has limited range (e.g. only 3m height); and Dense Depth GT, that uses EmerNerf to define dense depth targets with large range. We compare against the same baselines as for rendering. For UniPAD we increase the maximum range to 80m and retrain the model. For SelfOcc, we evaluate two model variants, SelfOcc* that supports depth prediction only (used in [22] ), and SelfOcc that also supports rendering (thus more similar to our method). Same as before, we evaluate over the second frame of each scene.Depth Comparison. Results in Table 2 and Fig. 6 show that while EmerNerf has superior depth accuracy by being optimized for each scene, our method outperforms prior SOTA generalizable NeRFs (SelfOcc and UniPAD). Specifically, while SelfOcc which only considers the depth prediction task shows high performance (noted as SelfOcc*), when we evaluate the model that supports both depth and rendering (noted as SelfOcc), the performance drops considerably. Looking at Fig. 6 , SelfOcc and UniPAD generate unreasonable depths for higher regions of the image, which is not reflected when evaluated against the sparse LiDAR ground truth. When evaluated on dense depth targets (Table 2b ), their performance drops, while our approach shows more consistent performance for the two sources of ground truth.Ablations. Consistently with previous results, distillation from offline NeRFs also improve depth estimation (Table 2 ). Quantitatively (Fig. 7 ), without depth distillation, we see inconsistent depth predictions between low and high regions of the image; without parameterized space, the model Setup. To evaluate the zero-shot downstream capabilities of DistillNeRF, we run evaluation on the Occ3D-nuScenes dataset [5] for 3d semantic occupancy prediction. The dataset comprises semantic occuapancy labels with 18 classes in the range [-40m, -40m, -1m, 40m, 40m, 5.4m] with voxel size 0.4m. We evaluate both binary and semantic 3d occupancy prediction. In DistillNeRF we use density thresholding (<0.001) to define whether a voxel is occupied. For semantic occupancy prediction, following [23] , we use a pre-trained open vocabulary model GroundedSAM [56, 57, 58] to generate 2D semantic masks for the input images. Then, we project the center of occupied voxels onto the 2D masks to get the semantic class, following [22] . We found that the resolution of input and output images is important for occupancy prediction with DistillNeRF, so we increased them to 400×228 and 200×114, respectively. We compare our method with SOTA self-supervised methods that do not use occupancy annotation: SimpleOcc [59] , OccNeRF [23] , and SelfOcc [22] . Following prior work, we report Intersectionover-Union (IoU) for each semantic category, mean IoU over all categories (mIoU), and geometry only IoU (G-IoU) for binary occupancy that ignores the semantic class. Additionally, we also report mean IoU over foreground categories (F-mIoU), that is, for categories excluding, drivable surface, sidewalk, terrain, other flat, and others.Results. We show the comparison of occupancy prediction in Table 4 . Our approach achieves competitive performance and excels on F-mIoU compared to the baselines, presumably because the sparse voxel representation emphasizes and better fits the foreground objects. SelfOcc (TPV) produces the highest mIoU and G-IoU, in part because it takes advantage of the fact that these metrics are dominated by ground-related classes (drive. surf., sidewalk, terrain), and it learns a prior for predicting the ground level as occupied even for non-visible regions (Fig. 4 in [22] ). Comparing ablations from DistillNeRF, we observe that distillation from offline NeRFs significantly improves performance (8.93 vs. 4.63 mIoU). 
conclusion: 
 We proposed a framework for generalizable 3D scene representation prediction from sparse multiview image inputs, using distillation from per-scene optimized NeRFs and visual foundation models. We also introduced a novel model architecture with spare hierarchical voxels. Our method achieved promising results in various downstream tasks.Our approach is not without limitations. First, we currently rely on LiDAR to train offline EmerNerfs for distillation. Second, our sparse voxel representation naturally trades off rendering efficiency for dense scene representation, and thus may not be suitable for all downstream tasks. An interesting idea would be to combine a low-resolution dense voxel with a sparse voxel, or explore respresentations similar to Gaussian Splatting instead of voxels. Finally, there are numerous exciting directions for future work, including introducing temporal input, learning static-dynamic decomposition similar to EmerNerf, and utilizing the learned rich 3D scene representation for downstream tasks, such as detection, tracking, mapping, and planning. ) Typical object-level indoor NeRF involves an "inward" multi-view setup, where numerous cameras are positioned around the object from various angles. This setup creates extensive view overlap and simplifies geometry learning. In contrast, the outdoor driving task uses an "outward" sparse-view setup, with only 6 cameras facing different directions from the car. The limited overlap between cameras significantly aggravates the ambiguity in depth/geometry learning.2) In the images captured from unbounded driving scenes, nearby objects occupy significantly more pixels than those far away, even if their physical sizes are identical. This introduces the difficulty in processing and coordinating distant/nearby objects. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We believe the key required details have been already included in the main text, but we also plan to release code to reproduce our results.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: we did not include statistical significance results due to the prohibitive cost of training both our model and baselines, and because for the majority of our baselines statistical significance results were not reported.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. Justification: The focus of this research is on scene representation and reconstruction for autonomous driving. We utilize publicly available datasets that may raise individual privacy concerns, as well as foundation models trained on diverse data, where the exact source of data is not always defined. The driving dataset used to train our model should aim to protect personal privacy, for example, by using advanced object detection technologies to anonymize faces and vehicle license plates. Additionally, a challenging aspect that requires special attention involves ensuring data diversity, including not just geographical variety, but also differences in population, architectural styles, and the timing of data collection.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: We believe our paper does not have higher risk for misuse; however, it inherits risks associated with foundation models, for which existing safeguards should be used. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: All sources and licences have been accredited to the best of our knowledge. Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: Releasing assets is not the main contribution of the paper, but we do plan to release code and trained checkpoints along with appropriate documentation.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: No experiments with crowdsourcing or human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: No experiments with crowdsourcing or human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Compared to object-centric indoor scenes, the online autonomous driving with sparse camera images poses two key challenges, which is also illustrated in Fig 5 : 1. Sparse views with limited overlap complicate depth estimation and geometry learning: Typical object-level indoor NeRF involves an "inward" multi-view setup, where numerous cameras are positioned around the object from various angles. This setup creates extensive view overlap and simplifies geometry learning. In contrast, the outdoor driving task uses an "outward" sparse-view setup, with only 6 cameras facing different directions from the car. The limited overlap between cameras increases the ambiguity in depth/geometry learning. To this end, following designs are made:• Depth distillation: depth images rendered from per-scene NeRF are used to supervise our model. These per-scene optimized depth images are high-quality, dense, and consistent spatially/temporally • Virtual camera distillation: virtual cameras are created as additional targets to artificially increase view overlaps • Two-stage Lift-Splat-Shoot strategy: proposed to capture more nuanced depth • Features from 2D pre-trained encoder: help the model learn better depth estimations and 2D image features 2. Difficulty in processing and coordinating distant/nearby objects in the unbounded scene: Unlike object-centric indoor problems, in the unbounded driving scene, the images usually contain unevenly distributed visual information based on distance: nearby objects occupy significantly more pixels than those far away, even if their physical sizes are identical. This is usually not the case in common NeRF benchmarks, and motivates multiple key designs:• Parameterized space: introduced to account for the unbounded scene, unlike Self-Occ/UniPAD with a limited range ( 50m) and losing geometry information for far-away scenes. • Density complement: far-away objects occupy pixels, and thus sampled rays could easily miss them (e.g. distant cars). Thus we propose to query densities from both fine/coarse voxel, and complement density. • Light-weight upsampling decoder: applied to rendered feature images, to 1) upsample the final RGB image without additional rendering cost; 2) enable robustness to noises in rendered features Following the discussions above, as shown in Tab 5, we ablated key components of our sparse voxel representation such as density complement, decoder, pre-trained 2D encoder, and the two-stage LSS. We remove one component each time to ablate its effect. In the last row, we also further add the depth distillation from offline NeRF, which represents the best performance of our full model.• No density complement: we observe a significant drop of PSNR from 28.01 to 22.76, demonstrating the importance of better coordination between low-level and high-level sparse voxels.• No decoder: we see a decent drop of PSNR from 28.01 to 25.76, showing the effectiveness of using the decoder for robustness to noises in rendered features.• No pre-trained 2D encoder: we see a significant drop of PSNR from 28.01 to 21.35, which is expected since using pre-trained 2D encoder has been a commonly acknowledged approach in the field, which SOTA methods such as UniPAD and SelfOcc all adopt.• No two-stage LSS: we observe a slight drop of PSNR from 28.01 to 27.40.• Add depth distillation from offline NeRF: we observe the jump of PSNR from 28.01 to 30.11.Note that every ablation above outperforms the SOTA methods UniPAD and SelfOcc, which have PSNR of 19.44 and 20.67 respectively.A.3 Discussion on Advantages of Different Ways of Lifting 2D to 3D.In this work, we employ lift-splat-shoot with convolutions to lift 2D features to 3D. In comparison, another common approach of lifting is 3D-to-2D feature projection, which is exploited in two threads of generalizable approaches: 1) image-based methods, such as GeoNeRF, IBRNet; 2) voxel-based methods, such as UniPAD, NeRF-Det.In the image-based generalizable methods, for one query point along the ray from novel views, features are extracted from the feature volume of surrounding source views via projection and interpolation. Our multi-view fusion/voxel grid approach possessed multiple advantages:• Explicit Representation: Voxel-based methods provide a direct/explicit 3D scene representation, allowing more straightforward manipulation, analysis, and understanding of spatial relationships in the scene (e.g. removing or replacing object for simulation use)• Scalability: As in our case, voxel-based methods can scale to different levels of detail by adjusting the voxel resolution. This scalability allows for efficient representation and rendering of both large scenes and fine details.In voxel-based generalizable works, a 3D voxel is created, and the feature of each voxel is generated by projecting the voxel coordinates to 2D image features and feature interpolation, where an explicit representation is offered. However, the issues with this approach include• As pointed out by NeRF-Det, this approach is equivalent to shooting a ray from the camera into the scene, and populating all voxels on the ray with the same 2D features, which could have strong depth ambiguity along the ray since all voxels on the ray have the same feature.• Besides, such 3D-to-2D projection introduces non-trivial challenges for 3D voxel grids sparsification, since every voxel is populated with features.Therefore, in this work, we choose to use the multi-view fusion approach and a lift-splat-shoot approach where• the model predicts the depth of each pixel, and only lifts 2D features to voxels around the depth of the pixel; • the model enables easy voxel sparsifications, which accelerates 3D encoding via sparse convolutions. • depth frustums are explicitly created for each view, and the distillation from Per-Scene NeRF depth would be more straightforward (applying depth loss or regularization on the frustums).As in our experiments, when comparing against UniPAD which takes the 3D-to-2D feature projection approach, our method shows better view synthesis and depth prediction performance, and offers higher inference efficiency. We hope to present our approach as an interesting and promising method that could be valuable to the community.In our paper, we distill the foundation model features into our model for enhanced semantics. The resulting model is able to create a 3D representation populated with foundation model features, and render 2D foundation model features from the 3D scene. However, to generate 2D foundation model feature images, there could be simpler baselines, e.g. rendering RGB and then feeding the RGB renderings into CLIP/DINO models. As shown in Tab 6, we compared the reconstruction accuracy and inference speed for the two approaches.As expected, given the high-quality rendered RGB images from our model, directly feeding these rendered RGB images into CLIP/DINO models generates a good original-view reconstruction accuracy (CLIP: 19.81 vs. 18.69, DINO: 21.70 vs. 18.48). However, this baseline introduces significantly higher inference latency and memory consumption due to the additional use of CLIP/DINO models. Specifically, for the CLIP model, the inference time is three times longer (1.656s vs. 0.501s, 3.3 times), and for the DINO model, the inference time is almost twice as long (0.948s vs. 0.501s, 1.89 time).Besides, we would like to emphasize that by distilling 2D foundation model features into our model, we not only enable it to render 2D foundation model feature images, but also lift 2D foundation models into 3D simultaneously. The resulting 3D voxel fields, similar to those in EmerNeRF [14] and LeRF [42] , contain rich semantic information. As demonstrated by prior works such as LeRF-ToGo [60] , ConceptFusion [61] and FeatureNeRF [20] , such 3D foundational features can greatly facilitate 3D multimodal grounding (e.g. bridging language via CLIP features) and effectively benefit downstream tasks such as segmentation, keypoint transfer, and robot planning in open world. At the end of Sec 3, we introduced the training loss of our method:EQUATIONwhere L rgb and L depth are rendering losses for RGB and depth; L density is a density entropy loss as in [14] ; L N eRF and L f ound denote distillation losses from offline NeRFs and foundation models.Here we introduce more details on each loss terms by describing their equations or pseudo code:• L rgb comprises L 1 loss and perceptual loss:EQUATION• L depth includes L 1 and MSE loss:EQUATIONwhere we normalize the absolute depth values to 0 1, and the L 1 and MSE losses are computed between ground truth depths (LiDAR points projected onto image planes) and predicted depths. • L density is a density entropy loss from EmerNeRF, which encourages the opacity of a ray to be 1: BEC(opacity, ones_like(opacity)), where opacity is the accumulated opacity per ray, and BCE is the binary cross entropy loss.• L nerf captures the distillation from per-scene NeRFs, including: 1) Dense depth distillation loss: The same depth loss equation as above, but computed with a different ground-truth, namely dense depth maps rendered from per-scene NeRFs in original camera views. 2) Novel camera view supervision: the same RGB and depth losses above, but between the per-scene NeRFs' rendered results and our online models' predictions. • L found : L 1 loss on the predicted foundation model features:EQUATIONA.6 Inference Time AnalysisIn our paper, we demonstrate that our method substantially outperforms other SOTA generalizable approaches, specifically SelfOcc and UniPad, with a reconstruction PSNR of 30.11 compared to 20.67 and 19.44, respectively. To provide a thorough evaluation, we also analyzed the inference speed of our method alongside SelfOcc and UniPad, and present the results in Tab. A.6. The table details the inference time for each component in our method: our model requires a total of 0.4867s for inference, with 0.3594s dedicated to voxel prediction from six image inputs and 0.1273s for rendering six images (achieving approximately 47 fps for a single camera). In comparison, SelfOcc and UniPad achieve total inference times of 0.1771s and 0.6514s, respectively.• As expected, SelfOcc achieves high inference speed, since it adopts an implicit representation where deformable cross-attention is used to aggregate information from the image features to generate a 3D SDF field. In comparison, our explicit voxel representation takes decent time for the 3D convolution operations, but offers additional benefits such as more straightforward manipulation (e.g. removing or replacing object for simulation use), and flexibility to scale to different levels of detail by adjusting the voxel resolution. • UniPad utilizes a voxel-based representation, making it more directly comparable to our approach. Our method shows faster inference speed, presumably due to key designs such as voxel sparsification, and the lightweight decoder that enables efficient rendering.Evaluations were conducted on the same desktop-grade system (13th Gen Intel(R) Core(TM) i7-13700KF, NVIDIA GeForce RTX 4090) and used the same image resolution (228 × 128). To ensure accurate inference time measurement, we performed CUDA synchronization to exclude the time associated with tensor transfers.The resolution of input RGB images, rendered RGB, and rendered depth are 114×228, 114×228, and 64×114 respectively. To favor downstream tasks (e.g. occupancy prediction typically considers the region of interest as [-40, -40, -1.6, 40, 40, 4.8]), in our parameterized neural field, the range of the inner voxel is 50 meters for the two horizontal directions and 6.4 meters for the vertical directions, and the proportion of the inner range α is set as 0.8. For the sparse voxel representation, we create two multi-resolution octrees with the finest levels of 7 and 9, respectively, where we employ the Kaolin library for implementation, ensuring robust and efficient handling of voxel data. [62] to generate multi-scale fused features, which are then concatenated with prior depth features from [63] as the final depth feature map for the image. Specifically, we extracted the feature before the output layer as the prior depth feature. To generate the density of each depth candidate in the second stage, we first embed the depth candidate, then concatenate it with the depth feature map from the first stage, and further embed the concatenated feature. To generate the 2D image features, we use the same strategy as in the first-stage depth prediction, namely feeding each image to feature pyramid network (FPN) [62] to generate multi-scale fused features, which are then concatenated with prior depth features from [63] .Regarding the virtual camera distillation, for each of the six cameras, we move the camera pose 1 meter away from the original camera pose in three directions (upward, leftward, and rightward), to render virtual depths/RGB images from offline NeRFs. During training, for each camera, we randomly sample one virtual view for supervision in addition to the original camera view. To facilitate CLIP/DINOv2 feature synthesis, we use the PCA matrix to reduce the feature dimension from 768 to 64. The PCA matrix is generated according to random samples from ground-truth feature images, and is applied to all data samples.We trained DistillNeRFs on 8x A100 GPUs, each with 80 GB memory, for around 4 days. We use a learning rate of 0.0002, and the Adam optimizer with an exponential decay rate of the moving averages β 1 = 0, β 2 = 0.99. We apply a gradient clip with a maximum 35 l 2 norm to stabilize training. At inference, our model takes around 1.708s for single-view encoding/lifting, and multi-view fusion, and around 0.685s for rendering RGB images, tested on a local machine with an NVIDIA TITAN RTX GPU, and an Intel(R) Core(TM) i9-10980XE CPU. Moderate training instabilities and oscillations were observed, particularly in depth prediction, which tended to fluctuate across epochs and runs. This behavior is likely due to the trade-off between geometry and rendering introduced by the sparse hierarchical design. Without depth distillation, we see inconsistent depth predictions between low and high regions of the image. Without parameterized space, the model can only predict depth in a limited depth range, while with parameterized space we can generate reasonable unbounded depth.Answer: [No] Justification: We are unable to release code at the time of submission, however, we do plan to do so shortly after.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 


Decision: The paper received four reviews from experts in the field. After the rebuttal and discussion phases, the final scores remain mixed. The rebuttal cleared up most open questions, added timings, additional ablations, and further clarifications.
Reviewer WxYD main remaining concern is that the promised changes will be included in the final version. The AC has read the paper, the reviews, the rebuttal, and the discussion and finds that the explanations provided in the rebuttal are sufficient evidence for inclusion in the paper. The author-reviewer discussion period is the exact mechanism to enable reviewers and authors to discuss issues of clarity and missing details. This involves a certain level of trust but is further encouraged by the fact that reviews, meta-reviews, and any discussion with the authors will be made public for accepted papers.

The AC thus recommends acceptance of the paper with the urge to include all necessary details to replicate the method in the text and not only in the source code.

Reviews: review: paper_summary: paper_summary: This work aims to enhance the understanding of 3D environments from limited 2D observations in autonomous driving scenarios. It achieves this by proposing a new generalizable NeRF pipeline, trained using distillation from per-scene NeRFs and foundation models. This pipeline can transform input RGB images into 3D feature volumes that can be decoded into foundation model features at inference time in a feed-forward manner. Extensive experiments validate that the proposed method outperforms the baselines across various 3D tasks.
strength_weakness: strength_weakness: 1. The new insights delivered by this work to the community are somewhat unclear. Specifically, this work combines various techniques, some of which have been partially employed in previous image-based rendering pipelines, into the proposed framework. However, it is unclear which techniques are particularly useful for the target autonomous driving scenario. If the authors aim to address the challenges specific to autonomous driving scenes, they should explicitly formulate these challenges and provide an analysis of which design is particularly useful for addressing each challenge. Although the detailed techniques may not be new, this analysis can significantly benefit the community when addressing similar scenes.

2. The differences between the components in the proposed pipeline and other image-based rendering pipelines need more clarification. Specifically, what are the key differences between the proposed method and the combination of GeoNeRF and FeatureNeRF? In my understanding, the key differences are the distillation from per-scene NeRF due to the lack of accurate depth and the adoption of multi-view fusion.

3. As a follow-up to point 2, I wonder what the advantages of adopting multi-view fusion plus a voxel grid are compared to 3D-to-2D feature projection for each sampled point along the ray in previous generalizable NeRFs.

4. Although this work mentions the real-time processing demand, the efficiency aspect of the proposed pipeline is not measured or analyzed.

5. I wonder whether the proposed method can be applied to common NeRF benchmarks, in addition to NuScenes, while still achieving leading generalizable reconstruction performance compared to other generalizable NeRF variants like IBRNet, GNT, and GeoNeRF.
questions: questions: My questions have been included in the weakness section. I'm willing to adjust my scores if my concerns are properly addressed.
limitations: limitations: This work does not suffer from notable negative societal impacts.
paper_summary: paper_summary: Appears to present a method for online 2D feature distillation in an autonomous driving configuration. Method appears to use some kind of pre-trained depth prediction network to build a frustum aligned grid of 2D features, which are somehow rasterized into a canonical sparse volumetric grid. There appears to be some use of networks ("embedding") to do inference on these features, which allow for 3D semantic segmentation and occupancy prediction. 

Method presents novel view synthesis, depth prediction and occupancy prediction metrics on the nuScenes dataset.
strength_weakness: strength_weakness: The papers greatest weakeness is the lack of enough detail descrbing the method to reproduce it, and therefore to be able to properly critique it.  Distilling 2D features into 3D via Neural Radiance Fields is not a new idea (“LERF” , and “FeatureNeRF" are cited.  “Decomposing NeRF for Editing via Feature Field Distillation”, NeurIPS 2020 is not cited), yet is given too much attention. The real contribution should be as to how this is done in real-time, but critical details as to how 2D features are lifted into 3D via depth predictions, are missing, how view frustums are combined into a canonnical 3D grid are missing, and how features are combined and "embedded" (suggesting the use of a neural network) missing. (See "Questions below for further deails"). I would have thought the use of sparse grids would more easily allow unbounded spatial discretization and the MipNeRF-360 style space contraction unnecessary ("Neural Field Parameterization"). This design choice appears to be unmotivated and un-ablated.

Precise (non-verbal) definitions of all terms in the final loss Eq (4). appear to be missing.

Method is presented as an online method, but provides no metrics indicating real-time performance.
questions: questions: #124 “depth feature map is further embedded”  - how do you perform this embedding? This would suggest you are using a neural network of some sort.

#127 “To this end, we first aggregate the frustum to predict a raw depth for each pixel, and then sample fine-grained depth candidates centered around the raw depth. Both depths are trained end-to-end with our model.” You provide equations for compositing (1) and expected depth (2) (which are relatively trivial and could probably be omitted), but exactly how you do the remainder appears to be missing.

#130 “the frustum is designed to contain the density value”. This would be the place to describe this design.

#136 “The depth feature map” It is unclear what you mean by “depth features”. Are intermediate features used to predict depth?  Bucketized depth probabilities? This might reference details that are buried inside “Depth Anything”, but not apparent on a cursory read of it.

#145 Unclear what you mean by “sparse quantization”. Quantization usually means discretizing the floating point value to a finite set of values. Is this what you mean? Also, what spatial interpolation and sampling technique are you using to sample from the frustum to the global grid?

#151 What do you mean by “two octrees with different quantization granularities,” An octree is by definition multi-resolution. Are you simply using a sparse grid? Do you have two sparse grids? If using an octree, please describe at least how many levels are in it. Further saying you apply “sparse convolutions” makes me think you actually mean sparse grid, because a sparse multi-resolution convolution on an octree is exceedingly difficult to implement.
limitations: limitations: Briefly mentions limitations in the final paragraph of "Conclusions". Though they state #334 "sparse voxel representation naturally trades off rendering efficiency for dense scene representation", no actual performance metrics appear to be given.

Societal impact is properly addressed.
paper_summary: paper_summary: This paper presents a method for 3d understanding from 2d observations for autonomous driving. The main technical contribution is a feed-forward model, which is trained by distilling RGB and depth from a per-scene optimized NeRF model. The proposed model predicts 3d feature volumes that enable volumetric rendering similar to NeRFs. Experimental results on NuScenes show improved performance compared to recent comparable generalizable methods.
strength_weakness: strength_weakness: - One of the main reasons for having this generalizable formulation instead of just using e.g. EmerNeRF is that it is faster, yet inference time is not discussed in the paper. Can the authors report this in the rebuttal? For Table 1, 2 and 3 inference time should be reported for all methods.
- The ablations are unclear. What exactly is meant by “Ours” in Table 1, 2 and 3? Is it the model without depth distillation, param space and visual cam distillation? Why is not the model that combines all components like “Ours (+ Depth Distillation + Param Space + Virtual Cam Distillation)” tested? Since “Ours” performs worse than “Ours (+ X)” it looks like it is not tested. Why not test “Ours (full)” and e.g. “Ours (- Depth Distillation)”, more like a standard ablation study?
- The part about distilling foundation models is incomplete. There are some simple baselines missing, e.g. rendering RGB and then computing CLIP/DINO from the RGB renderings. How much slower is it and how does the reconstruction accuracy compare? It is also unclear why the foundation model reconstruction is only reported for the model variant “Ours (+ Depth Distillation)” in Table 1 and not the others.
questions: questions: Please see weaknesses
limitations: limitations: This is adequately addressed
paper_summary: paper_summary: The paper introduces a self-supervised learning framework named DistillNeRF for understanding 3D environments from limited 2D observations. This framework is designed for autonomous driving and leverages per-scene optimized Neural Radiance Fields (NeRFs) and features distilled from pre-trained 2D foundation models such as CLIP and DINO. The model predicts rich 3D feature volumes from single-frame multi-view camera inputs, supporting various downstream tasks like scene reconstruction, novel view synthesis, and zero-shot 3D semantic occupancy prediction. Experimental results on the NuScenes dataset demonstrate the effectiveness of DistillNeRF over existing methods.
strength_weakness: strength_weakness: 1. Computational Complexity and Inference Speed: It would be beneficial for the authors to include the training time for both per-scene EmerNeRF and DistillNeRF, as well as the inference speed of DistillNeRF. This will help to better assess the efficiency of the proposed method.
2. Results on Other Datasets: Currently, the experiments are conducted only on the nuScenes dataset, whereas EmerNeRF has been tested on the Waymo dataset as well. Including experiments on the Waymo dataset would strengthen the claims regarding the generalizability of the method.
questions: questions: See the "Weakness" section.
1. Since the nerf-based method requires extra training time, is it possible to extend this method to support 3DGS distillation?
limitations: limitations: See the "Weaknesses" section. Additionally, please note that the final score for this study is not solely determined by the peer reviewers' discussion. If the authors can address my main concerns, I would be willing to raise the score.

Comments: title: Response to reviewer's comment
Response to reviewer's comment
Update on the requested ablation studies
Reference in our response
Response to Reviewer WxYD
Detailed comparison of our DistillNeRF with GeoNeRF and FeatureNeRF
Settings for the under-training ablation studies.
comment: Dear reviewer, thanks for the reply. 

We want to clarify, please do not get us wrong, our response is fully respectful to the reviewer and the reviewer's efforts, just like the word literally means "respect/appreciate", "kindly", and just as how we respond to reviewer's questions one by one in detail during the rebuttal stage, such as explaining the definition of feature maps, embeddings, voxel quantizations, write down the mathematical definition of l1 loss to address the reviewer’s question, and introduce the common practice for writing and code sharing in the community. Apologies if any of these comments look improper from some angles, that is not what we mean.

We will incorporate the clarifications into our paper, just as we will incorporate every other reviewer's suggestions into our paper, and sincerely appreciate every reviewer's great suggestions/perspectives to improve our paper and contribution to the conference and community. Specifically, for the losses, we reassure the reviewer that we will update our paper to include them, just as how we responded to you in detail during our rebuttal. 

Finally, we thank the reviewer again for the precious time and effort in reviewing our work, and engaging in the discussion stage.
I thank the authors for their response, but not their combative tone ("It’s a pity to hear that you do not...", "but we respect/appreciate your unique angle...", "We kindly request you carefully read our response, and make informed comments"). 

There may be a cognitive bias to apply a perception of laziness and mis-understanding to the reader, but I would suggest that a more productive tone would be to ask if what your are intending to communicate is actually what you are communicating?

As a meta-example, you state "[you] will incorporate these details into the paper". Yet closer to your response on loss terms you further state: "Due to the character limit, we briefly introduce them here, and refer to the code for full details." This reads as a) an excuse as to why these details weren't initially included, and b) an indication that the source code will be the documentation for these losses. There is no statement that you intend to include these loss terms in the paper. The wording "brielfy introduce them here" is ambiguous. What does "here" mean? The rebuttal? or the paper? A clearer response would have been: "We agree the loss equations are essential details originally omitted due to space constraints. We will amend our submission to include the terms listed below". 

Ultimately, you state "we firmly believe that our original submission has already described the method in sufficient detail, that is already up to the standards of the field." Which I disagree with (with all due respect to my peer reviewers who may feel the opposite). I do not hear the authors stating unambigiously they intend to clarify their presentation.
I thank the authors for providing answers to all my questions. I keep my rating as weak accept.
Dear reviewer, first we would like to extend our sincere gratitude for taking the time to read through the feedback and engage in the discussion for our paper. We greatly value your precious time/efforts that you've dedicated to contributing to the conference and the broader community.

We would like to clarify the point mentioned by R-WxYD, who stated that “this feels like writing the paper after the publication deadline”. However, as mentioned in our new response to R-WxYD, we are currently at the rebuttal stage rather than the publishing stage. We firmly believe, “reviewers provide constructive comments and authors improve the paper accordingly”, is one key reason to set up the rebuttal stage, and one important part of the review cycle where reviewers and authors work together to present greater works and contributions to the community. We kindly refer you to the NeurIPS reviewer guidelines, which emphasize the importance of this stage.

Since you are taking reviewer WxYD's comments into your assessment, we would also gently suggest you read our responses to R-WxYD, as it could provide additional contexts that could be helpful to your evaluation. 

Thank you again for your time and thoughtful consideration.
Dear reviewer, may we first extend heartfelt gratitude for spending the time reviewing and engaging in the discussion stage, it never goes unappreciated. 

As promised, now we are happy to present the ablation studies requested, as in the table below.

| Density complement | Decoder | Pretrained encoder | Two-stage LSS | Depth Distillation | PSNR  | SSIM  |
|:------------------:|:-------:|:------------------:|:-------------:|:------------------:|:-----:|:-----:|
| ✗                  | ✓       | ✓                  | ✓             | ✓                  | 22.76 | 0.669 |
| ✓                  | ✗       | ✓                  | ✓             | ✓                  | 25.34 | 0.839 |
| ✓                  | ✓       | ✗                  | ✓             | ✓                  | 21.35 | 0.536 |
| ✓                  | ✓       | ✓                  | ✗             | ✓                  | 27.40 | 0.859 |
| ✓                  | ✓       | ✓                  | ✓             | ✗                  | 28.01 | 0.872 |
| ✓                  | ✓       | ✓                  | ✓             | ✓                  | 30.11 | 0.917 |

Specifically, we ablated key components of our sparse voxel representation such as density complement, decoder, pre-trained 2D encoder, and the two-stage LSS. We remove one component each time to ablate its effect. In the last row, we also further add the depth distillation from offline NeRF, which represents the best performance of our full model. 
- No density complement: we observe a significant drop of PSNR from 28.01 to 22.76, demonstrating the importance of better coordination between low-level and high-level sparse voxels.
- No decoder: we see a decent drop of PSNR from 28.01 to 25.76, showing the effectiveness of using the decoder for robustness to noises in rendered features.
- No pre-trained 2D encoder: we see a significant drop of PSNR from 28.01 to 21.35, which is expected since using pre-trained 2D encoder has been a commonly acknowledged approach in the field, which SOTA methods such as UniPAD and SelfOcc all adopt.
- No two-stage LSS: we observe a slight drop of PSNR from 28.01 to 27.40.
- Add depth distillation from offline NeRF: we observe the jump of PSNR from 28.01 to 30.11. 

Note that every ablation above outperforms the SOTA methods UniPAD and SelfOcc, which have PSNR of 19.44 and 20.67 respectively. Among the above techniques, 1) density complement, 2) rendering decoder, 3) two-stage LSS, and 4) distillation from offline NeRF are the novel and unique methods proposed by our paper. With these ablations and analysis for each technique, we believe the key message and contribution of our paper to the community are much clearer.

Finally, the authors want to take a moment to acknowledge your strong expertise and foundations in the field. Your comments are among the deepest and the most constructive, which effectively helps us improve our paper. We acknowledge your precious time spent on the review and discussion.
[1] Zhang, Chris, Wenjie Luo, and Raquel Urtasun. "Efficient convolutions for real-time semantic segmentation of 3d point clouds." 2018 International Conference on 3D Vision (3DV). IEEE, 2018.

[2] Qian, R., Garg, D., Wang, Y., You, Y., Belongie, S., Hariharan, B., Campbell, M., Weinberger, K.Q. and Chao, W.L., 2020. End-to-end pseudo-lidar for image-based 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 5881-5890).

[3] Huang, Lila, et al. "Octsqueeze: Octree-structured entropy model for lidar compression." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.

[4] Chen, Xiaozhi, et al. "Multi-view 3d object detection network for autonomous driving." Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2017.

[5] Ku, Jason, et al. "Joint 3d proposal generation and object detection from view aggregation." 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2018.

[6] Liang, Ming, et al. "Deep continuous fusion for multi-sensor 3d object detection." Proceedings of the European conference on computer vision (ECCV). 2018.

[7] Yang, Bin, Wenjie Luo, and Raquel Urtasun. "Pixor: Real-time 3d object detection from point clouds." Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. 2018.

[8] Zhou, Yin, and Oncel Tuzel. "Voxelnet: End-to-end learning for point cloud based 3d object detection." Proceedings of the IEEE conference on computer vision and pattern recognition. 2018.

[9] Yu, A., Ye, V., Tancik, M. and Kanazawa, A., 2021. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4578-4587).

[10] Charatan, D., Li, S.L., Tagliasacchi, A. and Sitzmann, V., 2024. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19457-19467).

[11] Wang, Q., Wang, Z., Genova, K., Srinivasan, P.P., Zhou, H., Barron, J.T., Martin-Brualla, R., Snavely, N. and Funkhouser, T., 2021. Ibrnet: Learning multi-view image-based rendering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4690-4699).
We thank you for the prompt response.

> Q1: "lack of enough detail descrbing the method to reproduce it"...
- First, to clarify, we firmly believe that our original submission has already described the method in sufficient detail, that is already up to the standards of the field. This is also reflected by the fact that, other reviewers unanimously rated the presentation as “3 good”, and explicitly mentioned “The paper is well-structured and clearly explains the methodology, including detailed descriptions of the model architecture, training process, and experiments” and “The proposed method exhibits good soundness, with the technical details of each component well elaborated”. 
- In such contexts, you raised questions such as the definitions of feature maps and embeddings. We firmly believe these are preliminary knowledge in deep learning/vision, but we respect/appreciate your unique angle, and were happy to respond to them one by one in the rebuttal phase.
- Third, the reviewer suggested including every fine-to-the-ground detail of the paper, for which we even wrote down the mathematical equation of L1 loss. This is not actionable: today’s AI fields are moving very fast and have grown to a level of complexity that every paper would rely on previous foundations. It has been a common writing paradigm for papers to 1) mostly emphasize the core novelty of the paper, since that’s the key message delivered to the community; 2) leave the commonly used techniques or implementation details to references, appendix, supplementary material, and ultimately the source code. We provide code not as a replacement for a clear description, but as an additional resource to aid easy reproducibility, which is common practice in the field. For example, all SOTA methods compared in our paper follow this paradigm (EmerNeRF, UniPAD, SelfOcc, OccNeRF, SimpleOcc)
- Based on the points above, we firmly believe that the reviewer’s concerns of clarification and reproducibility are already well addressed.

> does not sound like you intend to include these losses in the main text.
- As in our rebuttal, we explicitly said “will incorporate these details into the paper”. We kindly request you carefully read our response, and make informed comments.

> I appreciate the extra descriptions of the method. But these are all descriptions that should have been included in the paper. This feels like writing the paper after the publication deadline. As do the extra timing ablations that should have been included in the paper.
- Note that we are at the rebuttal stage instead of the publishing stage. We firmly believe, “reviewers provide constructive comments and authors improve the paper accordingly”, is one key reason to set up the rebuttal stage, and one important part of the process where reviewers and authors work together to present greater works and contributions to the community. Please also see the NeurIPS reviewer guidelines. It’s a pity to hear that you do not recognize the responses and improvements made during the rebuttal, even if they well address your concerns. 

> "... [we] ... also offline-trained NeRFs" I'm unclear what you are trying to say. Are you trying to say that you distilled features offline?
- This is the key/core method of our paper, namely distillation from offline-trained NeRFs, which is clearly indicated by the title of our paper (DistillNeRF), and extensively elaborated/evaluated throughout the paper (Intro/abs, Fig.1, Sec 3.2, Table 1/2/3). Again, we kindly request you read our paper carefully and make informed comments.

> I have always seen "quantization" refer to the quantization of floating point values, and not the sparsity structure of the grid. Can you provide a reference that actually uses this term the same way?
- In 3D vision, voxel quantization, or sometimes called voxelization, is a basic operation widely used [1,2,3,4,5,6,7,8]. [1] introduced “Commonly, point clouds are first quantized in a process known as voxelization, with the resulting voxel grid being used as input to 3D CNNs”. Also, see Sec.3.1 in [2] for a whole section of descriptions. Fig.4 in [2] and Fig.1 in [3] also show excellent illustrations for dense and sparse quantization, respectively.

> I also have a hard time accepting 2fps at a very low resolution of 228x128 as a "realtime" or "online" method
- First, we use the term “online model” in line with the literature, referring to reconstructing the scene with one model forward pass [9,10,11]. This contrasts with “offline" NeRFs, which obtain a single scene representation through dedicated optimization. Our method renders 6 images in 0.486 seconds, compared to an offline NeRF like EmerNeRF, which takes 1.5–2.5 hours to achieve the same.
- Second, we need to point out that our model generates 6 images with 0.486s, showing 12fps instead of 2fps. The pure rendering takes 0.127s with 47fps. Again, please read our response carefully and make informed comments.
Thank you to the authors for providing the detailed response. i will keep my rating and listen to our reviewers' opinions.
Thanks to the author for providing the rebuttal. However, I agree with Reviewer WxYD that this work needs to be improved by including the materials in the rebuttal. Considering this point, I tend to lower my rating.
I thank the authors for their rebuttal. 

I remain very negative over the "lack of enough detail descrbing the method to reproduce it". I appreciate the authors inclusion of the source code, but there's no reason a well written Methods section _couldn't_ have included include all salient details so that I wouldn't _have to_ look at the source code. A paper should do the work of digesting the method for the reader instead of forcing them to go a level deeper and dissect the source code. Source code inclusion should only be for reproducibility purposes, and not as the primary documentation of the method. Furthermore, the authors rebuttal of "Due to the character limit, we briefly introduce them here, and refer to the code for full details." does not sound like you intend to include these losses in the main text.

I appreciate the extra descriptions of the method. But these are all descriptions that should have been included in the paper. This feels like writing the paper after the publication deadline. As do the extra timing ablations that should have been included in the paper.

I'm holding up the terminology of "sparse quantization" as unclear writing. I have always seen "quantization" refer to the quantization of floating point values, and not the sparsity structure of the grid. Can you provide a reference that actually uses this term the same way? 

I also have a hard time accepting 2fps at a very low resolution of 228x128 as a "realtime" or "online" method 

"... [we] ... also offline-trained NeRFs" I'm unclear what you are trying to say. Are you trying to say that you distilled features offline? Again, the language is still unclear.

I appreciate the authors rebuttal but I maintain my rating as "reject".
|                | Reconstruction  | Offline NeRF Distillation | Multi-View Fusion | Multi-view inputs | Found. Model Lifting | Downstream Task |
|:--------------:|:---------------:|:-------------------------:|:-----------------:|:------------------:|:--------------------:|:---------------:|
|    GeoNeRF     |  Object level   |            ✗              |         ✗         |         ✓          |          ✗           |        ✗        |
|  FeatureNeRF   |  Object level   |            ✗              |         ✗         |         ✗          |          ✓           |        ✓        |
|     Ours       |  Scene level    |            ✓              |         ✓         |         ✓          |          ✓           |        ✓        |
| Two-stage LSS | Pretrained encoder | Density complement | Decoder | Metrics |
|:-------------:|:------------------:|:------------------:|:-------:|:-------:|
|       ✗       |         ✓          |         ✓          |    ✓    |         |
|       ✓       |         ✗          |         ✓          |    ✓    |         |
|       ✓       |         ✓          |         ✗          |    ✓    |         |
|       ✓       |         ✓          |         ✓          |    ✗    |         |
|       ✓       |         ✓          |         ✓          |    ✓    |         |