To View Transform or Not to View Transform: NeRF-based Pre-training Perspective
Authors: Hyeonjun Jeong
Juyeb Shin
Dongsuk Kum
Venue: ICLR 2026 Poster (2026)
Keywords: Autonomous driving
Pre-training
Self-supervised learning
Neural radiance fields
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: NeRP3D: NeRF-Resembled Point-based 3D detector for effective NeRF pre-training and continuous representation

Abstract: Neural radiance fields (NeRFs) have emerged as a prominent pre-training paradigm for vision-centric autonomous driving, which enhances 3D geometry and appearance understanding in a fully self-supervised manner. To apply NeRF-based pre-training to 3D perception models, recent approaches have simply applied NeRFs to volumetric features obtained from view transformation. However, coupling NeRFs with view transformation inherits conflicting priors; view transformation imposes discrete and rigid representations, whereas radiance fields assume continuous and adaptive functions. When these opposing assumptions are forced into a single pipeline, the misalignment surfaces as blurry and ambiguous 3D representations that ultimately limit 3D scene understanding. Moreover, the NeRF network for pre-training is discarded during downstream tasks, resulting in inefficient utilization of enhanced 3D representations through NeRF. In this paper, we propose a novel NeRF-Resembled Point-based 3D detector that can learn continuous 3D representation and thus avoid the misaligned priors from view transformation. NeRP3D preserves the pre-trained NeRF network regardless of the tasks, inheriting the principle of continuous 3D representation learning and leading to greater potentials for both scene reconstruction and detection tasks. Experiments on nuScenes dataset demonstrate that our proposed approach significantly improves previous state-of-the-art methods, outperforming not only pretext scene reconstruction tasks but also downstream detection tasks.

TO VIEW TRANSFORM OR NOT TO VIEW TRANSFORM: NERF-BASED PRE- PRAIIING PERSPECTIVE
Hyeonjun Jeong, Juyeb Shin, Dongsuk Kum KAIST, Daejeon, Korea
(hyeonjun, jeong, juyebshin, dskum @kaist
ABSTRACT
Neural radiance fields (NeRFs) have emerged a prominent pre-training paradigm for
autonomous driving, which enhances 3D geometry and appear- ance understanding a fully self-supervised manner. To apply NeRF-based pre- training to 3D perception models, recent approaches have simply applied NeRFs to volumetric features obtained from view transformation. However, coupling NeRFs with view transformation inherits conflicting priors; view transformation imposes discrete and rigid representations, whereas radiance fields assume continuous and adaptive functions. When these opposing assumptions are forced into single pipeline, the misalignment surfaces as blurry and ambiguous 3D representations that ultimately limit 3D scene understanding. Moreover, the NeRF network for pre-training is discarded during downstream tasks, resulting in inefficient utiliza- tion of enhanced 3D representations through NeRF. In this paper, propose novel NeRF-Resembled Point-based 3D detector that can learn continuous 3D representation and thus avoid the misaligned priors from view transformation. NeRP3D preserves the pre-trained NeRF network regardless of the tasks, inheriting the principle of continuous 3D representation learning and leading to greater poten- tials for both scene reconstruction and detection tasks. Experiments on nuScenes dataset demonstrate that our proposed approach significantly improves previous state-of-the-arr methods, outperforming not only pretext scene reconstruction tasks but also downstream detection tasks.
INTRODUCTION
Accurate and fine-grained 3D scene understanding is essential for autonomous driving, supporting
critical tasks such as 3D object detection Reading et al. (202I): Li et al. (2023 2024). high-definition (HD) map construction Liao
al. (023);; Shhin et al. (202)), and occupancy prediction Tong
et al.
y ;
Tian et al. (2023). To facilitate these worndworld perceptions, view transformation
backbones Li al. (2023 2024 2022) have drawn great attention, which project multi-view 2D
image features into unified 3D representation on bird' s-yee-veew (BEV) or voxel space. A unified
3D representation, aligning various modalities [Lu al. (2023b); [i et al. (2022) Yan al. (2023)
Kim et al. (2023) in common metric frame, provides a single le 3D canvas that can be leveraged across
diverse downstream tasks Hu et al. (2033) Jiang et (2023) : Weng al. (2024).
In parallel, neural fields, such as NeRFs Mildenhall et al. 1202 and 3DGS Kerbl et al. 2023), have emerged dominant paradigm for reconstructing 3D representation and synthesizing novel views by learning continuous field of color and volume density in suff-super manner. Sharing the goal of understanding the 3D environment, recent studies Yang et al. (2024); Huang al. (2024); Xu et al. (2024) proposed combining NeRFs or 3DGS with view transformation, enabling self-supervised pre-training through photometric and depth reconstruction without the need for expensive manual annotations.
Although both view transformation and NeRFs ultimately aim to reconstruct a 3D representation of the world from 2D signals, they embody conflicting priors. Existing approaches Yang et al. (202)): Huang et al. (2024) extract point features for radiance fields by interpolating discretized and fixed voxel features from view transformation backbone, and then pre-train the backbone through photometric and depth errors rendered from those point features. However, this pipeline inevitably leads to NeRF inheriting the discrete and rigid priors of the view transformation, which conflicts

---

GT RGB
UniPAD
SelfOcc
NeRP3D
DINO
Figure 1: Comparison of 2D feature maps (left) and their instance segmentation (right) results using SAM Kirillov et al. (023);; Ren et al. (0024); Ravi et al. (2024) across different methods. All 2D feature maps, except for ground truth RGB (row 1) and DINO et al. (2021): Oquab ct al. (2023) feature (row 5), are obtained by accumulating 3D point- wise representations along each ray onto the image plane with predicted density. They extracted directly after radiance field pre-training without any task-specific fine-tuning. UniPAD Yang et al.] (2024) (row 2) and SelfOcc Huang et al. 2024) (row 3) produce blurry and inaccurate features that fail separate nearby or crowded objects, resulting under-segmented instances. In contrast, NeRP3D (row 4) produces precise and well-localized features with distinct object boundaries without any distillation or fine-tuning from 2D foundation models, comparable to those from DINO features. Consequently, we observe the potential for the enhancement of 3D representation be reflected in the improved instance segmentation quality.
with the continuous radiance fields and restricts the fidelity of the reconstructed 3D representation. Moreover, the pre-trained NeRF discarded during downstream tasks, preventing effective transfer of NeRF knowledge and limiting the exploitation of enhanced 3D representations from pre-training. As result, distinct objects can be collapsed into single blurry blob, as shown in Fig.
In this paper, we introduce NeRP3D, novel NeRF-Resembled Point-based 3D detector that fully inherits the continuous function of neural radiance fields Mildenhall et al. (202 Wang et al. (2021) )
effectively overcoming the inherent discrepancy with view transformation. Unlike methods rel relying rigidly discretized voxel-based representations, NeRP3D directly models 3D scenes as continuous 3DD features, geometry, and appearance from any continuous 3DD location in feedforward manner,
2

---

Discretized Vovel Feat.
Pts for NeRF
View Transform
Pre Processingr
NeRF
Rendering Head
NeRP3D
Rendering Head
NeRF Discarded after Pre-training
Images
Perception
Pts Query
Perception
pre-training
Head
fine fine-tuning
Continuous Points
NeRF-based Pre-training with View Transform
NeRf-Resembled Point-based Detector
Figure 2: Comparison of the previous NeRF-based pre- pre-training methods and our NeRP3D pipeline.
illustrated in Fig. Experiments on the nuScenes Caesar et al. (0200) benchmark demonstrate that our approach significantl improves not only the rendcring quality but also the downstream perception tasks for autonomous driving compared to previous approaches that simply incorporate NeRF-basee pre- pre-training into view transformation frameworks. These findings highlight the importance of aligning the 3D backbone with the pre- trainini model as well as continuous 3D representation learning in advancing NeRF-based pre- traitrain for enhanced 3D scene understanding.
In summary, our contributions are:
NeRP3D preserves the full knowledge from pre-training, since the NeRF-resembled design makes it effectively inherit and utilize continuous and fine-grained representations for both pretext and downstream tasks.
Regardless of tasks, NeRP3D provides unified framework allowing for consistent feature extraction with adaptive sampling, ray-wise and uniform spatial sampling, available through our proposed continuous function.
2 RELATED WORK
Neural Radiance Fields Neural radiance fields (NeRFs) Mildenhall et al. (2021) and their vari- ants Wang et al. (002); Fridovich-Keil et al. (2022); Müller et al. (2022); Barron et al. (2022) 2023) have established powerful paradigm for 3D scene reconstruction by learning continuous volumetric functions from posed multi-view images. NeRFs are typically trained in a self-supervieed manner, minimizing photometric reconstruction loss across multiple views. These prior works have demonstrated their ability to understand and enhance fine 3D geometry and appearance through high-fidelity novel view synthesis and 3D reconstruction. To move from dense toward sparse image sets, conditioning the radiance fields with image features Yu et al. (2021): [Chen et al. (202)222)))) Liu et al. (20226) shows reliable novel view synthesis results, demonstrating that generic 2D representations guide NeRF training. Moreover, depth supervision Roessle et al. (2022); Wei et al. (2023a); Deng et al. (002);; Wei et al. (202T) is incorporated to understand more accurate geometry. NeRF's enhanced 3D understanding is increasingly being extended to autonomous driving applications, and NeRP3D aims to fully leverage these capabilities.
Neural Radiance Fields with Autonomous Driving The inherent ability of NeRFs Mildenhhl et al. (202I): Wang et al. (2021): Barron al. (2022) 2023) to capture 3D scene structure from multi-view 2D observations in self-supervised manner has positioned them as a promising foundation for various autonomous driving applications. For sensor simulation in driving environments, offline scene reconstruction methods Yang et al. (2023c); Tonderski et al. (2024); Yang et al. (202b)) have demonstrated NeRFs' capability synthesize realistic camera images, generate scenarios through object manipulation, and decompose static-dynamic scenes. Moreover, DistilINeRF Wang al. (2024) builds upon EmerNeRF Yang al. (2023b) by extending into feed-forward model, while feature distillation from 2D foundation models Radford et al. (2021): Oquab et al. (2023) further enhances 3D scene understanding.
The most relevant branch of this paper is the integration of NeRFs in pre-training to improve down- stream perception tasks. UniPAD Yang et al. (2024) introduces a universal NeRF-based pre-traue pre-training framework to enhance the 3D object detection downstream task. Occupancy predictions Huang et al.) (2024); Zhang et al. (2023) are also integrated with NeRF, which is optimized through multi-view
consistency Godard al. 2017 2019) Zhou al. (2017). GaussianPretrain Xu al. (024) has

---

RGB
SDF
Det
Map
Oco
0
Pretrain
Cross Attention
Finetune
Multi-view Images
Positional Encoder
O O
Ray-wise Sampling
Spatial Sampling
Figure 3: Overview NeRP3D, illustrating both pre-training for rendering (orange) and fine-tuning for downstream (blue) pipelines. Through NeRF-resembled design, our method maintains a coherent 3D understanding from scattered points across diverse tasks while accommodating task-specific point sampling strategies, enabling the model to effectively leverage underlying geometric and appearance information while allowing for task-dependent feature specialization.
demonstrated the feasibility of 3D Gaussian Splatting Kerbl et al. 20233 for pre-training 3D scene representations in driving environments. However, cxisting methods Yang et al. (2024): Huang et al. (2024) Tian et al. (2023) Xu et al. (2024), which rely on view transformation. have inherent con- straints that diminish NeRF's capacity for continuous and fine-grained 3D representation. Moreover, pre-trained NeRF is discarded during downstream tasks, resulting in suboptimal 3D representations enhancement from pre-training. In contrast, NeRP3D fully inherits pre-trained NeRF knowledge and utilizes continuous and fine-grained representations through its NeRF-resembled design.
METHOD
NeRP3D is simple and effective NeRF-resembled architecture that unifies scene reconstruction and perception tasks from ainle-timestey multi-view images. As illustrated in Fig. 3 our framework operates in two distinct stages within unified architecture, without discarding adding modules depending on stage or task requirements. This unified architecture enables adaptive exploration of regions of interest tailored specific processing efficiency, while maintaining coherent 3D>
understanding across diverse tasks.
3.1 ADAPTIVE SAMPLING & REPRESENTATION OF POINT
To reconstruct accurate 3D representations from sparse and dynamic multi-view inputs, NeRP3D directly samples 3D points of interest at arbitrary spatial locations and predicts the representation of sampled points with 2D image features to cope with dynamic driving scenes, without processing voxelized feature grids or any interpolation from them.
NeRP3D first samples 3D points x E I&' using one of two distinct strategies tailored to different processing phases, view-dependent ray-wise sampling and uniform spatial sampling. For volumetric rendering, we follow the standard NeRF Specifically, for each pixel in the multi-view images, we define camera ray r, based on its origin o, and direction d; which are derived from camera intrinsics and extrinsics. Along each ray, we sample set of points (x, o; t,d,) at regular or stratified distances within a defined range [tyli = 1, ..., D,t, < tt+1). These sampled points are then integrated into rendered color and depth along the ray for differentiable volumetric rendering. In contrast, for downstream tasks, where the goal is to utilize the learned 3D representation for autonomous driving tasks such 3D object detection occupancy prediction, sample points across the scene volume rather than following camera rays. We sample points Xayz uniformly in 3D space around the vehicle, covering regions relevant to perception tasks.
4

---

Despite the difference in sampling methods, all 3D points, whether sampled along camera rays or spatially, are represented in the identical system, ensuring consistency across tasks and sharing a unified spatial understanding. In addition, we parameterize 3D coordinates to account for unbounded environments, inspired by |Brrron et al. 2202);
p(x') = { ax'
X x > 1
S 1
(1- (1-
where p(-) denotes parameterized function that preserves real-scale coordinates for points within the inner range, while distributing distant points proportionally to disparity, including those at infinite distance. x' denotes normalized x to the range [0. 1] and a E [0, 1] denotes the contraction ratio.
After sampling 3D points. a set of 3I) points {x} is conditioned with sparse 2D observations to represent 3D dynamic environments in feed-forward manner. Given multi-view images {Te)==>> we feed each image the image backbone to obtain 2D image features F R-NxHxWxC Then, to enhance 3D point representations with image- -aigged context, we adopt a deformable cross- attention Zhu al. (2020) with 2D image features F. We first encode each 3D query point x by 7(-) and learn set of N₂ sampling offsets {AT, I ()), focusing interaction with relevant image regions. The final representation z of 3D point defined as:
1, ..... N₂) relative to its projected 2D location
N
N, WR
2
W. W ATh,()(p((()1))
(2)
where N₂ denotes the number of heads for multi-head attention. Wh E (CX(C/NN) and W RCC/N))xC) denotes learnable weights and Ab. denotes the attention weights which are normalized as Σ. A The resulting point embedding serves as input to both rendering heads and detection heads described in the following sections.
3.2 POINT-BASED 3D SCENE RECONSTRUCTION & PERCEPTION
Volumetric Rendering To support 3D scene understanding for downstream tasks autonomous driving, first optimize radiance fields in self self-suuer manner Yang et al. (2024), using the signed distance function (SDF) and RGB reconstruction to represent 3D geometry and appearance. Given a of sampled points along each ray and its embedded features (2z)) RGB color values of 3D points Xy are predicted by cj = (z,, d,), and its signed distance sy extracted by signed distance function oodf (z,) transformed into opacity a, derived with:
@ (Geaf (zg) ( (Zj (2g-1)) .0).
aj max
L ( (z,))
where (x) = (1 + e - is the sigmoid function with a learnable parameter w. Then, the unbiased and occlusion-awee aware weights Wang et al. (2021) wy TJa is computed from aj, where T, = - -ax) ) is the accumulated transmittance. The final color and depth values are computed by accumulating the contributions of 3D points sampled along ray r,, weighted by the probability distribution {w
D C(r,) = Emnjo myjj D(r:) = Σ w5li f=1
D
1=1
where Č(r,) and D(r,) denote the predicted color and depth corresponding to the ray r,, respectively.
To optimize the neural radiance field, we employ a combination of RGB reconstruction, depth supervision, and multi-view consistency losses. We adopt the standard volumetric rendering loss from NeRFs, comparing the rendered color C(r:) against the ground truth pixel color C(r,) for sampled {r:). To further constrain the 3D geometry, we leverage explicit depth supervision Deng et al. 2022 Yang al. 22024) for r; against LiDAR measurements Dridrr (r:) where available. Furthermore, while LiDAR provides direct supervision, it suffers from sparse scan patterns and cannot capture regions such as the sky, transparent surfaces (e.g., windows), distant backgrounds where depth is undefined or unprojectable. To address this without additional annotations Yang et al. (2023b) or distillation from 2D foundation models Oquab et al. (2023): Kirillov et al. (2023). further
rays R-
5

---

enforce multi-view consistency Godard al. (2019); Cao & De Charette (2023) by minimizing the discrepancy in predicted depth distributions across different views as:
1
Creproj
Σ EERRYERR
wylT(())) (xx(x))))).
[R]
where T (1;) denotes the color value of a pixel in target or current image I; corresponding to the ray Fy. (x) denotes the projection matrix from 3D points to 2D pixels on a source image I,, such as + t,d, along the
previous 1--1 future image 1++1. Consequently, the sampled 3D point x, ray r, is projected on the source image, and the corresponding pixel color (T (x,) ) compared with I (r,)) in weighted sum {w, ). The overall loss for pre-training consists of RGB reconstruction loss, depth supervision loss, and reprojection loss:
Ppretrain Arghhegb + Adepth Areproj freproj
where Argb, Adepth and Areproj are the loss scale factors for each pre-training loss. Ligb is RGB reconstruction loss and Leepth is depth estimation loss directly supervised by LiDAR measurements.
Open-Workee Perception Unlike view-dependent volumetric rendering, perception tasks require comprehensive spatial coverage of the vehicle's surroundings. All need to do with NeRP3D scatter the points {x) IR Nx3 throughout the space and reshape the resulting representations {z) R NxC from Eq. to be compatible with task-specific detection heads, for example, {2} R(XxYxZ)xC for occupancy prediction. This straightforward adaptation maintains the enhanced geometric and appearance information learned during pre-training while enabling seamless integration with established perception architectures.
EXPERIMENTS
We demonstrate NeRP3D on the nuScenese Caesar al. (2020) dataset against the state-of oo-thhe-aaet- the-art NeRF-based pre-training approaches as well as comparable methods. Our experiments are designed to assess both pre-trained 3D representations by scene reconstruction and the effectiveness of finetuning for downstream tasks.
4.1 DATASET
We conduct experiments using the nuScenes dataset Caesar et al. (2020), which provides 700, 150, and 150 scenes for training, validation, and testing, respectively. We follow this data split for both the pretext and downstream tasks. Each scene provides 6 RGB camera images that cover a full 360 field of view, along with 32-beam LiDAR point cloud and 3D radar point cloud data. The key samples are annotated at 2 Hz and support multiple tasks for autonomous driving, including 3D object detection, HD map construction, and segmentation. Recently, the annotations for occupancy prediction have been made available through Occ3D Tian al. (2023) and SurroundOce Wei et al. (2023b), providing dense 3D semantic occupancy labels. In our experiments, we adopt the Oce3D benchmark for the occupancy prediction.
Moreover, to evaluate generalization across different data distributions and sensor configurations, additionally utilize Argoverse 2 (AV2) Wilson al. (2023) dataset. AV2 provides 1,000 driving sequences with distinct sensor suite comprising seven high-resolution ring cameras (2048 x 1550) covering 360 field of view and two 32-bean LiDARs. This setup introduces significant domain shifts environmental statistics and sensor layouts compared to nuScenes Caesar et al. (2020). This distinct setup serves to assess the model's robustness domain changes and its data efficiency under limited supervision. For our experiments, we resized the input images to 800 x 450 and utilized only 1/4 subset of the training data.
4.2 EVALUATION METRICS
We evaluate performance across two pretext scene reconstruction tasks and three downstream 3D perception tasks by following standard evaluation protocols for each task to ensure comparability with existing methods.

---

Table 1: Downstream detection performance
(a) 3D object detection
(b) Occ prediction
Method
Pre-train
NDSt
mAPt
Method
mloU
UVTR-C
ImageNet
44.1
37.2
BEVDet
19.38
BEVFormerV2
ImageNet
46.7
39.6
BEVFormer
26.88
FoVrerr
27.83
TPVFormer!
SelfOce
33.5
31.0
CTF-Occ
28.53
UVTR-CT
UniPAD
37.1
33.7
NeRP3D
Ours
39.2
35.8
SelfOce
29.65
UniPAD
34.05
UVTR-C
UniPAD
45.5
41.6
NeRP3D
35.49
NeRP3D
Ours
47.3
42.8
(c) HD map construction
Method
Pre-train
Epochs
mAP
HDMapNet
ImageNet
30
23,0
VectorMapNet
ImageNet
110
40,9
MapTR-tiny
ImageNet
49,9
TPVFrreer
SelfOcc UniPAD
24
53.9
UVTR-C
24
57.8
NeRP3D
Ours
59.1
Scene Reconstruction Tasks We compare scene reconstruction quality by generating rendered RGB and depth images 1:2 the size of the input images. RGB reconstructed images are evaluated for all rendered pixels by Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS), following standard NeRF evaluation protocols. For depth estimation, we report relative errors (AbsRel & SqRel), root mean squared error (RMSE& & RMSE log), and accuracy under threshold 8 metrics up 80m, only for pixels where the lidar point cloud with 20 sweeps is projected.
Downstream Tasks We evaluate the performance of 3D object detection using the mean Average Precision (mAP) and nuScenes Detection Score (NDS) under the standard nuScenes evaluation protocol. The perception range for object detection set to |-1..2m, 51.2m) along both the X and Y axes. For vectorized HD map construction, and report mAP under Chamfer distance thresholds (r E (0.5, 1.0, 1.5)). The evaluation range is to 15.0m, 15.0m] for the X axis and [-30.0m, 30.0m] for the Y axis. Occupancy prediction aims to predict the semantic classes of 0.4m x 0.4m x 0.4m voxels covering [-40m, 40m] in both the X and Y axes and [-100m, 5.4m] along the Z axis. The prediction result evaluated using mean Intersection over Union (mloU) across 17 semantic classes.
follow the nuScenes map annotation benchmark
4.3 IMPLEMENTATION DETAILS
To ensure fair comparison with prior works Yang et al. (004)) Huang et al. (2024), we adopt identical
pre-training architectures and detection heads. We leverage NeuS Wang al. (2021) for radiance field
pre-training, following previous studies. Furthermore, we conduct downstream tasks based on UVTR- Ci al. (2022). MapTR Liao al. (2023), and Occ3D (CTF-Oce) Tian al. (023) for 3D object detection, HD map construction, and occupancy prediction, respectively. Class bala-cealceed sampling (CBGS) or specialized data augmentations are not applied for finetuning, and all downstream tasks are trained and evaluated using
images only, without temporal information or frame
stacking.
Our implementation builds upon the MMDettection3DIConnnnee conducted on 4 NVIDIA A6000 GPUs. The input image resolution varies by tasks, set to 1600 x 900 for object detection and 800 x 450 for rendering, HD map construction, and occupancy prediction. We both pre- train and fine-tune the model for 24 epochs using the Adam optimizer, with initial learning rate of 2e-4 and weight decay of 0.01. The loss scale factors set to Argb Adepth = Apeprej = 10. Unless otherwise specified, we fine-tune the models on a 1/2 subset for 12 epochs with 800 x 450 images in ablation studies.
(2000) framework, and training

---

GT RGB
UniPAD
SelfOcc
NeRD3D
Figure 4: Qualitative comparison rendered RGB & depth. NeRP3D outperforms state-of of-the-ar methods on both RGB and depth reconstruction. Our approach maintains high fidelity in urban scenes without any blur and pattern artifacts. For depth estimation, NeRP3D distinguishes individual people in crowded areas rather than merging them into indistinct blobs, and precisely captures thin structures such poles that are often missed or reconstructed thick structures by competing methods.
Table Pretext scene reconstruction performance
(a) Depth estimation
(b) RGB reconstruction
Method
Abs Rell
Sq Rell
RMSEJ
RMSE logl
Method
PSNR1
SSIM
LPIPS
SelfOce
0,311
3.808
8.503
0.391
SelfOce
18,82
0.536
0.657
SelfOce*
0.215
2.743
6.706
0.316
UniPAD
21.14
0.549
0.634
UniPAD
0.218
2.512
7.937
0.356
NeRP3D
33.42
0.969
0.070
NeRP3D
0.183
2.274
7.884
0.353
4.4
MAIN RESULTS
3D Object Detection
We compare NeRP3D with previous 3D object detection approaches Li et al. 2022 Liu et al. (202aa) Shu et al.
2023 ; Yang et al. (202a): Yan et al. (2023) the nuScenes val set. To compare with previous NeRF based pre-training methods on detection, we follow the fine-tuning framework of UniPAD Yang al. (2024) and also reproduce the results of both UVTR-C (UniPAD) Li (202)); Yang et al. (2024) and TPVFormer (SelfOcc) Huang et al. 20023
(2044
2024) by replacing the NeRF network for pre-training with UVTR's object detection head. in Tab (a) denotes the result evaluated on input resolutions of 800 x 450. Compared to the state-of-the-ar
-eRe- self-supervision methods, our method outperforms 1.8 mAP and 2.1 NDS 800 450 1.2 mAP and 1.8 NDS on 1600 x 900 over UniPAD, shown in Tab. I (a). This improvement stems from NeRP3D's s ability learn fine- seee-grrined 3D representations, which enables more precise localization of bounding boxes and better separation of nearby objects, as qualitatively suggested by detailed features Fig. and sharp reconstructions Fig.
Occupancy Prediction In Tab. (b), we evaluate the performance of our method on Oce3D- nuScenes for 3D occupancy prediction. Similar to 3D object detection, fine-tune the backbones with the same occupancy prediction head Tian et al. (2023) after pre- mre-trainnng. Our approach inherits FsRF's strength in modeling fine-grained representations, leading to improved mloU and consistent
Tian
8

---

Table 3: Zero-shot scene reconstruction performance (Argoverse 2 - nuScenes)
Method
Abs Rell
Sq Rell
RMSEL
RMSE log!
PSNRT
SSIMI
LPIPS!
UniPAD 0.985
11.767
14.963
4.390
18.665
0.432
0.577
NeRP3D
0.626
6.251
10.728
0.921
28.238
0.905
0.111
gains over UniPAD and SelfOcc. As result, our NeRP3D outperforms UniPAD and SelfOcc by 2.8 and 9.2 mloUs, respectively. The continuous and high-fidelity representations learned by NeRP3D are particularly beneficial for this dense prediction task, enabling the model to accurately discern object boundaries and capture intricate geometric details often missed by other methods.
HD Map Reconstruction We evaluate the accuracy of HD map construction to assess each method's capability for understanding static driving environments, particularly in detecting road boundaries, dividers, and pedestrian crossings. To facilitate this task, we commonly utilized the detection head of MapTR Liao et al. (2023) for fair comparison. As shown in Tab. (c), our method achieves improved mAP compared to both UniPAD and SelfOec, with gains of I.3 and 5.2 mAP, respectively. HD map reconstruction is particularly challenging as it requires a nuanced semantic understanding differentiate map elements like pedestrian crossings that geometrically coplanar with the drivable surface. As visually evidenced in Fig. [ the feature representations from NeRP3I) make these elements distinctly separable, which is critical for precise map construction.
RGB & Depth Reconstruction To validate the effectiveness of the pre-training, the performance of NeRP3D on the pretext tasks is also compared with the previous NeRF-based pre-training meth- ods Yang et al. (2024): Huang et al. (2024) on the nuScenes val set. As shown in Tab. 2 NeRP3I) achieves remarkable enhancements in both depth estimation and RGB reconstruction. More specif- ically, the qualitative depth maps in Fig. consistently demonstrate that our method yields more accurate and detailed depth estimations, particularly in complex regions, whereas UniPAD and Self- Oce struggle to resolve fine structures and depth discontinuities. For RGB reconstruction, UniPAD generates blurry and imprecise reconstructions lacking detailed textures, while SelfOce produces grayish images with unidentified vertical patterns. In contrast, our approach reconstructs sharper images with rich colors, closely matching the ground truth without introducing patterned signals.
Generalization To assess the robustness of our method against domain shifts and varying sensor configurations, we conducted cross-dataset transfer experiments using Argoverse (AV2) Wilson al. 2023) for pre-training and nuScenes for evaluation. AV2 possesses distinct camera geometries and environmental statistics compared to nuScenes, serving as a rigorous testbed for generalization.
| We first evaluated zero-shot scene reconstruction by directly applying the AV2 Wilson et al. (2023)
pre- pre-trained weights to nuScenes Caesar et al. (2020) without any fine-tuning. As presented in Tab. 3 NeRP3D demonstrates remarkable robustness. achieving an Abs Rel of 0.626 and PSNR of 28. 28.24, significantly outperforming UniPAD (Abs Rel 0.985, PSNR 18.67). While the view transformation method Yang al. (2024) suffers from severe degradation due to its dependency aligned with specific sensor layouts. NeRP3D's s continuous point-based architecture effectively adapts new sensor geometries. Qualitative results in Fig. of Appendix further visualize this, showing that NeRP3D preserves structural details while the view transformation method produces blurry artifacts.
fixed grid priors
Moreover, we evaluated the transferability for 3D object detection. When pre-trained on AV2 Wilson al. 2022) and fine-tuned on nuScenes Caesar et al. (2020). NeRP3D achieved 27.46 mAP, surpass- ing UniPAD Yang et al. (2024) (26.29 mAP) by significant margin. This confirms that NeRP3D learns universal geometric representations that are not overfitted to specific sensor configurations or dataset distributions but are effectively transferable across domains.
Overall, these results demonstrate that our approach effectively leverages the inherent advantages of continuous and fine-grained representations derived from NeRF. NeRP3D not only significantly benefits pretext scene reconstruction tasks and downstream detection tasks but also ensures robust generalization across different sensor configurations and data distributions for autonomous driving. More comprehensive comparison and quantitative analysis of the experimental results are provided by Tab. 8 in Appendix A and

---

4.5
ABLATION STUDIES
We conduct comprehensive ablation studies to analyze different model variants and evaluate their impact. Ablation results are reported in Appendix C and summarized in the following sections.
Cross- Cross-Task Generalization We further investigate whether the learned 3D representation remains valid across different task objectives. By performing volumetric rendering using the backbone fine-tuned for occupancy prediction, observe that NeRP3D successfully retains structural details, whereas view-transformation methods suffer from catastrophic forgetting, collapsing into mean regression. This confirms that NeRP3D learns task-agnostic continuous representation that preserves geometric fidelity regardless of downstream optimization pressure.
Adaptability View transformation is dependent on the range and voxel size, leading to severe performance degradation if the voxel-related parameters are changed against pre-training. In contrast, NeRP3D aims for a continuous representation without voxel-related parameters, and variations only correspond simple changes in the range of interest.
Effectivness We analyze the effectiveness NeRP3D reducing the reliance on annotations by comparing previous works, ranging from the full dataset to a 1/8 subset. Consequently, NeRP3D maintains strong detection performance even with significantly reduced supervision, indicating the robustness of its NeRF-based pre-training.
Multi-view Consistency LiDAR-based supervision ensures consistent depth estimation accuracy. However, we found that the sparsity and sean patterns of LiDAR are ultimately insufficient for reconstructing dense 3D geometry. To address LiDAR's sparsity and patterns, only rely on LiDAR supervision but also consider multi-view consistency and our sampling strategy tailored to this approach.
Design Validation We verify the necessity of our architectural choices through comprehensive comparisons. First, applying NeRF pre-training to existing point-based detectors Liu et al. (2023a) fails to transfer knowledge due to query mismatch, confirming the importance of our unified design. Second, comparisons between SDF (NeuS) Wang al. (2021) and density (standard NeRFMiildenhn al. (202T) priors validate that SDF enforees clearer object boundaries beneficial for perception. Finally, demonstrate that deformable attention outperforms standard attention by providing necessary locality inductive bias, ensuring that the points remain faithful to their local spatial context.
CONCLUSION
In this paper, we present NeRP3D,, novel point-based 3D architecture for scene reconstruction and downstream perception tasks for autonomous driving. Our approach addresses the fundamental misalignment between view transformation and neural radiance fields. Through its NeRF-resembled design, NeRP3D fully inherits NeRF's continuous representation capabilities, enabling the model to maintain consistent geometric and appearance information arbitrary spatial locations across both scene reconstruction and open-world perceptions. Although NeRP3D outperforms previous approaches, struggles with depth beyond its ROI, relying LiDAR. Additionally, its point-based architecture incurs high computational costs from adapting NeRF's output to existing detection heads. Future enhancements include temporal RGB reconstruction for consistency, density/opacity filtering for efficiency, and Gaussian splatting for real-time performance with point queries.
ACKNOWLEDGEMENT
This work supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government (MSIT) and the Ministry of Trade, Industry and Resources (MOTIR) of the Republic of Korea. (2022R1 A2C200494414, RS-2025-25448249)
10

---

REFERENCES
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5470-5479, 2022.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19697-19705, 2023.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1621-11631, 2020.
Anh-Quan Cao and Raoul De Charette. Scenerf: Self-supervised monocular 3d scene reconstruction with radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9387-9398, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings the IEEF/CVF international conference on computer vision, pp. 9650-9660, 2021.
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEF/CVF international conference computer vision, pp. 14124-14133, 2021.
MMDetection3D Contributors. MMDetectionn3D: Open next-generation platform for general 3D object detection. https: com/ open wwt.a//aammeeeeeeeeiioooo.o.. 2020.
Kangle Deng, Andrew Liu, Jun-Yan Zhu, and Deva Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12882-12891, 2022.
Sara Sridovich-Keil. Alex Yu. Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5501-5510, 2022.
Clément Godard, Oisin Mac Aodha, and Gabriel J Brostow. Unsupervised monocular depth estimation with left-right consistency. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 270-279,40 2017. 2017.
Clément Godard, Oisin Mae Aodha, Michael Firman, and Gabriel J Brostow. Digging into self- supervised monocular depth estimation. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3828-3838 38388, 2019.
Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 17853-17862, 2023.
Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri-perspective view for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9223-9232, 2023.
Yuanhui Huang, Wenzhao Zheng, Borui Zhang, Jie Zhou, and Jiwen Lu. Selfocc: Self-supervised vision-basee 3d occupancy prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19946-19956 2024.
Bo Jiang, Shaoyu Chen, Qing Xu, Bencheng Liao, Jiajie Chen, Helong Zhou, Qian Zhang, Wenyu Liu, Chang Huang, and Xinggang Wang. Vad: Vectorized scene representation for efficient autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8350-8350, 2023.
11

---

Bernhard Kerbl, Georgios Kopanas, Thomas Leimktihler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(():139-1, 2023.
Youngseok Kim, Juyeb Shin, Sanmin Kim, In-Jae Lee, Jun Won Choi, and Dongsuk Kum. Crn: Camera radar net for accurate, robust, efficient 3d perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 17615-17626, 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yea Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference computer vision, pp. 4015 4026-4026, 2023.
Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, and Jiaya Jia. Unifying voxel-based representation with transformer for 3d object detection. Advances in Neural Information Processing Systems, 35:18442-115 18455, 2022.
Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. In Proceedings of the AAAI conference on artificial intelligence, volume 37, pp. 1477-1485, 2023.
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng
Dai. Bevformer: learning eeye---we representation from camera via spatiotemporal transformers. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
Bencheng Liao, Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Wenyu Liu, and Chang Huang. Maptr: Structured modeling and learning for online vectorized hd map construction. In International Conference on Learning Representations, 2023.
Yingfei Liu, Tiancai Wang, Xiangyu Zhang. and Jian Sun. Petr: Position embedding transformation for vulti-view 3d object detection. In European conference on computer vision, pp. 531-548. Springer, 2022a.
Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, and Xiangyu Zhang. Petrv2: A unified framework for 3d perception from multi-camera images. In Proceedings of the IEFE/CVF International Conference on Computer Vision, pp. 3262-3272, 2023a.
Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang. Neural rays for occlusion-aware image-based rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7824-7833, 2022b.
Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang. Huizi Mao, Daniela I. Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's-eye view representation. In 2023 IEEE international conference on robotics and automation (ICRA), pp. 2774-2781. IEEE, 2023b.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG), 41(4): 1-15, 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXtv:2304.07193, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp- 8748-8763. PmLR, 2021.
Nikhila Ravi, Valentin Gabeur, Yuan-Tine Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint 2xxi:2448.007144 2024.
12

---

Cody Reading, Ali Harakeh, Julia Chae, and Steven L Waslander. Categorical depth distribution network for monocular 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8555-8564, 2021.
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint arXiv:2401. 1491. 2024.
Barbara Roessle, Jonathan T Barron, Ben Mildenhall, Pratul Srinivasan, and Matthias NieBner. Dense depth priors for neural radiance fields from sparse input views. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12892-12901, 2022.
Juyeb Shin, Hyeonjun Jeong, Francois Rameau, and Dongsuk Kum. Instagram: Instance-level graph modeling for vectorized hd map learning, IEEE Transactions on Intelligent Transportation Systems, 2025.
Changyong Shu, Jiajun Deng, Fisher Yu, and Yifan Liu. 3dppe: 3d point positional encoding for transformer-based multi-camera 3d object detection, Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 3580-3589, 2023.
Xiaoyu Tian, Tao Jiang, Longfei Yun, Yucheng Mao, Huitong Yang, Yue Wang, Yilun Wang, and Hang Zhao. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. Advances in Neural Information Processing Systems, 11664318-43335 2023.
Adam Tonderski, Carl Lindström, Georg Hess, William Ljungbergh, Lennart Svensson, and Christof- fer Petersson. Neurad: Neural rendering for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14895-1 14904, 2024.
Wenwen Tong, Chonghao Sima, Tai Wang. Li Chen, Silei Wu, Hanming Deng, Yi Gu, Lewei Lu, Ping Luo, Dahua Lin, et al. Scene as occupancy. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 8406-8411 2023.
Letian Wang, Seung Wook Kim, Jiawei Yang, Cunjun Yu, Boris Ivanovic, Steven Waslander, Yue Wang, Sanja Fidler, Marco Pavone, and Peter Karkus. Distillnerf: Perceiving 3d scenes from single-glance images by distilling neural fields and foundation model features. Advances in Neural Information Processing Systems, 37:62334 62361, 2024.
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv:2106. 10689, 2021.
Yi Wei, Shaohui Liu, Yongming Rao, Wang Zhao, Jiwen Lu, and Jie Zhou. Nerfingmvs: Guided optimization of neural radiance fields for indoor multi-view stereo. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 5610-5619, 2021.
Yi Wei, Shaohui Liu, Jie Zhou, and Jiwen Lu. Depth-guided optimization of neural radiance fields for indoor multi-view stereo. IEEE Transactions Pattern Analysis and Machine Intelligence, 45 (9):10835-1084, 2023a.
Yi Wei, Linqing Zhao, Wenzhao Zheng, Zheng Zhu, Jie Zhou, and Jiwen Lu. Surroundoce: Multi- camera 3d occupancy prediction for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21729-21740, 2023b.
Xinshuo Weng, Boris Ivanovic, Yan Wang, Yue Wang, and Marco Pavone. Para-drive: Parallelized architecture for real-time autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15449-15458, 2024.
Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting. arXiv preprint arXiv:2301.00499, 2023.
13

---

Shaoqing Xu, Fang Li, Shengyin Jiang, Ziying Song. Li Liu, and Zhi-xin Yang. Gaussianpretrain: A simple unified gaussian representation for visual pre-training in autonomous driving. arXiv preprint arXiv:2411.1222 2024.
Junjie Yan, Yingfei Liu, Jianjian Sun, Fan Jia, Shuailin Li, Tiancai Wang, and Xiangyu Zhang, Cross modal transformer: Towards fast and robust 3d object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 18268-18277 2023.
Chenyu Yang. Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang. Hongyang Li, Yu Qiao, Lewei Lu, al. Bevformer v2: Adapting modern image backbones to bird's s-eye-view recognition via perspective supervision. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 17830- 17839, 2023a.
Honghui Yang, Sha Zhang, Di Huang, Xiaoyang Wu, Haoyi Zhu, Tong He, Shixiang Tang, Heng- shuang Zhao, Qibo Qiu, Binbin Lin, et al. Unipad: A universal pre-training paradigm for autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 15238-15255 2024.
Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng. Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, et al. Emernerf: Emergent spatial-temporal scene decomposition via self-supervision. arXiv preprini arXiv:2311.0000 2023b.
Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1389-1399, 2023c.
Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4578 4587, 2021.
Chubin Zhang, Juncheng Yan, Yi Wei, Jiaxin Li, Li Liu, Yansong Tang, Yueqi Duan, and Jiwen Lu. Occnerf: Self-supervised multi-camera occupancy prediction with neural radiance fields. CoRR, 2023.
Haiming Zhang, Wending Zhou, Yiyao Zhu, Xu Yan, Jiantao Gao, Dongfeng Bai, Yingjie Cai, Bingbing Liu, Shuguang Cui, and Zhen Li. Visionpad: A vision-centric pre-training paradigm for autonomous driving. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 17165-17175, 2025.
Tinghui Zhou, Matthew Brown, Noah Snavely, and David G Lowe. Unsupervised learning of depth and ego-motion from video. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1851-1858, 2017.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:20/0.04159, 2020.
14


Reviews:
[{"review_id": "13SwtpA21G", "reviewer": "Reviewer_6P5j", "summary": "This paper critiques existing NeRF-based pre-training methods in autonomous driving due to their reliance on rigid view transformations, creating conflicts with NeRF's continuous representation capabilities. The authors then introduce NeRP3D, a point-based 3D detector that retains continuous NeRF representation from pre-training through downstream tasks. Experimental results on the nuScenes dataset demonstrate considerable improvements in both reconstruction quality and detection performance compared to prior methods.", "questions": "1. Compute trade-offs. Please provide FPS, peak memory, and point count vs. accuracy curves for NeRP3D and UniPAD/TPV across identical image resolutions and heads. This will show whether the continuous design is cost-effective at scale. \n\n2. Influence of SDF Prior: The pre-training uses NeuS (SDF), while competitors use standard NeRF (density). How much of the performance gain, especially in geometric fidelity, is attributable to the SDF prior versus the continuous architecture? What happens if NeRP3D is pre-trained with a standard density-based NeRF loss?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper's primary strength is its core insight. Identifying the \"misaligned prior\" between discrete view-transformation and continuous radiance fields is a valuable critique of existing methods. This moves beyond simply \"adding NeRF\" and asks a more fundamental question about how it should be integrated.\n\n2. The proposed NeRP3D architecture is an elegant solution to the problem it identifies. The unification of the pre-training and fine-tuning architectures into a single, continuous, point-queryable function is conceptually clean. This design, which preserves the pre-trained network rather than discarding it, is a good contribution to the field of 3D pre-training.\n\n3. The experiments are solid and provide comprehensive evidence for the paper's claims.", "weaknesses": "Overall I think this is a good work. Only some minor weaknesses remain.\n\n1. Computational Cost: The paper admits in the conclusion that the point-based architecture \"incurs high computational costs\". However, the paper provides no quantitative analysis (e.g., FLOPS, latency, memory) to compare NeRP3D against the view-transform-based baselines. This is a critical omission for a paper aimed at practical autonomous driving applications. \n\n2. Generalization: While results on the nuScenes dataset are promising, evaluation on additional datasets could help establish broader applicability and robustness of the method.\n\n3. Gains vs. the closest continuous baseline are small. Against GaussianPretrain, the margin on detection is +0.1 NDS / +1.1 mAP, which is within typical variance unless confidence intervals are shown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UG6SxCa3cU", "reviewer": "Reviewer_wj2C", "summary": "This paper proposes NeRP3D, a NeRF-Resembled Point-based 3D detector for autonomous driving that avoids view transformation by directly modeling continuous 3D representations. The authors argue that existing NeRF-based pre-training methods (UniPAD, SelfOcc) suffer from conflicting priors: view transformation imposes discrete/rigid representations while NeRFs assume continuous functions. NeRP3D preserves the pre-trained NeRF network during downstream tasks and supports adaptive sampling strategies (ray-wise for rendering, spatial for detection). Experiments on nuScenes show modest gains ($1-2\\%$) over UniPAD in 3D detection, occupancy, and HD map tasks.", "questions": "Please see the critical issues and required clarifications detailed in the Weaknesses section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Strong Problem Diagnosis. The paper clearly identifies the fundamental tension between discrete View Transformation and continuous NeRF priors. The critique of this \"conflicting prior\" (Fig 1) is insightful and provides a solid motivation.\n\n2. Architectural Coherence. The unified framework for pre-training (ray-wise sampling) and downstream tasks (spatial sampling) is elegant. It effectively preserves the pre-trained NeRF representation without an intermediate discrete stage.\n\n3. Comprehensive experimental validation. The paper evaluates across multiple pretext tasks (RGB/depth reconstruction) and three downstream tasks", "weaknesses": "1. Limited Methodological Novelty. The core idea of querying 2D features at 3D points to maintain a continuous representation is not new (e.g., pixelNeRF [1], MVSNeRF [2]). The contribution is an architectural choice (avoiding an intermediate voxel grid) rather than a fundamental new modeling approach.\n\n2. Insufficient Ablations. Key design choices are unjustified. (a) Why deformable cross-attention (Eq 2) over standard attention? (b) The point sampling density trade-off (accuracy vs. cost) is not analyzed.\n\n3. Incomplete Cost Analysis. The paper acknowledges high costs (line 484) but fails to quantify them. Key metrics are missing: (a) Inference latency (ms/frame) and (b) Scalability analysis (i.e., the accuracy/cost trade-off vs. the number of sampled points).\n\n4. Missing Comparisons to Key Baselines. The paper only compares against NeRF-pretrain (UniPAD) and VT-based (BEVFormer) methods. It critically omits comparisons to point-based detectors (e.g., DETR3D[3], PETR[4]). These methods share NeRP3D's core philosophy of querying continuous 3D locations. Without this comparison, it is impossible to know if the gains come from NeRF pre-training or simply from the point-based representation itself.\n\n5. Single-Dataset Evaluation. All experiments are confined to nuScenes. Without cross-dataset validation (e.g., Waymo, KITTI, Argoverse), the method's robustness to different sensors, environments, and camera layouts is entirely unverified.\n\n[1] Yu, Alex, et al. \"pixelnerf: Neural radiance fields from one or few images.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\n[2] Chen, Anpei, et al. \"Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n\n[3] Wang, Yue, et al. \"Detr3d: 3d object detection from multi-view images via 3d-to-2d queries.\" Conference on robot learning. PMLR, 2022.\n\n[4] Liu, Yingfei, et al. \"Petr: Position embedding transformation for multi-view 3d object detection.\" European conference on computer vision. Cham: Springer Nature Switzerland, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hQkKU8VJGt", "reviewer": "Reviewer_4CMH", "summary": "The paper proposes NeRP3D, a new framework that rethinks how NeRF-based pre-training is used for 3D perception. It argues that existing methods with view transformations (e.g., BEV or voxel space), which introduces a mismatch between NeRF’s continuous radiance fields and discrete voxel priors. NeRP3D removes this step entirely and directly models continuous 3D points, fusing multi-view image features through deformable attention. The same network is used for both pre-training (via RGB and signed distance function) and downstream tasks like 3D detection and occupancy prediction. Experiments on nuScenes show clear improvements across all benchmarks and much sharper scene reconstructions.", "questions": "1. How does NeRP3D compare in terms of training time and inference speed to existing voxel-based models?\n2. How sensitive is the model to the number and distribution of sampled points during training?\n3. Is there any evidence that the pre-trained NeRF-like encoder retains its radiance modeling ability after fine-tuning for detection or mapping?\n4. Could this approach be extended to incorporate LiDAR or radar inputs directly, given that it operates in continuous 3D space?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper identifies a conceptual flaw in current NeRF-based pre-training methods and proposes a simple but elegant fix. Removing view transformation aligns the training process with the continuous nature of NeRFs.\n2. The unified architecture is clean and consistent, avoiding the typical “throwaway pre-training” stage found in earlier works.\n3. Experimental results are strong and consistent. The performance gains are not marginal, especially in image reconstruction and occupancy prediction.", "weaknesses": "1. The paper is conceptually motivated by the mismatch between discrete view-transformed features and continuous NeRF representations. However, it does not provide direct quantitative evidence demonstrating how this conflict affects model performance or how NeRP3D resolves it. Including controlled experiments that isolate this factor, such as measuring representational smoothness or reconstruction error across discretization levels—would strengthen the argument and make the theoretical claim more concrete.\n2. The paper lacks an evaluation of runtime, computational cost, or scalability. Because NeRP3D relies on dense point sampling and deformable cross-attention, it may have higher memory and compute overhead compared to voxel-based methods. Reporting training and inference times, FLOPs, and GPU memory usage, along with a discussion of potential optimization strategies (e.g., sampling reduction or pruning), would help clarify whether the approach is practical for real-time or large-scale autonomous driving applications.\n3. All experiments are conducted solely on the nuScenes dataset. While the results there are strong, it remains unclear how well NeRP3D generalizes to different environments, sensor setups, or data distributions. Evaluating the model on additional benchmarks such as Waymo Open Dataset or at least providing cross-dataset transfer results, would greatly enhance the paper’s credibility and demonstrate the robustness of the proposed representation across domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)