FreeVS: Generative View Synthesis on Free Driving Trajectory
Authors: Qitai Wang
Lue Fan
Yuqi Wang
Yuntao Chen
Zhaoxiang Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Novel View Synthesis
Driving Scene
Free Trajectory
Image Generation
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose FreeVS, a novel fully generative approach that can synthesize camera views on free new trajectories in real driving scenes.

Abstract: Existing reconstruction-based novel view synthesis methods for driving scenes focus on synthesizing camera views along the recorded trajectory of the ego vehicle. 
Their image rendering performance will severely degrade on viewpoints falling out of the recorded trajectory, where camera rays are untrained.
We propose FreeVS, a novel fully generative approach that can synthesize camera views on free new trajectories in real driving scenes. 
To control the generation results to be 3D consistent with the real scenes and accurate in viewpoint pose, we propose the pseudo-image representation of view priors to control the generation process.
Viewpoint translation simulation is applied on pseudo-images to simulate camera movement in each direction.
Once trained, FreeVS can be applied to any validation sequences without reconstruction process and synthesis views on novel trajectories.
Moreover, we propose two new challenging benchmarks tailored to driving scenes, which are novel camera synthesis and novel trajectory synthesis, emphasizing the freedom of viewpoints.
Given that no ground truth images are available on novel trajectories, we also propose to evaluate the consistency of images synthesized on novel trajectories with 3D perception models.
Experiments on the Waymo Open Dataset show that FreeVS has a strong image synthesis performance on both the recorded trajectories and novel trajectories. 
The code is released. Project page: https://freevs24.github.io/.

Published as a conference paper at ICLR 2025
FREEVS: GENERATIVE VIEW SYNTHESIS ON FREE
DRIVING TRAJECTORY
Qitai Wang! .2 Lue Fan Yugi Wang Yuntao Chen 450
Zhaoxiang Zhang!.2.4
1,245
Schooo of Future Technology, University of Chinese Academy of Sciences (UCAS), NNLRR, MAIS, Institute of Automation, Chinese Academy of Sciences (CASIA), CUHK *Center for Artificial Intelligence and Roboties, HKISI, CAS (wanggitai20200, lue fan, wangyuqi2020, zhaoxiang zhang)@ia. cn,
chenyunta o0 8 gma i 1 com
Project Page & Code: ps / freevs24, wtt.vs224.ithuu...o.. github.
ABSTRACT
Existing coanstrruttoonbaseee novel view synthesis methods for driving scenes focus synthesizing camera views along the recorded trajectory of the ego ve- hicle. Their image rendering performance will severely degrade on viewpoints falling of the recorded trajectory, where camera rays are untrained. We pro- pose Vre a novel fully generative approach that can synthesize camera views on free new trajectories in real driving scenes. To control the generation results to be 3D consistent with the real scenes and accurate in viewpoint pose, we pro- pose the sneudo-image representation of view priors to control the generation pro- cess. Viewpoint transformation simulation is applied on pseudo-images to simu- late camera movement in each direction. Once trained, FreeVS can be applied to any validation sequences without reconstruction process and synthesis views on novel trajectories. Moreover, we propose two new challenging benchmarks tailored to driving scenes, which are novel camera synthesis and novel trajectory synthesis, emphasizing the freedom of viewpoints. Given that no ground truth novel trajectories,
images are available sistency of images synthesized on novel trajectories with 3D perception models. Experiments on the Waymo Open Dataset show that VS has strong image synthesis performance on both the recorded trajectories and novel trajectories. The code is released.
also propose to evaluate the con-
EmerNerf
Street Gaussians
HUGS
Ours
Origin
Right m
Up m.0m
Figure 1: Synthesis results comparison on the Waymo Open Dataset(Sun et al., 2020). We show the camera views synthesized by NVS methods on the original front view (first row), viewpoint 1.0 m to the right (second row), and viewpoint 1.0 m above (third tow). Our method significantly outperforms previous NVS methods viewpoint outside the existing ego trajectory.

---

Published as a conference paper at ICLR 2025
INTRODUCTION
Scene reconstruction and novel view synthesis (NVS) have gained special attention in embodied AI due to their potential to develop closed-loop simulations for embodied systems. Recent advances have led to remarkable improvements in the reconstruction quality of general scenes using multi- pass and multi-view recordings. However, reconstructing driving scenes presents distinct challenges due to the sparse observations inherent in their less controlled, real-world recording conditions.
Unlike general scene reconstruction settings, which typically leverage excessive views surrounding the scene, driving scene reconstruction generally only has access to image views along the single- pass ego driving trajectory. This limitation raises an important question: How well does driving scene reconstruction perform for novel viewpoints outside the recorded trajectory?
Currently, existing driving scene NVS works(Guo et al., 2023; Wu et al., 2023e; Xie et al., 2023; Turki et al., 2023; Yang et al., 2023a; Zhou et al., 2024b; Chen et al., 2023; Yan et al.. 2024) only evaluate their image rendering quality along the recorded trajectory, leaving this question largely unanswered. As shown in Fig. 1, the quality of rendering results of existing representative NVS methods degrades drastically when the rendering camera moves away from its recording trajectory. This is because, in driving scenes, recorded camera viewpoints are sparse in 3D space and homoge- neous in their positions along the recorded trajectories. The sparsity and homogeneity of recorded camera views cause the camera rays shooting from the camera centers on novel trajectories largely untrained.
We propose FreeVS to address this issue, which is fully generative NVS method that syn- thesize high-quality camera views inside and beyond the recorded driving trajectory. We face two core challenges when building the Fr The first challenge is accurately controlling the camera poses while maintaining the 3D geometry consistency of the generated views. Although previous diffusion-based methods(WWang et al., 2023a; Lu et al., 2023b; Hu et al., 2023; Wang al., 2024b; Yang et al., 2024a) are capable of controlling the camera motion in a coarse trajectory, their control precision far from enough for safety-critical simulation purposes. The second challenge is the ground truth images in the novel trajectories are unavailable, making it difficult to directly train a model synthesize novel views beyond recorded trajectories.
To tackle the two challenges, the proposed FS leverages pseudo-image representation, a sparse yet accurate representation 3l) scenes obtained through colored 3D points projection. Specifically, for each existing view, we create its pseudo-image counterpart by projecting colored point clouds into this view. Here the colored points can be easily obtained by projecting point clouds to any valid images. In this way, we obtain training data pairs to train generative model that can generate a real image from its pseudo-image image counterpart. Since we create the pseudo images using ground truth camera models, they contain sparse but highly accurate appearance and geometry, sidestepping the tough challenge of accurately controlling the camera poses. At inference time, we could create a pseudo-image for a novel viewpoint beyond the recorded trajectory and then synthesize the novel view using the trained generative model. This design greatly narrowed the gap between synthesizing views inside and beyond the recorded trajectory.
To reveal the practicality of FreeVS, we propose two challenging benchmarks for evaluating the performance of NVS methods in driving scenes, which is more practically meaningful than the conventional evaluation on the recorded trajectories. (i) On the recorded trajectories, we propose the novel camera synthesis benchmark. Instead of evaluating synthesis results on test frames sampled at intervals from video sequences (i.e. novel frame synthesis), we propose to drop all images of certain camera view (e.g. the front-side view) in the whole trajectory and synthesize the images of the dropped camera view. (ii) We further propose the novel trajectory synthesis benchmark. With no ground truth images available on novel trajectories, we propose to evaluate the geometry consistency of synthesized views through the performance of off-the-shelf 3D detectors. The experiments on the Waymo Open Dataset (WOD) demonstrate that VS outperforms previous NVS methods by a large margin in the two more practical benchmarks as well as in the traditional novel frame synthesis benchmark.
Our contributions are summarized follows:
2

---

Published as a conference paper at ICLR 2025
1. We propose VSeeVS, fully generative view synthesis method for driving scenes that gen- erate high-quality 3D-coherent novel views both for recorded and novel trajectories without time-cost reconstruction.
2. We devise two new benchmarks for evaluating driving NVS methods novel trajectories beyond recorded ones.
3. Experiments on WOD show that FreeVS achieves leading performance on synthesizing camera views inside and beyond the recorded trajectory.
2 RELATED WORK
2.1 NOVEL VIEW SYNTHESIS THROUGH RECONSTRUCTION
Recently, rapid progress has been achieved in novel view symthesis through 3D reconstruction and radiance field rendering. Neural Radiance Fields (NeRF) (Mildenhall al., 2020) utilizes multi- layer perceptrons to represent continuous volumetric scenes and achieve a breakthrough in rendering quality. Many works have extended NeRF to unbounded. dynamic urban scenes (Tancik al., 2022; Barron et al., 2022; Ost et al., 2022; Rematas et al., 2022: Turki et al., 2022; Lu et al., 2023a; Guo et al., 2023; Liu et al., 2023a; Wu al., 2023e; Xie et al., 2023; Turki et al., 2023; Yang et al., 2023b; Wang et al., 2023b; Ost et al., 2021; Tonderski et al., 2024). Authors of MapNeRF (Wu et al., 2023a) noticed the problem of NeRF in generating extrapolated views and proposed incorporating map priors to guide the training of radiance fields. 3D Gaussian Splatting (Kerbl et al., 2023) (3D GS) models scenes with numerous 3D Gaussians. Recently, some researchers have extended 3D GS to dynamic scenes (Luiten al., 2024; Wu al., 2023b; Yang et al., 2023d;c) and driving scenes (Zhou et al., 2024b; Chen et al., 2023; Yan et al., 2024). HUGS (Zhou et al., 2024a) further jointly model the geometry, appearance, motion, and semanties in 3D scenes for better scene understanding.
Another line of work in NVS through reconstruction focuses on fast scene reconstruction or gener- alizable red-ffrwwrr reconstructton((lymn et al., 2019; Chen et al., 2021; Liu et al., 2022a; Johari et al., 2022; Lin et al., 2022; Varma et al., 2022; Xu et al., 2024; Chen et al., 2025a; Charatan et al., 2024; Wang et al., 2024a; Ren al., 2024b;a; Zhang al., 2025; Liu et al., 2025), which are primarily applied to object-centric or small scenes with few source observations. Recently, SCube(Ren et al., 2024c) and DrivingRecon(?) achieved the feed-forward reconstruction in driving scenes with generalizable models. However, the reconstruction accuracy and rendering resolution of forward reconstruction models in driving scenes still significantly fall short compared to cur- rent per-scene reconstruction methods. G3R(Chen et al., 2025b) achieved the fast reconstruction (in a few minutes) of large scenes by efficiently updating a 3D scene representation with generalizable modules that take gradient feedback signals from differentiable rendering input. Compared to G3R, FreeVS completely does not require a scene reconstruction process, but its image synthesis efficiency is constrained by the efficiency of existing video generation models.
2.2 NOVEL VIEW SYNTHESIS THROUGH GENERATION
Novel view synthesis through image generation has greatly benefitted from the advancements in image generation models(Ho et al., 2020; Song et al., 2020; Rombach et al., 2022; Blattmann et al., 2023). Free View Synthesis(Riegler & Koltun, 2020) conditions the image generation process on wrapped image features. Zero-1-to-3(Liu et al., 2023b) and ZeroNVS(Liu et al., 2023b) gener- novel views with a diffusion process conditioned on the reference image and the target camera pose embedded as a text embedding. GeNvS(Chan et al., 2023) condition the diffusion process on volume-renderee feature images. Reconfusion(Wu et al., 2024) uses the diffusion model re- fine images rendered by the reconstruction model as extra supervision to the reconstruction process. Similarly, KealFusionnMMela- Kyriazi et al., 2023) uses a diffusion model to provide an extra per- spective view for object-centric reconstruction. Yu et al. (2024a) use the Stable Video Diffusion mldet(nl et al., 2023) to iteratively refine the rendered video along a novel camera trajec- tory based on the partial image wrapped from the reference view to the target view. Most of the previous novel view synthesis through generation works are designed for object-centrie(Lin et al., 2023b; Chan et al., 2023; Wu et al., 2024; Melas-Kyriazi et al., 2023; Yu et al., 2024a) indoor(Liu et al., 2023b; Chan et al., 2023; Wu et al., 2024; Yu et al., 2024a) scenes. MagicDrive3DDGaa et al., 2024) places image generation(Gao al., 2023) upstream of scene reconstruction obtain the 3D>
3

---

Published as a conference paper at ICLR 2025
representation of an imagined scene. For driving scenes, Yu et al. (2024b) proposes SGD which generates novel views with a diffusion process conditioned on reference images and depth maps in the target view. However, SGD still can only synthesizes camera views along the recorded trajectory of the ego vehicle.
FREEVS
We introduce the detailed design of our proposed FreeVS in this section. We summarize the algo- rithm pipeline of VS in Fig. 2.
Overview of VS. FreeVS is fully generative model that synthesizes new camera views on novel trajectories based on observations of 3D scenes from recorded trajectories. FSeeVS is imple- mented as a conditional video diffusion model. To ensure the model generates views from accurate viewpoints with consistent appearance attributes and 3D geometries as the real 3D scene, we for- mulate all essential priors regarding the 3D scene pseudo-images to control the diffusion process. Based on view prior conditions, Free VS is learned by denoising noised target views at training time and synthesizing target views from pure noise inference time.
3.1 VIEW PRIORS FOR VIEW GENERATION
Unified view prior representation. One major challenge of generative novel view synthesis is to ensure the generated images are consistent with the priors in the novel view. Here the view priors include the observed colors, 3D geometry. and camera pose of this view. However, the different types of priors are totally different modalities, posing significant challenge for diffusion models to precisely encode them. For example, as discussed in Sec. 1, diffusion models cannot precisely control the camera motions (i.c., poses). To tackle this challenge, we propose a pseudo-image representation that unifies all types of view priors in one modality. Pseudo-images are obtained through colored point cloud projection. Specifically, for each frame driving sequence, we first merge LiDAR points across the nearby frames. LiDAR points moving objects will be merged along the moving trajectory of the object based on their 3D bounding box annotations. Finally, we project the merged and colored LiDAR point cloud into the target camera viewpoints as pseudo- images. In this way, we encode color information, geometry information, and the view pose into unified pseudo-image, largely facilitating the learning of generative models.
Compared with directly providing reference images and viewpoint transformations to the diffusion process, the pseudo-image representation greatly simplified the optimization objective of the gener- ative model: With the former inputs, the model is required to have a correct understanding of the 3D scene geometry as well as the transformation of viewpoint to generate correct view based on the reference image. In contrast, with pseudo-image as input, FS only needs to recover target views based on sparse valid pixels, which more akin to a basic image completion task. The sim- plification of the training objective greatly enhances the model's robustness unfamiliar viewpoint transformations, since the generated image is completed from sparse but geometrically accurate pixel points.
Viewpoint transformation simulation. Another challenge of novel view generation on new trajec- tories stems from the absence of ground truth views beyond recorded trajectories. We can only train the generative model on recorded trajectories, where the diversity of viewpoint transformations is extremely limited. For example, we have no access the training sample where the frontal cam- era is moved laterally. However, such viewpoint transformation is essential for synthesizing views on novel trajectories inference time. This brings significant gap between training and infer- ence for the generative model. Moreover, we propose the viewpoint transformation simulation with pseudo-images. At training time, we sample color and LiDAR priors from frames mismatched with the training image frames. That is to say, we force the generative model to recover current camera views based on observations from nearby frames. Through this, we simulate the camera movement in each direction as strong data augmentation on pseudo-image priors. For example, as the ego ve- hicle moves along its heading direction, the side cameras are actually moving to their front or right, Therefore although we have access to the training data where the front camera is moved laterally, we can still simulate lateral camera movement by training FreeVS on side-views with mismatched observation-superviss frames.
4

---

Published as a conference paper at ICLR 2025
Noise
Loss
EyE
Denoising U-Net
Dyae
Vr+e
x
Training View
inages Viwe
V
VT
LIDAR Ports
VTTk
Concatenate
Frozen
Projection
Ep
Training
Inference
Projection
Novel Trejectory
Recorded Trejectory
V
Training view
Inference view
Pseudo images
Figure 2: Method pipeline of FreeVS. We propose encode the view priors in driving scenes including appearance, 3D geometry. and pose of target viewpoints all in one modality as the pseudo- images. Best viewed in color. The diffusion model is trained to synthesize target views solely based on the unified pseudo-image priors.
3.2 DIFFUSION MODEL FOR NVS
Training of FreeVS. In each training iteration of FreeVS. we randomly sample a colored LiDAR point cloud sequence p (P1. ...Pr.) from the driving scene dataset. p sequence colored point clouds, each point cloud frame p: E R x6 contains a set of 6-dimension 3D points. 3D points are recorded with their positions in the world reference frame and their visible colors. From the driving sequence, we also sample target camera viewpoint sequence of n frames and m surrounding cameras. Each camera parameter 2 stands for the intrinsics and extrinsics of a camera viewpoint. For a viewpoint the target video with camera parameter vy we project the colored LiDAR point cloud p; into the viewpoint as pseudo-image sf = Proj v The training target of FreeVS in each iteration ([s).....s") s""),..... (s....... s)]) E IR" xx
([rt,... .... vf") ]. ..... ["/,..., .... "("))
recover the target images at sampled viewpoints
based on the pseudo-image sequence
During the training process of FreeVS, the ground truth camera views R=xmx3xx* sampled along the viewpoint sequence v. The ground truth camera views are encoded target video latent representation (x) = y E B=xmxexxxx through an frozen VAE encoder. Then We have the diffused inputs y, y.,y + 0,,6, (O0 I), here a, and a, is noise schedule at the diffusion time step т. We also encode the pscudo-images into the latent representation Es = 2 e 2axmxexhxe with 2D encoder trained simultaneously with the diffusion model. We concatenate to the diffusion model to predict the noise upon y. We have a denoising model fa with parameters that take y. as inputs and optimized by minimizing the following denoising objective:
is also
yr and z the input k E [R*xmx2exk>>
ER T. e - [||c fo (k; 1).
(1)
Where is the description conditions generated by encoding the reference camera views with an off- the-shelf CLIP-vision model(Radford al., 2021), following the convention of diffusion models. p- a uniform distribution over the diffusion time
Synthesizing views novel trajectories with FreeVS. During the inference process of FS we project the colored LiDAR points in each frame into the targeted camera poses to generate pseudo-image sequence for image synthesis. The diffusion model fed with pure noise latents concatenated with pseudo-image latents, The diffused latent will be decoded as synthesized views through an off-the-stee the- VAE decoder DVAE-
3.3 EVALUATING NVS ON NOVEL CAMERA AND NOVEL TRAJECTORY SYNTHESIS
To thoroughly demonstrate the view generalization capability of our FreeVS, which can truly meet the demands of closed-loop embodied simulation, we present a comprehensive discussion of eval- uation benchmarks for novel view synthesis in driving scenes. Fig. 3 illustrates this: panels (a) and (b) summarize existing evaluation benchmarks, while panels (c) and (d) introduce our two new challenging NVS benchmarks.
5

---

Published as a conference paper at ICLR 2025
Train Views
Test Views
Ego Poses
Detection Targets
(a) Front-view novel frame synthesis
(b) Multi-view novel frame synthesis
(c) Multi-view novel camera synthesis
(d) Novel trajectory synthesis
Figure 3: Benchmarks for evaluating NVS methods in driving scenes. We conclude the previous NVS evaluation benchmarks for driving scenes as (a) and (b). We propose two novel evaluation benchmarks: the novel camera synthesis benchmark as in (c), and the novel trajectory synthesis benchmark as in (d). Best viewed in color.
Evaluating NVS on recorded trajectories. All current NVS works for driving scenes evaluate their NVS performance on test frames sampled periodically along the recorded trajectory. Some previ- ous driving scene NVS methods, such Street Gaussians Yan et al., 2024), NSG(Ost et al., 2021), and Mars (Wu et al., 2023c), evaluate their performance with only front camera views considered, as illustrated in Fig. 3(a). Other NVS methods take the multi-view cameras into consideration as illustrated in Fig. 3()), such as DrivinGaussian et al., 2024b), PVG(Chen et al., 2023), EmerN- et al., 2024), S-Nerf(Xie et al., 2023), and SUDS(Turki
erf((Yngg et al., 2023a), SeuRADDTonderrss et al., 2023). All these two evaluation benchmarks sample test frames periodically along the trajec- tory, i.e. novel frame synthesis. In such cases, camera views in test frames can be directly inferred from the adjacent frames, especially for datasets with a high video frame rate (e.g. 10Hz for the WOD dataset). To provide a more challenging evaluation setting for driving scene NVS methods, we propose the novel camera synthesis benchmark as illustrated in Fig. 3(C). Instead of period- ically sampling test frames, we drop images collected by certain multi-view cameras throughout a driving sequence as test views. For example, for driving sequence in the WOD dataset, we provide NVS methods with the front, and side camera views as training views and evaluate the synthe- sis results on front-left and front-right views. Under the novel camera synthesis benchmark, NVS methods are required to synthesize views on unseen camera poses, which places higher demands accurately modeling the 3D scene. We ensure in the validation sequences, most 3D contents in front-side cameras are observed in the front or side camera views along the ego trajectory.
Novel trajectory synthesis. On test views sampled from the recorded trajectories, the ground truth camera images are available for evaluating the synthesized images with image similarity metrics including SSIM, PSNR, and LPIPS(Zhang al., 2018). Differently, in driving scenes, there are no ground truth images available on novel trajectories. The Fréchet Inception Distance (FID)Seeitzer, 2020) metric can compare the overall image distribution between synthesized images on novel tra- jectories and ground truth images on recorded trajectories, but it can not assess the fidelity of the synthesized images the 3D scenes all. In addition to qualitative visualization comparisons, we also propose the perceptual robustness evaluation to assess the geometry consistency performance of NVS methods on new ego trajectories.
In driving scenes, modern image-based 3D perception models have achieved high robustness. As shown in Fig. 3()), assuming an NVS method can synthesize views on a novel trajectory with ideal image quality, the perception model feed with synthesized views should still be able to produce accurate predictions on the novel trajectory. With such an assumption, we believe that the perfor- mance of an off tff-thh-sshef perception model on novel trajectories reflect the quality of images synthesized by the NVS methods. Under the novel trajectory synthesis benchmark, we feed the synthesized images and camera poses the novel trajectory off-the-shelf 3D camera-based

---

Published as a conference paper at ICLR 2025
detector. The detection results are evaluated with the longitudinal error tolerant mean average preci- sion (HEun et al., 2022) metric on the WOD dataset. For all NVS methods, we modify novel trajectories by laterally shifting the ego positions in each frames. We shift the trajectories by 1.0 m, 2.0 and 4.0 and report the mean evaluated results mAP 1.0m 1.0....mmpppp...... 2.0m and mAP 4.0m mAPEET
4 EXPERIMENTS
In this section, we first introduce our experimental setup including datasets, evaluation benchmarks, method implementation details, and counterpart methods. Then provide the quantitative and qualitative experiment results.
Datasets. We perform experiments the WOD dataset(Sun al., 2020). We select 12 driving sequences for evaluating NVS methods. We ensure that there is ample space on both sides of the ego vehicle in most frames of the selected sequence to simulate novel trajectories by lateral moving the ego vehicle. For each sequence, all 200 data frames in 10Hz are used.
Evaluation of NVS methods. We compare Free with NVS counterparts under all the experiment benchmarks shown in Fig. 3. For the front-view or multi-view novel frame synthesis benchmark (Fig. 3(a) and (b)), we sample every fourth frame in driving sequences as test frames. All the remaining frames are used for training NVS counterparts, or input frames for FreeVS. On, we report metrics including SSIM, PSNR, and LPIPS. Under the novel camera synthesis benchmark, we reserve all the front-side camera views as test views and use the front and side camera views train views throughout each sequence. Note that for FS which does not require scene reconstruction on validation sequences, only take information from the train views to generate test views.
Moreover, we also evaluate NVS methods on novel trajectories with the FID score and the proposed perceptual robustness evaluation method. We take ANV-FCOS3D+<WWain strong multi-view camera-based 3D detector as our baseline detection model. We follow most of the settings of the official open-sourced implementation of NV-FCCS3D+++ We train MV-FCOS3D++ for 24 epochs on the WOD training set (except for the validation sequences in our experiments) obtain the baseline detector. Following (Wang al., 2022), initialize MV-FCOS3D++ from an FCOS3D++ checkpoint, which is also trained on the above training sequences. We feed the camera views synthesized on novel trajectories the baseline detector. We report camera-based 3D detection metries LET- LEE-uAA(HHung et al., 2022) the vehicle class as mApLET
al., 2022), basic yet
Method details. We implement the proposed FreeVS pipeline based on Stable Video Diffusion (SVD)(Blantman al., 2023). We initialize the diffusion model from pre-trained Stable Diffu- sion checkpoint(Rombact et al., 2022). FS is trained on the WOD training set, except for the selected validation sequences. To generate pseudo-image across the adjacent +2 frames of each frame. If a 3D LiDAR point has more than one projected 2D point in multi-view images, the mean color of its projected image points will be recorded. For viewpoint transformation simulation, we randomly sample the target viewpoint sequence starting from the adjacent #4 frames the first frame of the source point cloud sequence. We employ ConVNext- et al., 2022b) backbone the pseudo-image encoder. We train the model for 40,000 iterations with a batch size of 8 and video frame length n 8. Please refer to the appendix for more training details.
we accumulate colored LiDAR points
4.1 SOTA COMPARISON UNDER THE PROPOSED CHALLENGING NEW BENCHMARKS.
Novel camera synthesis. We first report the performance of NVS methods under the proposed multi-view novel camera synthesis benchmark in Table 1. FS achieves leading performance on all metrics by a large margin. Previous NVS methods tend to render images with severe image distortion massive unnatural artifacts when facing severe scene information loss on the target views, as shown in Fig. 4. Meanwhile, FreeVS can generate camera views close to ground truth views based on limited 3D scene observations.
Novel trajectory synthesis. We also report the FID and perceptual robustness performance of NVS methods novel trajectories in Table 2. The proposed Free outperforms previous NVS methods on almost all metrics with different lateral offsets applied to the viewpoints. Compared to previous NVS methods, the proposed Free has a very strong performance the FID metric. This is mainly

---

Published as a conference paper at ICLR 2025
Table 1:
Comparison with NVS counterparts on novel camera synthesis. For all NVS methods, use all front and side camera views source views to synthesize the front-side camera views.
Front-side camera syntheising
Methods
SSIM
PSNR1
I.PPSS
3D-GS(Kerbl et al., 2023)
0.484
15.97
0.442
EmerNeer((aamm ct al., 2023a)
0.603
19.61
0.330
StreetGaussian( Yan al., 2024)
0.531
17.35
0.397
Ours
0.628
20.70
0.283
Table Comparison with NVS counterparts on novel trajectories. The axis defined lateral
to the ego vehicle's heading direction. t: performance of baseline detector ground truth images.
y/ 0.0m
y 1.0m
y/ + 2.0m
y + 4.0m
Methods
FID
mAPLEr
FID|
MAPLET 1.3m 1
FID
mAP
FID
mAPE.
GT
0.895
3D-GS
34.79
0.729
52.07
0.605
61.16
0.581
86,21
0,452
EmerNerf
53.88
0.600
58.26
0,510
69.50
0.478
84,81
0.464
Strectausssia
21.62
0.826
41.17
0.738
55.71
0.682
80.44
0.544
Ours
11.18
0.816
13.45
0.786
16.60
0.724
22.08
0.612
because the proposed FreeVS is nearly free from image degradation and artifacts when synthesizing images on novel trajectories. FreeVS also has the strongest mAPlET performance among all NVS methods, which indicates that generation-based method, FreeVSS is of even higher fidelity to the 3D scene geometry compared with previous reconstruction-bar based methods when rendering views on novel trajectories. We also provide a visualization comparison example in Fig. 5.
While FreeVS relies on LiDAR point inputs, EmerNerf and Street Gaussians also rely on LiDAR depth supervision during their training process. Therefore FreeVS did not gain any information advantages our experiments. Moreover, as a fully generative method, Free does not require any scene reconstruction process when applied to validation sequences. From this aspect, inference time, FreeVS costs less computational resources even compared with 3DGS-basee methods, which usually take 1-2 hours to model validation sequence of 20s.
4.2 SOTA COMPARISON NOVEL FRAME SYNTHESIS
We also report the performance of NVS methods under the traditional front-view novel frame syn- thesis or multi-view novel frame synthesis benchmark in Table 3. The performance of previous NVS methods is strong when only the front-view camera is considered. However, when it comes to the multi-view setting which is more aligned with the current autonomous driving scenes, the perfor- mance of previous NVS methods is surpassed by the proposed FreeVS by a large margin. It is worth mentioning that Table 3, all previous reconstruction-baser NVS methods exhibit a significant performance drop when multi-view cameras are considered. We think this is due to the increased number of training views, the expanded range of the visible 3D scene, and the rapidly changing content in lateral views, all of which make the convergence of reconstruction models more difficult.
4.3 ABLATION STUDIES
Ablation on view prior condition. We first ablate on the representation of view prior as conditions for the diffusion process, shown in Table 4. We apply breakdown experiment on the pseudo- image representation. Models are trained for 20,000 iterations. We start by dropping the color information in the pseudo-image representation, represented by Table 4(b). Dropping the color nearly does not affect the geometric accuracy of rendered results, but has considerable impact on the image similarity metrics. Then we experiment with dropping the LiDAR inputs (c), where the reference images and camera pose transformation matrix (from the reference view to the target view) are independently encoded by a VAE or MLP encoder. Under this setting, we found the diffusion model unable to accurately synthesize views the target viewpoint. Most time, the model ignores the pose condition and moves the camera viewpoint by its familiar viewpoint transformation. (e.g. always move the frontal camera forward backward, or move the side camera left right.) Based
8

---

Published as a conference paper at ICLR 2025
Table 3:
Comparison with NVS counterparts on recorded trajectories. We report the perfor- mance of NVS methods when only the front-view cameras are considered or when all multi-view cameras are considered. Reconstruction time cost and FPS are measured under the multi- set-
ting, with single NVIDIA L20 GPU.
Front View
Multi-view PSNR
Reconstruction
Methods
FPS
SSIM
PSNR
LPIPS
SSIM
LPIPS.
time
3D-GS(Kerbl et al., 2023)
0.799
26.31
0.143
0.586
19.21
0.366
2-3hF
61.2
EmerNerf( Yang et al., 2023a)
0.869
30.28
0.155
0.689
24.68
0.347
2-3h
0.2
StreetGiassian( Yan et al., 2024)
0.903
30.80
0.096
0.702
22.47
0.314
1-2h
52.6
Ours
0.787
25.30
0.139
0.730
24.96
0.179
0.9
Table
Ablation study on view prior condition. We conduct a breakdown ablation on the pro- posed pseudo-image representation of view priors. The axis is defined lateral to the ego vehicle's heading direction.
Multi-view PSNR1
+ 2.0m
View priors
Encoders
SSIM
LPIPS..
FID
mAP2.0m
full priors
2D>CCnnv
0.704
23.28
0.203
21.27
0.690
(b)
w/o, color
2D-Conv
0.701
23.05
0.231
23.13
0.687
w/o. LiDAR
2D-Conv ML.P
0.613
19.86
0.288
21.25
0,013
w/o. projection
2D-Conv + 3D MLP
0.609
19.88
0.284
21.32
0.028
on (c). we experiment with preserving all view prior imputs but do not unify them as pseudo-images (d). The LiDAR points are encoded as latents with a point cloud backbone Yan et al., 2018). The experiment result shows the model fails at utilizing LiDAR inputs due its significant gap with images, the model trained under setting (d) has identical performance as the mode trained under setting (c). Due to the wrong viewpoint of most generated images, trained models under settings (c) and (d) have extremely poor perceptual robustness performance. Through this observation, we can conclude that the pseudo-image representation greatly improved the overall quality and viewpoint controllability of images generated by the diffusion model.
We report the results of ablation studies on viewpoint transformation simulation with speudo-imaged in Table 5. We report the performance of FreeVS under the multi-view novel frame synthesis setting and on novel trajectories generated by applying 2.0 m lateral offsets the recorded trajectories. As shown in Table 5, sampling target frames from adjacent +2 or +4 frames from the source frame can boost the view-synthesize per- formance of FreeVS on novel trajectories. When the temporal sampling window size exceeds 14 frames, the view-synthesize performance of Free on the recorded trajectory will be negatively affected. We believe this is due to the large timestamp mismatch between view priors and target images hindering the model's convergence. We also present a visualization illustration of the effect of viewpoint transformation simulation in the appendix.
Ablation study on viewpoint transformation simulation.
4.4 VISUALIZATION COMPARISON
We show visualization comparisons between NVS methods under the novel camera synthesis bench- mark in Fig. 4, and under the novel trajectory synthesis benchmark in Fig. 5.
4.5 CONCLUSION
We present FreeVS, a novel fully generative method for synthesizing camera views on free driving trajectory. We propose the pseudo-image representation of view priors, which conveys accurate 3D scene geometry and viewpoint conditions through colored point projection. The diffusion model trained to synthesize target views solely based on pseudo-images. In this paper, we fully discuss the evaluation benchmarks for driving scene NVS. We propose two novel evaluation benchmarks includ- ing the novel camera synthesis benchmark and the novel trajectory synthesis benchmark. We also propose the perceptual robustness evaluation method for assessing the performance of NVS meth- ods novel trajectories. Experiments across several experiment benchmarks show that FreeVS achieves leading performance in synthesizing camera views inside or beyond recorded trajectories.

---

Published as a conference paper at ICLR 2025
Table Ablation study on viewpoint translation simulation. The y axis is defined lateral to the ego vehicle's heading direction.
Multi-view
- 2.0m
Temporal sampling window size
SSIM
PSNR
LPIPS
FID
mAP FT
0.733
25.04
0.180
16.93
0.707
22 frames
0.734
25.04
0.179
16.77
0.713
44 frames
0.730
24.96
0.179
16.60
0.724
88 frames
0.717
24.82
0.188
16.53
0.721
Quss
EmerNat
Scneet Grarstions
Groundd Truth
Figure 4: Visualization comparison on novel-camera synthesis benchmark. We show the front- side camera views synthesized from front and side camera views with NVS methods.
Original Vieapoint
Viewpoint 2e.t
Viewpoint at right 2.0
Emeetteet
Figure Visualization comparison on novel trajectories. We show the camera views synthesized by NVS methods on the original training viewpoint, viewpoint 2.0 left of the original viewpoint, and viewpoint 2.0 right of the original viewpoint.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported in part by the National Key R&D Program of China (No. 20222D0116500) the National Natural Science Foundation of China (No. U21B2042, No. 62320106010), and in part by the 2035 Innovation Program of CAS.
REFERENCES
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVE Conference Computer Vision and Puttern Recognition, pp. 5470-5479, 2022. 3
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint ar arXi::231..55177 2023. 3,
Eric R Chan, Koki Nagano, Matthew A Chan, Alexander W Bergman, Jeong Joon Park, Axel Levy, Miik Aittala, Shalini De Mello, Tero Karras, and Gordon Wetzstein. Generative novel view syn- thesis with 3d-aware diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2217-4229, 2023. 3
David Charatan, Sizhe Lester Li, Andrea Tagliasacchi, and Vincent Sitzmann. pixelsplat: 3d gaus- sian splats from image pairs for scalable generalizable 3d reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 19457-19467, 2024. 3, 22
Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang, Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF international conference computer vision, pp. 14124-14133 2021.
Yuedong Chen, Haofei Xu, Chuanxia Zheng, Bohan Zhuang, Mare Pollefeys, Andreas Geiger, Tat- Jen Cham, and Jianfei Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. In European Conference Computer Vision, pp. 370-3330000 386. Springer, 2025a. 3
Yun Chen, Jingkang Wang, Ze Yang, Sivabalan Manivasagam, and Raquel Urtasun. G3r: Gradient guided generalizable reconstruction. In European Conference on Computer Vision, pp. 305-323. Springer, 2025b. 21, 22
Yurui Chen, Chun Gu, Junzhe Jiang, Xiatian Zhu, and Li Zhang. Periodie vibration gaussian: Dynamic urban scene reconstruction and real-time rendering. arXiv preprint arXiv:2311.1851,, 2023. 2, 3, 6, 19
Kai Cheng, Xiaoxiao Long, Wei Yin, Jin Wang, Zhiqiang Wu, Yuexin Ma, Kaixuan Wang, Xiaozhi Chen, and Xuejin Chen. Uc-nerf: Neural radiance field for under-calibrated multi-view cameras in autonomous driving, arXiv preprint arXiv:2311.1644 2023. 18
John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gradient descent. In 2019 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2019. doi: 10. 10.11009
URL http: /ttp///xx//// dx doi. www/oorg./...o...oo 1109 cvpr. 2019. 00247.
Ruiyuan Gao, Kai Chen, Enze Xie, Lanqing Hong, Zhenguo Li, Dit- Din Yeung, and Qiang Xu. Magicdrive: Street view generation with diverse 3d geometry control. arXiv preprint arXiv:2310.02601, 2023. 3
Ruiyuan Gao, Kai Chen, Zhihao Li, Lanqing Hong, Zhenguo Li, and Qiang Xu. Magicdrive3d: Con- trollable 3d generation for any- rendering in street scenes. arXiv preprint arXiv:2405. 14475, 2024. 3
Jianfei Guo, Nianchen Deng, Xinyang Li, Yeqi Bai, Botian Shi, Chiyu Wang. Chenjing Ding, Dongliang Wang, and Yikang Li. Streetsurf: Extending multi-view implicit surface reconstruction to street views. arXiv preprint arXiv:2306.04900 2023. 2,3

---

Published as a conference paper at ICLR 2025
Jonathan Ho and Tim Salimans. 1rXi::2207.125900 2022. 16
Classifier-fre
diffusion
guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 68500 2020.
Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shot- ton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309. 17880, 2023. 2
Wei-Chih Hung, Henrik Kretzschmar, Vincent Casser, Jyh-Jing Hwang, and Dragomir Anguelov. Let-3d-ap: Longitudinal error tolerant 3d average precision for camera-only 3d detection. arXiv
preprint arXiv:2206.07775, 2022. 7
Mohammad Mahdi Johari, Yann Lepoittevin, and François Fleuret. Geonerf: Generalizing nerf with geometry priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18365,1837, 2022. 3
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023, 3, 8, 9, 16
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014. 16
Hao Li, Ming Yuan, Yan Zhang, Chenming Wu, Chen Zhao, Chunyu Song, Haocheng Feng, Errui Ding, Dingwen Zhang, and Jingdong Wang. XId: cross-lane dataset for benchmarking novel driving view synthesis. arXiv preprint arXiv:2406. 18360, 2024a. 20, 23
Zhuopeng Li, Yilin Zhang, Chenming Wu, Jianke Zhu, and Liangjun Zhang. Ho-gaussian: Hybrid optimization of 3d gaussian splatting for urban scenes. arXiv preprint 2rXiv:24433.20032, 2024b.
18
Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Efficient neural radiance fields for interactive free-viewpoint video. In SIGGRAPH Asia 2022 Conference Papers, pp. 1-9, 2022. 3, 22
Jeffrey Yunfan Liu, Yun Chen, Ze Yang, Jingkang Wang, Sivabalan Manivasagam, and Raquel Ur- tasun. Neural scene rasterization for large scene rendering in real time. In ICCV, 2023a. 3
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-I-to-3; Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 92989999999999 9309, 2023b.
Tianqi Liu, Guangcong Wang, Shoukang Hu, Liao Shen, Xinyi Ye, Yuhang Zang, Zhiguo Cao, Wei Li, and Ziwei Liu. Mvsgaussian: Fast generalizable gaussian splatting reconstruction from multi-view stereo. In European Conference on Computer Vision, pp. 37-53. Springer, 2025. 3
Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang. Neural rays for occlusion-aware image-based rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7824-7833, 2022a. 3
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11976-11986, 2022b. 7, 16
Fan Lu, Yan Xu, Guang Chen, Hongsheng Li, Kwan Yee Lin, and Changjun Jiang. Urban radiance field representation with deformable neural mesh primitives. ICCV, 2023a. 3
Jiachen Lu, Ze Huang. Jiahui Zhang, Zeyu Yang, and Li Zhang. Wovogen: World volume- dif- fusion for controllable multi-camert driving scene generation. arXiv preprint arXiv:2312.02944, 2023b.
12

---

Published as a conference paper at ICLR 2025
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 3DV, 2024." 3
Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Realfusion: 360deg reconstruction of any object from a single image. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 8446-844 841668845,220 2023. 3
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural scene graphs for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1856-2865, 2021. 3,6
Julian Ost, Issam Laradji, Alejandro Newell, Yuval Bahat, and Felix Heide. Neural point light fields. CVPR, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PMLR, 2021. 5
Konstantinos Rematas, Andrew Liu, Pratul P Srinivasan, Jonathan Barron, Andrea Tagliasacchi, Thomas Funkhouser, and Vittorio Ferrari. Urban radiance fields. In CVPR, 2022. 3
Jiawei Ren, Kevin Xie, Ashkan Mirzaci, Hanxue Liang, Xiaohui Zeng, Karsten Kreis, Ziwei Liu, Antonio Torralba, Sanja Fidler, Seung Wook Kim, et al. L4gm: Large 4d gaussian reconstruction model. arXiv preprint arXiv:2406. 10324, 2024a.
Xuanchi Ren, Jiahui Huang, Xiaohui Zeng, Ken Museth, Sanja Fidler, and Francis Williams. Xcube: Large-scale 3d generative modeling using sparse voxel hicrarchies. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4209 4219, 2024b.
Xuanchi Ren, Yifan Lu, Hanxue Liang, Jay Zhangjie Wu, Huan Ling, Mike Chen, Francis Fidler, Sanja annd Williams, and Jiahui Huang. Scube: Instant large-scale scene reconstruction using voxsplats. In The Thirty-eighth Annual Conference Neural Information Processing Systems, 2024c. 3
Gemnot Riegler and Vladlen Koltun. Free view synthesis. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIX 16, pp. 623- 640. Springer, 2020. 3
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022. 7,
Maximilian Seitzer. pytorch-fid: fid: FID Score for PyTorch. https::/github.com htt.:u...coommmeeeeaeaae
secorrch-iid, August 2020. Version 0.3.0. 6
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020. 3
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for au- tonomous driving: Waymo open dataset. In CVPR, 2020. 1,
Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srini- vasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8248 8258, 2022. 3
13

---

Published as a conference paper at ICLR 2025
Adam Tonderski, Carl Lindström, Georg Hess, William Ljungbergh, Lennart Svensson, and Christoffer Petersson. Neurad: Neural rendering for autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14895-14904, 2024. 3,
Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan, Mega-nerf: Scalable construction of large-scale nerfs for virtual fly-throughs. In CVPR, 2022.
Haithem Turki, Jason Y Zhang, Francesco Ferroni, and Deva Ramanan. Suds: Scalable urban dynamic scenes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 22375-12385, 2023. 2, 3,
Mukund Varma, Peihao Wang, Xuxi Chen, Tianlong Chen, Subhashini Venugopalan, and Zhangyang Wang. Is attention all that nerf needs? In The Eleventh International Conference on Learning Representations, 2022. 3,22
Shuzhe Wang, Vincent Leroy, Yohann Cabon, Boris Chidlovskii, and Jerome Revaud. Dust3r: Ge- ometric 3d vision made easy. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 20697-20709, 2024a. 3
Tai Wang, Qing Lian, Chenming Zhu, Xinge Zhu and Wenwei Zhang. MV-FCOS3D++: Multi- View camera-only 4d object detection with pretrained monocular backbones. arXiv preprint, 2022.
Xiaofeng Wang. Zheng Zhu, Guan Huang, Xinze Chen, Jiagang Zhu, and Jiwen Lu. Drive- dreamer: Towards weal-world-driver world models for autonomous driving. arXiv preprint arXiv:2309.09777. 2023a. 2
Yuqi Wang. Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
14749- 14759, 2024b. 2
Zian Wang. Tianchang Shen, Jun Gao, Shengyu Huang, Jacob Munkberg, Jon Hasselgren, Zan Gojeic, Wenzheng Chen, and Sanja Fidler. Neural fields meet explicit geometric representations for inverse rendering of urban scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8370-8388 2023b. 3
Chenming Wu, Jiadai Sun, Zhelun Shen, and Liangjun Zhang. Mapnerf: Incorporating map priors into neural radiance fields for driving view simulation. In 2023 JEEE/RSJ International Confer- ence on Intelligent Robots and Systems (IROS), pp- 1082-7088. IEEE, 2023a.
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Wang Xinggang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint 1rXiv:2310.08528 2023b. 3
Rundi Wu, Ben Mildenhall, Philipp Henzler, Keunhong Park, Ruiqi Gao, Daniel Watson, Pratul P Srinivasan, Dor Verbin, Jonathan Barron, Ben Poole, et al. Reconfusion: 3d reconstruction with diffusion priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern 2024. 3
Recognition, pp.
Zirui Wu, Tianyu Liu, Liyi Luo, Zhide Zhong, Jianteng Chen, Hongmin Xiao, Chao Hou, Haozhe Lou, Yuantao Chen, Runyi Yang, Yuxin Huang, Xiaoyu Ye, Zike Yan, Yongliang Shi, Yiyi Liao, and Hao Zhao. Mars: An instance-aware, modular and realistic simulator for autonomous driving. CICAI, 2023c. 2, 3,
Ziyang Xie, Junge Zhang, Wenye Li, Feihu Zhang, and Li Zhang. S-nerf: Neural radiance fields for street views. arXiv preprint arXiv:2303.0072 2023. 2, 3,
Haofei Xu, Songyou Peng, Fangjinhua Wang, Hermann Blum. Daniel Barath, Andreas Geiger, and Mare Pollefeys. Depthsplat: Connecting gaussian splatting and depth. arXiv preprint a41iv:2410. 13862, 2024. 3
14

---

Published as a conference paper at ICLR 2025
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018.
Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. Street gaussians: Modeling dynamic urban scenes with gaussian splatting. In ECCV, 2024. 2, 3, 6, 8, 9, 16
Jiawei Yang. Boris Ivanovic, Or Litany, Xinshuo Weng, Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, et Emernerf: Emergent spatial-temporal scene de- composition via self-supervision. arXiv preprini arXiv:2311.02077, 2023a. 2, 6, 8, 9, 16, 18
Jiazh Yang, Shenyuan Gao, Yihang Qiu, Li Chen, Tianyu Li, Bo Dai, Kashyap Chitta, Penghao Wu, Jia Zeng. Ping Luo, et al. Generalized predictive model for autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14662-14672. 2024a. 2
Lihe Yang. Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv preprint arXiv:2406. 09414, 2024b. 20
Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1389-1399, 2023b.
Zeyu Yang, Hongye Yang, Zijie Pan, Xiatian Zhu, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint arXiv 23 10642, 2023c. 3
Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuging Zhang, and Xiaogang Jin. De- formable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309. 13101, 2023d.
Wangbo Yu, Jinbo Xing, Li Yuan, Wenbo Hu, Xiaoyu Li, Zhipeng Huang, Xiangjun Gao, Tien- Tsin Wong, Ying Shan, and Yonghong Tian. Viewerafter: Taming video diffusion models for high-fidelity novel view synthesis. arXiv preprint arXiv:24099 240.22409000448, 2024a. 3
Zhongrui Yu, Haoran Wang, Jinze Yang, Hanzhang Wang, Zeke Xie, Yunfeng Cai, Jiale Cao, Zhong Ji, and Mingming Sun. Sgd: Street view synthesis with gaussian splatting and diffusion prior. arXiv preprint arXiv:2433.2007, 2024b.
Kai Zhang, Sai Bi, Hao Tan, Yuanbo Xiangli, Nanxuan Zhao, Kalyan Sunkavalli, and Zexiang Xu. Gs-Irm: Large reconstruction model for 3d gaussian splatting. In European Conference Computer Vision, pp. 1-19. Springer, 2025.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018. 6
Hongyu Zhou, Jiahao Shao, Lu Xu, Dongfeng Bai, Weichao Qiu, Bingbing Liu, Yue Wang, Andreas Geiger, and Yiyi Liao. Hugs: Holistic urban 3d scene understanding via gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21336-21345, 2024a.
Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming- Ming-Hsuan Yang. Driv- inggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
21634-21643, 2024b. 2,3,
15

---

Published as a conference paper at ICLR 2025
Frame T
Frame T+10
Frame T+20
Ours
EmerNerr
Street Gaussians
Figure A: Visualization comparison on moving objects. We compare the performance of NVS methods where moving objects are visible in camera views. In the shown case, our proposed method can generate more accurate images of vehicles driving the opposite lane. Images are synthesized on training views.
A APPENDIX
A.1 IMPLEMENTATION DETAILS
Implementation details of FS. We employ ConVNext-TTLLi al., 2022b) encoder to code pseudo-images. For training FreeVS, the diffusion model initialized with Stable Diffusion checkpoints (Rombach et al., 2022). We train the model for 40,000 iterations with a batch size of and video frame length n = 8. We use the AdamW optimizer (Kingma & Ba, 2014) with learning rate 10-4, During training time, we randomly drop the pseudo-image condition latent as well as the CLIP text description latent with a probability of 20%. We enable the viewpoint transformation simulation with a probability of 50%. During inference, we set the number of sampling steps as 25 and stochasticity 111.00 When synthesizing images on the existing trajectory, we set the classifier- free guidance (Ho & Salimans, 2022) guidance scale to 1.0. For synthesizing images on novel cameras and new trajectories, enlarge the CFG guidance scale to 2.0 to strengthen the control of 3D prior conditions over the generated results.
Implementation details of NVS counterparts. We compare novel view synthesis method with the 3DGS(Kerbl et al., EmerNeRF( Yang al., 2023a), and Street Gaussians (Yan et al., 2024). implemented based on their official implementation. For 3DGS which
All counterpart methods is not initially designed for unbounded driving scenes, we largely increase its max training iterations for better convergence of the model. Please check the appendix for on NVS counterparts. For 3DGS which is not initially designed for unbounded driving scenes, we optimize its performance by adjusting its hyperparameters. including setting the densification interval to 500 iterations, setting the opacity reset interval to 10000, and training the 3DGS models for 100000 iterations while densifying 3D Gaussians until 50000 iterations. We also noticed that the Street Gaussians models have convergence issue when training with all 200 frames of each sequence. Therefore, we split the validation sequences into two 100-frame sequences for training the Street Gaussians models, following its official configuration Yan et al., 2024). We resize the input images on WOD from a resolution of 1920 x 1280 to 1248 x 832 and clipped the sky region in images (top 40% area in front views and top 13% area in side views) when training Free and all NVS counterparts. The secudo-iaages are generated in the same resolution. All experiments are conducted on NVIDIA 1.20 GPUs.
implementation details
16

---

Published as a conference paper at ICLR 2025
(a) Training camera
[b] Training pseudo-mage wio temporal augmentaton
(c) Training pseudo-inage temporal
augmentaton
Validation pseudo image original trajectory
Validation pseudo-inage -image shilled braeectoyy
ynthesized image shifted trajectory
Figure B: Visualization of the viewpoint transformation simulation. For a training sample, we show the (a) source camera view, (b) pseudo-image generated from current LiDAR observation, and (c) pseudo-image generated from previous LiDAR observation to simulate viewpoint transforma- tion. We also show a validation case of (d) pseudo-image on the original viewpoint, (c) pseudo- image the shifted viewpoint, and (ff) generated image the shifted viewpoint. Note the image areas with missing or overlapped 3D observation circled by red boxes in (c) and (f). The proposed viewpoint transformation simulation on pseudo-images well-stimulate the insufficient 3D prior observations brought by the shift in viewpoint.
A.2 VIDEO COMPARISON.
We present a video comparison of novel view sequences synthesized by NVS methods on the mod- ified trajectory in driving sequence. Please check the video file submitted as supplementary mate- rial.
A.3 VISUALIATION COMPARISON ON DYNAMIC OBJECTS.
Without the scene reconstruction process, FreeVS is free from complex cross-frame optimization of dynamic objects. FreeVS can synthesize images of dynamic objects in high accuracy, as shown in Fig.A. In comparison, despite with specific design, current reconstruction based NVS methods still suffer from dynamic object modeling.
A.4 VISUALIZATION OF THE VIEWPOINT TRANSFORMATION SIMULATION.
Besides quantitatively ablating the impact of the proposed viewpoint transformation simulation with pseudo-images, also provide visualization case in Fig.B to illustrate its effect.
When facing the translation of viewpoints, pseudo images generated from the existing viewpoints cannot provide complete 3D priors, such as areas circled by red boxes in F(g. To strengthen the robustness of FreeVS towards those patterns in meeuoo-maaees we propose to employ viewpoint transformation simulation with pseudo-images by generating pseudo-images in training views from mismatched LiDAR observations. As shown in F(). we can stimulate the pseudo-image areas with insufficient 3D priors. The generative model trained with the proposed viewpoint transforma- tion simulation with pseudo-image can render images of high quality when facing insufficient 3D prior inputs, as shown in B(g.
A.5 VISUALIZATION RESULTS OF COMPONENT ABLATION EXPERIMENT ON PSEUDO-IMAGE REPRESENTATION.
For the ablation setting (a) and (d) introduced in Sec.4.3, whose quantitative results are reported in Table 4, present visualization comparison in Fig. C. Setting (a) is the baseline setting of feeding diffusion models with sreudo-imagee scene representations for view synthesis. Under setting (d), feed the diffusion model with the reference image, LiDAR point cloud, and the transformation
17

---

Published as a conference paper at ICLR 2025
Pose encoded in pseudo-image
Pose as transforation matrix
Original Viewpoint
Left 2. O.0m
Figure C: Visualization comparison on the encoding of camera viewpoint condition. We qual- itatively compare the image synthesis performance of the diffusion model with pseudo-image as input (Table 4(a)), or with the reference image, LiDAR points, and viewpoint transformation matrix as input (Table 4(d). When feeding model with pose transformation matrices, the diffusion models often fail to generate views on the targeted viewpoint, as shown in the second column.
Table A: Comparison with UC-NeRF on recorded trajectories. We report the performance of FreeVS, UC-NeRF, and EmerNerf when the three frontal view cameras are considered.
Methods
SSIM
PSNR1
LPIPS..
EmerNerf( Yang et al., 2023a)
0.764
25.16
0.282
UC-NeRF(Chenn et al., 2023)
0.770
25.91
0.250
Ours
0.761
25.47
0.146
matrix from the reference view to the target view. As shown in Fig.C, model feed with pseudo- image can precisely synthesize image on the target viewpoint, while model feed with raw camera pose fails to follow the viewpoint condition. Given that the diffusion model can only be trained on recorded trajectories, we found the diffusion model fed with reference images and viewpoint pose tends to overfit the specific camera movement pattern in each camera position. As shown in Fig.C, the model fed with raw 3D prior inputs will only move the frontal camera view forward or backward, ignoring the viewpoint pose condition. This is due the absence of training sample where the frontal camera is moved laterally. By modifying the novel view synthesis task as an image completion task based the pseudo-image representation of 3DD priors well applying the proposed viewpoint transformation simulation with pseudo-images, we can overcome this training data shortage.
A.6 COMPARE WITH METHOODS WITH IMAGE WARPING
The baseline methods in the main paper, such EmerNeRF and StreetGaussian, do not possess spe- cial designs for rendering views out of the recorded trajectories. On the other hand, some previous works including UC-NeRF(Cheng al., 2023) and HO-Gaussian(Li al., 2024b) use image wrap- ping based on predicted depth maps to strengthen their reconstructed results. Such design might alleviate the overfitting problem of reconstruction methods on recorded trajectories, although they are not designed to strengthen the performance of those methods on new trajectories. Here we report the performance of UC-NeRF on our validation segments as supplement to the experiment results in the main paper. We only consider the three frontal cameras in the following supplementary ex- periments, due to UC-NeRF's mostly hard-coded implementations and hyper-parameten optimized for the 3-camera setting. We compare FS with UC-NeRF under the novel-frand synthesis set- ting on the recorded trajectories and under the novel-trajecee trajectory synthesis setting. Considering that dropping the side cameras will have some impact the overall performance of each method,
18

---

Published as a conference paper at ICLR 2025
Table B: Comparison with UC-NeRF on new trajectories. We report the performance of FreeVS. UC-NeRF, and EmerNerf when the three frontal view cameras are considered.
+ 1.0m mAPLET
y + 2.0m
y = 4.0m
Methods
FID4
MAP1.
FID
mAP2.0m 1
FID
mAP 1
EmerNerf
50.58
0.541
62.86
0.506
79.61
0.418
UC-NeRF
42.08
0.592
75,56
0.447
88.47
0.370 24.07
Ours 15.17
0.781
18.49
0.761
0.696
Table C: Performance on the nuScenes dataset. The y axis is defined lateral to the ego vehicle's heading direction.
+ 0.0m
y y 2.0m FID4
y 4.0m FID
Methods
FID4
PVG
65.08 35,32
87.65 39.49
113.41 43.31
Ours
also provide the performance of EmerNerf under the 3-camera setting reference. We report the performance of NVS methods on the recorded trajectories in Table and on new trajectories in Table B. On the recorded trajectories, UC-NeRF and FreeVS have similar performance (UC- NeRF has slightly better SSIM/PSNR performance, while FreeVS has significantly better LPIPS performance.) On the new trajectories, VS significantly outperforms UC-NeRF.
A.7 EXPERIMENT ON THE NUSCENES DATASET
We report the performance of FreeVS on the nuScenes dataset in Table C. We randomly select sequences from the nuScenes dataset for experiments. We also report the performance of PVGGChhen et al., 2023) following its unofficial implementation' on the nuScenes dataset. resize the input images half of the original resolution (800 450) when training Free PVG models trained under full-resolution image inputs. On the nuScenes dataset, FreeVS still demonstrates the ability trajectories. We also show the novel trajectory synthesis
synthesize high-quality images examples of Free on the nuScenes dataset in Fig.
Original
Viewpoint
Viewpeint
TO FIFE
Figure D: Qualitative results on the nuScenes dataset.
19

---

Published as a conference paper at ICLR 2025
Table D: Comparison with NVS counterparts on the XLD dataset. We report the performance of previous NVS methods according to Li ct al. (2024a).
Methods
V- 4.0m
y I 2.0m
V 1.0m
y t 0.0mm.
PSNRT
SSIM1
LPIPS,
PSNRT
SSIM1
LPIPS.
PSNR
SSIM1
LPIPS,
PSNRT
SSIM1
LPIPS.
UC-NeRR
22.89
0.768
0.420
25.17
0.863
0.367
30.07
0.896
0.355
35.95
0.936
0.311
MARS
23.29
0.818
0.235
24.95
0.847
0.194
27.40
0.851
0.169
30.21
0.873
0.146
EmerNeRF
24.80
0.837
0.203
26.05
0.852
0.182
28.66
0.878
0.150
31.76
0.907
0.126
PVG
23.17
0.841
0.353
24.42
0.854
0.335
26.84
0.882
0.296
37.78
0.960
0.189
Ours
26.52
0.865
0.151
27.65
0.876
0.135
29.08
0.894
0.120
30.06
0.908
0.079
A.8 EXPERIMENT ON THE XLD DATASET
The XLD dataset(Li et al., 2024a) is synthesized dataset with ground-truth camera views tra- jectories with 1.0m, 2.0m, or 4.0m offset from the training trajectories. On the XLD dataset, we can evaluate the performance of FreeVS with image-recovering metries including PSNR, SSIM, and LPIPIS. Here we report the performance of FS on XLD dataset on trajectories with 0.0m, 1.0m, 2.0m or 4.0m offsets from the training trajectories in Table D.
Experiments show that Free outperforms all previous methods on all metrics on trajectories with 2m or 4m offsets. On trajectories with Im offsets, the performance of Free is still better than most previous methods. On trajectories with no offset, reconstruction-batt based methods have better PSNR and SSIM performance. Still, we think the high performance of reconstruction-but based methods on the original trajectories comes from their overfitting on the training views, considering the huge performance gap between their performance on trajectories with no offset and trajectories with a slight Im offset (-5.88 PSNR for UC-NeRF,- 10.94 PSNR for PVG).
is also worth emphasizing that FreeVS has a significantly better LPIPS performance on all valida- tion trajectories, even including trajectories with no offsets. As perceptual metric. LPIPS is more aligned with human perception compared with pixel-error metrics such as PSNR/SSIM. As the per- formance degradation of the reconstruction- based methods mainly comes from artifacts in the image when facing out-of-domain test views, the performance degradation of FreeVS mainly comes from the loss of high- tighffrequency details such as the brick patterns on wall or the number of leaves tree. We believe this is why FS exhibits significantly better visual appeal in visualized results.
We also present qualitative comparison between FreeVS and reconstruction- based methods in Figure F. We compare the generation results of FreeVS with all visualization results shown in Figure 3 in the XLD dataset paper(Li et al., 2024a).
A.9 LIMITATION DISCUSSION
As discussed above, the pseudo-image representation of 3D priors can handle rigid dynamic objects like vehicles well by accumulating their 3D points along their moving trajectory. On the other hand, the pseudo-image representation cannot handle non-rigid dynamic objects (like pedestrians and cyclists) as well as rigid objects since their 3D points are hard to be correctly accumulated across frames. However, thanks to the generalization ability of video generation models, views of pedestrians and cyclists synthesized by VS are still satisfying as shown in Figure 5 in the paper.
As we use ground truth LiDAR points for the main experiments, we found the LiDAR inputs for FS can be replaced by pseudo-LiDDA points generated from estimated depths. We experi- mented with applying an off-the-shelf Depth-Anything V2 (Yngg al., 2024b) model on reference images and sample pseudo LiDAR points from the obtained depth map. Here we show a success case as well failure case of generation with pseudo LiDAR points in Figure ??. According to our observation, the proposed FSee pipeline can be applied to pseudo LiDAR points and recover most scene contents correctly. However, we found the depth prediction model sometime has problems in predicting the depth of objects very near to the camera, as shown in the failure case, circled with red. We believe this issue is caused by directly applying the dataset. A depth predictor that is fully converged on the WOD dataset should be able to avoid this problem. As for VS, the visualized results show that our method has satisfying performance on most scene contents with about the right depth.
V2 model to the WOD
20

---

Published as a conference paper at ICLR 2025
(+) CFG scale =0.1
(b) CFG scale 0.5
(c) CFG scale =1.0
scale =20
(e) scale
(f) CFG scale
Figure Visualization of the effect of classifier-free guidance (CFG). We show generation results with the same imput pseudo-image and different CFG scales. The larger CFG scale is set, the stronger the generation result constrained by the 3D prior condition.
Note that we generate the above-provided results by directly applying the FreeVS model trained with GT LiDAR points the pseudo LiDAR points. The different valid pixel patterns and pixel density of pseudo image generated from pseudo LiDAR points lead to some slight imaging style changes of the generated result, which is totally avoidable by training VS on pseudo image generated from pseudo LiDAR points.
A. EFFECT OF CLASSIFIER- GUIDANCE
As a diffusion model, FS can use the classifier-free guidance (CFG) technique to adjust the control effect of input 3D prior condtions. We show the impact of CFG with different CFG scales in Fig. E.
A.11 VALIDATION SEQUENCES
We list the selected validation sequences from the WOD dataset here with their official individual file names:
baattthcamrr_labaae .frecodd
segment-
222.2882255333394444976322222202 wtt.://.......00.ii...mmeeerrea. haamera.labels.trreaeec
segment-
(6M116666611111912220099999 2480 000 2500 000 with camera labels.tfrrecore
1ggneen=111177777706600222222222 000.3900. 19000000_wwe wwth00000..hhh.0.
100.with.cammeaa camera .freood,
[ggneem||1B444391152224666601333 3560.000.withh camera waw.merr.lablls.ffeecoooo
[ggmeett||8111887798871113388383 000_340_000.
$300.with.camerallbbe .freecodd
sggmeenn_11055552229933133333303 1924 000. 1944 000 with camera labels.tfrecord
sggmemm-11211122244866677795555 1813 380 1833 380 with camera labels.tfrecord,
A.12 EFFICIENCY COMPARISON WITH GENERALIZABLE RECONSTRUCTION METHODS.
Here we provide an efficiency comparison between FreeVSS and generalizable 3D reconstruction methods in Table E. Note that report the performance of generalizable 3D reconstruction methods in the PandaSet dataset according to (Chen et al., 2025b), and the performance FreeVS in the WOD dataset. The PandaSet dataset consists of video sequences in 80 frames (on average) and
21

---

Published as a conference paper at ICLR 2025
cameras, while WOD dataset consists of video sequences in 200 frames and cameras, therefore the comparison experiment setting is not completely aligned. The 3D scale of scenes in PandaSet WOD seems similar. The training time cost, reconstruction time cost, and rendering FPS are measured on RTX 3090 GPU according to (Chen et al., 2025b), and the computational costs of VS are measured single NVIDIA 1.20 GPU.
Table E: Efficiency comparison with generalizable reconstruction methods. t: methods that need to reconstruct the scene again with different source images when rendering each new view. *; The training costs of generalizable reconstruction methods are measured while the training cost of FS is measured on 8 NVIDIA L20 GPUs. Samely, the inference efficiency of previous methods FreeVS is measured on 3090 / L20 GPU.
RTX 3090 GPUs,
Methods
Train Time
Train mem
Recon mem
Recon time
Render FPS
ENcRF(Lin et al., 2022)
108h#2
24GB
10GB
0.11st
2.65
GNT(Varma
GNT Varrm et al., 2022)
49h
23GB
21GB
0.35sf
0.00249
PixelSplat(Charee Charatan al., 2024)
110h*2
48GB
11GB
1.14s
176
G3R(Chen al., 2025b)
60h
20GB
24GB
210s
97.0
FreeVS
40h*8
45GB
14GB
0.9
22

---

Published as a conference paper at ICLR 2025
Offseet-m
FreeVS
UC-NeRF
EmerNerf
GT
PVG
GaussianPro
Offset-2m
FS
UC-NeRF
EmerNerf
GT
PVG
GaussianPro
Offset-4m
FreeVS
UC-NeRF
EmerNerf
GT
PVG
CaussianPPo
Figure F: Visualization comparison with NVS counterparts on the XLD dataset. We qualita- tively compare the image synthesis performance of Free VS with previous NVS methods on the XLD dataset. Visualization results of previous NVS methods are referenced from Li et al. (2024a).
23


Reviews:
[{"review_id": "7l7zXW7cVk", "reviewer": "Reviewer_6Zxi", "summary": "The authors present FreeVS -- a Video Stable Diffusion-based generative view synthesis method for driving scenes that can synthesize high-quality camera views both on and beyond recorded trajectories. \n\nThe key innovation is using pseudo-images created by projecting colored point clouds as a unified representation for view priors. As opposed to recent contenders that rely on gaussian splatting or nerfs to represent the scene, the authors train a diffusion model on colored LiDAR point clouds.\n\nThe authors introduce two new benchmarks to evaluate the generated images that are far from the original poses. The method outperforms previous approaches for both traditional and newly proposed benchmarks on the Waymo dataset.", "questions": "1. Why is there no ablation study for **no priors**? This should be as close as possible to the vanilla Video Stable Diffusion.\n2. Have you experimented with LiDAR noise / pseudo lidar from MDE methods or maybe a mesh-based method? Otherwise this pipeline is bound to an expensive data acquisition pipeline.\n3. Training/inference times are unclear. How can it be that your method is faster than splatting-based methods at 0.9 FPS? The training time should also be discussed but it's not.\n4. Could moving objects benefit from a special treatment? I.e, tracking/ inpainting. It doesn't look like the method handles well uncertain areas.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "### Novelty\nClever use of pseudo-images obtained through colored point cloud projection as a unified representation for all view priors, simplifying the learning objective for the generative model.\n### Evaluation\nIntroduces two new challenging benchmarks - novel camera synthesis and novel trajectory synthesis.\n\n### Efficiency\nThe authors claim it takes less computational resources at inference time compared to splatting-based models.\n\n### Performance\nBetter performance versus contenting methods, especially on poses far away from the original camera poses.", "weaknesses": "### Novelty\nEngineering work -- it boils down to an addon for Video Stable Diffusion that has colored LiDAR point features concatenated.\n\n\n### Different type of artfacts\nThe method trades the gaussian and nerf artifacts with the diffusion ones. While there is no denying that FreeVS works better than the previous attempts from novel views, for single front view, splatting still yields significantly better results (Table 3, front view).\n\n### Evaluation\nA single dataset is benchmarked (Waymo Open Dataset).\n\n### Paper quality\nSome tables missing numbers -- Table 3, reconstruction time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IHkarajXuv", "reviewer": "Reviewer_YUkf", "summary": "This paper approaches the task of novel view synthesis of outside-trajectory viewpoints on driving videos. It does so by training a conditional video diffusion model on outside-trajectory views created through projection of existing 3D point clouds. It evaluates generated images outside trajectories using off-the-shelf models. Compared to baselines StreetGaussian, EmerNeRF and 3D-GS, it shows superior results on novel camera synthesis, multi-view novel frame synthesis, and outstanding results on novel trajectory synthesis. Qualitative video results show a significant improvement over baselines.", "questions": "See Weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Impressive results on a challenging task\n- Results are clearly better than baselines in novel camera synthesis, multi-view novel frame synthesis, and significantly better in novel trajectory synthesis (FID drops by towards 75%)\n- Qualitative comparisons clearly back these results\n- Video results show impressive generation well outside input trajectory, while other methods have severe artifacts / fail entirely\n\nCreative use of 3D and off-the-shelf models to enable a non-conventional setup\n- Novel View Synthesis is so often limited to input trajectories. In the case of cars, this makes the task fairly straightforward and limited due to constraints on using positions connected to the car.\n- Instead, this work approaches prediction several meters away from car trajectories. It does so by utilizing colored LiDAR across multiple views to create point clouds it can project into pseudo-images. This is a nontrivial trick to implement effectively. This work shows it can be useful for training a generative model!\n- Evaluation is also tricky in the pseudo-image setting, but FID and 3D Detection mAP are suitable metrics; while of course qualitative results are most important.\n\n\n***Post-Rebuttal Update*** I leave my score at 6. The reviewers did a good job of addressing my concerns and I feel the paper should be accepted as it offers good contributions in 3D and video generation to yield an effective method. See my response to the rebuttal for more detail.", "weaknesses": "Could use clearer argument for method leading to performance gain\n- Numbers in the ablations table do not match that in comparisons to baselines. Why not?\n- Ablations show little impact on performance. When the FID of this method is less than a third of that of baselines, surely more than 10% of performance can be explained by choices. For example, how does training data impact performance? What about pretraining or architecture? If these are important, it feels the architecture should be described in more detail. \n- I infer a lot of the performance is coming from training data, yet the main paper has little information about this. How big is the train set in terms of sequences, if they are chosen from WOD? \n- The alternative explanation I fear is most of the performance is coming from Stable Video Diffusion. It should be very clear how strong this baseline is without the proposed contributions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SoH8FKpi1w", "reviewer": "Reviewer_NG25", "summary": "This work focus on novel view synthesis on extrpolated view (i.e. rendered views are far away from source training views).\nThe authors propose to use reprojected colored lidar points as condition, using a freezed diffusion decoder to achieve high quality synthesized images.\nExperiments are conducted on Waymo, demonstrating way better results over other 3d-optimization based approaches (emernerf, streetgaussian)", "questions": "This idea is quite similar to Free View Synthesis which both project source views into target view, you may consider add it as a baseline, at least cite it.\n\n@article{riegler2020free,\n  title     = {Free View Synthesis},\n  author    = {Gernot Riegler and V. Koltun},\n  journal   = {European Conference on Computer Vision},\n  year      = {2020},\n  doi       = {10.1007/978-3-030-58529-7_37},\n  bibSource = {Semantic Scholar https://www.semanticscholar.org/paper/49fae04a4e9383080788759f63dba75c86bd21b0}\n}\n\n\nYou may also want to cite magicDrive and MagicDrive3D, as both work on generative driving scenes.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Much better (more robust) results over 3D-optimization based approaches (EmerNerf, streetgaussian) on far-away novel veiws, because they typically have overfitting issues.\n2. A combination of 3D informtion and 2D diffusion  model that provides both controllability and decent rendering results.", "weaknesses": "1. The rendering speed is very slow,  while 3DGS which can render at real time (50+fps), this hinder the downstream applications that requires realtime efficiency.\n2. Inconsistency results because of using large decoder\n3. Worse performance compared to 3D-optimization approaches if the novel view are close to source views;", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v4buSBohxQ", "reviewer": "Reviewer_zGyV", "summary": "The paper introduces FreeVS, an approach to view synthesis for driving scenes that overcomes limitations of existing methods, which primarily focus on synthesizing camera views along pre-recorded vehicle trajectories. Traditional methods tend to degrade in performance when synthesizing viewpoints deviate from the recorded trajectory, as these viewpoints lack of groundtruth data. \n\nTo ensure that the generated images remain consistent with the 3D structure of the scene and accurate in terms of viewpoint pose, the authors introduce a pseudo-image representation of view priors based on LiDAR. This representation controls the generation process and allows for the simulation of viewpoint transformations, enabling the model to mimic camera movement in different directions.\n\nThe authors proposed two benchmarks for evaluating novel camera synthesis and novel trajectory synthesis. The proposed method is evaluated on Waymo Open Dataset.", "questions": "See above. I personally like the idea of the paper, but I still have many concerns and would provide a final rating based on the authors' responses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well motivated, shows great potential for the real-application of autonomous driving simulation.\n- Proposed a \"psuedo LiDAR controlnet\" for SVD, which is easy yet effective.\n- The experimental results and demo video demonstrate the effectiveness of the proposed method.", "weaknesses": "- The evaluation is not very comprehensive. The baseline methods are not *specifically* designed for the similar purpose of the paper. 3D-GS is the basic method, EmerNeRF and StreetGaussian are more for the dynamic NeRF/3DGS. There are works that use virtual warping for improving the novel view quality such as [1] [2], that might be better for the baselines.\n- The benchmark of novel trajectory synthesis looks interesting to me, however, the authors only show the FID results, while FID is well-known for its instability and unreliability by changing the resolutions, etc. This reminds me of the existing novel trajectory synthesis benchmark [3], the authors should test their methods on such a dataset and demonstrate the *absolute* performance gain using the metrics of PSNR, SSIM, etc.\n- I would like to further know the zero-shot generalization of the trained SVD, since the Waymo Open dataset is quite clean for its camera/LiDAR extrinsic calibration, and the number of LiDAR beams is relatively high, it is interesting to show the results of genelizating it to other scenes or datasets, and how the SVD benefits the downstream 3DGS reconstruction.\n- What are the results of applying the proposed methods on dynamic scenes?\n\n[1] UC-NeRF: Neural Radiance Field for Under-Calibrated Multi-view Cameras in Autonomous Driving, ICLR 2024.\n\n[2] HO-Gaussian: Hybrid Optimization of 3D Gaussian Splatting for Urban Scenes, ECCV 2024.\n\n[3] XLD: A Cross-Lane Dataset for Benchmarking Novel Driving View Synthesis, arxiv, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1ivxb6jncW", "reviewer": "Reviewer_BdtD", "summary": "This paper addresses the free-view synthesis problem by training a diffusion model conditioned on synthetic images through point cloud projection. During training, it project colored LiDAR points from nearby views into the reference view as a condition image, and finetune a diffusion model so that it can generate the reference view. During inference, condition image can be synthesized for a novel view, which is taken by the diffusion model to generate a realistic image of that view.", "questions": "See weakness.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is easy to read and the idea is clear.\n\nThe method is solid and reasonable, potentially showing good generalizability without training very big models (e.g. video foundation model). Conditioning on pseudo-images can help improve 3D consistency.", "weaknesses": "1. the proposed method seems too normal, and I read tons of paper doing similar things -- fine-tune the result based on a synthesized image. The authors also mentioned several such works in the related work. The only difference seems to be that they deal with objects while the authors deal with the driving scene.\n\n2. it seems that the paper did not seriously consider temporal information. There is no way to ensure temporal consistency. Also, it seems that moving objects can cause inconsistency and troubles during pseudo image synthesis while the authors only have one sentence discussing how to address it. I believe it can cause a lot of troubles.\n\n3. There is no discussion about the requirement of the pseudo images. For examples, if the novel view is far from captured views, it surely can cause problems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)