Efficient Depth-Guided Urban View Synthesis
Sheng Miao1∗ , Jiaxin Huang1∗ , Dongfeng Bai2 , Weichao Qiu2 , Bingbing Liu2 ,Andreas Geiger3,4 , and Yiyi Liao1B
1

Huawei Noah’s Ark Lab 3 University of Tübingen
4
Tübingen AI Center
https://xdimlab.github.io/EDUS/

Zhejiang University

2

Abstract. Recent advances in implicit scene representation enable highfidelity street view novel view synthesis. However, existing methods optimize a neural radiance field for each scene, relying heavily on dense
training images and extensive computation resources. To mitigate thisshortcoming, we introduce a new method called Efficient Depth-Guided
Urban View Synthesis (EDUS) for fast feed-forward inference and efficient per-scene fine-tuning. Different from prior generalizable methods
that infer geometry based on feature matching, EDUS leverages noisy
predicted geometric priors as guidance to enable generalizable urban view
synthesis from sparse input images. The geometric priors allow us to apply our generalizable model directly in the 3D space, gaining robustness
across various sparsity levels. Through comprehensive experiments on
the KITTI-360 and Waymo datasets, we demonstrate promising generalization abilities on novel street scenes. Moreover, our results indicatethat EDUS achieves state-of-the-art performance in sparse view settings
when combined with fast test-time optimization.
Keywords: urban view synthesis, generalizable NeRF, sparse view1

Introduction

Novel View Synthesis (NVS) of street scenes is a key issue in autonomous driving and robotics. Recently, Neural Radiance Fields (NeRF) [25] have evolved asa popular scene representation for NVS. Several works [30, 43, 46, 50, 54] subsequently demonstrate promising NVS performance for street scenes.
However, these techniques are not applicable to sparse image settings, as
they are typical for autonomous driving. When vehicles move at high speeds, a
substantial portion of the content is captured from merely two or three viewpoints. This insufficient overlap between consecutive views significantly hinders
the performance of existing methods. Furthermore, as the cameras are in forward
motion, the parallax angle between frames is typically small. This reduction inparallax angle further amplifies the uncertainty of reconstruction.
Several recent methods address the sparse view setting: One line of works[6,33,41] exploits external geometric priors or regularization terms, such as depth⋆

Equally contributed. B Corresponding author.

2

S.Miao, J.Huang et al.

Fig. 1: Illustration. Left: Most existing generalizable NeRF methods rely on feature
matching for recovering the geometry, e.g., by constructing local cost volumes, potentially overfitting certain reference camera pose distributions. Right: Our method lifts
geometric priors to the 3D space and fuses them into a global volume to be processedby a generalizable network. This enhances robustness as the geometric priors are unaffected by the reference image poses. We show synthesized images and depth maps
through a feed-forward inference in the middle.

maps, geometric continuity, and color cues, to enhance model performance. However, these approaches require per-scene optimization, which may take up to afew hours to converge on a single scene.
Another line of research [4, 14, 44, 57], known as generalizable NeRF, resorts
to pre-training on large-scale datasets to gain domain-specific prior knowledge.
In new scenarios, these data-driven methods are able to efficiently infer novel
views from sparse inputs in a feed-forward manner, with the option for additional test-time optimization. Most of these methods rely on feature matching
to learn the geometry, which suffers from insufficient overlap and texturelessregions given sparse street views shown in Fig. 1. Furthermore, these methods
typically retrieve the nearest reference images for feature matching, potentiallyleading to overfitting to specific camera pose configurations. This hampers theirgeneralization capability to unseen sparsity levels and various lengths of camera
baselines, especially in highly sparse settings. We hence ask the following question: Can we develop an efficient and generalizable urban view synthesis method
capable of robustly handling diverse sparsity levels?
To tackle this challenge, we propose Efficient Depth-Guided Urban View
Synthesis (EDUS) which mitigates overfitting to a certain reference image poseconfiguration. This is achieved by lifting the reference images to 3D space utilizing approximate geometric priors, which are then processed by a feed-forward
network directly applied in 3D space. In contrast to existing generalizable NeRF
methods relying on feature matching for geometry prediction, whose performance is dependent on the relative poses between reference images and the target
frame [4, 44], our method explicitly leverages geometric priors, e.g., monoculardepth. These priors remain unaffected by the reference image poses, making our
method more robust to various sparsity levels. In addition, our method allows
for accumulating geometric priors of reference frames within a large window(60 meters) to learn a generalizable model in this global volume. This further
enables efficient per-scene fine-tuning by directly updating the global volume,

Efficient Depth-Guided Urban View Synthesis

3

circumventing the need to fine-tune numerous local volumes [4] or the entire
network [44, 47].
In order to leverage the geometric prior within a limited distance while being
able to perform feed-forward inference for the full unbounded scene, we further
decompose the scene into three components: foreground regions, background,and the sky, with a separate generalizable module designed for each. Specifically,
assuming a noisy point cloud of the foreground regions is provided by accumulating monocular/stereo depth estimations of the reference views, we train ageneralizable 3D CNN to predict a feature volume. This 3D-to-3D generalizable
module enhances robustness against various reference pose configurations as its
input is the point cloud represented in world coordinates. While the 3D CNN
leads to good geometry prediction, it encounters challenges in capturing highfrequency appearance details due to its inherent smoothness bias. To address thislimitation, we augment the model with 2D features retrieved from nearby input
views, enhancing the model’s ability to predict detailed appearance. Finally, weleverage image-based rendering for the background and sky.
Our contributions in this paper can be summarized as follows: 1) We investigate the utility of noisy geometric priors and 3D-to-3D refinement for generalizable NeRF models, achieving promising feed-forward inference on various sparse
urban settings. Our global volume-based representation further allows for efficient per-scene optimization which converges within five minutes. 2) We build ourgeneralizable model tailored for unbounded street scenes by utilizing three separate generalizable components for the foreground, background, and the sky. This
allows us to leverage the geometric priors within limited distances while being
able to represent the full unbounded scene. 3) Our proposed approach outperforms baselines for sparse novel view synthesis on KITTI-360 dataset, including
geometric regularization-based methods and generalization-based methods. We
further demonstrate that our model trained on KITTI-360 shows convincing
generalization performance on the Waymo dataset.

2

Related Work

Novel View Synthesis: Neural Radiance Fields [25] revolutionized the taskof Novel View Synthesis (NVS). Many follow-up works improve NeRF to achieve
real-time rendering, fast training [7,22,35,36], scene editing [11], etc. In this work,
we focus on generalizable NeRF on street views which has many applications inautonomous driving.
Street Scene NeRF: There are a few NeRF-based methods carefully designed
to deal with street scenes [19,27,30,46]. One line of methods [30,46,54] combines
LiDAR and RGB data for street scene reconstruction. In this work, we optionallyleverage LiDAR in the generalizable training stage and consider an RGB-only
setting for test scenes. Another line of methods consider the street scenes basedon RGB images [8, 9, 19, 27, 45]. All aforementioned methods takes relatively
dense images as input and consider per-scene optimization. Our method, in contrast, focuses on generalizable street scenes NeRF which is applicable to various

4

S.Miao, J.Huang et al.

sparsity levels. Methods for unbounded scene representation are also relevant
for street scenes [2, 3, 24, 43]. Our method takes inspiration from these per-scene
optimization methods and decomposes the scene into foreground, background,and sky with a generalizable module for each.
Sparse View NeRF: One line of works leverages depth or appearance regularization [6, 15, 26, 29, 33, 41, 51] to address the challenging task of sparse view
synthesis. While leading to improvement in performance, these methods sufferfrom long training time. In contrast, our method takes the geometric priors as
input to train a generalizable NeRF, enabling feed-forward inference and fast perscene optimization. Generalizable NeRF approaches are also intensively studied
in the literature [4, 16, 23, 44, 47, 48, 57]. Many works are built on image-based
rendering [44,57] or local cost volume [4,16,23]. As suggested in the current work
MuRF [47], these methods struggle to generalize well to large baselines. MuRF
tackles this issue by constructing a cost volume at the target view and achieves
strong generalization to multiple baselines. However, it requires a long time for
per-scene optimization, as the full computationally expensive model needs tobe fine-tuned. Our method is more closely related to PointNeRF [48] and NeO
360 [14], both reconstructing a scene representation in a global world coordinate. PointNeRF leverages point clouds from a multi-view stereo algorithm andbuilds a point-based radiance field. We instead apply a 3D CNN to the input
point cloud for refinement, allowing for better toleration of noisy point cloud
input. NeO 360 [14] proposes to learn a tri-plane-based global scene representation from sparse RGB images, enabling 360◦ on the synthetic dataset. However,when it comes to complex street scenes with forward-moving cameras, NeO 360
struggles to directly predict the global scene representation. Our method tackles
this challenge by incorporating explicit geometric priors.
NVS with Point Cloud: There are many existing NVS methods that take
explicit geometric priors into account. [17,48] learn a scene starting from a coarse
point cloud, which can be obtained from multi-view stereo techniques [32] orlearning-based methods [52, 55, 56] by unprojecting depth maps. These pointbased works combine explicit geometric priors and implicit networks to regresscolor and other attributes of each point in the point cloud, and then render 2D
images through volume rendering or rasterization. Other methods [20,31] project
the point cloud into feature images first and then learn a model to fill holes duringneural rendering. Instead, we only take as input the point cloud, extracting
geometry and appearance guidance for the generalizable reconstruction.
Some methods utilize encoders to process point clouds, followed by a volumebased neural rendering procedure. [13,58] mainly focus on pre-trained point cloud
encoder for downstream tasks, and do not consider cases like sparse street views
into consideration. [49] reconstructs a driving scene well with either 2D images or
3D point cloud inputs, but it is limited by per-scene optimization. In contrast,
our work is a generalizable method designed for autonomous driving scenes,
which has a fast convergence speed and can obtain good reconstruction results
even in extremely sparse viewpoints with fine-tuning.

Efficient Depth-Guided Urban View Synthesis

5

Fig. 2: EDUS. Our model takes as input sparse reference images and renders a color
image at a target view point. This is achieved by decomposing the scene into three
generalizable modules: 1) depth-guided generalizable foreground fields to model thescene within a foreground volume; 2) generalizable background fields to model thebackground objects and stuff; and 3) generalizable sky fields. Our model is trained on
multiple street scenes using RGB supervision Lrgb and optionally LiDAR supervision
Llidar . We further apply Lsky to decompose the sky from the other regions and Lentropy
to penalize semi-transparent reconstructions.

3

MethodThis paper explores the learning of a generalizable Neural Radiance Field (NeRF)
for rendering novel views from sparse reference images in unbounded street
scenes, with the option of test-time optimization. An overview of our EDUS ispresented in Fig. 2. Initially, we partition the scene into foreground, background,and sky components. This partitioning facilitates the characterization of foreground areas with restricted extent utilizing a 3D-to-3D adaptable framework,
while the remaining distant regions are modeled through image rendering techniques. Throughout this paper, the term foreground denotes the region confinedwithin a predetermined volume, while background encompasses substances andobjects outside of this volume, excluding the sky. We develop distinct generalizable modules for each component. The core of our method is the depth-guided
generalizable foreground field, effectively combining noisy geometric priors andimage-based rendering for representing close-range regions (Sec. 3.1). We further exploit image-based rendering to model the background regions and the sky(Sec. 3.2). Finally, these three parts are composed to represent the unbounded
street scenes (Sec. 3.3). Our method is trained on multiple street scenes andenables feed-forward NVS on unseen validation scenes, with optional fine-tuning
for further improvement (Sec. 3.4).
3.1

Depth-Guided Generalizable Foreground Fields

Depth Estimation: We explore geometric priors of existing powerful depth
estimation methods to enhance our generalizable NeRF for foreground regions,

6

S.Miao, J.Huang et al.

i.e., regions within a close-range volume. Given N input images {Ii }N
i=0 for asingle scene, we exploit off-the-shelf depth estimators to predict metric depth
maps {Di }N
i=0 . The depth estimator can be a stereo depth model [34] when the
input contains stereo pairs. Otherwise, we adopt a metric monocular depth estimator [56] which provides less accurate but plausible metric depth estimations.
Point Cloud Accumulation: With the provided camera intrinsic matrices
N{Ki }Ni=0 and camera poses {Ti }i=0 , we unproject the predicted depth maps
N{Di }i=0 into 3D space, and accumulate them in the world coordinate system toform a scene point cloud P ∈ RNp ×3 . Specifically, for each pixel with homogeneous coordinate u froma given frame, its world coordinate x is computed as:
x = dRi K−1 u + ti , where d denotes the pixel’s estimated depth and (Ri , ti )is derived from the ith frame’s cam-to-world transformation Ti . Additionally, weassign a 3-channel color value to each unprojected point by retrieving the color
from the RGB images. Note that we keep points that fall within the foregroundvolume and omit the rest, as depth predictions are usually untrustworthy at far
regions.
We also utilize depth consistency check to filter noise. Specifically, we unproject a depth map Di of ith frame to 3D and reproject it to a nearby view j,obtaining a projected depth Di→j . Next, we compare Di→j and Dj , and maskout pixels if the difference exceeds an empirical threshold σ = 0.2m.
Modulation-based 3D Feature Extraction: The resulting noisy foregroundpoint cloud serves as input for our generalizable feature extraction network fθ3D .
We first discretize the point cloud P into a volume P ∈ RH×W ×D×3 , where
H ×W ×D denotes the spatial resolution. The discretized point cloud is mapped
to a feature volume F ∈ RH×W ×D×F , where F denotes the feature dimension:
  {\bF }=f_\theta ^\text {3D}(\bP ). 

(1)

Notably, the design choice of fθ3D affects the generalization performance. We
experimentally observe that a conventional 3D U-Net with encoder-decoder architecture results in blurry artifacts when generalizes to new scenes, as shownin supplementary material. Instead, we draw inspiration from [28, 53] to employ
a 3D Spatially-Adaptive Normalization convolutional neural network (SPADE
CNN) to modulate feature volume generation at multiple scales, which is a popular strategy in semantic image synthesis. More specifically, our SPADE CNN
discards the encoder part and comprises 3 SPADE residual blocks and upsampling layers. For each block, SPADE CNN downsamples the input volume P tothe corresponding spatial resolution and uses it to modulate layer activations
with learned scale and bias parameters. We hypothesize that SPADE CNN ismore effective in preserving the appearance information encoded in P thanks
to its multi-resolution modulation. Given a sampled point x ∈ R3 in the 3D
3D
space, we retrieve its feature fbg∈ RF from F using trilinear interpolation.
More network details are provided in the supplementary materials.
Image-based 2D Feature Retrieval: While the SPADE CNN takes as input
the noisy depth predictions and achieves reasonable generalization, we observe
3D
that using fbg
alone struggles to provide high-frequency details in appearance.

Efficient Depth-Guided Urban View Synthesis

7

This is due to two reasons: 1) the point cloud discretization limits the spatial
resolution; 2) the 3D CNN suffers from an inductive smoothness bias. Therefore,
we propose to additionally exploit the idea of image-based rendering to achieve
better generalization of the appearance. Inspired by [44], we first select n nearest
neighboring views from the target view based on the distances of the camera
poses. This forms a group of nearest reference images Î = {Ik }K
k=0 . For eachsample point x, we project it to the reference frame in Î. Next, we retrieve thecolors from the reference frames based on bilinear interpolation, and concatenate
2D
them to form a 2D feature vector ffg∈ R3K . We set K = 3 in this paper.
Color and Density Decoder: Given a 3D point confined to the foregroundvolume, we predict its density σfg and color cfg based on the volumetric feature
3D
2D
ffg
and the 2D feature ffg. More specifically, the density σfg is predicted by
3D
applying a tiny MLP gθ to the volumetric feature ffg.
3D
For prediction of the color cfg , we concatenate ffg and ff2D
g and feed theminto a six-layer MLP. The 3D location x and the viewing direction d are further
considered as input. This process can be expressed as:  \sigma _\text {fg}=g_\theta (\bff _\text {fg}^\text {3D}) , \quad \bc _\text {fg} =h_\theta (\bff _\text {fg}^\text {3D},\bff _\text {fg}^\text {2D},\gamma (\bx ),\bd ) \label {eq:foreground} 

(2)

Different from the original NeRF [25], we apply positional encoding γ(·) to the location x but not to the viewing direction d as we expect non-Lambertion objects
2D
to be mainly modeled by the 2D feature ffg. Please refer to the supplementarymaterial for more details.
3.2

Generalizable Background and Sky Fields

As mentioned above, the foreground volume only covers a limited street scene
and falls short in representing the distant landscape (often hundreds of metersaway) and the sky. Therefore, we model the background region and the sky
separately.
Image-based Background Modelling: An object placed outside the foreground volume is regarded as a background landscape. Due to perspective projection, distant objects occupy only a small portion of the image and have lessdetailed appearance. In this case, we observe that image-based rendering is sufficient to faithfully reconstruct the background regions as relative depth changes
are small. Thus, we employ an MLP hbg
θ to jointly predict the background density
2D
σbg and the color cbg based on 2D image-based feature fbg:
  \sigma _\text {bg}, \bc _\text {bg} =h_\theta ^{bg}(\bff _\text {bg}^\text {2D},\gamma (\bx ),\bd ) \label {eq:background} (3)

where x is spatial location and d is the view direction. Same to the foreground,2D
we retrieve color from nearby images and concatenate them to form vector fbg.
Sky Modelling: Street scenes invariably contain the infinite sky region whererays do not collide with any physical objects, leading to minimal changes inappearance while moving forward. Thus, we omit the positional influence andrepresent the sky as a view-dependent environment map. Given that the sky’s

8

S.Miao, J.Huang et al.

Sky

Background

Full Image

Fig. 3: Scene Decomposition Visual Results.

appearance between consecutive frames is remarkably similar, we retrieve the 2D
2D
image feature fsky
from reference frames, as we discussed in Sec. 3.1, and thenblend the sky color using a single-layer MLP hsky
θ conditioned on view directions:  \mathbf {c}_{\text {sky}} = h_{\theta }^{\text {sky}}(\mathbf {f}_{\text {sky}}^{\text {2D}}, \mathbf {d}). 
3.3

(4)

Scene Decomposition

The composition of the foreground, background, and sky leads to our full modelfor the unbounded street scenes, as shown in Fig. 3. Here, we elaborate on thecompositional volume rendering for each ray. To render a pixel, we cast a rayr = ro +td from the camera center ro along its view direction d. We sample points
along the ray to query their color and density. The color and density of points
falling within the foreground volume are queried through Eq. (2), otherwise thebackground model in Eq. (3) is applied.
We accumulate all foreground and background points to obtain the color andthe accumulated alpha mask:
  {\bC }^{(\mathrm {fg} + \mathrm {bg})}=\sum _{i=1}^{M} T_k\left (1-\exp \left (-\sigma _i\right )\right ) \mathbf {c}_i, \quad {\alpha }^{(\mathrm {fg} + \mathrm {bg})}=\sum _{i=1}^{M} T_k\left (1-\exp \left (-\sigma _i\right )\right ) 

(5)

 P

i−1
where Ti = exp − j=1 σj denotes the accumulated transmittance. Finally,
we add the sky color of the ray assuming it is located at an infinite distance:
  \bC = {\bC }^{(\mathrm {fg} + \mathrm {bg})} + (1-\alpha ^{(\mathrm {fg} + \mathrm {bg})} )\bc _\text {sky} 

3.4

(6)

Training and Fine-tuning

we use multiple training losses to train on different scenes and (optional) testtime optimization on a specific scene. The most common sensors used in selfdriving cars are cameras and LiDAR. Compared to cameras, LiDAR data are
more expensive to acquire and may not be deployed in unseen test scenarios.
As a result, we use LiDAR information only for training, to enhance geometric
understanding of the model. This leads to the training loss function and finetuning loss functions as follows:  \mathcal {L}_{\text {training }} & =\mathcal {L}_{\text {rgb}}+\lambda _1 \mathcal {L}_{\text {lidar}}+\lambda _2 \mathcal {L}_{\text {sky}}+\lambda _3 \mathcal {L}_{\text {entropy }} \\ \mathcal {L}_{\text {fine-tuning }} &=\mathcal {L}_{\text {rgb}}+\lambda _2 \mathcal {L}_{\text {sky}}+\lambda _3 \mathcal {L}_{\text {entropy }}(8)

Efficient Depth-Guided Urban View Synthesis

9

We now elaborate on the loss terms.
Photometric loss: We use the L2 loss to minimize the photometric loss between the rendered colors C and ground truth pixel color C:  \mathcal {L}_{\text {rgb}}=\left \|{\bC }-\bC _{gt}\right \|_2^2 (9)Sky loss: In order to separate the sky from the foreground and backgroundsolid structures, we leverage a pre-trained segmentation model [40] to provide
pseudo sky mask M sky , and supervise the rendered sky mask with binary crossentropy (BCE) loss:  \mathcal {L}_{\text {sky}} = \text {BCE}\left (1-\alpha ^{(\text {fg}+\text {bg})}, M^{\text {sky}}\right ) (10)

LiDAR loss: During training, we optionally exploit LiDAR as an extra sensor to improve the reconstruction of our method. Given a set of collected lidar
samples r(z) = oℓ + zdℓ , Urban Radiance Fields [30] proposes line-of-sight loss
which encourages
the ray distribution to approximate the Gaussian Distribution
N 0, ϵ2 , with ϵ being the bound width of the solid structure. As the assumptionof Gaussian distribution may not hold for all rays, we experimentally observethat the loss in [30] leads to unstable training in our setting. Hence, we propose
to soft the strict constraints as follows:  \mathcal {L}_{\text {empty }}=\int _{t_n}^{z-\epsilon } w(t)^2 d t,~ \mathcal {L}_{\text {near }}=1 - \int _{z-\epsilon }^{z+\epsilon }{w(t)} d t, ~ \mathcal {L}_{\text {dist }}=\int _{z+\epsilon }^{t_f} w(t)^2 d t  (11)where the bound width ϵ is set to 0.5 meters as initialization and we apply exponential decay to the minimum 0.1 meters, which enables gradually decreasing
ambiguities along the training step. Note that the second term Lnear encourages our model to increase volumetric density within a certain range but without
specifying its distribution, whereas the first and third terms are required to keep
the space empty for the rest of the regions.
Entropy regularization loss: To penalize our model representing the distantlandscape as semi-transparent, we introduce the entropy regularization loss toencourage opaque rendering followed by StreetSurf [10]:  \mathcal {L}_{\text {entropy }}=-(\alpha ^{\mathrm {fg}} \ln \alpha ^{\mathrm {fg}}+(1-\alpha ^{\mathrm {fg}}) \ln (1-\alpha ^{\mathrm {fg}})) 

4

(12)

Implementation details

Input Volume Masking: In the training stage, we randomly mask small
regions of the input volume P. Similar to Masked AutoEncoders [12], this trick
improves the completion capability when given a sparse and incomplete point
cloud. More details and ablation results are shown in the supplementary material.
Hierarchical Sampling: For rendering a ray, we follow NeuS [42] to perform
hierarchical sampling during training and inference. This is achieved by distributing the initial half of these samples uniformly, followed by iterative importancesampling for the rest.

10

S.Miao, J.Huang et al.

Appearance Embedding: The images collected from urban scenarios frequently present variable illumination or other environmental changes. Inspiredby [24, 38], we add per-frame appearance embedding to account for exposure
variations. For feed-forward inference on unseen scenes, we set the appearance
embeddings as the mean embedding averaged across all training frames. Oncefine-tuning is considered, we learn the appearance embedding of the reference
images and interpolate them for the test views.
Training Details: We train our neural network end-to-end using Adam optimizer [18] with initial learning rate 5 × 10−3 . We set λ1 = 0.1, λ2 = 1 andλ3 = 0.002 as loss weights in our work. During training, we randomly select asingle image and randomly sample 4096 pixels as a batch. Our EDUS is trainedon one RTX4090 GPU with 500k steps, which takes about 2 days.

5

Experiment

Dataset: We conduct generalizable training on the KITTI-360 dataset [21].
Specifically, we gather 80 sequences encompassing varied street perspectives fortraining purposes, with each sequence comprising 60 stereo images alongside
their respective camera poses. In the case of training scenes, we optionally utilize accumulated LiDAR points to supervise the geometry. Across all scenes,
we standardize the pose by centering the translation around the origin and establish the axis-aligned bounding box (AABB) to decompose the street scene.
The foreground range is set to [−12.6, 12.6 m] for X axes, [−3, 9.8 m] for Y
axes, [−20, 31.2 m] for Z axes, and dimension of the voxelized pointcloud is
128 × 64 × 256 since the voxel size is set to (0.2 m, 0.2 m, 0.2 m).
We use five public validation sets from the KITTI-360 dataset and five scenes
sourced from Waymo [37] as our test scenarios, with the Waymo dataset serving as an out-of-domain test set. It is important to note that the test sets fromKITTI-360 have no overlap with the training scenes. Each test sequence comprises 60 images covering approximately 60 meters. To further assess our robustness against various levels of sparsity, we consider three sparsity settings: 50%,80%, and 90% drop rates, where a higher drop rate indicates a more sparse setof reference images. To ensure fairness in comparison, we employ the same test
images across all drop rates.
Depth: In this paper, we use both monocular and binocular depth to conduct
our experiment. For test sequences in KITTI-360, depth used as both supervisionand input is stereo depth generated by MobileStereoNet [34]. We only supervise
the regularization-based methods by left-eye depth. For the geometry-guided
methods, we use the left-eye depth values of corresponding training images andaggregate them into a point cloud after depth consistency check [5]. For experiments on Waymo, we employ Metric3D [56] to get metric monocular depth maps
and unproject them to form point clouds for evaluation.
Baselines: We conduct a comparative analysis of our method against two lines
of approaches. Firstly, we compare our model with generalizable approaches:
IBRNet [44], MVSNeRF [4], NeO 360 [14] and MuRF [47]. We train each modelEfficient Depth-Guided Urban View Synthesis

Methods

Setting

11

KITTI360drop50%
KITTI360drop80%
Waymodrop50%
PSNR↑SSIM↑ LPIPS↓ PSNR↑SSIM↑ LPIPS↓ PSNR↑SSIM↑ LPIPS↓

IBR-NetMVSNeRFNo
Neo360per-scene
MuRF
opt.
Ours

19.99 0.624 0.217
17.73 0.618 0.328
13.73 0.394 0.624
22.19 0.741 0.264
21.93 0.745 0.178

15.96 0.469 0.354
16.50 0.577 0.365
12.98 0.357 0.659
18.69 0.639 0.353
19.63 0.668 0.244

21.28 0.777 0.199
19.58 0.662 0.278
14.07 0.541 0.708
23.12 0.779 0.318
23.16 0.761 0.189

IBR-NetMVSNeRF
Per-scene
Neo360
opt.
MuRF
Ours

21.17 0.657 0.199
19.47 0.647 0.310
17.92 0.489 0.566
23.71 0.762 0.233
24.43 0.793 0.136

17.98 0.529 0.279
18.06 0.602 0.353
17.51 0.445 0.581
19.70 0.666 0.321
20.91 0.712 0.220

23.39 0.825 0.163
24.28 0.759 0.207
22.59 0.670 0.522
28.30 0.846 0.175
28.45 0.834 0.132

GT

Ours

MuRF IBRNet

Table 1: Quantitative Comparison on five test scenes among generalizable methods. All models are trained on the KITTI-360 dataset using drop 50% sparsity level.

Drop 50% feed-forward

Drop 80% feed-forward

Drop 80% fine-tuned

Fig. 4: Qualitative Comparison with generalizable baselines.

from scratch on the 80 training sequences of KITTI-360, and assess their performance both with and without per-scene fine-tuning. For all generalizable methods, we use three reference frames for training and inference. Secondly, we compare with test-time optimization methods, including methods for sparse viewsettings: MixNeRF [33], SparseNeRF [41], DS-NeRF [6], and one taking point
clouds as input: 3DGS [17]. To maintain parity in our evaluation, we furnish thedepth maps utilized in our method to train depth-supervision approaches [6,41],and employ our accumulated point cloud as the initialization for 3DGS [17].
5.1Comparison with Generalizable NeRFs

Feed-Foward Inference: We train all generalizable methods under 50% drop
rate setting on KITTI-360 and assess their performance using both the 50% and
80% drop rate sparsity levels on KITTI-360. Tab. 1 and Fig. 4 show the qualitative and quantitative comparisons respectively. The results show that our proposed EDUS achieves photorealistic novel view synthesis on the complex street

12

S.Miao, J.Huang et al.

Fig. 5: Zero-shot Inference on Waymo.

Fig. 6: Training Time.

view using only feed-forward inference. More importantly, our method outperforms the baselines with a larger gap when evaluated at a higher unseen sparsity
level (drop 80%), as our method is less sensitive to the camera pose distribution.
IBR-Net and MVSNeRF are mainly suitable for small-scale, object-level scenes
and have difficulty handling unbounded street views, resulting in blurry images.
NeO 360 directly predicts a tri-plane-based radiance field from RGB images,
making it struggle to reconstruct the street scene of 60 meters driving distance.
Our concurrent work MuRF [47] achieves comparable PSNR and SSIM in drop
50% but sometimes shows blocky artifacts, resulting in worse LPIPS. Furthermore, our methods outperform all baselines on the unseen Waymo datasets,
demonstrating the generalization capability empowered by the geometric priors,
see Fig. 5.
Per-Scene Fine-Tuning: We further fine-tune all generalizable methods under
different sparsity levels, as shown in Tab. 1. For each novel street sequence, we
freeze our 3D CNN and directly fine-tune the feature volume to enable efficientconvergence. Note that our feature volume is initialized by running a single feedforward pass of the 3D CNN, yielding a good initialization and subsequently
yielding improved performance than baselines after fine-tuning.
Note that MuRF constructs local volume at each target view and requires
complete propagation of gradients, resulting in slower convergence during perscene optimization. In contrast, our global volume-based approach demonstrates
faster convergence speed, see Fig. 6, making it more suitable for urban scenes.
Besides, EDUS shows superior memory efficiency when handling full-resolution(376 × 1408) inference with 6GB while MuRF requires splitting the same size
image into 4 patches, each consuming 16.2GB on a single RTX4090 (24GB).
5.2

Comparison with Test-Time Optimization NeRFs

We proceed to compare our method with test-time optimization approaches
across various sparsity levels. Tab. 2 and Fig. 7 demonstrate that our method
achieves state-of-the-art performance at high sparsity levels, attributable to therobust initialization provided by our generalizable model. In the drop 50% setting, the recent state-of-the-art method 3DGS [17] achieves the best SSIM and
LPIPS, but their performance experiences significant degradation in the 80%
and 90% settings. Our method achieves superior performance than other geometric regularization [33] or depth-supervision [6, 41] methods. Although theyEfficient Depth-Guided Urban View Synthesis
Setting

Methods
MixNeRF

GeometricReg.

SparseNeRF

DS-NeRF

3D GS
Depth Input
Ours

Drop Rate

13

Cost Time PSNR↑ SSIM↑ LPIPS↓

50%
80%
90%
50%
80%
90%
50%
80%
90%

∼ 51min∼ 38min∼ 31min∼ 35min∼ 27min∼ 21min∼ 36min∼ 28min∼ 22min

21.50
18.89
17.89
21.34
19.18
17.94
20.40
18.91
17.91

0.625
0.558
0.529
0.642
0.607
0.559
0.604
0.581
0.564

0.357
0.404
0.416
0.552
0.559
0.608
0.601
0.620
0.632

50%
80%
90%
50%
80%
90%

∼ 29min∼ 18min∼ 10min∼ 5min< 5min< 5min

24.37
19.80
17.46
24.43
20.91
19.16

0.804
0.679
0.597
0.793
0.712
0.657

0.109
0.234
0.332
0.136
0.220
0.271

Table 2: Test-Time Optimization Comparison.

MixNeRF

SparseNeRF

DS-NeRF

3DGS
Ours
GT
Fig. 7: Qualitative Comparison with Test-Time Optimization baselines.

show good performance on small-scale datasets (e.g. DTU, LLFF datasets), theystruggle with urban scenes which contain more complex geometry layouts. In asparser setting, we maintain our advantages and achieve a PSNR of 20.91dB
in 80% drop rate and 19.16dB PSNR in 90% drop rate, respectively. Anotheradvantage of our work is that we achieve good quality with fast fine-tuning on
novel scenes. Given the same computation recourse, our training time (5 minutes) outperforms other baselines by a large margin, 5× faster than DS-Nerf [6]and SparseNerf [41] and 10× faster than MixNeRF [33].
5.3

Ablation StudyIn Tab. 6, we ablate the design choices of our method. We report the results on
KITTI-360 validation dataset with 50% drop rate. First, we replace the modulated SPADE CNN with the traditional 3D U-Net, yielding the results (3DU-Net). This worsens feed-forward inference results, indicating that the modulated convolutions better preserve the information of the input point cloud.

14

S.Miao, J.Huang et al.
Feed-foward NVS
PSNR↑ SSIM↑ LPIPS↓

Finetuning NVS
PSNR↑ SSIM↑ LPIPS↓

3D U-Net3D
w/o ffg
2D
w/o ffg
w/o decomposition
w/o Llidar

21.49
15.71
18.28
20.65
21.75

0.716
0.532
0.593
0.684
0.742

0.194
0.441
0.342
0.195
0.189

24.14
18.91
21.41
22.54
23.77

0.789
0.609
0.648
0.754
0.788

0.145
0.343
0.315
0.175
0.139

Full model

21.93

0.745

0.178

24.43

0.793

0.136

Ablation

Table 3: Ablation study. Metrics are averaged over the 5 test scenes.

Next, we remove the 3D feature from the full model, yielding a baseline similar
3D
to IBRNet but augmented with scene decomposition (w/o ffg). This leads toa significant drop in performance, indicating the importance of the geometricpriors. We also study the effect of the 2D image-based feature, and observe that
2D
removing it leads to blurry rendering (w/o ffg). This demonstrates that the
3D
3D feature ffg alone is insufficient due to the inherent inductive bias of the
3D CNN. When removing the street decomposition and representing the entire
scene as foreground (w/o decomposition), it leads to artifacts in background regions due to the incorrect geometric modeling. Finally, we demonstrate that the
LiDAR supervision of the training sequences leads to slight improvements (w/o
Llidar ). We observe that the LiDAR supervision improves details, leading to alarger improvement in LPIPS compared to PSNR and SSIM. Note that LiDAR
supervision is not applied during test, but only applied to the training sequences
to enhance our generalizable geometry prediction against noisy geometric priors.
Please refer to our supplementary material for qualitative comparisons.

6

Conclusion

The paper presents EDUS, a generalizable and efficient method for sparse urbanview synthesis. By integrating geometric priors into a generalizable model, we
demonstrate that our method achieves robust performance across various density levels and datasets. It further enables efficient fine-tuning and outperforms
existing sparse view methods based on good initialization. We will investigate
handling dynamic objects in street views in future work, which poses challenges
to generalizable approaches.

Acknowledgements
We would like to thank Dzmitry Tsishkou for his useful suggestions and tips. This
work was supported by the National Key R&D Program of China under Grant
2021ZD0114500, and NSFC under grant 62202418 and U21B2004. A. Geiger was
supported by the ERC Starting Grant LEGO-3D (850533) and the DFG EXC
number 2064/1 - project number 390727645.

Efficient Depth-Guided Urban View Synthesis

15

References
1. Barron, J.T., Mildenhall, B., Tancik, M., Hedman, P., Martin-Brualla, R., Srinivasan, P.P.: Mip-nerf: A multiscale representation for anti-aliasing neural radiancefields. In: Proceedings of the IEEE/CVF International Conference on Computer
Vision. pp. 5855–5864 (2021)
2. Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Mipnerf 360: Unbounded anti-aliased neural radiance fields. In: Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5470–
5479 (2022)
3. Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P., Hedman, P.: Zip-nerf:
Anti-aliased grid-based neural radiance fields. arXiv preprint arXiv:2304.06706(2023)
4. Chen, A., Xu, Z., Zhao, F., Zhang, X., Xiang, F., Yu, J., Su, H.: Mvsnerf: Fast
generalizable radiance field reconstruction from multi-view stereo. In: Proceedingsof the IEEE/CVF International Conference on Computer Vision. pp. 14124–14133(2021)
5. Cheng, K., Long, X., Yin, W., Wang, J., Wu, Z., Ma, Y., Wang, K., Chen, X.,
Chen, X.: Uc-nerf: Neural radiance field for under-calibrated multi-view cameras.
In: The Twelfth International Conference on Learning Representations (2023)
6. Deng, K., Liu, A., Zhu, J.Y., Ramanan, D.: Depth-supervised nerf: Fewer views andfaster training for free. In: Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition. pp. 12882–12891 (2022)
7. Fridovich-Keil, S., Yu, A., Tancik, M., Chen, Q., Recht, B., Kanazawa, A.: Plenoxels: Radiance fields without neural networks. In: CVPR (2022)
8. Fu, X., Zhang, S., Chen, T., Lu, Y., Zhou, X., Geiger, A., Liao, Y.:
Panopticnerf-360: Panoramic 3d-to-2d label transfer in urban scenes. arXiv
preprint arXiv:2309.10815 (2023)
9. Fu, X., Zhang, S., Chen, T., Lu, Y., Zhu, L., Zhou, X., Geiger, A., Liao, Y.:
Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation. In:
2022 International Conference on 3D Vision (3DV). pp. 1–11. IEEE (2022)
10. Guo, J., Deng, N., Li, X., Bai, Y., Shi, B., Wang, C., Ding, C., Wang, D., Li, Y.:
Streetsurf: Extending multi-view implicit surface reconstruction to street views.
arXiv preprint arXiv:2306.04988 (2023)
11. Haque, A., Tancik, M., Efros, A.A., Holynski, A., Kanazawa, A.: Instruct-nerf2nerf:
Editing 3d scenes with instructions. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 19740–19750 (2023)
12. He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders arescalable vision learners. In: Proceedings of the IEEE/CVF conference on computer
vision and pattern recognition. pp. 16000–16009 (2022)
13. Huang, D., Peng, S., He, T., Yang, H., Zhou, X., Ouyang, W.: Ponder: Point cloud
pre-training via neural rendering. In: Proceedings of the IEEE/CVF International
Conference on Computer Vision. pp. 16089–16098 (2023)
14. Irshad, M.Z., Zakharov, S., Liu, K., Guizilini, V., Kollar, T., Gaidon, A., Kira, Z.,
Ambrus, R.: Neo 360: Neural fields for sparse view synthesis of outdoor scenes. In:
Proceedings of the IEEE/CVF International Conference on Computer Vision. pp.
9187–9198 (2023)
15. Jain, A., Tancik, M., Abbeel, P.: Putting nerf on a diet: Semantically consistent
few-shot view synthesis. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 5885–5894 (2021)

16

S.Miao, J.Huang et al.

16. Johari, M.M., Lepoittevin, Y., Fleuret, F.: Geonerf: Generalizing nerf with geometry priors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition. pp. 18365–18375 (2022)
17. Kerbl, B., Kopanas, G., Leimkühler, T., Drettakis, G.: 3d gaussian splatting forreal-time radiance field rendering. ACM Transactions on Graphics 42(4) (2023)
18. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.6980 (2014)
19. Kundu, A., Genova, K., Yin, X., Fathi, A., Pantofaru, C., Guibas, L.J., Tagliasacchi, A., Dellaert, F., Funkhouser, T.: Panoptic neural fields: A semantic objectaware neural scene representation. In: Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition. pp. 12871–12881 (2022)
20. Li, Z., Li, L., Zhu, J.: Read: Large-scale neural scene rendering for autonomous
driving. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 37,
pp. 1522–1529 (2023)
21. Liao, Y., Xie, J., Geiger, A.: Kitti-360: A novel dataset and benchmarks for urban
scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis andMachine Intelligence 45(3), 3292–3310 (2022)
22. Liu, L., Gu, J., Zaw Lin, K., Chua, T.S., Theobalt, C.: Neural sparse voxel fields.
Advances in Neural Information Processing Systems 33, 15651–15663 (2020)
23. Liu, Y., Peng, S., Liu, L., Wang, Q., Wang, P., Theobalt, C., Zhou, X., Wang,
W.: Neural rays for occlusion-aware image-based rendering. In: Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7824–
7833 (2022)
24. Martin-Brualla, R., Radwan, N., Sajjadi, M.S., Barron, J.T., Dosovitskiy, A., Duckworth, D.: Nerf in the wild: Neural radiance fields for unconstrained photo collections. In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition. pp. 7210–7219 (2021)
25. Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng,
R.: Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 65(1), 99–106 (2021)
26. Niemeyer, M., Barron, J.T., Mildenhall, B., Sajjadi, M.S., Geiger, A., Radwan, N.:
Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs.
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition. pp. 5480–5490 (2022)
27. Ost, J., Mannan, F., Thuerey, N., Knodt, J., Heide, F.: Neural scene graphs for
dynamic scenes. In: Proceedings of the IEEE/CVF Conference on Computer Vision
and Pattern Recognition. pp. 2856–2865 (2021)
28. Park, T., Liu, M.Y., Wang, T.C., Zhu, J.Y.: Semantic image synthesis withspatially-adaptive normalization. In: Proceedings of the IEEE/CVF conference on
computer vision and pattern recognition. pp. 2337–2346 (2019)29. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G.,
Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models fromnatural language supervision. In: International conference on machine learning. pp.
8748–8763. PMLR (2021)
30. Rematas, K., Liu, A., Srinivasan, P.P., Barron, J.T., Tagliasacchi, A., Funkhouser,T., Ferrari, V.: Urban radiance fields. In: Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition. pp. 12932–12942 (2022)
31. Rückert, D., Franke, L., Stamminger, M.: Adop: Approximate differentiable onepixel point rendering. ACM Transactions on Graphics (ToG) 41(4), 1–14 (2022)

Efficient Depth-Guided Urban View Synthesis

17

32. Schonberger, J.L., Frahm, J.M.: Structure-from-motion revisited. In: Proceedingsof the IEEE conference on computer vision and pattern recognition. pp. 4104–4113(2016)
33. Seo, S., Han, D., Chang, Y., Kwak, N.: Mixnerf: Modeling a ray with mixture density for novel view synthesis from sparse inputs. In: Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition. pp. 20659–20668 (2023)
34. Shamsafar, F., Woerz, S., Rahim, R., Zell, A.: Mobilestereonet: Towards lightweight
deep networks for stereo matching. In: Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. pp. 2417–2426 (2022)
35. Sun, C., Sun, M., Chen, H.T.: Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction (2022)
36. Sun, C., Sun, M., Chen, H.T.: Improved direct voxel grid optimization for radiancefields reconstruction. arXiv preprint arXiv:2206.05085 (2022)
37. Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo,
J., Zhou, Y., Chai, Y., Caine, B., et al.: Scalability in perception for autonomous
driving: Waymo open dataset. In: Proceedings of the IEEE/CVF conference on
computer vision and pattern recognition. pp. 2446–2454 (2020)
38. Tancik, M., Casser, V., Yan, X., Pradhan, S., Mildenhall, B., Srinivasan, P.P., Barron, J.T., Kretzschmar, H.: Block-nerf: Scalable large scene neural view synthesis.
In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition. pp. 8248–8258 (2022)
39. Tancik, M., Weber, E., Ng, E., Li, R., Yi, B., Wang, T., Kristoffersen, A., Austin, J.,
Salahi, K., Ahuja, A., et al.: Nerfstudio: A modular framework for neural radiancefield development. In: ACM SIGGRAPH 2023 Conference Proceedings. pp. 1–12
(2023)
40. Tao, A., Sapra, K., Catanzaro, B.: Hierarchical multi-scale attention for semantic
segmentation. arXiv preprint arXiv:2005.10821 (2020)
41. Wang, G., Chen, Z., Loy, C.C., Liu, Z.: Sparsenerf: Distilling depth ranking forfew-shot novel view synthesis. arXiv preprint arXiv:2303.16196 (2023)
42. Wang, P., Liu, L., Liu, Y., Theobalt, C., Komura, T., Wang, W.: Neus: Learningneural implicit surfaces by volume rendering for multi-view reconstruction. arXiv
preprint arXiv:2106.10689 (2021)
43. Wang, P., Liu, Y., Chen, Z., Liu, L., Liu, Z., Komura, T., Theobalt, C., Wang,
W.: F2-nerf: Fast neural radiance field training with free camera trajectories. In:
Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4150–4159 (2023)
44. Wang, Q., Wang, Z., Genova, K., Srinivasan, P.P., Zhou, H., Barron, J.T., MartinBrualla, R., Snavely, N., Funkhouser, T.: Ibrnet: Learning multi-view image-based
rendering. In: Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition. pp. 4690–4699 (2021)
45. Wu, Z., Liu, T., Luo, L., Zhong, Z., Chen, J., Xiao, H., Hou, C., Lou, H., Chen,Y., Yang, R., et al.: Mars: An instance-aware, modular and realistic simulator forautonomous driving. In: CAAI International Conference on Artificial Intelligence.
pp. 3–15. Springer (2023)
46. Xie, Z., Zhang, J., Li, W., Zhang, F., Zhang, L.: S-nerf: Neural radiance fields forstreet views. arXiv preprint arXiv:2303.00749 (2023)
47. Xu, H., Chen, A., Chen, Y., Sakaridis, C., Zhang, Y., Pollefeys, M., Geiger, A., Yu,
F.: Murf: Multi-baseline radiance fields. arXiv preprint arXiv:2312.04565 (2023)
48. Xu, Q., Xu, Z., Philip, J., Bi, S., Shu, Z., Sunkavalli, K., Neumann, U.: Point-nerf:
Point-based neural radiance fields. In: Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition. pp. 5438–5448 (2022)

18

S.Miao, J.Huang et al.

49. Yang, H., Zhang, S., Huang, D., Wu, X., Zhu, H., He, T., Tang, S., Zhao, H.,
Qiu, Q., Lin, B., et al.: Unipad: A universal pre-training paradigm for autonomous
driving. arXiv preprint arXiv:2310.08370 (2023)
50. Yang, J., Ivanovic, B., Litany, O., Weng, X., Kim, S.W., Li, B., Che, T., Xu, D.,
Fidler, S., Pavone, M., et al.: Emernerf: Emergent spatial-temporal scene decomposition via self-supervision. arXiv preprint arXiv:2311.02077 (2023)
51. Yang, J., Pavone, M., Wang, Y.: Freenerf: Improving few-shot neural renderingwith free frequency regularization. In: Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition. pp. 8254–8263 (2023)
52. Yang, L., Kang, B., Huang, Z., Xu, X., Feng, J., Zhao, H.: Depth anything: Unleashing the power of large-scale unlabeled data. arXiv preprint arXiv:2401.10891(2024)
53. Yang, Y., Yang, Y., Guo, H., Xiong, R., Wang, Y., Liao, Y.: Urbangiraffe: Representing urban scenes as compositional generative neural feature fields. arXiv
preprint arXiv:2303.14167 (2023)
54. Yang, Z., Chen, Y., Wang, J., Manivasagam, S., Ma, W.C., Yang, A.J., Urtasun, R.:
Unisim: A neural closed-loop sensor simulator. In: Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition. pp. 1389–1399 (2023)
55. Yao, Y., Luo, Z., Li, S., Fang, T., Quan, L.: Mvsnet: Depth inference for unstructured multi-view stereo. In: Proceedings of the European conference on computer
vision (ECCV). pp. 767–783 (2018)56. Yin, W., Zhang, C., Chen, H., Cai, Z., Yu, G., Wang, K., Chen, X., Shen, C.: Metric3d: Towards zero-shot metric 3d prediction from a single image. In: Proceedingsof the IEEE/CVF International Conference on Computer Vision. pp. 9043–9053(2023)
57. Yu, A., Ye, V., Tancik, M., Kanazawa, A.: pixelnerf: Neural radiance fields fromone or few images. In: Proceedings of the IEEE/CVF Conference on Computer
Vision and Pattern Recognition. pp. 4578–4587 (2021)
58. Zhu, H., Yang, H., Wu, X., Huang, D., Zhang, S., He, X., He, T., Zhao, H., Shen,
C., Qiao, Y., et al.: Ponderv2: Pave the way for 3d foundataion model with auniversal pre-training paradigm. arXiv preprint arXiv:2310.08586 (2023)

Efficient Depth-Guided Urban View Synthesis

19

Supplementary Material for Efficient
Depth-Guided Urban View Synthesis
Abstract. In this supplementary document, we initially present an
in-depth overview of our network architecture in Appendix A. Subsequently, we elaborate on our implementation details in Appendix B, encompassing the random masking strategy, the sampling strategy, and thepoint cloud accumulation configurations. Following this, we introduce thebaselines for comparison in Appendix C, and discuss the datasets utilized in our experiments in Appendix D. Finally, we present additional
experimental results in Appendix E. Besides, the supplementary video
showcases our feed-forward and fine-tuned results across various scenes,
along with novel view renderings compared against multiple baselines.

A

Network Architecture

A.1

Modulation-based CNN

Our 3D Spatially-Adaptive Normalization convolutional neural network (SPADE
CNN) modulates the feature volume generation at multiple resolutions. It takes
a single voxelized point cloud with dimension 64 × 128 × 256 as input. The
appearance information of the input point cloud is injected into the featurevolume at each resolution. Fig. 8 presents the detailed network architecture.

Fig. 8: Architecture of our 3D SPADE CNN.

20

S.Miao, J.Huang et al.

A.2

Foreground Network Architecture

The feature volume generated by the 3D SPADE CNN provides both geometricpriors and appearance information for foreground contents. Based on a feature
3D
vector ffg∈ R16 extracted from this feature volume, our density decoder gθpredicts the density value. In order to capture high-frequency appearance during
3D
feedforward inference, the color decoder hθ combines the 3D feature ffg
∈ R16
2D
9
and 2D features queried from nearest views ffg ∈ R for color prediction. In
addition, the color decoder further takes the positional encoding of x as input,
setting the number of frequencies to 10 following [25]. Furthermore, we inject
viewing direction d ∈ R3 and a per-frame appearance embedding w ∈ R32 . We
present the detailed network architecture in Tab. 4.

Density Decoder
Layer
in channels
LinearRelu0
16
LinearRelu1
64
Color Decoder
Layer
in channels
LinearRelu0
16+9+63
LinearRelui=1,2
128
LinearRelu3
128+32+3
LinearRelu4
128
LinearRelu564

out channels
641

description
3D
in: ffg
out: σfg

out channels
128
128
128
64
3

description
3D 2D, ffg , γ(x)in: ffg/
in: + d, w/
out: cfg

Table 4: Density and Color Decoder Architectures. From top to bottom: twolayer MLP for density prediction and six-layer MLP for color prediction.

A.3

Background and Sky Network Architectures

We also build the generalizable model for the background and sky region via theimage-based rendering paradigm as we mentioned in the main paper. We now
show the details of the background and sky network details in Tab. 5.

BImplementation Details

B.1

Input Volume Random Masking

Our target is to perform novel view synthesis from sparse urban images. When
provided with sparse images, the accumulated point clouds are often incomplete
and contain holes due to occlusions or insufficient overlap across the input frames.
To equip EDUS with the capability to handle sparse and incomplete point clouds
by filling in the missing parts, we randomly mask portions of the point cloudEfficient Depth-Guided Urban View Synthesis

21

Background network architecture
Layer
in channels out channels
description
2D
LinearRelu0
63+9
128
in: ffg, γ(x)
LinearRelui=1,2,3
128
128
/
LinearRelu4
128
16
out: σbg , fe
LinearRelu5
15+3
64
/
LinearRelu664
64/LinearRelu7
643
out: cbg
Sky network architecture
Layer
in channels out channels
description
2D
LinearRelu0
9+3
3
in: ffg, d; out: csky
Table 5: Background and Sky Network Architectures. The fe ∈ R15 represents
the geometric embedding vector and serves as the input for background color prediction.

during the training stage. Specifically, we randomly select an 8m × 8m × 12m
cuboid in each iteration, corresponding to a volume of size [40 × 40 × 60], andeliminate the point cloud within this area. Note that this is not applied in feedforward inference nor per-scene fine-tuning. This simple yet effective strategy
enhances the generalizable ability on novel scenes, as shown in our ablation
study in Appendix E.2.
B.2

Hierarchical Sampling

The total number of samples located at the ray is 160 with 128 sampled fromthe foreground and 32 from the background. Specifically, we first select 80 points
along the ray within the foreground volume, distributing one-half of these samples uniformly, while the other half is allocated linearly based on disparity spacing, following Nerfacto [39]. Next, we iteratively perform importance sampling
to sample valid regions containing solid contents based on the coarse density
prediction following NeuS [42], with three iterations and 16 samples for each
iteration. For the background, we additionally uniformly sample 32 points outof the foreground volume.
B.3

Accumulated Point CloudWe design our accumulated point cloud to encourage the model to learn completion. Considering that the sparsity level in real-world driving data is unknown
in advance, we aim to train a single generalizable model and evaluate its performance at different sparsity levels. In our main experiments, all generalizable
methods are trained at the sparsity level of the drop50 setting, i.e., 50% images
are available for supervision. Instead of directly using the 50% reference images
to create the accumulated point cloud, we create a more sparse point cloud fortraining to encourage the model to learn scene completion. Specifically, the accumulated point cloud we use for training is from every fifth of the stereo pairs.

22

S.Miao, J.Huang et al.

Note that during the feed-forward inference or per-scene optimization, we always use the given reference images of different sparsity levels (50%, 80%, or
90%) to generate the corresponding accumulated point cloud. This ensures faircomparison to the baselines and further verifies our method’s robustness to thedensity of the input point cloud.

C

Baselines

Generalizable NeRFs We utilize the official implementations of IBR-Net,
MVSNeRF, Neo360, and MuRF. For each method, we pretrain the model using
80 scenes from the KITTI360 dataset [21] under a 50% drop rate. In the case of
IBR-Net, MVSNeRF, and MuRF, we select the nearest three training frames of
the target view as reference frames. As MVSNeRF constructs local cost volumes
defined at reference views, we adjust the fine-tuning strategy of MVSNeRF to
dynamically use the nearest reference frames to construct the cost volume, similar to the pre-training phase, rather than employing a fixed set of frames as intheir original implementation.
Test-Time Optimization Methods The test-time optimization approaches
are individually trained for each scene under three sparsity levels. We employ
the official codebase of MixNeRF [33] and implement SparseNeRF [41] and DSNeRF [6] by integrating stereo depth supervision onto Nerfstudio’s Mip-NeRF
model [1, 39]. Additionally, we utilize the splatfacto model of Nerfstudio to train
our 3D GS [17]. To ensure a fair comparison, we provide the depth maps usedin our approach to train the depth-supervision methods [6, 41]. When it comesto point cloud-based methods [17], we find that the missing of sky may lead to a
decline of performance, so we keep the sky in the point cloud and initialize themodel with it.

DDatasets

D.1

KITTI-360

We follow the 50% drop rate (Drop50) and 90% drop rate (Drop90) settingsof the KITTI-360 dataset and expand them into 80% drop rate (Drop80). To
be more specific, we take a pair of left and right eye image pairs as the unit.
In the Drop50 setting, we use odd-numbered image pairs as the training set.
Similarly, in the Drop80 and Drop90 settings, we use every fifth or every tenthpair of images for training, respectively. Note that all test frames are the same
to allow comparison across different sparsity levels. For example, consideringten consecutive image pairs, Drop50 refers to using pairs of id 0, 2, 4, 6, 8 fortraining, whereas Drop80 and Drop90 refer to using 0, 5 and 0, respectively. The
test frames are selected as the left-eye image of 1, 3, 7, 9 pairs to avoid overlappingwith the training frames.

Efficient Depth-Guided Urban View Synthesis(a) 3D U-Net

2D(b) w/o ffg(c) w/o decomposition(d) Full model

23(e) GT

Fig. 9: Qualitative Results of Ablation Study. The experiments are conducted
on KITTI-360 under Drop50 sparsity level.

D.2

Waymo

We use the same drop rate setting and point cloud accumulation scheme as
the KITTI-360 dataset for Waymo. Note that we only use the front camera of
the Waymo dataset. Therefore, every odd-numbered image is used for training
for the Drop50 setting, and every fifth or tenth image is used for Drop80 and
Drop90, respectively. The test views are configured to encompass frames at the
3rd and 7th positions, as well as frames with id equivalent to 3 or 7 plus themultiples of 10, which has no overlapping with the training set.

E

Additional Experimental Results

E.1

Qualitative Results of Ablation Study

Fig. 9 demonstrates the qualitative outcomes of our ablation study. These results
are all derived from feed-forward inference images under the Drop50 sparsity
level, which align seamlessly with the corresponding quantitative results in Table
3 of the main paper. The substitution of the modulated SPADE CNN with aconventional 3D U-Net leads to less detailed prediction and yields blurrinessin some regions. The elimination of the 2D image-based feature results in thehigh-frequency contents losing their sharpness and becoming noticeably blurry.
Moreover, if we solely represent the scene as foreground, it results in artifacts
in the distant region, due to the wrong modeling of the scene geometry. Our
fully integrated model exhibits the highest quality of reconstruction, effectively
validating our design choices.

24

S.Miao, J.Huang et al.
KITTI360drop50
PSNR↑ SSIM↑ LPIPS↓KITTI360drop80
PSNR↑ SSIM↑ LPIPS↓

w/o random masking

20.62

0.694

0.228

18.54

0.641

0.288

Full model

21.93

0.745

0.178

19.63

0.668

0.244

Ablation

Table 6: Ablation Study of Input Volume Random Masking. The results showthat the random masking strategy in the training stage improves the feed-forward
inference performance using different sparse settings.

E.2

Additional Ablation of Input Volume Random Masking

As shown in Tab. 6, the random mask strategy helps improve our feed-forward
performance, which means our model is equipped with filling capacity to process
the incomplete point cloud by adding random masking.
E.3

Additional Comparison on Waymo Dataset

We conduct zero-shot NVS under a drop rate of 50% using the model trainedon the KITTI-360 dataset, and the rendered images are shown in Fig. 10. To
further verify our method’s performance given different training scenes, we train
all generalizable methods on the Waymo dataset and evaluate them on Waymo
and KITTI-360, see Tab. 7. Specifically, all models are trained with the Drop50
setting on the Waymo dataset. Our model has better performance under 80%
drop rate thanks to the depth-guided global volume and achieves the best metrics
after fine-tuning. Note that our depth guidance of the Waymo dataset is obtained
from a monocular depth estimation method [56], demonstrating that we are able
to learn an effective generalizable model guided by noisy monocular depth priors.
Fig. 11 shows the feed-forward and fine-tuned qualitative results of our method
trained and evaluated on the Waymo dataset.
E.4

Additional Qualitative Analysis of Geometry Prediction

Fig. 12 shows the novel view synthesis and depth comparison of our method
to other optimization-based methods under Drop90 setting. The quantitative
comparisons are reported in Table 2 of the main paper. Note that all methods except for Mix-NeRF use the same depth maps as supervision (DS-NeRF,
SparseNeRF) or as input (3DGS, Ours). Although 3DGS [17] renders relatively
good depth maps, it shows obvious flaws in appearance under sparse settings.
Our model shows the best geometry thanks to our generalizable prior. As shownin Fig. 13, we illustrate the geometry extracted from the feed-forward inference,
demonstrating that our EDUS is able to refine the noisy input point cloud andfill holes with generalizable 3D CNN.

Efficient Depth-Guided Urban View Synthesis

25

Fine-tuned

Feed-forward

Fig. 10: Zero-shot Inference on Waymo dataset. Our model is trained on the KITTI360 dataset using Drop50 sparsity level.

Fig. 11: Qualitative Results trained and evaluated on Waymo dataset. Our modelis trained on Waymo dataset using Drop50 sparsity level. The feed-forward inference
and fine-tuning are performed under the Drop80 setting.

E.5

Number of Reference Views for Generalizable NeRFs

As we have mentioned in the main paper, our method takes input as a point
cloud, which accumulates multi-frame information in addition to the reference
frames. To verify fairness and our advancement compared with other referencebased methods, we evaluate the state-of-the-art approach MuRF [47] using different numbers of reference images. Specifically, we gradually increase the numberof reference frames which is used for feature extraction and conduct feed-forward
inference on our test sequence of the KITTI-360 dataset under Drop80 setting.

26

S.Miao, J.Huang et al.
Waymodrop50
Waymodrop80
KITTI360drop50
PSNR↑SSIM↑ LPIPS↓ PSNR↑SSIM↑ LPIPS↓ PSNR↑SSIM↑ LPIPS↓

Methods

Setting

IBR-NetMVSNeRF
Neo360
MuRF
Ours

22.90 0.739 0.148
18.05 0.595 0.378
No
per-scene 16.28 0.505 0.569
23.66 0.746 0.256
opt.
23.41 0.769 0.147

17.90 0.619 0.242
17.41 0.581 0.404
15.83 0.505 0.584
21.25 0.664 0.327
21.64 0.703 0.204

18.88 0.606 0.272
15.35 0.524 0.389
10.99 0.204 0.687
19.83 0.669 0.340
20.13 0.659 0.257

IBR-NetMVSNeRF
Neo360
MuRF
Ours

23.71 0.861 0.121
22.42 0.699 0.309
Per-scene
21.76 0.647 0.517
opt.
24.06 0.753 0.224
26.88 0.839 0.109

18.58 0.741 0.222
20.81 0.657 0.348
20.66 0.631 0.533
22.85 0.711 0.260
24.33 0.776 0.164

20.78 0.650 0.205
18.80 0.611 0.354
17.89 0.480 0.571
21.24 0.710 0.265
24.35 0.787 0.145

Table 7: Quantitative Comparison on five test scenes among generalizable methods. All models are trained on the Waymo dataset using Drop50 sparsity level.

(a) Mix-NeRF(b) DS-NeRF(c) SparseNeRF(d) 3DGS(e) Ours(f ) GT

Fig. 12: Qualitative Depth Comparison with test-time optimization methods.

Efficient Depth-Guided Urban View Synthesis

27(a) Filling holes in input point clouds.

(b) Removing noise from input point clouds.

Fig. 13: Qualitative Results of density volume extracted from our feed-forward
inference.

We find that increasing the number of reference frames does not improve themodel’s performance. This may be due to the fact that further reference frames
have stronger occlusions, hence increasing the difficulty of feature matching. On
the other hand, the usage of more input frames also increases the burden on computing resources and causes out-of-memory when the number of views achieves
10. On the contrary, our method utilizes a fixed size of point cloud and manages
to combine multi-frame guidance to predict accurate scene geometry. The resultsof MuRF using different numbers of reference views are shown in Tab. 8.
E.6

Limitations

Although our approach achieves superior performance on NVS from sparse urban
images, it still suffers degeneration when there is less overlap between the target
image and reference images. In this case, 3D sample points will not query accurate appearance information from nearby images, leading to blurry rendering at
the bottom of the test images, as illustrated in Fig. 14. This problem may be
mitigated in future work by improving the 3D CNN’s capacity to directly predicthigh-frequency appearance instead of querying color from reference images.

28

S.Miao, J.Huang et al.

Num. of Views

PSNR↑

SSIM↑

LPIPS↓

2
3
4
6
8
10

18.06
18.69
18.47
16.61
15.01
/

0.606
0.639
0.633
0.589
0.544
/

0.374
0.353
0.361
0.410
0.459
/

Table 8: Number of Reference Views for MuRF. We evaluate MuRF using
various numbers of reference views. Note that the performance of MuRF does notincrease wrt. the number of reference views.

Fig. 14: Limitations in Drop90 Feed-Foward Inference. Our performance degenerates in regions with insufficient overlap between reference and target frames, e.g.,
the bottom of the images.