NeLF-Pro: Neural Light Field Probes for Multi-Scale Novel View Synthesis
Zinuo You1
1
ETH Zürich

Andreas Geiger2
Anpei Chen†,1,2
2
University of Tübingen, Tübingen AI Centerzinyou@ethz.ch{a.geiger,anpei.chen}@uni-tuebingen.de

Abstract
We present NeLF-Pro, a novel representation to modeland reconstruct light fields in diverse natural scenes thatvary in extent and spatial granularity. In contrast to previous fast reconstruction methods that represent the 3D
scene globally, we model the light field of a scene as aset of local light field feature probes, parameterized withposition and multi-channel 2D feature maps. Our central
idea is to bake the scene’s light field into spatially varying learnable representations and to query point features byweighted blending of probes close to the camera - allowing
for mipmap representation and rendering. We introduce anovel vector-matrix-matrix (VMM) factorization techniquethat effectively represents the light field feature probes as
products of core factors (i.e., VM) shared among local feature probes, and a basis factor (i.e., M) - efficiently encodinginternal relationships and patterns within the scene. Experimentally, we demonstrate that NeLF-Pro significantly
boosts the performance of feature grid-based representations, and achieves fast reconstruction with better rendering quality while maintaining compact modeling. Project
page: https://sinoyou.github.io/nelf-pro.

Figure 1. NeLF-Pro represent a Scene as spatially distributed
Light Field Probes for faithful novel view synthesis in diverseand spatially inhomogeneous natural scenes.

scene into bounded MLP and feature grids, scene contraction [6, 41, 62, 74] is commonly used to warp the
input domain within a ball or cube of fixed size. Despite their impressive performance, the basic representations they use, such as MLPs [6], VMGrids [12], and HashGrids [41, 62], are constrained to small-scale scenes (e.g.,
mip-NeRF360 and Free dataset), because these representations rely on a bounded centralized space. To represent
large-scale scenes, a common strategy is to subdivide thetarget scene into cells, each endowed with individual representations [19, 37, 53, 58, 66]. However, determining theideal position and dimensions of the local bounding boxes
is not trivial. Employing bounding boxes with hard partitions can precipitate unstable optimization, giving rise to“floater” artifacts as demonstrated in KiloNeRF [47]. To
solve this, regularization terms such as geometrical pixel
partition [58] or total variance loss [67] are usually needed
to stabilize the reconstruction. On the other hand, soft partitions with overlaps require smoothing filters, such as appearance matching [53] or per-frame blending [37] for thecolors predicted from different local representations, resulting in slow optimization.
In this work, we propose Neural Light Field Probes(aka, NeLF-Pro), a novel representation that is compatible with small and large scale scenes, and at the same timeachieves on-par-with or better-than current state-of-the-art
novel view rendering quality. Unlike previous methods that

1. Introduction
Efficiently representing and reconstructing 3D scenes that
support high-fidelity novel view synthesis is critical forcomputer vision and graphics applications, including virtualand augmented reality, cinematography, robotics and autonomous navigation. Neural Radiance Fields (NeRF) [40]have emerged as an effective representation to model realworld objects/scenes by incorporating density with viewdependent radiance.
Despite the high rendering quality, NeRF takes days
to reconstruct an object or scene. The follow-up works
demonstrated significant success in speeding up the training process by relaxing the input domain to optimizable feature grids [2, 12, 41, 52]. To fit an unbounded† Corresponding author.

1

encode positions globally into multilevel inputs, we decompose the scene into local probes, allowing for scale- andtrajectory-agnostic modeling. The core idea of our representation is to jointly encode the light field and geometry using Light Field Probes and optimize the probes by gradientdescent. Unlike standard light field methods that model rays
using global parametrization, our light field probes capture
visible rays in local areas, achieving occlusion-aware modeling and rendering.
To do so, we first decompose the global coordinates into
local ones. We observe that encoding a large scene globally
within a compact representation, such as VMGrid or HashGrid, necessitates a substantial number of factors or levels. Consequently, the entire representation must be loaded
when querying features, leading to inefficiencies in computations and GPU memory utilization. To address theseissues, we propose to decompose the scene into multiple
local representations based on the camera trajectory. This
leads to flexible scene modeling when dealing with diverse
natural scenes that vary in extent and spatial granularity.
We leverage the fact that a full light field can be seen
as an integration of light field samples, where each lightfield sample can be approximated by blending a collectionof discrete light field images - which have been widely applied in Image Based Rendering (IBR) [28]. We therefore
present a novel light field feature probe representation to encode light field radiance and geometry as multiple-channel
spherical feature maps. In this case, light field samples canbe approximated by aggregating probe features. Unlike therecent popular scene contraction mechanism [6], our work
relaxes the assumption of a single scene center by retaining multiple representation centers. With this approach,
we locally encode the radiance and geometry of the scene,
thereby facilitating decomposition and reconstruction.
Our work aims to improve grid-based representations tohandle scenes with arbitrary scales within a compact andefficient framework. Towards this goal, we present a novel
Vector-Matrix-Matrix (VMM) decomposition techniquethat effectively represents the light field feature probes as
the products of a core factor (i.e., VM) which is shared
among local feature probes, and a basis factor (i.e., M)
which is designated exclusively for each probe. The spatial sharing mechanism facilitates the uncovering of hidden
relationships and patterns within the scene, while reducing
the number of light field feature probes required to maintainexpressivity. With VMM factorization, our work compactly
encodes the entire scene content within dispersed light fieldfeature probes across the spatial domain, achieving compact
encoding while also preserving locality.
We further propose a soft localization and blending algorithm to enable fast reconstruction and aliasing-free rendering. Specifically, we aim to construct a target image by
selecting K neighboring feature probes based on the dis-

tance between the target camera center and the probe centers. Then, instead of representing each probe feature individually through factor multiplication, we apply efficientweighted blending on each factor group before multiplication. It should be noted that our feature query differs ina principled way from previous grid-based representations.
We index point features on camera-adjacent probes rather
than point-adjacent probes. Together with the local spherical projections, this enables storing of a 3D point at different probes in a form of multi-scale pyramid (also known asmipmaps) - nearby areas are modeled at higher resolution,
while distant areas are represented at lower resolution. This
conforms to the perspective projection and allows querying
the mip feature without tracing cones [6, 21] or multiple
samples [8]. Additionally, the neighboring probe selection
scheme demonstrates 3D scene locality and is able to facilitate training and rendering on a large scale without the needto load the entire representation into memory. For continuous light field factorization, as inspired by TensoRF [12],we apply linear factor interpolation for the query points.
We conduct extensive experiments on diverse scenes, including both small-scale, i.e., mip-NeRF360 [6], Free [62],middle-scale KITTI360 [30] dataset) and large-scale, i.e.,
Google Earth [66], KITTI-360-large [30] and website
video. We demonstrate that all models are able to achieve
realistic novel view synthesis results that are on-par-withor better than state-of-the-art methods, especially on largescale scenarios, providing noticeable improvements while
preserving fast reconstruction and high compactness.

2. Related Works
Light fields. Light fields capture the spatial-angular distribution of radiance in a scene. They were first introduced
by Levoy and Hanrahan and enable novel view synthesis
without the need for explicit 3D models [28]. Gortler et
al. [11, 17, 20, 65] extended this idea with proxy geometry
and improved the rendering quality using approximate geometry. The advent of neural networks brought a paradigm
shift in light field rendering and early works [16, 43] use
MLPs to implicitly model light fields and directly predictray colors. [29, 60] propose novel light field representations for efficient rendering. Similarly, [39, 75] propose topredict blending weights based on image features and ray
distances.
Directly representing a full light field with a neural network is extremely difficult because of its complexity. To
address this problem, Light Field Networks [50] leverage an implicit MLP together with 6D Plücker parameterization of the space of light rays, allowing 360-degree
light field modeling and demonstrating success in representing scenes and objects with simple texture. Subsequent
works [4, 23, 51, 63, 64] incorporate light field rendering
2with volumetric integration to achieve high fidelity rendering, especially in high reflectent and translucent regions.
However, they are limited to forward-facing or small-scale
synthetic scenes.
Light Field Probes were designed to encode a scene’s fulllight field and internal visibility by placing a probe array inthe scene, allowing a quick rendering of global illumination
effects [35, 49]. Our work is inspired by probe representation. However, unlike traditional rendering setups, our work
assumes that the geometry and radiance of the scene are unknown and attempts to learn and reconstruct the scene withgradient descent. We refer the readers to [46, 55] for a morecomprehensive view.

from the Internet. While [27] improve accuracy, they still
mainly focus on reconstructing point clouds with diffuse
color, and use Poisson reconstruction to obtain meshes,lacking the ability to render realistic images from reconstructed representations.
Recent studies [37, 38, 53, 58, 66, 67] have integrated
NeRF with the concept of decomposing large-scale scenes
into a union of local blocks while maintaining individual
representations for each block. In practice, this localized
approach introduces optimization challenges, often requiring the incorporation of smoothing regularization or additional supervision to enhance the stability. To address challenges such as inconsistent exposure and moving objects
in the capture of large scenes, NeRF-W [33] and BlockNeRF [53] present an uncertainty and latent appearance
modeling, which significantly reduce the reconstruction“floaters”. It is worth mentioning that 3D Gaussian Splatting [24] doesn’t rely on scene contraction but with a projection mechanism, showing potential for large-scale scene reconstruction. However, the primitive-based representation
can easily overfit to the training views and thus requires agood initial point cloud and dense training views. Our representation is between local radiance fields and primitive
representations, utilizes only a few dozen local probes, anddoes not require a point cloud prior.

Neural radiance field. Advanced neural representations
are revolutionizing the reconstruction field as a promising
replacement for traditional representations. They show significant progress in improving reconstruction quality and efficiency in a wide range of graphics and vision applications,
including novel view synthesis [3, 12, 31, 32, 40, 56, 59,
68, 75], surface reconstruction [42, 61, 69–71], appearance
acquisition [9, 10, 22, 72, 73], etc. In contrast to standard
explicit representations such as meshes, point clouds, implicit representations [15, 36, 44] provide continuous andsmooth representations of surfaces.
NeRF [40] represents the neural radiance field for novel
view synthesis using MLPs with positional encoding forscene representation. However, its reconstruction process istime-consuming. To address this, recent works have introduced learnable feature grids [2, 12, 41, 52] for faster optimization, achieving reconstructions in just 5 minutes. However, these representations, along with state-of-the-art methods [6, 8], assume object-centric scenarios and use scene
contraction, which limits their scalability. To achieve compact reconstruction and constrain the reconstruction process, TensoRF uses various matrix factorizations to modela radiance field. However, it is scene-centric and fails tomodel large-scale scenes. In contrast, F2 -NeRF [62] identifies a 3D subspace where the distance between any two
points is equal to the sum of their 2D perspective distancesacross all visible cameras. However, F2 -NeRF still depends
on the HashGrid representation, which limits its suitability
for long trajectories or large-scale scenes. Our approach, on
the other hand, defines the target space as a set of local 2D
subspaces (light field probes) rather than a homogeneous
3D space. This naturally preserves the perspective distancesand provides flexibility when modeling scenes with different scales.

3. Neural Light Field Probes
Our goal is to synthesize the novel views of the diverse
scenes that vary in scale and spatial granularity, from acollection of posed multi-view images. In the following
sections, we first present our scene parameterization which
bridges the gap between light field probes and volume rendering. We then introduce a factorization algorithm that efficiently factorizes radiance and geometric information forlight field queries. Finally, we describe in detail how to
distribute the probes during initialization and select themduring querying for high-fidelity rendering.

3.1. Volumetric Light Field Rendering
A continuous light field L(x, d) encodes the spatial-angular
distribution of radiance for all receivers x ∈ R3 over theunit outgoing directions d ∈ S2 . We build a sparse setof discrete representations of this distribution: Ll (d) :=L(pl , d), l ∈ [1, · · · , L] (Light Field Probes), where eachprobe captures the angular variation as seen from the probe
position pl . The light field at any receiver x can be reconstructed by calculating a spatial average from the radianceof the probes.

Large-scale 3D reconstruction. Numerous works havesucceeded in reconstructing large-scale scenes using standard structure-from-motion techniques or recent neural radiance fields. Early work [1] presents a system that canmatch and reconstruct entire cities from images collected  \LF \approx \sum _{l=1}^L \mathrm {w}_l(\Dir ) \cL _l(\Dir ) \label {eqn:light_field} 3(1)

Figure 2. NeLF-Pro pipeline. Our approach models a scene using light field probes represented by a set of core vectors V̄, matricesM̄ as well as a set of basis matrices M. We project the sample points onto local probes to obtain local spherical coordinates and thenquery factors from probes near the camera. These factors are aggregated using a self-attention mechanism and combined using Hadamard
products. The resulting factor τl (x) and the viewing direction d are used to calculate the density σ and the light field radiance L(x, d).

P
where wl (d) = w̃l (d)/ k w̃k (d), which denotes the normalization weighting depending on the distance between x
and pl .
Note that the main difference between light field and radiance field [40] rendering is that, light field rendering directly utilizes the distribution of light in ray space, while
radiance field reconstructs the light distribution in a volumetric space by integration.
Volumetric light field rendering combines the advantagesof both methods and proposes to divide the target ray into
a sequence of intervals between the near and far ranges andto sum over the light field segments. The light segment ofeach interval is determined by weighted blending of lightfield samples from the multi-view images:

However, these neural Image Based Rendering methods
like IBRNet [63] can easily lead to jumping artifacts whenmoving the camera and fail to accurately estimate geometry
due to appearance bias. Instead of directly using images,
we introduce light probes to model light field samples. Our
work assumes that the fusion network Φ and the light fieldprobes Ll (v) are both unknown and optimizes these parameters by gradient descent. In practice, the light field probes
and the fusion network can easily be overfitted to the multiview inputs due to the strong flexibility and view-dependent
nature of the light field.

3.2. Light Field Factorization
We now present our novel light field probe representation
for stable light field reconstruction. Our model consists ofspatially distributed feature probes, each representing theradiance and geometry as multi-channel features.
We consider the discrete spherical probes as a set of 3D
tensors T = {τl ∈ RH×W ×D }L
l=1 , where each local probeτl denotes a D-layered spherical grid located at pl , W, H
are the spherical dimensions, providing full angular coverage. We propose to use low-dimensional factors to represent the light field tensors T for compact modeling, and introduce a novel vector-matrix-matrix (VMM) factorization,
where a factor V̄ , M̄ , serves as core factors and captures themultilinear interactions between different factors, accompanied by basis factors M located at position x. VMM factorization factorizes each tensor element τlijk as a sum of R
scalar products:

 \LF \approx \sum _{i=1}^{\numsamples } \Transmit _i \alpha _i \cL (\bx _i,\Dir ) \label {eqn:volume_rendering}\\ \text {where} \quad \cL (\bx _i,\Dir ) = \sum ^L_{l=1} w_{i,l}\cL _l(\bv _{i,l})(3)where Ti are the Q
approximated accumulated transmittancesi−1defined by Ti = j=1 (1 − αj ), and αi = 1 − exp (−σi ∆i )is the approximated opacity value of sub-interval ∆i =ti+1 −ti , and σi denotes the volume density of point sample
xi = x + ti d. vi,l = (xi − pl )/ ∥xi − pl ∥2 is the normalized direction between the sample point and the position oflight field probes, wi,l are the normalized blending weightof the intervals.
To obtain the density and blending weight at position xi ,a common solution is to reuse the input images as light fieldsamples and to train a neural network Φ to predict the density and blended radiance from a set of light field queries
Ll (xi , vi,l ): \alpha _i, w_{i,l} = \Phi (\{\cL _l(\bv _{i,l})\}) \label {eqn:network_weighting}  \tensor _l^{ijk} = \sum ^R_{r=1}\ddot {\bV }_{c,r}^{k} \circ \ddot {\bM }_{c,r}^{ij} \circ \bM _{l,r}^{ij} \label {eqn:light_field_factorization} (5)where the factor V̄c,r ∈ RD is a vector of length D, and thematrix M̄c,r ∈ RH×W , Ml,r ∈ RH×W are matrix tensors,◦ denotes the Hadamard product. We use c, l as the index(4)4

indices core and basis factors respectively, i, j, k denote theindices of the three modes.

To simplify Eq. (8), we concatenate all scalar values into an R-channel vector, this yields V̄c (t̄c ) =[V̄c,1 (t̄c ), · · · , V̄c,R (t̄c )]. Similarly, we define M̄c (v̄c ) andMl (vl ). In the following, we denote Gl (x) = V̄c (t̄c ) ◦M̄c (v̄c ) ◦ Ml (vl ). If necessary, we can recover τlP(x) from
Gl (x) using a simple sum operation τl (x) =Gl (x),
though this operation is omitted with the inclusion of a neural decoder.

Distributing factors We distribute both the core and basis
factors as scene representation in 3D, and apply the Farthest
Point Sampling (FPS) [18] algorithm to select L positions
for the basis factor from the input cameras, and then use
the K-means algorithm to select C positions for the corefactors. The number of factors C, L can be set differently
and should be chosen depending on the complexity of thescene. Inspired by the factor sharing mechanism in Tucker
Decomposition [57], we propose to share the core factor
across different basis factors, resulting in C ≪ L. The
sharing mechanism enables the discovery of hidden relationships and patterns within the scene.
Unlike previous scene representations, which are usuallycentered at the scene, our approach exploits the prior information provided by the input camera positions, facilitating
the handling of scenes with arbitrary shapes. By distributing local scene representations in these regions, our methodenhances representation efficiency.

Localization. We now detail how to form an image using
a subset of probes, resulting in low memory consumption
and scalable representation. When the scene size and consequently the number of necessary probes increases, rendering becomes inefficient.
Therefore, we propose to select the Ĉ nearest core factors and the L̂ basis factors out of C and L based on the
distance between pl and the position of the target camera.
After that, only the selected factors are used when rendering a target view, enabling reconstruction and rendering oflarge scenes or using computationally restricted hardware.
In addition, such a strategy implicitly mimics a mipmap
representation. Due to the spatially distributed light fieldprobes and spherical projection transformation, the lightfield features of a 3D point are captured at multiple resolutions across different probes, where the representation
resolution is automatically adjusted based on the relative
distance between the 3D point and the probe’s center. Rendering with the feature in neighboring probes naturally encodes a proper level of scene features, where 3D content
close to the target view is represented at high resolution due
to the nature of the spherical projection, as described in Section 3.2.

Continuous factorization. To achieve a continuous field,
we employ linear interpolation on each vector and matrix
while querying the tensor grid features of a target position
x, combining with Eq. (5) we obtain,
  \tensor _l(\bx ) = \sum ^R_{r=1} \ddot {\bV }_{c,r}(\bx ) \circ \ddot {\bM }_{c,r}(\bx ) \circ \bM _{l,r}(\bx ) \label {eqn:continue_factorization} (6)where (·) denotes linear interpolation.
Coordinate transformation. To query values on the lowdimensional factors, we transform the input coordinates x
to local coordinates using a projection transformation. For
a 3D location denoted by x = (x, y, z), we compute its
distance to a factor center t̃ = ∥x − p∥2 and transform it
into local normalized depth and spherical coordinates as follows:  t = \frac {1}{\Tilde {t}+1}, \Tilde {\bv } = \left (\arccos \left (\frac {z}{\Tilde {t}}\right ),\arctan \left (\frac {y}{x}\right )\right ) \label {eqn:transformation} 

Factor aggregation.
Our work leverages local modeling and regionally selects factors for target view rendering. Naively concatenating the selected factors can introduce a permutation-variant problem. To address this problem, we propose a permutation-invariant factor blending
scheme. Given the Ĉ selected shared core and L̂ basis factors for each query point, we predict a weight for each factor
and blend the factors of same groups using a self-attention
mechanism:(7)

Following Factor Fields [13, 14], we apply a scaling factor
and a periodic function to the projected coordinate to effectively utilize the empty regions within the spherical grids,v = γ(ν · ṽ), where ν determines the frequency and γ isthe sawtooth function.
Therefore, the tensor values at location x denoted in
Eq. (6) are given by:
  \tensor _l(\bx ) = \sum ^R_{r=1} \ddot {\bV }_{c,r}(\ddot {t}_{c}) \circ \ddot {\bM }_{c,r}(\ddot {\bv }_c) \circ \bM _{l,r}(\bv _{l}) \label {eqn:feature_intepolation} 

  \cG (\bx ) = \ddot {\cV } \circ \ddot {\cM } \circ \cM ,\,\text {where} \left \{ \footnotesize \begin {aligned} \ddot {\cV } &= \sum _{i=1}^{\hat {C}} w_i^{\ddot {\bV }}\ddot {\bV }_{i}(\ddot {t}_i)\\ \ddot {\cM } &= \sum _{i=1}^{\hat {C}} w_i^{\ddot {\bM }}\ddot {\bM }_{i}(\ddot {\bv }_i)\\ \cM &= \sum _{j=1}^{\hat {L}} w_j^{\bM }\bM _{j}(\bv _j) \end {aligned} \right . \label {eqn:vm_fusion} 

(9)

(8)
where the G(x) ∈ RR is the aggregated vector.
we obtain the scaler blending weight w by a trainable

where t̄ and v̄ represent the projected coordinates of thecore factors.
5

linear transformation with sigmoid activation,
  w = \text {Sigmoid}\left (\bW f + b \right ) \label {eqn:tensor_blending_weight} (10)

here, f = V̄i (t̄i ), M̄i (v̄i ), and Mj (vj ) respectively.
To further boost the compactness, we reduce the component dimension of the basis factor from R to F by applying a linear layer to Mj (vj ) before factor aggregation, as
shown in Fig. 2.
Decoder. So far, we have introduced the factorization of thelight field features. However, unlike the direct light field
rendering in Eqn.(2), which works with radiance samples,
our light field representation is embedded in feature space.
To obtain multi-dimensional outputs, we introduce a MLP
network P to obtain the radiance and geometric density:(a) small 360 degree circling
mip-NeRF360 [6](b) short forwarding
Free [62] and KITTI360 [30](c) aerial multi-scale circling
Google Earth [66](d) free forwarding
KITTI360-big [30], City Walk

Figure 3. Datasets and trajectories used for our evaluation.

  \alpha _i, \cL (\bx _i,\Dir ) = \cP (\cG (\bx _i),\Dir ) \label {eqn:projection_network} 

(11)
We implement the core and basis factors of light fieldprobes with feature grids. We set the resolution of the core
vector V̄ to 64 and the matrix M̄ to (64, 128), progressively upscaling them to 1024 and (128, 256), respectively.
For the basis factor, we opt for a lower resolution, fixed at(256, 512) during the optimization process. For all experiments, we maintain the factor component number at R = 32
and F = 2.
Due to the difference in scale, the probe number
L and the core tensor number C vary for different
datasets. For small scale scenes, we set (L, C) =(64, 3);(100, 5);(256, 16) for the small, medium, and large
scale scenario respectively. We use L̂ = 16 neighboring basis factors to represent the target views. The core factor forneighbor selection Ĉ is 3 for centralized camera trajectories(mip-NeRF360 and 56Leonard) and 2 for others. The periodic factor ν is (5, 5) for the mip-NeRF360 dataset, (5, 2)
for KITTI middle scale and (4, 4) for others.

3.3. Optimization
We optimize the light field feature probes and the projection
network using gradient descent, and minimize a L2 loss between the reconstructed images and the multiview inputs.
Using global and local factorization as basis factors that correlate and regularize the entire field during optimization, ourmethod achieves efficient reconstruction. Furthermore, we
include the interlevel loss from mip-NeRF 360 [6] in our
optimization process.

4. Experiments
4.1. Experimental Settings
Datasets We evaluate our method on four datasets spanning
diverse scene scales and camera trajectories, including the
mip-NeRF360 dataset [6], the Free dataset [62], BungeeNeRF [66], and KITTI-360 [30], as well as a city walk videofrom Youtube, their trajectories are shown in Fig. 3. For the
KITTI-360 dataset, we use two different scales: medium
and large scales. For medium-scale scenes, we used fivestatic sequences from Novel View Appearance Synthesis(50% Drop), with a distance of about 50 meters, labeled
as KITTI-360. In addition, we selected a long trajectorywith a driving distance of approximately 1500 meters for alarge-scale evaluation, labeled as KITTI-360-big.

4.2. Comparisons
Small Scale Scenes. We consider the mip-NeRF360
and Free datasets as small-scale datasets and compare
our method with MLP and primitive-based representations,partly with voxel-grid methods, as shown in Table 1.
Our NeLF-Pro significantly minimizes the known performance gap between feature grid-based representations
and the well-trained MLP SOTA on the unbounded scenes,
achieving SOTA performance over feature grid-based methods [41, 62] on both the Free and mip-NeRF360 datasets,
while it is on-par-with mip-NeRF360 [6] on the Free
dataset.
The qualitative results are shown in Fig. 4, our approachis capable of providing better renderings, especially in recovering thin structures. Thanks to the local representation,
our method models specular reflections more accurately.

Implementations We implement our NeLF-Pro with PyTorch within the SDFStudio framework [54, 70], without
customized CUDA kernels. We train our model on a single
RTX 3090 GPU with Adam optimizer [25] and a learning
annealed in multiple steps from 10−2 to 3.5×10−4 . We use
a two-layer proposal model [6] for sampling points along
the rays, with [256, 96, 48] samples per stage. The proposalmodel is represented with our sparse core factor and is optimized by a distillation loss [6].
6Free Dataset
mip-NeRF360 Dataset
MethodIterations Batch Size Time ↓ #Params ↓ PSNR↑ SSIM↑ LPIPS ↓ PSNR↑ SSIM↑ LPIPS ↓NeRF++ [74]
250K
2K
hours
23.47
0.603
0.499
26.21
0.729
0.348
mip-NeRF360 [6]
250K
16K
hours
8.6M
27.01
0.766
0.295
28.94
0.837
0.208
Plenoxels [2]
128K
5K
25m
Dynamic
19.13
0.507
0.543
23.35
0.651
0.471
DVGO [52]
30K
5K
21m
153M
23.90
0.651
0.455
25.42
0.695
0.429
6m
12M
24.43
0.677
0.413
26.24
0.716
0.404
Instant-NGP [41]20K
10-85KF2 -NeRF [62]20K
10-85K
12m
18M
26.32
0.779
0.276
26.39
0.746
0.361
NeLF-Pro (ours)
20K
4K
16m
20M
27.02
0.802
0.256
27.27
0.753
0.360
Table 1. Results on the Free and mip-NeRF360 dataset. The scores of the baseline methods are taken from F2 -NeRF [62]. Our training
times are evaluated on an RTX3090 GPU.

MethodPSNR↑ SSIM↑ LPIPS ↓ Time ↓NeRF [40]
21.28
0.779
0.343
days
mip-NeRF [5]
21.54
0.778
0.365
days
22.07
0.820
0.221
days
PNF [26]
NeLF-Pro (ours) 22.55
0.829
0.201
45minsTable 2. Quantitative result on KITTI360-NVS-appearance50% task [30]. We report SSIM and LPIPS from the leaderboard.
BicycleNeLF-Pro (ours)

mip-NeRF

PNF

NeLF-Pro

Figure 5. Results on KITTI-360 NVS Task. Our approach provides more sharp renderings than the baselines.
Countersults to the leaderboard to obtain evaluation scores. In Table 2, we compare our method with other submissions thatused the same setting as ours, i.e., RGB supervision only.
Note that, in compliance with KITTI-360’s guidelines, we
couldn’t submit results for other methods. Thus, our comparison is limited to previous submissions. Our approachconsistently outperforms all baseline methods for all three
metrics and also provides more than 20 times faster training
speed. Fig. 5 demonstrates that our approach yields sharper
renderings compared to the baseline methods.

Grass

Hydrant

Large Scale Scenes. We also show the results of largescale reconstruction for three distinctly different types ofscenes: 56-Leonard with an input level of detail that includes a bird-eye-view, Scuol with a town in which randomly walking trajectory, and KITTI-360-big with a longstraight driving style. In our experiments, we use a four-foldlarger model with three-fold more iterations (60k iterations)
to reconstruct large scenes. We applied a similar increase inmodel size to the comparison methods.
The quantitative Table 3 and visual results Fig. 6 clearly
demonstrate the high efficiency and strong scalability of ourapproach. Impressively, we found that our approach is able
to better preserve structured patterns, which benefit fromlocal representation and modeling of core factors.

Road
Road

NeLF-Pro (ours)

Instant-NGP

F2-NeRFNeLF-Pro

GT

Figure 4. Qualitative results on mip-NeRF360 [6] and Free
dataset [62]. Our approach is able to better reconstruct thin structure and appearance details than the baseline methods (InstantNGP [41], F2 -NeRF [62]).

Furthermore, our method is efficient and shows a training
time comparable to Instant-NGP [41] and F2 -NeRF [62],both implemented with customized CUDA kernels. Notethat F2 -NeRF requires precomputing near-far values foreach training view from the calibration point cloud, while
we instead use constant near-far values without the need fora sparse point cloud.

4.3. Ablation StudyIn Table 4 we evaluate several ablations of NeLF-Pro: a-c)partial or complete removal of core factors V̄ modeling; d)“point neighborhood” queries the nearest neighbor factors

Medium Scale Scenes. Since the training views are accessible for the KITTI 360 dataset, we submitted our re7

56-Leonard
Scuol
KITTI-360-big
MethodTime ↓ #Params ↓ PSNR↑ SSIM↑ LPIPS ↓ PSNR↑ SSIM↑ LPIPS ↓ PSNR↑ SSIM↑ LPIPS ↓mip-NeRF360 [6]
13.3h
8.6M
26.22
0.829
0.200
25.20
0.800
0.381
16.63
0.537
0.527
6.9h
276M
23.42
0.731
0.318
23.65
0.763
0.510
18.55
0.548
0.607
GridNeRF [67]
3DGS [24]
2.5h
225M
29.23
0.942
0.075
22.74
0.835
0.301
15.17
0.510
0.601
F2 -NeRF-Big [62]
1h
148M
27.07
0.930
0.102
26.43
0.842
0.385
21.36
0.686
0.446
NeLF-Pro (ours)
1.6h
100M
28.86
0.930
0.108
27.84
0.855
0.368
22.68
0.720
0.399
Table 3. We show averaged results for large-scale scenes. Bold score for the first and underlined score for the second. Time andparameter number are the mean across all the scenes. In general, our method is on-par-with or outperforms other reconstruction methods.
While 3DGS and F2 -NeRF-Big achieve similar or better results than ours on the 56-Leonard scene, they perform better near the centerof the scene and less effectively for zoomed-out views. In contrast, our approach maintains high rendering quality throughout the entire
trajectory due to its distributed representation. Our representation is also more compact and requires fewer parameters.
56 Leonard

Scuol

KITTI360

NeLF-Pro (ours)

mip-NeRF360

GridNeRF

3DGS

F2-NeRFGT

NeLF-Pro

Figure 6. Results on large-scale scenes. Our representation better preserves fine structural details.

with the distance between the sample point and the centerof probes, instead of the proposed camera neighborhood; e)“random centers” distributes the light field feature probes
by randomly selecting positions from the centers of the input cameras; f) “w/o coordinate scaling ν” disables the scali
ng of the spherical coordinates by setting the scaling factorν = (1, 1); g) “factor concatenation” concatenates the selected factors before the products.
We observe that disabling the core factor V̄ results in atiny loss of quality in small scene, while disabling core factor M̄ , or the simultaneous removal of both factors leads to asignificant drop in performance. Moreover, selecting neighboring factors with the target camera outperforms the “pointneighborhood” strategy in the scene with multi-level images, benefiting from our mipmap-like representation. Our
factor aggregation model delivers significant performance
improvements over the factor concatenation baseline.

a)
b)
c)
d)
e)
f)
g)

full model
w/o core factor V̄
w/o core factor M̄
w/o core factors V̄, M̄
point neighborhood
random centers
w/o coordinate scaling ν
factor concatenation

PSNR↑
30.20
29.74
28.16
23.78
28.97
30.24
28.65
27.68

Room
SSIM↑
0.871
0.864
0.818
0.655
0.845
0.871
0.830
0.764

LPIPS ↓
0.367
0.382
0.434
0.550
0.414
0.365
0.465
0.477

PSNR↑
27.84
27.59
24.55
22.76
26.72
27.81
26.97
24.78

Scuol
SSIM↑
0.855
0.846
0.741
0.639
0.824
0.853
0.827
0.694

LPIPS ↓
0.368
0.382
0.518
0.595
0.417
0.371
0.421
0.547

Table 4. Ablation study on a 360-degree surrounding room scene
and the Scuol scene with free camera trajectories.

decomposes light field probes into products of local factors,
resulting in a compact representation with novel view synthesis on par with or better than state-of-the-art methods indiverse scenarios.

6. Acknowledgement
Andreas Geiger and Anpei Chen are supported by the ERC
Starting Grant LEGO-3D (850533) and DFG EXC number
2064/1 - project number 390727645. We thank Zehao Yu
and Binbin Huang for proofreading and their insightful suggestions. We also thank Siyu Tang and Kaifeng Zhao for
their valuable discussions and help. Special thanks to The
Flying Dutchman for the city walk video.

5. Conclusion
We presented a novel volumetric light field probe representation for high-quality scene reconstruction and rendering.
We have also proposed a VMM factorization technique that8

References[14] Anpei Chen, Zexiang Xu, Xinyue Wei, Siyu Tang,
Hao Su, and Andreas Geiger. Factor fields: A unified framework for neural fields and beyond. arXiv
preprint arXiv:2302.01226, 2023. 5[15] Zhiqin Chen and Hao Zhang. Learning implicit fields
for generative shape modeling. In CVPR, 2019. 3[16] ChenandAnpei, WuandMinye, ZhangandYingliang,
LiandNianyi, LuandJie, GaoandShenghua, andYuandJingyi. Deep surface light fields. In ACM
Trans. on Graphics, 2018. 2[17] Greg Coombe, Chad Hantak, Anselmo Lastra, andRadek Grzeszczuk. Online construction of surface
light fields. In ESRT, 2005. 2[18] Yuval Eldar, Michael Lindenbaum, Moshe Porat, andYehoshua Y. Zeevi. The farthest point strategy for progressive image sampling. TIP, 1997. 5[19] Quankai Gao, Qiangeng Xu, Hao su, Ulrich Neumann,and Zexiang Xu. Strivec: Sparse tri-vector radiancefields. In ICCV, 2023. 1[20] Steven J Gortler, Radek Grzeszczuk, Richard Szeliski,
and Michael F Cohen. The lumigraph. In Seminal
Graphics Papers: Pushing the Boundaries. 2023. 2[21] Wenbo Hu, Yuling Wang, Lin Ma, Bangbang Yang,
Lin Gao, Xiao Liu, and Yuewen Ma. Tri-miprf: Trimip representation for efficient anti-aliasing neural radiance fields. In ICCV, 2023. 2[22] Haian Jin, Isabella Liu, Peijia Xu, Xiaoshuai Zhang,
Songfang Han, Sai Bi, Xiaowei Zhou, Zexiang Xu,
and Hao Su. Tensoir: Tensorial inverse rendering. In
CVPR, 2023. 3[23] KellnhoferandPetr, JebeandLars C, JonesandAndrew,
SpicerandRyan, PulliandKari, and WetzsteinandGordon. Neural lumigraph rendering. In CVPR, 2021. 2[24] Bernhard Kerbl, Georgios Kopanas, Thomas
Leimkühler, and George Drettakis. 3d gaussian
splatting for real-time radiance field rendering. ACM
Trans. on Graphics, 2023. 3, 8
[25] Diederik P. Kingma and Jimmy Ba. Adam: A method
for stochastic optimization. In ICLR, 2015. 6[26] Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza
Fathi, Caroline Pantofaru, Leonidas Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas
Funkhouser. Panoptic Neural Fields: A Semantic
Object-Aware Neural Scene Representation. In CVPR,
2022. 7[27] Florent Lafarge and Clément Mallet. Creating largescale city models from 3d-point clouds: a robust approach with hybrid representation. International journal of computer vision, 2012. 3[28] Marc Levoy and Pat Hanrahan. Light field rendering.
In ACM Trans. on Graphics, pages 31–42, 1996. 2[29] Zhong Li, Liangchen Song, Celong Liu, Junsong
Yuan, and Yi Xu. Neulf: Efficient novel view syn-[1] Sameer Agarwal, Noah Snavely, Ian Simon, Steven M.
Seitz, and Richard Szeliski. Building rome in a day.
In ICCV, 2009. 3[2] Alex Yu and Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo
Kanazawa. Plenoxels: Radiance fields without neural networks. CVPR, 2022. 1, 3, 7, 4
[3] Kara-Ali Aliev, Dmitry Ulyanov, and Victor S. Lempitsky. Neural point-based graphics. arXiv.org,
1906.08240, 2019. 3[4] Benjamin Attal, Jia-Bin Huang, Michael Zollhöfer,
Johannes Kopf, and Changil Kim. Learning neurallight fields with ray-space embedding networks. In
CVPR, 2022. 2[5] Jonathan T. Barron, Ben Mildenhall, Matthew Tancik,
Peter Hedman, Ricardo Martin-Brualla, and Pratul P.
Srinivasan. Mip-nerf: A multiscale representation foranti-aliasing neural radiance fields. In ICCV, 2021. 7[6] Jonathan T. Barron, Ben Mildenhall, Dor Verbin,
Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360:
Unbounded anti-aliased neural radiance fields. 2022.
1, 2, 3, 6, 7, 8
[7] Jonathan T Barron, Ben Mildenhall, Dor Verbin,
Pratul P Srinivasan, and Peter Hedman. Mip-nerf
360: Unbounded anti-aliased neural radiance fields.
In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470–
5479, 2022. 1, 3, 4
[8] Jonathan T. Barron, Ben Mildenhall, Dor Verbin,
Pratul P. Srinivasan, and Peter Hedman. Zip-nerf:
Anti-aliased grid-based neural radiance fields. ICCV,
2023. 2, 3, 1
[9] Mark Boss, Raphael Braun, Varun Jampani,
Jonathan T Barron, Ce Liu, and Hendrik Lensch.
Nerd: Neural reflectance decomposition from image
collections. In ICCV, 2021. 3[10] Mark Boss, Varun Jampani, Raphael Braun, Ce Liu,
Jonathan Barron, and Hendrik Lensch. Neural-pil:
Neural pre-integrated lighting for reflectance decomposition. Advances in Neural Information Processing
Systems, 2021. 3[11] Chris Buehler, Michael Bosse, Leonard McMillan,
Steven J. Gortler, and Michael F. Cohen. Unstructured lumigraph rendering. In ACM Trans. on Graphics, 2001. 2[12] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu,
and Hao Su. Tensorf: Tensorial radiance fields. In
ECCV, 2022. 1, 2, 3
[13] Anpei Chen, Zexiang Xu, Xinyue Wei, Siyu Tang,
Hao Su, and Andreas Geiger. Dictionary fields: Learning a neural basis decomposition. In ACM Trans. on
Graphics, 2023. 5
9thesis with neural 4d light field. arXiv preprint
arXiv:2105.07112, 2021. 2[30] Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360: A
novel dataset and benchmarks for urban scene understanding in 2d and 3d. PAMI, 2022. 2, 6, 7
[31] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng
Chua, and Christian Theobalt. Neural sparse voxel
fields. In NeurIPS, 2020. 3[32] LombardiandStephen, SimonandTomas, SaragihandJason, SchwartzandGabriel, LehrmannandAndreas,
and SheikhandYaser. Neural volumes: Learning dynamic renderable volumes from images. In ACM
Trans. on Graphics, 2019. 3[33] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M.
Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, andDaniel Duckworth. Nerf in the wild: Neural radiancefields for unconstrained photo collections. arXiv.org,
2008.02268, 2020. 3[34] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M.
Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, andDaniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In
CVPR, 2021. 1[35] Morgan
McGuire,
Mike
Mara,
Derek
Nowrouzezahrai, and David Luebke.
Real-time
global illumination using precomputed light fieldprobes. 2017. 3[36] Lars Mescheder, Michael Oechsle, Michael Niemeyer,
Sebastian Nowozin, and Andreas Geiger. Occupancy
networks: Learning 3d reconstruction in function
space. In CVPR, 2019. 3[37] Andreas Meuleman, Yu-Lun Liu, Chen Gao, Jia-Bin
Huang, Changil Kim, Min H. Kim, and Johannes
Kopf. Progressively optimized local radiance fields
for robust view synthesis. In CVPR, 2023. 1, 3[38] Zhenxing Mi and Dan Xu. Switch-nerf: Learning
scene decomposition with mixture of experts for largescale neural radiance fields. In ICLR, 2023. 3[39] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz
Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: practical view synthesis with prescriptive sampling guidelines. In ACM Trans. on Graphics, 2019.
2[40] MildenhallandBen, SrinivasanandPratul P, TancikandMatthew, BarronandJonathan T, RamamoorthiandRavi, and NgandRen. NeRF: Representing scenes
as neural radiance fields for view synthesis. In ECCV,
2020. 1, 3, 4, 7
[41] Thomas Müller, Alex Evans, Christoph Schied, andAlexander Keller. Instant neural graphics primitives
with a multiresolution hash encoding. ACM Trans. on
Graphics, 2022. 1, 3, 6, 7, 4[42] Michael Niemeyer, Lars M. Mescheder, Michael
Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations
without 3d supervision. In CVPR, 2020. 3[43] OechsleandMichael,
NiemeyerandMichael,MeschederandLars, StraussandThilo, and GeigerandAndreas. Learning implicit surface light fields.
arXiv.org, 2003.12406, 2020. 2[44] Jeong Joon Park, Peter Florence, Julian Straub,
Richard A. Newcombe, and Steven Lovegrove.
Deepsdf: Learning continuous signed distance functions for shape representation. In CVPR, 2019. 3[45] Marie-Julie Rakotosaona,
Fabian Manhardt,
Diego Martı́n Arroyo, Michael Niemeyer, Abhijit Kundu, and Federico Tombari. Nerfmeshing:Distilling neural radiance fields into geometricallyaccurate 3d meshes. arXiv.org, 2023. 1[46] Ravi Ramamoorthi. Nerfs: The search for the best 3d
representation. arXiv.org, 2023. 3[47] Christian Reiser, Songyou Peng, Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiancefields with thousands of tiny mlps. In ICCV, 2021. 1[48] Sara Sabour, Suhani Vora, Daniel Duckworth, Ivan
Krasin, David J. Fleet, and Andrea Tagliasacchi. Robustnerf: Ignoring distractors with robust losses. In
CVPR, 2023. 1[49] Ari Silvennoinen and Jaakko Lehtinen. Real-time
global illumination by precomputed local reconstruction from sparse radiance probes. ACM Trans. on
Graphics, 2017. 3[50] Vincent Sitzmann, Semon Rezchikov, William T.
Freeman, Joshua B. Tenenbaum, and Fredo Durand.
Light field networks: Neural scene representations
with single-evaluation rendering. In NeurIPS, 2021.
2[51] Mohammed Suhail, Carlos Esteves, Leonid Sigal, andAmeesh Makadia. Light field neural rendering. In
CVPR, 2022. 2[52] Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct
voxel grid optimization: Super-fast convergence forradiance fields reconstruction. CVPR, 2022. 1, 3, 7, 4[53] Matthew Tancik, Vincent Casser, Xinchen Yan,
Sabeek Pradhan, Ben P. Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, and Henrik Kretzschmar.
Block-nerf: Scalable large scene neural view synthesis. In CVPR, 2022. 1, 3
[54] Matthew Tancik, Ethan Weber, Evonne Ng, Ruilong
Li, Brent Yi, Justin Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa.
Nerfstudio: A modular framework for neural radiancefield development. In ACM Trans. on Graphics, 2023.
6
10

[55] Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul
Srinivasan, Edgar Tretschk, Wang Yifan, Christoph
Lassner, Vincent Sitzmann, Ricardo Martin-Brualla,
Stephen Lombardi, et al. Advances in neural rendering. In Computer Graphics Forum, 2022. 3[56] Justus Thies, Michael Zollhöfer, and Matthias
Nießner. Deferred neural rendering: image synthesis
using neural textures. ACM Trans. on Graphics, 2019.
3[57] Ledyard R Tucker. Some mathematical notes on threemode factor analysis. Psychometrika, 1966. 5[58] Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. Mega-nerf: Scalable construction of largescale nerfs for virtual fly-throughs. In CVPR, 2022. 1,
3[59] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd
Zickler, Jonathan T. Barron, and Pratul P. Srinivasan.
Ref-NeRF: Structured view-dependent appearance forneural radiance fields. 2022. 3[60] Huan Wang, Jian Ren, Zeng Huang, Kyle Olszewski,
Menglei Chai, Yun Fu, and Sergey Tulyakov. R2l:
Distilling neural radiance field to neural light field for
efficient novel view synthesis. In European Conference on Computer Vision, pages 612–629. Springer,
2022. 2[61] Peng Wang, Lingjie Liu, Yuan Liu, Christian
Theobalt, Taku Komura, and Wenping Wang. Neus:
Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In NeurIPS, 2021.
3[62] Peng Wang, Yuan Liu, Zhaoxi Chen, Lingjie Liu, Ziwei Liu, Taku Komura, Christian Theobalt, and Wenping Wang. F2-nerf: Fast neural radiance field training
with free camera trajectories. 2023. 1, 2, 3, 6, 7, 8, 4[63] Qianqian Wang, Zhicheng Wang, Kyle Genova,
Pratul P. Srinivasan, Howard Zhou, Jonathan T. Barron, Ricardo Martin-Brualla, Noah Snavely, andThomas A. Funkhouser. Ibrnet: Learning multi-view
image-based rendering. In CVPR, 2021. 2, 4[64] Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasorn Suwajanakorn.
Nex: Real-time view synthesis with neural basis expansion. In CVPR, 2021. 2[65] Daniel N. Wood, Daniel I. Azuma, Ken Aldinger,
Brian Curless, Tom Duchamp, David Salesin, andWerner Stuetzle. Surface light fields for 3d photography. In ACM Trans. on Graphics, 2000. 2[66] Yuanbo Xiangli, Linning Xu, Xingang Pan, Nanxuan Zhao, Anyi Rao, Christian Theobalt, Bo Dai, andDahua Lin. Bungeenerf: Progressive neural radiance field for extreme multi-scale scene rendering. In
ECCV, 2022. 1, 2, 3, 6

[67] Linning Xu, Yuanbo Xiangli, Sida Peng, Xingang
Pan, Nanxuan Zhao, Christian Theobalt, Bo Dai, andDahua Lin. Grid-guided neural radiance fields forlarge urban scenes. In CVPR, 2023. 1, 3, 8
[68] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi,
Zhixin Shu, Kalyan Sunkavalli, and Ulrich Neumann.
Point-nerf: Point-based neural radiance fields. In
CVPR, 2022. 3[69] YarivandLior, GuandJiatao, KastenandYoni, and LipmanandYaron. Volume rendering of neural implicit
surfaces. In NeurIPS, 2021. 3[70] Zehao Yu, Anpei Chen, Bozidar Antic, Songyou
Peng, Apratim Bhattacharyya, Michael Niemeyer,
Siyu Tang, Torsten Sattler, and Andreas Geiger. Sdfstudio: A unified framework for surface reconstruction,
2022. 6[71] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten
Sattler, and Andreas Geiger. Monosdf: Exploring
monocular geometric cues for neural implicit surface
reconstruction. In NeurIPS, 2022. 3[72] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala,
and Noah Snavely. Physg: Inverse rendering withspherical gaussians for physics-based material editing
and relighting. In CVPR, 2021. 3[73] Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu,
Rongfei Jia, and Xiaowei Zhou. Modeling indirect
illumination for inverse rendering. In CVPR, 2022. 3[74] ZhangandKai, RieglerandGernot, SnavelyandNoah,and KoltunandVladlen. Nerf++: Analyzing and improving neural radiance fields. arXiv.org, 2010.07492,
2020. 1, 7, 3, 4[75] Tinghui Zhou, Richard Tucker, John Flynn, Graham
Fyffe, and Noah Snavely. Stereo magnification: learning view synthesis using multiplane images. ACM
Trans. on Graphics, 2018. 2, 3

11

NeLF-Pro: Neural Light Field Probes for Multi-Scale Novel View Synthesis
Supplementary Material
In this supplementary document, we present visualizations of probe distribution, provide detailed analyses of theablations along with observations, discuss the limitations
and future work, and offer more quantitative and qualitative
results.

model provides high-quality reconstruction and better stability for diverse natural scenes that vary in extent and spatial granularity.

C. Limitations and Future Work.
Our NeLF-Pro demonstrates a remarkable ability to achieve
high-fidelity novel view synthesis across a wide range ofnatural scenes, with various levels of spatial granularity.
However, our approach currently cannot handle distractions, such as exposure changes and moving objects, andour method tends to produce ‘floaters’ in these regions.
Combining our method with techniques like feature space
regularization [2], appearance embedding [34] or pixel-wiseloss reweighting [48] could be beneficial. In addition, ourwork cannot support fast rendering, it still takes about 2.6
seconds to render a 960×540 image. Incorporating fast rendering techniques, such as post-baking [45] is an orthogonal
direction to our work.

A. Light Field Probes Distribution.
We propose distributing light field feature probes near thecamera trajectory, utilizing the Farthest Point Sampling(FPS) algorithm to selectively determine L̂ positions. The
distribution of these feature probes within the scene andtheir corresponding content is depicted in Fig. 7. To enhance the visualization of the probe distribution within target scenes, we additionally present the calibration point
cloud. These calibration point clouds are not used in training and are only for visualization purposes.

B. Ablation Details.

Future Work. To further enhance the per-scene reconstruction quality, a possible solution is to integrate our representation with advanced multi-level modeling [41] or cone
tracing point sampler [7, 8]. We leave this combination as
future work. Furthermore, while this paper demonstrates
success in per-scene optimization, another interesting direction for future work involves the exploration and learning ofgeneral core and basis factors across various scenes. By analyzing a large-scale dataset, we can leverage data priors
not only to improve quality but also to potentially enable
new applications, such as in the development of generativemodels. In the ablation study, a notable finding is that matrix factors alone, without the vector core factor, can yieldsatisfactory reconstructions in small-scale scenes. This aspect is particularly advantageous for convolutional neuralnetworks.

We conduct ablations on key components, including thepoint query scheme, probe distributions, period factor, andfactor aggregation module. These ablations are performed
on a small indoor scene and a larger scene encompassing
distant and close viewpoints.
Our observations are as follows:• Core factor modeling is critical. Disabling either factor V̄
or M̄ hampers reconstruction capabilities. Factor M̄ significantly influences overall quality, while factor V̄ improves fine-grained detail reconstruction qualitatively.
• Selecting factors close to the target camera yields substantial improvements, particularly for the objects withmulti-scale appearances, where the camera neighborhood
strategy effectively simulates a mipmap representation.
• Smaller scenes show less sensitivity to probe distribution
than larger scenes. Farthest point position sampling (FPS)
provides a more uniformly distributed set of probes, facilitating stable reconstruction across different areas.
• Coordinate scaling notably improves performance, especially in scenes captured from a small angle range. For
instance, in the Free dataset, where most images are captured from a single viewing direction (as depicted in the
middle of Fig. 7). The scale factor enables a more efficient use of the model capabilities.
• Factor aggregation delivers significant performance
boosts than naive feature concatenation, benefiting fromthe order-invariant weighted fusion design.

D. More Visual Results.
We show more qualitative results in Fig. 9, Fig. 10, andFig. 11. Our NeLF-Pro is able to better preserve detail,sharpness, and thin structures more effectively than thebaselines.

E. Per-scene Breakdown.
In Table 5 and Table 6, we provide breakdowns of the quantity metrics for the Free dataset [62] and the mip-NeRF360
dataset [7]. Our NeLF-Pro achieves consistently better rendering quality compared to previous grid-based approaches.

In Fig. 8, we qualitatively illustrate the performance differences resulting from various design choices. Our full1

(a) Garden(c) Scuol(b) Grass

Figure 7. The distribution of light field feature probes. The green spheres denote basis factors and the blue dots are core factors. We
densely distribute the basis factors in the scene and sparsely distribute the core factors, only using 3 for the small scenes, and 16 for the
Scuol scene shown on the right side.

full

GT

full

w/o core!
factor 𝐕

w/o core!
factor 𝐌

w/o core!, 𝐌!
factor 𝐕

point
neighboring

randomcenters

w/o coordinate
factorscaling
concatenation

Figure 8. Qualitative results for ablation study. We conduct the ablations using one indoor-small-scene and outdoor-large-scene.

2Hydrant

Lab

Road

Sky

Grass

NeLF-Pro (ours)

F2-NeRF

Instant-NGP

NeLF-Pro (ours)

GT

Figure 9. More Qualitative Results on the Free Dataset [62].

MethodNeRF++ [74]
mip-NeRF360 [7]Plenoxels [2]
DVGO [52]Instant-NGP [41]F2 -NeRF [62]NeLF-Pro (ours)
Our SSIM
Our LPIPS

Hydrant
22.21
25.03
19.82
22.10
22.30
24.34
24.92
0.770
0.260

Lab
21.82
26.57
18.12
23.78
23.21
25.92
26.39
0.834
0.251

Pillar
25.73
29.22
18.74
26.22
25.88
28.76
29.56
0.818
0.233

Road
23.29
27.07
21.31
23.53
24.24
26.76
27.65
0.834
0.231

Sky
23.91
26.99
18.22
24.26
25.80
26.41
27.06
0.873
0.217

Stair
26.08
29.79
21.41
26.65
27.79
29.19
29.55
0.853
0.203

Grass
21.26
24.39
16.28
20.75
21.82
22.87
24.00
0.629
0.398

Avg
23.47
27.01
19.13
23.90
24.43
26.32
27.02
0.802
0.256

Table 5. Per-Scene breakdown on the Free dataset. The baseline method scores are sourced from F2 -NeRF; however, F2 -NeRF does not
furnish detailed per-scene breakdowns for SSIM and LPIPS metrics.

3BonsaiCounter

Garden

Room

Stump

F2-NeRF

Instant-NGP

NBeLF-Pro (ours)

NeLF-Pro (ours)

GT

Figure 10. More novel view synthesis results on mip-NeRF360 Dataset [7].

MethodNeRF++ [74]
mip-NeRF360 [7]Plenoxels [2]
DVGO [52]Instant-NGP [41]F2 -NeRF [62]NeLF-Pro (ours)
Our SSIM
Our LPIPS

Bicycle
22.64
23.99
21.39
22.12
22.08
22.11
22.42
0.496
0.500

Bonsai
29.15
33.06
23.65
27.80
29.86
29.65
31.28
0.907
0.289

Counter
26.38
29.51
25.23
25.76
26.37
25.36
27.52
0.823
0.366

Garden
24.32
26.10
22.71
24.34
24.26
24.76
25.08
0.691
0.331

Kitchen
27.80
32.13
24.00
26.00
28.27
28.97
29.79
0.868
0.240

Room
28.87
31.53
26.38
28.33
28.90
29.30
30.10
0.871
0.367

Stump
24.34
26.27
20.08
23.59
23.93
24.60
24.58
0.615
0.427

Avg
26.21
28.94
23.35
25.42
26.24
26.39
27.27
0.753
0.360

Table 6. Per-Scene breakdown on the mip-NeRF360 dataset. The baseline method scores are sourced from F2 -NeRF; however, F2 -NeRF
does not furnish detailed per-scene breakdowns for SSIM and LPIPS metrics.

4

56 Leonard

56 Leonard

Scuol

Scuol

Scuol

KITTI360

KITTI360

KITTI360

NeLF-Pro (ours)

mip-NeRF360

GridNeRF

3DGS

F2-NeRF

Figure 11. More novel view synthesis results on the Large Scale scenes.

5

NeLF-Pro

GT