Factor Fields: A Unified Framework for Neural Fields and Beyond
Anpei Chen1,4

Zexiang Xu2
Xinyue Wei3
Siyu Tang1
Hao Su3
Andreas Geiger4,51
2
3
ETH Zürich
Adobe Research
University of California, San Diego
4
5
University of TübingenTübingen AI Center

Point xPolynomial

MLPs

f1 (γ1 (x))

Input Domain···
Coordinate Transformation
γ(x)

Grids

fN (γN (x))· · · Vectors

Field Representation
f (x)

···
Field Connector◦

Projection
P(x)Figure 1: Factor Fields is a framework which unifies many recently proposed neural field representations (e.g., NeRF, EG3D,
Instant-NGP, TensoRF) and allows for the creation of powerful new ones such as the proposed Dictionary Factorization. In
particular, Factor Fields decomposes a signal into N factors f1 to fN (top-center), each of which is represented by one out ofmany different field representations operating on coordinate transformations γ1 to γN . The resulting product field is passed toa projection function (e.g., MLP) which maps it to the signal’s output.

Abstract
We present Factor Fields, a novel framework for modelingand representing signals. Factor Fields decomposes a signal
into a product of factors, each represented by a classical orneural field representation which operates on transformed
input coordinates. This decomposition results in a unified
framework that accommodates several recent signal representations including NeRF, Plenoxels, EG3D, Instant-NGP,
and TensoRF. Additionally, our framework allows for thecreation of powerful new signal representations, such as
the ”Dictionary Field” (DiF) which is a second contribution of this paper. Our experiments show that DiF leads toimprovements in approximation quality, compactness, andtraining time when compared to previous fast reconstruction methods. Experimentally, our representation achieves
better image approximation quality on 2D image regression tasks, higher geometric quality when reconstructing 3D
signed distance fields, and higher compactness for radiancefield reconstruction tasks. Furthermore, DiF enables generalization to unseen images/3D scenes by sharing bases
across signals during training which greatly benefits use
cases such as image regression from sparse observations
and few-shot radiance field reconstruction. Our code isavailable at https://apchenstu.github.io/FactorFields/.

1. Introduction
Effectively representing multi-dimensional digital content– like 2D images or 3D geometry and appearance – is criticalfor computer graphics and vision applications. These digital signals are traditionally represented discretely as pixels,
voxels, textures, or polygons. Recently, significant headway
has been made in developing advanced neural representations [36, 51, 37, 10, 49], which demonstrated superiority inmodeling accuracy and efficiency over traditional representations for different image synthesis and scene reconstruction
applications.

In order to gain a better understanding of existing representations, make comparisons across their design principles, and create powerful new representations, we propose
Factor Fields, a novel mathematical framework that unifies
many previous neural representations for multi-dimensionalsignals. This framework offers a simple formulation formodeling and representing signals.
Our framework decomposes a signal by factorizing it
into multiple factor fields (f1 , . . . , fN ) operating on suitably chosen coordinate transformations (γ1 , . . . , γN ) as illustrated in Fig. 1. More specifically, each factor field decodes multi-channel features at any spatial location of acoordinate-transformed signal domain. The target signal isthen regressed from the factor product via a learned projection function (e.g., MLP).
Our framework accommodates most previous neural representations. Many of them can be represented in our framework as a single factor with a domain transformation – forexample, the MLP network as a factor with a positional encoding transformation in NeRF [36], the tabular encoding
as a factor with a hash transformation in Instant-NGP [37],and the feature grid as a factor with identity transformation in DVGO [51] and Plenoxels [16]. Recently, TensoRF[10] introduced a tensor factorization-based representation,
which can be seen as a representation of two Vector-Matrix
or three CANDECOMP-PARAFAC decomposition factors
with axis-aligned orthogonal 2D and 1D projections as transformations. The potential of a multi-factor representation
has been demonstrated by TensoRF, which has led to superior quality and efficiency on radiance field reconstruction
and rendering, while being limited to orthogonal transformations.
This motivates us to generalize previous classic neuralrepresentations via a single unified framework which enables
easy and flexible combinations of previous neural fields andtransformation functions, yielding novel representation designs. As an example, we present Dictionary Field (DiF),
a two-factor representation that is composed of (1) a basis
function factor with periodic transformation to model thecommonalities of patterns that are shared across the entire
signal domain and (2) a coefficient field factor with identity
transformation to express localized spatially-varying featuresin the signal. The combination of both factors allows for an
efficient representation of the global and local properties ofthe signal. Note that, most previous single-factor representations can be seen as using only one of such functions – eithera basis function, like NeRF and Instant-NGP, or a coefficient
function, like DVGO and Plenoxels. In DiF, jointly modeling
two factors (basis and coefficients) leads to superior quality
over previous methods like Instant-NGP and enables compact and fast reconstruction, as we demonstrate on various
downstream tasks.
As DiF is a member of our general Factor Fields family,

we conduct a rich set of ablation experiments over the choiceof basis/coefficient functions and basis transformations. We
evaluate DiF against various variants and baselines on threeclassical signal representation tasks: 2D image regression,
3D SDF geometry reconstruction, and radiance field reconstruction for novel view synthesis. We demonstrate that
our factorized DiF representation is able to achieve state-ofthe-art reconstruction results that are better or on par withprevious methods, while achieving superior modeling efficiency. For instance, compared to Instant-NGP our method
leads to better reconstruction and rendering quality, while
effectively halving the total number of model parameters(capacity) for SDF and radiance field reconstruction, demonstrating its superior accuracy and efficiency.
Moreover, in contrast to recent neural representations
that are designed for purely per-scene optimization, ourfactorized representation framework is able to learn basis
functions across different scenes. As shown in preliminary
experiments, this enables learning across-scene bases frommultiple 2D images or 3D radiance fields, leading to signal
representations that generalize and hence improve reconstruction results from sparse observations such as in the
few-shot radiance reconstruction setting. In summary,• We introduce a common framework Factor Fields thatencompasses many recent radiance field / signal representations and enables new models from the Factor
Fields family.
• We propose DiF, a new member of the Factor Fields
family representation that factorizes a signal into coefficient and basis factors which allows for exploiting
similar signatures spatially and across scales.
• Our model can be trained jointly on multiple signals,
recovering general basis functions that allow for reconstructing parts of a signal from sparse or weak observations.
• We present thorough experiments and ablation studies that demonstrate improved performance (accuracy,
runtime, memory), and shed light on the performance
improvements of DiF vs. other models in the Factor
Fields family.

2. Factor Fields
We seek to compactly represent a continuous Qdimensional signal s : RD → RQ on a D-dimensional
domain. We assume that signals are not random, but structured and hence share similar signatures within the same
signal (spatially and across different scales) as well as between different signals. In the following, we develop ourfactor fields model step-by-step, starting from a standard
basis expansion.

We generalize our model to Q-dimensional signals f (x) byintroducing a projection function P : RK → RQ and replacing the inner product with the element-wise/Hadamard
product (denoted by ◦ in the following):
  \hat {\bs }(\bx ) = \cP \left (\bc (\bx ) \circ \bb \left (\bgamma (\bx )\right )\right ) \label {eq:expansion_3} (a) 1D Illustration(b) 2D Example

Figure 2: Local Basis. (a) Choosing a (periodic) coordinate transformation γ(x) allows for applying the same basis
function b(x) at multiple spatial locations and scales. For
clarity, we have chosen constant coefficients c = 1. (b)Composing multiple bases at different spatial resolutions
with their respective coefficients yields a powerful representation for signal s(x). In practice, we use multiple bases andcoefficient fields at each resolution.

We refer to Eq. (3) as Dictionary Field (DiF). Note that incontrast to the scalar product c⊤ b in Eq. (2), the output ofc ◦ b is a K-dimensional vector which comprises the individual coefficient-basis products as input to the projection
function P which itself can be either linear or non-linear.
In the linear case, we have P(x) = Ax with A ∈ RQ×K .
Furthermore, note that for Q = 1 and A = (1, . . . , 1) werecover Eq. (2) as a special case. The projection operator
P can also be utilized to model the volumetric rendering
operation when reconstructing a 3D radiance field from 2D
image observations as discussed in Section 2.3.
Factor Fields: To allow for more than 2 factors, we generalize Eq. (3) to our full Factor Fields framework by replacing
the coefficients c(x) and basis b(x) with a set of factor fields{fi (x)}N
i=1 :  \hat {\bs }(\bx ) = \cP \left (\prod _{i=1}^N \bff _i\left (\bgamma _i(\bx )\right )\right ) \label {eq:expansion_4} 

Dictionary Field (DiF): Let us first consider a 1D signals(x) : RD → R. Using basis expansion, we decomposes(x) into a set of coefficients c = (c1 , . . . , cK )⊤ with ck ∈R and basis functions b(x) = (b1 (x), . . . , bK (x))⊤ withbk : RD → R:
  \hat {s}(\bx ) = \bc ^\top \bb (\bx ) \label {eq:expansion_1} (1)Note that we denote s(x) as the true signal and ŝ(x) as its
approximation.
Representing the signal s(x) using a global set of basis functions is inefficient as information cannot be sharedspatially. We hence generalize the above formulation by(i) exploiting a spatially varying coefficient field c(x) =(c1 (x), . . . , cK (x))⊤ with ck : RD → R and (ii) transforming the coordinates of the basis functions via a coordinatetransformation function γ : RD → RB :  \hat {s}(\bx ) = \bc (\bx )^\top \bb \left (\bgamma (\bx )\right ) \label {eq:expansion_2} (2)When choosing γ to be a periodic function, this formulation
allows us to apply the same basis at multiple spatial locations
and optionally also at multiple different scales while varying
the coefficients c, as illustrated in Fig. 2. Note that in general
B does not need to match D, and hence the domain of thebasis functions also changes accordingly: bk : RB → R.
Furthermore, we obtain Eq. (1) as a special case when settingc(x) = c and γ(x) = x.
So far, we have considered a 1D signal s(x). However,many signals have more than a single dimension (e.g., 3 inthe case of RGB images or 4 in the case of radiance fields).

(3)

(4)

Q
Here,
denotes the element-wise product of a sequenceof factors. Note that in this general form, each factor
fi : RFi → RK may be equipped with its own coordinatetransformation γi : RD → RFi .
We obtain DiF in Eq. (3) as a special case of our Factor Fields framework in Eq. (4) by setting f1 (x) = c(x),γ1 (x) = x, f2 (x) = b(x) and γ2 (x) = γ(x) with N = 2.
Besides DiF, the Factor Fields framework generalizes many
recently proposed radiance field representations in one unified model as we will discuss in Section 3.
In our formulation, {γi } are considered deterministic
functions while P and {fi } are parametric mappings (e.g.,polynomials, multi-layer perceptrons or 3D feature grids)
whose parameters (collectively named θ below) are optimized. The parameters θ can be optimized either for a single
signal or jointly for multiple signals. When optimizing for
multiple signals jointly, we share the parameters of the projection function and basis factors (but not the parameters of
the coefficient factors) across signals.

2.1. Factor Fields fi
For modeling the factor fields fi : RFi → RK , we consider various different representations in our Factor Fields
framework as illustrated in Fig. 1 (bottom-left). In particular,
we consider polynomials, MLPs, 2D and 3D feature grids
and 1D feature vectors.
MLPs have been proposed as signal representations in
Occupancy Networks [35], DeepSDF [40] and NeRF [36].

(1) Sawtooth(2) Triangular(3) Sinusoidal(4) Hashing(5) Orthogonal-1D(6) Orthogonal-2D

Figure 3: Coordinate Transformations. We show various
periodic (top) and non-periodic (bottom) coordinate transformations γ used in our framework.
While MLPs excel in compactness and induce a usefulsmoothness bias, they are slow to evaluate and hence increase training and inference time. To address this, DVGO
[51] proposes a 3D voxel grid representation for radiancefields. While voxel grids are fast to optimize, they increase
memory significantly and do not easily scale to higher dimensions. To better capture the sparsity in the signal, InstantNGP [37] proposes a hash function in combination with 1D
feature vectors instead of a dense voxel grid, and TensoRF[10] decomposes the signal into matrix and vector products. Our Factor Fields framework allows any of the above
representations to model any factor fi . As we will see inSection 3, many existing models are hence special cases ofour framework.

2.2. Coordinate Transformations γi
The coordinates input to each factor field fi are transformed by a coordinate transformation function γi : RD →RF i .
Coefficients: When the factor field fi represents coefficients, we use the identity γi (x) = x for the corresponding
coordinate transformation since coefficients vary freely overthe signal domain.
Local Basis: The coordinate transformation γi enables
the application of the same basis function fi at multiple
locations as illustrated in Fig. 2. In this paper, we consider
sawtooth, triangular, sinusoidal (as in NeRF [36]), hashing(as in Instant-NGP [37]) and orthogonal (as in TensoRF [10])transformations in our framework, see Fig. 3.
Multi-scale Basis: The coordinate transformation γi also
allows for applying the same basis fi at multiple spatial
resolutions of the signal by transforming the coordinates x
with (periodic) transformations of different frequencies as
illustrated in Fig. 2. This is crucial as signals typically carryboth high and low frequencies, and we seek to exploit ourbasis representation across the full spectrum to model finedetails of the signal as well as smooth signal components.

Specifically, we model the target signal with a set of
multi-scale (field of view) basis functions. We arrange thebasis into L levels where each level covers a different scale.
Let [u, v] denote the bounding box of the signal along one
dimension. The corresponding scale is given by (v − u)/flwhere fl is the frequency at level l. A large scale basis (e.g.,level 1) has a low frequency and covers a large region of
the target signal while a small scale basis (e.g., level L) has
a large frequency fL covering a small region of the target
signal.
We implement our multi-scale representation (PR) by
multiplying the scene coordinate x with the level frequency
fl before feeding it to the coordinate transformation function
γi and then concatenating the results across the different
scale l = 1, . . . , L:  \bgamma ^{\text {PR}}_i(\bx ) = \left (\bgamma _i(\bx \, f_1),\dots ,\bgamma _i(\bx \, f_L)\right ) (5)Here, γi is any of the coordinate transformations in Fig. 3,
and γPR is the final coordinate transform of our multi-scale
representation.
As illustrated in Fig. 2 (b), when considering one coefficient factor f1 (x) = c(x) and one basis factor f2 (x) = b(x)with coordinate transformation γ2PR (x) results in the target signal s(x) being decomposed as the product of spatial
varying coefficient maps and multi-level basis maps which
comprise repeated local basis functions.

2.3. Projection P
To represent multi-dimensional signals, we introduced aprojection function P : RK → Q
RQ that maps from the Kdimensional Hadamard product i fi to the Q-dimensionaltarget signal. We distinguish two cases in our framework:
The case where direct observations from the target signal
are available (e.g., pixels of an RGB image) and the indirectcase where observations are projections of the target signal(e.g., pixels rendered from a radiance field).
Direct Observations: In the simplest case, the projection
function realizes a learnable linear mapping P(x) = Ax
with parameters AQ∈ RQ×K to map the K-dimensional
Hadamard product i fi to the Q-dimensional signal. However, a more flexible model is attained if P is represented bya shallow non-linear multi-layer perceptron (MLP) which isthe default setting in all of our experiments.
Indirect Observations: In some cases, we only have access to indirect observations of the signal. For example,
when optimizing neural radiance fields, we typically only
observe 2D images instead of the 4D signal (density andradiance). In this case, extend P to also include the differentiable volumetric rendering process. More concretely, we
first apply a multi-layer perceptron
Q to map the view directiond ∈ R3 and the product features i fi at a particular locationx ∈ R3 to a color value c ∈ R3 and a volume density σ ∈ R.

Next, we follow Mildenhall et al. [36] and approximate theintractable volumetric projection integral using numerical integration. More formally, let {(ci , σi )}N
i=1 denote the color
and volume density values of N random samples along acamera ray. The RGB color value c at the corresponding
pixel is obtained using alpha composition

regularizer using a random binary vector Q
which we multiply
element-wise with the factor product: ◦ i fi .

(6)
where Ti and αi denote the transmittance and alpha valueof sample i and δi = ∥xi+1 − xi ∥2 is the distance between
neighboring samples. The composition of the learned MLP
and the volume rendering function in Eq. (6) constitute theprojection function P.

Multiple Signals: When optimizing for multiple signals
jointly, we share the parameters of the projection function
and basis factors (but not the parameters of the coefficient
factors) across signals. As evidenced by our experiments
in Section 4.3, sharing bases across different signals while
encouraging sparse coefficients improves generalization andenables reconstruction from sparse observations.

Initialization: During all our experiments, we initialize
the basis factors using the discrete cosine transform (DCT),
while initializing the parameters of the coefficient factors
and projection MLP randomly. We experimentally foundthis to improve the quality of the solution as illustrated in  \bc _r = \sum _{i=1}^N \, T_i \, \alpha _i \, \bc _i \hspace {0.5cm} T_i = \prod _{j=1}^{i-1}\left (1-\alpha _j\right )\hspace {0.5cm} \alpha _i = 1-\exp \left (-\sigma _i\delta _i\right ) \label {eq:alphacomp} our ablation study in Table 3a to Table 3e.

2.4. Space Contraction
We normalize the input coordinates x ∈ RD to [0, 1]
before passing them to the coordinate transformations γi (x)by applying a simple space contraction function to x. We
distinguish two settings:
For bounded signals with D-dimensional bounding box[u, v] (where u, v ∈ RD ), we utilize a simple linear mapping to normalize all coordinates to the range [0, 1]:
  \text {contract}(\bx ) = \frac {\bx -\bu }{\bv -\bu } (7)For unbounded signals (e.g., an outdoor radiance field), we
adopt Mip-NeRF 360’s [3] space contraction function:
  \text {contract}(\bx ) = \begin {cases} \bx & {\Vert \bx \Vert }_2\leq 1 \\ \left (2-\frac {1}{{\Vert \bx \Vert }_2}\right ) \left (\frac {\bx }{{\Vert \bx \Vert }_2}\right ) & {\Vert \bx \Vert }_2 > 1 \end {cases} \label {eq:mipnerf360} (8)2.5. Optimization
Given samples {(x, s(x))} from the signal, we minimize
  \argmin _\theta \nE _{\bx } \left [ {\Vert \bs (\bx ) - \hat {\bs }_\theta (\bx ) \Vert }_2 + \Psi (\theta ) \right ] (9)where Ψ(θ) is a regularizer on the model parameters. We
optimize this objective using stochastic gradient descent.
Sparsity Regularization: While using the ℓ0 norm forsparse coefficients is desirable, this leads to a difficult optimization problem. Instead, we use a simpler strategy which
we found to work surprisingly well. We regularize our objective by randomly dropping a subset of the K features of ourmodel by setting them to zero with probability µ. This forces
the signal to be represented with random combinations offeatures at every iteration, encouraging sparsity and preventing co-adaptation of features. We implement this dropout
In our implementation, we slightly modify Eq. (8) to map coordinates to aunit ball centered at 0.5 which avoids negative coordinates when indexing
feature grids.

3. Factor Fields As A Common Framework
Advanced neural representations have emerged as apromising replacement for traditional representations andbeen applied to improve the reconstruction quality and efficiency in various graphics and vision applications, suchas novel view synthesis [65, 32, 54, 2, 36, 31, 10, 58, 55],
generative models [47, 9, 8, 17], 3D surface reconstruction[39, 60, 56, 62, 25], image processing [13], graphics asset
modeling [45, 27, 66], inverse rendering [5, 4, 63, 6, 7, 64],
dynamic scene modeling [44, 30, 41, 29], and scene understanding [42, 34] amongst others.
Inspired by classical factorization and learning techniques, like sparse coding [57, 59, 18] and principal component analysis (PCA) [46, 33], we propose a novel neuralfactorization-based framework for neural representations.
Our Factor Fields framework unifies many recently published neural representations and enables the instantiationof new models in the Factor Fields family, which, as we
will see, exhibit desirable properties in terms of approximation quality, model compactness, optimization time andgeneralization capabilities. In this section, we will discuss
the relationship to prior work in more detail. A systematic
performance comparison of the various Factor Field model
instantiations is provided in Section 4.4.
Occupancy Networks, IMNet and DeepSDF: [35, 14, 40]
represent the surface implicitly as the continuous decisionboundary of an MLP classifier or by regressing a signed
distance value. The vanilla MLP representation provides acontinuous implicit 3D mapping, allowing for the extractionof 3D meshes at any resolution. This setting corresponds
to our Factor Fields model when using a single factor (i.e.,
N = 1) with γ1 (x) = x, f1 (x) = x, P(x) = MLP(x), thusŝ(x) = MLP(x). While this representation is able to generate high-quality meshes, it fails to model high-frequency
signals, such as images due to the implicit smoothness bias

of MLPs.

for higher compression rates.

NeRF: [36] proposes to represent a radiance field via an
MLP in Fourier space by encoding the spatial coordinateswith a set of sinusoidal functions. This corresponds to
our Factor Fields setting when using a single factor with
γ1 (x) = (sin(xf1 ), cos(xf1 ), . . . , sin(xfL ), cos(xfL )),f1 (x) = x and P(x) = MLP(x). Here, the coordinatetransformation γ1 (x) is a sinusoidal mapping as shown in
Fig. 3 (3), which enables high frequencies.

TensoRF: [10] factorizes the radiance fields into the products of vectors and matrices (TensoRF-VM) or multiple
vectors (TensoRF-CP), achieving efficient feature queries at
low memory footprint. This setting instantiates our Factor
Fields framework for N = 2, γ1 (x) = Orthogonal-1D(x),f1 (x) = Vectors(x), γ2 (x) = Orthogonal-2D(x), f2 (x) =2D-Maps(x) for VM decomposition and N = 3, γi (x) =Orthogonal-1D(x), fi (x) = Vectors(x) for CP decomposition. Moreover, TensoRF uses both SH and MLP models forthe projection P. Similar to ConvONet and EG3D, TensoRF
is sensitive to the orientation of the coordinate system dueto the use of an orthogonal coordinate transformation function. Note that, with the exception of TensoRF [10], all of
the above representations factorize the signal using a single
factor field, that is N = 1. As we will show in Table 3a toTable 3d, using multiple factor fields (i.e., N > 1) provides
stronger model capacity.

Plenoxels: [16] use sparse voxel grids to represent 3D
scenes, allowing for direct optimization without neural networks, resulting in fast training. This corresponds to our
Factor Fields framework when setting N = 1, γ1 (x) = x,
f1 (x) = 3D-Grid(x), and P(x) = x for the density field
while P(x) = SH(x) (Spherical Harmonics) for the radiance field. In related work, DVGO [51] proposes a similar
design, but replaces the sparse 3D grids with dense grids anduses a tiny MLP as the projection function P. While dense
grid modeling is simple and leads to fast feature queries, it
requires high spatial resolution (and hence large memory) torepresent fine details. Moreover, optimization is more easily
affected by local minima compared to MLP representations
that benefit from their inductive smoothness bias.
ConvONet and EG3D: [43, 8] use a tri-plane representation to model 3D scenes by applying an orthogonal coordinate transformation to spatial points within a bounded scene,
and then representing each point as the concatenation offeatures queried from a set of 2D feature maps. This representation allows for aggregating 3D features using only
2D convolution, which significantly reduces memory footprint compared to standard 3D grids. The setting can beviewed as an instance of our Factor Fields framework, with
N = 1, γ1 (x) = Orthogonal-2D(x), f1 (x) = 2D-Maps(x)and P(x) = MLP(x). However, while the axis-aligned
transformation allows for dimension reduction and feature
sharing along the axis, it can be challenging to handle complicated structures due to the axis-aligned bias of this representation.
Instant-NGP: [37] exploits a multi-level hash grid to efficiently model internal features of target signals by hashing spatial locations to 1D feature vectors. This approachcorresponds to our Factor Fields framework when using
N = 1, γ1 (x) = Hashing(x), f1 (x) = Vectors(x) andP(x) = MLP(x) with L = 16 scales. However, the multilevel hash mappings can result in dense collisions at finescales, and the one-to-many mapping forces the model to
distribute its capacity bias towards densely observed regions
and noise in areas with fewer observations. The concurrent
work VQAD [52] introduces a hierarchical vector-quantizedauto-decoder (VQ-AD) representation that learns an index
table as the coordinate transformation function which allows

ArXiv Preprints: The field of neural representation learning is advancing fast and many novel representations havebeen published as preprints on ArXiv recently. We now
briefly discuss the most related ones and their relationshipto our work: Phasorial Embedding Fields (PREF) [19] proposes to represent a target signal with a set of phasor volumes
and then transforms them into the spatial domain with aninverse fast Fourier Transform (iFFT) for compact representation and efficient scene editing. This method shares asimilar idea with DVGO and extends the projection function
P in our formulation with an iFFT function. Tensor4D [48]extends the triplane representation to 4D human reconstruction by using 3 triplane Factor Fields and indexing planefeatures with 3 orthogonal coordinate transformations (i.e.,
Orthogonal-2D(x) = (xy, xt, yt), Orthogonal-2D(x) =(xz, xt, zt), Orthogonal-2D(x) = (yz, yt, zt)). NeRFPlayer [50] represents dynamic scenes via deformation, newness and decomposition fields, using multiple factors similar
to TensoRF. It further extends the features by a time dimension. D-TensoRF [20] reconstructs dynamic scenes using
matrix-matrix factorization, similar to the VM factorizationof TensoRF, but replacing γ1 (x) = Orthogonal-1D(x) andf1 (x) = Vectors(x) with γ1 (x) = Orthogonal-2D(x) andf1 (x) = 2D-Maps(x). Quantized Fourier Features (QFF)[28] factorizes internal features into bins of Fourier features,
corresponding to our Factor Fields framework when using
N = 1, γ1 (x) = sinusoidal(x), f1 (x) = 2D-Maps(x), andP(x) = MLP(x) for the 2D signal representation.
Dictionary Field (DiF): Besides existing representations,
our Factor Fields framework enables the design of novel
representations with desirable properties. As an example, we
now discuss the DiF representation which we have alreadyintroduced in Eq. (3). DiF offers implicit regularization,compactness and fast optimization while also generalizing

across multiple signals. The central idea behind the DiF
representation is to decompose the target signals into two
fields: a global field (i.e., the basis) and a local field (i.e.,the coefficient). The global field promotes structured signal
signatures shared across spatial locations and scales as well
as between signals, while the local field allows for spatiallyvarying content. More formally, DiF factorizes the target signal into coefficient fields f1 (x) = c(x) and basis functions
f2 (x) = b(x) which differ primarily by their respective coordinate transformation: We choose the identity mapping
γ1 (x) = x for c(x) and a periodic coordinate transformation γ2 (x) for b(x), see Fig. 3 (top). As representation of
the two factor fields f1 and f2 we may choose any of the ones
illustrated in Fig. 1 (bottom-left). To facilitate comparison
with previous representations, we use the sawtooth function
as the basis coordinate transformation γ2 and uniform grids(i.e., 2D Maps for 2D signals and 3D Grids for 3D signals)
as the representation of the coefficient fields f1 and basis
functions f2 for most of our experiments. Besides, we alsosystematically ablate the number of factors, the number oflevels, the coordinate transformation and the field representation in Section 4.4.

4. Experiments
We now present extensive evaluations of our Factor Fields
framework and DiF representation. We first briefly discuss
our implementation and hyperparameter configuration. We
then compare the performance of DiF with previously proposed representations on both per-signal reconstruction (optimization) and across-signal generalization tasks. At theend of this section, we examine the properties of our Factor
Fields framework by varying the number of factors N , level
number L, different types of transformation functions γifield representation fi , and field connector ◦.

4.1. Implementation
We implement our Factor Fields framework using vanilla
PyTorch without customized CUDA kernels. Performanceis evaluated on a single RTX 6000 GPU using the Adam
optimizer [23] with a learning rate of 0.02.
We instantiate DiF using L = 6 levels with frequencies(linearly increasing) fl ∈ [2., 3.2, 4.4, 5.6, 6.8, 8.], and feature channels K = [4, 4, 4, 2, 2, 2]⊤ · 2η , where η controls
the number of feature channels. We use η = 3 for our 2D
experiments and η= 0 for our 3D experiments. The model
parameters θ are distributed across 3 model components:
coefficients θc , basis θb , and projection function θP . The
size of each component can vary greatly depending on thechosen representation.
In the following experiments, we refer to the default
model setting as “DiF-Grid”, which implements the coefficients c and bases b with learnable tensor grids, P(x) =MLP(x), and γ(x) = Sawtooth(x), where Sawtooth(x) =x mod 1.0. In the DiF-Grid setting, the total number of optimizable parameters is mainly determined by the resolutionof the coefficient and basis grids Mcl , Mbl :  \begin {split} |\theta | &= |\theta _{\cP }| + |\theta _{\bc }| + |\theta _{\bb }| = |\theta _{\cP }| + \sum _{l=1}^L {M_{\bc }^l}^D + \, K_l \cdot {M_{\bb }^l}^D \label {eqn:model_size} \end {split} (10)
We implement the basis grid using linearly increasing resolutions Mbl ∈ [32, 128]T · min(v−u)
with interval [32, 128] and1024
scene bounding box [u, v]. This leads to increased resolution
for modeling higher-resolution signals in our experiments.
We use the same coefficient grid resolution Mcl across all L
levels for query efficiency and to lower per-signal memory
footprint.
In the following, the different model variants of our Dictionary factorization are labeled ”DiF-xx”, where ”xx” indicates the differences from the default setting “DiF-Grid”.
For example, ”-MLP-B” refers to using an MLP basis representation, and ”SL” stands for single level.

4.2. Single Signals
We first evaluate the accuracy and efficiency of our DiFGrid representation on various multi-dimensional signals,
comparing it to several recent neural signal representations.
Towards this goal, we consider three popular benchmark
tasks for evaluating neural representations: 2D image regression, 3D Signed Distance Field (SDF) reconstruction
and Radiance Field Reconstruction / Novel View Synthesis. We evaluate each method’s ability to approximate highfrequency patterns, interpolation quality, compactness, androbustness to ambiguities and sparse observations.
2D Image Regression: In this task, we directly regress
RGB pixel colors from pixel coordinates. We evaluate our
DiF-Grid on fitting four complex high-resolution images,
where the total number of pixels ranges from 4 M to 213 M.
In Fig. 4, we show the reconstructed images with the corresponding model size, optimization time, and image PSNRs,
and compare them to Instant-NGP [37], a state-of-the-art
neural representation that supports image regression and has
shown superior quality over prior art including Fourier Feature Networks [53] and SIREN [49]. Compared to InstantNGP, our model consistently achieves higher PSNR on all
images when using the same model size, demonstrating thesuperior accuracy and efficiency of our model. On the other
hand, while Instant-NGP achieves faster optimization owing to its highly optimized CUDA-based framework, ourmodel, implemented in pure PyTorch, leads to comparably
fast training while relying on a vanilla PyTorch implementation without custom CUDA kernels which simplifies future
extensions.
Signed-Distance Field Reconstruction: Signed DistanceSummer Day

AlbertPlutoGirl with a Pearl Earring

6114 × 3734 × 3 (resolution) / 35.42 M (params)
0:46 vs. 4:13 (mm:ss) / 42.4 vs. 49.0 dB (PSNR)

1024 × 1024 × 4 / 1.36 M
0:30 vs. 1:13 / 51.0 vs. 62.7

8000 × 8000 × 3 / 58.60 M
0:50 vs. 5:32 / 44.30 vs. 46.19

8000 × 9302 × 3 / 68.52 M
1:05 vs. 6:06 / 37.4 vs. 38.9

PSNR (dB)params (M)

PSNR (dB)

PSNR (dB)params (M)params (M)

PSNR (dB)params (M)Figure 4: 2D Image Regression. This figure shows images represented using our DiF-Grid model. The respective image
resolutions and numbers of model parameters are shown below each image. Moreover, we also report a comparison to
Instant-NGP (first number) in terms of optimization time and PSNR metrics (Instant-NGP vs Ours) at the bottom using thesame number of model parameters. Note that our method achieves better reconstruction quality on all images when using
the same model size. While optimization is slower than Instant-NGP, we use a vanilla PyTorch implementation without
customized CUDA kernels. “Summer Day” credit goes to Johan Hendrik Weissenbruch and rijksmuseum. “Albert” credit goesto Orren Jack Turner. “Pluto” credit goes to NASA. “Girl With a Pearl Earring” renovation ©Koorosh Orooj (CC BY-SA 4.0).
Function (SDF), as a classic geometry representation, describes a set of continuous iso-surfaces, where a 3D surface isrepresented as the zero level-set of the function. We evaluate
our DiF-Grid on modeling several challenging object SDFs
that contain rich geometric details and compare with previous state-of-the-art neural representations, including Fourier
Feature Networks [53], SIREN [49], and Instant-NGP [37].
To allow for fair comparisons in terms of the training set
and convergence, we use the same training points for all
methods by pre-sampling 8 M SDF points from the target
meshes for training, with 80% points near the surface and theremaining 20% points uniformly distributed inside the unit
volume. Following the evaluation setting of Instant-NGP, we
randomly sample 16 M points for evaluation and calculate
the geometric IOU metric based on the SDF sign
  gIoU = \frac {\sum (s(\bX )>0) \cap (\hat {s}(\bX )>0)}{\sum (s(\bX )>0) \cup (\hat {s}(\bX )>0)} \label {eqn:iou} 

(11)

where X is the evaluation point set, s(X) are the groundtruth SDF values, and ŝ(X) are the predicted SDF values.
Fig. 5 shows a quantitative and qualitative comparisonof all methods. Our method leads to visually better results,
it recovers high-frequency geometric details and contains
less noise on smooth surfaces (e.g., the elephant face). The
high visual quality is also reflected by the highest gIoU valueof all methods. Meanwhile, our method also achieves thefastest reconstruction speed, while using less than half ofthe number of parameters used by CUDA-kernel enabled
Instant-NGP, demonstrating the high accuracy, efficiency,
and compactness of our factorized representation.
Radiance Field Reconstruction: Radiance field reconstruction aims to recover the 3D density and radiance of eachvolume point from as multi-view RGB images. The geometry and appearance properties are updated via inverse volume
rendering, as proposed in NeRF [36]. Recently, many encoding functions and advanced representations have been
proposed that significantly improve reconstruction speed andquality, such as sparse voxel grids [16], hash tables[37] andtensor decomposition[10].
In Table 1, we quantitatively compare DiF-Grid withseveral state-of-the-art fast radiance field reconstruction
methods (Plenoxel [16], DVGO [51], Instant-NGP [37]and TensoRF-VM [10]) on both synthetic [36] as well as
real scenes (Tanks and Temple objects) [24]. Our method
achieves high reconstruction quality, significantly outperforming NeRF, Plenoxels, and DVGO on both datasets, while
being significantly more compact than Plenoxels and DVGO.
We also outperform Instant-NGP and are on par with TensoRF regarding reconstruction quality, while being highly
compact with only 5.1 M parameters, less than one-third of

DiF (ours)

SIREN

Frequency

Instant-NGP

DiF-Grid

Reference

Lucy

Armadillo

DiF (ours)

Statuette

1.0 M (params)
36:53 (mm:ss)
2.414
0.9169 (gIoU)

1.0 M
29:22
1.561
0.966311.6 M
0:41
1.442
0.97225.10 M
0:31
1.395
0.9795model size (M)speed (mm:ss)
CD (1e − 3)
gIoU

DragonFigure 5: Signed-Distance Field Reconstruction. We reconstruct SDFs from 8.0 M training points. We show qualitativevisual comparisons on the top and quantitative comparisons on the bottom including the number of parameters, reconstruction
time and gIoU. DiF-Grid and iNGP [37] are trained for 10k iterations, while SIREN [49] and NeRF with Frequency
Encodings [53] are trained for 200k iterations.
TensoRF-VM and one-half of Instant-NGP. Our DiF-Gridalso optimizes faster than TensoRF, at slightly over 10 minutes, in addition to our superior compactness. Additionally,
unlike Plenoxels and Instant-NGP which rely on their own
CUDA framework for fast reconstruction, our implementation uses the standard PyTorch framework, making it easily
extendable to other tasks.
In general, our model leads to state-of-the-art results on
all three challenging benchmark tasks with both high accuracy and efficiency. Note that the baselines are mostly
single-factor, utilizing either a local field (such as DVGO
and Plenoxels) or a global field (such as Instant-NGP). In
contrast, our DiF model is a two-factor method, incorporating both local coefficient and global basis fields, hence
resulting in better reconstruction quality and memory efficiency.

4.3. Generalization
Recent advanced neural representations such as NeRF,
SIREN, ACORN, Plenoxels, Instant-NGP and TensoRF optimize each signal separately, lacking the ability to model
multiple signals jointly or learning useful priors from mul-Figure 6: Radiance Field Reconstruction. We evaluate our
DiF using NeRF-Synthetic and Tanks and Temples datasets,
our method is able to reconstruct high-quality surface details.

tiple signals. In contrast, our DiF representation not only
enables accurate and efficient per-signal reconstruction (asdemonstrated in Section 4.2) but it can also be applied to generalize across signals by simply sharing the basis field acrosssignal instances. We evaluate the benefits of basis sharing
by conducting experiments on image regression from partial
pixel observations and few-shot radiance field reconstruction.

Synthetic-NeRF
MethodNeRF [36]Plenoxels [16]DVGO [51]Instant-NGP [37]TensoRF-VM [10]DiF-Grid (Ours)

BatchSize
4096
5000
5000
10k-85k
4096
4096

Steps
300k
128k
30k
30k
30k
30k

Time ↓
∼35h
11.4m
15.0m
03.9m
17.4m
12.2m

Size (M)↓
01.25
194.5
153.0
11.64
17.95
05.10

Tanks and Temples

PSNR↑
31.01
31.71
31.95
32.59
33.14
33.14

SSIM↑
0.947
0.958
0.957
0.960
0.963
0.961

PSNR↑
25.78
27.43
28.41
27.09
28.56
29.00

SSIM↑
0.864
0.906
0.911
0.905
0.920
0.938

Table 1: Novel View Synthesis with Radiance Fields. We compare our method to previous radiance field reconstruction
methods on the Synthetic-NeRF[36] and Tanks and Temples [24] datasets. We report the scores reported in the original papers
whenever available. We also show average reconstruction time and model size for the Synthetic-NeRF dataset to compare theefficiency of the methods.
For these experiments, instead of DiF-Grid, we adopt DiFMLP-B (i.e., (5) in the Table 3d) as our DiF representation,
where we utilize a tensor grid to model the coefficient and6 tiny MLPs (two layers with 32 neurons each) to model
the basis. We find that DiF-MLP-B performs better than
DiF-Grid in the generalization setting, owing to the strong
inductive smoothness bias of MLPs.
Image Regression from Sparse Observations: Unlike theimage regression experiments conducted in Sec. 4.2 which
use all image pixels as observations during optimization, thisexperiment focuses on the scenario where only part of the
pixels are used during optimization. Without additional priors, a single-signal optimization easily overfits in this setting
due to the sparse observations and the limited inductive bias,
hence failing to recover the unseen pixels.
We use our DiF-MLP-B model to learn data priors by pretraining it on 800 facial images from the FFHQ dataset[21]while sharing the MLP basis and projection function parameters. The final image reconstruction task is conducted by
optimizing the coefficient grids for each new test image.
In Fig. 7, we show the image regression results on threedifferent facial images with various masks and compare themto baseline methods that do not use any data priors, including Instant-NGP and our DiF-MLP-B without pre-training.
As expected, Instant-NGP can accurately approximate thetraining pixels but results in random noise in the untrainedmask regions. Interestingly, even without pre-training andpriors from other images, our DiF-MLP-B is able to capture
structural information to some extent within the same image being optimized; as shown in the eye region, the model
can learn the pupil shape from the right eye and regress
the left eye (masked during training) by reusing the learned
structures in the shared basis functions. As shown on theright of Fig. 7, our DiF-MLP-B with pre-trained prior clearly
achieves the best reconstruction quality with better structuresand boundary smoothness compared to the baselines, demonstrating that our factorized DiF model allows for learning

Input

Instant-NGP

DiF-MLP-B

DiF-MLP-B∗

Figure 7: Image Regression from Sparse Observations.
Results obtained by fitting each model to all unmasked pixels.
We use randomly placed black squares as masks for thebottom two rows and an image of text and small icons asmask for the top row. The symbol ∗ denotes pre-trainingof the basis factors using the FFHQ facial image set. Our
pre-trained model (DiF-MLP-B∗ ) learns robust basis fields
which lead to better reconstruction compared to the per-scene
baselines Instant-NGP and DiF-MLP-B.

and transferring useful prior information from the training
set.
Few-Shot Radiance Field Reconstruction: Reconstructing radiance fields from few-shot input images with sparse
viewpoints is highly challenging. Previous works address
this by imposing sparsity assumptions [38, 22] in per-scene
optimization or training feed-forward networks [61, 12, 26]from datasets. Here we consider 3 and 5 input views per
scene and seek a novel solution that leverages data priors
in pre-trained basis fields of our DiF model during the optimization task. It is worth-noting that the views are chosen ina quarter sphere, thus the overlapping region between views
is quite limited.
Specifically, we first train DiF models on 100 Google
Scanned Object scenes [15], which contains 250 views per
scene. During cross-scene training, we maintain 100 perscene coefficients and share the basis c and projection function P. After cross-scene training, we use the mean coefficient values of pre-trained coefficient fields as the initialization, while fixing the pre-trained functions (c and P) andfine-tuning the coefficient field for new scenes with few-shotobservations. In this experiment, we compare results fromboth DiF-MLP-B and DiF-Grid with and without the pretraining. We also compare with Instant-NGP and previous
few-shot reconstruction methods, including PixelNeRF [61]and MVSNeRF [11], re-train with the same training set andtest using the same 3 or 5 views. As shown in Table 2 andFig. 8, our pre-trained DiF representation with MLP basis
provides strong regularization for few-shot reconstruction,
resulting in fewer artifacts and better reconstruction quality than the single-scene optimization methods without datapriors and previous few-shot reconstruction methods that
also use pre-trained networks. In particular, without any
data priors, single-scene optimization methods (Instant-NGP
and ours w/o prior) lead to a lot of outliers due to overfitting to the few-shot input images. Previous methods like
MVSNeRF and PixelNeRF achieve plausible reconstructions
due to their learned feed-forward prediction which avoids
per-scene optimization. However, they suffer from blurry
artifacts. Additionally, the strategy taken by PixelNeRF and
MVSNeRF assumes a narrow baseline and learns correspondences across views for generalization via feature averaging
or cost volume modeling which does not work as effectively
in a wide baseline setup. On the other hand, by pre-training
shared basis fields on multiple signals, our DiF model canlearn useful data priors, enabling the reconstruction of novelsignals from sparse observations via optimization.

4.4. Influence of Design Choices in Factor Fields
Factor Fields is a general framework that unifies many
previous representations with different settings. In this section, we aim to analyze the properties of these variations andoffer a comprehensive understanding of the components ofthe proposed representation framework. We conduct extensive evaluations on the four main components of our Factor
Fields framework: factor number N , level number L, coordinate transformation function γi , field representation fi , andfield connector ◦.
We present a comprehensive assessment of the representations’ capabilities in terms of efficiency, compactness, reconstruction quality, as well as generalizability, with a rangeof tasks including 2D image regression (with all pixels), andper-scene and across-scene 3D radiance field reconstruction.
Note that, the settings in per-scene and across-scene radiance

Method
Time↓
iNGP
03:38
DiF-Grid
13:39
DiF-MLP-B 18:24
PixelNeRF
00:00
MVSNeRF
00:00
PixelNeRF-ft 25:18
MVSNeRF-ft 13:06
DiF-Grid∗
13:18
DiF-MLP-B∗ 18:44

3 views

5 views

PSNR↑ SSIM↑
14.74 0.776
18.13 0.805
16.31 0.804
21.37 0.878
20.50 0.868
22.21 0.882
18.51 0.864
20.77 0.871
21.96 0.891

PSNR↑ SSIM↑
20.79 0.860
20.83 0.847
22.26 0.900
22.73 0.896
22.76 0.891
23.67 0.895
20.49 0.887
25.41 0.915
26.91 0.927

Table 2: Few-shot Radiance Field Reconstruction. We
show quantitative comparisons of few-shot radiance field reconstruction from 3 or 5 viewpoints regarding optimization
time and novel view synthesis quality (PSNRs and SSIMs).
Results are averaged across 8 test scenes. The results of
Instant-NGP and our DiF models are generated based on perscene optimization, while DiF models with ∗ use pre-trained
basis factors across scenes. We train the feed-forward networks of PixelNeRF and MVSNeRF using the same dataset
we learn our shared basis factors, and the results of PixelNeRF and MVSNeRF are generated from the networks via
direct feed-forward inference. Our DiF-MLP-B∗ with pretrained MLP basis factors leads to the best reconstruction
quality.

field reconstruction are the same as introduced in Section 4.2
and Section 4.3, while for the 2D image regression task, we
use the same model setting as in Section 4.2 and test on 256
high fidelity images at a resolution of 1024 × 1024 from theDIV2K dataset [1]. To enable meaningful comparisons, weevaluate the variations within the same code base and reporttheir performance using the same number of iterations number, batch size, training point sampling strategy and pyramid
frequencies. Correspondingly, the results for Instant-NGP,
EG3D, OccNet, NeRF, DVGO, TensoRF-VM/-CP in Tab. 3
are based on our reimplementation of the original methods inour unified Factor Fields framework with the corresponding
design parameters shown in the tables.
Factor Number N : As illustrated in Eq. (4), the factor
number N refers to the number of factors used to represent
the target signal. We show comparisons between singlefactor models (1,3,5) and two-factor models (2,4,6) in Table 3a. Compared to the models (1) iNGP, (3) EG3D and(5) DiF-no-C which only use a single factor, the models(2) DiF-Hash-B, (4) TensoRF-VM and (6) DiF-Grid use thesame factor as (1), (3), (5) respectively, but are extended to
two-factor models with an additional factor, leading to 3dB
and 0.35 dB PSNR improvement in image regression and
3D radiance field reconstruction tasks, while also increasingPixelNeRF MVSNeRF
iNGP
w/o priorw/ prior
GT

Figure 8: Radiance Fields from 5 Views. We visualize
novel view synthesis results of six test scenes, corresponding to the quantitative results in Tab. 2. We show our DiFMLP-B model w/ and w/o pre-trained data priors (bottom
two rows) and compare it to Instant-NGP, PixelNeRF and
MVSNeRF (top three rows). Our model with pre-trained
basis factors can effectively utilize the learned data priors,
resulting in superior qualitative results with fewer outliers
compared to single-scene models (iNGP and ours w/o priors), as well as sharper details compared to feed-forward
models (PixelNeRF and MVSNeRF).

training time (∼ 10%) and model size (∼ 5%) as expected.
Despite the marginal computational overhead, introducing
additional factors in the framework clearly leads to betterreconstruction quality and represents the signal more effectively. The multiplication between factors allows the two
factor fields to modulate each other’s feature encoding andrepresent the entire signal more flexibly in a joint manner, alleviating the problem of feature collision in Instant-NGP and
related problems of other single-factor models. Additionally,as shown in Table 1, multi-factor modeling (e.g., N >= 2)
is able to provide more compact modeling while maintaining
a similar level of reconstruction quality. Moreover, it allows
for generalization by partially sharing the fields across instances, such as the across-scene radiance field modeling in
the few-shot radiance reconstruction task.
Level Number L: Our DiF model adopts multiple levelsof transformations to achieve pyramid basis fields, similar to
the usage of a set of sinusoidal positional encoding functions

in NeRF [36]. We compare multi-level models (including
DiF and NeRF) with their reduced single-level versions that
only use a single transformation level in Table 3b. Note that
Occupancy Networks (OccNet, row (1)) do not leverage positional encodings and can be seen as a single-level version of
NeRF (row (2)) while the model with multi-level sinusoidal
encoding functions (NeRF) leads to about 10dB PSNR performance boost for both 2D image and 3D reconstruction
tasks. On the other hand, the single-level DiF models arealso consistently worse than the corresponding multi-level
models in terms of speed and reconstruction quality, despite
the performance drops being not as severe as those in purely
MLP-based representations.
Coordinate Transformation γi : In Table 3c, we evaluate four coordinate transformation functions using our DiF
representation. These transformation functions include sinusoidal, triangular, hashing and sawtooth. Their transformation curves are shown in Fig. 3. In general, in contrast
to the random hashing function, the periodic transformation
functions (2, 3, 4) allow for spatially coherent information
sharing through repeated patterns, where neighboring points
can share spatially adjacent features in the basis fields, hence
preserving local connectivity. We observe that the periodic
basis achieves clearly better performance in modeling dense
signals (e.g., 2D images). For sparse signals such as 3D radiance fields, all four transformation functions achieve highreconstruction quality on par with previous state-of-the-art
fast radiance field reconstruction approaches [51, 37, 10].
Field Representation fi : In Table 3d, we compare various functions for representing the factors in our framework(especially our DiF model) including MLPs, Vectors, 2D
Maps and 3D Grids, encompassing most previous representations. Note that discrete feature grid functions (3D
Grids, 2D Maps, and Vectors) generally lead to faster reconstruction than MLP functions (e.g. DiF-Grid is fasterthan DiF-MLP-Band DiF-MLP-C). While all variants canlead to reasonable reconstruction quality for single-signal
optimization, our DiF-Grid representation that uses grids forboth factors achieves the best performance on the image regression and single-scene radiance field reconstruction tasks.
On the other hand, the task of few-shot radiance field reconstruction benefits from basis functions that impose stronger
regularization. Therefore, representations with stronger inductive biases (e.g., the Vectors in TensoRF-VM and MLPs
in DiF-MLP-B) lead to better reconstruction quality compared to other variants.
Field Connector ◦: Another key design choice of our
Factor Fields framework and DiF model is to adopt theelement-wise product to connect multiple factors. Directly
concatenating features from different components is an alternative choice and exercised in several previous works[36, 8, 37]. In Table 3e, we compare the performance of

the element-wised product against the direct concatenationin three model variants. Note that the element-wise product consistently outperforms the concatenation operation interms of reconstruction quality for all models on all applications, demonstrating the effectiveness of using the proposed
product-based factorization framework.

5. Conclusion and Future WorkIn this work, we present a novel unified framework for(neural) signal representations which factorizes a signal as a
product of multiple factor fields. We demonstrate that Factor
Fields generalizes many previous neural field representations(like NeRF, Instant-NGP, DVGO, TensoRF) and enables new
representation designs. In particular, we propose a novel
representation – DiF – with Dictionary factorization, as a
new model in the Factor Fields family, which factorizes asignal into a localized coefficient field and a global basis
field with periodic transformations. We extensively evaluate
our DiF model on three signal reconstruction tasks including
2D image regression, 3D SDF reconstruction, and radiancefield reconstruction. We demonstrate that our DiF modelleads to state-of-the-art reconstruction quality, better or on
par with previous methods on all three tasks, while achieving faster reconstruction and more compact model sizes thanmost methods. Our DiF model is able to generalize acrossscenes by learning shared basis field factors from multiple
signals, allowing us to reconstruct new signals from sparse
observations. We show that, using such pre-trained basis factors, our method enables high-quality few-shot radiance fieldreconstruction from only 3 or 5 views, outperforming previous methods like PixelNeRF and MVSNeRF in the sparseview / wide baseline setting. In general, our framework takesa step towards a generic neural representation with high accuracy and efficiency. We believe that the flexibility of ourframework will help to inspire future research on efficientsignal representations, exploring the potential of multi-factor
representations or novel coordinate transformations.

References[1] Eirikur Agustsson and Radu Timofte. Ntire 2017 challengeon single image super-resolution: Dataset and study. In CVPR
Workshops, 2017. 11[2] Kara-Ali Aliev, Dmitry Ulyanov, and Victor S. Lempitsky.
Neural point-based graphics. arXiv.org, 1906.08240, 2019. 5[3] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P.
Srinivasan, and Peter Hedman. Mip-nerf 360: Unboundedanti-aliased neural radiance fields. In CVPR, 2022. 5[4] Sai Bi, Zexiang Xu, Pratul Srinivasan, Ben Mildenhall,
Kalyan Sunkavalli, Milo Haan, Yannick Hold-Geoffroy,
David Kriegman, and Ravi Ramamoorthi. Neural reflectancefields for appearance acquisition, 2020. 5[5] Sai Bi, Zexiang Xu, Kalyan Sunkavalli, Miloš Hašan, Yannick
Hold-Geoffroy, David Kriegman, and Ravi Ramamoorthi.

Deep reflectance volumes: Relightable reconstructions frommulti-view photometric images. 2020. 5[6] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. Nerd: Neural reflectancedecomposition from image collections. In Proceedings ofthe IEEE/CVF International Conference on Computer Vision,
pages 12684–12694, 2021. 5[7] Mark Boss, Varun Jampani, Raphael Braun, Ce Liu, Jonathan
Barron, and Hendrik Lensch. Neural-pil: Neural preintegrated lighting for reflectance decomposition. Advancesin Neural Information Processing Systems, 2021. 5[8] Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano,
Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas J.
Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, andGordon Wetzstein. Efficient geometry-aware 3d generative
adversarial networks. In CVPR, 2022. 5, 6, 12[9] Eric R. Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu,
and Gordon Wetzstein. Pi-gan: Periodic implicit generative
adversarial networks for 3d-aware image synthesis. In CVPR,
2021. 5[10] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, andHao Su. Tensorf: Tensorial radiance fields. In ECCV, 2022.
1, 2, 4, 5, 6, 8, 10, 12[11] Anpei Chen, Zexiang Xu, Fuqiang Zhao, Xiaoshuai Zhang,
Fanbo Xiang, Jingyi Yu, and Hao Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo.
In ICCV, 2021. 11[12] Xu Chen, Yufeng Zheng, Michael Black, Otmar Hilliges, andAndreas Geiger. Snarf: Differentiable forward skinning foranimating non-rigid neural implicit shapes. In ICCV, 2021.
10[13] Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function.
In CVPR, 2021. 5[14] Zhiqin Chen and Hao Zhang. Learning implicit fields forgenerative shape modeling. In CVPR, 2019. 5[15] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh,and Vincent Vanhoucke. Google scanned objects: A highquality dataset of 3d scanned household items. arXiv preprint
arXiv:2204.11918, 2022. 11[16] Sara Fridovich-Keil, Alex Yu, Matthew Tancik, QinhongChen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels:
Radiance fields without neural networks. In CVPR, 2022. 2,
6, 8, 10[17] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen,
Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja
Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. In Advances In Neural
Information Processing Systems, 2022. 5[18] Felix Heide, Wolfgang Heidrich, and Gordon Wetzstein. Fastand flexible convolutional sparse coding. In CVPR, 2015. 5[19] Binbin Huang, Xinhao Yan, Anpei Chen, Shenghua Gao, andJingyi Yu. PREF: phasorial embedding fields for compact
neural representations. arXiv.org, 2022. 6[20] Hankyu Jang and Daeyoung Kim. D-tensorf: Tensorial radiance fields for dynamic scenes. arXiv.org, 2022. 6

Design

PerformanceName
N fi (x)γi (x)TimeSize
2D Images(1) iNGP∗
1 Vectors
Hashing(x)00:45/12:02/ - 0.92/6.80/ - 34.73/0.906(2) DiF-Hash-B 2 Vectors; 3D grids Hashing(x)00:55/13:10/04:45 1.09/4.37/3.28 37.53/0.949(3) EG3D∗
1 2D Maps
Orthog2D (x)- /14:17/ - /4.54/ - / (4) TensoRF-VM∗ 2 2D Maps; Vectors Orthog1,2D (x) - /16:20/13:06 - /4.55/4.93 - / (5) DiF-no-C
1 3D Grids
Sawtooth(x) 00:41/12:55/ - 0.82/4.55/ - 22.28/0.479(6) DiF-Grid
2 3D Grids; 3D Grids Sawtooth(x) 01:13/12:10/11:35 0.87/5.10/7.32 39.51/0.963
DesignRadiance Field Few-shot RF
32.56/0.958- / 32.80/0.960 26.62/0.919
30.01/0.935- / 30.47/0.940 26.79/0.908
31.10/0.947- / 33.14/0.961 25.41/0.915(a) Design Study on Number of Factors N .
PerformanceName
N fi (x)γi (x)TimeSize
2D Images(1) OccNet∗
1xx
02:17/100:23/ - 0.38/0.43/ - 13.90/0.437(2) NeRF∗
1x
Sinusoidal(x) 02:18/65:50/ - 0.39/0.44/ - 28.99/0.816(3) DiF-Hash-B
2 Vectors; 3D Grids Hashing(x) 00:55/13:10/4:45 1.09/4.37/3.28 37.53/0.949(4) DiF-Hash-B-SL 2 Vectors; 3D Grids Hashing(x) 00:41/13:21/03:03 0.89/4.54/4.59 30.97/0.891(5) DiF-Grid
2 3D Grids; 3D Grids Sawtooth(x) 01:13/12:10/11:35 0.99/5.10/7.32 39.51/0.963(6) DiF-Grid-SL 2 3D Grids; 3D Grids Sawtooth(x) 00:49/22:35/13:12 0.98/5.11/7.25 38.73/0.973
DesignRadiance Field Few-shot RF
20.60/0.849- / 27.81/0.919- / 32.80/0.960 26.62/0.919
31.11/0.941 24.13/0.881
33.14/0.961 25.41/0.915
31.08/0.942 23.88/0.882(b) Design Study on Pyramid Levels L.
PerformanceName
N fi (x)γi (x)TimeSize
2D Images(1) DiF-Hash-B 2 Vectors; 3D Grids Hashing(x) 00:55/13:10/4:45 1.09/4.37/3.28 37.53/0.949(2) DiF-Sin-B 2 3D Grids; 3D Grids Sinusoidal(x) 00:55/12:19/08:28 0.99/5.10/7.32 38.21/0.953(3) DiF-Tri-B 2 3D Grids; 3D Grids Triangular(x) 00:55/12:49/08:44 0.99/5.10/7.32 39.38/0.962(4) DiF-Grid 2 3D Grids; 3D Grids Sawtooth(x) 01:13/12:10/11:35 0.99/5.10/7.32 39.51/0.963

Radiance Field Few-shot RF
32.80/0.960
26.62/0.919
32.85/0.961
25.43/0.908
32.95/0.960
24.78/0.904
33.14/0.961
25.41/0.915(c) Design Study on Coordinate Transformations γi .
DesignPerformanceName
N fi (x)γi (x)TimeSize
2D Images(1) TensoRF-VM∗ 2 2D Maps; Vectors Orthog1,2D (x) - /16:20/13:06 - /4.55/4.93 - / (2) DiF-Grid
2 3D Grids; 3D Grids Sawtooth(x) 01:13/12:10/11:35 0.99/5.10/7.32 39.51/0.963(3) DiF-DCT
2 3D Grids; 3D Grids Sawtooth(x) 00:53/ - / 0.18/ - / - 23.16/0.606(4) DiF-Hash-B 2 Vectors; 3D Grids Hashing(x)00:55/13:10/10.15 1.09/4.37/3.28 37.53/0.949(5) DiF-MLP-B 2 MLP; 3D Grids
Sawtooth(x) 01:24/18:18/18:23 0.18/0.62/2.53 28.76/0.819(6) DiF-MLP-C 2 3D Grid; MLP
Sawtooth(x) 01:13/13:38/08:23 0.87/4.54/4.86 34.72/0.910(7) TensoRF-CP∗ 3 Vectors×3
Orthog1D (x) 00:43/28:05/12:42 0.39/0.29/0.29 33.79/0.899

Radiance Field Few-shot RF
30.47/0.940 26.79/0.908
33.14/0.961 25.41/0.915- / - / 32.80/0.960 26.53/0.924
29.62/0.932 26.91/0.927
32.57/0.956 23.54/0.875
31.14/0.944 22.62/0.867(d) Design Study on Field Representations fi .
DesignPerformanceName
N fi (x)γi (x)TimeSize
2D Images(1) TensoRF-CP∗
3 Vectors×3
Orthog1D (x) 00:43/28:05/10:11 0.39/0.29/0.2933.79/0.899(2) TensoRF-CP∗ -Cat 3 vectors×3
Orthog1D (x) 00:47/39:05/09:47 0.40/0.31/0.3125.67/0.683(3) TensoRF-VM∗
2 2D Maps; vectors Orthog1,2D (x) - /16:20/12:52 - /4.55/4.93 - / (4) TensoRF-VM∗ -Cat 2 2D Maps; vectors Orthog1,2D (x) - /18:35/07:02 - /4.56/4.94 - / (5) DiF-Grid
2 3D Grids; 3D GridsSawtooth(x) 01:13/12:10/11:35 0.99/5.10/7.3239.51/0.963(6) DiF-Grid-Cat
2 3D Grids; 3D GridsSawtooth(x) 00:51/11:35/06:471.00/5.10/7.3237.76/0.946

Radiance FieldFew-shot RF
31.14/0.944 23.19/0.879
26.75/0.905 21.43/0.856
30.47/0.940 26.99/0.911
29.86/0.939 24.67/0.885
33.14/0.961 25.41/0.915
32.95/0.960 24.71/0.894(e) Performance comparison on element-wise product ◦ vs. concatenation.

Table 3: Influence of Design Choices in the Factor Fields Framework. The comparison is done using the same code baseand hyperparameter configuration including number of levels, frequency of each level, etc. Prior methods represented in ourframework are labeled with ∗ due to minor differences with respect to the original publications. Runtime and model size arereported separately for 2D Images / Radiance Fields / Few-shot Radiance Fields.

[21] Tero Karras, Samuli Laine, and Timo Aila. A style-based
generator architecture for generative adversarial networks.

arXiv.org, 2018. 10[22] Mijeong Kim, Seonguk Seo, and Bohyung Han. Infonerf: Rayentropy minimization for few-shot neural volume rendering.
In CVPR, 2022. 10[23] Diederik P. Kingma and Jimmy Ba. Adam: A method forstochastic optimization. In ICLR, 2015. 7[24] Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen
Koltun. Tanks and temples: Benchmarking large-scale scenereconstruction. ACM Trans. on Graphics, 36(4), 2017. 8, 10[25] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann.
Decomposing nerf for editing via feature field distillation. In
NIPS, 2022. 5[26] Jonas Kulhanek, Erik Derner, Torsten Sattler, and RobertBabuska. Viewformer: Nerf-free neural rendering from few
images using transformers. In Shai Avidan, Gabriel J. Brostow, Moustapha Cisse, Giovanni Maria Farinella, and Tal
Hassner, editors, ECCV, 2022. 10[27] Alexandr Kuznetsov, Krishna Mullia, Zexiang Xu, Miloš
Hašan, and Ravi Ramamoorthi. Neumip: multi-resolution
neural materials. ACM Trans. on Graphics, 2021. 5[28] Jae Yong Lee, Yuqun Wu, Chuhang Zou, Shenlong Wang,
and Derek Hoiem. QFF: quantized fourier features for neuralfield representations. arXiv.org, 2022. 6[29] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green,
Christoph Lassner, Changil Kim, Tanner Schmidt, Steven
Lovegrove, Michael Goesele, and Zhaoyang Lv. Neural 3dvideo synthesis. arXiv.org, 2103.02597, 2021. 5[30] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang.
Neural scene flow fields for space-time view synthesis ofdynamic scenes. arXiv.org, 2011.13084, 2020. 5[31] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, andChristian Theobalt. Neural sparse voxel fields. In NeurIPS,
2020. 5[32] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel
Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neuralvolumes: Learning dynamic renderable volumes from images.
In ACM Trans. on Graphics, 2019. 5[33] Julien Mairal, Francis R. Bach, Jean Ponce, and Guillermo
Sapiro. Online dictionary learning for sparse coding. In
Andrea Pohoreckyj Danyluk, Léon Bottou, and Michael L.
Littman, editors, ICML, 2009. 5[34] Matthew, Aditya Kusupati, Alex Fang, Vivek Ramanujan,
Aniruddha Kembhavi, Roozbeh Mottaghi, and Ali Farhadi.
Neural radiance field codebooks. arXiv.org, 2023. 5[35] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks:
Learning 3d reconstruction in function space. In CVPR, 2019.
3, 5[36] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik,
Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF:
Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020. 1, 2, 3, 4, 5, 6, 8, 10, 12[37] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM Trans. on Graphics, 2022. 1, 2,
4, 6, 7, 8, 9, 10, 12[38] Michael Niemeyer, Jonathan Barron, Ben Mildenhall, Mehdi
S. M. Sajjadi, Andreas Geiger, and Noha Radwan. Regnerf:
Regularizing neural radiance fields for view synthesis fromsparse inputs. In CVPR, 2022. 10[39] Michael Niemeyer, Lars M. Mescheder, Michael Oechsle,
and Andreas Geiger. Differentiable volumetric rendering:
Learning implicit 3d representations without 3d supervision.
In CVPR, 2020. 5[40] Jeong Joon Park, Peter Florence, Julian Straub, Richard A.
Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In
CVPR, 2019. 3, 5[41] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien
Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo
Martin-Brualla. Nerfies: Deformable neural radiance fields.
In ICCV, 2021. 5[42] Songyou Peng, Kyle Genova, Chiyu ”Max” Jiang, Andrea
Tagliasacchi, Marc Pollefeys, and Thomas A. Funkhouser.
Openscene: 3d scene understanding with open vocabularies.
arXiv.org, 2022. 5[43] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc
Pollefeys, and Andreas Geiger. Convolutional occupancy
networks. In ECCV, 2020. 6[44] Albert Pumarola, Enric Corona, Gerard Pons-Moll, andFrancesc Moreno-Noguer. D-NeRF: Neural radiance fields
for dynamic scenes. In CVPR, 2021. 5[45] Gilles Rainer, Wenzel Jakob, Abhijeet Ghosh, and Tim
Weyrich. Neural btf compression and interpolation. In Computer Graphics Forum, 2019. 5[46] Ron Rubinstein, Michael Zibulevsky, and Michael Elad. Efficient implementation of the k-svd algorithm using batch
orthogonal matching pursuit. Technical report, Computer
Science Department, Technion, 2008. 5[47] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas
Geiger. Graf: Generative radiance fields for 3d-aware image
synthesis. In NeurIPS, 2020. 5[48] Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. Tensor4d : Efficient neural 4d
decomposition for high-fidelity dynamic reconstruction andrendering. arXiv.org, 2022. 6[49] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman,David B. Lindell, and Gordon Wetzstein. Implicit neuralrepresentations with periodic activation functions. In NIPS,
2020. 1, 7, 8, 9[50] Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele
Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. Nerfplayer:
A streamable dynamic scene representation with decomposed
neural radiance fields. arXiv.org, 2022. 6[51] Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel
grid optimization: Super-fast convergence for radiance fields
reconstruction. In CVPR, 2022. 1, 2, 4, 6, 8, 10, 12[52] Towaki Takikawa, Alex Evans, Jonathan Tremblay, ThomasMüller, Morgan McGuire, Alec Jacobson, and Sanja Fidler.
Variable bitrate neural fields. In ACM Trans. on Graphics,
2022. 6[53] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara
Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let
networks learn high frequency functions in low dimensional
domains. In NeurIPS, 2020. 7, 8, 9

[54] Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: image synthesis using neural textures.
ACM Trans. on Graphics, 2019. 5[55] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler,
Jonathan T Barron, and Pratul P Srinivasan. Ref-nerf: Structured view-dependent appearance for neural radiance fields.
arXiv.org, 2021. 5[56] Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku
Komura, and Wenping Wang. Neus: Learning neural implicit
surfaces by volume rendering for multi-view reconstruction.
In NeurIPS, 2021. 5[57] John Wright, Allen Y. Yang, Arvind Ganesh, S. Shankar
Sastry, and Yi Ma. Robust face recognition via sparse representation. PAMI, 2009. 5[58] Qiangeng Xu, Zexiang Xu, Julien Philip, Sai Bi, Zhixin Shu,
Kalyan Sunkavalli, and Ulrich Neumann. Point-nerf: Pointbased neural radiance fields. In CVPR, 2022. 5[59] Jianchao Yang, John Wright, Thomas S. Huang, and Yi Ma.
Image super-resolution via sparse representation. TIP, 2010.
5[60] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. In NeurIPS, 2021.
5[61] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa.
pixelnerf: Neural radiance fields from one or few images. In
CVPR, 2021. 10, 11[62] Zehao Yu, Songyou Peng, Michael Niemeyer, Torsten Sattler, and Andreas Geiger. Monosdf: Exploring monoculargeometric cues for neural implicit surface reconstruction. In
NeurIPS, 2022. 5[63] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, andNoah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In
CVPR, 2021. 5[64] Yuanqing Zhang, Jiaming Sun, Xingyi He, Huan Fu, Rongfei
Jia, and Xiaowei Zhou. Modeling indirect illumination for
inverse rendering. In CVPR, 2022. 5[65] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe,and Noah Snavely. Stereo magnification: learning view synthesis using multiplane images. ACM Trans. on Graphics,
2018. 5[66] Junqiu Zhu, Yaoyi Bai, Zilin Xu, Steve Bako, Edgar
Velázquez-Armendáriz, Lu Wang, Pradeep Sen, Miloš Hašan,and Ling-Qi Yan. Neural complex luminaires: representation
and rendering. ACM Trans. on Graphics, 2021. 5