UNISURF: Unifying Neural Implicit Surfaces andRadiance Fields for Multi-View Reconstruction

1

Michael Oechsle1,2,3 Songyou Peng1,4 Andreas Geiger1,2
2
Max Planck Institute for Intelligent Systems, Tübingen
University of Tübingen
3
4
ETAS GmbH, Stuttgart
ETH Zurich{firstname.lastname}@tue.mpg.de

Abstract
Neural implicit 3D representations have emerged as a
powerful paradigm for reconstructing surfaces from multiview images and synthesizing novel views. Unfortunately,
existing methods such as DVR or IDR require accurate perpixel object masks as supervision. At the same time, neural radiance fields have revolutionized novel view synthesis.
However, NeRF’s estimated volume density does not admit
accurate surface reconstruction. Our key insight is that implicit surface models and radiance fields can be formulated
in a unified way, enabling both surface and volume rendering using the same model. This unified perspective enables
novel, more efficient sampling procedures and the ability
to reconstruct accurate surfaces without input masks. We
compare our method on the DTU, BlendedMVS, and a synthetic indoor dataset. Our experiments demonstrate that weoutperform NeRF in terms of reconstruction quality while
performing on par with IDR without requiring masks.

1. Introduction
Capturing the geometry and appearance of 3D scenes
from a set of images is one of the cornerstone problems incomputer vision. Towards this goal, coordinate-based neural models have emerged as a powerful tool for 3D reconstruction of geometry and appearance within the last years.
Many recent methods employ continuous implicit functions parameterized with neural networks as 3D representations of geometry [3, 8, 12, 31, 32, 37, 41, 43, 47, 57] or appearance [34, 38, 39, 40, 47, 52, 61]. These neural 3D representations have shown impressive performance on geometry reconstruction and novel view synthesis from multi-view
images. Besides the choice of the 3D representation (e.g.,
occupancy field, unsigned or signed distance field), one key
element for neural implicit multi-view reconstruction is therendering technique. While some of these works represent
the implicit surface as level set and hence render the appearance from surfaces [38, 52, 61], others integrate densities by

Figure 1: Illustration. Implicit models based on surface
rendering [38, 61] require input masks and radiance fields[34] do not optimize implicit surfaces directly. UNISURF
provides a principled unified formulation, enabling accurate
surface reconstruction from images without input masks.
drawing samples along the viewing rays [22, 34, 49].
In existing work, surface rendering techniques haveshown impressive performance in 3D reconstruction [38,61]. However, they require per-pixel object masks as input
and an appropriate network initialization since surface rendering techniques only provide gradient information locallywhere a surface intersects with a ray. Intuitively speaking,
optimizing wrt. local gradients can be seen as an iterative
deformation procedure applied to an initial neural surface
which is often initialized as a sphere. Additional constraints
such as mask supervision are necessary for converging to avalid surface, see Fig. 2 for an illustration. Due to theirreliance on masks, surface rendering methods are limited toobject-level reconstruction and do not scale to larger scenes.
On contrary, volume rendering methods like NeRF [34]have shown impressive results for novel view synthesis, also
for larger scenes. However, surfaces extracted as level setsof the underlying volume density are usually non-smooth2. Related WorkIn this section, we first discuss related work from the domain of 3D reconstruction from multi-view images. Next,
we provide an overview of recent works on neural implicit
representations as well as differentiable rendering.

Random Init.
No Masks

Sphere Init.
No Masks1

Sphere Init.
With Masks

Figure 2: Surface Rendering (IDR Results). State-of-theart methods like IDR [61] require object masks and careful
initialization for capturing accurate geometry.

σ=1

σ = 50σ = 500

Figure 3: Volume Rendering (NeRF Results). We showlevel sets of the recovered density volume for a trained
NeRF model [34] using different density thresholds σ.
and contain artifacts due to the flexibility of the radiancefield representation which does not sufficiently constrain the3D geometry in the presence of ambiguities, see Fig. 3.
Contributions: In this paper, we propose UNISURF (UNIfied Neural Implicit SUrface and Radiance Fields) a principled unified framework for implicit surfaces and radiance fields, with the goal of reconstructing solid (i.e., nontransparent) objects from a set of RGB images. Our framework combines the benefits of surface rendering with thoseof volume rendering, enabling the reconstruction of accurate geometry from multi-view images without masks. By
recovering implicit surfaces, we are able to gradually decrease the sampling region for volume rendering during optimization. Starting with large sampling regions enables
capturing coarse geometry and resolving ambiguities during early iterations. At a later stage, we draw samples closer
to the surface which improves reconstruction accuracy. We
show that our approach enables capturing accurate geometry without mask supervision on the DTU MVS dataset [1],attaining results competitive with state-of-the-art implicit
neural reconstruction methods like IDR [61] which use
strong mask supervision. Moreover, we also demonstrate
our method on scenes from the BlendedMVS dataset [60] as
well as synthetic indoor scenes from SceneNet [29]. Code isavailable at https://github.com/autonomousvision/unisurf.
1 We remark that this result w/o masks from IDR [61] was the only
successful case out of several failure cases.

3D Reconstruction from Multi-View Images: Reconstructing 3D geometry from multiple images has been alongstanding computer vision problem [14]. Before the eraof deep learning, classic multi-view stereo (MVS) methods [2, 4, 5, 7, 20, 20, 48, 50, 51] focus on either matching
features across views [4, 48] or representing shapes with avoxel grid [2, 5, 7, 20, 27, 42, 50, 54, 55]. The former approaches usually have a complex pipeline requiring additional steps like fusing depth information [9, 30] and meshing [18, 19], while the latter ones are limited to low resolution due to cubic memory requirements. In contrast, neural
implicit representations for 3D reconstruction do not sufferfrom discretization artifacts as they represent surfaces by
the level set of a neural network with continuous outputs.
Recent learning-based MVS methods attempt to replacesome parts of the classic MVS pipeline. For instance, someworks learn to match 2D features [15, 21, 26, 56, 62], fusedepth maps [11, 46], or infer depth maps [16, 58, 59] frommulti-view images. Contrary to these learning-based MVS
approaches, our method only requires weak 2D supervision during optimization. Moreover, our method yields
high-quality 3D geometry and synthesizes photorealistic
and consistent novel views.
Neural Implicit Representations: Recently, neural implicit functions have emerged as an effective representation of 3D geometry [3, 8, 12, 31, 32, 37, 41, 43, 47, 57]
and appearance [22, 24, 34, 38, 39, 40, 47, 49, 52] as they
represent 3D content continuously and without discretization while simultaneously having a small memory footprint. Most of these methods require 3D supervision.
However, several recent works [23, 34, 38, 52, 61] demonstrated differentiable rendering for training directly fromimages [23, 34, 38, 52, 61]. We divide these methods into
two groups: surface rendering and volume rendering.
Surface rendering approaches, including DVR [38] andIDR [61], determine the radiance directly on the surface ofan object and provide a differentiable rendering formulation
using implicit gradients. This allows for optimizing neural
implicit surfaces from multi-view images. Conditioning on
the viewing direction allows IDR to capture a high levelof detail, even in the presence of non-lambertian surfaces.
However, both DVR and IDR require pixel-accurate objectmasks for all views as input. In contrast, our method leads
to similar reconstructions without requiring masks.
NeRF [34] and follow-ups [6,28,35,36,44,45,49,53,63]
use volume rendering by learning alpha-compositing of aradiance field along rays. This method has shown impres-sive results on novel view synthesis and does not requiremask supervision. However, the recovered 3D geometry is
far from satisfactory, see Fig. 3. Several follow-up works(Neural Body [44] D-NeRF [45] and NeRD [6]) extract
meshes using the volume density from NeRF, but none ofthem considers optimizing surfaces directly. Unlike theseworks, we aim at capturing accurate geometry and propose
a volume rendering formulation that provably approaches
surface rendering in the limit.

3. Background
The two main ingredients for learning neural implicit 3D
representations from multi-view images are the 3D representation and the rendering technique linking the 3D representation and the 2D observations. This section provides
the relevant background on implicit surface and volumetricradiance representations which we unify in this paper forthe case of solid (non-transparent) objects and scenes.
Implicit Surface Models: Occupancy Networks [31, 38]
represent surfaces as the decision boundary of a binary occupancy classifier, parameterized by a neural network  o_\theta (\bx ): \mathbb {R}^3 \rightarrow [0,1] \label {eq:onet} 

(1)

where x ∈ R3 is a 3D point and θ are the model parameters.
The surface is defined as the set of all 3D points where theoccupancy probability is one half: S = {xs |oθ (xs ) = 0.5}.
To associate a color with every 3D point xs on the surface, acolor field cθ (xs ) can be learned jointly with the occupancy
field oθ (x). The color for a particular pixel/ray r is thus
predicted as follows
  \hat {C}(\br ) &= c_\theta (\bx _s)(2)where xs is retrieved by root finding along ray r, see [38]for details. The parameters2 θ of the occupancy field oθ andthe color field cθ are determined by optimizing a reconstruction loss via gradient descent as described in [24, 38, 61].
While surface rendering allows for accurately estimating geometry and appearance, existing approaches strongly
rely on supervision with object masks as surface rendering
methods are only able to reason about rays that intersect asurface.
Volumetric Radiance Models: In contrast to implicit surface models, NeRF [34] represents scenes as colorized volume densities and integrates radiance along rays via alphablending [25, 34]. More specifically, NeRF uses a neuralnetwork to map a 3D location x ∈ R3 and a viewing direction d ∈ R3 to a volume density σθ (x) ∈ R+ and a color
value cθ (x, d) ∈ R3 . Conditioning on the viewing direction
d allows for modeling view-dependent effects such as specular reflections [34,40] and improves reconstruction quality
2 For convenience, we use the same symbol θ for all model parameters.

Figure 4: Concept and Notation. Our rendering consistsof two steps: First, we seek the surface xs (green) in the occupancy field oθ . Second, we define an interval around thesurface to sample points {xi } (red) for volume rendering.

in case the Lambertian assumption is violated [61]. Let odenote the location of the camera center. Given N samples{xi } along ray r = o + td, NeRF approximates the colorof pixel/ray r using numerical quadrature:  \hat {C}(\br ) &= \sum _{i=1}^{N} T_i \, \left (1 - \exp \left ( - \sigma _\theta (\bx _i)\,\delta _i \right )\right ) \, c_\theta (\bx _i, \bd ) \label {eq:nerf_original}\\ T_i &= \exp \left (- \sum _{j<i} \sigma _\theta (\bx _j) \, \delta _j\right ) \label {eq:nerf_transmittance}(4)Here, Ti is the accumulated transmittance along the ray andδi = |xi+1 − xi | is the distance between adjacent samples.
As Eq. (3) is differentiable, the parameters θ of the density
field σθ and the color field cθ can be estimated by optimizing a reconstruction loss. We refer to [34] for details.
While NeRF does not require object masks for training
due to its volumetric radiance representation, extracting thescene geometry from the volume density requires careful
tuning of the density threshold and leads to artifacts due tothe ambiguity present in the density field, see Fig. 3.

4. Method
We now describe our main contribution. In contrast to
NeRF which is also applicable to non-solid scenes (e.g.,fog, smoke), we restrict our focus to solid objects that can be
represented by 3D surfaces and view-dependent surface colors. Our method exploits both, the power of volumetric radiance representations to learn coarse scene structure without mask supervision as well as surface rendering whichacts as an inductive bias to represent objects by a set of precise 3D surfaces, leading to accurate reconstructions.

4.1. Unifying Surface and Volume Rendering

4.2. Loss Function

We start by noting that Eq. (3) can be rewritten as3

We optimize the following regularized loss function

  \hat {C}(\br ) = \sum _{i=1}^{N} \alpha _i(\bx _i) \prod _{j<i} \bigl (1-\alpha _j(\bx _j)\bigr ) \, c(\bx _i, \bd ) \label {eq:nerf_alpha} (5)with alpha values αi (x) = 1 − exp (−σ(x) δi ). Assuming solid objects, α becomes a discrete occupancy indicatorvariable o ∈ {0, 1} which either takes o = 0 in free spaceand o = 1 in occupied space as value:  \hat {C}(\br ) = \sum _{i=1}^{N} o(\bx _i) \prod _{j<i} \bigl (1-o(\bx _j)\bigr ) \, c(\bx _i, \bd ) \label {eq:nerf_occupancy}   \hat {C}_v(\br ) &= \sum _{i=1}^{N} o_\theta (\bx _i) \prod _{j<i} \bigl (1-o_\theta (\bx _j)\bigr ) c_\theta (\bx _i, \bn _i, \bh _i,\bd ) \label {eq:ours_volume}\\ \hat {C}_s(\br ) &= c_\theta (\bx _s,\bn _s,\bh _s,\bd ) \label {eq:ours_surface}(8)
where xs is retrieved by root-finding along ray r and ns , hs
denote the normal and geometry features at xs , respectively.
Note that xs depends on the occupancy field oθ , but we have
dropped this dependency here for clarity. For further details,
we refer the reader to the supplementary material.
The advantage of this unified formulation is that it allows for both rendering on the surface directly and rendering throughout the entire volume which enables gradually
removing ambiguities during optimization. As evidenced
by our experiments, it is indeed critical to combine bothfor obtaining accurate reconstructions without mask supervision. Being able to quickly recover the surface S via
root-finding enables more effective volume rendering, successively focusing on and refining the object surfaces as we
will describe in Section 4.3. Furthermore, surface rendering
enables faster novel view synthesis as illustrated in Fig. 5.

(9)with ℓ1 reconstruction loss and ℓ2 surface regularization
which encourages the normal of a surface point xs and apoint sampled in its neighborhood to be similar:
  \mathcal {L}_{rec} &= \sum _{\br \in \cR } {\lVert \hat {C}_v(\br ) - C(\br ) \rVert }_1 \label {eq:loss_rec}\\\mathcal {L}_{reg} &= \sum _{\bx _s \in \cS } {\lVert \bn (\bx _s) - \bn (\bx _s+\bepsilon ) \rVert }_2

(11)

(6)

We recognize this expression as the image Q
formation model
for solid objects [55] where the term o(xi ) j<i 1−o(xj )evaluates to 1 for the first occupied
 ray
Q sample xi alongr and to 0 for all other samples.
j<i 1 − o(xj ) is anindicator for visibility which is 1 if there exists no occupied
sample xj with j < i before sample xi . Thus, Ĉ(r) takes
the color c(xi , d) of the first occupied sample along ray r.
To unify implicit surface and volumetric radiance models, we parameterize o directly by a continuos occupancy
field oθ (1) as opposed to predicting volume density σ. Following [61], we condition the color field cθ on the surface
normal n and a feature vector h of the geometry network
which empirically induces a useful bias as also observedin [61] for the case of implicit surfaces. Importantly, ourunified formulation allows for both volume and surface rendering

3 We drop dependencies on the model parameters for clarity.

  \mathcal {L} = \cL _{rec} + \lambda \,\cL _{reg} \label {eq:loss} 

Here, R denotes the set of all pixels/rays in the minibatch,
S is the set of corresponding surface points, C(r) is the observed color for pixel/ray r and ϵ is a small random uniform
3D perturbation. The normal at xs is given by  \bn (\bx _s) = \frac {\nabla _{\bx _s} o_\theta (\bx _s)}{\lVert \nabla _{\bx _s} o_\theta (\bx _s)\rVert _2} (12)which can be computed using double backpropagation [38].

4.3. Optimization
The key hypothesis of implicit surface models [38, 61] isthat only the region at the first intersection point with thesurface contributes to the rendering equation. However, thisassumption is not true during early iterations where the surface is not well defined. Consequently, existing methods[38, 61] require strong mask supervision. Conversely, during later iterations, knowledge of the approximate surface
is valuable for drawing informative samples when evaluating the volume rendering equation in Eq. (7). Therefore, we
utilize a training schedule with a monotonically decreasing
sampling interval for drawing samples during volume rendering, as visualized in Fig. 4. In other words, during early
iterations, the samples {xi } cover the entire optimization
volume, effectively bootstrapping the reconstruction process using volume rendering. During later iterations, thesamples {xi } are drawn closer around the estimated surface. As the surface can be estimated directly from the occupancy field oθ via root-finding [38], this eliminates theneed for hierarchical two-stage sampling as in NeRF. Our
experiments demonstrate that this procedure is particularly
effective for estimating accurate geometry, while it allows
for resolving ambiguities during early iterations.
More formally, let xs = o + ts d. We obtain samples
xi = o + ti d by drawing N depth values ti using stratified
sampling within the interval [ts − ∆, ts + ∆] centered at ts :  t_i \sim \mathcal {U}\left [ t_s +\left ( \frac {2i-2}{N}-1\right ) \Delta , t_s +\left ( \frac {2i}{N}-1\right ) \Delta \right ]  (13)During training, we start with a large interval ∆max andgradually decrease ∆ for more accurate sampling and optimization of the surface using the following decay schedule  \Delta _k = \max (\Delta _\text {max} \exp (-k \, \beta ), \Delta _\text {min}) \label {eq:adaptive} (14)where k denotes the iteration number and β is a hyperparameter. In fact, it can be shown that for ∆ → 0 andN → ∞, volume rendering (7) indeed approaches surface
rendering (8): Ĉv (r) → Ĉs (r). A formal proof of this limit
is provided in the supplementary material.
As evidenced by our experiments, the decay schedule in(14) is critical for capturing detailed geometry as it combines volume rendering of large and uncertain volumes inthe beginning of training with surface rendering towards theend of training. To reduce free space artifacts, we combine
these samples with points sampled randomly between thecamera and the surface. For rays without surface intersection, we use stratified sampling on the entire ray.

4.4. Implementation Details
Architecture: Similar to Yariv et al. [61], we use an 8layer MLP with a Softplus activation function and a hidden
dimension of 256 for the occupancy field oθ . We initialize
the network such that the decision boundary is a sphere [13].
In contrast, the radiance field cθ is parameterized as a 4layer ReLU MLP. We encode the 3D location x and theviewing direction d using Fourier features [34] at k octaves.
We empirically found k = 6 for the 3D location x and k =4 for the viewing direction d to work best.
Optimization: In all experiments, we fit our model tomulti-view images of a single scene. During optimization of
the model parameters, we first randomly sample a view andthen M pixels/rays R from this view based on the camera
intrinsics and extrinsics. Next, we render all rays to compute the loss function in Eq. (9). For root-finding, we use
256 uniform sampled points and apply the secant methodwith 8 steps [31]. For our rendering procedure, we use
N = 64 query points inside the interval and 32 in the freespace between the camera and the lower bound of the interval. The interval decay parameters are β = 1.5e − 5,∆min = 0.05 and ∆max = 1.0. We use Adam with a learning rate of 0.0001 and optimize for M = 1024 pixels periteration with two decay steps after 200k and 400k iterations. In total, we train our models for 450k iterations.
Inference: Our method allows for inferring 3D shapes aswell as for synthesizing novel view images. For synthesizing images, we can render our representation in two different ways, we can either use volume rendering or surface rendering. In Fig. 5, we show that both rendering ap4 We used a single NVIDIA V100 GPU to measure the inference time.

GTVR (178 s)432/0.79/0.33

SR (84 s)
30/0.78/0.32

Figure 5: Volume vs. Surface Rendering. We compare
images generated using volume rendering (VR, ∆ = ∆min )and surface rendering (SR), reporting three image metrics(PSNR↑/SSIM↑/LPIPS↓). Both approaches yield similar
results while surface rendering is twice as fast.
proaches lead to similar results. However, we observe that
surface rendering is faster than volume rendering.
To extract meshes, we apply the Multiresolution IsoSurface Extraction (MISE) algorithm from [31]. We use 643
as the initial resolution and up-sample the mesh in 3 steps
without gradient-based refinement.

5. Experimental Evaluation
We conduct experiments on multi-view 3D reconstruction to assess our method. First, we provide a qualitative and quantitative comparison of our approach to existing methods (IDR [61], NeRF [34], COLMAP [48]) on the
widely used DTU MVS dataset [17]. Second, we show aqualitative comparison for samples from the BlendedMVS
dataset [60] and synthetic renderings of scenes from the
SceneNet dataset [29]. Third, we analyze our rendering procedure and loss function in an ablation study. In the supplementary, we provide results on the LLFF dataset [33].

5.1. Baselines
To validate the effectiveness of our method, we compare
it to three different baselines.
COLMAP [48]: We consider COLMAP [48] as a classical MVS baseline, as it shows strong performance on
multi-view reconstruction and is widely used in related
works [38, 61]. We reconstruct a mesh from the output of
COLMAP using screened Poisson Surface reconstruction(sPSR) [19]. Following [38, 61], we show results for thequantitatively best trim parameter 7 and for the setting that
results in watertight meshes (trim parameter 0).
NeRF [34]: Although NeRF targets novel view synthesis,
its volume density admits the extraction of geometry. To
extract meshes from NeRF, we define a density threshold of
50. We validate this choice in the supplementary.
IDR [61]: IDR is the state-of-the-art multi-view reconstruction method for neural implicit surfaces. IDR reconstructs
surfaces with an impressively high level of detail and han-dles specular surfaces, but requires input masks. We do notcompare to DVR [38] as IDR’s view-dependent modelinghas been demonstrated to be superior to DVR, see [61].

5.2. Datasets
DTU MVS Dataset [17]: The DTU MVS dataset contains
49 to 64 images at a resolution of 1200 × 1600 as well as
extrinsic and intrinsic camera parameters for all views. The
dataset consists of objects with different shapes and appearances. Non-lambertian appearance effects make some ofthe objects particularly challenging. For each scan, groundtruth 3D shapes, as well as the official evaluation procedure,
are available5 . Like previous works [38, 61], we use the”Surface” method of the evaluation script, and evaluate all
methods on meshes cleaned with the respective masks. The
official evaluation procedure calculates the Chamfer distance between sampled points of the predicted shapes andground truth shapes provided in the dataset. For evaluating
IDR [61], we use the pixel-accurate masks for all images
which are provided by the authors of IDR.
BlendedMVS Dataset [60]: The BlendedMVS dataset is alarge-scale dataset containing multi-view images with respective camera extrinsics and intrinsics. We use examples
from the BlendedMVS low-res set with an image resolutionof 768 × 576. These examples contain 24 to 64 different
views of unmasked images. We define the scene as suchthat the object is in the center and the closest camera lies
near the unit sphere.
SceneNet Dataset [29]: For testing our model on complex
indoor scenes, we take two scenes from [29] for evaluation.
BlenderProc [10] is applied for rendering images of a partof the scenes containing multiple objects. The first scene
is a bedroom scene with a bed, a lamp and a bedside table.
The other scene is a living room scene with a sofa, a curtain
and a round table. We use 83 and 40 images, respectively.

5.3. Comparison on DTU
In Table 1, we quantitatively compare our method to thebaselines on the DTU MVS dataset. While the COLMAP
baseline with trim parameter ζ = 7 shows the best performance on the Chamfer distance, it produces non-watertightmeshes with incomplete surfaces. Our method performs
nearly on par with the state-of-the-art neural implicit model
IDR while not relying on strong mask supervision. NeRF
and COLMAP (ζ = 0) also do not use input masks but exhibit worse performance in terms of Chamfer distance.
In Fig. 6, we show qualitative results for our method andthe baselines. While COLMAP provides detailed reconstructions, it leads to incomplete geometry due to trimming.
For NeRF, holes and noise artifacts can be observed in thereconstructions. In contrast, our approach and IDR (with
5 https://roboimagedata.compute.dtu.dk

masks
trim

COLMAP
✗
7

IDR
✓
-

COLMAP
✗
0

NeRF
✗
-

Ours
✗
-

scan24
scan37
scan40
scan55
scan63scan65
scan69scan83scan97
scan105
scan106
scan110
scan114
scan118
scan122

0.45
0.91
0.37
0.37
0.90
1.00
0.54
1.22
1.08
0.64
0.48
0.59
0.32
0.45
0.43

1.63
1.87
0.63
0.48
1.04
0.79
0.77
1.33
1.16
0.76
0.67
0.90
0.42
0.51
0.53

0.81
2.05
0.73
1.22
1.79
1.58
1.02
3.05
1.40
2.05
1.00
1.32
0.49
0.78
1.17

1.90
1.60
1.85
0.58
2.28
1.27
1.47
1.67
2.05
1.07
0.88
2.53
1.06
1.15
0.96

1.32
1.36
1.72
0.44
1.35
0.79
0.80
1.49
1.37
0.89
0.59
1.47
0.46
0.59
0.62

mean

0.65

0.90

1.36

1.49

1.02

Table 1: Quantitative Comparison on DTU [17]. Weshow a quantitative comparison against the baselines for thereconstructed geometry on 15 scans from the DTU dataset.
Our method performs on par with IDR, even though IDR
uses masks as input. Bold numbers are only considering
methods without mask supervision and trimming.
input masks) produce accurate surfaces with high-quality
details. We remark that our model captures the overall spatial arrangement of the scene accurately while being able
to also capture geometric details, e.g., the teeth of the skull
and other surface details.

5.4. Comparison on BlendedMVS and SceneNet
To show our model’s capabilities on more diverse scenes,
we use samples of the BlendedMVS dataset and indoor
scenes from SceneNet. As there exist no object masks
for these scenes, we only consider COLMAP and NeRF
as baseline methods. While we have tried running IDR,
none of the scenes converged, resulting in degenerate outputs. For scenes with complex backgrounds, we use a background model that learns to capture appearance information
outside of our region interest, we refer the reader to the supplementary material for more details.
Our qualitative results in Fig. 8 provide evidence that,
unlike existing implicit surface models, our method is able
to reconstruct plausible geometry for complex scenes withmultiple objects and backgrounds. While COLMAP works
well on indoor scenes, it shows artifacts on uniformly colored regions (e.g., round table in the second row). As for the
BlendedMVS experiment, NeRF can reason about the overall spatial structure but shows less accurate surfaces withsignificantly higher levels of noise compared to UNISURF.
More results can be found in the supplementary material.

Without Input Masks

GT view

COLMAP [48](ζ = 7)

NeRF [34]With Input Masks

Ours

IDR [61]

Figure 6: Reconstructed 3D Shapes for DTU. We show a qualitative comparison of reconstructed surfaces. We use
COLMAP [48], NeRF [34] and IDR [61] as baselines. While IDR requires pixel-accurate object masks for all images,
the other methods reason about the full scene extent. Our method performs visually on par with IDR while not requiring
masks.

SR
Uniform VR
HVR
Ours w/o Surf Reg
Ours
Chamfer Distance (not converging)
3.70
3.22
0.48
0.46
Figure 7: Ablation Study. We compare our method to different rendering procedures: Surface rendering in isolation (SR),volume rendering with uniformly sampled points (Uniform VR) and hierachical volume rendering (HVR). This analysis on
the Buddha scene (DTU) shows that our full method performs best, both visually and quantitatively. Moreover, our model
without surface regularizer (Ours w/o Surf Reg) leads to non-smooth regions, especially at the textureless flat table.
and bloated shapes with missing details. This provides evidence that the proposed unified model leads to more accurate reconstructions compared to the baselines.
Losses: In Fig. 7, we also show an ablation study of oursurface regularization term in (9). Without this regularizer
surfaces become less smooth in flat and ambiguous areas,e.g., at the table. This regularization term is particularly
useful for regions that are observed less frequently, as it incorporates an inductive bias towards smooth surfaces.

6. Discussion and Conclusion

GT view

COLMAP [48] NeRF [34]Ours

Figure 8: Reconstructions for Indoor Scenes [29] andBlendedMVS [60]. We show a qualitative comparison ofreconstructed surfaces on indoor scenes (1st and 2nd row)and BlendedMVS (3rd, 4th and 5th row).

5.5. Ablation study
We finally investigate the impact of rendering design
choices and show ablations of the loss function.
Rendering Procedure: We argue for our choices of therendering procedure by comparing different variations in
Fig. 7: First, we consider a baseline which only uses surface rendering during optimization (SR). We use an ℓ1 reconstruction loss on the surface color and backpropagate
through implicit differentiation following [38]. A second baseline uses uniform volume rendering with 96 query
points (Uniform VR). Third, we use NeRF’s hierarchicalvolume sampling (HVR) [34] with 64+64 sample points forquerying the occupancy field. While surface rendering does
not converge, Uniform VR and HVR result in overly smoothThis work presents UNISURF, a unified formulation of
implicit surfaces and radiance fields for capturing highquality implicit surface geometry from multi-view images
without input masks. We believe that neural implicit surfaces and advanced differentiable rendering procedures playa key role in future 3D reconstruction methods. Our unified
formulation shows a path towards optimizing implicit surfaces in a more general setting than possible before.
Limitations: By design, our model is limited to represent
solid, non-transparent surfaces. Overexposed and textureless regions are also limiting factors that lead to inaccuraries and non-smooth surfaces. Furthermore, the reconstructions are less accurate at rarely visible regions in theimages. Limitations are discussed in more detail in the supplementary.
In future work, for resolving ambiguities from rarely visible and texture-less regions, a prior is necessary for reconstruction. While we incorporate an explicit smoothness
prior during optimization, learning a probabilistic neuralsurface model which captures regularities and uncertainty
across objects would help to resolve ambiguities, leading tomore accurate reconstructions.
Acknowledgement This work was supported by an NVIDIA
research gift, the ERC Starting Grant LEGO-3D (850533) and the
DFG EXC number 2064/1 - project number 390727645. Songyou
Peng is supported by the Max Planck ETH Center for Learning
Systems.

References[1] Henrik Aanæs, Rasmus Ramsbøl Jensen, George Vogiatzis,
Engin Tola, and Anders Bjorholm Dahl. Large-scale data for
multiple-view stereopsis. International Journal of Computer
Vision (IJCV), 120(2):153–168, 2016. 2[2] Motilal Agrawal and Larry S Davis. A probabilistic framework for surface reconstruction from multiple images. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2001. 2[3] Matan Atzmon, Niv Haim, Lior Yariv, Ofer Israelov, HaggaiMaron, and Yaron Lipman. Controlling neural level sets. In
Advances in Neural Information Processing Systems (NIPS),
2019. 1, 2
[4] Michael Bleyer, Christoph Rhemann, and Carsten Rother.
Patchmatch stereo - stereo matching with slanted support
windows. In Proc. of the British Machine Vision Conf.
(BMVC), 2011. 2[5] JS De Bonet and Paul Viola. Poxels: Probabilistic voxelized
volume reconstruction. In Proc. of the IEEE International
Conf. on Computer Vision (ICCV), 1999. 2[6] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T Barron, Ce Liu, and Hendrik Lensch. Nerd: Neural reflectancedecomposition from image collections. arXiv.org, 2020. 2, 3
[7] A Broadhurst, T W Drummond, and R Cipolla. A probabilistic framework for space carving. In Proc. of the IEEE
International Conf. on Computer Vision (ICCV), 2001. 2[8] Liang-Chieh Chen, Yukun Zhu, George Papandreou, FlorianSchroff, and Hartwig Adam. Encoder-decoder with atrous
separable convolution for semantic image segmentation. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 833–851, 2018. 1, 2
[9] Brian Curless and Marc Levoy. A volumetric method forbuilding complex models from range images. In ACM Trans.
on Graphics, 1996. 2[10] Maximilian Denninger, Martin Sundermeyer, Dominik
Winkelbauer, Youssef Zidan, Dmitry Olefir, Mohamad Elbadrawy, Ahsan Lodhi, and Harinandan Katam. Blenderproc. arXiv.org, 2019. 6[11] Simon Donne and Andreas Geiger. Learning non-volumetric
depth fusion using successive reprojections. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2019. 2[12] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna,
William T Freeman, and Thomas Funkhouser. Learning
shape templates with structured implicit functions. In Proc.
of the IEEE International Conf. on Computer Vision (ICCV),
2019. 1, 2
[13] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, andYaron Lipman. Implicit geometric regularization for learning shapes. In Proc. of the International Conf. on Machinelearning (ICML), 2020. 5[14] Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge university press,
2003. 2[15] Wilfried Hartmann, Silvano Galliani, Michal Havlena, Luc
Van Gool, and Konrad Schindler. Learned multi-patch simi-

larity. In Proc. of the IEEE International Conf. on Computer
Vision (ICCV), 2017. 2[16] Po-Han Huang, Kevin Matzen, Johannes Kopf, Narendra
Ahuja, and Jia-Bin Huang. Deepmvs: Learning multi-view
stereopsis. In 2018 IEEE Conference on Computer Vision
and Pattern Recognition, CVPR 2018, Salt Lake City, UT,
USA, June 18-22, 2018, pages 2821–2830, 2018. 2[17] Rasmus Ramsbøl Jensen, Anders Lindbjerg Dahl, George
Vogiatzis, Engil Tola, and Henrik Aanæs. Large scale multiview stereopsis evaluation. In Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR), 2014. 5, 6
[18] Michael M. Kazhdan, Matthew Bolitho, and Hugues Hoppe.
Poisson surface reconstruction. In Proceedings of the
Fourth Eurographics Symposium on Geometry Processing,Cagliari, Sardinia, Italy, June 26-28, 2006, volume 256,
pages 61–70, 2006. 2[19] Michael M. Kazhdan and Hugues Hoppe. Screened poisson
surface reconstruction. ACM Trans. on Graphics, 32(3):29,
2013. 2, 5[20] Kiriakos N. Kutulakos and Steven M. Seitz. A theory ofshape by space carving. International Journal of Computer
Vision (IJCV), 38(3):199–218, 2000. 2[21] Vincent Leroy, Jean-Sébastien Franco, and Edmond Boyer.
Shape reconstruction using volume sweeping and learned
photoconsistency. In Proc. of the European Conf. on Computer Vision (ECCV), 2018. 2[22] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, andChristian Theobalt. Neural sparse voxel fields. In Advancesin Neural Information Processing Systems (NeurIPS), 2020.
1, 2[23] Shichen Liu, Shunsuke Saito, Weikai Chen, and Hao Li.
Learning to infer implicit surfaces without 3d supervision. In
Advances in Neural Information Processing Systems (NIPS),
2019. 2[24] Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Marc
Pollefeys, and Zhaopeng Cui. Dist: Rendering deep implicit
signed distance function with differentiable sphere tracing.
In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 2, 3
[25] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel
Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images.
In ACM Trans. on Graphics, 2019. 3[26] W. Luo, A. Schwing, and R. Urtasun. Efficient deep learningfor stereo matching. In Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2016. 2[27] David Marr and Tomaso Poggio. Cooperative computationof stereo disparity. Science, 1976. 2[28] Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi,
Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR), 2021. 2[29] John McCormac, Ankur Handa, Stefan Leutenegger, andAndrew J. Davison. Scenenet RGB-D: can 5m synthetic images beat generic imagenet pre-training on indoor segmentation? In Proc. of the IEEE International Conf. on Computer
Vision (ICCV), 2017. 2, 5, 6, 8

[30] Paul Merrell, Amir Akbarzadeh, Liang Wang, Philippos
Mordohai, Jan-Michael Frahm, Ruigang Yang, David Nistér,
and Marc Pollefeys. Real-time visibility-based fusion of
depth maps. In Proc. of the IEEE International Conf. on
Computer Vision (ICCV), pages 1–8, 2007. 2[31] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks:
Learning 3d reconstruction in function space. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2019. 1, 2, 3, 5[32] Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack,
Mahsa Baktashmotlagh, and Anders Eriksson. Implicit surface representations as layers in neural networks. In Proc.
of the IEEE International Conf. on Computer Vision (ICCV),
2019. 1, 2
[33] Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon,
Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, andAbhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Trans. on
Graphics, 2019. 5[34] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik,
Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF:
Representing scenes as neural radiance fields for view synthesis. In Proc. of the European Conf. on Computer Vision(ECCV), 2020. 1, 2, 3, 5, 7, 8[35] Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas
Kurz, Chakravarty R. Alla Chaitanya, Anton Kaplanyan,
and Markus Steinberger. Donerf: Towards real-time rendering of neural radiance fields using depth oracle networks.
arXiv.org, 2021. 2[36] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative neural feature fields.
In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2021. 2[37] Michael Niemeyer, Lars Mescheder, Michael Oechsle, andAndreas Geiger. Occupancy flow: 4d reconstruction bylearning particle dynamics. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), 2019. 1, 2[38] Michael Niemeyer, Lars Mescheder, Michael Oechsle, andAndreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 1, 2, 3, 4, 5, 6, 8
[39] Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo
Strauss, and Andreas Geiger. Texture fields: Learning texture representations in function space. In Proc. of the IEEE
International Conf. on Computer Vision (ICCV), 2019. 1, 2[40] Michael Oechsle, Michael Niemeyer, Christian Reiser, Lars
Mescheder, Thilo Strauss, and Andreas Geiger. Learning implicit surface light fields. In Proc. of the International Conf.
on 3D Vision (3DV), 2020. 1, 2, 3[41] Jeong Joon Park, Peter Florence, Julian Straub, Richard A.
Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation.
In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2019. 1, 2[42] Despoina Paschalidou, Ali Osman Ulusoy, Carolin Schmitt,
Luc van Gool, and Andreas Geiger. Raynet: Learning volu-

metric 3d reconstruction with ray potentials. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2018. 2[43] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc
Pollefeys, and Andreas Geiger. Convolutional occupancy
networks. In Proc. of the European Conf. on Computer Vision (ECCV), 2020. 1, 2
[44] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang,
Qing Shuai, Hujun Bao, and Xiaowei Zhou. Neural body:
Implicit neural representations with structured latent codes
for novel view synthesis of dynamic humans. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2020. 2, 3[45] Albert Pumarola, Enric Corona, Gerard Pons-Moll, andFrancesc Moreno-Noguer. D-nerf: Neural radiance fields for
dynamic scenes. arXiv.org, 2020. 2, 3
[46] Gernot Riegler, Ali Osman Ulusoy, Horst Bischof, and Andreas Geiger. OctNetFusion: Learning depth fusion from
data. In Proc. of the International Conf. on 3D Vision (3DV),2017. 2[47] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned
implicit function for high-resolution clothed human digitization. In Proc. of the IEEE International Conf. on Computer
Vision (ICCV), 2019. 1, 2
[48] Johannes Lutz Schönberger, Enliang Zheng, Marc Pollefeys,
and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In Proc. of the European Conf.
on Computer Vision (ECCV), 2016. 2, 5, 7, 8
[49] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas
Geiger. Graf: Generative radiance fields for 3d-aware image
synthesis. In Advances in Neural Information Processing
Systems (NeurIPS), 2020. 1, 2[50] S.M. Seitz and C.R. Dyer. Photorealistic scene reconstruction by voxel coloring. In Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR), 1997. 2[51] Steven M. Seitz, Brian Curless, James Diebel, Daniel
Scharstein, and Richard Szeliski. A comparison and evaluation of multi-view stereo reconstruction algorithms. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2006. 2[52] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3dstructure-aware neural scene representations. In Advancesin Neural Information Processing Systems (NIPS), 2019. 1,
2[53] Pratul Srinivasan, Boyang Deng, Xiuming Zhang, Matthew
Tancik, Ben Mildenhall, and Jonathan T. Barron. NeRV:Neural reflectance and visibility fields for relighting andview synthesis. In Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2021. 2[54] Shubham Tulsiani, Tinghui Zhou, Alexei A Efros, and Jitendra Malik. Multi-view supervision for single-view reconstruction via differentiable ray consistency. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2017. 2[55] Ali Osman Ulusoy, Andreas Geiger, and Michael J. Black.
Towards probabilistic volumetric reconstruction using ray

potentials. In Proc. of the International Conf. on 3D Vision(3DV), 2015. 2, 4[56] Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Ilg, Alexey Dosovitskiy, and Thomas
Brox. Demon: Depth and motion network for learning
monocular stereo. In Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2017. 2[57] Weiyue Wang, Xu Qiangeng, Duygu Ceylan, RadomirMech, and Ulrich Neumann. DISN: Deep implicit surface
network for high-quality single-view 3d reconstruction. Advances in Neural Information Processing Systems (NIPS),
2019. 1, 2
[58] Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan.
Mvsnet: Depth inference for unstructured multi-view stereo.
Proc. of the European Conf. on Computer Vision (ECCV),
2018. 2[59] Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang,and Long Quan. Recurrent mvsnet for high-resolution multiview stereo depth inference. Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR), 2019. 2[60] Yao Yao, Zixin Luo, Shiwei Li, Jingyang Zhang, Yufan Ren,
Lei Zhou, Tian Fang, and Long Quan. Blendedmvs: A largescale dataset for generalized multi-view stereo networks. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020. 2, 5, 6, 8
[61] Lior Yariv, Matan Atzmon, and Yaron Lipman. Universal differentiable renderer for implicit neural representations.
arXiv.org, 2003.09852, 2020. 1, 2, 3, 4, 5, 6, 7
[62] Sergey Zagoruyko and Nikos Komodakis. Learning to compare image patches via convolutional neural networks. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2015. 2[63] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen
Koltun. Nerf++: Analyzing and improving neural radiancefields. arXiv.org, 2020. 2