NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild
Authors: Jason Y. Zhang
Gengshan Yang
Shubham Tulsiani
Deva Ramanan
Venue: NeurIPS 2021 Poster (2021)
Keywords: Computer Vision

Abstract: Recent history has seen a tremendous growth of work exploring implicit representations of geometry and radiance, popularized through Neural Radiance Fields (NeRF).  Such works are fundamentally based on a (implicit) {\em volumetric} representation of occupancy, allowing them to model diverse scene structure including translucent objects and atmospheric obscurants. But because the vast majority of real-world scenes are composed of well-defined surfaces, we introduce a {\em surface} analog of such implicit models called Neural Reflectance Surfaces (NeRS). NeRS learns a neural shape representation of a closed surface that is diffeomorphic to a sphere, guaranteeing water-tight reconstructions. Even more importantly, surface parameterizations allow NeRS to learn (neural) bidirectional surface reflectance functions (BRDFs) that factorize view-dependent appearance into environmental illumination, diffuse color (albedo), and specular “shininess.” Finally, rather than illustrating our results on synthetic scenes or controlled in-the-lab capture, we assemble a novel dataset of multi-view images from online marketplaces for selling goods. Such “in-the-wild” multi-view image sets pose a number of challenges, including a small number of views with unknown/rough camera estimates. We demonstrate that surface-based neural reconstructions enable learning from such data, outperforming volumetric neural rendering-based reconstructions. We hope that NeRS serves as a first step toward building scalable, high-quality libraries of real-world shape, materials, and illumination.

NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild
Jason Zhang
Shubham Tulsiani* Robotics Institute, Carnegie Mellon University
Gengshan Yang
Deva Ramanan"
Abstract
Recent history has seen a tremendous growth of work exploring implicit repre- sentations of geometry and radiance, popularized through Neural Radiance Fields (NeRF). Such works are fundamentally based on a (implicit) volumetric repre- sentation of occupancy, allowing them to model diverse scene structure including translucent objects and atmospheric obscurants. But because the vast majority real-world scenes are composed of well-defined surfaces, we introduce surface analog of such implicit models called Neural Reflectance Surfaces (NeRS). NeRS learns neural shape representation of closed surface that is diffeomorphic to a sphere, guaranteeing water-tight reconstructions. Even more importantly, surface paramcterizations allow NeRS to Iearn (neural) bidirectional surface reflectance functions (BRDFs) that factorize view-dependent appearance into environmental illumination, diffuse color (albedo), and specular "shininess." Finally, rather than illustrating our results on synthetic scenes or controlled in-the-lab capture, we assemble novel dataset of multi-view images from online marketplaces for selling goods. Such "in-the-wiil multi-view image sets pose a number of challenges, including small number of views with unknown/rough camera estimates. We demonstrate that surface-based neural reconstructions enable learning from such data, outperforming volumetric neural rendering- based reconstructions. We hope that NeRS serves as first step toward building scalable, high-quality libraries of real-world shape, materials, and illumination. The project page with code and video visualizations can be found at jasonyzhang.com// com/nnes.
Input Images
Initial Mesh
Recovered Shape and Cameras
Novel Views
Output
Texture
Illumination
Diverse 3D Reconstructions
Figure 1: 3D view synthesis in the wild. From several multi-vies internet images of a truck and coarse initial mesh (top left), we recover the camera poses, 3D shape, texture, and illumination (top right). We demonstrate the scalability of our approach wide variety of indoor and outdoor object categories (second row). [Video]
denotes equal coding. Corresponding email: Sssonyzhan@@emm.com
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Introduction
Although we observe the surrounding world only via 2D percepts, it is undeniably 3D. The goal of recovering this underlying 3D from 2D observations has been a longstanding one in the vision community, and any computational approach aimed at this task must answer a central question about representation--hov should we model the geometry and appearance of the underlying 3D structure?
An increasingly popular answer to this question is to leverage neural volumetric representations of density and radiance fields (Mildenhall al., 2020). This allows modeling structures from rigid objects to translucent fluids, while further enabling arbitrary view-dependent lighting effects. However, is precisely this unconstrained expressivity that makes less robust and unsuitable for modeling 3D objects from sparse views in the wild. While these neural volumetric representations have been incredibly successful, they require hundreds of images, typically with precise camera poses, to model the full 3D structure and appearance of real real-world objects. In contrast, when applied to 'in-the-wild' settings e.g. a sparse set of images with imprecise camera estimates from off-the-shelff systems (see Fig. 1), they are unable to infer a coherent 3D representation. We argue this is because these neural volumetric representations, by allowing arbitrary densities and lighting, are too flexible.
Is there a robust alternative that captures real-world 3D structure? The vast majority of worl-world objects and scenes comprise of well-defined surfaces. This implies that the geometry. rather than being an unconstrained volumetric function, can be modeled as a 2D manifold embedded in euclidean 3D space- and thus encoded via a (neural) mapping from a 21 manifold to 3D. Indeed, such meshed surface manifolds form the heart of virtually all rendering engines (Foley et al., 1996). Moreover, instead of allowing arbitrary view-dependent radiance, the appearance of such surfaces can be described using (neural) bidirectional surface reflection functions (BRDFs), themselves developed by the computer graphics community over decades. We operationalize these insights into Neural Reflectance Surfaces (NeRS), a surface-basel neural representation for geometry and appearance.
NeRS represents shape using a neural displacement field over canonical sphere, thus constraining the geometry to be a watertight surface. This representation crucially associates surface normal to each point, which enables modeling view-dependent lighting effects in a physically grounded manner. Unlike volumetric representations which allow unconstrained radiance, NeRS factorizes surface appearance using combination of diffuse color (albedo) and specularity. It does so by learning neural texture fields over the sphere to capture the albedo at each surface point, while additionally inferring an environment map and surface material properties. This combination of surface constraint and a factored appearance allows NeRS to learn efficiently and robustly from sparse set of images in the wild, while being able to capture varying geometry and complex view-dependent appearance.
Using only a coarse category-level template and approximate camera poses, NeRS can reconstruct instances from a diverse set of classes. Instead of evaluating in synthetic setup, we introduce a dataset sourced from marketplace settings where multiple images of varied set of real-world objects under challenging illumination are easily available. We show NeRS significantly outperforms neural volumetric or classic mesh-baseed approaches in this challenging setup, and as illustrated in Fig. 1, able to accurately model the view-dependent appearance via its disentangled representation. Finally, as cameras recovered in the wild are only approximate, we propose a new evaluation protocol for in-the-wild novel view synthesis in which cameras can be refined during both training and evaluation. We hope that our approach and results highlight the several advantages that neural surface representations offer, and that our work serves as a stepping stone for future investigations.
2 Related Work
Surface-based 3D Representations. As they enable efficient representation and rendering, polygo- nal meshes are widely used in vision and graphics. In particular, morphable models (Blanz and Vetter, 1999) allow parametrizing shapes as deformations of canonical template and can even be learned from category- image collections (Cashman and Fitzgibbon, 2012; Kar et al., 2015). With the advances differentiable rendering (Kato et al., 2018; Laine et al., 2020; Ravi et al., 2020), these have also been leveraged in learning based frameworks for shape prediction (Kanazawa et al., 2018; Gkioxari et al., 2019; Goel et al., 2020) and view synthesis (Riegler and Koltun, 2020). Whereas these approaches use an explicit discrete mesh some recent methods have proposed using continuous

---

neural surface parametrization like ours to represent shape (Groueix et al., 2018) and texture (Tulsiani et al., 2020; Bhattad al., 2021).
However, all of these works leverage such surface representations for (coarse) single-view 3D prediction given category-level training dataset. In contrast, our aim is to infer such a representation given multiple images of single instance, and without prior training. Closer to this goal of representing a single instance in detail, contemporary approaches have shown the benefits of using videos (Li et al., 2020; Yang et al., 2021) to recover detailed shapes, but our work tackles a more challenging setup where correspondence/for across images is not easily available. In addition, while these prior approaches infer the surface texture, they do not enable the view-dependent appearance effects that our representation can model.
Volumetric 3D and Radiance Fields. Volumetric representations for 3D serve as a common, and arguably more flexible alternative surface based representations, and have been very popular for classical multi-view reconstruction approaches (Furukawa and Hernández, 2015). These have since been incorporated in deep-learning frameworks for shape prediction (Girdhar et al., 2016; Choy et al., 2016) and differentiable rendering (Yan et al., 2016; Tulsiani et al., 2017). Although these initial approaches used discrete volumetric grids, their continuous neural function analogues have since been proposed to allow finer shape (Mescheder et al., 2019; Park et al., 2019) and texture modeling (Oechsle et al., 2019).
Whereas the above methods typically aimed for category-level shape representation, subsequent approaches have shown particularly impressive results when using these representations to model a single instance from images (Sitzmann et al., 2019a; Thies et al., 2019; Sitzmann et al., 2019h) which is the goal of work. More recently, by leveraging an implicit representation in the form of a Neural Radiance Field, Mildenhall et al. (2020) showed the ability to model complex geometries and illumination from images. There has since been flurry of impressive work to further push the boundaries of these representations and allow modeling deformation (Pumarola et al., 2020; Park al., 2021), lighting variation Martin-Bruall, et al., 202 1),1,, and similar to ours, leveraging insights from surface rendering to model radiance (Yariv et al., 2020: Boss et al., 2021: Oechsle et al., 2021; Srinivasan et al., 2021: Zhang et al., 2021b; Wang et al., 2021: Yariv et al., 2021). However, unlike our approach which can efficiently learn from a sparse set of images with coarse cameras, these approaches rely on a dense set of multi-view images with precise camera localization to recover a coherent 3D structure of the scene. DietNeRF (Jain et al., 2021) reduces the number of images but requires precise cameras and semantic supervision. BARF (Lin et al., 2021) relaxes the constraint of precise cameras while foregoing riew-dependent appearance and requiring a dense set of images. Other approaches (Bi et al., 2020; Zhang et al., 2021a) that learn material properties from sparse views require specialized illumination rigs.
Multi-view Datasets. Many datasets study the longstanding problem of multi-view reconstruction and view synthesis. However, they are often captured in controlled setups, small in scale, and not diverse enough to capture the span of real world objects. Middlebury (Seitz et al., 2006) benchmarks multi-view reconstruction, containing two objects with nearly Lambertian surfaces. DTU (Aanæs et al., 2016) contains eighty objects with various materials but is still captured a lab with controlled lighting. Freiburg cars (Sedaghat and Brox, 2015) captures 360 degree videas of fifty-two outdoor cars for multi-view reconstruction. ETH3D (Schöps et al., 2019) and Tanks and Temples (Knapitsch et al., 2017) contain both indoor and outdoor scenes but are small in scale. Perhaps most relevant are large-scale datasets of real-world objects such as Redwood (Choi et al., 2016) and Stanford Products (Oh Song et al., 2016), but the data is dominated by single-views small baseline videos. In contrast, our Multi-view Marketplace Cars (MVMC) dataset contains thousands of multi-view captures of and benchmarking algorithms for multi-view reconstruction, view synthesis, and inverse rendering.
objects under various illumination conditions, making it suitable for studying
3 Method
Given sparse set of input images of an object under natural lighting conditions, we aim to model its shape and appearance. While recent neural volumetric approaches share a similar goal, they require a dense set of views with precise camera information. Instead, our approach relies only on approximate camera pose estimates and a coarse category-level shape template. Our key insight is that instead of allowing unconstrained densities popularly used for volumetric representations,
enforce
3

---

(u, v)
Stex
(2.
(a, U, 2, r,
+
grgg.bb b
Sshape
Sphere Template
Predicted Shape
Predicted Texture
Textured Mesh
Figure 2:
Neural Surface Representation. We propose an implicit, continuous representation of shape and lexture. We model shape as deformation of a unit sphere via neural network Salupe. and texture as learned per-uv color value via neural network fixx. We discretize fatage and fee produce the textured mesh above.
a surface-based 3D representation. Importantly, this allows view-dependent appearance variation by leveraging constrained reflection models that decompose uppearance into diffuse and specular components. In this section, we first introduce our (neural) surface representation that captures the object's shape and texture, and then explain how illumination and specular effects can be modeled for rendering. Finally, we describe how our approach can learn using challenging in-the-wild images.
3.1 Neural Surface Representation
We represent object shape via deformation of unit sphere. Previous works (Kanazawa al., 2018; Goel et al., 2020) have generally modeled such deformations explicitly: the unit sphere is discretized some resolution as 3D mesh with V vertices. Predicting the shape deformation thus amounts to predicting vertex offsets 0 E IR TRVx x3 Such explicil discrete representations have several drawbacks. First, they can be computationally expensive for dense meshes with fine details. Second, they lack useful spatial inductive biases as the vertex locations are predicted independently. Finally, the learned deformation model is fixed a specific level of discretization, making it non-trivial, for instance, to allow for more resolution as needed in regions with richer detail. These limitations also extend texture parametrization commonly used for such discrete mesh representations using either per-vertex or per-face texture samples (Kato et al., 2018), or fixed resolution texture map, limits the ability to capture finer details.
Camera
Inspired by Groueix et al. (2018); Tulsiani et al. (2020), we address these challenges by adopting continuous surface representation via a neural network. We illustrate this representation in Fig. 2. For any point u on the surface of unit sphere S², we represent its 3D deformation E R using the mapping fahape (u) = x where fahape parameterized as multi-layer perceptron. This network therefore induces a deformation field over the surface of the unit sphere, and this deformed surface serves as our shape representation. We represent the surface texture in a similar manner-ar as a neural vector field over the surface of the sphere: fiex (w) = E RR3 This surface texture can be interpreted an implicit UV texture map.
Ω
Surface S
Figure 3: Notation and conven- tion for viewpoint and illumi- nation parameterization. The camera e looking at point on the surface S. v denotes the direction of the camera w.r.t and is the normal of S at Ω denotes the unit hemisphere cen- tered about n.. We compute light arriving in the direction of every w E Ω, and r is the reflec- tion of about
3.2 Modeling Illumination and Specular Rendering
Surface Rendering. The surface geometry and texture are not suf- ficient to infer appearance of the object e.g. a uniformly red car may appear darker on one side, and lighter the other depending the viewing direc-
direction of incident light. In addition. depending tion and material properties, one may observe different appearance for the same 3D point e.g. shiny highlight from certain viewpoints. More formally, assuming that surface does not emit light, the outgoing radiance T., in direction u from surface point xr can be described by the rendering equation (Kajiya, 1986; Immel et al., 1986):
(x, v) / fx v. (z, n)dw
where Q is the unit hemisphere centered surface normal n, and w denotes the negative direction of incoming light. fa v, ) is the bidirectional reflectance function (BRDF) which captures material

---

Elevation
Azimmuhh Texture (Fux)
Reference View (1)
Output Radiance (Lo)
Pred. Texture (2)
View Indep, (TQO Ldiifow)
High a, Low ly
Elecation
Normals (n)
Diffuse Lighting (Jaiuus))
Specular Lighting (Fapecular)
Illum. Mean Tex.
Map (fmvr
(T Jax + kydspee)
High k,
Figure 4: Components of learned illumination model. Given a query camera viewpoint (illustrated via the reference image I), we recover the radiance output Lo.. computed using Phong shading (Phong, 1975). Here, show the full decomposition learned components. From the environment map fane and normals n. we compute diffuse (Tamee) and specular lighting (Aypcuha)) component ("View Indep.") and the specular lighting (weighted by the specular coefficient k.,) forms the view-dependent component of the radiance. Altogether, the output radiance L. T O Taituss + h. Tppeusaaayy (4). We also visualize the radiance using the mean texture. which is used to help learn plausible illumination. yellow box, we visualize the effects nessruheesss of the surface. The specular coefficient k, controls the intensity of the specular highlights.
texture and diffuse lighting form view-independent
two specularity parameters.
shininess controls
properties (e.g, color and shininess) of surface at x, and (, w) is the radiance coming toward from (Refer to Fig. 3). Intuitively, this integral computes the total effect of the reflection of every possible light ray w hitting ø bouncing in the direction v.
We thus need to infer the environment lighting and surface material properties to allow realistic renderings. However, learning arbitrary lighting I, or reflection models f, infeasible given sparse views, and we need to further constrain these to allow learning. Inspired by concurrent work (Wu al., 2021) that demonstrated its efficacy when rendering rotationally symmetric objects, we leverage the Phong reflection model (Phong, 1975) with the lighting represented as neural environment map.
Neural Environment Map. An environment map intuitively corresponds to the assumption that all the light sources are infinitely far away. This allows simplified model of illumination, where the incoming radiance only depends on the direction w and is independent of the position æ i.e. L,(x, w) Iu- We implement this predict the incoming radiance for any query direction (x(,w) = L. = fenv (w). Note that there is a fundamental ambiguity between material properties and illumination, e.g. car that appears red could be a white car under red illumination, or red car under white illumination. To avoid this, we follow Wu al. (2021), and further constrain the environment illumination to be grayscale, i.e. fenv (w) R.
neural spherical environment map faw which learns to
Appearance under Phong Reflection. Instead of allowing an arbitrary BRDF fr, the Phong reflection model decomposes the outgoing radiance from point in direction into the diffuse and specular components. The view-independent portion the illumination modeled by the diffuse component:
Ldiffuse
Σ (w nn)
(2)
WES
while the view-dependent portion of the illumination is modeled by the specular component:
Ispecutar (x, v)
Σ (Twin - v)"
WER
where Tw,n 2(w n))n w is the reflection w about the normal n. The shininess coefficient (0, oo) is property of the surface material and controls the "mirror-ness" of the surface. If is high, the specular highlight will only be visible if o aligns closely with Tw. Altogether, we compute the radiance of in direction as:
Lo(z, v) T(x) Laittuse (2) + k2 -Lpoccuaar v)
(4)
5

---

Images
Init. Shape
Pred. Shape
Reconstructions from Novel Views
Figure 5: Qualitative results on various household objects. We demonstrate the versatility of our approach on an espresso machine, a bottle of ketchup, game controller, and a fire hydrant. Each instance has 7-10 input views. We find that coarse, cuboid mesh is sufficient as an initialization to learn detailed shape and texture. We initialize the camera poses by hand, roughly binning in increments of 45 degrees azimuth. [Video]
where the specularity coefficient k2 is another surface material property that controls the intensity of the specular highlight. T(x) ) is the texture value at computed by fex- For the sake of simplicity, and hs are shared across the entire instance. See Fig. 4 for a full decomposition of these components.
3.3 Learning NeRS in the Wild
Given a sparse set of images the-wiidd, our approach aims to infer a NeRS representation, which when rendered, matches the available input. Concretely, our method takes as input (typically 8) images of the same instance {T.) noisy camera rotations {R.) and category-specific mesh initialization M. Using these, we aim to optimize full perspective cameras (1I),N as well as the neural surface shape fshupe, surface texture fux. and environment map fem- In addition, we also recover the material properties of the object, parametrized by specularity coefficient k, and shininess coefficient a.
Note that both the camera poses and mesh initialization are only required to be coarsely accurate. We use off-the-shelf approach (Xiao et al., 2019) predict camera rotations, and we find that a cuboid is sufficient as an initialization for several instances (See Fig. 5). We use off-the-shelf approaches (Rother al., 2004; Kirillov al., 2020) to compute masks [M, J.N We assume that all images were taken with the same camera intrinsics. We initialize the shared global focal length correspond to a field of view of 60 degrees, and set the principal point at center of each image. We initialize the camera pose with the noisy initial rotations RG and a translation é, such that the object fully in view. We pre-train fauuee to output the template mesh M.
Initialization.
Rendering. To render an image, NeRS first discretizes the neural shape model fshape (u) over spherical coordinates u to construct an explicit triangulated surface mesh. This triangulated mesh and camera II, are fed into Py Torch3D's differentiable renderer (Ravi et al., 2020) to obtain per-pixel (continuous) spherical coordinates and associated surface properties:
UV, N, M, = RasterizeeTss
(5)
where UV[2), Nip), and M[p are (spherical) uv-coordinates, normals, and binary foreground- background labels corresponding to each image pixel p. Together with the environment map Jem and specular material parameters (a, k.), these quantities are sufficient to compute the outgoing radiance at each pixel p under camera viewpoint 1I, using (4). In particular, denoting by v(II, p) the viewing

---

direction for pixel p under camera II. and using UVpp), = [pp for notational brevity, the intensity pixel p can be computed as:
[p] = f2)
(E(w
(w.n) n2) fenu(w) + Ks WEn
Σ WESt
(Twnn eII, p)) fenr(w) (w)(w)
(6)
Image loss. We compute a perceptual loss (Zhang et al., 2018) L pperceptual (4,, 1+) that compares the distance between the rendered and true image using off- off-the-shee the-thelf VGG deep features. Note that being able to compute perceptual loss is significant benefit of surface-base volumetric approaches such as NeRF (Mildenhall et al., 2020), which operate on batches of rays rather than images, due to the computational cost of volumetric rendering. Similar to Wu et al. (2021), we find that an additional rendering loss using the mean texture (see Fig. 4 and Fig. 8 for examples with details in the supplement) helps learn visually plausible lighting.
representations over
Mask Loss. To measure disagreement between the rendered and measured silhouettes, we compute a mask loss Lmusk- N >i= IIM, M.llg. distance transform loss Lar N Σ D; O M,, and 2D chamfer loss Lehanfer = & Σ EME)(M)) minpess lpp - 211.. D; refers to the Euclidean distance transform of mask M,. E.) computes the 2D pixel coordinates of the edge of a mask, and ý is every pixel coordinate in the predicted silhouette.
Regularization. Finally, to encourage smoother shape whenever possible, we incorporate a mesh regularization loss Lregularize Lurmalss Lisplaciu consisting of normals consistency and Laplacian smoothing losses (Nealen et al., 2006; Desbrun et al., 1999). Note that such geometry regularization is another benefit of surface representations over volumetric ones. Altogether, we minimize:
L = A Lmaak + A2 + Aehamfer + Xa Irperceppullll + As Iregularize
w.r.t 11, [Rg. (i, a, kg, and the weights of fahapes fex. and fewv_map.
Optimization. We optimize (7) in coarse-to-fin fashion, starting with few parameters and slowly increasing the number of free parameters. We initially optimize (7), w.r.t only the camera parameters II After convergence, we sequentially optimize fshupe. fux. and fem//a/ks, kg- We find it helpful to sample new set of spherical coordinates u each iteration when rasterizing. This helps propagate gradients over a larger surface and prevent aliasing. With 4 Nvidia 1080TI GPUs, training NeRS requires approximately 30 minutes. Please see Sec. ?? for hyperparameters and additional details.
4 Evaluation
In this section, we demonstrate the versatility of Neural Reflectance Surfaces to recover meaningful shape, texture, and illumination from in-the-wild indoor and outdoor images.
Multi-view Marketplace Dataset. To address the shortage of in-the-wild multi-view datasets, we introduce a new dataset, Multi-view Marketplace Cars (MVMC), collected from an online marketplace with thousands of car listings. Each user-submitted listing contains seller images of the same car instance. In total, we curate a subset of size 600 with at least 8 exterior views (averaging 10 exterior images per listing) along with 20 instances for an evaluation set (averaging 9.1 images per listing). We use Xiao et al. (2019) to compute rough camera poses. MVMC contains a large variety of cars under various illumination conditions (e.g. indoors, overcast, sunny, snowy, etc). The filtered dataset with anonymized personally identifiable information (e.g. license plates and phone numbers), masks, initial camera poses, and optimized NeRS cameras will be made available on the project page.
Novel View Synthesis. Traditionally, novel view synthesis requires accurate target cameras to use as queries. Existing approaches use COLMAP (Schönberger and Frahm, 2016) recover ground truth cameras, but this consistently fails on MVMC due to specularities and limited views. On the other hand, we can use learning-basee based methods (Xiao al., 2019) recover camera poses for both training and test views. However, as these are inherently approximate, this complicates training and evaluation. To account for this, we explore two evaluation protocols. First, to mimic the traditional evaluation setup, we obtain pseudo-groume ground truth cameras (with manual correction) and freeze them during training and evaluation. While this evaluates the quality the 3D reconstruction, it does not evaluate the method's ability to jointly recover cameras. As more realistic setup for evaluating view synthesis in the wild, we evaluate each method with approximate shef-) cameras, while allowing them to be optimized.
7

---

Input Images
Target View
NeRF
MetaNeRF
F-ff
IDR
NeRS (Ours)
Figure 6: Qualitative comparison with fixed cameras. We evaluate all baselines the task of novel view synthesis on Multi-view Marketplace Cars trained and tested with fixed, pseudo-ground truth cameras. One image is held out during training. Since we do not have ground truth cameras, we treat the optimized cameras from optimizing over all images as the ground truth cameras. We train a modified version (See Sec. 4) of NeRF (Mildenhall et al., 2020) that is more competitive with sparse views (NeRF*). We also evaluate against meta-learned initialization of NeRF with and without finetuning until convergence (Tancik al., 2021), but found poor results perhaps due to the domain shift from Shapenet cars, Finally. IDR (Yariv et al., 2020) extracts a surface from an SDF representation, but struggles to produce a view-consistent output given limited input views. We find that NeRS synthesizes novel views that are qualitatively closer to the target. The red truck has 16 total views while the blue SUV has 8 total views. [Video]
Target View
N
IDR
NeRS (Ours) Target View
NeRF
IDR
NeRS (Ours)
Figure 7: Qualitative results for in-the-wild novel view synthesis. Since off-the-shelf camera poses are only approximate for both training and test images, we allow cameras to he optimized during both training and evaluation (See Tab. 2 and Sec. 4). We find that NeRS generalizes better than the baselines in this unconstrained but more realistic setup. [Video]
Novel View Synthesis with Fixed Cameras. In the absence of ground truth cameras, create pseudo-ground truth by manually correcting cameras recovered by jointly optimizing over all images for each object instance. For each evaluation, we treat one image-camera pair the target and the remaining pairs for training. We repeat this process for each image in the evaluation set (totaling 182). Unless otherwise noted, qualitative results use approximate cameras and not the pseudo-groum truth.
Novel View Synthesis in the Wild. While the above evaluates the quality of the 3D reconstructions, it is not representative of in-the-wild settings where the initial cameras are unknown/approximat and should be optimized during training. Because even the test camera is approximate, each method is similarly allowed to refine the test camera to better match the test image while keeping the model fixed. Intuitively, this measures the ability of a model to synthesize a target view under some camera. See implementation details in the supplement.
Baselines. We evaluate our approach against Neural Radiance Fields (NeRF) (Mildenhall et al., 2020), which learns a radiance field conditioned on viewpoint and position and renders images using raymarching. We find that the vanilla NeRF struggles in our an-the-wild low-data regime. As such, we make a number of changes to make the NeRF baseline (denoted NeRF*) as competitive as possible, including a mask loss and a canonical volume. Please see the appendix for full details. We also evaluate simplified NeRF with meta-learned initialization for cars from multi-view images (Tancik et al., 2021), denoted as MctaNeRF. MetaNeRF meta-lcarns an initialization such that with just a few gradient steps, it can Iearn NeRF model. This allows the model to learn data-driven prior over the shape of cars. Note that MetaNeRF is trained on ShapeNet (Chang et al., 2015) and thus has approaches. We find that the default number
seen more data than the other test-time-optmmizate of gradient steps was insufficient for MetaNeRF to converge on images from MVMC, so we also evaluate MetaNeRF-ft, which is finetuned until convergence. Finally, we evaluate IDR (Yariv et al., 2020), which represents geometry by extracting a surface from a signed distance field. IDR learns a neural renderer conditioned on the camera direction, position, and normal of the surface.
Metries. We evaluate all approaches using the traditional image similarity metries Mean-Square Error (MSE), Peak Signal-to- -Nisee Ratio (PSNR), and Structural Similarity Index Measure (SSIM).
8

---

Input Images
Novel View 1
Novel View 2
Novel View 3
Output Illum of Mean Tex. Output
Illum, of Mean Tex.
Output
Illom. Mean Tex.
Figure 8: Qualitative results on our in-the-wild Multi-view Marketplace Cars dataset. Here we visualize the NeRS outputs as well as the illumination of the mean texture on of listings from the MVMC dataset. We find that NeRS recovers detailed textures and plausible illumination. Each instance has input views. [Video]
Method
MSE 4
PSNR f
SSIM t
LPIPS +
FID
NeRF" (Mildenhall al., 2020)
0.0393
16.0
0.698
0.287
231.7
MetaNeRF (Tancik et al., 2021)
0,0755
11.4
0.345
0.666
394.5
MetaNeRF-fi (Tancik et al., 2021)
0,0791
11.3
0.500
0,542
326.8
IDR (Yariv al., 2020)
0,0698
13.8
0.658
0.328
190.1
NeRS (Ours)
0.0254
16.5
0.720
0.172
60.9
Table 1: Quantitative evaluation of novel-view synthesis on MVMC fixed pseudo-ground truth cam- eras. To evaluate novel view synthesis in a manner consistent with previous works that assume known cameras, we obtain pseudo-ground truth cameras by manually correcting off-the-shelf recovered cameras. We evaluate against modified NeRF (NeRF*), meta-learned initialization NeRF with and without finetuning (MetaN- eRF), and the volumetric surface-based IDR. NeRS significantly outperforms the haselines on all metrics on the task of novel-view synthesis with fixed cameras. See Fig. for qualitative results.
Method
MSE
PSNR +
SSIM 1
LPIPS
FID
NeRF" (Mildenhall et al., 2020)
0.0464
14.7
0.660
0.335
277.9
IDR (Yariv et al., 2020)
0.0454
14.4
0.685
0.297
242.3
NeRS (Ours)
0.0338
15.4
0.675
0.221
92.5
Table 2: Quantitative evaluation of in-the-wild novel-view synthesis on MVMC. Off- off-the-shelf cameras estimated for in wild data are inherently erroneous. This means that both training and test cameras are approximate, complicating training and evaluation. To compensate for approximate test cameras, allow methods to refine the test camera given the test image with the model fixed (see See. ?? for details). Intuitively this measures the ability a method to synthesize a test image under some camera. We evaluate against NeRF and IDR, and find that NeRS outperforms the baselines across all metries. See Fig. 7 for qualitative results.
We also compute the Learned Perceptual Image Patch Similarity (LPIPS) (Zhang et al., 2018) which correlates more strongly with human perceptual distance. Finally, we compute the Fréchet Inception Distance (Heusel et al., 2017) between the novel view renderings and original images as a measure of visual realism. In Tab. and Tab. 2, we find that NeRS significantly outperforms the baselines in all metrics across both the fixed camera and in- the-wild novel-view synthesis evaluations. See Fig. 6 and Fig. 7 for a visual comparison of the methods.
Qualitative Results. In Fig. 8, we show qualitative results on our Multi-view Marketplace Cars dataset. Each car instance has between 8 and 16 views. We visualize the outputs of our reconstruction from novel views. We show the rendering for both the full radiance model and the mean texture. Both of these renderings are used to compute the perceptual loss (See Sec. 3.2). We find that NeRS recovers detailed texture information and plausible illumination parameters. To demonstrate the scalability of our approach, we also evaluate on various household objects in Fig. 5. We find that coarse, cuboid mesh is sufficient as an initialization to recover detailed shape, texture, and lighting conditions. Please refer to the project webpage for 360 degree visualizations.

---

Discussion
We present NeRS, an approach for learning neural surface models that capture geometry and sur- face reflectance. In contrast to volumetric neural rendering, NeRS enforces watertight and closed manifolds. This allows NeRS model surface-base appearance affects, including view-dependant specularities and normal- dependant diffuse appearance. We demonstrate that such regularized re- constructions allow for learning from sparse in-the-wild multi-view data, enabling reconstruction of objects with diverse material properties across variety of indoor/outdoor illumination conditions. Further, the recovery of accurate camera poses in the wild (where classic structure- frotturrrrooommooioonn fails) remains unsolved and serves as significant bottleneck for all approaches, including ours. We tackle this problem by using realistic but approximate off-the-shelf camera poses and by introducing a new evaluation protocol that accounts for this. We hope NeRS inspires future work that evaluates in the wild and enables the construction of high-quality libraries of real-world geometry, materials, and environments through better neural approximations of shape, reflectance, and illuminants.
Limitations. Though NeRS makes use of factorized models of illumination and material reflectance, there exists some fundamental ambiguities that are difficult from which to recover. For example, it difficult to distinguish between an image
gray car under bright illumination and an image of white car under dark illumination. We visualize such limitations in the supplement. In addition, because the neural shape representation of NeRS diffcomorphic to a sphere, it cannot model objects of non-genus-zett topologies.
Broader Impacts. NeRS reconstructions could be used reveal identifiable or proprietary in- formation (e.g., license plates). We made our best effort to blur out all license plates and other personally-identffabbl information the paper and data.
Acknowledgements. We would like to thank Angjoo Kanazawa and Leonid Keselman for useful discussion and Sudeep Dasari and Helen Jiang for helpful feedback drafts of the manuscript. This work was supported in part by the NSF GFRP (Grant No. DGE1745016), Singapore DSTA, and CMU Argo Al Center for Autonomous Vehicle Research.
References
Henrik Aanws, Rasmus Ramsbol Jensen, George Vogiatzis, Engin Tola, and Anders Bjorholm Dahl. Large-Scale Data for Multiple-View Stereopsis. ICCV, 2016.
Anand Bhattad, Aysegul Dundar, Guilin Liu, Andrew Tao, and Bryan Catanzaro. View Generalization for Single Image Textured 3D Models. In CVPR, 2021.
Sai Bi, Zexiang Xu, Kalyan Sunkavalli, David Kriegman, and Ravi Ramamoorthi. Deep 3D Capture: Geometry and Reflectance from Sparse Multi-view Images. In CVPR, 2020.
Volker Blanz and Thomas Vetter. A Morphable Model for the Synthesis of 3D Faces. In SIGGRAPH, 1999.
Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron, Ce Liu, and Hendrik P.A. Lensch. NeRD: Neural Reflectance Decomposition from Image Collections. In ICCV, 2021.
Thomas J Cashman and Andrew W Fitzgibbon. What Shape are Dolphins? Building 3D Morphable Models from 2D Images. TPAMI, 35(1):232-244, 2012.
Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An Information-rich 3D Model Repository. arXiv:1512.03012. 2015.
Sungjoon Choi, Qian-Yt Yi Zhou, Stephen Miller, and Vladlen Koltun. A Large Dataset of Object Scans. arXiv:1602.02481, 2016.
Christopher B Choy, Danfei Xu, Jun Young Gwak, Kevin Chen, and Silvio Savarese. 3D-R2N2: A Unified Approach for Single and Multi-view 3D Object Reconstruction. In ECCV, 2016.
Mathieu Desbrun, Mark Meyer, Peter Schröder, and Alan H Barr. Implicit Fairing of Irregular Meshes using Diffusion and Curvature Flow. In SIGGRAPH, 1999.
10

---

James D Foley, Foley Dan Van, Andries Van Dam, Steven K Feiner, John F Hughes, Edward Angel, and J Hughes. Computer Graphics: Principles and Practice, volume 121 10. Addison- Addison-Wesley Professional, 1996.
Yasutaka Furukawa and Carlos Hernández. Multi-view Stereo: A Tutorial. Foundations and Trends in Computer Graphics and Vision, (11-): 2015.
Rohit Girdhar, David F Fouhey, Mikel Rodriguez, and Abhinav Gupta. Learning Predictable and Generative Vector Representation for Objects. ECCV, 2016.
Georgia Gkioxari, Jitendra Malik, and Justin Johnson. Mesh R-CNN. In ICCV, 2019.
Shubham Goel, Angjoo Kanazawa, and Jitendra Malik, Shape and Viewpoints without Keypoints. In ECCV, 2020.
Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathicu Aubry. A Papier-måché Approach to Learning 3D Surface Generation. In CVPR, 2018
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans Trained by Two Time-scale Update Rule Converge to Local Nash Equilibrium. In NeurIPS. 2017.
David S Immel, Michael F Cohen, and Donald P Greenberg. A Radiosity Method for Non-diffuse Environments. In SIGGRAPH, 1986.
Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting NeRF on a Diet: Semantically Consistent Few-Shot View Synthesis. In ICCV, 2021.
James T Kajiya. The Rendering Equation. In SIGGRAPH, 1986.
Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, and Jitendra Malik. Learning Category- Specific Mesh Reconstruction from Image Collections. In ECCV, 2018.
Abhishek Kar, Shubham Tulsiani, Joao Carreira, and Jitendra Malik. Category-speciffe Object Reconstruction from Single Image. In CVPR, 2015.
Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3D Mesh Renderer. In CVPR, 2018.
Alexander Kirillov, Yuxin Wu, Kaiming He, and Ross Girshick. Pointrend: Image Segmentation as Rendering- In CVPR, 2020.
Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and Temples: Benchmarking Large-scale Scene Reconstruction. ACM Transactions on Graphics, 36(4):1-13, 2017.
Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Scol, Jaakko Lehtinen, and Timo Aila. Modular Primitives for High-Peeforrance Differentiable Rendering. ACM Transactions on Graphics, 39(6), 2020.
Xueting Li. Sifei Liu, Shalini De Mello, Kihwan Kim, Xiaolong Wang, Ming-Hsuan Yang, and Jan Kautz. Online Adaptation for Consistent Mesh Reconstruction in the Wild. In NeurIPS, 2020.
Chen-Hsuan Lin, Wei-Chiu Ma. Antonio Torralba, and Simon Lucey. BARF: Bundle-Adjusting Neural Radiance Fields. In ICCV, 2021.
Ricardo Bauuila-ruaala,, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR, 2021.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy Networks: Learning 3D Reconstruction in Function Space. In CVPR, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In ECCV, 2020.
11

---

Andrew Nealen, Takeo Igarashi, Olga Sorkine, and Mare Alexa. Laplacian Mesh Optimization. In GRAPHITE. 2006.
Michael Oechsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. Texture Fields: Learning Texture Representations in Function Space. In ICCV, 2019.
Michael Oechsle, Songyou Peng, and Andreas Geiger. UNISURF: Unifying Neural Implicit Surfaces and Radiance Fields for Multi-View Reconstruction. ICCV, 2021.
Hyun Oh Song. Yu Xiang, Stefanie Jegelka, and Silvio Savarese. Deep Metric Learning via Lifted Structured Feature Embedding. In CVPR, 2016.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove, DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation. In CVPR, 2019.
Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan Goldman, Steven M. Seitz, and Ricardo
Nerfies: Deformable Neural Radiance Fields. In ICCV, 2021.
Bui Tuong Phong. Illumination for Computer Generated Pictures. Communications of the ACM, 18 (6), 1975.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. In CVPR, 2020.
Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan- Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3D Deep Learning with PyTorch3D. arXiv:2007.08501, 2020.
Gernot Riegler and Vladlen Koltun. Free View Synthesis. In ECCV, 2020.
Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. "GrabCut" Interactive Foreground Extraction using Iterated Graph Cuts. ACM Transactions on Graphics, 23(3):30--314 2004.
Johannes Lutz Schönberger and fan-Michael Frahm. Pnneturrefrom-MMttion Revisited. CVPR, 2016.
Thomas Schöps, Torsten Sattler, and Mare Pollefeys. BAD SLAM: Bundle Adjusted Direct RGB-D SLAM. In CVPR, 2019.
Nima Sedaghat and Thomas Brox. Unsupervised Generation of a Viewpoint Annotated Car Dataset from Videos. In ICCV, 2015.
Steven M Seitz, Brian Curless, James Dicbel, Daniel Scharstein, and Richard Szeliski. A Comparison and Evaluation of Multi-viww Sterco Reconstruction Algorithms. In CVPR, 2006.
Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhofer. DeepVoxels Learning Persistent 3D Feature Embeddings. In CVPR. 2019a.
Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene Representation Networks: Continuous 3D-Structure-Awarc Neural Scene Representations. In NeurlPS, 2019b.
Pratul Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan Barron. NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis. In CVPR, 2021.
Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul Srinivasan, Jonathan Bar- ron, and Ren Ng. Learned Initializations for Optimizing Coordinate-Basse Based Neural Representations. In CVPR, 2021.
Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred Neural Rendering: Image Synthesis using Neural Textures. ACM Transactions Graphics, 38(4):1-12, 2019.
Shubham Tulsiani, Tinghui Zhou, Alexei A Efros, and Jitendra Malik. Multi-view Supervision for Single-view Reconstruction via Differentiable Ray Consistency. In CVPR, 2017.
Shubham Tulsiani, Nilesh Kulkarni, and Abhinav Gupta. Implicit Mesh Reconstruction from Unannotated Image Collections. 10Xi::007.008544 2020.
12

---

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. In NeurlPS, 2021.
Shangzhe Wu, Ameesh Makadia. Jiajun Wu, Noah Snavely, Richard Tucker, and Angjoo Kanazawa. De-rendering the World's 's Revolutionary Artefacts. In CVPR, 2021.
Yang Xiao, Xuchong Qiu, Pierre-Alain Langlois, Mathieu Aubry, and Renaud Marlet. Pose from Shape: Deep Pose Estimation for Arbitrary 3D Objects. In BMVC, 2019.
Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective Transformer Nets: Learning Single-view 3D Object Reconstruction without 3D Supervision. In NeurIPS, 2016.
Gengshan Yang, Deqing Sun, Varun Jampani, Daniel Vlasic, Forrester Cole, Huiwen Chang, Deva Ramanan, William T Freeman, and Ce Liu. LASR: Learning Articulated Shape Reconstruction from Monocular Video. In CVPR, 2021.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance. In NeurlPS, 2020.
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume Rendering of Neural Implicit Surfaces. arXiv:2106. 12..6.220555 /2052, 2021.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Xiuming Zhang, Sean Fanello, Yun-Ta Tsai, Tiancheng Sun, Tianfan Xue, Rohit Pandey, Sergio Orts-Escolano, Philip Davidson, Christoph Rhemann, Paul Debevec, et al. Neural Light Transport for Relighting and View Synthesis. ACM Transactions on Graphics, 40(1):1-17, 2021a.
Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William Freeman, and Jonathan T Barron. NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination. In SIGGRAPH Asia, 2021b.
13


Reviews:
[{"review_id": "lLoVQcWX33D", "reviewer": "Reviewer_h3Zr", "summary": "This paper tackles the problem of recovering 3D shape, surface albedo, specular material (Phong model), and environment lighting map from a sparse set of images from ~8 viewpoints, given a rough template shape as initialization and automatically obtained masks and camera poses. This is achieved by coupling a AtlasNet-like representation for shape and texture with explicit material and lighting models in an optimization pipeline that reconstructs the input observations.\n\nThe paper shows plausible decomposition results on multi-view image sets from the Internet with rough camera estimates and mask predictions, whereas existing NeRF-based methods fail dramatically on such few inputs.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n## Strengths\n### S1 - Method\n- Overall, the proposed method is carefully designed and achieves compelling decomposition results.\n- It uses an AtlasNet like representation for 3D shape, texture and surface material, which is discretized and rendered into 2D images through a differentiable renderer. In this view, the proposed method essentially extends an AtlasNet-based multi-view reconstruction model with material and lighting components.\n- The environment lighting map is modeled similarly with a sphere AtlasNet, and sampled for rendering.\n\n### S2 - Results\n- The decomposition results are visually appealing, and no existing methods seem to be able to achieve similar results on real world photo sets \"in the wild\". The comparisons against NeRF-based methods also show significant performance gap.\n- The ablation results with naive NeRF-like view-conditioning texture prediction reported in the supplementary material are very insightful, confirming the effectiveness of enforcing explicit shading models for few-view reconstruction.\n\n## Weaknesses\n### W1 - Initialization\nOne significant limitation of the proposed method is need of a category shape template for shape initialization. It also relies on another method for estimating coarse initial camera poses. These additional information has significantly simplified the problem.\n\n### W2 - Over-claim on shape representation\nThe shape representation they are using is essentially an AtlasNet over a sphere. I think its novelty has been over-claimed by focusing the comparison instead on NeRF and its rendering procedure.\n\n### W3 - Simplistic material model\n- The method assumes a Phong model with a global shininess to model specularity, which cannot model sophisticated real world illumination effects, resulting from complex spatially-varying BRDF.\n- It also assumes a gray-scale environment map, which is not realistic.\n- It seems such explicit material and lighting models, especially with the unrealistic Phong model, also limit the image quality, resulting in relatively low faithfulness scores in novel view synthesis as shown in Table 1.\n\n### W4 - Limited comparison\n- The method is only compared to NeRF-based models. How does the method compare to IDR and other photometric stereo methods? Can this CVPR20 work [A] handle these cases, which also recovers 3D shape and SVBRDF from a small set of images, but seems to assume collocated flash light?\n\n[A] \"Deep 3D Capture: Geometry and Reflectance from Sparse Multi-View Images\". Sai Bi, Zexiang Xu, Kalyan Sunkavalli, David Kriegman, and Ravi Ramamoorthi. CVPR 2020.\n\n## Additional comments\n- How is the shape network $f_\\text{shape}$ pretrained with the template mesh? I suppose the mapping from sphere to surface is unknown. Is this achieved using for example an occupancy loss?\n- How well is the environment lighting estimated? There is only one single example in Fig 4. It would be helpful to show more visualizations.\n- What are the resolutions of the discretized mesh and environment map?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "\nA few limitations are discussed, and visual examples are provided in the supplementary material. It would also be interesting to show how the method fails without sufficiently accurate initial mesh.\n", "review_text": "", "extra_scores": {"ethical_concerns": "\nI do not see immediate concerns.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "kMAlQZZt6wp", "reviewer": "Reviewer_3udv", "summary": "This work introduces a method to reconstruct a scene from a small number of views. Rather than using a volumetric representation, it uses an explicit surface that is defined as a sphere with perturbed vertices. This restriction allows the model to learn to decompose the reflectance into a Phong lighting model. A contribution of the work is the introduction of a large, in-the-wild multiview dataset of cars from craigslist. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n+ Surface representations deserve more exploration, and using the additional constraints for few view reconstruction makes sense.\n\n+ Real world dataset with few views is a great contribution and space to work in. I really like the idea of scraping marketplaces for such data, and this work introduces a new dataset consisting of millions of multi-view captures, which is great. The main downside is that as far as I can tell, they are only for cars. \n\n+ BRDF factorization from sparse views is very hard.\n\n+ Can work with only coarse cameras (by jointly optimizing camera parameters during reconstruction).\n\n+ Formulation is clean and elegant, everything defined on the surface of a sphere (geometry, env map, texture).\n\n+ Can use better reconstruction losses due to entire images generated at train time. \n\nCons: \n\n- My biggest concern is that this approach is missing comparisons to other surface based models! for example: [Yariv et al] with code available, why was this not compared to? This would be key to showing that the \"deformed sphere\" approach is better than an SDF representation, as it comes with some limitations in terms of what kinds of models you can represent. \n\n- Results are only okay, they tend to look like \"balloon\" versions of the original, and the Phong shading model does not capture specular reflections well (at all?). \n\n- Requires a mask loss (like Yariv et al), but it seems like the mask loss might already get you a decent starting model simply by reconstructing the visual hull from the masks for convex objects like cars. Is it possible to compare the result to this simple solution?  \n\n- Requires per-category initialization mesh. In fact, this part was a bit unclear to me, what do you use for cars? How do you show motorcycle results? Where do the other non-car examples and initialization meshes come from (e.g., the coffee machine)? What is the importance of the initialization mesh? Is it just for camera pose estimation, or do you use it to initialize the geometry network as well?  \n\nAdditional Questions:\n\nWhen comparing to baselines, the final camera positions that are derived by optimizing this method's objective are used. In some senses, those may be optimal for this approach. How much does the camera change 1) between initialization and the pre-training, 2) between pre-training and final training? \n\nIt is interesting that Nerf outperforms with respect to LPIPS but seems clearly worse in the results. Do you have any intuition why this is? \n\nHow do you model shadows? This seems important given that all images are outdoor, and you often see the object shadowing itself. Does the neural renderer correctly model these? \n\nTL/DR: I like this work, but I think its missing some comparisons (e.g., to another surface based method), and the overhead of category specific steps may make it hard to extend to new classes in the wild.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I believe that few view 3D reconstruction has the ability to give non-professional users the ability to create 3D assets. Negative social impact is adequately discussed.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "hrD_0FC0ti", "reviewer": "Reviewer_kaid", "summary": "The submission studies the problem of 3D geometry and appearance reconstruction from a sparse set of 2D images, and proposes a new neural implicit model called Neural Reflectance Surfaces (NeRS). The main difference from prior work (specifically, NeRF) is the following: \n1. using a much sparser set of 2D observations as input - typically around 10 images per instance \n2. a neural surface model for geometry instead of neural volumetric representations, \n3. a neural environment map to model lightings, conditioned only on the viewing direction,\n4. BRDFs representation for surface specularity to model view-dependent effects.\n\nIn particular, the shape geometry is modeled as a continuous and implicit deformation of a unit sphere, parametrized by a multi-layer perceptron (MLP). Similarly, the diffuse color is parametrized by a second MLP with spherical coordinates as input. To render an image, a continuous unit sphere in 3D is first discretized into a set of vertices (spherical coordinates), which are then passed as input to the shape MLP to construct a triangulated surface mesh. The per-vertex diffuse colors are computed from a second MLP using the same set of spherical coordinates. Together with the neural environment map (from a third MLP), the final pixel intensity can be rendered using a Phong reflection model. These 3 different MLPs are optimized sequentially and iteratively using a combination of 2D image loss, a mask loss, and mesh regularization. The method is compared against different variants of NeRF on a new \"Craigslist Multiview\" (CMV) dataset. Both quantitative and qualitative comparisons show that the proposed NeRS method outperforms NeRF under this few-shot experiment setup.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "#### Originality\n- This submission is slightly different from NeRF in the problem setup - it focuses on the few-shot scenario, and reconstructs only a single instance of objects.\n- Some of the main technical components are from Tulsiani et al. (2020), including the mesh and color parametrization and the loss function. There is a high level of similarity between the two methods, with some differences in using environmental map and BRDFs as well as the number of inputs. \n- Related Work: \n    - Please elaborate on the differences from Tulsiani et al. (2020). \n    - The idea of using surface representation is relevant to \"Deferred neural rendering: Image synthesis using neural textures\" (ToG, 2019). Please consider adding this (and its follow-up works) to the related work section. \n    - Some papers on modeling reflection in the context of NeRF is also relevant, including but not limited to \"NeRV: Neural Reflectance and Visibility Fields for Relighting and View Synthesis\" and \"Neural Light Transport for Relighting and View Synthesis\"\n\n#### Quality\n- The submission is technically sound.\n- Both the quantitative and qualitative results show that the proposed method significantly outperforms NeRF in the sparse-input regime on the new dataset. \n- However, the experiment section is limited in several major ways:\n    - The submission is only tested on one new dataset - it would be much more convincing if the authors can include experiments on ShapeNet and/or other datasets that are widely used in the community.  \n    - There are a number of variables in the optimization problem, including geometry, textures, camera poses, and specularity. An ablation study on the contribution of these optimization variables to the final accuracy is needed - for instance, how important is it to include the camera poses and the specularity in this optimization framework, or can we remove them without hurting the accuracy? Showing the quality metrics in Table 2 (in the supplementary materials) after each optimization stage would be helpful.\n    - It is unclear why the environment map requires a MLP representation. Can we achieve the same performance with bilinear sampling from an optimizable tensor? Please add an ablation study to support your design choice. \n    - Another great ablation study to include is a curve of reconstruction quality vs. number of input images. This helps readers understand how many images are needed for their use cases.\n- While this submission is interesting to the community, it seems like a work-in-progress rather than a complete piece of work, especially given the limited experiments.\n- Instead of directly claiming NeRF fails, the authors have made an effort to improve NeRF with several modifications to ensure a fair comparison. The extensive visualization provided by the authors is quite helpful. \n\n#### Clarity\n- The paper is overall clearly written and well organized, and the supplementary materials are also helpful. \n- I believe it is largely reproducible based on the technical description. \n- A few comments that might help improve the writing:\n    - Please consider adding new paragraph in Section I as a summary of the main contributions, as well as the differences from NeRF and Tulsiani et al. (2020).\n    - It is not crystal clear from the paper whether the specular material parameters $(k_s, \\alpha)$ are defined on an instance level (e.g., shared by the entire surface of an object instance) or on a per-pixel level (parametrized by another MLP). My understanding is the former, but an explicit explanation would help.\n    - One missing implementation detail is how the authors sample the view directions $\\omega \\in \\Omega$.\n\n#### Significance\n- The results are significant, since it might help enable 3D reconstruction from a sparse set of 2D images. I believe the topic is of great relevance to the machine learning / computer vision / computer graphics community and beyond. \n- I can see practitioners using this technique to build new applications that require less intensive data capturing from users. Researchers could also build on top of the submission in terms of complexity, extending from single objects to complete scenes.\n- Compared to NeRF, the submission's task is more challenging in the input size but at the same time simpler in scene complexity (single object vs. entire scene). It is addressing a simpler task than Tulsiani et al. (2020) given that it is using more input images.\n- It improves the state of the art on few-shot reconstruction in a clear and demonstrable way.\n- It also provides a new, unique dataset for few-shot reconstruction, establishing a new benchmark for future exploration.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "SlOxFYiJna2", "reviewer": "Reviewer_MM89", "summary": "This paper presents a method for surface geometry and surface reflectance estimation from multiple photographs in the wild.  The central idea can be viewed as an extension of the NerF-type neural implicit volumetric representation, here the goal is instead to learn neural implicit surface  representation for both object geometry and surface reflectance.    ", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper  extends the idea of NeRF from volumetric  model to neural surface modelling.   This idea is very similar to the work of Yariv et al in NeurlPS2020 for disentangled multi-view object geometry  recovery (also represented in the form of neural surface) and object reflectance estimation (also used a similar photometric model).    The overall computation pipeline adopts the inverse graphics rendering procedure where a pre-defined mesh (here a simple mesh) is gradually deformed from a canonical shape (a sphere)  to the target shape to conform with the multi-view image measurements.    Rather than directly predicting 3D vertex positions on the mesh,  the paper propose to predict a displacement field (i.e. mesh flows).   \n\nAdvantages:   \n1. impressive experiment results: from a sparse set of multi-view images ( can be as few as 8), the method is able to recovery the shape and texture of objects with general environment lighting. In contrast, NerF often needs more images (>20~40).\n 2.  Estimating displacement field rather than shape itself will in principle make the neural net more generalisable to more complex shapes.  ( though no ablation study is provided  in the paper).\n3.  Contribute a new  \"in the wild\" multi-view object dataset (the Craiglist Multi-View dataset). \n\nLimitations: \n\n1. Limited originality:   The originality and novelty of the main idea of the paper is relatively limited.  Apart from some significant overlaps with Yariv NeurlPs 2020,  as well as with Piex2Mesh++ (in CVPR2019),  the idea of estimation mesh deformation field (i.e. Mesh Flow) rather than mesh itself was also proposed in \"Kunal Gupta and Manmohan Chandraker, Neurla MeshFlow\"  (Neurlps 2020) for the very same task of  image2mesh reconstruction by deforming a canonical initial mesh.    However, no comparison is presented in the paper.   The Pixel2Mesh++ in CVPR2019 also used the same deforming a sphere and rendering and comparing it with the input multi-view images, despite using an explicit surface representation (rather than implicit neural surface representation).   The NeuralMeshFlow pipeline is almost identical to the paper's, which also estimates  a deformation field for neural surface representation.   Doing the above comparisons is imo essential for demonstrating the value of this work.\n\n\n2. Insufficient validation:    As shown in the figures of the paper,  most of the objects being reconstructed have rather simple  (sometimes convex) geometry shapes  ( e.g. ketchup bottle, a sedan/car,  a rice cooker, a coffee machine,  etc.).   For these simple shapes,  the method of \"shape from multi-view silhouettes\" may already work well to recover the shape.   Indeed,  I notice the \"object mask-loss\" used in your method is precisely for this purpose.   Unfortunately, there is no ablation study, nor comparison in your paper to validate this.  It is unclear whether the good result is largely due to the use of the object mask-loss alone.  Which terms plays what role ?     In light of this, the paper is clearly pre-mature, and more work is needed to  validate the main idea.   \n\n3. Unclear analysis:  Similarly,  although the paper claims it also estimates a spherical light map (environment map), the efficacy of this env-map has never been demonstrated or validated in the paper.   After all, the Phong shading model is an overly simplified  lighting model not too far away from the dichromatic model.   The recovered texture map shown in the paper (figure-7 in the main paper, as well as in the demo videos) show  strong highlight/specularity-- suggesting  the reflectance estimation is quite off by your method.   Have you evaluated how accurate the recovered env-map, or the recovered Phong parameters ? \n\nMore importantly,  it is unclear how this estimated spherical environment map resolves the \"fundamental ambiguity\" (in reflectance and lighting).  Given the env-map is an area light source,   how can the network distinguish a low-textured object illuminated by a high-frequency env-map, and a highly-textured object illuminated by a smooth env-map ?   To speak of the least,  there is no account nor experiments about how well/accurate the estimated env-map looks like or works.  \n\n5.  The paper claims ( in line 120) the method only requires approximately calibrated camera parameters to start with.  However, based on the description in the experiment section (line 236-243), it seems the camera parameters are optimized BEFORE optimizing the other parameters (f_shape and f_env and f_tex), and kept fixed afterward.     This is strange, because if you do not have a good enough initial f_shape, f_texture, and F_env the environment map estimations,   how can you expect to get the camera calibration parameters right or accurately ?    The experiments do not seem to support your claim. \n \n6.  Most of the objects shown in the paper (as well as in the video) have the spherical topology.  While the implicit surface representation naturally accepts non-sphere topologies,  the rasterization graphics rendering step often has issue with surface with self-intersection or holes.   The paper failed to analyze this point.  Tests on more complex shapes  (e.g. with non-spherical topology) is necessary.  \n  \n\nOverall,  the paper combines multiple interesting ideas, and experiment results looks good, but its execution (of those ideas) is rather poor,  weak and insufficient.   Its theoretical novelty is limited, and experiment  are also lacking (missing necessary comparisons and ablation studies).   The quality of the paper is okay but not good enough.  It is clearly under-developed and pre-mature, making it just below the bar for NeulPS. \n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 hours reading + 1 hours writing. "}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received borderline reject initial reviews, with the reviewers asking for comparisons with IDR etc. and more discussion on the environment maps. The authors provided significant new results which addressed most of the reviewers' concerns. Limitations remain both on the  accuracy of the illumination/reflectance decomposition and on the range of shapes that the method handles. 
However, based on the current interest in the area, the novel experimental set-up and dataset and the fact that the authors have demonstrated better results in the case of a challenging sparse view set-up, the metareviewer recommends acceptance. The authors are urged to include the comparisons from the rebuttal to the final paper.
Decision: Accept (Poster)