NeRS: Neural Reflectance Surfaces for Sparse-view 3D Reconstruction in the Wild

∗ denotes equal coding . Corresponding email : jasonyzhang @ cmu.edu . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1 Introduction . Although we observe the surrounding world only via 2D percepts , it is undeniably 3D . The goal of recovering this underlying 3D from 2D observations has been a longstanding one in the vision community , and any computational approach aimed at this task must answer a central question about representation—how should we model the geometry and appearance of the underlying 3D structure ? An increasingly popular answer to this question is to leverage neural volumetric representations of density and radiance fields ( Mildenhall et al. , 2020 ) . This allows modeling structures from rigid objects to translucent fluids , while further enabling arbitrary view-dependent lighting effects . However , it is precisely this unconstrained expressivity that makes it less robust and unsuitable for modeling 3D objects from sparse views in the wild . While these neural volumetric representations have been incredibly successful , they require hundreds of images , typically with precise camera poses , to model the full 3D structure and appearance of real-world objects . In contrast , when applied to ‘ in-the-wild ’ settings e.g . a sparse set of images with imprecise camera estimates from off-the-shelf systems ( see Fig . 1 ) , they are unable to infer a coherent 3D representation . We argue this is because these neural volumetric representations , by allowing arbitrary densities and lighting , are too flexible . Is there a robust alternative that captures real-world 3D structure ? The vast majority of real-world objects and scenes comprise of well-defined surfaces . This implies that the geometry , rather than being an unconstrained volumetric function , can be modeled as a 2D manifold embedded in euclidean 3D space—and thus encoded via a ( neural ) mapping from a 2D manifold to 3D . Indeed , such meshed surface manifolds form the heart of virtually all rendering engines ( Foley et al. , 1996 ) . Moreover , instead of allowing arbitrary view-dependent radiance , the appearance of such surfaces can be described using ( neural ) bidirectional surface reflection functions ( BRDFs ) , themselves developed by the computer graphics community over decades . We operationalize these insights into Neural Reflectance Surfaces ( NeRS ) , a surface-based neural representation for geometry and appearance . NeRS represents shape using a neural displacement field over a canonical sphere , thus constraining the geometry to be a watertight surface . This representation crucially associates a surface normal to each point , which enables modeling view-dependent lighting effects in a physically grounded manner . Unlike volumetric representations which allow unconstrained radiance , NeRS factorizes surface appearance using a combination of diffuse color ( albedo ) and specularity . It does so by learning neural texture fields over the sphere to capture the albedo at each surface point , while additionally inferring an environment map and surface material properties . This combination of a surface constraint and a factored appearance allows NeRS to learn efficiently and robustly from a sparse set of images in the wild , while being able to capture varying geometry and complex view-dependent appearance . Using only a coarse category-level template and approximate camera poses , NeRS can reconstruct instances from a diverse set of classes . Instead of evaluating in a synthetic setup , we introduce a dataset sourced from marketplace settings where multiple images of a varied set of real-world objects under challenging illumination are easily available . We show NeRS significantly outperforms neural volumetric or classic mesh-based approaches in this challenging setup , and as illustrated in Fig . 1 , is able to accurately model the view-dependent appearance via its disentangled representation . Finally , as cameras recovered in the wild are only approximate , we propose a new evaluation protocol for in-the-wild novel view synthesis in which cameras can be refined during both training and evaluation . We hope that our approach and results highlight the several advantages that neural surface representations offer , and that our work serves as a stepping stone for future investigations . 2 Related Work . Surface-based 3D Representations . As they enable efficient representation and rendering , polygonal meshes are widely used in vision and graphics . In particular , morphable models ( Blanz and Vetter , 1999 ) allow parametrizing shapes as deformations of a canonical template and can even be learned from category-level image collections ( Cashman and Fitzgibbon , 2012 ; Kar et al. , 2015 ) . With the advances in differentiable rendering ( Kato et al. , 2018 ; Laine et al. , 2020 ; Ravi et al. , 2020 ) , these have also been leveraged in learning based frameworks for shape prediction ( Kanazawa et al. , 2018 ; Gkioxari et al. , 2019 ; Goel et al. , 2020 ) and view synthesis ( Riegler and Koltun , 2020 ) . Whereas these approaches use an explicit discrete mesh , some recent methods have proposed using continuous neural surface parametrization like ours to represent shape ( Groueix et al. , 2018 ) and texture ( Tulsiani et al. , 2020 ; Bhattad et al. , 2021 ) . However , all of these works leverage such surface representations for ( coarse ) single-view 3D prediction given a category-level training dataset . In contrast , our aim is to infer such a representation given multiple images of a single instance , and without prior training . Closer to this goal of representing a single instance in detail , contemporary approaches have shown the benefits of using videos ( Li et al. , 2020 ; Yang et al. , 2021 ) to recover detailed shapes , but our work tackles a more challenging setup where correspondence/flow across images is not easily available . In addition , while these prior approaches infer the surface texture , they do not enable the view-dependent appearance effects that our representation can model . Volumetric 3D and Radiance Fields . Volumetric representations for 3D serve as a common , and arguably more flexible alternative to surface based representations , and have been very popular for classical multi-view reconstruction approaches ( Furukawa and Hernández , 2015 ) . These have since been incorporated in deep-learning frameworks for shape prediction ( Girdhar et al. , 2016 ; Choy et al. , 2016 ) and differentiable rendering ( Yan et al. , 2016 ; Tulsiani et al. , 2017 ) . Although these initial approaches used discrete volumetric grids , their continuous neural function analogues have since been proposed to allow finer shape ( Mescheder et al. , 2019 ; Park et al. , 2019 ) and texture modeling ( Oechsle et al. , 2019 ) . Whereas the above methods typically aimed for category-level shape representation , subsequent approaches have shown particularly impressive results when using these representations to model a single instance from images ( Sitzmann et al. , 2019a ; Thies et al. , 2019 ; Sitzmann et al. , 2019b ) – which is the goal of our work . More recently , by leveraging an implicit representation in the form of a Neural Radiance Field , Mildenhall et al . ( 2020 ) showed the ability to model complex geometries and illumination from images . There has since been a flurry of impressive work to further push the boundaries of these representations and allow modeling deformation ( Pumarola et al. , 2020 ; Park et al. , 2021 ) , lighting variation ( Martin-Brualla et al. , 2021 ) , and similar to ours , leveraging insights from surface rendering to model radiance ( Yariv et al. , 2020 ; Boss et al. , 2021 ; Oechsle et al. , 2021 ; Srinivasan et al. , 2021 ; Zhang et al. , 2021b ; Wang et al. , 2021 ; Yariv et al. , 2021 ) . However , unlike our approach which can efficiently learn from a sparse set of images with coarse cameras , these approaches rely on a dense set of multi-view images with precise camera localization to recover a coherent 3D structure of the scene . DietNeRF ( Jain et al. , 2021 ) reduces the number of images but requires precise cameras and semantic supervision . BARF ( Lin et al. , 2021 ) relaxes the constraint of precise cameras while foregoing view-dependent appearance and requiring a dense set of images . Other approaches ( Bi et al. , 2020 ; Zhang et al. , 2021a ) that learn material properties from sparse views require specialized illumination rigs . Multi-view Datasets . Many datasets study the longstanding problem of multi-view reconstruction and view synthesis . However , they are often captured in controlled setups , small in scale , and not diverse enough to capture the span of real world objects . Middlebury ( Seitz et al. , 2006 ) benchmarks multi-view reconstruction , containing two objects with nearly Lambertian surfaces . DTU ( Aanæs et al. , 2016 ) contains eighty objects with various materials but is still captured in a lab with controlled lighting . Freiburg cars ( Sedaghat and Brox , 2015 ) captures 360 degree videos of fifty-two outdoor cars for multi-view reconstruction . ETH3D ( Schöps et al. , 2019 ) and Tanks and Temples ( Knapitsch et al. , 2017 ) contain both indoor and outdoor scenes but are small in scale . Perhaps most relevant are large-scale datasets of real-world objects such as Redwood ( Choi et al. , 2016 ) and Stanford Products ( Oh Song et al. , 2016 ) , but the data is dominated by single-views or small baseline videos . In contrast , our Multi-view Marketplace Cars ( MVMC ) dataset contains thousands of multi-view captures of in-the-wild objects under various illumination conditions , making it suitable for studying and benchmarking algorithms for multi-view reconstruction , view synthesis , and inverse rendering . 3 Method . Given a sparse set of input images of an object under natural lighting conditions , we aim to model its shape and appearance . While recent neural volumetric approaches share a similar goal , they require a dense set of views with precise camera information . Instead , our approach relies only on approximate camera pose estimates and a coarse category-level shape template . Our key insight is that instead of allowing unconstrained densities popularly used for volumetric representations , we can enforce Shape Deformation Network Texture Prediction Network ( a ) ( b ) ( c ) ( d ) ( e ) ( f ) Canonical Sphere Textured Mesh Predicted Shape Predicted Texture a surface-based 3D representation . Importantly , this allows view-dependent appearance variation by leveraging constrained reflection models that decompose appearance into diffuse and specular components . In this section , we first introduce our ( neural ) surface representation that captures the object ’ s shape and texture , and then explain how illumination and specular effects can be modeled for rendering . Finally , we describe how our approach can learn using challenging in-the-wild images . 3.1 Neural Surface Representation . We represent object shape via a deformation of a unit sphere . Previous works ( Kanazawa et al. , 2018 ; Goel et al. , 2020 ) have generally modeled such deformations explicitly : the unit sphere is discretized at some resolution as a 3D mesh with V vertices . Predicting the shape deformation thus amounts to predicting vertex offsets δ ∈ RV×3 . Such explicit discrete representations have several drawbacks . First , they can be computationally expensive for dense meshes with fine details . Second , they lack useful spatial inductive biases as the vertex locations are predicted independently . Finally , the learned deformation model is fixed to a specific level of discretization , making it non-trivial , for instance , to allow for more resolution as needed in regions with richer detail . These limitations also extend to texture parametrization commonly used for such discrete mesh representations—using either per-vertex or per-face texture samples ( Kato et al. , 2018 ) , or fixed resolution texture map , limits the ability to capture finer details . Inspired by Groueix et al . ( 2018 ) ; Tulsiani et al . ( 2020 ) , we address these challenges by adopting a continuous surface representation via a neural network . We illustrate this representation in Fig . 2 . For any point u on the surface of a unit sphere S2 , we represent its 3D deformation x ∈ R3 using the mapping fshape ( u ) = x where fshape is parameterized as a multi-layer perceptron . This network therefore induces a deformation field over the surface of the unit sphere , and this deformed surface serves as our shape representation . We represent the surface texture in a similar manner–as a neural vector field over the surface of the sphere : ftex ( u ) = t ∈ R3 . This surface texture can be interpreted as an implicit UV texture map .

Summary: The submission studies the problem of 3D geometry and appearance reconstruction from a sparse set of 2D images, and proposes a new neural implicit model called Neural Reflectance Surfaces (NeRS). The main difference from prior work (specifically, NeRF) is the following:  1. using a much sparser set of 2D observations as input - typically around 10 images per instance  2. a neural surface model for geometry instead of neural volumetric representations,  3. a neural environment map to model lightings, conditioned only on the viewing direction, 4. BRDFs representation for surface specularity to model view-dependent effects.  In particular, the shape geometry is modeled as a continuous and implicit deformation of a unit sphere, parametrized by a multi-layer perceptron (MLP). Similarly, the diffuse color is parametrized by a second MLP with spherical coordinates as input. To render an image, a continuous unit sphere in 3D is first discretized into a set of vertices (spherical coordinates), which are then passed as input to the shape MLP to construct a triangulated surface mesh. The per-vertex diffuse colors are computed from a second MLP using the same set of spherical coordinates. Together with the neural environment map (from a third MLP), the final pixel intensity can be rendered using a Phong reflection model. These 3 different MLPs are optimized sequentially and iteratively using a combination of 2D image loss, a mask loss, and mesh regularization. The method is compared against different variants of NeRF on a new "Craigslist Multiview" (CMV) dataset. Both quantitative and qualitative comparisons show that the proposed NeRS method outperforms NeRF under this few-shot experiment setup.