Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering

1 Introduction . A fundamental problem across computer graphics , computer vision , and artificial intelligence is to infer a representation of a scene ’ s 3D shape and appearance given impoverished observations such as 2D images of the scene . Recent contributions have advanced the state of the art for this problem significantly . First , neural implicit representations have enabled efficient representation of local 3D scene properties by mapping a 3D coordinate to local properties of the 3D scene at that coordinate [ 1– 6 ] . Second , differentiable neural renderers allow for the inference of these representations given only 2D image observations [ 3 , 4 ] . Finally , leveraging meta-learning approaches such as hypernetworks or gradient-based meta-learning has enabled the learning of distributions of 3D scenes , and therefore reconstruction given only a single image observation [ 3 ] . This has enabled a number of applications , such as novel view synthesis [ 7 , 3 , 6 ] , 3D reconstruction [ 5 , 3 ] semantic segmentation [ 8 , 9 ] , and SLAM [ 10 ] . However , 3D-structured neural scene representations come with a major limitation : Their rendering is prohibitively expensive , on the order of tens of seconds for a single 256 × 256 image for state-of-the-art approaches . In particular , parameterizing the scene in 3D space necessitates ∗These authors contributed equally to this work . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . the discovery of surfaces along camera rays during rendering . This can be solved either by encoding geometry as a level set of an occupancy or signed distance function , or via volumetric rendering , which solves an alpha-compositing problem along each ray . Either approach , however , requires tens or even hundreds of evaluations of the 3D neural scene representation in order to render a single camera ray . We propose a novel neural scene representation , dubbed Light Field Networks or LFNs . Instead of encoding a scene in 3D space , Light Field Networks encode a scene by directly mapping an oriented camera ray in the four dimensional space of light rays to the radiance observed by that ray . This obviates the need to query opacity and RGB at 3D locations along a ray or to ray-march towards the level set of a signed distance function , speeding up rendering by three orders of magnitude compared to volumetric methods . In addition to directly encoding appearance , we demonstrate that LFNs encode information about scene geometry in their derivatives . Utilizing the unique flexibility of neural field representations , we introduce the use of Plücker coordinates to parameterize 360-degree light fields , which allow for storage of a-priori unbounded scenes and admit a simple expression for the depth as an analytical function of an LFN . Using this relationship , we demonstrate the computation of geometry in the form of sparse depth maps . While 3D-structured neural scene representations are multi-view consistent by design , parameterizing a scene in light space does not come with this guarantee : the additional degree of freedom enables rays that view the same 3D point to change appearance across viewpoints . For the setting of simple scenes , we demonstrate that this challenge can be overcome by learning a prior over 4D light fields in a meta-learning framework . We benchmark with current state-of-the-art approaches for single-shot novel view synthesis , and demonstrate that LFNs compare favorably with globally conditioned 3D-structured representations , while accelerating rendering and reducing memory consumption by orders of magnitude . In summary , we make the following contributions : 1 . We propose Light Field Networks ( LFNs ) , a novel neural scene representation that directly parameterizes the light field of a 3D scene via a neural network , enabling real-time rendering and vast reduction in memory utilization . 2 . We demonstrate that we may leverage 6-dimensional Plücker coordinates as a parameterization of light fields , despite their apparent overparameterization of the 4D space of rays , thereby enabling continuous , 360-degree light fields . 3 . By embedding LFNs in a meta-learning framework , we demonstrate light field reconstruction and novel view synthesis of simple scenes from sparse 2D image supervision only . 4 . We demonstrate that inferred LFNs encode both appearance and geometry of the underlying 3D scenes by extracting sparse depth maps from the derivatives of LFNs , leveraging their analytical differentiability . Scope . The proposed method is currently constrained to the reconstruction of simple scenes , such as single objects and simple room-scale scenes , in line with recent work on learning generative models in this regime [ 3 , 11 ] . 2 Related Work . Neural Scene Representations and Neural Rendering . A large body of work addresses the question of inferring feature representations of 3D scenes useful to downstream tasks across graphics , vision , and machine learning . Models without 3D structure suffer from poor data efficiency [ 12 , 13 ] . Voxel grids [ 14–20 ] offer 3D structure , but scale poorly with spatial resolution . Inspired by neural implicit representations of 3D geometry [ 1 , 2 ] , recent work has proposed to encode properties of 3D scenes as neural fields ( also implicit- or coordinate-based representations , see [ 21 ] for an overview ) , neural networks that map 3D coordinates to local properties of the 3D scene at these coordinates . Using differentiable rendering , these models can be learned from image observations only [ 3 , 4 , 22 , 11 ] . Reconstruction from sparse observations can be achieved by learning priors over the space of neural fields [ 3 , 5 , 11 , 23–25 ] or by conditioning of the neural field on local features [ 6 , 26 , 27 ] . Differentiable rendering of such 3D-structured neural scene representations is exceptionally computationally intensive , requiring hundreds of evaluations of the neural representation per ray , with tens of thousands to millions of rays per image . Some recent work seeks to accelerate test-time rendering , but either does not admit generalization [ 28–30 ] , or does not alleviate the cost of rendering at training/inference time [ 31–33 ] . With Light Field Networks , we propose to leverage 360- degree light fields as neural scene representations . We introduce a novel neural field parameterization of 360-degree light fields , infer light fields via meta-learning from as few as a single 2D image observation , and demonstrate that LFNs encode both scene geometry and appearance . Light fields and their reconstruction . Light fields have a rich history as a scene representation in both computer vision and computer graphics . Adelson et al . [ 34 ] introduced the 5D plenoptic function as a unified representation of information in the early visual system [ 35 ] . Levoy et al . [ 36 ] and , concurrently , Gortler et al . [ 37 ] introduced light fields in computer graphics as a 4D sampled scene representation for fast image-based rendering . Light fields have since enjoyed popularity as a representation for novel view synthesis [ 38 ] and computational photography , e.g . [ 39 ] . Light fields enable direct rendering of novel views by simply extracting a 2D slice of the 4D light field . However , they tend to incur significant storage cost , and since they rely on two-plane parameterizations , they make it hard to achieve a full 360-degree representation without concatenating multiple light fields . A significant amount of prior work addresses reconstruction of fronto-parallel light fields via handcrafted priors , such as sparsity in the Fourier or shearlet domains [ 40–42 ] . With the advent of deep learning , approaches to light field reconstruction that leverage convolutional neural networks to in-paint or extrapolate light fields from sparse views have been proposed [ 43 , 7 , 44 ] , but similarly only support fronto-parallel novel view synthesis . We are instead interested in light fields as a representation of 3D appearance and geometry that enables efficient inference of and reasoning about the properties of the full underlying scene . 3 Background : 3D-structured Neural Scene Representations . Recent progress in neural scene representation and rendering has been driven by two key innovations . The first are neural fields , often also referred to as neural implicit- or coordinate-based scene representations Φ3D [ 3 , 4 ] , which model a scene as a continuous function , parameterized as an MLP which maps a 3D coordinate to a representation v of whatever is at that 3D coordinate : Φ3D : R3 → Rn , x 7→ Φ3D ( x ) = v. ( 1 ) The second is a differentiable renderer m , which , given a ray r in R3 , and the representation Φ3D , computes the value of the color c of the scene when viewed along r : m ( r , Φ3D ) = c ( r ) ∈ R3 . ( 2 ) Existing rendering methods broadly fall into two categories : sphere-tracing-based renderers [ 3 , 45 , 5 , 46 ] and volumetric renderers [ 19 , 4 ] . These methods require on the order of tens or hundreds of evaluations of the values of Φ3D along a ray r to compute c ( r ) . This leads to extraordinarily large memory and time complexity of rendering . As training requires error backpropagation through the renderer , this impacts both training and test time . 4 The Light Field Network Scene Representation . We propose to represent a scene as a 360-degree neural light field , a function parameterized by an MLP Φφ with parameters φ that directly maps the 4D space L of oriented rays to their observed radiance : Φφ : L → R3 , r 7→ Φφ ( r ) = c ( r ) . ( 3 ) A light field completely characterizes the flow of light through unobstructed space in a static scene with fixed illumination . Light fields have the unique property that rendering is achieved by a single evaluation of Φ per light ray , i.e. , no ray-casting is required . Moreover , while the light field only encodes appearance explicitly , its derivatives encode geometry information about the underlying 3D scene [ 47 , 34 , 35 ] . This makes many methods to extract 3D geometry from light fields possible [ 48–51 ] , and we demonstrate efficient recovery of sparse depth maps from LFNs below . 4.1 Implicit representations for 360 degree light fields . To fully represent a 3D scene requires a parameterization of all light rays in space . Conventional light field methods are constrained to leverage minimal parameterizations of the 4D space of rays , due to the high memory requirements of discretely sampled high-dimensional spaces . In contrast , our use of neural field representations allows us to freely choose a continuous parameterization that is mathematically convenient . In particular , we propose to leverage the 6D Plücker parameterization of the space of light rays L for LFNs . The Plücker coordinates ( see [ 52 ] for an excellent overview ) of a ray r through a point p in a normalized direction d are r = ( d , m ) ∈ R6 where m = p× d , for d ∈ S2 , p ∈ R3 . ( 4 ) where × denotes the cross product . While Plücker coordinates are a-priori 6-tuples of real numbers , the coordinates of any ray lie on a curved 4-dimensional subspace L. Plücker coordinates uniformly represent all oriented rays in space without singular directions or special cases . Intuitively , a general ray r together with the origin define a plane , and m is a normal vector to the plane with its magnitude capturing the distance from the ray to the origin ; if m = 0 then the ray passes through the origin and is defined by its direction d. This is in contrast to conventional light field parameterizations : Fronto-parallel two-plane or cylindrical parameterizations can not represent the full 360-degree light field of a scene [ 36 , 53 ] . Cubical two-plane arrangements [ 37 , 38 ] are not continuous , complicating the parameterization via a neural implicit representation . In contrast to the two-sphere parameterization [ 54 ] , Plücker coordinates do not require that scenes are bounded in size and do not require spherical trigonometry . The parameterization via a neural field enables compact storage of a 4D light field that can be sampled at arbitrary resolutions , while non-neural representations are resolution-limited . Neural fields further allow the analytical computation of derivatives . This enables the efficient computation of sparse depth maps , where prior representations of light fields require finite-differences approximations of the gradient [ 48–50 ] . Rendering LFNs . To render an image given an LFN , one computes the Plücker coordinates ru , v of the camera rays at each u , v pixel coordinate in the image according to Equation 4 . Specifically , given the extrinsic E = [ R|t ] ∈ SE ( 3 ) and intrinsic K ∈ R3×3 camera matrices [ 55 ] of a camera , one may retrieve the Plücker coordinates of the ray ru , v at pixel coordinate u , v as : ru , v = ( du , v , t× du , v ) /‖du , v‖ , where du , v = RK−1 ( u v 1 ) + t , ( 5 ) where we use the world-to-camera convention for the extrinsic camera parameters . Rendering then amounts to a single evaluation of the LFN Φ for each ray , cu , v = Φ ( ru , v ) . For notational convenience , we introduce a rendering function ΘΦE , K : R ` → RH×W×3 ( 6 ) which renders an LFN Φφ with parameters φ ∈ R ` when viewed from a camera with extrinsic and intrinsic parameters ( E , K ) into an image .

Summary: The paper presents a new neural scene representation based on the idea of light fields. Rather than predicting properties (e.g. occupancies, colors) for points in space, the paper proposes to predict such entities for all rays in a scene using simple MLP networks. For the network input the authors propose to use Plucker coordinates to canonically parametrize the viewing rays independently of a point offset.  To encourage multi-view consistency of the network predictions, the authors propose a meta-learning approach allows to decouple the rendering from the latent code optimization.  In contrast to volumetric neural scene representations that require expensive sampling with multiple network predictions per ray for rendering a novel view, the proposed method only requires a single network evaluation per ray while achieving state-of-the-art rendering results.