Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering
Year: 2022

Abstract:
Inferring representations of 3D scenes from 2D observations is a fundamental problem of computer graphics, computer vision, and artificial intelligence. Emerging 3D-structured neural scene representations are a promising approach to 3D scene understanding. In this work, we propose a novel neural scene representation, Light Field Networks or LFNs, which represent both geometry and appearance of the underlying 3D scene in a 360-degree, four-dimensional light field parameterized via a neural network. Rendering a ray from an LFN requires only a single network evaluation, as opposed to hundreds of evaluations per ray for ray-marching or volumetric based renderers in 3D-structured neural scene representations. In the setting of simple scenes, we leverage meta-learning to learn a prior over LFNs that enables multi-view consistent light field reconstruction from as little as a single image observation. This results in dramatic reductions in time and memory complexity, and enables real-time rendering. The cost of storing a 360-degree light field via an LFN is two orders of magnitude lower than conventional methods such as the Lumigraph. Utilizing the analytical differentiability of neural implicit representations and a novel parameterization of light space, we further demonstrate the extraction of sparse depth maps from LFNs.

heading: 1 Introduction
text: A fundamental problem across computer graphics, computer vision, and artificial intelligence is to infer a representation of a scene’s 3D shape and appearance given impoverished observations such as 2D images of the scene. Recent contributions have advanced the state of the art for this problem significantly. First, neural implicit representations have enabled efficient representation of local 3D scene properties by mapping a 3D coordinate to local properties of the 3D scene at that coordinate [1– 6]. Second, differentiable neural renderers allow for the inference of these representations given only 2D image observations [3, 4]. Finally, leveraging meta-learning approaches such as hypernetworks or gradient-based meta-learning has enabled the learning of distributions of 3D scenes, and therefore reconstruction given only a single image observation [3]. This has enabled a number of applications, such as novel view synthesis [7, 3, 6], 3D reconstruction [5, 3] semantic segmentation [8, 9], and SLAM [10]. However, 3D-structured neural scene representations come with a major limitation: Their rendering is prohibitively expensive, on the order of tens of seconds for a single 256 × 256 image for state-of-the-art approaches. In particular, parameterizing the scene in 3D space necessitates
∗These authors contributed equally to this work.
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
the discovery of surfaces along camera rays during rendering. This can be solved either by encoding geometry as a level set of an occupancy or signed distance function, or via volumetric rendering, which solves an alpha-compositing problem along each ray. Either approach, however, requires tens or even hundreds of evaluations of the 3D neural scene representation in order to render a single camera ray.
We propose a novel neural scene representation, dubbed Light Field Networks or LFNs. Instead of encoding a scene in 3D space, Light Field Networks encode a scene by directly mapping an oriented camera ray in the four dimensional space of light rays to the radiance observed by that ray. This obviates the need to query opacity and RGB at 3D locations along a ray or to ray-march towards the level set of a signed distance function, speeding up rendering by three orders of magnitude compared to volumetric methods. In addition to directly encoding appearance, we demonstrate that LFNs encode information about scene geometry in their derivatives. Utilizing the unique flexibility of neural field representations, we introduce the use of Plücker coordinates to parameterize 360-degree light fields, which allow for storage of a-priori unbounded scenes and admit a simple expression for the depth as an analytical function of an LFN. Using this relationship, we demonstrate the computation of geometry in the form of sparse depth maps. While 3D-structured neural scene representations are multi-view consistent by design, parameterizing a scene in light space does not come with this guarantee: the additional degree of freedom enables rays that view the same 3D point to change appearance across viewpoints. For the setting of simple scenes, we demonstrate that this challenge can be overcome by learning a prior over 4D light fields in a meta-learning framework. We benchmark with current state-of-the-art approaches for single-shot novel view synthesis, and demonstrate that LFNs compare favorably with globally conditioned 3D-structured representations, while accelerating rendering and reducing memory consumption by orders of magnitude.
In summary, we make the following contributions:
1. We propose Light Field Networks (LFNs), a novel neural scene representation that directly parameterizes the light field of a 3D scene via a neural network, enabling real-time rendering and vast reduction in memory utilization.
2. We demonstrate that we may leverage 6-dimensional Plücker coordinates as a parameterization of light fields, despite their apparent overparameterization of the 4D space of rays, thereby enabling continuous, 360-degree light fields.
3. By embedding LFNs in a meta-learning framework, we demonstrate light field reconstruction and novel view synthesis of simple scenes from sparse 2D image supervision only.
4. We demonstrate that inferred LFNs encode both appearance and geometry of the underlying 3D scenes by extracting sparse depth maps from the derivatives of LFNs, leveraging their analytical differentiability.
Scope. The proposed method is currently constrained to the reconstruction of simple scenes, such as single objects and simple room-scale scenes, in line with recent work on learning generative models in this regime [3, 11].

heading: 2 Related Work
text: Neural Scene Representations and Neural Rendering. A large body of work addresses the question of inferring feature representations of 3D scenes useful to downstream tasks across graphics, vision, and machine learning. Models without 3D structure suffer from poor data efficiency [12, 13]. Voxel grids [14–20] offer 3D structure, but scale poorly with spatial resolution. Inspired by neural implicit representations of 3D geometry [1, 2], recent work has proposed to encode properties of 3D scenes as neural fields (also implicit- or coordinate-based representations, see [21] for an overview), neural networks that map 3D coordinates to local properties of the 3D scene at these coordinates. Using differentiable rendering, these models can be learned from image observations only [3, 4, 22, 11]. Reconstruction from sparse observations can be achieved by learning priors over the space of neural fields [3, 5, 11, 23–25] or by conditioning of the neural field on local features [6, 26, 27]. Differentiable rendering of such 3D-structured neural scene representations is exceptionally computationally intensive, requiring hundreds of evaluations of the neural representation per ray, with tens of thousands to millions of rays per image. Some recent work seeks to accelerate test-time rendering, but either does not admit generalization [28–30], or does not alleviate the cost of
rendering at training/inference time [31–33]. With Light Field Networks, we propose to leverage 360- degree light fields as neural scene representations. We introduce a novel neural field parameterization of 360-degree light fields, infer light fields via meta-learning from as few as a single 2D image observation, and demonstrate that LFNs encode both scene geometry and appearance.
Light fields and their reconstruction. Light fields have a rich history as a scene representation in both computer vision and computer graphics. Adelson et al. [34] introduced the 5D plenoptic function as a unified representation of information in the early visual system [35]. Levoy et al. [36] and, concurrently, Gortler et al. [37] introduced light fields in computer graphics as a 4D sampled scene representation for fast image-based rendering. Light fields have since enjoyed popularity as a representation for novel view synthesis [38] and computational photography, e.g. [39]. Light fields enable direct rendering of novel views by simply extracting a 2D slice of the 4D light field. However, they tend to incur significant storage cost, and since they rely on two-plane parameterizations, they make it hard to achieve a full 360-degree representation without concatenating multiple light fields. A significant amount of prior work addresses reconstruction of fronto-parallel light fields via handcrafted priors, such as sparsity in the Fourier or shearlet domains [40–42]. With the advent of deep learning, approaches to light field reconstruction that leverage convolutional neural networks to in-paint or extrapolate light fields from sparse views have been proposed [43, 7, 44], but similarly only support fronto-parallel novel view synthesis. We are instead interested in light fields as a representation of 3D appearance and geometry that enables efficient inference of and reasoning about the properties of the full underlying scene.

heading: 3 Background: 3D-structured Neural Scene Representations
text: Recent progress in neural scene representation and rendering has been driven by two key innovations. The first are neural fields, often also referred to as neural implicit- or coordinate-based scene representations Φ3D [3, 4], which model a scene as a continuous function, parameterized as an MLP which maps a 3D coordinate to a representation v of whatever is at that 3D coordinate:
Φ3D : R3 → Rn, x 7→ Φ3D(x) = v. (1) The second is a differentiable renderer m, which, given a ray r in R3, and the representation Φ3D, computes the value of the color c of the scene when viewed along r:
m(r,Φ3D) = c(r) ∈ R3. (2)
Existing rendering methods broadly fall into two categories: sphere-tracing-based renderers [3, 45, 5, 46] and volumetric renderers [19, 4]. These methods require on the order of tens or hundreds of
evaluations of the values of Φ3D along a ray r to compute c(r). This leads to extraordinarily large memory and time complexity of rendering. As training requires error backpropagation through the renderer, this impacts both training and test time.

heading: 4 The Light Field Network Scene Representation
text: We propose to represent a scene as a 360-degree neural light field, a function parameterized by an MLP Φφ with parameters φ that directly maps the 4D space L of oriented rays to their observed radiance: Φφ : L → R3, r 7→ Φφ(r) = c(r). (3) A light field completely characterizes the flow of light through unobstructed space in a static scene with fixed illumination. Light fields have the unique property that rendering is achieved by a single evaluation of Φ per light ray, i.e., no ray-casting is required. Moreover, while the light field only encodes appearance explicitly, its derivatives encode geometry information about the underlying 3D scene [47, 34, 35]. This makes many methods to extract 3D geometry from light fields possible [48–51], and we demonstrate efficient recovery of sparse depth maps from LFNs below.

heading: 4.1 Implicit representations for 360 degree light fields
text: To fully represent a 3D scene requires a parameterization of all light rays in space. Conventional light field methods are constrained to leverage minimal parameterizations of the 4D space of rays, due to the high memory requirements of discretely sampled high-dimensional spaces. In contrast, our use of neural field representations allows us to freely choose a continuous parameterization that is mathematically convenient. In particular, we propose to leverage the 6D Plücker parameterization of the space of light rays L for LFNs. The Plücker coordinates (see [52] for an excellent overview) of a ray r through a point p in a normalized direction d are
r = (d,m) ∈ R6 where m = p× d, for d ∈ S2,p ∈ R3. (4) where × denotes the cross product. While Plücker coordinates are a-priori 6-tuples of real numbers, the coordinates of any ray lie on a curved 4-dimensional subspace L. Plücker coordinates uniformly represent all oriented rays in space without singular directions or special cases. Intuitively, a general ray r together with the origin define a plane, and m is a normal vector to the plane with its magnitude capturing the distance from the ray to the origin; if m = 0 then the ray passes through the origin and is defined by its direction d. This is in contrast to conventional light field parameterizations: Fronto-parallel two-plane or cylindrical parameterizations cannot represent the full 360-degree light field of a scene [36, 53]. Cubical two-plane arrangements [37, 38] are not continuous, complicating the parameterization via a neural implicit representation. In contrast to the two-sphere parameterization [54], Plücker coordinates do not require that scenes are bounded in size and do not require spherical trigonometry.
The parameterization via a neural field enables compact storage of a 4D light field that can be sampled at arbitrary resolutions, while non-neural representations are resolution-limited. Neural fields further allow the analytical computation of derivatives. This enables the efficient computation of sparse depth maps, where prior representations of light fields require finite-differences approximations of the gradient [48–50].
Rendering LFNs. To render an image given an LFN, one computes the Plücker coordinates ru,v of the camera rays at each u, v pixel coordinate in the image according to Equation 4. Specifically, given the extrinsic E = [ R|t ] ∈ SE(3) and intrinsic K ∈ R3×3 camera matrices [55] of a camera, one may retrieve the Plücker coordinates of the ray ru,v at pixel coordinate u, v as:
ru,v = (du,v, t× du,v)/‖du,v‖, where du,v = RK−1 ( u v 1 ) + t, (5)
where we use the world-to-camera convention for the extrinsic camera parameters. Rendering then amounts to a single evaluation of the LFN Φ for each ray, cu,v = Φ(ru,v). For notational convenience, we introduce a rendering function
ΘΦE,K : R` → RH×W×3 (6)
which renders an LFN Φφ with parameters φ ∈ R` when viewed from a camera with extrinsic and intrinsic parameters (E,K) into an image.

heading: 4.2 The geometry of Light Field Networks
text: We will now analyze the properties of LFNs representing Lambertian 3D scenes, and illustrate how the geometry of the underlying 3D scene is encoded. We will first derive an expression that establishes a relationship between LFNs and the classic two-plane parameterization of the light field. Subsequently, we will derive an expression for the depth of a ray in terms of the local color gradient of the light field, therefore allowing us to efficiently extract sparse depth maps from the light field at any camera pose via analytical differentiation of the neural implicit representation. Please see Figure 2 for an overview.
Locally linear slices of the light field. We derive here a local parametrization that will allow us to work with an LFN as if it were a conventional 2-plane light field. Given a ray r in Plücker coordinates, we pick two points x,x′ ∈ R3 along this ray. We then find a normalized direction d ∈ S2 not parallel to the ray direction - a canonical choice is a direction orthogonal to the ray direction. We may now parameterize two parallel lines a(s) = x + sd and b(t) = x′ + td that give rise to a local two-plane basis of the light field with ray coordinates s and t. r intersects these lines at the two-plane coordinates (s, t) = (0, 0). This choice of local basis now assigns the two-plane coordinates (s, t) to the ray r from a(s) to b(t). In Figure 2, we illustrate this process on a simple 2D scene.
Epipolar Plane Images and their geometry. The Plücker coordinates (see Eq. 4) enable us to extract a 2D slice from an LFN field by varying (s, t) and sampling Φ on the Plücker coordinates of the rays parametrized pairs of points on the lines a(s) and b(t):
c(s, t) = Φ (r(s, t)) ,where r(s, t) = −−−−−→ a(s)b(t) = ( b(t)− a(s) ‖b(t)− a(s)‖ , a(s)× b(t) ‖b(t)− a(s)‖ ) . (7)
The image of this 2D slice c(s, t) is well-known in the light field literature as an Epipolar Plane Image (EPI) [47]. EPIs carry rich information about the geometry of the underlying 3D scene. For example, consider a point p on the surface of an object in the scene; please see Figure 2 for a diagram. A point p ∈ R2 has a 1-dimensional family of rays going through the point, which correspond to a (green) line Lp in the EPI. In a Lambertian scene, all rays that meet in this point and that are not occluded by other objects must observe the same color. Therefore, the light field is constant along this line. As one travels along Lp, rotating through the family of rays through p, one eventually reaches a (magenta) tangent ray τ to the object. At a tangent ray, the value of the EPI ceases to be constant, and the light field changes its color to whatever is disoccluded by the object at this tangent ray. Because objects of different depth undergo differing amounts of parallax, EPIsRGB Gradients Depthsthe slope of the segment of Lp along which cis constant determines the 3D coordinates of p. Finally, by observing that we may extract EPIs from any perspective, it is clear that an LFN encodes the full 3D geometry of the underlying scene. Intuitively, this may also be seen by con-
sidering that one could render out all possible perspectives of the underlying scene, and solve a classic multi-view stereo problem to retrieve the shape.
Extracting depth maps from LFNs. A correctly inferred light field necessarily contains accurate 3D geometry information, although the geometry is encoded in a nontrivial way. To extract 3D geometry from an LFN, we utilize the property of the 2-plane parameterization that the light field is constant on segments Lp, the slopes of which determine p. In the supplemental material, we derive Proposition 1. For a Lambertian scene, the distance d along r = −−−−−→ a(s)b(t) from a(s) to the point p on the object is
d(r) = D ∂tc(s, t)
∂sc(s, t) + ∂tc(s, t) . (8)
where a(s) and b(t) are as above, c(s, t) is defined by (7), D is the distance between the lines a(t) and b(t). Thus p = a(s) + d(r) b(t)−a(s)‖b(t)−a(s)‖ , and ∂x denotes the partial derivative by variable x.
This result yields meaningful depth estimates wherever the derivatives of the light fields are nonzero along the ray. In practice, we sample several rays in a small (s, t) neighborhood of the ray r and declare depth estimates as invalid if the gradients have high variance-please see the code for implementation details. This occurs when r hits the object at a point where the surface color is changing, or when r is a tangent ray. We note that there is a wealth of prior art that could be used to extend this approach to extract dense depth maps [48–51].

heading: 4.3 Meta-learning with conditional Light Field Networks
text: We consider a dataset D consisting of N 3D scenes
Si = {(Ij ,Ej ,Kj)}Kj=1 ∈ RH×W×3 × SE(3)× R3×3, i = 1 . . . N (9)
with K images Ij of each scene taken with cameras with extrinsic parameters Ej and intrinsic parameters Kj [55]. Each scene is completely described by the parameters φi ∈ R` of its corresponding light field MLP Φi = Φφi .
Meta-learning and multi-view consistency. In the case of 3D-structured neural scene representations, ray-marching or volumetric rendering naturally ensure multi-view consistency of the reconstructed 3D scene representation. In contrast, a general 4D function Φ : L → R3 is not multi-view consistent, as most such functions are not the light fields of any 3D scene. We propose to overcome this challenge by learning a prior over the space of light fields. As we will demonstrate, this prior can also be used to reconstruct an LFN from a single 2D image observation. In this paradigm, differentiable ray-casting is a method to force the light field of a scene to be multi-view consistent, while we instead impose multi-view consistency by learning a prior over light fields.
Meta-learning framework. We propose to represent each 3D scene Si by its own latent vector zi ∈ Rk. Generalizing to new scenes amounts to learning a prior over the space of light fields that is concentrated on the manifold of multi-view consistent light fields of natural scenes. To represent this latent manifold, we utilize a hypernetwork [56, 3]. The hypernetwork is a function, represented as an MLP
Ψ : Rk → R`,Ψψ(zi) = φi (10)
with parameters ψ which sends the latent code zi of the i-th scene to the parameters of the corresponding LFN.
Several reasonable approaches exist to obtain latent codes zi. One may leverage a convolutionalor transformer-based image encoder, directly inferring the latent from an image [11, 5], or utilize gradient-based meta-learning [23]. Here, we follow an auto-decoder framework [1, 3] to find the latent codes zi, but note that LFNs are in no way constrained to this approach. We do not claim that this particular meta-learning method will out-perform other forms of conditioning, such as gradient-based meta-learning [57, 23] or FILM conditioning [58], but perform a comparison to a conditioning-by-concatenation approach in the appendix. We assume that the latent vectors have
a Gaussian prior with zero mean and a diagonal covariance matrix. At training time, we jointly optimize the latent parameters zi together with the hypernetwork parameters ψ using the objective
arg min {zi},ψ ∑ i ∑ j ‖ΘΦEj ,Kj (Ψψ(zi))− Ij‖ 2 2 + λlat‖zi‖22. (11)
Here the ΘΦ is the rendering function (Equation 6), the first term is an `2 loss penalizing the light fields that disagree with the observed images, and the second term enforces the prior over the latent variables. We solve Equation 11 using gradient descent. At test time, we freeze the parameters of the hypernetwork and reconstruct the light field for a new scene S given a single observation of the scene {(I,E,K)} by optimizing, using gradient descent, the latent variable zS of the scene, such that the reconstructed light field ΦΨψ(zS) best matches the given observation of the scene:
zS = arg min z ‖ΘΦE,K (Ψψ(z))− I)‖22 + λlat‖z‖22. (12)
Global vs. local conditioning The proposed meta-learning framework globally conditions an LFN on a single latent variable z. Recent work instead leverages local conditioning, where a neural field is conditioned on local features extracted from a context image [26, 6, 27]. In particular, the recently proposed pixelNeRF [6] has achieved impressive results on few-shot novel view synthesis. As we will see, the current formulation of LFNs does not outperform pixelNeRF. We note, however, that local conditioning methods solve a different problem. Rather than learning a prior over classes of objects, local conditioning methods learn priors over patches, answering the question “How does this image patch look like from a different perspective?”. As a result, this approach does not learn a latent space of neural scene representations. Rather, scene context is required to be available at test time to reason about the underlying 3D scene, and the representation is not compact: the size of the conditioning grows with the number of context observations. In contrast, globally conditioned methods [3, 11, 1, 2] first infer a global representation that is invariant to the number of context views and subsequently discard the observations. However, local conditioning enables better generalization due to the shift-equivariance of convolutional neural networks. An equivalent to local conditioning in light fields is non-obvious, and an exciting direction for future work.

heading: 5 Experiments
text: We demonstrate the efficacy of LFNs by reconstructing 360-degree light fields of a variety of simple 3D scenes. In all experiments, we parameterize LFNs via a 6-layer ReLU MLP, and the hypernetwork as a 3-layer ReLU MLP, both with layer normalization. We solve all optimization problems using the ADAM solver with a step size of 10−4. Please find more results, as well as precise hyperparameter, implementation, and dataset details, in the supplemental document and video.
Reconstructing appearance and geometry of single-object and room-scale light fields. We demonstrate that LFN can parameterize 360-degree light fields of both single-object ShapeNet [59] objects and simple, room-scale environments. We train LFNs on the ShapeNet “cars” dataset with 50 observations per object from [3], as well as on simple room-scale environments as proposed in [13]. Subsequently, we evaluate the ability of LFNs to generate novel views of the underlying 3D scenes. Please see Figure 3 for qualitative results. LFNs succeed in parameterizing the 360-degree light field, enabling novel view synthesis at real-time frame-rates (see supplemental video). We further demonstrate that LFNs encode scene geometry by presenting Epipolar Plane Images and leveraging the relationship derived in Equation 8 to infer sparse depth maps. We highlight that both rendering and depth map extraction do not require ray-casting, with only a single evaluation of the network or the network and its gradient respectively.
Multi-class single-view reconstruction. Following [5, 6], we benchmark LFNs with recent global conditioning methods on the task of single-view reconstruction and novel view synthesis of the 13 largest ShapeNet categories. We follow the same evaluation protocol as [60] and train a single model across all categories. See Figure 4 for qualitative and Table 1 for quantitative baseline comparisons. We significantly outperform both Differentiable Volumetric Rendering (DVR) [5] and Scene Representation Networks (SRNs) [3] on all but two classes by an average of 1dB, while requiring more than an order of magnitude fewer network evaluations per ray. Qualitatively, we find that the reconstructions from LFNs are often crisper than those of either Scene Representation Networks or DVR. Note that DVR requires additional ground-truth foreground-background segmentation masks.
Class-specific single-view reconstruction. We benchmark LFNs on single-shot reconstruction on the Shapenet “cars” and “chairs” classes as proposed in SRNs [3]. See Figure 5 for qualitative and quantitative results. We report performance better than SRNs in PSRN and on par in terms of SSIM
on the “cars” class, and worse in PSNR but better in terms of SSIM on the “chairs” class, while requiring an order of magnitude fewer network evaluations and rendering in real-time. We attribute the drop in performance compared to multi-class reconstruction to the smaller dataset size, causing multi-view inconsistency.
Global vs. local conditioning and comparison to pixelNeRF [6]. We investigate the role of global conditioning, where a single latent is inferred to describe the whole scene [3], to local conditioning, where latents are inferred per-pixel in a 2D image and leveraged to locally condition a neural implicit representation [26, 27, 6]. We benchmark with the recently proposed pixelNeRF [6]. As noted above (see Section 4.3), local conditioning does not infer a compact neural scene representation of the scene. Nevertheless, we provide the comparison here for completeness. See Figure 6 for qualitative and quantitative results. On average, LFNs perform 1dB worse than pixelNeRF in the single-class case, and 2dB worse in the multi-class setting.
Real-time rendering and storage cost. See Table 2 for a quantitative comparison of the rendering complexity of LFN compared with that of volumetric and ray-marching based neural renderers [3, 45, 19, 4, 6]. All clock times were collected for rendering 256× 256 images on an NVIDIA RTX 6000 GPU. We further compare the cost of storing a single LFN with the cost of storing a conventional light field. With approximately 400k parameters, a single LFN requires around 1.6 MB of storage, compared to 146 MB required for storing a 360-degree light field at a resolution of 256×256×17×17 in the six-plane Lumigraph configuration.
Multi-view consistency as a function of training set size. We investigate how multi-view consistency scales with the amount of data that the prior is trained on. Please find this analysis in the supplementary material.
Overfitting of single 3D scenes. We investigate overfitting a single 3D scene with a Light Field Network with positional encodings / sinusoidal activations [24, 61]. Please find this analysis in the supplementary material.
Evaluation of Reconstructed Geometry. We investigate the quality of the geometry that can be computed from an LFN via Eq. 8. For every sample in the class-specific single-shot reconstruction experiment, we extract its per-view sparse depth map. We then backproject depth maps from four views into 3D to reconstruct a point cloud, and benchmark mean L1 error on valid depth estimates with Scene Representation Networks [3]. Fig. 7 displays qualitative and quantitative results. Qualitatively, point clouds succeed in capturing fine detail such as the armrests of chairs. Quantitatively, LFNs outperform SRNs on both cars and chairs. We note that LFNs have a slight advantage in this comparison, as we can only benchmark on the sparse depth values, for which LFNs have high confidence. This includes occlusion boundaries, which are areas where the sphere-tracing based SRNs incurs high error, as it is forced to take smaller and smaller steps and may not reach the surface. We highlight that we do not claim that the proposed method is competetive with methods designed for geometry reconstruction in particular, but that we only report this to demonstrate that the proposed method is capable to extract valid depth estimates from an LFN.
Limitations. First, as every existing light field approach, LFNs store only one color per oriented ray, which makes rendering views from cameras placed in between occluding objects challenging, even if the information may still be stored in the light field. Second, though we outperform globally-conditioned methods, we currently do not outperform the locally conditioned pixelNeRF. Finally, as opposed to 3D-structured representations, LFNs do not enforce strict multi-view consistency, and may be inconsistent in the case of small datasets.

heading: 6 Discussion and Conclusion
text: We have proposed Light Field Networks, a novel neural scene representation that directly parameterizes the full 360-degree, 4D light field of a 3D scene via a neural implicit representation. This enables both real-time neural rendering with a single evaluation of the neural scene representation per ray, as well as sparse depth map extraction without ray-casting. Light Field Networks outperform globally conditioned baselines in single-shot novel view synthesis, while being three orders of magnitude faster and less memory-intensive than current volumetric rendering approaches. Exciting avenues for future work include combining LFNs with local conditioning, which would enable stronger out-of-distribution generalization, studying the learning of non-Lambertian scenes, and enabling camera placement in obstructed 3D space. With this work, we make important contributions to the emerging fields of neural rendering and neural scene representations, with exciting applications across computer vision, computer graphics, and robotics.
Societal Impacts. Potential improvements extending our work on few-observation novel view synthesis could enable abuse by decreasing the cost of non-consensual impersonations. We refer the reader to a recent review of neural rendering [22] for an in-depth discussion of this topic.
Acknowledgements and Disclosure of Funding
This work is supported by the NSF under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, http://iaifi.org/), ONR under 1015
G TA243/N00014-16-1-2007 (Understanding Scenes and Events through Joint Parsing, Cognitive Reasoning and Lifelong Learning), Mitsubishi under 026455-00001 (Building World Models from some data through analysis by synthesis), DARPA under CW3031624 (Transfer, Augmentation and Automatic Learning with Less Labels), as well as the Singapore DSTA under DST00OECI20300823 (New Representations for Vision). We thank Andrea Tagliasacchi, Tomasz Malisiewicz, Prafull Sharma, Ludwig Schubert, Kevin Smith, Bernhard Egger, Christian Richardt, Manuel Rey Area, and Jürgen and Susanne Sitzmann for interesting discussions and feedback, and Alex Yu for kindly sharing the outputs of pixelNeRF and baselines with us.

Review summaries:
The authors propose a novel neural scene representation called Light Field Networks (LFNs), where geometry and appearance of the considered scene are represented in a 360-degree, 4D light field that is parameterized via a neural implicit representation. In contrast to other ray-marching-based or volumetric-rendering-based techniques that rely on hundreds of evaluations per ray in similar tasks, the proposed LFN only requires a single evaluation per ray, thereby significantly improving efficiency and enabling real-time rendering at low memory requirements. Key aspects to achieve this are the parametrization of the space of light rays based on Plücker coordinates. In addition, the authors embed LFNs in a meta-learning framework to allow novel view synthesis from solely sparse 2D image supervision.    The overall approach seems novel and interesting. While the complexity benefits have been demonstrated by the authors, the current approach seems to be limited to simple scenes.  

The paper presents a new neural scene representation based on the idea of light fields. Rather than predicting properties (e.g. occupancies, colors) for points in space, the paper proposes to predict such entities for all rays in a scene using simple MLP networks. For the network input the authors propose to use Plucker coordinates to canonically parametrize the viewing rays independently of a point offset.  To encourage multi-view consistency of the network predictions, the authors propose a meta-learning approach allows to decouple the rendering from the latent code optimization.  In contrast to volumetric neural scene representations that require expensive sampling with multiple network predictions per ray for rendering a novel view, the proposed method only requires a single network evaluation per ray while achieving state-of-the-art rendering results.  

The paper suggests a new representation of the 4D light field using a neural scene representation that predicts the light field from the 6D Plucker coordinates. The new scene representation enables the pixel radiance estimation using a single network query, whereas previous works needed to follow ray-marching or volume rendering procedures that require multiple queries along the ray. The presented scene representation is then utilized to learn multiple shapes or scenes using hypernetworks, in order to learn a prior of multiview consistency. Novel view synthesis results are presented for the learned scenes from the dataset, as well as generalizations from only single view supervision. Moreover, the authors demonstrate how sparse depth maps can be extracted from the learned light field in the case of Lambertian scenes.

Light Field Networks encode the light field of a 3D scene, which comes with certain restrictions on where an observer can be placed. LFNs are coordinate-based MLPs that use the Plucker parameterization of directed rays in 3D space to represent rays. Thus they can output the final color of a pixel with a single network evaluation, massively speeding up rendering relative to NeRF but sacrificing "hard" multi-view consistency, which now needs to be learned. LFNs here are trained via a hypernetwork and scene conditioning happens via an auto-decoded latent code for the hypernetwork. While light fields only represent appearance directly, it is possible to use gradients to extract depth along appearance edges. Results are presented on ShapeNet and very simple synthetic rooms.