Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering
Authors: Vincent Sitzmann
Semon Rezchikov
William T. Freeman
Joshua B. Tenenbaum
Fredo Durand
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Neural Fields
Neural Scene Representations
Neural Implicit Representations
Coordinate-based representations
Neural Rendering
Light Fields

Abstract: Inferring representations of 3D scenes from 2D observations is a fundamental problem of computer graphics, computer vision, and artificial intelligence. Emerging 3D-structured neural scene representations are a promising approach to 3D scene understanding. In this work, we propose a novel neural scene representation, Light Field Networks or LFNs, which represent both geometry and appearance of the underlying 3D scene in a 360-degree, four-dimensional light field parameterized via a neural implicit representation.  Rendering a ray from an LFN requires only a *single* network evaluation, as opposed to hundreds of evaluations per ray for ray-marching or volumetric based renderers in 3D-structured neural scene representations.  In the setting of simple scenes, we leverage meta-learning to learn a prior over LFNs that enables multi-view consistent light field reconstruction from as little as a single image observation. This results in dramatic reductions in time and memory complexity, and enables real-time rendering. The cost of storing a 360-degree light field via an LFN is two orders of magnitude lower than conventional methods such as the Lumigraph.  Utilizing the analytical differentiability of neural implicit representations and a novel parameterization of light space, we further demonstrate the extraction of sparse depth maps from LFNs. 

Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering
Vincent Sitzmann +* si tzmann@mi edu
Semon Rezchikoy skr@math. columbia wwluuaaaaeiii..... edu
William T. Freeman
1.4.5 Joshua B. Tenenbaum 1.4.5
Frédo Durand'
billffmit. edd
jbt@mit. edu
fredo@mit edu
MIT CSAIL
Columbia University IAFI *MIT BCS
*CBMM
vsitzmann meatttann.githubb folhf..//ff/////////////
Abstract
Inferring representations of 3D scenes from 2D observations is fundamental prob- lem of computer graphics, computer vision, and artificial intelligence. Emerging 3D-structured neural scene representations are promising approach to 3D scene understanding. In this work, we propose novel neural scene representation, Light Field Networks LFNs, which represent both geometry and appearance of the underlying 3D scene in a 360-degree, diunnsimensioaal light field parameterized via neural network. Rendering ray from an LFN requires only single network evaluation, as opposed to hundreds of evaluations per ray for ray- ray-marching volumetric based renderers 3D-structured neural scene representations. setting of simple scenes, we leverage meta-learning to learn prior over LFNs that enables multi-view consistent light field reconstruction from as little as a single image observation. This results in dramatic reductions in time and memory complexity, and enables real-time rendering. The cost of storing a 360-degree light field via an LFN is two orders of magnitude lower than conventional methods such as the Lumigraph. Utilizing the analytical differentiability of neural implicit representations and a novel parameterization of light space, further demonstrate the extraction of sparse depth maps from LFNs.
the
Introduction
fundamental problem across computer graphics, computer vision, and artificial intelligence is to infer a representation of scene's 3D shape and appearance given impoverished observations such as 2D images of the scene. Recent contributions have advanced the state of the art for this problem significantly. First, neural implicit representations have enabled efficient representation of local 3D scene properties by mapping a 3D coordinate to local properties of the 3D scene at that coordinate [1- 6]. Second, differentiable neural renderers allow for the inference of these representations given only 2D image observations [3, 4]. Finally, leveraging meta-learning approaches such as hypernetworks or gradient-based meta-learning has enabled the learning of distributions of 3D scenes, and therefore reconstruction given only single image observation
[3]. This has enabled a number of applications, such novel view synthesis [7, 3, 6). 3D reconstruction [5, 3] semantic segmentation [8, 9], and SLAM [10]. However, 3D-structured neural scene representations come with a major limitation: Their rendering is prohibitively expensive, on the order of tens of seconds for a single 256 256 image for state-of-the-ar approaches. In particular, parameterizing the scene in 3D space necessitates
'These authors contributed equally to this work.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

the discovery of surfaces along camera rays during rendering. This can be solved either by encoding geometry as a level set of an occupancy or signed distance function, or via volumetric rendering. which solves an alpha-compositing problem along each ray. Either approach, however, requires tens or even hundreds of evaluations of the 3D neural scene representation in order to render a single camera ray.
We propose a novel neural scene representation, dubbed Light Field Networks or LFNs. Instead of encoding a scene in 3D space, Light Field Networks encode a scene by directly mapping an oriented camera ray in the four dimensional space of light rays to the radiance observed by that ray. This obviates the need to query opacity and RGB at 3D locations along ray or to ray- ray-march towards the level set of a signed distance function, speeding up rendering by three orders of magnitude compared to volumetric methods. In addition to directly encoding appearance, we demonstrate that LFNs encode information about scene geometry in their derivatives. Utilizing the unique flexibility of neural field representations, we introduce the use of Plücker coordinates to parameterize 360-de light fields, which allow for storage of a-priori unbounded scenes and admit a simple expression for the depth as analytical function of an LFN. Using this relationship, we demonstrate the computation of geometry in the form of sparse depth maps. While 3D-structured neural scene representations are multi-view consistent by design, parameterizing scene in light space does not come with this guarantee: the additional degree of freedom enables rays that view the same 3D point to change appearance across viewpoints. For the setting of simple scenes, can be overcome by learning a prior over 4D light fields in a meta-learning framework. We benchmark with current state-of-ther off-th---rrtttt art approaches for single-shot novel view synthesis, and demonstrate that LFNS compare favorably with globally conditioned 3D-structured representations, while accelerating rendering and reducing memory consumption by orders of magnitude.
demonstrate that this challenge
In summary, we make the following contributions:
1. We propose Light Field Networks (LFNss, a novel neural scene representation that directly parameterizes the light field of a 3D scene via a neural network, enabling real-time rendering and vast reduction in memory utilization.
2. We demonstrate that we may leverage 6-dimensional Plücker coordinates as a parameteri- zation of light fields, despite their apparent overparameterrizatoon of the 4D space of rays, thereby enabling continuous, 360-degree light fields.
3. By embedding LFNs in a meta-learning framework, we demonstrate light field reconstruction and novel view synthesis of simple scenes from sparse 2D image supervision only.
4. We demonstrate that inferred LFNs encode both appearance and geometry of the underlying 3D scenes by extracting sparse depth maps from the derivatives of LFNs, leveraging their analytical differentiability.
Scope. The proposed method is currently constrained to the reconstruction of simple scenes, such as single objects and simple room-scale scenes, in line with recent work on learning generative models in this regime [3, 11].
2 Related Work
Neural Scene Representations and Neural Rendering. large body of work addresses the question of inferring feature representations of 3D scenes useful to downstream tasks across graphics, vision, and machine learning. Models without 3D structure suffer from poor data efficiency [12, 13). Voxel grids [14-20] offer 3D structure, but scale poorly with spatial resolution. Inspired by neural implicit representations of 3D geometry [1. 2], recent work has proposed to encode properties of 3D scenes as neural fields (also implicit- or coordinate-basee overview), neural networks that map 3D coordinates local properties of the 3D scene these coordinates. Using differentiable rendering, these models can be learned from image observations only [3, 4. 22, 11]. Reconstruction from sparse observations can be achieved by learning priors over the space of neural fields [3, 5. 11, 23-25] or by conditioning of the neural field on local features [6, 26, 27]. Differentiable rendering of such 3D-structured neural scene representations is exceptionally computationally intensive, requiring hundreds of evaluations of the neural representation per ray, with tens of thousands to millions of rays per image. Some recent work seeks to accelerate test-time rendering, but either does not admit generalization [28-30], or does not alleviate the cost of
representations, see [21] for an
2

---

Volumetric & Ray-marching based remdering: Hundrods of samples per ray
X2
X3
X
Ray Color
c
$
(x)
Light Field Networka: Single sample ray
X
Plücker Coords.
Ray Color
r = (d, x x d)
Observed
Conditioning
Image
Pace (r)
Figure 1: Overview. We propose Light Field Networks or LFNs, which encode the full 360-degree light field of a 3D scene in the weights of a fully connected neural network & (with weights ø conditioned latent code 2) that maps oriented ray to the radiance с observed by that ray. Rendering an LFN & only requires evaluating the underlying MLP once per ray. in contrast to 3D-structured neural scene representations aaD such SRNs [3], NeRF [4], or DVR [5] that require hundreds of evaluations per ray. We leverage meta-learning to learn a multi-view consistent space of LFNs. Once trained, this enables reconstruction of 360-degree light field and subsequent real-time novel view synthesis of simple scenes from only single observation.
rendering training/inference time [31-33]. With Light Field Networks, propose to leverage 360- degree light fields as neural scene representations. We introduce a novel neural field parameterization of 360-degree light fields, infer light fields via meta-learning from as few single 2D image observation, and demonstrate that LFNs encode both scene geometry and appearance.
Light fields and their reconstruction. Light fields have rich history as a scene representation in both computer vision and computer graphics. Adelson et al. [34] introduced the 5D plenoptic function as unified representation of information in the early visual system [35]. Levoy et al. [36] and, concurrently, Gortler et al. [37] introduced light fields in computer graphics as a 4D sampled scene representation for fast image-based rendering. Light fields have since enjoyed popularity representation for novel view synthesis [38] and computational photography, e.g. [39]. Light fields enable direct rendering of novel views by simply extracting a 2D slice of the 4D light field. However, they tend to incur significant storage cost, and since they rely on two two-plane parameterizations. they make it hard achieve a full 360-degree representation without concatenating multiple light fields. A significant amount of prior work addresses reconstruction of fronto-parale light fields via hand- crafted priors, such as sparsity in the Fourier shearlet domains [40-42]. With the advent of deep learning, approaches to light field reconstruction that leverage convolutional neural networks to in-paint or extrapolate light fields from sparse views have been proposed [43, 7, 44], but similarly only support montoo-paralle novel view synthesis. We are instead interested in light fields representation 3D appearance and geometry that enables efficient inference of and reasoning about the properties of the full underlying scene.
3 Background: 3D-structured Neural Scene Representations
Recent progress in neural scene representation and rendering has been driven by two key innovations. The first are neural fields, often also referred to neural implicit- or coordinate-based scene representations 3DD [3, 4], which model a scene as a continuous function, parameterized as an MLP which maps 3D coordinate to representation of whatever is that 3D coordinate:
3DD E23 + R", x 3D (x) V.
(1)
The second is differentiable renderer m, which, given ray in R3. and the representation Φ3D computes the value the color e the scene when viewed along r:
m(r 3DD) c(r) R
Existing rendering methods broadly fall into two categories: sphere-tracing- based renderers [3, 45, 5, 46] and volumetric renderers [19, 4]. These methods require the order of tens or hundreds of
3

---

evaluations of the values of 430 along ray to compute c(r). This leads to extraordinarily large memory and time complexity of rendering. As training requires error backpropagation through the renderer, this impacts both training and test time.
4 The Light Field Network Scene Representation
We propose to represent scene 360-degree neural light field, function parameterized by an MLP Φg with parameters & that directly maps the 4D space L of oriented rays to their observed radiance:
R R Po(r) c(r).
(3)
A light field completely characterizes the flow of light through unobstructed space in a static scene with fixed illumination. Light fields have the unique property that rendering is achieved by a single evaluation of & per light ray, i.e., no ray-casting is required. Moreover, while the light field only encodes appearance explicitly, its derivatives encode geometry information about the underlying 3D scene [47, 34, 35J. This makes many methods to extract 3D geometry from light fields possible [48-51], and we demonstrate efficient recovery of sparse depth maps from LFNs below.
4.1 Implicit representations for 360 degree light fields
To fully represent a 3D scene requires a parameterization of all light rays in space. Conventional light field methods are constrained leverage minimal parameterizations of the 4D space of rays, due to the high memory requirements of discretely sampled high-dimensional spaces. In contrast, our use of neural field representations allows us to freely choose a continuous parameterization that is mathematically convenient. In particular, we propose to leverage the 6D Plücker parameterization of the space of light rays L for LFNs. The Plücker coordinates (see [52] for an excellent overview) of a ray r through point p in a normalized direction d are
r = (d, m) E R where m = x d, for d S2 P
(4)
where x denotes the cross product. While Plücker coordinates are a-priori 6-tuples of real numbers, the coordinates of any ray lie on a curved 4-dimensional subspace L. Plücker coordinates uniformly represent all oriented rays in space without singular directions or special cases. Intuitively, a general ray r together with the origin define a plane, and m is a normal vector to the plane with its magnitude capturing the distance from the ray to the origin; if through the origin and is defined by its direction d. This is in contrast to conventional light field parameterizations: Pronto-parallel two-plane or cylindrical parameterizations cannot represent the full 360-degree light field of scene [36, 53]. Cubical two-plane arrangements [37, 38] are not continuous, complicating the parameterization via a neural implicit representation. In contrast to the two-sphere parameterization [54], Plücker coordinates do not require that scenes are bounded in size and do not require spherical trigonometry.
then the ray passes
The parameterization via a neural field enables compact storage of a 4D light field that can be sampled at arbitrary resolutions, while non-neural representations are essolution-limited Neural fields further allow the analytical computation of derivatives. This enables the efficient computation of sparse depth maps, where prior representations of light fields require finiteddifference approximations of the gradient [48-50].
Rendering LFNs.
To render image given LFN, one computes the Plücker coordinates ra,v of the camera rays at each u. pixel coordinate in the image according to Equation 4. Specifically, given the extrinsie E- R|t] SE)) and intrinsic K E gax3 camera matrices [55] of a camera, one may retrieve the Plücker coordinates of the ray Fu,v at pixel coordinate u, v as:
F v (du,,, t d,,,:/|(dd..) where d R.K
t,
(5)
where we use the world-to-camera convention for the extrinsic camera parameters. Rendering then amounts to single evaluation of the LFN for each ray, Cu,t = ((r,,)). For notational convenience, we introduce rendering function
: R Xx
E.
which renders an LFN & with parameters & E RC when viewed from a camera with extrinsic and intrinsic parameters (E, K) into an image.

---

b (()
(t))
c(s, t)
ss
Lp
Lp
(s)
(s))
Figure 2: Given a 3D scene (left) and a light ray r (blue), we can slice the scene along 2D plane containing the ray (light blue), yielding a 2D scene (center). The light field of all rays in a 2D plane, the Epipolar Plane Image (EPI) c(s, (right). can be analytically computed from our 360-degree LFN. The family of rays Lp (green) going through a point on an object in the scene defines a straight line in the EPL. See below for further discussion of EPI geometry.
4.2 The geometry of Light Field Networks
We will now analyze the properties of LFNs representing Lambertian 3D scenes, and illustrate how the geometry of the underlying 3D scene is encoded. We will first derive an expression that establishes a relationship between LFNs and the classic two-plane parameterization of the light field. Subsequently, we will derive expression for the depth of ray in terms of the local color gradient of the light field, therefore allowing us to efficiently extract sparse depth maps from the light field at any camera pose via analytical differentiation of the neural implicit representation. Please see Figure/ for an overview.
Locally linear slices of the light field. We derive here local parametrization that will allow us to work with an LFN as if it were a conventional 2-plane light field. Given ray in Plücker coordinates, we pick two points x, E RR3 along this ray. We then find normalized direction d $ not parallel to the ray direction may now parameterize two parallel lines a(s) = x + sd and b(t) x' td that give rise to local two-plane basis of the light field with ray coordinates and l. intersects these lines at the two-plane coordinates (s, () = (0, 0). This choice of local basis now assigns the two-plane coordinates (8, () to the ray from a(s) to b(l). In Figure 2, we illustrate this process on a simple 2D scene.
canonical choice is a direction orthogonal to the ray direction. We
Epipolar Plane Images and their geometry. The Plücker coordinates (see Eq. 4) enable us to extract 2D slice from an LFN field by varying ts and sampling T the Plücker coordinates of the rays parametrized pairs of points on the lines a(s) and b(t):
b(t) - a(s) a(s) x b(t) bb(() a())ll [[((t) a(s)l)
c(s, t) = db (r(s,t)) where r(s, t) a())b(t)
(7)
The image of this 2D slice c(s, is well-known in the light field literature
Epipolar Plane Image (EPI) [47]. EPIs carry rich information about the geometry of the underlying 3D scene. For example, consider point p on the surface of an object in the scene; please see Figure 2 for a diagram. A point E R²2 has a l-dimensional family of rays going through the point, which correspond (green) line Tp in the EPI. In Lambertian scene, all rays that meet in this point and that are not occluded by other objects must observe the same color. Therefore, the light field constant along this line. As one travels along 4p. rotating through the family of rays through p. one eventually reaches a (magenta) tangent ray to the object. At a tangent ray, the value of the EPI ceases to be constant, and the light field changes its color to whatever disoccluded by the object at this tangent ray. Because objects of different depth undergo differing amounts of parallax, the slope of the segment of Lpp along which is constant determines the 3D coordinates of p. Finally, by observing that we may extract EPIs from any perspective, it is clear that an LFN encodes the full 3D geometry of the underlying scene. Intuitively, this may also be seen by con-
RGB
EPIs
Gradients
Depths
5

---

sidering that one could render out all possible perspectives of the underlying scene, and solve classic multi-view stereo problem retrieve the shape.
Extracting depth maps from LFNs. A correctly inferred light field necessarily contains accurate 3D geometry information, although the geometry is encoded in a nontrivial way. To extract 3D geometry from an LFN, we utilize the property of the 2-plane parameterization that the light field constant segments Lp. the slopes of which determine p. In the supplemental material, we derive
Proposition For a Lambertian scene, the distance d along r = b()) from a(s) to the point p on the object is
0,c(s,t) d(r) d, + d,c(s,t)
(8)
where a(s) and btt) are above, c(s, t) is defined by (7), D is the distance between the lines a(t) and b(t). Thus p = a(s) + d(r) [b()) [| and 0g denotes the partial derivative by variable x.
b(t) m(s)
This result yields meaningful depth estimates wherever the derivatives of the light fields are nonzero along the ray. practice, we sample several rays a small (8, () neighborhood of the ray r and declare depth estimates invalid if the gradients have high variance- implementation details. This occurs when r hits the object at a point where the surface color is changing, when is a tangent ray. We note that there is wealth of prior art that could be used to extend this approach to extract dense depth maps [48-51].
the code for
4.3 Meta-learning with conditional Light Field Networks
We consider a dataset D consisting of N 3D scenes
S = {(T, E,, K "xxx3 x SF((3) x 3x3 i
with K images I, of each scene taken with cameras with extrinsic parameters E, and intrinsic param- eters K, [55]. Each scene is completely described by the parameters Φ; E R'' of its corresponding light field MLP 4, Lo.
Meta-learning and multi-view consistency. In the case of 3D-structured neural scene representa- tions, ray-marching or volumetric rendering naturally ensure multi-view consistency of the recon- structed 3D scene representation. In contrast, a general 4D function consistent, as most such functions are not the light fields of any 3D scene. We propose to overcome this challenge by learning a prior over the space of light fields. As we will demonstrate, this prior can also be used to reconstruct an LFN from a single 2D image observation. In this paradigm, differentiable ray-casting is a method to force the light field of a scene to multi-view consistent, while we instead impose multi-view consistency by learning a prior over light fields.
C - R33 is not multi-view
Meta-learning framework. We propose to represent each 3D scene S, by its own latent vector Z, E R Generalizing to new scenes amounts to learning a prior over the space of light fields that is concentrated on the manifold of multi-viee consistent light fields of natural scenes. To represent this latent manifold, we utilize a hypernetwork [56, 3]. The hypernetwork a function, represented as an MLP
R R (Z.) Φb
(10)
with parameters which sends the latent code z, of the i-th scene to the parameters of the corre- sponding LFN.
Several reasonable approaches exist to obtain latent codes z; One may leverage convolutional- or transformer- based image encoder, directly inferring the latent from an image [11, 5], or utilize
basedd meta-learning [23]. Here, we follow an auto-decoder framework [1, 3] to find the latent codes z;, but note that LFNs are in no way constrained to this approach. We do not claim that this particular meta-learning method will out-perform other forms of conditioning, such as gradient- meta-learning [57, 23] or FILM conditioning [58], but perform comparison to conditioning-by
bundiioninggby-coocaatttttoo con.atenatttnnnnnnnnccnnmm approach in the appendix. We assume that the latent vectors have

---

3D Light Field Slice
RGB
Novel Views
Depth
100
RGB
Local Light Field
Novel Views
Depth
Figure 3: 360-degree light field parameterization. Top left: 3D slice in the style of the Lumi- graph [37] of an LFN of room-scale scene. Bottom left: nearby views of
well vertical (right, red) and horizontal (top, green) Epipolar Plane Images, sampled from an LFN. Reconstructed from training set, 50 and 15 views, respectively. Right: LFNS enable rendering from arbitrary, 360-degree camera perspectives as well as sparse depth map extraction from only a single sample per ray. Please see the supplemental video for more qualitative results.
Gaussian prior with zero mean and diagonal covariance matrix. training time, we jointly optimize the latent parameters z; together with the hypernetwork parameters t using the objective
arg min (x: ).v
ΣΣ 10 IEIGE,E K (z.)) 1, 11 + Atat 12:12
(1I)
Here the * is the rendering function (Equation 6), the first term is an &g loss penalizing the light fields that disagree with the observed images, and the second term enforces the prior over the latent variables. We solve Equation 11 using gradient descent. At test time, we freeze the parameters of the hypernetwork and reconstruct the light field for a new scene given a single observation of the scene {(I, E, K)} by optimizing, using gradient descent, the latent variable 2ş of the scene, such that the reconstructed light field &e)()s) best matches the given observation of the scene:
255 arg min | (z)( - 1))1 + Atar Mall
(12)
Global vs. local conditioning The proposed meta-learning framework globally conditions an LFN on single latent variable z. Recent work instead leverages local conditioning, where neural field is conditioned on local features extracted from a context image [26, 6, 27]. In particular, the recently proposed pixeINeRF [6] has achieved impressive results few-shot novel view synthesis. As will see, the current formulation of I_FNs does not outperform pixeINeRF. We note, however, that local conditioning methods solve different problem. Rather than learning a prior over classes of objects, local conditioning methods learn priors over patches, answering the question "How does this image patch look like from a different perspective?"' As result, this approach does not learn a latent space of neural scene representations. Rather, scene context is required to be available at test time to reason about the underlying 3D scene, and the representation not compact: the size of the conditioning grows with the number of context observations. In contrast, globally conditioned methods [3, 11, 1, 2] first infer global representation that is invariant to the number of context views and subsequently discard the observations. However, local conditioning enables better generalization due to the shift-equivariaacee convolutional neural networks. equivalent to local conditioning in light fields is non-obvious, and exciting direction for future work.

---

Input
DVR
SRNs
Ours
GT
Input
DVR
SRNs
Ours
GT
Input
DVR
SRNs
Ours
GT
<
Figure 4: Category-agnostic single-shot reconstruction. We train a single model on the 13 largest Shapenet classes. LFNs consistently outperform global conditioning baselines. Baseline results courtesy of the pixelNeRF [6] authors. Please see the supplemental video for more qualitative results.
Table I: Single-shot multi-class reconstruction results. We benchmark LFNs with SRNs [3] and DVR [5] on the task of single-shot (auto-decoding with a single view), multi-class reconstruction of the 13 largest ShapeNet [59] classes. LFNs significantly outperform DVR and SRNs on almost all classes, on average by more than 1dB, while only requiring single network evaluation per ray. Note that DVR requires additional supervision in the form of foreground-backgroundd masks. Means across classes are weighted equally.
plane
bench
chnt.
chair
lamp
spkr.
rifle
sofa
table
phone
boat
DVR [S]
25.29
22.64
24.47
23.95
19.91
20.86
23.27
20.78
23.44
23.35
21.53
24.18
25.09
22.70
PSNR
SRN [3]
26.62
22.20 20
23.42
24.40
21.85
19.07
22.17
21.04
24.95
23.65
22.45
20.87
25.86
23.28
LPN
29.95
23.21
25.91
28.04
22.94
20.64
24.56
22.54
27.50
25.15
24.58
22.21
27.16
24.95
DVR
0.905
0.866
0.877
0,909
0.787
0.814
0.849
0.798
0.916
0.868
0.840
0.892
0.902
0,860
1 1SSM
SRN [3]
0.901
0.837
0.831
0.897
0.814
0.744
0.801
0.779
0.913
0.851
0.828
0.811
0.898
0.849
LFN
0.932
0.855
0.871
0,943
0.835
0.786
0.844
0.808
0.939
0.874
0.868
0.844
0,907
0,870
Experiments
We demonstrate the efficacy of LFNs by reconstructing 360-degree light fields of variety of simple 3D scenes. In all experiments, we parameterize LFNs via a 6-layer ReLU MLP, and the hypernetwork as 3-layer ReLU MLP, both with layer normalization. We solve all optimization problems using the ADAM solver with a step size of 10-4. Please find more results, as well as precise hyperparameter, implementation, and dataset details, in the supplemental document and video.
Reconstructing appearance and geometry of single-object and room-scale light fields. We
demonstrate that LFN can parameterize 360-degree light fields of both single-object ShapeNet [59] objects and simple, room-scale environments. We train LFNs on the ShapeNet "cars" dataset with 50 observations per object from [3], well as on simple room-scale environments as proposed in [13]. Subsequently, we evaluate the ability of LFNs to generate novel views of the underlying 3D scenes. Please see Figure for qualitative results. LFNS succeed in parameterizing the 360-degree light field, enabling novel view synthesis at real-time frame-rates (see supplemental video). We further demonstrate that LFNs encode scene geometry by presenting Epipolar Plane Images and leveraging the relationship derived in Equation to infer sparse depth maps. We highlight that both rendering and depth map extraction do not require ray-casting, with only single evaluation of the network or the network and its gradient respectively.
Multi-class single-view reconstruction. Following [5, 6), we benchmark LFNs with recent global conditioning methods on the task of single-view reconstruction and novel view synthesis of the 13 largest ShapeNet categories. We follow the same evaluation protocol as [60] and train a single model across all categories. See Figure 4 for qualitative and Table for quantitative baseline compar- isons. We significantly outperform both Differentiable Volumetric Rendering (DVR) [5] and Scene Representation Networks (SRNs) [3] on all but two classes by an average of 1dB, while requiring more than an order of magnitude fewer network evaluations per ray. Qualitatively, we find that the reconstructions from LFNs are often crisper than those of either Scene Representation Networks or DVR. Note that DVR requires additional ground-truth bregrounddbackgroone segmentation masks.
Class-specific single-view reconstruction. We benchmark LFNs on single-shot reconstruction on the Shapenet "cars" and "chairs" classes as proposed in SRNs [3]. See Figure 5 for qualitative and quantitative results. We report performance better than SRNs in PSRN and on par terms of SSIM
8

---

Input
SRNs
Ours
GT
Input
SRNs
Ours
GT
Chairs
Cars
SRNs [3] LFN
22.89 /.89
22.25 0.89
22.26 0.90
22.42/ 0.89
Figure Class-specific single-shot reconstruction. LFN performs approximately on par with SRNs [3] in the single-class single-shot reconstruction case, while requiring an order of magnitude fewer network evaluations, memory, and rendering time, Quantitative results report (PSNR, SSIM).
Inpur
Ours pixeINeRF GT
Inpur Ours NixeINeRF GT
Class-specific
Multi-Class
LFN
22.34/ 0.90
24.95 0.87
pixelNeRF 23.45 0.91
26.80 0.91
Figure 6: Global vs. local conditioning. The locally conditioned pixelNeRF outperforms LFNS in single-shot reconstruction, though LFNs require three orders magnitude less rendering time and memory. Qualitatively, for many objects LFNs are on par with pixeINERF (left), but confuse the object class on others (right). Quantitative results report (PSNR, SSIM). "Class-specific" refers to average over cars and chairs as in Fig. while 'multi-class' refers to the average over all ShapeNet classes in Fig.
on the "cars" class, and worse in PSNR but better in terms of SSIM on the "chairs" class, while requiring order of magnitude fewer network evaluations and rendering in real-time. We attribute the drop in performance compared to multi-class reconstruction to the smaller dataset size, causing multi-view inconsistency.
Global vs. local conditioning and comparison to pixeINeRF [6]. We investigate the role of global conditioning, where a single latent is inferred to describe the whole scene [3], to local condi- tioning, where latents inferred per-pixel in a 2D image and leveraged to locally condition a neural implicit representation [26, 27, 6). We benchmark with the recently proposed pixelNeRF [6]. As noted above (see Section 4.3), local conditioning does not infer compact neural scene representation of the scene. Nevertheless, we provide the comparison here for completeness. See Figure 6 for qualitative and quantitative results. On average, LFNs perform IdB worse than pixelNeRF in the single-class case, and 2dB worse in the multi-class setting.
Real-time rendering and storage cost. See Table 2 for quantitative comparison of the rendering complexity of LFN compared with that of volumetric and ray-marching based neural renderers [3, 45, 19, 4, 6]. All clock times were collected for rendering 256 x 256 images on an NVIDIA RTX 6000 GPU. We further compare the cost of storing a single LFN with the cost of storing a conventional light field. With approximately 400k parameters, single LFN requires around 1.6 MB of storage, compared to 146 MB required for storing 360-degree light field a resolution of 256 x 256 x x7 in the six-plane Lumigraph configuration.
Multi-view consistency function of training set size. We investigate how multi-vien view con- sistency scales with the amount of data that the prior is trained on. Please find this analysis in the supplementary material.
Table 2: Comparison of rendering complexity. LFNs require three orders of magnitude less compute than volumetric rendering based approaches, and admit real-time rendering. Please see Table of supplemental material for comparison on larger images based on data in
LFNs
SRNs [3]
PixeINeRF [6]
# evaluations per ray
1
11
192
clock time for 256 256 image (ms)
2.1
120
30e3

---

Method
Cars
Chairs
SRNs
0.071
0.20
LFN
0.059
0.130
Figure 7: Geometry reconstruction with LFNs. By backprojecting extracted depth into 3D, LFNs enable reconstruction of 3D pointclouds (left). For rays were depth valid, LFNs achieve lower mean LI depth error than Scene Representation Networks (right).
Overfitting of single 3D scenes. We investigate overfitting single 3D scene with a Light Field Network with positional encodings sinusoidal activations [24, 61]. Please find this analysis in the supplementary material.
Evaluation of Reconstructed Geometry. We investigate the quality of the geometry that can be computed from LFN via Eq. 8. For every sample in the class-speciffs single-shot reconstruction experiment, we extract its per-view sparse depth map. We then backproject depth maps from four views into 3D reconstruct point cloud, and benchmark mean LI error valid depth estimates with Scene Representation Networks [3]. Fig. 7 displays qualitative and quantitative results. Qualitatively, point clouds succeed in capturing fine detail such as the armrests of chairs. Quantitatively, LFNs outperform SRNs on both cars and chairs. We note that LFNs have a slight advantage in this comparison,
can only benchmark confidence. This includes occlusion boundaries, which are areas where the sphere-tracing based SRNs incurs high error, it is forced to take smaller and smaller steps and may not reach the surface. We highlight that we do not claim that the proposed method is competetive with methods designed for geometry reconstruction in particular, but that we only report this demonstrate that the proposed method is capable to extract valid depth estimates from an LFN.
the sparse depth values, for which LFNs have high
c(r)
c(r)
Limitations. First, as every existing light field approach, LFNs store only one color per oriented ray, which makes rendering views from cameras placed in between occlud- ing objects challenging, even the information may still be stored the light field. Second, though outperform loballl-conditioned methods, currently do not outperform the locally condi- tioned RiR. Finally, as opposed to 3D-structured representations, LFNs do not enforce strict multi-view consistency, and may be inconsistent in the case of small datasets.
6 Discussion and Conclusion
We have proposed Light Field Networks, a novel neural scene representation that directly parameter- izes the full 360-degree, 4D light field of a 3D scene via neural implicit representation. This enables both real-time neural rendering with a single evaluation of the neural scene representation per ray, as well as sparse depth map extraction without ray rastins.ing. Light Field Networks outperform globally conditioned baselines in single-shot novel view synthesis, while being three orders of magnitude faster and less memory-intensive than current volumetric rendering approaches. Exciting avenues for future work include combining LFNs with local conditioning, which would enable stronger dittrf-distribution generalization, studying the learning of non-Lambertian scenes, and enabling camera placement in obstructed 3D space. With this work, we make important contributions to the emerging fields of neural rendering and neural scene representations, with exciting applications across computer vision, computer graphics, and robotics.
Societal Impacts. Potential improvements extending work on few-observation novel view synthesis could enable abuse by decreasing the cost non-consensual impersonations. We refer the reader recent review of neural rendering [22] for in-depth discussion of this topic.
Acknowledgements and Disclosure of Funding
This work supported by the NSF under Cooperative Agreement PHY-2019786 (The NSF AI Institute for Artificial Intelligence and Fundamental Interactions, attp://iaite http////iiiii.o......//.//. ONR under 1015
10

---

G TTA243/0000144-611200000 (Understanding Scenes and Events through Joint Parsing, Cognitive Reasoning and Lifelong Leamning, Mitsubishi under 026455-0000) (Building World Models from some data through analysis by synthesis), DARPA under CW3031 1624 (Transfer, Augmentation and Automatic Learning with Less Labels), as well as the Singapore DSTA under DSTM00FC120000823 (New Representations for Vision). We thank Andrea Tagliasacchi, Tomasz Malisiewicz, Prafull Sharma, Ludwig Schubert, Kevin Smith, Bernhard Egger, Christian Richardt, Manuel Rey Area, and Jürgen and Susanne Sitzmann for interesting discussions and feedback, and Alex Yu for kindly sharing the outputs of pixelNeRT and baselines with us.
References
[1] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: I.carning continuous signed distance functions for shape representation. In Proc. CVPR, 2019.
[2] Lars Mescheder, Michael Occhsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proc. CVPR, 2019.
[3 Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous
3d structure- aware neural scene representations. In Proc. NeurlPS 2019, 2019.
[4] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik. Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields view synthesis. In Proc. ECCV. 2020.
[5] Michael Niemeyer, Lars Mescheder, Michael Oechsle. and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In Proc. CVPR, 2020.
[6] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one or few images. Proc. CVPR, 2020.
[7) Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Trunsactions Graphics (TOG), 38(4):1-14, 2019.
[8] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew Davison. In-place scene labelling and understanding with implicit scene representation. arXiv preprint arXiv:2 15875, 2021.
[9] A. Kohli, V. Sitzmann, and G. Wetzstein. Semantic Implicit Neural Scene Representations with Semi- supervised Training. In Proc. 3DV, 2020.
[10] Edgar Sucar, Shikun Liu, Joseph Ortiz, and Andrew Davison. imap: Implicit mapping and positioning in real-time. arXiv preprint arXiv:2103. /2352, 2021.
[11] Adam R Kosiorek, Heiko Strathmann, Daniel Zoran, Pol Moreno, Rosalia Schneider, Soña Mokrá,
and Danilo J Rezende. Nerf-vae: A geometry aware 3d scene generative model. arXiv preprint arXiv:2104. 005800077777 2021.
[12] Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov. and Gabriel J Brostow. Interpretable transformations with encoder-decoder networks. In Proc. ICCV, volume 2017.
[13] SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, 363((33))))1166116 1204-1210, 2018.
[14] Vincent Sitzmann, Justus Thies, Felix Heide, Matthias Nießner, Gordon Wetzstein, and Michael Zollhöfer. Deepvoxels: Learning persistent 3d feature embeddings. In Proc. CVPR, 2019.
[15] Hsiao- Fish Tung, Ricson Cheng, and Katerina Fragkiadaki. Learning spatial common sense with geometry-awand recurrent networks. Proc. CVPR, 2019.
[16] Thu H Nguyen- Phuoc, network for differentiable rendering from 3d shapes. In Proc. NIPS. 2018
Nguyen-Phuoc, Chuan Li, Stephen Balaban, and Yongliang Yang. Rendernet: A deep convolutional
[17] Jun Yan Zhu, Zhoutong Zhang, Chengkai Zhang, Jiajun Wu, Antonio Torralba, Josh Tenenbaum, and Bill Freeman. Visual object networks: image generation with disentangled 3d representations. In Proc. NIPS, pages 118-129, 2018
[18] Thu Nguyen-Phuoe, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsuper- vised learning of 3d representations from natural images. In Proc. ICCV, 2019.
[19] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic rendcrable volumes from images. arXiv preprint 190.:19906..771 2019.
[20] Danilo Jimenez Rezende, S. M. Ali Eslami, Shakir Mohamed, Peter Battaglia. Max Jaderberg, and Nicolas Heess. Unsupervised learning of 3d structure from images. In Proc. NIPS. 2016.
11

---

[21] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari. James Tompkin, Vincent Sitzmann, and Srinath Sridhar. Neural fields in visual computing and beyond. In arXiv, 2021.
[22] Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Brualla Tomas Simon, Jason Saragih, Matthias Nießner, al. State of the on neural rendering. In Computer Graphics Forum, volume 39, pages 701-727. Wiley Online Library, 2020.
[23] Vincent Sitzmann, Eric R Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstcin. Mctasdf: Mcta-
learning signed distance functions. Proc. NeurlPS, 2020.
[24] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In Proc. NewrIPS, 2020.
[25] Konstantinos Rematas, Ricardo Martin-Bruall,, and Vittorio Ferrari. Sharf: Shape-conditioned radiance fields from a single view. In Proc. ICMI,, 2021.
[26] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In Proc. ICCV, pages 2304-2314, 2019.
[27] Alex Trevithick and Bo Yang. Grf: Learning general radiance field for 3d scene representation and rendering. arXiv preprint arXiv:2010.05595, 2020.
[28] Christian Reiser. Songyou Peng. Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mips. arXiv preprint urXiv:2103. 131::2403.137444, 2021.
[29] Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. arXiv preprint arXiv:2103. 14024, 2021.
[30] Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. Proc. NeurIPS, 2020.
[31] David B Lindell, Julien NP Martel, and Gordon Wetzstein. Autoint: Automatic integration for fast neural volume rendering. Proc. CVPR, 2020.
[32] Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Chakravarty R Alla Chaitanya, Anton Kaplanyan, and Markus Steinberger. Donerf: Towards real-time rendering of neural radiance fields using depth oracle networks. arXiv e-prints, pages mrXiv-210x 2021.
[33] Petr Kellnhofer, Lars Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, and Gordon Wetzstein. Neural lumigraph rendering. Proc. CVPR, 2021.
[34] Edward H Adelson, James R Bergen, et al. The plenoptic function and the elements of early vision, volume 2. Vision and Modeling Group, Media Laboratory, Massachusetts Institute of Technology, 1991.
[35] Edward H Adelson and John YA Wang. Single lens stereo with a plenoptic camera. IEEE transactions on pattern analysis and machine intelligence, 14(2):99-106, 1992.
[36] Mare Levoy and Pat Hanrahan. Light field rendering. In Proc. SIGGRAPH, 1996.
[37] Steven J Gortler, Radek Grzeszezuk, Richard Szeliski, and Michael Cohen. The lumigraph. Proc. SIGGRAPHI 1996.
[38] Chris Buchler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. Unstructured lumigraph rendering. In Proceedings of the 28th annual conference on Computer graphics and interactive techniques, pages 425 432,, 2001.
[39] Ren Ng et al. Digital light field photography, volume 7. stanford university Stanford, 2006.
[40] Suren Vagharshakyan, Robert Bregovic, and Atanas Gotchev. Light field reconstruction using shearlet transform. Proc. PAMI, 40(1):133-147, 2017.
[41] Lixin Shi, Haitham Hassanich, Ahe Davis, Dina Katabi, and Fredo Durand. Light field reconstruction using sparsity the continuous fourier domain. ACM Transactions on Graphics (TOG), 34(1):1-13, 2014.
[42] Anat Levin and Fredo Durand. Linear view synthesis using dimensionality gap light field prior. In Proc. CVPR, pages 1831-1838. IEEE, 2010.
[43] Nima Khademi Kalantari, Ting-Chun Chun Wang, and Ravi Ramamoorthi. Learning light field cameras. ACM Trans. Graph. (SIGGRAPH Asia). 35(6): 193, 2016.
Learning-based based view synthesis for
[44] Mojtaba Bemana. Karol Myszkowski, Hans-Peter Seidel, and Tobias Ritschel. X-fields: Implicit neural view-, light- and time-image interpolation. Proc. SIGGRAPH Asia 2020, 39(6), 2020.
[45] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. Proc. NeurlPS, 2020.
[46] Qiangeng Xu, Weiyue Wang. Duygu Ceylan. Radomír Mech, and Ulrich Neumann. DISN: deep implicit surface network for high-qualin single-view view 3d reconstruction. In Proc. NIPS, 2019.
12

---

[47] Robert C Bolles, H Harlyn Baker, and David H Marimont. Epipolar-plane image analysis: An approach to determining structure from motion. Proc. IJCV, 1(1):7-55, 1987.
[48] Don Dansereau and Len Bruton. Gradient-based depth estimation from 4d light fields. In Proc. IEEE International Symposium on Circuits and Systems, volume 3, pages 549, IEEE, 2004,
[49] Ivana Tosic and Kathrin Berkner. Light field scale-depth space transform for dense depth estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 435-442, 2014.
[50] Changil Kim, Henning Zimmer, Yacl Pritch, Alexander Dorkine-Hornung and Markus H Gross. Scene reconstruction from high spatio-angular resolution light fields. ACM Transactions on Graphics (TOG), 32(4):73-1, 2013.
[51] Lipeng Si and Qing Wang. Dense depth-map estimation and geometry inference from light fields via global optimization. Asian Conference on Computer Vision, pages 83-98. Springer, 2016.
[52] Yan-Bin Jia. Plücker coordinates for lines in the space. Problem Solver Techniques
for Applied Computer Science, Com-S-477/577 Course Handout. Iowa State University. https://aaulyy..iiess.aaaa www.aittes.ittttteeeeeeeeeeeeeee http://ffeeiiileeeeeeeeeeaeerooe
2020.
[53] Bernd Krolla, Maximilian Diehold, Bastian Goldlücke, and Didier Stricker. Spherical light fields. In Proc.
BMVC, 2014.
[54] Insung Ihm, Sanghoon Park, and Rae Kyoung Lee. Rendering of spherical light fields. In Proceedings The Fifth Pucific Conference on Computer Gruphics and Applications, pages 59-68. 1997.
[55] Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, 2nd edition, 2003.
[56] David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. In Proc. ICLR, 2017.
[57]
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML' 17, page 1126-1135. JMLR.org. 2017.
[58] Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. Film: Visual reasoning with general conditioning layer. In AAAI, 2018.
[59] Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat. Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: 1512.03012 2015.
[60] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In Proc. CVPR, pages 3907-3916, 2018
[61] Matthew Tancik, Pratul P Srinivasan, Ben Mildenhall, Sara Fridovich-Kann Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng. Fourier features let networks learn high frequency functions low dimensional domains. Proc. NeurlPS, 2020.
13


Reviews:
[{"review_id": "wfpn8vxvw0f", "reviewer": "Reviewer_9CQx", "summary": "The paper presents a new neural scene representation based on the idea of light fields. Rather than predicting properties (e.g. occupancies, colors) for points in space, the paper proposes to predict such entities for all rays in a scene using simple MLP networks.\nFor the network input the authors propose to use Plucker coordinates to canonically parametrize the viewing rays independently of a point offset.\n\nTo encourage multi-view consistency of the network predictions, the authors propose a meta-learning approach allows to decouple the rendering from the latent code optimization.\n\nIn contrast to volumetric neural scene representations that require expensive sampling with multiple network predictions per ray for rendering a novel view, the proposed method only requires a single network evaluation per ray while achieving state-of-the-art rendering results.\n\n", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Paper Strengths:\n\n#Originality\nThe paper presents several novel ideas for new ways of encoding neural scene representations and demonstrates their viability.\nNeural scene representations are an important and vibrant research direction and this paper contributes nicely by introducing a new model which tackles common scalability issues although it also introduces new limitations.\n\n#Quality\nThe paper is well structured and written. Illustrative figures support the explanations in the text well. The paper also presents competitive state-of-the-art results.\n\n#Clarity\nThe paper is clearly written and the mathematical model description is sound (apart from a few small mistakes - see comments below).\n\n#Significance\nI believe the paper contains several valuable ideas and thoughts that should be shared with the community and hence merit publication.\n\n\n\n\nPaper weaknesses / questions:\n\n1. The experimental evaluation could be stronger in my opinion.\n\n1.1. Ablations: One can certainly assume that the method will not work without the proposed meta-learning network architecture, but it is not explicitly stated or empirically shown that/why a simpler network architecture would not work.\nMoreover, it is unclear how changes of the latent code size or the network size would affect the output quality, as well as overfitting vs. generalization properties.\n\n1.2. Stronger evaluations: One of the major paper claims is that LFNs are able to encode both the geometry and the appearance of a scene (L66). The evaluation of these two entities is not very strong.\n\n1.2.1 For the appearance, all scenes contain only simple piece-wise constant textures with barely any challenging high-frequency details. Since, the network sometimes already struggles to recover these simple scene sharply, one can assume that more complex textures cannot be well recovered.\nThus, the paper does not really show that LFNs are able to encode real-world textures. It would still be nice to see some results on real images.\nFurther, it would be interesting to know/discuss whether the appearance modeling quality could be improved with an increased network capacity / larger latent codes.\n\n1.2.2 For the geometry, there is no quantitative evaluation at all.\nAlthough the depth values are only sparsely recovered, it is still possible to compute error values for corresponding masked depth maps. Otherwise the quality of computed depth maps (Eq. (8)) is difficult to assess.\nBesides showing depth maps, one could also (sparsely) evaluate the quality of surface normals to better assess the geometric reconstruction quality.\n\n\n\n\nMinor comments:\n\n- several parameters have not been specified in the paper:\n  what is the latent vector size k (Eq. (9))?\n  what is the LFN parameter size \\ell (Eq. (9))?\n  what is the value of \\lambda_lat in Eqs. (10), (11) ?\n  \nAlthough some parameters are described in the supp. mat., it does not need much space to state their values directly in the paper to make it better self-contained and give the reader a feeling for the network and optimization parameters.\n\\lambda_lat is only called \\lambda in the supp. mat.\n\n\n- Fig. 2 is very helpful. Although, the figure illustrates several complex concepts which are detailed in the text below one could improve the caption to make the figure more self-contained and briefly explain the symbols as many of them are not explained in the caption.\n\n- L194: it is better to use a different symbol than \\ell to denote a ray since the same symbol is used to denote the number of LFN parameters in Eq. (6) / L154.\n\n- L202.5: This should probably also be a numbered equation. Even if you do not reference it, other might want to.\nThere is a slight error in there: The right hand side of the “element of” operator describes the space of a tuple, while the left hand side is not a tuple, but a set of tuples.\n\n- L219: “which sends the” -> “which maps the” ?\n\n- Eq. (10): z_j -> z_i ? (z_j does not seem to make sense here)\n\n- Table 1: best numbers are incorrectly highlighted in 2nd and 3rd last columns\n\n\n--------------------------------------------------------------------------------------------------------------\n#Post-Rebuttal:\nI agree with the other reviewers that the authors did an excellent job in responding to the reviewer concerns.\nThanks a lot for this great effort!\nOverall, I am very happy with the responses not only to my questions, but also to the ones of the other reviewers.\nTherefore, I will keep my positive rating to accept the paper as I believe it has a lot of valuable insights that should be shared with the community and which merit publication. Thanks for the great work!\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations and societal impact have been sufficiently discussed.\n\n", "review_text": "", "extra_scores": {"ethical_concerns": "There are no ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "pFePOKpWpjS", "reviewer": "Reviewer_8NSa", "summary": "The paper suggests a new representation of the 4D light field using a neural scene representation that predicts the light field from the 6D Plucker coordinates. The new scene representation enables the pixel radiance estimation using a single network query, whereas previous works needed to follow ray-marching or volume rendering procedures that require multiple queries along the ray. The presented scene representation is then utilized to learn multiple shapes or scenes using hypernetworks, in order to learn a prior of multiview consistency. Novel view synthesis results are presented for the learned scenes from the dataset, as well as generalizations from only single view supervision. Moreover, the authors demonstrate how sparse depth maps can be extracted from the learned light field in the case of Lambertian scenes.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Those are the main strengths and novelty I find in the paper:\n1. The paper is well written and has a good flaw. Except for Figure 3, which I find to be a bit confusing and crowded, all the figures as well the supplied video tells the story nicely and helps to understand the method concepts.\n2. The idea of representing the light field using a unique coordinate system for rays is novel and interesting, but most importantly, it allows fast rendering which is currently the significant bottleneck of the existing rendering techniques. This parametrization is well motivated by the authors for 360 degrees scenes, where it is natural to assume that the radiance is constant along the ray.\n3. The authors exploited the epipolar plane images properties in Lambertian surfaces and developed an equation to extract sparse depth maps from the learned light field using a single evaluation of the network and its gradient. I believe it presents a novel approach that can be leveraged in future works with neural scene representations. Nonetheless, the depth maps results are impressive given that the light field is learned without inherent representation of the geometry.\n\nFollowing are the weaknesses I find in the presented method and the concerns which I would like to be addressed by the authors:\n1. Although real-time rendering is a well-desired property for the task of view-synthesis, so as inherent multiview consistency, that the authors agreed that their representation does not possess. The lack of explicit 3D geometry representation makes the suggested representation not multiview consistent by design, unlike previous works, which I find to be the major limitation of the method. \n2. As the authors mentioned, their representation encodes both geometry and appearance together. Hence I suspect it can mismatch between the two and compensate over geometry properties using the learned appearance. I want to emphasize that this comment differs from 1, where both limitations come from the fact the suggested representation does not model explicit geometry properties.\n3. The authors suggest that a future direction for their work would be working on real data. With the above (1,2) said, it is not clear how those limitations can be addressed in the real data settings, where in those cases the need for multiview consistency is more crucial, and need to store more than one color per ray. Moreover, I'm interested to know how the representation performs on a single scene multiview reconstruction.\n4. The comparison between global and local conditioning in the context of shape space learning is important and the authors describe the strengths and limitations of each option properly. However, in the context of the suggested scene representation (which is the paper's main contribution), I find the comparison to PixelNeRF irrelevant. A more proper baseline would be PixelNerF with global conditioning (meaning, auto-decoder as in the method of the paper with rendering function of NeRF). The source for that PixelNeRF overcomes LFN is not clear- is it due to the scene representation or due to the conditioning learning method? I believe there is a need to separate between those two observations to strengthen the paper's contribution. Another point of difference is that PixelNeRF utilizes positional encoding, which enables to learn higher frequencies in the light field, and I wonder why the authors did not use that. \n5. In several results, it seems that thin areas are model incorrectly (smoothed or mixed), for example - figure 5 bottom chair, or the tables and benches legs in the single-shot results (10:45-55 in the supplied video). I find possible reasons for that: the LFN fails to learn the high discontinuity (frequency) of the light field where a small change in the ray change from the table leg to the background; the multiview consistency is not modeled correctly in this area; due to the low-resolution data; compensation of the learned appearance over geometry; wrong generalization of tests views. I would appreciate it if the authors could address this concern, and I suggest they show depth evaluation for the more complicated geometry areas.\n6. The addition of extracting the sparse depth maps is another good contribution of the methods, however, it needs to be addressed by the authors that it is limited to Lambertian surfaces and that a generalization to secular scenes is unclear.\n\n\n_________________________\nAfter reading the other reviews and the author's response, I decided to update my rating to accept.\n\nHowever, there are few points I suggest the authors clarify in the revised paper:\n* In section 1.2 in the authors' response, I believe that the comparison to SDF can be misleading. Compares to LFNs, SDF is not limited to Lambertian surfaces for representing scene geometry, and the geometry representation is not partial (as the sparse depth maps extracted from LFNs).\n* In section 1.4 in the authors' response, running methods like SRN and DVR with a fixed compute budget will probably yield worsened results. However, they will still be multi-view consistent, meaning that the learned scene from a novel view will correspond to the learned scene geometry.\n\nAlso, I believe incorporating all the new results presented in the rebuttal (also the new Fern results in 2.5) would serve this paper well.\n\nOverall, I'm convinced by the novelty and contribution of this new representation, and I'm intrigued by the future works it opens up.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors did address the main limitations of the method and its potential social impacts. I believe the authors should be more apparent about extracting the (partial) depth maps only for Lambertian surfaces and that a generalization for non-Lambertian surfaces (e.g., with specular effects) is not trivial and non-obvious.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "ck72_SypD8Q", "reviewer": "Reviewer_NEE3", "summary": "Light Field Networks encode the light field of a 3D scene, which comes with certain restrictions on where an observer can be placed. LFNs are coordinate-based MLPs that use the Plucker parameterization of directed rays in 3D space to represent rays. Thus they can output the final color of a pixel with a single network evaluation, massively speeding up rendering relative to NeRF but sacrificing \"hard\" multi-view consistency, which now needs to be learned. LFNs here are trained via a hypernetwork and scene conditioning happens via an auto-decoded latent code for the hypernetwork. While light fields only represent appearance directly, it is possible to use gradients to extract depth along appearance edges. Results are presented on ShapeNet and very simple synthetic rooms.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- LFNs are novel and an interesting take on neural radiance fields that trades multi-view-consistency-by-design for orders of magnitude higher rendering speed.\n- Results in the supplemental video are random and not cherry-picked.\n- Qualitative results seem overall similar to DVR to me. But since LFNs are much faster, they offer an advantage over DVR in terms of results.\n- The paper is well written.\n\n- I would have liked a geometrical intuition as to why Plucker coordinates are independent of the specific point on the ray. The origin and the ray (i.e. all points p on the ray) define one single plane and p x d is the normal vector of that plane. Rays through the origin are a corner case in that intuition, but they are directly identified by their direction.\n\n- Interestingly, when comparing single-class vs. multi-class, chairs have better results overall on single-class while cars are better in multi-class. Looking at more classes in single-class experiments would give some indication on whether the claim in line 280 (smaller dataset is the issue) is correct, which would mean that chairs is an outlier class.\n- How many objects per class are there roughly in the single-class and multi-class settings? Hundreds?\n- It would also be interesting to see how multi-view consistency (e.g. measured naively via PSNR/SSIM of novel views on a random but fixed set of e.g. 10 shapes per class) improves with an increasing training set size (e.g. 1, 10, 100, 1k, all per class), both in the single-class and multi-class setting. That would give a better idea on how much the hypernetwork learns multi-view consistency as an abstract property, which is the main motivation for using a hypernetwork (lines 10-12).\n- In general, more experiments than just straightforward reconstruction results would add insights. E.g. an ablation of the hypernetwork setting vs. a simpler latent-code conditioning setting with a single, larger LFN. Or a quantitative evaluation of the sparse depth maps, even if only at the sparse points where depth is extractable. Especially the latter should be added since all of page 5 and Sec. 1 of the supplement discuss depth extraction. A comparison to DVR and SRNs, if possible, would offer a baseline. Or backproject multiple depth renderings of a single object and show the merged point cloud in the video, for example. Some experiment that gives a better idea of how well geometry extraction works.\n- Since light fields mostly support inside-out scenes similar to the room scenes and outside-in scenes like the ShapeNet objects, and since both settings are spherical, a simple two-sphere parametrization (e.g. a view-conditionined NeRF that is only evaluated once on a single 3D sphere) would be enough for the results presented in the paper. Better scenes that highlight the strengths of the particular parametrization would have been nice, namely unbounded scenes, e.g. some very long corridor (lines 141-142).\n\nOverall:\n\nLNFs take a valuable route towards better neural graphics representations that is complementary to the slow and hard-constrained setting of NeRF. I hope that future work will be able to re-introduce the multi-view consistency by design instead of learning it as a soft constraint. The main issue I have with the paper in its current form is the experimental investigation. I believe that depth evaluation is necessary, that an experiment investigating multi-view consistency vs. training set size is at least strongly recommendable, and that a hypernetwork vs. simple conditioning experiment and evaluating scenes benefiting from the Plucker parametrization would strengthen the paper noticeably but aren't required for acceptance.\n\nMajor clarity:\n\n- What are class-specific and multi-class in the table in Fig. 6 referring to? Means across all classes? Or what class was evaluated for the class-specific setting?\n- Related to that, means across all classes (weighted equally or by class size, should be mentioned) could be added to Table 1.\n- line 299: \"more difficult\" sounds too inaccurate to me. It is only even possible in a few carefully designed cases (e.g. line 68 in the supplement) and otherwise simply impossible (e.g. placing an arbitrarily rotating observer inside the convex hull of an outside-in scene like chairs from ShapeNet).\n- For single-class single-shot car reconstruction, the video states that LFNs offer slightly more detail, but I am unable to confirm that. To my eye, LFNs and SRNs are qualitatively on par and quantitatively LFNs are only slightly better. That is okay considering the speed difference.\n- How long does auto-decoding in the single-view setting take per scene at test time? Lines 86 and 97 in the supplement state \"until convergence\". How long does that take in practice?\n\nMinor clarity:\n\n- \"Moreoever\" in line 123\n- I don't understand why line 212 is emphasized. Wouldn't line 213 be more appropriate?\n- Eq. 10: there's an unnecessary closing paranthesis (same in Eq. 11) and it should be z_i in the second term. It would also be easier to parse if the arg min is over \\{z_i\\} instead of z_i.\n- What does \"single-shot\" in Table 1 refer to? Auto-decoding with a single view?\n- Table 1 boat SSIM is incorrectly bolded.\n- Fig. 5 Cars SSIM: LFN and SRNs are on par w.r.t. the two reported digits. That's wrongly bolded and incorrectly stated in line 277.\n\n----------\n\nThe authors have provided a very thorough rebuttal. They addressed my concerns very well. I read the other reviewers' concerns and I am satisfied by the authors' response to those. Given the improvements from the rebuttal, I have increased my score from a 5 to an 8.\n\nThis work opens up a number of interesting directions, I especially liked the remark at the end of Sec. 1.4 of the rebuttal: \"We argue that LFNs offer an intriguing orthogonal approach to neural rendering that may in future work inform or even be combined with 3D-structured neural representations.\"\n\nThe only concern that remains for me with the work is Sec. 2.5 in the rebuttal regarding real-world scenes, which I did not find as convincing as the other points. That LFNs struggle in the overfitting regime, which general real-world scenes fall into, is an unfortunate downside that future work hopefully addresses.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are addressed well and potential negative societal impact is shortly mentioned.", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "R9f0UVSGVrs", "reviewer": "Reviewer_Dh8e", "summary": "The authors propose a novel neural scene representation called Light Field Networks (LFNs), where geometry and appearance of the considered scene are represented in a 360-degree, 4D light field that is parameterized via a neural implicit representation. In contrast to other ray-marching-based or volumetric-rendering-based techniques that rely on hundreds of evaluations per ray in similar tasks, the proposed LFN only requires a single evaluation per ray, thereby significantly improving efficiency and enabling real-time rendering at low memory requirements. Key aspects to achieve this are the parametrization of the space of light rays based on Plücker coordinates. In addition, the authors embed LFNs in a meta-learning framework to allow novel view synthesis from solely sparse 2D image supervision. \n \nThe overall approach seems novel and interesting. While the complexity benefits have been demonstrated by the authors, the current approach seems to be limited to simple scenes.\n\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality\nThe approach seems novel and reasonable. In particular, the complexity benefits have also been demonstrated by the authors. \n\n\nEvaluation\n- In the paragraph on 'real-time rendering and storage cost' the statement might be misleading. Only references 3 and 6 have been included in Table 2, which might confuse less experienced readers. In this comparison, the authors provide an evaluation of the computational complexity w.r.t. to SRNs and pixelNeRF, where the proposed approach shows clear benefits. However, insights on the timings for different image resolutions have not been provided.\n- The authors provide quantitative and qualitative results for single-shot multi-class reconstruction and class-specific single-shot reconstruction. However, the comparison only includes DVR and SRNs which seem to not be state-of-the-art approaches any more.\nThe qualitative comparison to further scene-overfitting approaches (i,e. NeRF-like approaches) beyond Figure 6 would be interesting as well. \n- For the evaluation of global vs. local conditioning, the authors provide only a comparison to the local conditioning by pixelNeRF, that results in better quality.\n- Another interesting aspect for the evaluation would be the discussion regarding which configurations of cameras can be handled, i.e. how many views are required and how the views may be distributed for a reliable scene representation.\n- Limitations have been discussed.\n\n\nExposition\n- The paper is well-structured and easy to follow. Figures/tables and captions are informative.\n- The approach is well-motivated.\n- There are a few typos that can be solved in a proof-reading. In equation 4 and the nearby text, the ray seems to be denoted inconsistently by r and l.\n- In Section 5, as I understood, the reference in the paragraph 'class-specific single-view reconstruction' should read Figure 5 (instead of Figure 4).\n- There is an unfinished sentence at the end of the caption of Figure 5 on page 8 and at the end of the caption of Figure 6 on page 9.\n\n\nReproducibility\n- The paper seems reproducible from the facts in the paper. In addition, the authors mentioned to release code upon acceptance.\n\nPost-Rebuttal:\nI thank the authors for provding a comprehensive feedback to the reviewer's comments and agree with the and agree with the other reviewers that this significantly improves the paper. I like the presented contributions and their potential for future developments and am looking for the inclusion of the insights provided in the rebuttal into the paper and supplemental. This also makes me increase my rating towards accept.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and societal impacts have been adequately addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4.5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission was thoroughly reviewed and discussed. All four reviewers support acceptance. The work was found to be stimulating and can usefully inform follow-up efforts in this area. The AC supports the reviewers' recommendation. The authors are encouraged to thoroughly address the reviewers' concerns and recommendations in the revision.
Decision: Accept (Spotlight)