Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper presents a new neural scene representation based on the idea of light fields. Rather than predicting properties (e.g. occupancies, colors) for points in space, the paper proposes to predict such entities for all rays in a scene using simple MLP networks.
For the network input the authors propose to use Plucker coordinates to canonically parametrize the viewing rays independently of a point offset.

To encourage multi-view consistency of the network predictions, the authors propose a meta-learning approach allows to decouple the rendering from the latent code optimization.

In contrast to volumetric neural scene representations that require expensive sampling with multiple network predictions per ray for rendering a novel view, the proposed method only requires a single network evaluation per ray while achieving state-of-the-art rendering results.


main_review: Paper Strengths:

#Originality
The paper presents several novel ideas for new ways of encoding neural scene representations and demonstrates their viability.
Neural scene representations are an important and vibrant research direction and this paper contributes nicely by introducing a new model which tackles common scalability issues although it also introduces new limitations.

#Quality
The paper is well structured and written. Illustrative figures support the explanations in the text well. The paper also presents competitive state-of-the-art results.

#Clarity
The paper is clearly written and the mathematical model description is sound (apart from a few small mistakes - see comments below).

#Significance
I believe the paper contains several valuable ideas and thoughts that should be shared with the community and hence merit publication.




Paper weaknesses / questions:

1. The experimental evaluation could be stronger in my opinion.

1.1. Ablations: One can certainly assume that the method will not work without the proposed meta-learning network architecture, but it is not explicitly stated or empirically shown that/why a simpler network architecture would not work.
Moreover, it is unclear how changes of the latent code size or the network size would affect the output quality, as well as overfitting vs. generalization properties.

1.2. Stronger evaluations: One of the major paper claims is that LFNs are able to encode both the geometry and the appearance of a scene (L66). The evaluation of these two entities is not very strong.

1.2.1 For the appearance, all scenes contain only simple piece-wise constant textures with barely any challenging high-frequency details. Since, the network sometimes already struggles to recover these simple scene sharply, one can assume that more complex textures cannot be well recovered.
Thus, the paper does not really show that LFNs are able to encode real-world textures. It would still be nice to see some results on real images.
Further, it would be interesting to know/discuss whether the appearance modeling quality could be improved with an increased network capacity / larger latent codes.

1.2.2 For the geometry, there is no quantitative evaluation at all.
Although the depth values are only sparsely recovered, it is still possible to compute error values for corresponding masked depth maps. Otherwise the quality of computed depth maps (Eq. (8)) is difficult to assess.
Besides showing depth maps, one could also (sparsely) evaluate the quality of surface normals to better assess the geometric reconstruction quality.




Minor comments:

- several parameters have not been specified in the paper:
  what is the latent vector size k (Eq. (9))?
  what is the LFN parameter size \ell (Eq. (9))?
  what is the value of \lambda_lat in Eqs. (10), (11) ?
  
Although some parameters are described in the supp. mat., it does not need much space to state their values directly in the paper to make it better self-contained and give the reader a feeling for the network and optimization parameters.
\lambda_lat is only called \lambda in the supp. mat.


- Fig. 2 is very helpful. Although, the figure illustrates several complex concepts which are detailed in the text below one could improve the caption to make the figure more self-contained and briefly explain the symbols as many of them are not explained in the caption.

- L194: it is better to use a different symbol than \ell to denote a ray since the same symbol is used to denote the number of LFN parameters in Eq. (6) / L154.

- L202.5: This should probably also be a numbered equation. Even if you do not reference it, other might want to.
There is a slight error in there: The right hand side of the “element of” operator describes the space of a tuple, while the left hand side is not a tuple, but a set of tuples.

- L219: “which sends the” -> “which maps the” ?

- Eq. (10): z_j -> z_i ? (z_j does not seem to make sense here)

- Table 1: best numbers are incorrectly highlighted in 2nd and 3rd last columns


--------------------------------------------------------------------------------------------------------------
#Post-Rebuttal:
I agree with the other reviewers that the authors did an excellent job in responding to the reviewer concerns.
Thanks a lot for this great effort!
Overall, I am very happy with the responses not only to my questions, but also to the ones of the other reviewers.
Therefore, I will keep my positive rating to accept the paper as I believe it has a lot of valuable insights that should be shared with the community and which merit publication. Thanks for the great work!

limitations_and_societal_impact: The limitations and societal impact have been sufficiently discussed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a novel neural scene representation called Light Field Networks (LFNs), where geometry and appearance of the considered scene are represented in a 360-degree, 4D light field that is parameterized via a neural implicit representation. In contrast to other ray-marching-based or volumetric-rendering-based techniques that rely on hundreds of evaluations per ray in similar tasks, the proposed LFN only requires a single evaluation per ray, thereby significantly improving efficiency and enabling real-time rendering at low memory requirements. Key aspects to achieve this are the parametrization of the space of light rays based on Plücker coordinates. In addition, the authors embed LFNs in a meta-learning framework to allow novel view synthesis from solely sparse 2D image supervision. 
 
The overall approach seems novel and interesting. While the complexity benefits have been demonstrated by the authors, the current approach seems to be limited to simple scenes.


main_review: Originality
The approach seems novel and reasonable. In particular, the complexity benefits have also been demonstrated by the authors. 


Evaluation
- In the paragraph on 'real-time rendering and storage cost' the statement might be misleading. Only references 3 and 6 have been included in Table 2, which might confuse less experienced readers. In this comparison, the authors provide an evaluation of the computational complexity w.r.t. to SRNs and pixelNeRF, where the proposed approach shows clear benefits. However, insights on the timings for different image resolutions have not been provided.
- The authors provide quantitative and qualitative results for single-shot multi-class reconstruction and class-specific single-shot reconstruction. However, the comparison only includes DVR and SRNs which seem to not be state-of-the-art approaches any more.
The qualitative comparison to further scene-overfitting approaches (i,e. NeRF-like approaches) beyond Figure 6 would be interesting as well. 
- For the evaluation of global vs. local conditioning, the authors provide only a comparison to the local conditioning by pixelNeRF, that results in better quality.
- Another interesting aspect for the evaluation would be the discussion regarding which configurations of cameras can be handled, i.e. how many views are required and how the views may be distributed for a reliable scene representation.
- Limitations have been discussed.


Exposition
- The paper is well-structured and easy to follow. Figures/tables and captions are informative.
- The approach is well-motivated.
- There are a few typos that can be solved in a proof-reading. In equation 4 and the nearby text, the ray seems to be denoted inconsistently by r and l.
- In Section 5, as I understood, the reference in the paragraph 'class-specific single-view reconstruction' should read Figure 5 (instead of Figure 4).
- There is an unfinished sentence at the end of the caption of Figure 5 on page 8 and at the end of the caption of Figure 6 on page 9.


Reproducibility
- The paper seems reproducible from the facts in the paper. In addition, the authors mentioned to release code upon acceptance.

Post-Rebuttal:
I thank the authors for provding a comprehensive feedback to the reviewer's comments and agree with the and agree with the other reviewers that this significantly improves the paper. I like the presented contributions and their potential for future developments and am looking for the inclusion of the insights provided in the rebuttal into the paper and supplemental. This also makes me increase my rating towards accept.

limitations_and_societal_impact: Limitations and societal impacts have been adequately addressed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper suggests a new representation of the 4D light field using a neural scene representation that predicts the light field from the 6D Plucker coordinates. The new scene representation enables the pixel radiance estimation using a single network query, whereas previous works needed to follow ray-marching or volume rendering procedures that require multiple queries along the ray. The presented scene representation is then utilized to learn multiple shapes or scenes using hypernetworks, in order to learn a prior of multiview consistency. Novel view synthesis results are presented for the learned scenes from the dataset, as well as generalizations from only single view supervision. Moreover, the authors demonstrate how sparse depth maps can be extracted from the learned light field in the case of Lambertian scenes.
main_review: Those are the main strengths and novelty I find in the paper:
1. The paper is well written and has a good flaw. Except for Figure 3, which I find to be a bit confusing and crowded, all the figures as well the supplied video tells the story nicely and helps to understand the method concepts.
2. The idea of representing the light field using a unique coordinate system for rays is novel and interesting, but most importantly, it allows fast rendering which is currently the significant bottleneck of the existing rendering techniques. This parametrization is well motivated by the authors for 360 degrees scenes, where it is natural to assume that the radiance is constant along the ray.
3. The authors exploited the epipolar plane images properties in Lambertian surfaces and developed an equation to extract sparse depth maps from the learned light field using a single evaluation of the network and its gradient. I believe it presents a novel approach that can be leveraged in future works with neural scene representations. Nonetheless, the depth maps results are impressive given that the light field is learned without inherent representation of the geometry.

Following are the weaknesses I find in the presented method and the concerns which I would like to be addressed by the authors:
1. Although real-time rendering is a well-desired property for the task of view-synthesis, so as inherent multiview consistency, that the authors agreed that their representation does not possess. The lack of explicit 3D geometry representation makes the suggested representation not multiview consistent by design, unlike previous works, which I find to be the major limitation of the method. 
2. As the authors mentioned, their representation encodes both geometry and appearance together. Hence I suspect it can mismatch between the two and compensate over geometry properties using the learned appearance. I want to emphasize that this comment differs from 1, where both limitations come from the fact the suggested representation does not model explicit geometry properties.
3. The authors suggest that a future direction for their work would be working on real data. With the above (1,2) said, it is not clear how those limitations can be addressed in the real data settings, where in those cases the need for multiview consistency is more crucial, and need to store more than one color per ray. Moreover, I'm interested to know how the representation performs on a single scene multiview reconstruction.
4. The comparison between global and local conditioning in the context of shape space learning is important and the authors describe the strengths and limitations of each option properly. However, in the context of the suggested scene representation (which is the paper's main contribution), I find the comparison to PixelNeRF irrelevant. A more proper baseline would be PixelNerF with global conditioning (meaning, auto-decoder as in the method of the paper with rendering function of NeRF). The source for that PixelNeRF overcomes LFN is not clear- is it due to the scene representation or due to the conditioning learning method? I believe there is a need to separate between those two observations to strengthen the paper's contribution. Another point of difference is that PixelNeRF utilizes positional encoding, which enables to learn higher frequencies in the light field, and I wonder why the authors did not use that. 
5. In several results, it seems that thin areas are model incorrectly (smoothed or mixed), for example - figure 5 bottom chair, or the tables and benches legs in the single-shot results (10:45-55 in the supplied video). I find possible reasons for that: the LFN fails to learn the high discontinuity (frequency) of the light field where a small change in the ray change from the table leg to the background; the multiview consistency is not modeled correctly in this area; due to the low-resolution data; compensation of the learned appearance over geometry; wrong generalization of tests views. I would appreciate it if the authors could address this concern, and I suggest they show depth evaluation for the more complicated geometry areas.
6. The addition of extracting the sparse depth maps is another good contribution of the methods, however, it needs to be addressed by the authors that it is limited to Lambertian surfaces and that a generalization to secular scenes is unclear.


_________________________
After reading the other reviews and the author's response, I decided to update my rating to accept.

However, there are few points I suggest the authors clarify in the revised paper:
* In section 1.2 in the authors' response, I believe that the comparison to SDF can be misleading. Compares to LFNs, SDF is not limited to Lambertian surfaces for representing scene geometry, and the geometry representation is not partial (as the sparse depth maps extracted from LFNs).
* In section 1.4 in the authors' response, running methods like SRN and DVR with a fixed compute budget will probably yield worsened results. However, they will still be multi-view consistent, meaning that the learned scene from a novel view will correspond to the learned scene geometry.

Also, I believe incorporating all the new results presented in the rebuttal (also the new Fern results in 2.5) would serve this paper well.

Overall, I'm convinced by the novelty and contribution of this new representation, and I'm intrigued by the future works it opens up.
limitations_and_societal_impact: The authors did address the main limitations of the method and its potential social impacts. I believe the authors should be more apparent about extracting the (partial) depth maps only for Lambertian surfaces and that a generalization for non-Lambertian surfaces (e.g., with specular effects) is not trivial and non-obvious.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: Light Field Networks encode the light field of a 3D scene, which comes with certain restrictions on where an observer can be placed. LFNs are coordinate-based MLPs that use the Plucker parameterization of directed rays in 3D space to represent rays. Thus they can output the final color of a pixel with a single network evaluation, massively speeding up rendering relative to NeRF but sacrificing "hard" multi-view consistency, which now needs to be learned. LFNs here are trained via a hypernetwork and scene conditioning happens via an auto-decoded latent code for the hypernetwork. While light fields only represent appearance directly, it is possible to use gradients to extract depth along appearance edges. Results are presented on ShapeNet and very simple synthetic rooms.
main_review: - LFNs are novel and an interesting take on neural radiance fields that trades multi-view-consistency-by-design for orders of magnitude higher rendering speed.
- Results in the supplemental video are random and not cherry-picked.
- Qualitative results seem overall similar to DVR to me. But since LFNs are much faster, they offer an advantage over DVR in terms of results.
- The paper is well written.

- I would have liked a geometrical intuition as to why Plucker coordinates are independent of the specific point on the ray. The origin and the ray (i.e. all points p on the ray) define one single plane and p x d is the normal vector of that plane. Rays through the origin are a corner case in that intuition, but they are directly identified by their direction.

- Interestingly, when comparing single-class vs. multi-class, chairs have better results overall on single-class while cars are better in multi-class. Looking at more classes in single-class experiments would give some indication on whether the claim in line 280 (smaller dataset is the issue) is correct, which would mean that chairs is an outlier class.
- How many objects per class are there roughly in the single-class and multi-class settings? Hundreds?
- It would also be interesting to see how multi-view consistency (e.g. measured naively via PSNR/SSIM of novel views on a random but fixed set of e.g. 10 shapes per class) improves with an increasing training set size (e.g. 1, 10, 100, 1k, all per class), both in the single-class and multi-class setting. That would give a better idea on how much the hypernetwork learns multi-view consistency as an abstract property, which is the main motivation for using a hypernetwork (lines 10-12).
- In general, more experiments than just straightforward reconstruction results would add insights. E.g. an ablation of the hypernetwork setting vs. a simpler latent-code conditioning setting with a single, larger LFN. Or a quantitative evaluation of the sparse depth maps, even if only at the sparse points where depth is extractable. Especially the latter should be added since all of page 5 and Sec. 1 of the supplement discuss depth extraction. A comparison to DVR and SRNs, if possible, would offer a baseline. Or backproject multiple depth renderings of a single object and show the merged point cloud in the video, for example. Some experiment that gives a better idea of how well geometry extraction works.
- Since light fields mostly support inside-out scenes similar to the room scenes and outside-in scenes like the ShapeNet objects, and since both settings are spherical, a simple two-sphere parametrization (e.g. a view-conditionined NeRF that is only evaluated once on a single 3D sphere) would be enough for the results presented in the paper. Better scenes that highlight the strengths of the particular parametrization would have been nice, namely unbounded scenes, e.g. some very long corridor (lines 141-142).

Overall:

LNFs take a valuable route towards better neural graphics representations that is complementary to the slow and hard-constrained setting of NeRF. I hope that future work will be able to re-introduce the multi-view consistency by design instead of learning it as a soft constraint. The main issue I have with the paper in its current form is the experimental investigation. I believe that depth evaluation is necessary, that an experiment investigating multi-view consistency vs. training set size is at least strongly recommendable, and that a hypernetwork vs. simple conditioning experiment and evaluating scenes benefiting from the Plucker parametrization would strengthen the paper noticeably but aren't required for acceptance.

Major clarity:

- What are class-specific and multi-class in the table in Fig. 6 referring to? Means across all classes? Or what class was evaluated for the class-specific setting?
- Related to that, means across all classes (weighted equally or by class size, should be mentioned) could be added to Table 1.
- line 299: "more difficult" sounds too inaccurate to me. It is only even possible in a few carefully designed cases (e.g. line 68 in the supplement) and otherwise simply impossible (e.g. placing an arbitrarily rotating observer inside the convex hull of an outside-in scene like chairs from ShapeNet).
- For single-class single-shot car reconstruction, the video states that LFNs offer slightly more detail, but I am unable to confirm that. To my eye, LFNs and SRNs are qualitatively on par and quantitatively LFNs are only slightly better. That is okay considering the speed difference.
- How long does auto-decoding in the single-view setting take per scene at test time? Lines 86 and 97 in the supplement state "until convergence". How long does that take in practice?

Minor clarity:

- "Moreoever" in line 123
- I don't understand why line 212 is emphasized. Wouldn't line 213 be more appropriate?
- Eq. 10: there's an unnecessary closing paranthesis (same in Eq. 11) and it should be z_i in the second term. It would also be easier to parse if the arg min is over \{z_i\} instead of z_i.
- What does "single-shot" in Table 1 refer to? Auto-decoding with a single view?
- Table 1 boat SSIM is incorrectly bolded.
- Fig. 5 Cars SSIM: LFN and SRNs are on par w.r.t. the two reported digits. That's wrongly bolded and incorrectly stated in line 277.

----------

The authors have provided a very thorough rebuttal. They addressed my concerns very well. I read the other reviewers' concerns and I am satisfied by the authors' response to those. Given the improvements from the rebuttal, I have increased my score from a 5 to an 8.

This work opens up a number of interesting directions, I especially liked the remark at the end of Sec. 1.4 of the rebuttal: "We argue that LFNs offer an intriguing orthogonal approach to neural rendering that may in future work inform or even be combined with 3D-structured neural representations."

The only concern that remains for me with the work is Sec. 2.5 in the rebuttal regarding real-world scenes, which I did not find as convincing as the other points. That LFNs struggle in the overfitting regime, which general real-world scenes fall into, is an unfortunate downside that future work hopefully addresses.
limitations_and_societal_impact: Limitations are addressed well and potential negative societal impact is shortly mentioned.
review_title: null

Meta-review: The submission was thoroughly reviewed and discussed. All four reviewers support acceptance. The work was found to be stimulating and can usefully inform follow-up efforts in this area. The AC supports the reviewers' recommendation. The authors are encouraged to thoroughly address the reviewers' concerns and recommendations in the revision.
Decision: Accept (Spotlight)