Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: summary: This paper presents an approach to improving the training performance and overall quality of the recently introduced Neural Lumigraph Rendering (NLR) framework for novel view synthesis for objects, using a meta-learning framework that allows for quickly computing a representation of new scenes that can be rendered in real time.

The framework encodes the object surface using a neural signed distance function (SDF) representation of the surface, which can then be efficiently decoded using sphere tracing techniques. The framework encodes features from the 2D input images, which are aggregated from images that are visible in the target and decoded into the reconstructed target view. View-dependent effects are achieved using an MLP that acts as a learned weight-mapping function for the features aggregated from each input view and processed by the decoder. This decoder, in processing these feature maps, can fix artifacts caused by imperfect geometry reconstruction in the surface aggregation function. The Reptile algorithm is used as part of a meta-learning framework to learn a good parameter initialization from multi-view datasets that allows for quickly converging to a representation for previously unseen objects. Evaluations suggest that this approach achieves a good balance between fast convergence on unseen objects, final image quality and fast rendering speed compared to recent alternatives including the original NLR.

Overall my impression is that the main contributions of this work are in the approach described to use more sophisticated shape and appearance representation compared to the original NLR, using image encoders and decoders and a more sophisticated weight-mapping function to achieve superior results while allowing for comparable inference speeds after training. My impression is that the meta-learning framework as described, which is largely based on the Reptile algorithm, is not as substantial a contribution.
main_review: Overall this work presents an approach that seems reasonable and attains reasonably good results. The clarity of exposition is generally quite good, and the supplemental information and video help to illustrate the strengths and limitations of this approach. However, I am not inclined to argue in favor of acceptance due to some key concerns that I have:

- I noticed that for this paper (and the prior Neural Lumigraph Rendering work it extends), all of the depicted results appear to be for synthesizing novel views from a limited range of mostly frontal input images, similar to those from LLFF dataset used by NeRF. However, NeRF also demonstrates reasonable results on full rotations around isolated objects, e.g. synthetically rendered objects for which arbitrary input training views sampled from a sphere or hemisphere around the target are available. Given that this approach appears to be targeted at isolated foreground objects, I'm curious whether this approach would work for a similar range of novel viewpoints, given appropriate training data.

- While the qualitative and quantitative results for this work appear to validate the authors' claims, and the efficient inference performance is an attractive trait of this work, its apparent limitation to segmented foreground objects is a fairly strong limitation compared to the other evaluated approaches that can also be applied to images of complete scenes, e.g. the NeRF results on LLFF image sets.

- The core approach of this work is largely derived from previous work, which limits the overall novelty of the contributions. The extensions to this approach seem reasonable, but many are also relatively straightforward, e.g. using a network pre-trained on large object datasets to allow for faster adaptation to new objects, and using the Reptile algorithm as described in the paper for meta-learning.

This overall approach to meta-learning from other scenes appears to have some similarities to that of pixelNeRF, except that the latter uses a network trained on a large number of objects/scenes to allow for novel view synthesis using as little as 1 new image without any additional training. I am curious whether allowing for further fine-tuning of such a network given a set of multiple new images to obtain comparable results. I understand that this is a significant deviation from their approach, and don't expect a direct comparison to what I describe, but I would be curious to hear any comments from the authors or other reviewers on the relationship between these 2 approaches.

- As noted by the authors, the overly smooth nature of the results and the requirement of object masks to supervise the ray-mask loss are fairly strong limitations compared to other methods that produce high-quality results for complex, unmasked scenes with multiple objects.

- The SDF and sphere-tracing approach seems reasonable for opaque objects, and allows for efficient rendering of such objects compared to the volume rendering approach used by recent works using neural radiance fields, but it appears that its ability to account for complex effects such as translucent objects (e.g. glass) is limited. While this is an understandable tradeoff given the performance benefits, it should be noted as a limitation compared to some of the alternative approaches that are discussed and evaluated. In contrast, recent work like Nex [33] can handle such effects on full shiny/translucent while also maintaining real-time inference performance using an MPI-based approach, making me question the overall efficacy and value of this alternative approach.

Some minor points:

- I think that the notation used for the camera parameters, e.g. { C_i, R_i | t_i }, could be shortened to make the notation in the equations easier to read and follow.
- L231: emphesize -> emphasize

limitations_and_societal_impact: Some broad limitations are discussed, although I think that some of the additional points raised in the main review above should also be addressed.

There are no particular issues w/ potentially negative societal impact that need to be addressed.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents MetaNLR++, a novel method for representing and rendering novel views of object-centric scenes. The scene representation combines a CNN-based feature extractor which extracts deep features from input views and a Neural SDF module which serves as a proxy geometry surface for transforming the extracted features to the novel views. The transformed features are then aggregated and processed by a CNN-based decoder to synthesize the output image. This use of image-based and surface-based scene representation enables faster rendering time compared to volume-based representation such as NeRF. A meta-learning technique based on learned initialization was leveraged to enable fast fitting of the model to new testing scenes. Experiments on two novel-view synthesis benchmarks demonstrate that the proposed method performs favorably compared to existing methods while requiring significantly less time for convergence.
main_review: 
Strength:
 + The paper is well-written and easy to follow. 
 + The proposed method is novel. It effectively addresses two important problems in novel-view synthesis: 1) Constructing the scene representation that enables fast rendering, and 2) Fitting such representation for novel scenes at test time in an efficient manner. The proposed method addresses (1) by combining classical image-based rendering pipeline (using a geometry proxy for transferring visual content from input views to novel views) with neural implicit function model for representing shapes and rendering. To address (2), a meta-learning technique is used which enables fast convergence time for fitting to a test scene.

Weaknesses:
 + There is no serious weakness from my view. However, I feel that more discussions can be added to help the readers better connect this work to existing knowledge in novel-view synthesis and image-based rendering. For example, it can be noted that the proposed method leverages a standard pipeline in image-based rendering that transfer visual information from input views to target views using a proxy shape, and it can be emphasized that the proxy shape in this case is a learned neural implicit function which is the novel contribution of this work. Also, the idea of using CNN-based features instead of color values to represent the information to be transferred from the input views has also been leveraged before for view synthesis. For example, [1] transfers deep features from a single input view to the target view using a learned depth map. I believe the readers can benefit from more discussions on the connection between this work and those existing works.
 + The idea of using meta-learning to address the training-time problem makes sense. The proposed method was shown effective in the experiments using NLR and DUT dataset. The number of scenes in those datasets, however, is relatively small. I'm curious how will different properties of the training data distribution (e.g. number of training scenes, similarity between the training scenes and the testing scene) affect the effectiveness of the meta learning step. I believe synthetic datasets such as ShapeNet can be used to design more controlled experiments to investigate such aspects.

[1] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, Justin Johnson. "SynSin: End-to-End View Synthesis From a Single Image". Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
limitations_and_societal_impact: The limitation and impacts have been discussed clearly in the paper.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper presents a method for capturing the 3D shape and appearance of objects using an implicit model for the shape, and an aggregation of reprojected CNN features for the appearance. The authors highlight two problems with existing approaches that they try to address - slow training and rendering times. The key contribution of the paper is the use of meta-learning, in the form of a learned initialization, to improve the efficiency of the training process. The results show that the proposed method achieves a marginal improvement in training efficiency.
main_review: **TL;DR:** A useful idea to speed up the training and rendering of neural scenes, but the practical improvements are not that impressive. In the main review I have labelled points as a strength (+), weakness (-) or mixed (+/-).

**Originality**: 

(+) The method is new as I am not aware of any existing work that uses a combination of neural level-set surfaces and aggregated CNN features for appearance. There are, however, existing works that have investigated the use of learned initializations for training NeRF-like models but these have not been used in combination with generalizable features from a CNN. 

(+) Although all of the components of the approach (the SDF representation, the CNN features and the learning initialization) are all well-known techniques, the combination is novel and valuable as they all work together to improve the training efficiency. 
The authors have done a good job at differentiating this work from previous contributions (such as MetaSDF and [78]) and, as far as I can tell, have adequately cited all the related work.

**Quality**: 

(+) The submission appears to be technically sound and the details of the method are well documented. One technicality that I am not so sure about is why in Section 3.2, out of the k images that are sampled, only one is used as the ground-truth target image. Surely all of the selected images can act as training targets? 

(+/-) The claims of improved training efficiency and quality are supported by the quantitative evaluation that compares the approach to recent methods like SVS, IBRNet, NLR. However, the improvements in training time and PSNR are not that major (see significance below). 

(+) Overall, the paper is definitely a complete piece of work. The method is fully developed and it has been evaluated and compared to recent approaches on two different datasets. The figures, results and writing are well polished.

(+) The approach has a number of weaknesses, such as the need to use object silhouettes as input (and therefore can only reconstruct single objects and not the whole scene). Another disadvantage is that the reconstructed geometry is rather coarse and not very detailed. However, the authors have done a very good job at explaining and justifying these limitations and providing ways in which they could be addressed in future work.
    
**Clarity**: 

(+) The submission is very clearly written and the overall structure of the paper is well organized. As far as I can tell the paper provides enough information about the method and the experimental setup for a reader to be able to reproduce its results.

**Significance**: 

(-) The results are important in the sense that fast training and rendering of neural scene representations is challenging and an open problem. This paper addresses the training speed in two ways - through the use of generalizable features extracted from a CNN and by using meta-learning, where the main contribution seems to be the use of the meta learning (as per the title). Although it is conceptually a good idea, the real improvements gained by the meta learning seem to not be that significant - for example 27.32dB vs 29.91dB in the best case in Table 2. Is this difference even perceptually significant? Are there ways in which the meta learning could be improved to get a more significant gain in PSNR?

(+/-) There are ways in which others are likely to use or build upon the ideas presented in the paper, but the seemingly minor improvement in the training speed and accuracy might dissuade many in doing so.   

(+) The method does advance the state of the art in terms of training speed (albeit in a very slight manner). In terms of this comparison I would say that IBRNet should be included in Table 1 as there are ways in which it could be be turned into a pre-computed volume representation (for example, by storing the projected features from the nearest images in an Octree).

# Post response and discussion
After considering the authors' response and the other reviews, most of my concerns have been addressed. I agree with rL4a that the core components of the proposed method come from elsewhere and there does not seem to be any special theoretical contributions to these in the proposed paper (most are used as is). However, I think the approach is well-engineered and it seems to achieve good results in terms of training efficiency vs quality which is an important problem for these types of neural reconstruction+rendering approaches.  My final rating is therefore "6: Marginally above the acceptance threshold".


limitations_and_societal_impact: The paper does a good job at describing the potential negative societal impacts.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper finds a set of weight values that is a good initialization to train a recent novel-view synthesis (NVS) method, Neural Lumigraph Rendering (NLR). A meta-learning library, Reptile, is used to optimize this initial weight vector across a corpus, the DTU multiview dataset. The paper shows, that with such an initialization, optimization proceeds much quicker and to similar quality. Additionally, it is suggested to change NLR to aggregate features from the input images instead of pixel values directly, a change that appears to be orthogonal to the meta-learning aspect. Results show that after much fewer optimization steps, a good result is achieved. The method is fast, but that is a property of NLR.
main_review: The topic of meta-learning is original, yet the same idea of finding a good init has now been applied to the novel-view synthesis based on overfit neural networks twice, once by Sitzman et al. and once by Tancik. et al. The difference between all three approaches is mostly that they are applied to different backbone NVS methods, but the use of Reptile itself to tune the init is the same. Hence the topic is novel, but within this new trend, this approach is not adding much new to the two existing papers, besides the insight that what worked for DeepSDF and for NeRF word for NLR, too. This might not be enough.

The quality of the result and experiments is high. All experiments are performed very convincingly and systematic. One could imagine comparing to Tancik et al. They do metalearning for NeRF, this does metalearning to NLR. Both can be compared on a novel-view task after N iterations as done in Fig. 3 with non-metalearned approaches.

The writing is exceptionally clear, modulo that what is described is not always very interesting in the sense that it does not matter so much to the main topic. Most readers will want to learn something interesting about metalearning, but it boils down to run Reptile and a mere init vector optimization. A good deal of the paper is about a change from pixel aggregation to feature aggregation, and also this is not quite interesting and had been done in many papers before. it really only is NLR++, an incremental change to NLR, inspired by many other recent papers.

The problem is significant, as training times are a large issue and quick adaption is a much desired goal.

Overall, reading the paper left this reviewer slightly disappointed. A first brief look is very promising. A second, skim, read identified three relevant takeaways: 1) Reptile applied to two NVS methods before can be applied to NLR, too 2) Aggregating features is better than aggregating pixels, also shown for many NVS works before 3) NLR++ is high-quality, fast and learned quickly. A third, detailed, read did not bring anything substantial to note, resulting in a borderline assessment.
limitations_and_societal_impact: No idea
review_title: null

Meta-review: All four reviewers attest clarity of presentation, importance of the problem and solid engineering. There is a large spread in the judgements of the significance of the work explaining the different scores of 3,5,6,8. Overall the paper has been defended sufficiently well..
Decision: Accept (Poster)