Fast Training of Neural Lumigraph Representations using Meta Learning
Authors: Alexander William Bergman
Petr Kellnhofer
Gordon Wetzstein
Venue: NeurIPS 2021 Poster (2021)
Keywords: neural rendering
meta learning
novel view synthesis

Abstract: Novel view synthesis is a long-standing problem in machine learning and computer vision. Significant progress has recently been made in developing neural scene representations and rendering techniques that synthesize photorealistic images from arbitrary views. These representations, however, are extremely slow to train and often also slow to render. Inspired by neural variants of image-based rendering, we develop a new neural rendering approach with the goal of quickly learning a high-quality representation which can also be rendered in real-time. Our approach, MetaNLR++, accomplishes this by using a unique combination of a neural shape representation and 2D CNN-based image feature extraction, aggregation, and re-projection. To push representation convergence times down to minutes, we leverage meta learning to learn neural shape and image feature priors which accelerate training. The optimized shape and image features can then be extracted using traditional graphics techniques and rendered in real time. We show that MetaNLR++ achieves similar or better novel view synthesis results in a fraction of the time that competing methods require.

Fast Training of Neural Lumigraph Representations using Meta Learning
Alexander W. Bergman
Petr Kellnhofer Stanford University
Gordon Wetzstein
Stanford University
Stanford University
awb0stanford. edu
pkelInho@stanffrdd edu
gordon www.ssteeneetttteoor....... edu
Abstract
Novel view synthesis is a long-standing problem in machine learning and computer vision. Significant progress has recently been made in developing neural scene representations and rendering techniques that synthesize photorealistic images from arbitrary views. These representations. however, are extremely slow to train and often also slow render. Inspired by neural variants of mmage-based rendering, we develop a new neural rendering approach with the goal of quickly learning high-quality representation which also be rendered in real-time. Our approach, MetaNLR++, accomplishes this by using a unique combination of neural shape representation and 2D CNN-based image feature extraction, aggregation, and re-projection. To push representation convergence times down minutes, we leverage meta learning learn neural shape and image feature priors which accelerate training. The optimized shape and image features can then be extracted using traditional graphics techniques and rendered in real time. We show that MetaNLR++ achieves similar or better novel view synthesis results in a fraction of the time that competing methods require.
Introduction
Learning 3D scene representations from partial observations captured by sparse set of 2D images fundamental problem machine learning, computer vision, and computer graphics. Such representation can be used to reason about the scene or to render novel views. Indeed, the latter application has recently received a lot of attention (c.g., [[). For this problem setting, the key questions are: (1) How do we parameterize the scene, and (2) how do infer the parameters from our observations efficiently? With our work, we offer new solutions to answer these questions.
Several classes of scene representation learning approaches have recently been proposed. One popular approach consists of coordinate- based neural networks combined with volume rendering, like NeRF [2]. Although these representations offer photorealistic quality for synthesized images, they are slow to train and render. Coordinate-based networks that implicitly model surfaces combined with sphere tracing-base based rendering are another popular approach [3-6]. One benefit of an implict surface is that, once trained, it can be extracted and rendered real time (5). However, training these representations is equally slow training volumetric representations. Finally, approaches that use a proxy geometry with on-surface feature aggregation are also fast to render [7 8), but the quality and runtime of these methods is limited by the traditional 3D computer vision algorithms that pre-compute the proxy shape, such as structure from motion (SfM) or multiview stereo (MVS).
Here, we develop a new framework for neural scene representation and rendering with the goal of enabling both fast training and rendering times. To optimize the training time of our framework. we do not learn representation network for the view-dependent radiance, as other neural volume
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

or surface methods do, but directly aggregate the features extracted from the source views on the surface of our learned proxy shape. To cut down on pre-prceesiing times required by SfM and MVS, we optimize a coordinate-basee
network representing the proxy shape end-to-end with our CNN-based feature encoder and decoder, and learned aggregation function. key contribution of our work is to combine this unique surface-based neural rendering framework with meta learning, which enables us to learn efficient initializations for all of the trainable parts of our framework and further minimize training time. Because our representation directly parameterizes implicit surface, it can be extracted and rendered in real time. This representation thus incorporates both the fast training benefits from generalizing over shape representations and image features, and the fast rendering capabilities of implicit surface-based methods. We demonstrate training of high-quality neural scene representations in minutes tens of minutes, rather than hours days, which can then be rendered at real-time framerates.
2 Related Work
Image-based rendering (IBR). Classic IBR approaches synthesize novel views of a scene by blending the pixel values of set of 2D images [9118]. Recent IBR techniques leverage neural
networks to learn the required blending weights 19 24
These neural IBR methods either use
proxy geometry, for example obtained by SfM or MVS 25 26 or depth estimation [27]. together
with on-surface feature aggregation 77 8) use learned pixel aggregation functions 28 29] for
geometry-free image-based view synthesis. Our approach is closely related to the geometry-assisted and interpoaatri view synthesis techniques. Many existing approaches, however, require the proxy geometry to be estimated as a pre-processing step, which can take minutes to hours for single scene, preventing fast processing of novel views. Other approaches which use monocular depth estimation to re- project and aggregate features such as SynSin [27]. are applied only to single input images, and cannot easily be scaled to multi-view data due to depth estimation inconsistencies between images of the scene. Instead, our approach estimates a coordinate -aaed neural shape representation from the input images. The shape representation is optimized end-to-end with the image feature extraction, aggregation, and decoding and is accelerated using meta learning.
Neural scene representations and rendering. Emerging neural rendering techniques 1 use explicit, implicit, or hybrid scene representations. Explicit representations, for example those using proxy geometry (see above), object-specific shape templates [00] multi-plane (2 31-4] or multi-
sphere 35 36 images, or volumes [37 38 are fast to evaluate but memory inefficient. Implicit representations, or coordinate-base networks, can be more memory efficient but are typically slow to evaluate 535 55) Hybrid representations make a trade-off between computational and memory efficiency (56 Among these, NeRF [60] and related methods (e.g.. (61H66]) use coordinate-baat network representations and volume rendering, which requires many samples to be processed along a ray, each requiring full forward pass through the network. Although recent work has proposed enhanced data structures [67/69), network factorizations [70], pruning [52], importance sampling [66], fast integration (71), and other strategies to accelerate the rendering speed, training times of all of these methods are extremely slow, on the order of hours or days for a single scene. DVR 3, IDR 4 NLR (5) and UNISURF (6) the other hand, leverage implicitly defined surface representations, which are faster to render than volumes but are equally slow to train. While concurrent work has improved the applicability of these methods by addressing limitations of surface-based methods in general, such as removing the object mask requirement [6] 72 73).
these approaches are still slow to train as they rely on hybrid volumetric and surface formulations to bootstrap the neural surface training. Our approach builds on generalization approaches for neural scene representations to accelerate the training time of 2D-supervised neural surface representations, and thus can be applied alongside advanced training strategies for neural surfaces and make these representations even more applicable for practical view synthesis.
Generalization with neural scene representations. Being able to generalize across neural rep- resentations of different scenes is crucial for learning priors and for 3D aeterraaiveaavessaaiii networks with coordinate-based network backbones (74-78). variety of different generalization approaches have been explored for neural scene representations, including conditioning by con- catenation [39 40) hypernetworks [47], modulation or feature- wise linear transforms [77] 79 [0)
and meta learning [81] 82). Inspired by these works, propose meta-learning strategy that
2

---

Source Images
Encoders F(-)
Neural SDF +
Decoder D()
Synthesized Image
CNN
On-surface
CNN
Feature Aggregation
Meta Learning
Figure 1: Overview of MetaNLR++.
allows us to quickly learn high-quality neural shape representation for given set of multi-view images. As opposed to the 3D point cloud supervision proposed by MetaSDF [81], we meta-learn signed distance functions (SDFs) using 2D images, and opposed to meta-learned initializations for
NeRF volumes [82], we operate with SDFs and features. Our approach is unique in enabling fast, high-quality shape representations to be learned from multi-view images that, once optimized, can be rendered in real time.
Method
3.1 Image formation model
In this section, we outline the NLR++ novel view synthesis image formation model, which is presented in Figure 1 NLR++ takes as inputs a set of images (4:).21 each with corresponding binary foreground object masks {M, and known camera intrinsic parameters and extrinsic parameters collectively referred to as O. At output, NLR+ synthesizes image I, of the scene from the target viewpoint Ct.
Drawing inspiration from classic image-based rendering methods, we define the image formation model learned pixel-wise aggregation (-)-) input image features EE (T.) and their target viewing direction E # on the surface of the object represented by the neural surface & To project the non-occluded, visible input features of Eç(I,) into the target view before aggregation, we define the function Pi- (E,( C.). These neurally aggregated features are then decoded into an image by decoder (-):
i, ( (E((()) V.)).
(1)
The feature encoder E and decoder D are implemented resolution-preservinn convolutional neural network (CNN) architectures
with learned parameters &, th:
Eg R 11 #xxxx
I xxWxd
D& RR:R "xWxd HxWx3
To aggregate the input image features from F. into target feature map to be decoded by D. we use learned feature aggregation (or blending) function Tç. which operates on the surface of our shape representation 4. To define the surface of our shape, we choose to SIREN [51] which represents the signed-distance function (SDF) in 3D space. This encodes the surface of the object as the zero-level set, Lo. of the network:
(p)) {i R (2(x) 0), b : R.
The aggregation is performed on surface for each pixel of the target image f, with camera G.- To find the point in Lo (be) corresponding each pixel ray, we perform sphere tracing on the neural SDF model R(2p, Ct), retaining gradients for the final step of evaluation [4] 53 54]. These individual rendered surface points are projected into the image plane of each of the N input image views and used to sample interpolated features from the source feature maps for each pixel, which can be arranged into N re-sampled feature maps corresponding to each input image [F.), NxH W xd To check whether or not feature is occluded, we use sphere tracing for each pixel from the input views (R( G.) and ensure that the target view surface position projected into each of these surfaces is at the same depth as the source view surface position. Occluded features are then discarded. These three steps of sphere tracing, feature sampling, and occlusion checking make up the function Pi-. (Eg (L+) which outputs each of the re-sampled feature maps {F, N

---

Once the re-samlled feature maps {Fi)e= have been generated, the aggregation function I'ç aggre- gates them into single target feature F; E R #xxxxx xd which can be processed by the decoder into an image. This is done by performing weighted averaging operation on the input features using the relative feature weights L E &NxR..
Fe : R NxHxwxd RHxxx3
- R xxxxd
N
VH Vi) L o
(4)
where o the Hadamard product between the feature and weight maps, broadcasted over the feature dimension d. The weight map L used in the feature aggregation function I is obtained from an MLP % which is applied pixel-wise to each of the N re-sampled feature maps and each pixels target viewing direction:
((F))) ViJ + ((((F)) Vi)) L.
(5)
Here, the dependence upon viewing direction allows for the modeling of view-dependent image properties. These pixel-wise operations making up I' result in a H W x feature map which can be input into the decoder D.
The usage of features instead of pixel values directly allows D some opportunity to inpaint and correct artifacts from imperfect geometry to create photorealistic novel view, unlike methods which render pixels individually 5.. Additionally, the use of the CNN encoder and decoder increases the receptive field of the image loss applied, allowing for more meaningful gradients to be propagated back into 4g
3.2 Supervision and training
Since NLR++ is end-to-end differentiable, we can optimize the parameters &, 1/, 0, end-to--nd to reconstruct target views. For each iteration of training we sample of N images { and designate one of these images to be the ground-trute target image used for supervision I,, and sample its corresponding binary object mask Mj and parameters C..
Using the NLR++ image formation model, generate synthesized target image î, from viewpoint C.. The loss evaluated on the synthesized image consists of three terms:
C((i, Mc), (4, Mc), 2b) CR(4, (4e, Mc)) + A1 (2a, M,) AgCx(d0).
(6)
The image reconstruction loss La is computed as masked L1 loss rendered images:
Cn (t) = Ep Melp)
Σ
[T(p]
[.[]]]]
pi pipp1111
To quickly bootstrap the neural shape learning from the object masks, apply soft mask loss the rendered image masks 51:
LM (Mo Mi) Ep Milp| p M. www.d kp
Σ
BCE(sigmoid(<oD0mmn..)) M. (p)))
(8)
where the notation & miin [p] denotes the minimum value of the SDF 4 along the ray traced from pixel p. is a threshold for whether the zero level set Lo( has been intersected, and a sofuness hyperparameter. Finally, regularize the shape representation to model valid SDF by enforcing the cikonal constraint on randomly sampled points p; E R in unit cube containing the scene:
P
CE Σ 150
(IIIV, - 1)
However, to make our training more efficient, we augment the loss supervision schedule and batching strategy for our model. Specifically, for each sampled batch of k images, instead of computing gradients for a single selected target image, we treat < images as target images, and each of them from the other images in the batch. The number of target images / is limited by the GPU memory

---

MM
Frame Quality (PENR)
25
+Nelf
NLA
1
Mee
15
10
0.0
20
Seconds
Seconds
Figure 2: We demonstrate that all training-times, MetaNLR+ is comparable
outperforms
all competitive representation learning methods, including both neural volumetric and surface repre- sentations in PSNR (left) and LPIPS. [86] (center). We also plot the render time versus converged image quality, showing that MetaNL can generate high-quality frames at real-time rates (right). The shaded area around each line represents the standard deviation the method across three DTU scenes.
when computing the loss for each target image. Since all views must be sphere traced and passed through E for single target view, this additional batching only adds additional forward passes to T and D. which are fast to evaluate. This batching strategy gives more accurate gradients for our model each iteration. Additionally, since Las requires us to find a minimum of d along ray, requires dense sampling of this network and accounts for most of the compute time of each forward pass. Thus, while optimizing NLR++ we propose to only enforce shape-related losses LM and LE for the first i iterations, and then every l2 iterations thereafter. This allows NLR+ to learn shape approximation in the first ( iterations, and then further refine as the feature encoding scheme with E, D, I get significantly better. This is only possible since, unlike prior Neural Lumigraph work [4] 5), the appearance modeling is outsourced to the feature extraction from input images and is more independent from the current shape than dense appearance representation in 3D space.
3.3 Generalization using meta learning
As goal is to learn scene representations quickly, encoding, decoding, aggregation, and shape representation using datasets of multi-view images. This prior is realized via the initialization of the networks Ee, D., I's and dh which dictates the network convergence properties during gradient descent optimization. For simplicity of notation and since we meta-learning the initializations for all networks in NLR++ define all NLR+ parameters as = [g, v C. 8.
meta learning learn prior over feature
Let O, denote the NLR++ parameters initialization, and O denote the parameters after iterations of optimization. For a fixed number of steps m optimization, O. will depend significantly on the initialization Oo, resulting in possibly significantly different NLR++ losses. We adopt the notation from [82], and will emphasize the dependence of parameters on initialization by writing O, (o,, T), where 7' is the particular scene which we would like to represent. We aim to optimize the initial weights Op that will result in the lowest possible expected loss after m iterations when optimizing NLR++ for an unseen object T, sampled from distribution of objects T. This expectation over objects denoted as ErsT, resulting in the meta learning objective of:
+ = arg min Pr-T(C(O,,,, Co
T))]
(10)
To learn this initialization for Og, we use the Reptile [85] algorithm. which computes the weight values Om (0,. I') for fixed inner loop step size of m. Each optimization of NLR++ for steps for a different task sampled from 7, ~ T referred to as an outer loop, and is indexed by j. To avoid having to compute second-order gradients through the NLR+ model, Reptile updates the initialization Og in the direction of the optimized task weights with the following equation:
o 0g (( (%) T3) 0/),
(11)
where 3 is the meta-learning rate hyperparameter. We label NI. with the meta meta-learning initializa- tions applied MetaNLR++.
5

---

5 min.
10 min.
Convergence
NLR
20.7
17.8
21.0
26.4
32,8
3166
NeRF
20.7
22.5
23.1
23.7
26.3
27.5
SVS*
28.4
25.6
28.7
26.5
27.3
26.1
NLR+
27.1
252
28.7
29.1
31.5
30.6
MetaNLR++
27.9
29.9
28.6
30.9
GT
Figure 3: Novel views synthesized using various methods after a set training time. MetaNLR++ outperforms other surface and volume representation methods, especially for a training time budget on the order of minutes, and does not sacrifice quality of the final converged result.
3.4 Implementation details
The source code and pre-trained models are available on our project website, and the full set of implementation details including hyperparameters, training schedules, and architectures are described in our supplement for each of the various datasets evaluated on. We implement MetaNL in Py PyTorch and use the Adam [87] optimizer for all optimization steps of NLR++ including for the inner-loop in meta learning, with starting learning rate of 1 x 10 for & and 5 x 10 for E, D, r. We use a 50, т = 1 x 10-3, and / = x 10-1 as starting hyperparameter values, which are progressively decayed (or increased in the case of a) through training (full schedules are described in the supplement). We use shape loss training hyperparameter values of 4 50 and l2 = 7, and loss weight parameters of λ = 10² /a. X2 3. In the case of NLR++, we initialize & as unit sphere of radius 0.5 by pre-training our SIREN to represent this shape. We train each of our models on single Nvidia Quadro RTX8000 GPU. We also use an Nvidia Quadro RTX6000 GPU for rendering and training iteration time computation. In total, we have an internal server system with four Nvidia Quadro RTX8000 GPUs and six Nvidia Quadro RTX6000 GPUs, of which used subset of three RTX8000s and one RTX6000.
4 Experiments
Baselines. Our main contribution is the rapid learning of representation which can be used to render high-quality novel views of a scene in real time. We demonstrate this by comparison to several of-e-ooftthearr methods. Specifically, we evaluate the volumetric representation of NeRF [60], a mesh- based representation similar to SVS [8), the neural signed distance function- lunction-based representations of IDR 4 and NLR (5) and the image-based rendering of IBRNet (29.. For SVS [ we use our own simplified implementation and denote it SVS*. Our implementation trains the same F, D, I as in MetaNL R++ but we replace the learnable shape by a surface reconstruction from COLMAP 25| 26

---

Training time vs. quality trade-off. For the following comparisons, we use the DTU dataset [4] [8), which has been made public by its creators, and contains multi-view images of various inanimate objects, none of which are offensive or personally identifiable. In Figure 2 we plot the average PSNR and LPIPS score of three held-out test views on three test DTU scenes as a function of training time as measured on Nvidía Quadro RTX6000.. Each of these representations are trained using only seven ground-truth views from the DTU scene. The meta-learned initializations Oj are optimized using complete view sets from another 15 DTU scenes, distinct from the testing scenes, In these plots, we see that MetaNLR+ maintains high reconstruction quality throughout the training process which results in predictable quality progression for conetconttr applications. Beyond PSNR we showcase results of the perceptual LPIPS metric [86] as we observe that PSNR is not robust to small inaccuracies in the object masks and prefers low-frequency images. This trade-off is exemplified in Table D where we show that MetaNI is able to reach the 25dB and 30dB PSNR milestones faster than any other learned scene representation. The difference is particularly large for the volumetric method NeRF that aggregates many radiance samples along each rendered ray, and the implicit surface- based method NLR which must simultaneously optimize a neural surface and neural representation of color densely in 3D. In the case of NLR, this is because every time the shape representation is updated, the color representation must also update to correctly reflect the appearance on the modified surface of the shape, leading to slow training time.
While IBRNet [29] is able
Table 1: Training time to reach specified PSNR level. The best times and PSNR values are bolded (second best is underlined) for meth- ods which can render at real-time framerates. The training times are included for NeRF and IBRNet as well, which are incapable of fast rendering. IBRNet pre-trained on multi-view data, and thus needs no further training to reach 25dB PSNR.
to generate high-quality novel views very quickly, it cannot be turned into a pre-computed mesh-based or volume representation and requires input images for each rendered frame, and is more aptly consid- ered feed-forward image- blending method instead of neural scene representa- tion. SVS* is initially offset by the runtime of COLMAP. Afterwards, it quickly fits E, D,T, but its maximum performance is limited by the initial geometry. Con-
25dB PSNR
30dB PSNR
Maximum PSNR
IDR NLR
24.73dB
14.7 min.
191.4 min.
32.95dB
MetaNLR SVS*
2.1 min.
176.8 min.
31.71dB
2.1 min.
19.B
NLR+
3.2 min.
37.4 min.
31.02dB
MetaNLR++
1.9 min.
22.5 min.
30,57dB
NeRF
33.3 min.
27.95dB
IBRNet
0 sec.
23.1 sec.
31.86dB
sequently, D must learn to inpaint the resulting holes which results in over-fitting to training views. 3 and Table [] where the quality of novel views quickly saturates or
This is notable in Figures 2 even degrades. Additionally. the meshing step time scales with the number of input views, and it can take up to 2 hours for scenes with dense view coverage as reported in [8]. MetaNLR is provided as a baseline which applies our meta learning method to the NLR formulation. This shows that simply applying meta learning to NLR does not lead to nearly as fast of training to high quality when compared to MetaNL and therefore the improvements in scene parameterization in NLR++ and the generalization via meta learning are necessary components which jointly contribute to achieving the desired goal of both fast training and rendering. Qualitative comparisons are shown in the supplementary document. We emphasize that learning a representation from only seven views is a difficult task. NeRF in particular has difficulty avoiding over-fitting to training views in this scenario. We show qualitative results in Figure 3 which highlight that MetaNL performs progressively higher quality view synthesis using the learned features and geometry the training advances.
Training to convergence. In Table I and Figure 3 we show that MetaNLR++ does not sacrifice on final converged quality for the sake of speed. Given unconstrained learning time, MetaNLR++ is still able to produce images which are competitive with converged results of state-of-in of-the-art representations. As noted in Table IBRNet is able to also perform high quality novel view synthesis, and can render images with 29. 20BB without fine-tuning. However, since the rendering time is based off of neural volume rendering, it cannot generate frames at nearly a real-time rate. Additional comparisons are provided in the supplementary document. A quantitative evaluation of image quality and runtime at rendering time is shown in Figure 2 where we see that our surface- based method is able to render in real time, unlike neural volume rendering methods.
7

---

Real-time rendering. Because MetaNLR++ extracts the appearance directly from the source images and transforms them to the novel view using a compact shape model, we can greatly accelerate the rendering of our trained representation by pre-computing these components. First, we use the marching cubes algorithm [89] to extract a mesh as an iso-surface of the SDF Second, we store the features computed from the set of input images by our encoder E as well as their associated depth maps. At render time, we simply need to aggregate these features based on our target viewing direction, and evaluate the decoder on the aggregated features for each frame. For the DTU images and network architecture, the decoder takes 8. 8.7,,, and the aggregation takes 22.3ms, so we are able to render frames real-time rates as shown in Figure
Meta-learning ablation. We investigate the efficacy of meta learning in speeding up learning of individual components of our representation. Specifically, we ablate the contribution of meta learning on the shape representation parameters (fo), and feature encoding network parameters (Co, Vo, Co).
Additionally, we also compare MetaNLR++ to variant with the encoder and decoder replaced by direct extraction of image RGB pixel values and with and without a learned aggregation function. Here we clarify that replacing the en- coder and decoder with pixel val- ues is still a variant of NLR++ where we have simply replaced the features with pixel values directly, and does not model a dense, continuous color function as in NLR. The results of this ablation study are shown in Table 2
Table 2: Meta-learning ablation study. We report average PSNR
of synthesized views after 10 minutes for each method.
Meta init. applied to:
No net.
c only
All net.
RGB pixels & fixed I
27.85dB
28.13dB
28.13dB
RGB pixels & learned
26.82dB
26.90dB
26.91dB
CNN E/D & fixed P
28.62dB
29.16dB
29.88dB
CNN EID & learned r
27.32dB
28.48dB
29.91dB
We see that meta-learning of each component contributes to fast learning of high-quality repre- sentation (last row). The learned I' improves the performance for the learned E/D but it decreases the performance for the directly extracted RGB values. This implies that the classical unstructured lumigraph blending [14] works well for direct pixel values, but the additional flexibility of the learned I' can be advantagcous with deep features. The learned performs well when meta-learned but has difficulty to accurately learn the angular dependencies in only 10 minutes of training otherwise. More details of this experiment and additional results are available in the supplementary document.
Test view
Neural shape
3 views
Input size ablation. We illustrate the robustness of MetaNLR++ to low numbers of available training views in Figure 4 Here we see that our view synthesis quality decreases very gracefully with decreasing number in- put views and it produces meaningful results even in the extreme case of three views. This is unlike COLMAP, which produces geometry with significant holes in oc- cluded regions. These holes are responsible for the highly variable performance of SVS* shown in Fig- ure Additionally, while SVS* is able to quantitatively perform well when evaluated within the ground-truth mask, the holes in the geometry result in inaccurate rendering masks and thus severely limit novel view synthesis in practice. Additional results showing this phenomenon are included in the supplementary doc- ument. Since this ablation is performed on the DTU dataset, the input images capture only one side of the object. However, our method is applicable in scenarios where images are distributed 360 around the object. Additional results demonstrating this capability on the ShapeNet (90) dataset are included in the supplemen- tary document. Our learned shape models are overly smooth relative to other methods and thus not partic- ularly quantitatively accurate, but provide sufficient
25.1
7 views
32.5
24 views
33,8
49 views
33.7
Figure 4: MetaNLR++ is robust to the num- ber of views captured, which is essential in many applications where capturing a dense dataset is infeasible. In all cases, the learned & provides an adequate support for projec- tion of our encoded appearance features.
8

---

34.21
37.30
35.87
Figure 5: We compute additional results using the NLR dataset. The frames highlighted are supervised, and the intermediate frames are interpolated viewpoints for this MetaNLR++ model. Please the supplement for additional results.
30 min. PSNR (LPIPS)
Convergence PSNR (LPIPS)
NLR++ MetaNLR++
28.22 (0. 145)
35.54 (0.046)
31.12 (0.083)
37.55 (0.034)
Table 3: Table comparing PSNR score on the supervised views of the NLR dataset. Since there are only 6 training views, we use all for training and report PSNR averaged on the three views shown in Figure which shows that we are able to interpolate between these views.
accuracy for the image-based feature blending method to model appearance information observed in synthesized views. The smoothness of the learned shape is dependent on the capacity of the CNN architecture used for the feature encoder and decoder and coordinate-base surface, as shown in the supplementary document. We opt to use higher capacity feature encoder and decoders in order model sharp image details for novel view synthesis, which is responsible for the trade-off of quantitative accuracy of the neural shape representation.
network modeling the
Additional results. To show that MetaNLR++ is robust to datasets beyond DTU, we evaluate using the multi-view dataset in NLR [5] This dataset has been publicly released, and while the faces of the subjects in the dataset are personally identifiable, the subjects are the authors of NLR and have provided their consent of making this dataset public. The scenes in this dataset each have 22 multi-view images, taken with various cameras. We opt to use the final 6 views taken with high-resolution cameras to train our representation. We use 5 scenes in this dataset learn the meta initialization Og and test a withheld sixth scene. The meta learning in this case leads to significantly improved performance in both PSNR and perceptual quality, despite only being able to learn prior from a small number of scenes. This trend demonstrates that when the meta-training data more accurately covers the testing domain, the prior learned through meta learning is more effective at speeding up training time. This trend is further demonstrated in the supplementary document the ShapeNet dataset, where the meta- ceta-training dataset is significantly larger and more closely related to the specialization data than in the case of DTU. With more comprehensive multi-view training datasets, this trend shows that MetaNLR++ could even further speed up neural representation training time.
In Table we report our fit on the training frames after 30 minutes of training as benchmarked on our system. Additionally, we show interpolated frame results in Figure demonstrating that MetaNLR++ capable generalizing to this scene and producing convincing novel-view synthesis results. Additional implementation information and results included in the supplementary document.
5 Discussion and Conclusion
The fundamental problem of learning 3D scene representations from sparse sets of 2D images is rooted in machine learning, computer vision, and computer graphics. We provide an answer this problem by proposing novel parameterization of 3D scene, and an efficient method for inferring these parameters from observations using meta learning. We demonstrate that our representation and training method are able to reduce representation training time consistently and render at real- time rates without sacrificing on image synthesis quality. This opens several exciting directions for future work in efficient training and rendering of representations, including using more advanced

---

generalization methods to learn representations in real time. With this work, we make important contributions the field of neural rendering.
Limitations and future work. While our method is able to produce compelling novel view syn- thesis results in a fraction of the time of other methods, we note that there are a few shortcomings. Specifically, in order to bootstrap the learning of a neural shape quickly, object masks are required to supervise the ray-mask loss. While these can be computed automatically for some data, this poses challenge in cluttered scenes, applications which could generalize to arbitrary scenes. Additionally, all of our experiments have used known camera poses to reconstruct the shape. Future work jointly optimizing camera pose with our representation is certainly possible, and step in the direction for general view synthesis. Our method is also limited by memory consumption, since the CNN feature encoder/decoders process the entire image at a time. This method could likely be improved by shifting to training and evaluating on image patches for higher resolution rendering. Finally, our method tends produce overly-smoothed shape models, which, while beneficial for feature aggregation, are not always representative of high-frequency scene geometry. This highlights one fundamental trade-off: the capacity of the feature generation method versus the quality of the shape. With feature or color generation methods which are sufficiently regularized [4] the model has no choice but to explain observed details the neural shape model. We opt to utilize the full capacity of the CNN feature processing, as learning detailed neural shape is slower than modeling fine details with features.
Conclusion. The question of how to trade-off image synthesis quality with representation training and rendering time of paramount importance to engineers, producers, or any other users of neural rendering technology. In the space of neural rendering methods, this work takes steps towards making representation learning and rendering more practical by optimizing this trade-off. Our novel scene parameterization and generalization method may provide starting point for future work in optimizing this trade-off: speeding up representation training and rendering time and bringing modern neural rendering to the forefront of industry standard techniques.
Broader Impact
Methods such MetaNLR++ for learning 3D scene representations from 2D representations allow for photorealistic image synthesis using only collections of other images. We have shown that MetaNLR++ improves upon the axes of training and rendering time of these representations, and as such may make it less computationally restrictive to use for individuals who want to learn and use 3D models from only collections of easily acquirable images. While this proliferation of neural rendering technology may be extremely helpful for many, it has the potential for misuse. As with any synthesis method, the technology could enable approaches to synthesis of deliberately misleading or offensive images, posing challenges similar to those posed by generative-adversatee models.
Acknowledgments and Disclosure of Funding
Alexander W. Bergman was supported by a Stanford Graduate Fellowship. Gordon Wetzstein was supported by a PECASE by the ARO. Other funding was provided by the NSF (1839974).
References
[1] Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason Saragih, Matthias Nießner, al. State of the art on neural rendering. Eurographics, 2020.
[2] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Trans. Graph. (SIGGRAPH), 38(4), 2019.
[3] Michael Niemeyer, Lars Mescheder, Michael Oechsle, and Andreas Geiger. Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision. In CVPR, 2020.
[4] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Ronen Basri, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. In NeurIPS, 2020.
10

---

[5] Petr Kellnhofer, Lars Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, and Gordon Wetzstein. Neural lumigraph rendering. In CVPR, 2021.
[6] Michael Occhsle, Songyou Peng, and Andreas Geiger. Unisurf: Unitying neural implicit surfaces and radiance fields for multi-view reconstruction. In /CCV, 2021.
[7] Gernot Riegler and Vladlen Koltun. Free view synthesis. In ECCV, pages 623 640. Springer, 2020.
[8] Gernot Riegler and Vladlen Koltun. Stable view synthesis. In CVPR, 2021.
[9] Shenchang Fric Chen and Lance Williams. View interpolation for image synthesis. In SIGGRAPH, pages 279-288, 1993.
[10] Mare Levoy and Pat Hanrahan. Light field rendering. In SIGGRAPH, pages 31-42, 1996.
[11] Steven Gortler, Radek Grzeszezuk, Richard Szeliski, and Michael F Cohen. The lumigraph. In SIGGRAPH, pages 43 54, 1996.
[12] Paul E Debevec, Camillo Taylor, and Jitendra Malik. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. In SIGGRAPH, pages 11-20, 1996.
[13] Jonathan Shade, Steven Gortler, Li-wei He, and Richard Szeliski. Layered depth images. In SIGGRAPH. pages 231 242, 1998.
[14] Chris Buchler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen, Unstructured lumigraph rendering. In SIGGRAPH, pages 425-432, 2001.
[15] Harry Shum and Sing Bing Kang. Review of image-based rendering techniques. In Visual Communications and Image Processing, volume 4067, pages 2-13, 2000.
[16] Gaurav Chaurasia, Sylvain Duchene, Olga borkine-Homunn, and George Drettakis. Depth synthesis and local warps for plausible image-based navigation. ACM Trans. Gnaph., 32(3):1-12, 2013.
[17] Eric Penner and Li Zhang. Soft 3d reconstruction for view synthesis. ACM Trans. Graph., 36(6):1-11, 2017.
[18] SM Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S Morcos, Marta Garnelo, Avraham Ruderman, Andrei A Rusu, Ivo Danihelka, Karol Gregor, et al. Neural scene representation and rendering. Science, (60(6394)) 1204-1200, 2018.
[19] Peter Hedman, Julien Philip. True Price, Jan-Michael Frahm, George Drettakis, Gabriel Brostow. Deep blending for froe-viewpoint image-based rendering. ACM Trans. Graph. (SIGGRAPH Asia), 37(6), 2018.
[20] Inchang Choi, Orazio Gallo, Alejandro Troccoli, Min H Kim, and Jan Kautz. Extreme view synthesis. In ICCV, pages 7781-7790, 2019.
[21] Xu Chen, Jie Song, and Oumar Hilliges. Monocular neural image based rendering with continuous view control. In CVPR, pages 4090-41000 2019.
[22] Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, and Ricardo Martin-Brualla, Neural rerendering the wild. In CVPR, pages 6878-6887, 2019.
[23] Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neural rendering: Image synthesis using neural textures. ACM Trans. Graph., 38(4):1-12, 2019.
[24] Mojtaba Bemana, Karol Myszkowski, Hans-Peter Seidel, and Tobias Ritschel. X-fields: implicit neural view-, light-and time-image interpolation. ACM Trans. Graph., 39(6):1-15, 2020.
[25] Johannes Lutz Schönherger and Jan-Michael Frahm. Structure-from-mmotee revisited. In CVPR, 2016.
[26] Johannes Lutz Schönberger, Enliang Zheng, Mare Pollefeys, and Jan-Michael Frahm. Pixelwise view selection for unstructured multi-view stereo. In ECCV, 2016.
[27] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, and Justin Johnson. SynSin: End-to-end view synthesis from a single image. In CVPR, 2020.
[28] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa, pixelnerf: Neural radiance fields from one few images. In CVPR, 2021.
[29] Qianqian Wang, Zhicheng Wang, Kyle Genova, Pratul Srinivasan, Howard Zhou, Jonathan Barron, Ricardo Martin- Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering. In CVPR, 2021.
[30] Angjoo Kanazawa, Shubham Tulsiani, Alexei A Efros, and Jitendra Malik. Learning category-specifie mesh reconstruction from image collections. In ECCV, pages 371-386, 2018.
[31] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. ACM Trans. Graph. (SIGGRAPHI, 2018.
[32] John Flynn, Michael Broxton, Paul Debevec, Matthew Vall Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gradient descent. In Proc. CVPR, pages 2367-237777 2376, 2019.
11

---

[33] Richard Tucker and Noah Snavely. Single-view view synthesis with multiplane images. In CVPR. pages 551-560, 2020.
[34] Suttisak Wizadwongsa, Pakkapon Phongthawee, Jiraphon Yenphraphai, and Supasom Suwajanakom. Nex: Real-time view synthesis with neural basis expansion. In CVPR, 2021.
[35] Michael Broxton, John Flynn, Ryan Overbeck, Daniel Erickson, Peter Hedman, Matthew Duvall, Jason Dourgarian, Jay Busch, Matt Whalen, and Paul Debevec. Immersive light field video with layered mesh representation. ACM Trans. Graph. (SIGGRAPH), 39(4), 2020.
[36] Benjamin Attal, Selena Ling, Aaron Gokaslan, Christian Richardt, and James Tompkin. Matry ODShka: Real-time 6DoF video view synthesis using multi-sphere images. In Proc. ECCV, August 2020. URL https://visual.ce cs. brown. edu. mdt./a/arrrraaaaaaa.
[37] Vincent Sizzmann, Justus Thics, Felix Heide, Matthias Nicßner, Gordon Wetzstein, and Michael Zollhöfer. Deepvoxels: Learning persistent 3d feature embeddings. Proc. CVPR, 2019.
[38] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: Learning dynamic renderable volumes from images. ACM Trans. Graph. (SIGGRAPH), 38(4), 2019.
[39] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: cearning continuous signed distance functions for shape representation. CVPR, 2019.
[40] Lars Mescheder, Michael Occhsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019.
[41] Kyle Genova, Forrester Cole. Daniel Vlasic, Aaron Sarna, William T Freeman, and Thomas Funkhouser. Learning shape templates with structured implicit functions. In ICCV, pages 7154-7164, 2019.
[42] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In CVPR, 2020.
[43] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In CVPR, pages 5939-5948 2019.
[44] Mateusz Michalkiewicz, Jhony K Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders Eriksson. Implicit surface representations as layers in neural networks. In ICCV, pages 4743-4752, 2019.
[45] Matan Atzmon and Yaron Lipman. Sal: Sign agnostic learning of shapes from raw data. In CVPR, 2020.
[46] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization. In ICCV, pages 2304-2314, 2019.
[47] Vincent Sitzmann, Michael Zollhöfer, and Gordon Wetzstein. Scene representation networks: Continuous 3d-structure-awe neural scene representations. In NeurlPS, 2019.
[48] Michael Occhsle, Lars Mescheder, Michael Niemeyer, Thilo Strauss, and Andreas Geiger. Texture fields: Learning texture representations in function space. In ICCV, 2019.
[49] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman. Implicit geometric regularization for learning shapes. In Proceedings of Machine Learning and Systems 2020, pages 3569-3579. 2020.
[50] Thomas Davies, Derek Nowrouzezahrai, and Alec Jacobson. Overfit neural networks compact shape representation, 2020.
[51] Vincent Sitzmann, Julien N.P. Martel, Alexander W. Bergman, David B. Lindell. and Gordon Wetzstein. Implicit neural representations with periodic activation functions. In NewrlPS, 2020.
[52] Lingjic Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Thcobalt. Neural sparse voxel fields. NeurlPS, 2020.
[53] Yue Jiang, Dantong Ji, Zhizhong Han, and Matthias Zwicker. Sdfdiff: Differentiable rendering of signed distance fields for 3d shape optimization. In CVPR, 2020.
[54] Shaohui Liu, Yinda Zhang, Songyou Peng, Boxin Shi, Mare Pollefeys, and Zhaopeng Cui. Dist: Rendering deep implicit signed distance function with differentiable sphere tracing. In CVPR, 2020.
[55] A. Kohli, V. Sitzmann, and G. Wetzstein. Semantic Implicit Neural Scene Representations with Semi- supervised Training. In International Conference on 3D Vision (3DV), 2020.
[56] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional
occupancy networks. In ECCV, 2020.
[57] Chiyu Max Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias NieBner, and Thomas Funkhouser. Local implicit grid representations for 3d scenes. In Proceedings IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
12

---

[58]
Rohan Chabra, Jan Eric Lenssen, Eddy 1lg. Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In ECCV, 2020.
[59]
Julien N.P. Martel, David B. Lindell, Connor Z. Lin, Eric R. Chan, Marco Montciro, and Gordon Wetzstein. Acor: Adaptive coordinate networks for neural representation. ACM Trans. Graph. (SIGGRAPH), 2021.
[60]
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
[61] Ricardo
Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, and Daniel Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR, 2021.
[62] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositional generative ncural feature fields. In CVPR, 2021.
[63] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. In CVPR, 2020.
[64] Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T. Barron. Nerv: Neural reflectance and visibility fields for relighting and view synthesis. In CVPR, 2021.
[65] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 2020.
[66] Thomas Neff, Pascal Stadlbauer, Mathias Parger, Andreas Kurz, Joerg H. Mueller, Chakravarty R. Alla
Chaitanya, Anton S. Kaplanyan, and Markus Steinberger. DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks. Computer Graphics Forum, 40(4), 2021.
[67] Alex Yu, Ruilong Li. Matthew Tancik, Hao Li, Ren Ng. and Angjoo Kanazawa. PlenOctrees for real- real-time rendering of neural radiance fields. In ICCV, 2021.
[68] Peter Hedman, Pratul P. Srinivasan, Ben Mildenhall, Jonathan T. Barron, and Paul Dehevec. Baking neural radiance fields for real-time view synthesis. In /CCV, 2021.
[69] Stephan J Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, and Julien Valentin. Fastnerf: High-fidelity neural rendering at 200fps. In ICCV, 2021.
[70] Christian Reiser, Songyou Peng. Yiyi Liao, and Andreas Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mips. In ICCV, 2021.
[71] David Lindell, Julien Martel, and Gordon Wetzstein. Autoint: Automatic integration for fast neural volume rendering. In CVPR, 2021.
[72] Lior Yariy, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. arXiv preprint arXiv:2106. 12052, 2021.
[73] Peng Wang. Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurIPS, 2021.
[74] Thu Nguyen- Nguyen-Phuoc, Chuan Li, Lucas Theis, Christian Richardt, and Yong-Liang Yang. Hologan: Unsuper- vised learning of 3d representations from natural images. In ICCV, 2019.
[75] Thu Mguyen-Phuue Christian Richardt, Long Mai, Yong-Liang Yang, and Niloy Mitra, Blockgan: Learning 3d object-aware scene representations from unlabelled images. In NewrIPS, 2020.
[76] Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. In NewrlPS, 2020.
[77] Eric Chan, Marco Monteiro, Petr Kelinhofer, Jiajun Wu, and Gordon Wetzstein. pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In CVPR, 2021.
[78] Michael Niemeyer and Andreas Geiger. Campari: Camera-aware decomposed generative neural radiance
fields. In International Conference 3D Vision (3DV), 2021.
[79] Yinbo Chen, Sifei Liu, and Xiaolong Wang. Learning continuous image representation with local implicit image function. In CVPR, 2021.
[80] Ishit Mehta, Michaöl Gharbi, Connelly Barnes, Eli Shechtman, Ravi Ramamoorthi, and Manmohan Chandraker. Modulated periodic activations for generalizable local functional representations. In ICCV, 2021.
[81] Vincent Sitzmann, Eric R. Chan, Richard Tucker, Noah Snavely, and Gordon Wetzstein. Metasdf: Meta- learning signed distance functions. In NewrIPS, 2020.
[82] Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P Srinivasan, Jonathan T Barron, and Ren Ng. Learned initializations for optimizing coordinate based neural representations. In CVPR, 2021.
13

---

[83] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR. 2016.
[84] Olaf Ronneberger, Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In Medical Image Computing and Computer-Asisste Intervention -MICAI 2015, pages 234 24412241222 2013.
[85] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint ar0iv 02999, 2018.
[86] Richard Zhang, Phillip Isola, Alexei Efros, Shechtman, and Oliver Wang. The unreasonable offectiveness of deep features as perceptual metric. In CVPR, pages 286-595, 2018.
[87] Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR, 2014.
[88] Rasmus Jensen, Anders Dahl, George Vogiatzis, Engil Tola, and Henrik Aanæs. Large scale multi-view stereopsis evaluation. In 2014 IEEE Conference Computer Vision and Puttern Recognition, pages 406-413. IEEE, 2014.
[89] William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. In ACM SIGGRAPH, page 163-169, 1987.
[90] Angel X. Chang. Thomas Funkhouser, Leonidas Guibas, Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song. Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rian 3D Model Repository. arXiv preprint arXiv: 2015.
2015.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section S
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as URL)? [No] We plan to release the code completely but have not yet with submission.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] See supplementary document and Section 3.4
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [No] As models take a significant amount of time to train, averaging over many seeds would be computationally impractical. We report standard deviation error bars with respect to multiple testing scenarios, which demonstrate the robustness of our method.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 3.4
4. If you using existing assets (e.g., code, data, models) or curating/releasing
assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] We were unable find the license of these datasets, but the authors have explicitly published their datasets to the public. We
also discuss the nature of the datasets including their content and consent in Section 4
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] See Section
14

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Section and supplementary document for additional discussion the datasets.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "yd7QS09z1j2", "reviewer": "Reviewer_LYqQ", "summary": "This paper presents a method for capturing the 3D shape and appearance of objects using an implicit model for the shape, and an aggregation of reprojected CNN features for the appearance. The authors highlight two problems with existing approaches that they try to address - slow training and rendering times. The key contribution of the paper is the use of meta-learning, in the form of a learned initialization, to improve the efficiency of the training process. The results show that the proposed method achieves a marginal improvement in training efficiency.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**TL;DR:** A useful idea to speed up the training and rendering of neural scenes, but the practical improvements are not that impressive. In the main review I have labelled points as a strength (+), weakness (-) or mixed (+/-).\n\n**Originality**: \n\n(+) The method is new as I am not aware of any existing work that uses a combination of neural level-set surfaces and aggregated CNN features for appearance. There are, however, existing works that have investigated the use of learned initializations for training NeRF-like models but these have not been used in combination with generalizable features from a CNN. \n\n(+) Although all of the components of the approach (the SDF representation, the CNN features and the learning initialization) are all well-known techniques, the combination is novel and valuable as they all work together to improve the training efficiency. \nThe authors have done a good job at differentiating this work from previous contributions (such as MetaSDF and [78]) and, as far as I can tell, have adequately cited all the related work.\n\n**Quality**: \n\n(+) The submission appears to be technically sound and the details of the method are well documented. One technicality that I am not so sure about is why in Section 3.2, out of the k images that are sampled, only one is used as the ground-truth target image. Surely all of the selected images can act as training targets? \n\n(+/-) The claims of improved training efficiency and quality are supported by the quantitative evaluation that compares the approach to recent methods like SVS, IBRNet, NLR. However, the improvements in training time and PSNR are not that major (see significance below). \n\n(+) Overall, the paper is definitely a complete piece of work. The method is fully developed and it has been evaluated and compared to recent approaches on two different datasets. The figures, results and writing are well polished.\n\n(+) The approach has a number of weaknesses, such as the need to use object silhouettes as input (and therefore can only reconstruct single objects and not the whole scene). Another disadvantage is that the reconstructed geometry is rather coarse and not very detailed. However, the authors have done a very good job at explaining and justifying these limitations and providing ways in which they could be addressed in future work.\n    \n**Clarity**: \n\n(+) The submission is very clearly written and the overall structure of the paper is well organized. As far as I can tell the paper provides enough information about the method and the experimental setup for a reader to be able to reproduce its results.\n\n**Significance**: \n\n(-) The results are important in the sense that fast training and rendering of neural scene representations is challenging and an open problem. This paper addresses the training speed in two ways - through the use of generalizable features extracted from a CNN and by using meta-learning, where the main contribution seems to be the use of the meta learning (as per the title). Although it is conceptually a good idea, the real improvements gained by the meta learning seem to not be that significant - for example 27.32dB vs 29.91dB in the best case in Table 2. Is this difference even perceptually significant? Are there ways in which the meta learning could be improved to get a more significant gain in PSNR?\n\n(+/-) There are ways in which others are likely to use or build upon the ideas presented in the paper, but the seemingly minor improvement in the training speed and accuracy might dissuade many in doing so.   \n\n(+) The method does advance the state of the art in terms of training speed (albeit in a very slight manner). In terms of this comparison I would say that IBRNet should be included in Table 1 as there are ways in which it could be be turned into a pre-computed volume representation (for example, by storing the projected features from the nearest images in an Octree).\n\n# Post response and discussion\nAfter considering the authors' response and the other reviews, most of my concerns have been addressed. I agree with rL4a that the core components of the proposed method come from elsewhere and there does not seem to be any special theoretical contributions to these in the proposed paper (most are used as is). However, I think the approach is well-engineered and it seems to achieve good results in terms of training efficiency vs quality which is an important problem for these types of neural reconstruction+rendering approaches.  My final rating is therefore \"6: Marginally above the acceptance threshold\".\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper does a good job at describing the potential negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "sT8f3fe0U_E", "reviewer": "Reviewer_cxjr", "summary": "The paper finds a set of weight values that is a good initialization to train a recent novel-view synthesis (NVS) method, Neural Lumigraph Rendering (NLR). A meta-learning library, Reptile, is used to optimize this initial weight vector across a corpus, the DTU multiview dataset. The paper shows, that with such an initialization, optimization proceeds much quicker and to similar quality. Additionally, it is suggested to change NLR to aggregate features from the input images instead of pixel values directly, a change that appears to be orthogonal to the meta-learning aspect. Results show that after much fewer optimization steps, a good result is achieved. The method is fast, but that is a property of NLR.", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The topic of meta-learning is original, yet the same idea of finding a good init has now been applied to the novel-view synthesis based on overfit neural networks twice, once by Sitzman et al. and once by Tancik. et al. The difference between all three approaches is mostly that they are applied to different backbone NVS methods, but the use of Reptile itself to tune the init is the same. Hence the topic is novel, but within this new trend, this approach is not adding much new to the two existing papers, besides the insight that what worked for DeepSDF and for NeRF word for NLR, too. This might not be enough.\n\nThe quality of the result and experiments is high. All experiments are performed very convincingly and systematic. One could imagine comparing to Tancik et al. They do metalearning for NeRF, this does metalearning to NLR. Both can be compared on a novel-view task after N iterations as done in Fig. 3 with non-metalearned approaches.\n\nThe writing is exceptionally clear, modulo that what is described is not always very interesting in the sense that it does not matter so much to the main topic. Most readers will want to learn something interesting about metalearning, but it boils down to run Reptile and a mere init vector optimization. A good deal of the paper is about a change from pixel aggregation to feature aggregation, and also this is not quite interesting and had been done in many papers before. it really only is NLR++, an incremental change to NLR, inspired by many other recent papers.\n\nThe problem is significant, as training times are a large issue and quick adaption is a much desired goal.\n\nOverall, reading the paper left this reviewer slightly disappointed. A first brief look is very promising. A second, skim, read identified three relevant takeaways: 1) Reptile applied to two NVS methods before can be applied to NLR, too 2) Aggregating features is better than aggregating pixels, also shown for many NVS works before 3) NLR++ is high-quality, fast and learned quickly. A third, detailed, read did not bring anything substantial to note, resulting in a borderline assessment.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No idea", "review_text": "", "extra_scores": {"ethical_concerns": "No idea", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "RNBRs_7J6VI", "reviewer": "Reviewer_UVXv", "summary": "This paper presents MetaNLR++, a novel method for representing and rendering novel views of object-centric scenes. The scene representation combines a CNN-based feature extractor which extracts deep features from input views and a Neural SDF module which serves as a proxy geometry surface for transforming the extracted features to the novel views. The transformed features are then aggregated and processed by a CNN-based decoder to synthesize the output image. This use of image-based and surface-based scene representation enables faster rendering time compared to volume-based representation such as NeRF. A meta-learning technique based on learned initialization was leveraged to enable fast fitting of the model to new testing scenes. Experiments on two novel-view synthesis benchmarks demonstrate that the proposed method performs favorably compared to existing methods while requiring significantly less time for convergence.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nStrength:\n + The paper is well-written and easy to follow. \n + The proposed method is novel. It effectively addresses two important problems in novel-view synthesis: 1) Constructing the scene representation that enables fast rendering, and 2) Fitting such representation for novel scenes at test time in an efficient manner. The proposed method addresses (1) by combining classical image-based rendering pipeline (using a geometry proxy for transferring visual content from input views to novel views) with neural implicit function model for representing shapes and rendering. To address (2), a meta-learning technique is used which enables fast convergence time for fitting to a test scene.\n\nWeaknesses:\n + There is no serious weakness from my view. However, I feel that more discussions can be added to help the readers better connect this work to existing knowledge in novel-view synthesis and image-based rendering. For example, it can be noted that the proposed method leverages a standard pipeline in image-based rendering that transfer visual information from input views to target views using a proxy shape, and it can be emphasized that the proxy shape in this case is a learned neural implicit function which is the novel contribution of this work. Also, the idea of using CNN-based features instead of color values to represent the information to be transferred from the input views has also been leveraged before for view synthesis. For example, [1] transfers deep features from a single input view to the target view using a learned depth map. I believe the readers can benefit from more discussions on the connection between this work and those existing works.\n + The idea of using meta-learning to address the training-time problem makes sense. The proposed method was shown effective in the experiments using NLR and DUT dataset. The number of scenes in those datasets, however, is relatively small. I'm curious how will different properties of the training data distribution (e.g. number of training scenes, similarity between the training scenes and the testing scene) affect the effectiveness of the meta learning step. I believe synthetic datasets such as ShapeNet can be used to design more controlled experiments to investigate such aspects.\n\n[1] Olivia Wiles, Georgia Gkioxari, Richard Szeliski, Justin Johnson. \"SynSin: End-to-End View Synthesis From a Single Image\". Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitation and impacts have been discussed clearly in the paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "QM0WvRHarN", "reviewer": "Reviewer_rL4a", "summary": "This paper presents an approach to improving the training performance and overall quality of the recently introduced Neural Lumigraph Rendering (NLR) framework for novel view synthesis for objects, using a meta-learning framework that allows for quickly computing a representation of new scenes that can be rendered in real time.\n\nThe framework encodes the object surface using a neural signed distance function (SDF) representation of the surface, which can then be efficiently decoded using sphere tracing techniques. The framework encodes features from the 2D input images, which are aggregated from images that are visible in the target and decoded into the reconstructed target view. View-dependent effects are achieved using an MLP that acts as a learned weight-mapping function for the features aggregated from each input view and processed by the decoder. This decoder, in processing these feature maps, can fix artifacts caused by imperfect geometry reconstruction in the surface aggregation function. The Reptile algorithm is used as part of a meta-learning framework to learn a good parameter initialization from multi-view datasets that allows for quickly converging to a representation for previously unseen objects. Evaluations suggest that this approach achieves a good balance between fast convergence on unseen objects, final image quality and fast rendering speed compared to recent alternatives including the original NLR.\n\nOverall my impression is that the main contributions of this work are in the approach described to use more sophisticated shape and appearance representation compared to the original NLR, using image encoders and decoders and a more sophisticated weight-mapping function to achieve superior results while allowing for comparable inference speeds after training. My impression is that the meta-learning framework as described, which is largely based on the Reptile algorithm, is not as substantial a contribution.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall this work presents an approach that seems reasonable and attains reasonably good results. The clarity of exposition is generally quite good, and the supplemental information and video help to illustrate the strengths and limitations of this approach. However, I am not inclined to argue in favor of acceptance due to some key concerns that I have:\n\n- I noticed that for this paper (and the prior Neural Lumigraph Rendering work it extends), all of the depicted results appear to be for synthesizing novel views from a limited range of mostly frontal input images, similar to those from LLFF dataset used by NeRF. However, NeRF also demonstrates reasonable results on full rotations around isolated objects, e.g. synthetically rendered objects for which arbitrary input training views sampled from a sphere or hemisphere around the target are available. Given that this approach appears to be targeted at isolated foreground objects, I'm curious whether this approach would work for a similar range of novel viewpoints, given appropriate training data.\n\n- While the qualitative and quantitative results for this work appear to validate the authors' claims, and the efficient inference performance is an attractive trait of this work, its apparent limitation to segmented foreground objects is a fairly strong limitation compared to the other evaluated approaches that can also be applied to images of complete scenes, e.g. the NeRF results on LLFF image sets.\n\n- The core approach of this work is largely derived from previous work, which limits the overall novelty of the contributions. The extensions to this approach seem reasonable, but many are also relatively straightforward, e.g. using a network pre-trained on large object datasets to allow for faster adaptation to new objects, and using the Reptile algorithm as described in the paper for meta-learning.\n\nThis overall approach to meta-learning from other scenes appears to have some similarities to that of pixelNeRF, except that the latter uses a network trained on a large number of objects/scenes to allow for novel view synthesis using as little as 1 new image without any additional training. I am curious whether allowing for further fine-tuning of such a network given a set of multiple new images to obtain comparable results. I understand that this is a significant deviation from their approach, and don't expect a direct comparison to what I describe, but I would be curious to hear any comments from the authors or other reviewers on the relationship between these 2 approaches.\n\n- As noted by the authors, the overly smooth nature of the results and the requirement of object masks to supervise the ray-mask loss are fairly strong limitations compared to other methods that produce high-quality results for complex, unmasked scenes with multiple objects.\n\n- The SDF and sphere-tracing approach seems reasonable for opaque objects, and allows for efficient rendering of such objects compared to the volume rendering approach used by recent works using neural radiance fields, but it appears that its ability to account for complex effects such as translucent objects (e.g. glass) is limited. While this is an understandable tradeoff given the performance benefits, it should be noted as a limitation compared to some of the alternative approaches that are discussed and evaluated. In contrast, recent work like Nex [33] can handle such effects on full shiny/translucent while also maintaining real-time inference performance using an MPI-based approach, making me question the overall efficacy and value of this alternative approach.\n\nSome minor points:\n\n- I think that the notation used for the camera parameters, e.g. { C_i, R_i | t_i }, could be shortened to make the notation in the equations easier to read and follow.\n- L231: emphesize -> emphasize\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some broad limitations are discussed, although I think that some of the additional points raised in the main review above should also be addressed.\n\nThere are no particular issues w/ potentially negative societal impact that need to be addressed.\n", "review_text": "", "extra_scores": {"ethical_concerns": "There are no particular ethical concerns about this paper.\n", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "~4 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All four reviewers attest clarity of presentation, importance of the problem and solid engineering. There is a large spread in the judgements of the significance of the work explaining the different scores of 3,5,6,8. Overall the paper has been defended sufficiently well..
Decision: Accept (Poster)