Fast Training of Neural Lumigraph Representations using Meta Learning

1 Introduction . Learning 3D scene representations from partial observations captured by a sparse set of 2D images is a fundamental problem in machine learning , computer vision , and computer graphics . Such a representation can be used to reason about the scene or to render novel views . Indeed , the latter application has recently received a lot of attention ( e.g. , [ 1 ] ) . For this problem setting , the key questions are : ( 1 ) How do we parameterize the scene , and ( 2 ) how do we infer the parameters from our observations efficiently ? With our work , we offer new solutions to answer these questions . Several classes of scene representation learning approaches have recently been proposed . One popular approach consists of coordinate-based neural networks combined with volume rendering , like NeRF [ 2 ] . Although these representations offer photorealistic quality for synthesized images , they are slow to train and render . Coordinate-based networks that implicitly model surfaces combined with sphere tracing–based rendering are another popular approach [ 3–6 ] . One benefit of an implict surface is that , once trained , it can be extracted and rendered in real time [ 5 ] . However , training these representations is equally slow as training volumetric representations . Finally , approaches that use a proxy geometry with on-surface feature aggregation are also fast to render [ 7 , 8 ] , but the quality and runtime of these methods is limited by the traditional 3D computer vision algorithms that pre-compute the proxy shape , such as structure from motion ( SfM ) or multiview stereo ( MVS ) . Here , we develop a new framework for neural scene representation and rendering with the goal of enabling both fast training and rendering times . To optimize the training time of our framework , we do not learn a representation network for the view-dependent radiance , as other neural volume 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . or surface methods do , but directly aggregate the features extracted from the source views on the surface of our learned proxy shape . To cut down on pre-processing times required by SfM and MVS , we optimize a coordinate-based network representing the proxy shape end-to-end with our CNN-based feature encoder and decoder , and learned aggregation function . A key contribution of our work is to combine this unique surface-based neural rendering framework with meta learning , which enables us to learn efficient initializations for all of the trainable parts of our framework and further minimize training time . Because our representation directly parameterizes an implicit surface , it can be extracted and rendered in real time . This representation thus incorporates both the fast training benefits from generalizing over shape representations and image features , and the fast rendering capabilities of implicit surface-based methods . We demonstrate training of high-quality neural scene representations in minutes or tens of minutes , rather than hours or days , which can then be rendered at real-time framerates . 2 Related Work . Image-based rendering ( IBR ) . Classic IBR approaches synthesize novel views of a scene by blending the pixel values of a set of 2D images [ 9–18 ] . Recent IBR techniques leverage neural networks to learn the required blending weights [ 19–24 ] . These neural IBR methods either use proxy geometry , for example obtained by SfM or MVS [ 25 , 26 ] or depth estimation [ 27 ] , together with on-surface feature aggregation [ 7 , 8 ] , or use learned pixel aggregation functions [ 28 , 29 ] for geometry-free image-based view synthesis . Our approach is closely related to the geometry-assisted and feature-interpolating view synthesis techniques . Many existing approaches , however , require the proxy geometry to be estimated as a pre-processing step , which can take minutes to hours for a single scene , preventing fast processing of novel views . Other approaches which use monocular depth estimation to re-project and aggregate features such as SynSin [ 27 ] , are applied only to single input images , and can not easily be scaled to multi-view data due to depth estimation inconsistencies between images of the scene . Instead , our approach estimates a coordinate-based neural shape representation from the input images . The shape representation is optimized end-to-end with the image feature extraction , aggregation , and decoding and is accelerated using meta learning . Neural scene representations and rendering . Emerging neural rendering techniques [ 1 ] use explicit , implicit , or hybrid scene representations . Explicit representations , for example those using proxy geometry ( see above ) , object-specific shape templates [ 30 ] , multi-plane [ 2 , 31–34 ] or multisphere [ 35 , 36 ] images , or volumes [ 37 , 38 ] , are fast to evaluate but memory inefficient . Implicit representations , or coordinate-based networks , can be more memory efficient but are typically slow to evaluate [ 3–5 , 39–55 ] . Hybrid representations make a trade-off between computational and memory efficiency [ 56–59 ] . Among these , NeRF [ 60 ] and related methods ( e.g. , [ 61–66 ] ) use coordinate-based network representations and volume rendering , which requires many samples to be processed along a ray , each requiring a full forward pass through the network . Although recent work has proposed enhanced data structures [ 67–69 ] , network factorizations [ 70 ] , pruning [ 52 ] , importance sampling [ 66 ] , fast integration [ 71 ] , and other strategies to accelerate the rendering speed , training times of all of these methods are extremely slow , on the order of hours or days for a single scene . DVR [ 3 ] , IDR [ 4 ] , NLR [ 5 ] and UNISURF [ 6 ] on the other hand , leverage implicitly defined surface representations , which are faster to render than volumes but are equally slow to train . While concurrent work has improved the applicability of these methods by addressing limitations of surface-based methods in general , such as removing the object mask requirement [ 6 , 72 , 73 ] , these approaches are still slow to train as they rely on hybrid volumetric and surface formulations to bootstrap the neural surface training . Our approach builds on generalization approaches for neural scene representations to accelerate the training time of 2D-supervised neural surface representations , and thus can be applied alongside advanced training strategies for neural surfaces and make these representations even more applicable for practical view synthesis . Generalization with neural scene representations . Being able to generalize across neural representations of different scenes is crucial for learning priors and for 3D generative-adversarial networks with coordinate-based network backbones [ 74–78 ] . A variety of different generalization approaches have been explored for neural scene representations , including conditioning by concatenation [ 39 , 40 ] , hypernetworks [ 47 ] , modulation or feature-wise linear transforms [ 77 , 79 , 80 ] , and meta learning [ 81 , 82 ] . Inspired by these works , we propose a meta-learning strategy that allows us to quickly learn a high-quality neural shape representation for a given set of multi-view images . As opposed to the 3D point cloud supervision proposed by MetaSDF [ 81 ] , we meta-learn signed distance functions ( SDFs ) using 2D images , and as opposed to meta-learned initializations for NeRF volumes [ 82 ] , we operate with SDFs and features . Our approach is unique in enabling fast , high-quality shape representations to be learned from multi-view images that , once optimized , can be rendered in real time . 3 Method . 3.1 Image formation model . In this section , we outline the NLR++ novel view synthesis image formation model , which is presented in Figure 1 . NLR++ takes as inputs a set of images { Ii } Ni=1 each with corresponding binary foreground object masks { Mi } Ni=1 and known camera intrinsic parameters and extrinsic parameters collectively referred to as Ci . At output , NLR++ synthesizes an image Ît of the scene from the target viewpoint Ct . Drawing inspiration from classic image-based rendering methods , we define the image formation model as a learned pixel-wise aggregation Γζ ( · ) of input image features { Eξ ( Ii ) } Ni=1 and their target viewing direction Vt ∈ RH×W×3 on the surface of the object represented by the neural surface Φθ . To project the non-occluded , visible input features of Eξ ( Ii ) into the target view before aggregation , we define the function Pi→t ( Eξ ( Ii ) ; Ci , Ct ) . These neurally aggregated features are then decoded into an image by decoder Dψ ( · ) : Ît = Dψ ( Γζ ( { Pi→t ( Eξ ( Ii ) ) } Ni=1 , Vt ) ) . ( 1 ) The feature encoder E and decoder D are implemented as resolution-preserving convolutional neural network ( CNN ) architectures [ 83 , 84 ] with learned parameters ξ , ψ : Eξ : RH×W×3 → RH×W×d , Dψ : RH×W×d → RH×W×3 ( 2 ) To aggregate the input image features from E into a target feature map to be decoded by D , we use a learned feature aggregation ( or blending ) function Γζ , which operates on the surface of our shape representation Φ . To define the surface of our shape , we choose to use a SIREN [ 51 ] which represents the signed-distance function ( SDF ) in 3D space . This encodes the surface of the object as the zero-level set , L0 , of the network : L0 ( Φθ ) = { x ∈ R3|Φθ ( x ) = 0 } , Φθ : R3 → R. ( 3 ) The aggregation is performed on surface for each pixel of the target image Ît with camera Ct. To find the point in L0 ( Φθ ) corresponding to each pixel ray , we perform sphere tracing on the neural SDF modelR ( Φθ , Ct ) , retaining gradients for the final step of evaluation [ 4 , 5 , 53 , 54 ] . These individual rendered surface points are projected into the image plane of each of theN input image views and used to sample interpolated features from the source feature maps for each pixel , which can be arranged into N re-sampled feature maps corresponding to each input image { Fi } Ni=1 ∈ RN×H×W×d . To check whether or not a feature is occluded , we use sphere tracing for each pixel from the input views { R ( Φθ , Ci ) } Ni=1 , and ensure that the target view surface position projected into each of these surfaces is at the same depth as the source view surface position . Occluded features are then discarded . These three steps of sphere tracing , feature sampling , and occlusion checking make up the function Pi→t ( Eξ ( Ii ) ) which outputs each of the re-sampled feature maps { Fi } Ni=1 . Once the re-sampled feature maps { Fi } Ni=1 have been generated , the aggregation function Γζ aggregates them into a single target feature Ft ∈ RH×W×d which can be processed by the decoder into an image . This is done by performing a weighted averaging operation on the input features using the relative feature weights L ∈ RN×H×W : Γζ : RN×H×W×d × RH×W×3 → RH×W×d , { Fi } Ni=1 , Vt 7→ Γζ ( { Fi } Ni=1 , Vt ) = N∑ i=1 ( Li/ N∑ j=1 Lj ) ◦ Fi = Ft , ( 4 ) where ◦ is the Hadamard product between the feature and weight maps , broadcasted over the feature dimension d. The weight map L used in the feature aggregation function Γζ is obtained from an MLP γζ which is applied pixel-wise to each of the N re-sampled feature maps and each pixels target viewing direction : [ { Fi } Ni=1 , Vt ] 7→ γζ ( [ { Fi } Ni=1 , Vt ] ) = L. ( 5 ) Here , the dependence upon viewing direction allows for the modeling of view-dependent image properties . These pixel-wise operations making up Γ result in a H ×W × d feature map which can be input into the decoder D. The usage of features instead of pixel values directly allows D some opportunity to inpaint and correct artifacts from imperfect geometry to create a photorealistic novel view , unlike methods which render pixels individually [ 4 , 5 ] . Additionally , the use of the CNN encoder and decoder increases the receptive field of the image loss applied , allowing for more meaningful gradients to be propagated back into Φθ .

Summary: This paper presents an approach to improving the training performance and overall quality of the recently introduced Neural Lumigraph Rendering (NLR) framework for novel view synthesis for objects, using a meta-learning framework that allows for quickly computing a representation of new scenes that can be rendered in real time.  The framework encodes the object surface using a neural signed distance function (SDF) representation of the surface, which can then be efficiently decoded using sphere tracing techniques. The framework encodes features from the 2D input images, which are aggregated from images that are visible in the target and decoded into the reconstructed target view. View-dependent effects are achieved using an MLP that acts as a learned weight-mapping function for the features aggregated from each input view and processed by the decoder. This decoder, in processing these feature maps, can fix artifacts caused by imperfect geometry reconstruction in the surface aggregation function. The Reptile algorithm is used as part of a meta-learning framework to learn a good parameter initialization from multi-view datasets that allows for quickly converging to a representation for previously unseen objects. Evaluations suggest that this approach achieves a good balance between fast convergence on unseen objects, final image quality and fast rendering speed compared to recent alternatives including the original NLR.  Overall my impression is that the main contributions of this work are in the approach described to use more sophisticated shape and appearance representation compared to the original NLR, using image encoders and decoders and a more sophisticated weight-mapping function to achieve superior results while allowing for comparable inference speeds after training. My impression is that the meta-learning framework as described, which is largely based on the Reptile algorithm, is not as substantial a contribution.