GRF: Learning a General Radiance Field for 3D Scene Representation and Rendering

1 INTRODUCTION . Understanding the precise 3D structure of a real-world environment and realistically re-rendering it from free viewpoints is a key enabler for many critical tasks , ranging from robotic manipulation and navigation to augmented reality . Classic approaches to recover the 3D scene geometry mainly include the structure from motion ( SfM ) ( Ozyesil et al. , 2017 ) and simultaneous localization and mapping ( SLAM ) ( Cadena et al. , 2016 ) pipelines . However , they are limited to reconstruct sparse and discrete 3D point clouds which are unable to contain geometric details . The recent advances in deep neural networks have yielded rapid progress in 3D modeling and understanding . Most of them focus on the explicit 3D shape representations such as voxel grids ( Choy et al. , 2016 ) , point clouds ( Fan et al. , 2017 ) , or triangle meshes ( Wang et al. , 2018 ) . However , these representations are discrete and sparse , limiting the recovered 3D structures to extremely low spatial resolution . In addition , these networks usually require large-scale 3D shapes for supervision , resulting in the trained models being over-fitting particular datasets and unable to generalize to novel scenes . In fact , it is also costly and even infeasible to collect high-quality 3D labels . Encoding geometries into multilayer perceptrons ( MLPs ) ( Mescheder et al. , 2019 ; Park et al. , 2019 ) recently emerges as a promising direction in 3D reconstruction and understanding from 2D images . Its key advantage is the ability to model 3D structures in a continuous way instead of discrete , and therefore it has the potential to achieve unlimited spatial resolution in theory . However , most methods of this pipeline focus on individual objects . In addition , many of them require 3D geometry for supervision to learn the 3D shapes from images . By introducing a recurrent neural network based renderer , SRNs ( Sitzmann et al. , 2019 ) is among the early work to learn implicit surface representations only from 2D images , but it fails to capture complicated scene geometries and renders over-smoothed images . Alternatively , by leveraging the volume rendering to synthesize new views for 2D supervision , the very recent NeRF ( Mildenhall et al. , 2020 ) directly encodes the radiance fields of complex 3D scenes within the weights of MLPs , achieving an unprecedented level of fidelity for challenging 3D scenes . Nevertheless , it has two major limitations : 1 ) since each 3D scene is encoded into all the weights of MLPs , the trained network ( i.e. , a learned radiance field ) can only represent a single scene , and is unable to generalize across novel scenarios ; 2 ) because the shape and appearance of each spatial 3D location along a light ray is only optimized by the available pixel RGBs , the learned implicit representations of that location are lack of the general geometric patterns , resulting in the synthesized images being less photo-realistic . In this paper , we propose a general radiance field ( GRF ) , a simple yet powerful implicit function that can represent and render complex 3D scenes where there may be multiple objects with cluttered background . Our GRF takes a set of 2D images with camera poses and intrinsics , a 3D query point and its query viewpoint ( i.e. , the camera location xyz ) as input , and predicts the RGB value and volumetric density of that query point . Basically , this neural function learns to represent a 3D scene from sparse 2D observations , and infers the shape and appearance of that scene from previously unobserved viewing angles . Note that , the inferred shape and appearance of any particular 3D query point explicitly take into account its local geometric patterns from the available 2D observations . In particular , the proposed GRF consists of four components : • Extracting the general 2D visual features for every light ray from the input 2D observations ; • Reprojecting the corresponding 2D features back to the query 3D point using the principle of multi-view geometry ; • Selecting and aggregating all the reprojected features for the query 3D point , while the visual occlusions are implicitly considered ; • Rendering the aggregated features of the query 3D point along a particular query viewpoint , and producing the corresponding RGB and volumetric density . These four components enable our GRF to be significantly different from all existing 3D scene representation approaches . 1 ) Compared with the classic SfM/SLAM systems , our GRF can represent the 3D scene structure with smooth and continuous surfaces . 2 ) Compared with the neural approaches based on explicit voxel grids , point clouds and meshes , our GRF learns continuous 3D representations without requiring 3D data for training . 3 ) Compared with the existing implicit representation methods such as SDF ( Park et al. , 2019 ) , SRNs ( Sitzmann et al. , 2019 ) and NeRF ( Mildenhall et al. , 2020 ) , our GRF can represent arbitrarily complicated 3D scenes and has remarkable generalization to novel scenarios . In addition , the learned 3D representations carefully consider the general geometric patterns for every 3D spatial location , allowing the rendered views to be exceptionally realistic with fine-grained details . Our key contributions are : • We propose a general radiance field to implicitly represent the 3D scene structure and appearance purely from 2D images . It has remarkable generalization to novel scenes in a single forward pass . • We explicitly integrate the principle of multi-view geometry to learn geometric details for each 3D query point along every query light ray . This allows the synthesized 2D views to be superior . • We demonstrate significant improvement over baselines on three large-scale datasets and provide intuition behind our design choices through extensive ablation studies . 2 RELATED WORK . Classic Multi-view Geometry . Classic approaches to reconstruct 3D geometry from images mainly include SfM and SLAM systems , which firstly extract and match hand-crafted geometric local features and then apply bundle adjustment for both shape and camera motion estimation ( Hartley & Zisserman , 2004 ) . Although they can recover visually satisfactory 3D models , the reconstructed shapes are usually sparse , discrete point clouds . In contrast , our GRF learns an implicit function to represent the continuous 3D structures from images . Notably , however , the principle of classic multi-view geometry is explicitly integrated into our GRF to learn accurate and general features for every spatial location . Geometric Deep Learning . The recent advances of deep neural nets have yielded impressive progress in recovering explicit 3D shapes from either single or multiple images , including the voxel grid ( Choy et al. , 2016 ; Yang et al. , 2019 ) , octree ( Riegler et al. , 2017 ; Christian et al. , 2017 ) , point cloud ( Fan et al. , 2017 ; Qi et al. , 2017 ) and triangle mesh ( Wang et al. , 2018 ; Groueix et al. , 2018 ; Nash et al. , 2020 ) based approaches . However , most of these methods only focus on individual 3D objects , whilst only few pipelines ( Song et al. , 2017 ; Tulsiani et al. , 2018 ; Gkioxari et al. , 2019 ) attempt to learn the structures for complex 3D scenes . Although these neural networks can predict realistic 3D structures of objects and scenes , they have two limitations . First , almost all of them require ground truth 3D labels to supervise the networks , resulting in the learned representations being unable to generalize to novel real-world scenes . Second , since the recovered 3D shapes are discrete , they are unable to preserve high-resolution geometric details . Being quite different , our GRF learns to infer continuous 3D shapes only from a set of 2D images with camera poses and intrinsics , which can be cheaply acquired and also allow better generalization across real-world scenes . Neural Implicit 3D Representations . The implicit representation of 3D shapes recently emerges as a promising direction to recover 3D geometries . It is initially formulated as level sets by optimizing neural nets which map xyz locations to an occupancy field ( Mescheder et al. , 2019 ; Saito et al. , 2019 ) or a distance function ( Park et al. , 2019 ) . The subsequent work ( Niemeyer et al. , 2020 ; Sitzmann et al. , 2019 ; Liu et al. , 2019b ; Atzmon & Lipman , 2020a ; b ) introduce differentiable rendering functions , allowing 2D images or raw 3D point clouds to supervise the networks , but only producing decent results on individual objects . Using neural radiance fields instead , the latest NeRF ( Mildenhall et al. , 2020 ) and the succeeding NSVF ( Liu et al. , 2020 ) , NeRF-wild ( Martin-Brualla et al. , 2020 ) , GRAF ( Schwarz et al. , 2020 ) demonstrate impressive results to represent complex 3D scenes . However , they do not take into account the local geometric patterns for spatial locations , thereby the rendered 2D images being less realistic and sub-optimal . Additionally , both NeRF and NeRF-wild can only represent a single scene , and are unable to generalize to novel scenarios . Uniquely , our GRF maps any set of images to the corresponding 3D scene structure with geometric details . Novel View Synthesis and Neural Rendering . Novel view synthesis involves generating unseen views of a scene captured by multiple images . Existing methods usually learn an embedding of a scene and then estimate a new image given a viewing angle , including GAN based methods ( Goodfellow et al. , 2014 ; Radford et al. , 2016 ) , variational auto-encoders ( Kingma & Welling , 2014 ) , autoregressive models ( Oord et al. , 2016 ) , and other generative frameworks ( Eslami et al. , 2018 ) . Although photo-realistic single images can be generated , these methods tend to learn the manifold of 2D images , instead of exploiting the underlying 3D geometry for consistent multi-view synthesis . Neural rendering techniques ( Fried et al. , 2020 ; Kato et al. , 2020 ) have recently been actively investigated and integrated into 3D reconstruction pipelines , where there are no ground truth 3D data available but only 2D images for supervision . To render the discrete voxel grids ( Yan et al. , 2016 ; Rezende et al. , 2016 ; Tulsiani et al. , 2017 ) , point clouds ( Insafutdinov & Dosovitskiy , 2018 ) , meshes ( Kato et al. , 2018 ; Chen et al. , 2019 ; Liu et al. , 2019a ) , and implicit surfaces ( Liu et al. , 2019b ; Remelli et al. , 2020 ) , most of these techniques are designed with differentiable and approximate functions , but sacrifice the sharpness of synthesized images and the accuracy of estimated 3D shapes . By contrast , our GRF leverages the existing volume rendering method ( Mildenhall et al. , 2020 ) which is naturally differentiable and can accurately render the RGB per light ray . 3 GRF . 3.1 OVERVIEW . Our GRF models the complex 3D scenes including the geometry and appearance as a neural function fW where W represent learnable parameters . This function takes a set of M images together with their camera poses and intrinsics { ( I1 , ξ1 , K1 ) · · · ( Im , ξm , Km ) · · · ( IM , ξM , KM ) } , which are sampled from a complex 3D scene , a query 3D point p = { xp , yp , zp } , and its query viewpoint Vp = { xvp , yvp , zvp } as input , and then predicts the RGB value { rp , gp , bp } and the volumetric density dp of that query 3D point p observed from the query viewpoint Vp . Formally , it is defined as below : ( rp , gp , bp , dp ) = fW ( { ( I1 , ξ1 , K1 ) · · · ( IM , ξM , KM ) } , { xp , yp , zp } , { xvp , y v p , z v p } ) ( 1 ) Basically , this function is a general radiance field ( GRF ) which parameterizes arbitrary 3D scenes observed by the input M images . In the meantime , it returns both the appearance and geometry , when being queried at any location p from any viewpoint V in 3D space . As illustrated in Figure 1 , the proposed GRF firstly extracts general features for each light ray hitting through every pixel , and then reprojects those features back to the query 3D point p. After that , the corresponding RGB value and volumetric density of p are inferred from those features given a specific query viewpoint . This simple design of GRF follows the principle of classic multi-view geometry ( Hartley & Zisserman , 2004 ) , therefore driving the learned implicit representations to be multi-view consistent . In particular , for any query 3D point in space , its features are always the same when being viewing from different angles . Hence , its rendered RGBs and volumetric densities Under review as a conference paper at ICLR 2021 Query Point p Query Viewpoint Vp Implicit 3D Scene Representation Observed Image I1 Observed Image IK Viewp oint V1 Viewpoint V K are more likely to be consistent from different viewpoints . To realize our neural function fW , we introduce four components : 1 ) A feature extractor for every 2D pixel ; 2 ) A reprojector to transform 2D features to 3D space ; 3 ) An aggregator to obtain general features for a 3D point ; and 4 ) A neural renderer to infer the appearance and geometry for that 3D point . All these components are connected and trained end-to-end with arbitrary scenes . 3.2 EXTRACTING GENERAL FEATURES FOR 2D PIXELS The whole input set of observed 2D images together describes the geometry and appearance of a 3D scene , while each pixel of those images describes a specific point of that scene . This module aims to extract the general features of each pixel , so to learn the regional description and geometric patterns for each light ray . A naive approach is to directly use the raw rgb values as the pixel features . However , this is sub-optimal because the raw rgb values are sensitive to lighting conditions , environmental noise , etc . In order to learn more general and robust patterns for each pixel , we turn to use a more powerful encoder-decoder based convolutional neural network ( CNN ) . As shown in Figure 2 , our CNN module is designed with the follow two features : • Instead of directly feeding raw RGB images into the CNN module , we stack ( duplicate ) the corresponding viewpoint , i.e. , the xyz location of the camera , to each pixel of the image . This allows the learned pixel features to be explicitly aware of its relative position in the 3D scene space . Note that , we empirically find that stacking the additional camera rotation and intrinsics to each pixel does not noticeably improve the performance , especially for scenes centered at the origin . • We use skip connections between the encoder and decoder to preserve high frequency local features for each pixel , while optionally integrating a couple of fully connected ( fc ) layers in the middle of the CNN module to learn global features . The mixture of hierarchical features tends to to be more general and representative , effectively aiding the network in practice . Details of the CNN module is presented in the appendix and all input images share the same CNN module . Note that , there are many ways to construct a pixel feature extractor , but identifying an optimal CNN module here is out the scope of this paper .

Summary: This paper presents a new neural learner for learning generic scene radiance function. Compared with existing methods such as NeRf which are scene-specific requiring per scene based training, the proposed new method is potentially able to generalize to novel unseen scenes or objects.   Specifically, the method represents scene as a neural radiance filed in terms of spatial coordinates of camera centre and a query point. An attention model is used for aggregating back-projected image features at every query location, and the aggregated feature vectors are then used for predicting its RGB-alpha radiance.