GRF: Learning a General Radiance Field for 3D Scene Representation and Rendering

1 INTRODUCTION . Understanding the precise 3D structure of a real-world environment and realistically re-rendering it from free viewpoints is a key enabler for many critical tasks , ranging from robotic manipulation and navigation to augmented reality . Classic approaches to recover the 3D scene geometry mainly include the structure from motion ( SfM ) ( Ozyesil et al. , 2017 ) and simultaneous localization and mapping ( SLAM ) ( Cadena et al. , 2016 ) pipelines . However , they are limited to reconstruct sparse and discrete 3D point clouds which are unable to contain geometric details . The recent advances in deep neural networks have yielded rapid progress in 3D modeling and understanding . Most of them focus on the explicit 3D shape representations such as voxel grids ( Choy et al. , 2016 ) , point clouds ( Fan et al. , 2017 ) , or triangle meshes ( Wang et al. , 2018 ) . However , these representations are discrete and sparse , limiting the recovered 3D structures to extremely low spatial resolution . In addition , these networks usually require large-scale 3D shapes for supervision , resulting in the trained models being over-fitting particular datasets and unable to generalize to novel scenes . In fact , it is also costly and even infeasible to collect high-quality 3D labels . Encoding geometries into multilayer perceptrons ( MLPs ) ( Mescheder et al. , 2019 ; Park et al. , 2019 ) recently emerges as a promising direction in 3D reconstruction and understanding from 2D images . Its key advantage is the ability to model 3D structures in a continuous way instead of discrete , and therefore it has the potential to achieve unlimited spatial resolution in theory . However , most methods of this pipeline focus on individual objects . In addition , many of them require 3D geometry for supervision to learn the 3D shapes from images . By introducing a recurrent neural network based renderer , SRNs ( Sitzmann et al. , 2019 ) is among the early work to learn implicit surface representations only from 2D images , but it fails to capture complicated scene geometries and renders over-smoothed images . Alternatively , by leveraging the volume rendering to synthesize new views for 2D supervision , the very recent NeRF ( Mildenhall et al. , 2020 ) directly encodes the radiance fields of complex 3D scenes within the weights of MLPs , achieving an unprecedented level of fidelity for challenging 3D scenes . Nevertheless , it has two major limitations : 1 ) since each 3D scene is encoded into all the weights of MLPs , the trained network ( i.e. , a learned radiance field ) can only represent a single scene , and is unable to generalize across novel scenarios ; 2 ) because the shape and appearance of each spatial 3D location along a light ray is only optimized by the available pixel RGBs , the learned implicit representations of that location are lack of the general geometric patterns , resulting in the synthesized images being less photo-realistic . In this paper , we propose a general radiance field ( GRF ) , a simple yet powerful implicit function that can represent and render complex 3D scenes where there may be multiple objects with cluttered background . Our GRF takes a set of 2D images with camera poses and intrinsics , a 3D query point and its query viewpoint ( i.e. , the camera location xyz ) as input , and predicts the RGB value and volumetric density of that query point . Basically , this neural function learns to represent a 3D scene from sparse 2D observations , and infers the shape and appearance of that scene from previously unobserved viewing angles . Note that , the inferred shape and appearance of any particular 3D query point explicitly take into account its local geometric patterns from the available 2D observations . In particular , the proposed GRF consists of four components : • Extracting the general 2D visual features for every light ray from the input 2D observations ; • Reprojecting the corresponding 2D features back to the query 3D point using the principle of multi-view geometry ; • Selecting and aggregating all the reprojected features for the query 3D point , while the visual occlusions are implicitly considered ; • Rendering the aggregated features of the query 3D point along a particular query viewpoint , and producing the corresponding RGB and volumetric density . These four components enable our GRF to be significantly different from all existing 3D scene representation approaches . 1 ) Compared with the classic SfM/SLAM systems , our GRF can represent the 3D scene structure with smooth and continuous surfaces . 2 ) Compared with the neural approaches based on explicit voxel grids , point clouds and meshes , our GRF learns continuous 3D representations without requiring 3D data for training . 3 ) Compared with the existing implicit representation methods such as SDF ( Park et al. , 2019 ) , SRNs ( Sitzmann et al. , 2019 ) and NeRF ( Mildenhall et al. , 2020 ) , our GRF can represent arbitrarily complicated 3D scenes and has remarkable generalization to novel scenarios . In addition , the learned 3D representations carefully consider the general geometric patterns for every 3D spatial location , allowing the rendered views to be exceptionally realistic with fine-grained details . Our key contributions are : • We propose a general radiance field to implicitly represent the 3D scene structure and appearance purely from 2D images . It has remarkable generalization to novel scenes in a single forward pass . • We explicitly integrate the principle of multi-view geometry to learn geometric details for each 3D query point along every query light ray . This allows the synthesized 2D views to be superior . • We demonstrate significant improvement over baselines on three large-scale datasets and provide intuition behind our design choices through extensive ablation studies . 2 RELATED WORK . Classic Multi-view Geometry . Classic approaches to reconstruct 3D geometry from images mainly include SfM and SLAM systems , which firstly extract and match hand-crafted geometric local features and then apply bundle adjustment for both shape and camera motion estimation ( Hartley & Zisserman , 2004 ) . Although they can recover visually satisfactory 3D models , the reconstructed shapes are usually sparse , discrete point clouds . In contrast , our GRF learns an implicit function to represent the continuous 3D structures from images . Notably , however , the principle of classic multi-view geometry is explicitly integrated into our GRF to learn accurate and general features for every spatial location . Geometric Deep Learning . The recent advances of deep neural nets have yielded impressive progress in recovering explicit 3D shapes from either single or multiple images , including the voxel grid ( Choy et al. , 2016 ; Yang et al. , 2019 ) , octree ( Riegler et al. , 2017 ; Christian et al. , 2017 ) , point cloud ( Fan et al. , 2017 ; Qi et al. , 2017 ) and triangle mesh ( Wang et al. , 2018 ; Groueix et al. , 2018 ; Nash et al. , 2020 ) based approaches . However , most of these methods only focus on individual 3D objects , whilst only few pipelines ( Song et al. , 2017 ; Tulsiani et al. , 2018 ; Gkioxari et al. , 2019 ) attempt to learn the structures for complex 3D scenes . Although these neural networks can predict realistic 3D structures of objects and scenes , they have two limitations . First , almost all of them require ground truth 3D labels to supervise the networks , resulting in the learned representations being unable to generalize to novel real-world scenes . Second , since the recovered 3D shapes are discrete , they are unable to preserve high-resolution geometric details . Being quite different , our GRF learns to infer continuous 3D shapes only from a set of 2D images with camera poses and intrinsics , which can be cheaply acquired and also allow better generalization across real-world scenes . Neural Implicit 3D Representations . The implicit representation of 3D shapes recently emerges as a promising direction to recover 3D geometries . It is initially formulated as level sets by optimizing neural nets which map xyz locations to an occupancy field ( Mescheder et al. , 2019 ; Saito et al. , 2019 ) or a distance function ( Park et al. , 2019 ) . The subsequent work ( Niemeyer et al. , 2020 ; Sitzmann et al. , 2019 ; Liu et al. , 2019b ; Atzmon & Lipman , 2020a ; b ) introduce differentiable rendering functions , allowing 2D images or raw 3D point clouds to supervise the networks , but only producing decent results on individual objects . Using neural radiance fields instead , the latest NeRF ( Mildenhall et al. , 2020 ) and the succeeding NSVF ( Liu et al. , 2020 ) , NeRF-wild ( Martin-Brualla et al. , 2020 ) , GRAF ( Schwarz et al. , 2020 ) demonstrate impressive results to represent complex 3D scenes . However , they do not take into account the local geometric patterns for spatial locations , thereby the rendered 2D images being less realistic and sub-optimal . Additionally , both NeRF and NeRF-wild can only represent a single scene , and are unable to generalize to novel scenarios . Uniquely , our GRF maps any set of images to the corresponding 3D scene structure with geometric details . Novel View Synthesis and Neural Rendering . Novel view synthesis involves generating unseen views of a scene captured by multiple images . Existing methods usually learn an embedding of a scene and then estimate a new image given a viewing angle , including GAN based methods ( Goodfellow et al. , 2014 ; Radford et al. , 2016 ) , variational auto-encoders ( Kingma & Welling , 2014 ) , autoregressive models ( Oord et al. , 2016 ) , and other generative frameworks ( Eslami et al. , 2018 ) . Although photo-realistic single images can be generated , these methods tend to learn the manifold of 2D images , instead of exploiting the underlying 3D geometry for consistent multi-view synthesis . Neural rendering techniques ( Fried et al. , 2020 ; Kato et al. , 2020 ) have recently been actively investigated and integrated into 3D reconstruction pipelines , where there are no ground truth 3D data available but only 2D images for supervision . To render the discrete voxel grids ( Yan et al. , 2016 ; Rezende et al. , 2016 ; Tulsiani et al. , 2017 ) , point clouds ( Insafutdinov & Dosovitskiy , 2018 ) , meshes ( Kato et al. , 2018 ; Chen et al. , 2019 ; Liu et al. , 2019a ) , and implicit surfaces ( Liu et al. , 2019b ; Remelli et al. , 2020 ) , most of these techniques are designed with differentiable and approximate functions , but sacrifice the sharpness of synthesized images and the accuracy of estimated 3D shapes . By contrast , our GRF leverages the existing volume rendering method ( Mildenhall et al. , 2020 ) which is naturally differentiable and can accurately render the RGB per light ray . 3 GRF . 3.1 OVERVIEW . Our GRF models the complex 3D scenes including the geometry and appearance as a neural function fW where W represent learnable parameters . This function takes a set of M images together with their camera poses and intrinsics { ( I1 , ξ1 , K1 ) · · · ( Im , ξm , Km ) · · · ( IM , ξM , KM ) } , which are sampled from a complex 3D scene , a query 3D point p = { xp , yp , zp } , and its query viewpoint Vp = { xvp , yvp , zvp } as input , and then predicts the RGB value { rp , gp , bp } and the volumetric density dp of that query 3D point p observed from the query viewpoint Vp . Formally , it is defined as below : ( rp , gp , bp , dp ) = fW ( { ( I1 , ξ1 , K1 ) · · · ( IM , ξM , KM ) } , { xp , yp , zp } , { xvp , y v p , z v p } ) ( 1 ) Basically , this function is a general radiance field ( GRF ) which parameterizes arbitrary 3D scenes observed by the input M images . In the meantime , it returns both the appearance and geometry , when being queried at any location p from any viewpoint V in 3D space . As illustrated in Figure 1 , the proposed GRF firstly extracts general features for each light ray hitting through every pixel , and then reprojects those features back to the query 3D point p. After that , the corresponding RGB value and volumetric density of p are inferred from those features given a specific query viewpoint . This simple design of GRF follows the principle of classic multi-view geometry ( Hartley & Zisserman , 2004 ) , therefore driving the learned implicit representations to be multi-view consistent . In particular , for any query 3D point in space , its features are always the same when being viewing from different angles . Hence , its rendered RGBs and volumetric densities Under review as a conference paper at ICLR 2021 Query Point p Query Viewpoint Vp Implicit 3D Scene Representation Observed Image I1 Observed Image IK Viewp oint V1 Viewpoint V K are more likely to be consistent from different viewpoints . To realize our neural function fW , we introduce four components : 1 ) A feature extractor for every 2D pixel ; 2 ) A reprojector to transform 2D features to 3D space ; 3 ) An aggregator to obtain general features for a 3D point ; and 4 ) A neural renderer to infer the appearance and geometry for that 3D point . All these components are connected and trained end-to-end with arbitrary scenes . 3.2 EXTRACTING GENERAL FEATURES FOR 2D PIXELS The whole input set of observed 2D images together describes the geometry and appearance of a 3D scene , while each pixel of those images describes a specific point of that scene . This module aims to extract the general features of each pixel , so to learn the regional description and geometric patterns for each light ray . A naive approach is to directly use the raw rgb values as the pixel features . However , this is sub-optimal because the raw rgb values are sensitive to lighting conditions , environmental noise , etc . In order to learn more general and robust patterns for each pixel , we turn to use a more powerful encoder-decoder based convolutional neural network ( CNN ) . As shown in Figure 2 , our CNN module is designed with the follow two features : • Instead of directly feeding raw RGB images into the CNN module , we stack ( duplicate ) the corresponding viewpoint , i.e. , the xyz location of the camera , to each pixel of the image . This allows the learned pixel features to be explicitly aware of its relative position in the 3D scene space . Note that , we empirically find that stacking the additional camera rotation and intrinsics to each pixel does not noticeably improve the performance , especially for scenes centered at the origin . • We use skip connections between the encoder and decoder to preserve high frequency local features for each pixel , while optionally integrating a couple of fully connected ( fc ) layers in the middle of the CNN module to learn global features . The mixture of hierarchical features tends to to be more general and representative , effectively aiding the network in practice . Details of the CNN module is presented in the appendix and all input images share the same CNN module . Note that , there are many ways to construct a pixel feature extractor , but identifying an optimal CNN module here is out the scope of this paper .

Summary: + This paper proposes a method for synthesizing 3D scenes in novel view by treating the network as a general radiance field. It seems that the method is more like traditional manner based on multiple view geometry. It enables the generalization ability of rendering unseen test data in contrast to the most related work, NeRF (Mildenhall et al., 2020).