OctField: Hierarchical Implicit Functions for 3D Modeling

1 Introduction . Geometric 3D representation has been central to the tasks in computer vision and computer graphics , ranging from high-level applications , such as scene understanding , object recognition and classification , etc , to low-level tasks , including 3D shape reconstruction , interpolation and manipulation . To accommodate with various application scenarios , a universal and effective 3D representation for 3D deep learning should have the following properties : ( 1 ) compatibility with arbitrary topologies , ( 2 ) capacity of modeling fine geometric details , ( 3 ) scalability to intricate shapes , ( 4 ) support efficient encoding of shape priors , ( 5 ) compact memory footprint , and ( 6 ) high computational efficiency . While explicit 3D representations have been widely used in recent 3D learning approaches , none of these representations can fulfill all the desirable properties . In particular , point cloud and voxel representations struggle to capture the fine-scale shape details – often at the cost of high memory ∗Contributed equally . †Corresponding author is Lin Gao ( gaolin @ ict.ac.cn ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . consumption . Mesh-based learning approaches typically rely on deforming a template model , limiting its scalability to handle arbitrary topologies . The advent of neural implicit function [ 44 , 7 , 37 ] have recently brought impressive advances to the state-of-the-art across a range of 3D modeling and reconstruction tasks . However , using only a global function for encoding the entirety of all shapes , the aforementioned methods often suffer from limited reconstruction accuracy and shape generality . To overcome these limitations , follow-up works have proposed to decompose the 3D space into regular grid [ 27 , 4 ] , or local supporting regions [ 19 ] , where each subdivided shape is approximated by a locally learned implicit function . The decomposition of scenes simplifies the shape priors that each local network has to learn , leading to higher reconstruction accuracy and efficiency . However , these approaches do not take into account the varying granularities of local geometry , resulting in two major shortcomings . Efficiency-wise , their memory usage grows cubically with the volume of the 3D scenes . Even a moderately dense decomposition could impose severe memory bottleneck . Scalability-wise , the regular gridding has difficulty scaling to high resolutions , limiting its expressiveness when dealing with intricate shapes with small and sharp geometric features ( Figure 4 ) . We observe that most 3D shapes are typically consisting of large smooth regions and smallscale sharp features . In addition , the surface of interest often consumes only a small portion of the entire space , leading to an extremely sparse space occupancy . Based on these observations , we propose a novel 3D representation called OctField , that introduces hierarchies to the organization of local implicit functions to achieve better memory efficiency and stronger modeling capacity . As shown in Figure 1 , OctField leverages a hierarchical data structure , Octree , to adaptively subdivide the 3D space according to the surface occupancy and the richness of geometrical details . In particular , regions enclosing intricate geometries will be further subdivided to allocate more implicit kernels for higher modeling accuracy . In contrast , we stop subdivision for octants containing smooth part geometry as a single implicit kernel would suffice for modeling . Further , we do not allocate any implicit functions in the unoccupied regions . Hence , OctField could obtain significantly higher representation accuracy with a slightly deeper octree subdivision , as the modeling capacity has been adaptively optimized to accommodate the varying granularity of surface details . As the octree structure is discrete and non-differentiable , it is non-trivial to directly employ octree in a learning-based framework . We propose a novel hierarchical network that recursively encodes and decodes both octree structure and geometry features in a differentiable manner . Specifically , at the decoder side , we model the subdivision of octree cells as a probabilistic process to make the learning of octree structure differentiable . We employ a classifier to predict whether to subdivide current cell based on its enclosed geometry features . We validate the efficacy of our new representation in a variety of tasks on shape reconstruction and modeling . Experimental results demonstrate the superiority of OctField over the state-of-the-art shape representations in terms of both shape quality and memory efficiency . Our contributions can be summarized as follows : • A learnable hierarchical implicit representation for 3D learning , named OctField , that combines the state-of-the-art hierarchical data structure with local implicit functions . • A novel hierarchical encoder-decoder network that models the construction of octree as a probabilistic process and is able to learn both discrete octree structure and surface geometry in a differentiable manner . • We achieve significantly higher surface approximation accuracy with reduced memory cost in the 3D modeling related tasks by using our proposed representation . 2 Related Works . Representations for 3D Shape Learning . Various 3D representations have been extensively studied in 3D deep learning [ 28 ] . These surveys [ 2 , 62 ] discuss various shape representations comprehensively . As the raw output of 3D scanning devices , point cloud [ 46 , 47 , 65 , 49 , 53 ] has received much attention in recent years . Despite for its simplicity , generating dense point clouds with high precision remains notoriously difficult . Unlike the other 3D representations , the convolutional network can be directly employed on 3D voxels [ 36 , 11 , 20 , 59 , 60 , 61 , 64 , 23 , 10 ] . Due to the prohibitive computational cost of generating voxels , recent works [ 55 , 57 , 56 , 52 , 32 ] have introduced octree to the voxel representation to reduce memory cost . The polygon mesh is the another widely used representation in modeling and surface reconstruction . However , current learning-based mesh generation approaches [ 6 , 54 , 50 , 24 , 29 , 21 , 12 , 43 ] mostly rely on deforming a template mesh , limiting its scalability to shapes with arbitrary topologies . Recent advances in neural implicit functions [ 44 , 37 , 7 , 27 , 14 , 63 , 8 , 45 ] have significantly improved the surface reconstruction accuracy thanks to its flexibility of handling arbitrary topologies . More recently , [ 27 , 4 , 19 ] have introduced shape decomposition and local implicit functions to further improve the modeling capacity by locally approximating part geometry . [ 33 ] introduces implicit moving least-squares ( IMLS ) surface formulation on discrete point-set to reconstruct high-quality surfaces . However , these methods mostly rely on a regular decomposition and can not account for sparse surface occupancy and the varying granularities of geometry details , imposing memory bottleneck when dealing with moderately dense subdivision . OpenVDB [ 42 ] incorporates B+ tree with implicit field to achieve hierarchical modeling . However , the goal of OpenVDB is pursuing extremely fast modeling speed with constant time access in 3D simulation . Hence , the B+ tree is non-differentiable and too complex to be incorporated into a learning-based framework . In contrast , OctField is a learnable hierarchical implicit representation that can be differentiably implemented in our hierarchical network . Further , our representation can achieve higher modeling accuracy with even less memory compared to the previous local implicit function approaches [ 27 , 4 , 19 ] . In a concurrent work NGLOD [ 51 ] , it proposes a similar idea of leveraging level of details to encode local SDFs hierarchically . A corresponding rendering algorithm is proposed to render the neural SDFs in an interactive rate . However , NGLOD can not learn the hierarchical structure of the underlying octree . In contrast , our hierarchical encoder-decoder network learns the non-differentiable structural information in differentiable manner via modeling it as a probabilistic process . We believe the structural information is crucial for improving the modeling accuracy and future applications ( e.g . 3D semantic understanding , editing ) . Learning-based Generative Models . The deep generative models , such as GAN [ 22 ] and VAE [ 30 ] , have shown promising ability of synthesizing realistic images in 2D domain . 3D learning approaches have strived to duplicate the success of 2D generative models into 3D shape generation . 3D-GAN [ 60 ] pioneers at applying the GAN technology on the 3D voxels to learn a deep generator that can synthesize various 3D shapes . Generative models on point cloud [ 1 , 15 , 67 ] mainly leverage MLP layers but struggle to generate dense point sets with high fidelity due to the large memory consumption and high computational complexity . Recent works on synthesizing 3D meshes either rely on deforming an initial mesh using graph CNN [ 54 , 58 ] or assembling surface patches [ 24 , 13 ] to achieve more flexible structure manipulation . To better model man-made objects composed of hierarchical regular shapes , structural relationship has been considered in [ 18 , 66 , 17 , 38 , 40 , 39 ] , where the box-like primitives are used for initial shape to enhance shape regularity . To fully exploit the modeling capacity of implicit surface generator , IM-Net [ 7 ] has experimented with both VAE and GAN models to learn stronger shape priors . In a concurrent work , DeepSDF [ 44 ] proposes a auto-decoder structure to train latent space and decoder without using a traditional shape encoder . Recently , the local implicit methods [ 27 , 4 ] combine regular space decomposition with local implicit generators for modeling 3D scenes with fine geometric details . ACORN [ 35 ] introduces an adaptive multiscale neural scene representation for 2D and 3D complex scenes , which enables to fit the targets faster and better in an optimized multiscale fashion . In our paper , we propose a hierarchical implicit generative model for 3D modeling . Compared to other methods , our approach can generate high-quality 3D surfaces with intricate geometric details in a memory-efficient manner . 3 Method . Overview . OctField combines the good ends of both localized implicit representation and the hierarchical data structure . By adaptively allocating local implicit functions according to the surface occupancy and the richness of geometry , OctField is able to achieve high modeling accuracy with a low memory and computation budget . In particular , we decompose the 3D space into hierarchical local regions using octree structure , where the finest octant encodes the partial shape within its enclosed space using a learned implicit function . Our decomposition protocol not only considers the surface occupancy but also the richness of geometry . As show in Figure 2 , the octants that carry an embedded implicit kernel will only be allocated around the surface . Moreover , only the octants containing intricate geometries will be further divided . This ensures an adaptive memory and computation allocation that the richer surface details will be captured with more local implicit functions – hence with higher modeling accuracy . In contrast , the unoccupied regions will not be allocated with any implicit kernels to save the memory and computational budget . The octree itself is a non-differentiable discrete data structure . We propose a novel differentiable hierarchical encoder-decoder network that learns both the octree structure and the geometry features simultaneously . In particular , we formulate the construction of octree as a probabilistic process where the probability of subdividing an octant is predicted by a MLP layer . This makes it possible to learn discrete octree structure in a fully differentiable manner . In addition , we train our network in a VAE manner such that the trained latent space and decoder can be used for a variety of downstream applications including shape reconstruction , generation , interpolation , single-view reconstruction , etc . We provide detailed description of the OctField representation and the proposed network in Section 3.1 and 3.2 respectively .

Summary: This paper presents a new efficient 3D shape representation to reduce computation cost while also being able to preserve geometric details. The authors propose to represent 3D shapes using octree structures such that the hierarchies focus on modeling the surface geometry, where the leaf nodes additionally predicts implicit functions that returns the occupancy for each query 3D point. Both the hierarchical inference and prediction are designed to be differentiable such that the octree structures can be discovered and learned from data. Experiments show that the proposed representation achieves the state of the art on the ShapeNet reconstruction with applications of shape interpolation and completion.