OctField: Hierarchical Implicit Functions for 3D Modeling

1 Introduction . Geometric 3D representation has been central to the tasks in computer vision and computer graphics , ranging from high-level applications , such as scene understanding , object recognition and classification , etc , to low-level tasks , including 3D shape reconstruction , interpolation and manipulation . To accommodate with various application scenarios , a universal and effective 3D representation for 3D deep learning should have the following properties : ( 1 ) compatibility with arbitrary topologies , ( 2 ) capacity of modeling fine geometric details , ( 3 ) scalability to intricate shapes , ( 4 ) support efficient encoding of shape priors , ( 5 ) compact memory footprint , and ( 6 ) high computational efficiency . While explicit 3D representations have been widely used in recent 3D learning approaches , none of these representations can fulfill all the desirable properties . In particular , point cloud and voxel representations struggle to capture the fine-scale shape details – often at the cost of high memory ∗Contributed equally . †Corresponding author is Lin Gao ( gaolin @ ict.ac.cn ) . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) , virtual . consumption . Mesh-based learning approaches typically rely on deforming a template model , limiting its scalability to handle arbitrary topologies . The advent of neural implicit function [ 44 , 7 , 37 ] have recently brought impressive advances to the state-of-the-art across a range of 3D modeling and reconstruction tasks . However , using only a global function for encoding the entirety of all shapes , the aforementioned methods often suffer from limited reconstruction accuracy and shape generality . To overcome these limitations , follow-up works have proposed to decompose the 3D space into regular grid [ 27 , 4 ] , or local supporting regions [ 19 ] , where each subdivided shape is approximated by a locally learned implicit function . The decomposition of scenes simplifies the shape priors that each local network has to learn , leading to higher reconstruction accuracy and efficiency . However , these approaches do not take into account the varying granularities of local geometry , resulting in two major shortcomings . Efficiency-wise , their memory usage grows cubically with the volume of the 3D scenes . Even a moderately dense decomposition could impose severe memory bottleneck . Scalability-wise , the regular gridding has difficulty scaling to high resolutions , limiting its expressiveness when dealing with intricate shapes with small and sharp geometric features ( Figure 4 ) . We observe that most 3D shapes are typically consisting of large smooth regions and smallscale sharp features . In addition , the surface of interest often consumes only a small portion of the entire space , leading to an extremely sparse space occupancy . Based on these observations , we propose a novel 3D representation called OctField , that introduces hierarchies to the organization of local implicit functions to achieve better memory efficiency and stronger modeling capacity . As shown in Figure 1 , OctField leverages a hierarchical data structure , Octree , to adaptively subdivide the 3D space according to the surface occupancy and the richness of geometrical details . In particular , regions enclosing intricate geometries will be further subdivided to allocate more implicit kernels for higher modeling accuracy . In contrast , we stop subdivision for octants containing smooth part geometry as a single implicit kernel would suffice for modeling . Further , we do not allocate any implicit functions in the unoccupied regions . Hence , OctField could obtain significantly higher representation accuracy with a slightly deeper octree subdivision , as the modeling capacity has been adaptively optimized to accommodate the varying granularity of surface details . As the octree structure is discrete and non-differentiable , it is non-trivial to directly employ octree in a learning-based framework . We propose a novel hierarchical network that recursively encodes and decodes both octree structure and geometry features in a differentiable manner . Specifically , at the decoder side , we model the subdivision of octree cells as a probabilistic process to make the learning of octree structure differentiable . We employ a classifier to predict whether to subdivide current cell based on its enclosed geometry features . We validate the efficacy of our new representation in a variety of tasks on shape reconstruction and modeling . Experimental results demonstrate the superiority of OctField over the state-of-the-art shape representations in terms of both shape quality and memory efficiency . Our contributions can be summarized as follows : • A learnable hierarchical implicit representation for 3D learning , named OctField , that combines the state-of-the-art hierarchical data structure with local implicit functions . • A novel hierarchical encoder-decoder network that models the construction of octree as a probabilistic process and is able to learn both discrete octree structure and surface geometry in a differentiable manner . • We achieve significantly higher surface approximation accuracy with reduced memory cost in the 3D modeling related tasks by using our proposed representation . 2 Related Works . Representations for 3D Shape Learning . Various 3D representations have been extensively studied in 3D deep learning [ 28 ] . These surveys [ 2 , 62 ] discuss various shape representations comprehensively . As the raw output of 3D scanning devices , point cloud [ 46 , 47 , 65 , 49 , 53 ] has received much attention in recent years . Despite for its simplicity , generating dense point clouds with high precision remains notoriously difficult . Unlike the other 3D representations , the convolutional network can be directly employed on 3D voxels [ 36 , 11 , 20 , 59 , 60 , 61 , 64 , 23 , 10 ] . Due to the prohibitive computational cost of generating voxels , recent works [ 55 , 57 , 56 , 52 , 32 ] have introduced octree to the voxel representation to reduce memory cost . The polygon mesh is the another widely used representation in modeling and surface reconstruction . However , current learning-based mesh generation approaches [ 6 , 54 , 50 , 24 , 29 , 21 , 12 , 43 ] mostly rely on deforming a template mesh , limiting its scalability to shapes with arbitrary topologies . Recent advances in neural implicit functions [ 44 , 37 , 7 , 27 , 14 , 63 , 8 , 45 ] have significantly improved the surface reconstruction accuracy thanks to its flexibility of handling arbitrary topologies . More recently , [ 27 , 4 , 19 ] have introduced shape decomposition and local implicit functions to further improve the modeling capacity by locally approximating part geometry . [ 33 ] introduces implicit moving least-squares ( IMLS ) surface formulation on discrete point-set to reconstruct high-quality surfaces . However , these methods mostly rely on a regular decomposition and can not account for sparse surface occupancy and the varying granularities of geometry details , imposing memory bottleneck when dealing with moderately dense subdivision . OpenVDB [ 42 ] incorporates B+ tree with implicit field to achieve hierarchical modeling . However , the goal of OpenVDB is pursuing extremely fast modeling speed with constant time access in 3D simulation . Hence , the B+ tree is non-differentiable and too complex to be incorporated into a learning-based framework . In contrast , OctField is a learnable hierarchical implicit representation that can be differentiably implemented in our hierarchical network . Further , our representation can achieve higher modeling accuracy with even less memory compared to the previous local implicit function approaches [ 27 , 4 , 19 ] . In a concurrent work NGLOD [ 51 ] , it proposes a similar idea of leveraging level of details to encode local SDFs hierarchically . A corresponding rendering algorithm is proposed to render the neural SDFs in an interactive rate . However , NGLOD can not learn the hierarchical structure of the underlying octree . In contrast , our hierarchical encoder-decoder network learns the non-differentiable structural information in differentiable manner via modeling it as a probabilistic process . We believe the structural information is crucial for improving the modeling accuracy and future applications ( e.g . 3D semantic understanding , editing ) . Learning-based Generative Models . The deep generative models , such as GAN [ 22 ] and VAE [ 30 ] , have shown promising ability of synthesizing realistic images in 2D domain . 3D learning approaches have strived to duplicate the success of 2D generative models into 3D shape generation . 3D-GAN [ 60 ] pioneers at applying the GAN technology on the 3D voxels to learn a deep generator that can synthesize various 3D shapes . Generative models on point cloud [ 1 , 15 , 67 ] mainly leverage MLP layers but struggle to generate dense point sets with high fidelity due to the large memory consumption and high computational complexity . Recent works on synthesizing 3D meshes either rely on deforming an initial mesh using graph CNN [ 54 , 58 ] or assembling surface patches [ 24 , 13 ] to achieve more flexible structure manipulation . To better model man-made objects composed of hierarchical regular shapes , structural relationship has been considered in [ 18 , 66 , 17 , 38 , 40 , 39 ] , where the box-like primitives are used for initial shape to enhance shape regularity . To fully exploit the modeling capacity of implicit surface generator , IM-Net [ 7 ] has experimented with both VAE and GAN models to learn stronger shape priors . In a concurrent work , DeepSDF [ 44 ] proposes a auto-decoder structure to train latent space and decoder without using a traditional shape encoder . Recently , the local implicit methods [ 27 , 4 ] combine regular space decomposition with local implicit generators for modeling 3D scenes with fine geometric details . ACORN [ 35 ] introduces an adaptive multiscale neural scene representation for 2D and 3D complex scenes , which enables to fit the targets faster and better in an optimized multiscale fashion . In our paper , we propose a hierarchical implicit generative model for 3D modeling . Compared to other methods , our approach can generate high-quality 3D surfaces with intricate geometric details in a memory-efficient manner . 3 Method . Overview . OctField combines the good ends of both localized implicit representation and the hierarchical data structure . By adaptively allocating local implicit functions according to the surface occupancy and the richness of geometry , OctField is able to achieve high modeling accuracy with a low memory and computation budget . In particular , we decompose the 3D space into hierarchical local regions using octree structure , where the finest octant encodes the partial shape within its enclosed space using a learned implicit function . Our decomposition protocol not only considers the surface occupancy but also the richness of geometry . As show in Figure 2 , the octants that carry an embedded implicit kernel will only be allocated around the surface . Moreover , only the octants containing intricate geometries will be further divided . This ensures an adaptive memory and computation allocation that the richer surface details will be captured with more local implicit functions – hence with higher modeling accuracy . In contrast , the unoccupied regions will not be allocated with any implicit kernels to save the memory and computational budget . The octree itself is a non-differentiable discrete data structure . We propose a novel differentiable hierarchical encoder-decoder network that learns both the octree structure and the geometry features simultaneously . In particular , we formulate the construction of octree as a probabilistic process where the probability of subdividing an octant is predicted by a MLP layer . This makes it possible to learn discrete octree structure in a fully differentiable manner . In addition , we train our network in a VAE manner such that the trained latent space and decoder can be used for a variety of downstream applications including shape reconstruction , generation , interpolation , single-view reconstruction , etc . We provide detailed description of the OctField representation and the proposed network in Section 3.1 and 3.2 respectively .

Summary: The paper proposes a hierarchical implicit grid representation for modeling implicit shape-spaces. While concurrent research has focused either on hierarchical representations in overfitting mode, or on fixed grids in shape-space mode, the paper proposes a technique that simultaneously satisfies these two objectives. The method is tested on a few classes from the synthetic SDF'ied shapenet against a few methods, and has seemingly good reconstruction performance.