OctField: Hierarchical Implicit Functions for 3D Modeling
Authors: Jia-Heng Tang
Weikai Chen
Jie Yang
Bo Wang
Songrun Liu
Bo Yang
Lin Gao
Venue: NeurIPS 2021 Poster (2021)
Keywords: Neural implicit function
Octree
3D representation

Abstract: Recent advances in localized implicit functions have enabled neural implicit representation to be scalable to large scenes.
However, the regular subdivision of 3D space employed by these approaches fails to take into account the sparsity of the surface occupancy and the varying granularities of geometric details. As a result, its memory footprint grows cubically with the input volume, leading to a prohibitive computational cost even at a moderately dense decomposition. In this work, we present a learnable hierarchical implicit representation for 3D surfaces, coded OctField, that allows high-precision encoding of intricate surfaces with low memory and computational budget. The key to our approach is an adaptive decomposition of 3D scenes that only distributes local implicit functions around the surface of interest. We achieve this goal by introducing a hierarchical octree structure to adaptively subdivide the 3D space according to the surface occupancy and the richness of part geometry. As octree is discrete and non-differentiable, we further propose a novel hierarchical network that models the subdivision of octree cells as a probabilistic process and recursively encodes and decodes both octree structure and surface geometry in a differentiable manner. We demonstrate the value of OctField for a range of shape modeling and reconstruction tasks, showing superiority over alternative approaches.

OctField: Hierarchical Implicit Functions for 3D Modeling
Jia-Heng Tang*l.2, Weikai , Jie Yang! Wang', Songrun Liu², Bo Yang', and Lin Gao (5z)
Beijjng Key Laboratory of Mobile Computing and Pervasive Device, Institute of Computing Technology, Chinese Academy of Sciences 2 University of Chinese Academy of Sciences
Tencent Games Digital Content Technology Center
tangjiaheng 9 i at..ac.o. cn (bohawkwang, songrunling brandonyang] (tapopyyynn@@nnnennnne @tencent.com
chenwk891 8gmai 1.com yangjie01@ictt wt..:.0@iic......o..o........... on
gaolinßics en
Abstract
Recent advances localized implicit functions have enabled neural implicit rep- resentation to be scalable to large scenes. However, the regular subdivision of 3D space employed by these approaches fails to take into account the sparsity the surface occupancy and the varying granularities of geometric details. As result, its memory footprint grows cubically with the input volume, leading to a prohibitive computational cost even moderately dense decomposition. In this work, we present a learnable hierarchical implicit representation for 3D surfaces, coded OctField, that allows nigh-preeision encoding of intricate surfaces with low memory and computational budget. The key to our approach is an adaptive de- composition of 3D scenes that only distributes local implicit functions around the surface of interest. We achieve this goal by introducing a hierarchical octree struc- adaptively subdivide the 3D space according the surface occupancy and the richness of part geometry. As octree is discrete and non-diff non-differentiable, we further propose a novel hierarchical network that models the subdivision of octree cells as a probabilistic process and recursively encodes and decodes both octree structure and surface geometry in a differentiable manner. We demonstrate the value of OctField for range of shape modeling and reconstruction tasks, show- ing superiority over alternative approaches.
Introduction
Geometric 3D representation has been central to the tasks in computer vision and computer graphics, ranging from high-level applications, such as scene understanding. object recognition and classifica- tion, etc, to low-level tasks, including 3D shape reconstruction, interpolation and manipulation. To accommodate with various application scenarios, a universal and effective 3D representation for deep learning should have the following properties: (1) compatibility with arbitrary topologies, (2) capacity modeling fine geometric details, (3) scalability to intricate shapes, (4) support efficient encoding of shape priors, (5) compact memory footprint, and (6) high computational efficiency.
While explicit 3D representations have been widely used in recent 3D learning approaches, none of these representations can fulfill all the desirable properties. In particular, point cloud and voxel representations struggle to capture the fine-scale shape details often at the cost of high memory
"Contributed equally.
'Comesponding author is Lin Gao (gaolin@ict.ac.com
35th Conference on Neural Information Processing Systems (NcurIPS 2021), virtual.

---

consumption. Mesh-based learning approaches typically rely on deforming template model, lim- iting its scalability to handle arbitrary topologies. The advent of neural implicit function [44, 7. 37] have recently brought impressive advances to the ofte-of-thhe-rr across range of 3D modeling and reconstruction tasks. However, using only global function for encoding the entirety of all shapes, the aforementioned methods often suffer from limited reconstruction accuracy and shape generality.
To overcome these limitations, follow-up works have proposed decompose the 3D space into regular grid [27, 4], or local supporting regions [19], where each subdivided shape is approximated by a locally learned implicit function. The decomposition of scenes simplifies the shape priors that each local network has to learn, leading to higher reconstruction accuracy and efficiency. However, these approaches do not take into account the varying granularities of local geometry, resulting in two major shortcomings. Efficiency-wist their memory usage grows cubically with the volume of the 3D scenes. Even a moderately dense decomposition could impose severe memory bottleneck. Scalabilitywwise, the regular gridding has difficulty scaling to high resolutions, limiting its expres- siveness when dealing with intricate shapes with small and sharp geometric features (Figure 4).
We observe that most 3D shapes are typically consisting of large smooth regions and small- scale sharp features. In addition, the surface of interest often consumes only a small portion of the entire space. leading to an extremely sparse space occupancy. Based on these observations, we propose a novel 3D representation called OctField, that introduces hierarchies to the or- ganization of local implicit functions to achieve better memory efficiency and stronger model- ing capacity. As shown in Figure 1, OctField leverages a hierarchical data structure, Octree, to adaptively subdivide the 3D space accord- ing to the surface occupancy and the richness of geometrical details. In particular, regions enclosing intricate geometries will be further subdivided allocate more implicit kernels for higher modeling accuracy. In contrast, we stop subdivision for octants containing smooth part geometry as a single implicit kernel would suf- fice for modeling. Further, we do not allocate any implicit functions in the unoccupied regions. representation accuracy with a slightly deeper octree subdivision, as the modeling capacity has been adaptively optimized to accommodate the varying granularity of surface details.
octant depth
Intermediate octamis depiii 1
Figure 1: OctField utilizes an octree structure to achieve hierarchical implicit representation, where part geometry enclosed by an octant is rep- resented by local implicit function. OctField achieves an adaptive allocation of modeling ca- pacity according to the richness of surface geom- etry. In particular, intricate parts such as jet en- gines, tail-planes and the undercarriage are au- tomatically subdivided to engage more implicit kernels for higher modeling accuracy, while parts with regular shapes on the fuselage is encoded us- ing a coarser-level representation that suffices.
Hence, OctField could obtain significantly higher
As the octree structure is discrete and non-differentiable, it is non-trivial to directly employ octree in a learning-based framework. We propose a novel hierarchical network that recursively encodes and decodes both octree structure and geometry features in a differentiable manner. Specifically, the decoder side, we model the subdivision of octree cells as a probabilistic process to make the learning of octree structure differentiable. We employ classifier to predict whether subdivide current cell based on its enclosed geometry features. We validate the efficacy of our new representation in variety of tasks on shape reconstruction and modeling. Experimental results demonstrate the superiority of OctField over the state-of-the-art shape representations in terms of both shape quality and memory efficiency. Our contributions can be summarized as follows:
A learnable hierarchical implicit representation for 3D learning, named OctField, that com- bines the state-of-the-an hierarchical data structure with local implicit functions.
A novel hierarchical encoder-decoder network that models the construction of octree probabilistic process and is able to learn both discrete octree structure and surface geometry in a differentiable manner.
We achieve significantly higher surface approximation accuracy with reduced memory cost in the 3D modeling related tasks by using our proposed representation.

---

2 Related Works
Representations for 3D Shape Learning. Various 3D representations have been extensively stud- ied in 3D deep learning [28]. These surveys [2, 62] discuss various shape representations compre- hensively. As the raw output of 3D scanning devices, point cloud [46, 47, 65, 49, 53] has received much attention in recent years. Despite for its simplicity, generating dense point clouds with high precision remains notoriously difficult. Unlike the other 3D representations, the convolutional net- work can be directly employed on 3D voxels [36, 11, 20, 59, 60, 61, 64, 23, 10]. Due to the pro- hibitive computational cost of generating voxels, recent works [55, 57, 56, 52, 32] have introduced octree to the voxel representation to reduce memory cost. The polygon mesh is the another widely used representation modeling and surface reconstruction. However, current learning-basee mesh generation approaches [6, 54, 50, 24, 29, 21, 12, 43] mostly rely on deforming a template mesh, limiting its scalability to shapes with arbitrary topologies. Recent advances in neural implicit func- tions [44, 37, 7, 27, 14, 63, 8, 45] have significantly improved the surface reconstruction accuracy thanks to its flexibility of handling arbitrary topologies. More recently, [27, 4, 19] have introduced shape decomposition and local implicit functions to further improve the modeling capacity by locally approximating part geometry. [33] introduces implicit moving least-squares (IMI.S) surface formu- lation on discrete point-set to reconstruct high-quality surfaces. However, these methods mostly rely on a regular decomposition and cannot account for sparse surface occupancy and the varying granu- larities of geometry details, imposing memory bottleneck when dealing with moderately dense sub- division. Open VDB [42] incorporates B+ tree with implicit field to achieve hierarchical modeling. However, the goal of Open VDB is pursuing extremely fast modeling speed with constant time access in 3D simulation. Hence, the B+ tree is non- -on-differentiable and too complex to be incorporated into a learning-bast -baee framework. In contrast, OctField is learnable hierarchical implicit representation that can be differentiably implemented in our hierarchical network. Further, our representation can achieve higher modeling accuracy with even less memory compared to the previous local implicit function approaches [27, 4, 19]. In a concurrent work NGLOD [51], it proposes a similar idea of leveraging level of details to encode local SDFs hierarchically.
corresponding rendering algo- rithm is proposed to render the neural SDFs in an interactive rate. However, NGLOD cannot learn the hierarchical structure of the underlying octree. In contrast, our hierarchical encoder-decoder net- work learns the non-differentiable structural information in differentiable manner via modeling it as a probabilistic process. We believe the structural information is crucial for improving the modeling accuracy and future applications(e.g. 3D semantic understanding, editing).
Learning-based Generative Models. The deep generative models, such as GAN [22] and VAE [30], have shown promising ability of synthesizing realistic images in 2D domain. 3D learning approaches have strived to duplicate the success of 21) generative models into 3D shape generation.
3D-GAN [60] pioneers at applying the GAN technology on the 3D voxels to learn a deep generator that can synthesize various 3D shapes. Generative models on point cloud [1. 15, 67] mainly lever- age MLP layers but struggle to generate dense point sets with high fidelity due to the large memory consumption and high computational complexity. Recent works on synthesizing 3D meshes either rely on deforming an initial mesh using graph CNN [54, 58] or assembling surface patches [24, 13] to achieve more flexible structure manipulation. To better model man mane-made objects composed of hierarchical regular shapes, structural relationship has been considered in [18, 66, 17, 38, 40, 39], where the box-like primitives are used for initial shape to enhance shape regularity. To fully exploit the modeling capacity of implicit surface generator, IM-Net [7] has experimented with both VAE and GAN models to learn stronger shape priors.
concurrent work, DeepSDF [44] proposes a auto-decoder structure to train latent space and decoder without using traditional shape encoder. Recently, the local implicit methods [27,4] combine regular space decomposition with local implicit generators for modeling 3D scenes with fine geometric details. ACORN [35] introduces adap- tive multiscale neural scene representation for 2D and 3D complex scenes, which enables to fit the targets faster and better in optimized multiscale fashion. In our paper, we propose hierarchical implicit generative model for 3D modeling. Compared to other methods, our approach can generate high-quality 3D surfaces with intricate geometric details in memory-efficicee manner.
3 Method
OctField combines the good ends of both localized implicit representation and the hi- erarchical data structure. By adaptively allocating local implicit functions according to the surface
Overview.
3

---

occupancy and the richness of geometry. OctField is able to achieve high modeling accuracy with low memory and computation budget. In particular, we decompose the 3D space into hierarchical local regions using octree structure, where the finest octant encodes the partial shape within its en- closed space using learned implicit function. Our decomposition protocol not only considers the surface occupancy but also the richness of geometry. As show in Figure 2, the octants that carry an embedded implicit kernel will only be allocated around the surface. Moreover, only the octants containing intricate geometries will be further divided. This ensures adaptive memory and com- putation allocation that the richer surface details will captured with more local implicit functions hence with higher modeling accuracy. In contrast, the unoccupied regions will not be allocated with any implicit kernels to save the memory and computational budget.
The octree itself is a non-differemee -ol-differentiable discrete data structure. We propose novel differentiable hierarchical encoder-decoder network that learns both the octree structure and the geometry features simultaneously. In particular, we formulate the construction of octree as probabilistic process where the probability of subdividing an octant is predicted by a MLP layer. This makes it possible to learn discrete octree structure in a fully differentiable manner. In addition, we train network in VAE manner such that the trained latent space and decoder can be used for a variety of downstream applications including shape reconstruction, generation, interpolation, single-view reconstruction, etc. We provide detailed description of the OctField representation and the proposed network in Section 3.1 and 3.2 respectively.
axpile Donada
Mouel CNN
.9
Deoole
Disal
ROOD
OctField Representation
Hierarchical OctField Encoder
Hierarchical OctField Decoder OctField Representation
Figure 2D illustration of our hicrarchical OctField network. We propose novel recursive encoder-decoder structure and train the network in VAE manner. We use the voxel 3D CNN to encode the octants' geometry, and recursively aggregate the structure and geometry features using hierarchy of local encoder }- The decoding is implemented using hierarchy of local decoders {D:} with mirrored structure with respect the encoder. Both the structure and geometry infor-
mation are recursively decoded and the local surfaces are recovered using the implicit octant decoder within each octant.
3.1 OctField Representation
Octree Construction. To build an octree for the input model, we first uniformly scale the 3D shape into an axis-aligned bounding box and then recursively subdivide the bounding region into child octants in breadth-first order. The octant to be subdivided has to satisfy two requirements simultaneously: (1) the octant encloses the surface of interest; and (2) its enclosed geometry needs to have sufficient complexity that is worth subdividing. We use the normal variance of the surface as an indicator of its geometric complexity. Specifically, we formulate the normal variation of a surface patch S follows:
V(S) E, (V(n(() + V((n() + V((n())
where the ni. miy. n. are the 2, y, c-component of the normal vector n' at the i-th sampling point on the surface; (n',) } denotes the collection of n V (-) calculates the variations of the input while E, (.) returns the expectation. In our experiments, we perform regular sampling on the surface where the sampling points pre-computed. We repeat the decomposition until the pre-defined depth d is reached or (SS) is smaller than a pre-set threshold т. We set 0.1 throughout our experiments.
Local Implicit Representation. The implicit function associated with each octant is designed to model only part of the entire shape. This enables more training samples and eases the training as most 3D shapes share similar geometry at smaller scales. At each octant, the enclosed surface is

---

continuously decoded from the local latent code. However, as the finest octant may have differ- ent sizes, when querying for the value of the local implicit function, we normalize the imput world coordinate x against the center of the octant Ty. Formally, we encode the surface occupancy as: f(;; 2)) = Do, (c, -5--a m4)), where Do, is the learned implicit decoder with trainable parameter a. C2 is the local latent code and .N (.) normalizes the input coordinate into the range of [-1, according to the bounding box of the octant. To prevent the discontinuities across the octant bound- aries, we propose to enlarge each octant such that it overlaps with its neighboring octant at the same level. In our implementation, we let each octant has 50% overlap along the axis direction with its neighbors. When the implicit value at the overlapping regions is queried, we perform tri tri-linear interpolation over all the octants that intersect with this query position.
3.2 Hierarchical OctField Network
To enable differentiable framework for learning the octree structure and its encoded geometry. we propose a novel hierarchical encoder-decoder network that organizes local encoders and decoders in a recursive manner. We embed both the octree structure information and the geometry feature into the latent code of each octant. As shown in right part of Figure 2, the latent code e; (gi, Ci, (3:) for octant O; is a concatenation of three parts: (1) a geometry feature 9i that encodes the local 3D shape; (2) binary occupancy indicator a; that indicates whether the octant encloses any 3D surface; and (3) binary geometry subdivision indicator Br that denotes whether the enclosed geometry is intricate enough that needs further subdivision. We will show in the following subsections that how this configuration of latent vector guides the recursive decoding and encoding in our network. Note that, unlike the prior tree pructure-based generative models [66, 38, 31], our approach does not require manually labeled part hierarchy, e.g. the PartNet [41] dataset, for training, and can generate the hierarchical structure automatically using our octree construction algorithm.
3.2.1 Hierarchical Encoder
As shown in Figure 2, the encoder E of our network is composed of hierarchy of local en- coders {& that encodes local geometry feature and octree structure into the latent code. While our framework supports general geometry en- coders, we employ a 3D voxel CNN for ex- tracting geometry features due to its simplicity of implementation. After constructing the oc- tree for input model, voxelize the surface enclosed in each octant in a resolution of 32
Hierarchical Encoder a
B g.
voctar
FC
FC
max pooling
Hicrarchical Decoder Dr
FC
FC
hrikden vedior
The encoding process starts from the octants at the finest level in bottom-up manner. For each octant O,, we first compute its binary indicators (at,, 3.) according to its enclosed geometry. In particular, ae is set to 1 if there exist surfaces inside O and if otherwise; 3r is set to if O O,'s enclosed geometry (if o; = 1) satisfies the subdivision criteria as detailed in Section 3.1 and O if otherwise. We then extract O,'s ge- ometry feature 9; by passing its enclosed vox- elized geometry G; the voxel CNN V. When proceeding to a higher level, our network will aggregate the children's 's latent features to its parent octant. In particular, for parent octant O₁, denote the octant features of its children as {ee, = (ge,> Beyy Be,) |cj E Ckk, J. where C₁ represents the child octants of Og. Its encoder & then aggregates the latent features of O₂ 's child octants into Os's 's geometry feature gl:
Figure 3: The architecture of hierarchical encoder & and decoder Dr. & gathers the structure (Oeey: Bey)) and geometry Ic, feature of child oc- tants its parent octant by a MLP, max- max-pooling operation, and another MLP, where cj E Cg. Dx decodes the parent octant feature gk to features {9e,) } and two indicators a Gr, of its child oc- tants by two MLPs and classifiers. Two indicators infer the probability of surface occupancy and the necessity of further subdivision, respectively.
gk Eg (Ee. E....... ,ee)))
We then obtain O₂'s 's latent feature by concatenating ga with O₂'s structure features (an,, We perform the recursive feature encoding and aggregation until the root node has been processed. Specifically, the encoder & consists of a multi-layer perceptron (MLP), one max pooling layer and another MLP for output. At the end of encoder, we leverage the VAE re- parameterrizaionnn technique
5

---

to encourage the distribution of the latent space to fit normal distribution. Note that all the local encoders & share its parameters leverage the similarity of local geometries and reduce the network parameters.
3.2.2 Hierarchical Decoder
The hierarchical decoder D aims to decode the octree structure and local octant codes from the input global feature. consists a hierarchy of local decoders [Di] with a mirrored structure with respect to the encoder E. On the contrary E, the decoding process starts from the root node and recursively decodes the latent code of its child octants in a top-down manner. Specifically, for parent octant O₂ with geometry feature gk we decode the geometry features of its child octants using the decoder D3:
(Ceo Cct : = Dx (gk)
(3)
where E Ck denotes the child octant of O₂ and Ccy = (g Cey Bey) stands for the geometric feature and two indicators of the child octant Oeg. The two indicators provide the probability of whether the child octants need to be decoded or subdivided. Note that we decode all the 8 child octants one time.
In particular, Dx consists of two MLPs and two classifiers (see Figure 3). We first decode gt. into hidden vectors Ve, for all 8 child octants by a MLP. To decode the structure information, two classifiers Ig and Z, to infer the probability of surface occupancy and the necessity of further subdivision, respectively. For child octant Oeg we feed its hidden vector Dey into Ig and In and calculate 022 = (V,,) and Beg In (v,,)) ). For predicting the Jcg, we apply the other MLP on Ve,- ae,, < 0.5, indicates that Oz, does not contain any geometry and will not be further processed. If Ce; > 0.5, it means that Oeg is occupied by the surface and we will further check the value of Bey- If Be, S 0.5, we will not further subdivide the octant and will infer its enclosed surface using the implicit octant decoder G and the geometric feature Jc,, If Be, 0.5, we will proceed to subdivide the octant by predicting the latent features of its child octants with the same procedure. We repeat this process until octants need to be subdivided.
apply
Implicit Octant Decoder. We use local implicit decoder G to reconstruct the 3D surface within the octant. For octant O,, we feed its latent geometry feature 9: and the 3D query location a to the implicit decoder for occupancy prediction. We train & with binary cross entropy loss on the where F F-) returns the ground-truth -tunh label (insidc/outside) for input point. Le (1.-) is the binary cross entropy loss, P denotes the set of sampling points, w, describes the inverse of sampling density near æ; for compensating the density change as proposed in [7]. Note that & is pre-trained on all the local shape crops to encode stronger shape prior.
ΣSEP (.)
point samples. The training loss for octant O, is: Lieo
Eeer My
In order to obtain stronger supervision, we strive recover the local geometry of all the octants that are occupied by the surface regardless belongs to the finest level. Hence, the total loss for training our hierarchical encoder-decoder network is formulate as follows:
Liotel = Eo,co + La Ls + 3KLI],
(4)
where La and La denote the binary cross entropy loss of classifying whether the octant contains geometry and needs to be subdivided, respectively, LK is the KL divergence loss, and [E|-] returns the expected value over the set of all octants O that enclose surface geometry. We set À 10.0, 8 0.01 throughout our experiments.
Experiments
In the section, we will first introduce our data preparation process and then evaluate our approach in variety of applications, including shape reconstruction, shape generation and interpolation, scene reconstruction and shape completion. We also provide ablation study, more comparisons and imple- mentation details the supplemental materials.
4.1 Data Preparation
Our network is trained and evaluated on the five biggest and commonly used object categories in the ShapeNet dataset [5]: chair, table, airplane, car, and sofa. For fair comparison, we use the

---

officially released training and testing data splits. All the shapes are normalized to fit unit sphere and converted into watertight meshes [26] for computing ground-truth signed distance field. We first build the octree for each model according to the protocol defined in Section 3.1. To account for the sparse occupancy of surfaces, we apply importance sampling to sample more points near the surface and exponentially decrease the point density as the distance to surface increases. We sample 10000 points in total for each octant, and calculate its corresponding signed distance to surface. To deploy 3D CNN, we adaptively voxelize the part shape in each octant to ensure it maintains the 32 resolution regardless the size of the host octant. We use the voxelization code provided by [25].
Metries We follow the commonly used reconstruction metrics: Chamfer distance (CD) [3] and Earth Mover's Distance (EMD) [48] for quantitative evaluation and comparison with prior methods.
4.2 Shape Reconstruction
In this section, we evaluate the performance on shape reconstruction and compare with
the following state-of-the-art approaches that are closely related with our method: IM- Net [7], OccNet [37], Local Implicit Grid
(LIG) [27], ConvOceNet [45], Adaptive O-
CNN (AOCNN) [57], and OGN [52]. In partic- ular, IM-Net and OceNet use a global implicit function to depict the the entirety of 3D shape while LIG decomposes the input shape into reg- ular grid and leverages local implicit kernels to approximate part geometry. AOCNN and OGN also utilizes the octree structure. However, in- stead of using local implicit functions, AOCNN uses single plane approximate the local ge- ometry enclosed in each octant. OGN predicts occupancy in octant without further local geometric feature. We show the visual comparisons in Figure 4. While IM-Net and OccNet are capable of reconstructing the global shape of the object, they fail reconstruct detailed structures. LIG can recover some of the fine-scale geometries but has difficulty in modeling sharp and thin structures as shown in the second row. Since AOCNN only uses a primitive plane to approximate local geometry, it suffers from the discrepancy between adja- cent octants and cannot recover complex local structures due to its limited approximation capability. In comparison, our approach achieves the best performance in categories and is able to faithfully reconstruct intricate geometry details, such as the slats of chair backs, the hollowed bases of the tables and and the wheels of the cars.
(g)
(h)
Figure 4: Shape reconstruction comparison with
the baseline methods ((a) Input, (b) AOCNN [57].
(c) OGN [52], (d) LIG [27]. (e) OccNet [37], (f)
ConvONet [45], (g) IM-Net [7], and (h) Ours).
We report the result of quantitative compar- isons in Table 2. 1. Our approach outper- forms the alternative approaches over all cate- gories and achieves the best mean accuracy in terms of CD, EMD, mloU, and F1 score met- rics. In particular, our reconstruction accuracy is significantly higher than IM-Net, OccNet and AOCNN over all categories.
Method
IM-Net
OccNet
LIG
Ours
mloUt
79.9 8
71.36
86.28
87.96 0.94
FIf
0.83
0.70
0.93
Table 1: Quantitative evaluation on shape recon- struction, we report mIoU and Fl score.
Computational Cost. In Table 3, we com- pare the computational cost with the local im- plicit approach using regular decomposition LIG [27]. We show the consumption of local implicit cells/octants used for surface modeling and the computation memory with an increas- ing decomposition level. Thanks to our adap- tive subdivision, our approach consumes signif- icantly less local kernels compared to LIG to achieve similar or even better modeling accu- racy. This lepp becomes more prominent with
level
1
2
4
LIG
8
64
512
4096
Number of cells
Ours
8
30
200
1000
LIG
0.1
0.6
40
Memory (GB)
Ours
0.2
1.2
4.8
23
Table 3: Comparisons of computational cost with LIG [27]. We show the consumption of the local cells and memory with respect to different levels of decomposition.
7

---

IM-Net
Oce-Net
LIG
AOCNN
ConvOceNet
OGN
Ours
Dutaset
CD
EMD
CD
EMD
CD
EMD
CD
EMD
CD
EMD
CD
EMD
CD
EMD
Plane
4.21
3.39
5.62
3.46
2.50
2.57
6.90
4.26
3.03
3.82
7.43
4.61
2.29
2.47
Car
15.14
4,46
13.54
4.93
5.46
4.08
16.61
5.63
10.04
5.66
16.24
6.23
4.84
2.79
Chair
6.99
3.77
7.87
4.16
2.37
2.18
10.80
6.76
3.98
3.19
10.77
5.65
2.19
2.13
Table
8.03
3.16
7.47
3.34
2.81
2.27
9.15
4.78
3.83
3.04
9.03
3.88
2.53
1.71
Sofa
7.95
2.51
8.6
2.81
3.23
2.06
9.39
3.49
4.03
2.85
8.79
4.32
3.02
1.84
Mean
8.46
3.45
8.62
3.74
3.27
2.63
10.57
4.98
4.98
3.712
10.45
4.94
2.97
2.19
Table 2: Quantitative evaluation on shape reconstruction. this table, we report the CD (x 1x10
and EMD 10 ?) scores (smaller is better) on five categories. OctField can achieve the best perfor- mance on average score and each category by comparing to six baselines (IM-Net [7], OceNet [37], Local Implicit Grids [27], Adaptive O-CNN [57], ConvOccNet [45] and OGN [52]).
the increasing level of decomposition. As OctField requires additional memory to maintain octree structure, low decomposition level, memory cost is slightly higher than LIG. However, with finer subdivision, our memory consumption drops significantly and becomes much lower than that of LIG. It indicates an increasing advantage of our method in modeling intricate geometry in higher resolution.
4.3 Shape Generation & Interpolation
As we train our network in a VAE manner, our model is able to generate diversified 3D shapes by feeding our pre-trained decoder with random noise vectors sampled from normal distribu- tion. Our network learns a smooth latent space that captures the continuous shape structures and geometry variations. To generate novel 3D shapes, randomly sample latent vector in our learned latent space, and decode it into shape space by extracting its zero-isosurface using MarchingCubes [34]. In Figure 5, we show the generated results on chair and table categories respectively. Despite the random sampling, our ap- proach is still able synthesize high-quality 3D shapes with complex structure and fine geometric details, e.g. the second and the fourth table in the second row.
Figure 5: Shape Generation. We show the results generated by randomly sampling the latent codes in the latent space.
Another approach to synthesize new shapes is to interpolate between the given shapes in the latent space. For two input shapes, we inter- polate their latent codes linearly and feed the obtained latent vectors the pre-trained de- coder for shape interpolation. Figure shows the interpolated results on the chair and table categories. Our approach can achieve smooth and continuous interpolation even between two highly diversified objects with distinct struc- tures. In addition, the sharp geometry features, e.g. the six iquareere--ae grid base of the table in the first row, can be well maintained during the interpolation. This indicates that our network is capable of learning a smooth manifold to generate novel shapes in high fidelity.
(b)
Figure 6: Shape Interpolation. The figure shows two interpolated results in two categories: table and chair. (a) is source shape, (f) target shape.
4.4 Scene Reconstruction
Compared with a single object, our representation is more advantageous when dealing with large scenes. Our representation head. In this section, we illustrate the superiority of OctField on large scene dataset 3D-Front [16]. Further, we compare with local implicit approach using regular decomposition LIG [27], convo- lutional occupancy network [45], NGLOD [51] and ACORN [35] quantitatively and qualitatively. In Figure 7, we present two camera views in a large scene from 3D-Front [16] dataset.
obtain better reconstruction details while saving computational over-
8

---

From the visualization results, we can observe that our results is capable of capturing more fine-grained geo- metric and structure details compared to LIG. The other two methods that introduce hierarchical structure also perform well. It is worth mention- ing that high-quality visualized re- sults can be generated by rendering directly from the SDF NGLOD[51J. However, extracting the mesh from the implicit field could cause loss of reconstruction aceuracy. Table shows quantitative comparison results. Our approach outperforms the alternative approaches over the large scene models and achieves the best perfor- mance in terms of CD and EMD metries.
Method
CDx10
EMDx10 2
LIG
7.1
36.1
ConvOceNet
10.5
22.3
NGLOD
11.3
33.1
ACORN
7.7
24.2
Ours
6.4
21.1
Table 4: Quantitative evaluation on scene reconstruction.
(d) NGLOD Figure 7: Scene Reconstruction and Comparison. In this figure, some large scene reconstruction and comparison with Local Implicit Grid [27], convolutional occupancy network (Conv OccNet) [45], NGLOD [51] and ACORN [35] are presented. We show that our method can provide more accurate reconstruction of geometric and structural details of large scenes. The experiment are performed on 3D-Front [16].
(a) Input Scene
(b) LIG
(c) COceNet
(e) ACORN
(f) Ours
4.5
Shape Completion
We evaluate our method in the task of shape completion. Furthermore, We compare with the IF-Net [9] and and demonstrate that method achieves more robust completion performance with less artifacts in Figure 8. Specif- ically, we first voxelize the partial point cloud and then map it to the latent space of OctField representa- tion via 3DCNN encoder. The trieved latent code is fed to our hierar- chical decoder for reconstructing the octree structure, as well as the geo- metric surface. For IF-Net, we also adopt same voxelization and map the partial voxels to the complete shape by its shape completion model.
(a) Input
(b) IF-Net
(c) O-CNN
(d) Ours
(e) GT
Figure 8: Shape Completion and Comparison with IF- Net [9] and O-CNN [56]. Our method is able to recover complete and faithful 3D shapes only from partial point clouds.
Shape completion results (see Figure 8 and Table 5) the table category show that our method achieves more robust completion per-
Method
IF-Net
O-CNN
Ours
CD(x10-1)
4.9
12.1
formance with less artifacts. Compared to other octree-based method [56], our method predicts the complete mesh of partial input rather than dense point cloud.
Table 5: Quantitative evaluation shape comple- tion.
4.6 Shape Editing
With the proposed differential octree generative model, our framework enables some potential ap- plications, such as part editing that modifies replaces only part of the target geometry. In order

---

to realize parting edit, we re- p--aameeterize the latent code of the partial local shape, introducing the local VAE to modify and replace the local geometric shape. In Figure we show the results of the part editing of our method comparing with a naive method of directly blending two implicit fields from the source and target shapes. Our approach can generate a smooth transition even between two distinct structures while the naive blending method cannot guarantee continuous connection for local shape editing.
5 Conclusions and Discussions
We have proposed a novel hierarchical implicit representation for 3D surfaces, coded OctField. OctField takes advantages of the sparse voxel octree representation adaptively generate lo- cal supporting regions around the surface of in- terest. By associating local implicit function with each octant cell, OctField is able to model large-scale shape with fine-level details using compact storage. To accommodate the non- differentiable nature of octree, we further pro- pose novel hierarchical network that models the octree construction as probabilistic pro- cess and recursively encodes and decodes both structural and geometry information in differ- entiable manner. The experimental results have shown superior performance of OctField over the alternative approaches in variety of tasks related shape modeling, reconstruction, and edit- ing. In the future, we would like to incorporate semantic meaning into the organization of octree encode structural information and enable flexible editing of part-level geometry. In addition, is also an interesting avenue to explore adaptive length of local latent code such that local implicit functions with higher modeling capacity only dealing with geometries with more intricate de- tails.
(b)
Figure 9: Part editing, The figure shows two edited chairs using our network. ((a) source, (b) blending the SDF directly, (c) our result)
6 Broader Impact
The proposed OctField can serve fundamental representation of 3D geometry and thus can have a positive impact a broad range of research fields, including computer vision, computer graphics, and human-computer interaction, etc. Specifically, due the cost-effective nature of our representation, our method can reduce the economic cost of 3D environment acquisition from raw scanning, while maintaining high-fidelit modeling performance. This could benefit number of real-world applications, including modeling large-scale 3D scenes, compressing and transmitting high-quality 3D models for relecommunication and telepresence. Our generative model also be used for low-cost 3D shape generation without the need of performing actual 3D scanning and post processing, which are expensive and cone-consumnng. However, at the same time, special care must be taken not to violate the privacy and security of the private scene owners during the process of data collection for our model training.
Acknowledgments. This work was supported by CCF-Tencent Open Fund, the National Natural Science Foundation of China (No. 61872440 and No. 62061136007), the Beijing Municipal Nat- ural Science Foundation (No. L182016), the Royal Society Newton Advanced Fellowship (No. NAF 192151) and the Youth Innovation Promotion Association CAS.
References
[1] Panos Achlioptas, Olga Diamanti, Toannis Mitliagkas, and Leonidas Guibas. Learning representations and generative models for 3D point clouds. In ICML, pages 40-49, 2018.
[2] Eman Ahmed, Alexandre Saint, Abd El Rahman Shabayek, Kseniya Cherenkova, Rig Das, Gleb Gusev, Djamila Aouada, and Bjöm Ottersten. Deep learning advances different 3D data representations: A survey. arXiv preprint arXiv: 1808.01462, 1, 2018. 3
10

---

[3] Harry G Barrow, Jay M Tenenbaum, Robert C Bolles, and Helen C Wolf. Parametric correspondence and chamfer matching: Two new techniques for image matching, In Proceedings: Image Understanding Workshop, pages 21-27. Science Applications, Inc Arlington, VA, 1977.
[4] Rohan Chabra, Jan E Lenssen, Eddy Ilg. Tanner Schmidt, Julian Straub, Steven Lovegrove, and Richard Newcombe. Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In European Conference on Computer Vision. pages 608-625. Springer, 2020. 2,3
[5] Angel X Chang. Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese. Manolis Savva, Shuran Song, Hao Su, et al. ShapeNet: An information-rice 3D model reposi- tory. arXiv preprint arXiv: 1512.33012, 2015.
[6] Zhiqin Chen, Andrea Tagliasacchi, and Hao Zhang. Bsp-net: Generating compact meshes via binary space partitioning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 45-54, 2020. 3
[7] Zhiqin Chen and Hao Zhang. Learning implicit fields for generative shape modeling. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5939-5948, 2019. 2, 3, 6, 7, 8 [8] Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll. Implicit functions feature space for 3d shape reconstruction and completion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6970-6981, 2020. 3
[9] Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll. Implicit functions in feature space for 3d shape reconstruction and completion. In IEFE Conference on Computer Vision and Pattern Recognition (CVPR). IREF, jun 2020. 9
[10] Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4D spatio-temporal convnets: Minkowski con- volutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pages 3075-3084 2019. 3
[11] Christopher B Choy, Danfei Xu, Young Gwak, Kevin Chen, and Silvio Savarese. 3D-R2N2: unified approach for single and multi-view 3D object reconstruction. In ECCV, pages 628-644. Springer, 2016.
[12] Angela Dai and Matthias Nießner. Scan2mesh: From unstructured range scans to 3D meshes. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3574-5583, 2019.
[13] Theo Deprelle, Thibault Groueix, Matthew Fisher, Vladimir G. Kim, Bryan C. Russell, and Mathieu Aubry. Learning elementary structures for 3D shape generation and matching. NeurlPS, 2019. 3
[14] Yueqi Duan, Haidong Zhu, He Wang, Li Yi, Ram Nevatia, and Leonidas J. Guibas. Curriculum DeepSDF, 2020. 3
[15] Haoqiang Fan, Hao Su, and Leonidas Guibas. A point set generation network for 3D object recon- struction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 605-613, 2017. 3
[16] Huan Fu, Bowen Cai, Lin Gao, Ling-Xisa Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfci Jia, Binqiang Zhao, et al. 3d-front: 3d furnished rooms with layouts and semanties. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pages
2021. 8,
[17] Lin Gao, Tong WU, Yu-Jie Yuan, Ming-Xian Lin, Yu-Kun Lai, and Hao Zhang. TM-NET: Deep generative networks for textured meshes. ACM Transactions Graphics (TOG), 40(6):263:1 263::15, 2021. 3
[18] Lin Gao, Jie Yang, Tong Wu, Yu-Jic Yuan, Hongbo Fu, Yu-Kun Lai, and Hao(Richard) Zhang. SDM- NET: Deep generative network for structured deformable mesh. ACM Transactions Graphics (TOG),
38(6):243:1-2243155, 2019. 3
[19] Kyle Genova, Forrester Cole, Avneesh Sud, Aaron Sarna, and Thomas Funkhouser. Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern
Recognition, pages 4857-4866, 2020. 2, 3
[20] Rohit Girdhar, David F Fouhey, Mikel Rodriguez, and Abhinav Gupta, Learning predictable and gener- ative vector representation for objects. In ECCV, pages 484-499. Springer, 2016. 3
[21] Georgia Gkioxari. Jitendra Malik, and Justin Johnson. Mesh R-CNN. In Proceedings of the IEEE Inter-
national Conference Computer Vision, pages 9785-9795, 2019. 3
[22] Ian Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing xystems, pages 2672-2680, 2014. 3
[23] Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3D semantic segmentation with sub- manifold sparse convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 9224-9232, 2018.
[24] Thibault Groueix, Matthew Fisher, Vladimir G Kim, Bryan C Russell, and Mathieu Aubry. AtlasNet: A papier-måché approach to learning 3D surface generation. In CVPR, 2018. 3
[25] Christian Häne, Shubham Tulsiani, and Jitendra Malik. Hierarchical surface prediction for 3d object reconstruction. In 20/7 International Conference on 3D) Vision (3DV), pages 412-420. IEEF, 2017. 7
11

---

[26] Jingwei Huang, Yichao Zhou, and Leonidas Guibas. Manifoldplus: A robust and scalable watertight manifold surface generation method for triangle soups. arXiv preprint arXiv:2005. 2020. 7
[27] Chiyu Jiang, Avneesh Sud, Ameesh Makadia, Jingwei Huang, Matthias NieBner. and Thomas Funkhouser. Local implicit grid representations for 3d scenes. In Proceedings of the IEEE/CVF Con-
ference Computer Vision and Pottern Recognition, pages 6001 2020. 2, 3, 7, 8,9
[28] Yiwei Jin, Diqiong Jiang. and Ming Cai. 3d reconstruction using deep learning: a survey. Communicutions in Information and Systems, 20(4):389-44 -13,13,,,,, 2020. 3
[29] Angjoo Kanazawa, Michael Black, David W Jacobs, and Jitendra Malik. End-to-end recovery of human shape and pose. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7122-7131, 2018. 3
[30] Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:
2013. 3
[31] Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer, Hao Zhang, and Leonidas Guibas. GRASS: Gen- erative recursive autoencoders for shape structures. ACM Transactions on Graphics (TOG), 36(4):1-14,
2017. 5
[32] Lingjic Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Thcobalt. Neural sparse voxel fields. NeurlPS, 2020.
[33] Shi-Lin Liu, HlaonXiang Guo, Hao Pan, Peng-Shuai Wang, Xin Tong, and Yang Liu. Deep implicit moving least-squares functions for 3d reconstruction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1788-1797, 2021.
[34] William E. Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM Transactions on Gruphics (TOG), 21 1664))163-169, 1987. 8
[35] Julien N. P. Martel, David B. Lindell, Connor Lin, Eric R. Chan, Marco Monteiro, and Gordon Wet- zstein. Acorn: Adaptive coordinate networks for neural scene representation. ACM Transactions on Graphics (TOG), 40(4), 2021. 3,8,9
[36] Daniel Maturana and Sebastian Scherer. VoxNet: A 3D convolutional neural network for real-time object recognition. In 2015 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 022-928.11 IEBE, 2015.
[37] Lars Mescheder, Michael Occhsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occu- pancy networks: Learning 3D reconstruction in function space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4460-4470, 2019. 2, 3, 7,8
[38] Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy Mitra, and Leonidas Guibas. Struc- tureNet: Hicrarchical graph networks for 3D shape generation. ACM Transactions on Graphics (TOG), 38(6):Article 242, 2019. 3,5
[39] Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka, Niloy J Mitra, and Leonidas J Guibas. Struete- dit: Learning structural shape variations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8859-8868, 2020. 3
[40] Kaichun Mo, He Wang. Xinchen Yan, and Leonidas Guibas. Pt2pe: Learning to generate 3d point cloud shapes from part tree conditions. In European Conference on Computer Vision, pages 683-701. Springer, 2020. 3
[41] Kaichun Mo, Shilin Zhu, Angel X Chang, Li Yi, Subarna Tripathi, Leonidas Guibas, and Hao Su. PartNet: large-scale benchmark for tine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 909-918, 2019.
[42] Ken Museth. Vdb: reghluesolution sparse volumes with dynamic topology. ACM transactions on graph- ics (TOG), 32(3):1-23 22, 2013.
[43] Charlic Nash, Yaroslav Ganin, SM Ali Eslami, and Peter Battaglia. Polygen: An autoregressive generative model of meshes. In International Conference on Machine Learning, pages 7220 7229, PMLR, 2020.
[44] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019. 2,
[45] Songyou Peng, Michael Niemeyer, Lars Mescheder, Marc Pollefeys, and Andreas Geiger. Convolutional occupancy networks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16. pages 523- 543-500. Springer. 2020. 3, 7, 8,
[46] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. PointNet: Deep learning on point sets for 3D classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 652-660. 2017.
[47] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. PointNet++: Deep hierarchical feature learning on point sets in metric space. In Advances in neural information processing systems, pages 5099-5108, 2017.
[48] Yossi Rubner, Carlo Tomasi, and Leonidas J Guibas. The earth mover's distance as metric for image retrieval. International journal of computer vision, 40(2):99- 121, 2000. 7
12

---

[49] Dong Wook Shu, Sung Woo Park, and Junseok Kwon. 3D point cloud generative adversarial network based on tree structured graph convolutions. In Proceedings of the IEEE International Conference on Computer Vision, pages 3859-3868, 2019.
[50] Ayan Sinha, Asim Unmesh, Qixing Huang, and Karthik Ramani. SurINet: Generating 3D shape surfaces using deep residual networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 6040-6049) 2017.
[51] Towaki Takikawa, Joey Litalien. Kangxue Yin, Karsten Kreis, Charles Loop. Derek Nowrouzezahraii Alee Jacobson. Morgan McGuire, and Sanja Fidler. Neural geometrie level of detail: Real-time rendering with implicit 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13558-1 1 11367, 2021. 3, 8, 9
[52] Maxim Tatarchenko, Alexey Dosovitskiy, and Thomas Brox, Octree generating networks: Efficient con- volutional architectures for high- righ-resolution 3D outputs. Proceedings of the IEEE International Con- ference on Computer Vision, pages 2088-2096, 2017. 3, 7,
[53]
Diego Valsesia, Giulia Fracastoro, and Enrico Magli. Learning localized generative models for 3d point clouds via graph convolution. International Conference on Learning Representations, 2019.
[54] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei Liu, and Yu-Gang Jiang. Pixel2mesh; Gener- ating 3D mesh models from single RGB images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 52-67, 2018. 3
[55] Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun- Yu Sun, and Xin Tong. O-enn: Octrec-based convolu- tional neural networks for 3d shape analysis. ACM Transactions Graphics (TOG), 36(4):1-11, 2017.
[56] Peng-Shuai Wang. Yang Liu, and Xin Tong. Deep cetree-based cnns with output-guided skip connections for 3d shape and scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 266-267, 2020. 3,9
[57] Peng-Shuai Wang, Chun-Yo Yu Sun, Yang Liu, and Xin Tong. Adaptive o-cnn: A patch-based deep repre- sentation of 3d shapes. ACM Transactions on Graphics (TOG), 37 2018. 3, 7,
[58] Chao Wen, Yinda Zhang, Zhuwen Li, and Yanwei Fu. Pixel2mesh++: Multi-view 3d mesh generation via deformation. In Proceedings of the IEEE International Conference on Computer Vision, pages 1042-
1051, 2019.
[59] Jiajun Wu, Yifan Wang, Tianfan Xue, Xingyuan Sun, Bill Freeman, and Josh Tenenbaum. MarrNet:
3D shape reconstruction via 2.5D sketches. In Proceedings of the Conference on Neural Information
Processing Systems, pages 540-550, 2017. 3
[60] Jajjun Wu, Chengkai Zhang. Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning probabilistic latent space of object shapes via 3D generative-adversaria modeling. In Proceedings of the Conference
on Neural Information Processing Systems, pages 82-90, 2016. 3
[61] Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao.
3D ShapeNets: A deep representation for volumetric shapes. In CVPR, pages 1912-1920, 2015. 3
[62]
Yun-Peng Xiao, Yu-Kun Lai, Lung-Lue Zhang, Chumpeng Li, and Lin Gao. A survey on deep geometry learning: From a representation perspective. Computational Visual Media, 6(2):1 2020.
[63] Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomir Mech, and Ulrich Neumann. DISN: Deep im- plicit surface network for high-quality single-view 3D reconstruction. In Advances in Neural Information
Processing Syxtems, pages 490-500, 2019. 3
[64] Xinchen Yan, Jimei Yang, Ersin Yumer, Yijie Guo, and Honglak Lee. Perspective transformer nets: Learning single-view 3D object reconstruction without 3D supervision. In Proceedings of the Conference on Neural Information Processing Systems, pages 1696 1704, 2016. 3
[65] Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Yu Liu, Serge Belongie, and Bharath Hariharan. Point- Flow: 3D point cloud generation with continuous normalizing flows. In Proceedings the IEEE Inter-
national Conference on Computer Vision, pages 4541 4550, 2019. 3
[66] Jie Yang, Kaichun Mo, Yu-Kun Lai, Leonidas J Guibas, and Lin Gao. Dsm-net: Disentangled structured mesh net for controllable generation of fine geometry. arXiv preprint arXiv:2008. 3, 2020. 3,5 [67] Yaoqing Yang, Chen Feng, Yiru Shen, and Dong Tian. NoldingNet: Point cloud auto-encoder via deep grid deformation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 206-215, 2018. 3
13


Reviews:
[{"review_id": "msrOKHoKpkz", "reviewer": "Reviewer_Nodw", "summary": "The paper proposes a hierarchical implicit grid representation for modeling implicit shape-spaces. While concurrent research has focused either on hierarchical representations in overfitting mode, or on fixed grids in shape-space mode, the paper proposes a technique that simultaneously satisfies these two objectives. The method is tested on a few classes from the synthetic SDF'ied shapenet against a few methods, and has seemingly good reconstruction performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The ability of modeling adaptive/learnable hierarchical representations is a very exciting research direction, and this paper is one the few that has been looking at it.\nHowever, the paper makes the mistake to focus the entire analysis on reconstruction accuracy, rather than showing the capabilities to interpolate between trees with drastically different topologies (and all the ablations / explanations that justify why a VAE model make this possible).\nThis, together with the low'ish quality of the writing (see the many, too many, questions and suggestions below), is what makes me rate the paper as a weak reject.\n**I might raise my score to a weak accept, as a way to be friendly to the fact that NGLOD partially scooped the work, as far as my questions are properly answered.** (RAISED)\n\n\n1. Q: what happens if you disable the VAE loss? I was amazed not to see this ablation, nor a visualization of how the topology of the representation changes during interpolation tasks.\n1. Q: it is very difficult to understand the quality of a latent space from screenshots... do you have a video? Further, in this video a side-by-side comparison to the IM-Net VAE shape-space (to understand the differential gain caused by the hierarchical model) is essential?\n1. Q: can you justify why you only use 5 ShapeNet classes? (this is usually a red flag that results were cherry picked, unless you provide a very strong reasoning)\n1. Q: while CD is commonly used, EMD is rarely seen, while mIOU (for solids) and F1 are much more common. Can you justify why were they not reported?\n1. Q: I am quite concerned about Table2, where I was expecting a **significantly** bigger advantage in terms of memory vs. LIG (main competitor with a fixed grid). You argue the gains are more evident as you consider deeper trees, but provide no rationale on why subdivisions at 5,6,7 levels were not reported? (note L7 is the standard Poisson reconstruction subdiv level). Further, it would have been good to see the CD scores in this analysis as well... Why were they omitted?\n1. Q: Discontinuities solved \"a-la LIG\" is not super elegant, increases memory usage (overlap) and causes visible artefacts (top-right of Fig.4). Why is the NSVF/NGLOD solution of interpolate latent codes not feasible in your setting?\n1. Q: how can one (me) be sure that the generative model has not just retrieved training samples? (show closest model in the training set w.r.t. CD from the generated result to certify this)\n\nFinally, while ACORN / NGLOD are contemporary, I was quite disappointed not to see some qualitative comparisons (would have not affected the rating).\n\n\n## Further questions (Q) and suggestions to the authors\n\n1. Q: what were the improvements that CVPR/TVCG asked, and that you applied? I was expecting to see this in the supplementary\n1. Q: why did you not discuss \"Neural Sparse Voxel Fields\" from NeurIPS'20? While this is radiance rather than SDF based, the underlying hierarchical structure is very similar \n1. There is a disturbing amount of grammatical mistakes that a careful proofread by a native speaker would be able to fix. e.g. at lines L{88,89,149,193,207,214,228,235,247,278}, and this list is not exhaustive\n1. please sort citations in `\\cite` environments! [3,18,26] vs. your [26,3,18], it makes it faster to look up what you are referring to\n1. I can see why you discuss generative modeling in the related works after reading the technical/results sections, but it was not clear at all when reading the introduction. The reasoning is that you are proposing a hierarchical shape-space, while ACORD/NGLOD are mostly overfitting. As such, I would **strongly** recommend you move the discussion of concurrent work to the intro, and mention there the focus on generative modeling? Currently the structure of intro/related sections confused me a little.\n1. Q: I am quite confused about the notation surrounding (1)... why do you need two aggregation operation? If $\\mathcal{V}$ is variance, then it contains an expectation, but then you also have $\\mathbb{E}$. Yet, the metric is the one of a patch, so I would expect only one expectation in this expression...?\n1. Q: Why is (1) more meaningful that a more classical the divergence of the gradients within the patch (which is then related to curvature)? This is what Poisson surface reconstruction uses, coupled with a density estimator (like what you do for weighting the computation of the loss)\n1. Fig.2, you'd mention that the encoders are shared, also, if they are... why bother with giving them indices? \n1. Q: I think you made a mistake in preparing the figure, as the VAE constraints should affect both mean and variance, while you only show it on the variance branch? why? \n1. L199: undefined $\\mathbf{E}$ in Fig.2?\n1. L204 and L204: overloaded CNN $\\mathcal{V}$ and variance operator $\\mathcal{V}$?\n1. Q: why is the VAE prior only applied to the **global** latent code? You have a hierarchy, how do you justify not having it at every level?\n1. L243: a bit more precision than `classifiers` would be good... is it just a linear layer with a two-class softmax?\n1. L260: the importance of pre-training is enormous (supplementary), and should be quickly mentioned in the main text.\n1. L273: summarizing the set of experiments in the supplementary is a good practice?\n1. Q: in Sec 4.2, I would recommend to call this is auto-encoding (reconstructing shape collections) rather than reconstruction (reconstructing a single shape/overfitting).\n1. L304: comparisons in \"c\"?\n1. L316: you are referencing the wrong table\n1. Q: can you justify the weird failure case of LIG in Fig.4? I would expect LIG to just have slightly smoother results than your method. If there is a mistake in how the comparison is performed, the metric in Table.1 (for LIG) could be completely different.\n1. Table2: very unclear whether this is an amortized results (average) or whether it's on a single object/scene?\n1. Q: Sec 4.4: while I am pretty sure in 4.2 you intend auto-encoding, I am similarly pretty sure in this section you are referring to overfitting. If so, why not compare to methods like SIREN?\n1. Q: Sec 4.5: why are you only showing qualitative results? If no quantitative is possible, a clear rationale should be provided! ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No, and it's fine.\n", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "DG5KJbt1K1Q", "reviewer": "Reviewer_tVrM", "summary": "The paper proposed a learnable hierarchical implicit representation for 3D learning, combined with a network architecture that can probabilistically predict hierarchical structure of 3D shape. Experiments on showed some improvements over baselines (although I have some questions, see below)", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. The paper is tackling an important and interesting problem for 3D shape modelling as the memory consumption can increase dramatically if the resolution goes up, and the idea proposed in the paper is intersting as it  can differentially generate hierarchical structure (octree) of implicit functions, which can potentially for other tasks.\n2. The paper had a wide range of experiments to demonstrate the performance including shape reconstruction, generation, interpolation, completion & scene reconstruction.\n3. The paper is well-written and easy to understand\n\nWeakness:\n1. The most related work for this paper is NGLOD [48], the paper cited and discussed in the related work section (Line 109-116), however, it is very hard to claim the paper is concurrent work, as NGLOD  is already accepted to CVPR, and released almost half a year before neuritis ddl, and the paper has already released the code. The paper also argued the advantages of making the occ-tree generation to be differentiable comparing with NGLOD[48], which used a fixed oct-tree, however, in the main experiments, there's no comparisons with NGLOD that can support this argument, I'd recommend the author to add this comparison.\n\n2. Novelty. Predicting hierarchical structure of 3D shape has been studied in many papers in computer vision literatures, as in the cited papers [49, 52, 53, 54], but they're operating on the voxels, the paper extends the ideas from these papers into the implicit function by adding an implicit decoder. Besides, considering many similar works (NeuralLoD[48], ACORN[32]), the contribution dosen't reach the high bar of neurips.\n\n3. Minor problems: a) in table 2, using different level of both LIG and the method in the paper would have different performance, could the paper also provide the preformance of both methods at different level? otherwise it's hard to say which one is better, b) in the scene recconstruction experiments, what's the input? what exactly this experiment is doing? c) in the shape completion experiments, could the author provide quantitative comparison? looking at only two shape would have bias.\n\nTypo: Line 314: Table 2", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There's no discussion on the social impact in the main paper", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "6jytSdrrFLx", "reviewer": "Reviewer_ULW2", "summary": "This paper describes a learned shape representation that is based on hierarchical oct trees that store local implicit functions at the voxels. The implicit functions represent the geometry. It offers higher fidelity and smaller memory footprint than using local implicits without oct trees (LIG, [26]). In comparison to other oct-tree based methods, the proposed approach offers the ability to learn oct-tree hierarchy. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The proposed method is technically sound and achieves good results and improvements over the state of the art. Ability to learn (encode and decode) the hierarchical structure of the octtree, seems like an interesting contribution that can be used in other spatial representations, beyond storing implicit functions. Quantitatively, the proposed method also offers improvements over previous work. \n\nI suggest authors revise their related work section to discuss prior learning techniques that used OctTrees in more details. Right now  only OpenVDB [39] and NGLOD [48] are discussed, while lots of highly relevant techniques [52, 54, 53, 49] are only casually mentioned on line 91. I think they need to be discussed more, especially [54] which is further used as a baseline.  Paper \"Hierarchical surface prediction for 3d object reconstruction\" [24] is only mentioned in the context of data pre-processing, but seems highly related. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Acceptable", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "53fJU0A8lRF", "reviewer": "Reviewer_2BPG", "summary": "This paper presents a new efficient 3D shape representation to reduce computation cost while also being able to preserve geometric details. The authors propose to represent 3D shapes using octree structures such that the hierarchies focus on modeling the surface geometry, where the leaf nodes additionally predicts implicit functions that returns the occupancy for each query 3D point. Both the hierarchical inference and prediction are designed to be differentiable such that the octree structures can be discovered and learned from data. Experiments show that the proposed representation achieves the state of the art on the ShapeNet reconstruction with applications of shape interpolation and completion.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n+ This work combines a neat idea of blending octrees (often used for efficient voxelized shape inference/prediction) and implicit surfaces (for modeling dense and granular surface structures). It saves unnecessary computational costs by terminating early in monotonic occupancy/freespace regions to focus more on modeling the surfaces. I like how the hierarchy can be handled with MLPs, which would be much easier to parallelize while being adaptive to different numbers of input nodes, unlike sparse 3D convolution which requires specialized designs.\n+ The use of additional variables $\\boldsymbol{\\alpha}$ and $\\boldsymbol{\\beta}$ for predicting recursive octree structures is nice, whose differentiability allows end-to-end training.\n+ The proposed method achieves the state of the art performance on the ShapeNet reconstruction benchmark, and the new representation shows good results on scene reconstruction of indoor room scales as well.\n\nWeaknesses:\n- It is unclear why $\\boldsymbol{\\alpha}$ and $\\boldsymbol{\\beta}$ are necessary in the encoder; maintaining architectural symmetry does not seem important here. Furthermore, it is not clear why the hierarchical encoder is important. One can preprocess the input shapes into different 3D representations (meshes, point clouds, naive voxels) and use other encoders to get the latent codes, and still use the hierarchical decoder for shape reconstruction. The effect of the proposed encoder architecture is not very related to 3D shape predictions as presented in the experiments. To validate that the new encoder indeed learns a better latent space, the authors should run another experiment on comparing the learned latent space, e.g. by fitting a linear classifier on the trained latent codes as a measure of discrimination and generalization.\n- The VAE reparametrization seems like an orthogonal component to the paper and only allows the application random shape generation. I would suggest removing it, as the experiments are more focused on shape reconstruction.\n- Are the implicit occupancy decoders (at the leaf nodes) shared? I am assuming yes (like the encoders), but it is not mentioned anywhere in the paper. Also, are the intermediate MLPs shared as well? If so, would it be possible to apply the MLPs (in the decoder part) for an arbitrary number of times until the desired level of details is reached, adaptive to the threshold $\\tau$?\n- An ablation on the $\\tau$ value should be included to better understand its effectiveness. The expectation is that when $\\tau=1$, the proposed decoder would become OccNet [34]; if $\\tau=0$, it would become OGN [49].\n- In Sec 4.1, was a single network trained on multiple shape categories together, or was it category-specific?\n- I'm confused by L280-282. Why was SDF computed where (binary) occupancy prediction is the target goal?\n- It would be helpful to also compare the ShapeNet reconstruction against OGN [49], as it shares a very similar octree generating architecture but for voxels.\n- The setting in the scene reconstruction experiment is unclear. A brief overview of the 3D-Front dataset should be provided (e.g. how many train/test examples). It would also be helpful to provide quantitative results and more visual comparisons for this experiment.\n\nOther minor problems:\n- It would be helpful to enlarge Fig. 4, as it is difficult to compare in detail. A page-width figure would be recommended. It would also be helpful to visualize important parts where the proposed method is significantly superior to the baseline methods (e.g. the bars of the chairs).\n- L129: DeepSDF should not be considered as concurrent work, which is a bit misleading. It would also be helpful to elaborate how this work is different from ACORN [32], as both works share a large similarity in hierarchical modeling of signals (e.g. 3D shapes).\n- L28-29: I don't think voxels are explicit representations since it encodes occupancy at 3D locations, which is different from meshes or point clouds.\n- L247: the notations $\\boldsymbol{\\alpha}$ and $\\boldsymbol{\\beta}$ are overloaded with that from the encoder. It is suggested to use a slightly different notation to discriminate, e.g. $\\boldsymbol{\\hat{\\alpha}}$.\n- An important reference is missing: OctNet [A], which was the first to use an octree structure for inferring 3D shapes.\n\n[A] Riegler et al. \"OctNet: Learning Deep 3D Representations at High Resolutions.\" CVPR 2017", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Adequately addressed", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5hr"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Post rebuttal, the paper was the subject of extensive discussion both between the authors and reviewers and between the reviewers themselves. The reviewers were overall generally in agreement with many facts about the paper, but had good faith disagreements in terms of where to draw boundaries for novelty and contribution. The AC examined the reviews, rebuttals, and discussion and is inclined to agree with the reviewers advocating for acceptance. The AC is persuaded by overall perspective of reviewers 2BPG and Nodw that the paper contributes valuable insights to the field (presented in committee discussions). Moreover, if the reviewers are satisfied that their clarity concerns can be addressed in a revision, then the AC is as well.

The AC would make the following requests:
- The authors should carefully read the comments of the reviewers and be sure each commented is addressed in the final version of the paper. The rebuttal and a promise of revision was persuasive in this case. 
- The authors should add a stronger discussion of the social impact in the paper.
Decision: Accept (Poster)