Learning Unsupervised Hierarchical Part Decomposition of 3D Objects
from a Single RGB Image
Despoina Paschalidou1,3,5 Luc van Gool3,4,5 Andreas Geiger1,2,5
1
Max Planck Institute for Intelligent Systems Tübingen
2
University of Tübingen 3 Computer Vision Lab, ETH Zürich 4 KU Leuven
5
Max Planck ETH Center for Learning Systems{firstname.lastname}@tue.mpg.devangool@vision.ee.ethz.ch

Abstract
Humans perceive the 3D world as a set of distinct objects that are characterized by various low-level (geometry,reflectance) and high-level (connectivity, adjacency, symmetry) properties. Recent methods based on convolutional
neural networks (CNNs) demonstrated impressive progressin 3D reconstruction, even when using a single 2D image
as input. However, the majority of these methods focuses onrecovering the local 3D geometry of an object without considering its part-based decomposition or relations between
parts. We address this challenging problem by proposing anovel formulation that allows to jointly recover the geometry of a 3D object as a set of primitives as well as theirlatent hierarchical structure without part-level supervision.
Our model recovers the higher level structural decomposition of various objects in the form of a binary tree of primitives, where simple parts are represented with fewer primitives and more complex parts are modeled with more components. Our experiments on the ShapeNet and D-FAUST
datasets demonstrate that considering the organization ofparts indeed facilitates reasoning about 3D geometry.

Figure 1: Hierarchical Part Decomposition. We consider
the problem of learning structure-aware representations that
go beyond part-level geometry and focus on part-level relationships. Here, we show our reconstruction as an unbalanced binary tree of primitives, given a single RGB image
as input. Note that our model does not require any supervision on object parts or the hierarchical structure of the 3D
object. We show that our representation is able to model different parts of an object with different levels of abstraction,
leading to improved reconstruction quality.

1. Introduction
Within the first year of their life, humans develop acommon-sense understanding of the physical behavior of
the world [2]. This understanding relies heavily on the ability to properly reason about the arrangement of objects ina scene. Early works in cognitive science [21, 3, 27] stipulate that the human visual system perceives objects as a hierarchical decomposition of parts. Interestingly, while thisseems to be a fairly easy task for the human brain, computer vision algorithms struggle to form such a high-levelreasoning, particularly in the absence of supervision.
The structure of a scene is tightly related to the inherent
hierarchical organization of its parts. At a coarse level, ascene can be decomposed into objects and at a finer level
each object can be represented with parts and these parts
with finer parts. Structure-aware representations go beyond
part-level geometry and focus on global relationships between objects and object parts. In this work, we propose astructure-aware representation that considers part relationships (Fig. 1) and models object parts with multiple levels of abstraction, namely geometrically complex parts aremodeled with more components and simple parts are modeled with fewer components. Such a multi-scale representation can be efficiently stored at the required level of detail,namely with less parameters (Fig. 2).
1

Recent breakthroughs in deep learning led to impressive progress in 3D shape extraction by learning a parametric function, implemented as a neural network, thatmaps an input image to a 3D shape represented as a mesh
[32, 17, 25, 53, 58, 39], a pointcloud [13, 43, 1, 24, 49, 58],
a voxel grid [5, 8, 14, 44, 45, 48, 55], 2.5D depth maps
[26, 19, 42, 11] or an implicit surface [34, 7, 40, 46, 56, 35].
These approaches are mainly focused on reconstructing thegeometry of an object, without taking into consideration
its constituent parts. This results in non-interpretable reconstructions. To address the lack of interpretability, researchers shifted their attention to representations that employ shape primitives [51, 41, 31, 10]. While these methods
yield meaningful semantic shape abstractions, part relationships do not explicitly manifest in their representations.
Instead of representing 3D objects as an unstructured
collection of parts, we propose a novel neural network architecture that recovers the latent hierarchical layout of an
object without structure supervision. In particular, we employ a neural network that learns to recursively partition an
object into its constituent parts by building a latent spacethat encodes both the part-level hierarchy and the part geometries. The predicted hierarchical decomposition is represented as an unbalanced binary tree of primitives. More
importantly, this is learned without any supervision neither
on the object parts nor their structure. Instead, our model
jointly infers these latent variables during training.
In summary, we make the following contributions: We
jointly learn to predict part relationships and per-part geometry without any part-level supervision. The only supervision required for training our model is a watertight meshof the 3D object. Our structure-aware representation yields
semantic shape reconstructions that compare favorably tothe state-of-the-art 3D reconstruction approach of [34], using significantly less parameters and without any additional
post-processing. Moreover, our learned hierarchies have asemantic interpretation, as the same node in the learned tree
is consistently used for representing the same object part.
Experiments on the ShapeNet [6] and the Dynamic FAUST(D-FAUST) dataset [4] demonstrate the ability of our model
to parse objects into structure-aware representations that are
more expressive and geometrically accurate compared toapproaches that only consider the 3D geometry of the object
parts [51, 41, 15, 9]. Code and data is publicly available1 .

2. Related Work
We now discuss the most related primitive-based andstructure-aware shape representations.
Supervised Structure-Aware Representations: Our work
is related to methods that learn structure-aware shape representations that go beyond mere enumeration of object’s
1 https://github.com/paschalidoud/hierarchical primitives

Figure 2: Level of Detail. Our network represents an object as a tree of primitives. At each depth level d, the target
object is reconstructed with 2d primitives, This results in a
representation with various levels of detail. Naturally, reconstructions from deeper depth levels are more detailed.
We associate each primitive with a unique color, thus primitives illustrated with the same color correspond to the same
object part. Note that the above reconstructions are derived
from the same model, trained with a maximum number of
24 = 16 primitives. During inference, the network dynamically combines representations from different depth levels
to recover the final prediction (last column).
parts and recover the higher level structural decompositionof objects based on part-level relations [36]. Li et al. [30]represent 3D shapes using a symmetry hierarchy [54] andtrain a recursive neural network to predict its hierarchicalstructure. Their network learns a hierarchical organizationof bounding boxes and then fills them with voxelized parts.
Note that, this model considers supervision in terms of segmentation of objects into their primitive parts. Closely related to [30] is StructureNet [37] which leverages a graphneural network to represent shapes as n-ary graphs. StructureNet considers supervision both in terms of the primitive
parameters and the hierarchies. Likewise, Hu et al. [22] propose a supervised model that recovers the 3D structure of acable-stayed bridge as a binary parsing tree. In contrast ourmodel is unsupervised, i.e., it does not require supervision
neither on the primitive parts nor the part relations.
Physics-Based Structure-Aware Representations: The
task of inferring higher-level relationships among parts has
also been investigated in different settings. Xu et al. [57] recover the object parts, their hierarchical structure and eachpart’s dynamics by observing how objects are expected tomove in the future. In particular, each part inherits the motion of its parent and the hierarchy emerges by minimizing
the norm of these local displacement vectors. Kipf et al.
[28] explore the use of variational autoencoders for learning the underlying interaction among various moving particles. Steenkiste et al. [52] extend the work of [16] on perceptual grouping of pixels and learn an interaction functionthat models whether objects interact with each other at multiple frames. For both [28, 52], the hierarchical structure
emerges from interactions at multiple timestamps. In contrast to [57, 28, 52], our model does not relate hierarchies tomotion, thus we do not require multiple frames for discovering the hierarchical structure.

Figure 3: Overview. Given an input I (e.g., image, voxel grid), our network predicts a binary tree of primitives P of maximum
depth D. The feature encoder maps the input I into a feature vector c00 . Subsequently, the partition network splits each featured+1
dd
representation cdk in two {cd+12k , c2k+1 }, resulting in feature representations for {1, 2, 4, . . . , 2 } primitives where ck denotes
the feature representation for the k-th primitive at depth d. Each cdk is passed to the structure network that ”assigns” a partof the object to a specific primitive pdk . As a result, each pdk is responsible for representing a specific part of the target shape,denoted as the set of points Xkd . Finally, the geometry network predicts the primitive parameters λdk and the reconstruction
quality qkd for each primitive. To compute the reconstruction loss, we measure how well the predicted primitives match thetarget object (Object Reconstruction) and the assigned parts (Part Reconstruction). We use plate notation to denote repetition
over all nodes k at each depth level d. The final reconstruction is shown on the right.
Supervised Primitive-Based Representations: Zou et al.
[59] exploit LSTMs in combination with a Mixture Density Network (MDN) to learn a cuboid representation from
depth maps. Similarly, Niu et al. [38] employ an RNN
that iteratively predicts cuboid primitives as well as theirsymmetry and connectivity relationships from RGB images.
More recently, Li et al. [31] utilize PointNet++ [43] for predicting per-point properties that are subsequently used for
estimating the primitive parameters, by solving a series oflinear least-squares problems. In contrast to [59, 38, 43],
which require supervision in terms of the primitive parameters, our model is learned in an unsupervised fashion. In
addition, modelling primitives with superquadrics, allows
us to exploit a larger shape vocabulary that is not limited tocubes as in [59, 38] or spheres, cones, cylinders and planes
as in [31]. Another line of work, complementary to ours,
incorporates the principles of constructive solid geometry(CSG) [29] in a learning framework for shape modeling[47, 12, 50, 33]. These works require rich annotations forthe primitive parameters and the sequence of predictions.
Unsupervised Shape Abstraction: Closely related to ourmodel are the works of [51, 41] that employ a convolutional neural network (CNN) to regress the parameters of
the primitives that best describe the target object, in an unsupervised manner. Primitives can be cuboids [51] or superquadrics [41] and are learned by minimizing the discrepancy between the target and the predicted shape, by either computing the truncated bi-directional distance [51] or
the Chamfer-distance between points on the target and thepredicted shape [41]. While these methods learn a flat arrangement of parts, our structure-aware representation decomposes the depicted object into a hierarchical layout ofsemantic parts. This results in part geometries with different levels of granularity. Our model differs from [51, 41]also wrt. the optimization objective. We empirically observe that for both [51, 41], the proposed loss formulations
suffer from various local minima that stem from the natureof their optimization objective. To mitigate this, we use themore robust classification loss proposed in [34, 7, 40] andtrain our network by learning to classify whether points lie
inside or outside the target object. Very recently, [15, 9] explored such a loss for recovering shape elements from 3D
objects. Genova et al. [15] leverage a CNN to learn to predict the parameters of a set of axis-aligned 3D Gaussians
from a set of depth maps rendered at different viewpoints.
Similarly, Deng et al. [9] employ an autoencoder to recover
the geometry of an object as a collection of smooth convexes. In contrast to [15, 9], our model goes beyond the
local geometry of parts and attempts to recover the underlying hierarchical structure of the object parts.

3. Method
In this section, we describe our novel neural networkarchitecture for inferring structure-aware representations.
Given an input I (e.g., RGB image, voxel grid) our goal is
to learn a neural network φθ , which maps the input to a setof primitives that best describe the target object. The target
object is represented as a set of pairs X = {(xi , oi )}N
i=1 ,where xi corresponds to the location of the i-th point andoi denotes its label, namely whether xi lies inside (oi = 1)
or outside (oi = 0) the target object. We acquire these N
pairs by sampling points inside the bounding box of the target mesh and determine their labels using a watertight meshof the target object. During training, our network learns topredict shapes that contain all internal points from the target mesh (oi = 1) and none of the external (oi = 0). We
discuss our sampling strategy in our supplementary.
Instead of predicting an unstructured set of primitives,
we recover a hierarchical decomposition over parts in theform of a binary tree of maximum depth D asd−1
P = {{pdk }2k=0| d = {0 . . . D}}(1)

where pdk is the k-th primitive at depth d. Note that for theand its
k-th node at depth d, its parent is defined as pd−1
bk
2c
d+1
d+1
two children as p2k and p2k+1 .
At every depth level, P reconstructs the target object
with {1, 2, . . . ,M} primitives. M is an upper limit to the
maximum number of primitives and is equal to 2D . Morespecifically, P is constructed as follows: the root node isassociated with the root primitive that represents the entire
shape and is recursively split into two nodes (its children)
until reaching the maximum depth D. This recursive partition yields reconstructions that recover the geometry of thetarget shape using 2d primitives, where d denotes the depth
level (see Fig. 2). Throughout this paper, the term node isused interchangeably with primitive and always refers to theprimitive associated with this particular node.
Every primitive is fully described by a set of parameters λdk that define its shape, size and position in 3D space.
Since not all objects require the same number of primitives,
we enable our model to predict unbalanced trees, i.e. stop
recursive partitioning if the reconstruction quality is sufficient. To achieve this our network also regresses a reconstruction quality for each primitive denoted as qkd . Basedon the value of each qkd the network dynamically stops therecursive partitioning process resulting in parsimonious representations as illustrated in Fig. 1.

3.1. Network Architecture
Our network comprises three main components: (i) the
partition network that recursively splits the shape representation into representations of parts, (ii) the structure network that focuses on learning the hierarchical arrangementof primitives, namely assigning parts of the object to theprimitives at each depth level and (iii) the geometry network that recovers the primitive parameters. An overview
of the proposed pipeline is illustrated in Fig. 3. The firstpart of our pipeline is a feature encoder, implemented witha ResNet-18 [20], ignoring the final fully connected layer.
Instead, we only keep the feature vector of length F = 512
after average pooling.

Partition Network: The feature encoder maps the input I
to an intermediate feature representation c00 ∈ RF that describes the root node p00 . The partition network implements
a function pθ : RF → R2F that recursively partitions thefeature representation cdk of node pdk into two feature repred+1
sentations, one for each children {pd+12k , p2k+1 }:d+1pθ (cdk ) = {cd+12k , c2k+1 }.

(2)Each primitive pdk is directly predicted from cdk without considering the other intermediate features. This implies thatthe necessary information for predicting the primitive parameterization is entirely encapsulated in cdk and not in any
other intermediate feature representation.
Structure Network: Due to the lack of ground-truth supervision in terms of the tree structure, we introduce thestructure network that seeks to learn a pseudo-ground truth
part-decomposition of the target object. More formally, itlearns a function sθ : RF → R3 that maps each feature
representation cdk to hdk a spatial location in R3 .
One can think of each hdk as the (geometric) centroid ofa specific part of the target object. We defined−1
H = {{hdk }2k=0| d = {0 . . . D}}(3)the set of centroids of all parts of the object at all depth
levels. From H and X , we are now able to derive the part
decomposition of the target object as the set of points Xkd
that are internal to a part with centroid hdk .
Note that, in order to learn P, we need to be able to partition the target object into 2d parts at each depth level. At theroot level (d = 0), h00 is the centroid of the target object and
X00 is equal to X . For d = 1, h10 and h11 are the centroidsof the two parts representing the target object. X01 and X11comprise the same points as X00 . For the external points,
the labels remain the same. For the internal points, however, the labels are distributed between X01 and X11 basedon whether h10 or h11 is closer. That is, X01 and X11 eachcontain more external labels and less internal labels compared to X00 . The same process is repeated until we reachthe maximum depth.
More formally, we define the set of points Xkd corresponding to primitive pdk implicitly via its centroid hdk :on(4)
Xkd = Nk (x, o) ∀(x, o) ∈ Xbd−1kc2

Here, Xbd−1
denotes the points of the parent. The functionk2c
Nk (x, o) assigns each (x, o) ∈ Xbd−1
to part pdk if it is closerk2c
to hdk than to hds(k) where s(k) is the sibling of k:(
Nk (x, o) =(x, 1) khdk − xk ≤ khds(k) − xk ∧ o = 1(x, 0) otherwise(5)where α = [α1 , α2 , α3 ] determine the size and  = [1 , 2 ]the shape of the superquadric. If f (x; λ) = 1.0, the given
point x lies on the surface of the superquadric, if f (x; λ) <1.0 the corresponding point lies inside and if f (x; λ) > 1.0
the point lies outside the superquadric. To account for numerical instabilities that arise from the exponentiations in(7), instead of directly using f (x; λ), we follow [23] anduse f (x; λ)1 . Finally, we convert the inside-outside function to an occupancy function, g : R3 → [0, 1]:g(x; λ) = σ (s (1 − f (x; λ)1 ))
(a)

(b)

Figure 4: Structure Network. We visualize the centroids
hdk and the 3D points Xkd that correspond to the estimatedpart pdk for the first three levels of the tree. Fig. 4b explains
visually Eq. (4). We color points based on their closest
centroid hdk . Points illustrated with the color associated toa part are labeled “internal” (o = 1). Points illustrated withgray are labeled “external” (o = 0).
Intuitively, this process recursively associates points to theclosest sibling at each level of the binary tree where the association is determined by the label o. Fig. 4 illustrates thepart decomposition of the target shape using H. We visualize each part with a different color.
Geometry Network: The geometry network learns a function rθ : RF → RK × [0, 1] that maps the feature representation cdk to its corresponding primitive parametrization λdk
and the reconstruction quality prediction qkd :rθ (cdk ) = {λdk , qkd }.

(6)

3.2. Primitive Parametrization
For primitives, we use superquadric surfaces. A detailed
analysis of the use of superquadrics as geometric primitives
is beyond the scope of this paper, thus we refer the reader to[23, 41] for more details. Below, we focus on the properties
most relevant to us. For any point x ∈ R3 , we can determine whether it lies inside or outside a superquadric using
its implicit surface function which is commonly referred toas the inside-outside function:
f (x; λ) =

x
α1

 2



2

+

y
α2

 2 ! 21



2

+

z
α3

 2

1

(7)

(8)

that results in per-point predictions suitable for the classification problem we want to solve. σ(·) is the sigmoid
function and s controls the sharpness of the transition of
the occupancy function. To account for any rigid body motion transformations, we augment the primitive parameterswith a translation vector t = [tx , ty , tz ] and a quaternion
q = [q0 , q1 , q2 , q3 ] [18], which determine the coordinate
system transformation T (x) = R(λ) x + t(λ). Note that in(7), (8) we omit the primitive indexes k, d for clarity. Visualizations of (8) are given in our supplementary.

3.3. Network Losses
Our optimization objective L(P, H; X ) is a weighted
sum over four terms:
L(P, H; X ) = Lstr (H; X ) + Lrec (P; X )+ Lcomp (P; X ) + Lprox (P)(9)

Structure Loss: Using H and X , we can decompose thetarget mesh into a hierarchy of disjoint parts. Namely, eachhdk implicitly defines a set of points Xkd that describe a specific part of the object as described in (4). To quantify how
well H clusters the input shape X we minimize the sum ofsquared distances, similar to classical k-means:
Lstr (H; X ) =

X
hd
k ∈H

1
2d − 1Xo kx − hdk k2 (10)(x,o)∈Xkd

Note that for the loss in (10), we only consider gradientswith respect to H as Xkd is implicitly defined via H. This results in a procedure resembling Expectation-Maximization(EM) for clustering point clouds, where computing Xkd isthe expectation step and each gradient updated corresponds
to the maximization step. In contrast to EM, however, we
minimize this loss across all instances of the training set,leading to parsimonious but consistent shape abstractions.
An example of this clustering process performed at trainingtime is shown in Fig. 4.
Reconstruction Loss: The reconstruction loss measures
how well the predicted primitives match the target shape.
Similar to [15, 9], we formulate our reconstruction loss as abinary classification problem, where our network learns topredict the surface boundary of the predicted shape by classifying whether points in X lie inside or outside the target
object. To do this, we first define the occupancy functionof the predicted shape at each depth level. Using the occupancy function of each primitive defined in (8), the occupancy function of the overall shape at depth d becomes:
Gd (x) = max gkd x; λdk(11)(a) Inputk∈0...2d −1

Note that (11) is simply the union of the per-primitive occupancy functions. We formulate our reconstruction loss wrt.
the object and wrt. each part of the object as follows
Lrec (P; X ) =

D
X X


L Gd (x), o +(12)(b) Prediction(c) Predicted Hierarchy(d) Input(x,o)∈X d=0
d

D 2X
−1
X

X

 
L gkd x; λdk , o (13)d=0 k=0 (x,o)∈X dkwhere L(·) is the binary cross entropy loss. The first term
is an object reconstruction loss (12) and measures how well
the predicted shape at each depth level matches the target
shape. The second term (13) which we refer to as part reconstruction loss measures how accurately each primitive
pdk matches the part of the object it represents, defined as thepoint set Xkd . Note that the part reconstruction loss enforces
non-overlapping primitives, as Xkd are non-overlapping byconstruction. We illustrate our reconstruction loss in Fig. 3.
Compatibility Loss: This loss measures how well ourmodel is able to predict the expected reconstruction quality
qkd of a primitive pdk . A standard metric for measuring thereconstruction quality is the Intersection over Union (IoU).
We therefore task our network to predict the reconstruction
quality of each primitive pdk in terms of its IoU wrt. the partof the object Xkd it represents:(e) Prediction(f) Predicted Hierarchy

Figure 5: Predicted Hierarchies on ShapeNet. Our model
recovers the geometry of an object as an unbalanced hierarchy over primitives, where simpler parts (e.g. base of thelamp) are represented with few primitives and more complex parts (e.g. wings of the plane) with more.
Proximity Loss: This term is added to counteract vanishing gradients due to the sigmoid in (8). For example, if theinitial prediction of a primitive is far away from the target
object, the reconstruction loss will be large while its gradients will be small. As a result, it is impossible to “move”
this primitive to the right location. Thus, we introduce a
proximity loss which encourages the center of each primitive pdk to be close to the centroid of the part it represents:dd

Lcomp (P; X ) =

D 2X
−1
X

2
qkd − IoU(pdk , Xkd )(14)

Lprox (P) =

D 2X
−1
X

kt(λdk ) − hdk k2(15)

d=0 k=0d=0 k=0

During inference, qkd allows for further partitioning primitives whose IoU is below a threshold qth and to stop if thereconstruction quality is high (the primitive fits the object
part well). As a result, our model predicts an unbalanced
tree of primitives where objects can be represented withvarious number of primitives from 1 to 2D . This resultsin parsimonious representations where simple parts are represented with fewer primitives. We empirically observe thatthe threshold value qth does not significantly affect our results, thus we empirically set it to 0.6. During training, we
do not use the predicted reconstruction quality qkd to dynamically partition the nodes but instead predict the full tree.

where t(λdk ) is the translation vector of the primitive pdk andhdk is the centroid of the part it represents. We demonstrate
the vanishing gradient problem in our supplementary.

4. Experiments
In this section, we provide evidence that our structureaware representation yields semantic shape abstractions
while achieving competitive (or even better results) thanvarious state-of-the-art shape reconstruction methods, suchas [34]. Moreover, we also investigate the quality of thelearned hierarchies and show that the use of our structureaware representation yields semantic scene parsings. Im-

OccNet [34]Chamfer-L1
SQs [41] SIF [15]CvxNets [9]Ours

OccNet [34]SQs [41]

IoU
SIF [15]

CvxNets [9]Ours

Category
airplanebenchcabinetcar
chairdisplaylamp
speaker
rifle
sofa
table
phone
vessel

0.147
0.155
0.167
0.159
0.228
0.278
0.479
0.300
0.141
0.194
0.189
0.140
0.218

0.122
0.114
0.087
0.117
0.138
0.106
0.189
0.132
0.127
0.106
0.110
0.112
0.125

0.065
0.131
0.102
0.056
0.192
0.208
0.454
0.253
0.069
0.146
0.264
0.095
0.108

0.093
0.133
0.160
0.103
0.337
0.223
0.795
0.462
0.106
0.164
0.358
0.083
0.173

0.175
0.153
0.087
0.141
0.114
0.137
0.169
0.108
0.203
0.128
0.122
0.149
0.178

0.571
0.485
0.733
0.737
0.501
0.471
0.371
0.647
0.474
0.680
0.506
0.720
0.530

0.456
0.202
0.110
0.650
0.176
0.200
0.189
0.136
0.519
0.122
0.180
0.185
0.471

0.530
0.333
0.648
0.657
0.389
0.491
0.260
0.577
0.463
0.606
0.372
0.658
0.502

0.598
0.461
0.709
0.675
0.491
0.576
0.311
0.620
0.515
0.677
0.473
0.719
0.552

0.529
0.437
0.658
0.702
0.526
0.633
0.441
0.660
0.435
0.693
0.491
0.770
0.570

mean

0.215

0.122

0.165

0.245

0.143

0.571

0.277

0.499

0.567

0.580

Table 1: Single Image Reconstruction on ShapeNet. Quantitative evaluation of our method against OccNet [34] andprimitive-based methods with superquadrics [41] (SQs), SIF [15] and CvxNets [9]. We report the volumeteric IoU (higher isbetter) and the Chamfer-L1 distance (lower is better) wrt. the ground-truth mesh.
experiment with the Dynamic FAUST (D-FAUST) dataset[4], which contains meshes for 129 sequences of 10 humans
performing various tasks, such as ”running”, ”punching” or”shake arms”. We randomly divide these sequences intotraining (91), test (29) and validation (9).
Baselines: Closely related to our work are the shape parsing methods of [51] and [41] that employ cuboids and superquadric surfaces as primitives. We refer to [41] as SQs
and we evaluate using their publicly available code2 . Moreover, we also compare to the Structured Implicit Function (SIF) [15] that represent the object’s geometry as theisolevel of the sum of a set of Gaussians and to the CvxNets[9] that represent the object parts using smooth convex
shapes. Finally, we also report results for OccNet [34],which is the state-of-the-art implicit shape reconstruction
technique. Note that in contrast to us, [34] does not consider part decomposition or any form of latent structure.
Figure 6: Predicted Hierarchies on D-FAUST. We visualize the input RGB image (a), the prediction (b) and thepredicted hierarchy (c). We associate each primitive with acolor and we observe that our network learns semantic mappings of body parts across different articulations, e.g. node(3, 3) is used for representing the upper part of the left leg,
whereas node (1, 1) is used for representing the upper body.
plementation details and ablations on the impact of various
components of our model are detailed in the supplementary.
Datasets: First, we use the ShapeNet [6] subset of Choyet al. [8], training our model using the same image renderings and train/test splits as Choy et al. Furthermore, we also

Evaluation Metrics: Similar to [34, 15, 9], we evaluate
our model quantitatively and report the mean Volumetric
IoU and the Chamfer-L1 distance. Both metrics are discussed in detail in our supplementary.

4.1. Results on ShapeNet
We evaluate our model on the single-view 3D reconstruction task and compare against various state-of-the-art
methods. We follow the standard experimental setup andtrain a single model for the 13 ShapeNet objects. Both ourmodel and [41] are trained for a maximum number of 64
2 https://superquadrics.com

(a) Input(b) SQs(c) Ours (d) Input(e) SQs(f) Ours

Figure 7: Single Image 3D Reconstruction. The input image is shown in (a, d), the other columns show the resultsof our method (c, f) compared to [41] (b, e). Additional
qualitative results are provided in the supplementary.
primitives (D = 6). For SIF [15] and CvxNets [9] the reported results are computed using 50 shape elements. The
quantitative results are reported in Table 1. We observethat our model outperforms the primitive-based baselines interms of the IoU as well as the OccNet [34] for the majority of objects (7/13). Regarding Chamfer-L1 , our model isthe second best amongst primitive representations, as [41]is optimized for this metric. This also justifies that [41] performs worse in terms of IoU. While our model performs on
par with existing state-of-the-art primitive representationsin terms of Chamfer-L1 , it also recovers hierarchies, which
none of our baselines do. A qualitative comparison of ourmodel with SQs [41] is depicted in Fig. 7. Fig. 5 visualizes
the learned hierarchy for this model. We observe that ourmodel recovers unbalanced binary trees that decompose a
3D object into a set of parts. Note that [51, 41] were originally introduced for volumetric 3D reconstruction, thus weprovide an experiment on this task in our supplementary.

(a) Input(b) SQs(c) Ours (d) Input(e) SQs(f) Ours

Figure 8: Single Image 3D Reconstruction. Qualitative
comparison of our reconstructions (c, f), to [41] that does
not consider any form of structure (b, e). The input RGB
image is shown in (a, d). Note how our representation
yields geometrically more accurate reconstructions, while
being semantic, e.g., the primitive colored in blue consistently represents the head of the human while the primitive
colored in orange captures the left thigh. Additional qualitative results are provided in the supplementary.

SQs [41]
Ours

IoU

Chamfer-L1

0.608
0.699

0.189
0.098

Table 2: Single Image Reconstruction on D-FAUST. We
report the volumetric IoU and the Chamfer-L1 wrt. theground-truth mesh for our model compared to [41].
performs [41] also in terms of Chamfer-L1. Due to lack ofspace, we only illustrate the predicted hierarchies up to thefourth depth level. The full hierarchies are provided in the
supplementary.

4.2. Results on D-FAUST

5. Conclusion

We also demonstrate results on the Dynamic FAUST (DFAUST) dataset [4], which is very challenging due to thefine structure of the human body. We evaluate our model on
the single-view 3D reconstruction task and compare with[41]. Both methods are trained for a maximum number of
32 primitives (D = 5). Fig. 6 illustrates the predicted hierarchy on different humans from the test set. We note thatthe predicted hierarchies are indeed semantic, as the samenodes are used for modelling the same part of the humanbody. Fig. 8 compares the predictions of our model with
SQs. We observe that while our baseline yields more parsimonious abstractions, their level of detail is limited. On
the contrary, our model captures the geometry of the humanbody with more detail. This is also validated quantitatively,from Table 2. Note that in contrast to ShapeNet, D-FAUST
does not contain long, thin (e.g. legs of tables, chairs) orhollow parts (e.g. cars), thus optimizing for either ChamferL1 or IoU leads to similar results. Hence, our method out-

We propose a learning-based approach that jointly predicts part relationships together with per-part geometries in
the form of a binary tree without requiring any part-level annotations for training. Our model yields geometrically accurate primitive-based reconstructions that outperform existing shape abstraction techniques while performing competitively with more flexible implicit shape representations.
In future work, we plan to to extend our model and predict hierarchical structures that remain consistent in time,thus yielding kinematic trees of objects. Another future direction, is to consider more flexible primitives such as general convex shapes and incorporate additional constraints
e.g. symmetry to further improve the reconstructions.

Acknowledgments
This research was supported by the Max Planck ETHCenter for Learning Systems and a HUAWEI research gift.

References[1] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, andLeonidas J. Guibas. Learning representations and generativemodels for 3d point clouds. In Proc. of the International
Conf. on Machine learning (ICML), 2018. 2[2] Renée Baillargeon. Infants’ physical world. Current directions in psychological science, 13(3):89–94, 2004. 1[3] Irving Biederman. Human image understanding: Recent research and a theory. Computer Vision, Graphics, and Image
Processing, 1986. 1[4] Federica Bogo, Javier Romero, Gerard Pons-Moll, andMichael J. Black. Dynamic FAUST: registering human bodies in motion. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2017. 2, 7, 8
[5] André Brock, Theodore Lim, James M. Ritchie, and Nick
Weston. Generative and discriminative voxel modeling withconvolutional neural networks. arXiv.org, 1608.04236, 2016.
2[6] Angel X. Chang, Thomas A. Funkhouser, Leonidas J.
Guibas, Pat Hanrahan, Qi-Xing Huang, Zimo Li, Silvio
Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong
Xiao, Li Yi, and Fisher Yu. Shapenet: An information-rich
3d model repository. arXiv.org, 1512.03012, 2015. 2, 7
[7] Zhiqin Chen and Hao Zhang. Learning implicit fields forgenerative shape modeling. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018. 2, 3
[8] Christopher Bongsoo Choy, Danfei Xu, JunYoung Gwak,
Kevin Chen, and Silvio Savarese. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction.
In Proc. of the European Conf. on Computer Vision (ECCV),
2016. 2, 7
[9] Boyang Deng, Kyle Genova, Soroosh Yazdani, Sofien
Bouaziz, Geoffrey Hinton, and Andrea Tagliasacchi.
Cvxnets: Learnable convex decomposition. arXiv.org, 2019.
2, 3, 5, 7, 8
[10] Theo Deprelle, Thibault Groueix, Matthew Fisher,
Vladimir G Kim, Bryan C Russell, and Mathieu Aubry.
Learning elementary structures for 3d shape generation andmatching. In Advances in Neural Information Processing
Systems (NIPS), 2019. 2[11] Simon Donne and Andreas Geiger. Learning non-volumetric
depth fusion using successive reprojections. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2019. 2[12] Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, andJoshua B. Tenenbaum. Learning to infer graphics programsfrom hand-drawn images. In Advances in Neural Information Processing Systems (NIPS), 2018. 3[13] Haoqiang Fan, Hao Su, and Leonidas J. Guibas. A point
set generation network for 3d object reconstruction from asingle image. Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), 2017. 2[14] Matheus Gadelha, Subhransu Maji, and Rui Wang. 3d shapeinduction from 2d views of multiple objects. In Proc. of theInternational Conf. on 3D Vision (3DV), 2017. 2[15] Kyle Genova, Forrester Cole, Daniel Vlasic, Aaron Sarna,
William T. Freeman, and Thomas A. Funkhouser. Learning

shape templates with structured implicit functions. In Proc.
of the IEEE International Conf. on Computer Vision (ICCV),
2019. 2, 3, 5, 7, 8
[16] Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber.
Neural expectation maximization. In Advances in Neural Information Processing Systems (NIPS), 2017. 2[17] Thibault Groueix, Matthew Fisher, Vladimir G. Kim,
Bryan C. Russell, and Mathieu Aubry. AtlasNet: A papiermâché approach to learning 3d surface generation. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2018. 2[18] William Rowan Hamilton. Xi. on quaternions; or on a new
system of imaginaries in algebra. The London, Edinburgh,
and Dublin Philosophical Magazine and Journal of Science,
33(219):58–60, 1848. 5[19] Wilfried Hartmann, Silvano Galliani, Michal Havlena, Luc
Van Gool, and Konrad Schindler. Learned multi-patch similarity. In Proc. of the IEEE International Conf. on Computer
Vision (ICCV), 2017. 2[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.
Deep residual learning for image recognition. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2016. 4[21] Donald D Hoffman and Whitman A Richards. Parts of recognition. Cognition, 18(1-3):65–96, 1984. 1[22] Fangqiao Hu, Jin Zhao, Yong Hunag, and Hui Li. Learning
structural graph layouts and 3d shapes for long span bridges
3d reconstruction. arXiv.org, abs/1907.03387, 2019. 2[23] Ales Jaklic, Ales Leonardis, and Franc Solina. Segmentation and Recovery of Superquadrics, volume 20 of Computational Imaging and Vision. Springer, 2000. 5[24] Li Jiang, Shaoshuai Shi, Xiaojuan Qi, and Jiaya Jia. GAL:
geometric adversarial loss for single-view 3d-object reconstruction. In Proc. of the European Conf. on Computer Vision(ECCV), 2018. 2[25] Angjoo Kanazawa, Shubham Tulsiani, Alexei A. Efros, andJitendra Malik. Learning category-specific mesh reconstruction from image collections. In Proc. of the European Conf.
on Computer Vision (ECCV), 2018. 2[26] Abhishek Kar, Christian Häne, and Jitendra Malik. Learninga multi-view stereo machine. In Advances in Neural Information Processing Systems (NIPS), 2017. 2[27] Katherine D Kinzler and Elizabeth S Spelke. Core systems inhuman cognition. Progress in brain research, 164:257–264,
2007. 1[28] Thomas Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max
Welling, and Richard Zemel. Neural relational inference forinteracting systems. In Proc. of the International Conf. on
Machine learning (ICML), 2018. 2[29] David H Laidlaw, W Benjamin Trumbore, and John F
Hughes. Constructive solid geometry for polyhedral objects.
In ACM Trans. on Graphics, 1986. 3[30] Jun Li, Kai Xu, Siddhartha Chaudhuri, Ersin Yumer,
Hao (Richard) Zhang, and Leonidas J. Guibas. GRASS:
generative recursive autoencoders for shape structures. ACM
Trans. on Graphics, 36(4), 2017. 2

[31] Lingxiao Li, Minhyuk Sung, Anastasia Dubrovina, Li Yi,
and Leonidas Guibas. Supervised fitting of geometric primitives to 3d point clouds. In Proc. IEEE Conf. on Computer
Vision and Pattern Recognition (CVPR), 2019. 2, 3[32] Yiyi Liao, Simon Donne, and Andreas Geiger. Deep marching cubes: Learning explicit surface representations. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2018. 2[33] Yunchao Liu, Zheng Wu, Daniel Ritchie, William T Freeman, Joshua B Tenenbaum, and Jiajun Wu. Learning to describe scenes with programs. In Proc. of the International
Conf. on Learning Representations (ICLR), 2019. 3[34] Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks:
Learning 3d reconstruction in function space. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2019. 2, 3, 6, 7, 8[35] Mateusz Michalkiewicz, Jhony Kaesemodel Pontes, Dominic Jack, Mahsa Baktashmotlagh, and Anders P. Eriksson.
Implicit surface representations as layers in neural networks.
In Proc. of the IEEE International Conf. on Computer Vision(ICCV), 2019. 2[36] Niloy J. Mitra, Michael Wand, Hao Zhang, Daniel CohenOr, Vladimir G. Kim, and Qi-Xing Huang. Structure-aware
shape processing. In ACM Trans. on Graphics, pages 13:1–
13:21, 2014. 2[37] Kaichun Mo, Paul Guerrero, Li Yi, Hao Su, Peter Wonka,
Niloy Mitra, and Leonidas Guibas. Structurenet: Hierarchical graph networks for 3d shape generation. In ACM Trans.
on Graphics, 2019. 2[38] Chengjie Niu, Jun Li, and Kai Xu. Im2struct: Recovering
3d shape structure from a single RGB image. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2018. 3[39] Junyi Pan, Xiaoguang Han, Weikai Chen, Jiapeng Tang, andKui Jia. Deep mesh reconstruction from single RGB images
via topology modification networks. In Proc. of the IEEE
International Conf. on Computer Vision (ICCV), 2019. 2[40] Jeong Joon Park, Peter Florence, Julian Straub, Richard A.
Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation.
In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2018. 2, 3[41] Despoina Paschalidou, Ali Osman Ulusoy, and Andreas
Geiger. Superquadrics revisited: Learning 3d shape parsing
beyond cuboids. In Proc. IEEE Conf. on Computer Vision
and Pattern Recognition (CVPR), 2019. 2, 3, 5, 7, 8[42] Despoina Paschalidou, Ali Osman Ulusoy, Carolin Schmitt,
Luc van Gool, and Andreas Geiger. Raynet: Learning volumetric 3d reconstruction with ray potentials. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
2018. 2[43] Charles R Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a
metric space. In Advances in Neural Information Processing
Systems (NIPS), 2017. 2, 3
[44] Danilo Jimenez Rezende, S. M. Ali Eslami, Shakir Mohamed, Peter Battaglia, Max Jaderberg, and Nicolas Heess.

Unsupervised learning of 3d structure from images. In Advances in Neural Information Processing Systems (NIPS),
2016. 2[45] Gernot Riegler, Ali Osman Ulusoy, and Andreas Geiger.
Octnet: Learning deep 3d representations at high resolutions.
In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2017. 2[46] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li. Pifu: Pixel-aligned
implicit function for high-resolution clothed human digitization. In Proc. of the IEEE International Conf. on Computer
Vision (ICCV), 2019. 2[47] Gopal Sharma, Rishabh Goyal, Difan Liu, Evangelos
Kalogerakis, and Subhransu Maji. Csgnet: Neural shapeparser for constructive solid geometry. In Proc. IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR), 2018.
3[48] David Stutz and Andreas Geiger. Learning 3d shape completion from laser scan data with weak supervision. In Proc.
IEEE Conf. on Computer Vision and Pattern Recognition(CVPR), 2018. 2[49] Hugues Thomas, Charles R. Qi, Jean-Emmanuel Deschaud,
Beatriz Marcotegui, François Goulette, and Leonidas J.
Guibas. Kpconv: Flexible and deformable convolution forpoint clouds. In Proc. of the IEEE International Conf. on
Computer Vision (ICCV), 2019. 2[50] Yonglong Tian, Andrew Luo, Xingyuan Sun, Kevin Ellis,
William T Freeman, Joshua B Tenenbaum, and Jiajun Wu.
Learning to infer and execute 3d shape programs. In Proc. of
the International Conf. on Learning Representations (ICLR),
2019. 3[51] Shubham Tulsiani, Hao Su, Leonidas J. Guibas, Alexei A.
Efros, and Jitendra Malik. Learning shape abstractions byassembling volumetric primitives. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2017. 2,
3, 7, 8[52] Sjoerd van Steenkiste, Michael Chang, Klaus Greff, andJürgen Schmidhuber. Relational neural expectation maximization: Unsupervised discovery of objects and their interactions. In Proc. of the International Conf. on Learning
Representations (ICLR), 2018. 2[53] Nanyang Wang, Yinda Zhang, Zhuwen Li, Yanwei Fu, Wei
Liu, and Yu-Gang Jiang. Pixel2mesh: Generating 3d meshmodels from single rgb images. In Proc. of the European
Conf. on Computer Vision (ECCV), 2018. 2[54] Yanzhen Wang, Kai Xu, Jun Li, Hao Zhang, Ariel Shamir,
Ligang Liu, Zhi-Quan Cheng, and Yueshan Xiong. Symmetry hierarchy of man-made objects. In EUROGRAPHICS,
2011. 2[55] Haozhe Xie, Hongxun Yao, Xiaoshuai Sun, Shangchen
Zhou, and Shengping Zhang. Pix2vox: Context-aware 3dreconstruction from single and multi-view images. In Proc.
of the IEEE International Conf. on Computer Vision (ICCV),
2019. 2[56] Qiangeng Xu, Weiyue Wang, Duygu Ceylan, Radomı́rMech, and Ulrich Neumann. DISN: deep implicit surface
network for high-quality single-view 3d reconstruction. In

Advances in Neural Information Processing Systems (NIPS),
pages 490–500, 2019. 2[57] Zhenjia Xu, Zhijian Liu, Chen Sun, Kevin Murphy,
William T Freeman, Joshua B Tenenbaum, and Jiajun Wu.
Unsupervised discovery of parts, structure, and dynamics. In
Proc. of the International Conf. on Learning Representations(ICLR), 2019. 2[58] Guandao Yang, Xun Huang, Zekun Hao, Ming-Yu Liu,
Serge J. Belongie, and Bharath Hariharan. Pointflow: 3d
point cloud generation with continuous normalizing flows.
In Proc. of the IEEE International Conf. on Computer Vision(ICCV), 2019. 2[59] C. Zou, E. Yumer, J. Yang, D. Ceylan, and D. Hoiem. 3dprnn: Generating shape primitives with recurrent neural networks. In Proc. of the IEEE International Conf. on Computer
Vision (ICCV), 2017. 3