Exploring Generative 3D Shapes
Using Autoencoder Networks
Nobuyuki Umetani

Motivation• Machine learning is booming

h6ps://docs.microsoC.com/en-us/azure/cogniEve-services/face/overview

Face recogniEon

h6p://theoatmeal.com/blog/
google_self_driving_car

Self-driving car

h6ps://www.engadget.com/2016/03/13/google-alphago-losesto-human-in-one-match/

Alpha Go

Motivation• Machine learning for 3D shape– Understanding a class of shapes
In my understanding, shapes of cars look like..

Motivation• Machine learning for 3D shapes– Finding low dimensional manifold in space: Datum

low-dimensional
manifold
1forward/backward
mapping01

Synthesizing New Shapes• Mapping low dimensional manifold to the shapes

High dimensional shape space

Low dimensional encoding101

Preview of the Result: Slider bar

Preview of the Result: Random

Interactive Exploration

Parameterization problem• Shape need to be represented by ﬁxed
dimensional vector/tensor
2D Image

01
53.3
B ..
CB .
CB
C
B 236.1 CBC
B .
C@ ..
A67.2

3D shape0153.3
B ..
CB .
CBCB 236.1 CB
CB .
C@ ..
A67.2

Parameterization problem• Triangle mesh / NURBS are not suitable for ML– Topology / #points are not constant

Triangle meshNURBS

Related Work: Voxel model• Diﬃcult to synthesize detailed 3D shape– Expensive memory cost, noise…

[FPNN, Li et al., 2017][O-CNN, Wang et al., 2017][Girdhar et al., 2016][3D GAN, 2016][OctNet, Riegler et al., 2017][liu et al., 2017]

Related Work: Multi-view model• Diﬃcult to synthesize 3D data

[Su et al., 2017][Tulsiani et al., 2017][Zhou et al., 2016][Soltani et al. 2017][Park et al., 2017][Lun et al. 2017]

Related Work: Point-based model• Diﬃcult to synthesize detailed 3D shape– Hard to deﬁne surface[PointNet, Qi et al, 2017][PointNet++, Qi et al, 2017][DeformNet, Kurenkov et.al, 2017][Fan et al, 2017][Gadelha et al.,2017

Our Approach: Mesh Representation• Quad mesh with constant topology• Deforming a template mesh into input shapedeformaEon

Our Approach: 3D Shape as Height Field• Storing XYZ coordinates is redundant• Height ﬁeld from a cube in its normalheight2.5 dimensional

3 dimensionalNaïve Projection Leads to Distortion• DistorEon means biased sampling density• Not robust to the occlusion
Never to be sampledocclusion

Never to be sampled

Hierarchical Projection• We repeat subdivision and projecEon• Key observaEon: concave shape is locally convex

iniEalizaEon

subdivision & projecEon

Hierarchical Projection in 3D
heightLevel 0Level 1

Level 2

Level 4

Level 3

Level 5+0.5m

heights

XYZ posiEons…………

……………………………………..

Parameter vector

T-0.2mChallenge• ProjecEng points into polygon soup
• Inverted triangle• IntersecEons• Gaps and holes• Internal structures

Depth Field Techniques• Depth ﬁeld can be eﬃciently computed with GPU• More orthogonal projecEon is more trusted

Parameterization

Over 1,200 Car Shapes from ShapeNet[chang et al. 2015]
Manual annotaEon (5min)

ParameterizaEon(0.3sec)

Data is available on my web page: h6p://nobuyuki-umetani.com/

Autoencoder Network• Input and output of network is as same as possible

6165 dim

10 dim

6165 dim

Minimize diﬀerenceAutoencoder Networkoutputinput• We could train the network preserving shapes

Autoencoder Network• Nonlinear dimensional reducEon
encoderdecoder

6165 dim

10 dim

6165 dimbo6le neckAutoencoder Network• We use Sigmoid funcEon for acEvaEon

6165 dim

10 dim

6165 dim• Decoder takes value between [0,1]6165 dim

{0.3 . . . 0.1, . . . 0.9}T

10 dimAutoencoder Network

Interactive Exploration• Pulling operaEon to guide the synthesis
Minimize
lengthOpEmizeparameters

Interactive Exploration• Our parameterizaEon is close form
– Easily diﬀerenEable for grad-based opEmizaEon
gradient

6165 dim

update{0.3 . . . 0.1, . . . 0.9}T

10 dimgradient
gradientLive Demo!

Limitation• Only handles one-class of shape– Diﬃcult to handle diﬀerent topologies• Only handles nearly convex shapes– StarEng from a template mesh otherthan cube– combinaEons of several meshesFuture work
• Advanced generaEon frameworks– GAN, VAE

h6ps://www.slideshare.net/vsevolodrodionov/itsubbotnik-rodionov-talk-neural-networks-in-js-jsitsubbotnik-2016• Filter operaEons on the subdivision– ConvoluEon operaEon– Gaussian Pyramid / Laplacian Pyramid
h6p://library.wolfram.com/infocenter/Demos/4532/

Acknowledgement• Anonymous reviewers• Ryan Schmidt and Alec Jacobson for many advice

Exploring Generative 3D Shapes
Using Autoencoder Networks

Nobuyuki Umetani