Intelligent Tools for Creative Graphics
SIGGRAPH 2020 Course

Niloy J. Mitra
1Intelligent Tools for Creative Graphics• Geometric Reasoning• Physical Constraints• Data-Driven Techniques and Machine Learning• Crowdsourcing

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)2

Geometry: Ideation, Design, Fabrication

3

Image from: The Next Leap: How A.I. will change the 3D industry, Andrew Pricecreating high-quality content is challenging and very expensive!!!

geometry
semantics
uv coordinates
mesh quality
motionmaterial assignment
illumination
functionality….

5

Overview• Physics-guided geometric reasoning• Proxy-guided geometric reasoning• Structure-guided geometric reasoning

6

Overview• Physics-guided geometric reasoning• Proxy-guided geometric reasoning• Structure-guided geometric reasoning

7

8

Learning Robustness from Simulations[Designing Chain ReacHon ContrapHons from Causal Graphs, Roussel et al., Siggraph, 2019]9Modeling High-dimensional Design Spaces

10Online Modeling11Simple Example

12

13

Design Options

1. sketching

2. sewing patterns

3. draped garment= interaction(sewing pattern, material, body shape)
14Different Domain Representations[Learning a Shared Shape Space for MulHmodal Garment Design, Wang et al., 2018]15

Multimodal Design16

Learning a Shared Latent Space17

Loss Function Terms

18

Supported Edits`

19

20

21

[Learning an intrinsic garment space for interacHve authoring of garment animaHon,
Wang et al., Siggraph Asia, 2019]22

Overview• Physics-guided geometric reasoning• Proxy-guided geometric reasoning• Structure-guided geometric reasoning

23

Image Manipulation[InteracHve Images: Cuboid Proxies for Smart Image ManipulaHon,
Zheng et al., Siggraph, 2012]24

Proxies: Arrangement of Cuboids

repetitions

co-planarity
25

Edit Examples

26

Proxies: Generalized Cylinders[3-Sweep: ExtracHng Editable Objects from a Single Photo,Chen et al., Siggraph Asia, 2013]

27

28

FrankenGAN Architecture[FrankenGAN: Guided Detail Synthesis for Building Mass Models Using
Style-Synchonized GANs, Kelly et al., 2018]29

FrankenGAN: ‘Procedural’ Steps

input

1st step:windorlayout… progressive modeling …

• No mode
collapse• Style and scalecontrol• Regularisation• Manual edits• High-res30

Input: Madrid31

Output: Madrid32Total:

artistours> $ 500’000< $ 1’000(for the whole scene)

33

Overview• Physics-guided geometric reasoning• Proxy-guided geometric reasoning• Structure-guided geometric reasoning

34

35

backrest

armrestseatleg[StructureNet: Hierarchical Graph Networks for 3D Shape GeneraHon,
Mo et al., Siggraph Asia, 2019]36

backrest

armrestseatleg37

structure: elements (parts) + relationships

38

Examples of Domains with Structure
3d shapes

2D layouts

Patterns

Scene Compositions& Scene Graphs

Li et al., GRAINS: Generative Recursive Autoencoders
for INdoor Scenes, TOG 201839

Why Structure?
Unstructured representation

Structured representation• Regular sample grids and point clouds• Have received more research• Provides additional information
for down-stream tasks

point clouds for man-made 3d objects

images for patterns

Change type of backrest? Transform sub-graph?

• Changes the data flow
and inductive bias of a network
w/o edges with edges

GT

w/o struc. with struc.

40

Goal: A Smooth, Explorable Shape Space

A smooth, common shape space allows forinterpolation, generation, exploration, …
… of both geometry and structure41With vs. Without Structure
targetwithstructure

withoutstructuresource42

Object Representation: Part Geometry43

Object Representation: Part Structure

44

Object Representation: Part Structure

45

Object Representation: Sibling Relationships
Reflectional Symmetry
Rotational Symmetry
Translational Symmetry
Adjacency

46

Object Representation: Part Structure…

47Architecture: Variational Autoencoder
reconstructedshape with structure

shape with structure

encoder

decoderlatent shape space48

A Hierarchy of Graphs
VAE with hierarchical graph encoder and decoder49

Applications
free generationinterpolation

50

Novelty
generatedclosest training samples

51

Comparison to GRASS

52

Applications
reconstruction

editing53

Structuring Shape Distributions:::

latent shape space

: ?

54

Shape Deltas[StructEdit: Learning Structural Shape VariaHon, Mo et al., CVPR, 2020]55

Shape Neighborhoods
geometric neighbors(similar geometry)

Learn conditional distributions

source shapestructural neighbors(similar structure)of deltas in all neighborhoods with a VAE

shape space

56

Infer Main Edit Modes…

latent shape space57

Transfer Analogous Edits:::latent shape space:58

Results: Transferring Analogous Edits59

Latent Space of Programs

60

Further details• Physics-guided geometric reasoning• Proxy-guided geometric reasoning• Structure-guided geometric reasoning

http://geometry.cs.ucl.ac.uk/publications.php
Special thanks to all our collaborators and group members.

63