Intelligent Tools for CreativeGraphics
using Data Driven Techniques
Ariel Shamir
The Interdisciplinary Center, Herzliya, IsraelIntelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Data Driven Understanding• Recent years have seen a growth in the amount of data available• Machine learning techniques utilize data for better understanding
and for designing new solutions

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Machine Learning Tasks• Classification• Clustering!
• Searching & pattern matchingesrouCsi• Semantic embeddingThtoN• Cross-modal distances• Generative models

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

The key question:• How can ML be utilized for tools and interfaces?

Fully AutomaticIntelligent ToolIntelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

All ManualWhat can you do with data?
1998: What can you do with a Web in your Pocket?
Sergey Brin , Rajeev Motwani , Lawrence Page , Terry Winograd

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)What can you do with data?
1998: What can you do with a Web in your Pocket?
Sergey Brin , Rajeev Motwani , Lawrence Page , Terry Winograd
2007: Scene Completion Using Millions of Photographs
James Hays, Alexei A. Efros.

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

1. Search (& Replace)
How do we utilize search and replace in an interactive environment?

Sketch2Photo: Internet Image Montage
ACM Transactions on Graphics
Volume 28, Number 5, (SIGGRAPH Asia
Conference Proceedings), 2009
Tao Chen, Ming-Ming Cheng, Ping Tan,
Ariel Shamir, Shi-Min Hu

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)Find, Cut & Paste+++=Your Favorite Search Engine:Filtering: Complex vs. Simple Images

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)Requirements
1. A database(Filtering = changing the metric)
2. A metric
3. An effecitve/fast searching mechanism
4. Compositing…

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)Modeling 3D Objects

Modeling & Designing by Parts

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

3D Objects Repositories• Much less available data
• More difficult to search• Virtual Objects

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Virtual vs. Real

Challenges in Fabrication• Includes many parts and small details• Need to connect all parts• Need to comply with physical constraints such as strength, stability...

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)Designing 3D objects

3-Sweep: Extracting Editable Objects
from a Single Photo
Tao Chen, Zhe Zhu, Ariel Shamir, ShiMin Hu, Daniel Cohen-Or
ACM Transactions on Graphics,
Volume 32, Number 6, (SIGGRAPH
Asia Conference Proceedings), Article
195, 2013

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)Geometric Reasonning: Parameters & Constraints• parallelism• orthogonality• collinear anchors• overlapping anchors• coplanar anchors• coplanar axes

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Parametric Objects

Data Driven: A Database of Example
hierarchical parametricobjects (templates)

Modeling System

Design and Fabrication by Example
Adriana Schulz, Ariel Shamir, David Levin, Pitchaya Sitthi-Amorn, Wojciech Matusik
ACM Transactions on Graphics, Volume 33(4), (SIGGRAPH) 2014

Approach

InputExpert
Data

Composition ToolFabrication

Part Based Modeling & SnappingSearching Connectors

Search Metric• Geometric similarity of components• Functional similarity of unit
• Material properties

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)Example Modeling Session

Interactive vs. Automatic?

Fully AutomaticIntelligent ToolIntelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

All ManualMakers Make Connectors

Makers Make Connectors[OpenSCAD]• Difficulty to design (yet again)• Need (want) customization

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Auto-Connect

AutoConnect: Computational Design of 3DPrintable Connectors
Yuki Koyama, Shinjiro Sueda, Emma Steinhardt,
Takeo Igarashi, Ariel Shamir, Wojciech Matusik
ACM Transactions on Graphics, Volume 34,
Number 6, (SIGGRAPH Asia ConferenceProceedings), Article No. 231, 2015

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Defining a Connector
Bridge partTwo holder parts connectedby a bridge bar…

Common Object
Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Special Object

Workflow

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Holders Database33

Parametrized Holderse.g. heighte.g. closeness34

Metric• Geometric Similarity of components• Physical properties (weight)

Data Driven Physics
SelectOptimizeparameters
DatabaseOutput• Need to optimize shapes• Need to estimate functionality (grip strength)36

Another Domains: Creating Sketches

Style and Abstraction in Portrait Sketching
Itamar Berger, Ariel Shamir, Moshe Mahler, Elizabeth Carter, Jessica Hodgins
ACM Transactions on Graphics, Volume 32, Number 4, (SIGGRAPH Conference
Proceedings), Article No. 55, 2013

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Learning the Style of an Artist’s Sketchings• Data Gathering:Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Database of SketchesIntelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Learning a Parametric SpaceIntelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Converting Images to Sketches in Different Styles

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)Video Editing

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)Text Editing + Database of Videos

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

2. Multi Modal Embedding• Given a word/sentence search for image/video

Embedding/Encoding

Neural Networks

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)Embedding/Encoding
Feature Space

Giraffe or Not?

Positive Feedback

Correct?

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)Embedding/Encoding
Feature Space

Giraffe or Not?

Negative Feedback

Incorrect?

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)Embedding/Encoding
Feature Space

Giraffe or Not?

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)Embedding/Encoding
Feature SpaceIntelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)Embedding/Encoding
Feature SpaceIntelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Multi-Modal Embedding/Encoding
Feature Space

GiraffeIntelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Interface

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)Shot Assembly
Text

Segment 1

1Segment 2

5Segment 3

6giraffe

elephant

Segment L N
Semantic SearchIntelligent Tools for Creative Graphics (SIGGRAPH 2020Course)

247Segment 4

Video Repository

3Video Montage8

Back to Automatic Tools

Fully AutomaticIntelligent ToolIntelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

All ManualSynthesis of Images

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

3. Generative Networks (e.g. GANs)

A Style-Based Generator Architecture forGenerative Adversarial Networks.
Tero Karras, Samuli Laine, Timo Aila
IEEE transactions on pattern analysis andmachine intelligence, 2020

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

GANs: Generative Adverserial Networks• Two networks: the Generator and the Discriminator that work
“against” each otherBillions of Images

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Discriminator = Simple Classifier
RealBillions of Images

Fake!

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Generator:
Tries to Create “Real Samples” from Noise

Billions of Images

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Discriminator:
Rewarded to Discriminate Between Real and Fake

Billions of Images

FeedbackIntelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Generator:
Rewarded to Fool the Discriminate

Billions of Images

FeedbackIntelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

GANs Training• At the end only the generator is used
Billions of Images

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

GANs: Synthesis• Synthesizing new content from random seeds

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)Challenge: Push GANs Towards Interactive Use

Fully AutomaticIntelligent ToolIntelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

All ManualConditional Gans

Condition

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Sketch Regions & Generate

Semantic Image Synthesis with Spatially-Adaptive Normalization
Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu.
in CVPR, 2019Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Data Driven Techniques

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Data Driven Techniques• Utilizing Databases: Search & Replace

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Data Driven Techniques• Utilizing Databases: Search & Replace• Buliding Semantic Parametric Spaces

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Data Driven Techniques• Utilizing Databases: Search & Replace• Buliding Semantic Parametric Spaces• Mapping to Feature Spaces• Createing Multi-modal Embedding Spaces

Intelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Data Driven Techniques• Utilizing Databases: Search & Replace• Buliding Semantic Parametric Spaces• Mapping to Feature Spaces• Createing Multi-modal Embedding Spaces• Learning Priors using Neural Networks• Generative Models - GANsIntelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)Collaborators
Shi-Min Hu, Daniel Cohen-Or,
Tao Chen, Ming-Ming Cheng, Ping Tan, Zhe Zhu,Adriana Schulz, David Levin, Pitchaya Sitthi-Amorn, Wojciech Matusik,
Yuki Koyama, Shinjiro Sueda, Emma Steinhardt, Takeo Igarashi,Itamar Berger, Moshe Mahler, Elizabeth Carter, Jessica Hodgins,
Miao Wang, Guo-Wei Yang, Shing-Tung YauIntelligent Tools for Creative Graphics (SIGGRAPH 2020 Course)

Thank You!

Ariel Shamir
http://www.faculty.idc.ac.il/arik