Introduction to Computational DesignPart 1: What is Computational Design?
Presenter: Yuki Koyama (AIST, Japan)

2021-05-12 | CHI 2021 Course | Online

Introduction: Digitally Designing Contents

Professional designers do various design using computers

Graphics

Everyday objects

Architectures

Non-professionals also do various design


• 



-" %

– )'1(07* 6&'4–  (-" " • !!
" -#%,=:A9>@-1&)– 5/(1" (!'$– 
+"48;<B9?A/%/%
4-#%

• %3

Photo composition and coloring

Web design

6&.!2

Talk slide designIs there any common process in designing these contents?

Let’s think about designing a name card

Adobe IllustratorAdobe Illustrator

A common task: adjustment of design parameters

Example: Adjustment of the font size

Not good
because it is too small

Not good
because it is too large

Not good
because it is too small(At least) better
than the other two

Not good
because it is too large

Need trials and errors to find the best font size

Not good
because it is too small(At least) better
than the other two

Not good
because it is too large

Goodness of design

Design parameters(font size)

Goodness of design

Some function

Design parameters(font size)

Goodness of design

Some function

Design parameters(font size)

An “optimal” font sizeWhat this means?


Trials and errors
on design parameters⇤x = arg max Goodness(x)Trials and errors
on design parametersx2X

A mathematicaloptimization problem

What is Computational Design?

A Definition of “Computational Design”

20

A Definition of “Computational Design”
A general definition:
Computational design is a paradigm in which design problems are formulated
mathematically and solved by computational techniques.

21

A Definition of “Computational Design”
A general definition:
Computational design is a paradigm in which design problems are formulated
mathematically and solved by computational techniques.
A more focused (narrower) definition:
Computational design is a paradigm in which design problems are formulated
as optimization problems and solved by computational techniques.
22Computational Design Research
Approach:• Formulate design processes that have been traditionally dependent on
individual skills as mathematical optimization problems, and• Support or augment design processes by devising new ways of utilizing
computing power and mathematical tools

Goal:• Enable eﬃcient design workflow or sophisticated design outcomes that are
impossible in traditional approaches relying purely on the human brain

23

Computational Design Research
Approach:• Formulate design processes that have been traditionally dependent on
individual skills as mathematical optimization problems, and• Support or augment design processes by devising new ways of utilizing
computing power and mathematical tools

Goal:• Enable eﬃcient design workflow or sophisticated design outcomes that are
impossible in traditional approaches relying purely on the human brain

24

Optimization: Basics
Note: If you are already familiar with numerical optimization,then you can take a rest for the next 5 min.

Optimization: Description in Wikipedia“Mathematical optimization […] is the selection of abest element, with regard to some criterion,
from some set of available alternatives. […] In thesimplest case, an optimization problem consistsof maximizing or minimizing a real function bysystematically choosing input values from within anallowed set and computing the value of the function.
[…]”

https://en.wikipedia.org/wiki/Mathematical_optimization
26

Optimization: Easy Example
Question:2

Find x that minimizes y = 2x + 8x + 6.

A possible answer:2

The function can be written as y = 2(x + 2) − 2.
This is a U-shaped parabola whose vertex is(−2, − 2). Thus, y is minimized when x = − 2.

27

Optimization: Easy Example
Question:

3

2

Find x that minimizes y = 2x + 8x + 6.

2

1

A possible answer:-4

2

The function can be written as y = 2(x + 2) − 2.
This is a U-shaped parabola whose vertex is(−2, − 2). Thus, y is minimized when x = − 2.

-3

-2

-1

0

1

2

-1

-2

-3

28

Optimization: Notation and Terminology

29

Optimization: Notation and Terminology

Optimalsolution

Searchvariable

Searchspace

Objectivefunction
30

Solve Optimization Problems with Computers• We could obtain the analytic solution
by hand in the easy example• However, most problems in the realworld cannot be solved analytically• Objective functions are not simple?

• Thus, we solve these problems using

numerical optimization algorithms
and obtain the numerical solution

31

Local Solution vs. Global Solution• Local solution• Minimum/maximum at least among theneighborhoods• Global solution• Minimum/maximum among all possible
variables• More diﬃcult to find (depending on
Local
solution

Local & global
solution

objective functions)32

Local Optimization Algorithms• Idea: Iteratively refine the solution• Example: Hill climbing algorithm• Given the current solution x, check its

Local
optimizationneighbors. If there is a neighborelement that improves the objective,then replace the current solution with it• Repeat this procedure untilconvergence

Jorge Nocedal and Stephen J. Wright. 2006. Numerical Optimization (2nd ed.). Springer Science+Business Media.
https://doi.org/10.1007/978-0-387-40065-5
33

Local Optimization Algorithms• Idea: Iteratively refine the solution• Example: Hill climbing algorithm• Given the current solution x, check its

Local
optimizationneighbors. If there is a neighborelement that improves the objective,then replace the current solution with it• Repeat this procedure untilconvergence

Jorge Nocedal and Stephen J. Wright. 2006. Numerical Optimization (2nd ed.). Springer Science+Business Media.
https://doi.org/10.1007/978-0-387-40065-5
34

Modeling Design as Optimization⇤x = arg max Goodness(x)Trials and errors
on design parametersx2X

A mathematicaloptimization problem

Modeling Design as Optimization

Optimal solution

Search variable

Search space

Objective function37Modeling Design as Optimization

Optimal solution

Search variable

Search space

Objective function

Design outcome
The result obtained
after the design
process38

Modeling Design as Optimization

Optimal solution

Search variable

Design outcome
The result obtained
after the design
process

Design parameters
The parameters we
can manipulate in
the design process

Search space

Objective function

39Modeling Design as Optimization

Optimal solution

Search variable

Search space

Design outcome
The result obtained
after the design
process

Design parameters
The parameters we
can manipulate in
the design process

Design space
The set of all
possible design
parameters

Objective function40Modeling Design as Optimization

Optimal solution

Search variable

Search space

Objective function

Design outcome
The result obtained
after the design
process

Design parameters
The parameters we
can manipulate in
the design process

Design space
The set of all
possible design
parameters

Design goal
The criterion about
which design isconsidered good41

Modeling Design as Optimization
Search strategy
Design strategy
The strategy on how
to manipulate thecurrent design

Optimal solution

Search variable

Search space

Objective function

Design outcome
The result obtained
after the design
process

Design parameters
The parameters we
can manipulate in
the design process

Design space
The set of all
possible design
parameters

Design goal
The criterion about
which design isconsidered good42

Where to Use Computational Techniques?

Baseline: Perform everything manually without using
any computational techniques43Where to Use Computational Techniques?
Searchstrategy

Search variable& search space

Objectivefunction44

Where to Use Computational Techniques?
Searchstrategy

Search variable& search space

Objectivefunction• Human performance model• Machine learning (e.g., regression)• Physical simulation• Heuristic rules• Human computation• Psychological experiment• …
45

Where to Use Computational Techniques?
Searchstrategy

Search variable& search space

Objectivefunction• Numerical optimization• Visualization• Recommendation• Data mining• Interactive systems• …

• Human performance model• Machine learning (e.g., regression)• Physical simulation• Heuristic rules• Human computation• Psychological experiment• …
46

Where to Use Computational Techniques?
Searchstrategy

Search variable& search space

Objectivefunction• Generative (deep) model• Dimensionality reduction• Fabrication method• Procedural modeling• …

• Numerical optimization• Visualization• Recommendation• Data mining• Interactive systems• …

• Human performance model• Machine learning (e.g., regression)• Physical simulation• Heuristic rules• Human computation• Psychological experiment• …
47

Computational Design: Examples

Computational Design: Examples
Functional Fabrication

Pteromys: Interactive Design and Optimization of Free-Form Airplanes
N. Umetani, Y. Koyama, R. Schmidt, and T. Igarashi
ACM Trans. Graph. (a.k.a. SIGGRAPH 2014)

Simulate and visualize the flight trajectories
in real-time

Flight trajectories(with diﬀ erent initial conditions)Automatic adjustment
by numerical optimization

automatic
adjustment
user’s edit

max Fly-ability(x)xautomatic
adjustment
user’s edit

max Fly-ability(x)xTarget parameters:- Wing position- Wing angle

automatic
adjustment
user’s edit

max Fly-ability(x)xTarget parameters:- Wing position- Wing angle

Evaluation enabled by- Physical simulation- Machine learningautomatic
adjustment
user’s edit

Search by- Manual exploration with visualization- Auto-adjustment by optimization

max Fly-ability(x)xTarget parameters:- Wing position- Wing angle

Evaluation enabled by- Physical simulation- Machine learningautomatic
adjustment
user’s edit

Search by- Manual exploration with visualization- Auto-adjustment by optimization

max Fly-ability(x)xTarget parameters:- Wing position- Wing angle

Evaluation enabled by- Physical simulation- Machine learning

Computational techniques could produce creative yet functional
designs that are otherwise diﬃcult to achieve

An Interactive Design System of Free-Formed Bamboo-Copters
M. Nakamura, Y. Koyama, D. Sakamoto, and T. Igarashi
Comput. Graph. Forum (a.k.a. Pacific Graphics 2016)

max {Fly-ability(x) + Spin-stability(x)}xParameters- Wing angle- Wing orientation- Wing shape- Stick lengthmax {Fly-ability(x) + Spin-stability(x)}xFunctionality #Parameters- Wing angle
Predicted by physics- Wing orientation and machine learning- Wing shape- Stick length

Functionality #- Moment of inertia alignment- Moment of inertia balance- Center of mass position

max {Fly-ability(x) + Spin-stability(x)}xFunctionality #Parameters- Wing angle
Predicted by physics- Wing orientation and machine learning- Wing shape- Stick length

Functionality #- Moment of inertia alignment- Moment of inertia balance- Center of mass position

When we have two (or more) objectives, a popular
approach is to take their (weighted) sum

Research

AutoConnect: Computational Design of 3D-Printable Connectors
Y. Koyama, S. Sueda, E. Steinhardt, T. Igarashi, A. Shamir, W. Matusik
ACM Trans. Graph. (a.k.a. SIGGRAPH Asia 2015)

Many skilled end-users design 3D-printable connectors
to connect two objects together
How can novices design 3D-printable connectors?

Guitar + iPhone

(a) Mug holder
at desk edges
Speaker+ Book shelf
SpeakerConnector

Shelf(c) Speaker hanger for a shelf

Necessary Functionality of a Connector
Stability from thephysics viewpoint

Stability from thegeometric viewpoint

Necessary Functionality of a Connector
Stability from thephysics viewpoint

Stability from thegeometric viewpoint

Generate connectors
by incorporating themas “constraints” into
optimization

max

Goodness(x)s.t.

Connect-ability(x)x0Constraint function:• Return positive values when the connect-ability is suﬃcient• Return negative values when the connect-ability is insuﬃcienthttp://visualcomputing.ist.ac.at/publications//DefSense/

Shape Changing Displays

CHI#chi4good, CHI 2016, San Jose, CA, USA

DefSense: Computational Design of Customized
Deformable Input Devices
Moritz Bächer1 , Benjamin Hepp2†, Fabrizio Pece2†
Paul G. Kry3 , Bernd Bickel4 , Bernhard Thomaszewski1 , Otmar Hilliges2
1
Disney Research, 2 ETH Zurich, 3 McGill University, 4 IST Austria

Figure 1. Design and fabrication of custom input devices (from left to right): the designer creates a set of example deformations (1) and roughly indicateswhere to place internal sensors. Our optimization algorithm then refines the sensor placement to maximize reconstruction accuracy (2). We fabricate
our designs by inserting piezoresistive wires in-between 3D-printed body parts (3), then calibrate using motion capture (4). Our customized flexible
input devices can be used in a variety of applications, e.g., to animate digital characters (5).

ABSTRACTINTRODUCTION

Fabricating objects with good deformation sensing capabilities
Piezoresistive wires (embedded)

Challenge: Finding a functional
wire layout (sensor placement)
Moritz Bächer, Benjamin Hepp, Fabrizio Pece, Paul G. Kry, Bernd Bickel, Bernhard Thomaszewski, and Otmar Hilliges. 2016. DefSense: Computational Designof Customized Deformable Input Devices. In Proc. CHI '16. 3806–3816. DOI:https://doi.org/10.1145/2858036.2858354

Fabricating objects with good deformation sensing capabilities
Piezoresistive wires (embedded)

WirelayoutChallenge: Finding a functional
wire layout (sensor placement)

The sensing capability (thatshould be maximized) is
modeled as the reconstruction
error (that should be
minimized) when the wirelayout is used for sensing

Moritz Bächer, Benjamin Hepp, Fabrizio Pece, Paul G. Kry, Bernd Bickel, Bernhard Thomaszewski, and Otmar Hilliges. 2016. DefSense: Computational Designof Customized Deformable Input Devices. In Proc. CHI '16. 3806–3816. DOI:https://doi.org/10.1145/2858036.2858354

Computational Design: Examples
Haptics

A Computational Approach to Magnetic Force Feedback Design
M. Ogata and Y. Koyama
CHI 2021

Background: Magnetic Force Feedback [Ogata, UIST 2018]• By embedding magnets into physicalinterfaces, we can provide complex
force feedback (i.e., haptics)

Haptics(visualized)• Problem: even if we have a desired

haptics in mind, we do not know how
to embed magnets such that they
reproduce the desired haptics

Masa Ogata. 2018. Magneto-Haptics: Embedding Magnetic ForceFeedback for Physical Interactions. In Proc. UIST '18. 737–743. DOI:https://doi.org/10.1145/3242587.3242615

Embedded magnets?

magnetposition/orientation81

Approach: Computational Design• Formulate the magnet arrangement problem as optimization• I.e., search for the optimal magnet arrangement that reproduces thedesired haptics

82Target parameters:
Magnet positions andorientations

Objective (to be minimized):
The distance between
the physically simulated haptics andthe target (desired) haptics

Target parameters:
Magnet positions andorientations

Objective (to be minimized):
The distance between
the physically simulated haptics andthe target (desired) haptics

Inverse design: Find the optimal parameters set that
reproduces the target functionality

Computational Design of Hand-Held VR Controllers Using Haptic Shape Illusion
E. Fujinawa, S. Yoshida, Y. Koyama, T. Narumi, T. Tanikawa, and M. Hirose
VRST 2017

A problem with general-purpose
VR controllers
Cannot provide appropriate weight feel,
which may reduce the immersiveness≠https://commons.wikimedia.org/wiki/File:Oculus-Rift-Touch-Controller-Left.jpg

Cannot be perceived as a “sword” shapeWe can create a custom
controller that matches
the shape exactly, but…
They can be too large or too
heavy for comfortable use

Haptic shape illusion: Provide perception of having a diﬀerent-shaped
object by manipulating the mass distribution of the controller carefully

Haptic shape illusion: Provide perception of having a diﬀerent-shaped
object by manipulating the mass distribution of the controller carefully

Hollows

Weights

57.7%

We could make controllers smaller while preserving the perceived shapes

51.6%

55.0%

min kPerceivedShape(x)x<latexit sha1_base64="KMSzsxvbKbYUKBM/RFmIcr9W7bk=">AAACjnicjVFNT9tAEF27XzTQ4pZjL6tGRfRQKw6oVKlQkXrhmAoCSNkQrTfjZMV6bXbHiNT1z+kf6o1/wzq21IJaqSOtdvTmvTejmThX0mKvd+v5jx4/efps7XlnfePFy83g1etTmxVGwEhkKjPnMbegpIYRSlRwnhvgaazgLL78WtfPrsFYmekTXOYwSflcy0QKjg6aBj+3KUulnpYs5biIk/KmqigbsAFlPyhDuMFyCK6VvIbZ8YLnUO38Zr6nH1rOCTdzwIbglBdl332s48wHLcOGGDbOjfsKPJbfHxgyBVf3PWvKNOhGYW8V9N9Jl7QxnAa/2CwTRQoaheLWjqNejpOSG5RCQdVhhYWci0s+h7FLNU/BTsrVOiv6ziEzmmTGPY10hf6pKHlq7TKNHbOe3D6s1eDfauMCk0+TUuq8QNCiaZQUimJG69vQmTQgUC1dwoWRblYqFtxwge6Cnf9bwmk/jHbD/re97uFFu4418oa8JTskIvvkkByRIRkR4a17kTfwPvuB/9E/8L80VN9rNVvkXvhHd43TxsQ=</latexit>

s.t. Size(x)  TargetSize

TargetShapek

2min kPerceivedShape(x)x<latexit sha1_base64="KMSzsxvbKbYUKBM/RFmIcr9W7bk=">AAACjnicjVFNT9tAEF27XzTQ4pZjL6tGRfRQKw6oVKlQkXrhmAoCSNkQrTfjZMV6bXbHiNT1z+kf6o1/wzq21IJaqSOtdvTmvTejmThX0mKvd+v5jx4/efps7XlnfePFy83g1etTmxVGwEhkKjPnMbegpIYRSlRwnhvgaazgLL78WtfPrsFYmekTXOYwSflcy0QKjg6aBj+3KUulnpYs5biIk/KmqigbsAFlPyhDuMFyCK6VvIbZ8YLnUO38Zr6nH1rOCTdzwIbglBdl332s48wHLcOGGDbOjfsKPJbfHxgyBVf3PWvKNOhGYW8V9N9Jl7QxnAa/2CwTRQoaheLWjqNejpOSG5RCQdVhhYWci0s+h7FLNU/BTsrVOiv6ziEzmmTGPY10hf6pKHlq7TKNHbOe3D6s1eDfauMCk0+TUuq8QNCiaZQUimJG69vQmTQgUC1dwoWRblYqFtxwge6Cnf9bwmk/jHbD/re97uFFu4418oa8JTskIvvkkByRIRkR4a17kTfwPvuB/9E/8L80VN9rNVvkXvhHd43TxsQ=</latexit>

TargetShapeks.t. Size(x)  TargetSize
The perceived shape from the haptic feedback
modeled by regression of experimental data2

min kPerceivedShape(x)x<latexit sha1_base64="KMSzsxvbKbYUKBM/RFmIcr9W7bk=">AAACjnicjVFNT9tAEF27XzTQ4pZjL6tGRfRQKw6oVKlQkXrhmAoCSNkQrTfjZMV6bXbHiNT1z+kf6o1/wzq21IJaqSOtdvTmvTejmThX0mKvd+v5jx4/efps7XlnfePFy83g1etTmxVGwEhkKjPnMbegpIYRSlRwnhvgaazgLL78WtfPrsFYmekTXOYwSflcy0QKjg6aBj+3KUulnpYs5biIk/KmqigbsAFlPyhDuMFyCK6VvIbZ8YLnUO38Zr6nH1rOCTdzwIbglBdl332s48wHLcOGGDbOjfsKPJbfHxgyBVf3PWvKNOhGYW8V9N9Jl7QxnAa/2CwTRQoaheLWjqNejpOSG5RCQdVhhYWci0s+h7FLNU/BTsrVOiv6ziEzmmTGPY10hf6pKHlq7TKNHbOe3D6s1eDfauMCk0+TUuq8QNCiaZQUimJG69vQmTQgUC1dwoWRblYqFtxwge6Cnf9bwmk/jHbD/re97uFFu4418oa8JTskIvvkkByRIRkR4a17kTfwPvuB/9E/8L80VN9rNVvkXvhHd43TxsQ=</latexit>

TargetShapeks.t. Size(x)  TargetSize
The perceived shape from the haptic feedback
modeled by regression of experimental data
Inverse design: Find the optimal parameter set that
reproduces the target functionality

2https://www.pdf.inf.usi.ch/projects/SurfaceStylusCoOpt/index.html

SIGGRAPH

Fabrication-in-the-Loop Co-Optimization of Surfaces and Styli for
Drawing Haptics
MICHAL PIOVARČI, Università della Svizzera italiana, Switzerland
DANNY M. KAUFMAN, Adobe Research
DAVID I. W. LEVIN, University of Toronto, Canada
PIOTR DIDYK, Università della Svizzera italiana, Switzerland

Desired Haptic Feedback

Optimization With Fabrication-In-The-Loop

Our Reproduction

Application

Fig. 1. We propose a data-driven method for mimicking haptic feedback of drawing tools. Our method uses fabrication-in-the-loop design enabled byour data-driven surrogate model which automatically handles exploration-exploitation trade-o�s and minimizes the amount of printed samples. The final
stylus-surface combinations are manufacturable on commonly available hardware and can be directly integrated into current digital drawing solutions.
Digital drawing tools are now standard in art and design work�ows. These
tools o�er comfort, portability, and precision as well as native integrationof haptic feedback, and demonstrate that they are often hard for users to
distinguish from their traditional drawing-tool analogs.

Haptic feedback reproduction of fabricated drawing tools

Michal Piovarči, Danny M. Kaufman, David I. W. Levin, and Piotr Didyk. 2020. Fabrication-in-the-loop co-optimization of surfaces and styli for drawing haptics.
ACM Trans. Graph. 39, 4, Article 116 (July 2020), 16 pages. DOI:https://doi.org/10.1145/3386569.3392467

Haptic feedback reproduction of fabricated drawing tools

Fabrication is necessary because of• the complexity to simulate haptics• the imperfections introduced in the
fabrication process→ Fabrication-in-the-loop optimization
Michal Piovarči, Danny M. Kaufman, David I. W. Levin, and Piotr Didyk. 2020. Fabrication-in-the-loop co-optimization of surfaces and styli for drawing haptics.
ACM Trans. Graph. 39, 4, Article 116 (July 2020), 16 pages. DOI:https://doi.org/10.1145/3386569.3392467

Haptic feedback reproduction of fabricated drawing tools
Use optimization techniques (Bayesian optimization) to eﬀectively
determine which samples should be fabricated for measurement

Fabrication is necessary because of• the complexity to simulate haptics• the imperfections introduced in the
fabrication process→ Fabrication-in-the-loop optimization
Michal Piovarči, Danny M. Kaufman, David I. W. Levin, and Piotr Didyk. 2020. Fabrication-in-the-loop co-optimization of surfaces and styli for drawing haptics.
ACM Trans. Graph. 39, 4, Article 116 (July 2020), 16 pages. DOI:https://doi.org/10.1145/3386569.3392467

Computational Design: Examples
Specific Fabrication Methods

https://visualcomputing.ist.ac.at/publications//CurveUp/

CurveUps:
Shaping Objects from Flat Plates with Tension-Actuated Curvature
RUSLAN GUSEINOV, EDER MIGUEL, and BERND BICKEL, IST Austria

Fig. 1. CurveUps are printed flat and actuated into doubly curved 3D shapes using pre-stretched elastic membranes.

SIGGRAPH

Designing 3D surfaces that can be fabricated as flat plates and thenself-actuated by tensions (4D printing)
Pre-stretched elastic sheets

Printed (flat)

Rigid tiles

Actuated
ResultActuation mechanism (cross-section view)

Ruslan Guseinov, Eder Miguel, and Bernd Bickel. 2017. CurveUps: shaping objects from flat plates with tension-actuated curvature. ACM Trans. Graph. 36, 4,
Article 64 (July 2017), 12 pages. DOI:https://doi.org/10.1145/3072959.3073709

Designing 3D surfaces that can be fabricated as a flat plate and thenself-actuated by tensions (4D printing)
Pre-stretched elastic sheets

Printed (flat)

Inverse design scheme

Rigid tiles

Actuated
ResultActuation mechanism (cross-section view)

Ruslan Guseinov, Eder Miguel, and Bernd Bickel. 2017. CurveUps: shaping objects from flat plates with tension-actuated curvature. ACM Trans. Graph. 36, 4,
Article 64 (July 2017), 12 pages. DOI:https://doi.org/10.1145/3072959.3073709

Designing 3D surfaces that can be fabricated as a flat plate and thenself-actuated by tensions (4D printing)
Pre-stretched elastic sheets

Printed (flat)

Inverse design scheme

Rigid tiles

The search variables:
Shapes, positions, andorientations of rigid tiles
Actuated
ResultActuation mechanism (cross-section view)

The search space is ruled by the
fabrication method

Ruslan Guseinov, Eder Miguel, and Bernd Bickel. 2017. CurveUps: shaping objects from flat plates with tension-actuated curvature. ACM Trans. Graph. 36, 4,
Article 64 (July 2017), 12 pages. DOI:https://doi.org/10.1145/3072959.3073709

Computational Design: Examples
User Interface Designhttps://dl.acm.org/doi/

.

/

.

CHI

Session: Mobile Text Entry

Possible(design(

CHI 2013: Changing Perspectives, Paris, France

Improving Two-Thumb Text Entryon Touchscreen Devices
M!

B!

P!

W!

Antti Oulasvirta, Anna Reichel, Wenbin Li,
Yan Zhang, Myroslav Bachynskyi
Max Planck Institute for Informatics
G!
H!

X!

C!

R!

Y!

S!

Z!

D!

N!

F!

V!

Keith Vertanen
Montana Tech of the
University of Montana

T!

O!

J!

I!

E!

!

U!

K!

A!

L!

Q!

Per Ola Kristensson
University of St Andrews

Figure 1. KALQ (pronounced as in “calculated”) is a soft keyboard designed to improve two-thumb text entry on tablet devices.
Its design considers grip, coordinated performance of the two thumbs, and linguistic and motor errors.
ABSTRACT

We study the design of split keyboards for fast text entry
with two thumbs on mobile touchscreen devices. The layoutof KALQ was determined through first studying how users
should grip a device with two hands. We then assigned let-tablet placed on a surface [15], 44 wpm with a stylus [22],and 60 wpm with two thumbs on a physical mini-QWERTY*
keyboard [4]. With such rates, two-thumb text entry on
touchscreens may be limited to simple tasks such as entryof messages, addresses, calendar events, and names [2].

Optimizing keyboard layout for two-thumb text entry on touchscreen

Keyboard layout

Antti Oulasvirta, Anna Reichel, Wenbin Li, Yan Zhang, Myroslav Bachynskyi, Keith Vertanen, and Per Ola Kristensson. 2013. Improving two-thumb text entry on
touchscreen devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). 2765–2774. DOI:https://doi.org/
10.1145/2470654.2481383

Optimizing keyboard layout for two-thumb text entry on touchscreen

Keyboard layout

Predicted by a human performance model(in this case, a modification of Fitts’s law)

Antti Oulasvirta, Anna Reichel, Wenbin Li, Yan Zhang, Myroslav Bachynskyi, Keith Vertanen, and Per Ola Kristensson. 2013. Improving two-thumb text entry on
touchscreen devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). 2765–2774. DOI:https://doi.org/
10.1145/2470654.2481383

Optimizing keyboard layout for two-thumb text entry on touchscreen
Automatic search by the simulated-annealing method

Keyboard layout

Predicted by a human performance model(in this case, a modification of Fitts’s law)

Antti Oulasvirta, Anna Reichel, Wenbin Li, Yan Zhang, Myroslav Bachynskyi, Keith Vertanen, and Per Ola Kristensson. 2013. Improving two-thumb text entry on
touchscreen devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). 2765–2774. DOI:https://doi.org/
10.1145/2470654.2481383

Optimizing keyboard layout for two-thumb text entry on touchscreen
Automatic search by the simulated-annealing method

Keyboard layout

Predicted by a human performance model(in this case, a modification of Fitts’s law)

User interface design can be optimized using predictive models ofhuman performance
Antti Oulasvirta, Anna Reichel, Wenbin Li, Yan Zhang, Myroslav Bachynskyi, Keith Vertanen, and Per Ola Kristensson. 2013. Improving two-thumb text entry on
touchscreen devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). 2765–2774. DOI:https://doi.org/
10.1145/2470654.2481383

https://www.cs.columbia.edu/CAVE/projects/optimizing_keyboards/

CHI

Gesture Elicitation & Recognition

CHI 2015, Crossings, Seoul, Korea

2SWLPL]LQJ7RXFKVFUHHQ.H\ERDUGVIRU*HVWXUH7\SLQJ%ULDQ$6PLWK ;LDRMXQ%L

&ROXPELD8QLYHUVLW\
1HZ<RUN1<86$
EULDQ#FVFROXPELDHGX

D 4ZHUW\

6KXPLQ=KDL*RRJOH,QF
0RXQWDLQ9LHZ&$86$
^[LDRMXQEL]KDL`#DFPRUJ

E *.'

F *.7)LJXUH.H\ERDUGVRSWLPL]HGIRUJHVWXUHW\SLQJ7KH R NH\LVVKDGHGWRPDUNWKHEHJLQQLQJRIWKHZRUGJHVWXUHVIRURUZKLWH DQGRXU EODFN  D 7KH4ZHUW\NH\ERDUGVXIIHUVIURPWKHSUREOHPRIJHVWXUHDPELJXLW\0DQ\SDLUVRIZRUGV VXFKDV³RU´DQG³RXU´VKRZQKHUH VKDUHWKHVDPHJHVWXUHRQ4ZHUW\ E 7KH*.'NH\ERDUG ³*HVWXUH.H\ERDUG²'RXEOHRSWLPL]HG´ 
LVWKHEHVWFRPSURPLVHIRUJHVWXUHFODULW\DQGJHVWXUHVSHHG+HUHWKHJHVWXUHVIRU³RU´DQG³RXU´DUHQRWLFHDEO\GLIIHUHQW F 7KH
*.7NH\ERDUG ³*HVWXUH.H\ERDUG²7ULSOHRSWLPL]HG´ LVWKHEHVWFRPSURPLVHIRUJHVWXUHFODULW\JHVWXUHVSHHGDQG4ZHUW\VLPLODULW\$JDLQWKHWZRJHVWXUHVDUHQRWLFHDEO\GLIIHUHQW
$%675$&7

'HVSLWHLWVJURZLQJSRSXODULW\JHVWXUHW\SLQJVXIIHUVIURPGHYLFHVVLQFHLWVFRQFHSWLRQLQWKHHDUO\¶V>@7RGD\JHVWXUHW\SLQJFDQEHIRXQGRQDOOPDMRUPRELOHFRPSXWLQJ

Optimizing keyboard layout for gesture typing on touchscreen
Motivation:
The Qwerty keyboard has a problem of not being able to distinguish“our” and “or” in gesture typing (inaccuracy)

Brian A. Smith, Xiaojun Bi, and Shumin Zhai. 2015. Optimizing Touchscreen Keyboards for Gesture Typing. In Proceedings of the 33rd Annual ACM Conferenceon Human Factors in Computing Systems (CHI '15). 3365–3374. DOI:https://doi.org/10.1145/2702123.2702357

Optimizing keyboard layout for gesture typing on touchscreen
Motivation:
The Qwerty keyboard has a problem of not being able to distinguish“our” and “or” in gesture typing (inaccuracy)
Computationally designed layouts:
For easier learning

Brian A. Smith, Xiaojun Bi, and Shumin Zhai. 2015. Optimizing Touchscreen Keyboards for Gesture Typing. In Proceedings of the 33rd Annual ACM Conferenceon Human Factors in Computing Systems (CHI '15). 3365–3374. DOI:https://doi.org/10.1145/2702123.2702357

Optimizing keyboard layout for gesture typing on touchscreen
Motivation:
The Qwerty keyboard has a problem of not being able to distinguish“our” and “or” in gesture typing (inaccuracy)
Computationally designed layouts:
For easier learning

Adding a diﬀerent objective provides a diﬀerent (yet optimal) solution(c.f., Pareto front)
Brian A. Smith, Xiaojun Bi, and Shumin Zhai. 2015. Optimizing Touchscreen Keyboards for Gesture Typing. In Proceedings of the 33rd Annual ACM Conferenceon Human Factors in Computing Systems (CHI '15). 3365–3374. DOI:https://doi.org/10.1145/2702123.2702357

Computational Design: Examples
Visual Designhttps://ieeexplore.ieee.org/abstract/document/

TVCG (IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS (TVCG), ACCEPTED, 20171

Towards Perceptual Optimizationof the Visual Design of Scatterplots
Luana Micallef, Gregorio Palmas, Antti Oulasvirta, and Tino Weinkauf
Abstract—Designing a good scatterplot can be difficult for non-experts in visualization, because they need to decide on many
parameters, such as marker size and opacity, aspect ratio, color, and rendering order. This paper contributes to research exploring theuse of perceptual models and quality metrics to set such parameters automatically for enhanced visual quality of a scatterplot. A key
consideration in this paper is the construction of a cost function to capture several relevant aspects of the human visual system,
examining a scatterplot design for some data analysis task. We show how the cost function can be used in an optimizer to search for the
optimal visual design for a user’s dataset and task objectives (e.g., “reliable linear correlation estimation is more important than classseparation”). The approach is extensible to different analysis tasks. To test its performance in a realistic setting, we pre-calibrated it forcorrelation estimation, class separation, and outlier detection. The optimizer was able to produce designs that achieved a level of speed
and success comparable to that of those using human-designed presets (e.g., in R or MATLAB). Case studies demonstrate that theapproach can adapt a design to the data, to reveal patterns without user intervention.
Index Terms—Scatterplot, optimization, perception, crowdsourcing.

F

1

I

I NTRODUCTION
N this paper, we investigate the design of scatterplots, one of

the most important methods for visualization of bivariate dataapproach, the plots in Figure 1(b) are automatically generated forthe meteorologist once she provides the dataset and specifies her)

Optimizing visual design of scatterplots

A weighted sum of heuristic rules(e.g., average opacity, amount ofoverlapping, etc.)

Default design

Optimal design

L. Micallef, G. Palmas, A. Oulasvirta and T. Weinkauf, "Towards Perceptual Optimization of the Visual Design of Scatterplots," in IEEE Transactions on
Visualization and Computer Graphics, vol. 23, no. 6, pp. 1588-1599, 1 June 2017, doi: 10.1109/TVCG.2017.2674978

Optimizing visual design of scatterplots

A weighted sum of heuristic rules(e.g., average opacity, amount ofoverlapping, etc.)

Default design

Optimal design

The weights are carefully calibrated
by either expert judgment orcrowdsourced experiments

L. Micallef, G. Palmas, A. Oulasvirta and T. Weinkauf, "Towards Perceptual Optimization of the Visual Design of Scatterplots," in IEEE Transactions on
Visualization and Computer Graphics, vol. 23, no. 6, pp. 1588-1599, 1 June 2017, doi: 10.1109/TVCG.2017.2674978

Computational Design: Examples
Robotics

https://la.disneyresearch.com/publication/computational-co-optimization-of-design-parameters-and-motiontrajectories-for-robotic-systems/

Computational Co-Optimization of
Design Parameters and Motion
Trajectories for Robotic Systems

The International Journal of Robotics Research

The International Journal of Robotics
Research
XX(X):1–14
c The Author(s) 2016
Reprints and permission:
sagepub.co.uk/journalsPermissions.nav
DOI: 10.1177/ToBeAssigned
www.sagepub.com/

Sehoon Ha1 and Stelian Coros2 and Alexander Alspach3
and Joohyung Kim1 and Katsu Yamane1

Abstract
We present a novel computational approach to optimizing the morphological design of robots. Our framework takes as
input a parameterized robot design as well as a motion plan consisting of trajectories for end-effectors and, optionally,
for its body. The algorithm optimizes the design parameters including link lengths and actuator placements while
concurrently adjusting motion parameters such as joint trajectories, actuator inputs, and contact forces. Our key insightis that the complex relationship between design and motion parameters can be established via sensitivity analysis
if the robot’s movements are modeled as spatio-temporal solutions to an optimal control problem. This relationshipbetween form and function allows us to automatically optimize the robot design based on specifications expressed asa function of actuator forces or trajectories. We evaluate our model by computationally optimizing four simulated robots
that employ linear actuators, four-bar linkages, or rotary servos. We further validate our framework by optimizing thedesign of two small quadruped robots and testing their performances using hardware implementations.

Optimizing robot designs so that they follow user specifications

User-specified constraint

Max. actuator torque2.08 Nm

Max. actuator torque1.42 Nm

Ha, S., Coros, S., Alspach, A., Kim, J., & Yamane, K. (2018). Computational co-optimization of design parameters and motion trajectories for robotic systems.
The International Journal of Robotics Research, 37(13–14), 1521–1536. https://doi.org/10.1177/0278364918771172

Optimizing robot designs so that they follow user specifications

User-specified constraint

Max. actuator torque2.08 Nm

Max. actuator torque1.42 Nm

Search parameters:• Design parameters (e.g., leg lengths)• Motion parameters (e.g., torques at a time t)

Ha, S., Coros, S., Alspach, A., Kim, J., & Yamane, K. (2018). Computational co-optimization of design parameters and motion trajectories for robotic systems.
The International Journal of Robotics Research, 37(13–14), 1521–1536. https://doi.org/10.1177/0278364918771172

Optimizing robot designs so that they follow user specifications

User-specified constraint

Max. actuator torque2.08 Nm

Max. actuator torque1.42 Nm

Search parameters:• Design parameters (e.g., leg lengths)• Motion parameters (e.g., torques at a time t)

It is better to perform joint optimization (design + motion),
because a robot design needs to be tightly coupled with its motion,
Ha, S., Coros, S., Alspach, A., Kim, J., & Yamane, K. (2018). Computational co-optimization of design parameters and motion trajectories for robotic systems.
The International Journal of Robotics Research, 37(13–14), 1521–1536. https://doi.org/10.1177/0278364918771172

Part1: Summary

[Recap]⇤

x = arg max Goodness(x)Trials and errors
on design parametersx2X

A mathematicaloptimization problem

A Definition of “Computational Design”[Recap]A general definition:
Computational design is a paradigm in which design problems are formulated
mathematically and solved by computational techniques.
A more focused (narrower) definition:
Computational design is a paradigm in which design problems are formulated
as optimization problems and solved by computational techniques.
127

Modeling Design as Optimization[Recap]Search strategy
Design strategy
The strategy on how
to manipulate thecurrent design

Optimal solution

Search variable

Search space

Objective function

Design outcome
The result obtained
after the design
process

Design parameters
The parameters we
can manipulate in
the design process

Design space
The set of all
possible design
parameters

Design goal
The criterion about
which design isconsidered good128

Where to Use Computational Techniques?
Searchstrategy

Search variable& search space

Objectivefunction• Generative (deep) model• Dimensionality reduction• Fabrication method• Procedural modeling• …

[Recap]• Numerical optimization• Visualization• Recommendation• Data mining• Interactive systems• …

• Human performance model• Machine learning (e.g., regression)• Physical simulation• Heuristic rules• Human computation• Psychological experiment• …
129

Introduction to Computational DesignPart 1: What is Computational Design?
Presenter: Yuki Koyama (AIST, Japan)

2021-05-12 | CHI 2021 Course | Online