Introduction to Computational DesignPart 3: Summary and Discussion
Presenter: Yuki Koyama (AIST, Japan)

2021-05-12 | CHI 2021 Courses | Online

Search for the best design

Search for the maximum

Analogy

Design parameter tweaking

Mathematical optimization

Computational design: a paradigm in which design problems are formulated as
optimization problems and solved by computational techniques.

A Definition of “Computational Design”[Recap]A general definition:
Computational design is a paradigm in which design problems are formulated
mathematically and solved by computational techniques.
A more focused (narrower) definition:
Computational design is a paradigm in which design problems are formulated
as optimization problems and solved by computational techniques.
3Computational Design Research[Recap]Approach:• Formulate design processes that have been traditionally dependent on
individual skills as mathematical optimization problems, and• Support or augment design processes by devising new ways of utilizing
computing power and mathematical tools

Goal:• Enable eﬃcient design workflow or sophisticated design outcomes that are
impossible in traditional approaches relying purely on the human brain

4

[Recap]

Computational design:
SIGGRAPH 2014

CHI 2021

Pacific Graphics 2016

SIGGRAPH Asia 2015

VRST 2017

Fly-ability

Haptics

Fly-ability

Connect-ability

Haptics

The design goal (objective) is the functionalityof the designed object

Design Goals (Objectives)[Recap]

Functionality

Aesthetic preference

We can compute the “goodness” of adesign by predictive simulation

We cannot compute the “goodness”
of a design as it is perceptual

Human assessment is necessary

A Systematic Approach to Input Human Assessment
Human Computation:“[…] a paradigm forutilizing humanprocessing power tosolve problems thatcomputers cannot yet
solve.” [von Ahn 05][Recap]

Function call
(Query)
0110
0101
1101

Return value

Processor

Human Processors

Either crowd workers or the single user
A. J. Quinn and B. B. Bederson. 2011. Human computation: a survey and taxonomy of a growing field. In Proc. CHI '11. 1403-1412.
von Ahn, L. Human Computation. Doctoral Thesis. UMI Order Number: AAI3205378, CMU, (2005).

Human-in-the-Loop Optimization[Recap]Query“I don’t know the objectivefunction, but I want to findthe maximizer…”

🤔

0110
0101
1101

Feedback

Human-in-the-loop
optimization is used
for solving problems
with “subjective”
objective functions(e.g., preference)

Iterative search process8Summary• Computational design is a paradigm in which design problems areformulated as mathematical optimization and solved using computational
techniques• HCI topics: Functional fabrication, haptics, user interface, visual design, etc.
• Human-in-the-loop optimization can be used for solving problems withperceptual objective functions (e.g., preference)• Tight integration between algorithm design and interface design would
be the key to achieve higher eﬃciency

9

Discussion

Discussion: Exploratory Design“Exploratory modeling is open-ended: the user begins the design process withan under-specified goal, and the precise form of the fi nal model isestablished through experimentation […]”

Jerry O. Talton, Daniel Gibson, Lingfeng Yang, Pat Hanrahan, and Vladlen Koltun. 2009. Exploratory modeling with collaborative design spaces. ACM Trans.
Graph. 28, 5 (December 2009), 1–10. DOI:https://doi.org/10.1145/1618452.1618513
11

Discussion: Exploratory Design“Exploratory modeling is open-ended: the user begins the design process withan under-specified goal, and the precise form of the fi nal model isestablished through experimentation […]”The objective function is unknown at
the beginning, and it can even change
during the design process
Adaptive techniques (e.g., online
machine learning) should be used
Jerry O. Talton, Daniel Gibson, Lingfeng Yang, Pat Hanrahan, and Vladlen Koltun. 2009. Exploratory modeling with collaborative design spaces. ACM Trans.
Graph. 28, 5 (December 2009), 1–10. DOI:https://doi.org/10.1145/1618452.1618513
12

Discussion: ”Optimization-in-the-Loop” Design Iteration

13

Discussion: ”Optimization-in-the-Loop” Design Iteration🤔
The designer decides
what to do next

min f (x)Designiterationx

Manualcontent edit

Automatic
optimization

14

Discussion: ”Optimization-in-the-Loop” Design Iteration
Sketchplore(DIS 2016)
Tool/UI Design

DIS 2016, June 4–8, 2016, Brisbane, Australia

Sketchplore: Sketch and Explore with a Layout Optimiser
Kashyap Todi1,2
Daryl Weir1
Antti Oulasvirta1
kashyap.todi@uhasselt.bedaryl.weir@aalto.fi
antti.oulasvirta@aalto.fi
1
Aalto University 2 Hasselt University - tUL - iMinds
Helsinki, Finland
Diepenbeek, Belgium
Real-Time Optimisation

MenuOptimizer(UIST 2013)
GUI

UIST’13, October 8–11, 2013, St. Andrews, UK

MenuOptimizer: Interactive Optimization of Menu Systems
Gilles Bailly1
Antti Oulasvirta1
Timo Kötzing 2
1
Max Planck Institute for Informatics and Saarland University

Sabrina Hoppe1
2
University of Jena

ExploreTimeline+
GlobalOptimiser

CanvasSearchSpace

Figure 1: MenuOptimizer assists in the design of menus: While the designer edits the menu (action in red), a model-based optimizer
updates itself to provide feedback and suggestions (in blue): A) Item feedback indicates the frequency (line width) and user performance over time (color gradient). B–C) Hotkeys and separators are automatically assigned. D) Item placements to improve user
performance are suggested. E) Designers can normally edit items (move, delete, etc.) and also F) lock items to constrain them together to accelerate optimization.

Local
Optimiser
Current Design

ABSTRACT

Design Space

Figure 1: Sketchplorer is an interactive layout sketching tool supported by real-time model-based optimisation. The tool isdesigned to facilitate the creative and problem-solving aspects of sketching without requiring extensive input. While a designer
is sketching, a design task is automatically inferred. The optimiser uses predictive models to make suggestions for local andglobal changes that improve usability and aesthetics. Suggestions appear on the side, and never override the designer’s work.
ABSTRACTThis paper studies a novel concept for integrating real-time
design optimisation to a sketching tool. Although optimisation methods can attack very complex design problems,
their insistence on precise objectives and a point optimum isa poor fit with sketching practices. Sketchplorer is a multitouch sketching tool that uses a real-time layout optimiser. It
automatically infers the designer’s task to search for both local improvements to the current design and global (radical)
alternatives. Using predictive models of sensorimotor performance and perception, these suggestions steer the designer
toward more usable and aesthetic layouts without overriding
the designer or demanding extensive input.
Author Keywords

Sketching; Model-based optimisation; Visual Layouts
ACM Classification Keywords

H.5.2. Information Interfaces and Presentation: User Interfaces
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from Permissions@acm.org.
DIS 2016, June 04–08, 2016, Brisbane, QLD, Australia.
Copyright is held by the owner/author(s). Publication rights licensed to ACM.

INTRODUCTION

This paper is motivated by the observation that optimisation
methods have great untapped potential in design tools. We
focus on the activity of sketching layouts, in which a designer
places, colours, organises, and defines elements on a canvas.
From a combinatorial perspective, the design of layouts is notoriously hard. For a canvas of 1024⇥768 pixels, divided intoa 24 ⇥ 32 grid, as in the tool presented here, there are 158,400
one-element layouts and a whopping 1041 eight-element layouts. Although algorithms may not be able to find the optimal solution in such large search spaces, they can “parallelise” search, and find candidate solutions and suggest themto designers. This could help designers in exploration, who
are known to be limited to a handful of designs per iteration[8]. Also, algorithms can complement designers by exploring design spaces neutrally without being constrained by pastexperiences, to produce designs that the designer might nototherwise conceive. Employing an optimiser might also improve the quality of designs for end-users (see [12, 33, 45]).
However, several hard research challenges emerge. First, layout design is a complex, multi-objective task addressing not
only usability but also aesthetic qualities [15, 44]. Presently
no algorithmic approach exists that can address both. Second, optimisation typically takes a long time, due to combinatorial complexity, and no solution has been shown for fastpaced, iterative design of layouts. Third, although optimisation methods can attack very complex design problems, theirSupporting Creativity through UX Design

CHI 2015, Crossings, Seoul, Korea

DesignScape: Design with Interactive Layout SuggestionsPeter O’Donovan1,2Dept. of Computer Science
University of Toronto
odonovan@dgp.toronto.edu

1Aseem Agarwala22
Adobe Research
Seattle, WA, USA
asagarwa@adobe.com

Aaron Hertzmann1,33
Adobe Research
San Francisco, CA, USA
hertzman@adobe.com

ABSTRACTSketchploration Environment

Predictive Models

DesignScape(CHI 2015)

Menu systems are challenging to design because design
spaces are immense, and several human factors affect user
behavior. This paper contributes to the design of menus
with the goal of interactively assisting designers with an
optimizer in the loop. To reach this goal, 1) we extend a
predictive model of user performance to account for expectations as to item groupings; 2) we adapt an ant colony
optimizer that has been proven efficient for this class ofproblems; and 3) we present MenuOptimizer, a set of interactions integrated into a real interface design tool (QtDesigner).- MenuOptimizer supports designers’ abilities tocope with uncertainty and recognize good solutions. It allows designers to delegate combinatorial problems to the
optimizer, which should solve them quickly enough without
disrupting the design process. We show evidence that satisfactory menu designs can be produced for complex problems in minutes.Author Keywords

Menus; Predictive models; Interactive optimization.
ACM Classification Keywords

H.5.m. Information interfaces and presentation (e.g., HCI):
Miscellaneous.
INTRODUCTION

Menu systems, consisting of menus, hotkeys or toolbars,
are widespread interfaces for selecting commands. Interface
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACM
must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.
UIST’13, October 8–11, 2013, St. Andrews, United Kingdom.

design strongly affects their usability. However, despite
apparent simplicity, designing usable menu systems is challenging because the number of alternative designs grows
superexponentially as a function of the number of commands. For instance, a linear menu with n items can, intheory, be organized in n! ways. However, professional
applications comprise hundreds of items organized in hierarchical menus. A menu hierarchy can be organized inabout (2n)! ways. For 50 items, the size of the search space
is a whopping 100! 10158. Design heuristics, such as placing frequently used items at the top [5], may be effective forsmall n but fail with larger n or if additional human factors
such as semantic relationships among items are considered.
Although experts can quickly generate a handful of solutions to hard design problems [6], they cannot examine allpromising solutions. Novices, known to search the space
depth-first [3], are likely to get stuck in a local search space.
Combinatorial optimization methods (e.g., [29]) have beensuccessfully used to generate user interfaces such as virtual
keyboards [8,21,24,34]. These methods explore a large
number of designs in order to find ones that minimize or
maximize a pre-specified objective function. Computation
time is on a scale of hours, days, or weeks. While empiricalevidence confirms improvements in usability in other contexts (e.g., [24,34]), there is reason to suggest that they may
be impractical for the design of menu systems. First, designers cannot be expected to wait days or weeks for a solution. Moreover, designers may not be able to define theoptimization problem completely in advance. Interaction
design, in general, is rather more an iterative process ofredefinition and refinement. Finally, predictive models formenu systems performance are only just emerging[5,20,25], are limited to linear menus, and have yet to cover
all important human factors that affect design choices.

Creating graphic designs can be challenging for novice users.
This paper presents DesignScape, a system which aids thedesign process by making interactive layout suggestions, i.e.,changes in the position, scale, and alignment of elements.
The system uses two distinct but complementary types ofsuggestions: refinement suggestions, which improve the current layout, and brainstorming suggestions, which change thestyle. We investigate two interfaces for interacting with suggestions. First, we develop a suggestive interface, where suggestions are previewed and can be accepted. Second, we develop an adaptive interface where elements move automatically to improve the layout. We compare both interfaces witha baseline without suggestions, and show that for novice designers, both interfaces produce significantly better layouts,
as evaluated by other novices.
Author Keywords

Graphic design, suggestion interfaces, adaptive design
ACM Classification Keywords

D.2.2 Software Engineering: Design Tools and Techniques
User Interfaces
INTRODUCTION

Graphic design is ubiquitous in modern life. Unfortunately,
creating designs can be difficult, particularly for novices, who
often wish to create simple posters, cards, or social media designs. Starting from a blank canvas can be overwhelming, andexploring alternatives is time-consuming. Novice designers
also make a variety of mistakes, from misalignment to incorrect emphasis of elements. Existing tools range from simple
template-based interfaces like PowerPoint, to complex systems like Illustrator. However, these tools provide no suggestions when modifying templates or designs.
This paper presents a novel system for graphic design using
layout suggestions, i.e., changes in the size, position, andalignment of elements. Our system proposes two complementary types of suggestions: refinements which improve thecurrent layout, and brainstorming suggestions which explore
alternative layouts with large changes in style (see Fig. 1).
Exploration and refinement are critical and complementary
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from Permissions@acm.org.
CHI 2015, April 18 - 23, 2015, Seoul, Republic of Korea

Figure 1. DesignScape Interface. The central canvas allows the user tocreate layouts in a simple editor. On the left, the system provides refinement suggestions, layouts similar to the canvas, but slightly improved.
On the right, the system provides brainstorming suggestions large-scalelayout changes in a variety of styles. Photos courtesy of Wilhelm Joys
Andersen and Martin Fisch.

tasks in design. However, exploration is difficult since a designer must imagine possible layouts, and modify many elements. Refinement is also difficult, since a single modification can necessitate many other changes. Our system includes
both types, allowing users to easily switch between exploringalternative layouts and refining the current layout.
We use an energy-based model to generate designs that encode design principles such as symmetry, alignment, andoverlap. User constraints are used to infer the designer’s intent, and to make refinement suggestions on the current layout. We also learn a “style space” from examples, which canbe used to generate new layouts in a variety of styles, providing starting points for design. The system can also retarget
layouts, allowing the user to easily modify the design size.
We also investigate different ways users can interact with suggestions. First, we develop a suggestive interface, where suggestions are previewed and accepted. Second, we develop an
adaptive interface which moves elements automatically. The
two modes are compared to a baseline without suggestions
by novice users on Mechanical Turk, and the quality of theresulting layouts are also evaluated. Both modes produce significantly better designs than the baseline on average. Lastly,
we demonstrate the system’s use for tablet-based design.
RELATED WORK

Exploring alternatives is a vital part of the design process.
Gross and Do [2] present a prototyping interface which allows
users to sketch drawings and store alternatives. Terry et al. [7]present an interaction technique which allows users to saveand embed alternatives during the design process, and easily
manipulate alternatives at a later point. Dow et al. [1] find that
forcing users to create multiple design alternatives, instead

15

Discussion: UX of Human-in-the-Loop Optimization[Zhou+, IUI 2021]
This paper investigates how
the user feels when thevariation of the candidates(exploration vs. exploitation)
can be manually controlled
C.f., Human-AI collaboration

Yijun Zhou, Yuki Koyama, Masataka Goto, and Takeo Igarashi. 2021. Interactive Exploration-Exploitation Balancing for Generative Melody Composition.
In Proceedings of the 26th International Conference on Intelligent User Interfaces (IUI '21), pp.43–47. DOI:https://doi.org/10.1145/3397481.3450663
16

C++ Implementation & Python Bindings Available on GitHub• https://github.com/yuki-koyama/mathtoolbox• Bayesian optimization (BO)• https://github.com/yuki-koyama/sequential-line-search• Preferential Bayesian optimization (PBO)• Sequential line search• https://github.com/yuki-koyama/sequential-gallery• Sequential gallery
17

Questions / Comments(URL REMOVED)
18

Acknowledgment: I thank all the collaborators!

•

Nobuyuki Umetani•

Shigeo Yoshida•

Ariel Shamir•

Ryan Schmidt•

Takuji Narumi•

Wojciech Matusik•

Takeo Igarashi•

Tomohiro Tanikawa•

Daisuke Sakamoto•

Morihiro Nakamura•

Michitaka Hirose•

Issei Sato•

Masa Ogata•

Shinjiro Sueda•

Masataka Goto•

Eisuke Fujinawa•

Emma SteinhardtIntroduction to Computational Design
Presenter: Yuki Koyama (AIST, Japan)

2021-05-12 | CHI 2021 Courses | Online