SEQUENTIAL GALLERY
FOR INTERACTIVE VISUALDESIGN OPTIMIZATION
Yuki Koyama, Issei Sato, and Masataka Goto
© 2020 SIGGRAPH. ALL RIGHTS RESERVED.

Sequential Gallery

Initial…

Optimized…

Sequential Gallery
for Interactive Visual Design Optimization
Yuki Koyama1, Issei Sato2, and Masataka Goto1
1. National Institute of Advanced Industrial Science and Technology (AIST)
2. The University of Tokyo

Overview
Background and Target Problem

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

3

Background: Parametric Visual Design is Everywhere
Photo color enhancement

Generative modeling

Procedural design… etc.

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

4

Background: Parametric Visual Design is Everywhere

Problem: Need many trials and errors• E.g., try a diﬀerent parameter combination, see the result, judge whether it isbetter or not, and then decide which combination to try next …

• This is a high-dimensional search task and can be tedious and timeconsuming

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

5

Background: Parametric Visual Design is Everywhere

Motivation: What computational support is possible?
Technical challenge:• Parameters need to be determined based on the user’s preference• It is diﬃcult to fully automate the parameter tweaking process
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)6

Overview
Proposed System: Sequential Gallery

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

7

Target:n design parameters(e.g., photo enhance)…

…
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

Output:An optimal
parameter set

8

Target:n design parameters(e.g., photo enhance)

Sequential Gallery:
An interactive optimization framework
where the user sequentially performs
2D search subtasks via a grid interface…

…
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

Output:An optimal
parameter set

9Sequential Gallery:
An interactive optimization framework
where the user sequentially performs
2D search subtasks via a grid interface…

2D search subtask #1

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

Output:An optimal
parameter set…

Target:n design parameters(e.g., photo enhance)10Sequential Gallery:
An interactive optimization framework
where the user sequentially performs
2D search subtasks via a grid interface…

2D search subtask #1

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

Output:An optimal
parameter set…

Target:n design parameters(e.g., photo enhance)

11Sequential Gallery:
An interactive optimization framework
where the user sequentially performs
2D search subtasks via a grid interface…

2D
2D searchsearch subtask
subtask #1#2

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

Output:An optimal
parameter set…

Target:n design parameters(e.g., photo enhance)

12Sequential Gallery:
An interactive optimization framework
where the user sequentially performs
2D search subtasks via a grid interface…

2D
2D searchsearch subtask
subtask #2#3

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

Output:An optimal
parameter set…

Target:n design parameters(e.g., photo enhance)

13Sequential Gallery:
An interactive optimization framework
where the user sequentially performs
2D search subtasks via a grid interface…

2D
2D searchsearch subtask
subtask #3#4

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

Output:An optimal
parameter set…

Target:n design parameters(e.g., photo enhance)

14Sequential Gallery:
An interactive optimization framework
where the user sequentially performs
2D search subtasks via a grid interface…

2D search subtask #4

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

Output:An optimal
parameter set…

Target:n design parameters(e.g., photo enhance)

15

Target:n design parameters(e.g., photo enhance)

Sequential Gallery:
An interactive optimization framework
where the user sequentially performs
2D search subtasks via a grid interface…

…
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

Output:An optimal
parameter set

16

Overview
Example Usage

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)17

Photo Color Enhancement (12D)
Brightness, contrast, saturation, shadows (RGB), midtones (RGB), and highlights (RGB)Original photograph

Enhanced photograph(after 4 iterations)Overview
Summary of Contributions

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)21

Summary of Contributions• Novel algorithm: Sequential plane search• A variant of preferential Bayesian optimization

...

...

(PBO), enabling user-in-the-loop optimization• Can find optimal solutions with fewer iterations
than the previous algorithm [Koyama+17]

Sequential plane search• Interactive system: Sequential Gallery• Use the sequential-plane-search algorithm incombination with a zoomable grid interface• Enable the user to eﬀectively explore the design
space and perform the optimization

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

Sequential Gallery
22Summary of Contributions• Novel algorithm: Sequential plane search• A variant of preferential Bayesian optimization

...

...

(PBO), enabling user-in-the-loop optimization• Can find optimal solutions with fewer iterations
than the previous algorithm [Koyama+17]

Sequential plane search• Interactive system: Sequential Gallery• Use the sequential-plane-search algorithm incombination with a zoomable grid interface• Enable the user to eﬀectively explore the design
space and perform the optimization

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

Sequential Gallery
23Summary of Contributions• Novel algorithm: Sequential plane search• A variant of preferential Bayesian optimization

...

...

(PBO), enabling user-in-the-loop optimization• Can find optimal solutions with fewer iterations
than the previous algorithm [Koyama+17]

Sequential plane search• Interactive system: Sequential Gallery• Use the sequential-plane-search algorithm incombination with a zoomable grid interface• Enable the user to eﬀectively explore the design
space and perform the optimization

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

Sequential Gallery
24

Problem Definition
From Mathematical Viewpoint

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)25

Problem Definition from Mathematical Viewpoint• Suppose that we have n sliders to adjustn• Let 𝒳 = [0,1] be the search space andx ∈ 𝒳 be a set of n parameter values• Let g : 𝒳 → ℝ be a perceptual

preference function (= goodness function)
which returns a goodness value• We want to solve an optimization problem:
x* = argmax g(x)x∈𝒳

[Koyama+, Computational Interaction (2018)] Yuki Koyama and Takeo Igarashi. 2018. Computational Design with Crowds. In Computational Interaction (Eds.
Antti Oulasvirta, Per Ola Kristensson, Xiaojun Bi, and Andrew Howes), Oxford University Press, pp.153—184.
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)26

Problem Definition from Mathematical Viewpoint• Suppose that we have n sliders to adjustn• Let 𝒳 = [0,1] be the search space andx ∈ 𝒳 be a set of n parameter values• Let g : 𝒳 → ℝ be a perceptual

preference function (= goodness function)
which returns a goodness value• We want to solve an optimization problem:
x* = argmax g(x)x∈𝒳

[Koyama+, Computational Interaction (2018)] Yuki Koyama and Takeo Igarashi. 2018. Computational Design with Crowds. In Computational Interaction (Eds.
Antti Oulasvirta, Per Ola Kristensson, Xiaojun Bi, and Andrew Howes), Oxford University Press, pp.153—184.
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)27

Interacting with Goodness Function
Absolute assessment
should not be used:
The user cannot directly
answer the function value
reliably [Brochu+10;
Koyama+18]
Relative assessment
should be used:
The user can answer
which option is better
among two (or more)
options

Absolute
assessment

A

B

Relativeassessment

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)28

Interacting with Goodness Function
Absolute assessment
should not be used:
The user cannot directly
answer the function value
reliably [Brochu+10;
Koyama+18]
Relative assessment
should be used:
The user can answer
which option is better
among two (or more)
options

Absolute
assessment

A

B

Relativeassessment

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)29

Interacting with Goodness Function
Absolute assessment
should not be used:
The user cannot directly
answer the function value
reliably [Brochu+10;
Koyama+18]
Relative assessment
should be used:
The user can answer
which option is better
among two (or more)
options

Absolute
assessment

A

B

Relativeassessment

Preferential feedback
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)30

Interacting with Goodness Function
Absolute assessment
should not be used:
The user cannot directly
answer the function value
reliably [Brochu+10;
Koyama+18]
Relative assessment
should be used:
The user can answer
which option is better
among two (or more)
options

Absolute
assessment

A

B

Relativeassessment

Preferential Bayesian optimization (PBO)
can run using relative assessment (i.e., preferential feedback)Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)31Preferential Bayesian Optimization (PBO)
Previous Techniques and Our New Technique

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)32

Basics: “Standard” Bayesian Optimization (BO)• Is a global “black-box” optimization algorithm• Can find optimal solutions with only a small
number of function evaluations• Thus, useful to handle expensive-to-evaluate
objective functions• Example applications: hyperparameter tuning formachine learning models [Akiba+, KDD 2019]

See [Shahriari+, Proc. IEEE 2016] for details
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)33

Preferential BO (PBO)• PBO is an extension of BO, which runs with relative assessment (orpreferential feedback), rather than absolute assessment of function values• PBO can find optimal solutions with only a small number of preferential
feedbacks• Note: human is expensive-to-query

A

B

Relativeassessment

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)34

[Brochu+, NIPS 2007]“Pairwise Comparison” Query[Koyama+, SIGGRAPH 2017]“Line Search” Query(Sequential Line Search)[Ours]“Plane Search” Query(Sequential Plane Search)

Query
type:

User
task:
Notes:

chosenx= argmax g(x)x∈{x A,xB}

Is the first PBO algorithm

chosenx= argmax g(x)x∈𝒮Needs fewer iterations than
[Brochu+ NIPS 2007]

chosenx= argmax g(x)x∈𝒫Needs even fewer iterations,and have a good compatibility
with grid interfaces

[Brochu+, NIPS 2007]“Pairwise Comparison” Query[Koyama+, SIGGRAPH 2017]“Line Search” Query(Sequential Line Search)[Ours]“Plane Search” Query(Sequential Plane Search)

Query
type:

User
task:
Notes:

chosenx= argmax g(x)x∈{x A,xB}

Is the first PBO algorithm

chosenx= argmax g(x)x∈𝒮Needs fewer iterations than
[Brochu+ NIPS 2007]

chosenx= argmax g(x)x∈𝒫Needs even fewer iterations,and have a good compatibility
with grid interfaces

[Brochu+, NIPS 2007]“Pairwise Comparison” Query[Koyama+, SIGGRAPH 2017]“Line Search” Query(Sequential Line Search)[Ours]“Plane Search” Query(Sequential Plane Search)

Query
type:

User
task:
Notes:

chosenx= argmax g(x)x∈{x A,xB}

Is the first PBO algorithm

chosenx= argmax g(x)x∈𝒮Needs fewer iterations than
[Brochu+ NIPS 2007]

chosenx= argmax g(x)x∈𝒫Needs even fewer iterations,and has a good compatibility
with grid interfacesSequential Gallery Workflow

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)38

2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

…

2D search subtask

2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

…

2D search subtask

2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

…

2D search subtask

User’s feedback

Next search plane2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

…

2D search subtask

2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

…

2D search subtask

2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

…

2D search subtask

2D search subtask

2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

…

2D search subtask

2D search subtask

2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

…

2D search subtask

2D search subtask

Next search plane
User’s feedback

2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

…

2D search subtask

2D search subtask

2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

…

2D search subtask

2D search subtask

2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

…

2D search subtask

2D search subtask

2D search subtask

2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)…

…

2D search subtask

2D search subtask

2D search subtask

2-dimensional search subspaces (= search planes)
determined by the sequential plane search algorithm
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)…

Interface: Zoomable Grid Interface for 2D Search Subtasks

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)53

Interface: Zoomable Grid Interface for 2D Search Subtasks

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)54

Interface: Zoomable Grid Interface for 2D Search Subtasks

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)55

Interface: Zoomable Grid Interface for 2D Search Subtasks

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)56

Interface: Zoomable Grid Interface for 2D Search Subtasks

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)57

Interface: Zoomable Grid Interface for 2D Search Subtasks

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)58

Interface: Zoomable Grid Interface for 2D Search Subtasks

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)59

Interface: Zoomable Grid Interface for 2D Search Subtasks

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)60

Benefits of Zoomable Grid Interface• Allows the user to easily grasp the available

options in the 2D subspace by just seeing the gridview

• WYSIWYG (What-You-See-Is-What-You-Get); do
not need to be aware of raw parameters• Compatibility with the sequential-plane-search task
(i.e., 2D search)• Minimum quantization errors (thanks to zooming
operations)

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)61

Sequential Plane Search
How to Determine the Next Plane in Each Iteration

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

62

Criterion to Select 2D Search Planes
We want to select the most eﬀective 2D subspace (=search plane) for the next query• Eﬀectiveness here means the degree to which it

is worth observing in the next iteration to find the
optimal solution?

Question: How can we define the eﬀectiveness?

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)63

Acquisition Function in (Standard) BO• Standard BO uses acquisition functions• An acquisition function evaluates how
eﬀective a point is as the next query• The maximizer of the acquisition function is
selected as the next point• We extend this point-wise acquisition function
to evaluate how eﬀective a search plane isas the next query (next slide)

Note: There are many variants in acquisition function definitions (e.g., “expected improvement”). Most of them can
automatically balance the “exploration” (i.e., favor unobserved regions) and “exploitation” (i.e., favor promising regions)strategies, which is the magic to enable BO to be successful. See [Shahriari+, Proc. IEEE 2016] for details.
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)64

Acquisition Function in (Standard) BO• Standard BO uses acquisition functions• An acquisition function evaluates how
eﬀective a point is as the next query• The maximizer of the acquisition function is
selected as the next point• We extend this point-wise acquisition function
to evaluate how eﬀective a search plane isas the next query (next slide)

Note: There are many variants in acquisition function definitions (e.g., “expected improvement”). Most of them can
automatically balance the “exploration” (i.e., favor unobserved regions) and “exploitation” (i.e., favor promising regions)strategies, which is the magic to enable BO to be successful. See [Shahriari+, Proc. IEEE 2016] for details.
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

65

Acquisition Function for Determining the Next Plane• Let 𝒟 be the accumulated preferential data so far obtained• Point-wise acquisition function (used in standard BO):
pointa(x; 𝒟) [Note 1]• Plane-wise acquisition function (proposed):
a

plane

(𝒫; 𝒟) =

∫𝒫

a

point

(x; 𝒟)dS

𝒫

x

𝒳

next

• We choose the next plane 𝒫 by solving a maximization problem:
next
plane
𝒫 = argmax a(𝒫; 𝒟) [Note 2]𝒫∈𝒳[Note 1]: Refer to [Koyama+, SIGGRAPH 2017] for the definition of a point in PBO; in short, it is based on the use of Gaussian process regression toestimate the goodness function landscape from the available preferential data.
[Note 2]: This maximization problem is diﬀerentiable, so L-BFGS can be used. Since it can have multiple local maxima, we solve this problem
multiple times with random initialization. It takes less than one sec. in most cases, which is acceptable from the interaction viewpoint.
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)66

Acquisition Function for Determining the Next Plane• Let 𝒟 be the accumulated preferential data so far obtained• Point-wise acquisition function (used in standard BO):
pointa(x; 𝒟) [Note 1]• Plane-wise acquisition function (proposed):
a

plane

(𝒫; 𝒟) =

∫𝒫

a

point

(x; 𝒟)dS

𝒫

x

𝒳

next

• We choose the next plane 𝒫 by solving a maximization problem:
next
plane
𝒫 = argmax a(𝒫; 𝒟) [Note 2]𝒫∈𝒳[Note 1]: Refer to [Koyama+, SIGGRAPH 2017] for the definition of a point in PBO; in short, it is based on the use of Gaussian process regression toestimate the goodness function landscape from the available preferential data.
[Note 2]: This maximization problem is diﬀerentiable, so L-BFGS can be used. Since it can have multiple local maxima, we solve this problem
multiple times with random initialization. It takes less than one sec. in most cases, which is acceptable from the interaction viewpoint.
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)67

Acquisition Function for Determining the Next Plane• Let 𝒟 be the accumulated preferential data so far obtained• Point-wise acquisition function (used in standard BO):
pointa(x; 𝒟) [Note 1]• Plane-wise acquisition function (proposed):
a

plane

(𝒫; 𝒟) =

∫𝒫

a

point

(x; 𝒟)dS

𝒫

x

𝒳

next

• We choose the next plane 𝒫 by solving a maximization problem:
next
plane
𝒫 = argmax a(𝒫; 𝒟) [Note 2]𝒫∈𝒳[Note 1]: Refer to [Koyama+, SIGGRAPH 2017] for the definition of a point in PBO; in short, it is based on the use of Gaussian process regression toestimate the goodness function landscape from the available preferential data.
[Note 2]: This maximization problem is diﬀerentiable, so L-BFGS can be used. Since it can have multiple local maxima, we solve this problem
multiple times with random initialization. It takes less than one sec. in most cases, which is acceptable from the interaction viewpoint.
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

68

Applications:
Possible Scenarios and Demonstrations

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

69

Potential Applications
Photo color enhancement

Generative modeling

Procedural texturing… and many other parametric design scenarios
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)70

Photo Color Enhancement (12D)
Brightness, contrast, saturation, shadows (RGB), midtones (RGB), and highlights (RGB)Original photograph

Enhanced photograph(after 5 iterations)Body Shaping (10D)Using the SMPL model [Loper+15] (the first 10 principal components)

“He was of medium height, solidly built,
wide in the shoulders, thick in the neck,with a jovial heavy-jawed red face […]”
Dashiell Hammett. 1930. The Maltese Falcon.

Evaluation [1/2]:
Optimization Performance Comparison

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)77Performance Comparison Using Synthetic Functions• Goals:• Evaluate the eﬃciency of our sequential plane searchcompared to the previous work [Koyama+17]• Confirm that BO (i.e., the use of the acquisition

function) contributes to the optimization performance

Gaussian (5D/15D)• How:• Use synthetic objective functions to simulate human
responses (shown on the right)• Algorithms to be compared: (next slide)
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

Rosenbrock (10D/20D)78

Algorithms to be Compared
Baseline 1:
SLS

Baseline 2:
SPS (Random)

Sequential line search[Koyama+17]

Sequential planesearch, but the planeis randomly chosen(instead of using BO)

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

SPS (Ours)Sequential planesearch (using BO)

79

Result: Performance Comparison [1/2]
Synthetic objective function: Gaussian function
15D(Lower is better)

5D

Performance: SLS < SPS (random) < SPS (ours)
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

80

Result: Performance Comparison [2/2]Synthetic objective function: Rosenbrock function
20D(Lower is better)

10D

Performance: SLS < SPS (random) < SPS (ours)
Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)81

Evaluation [2/2]:
Informal User Study

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)82

Informal User Study• Participants: Six students/researchers (five novices and one expert)• Task: Enhance photographs using our system (12D)• Results:• All the participants could successfully perform optimization and findsatisfactory results in 5.36 iterations in average (SD = 2.69)• Participants appreciated the grid view since they could get inspiration forpossible color variations easily and quickly

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)83

Discussions
Limitations and Future Work

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)84

Discussions: High Dimensionality• BO (and thus PBO, too) is inherently not goodat handling very high-dimensional problems[Wang+16]• We assumed the dimensionality is at most 20• To overcome the dimensionality issue,

application-specific extensions should be made• E.g., for generative modeling (e.g., GANs),[Chiu+, SIGGRAPH 2020] is a possible option
to combine with Sequential Gallery

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

C.f., A method specific togenerative modeling[Chiu+, SIGGRAPH 2020]85

Discussion: Others• Initial plane selection: we randomly choose initial planes, but otherstrategies are possible (e.g., use Design Gallery [Marks+97] for the
first iteration and then start the sequential plane search iterations)• Time-changing preference: we assume that the user’s preference

does not change during the iteration, but this is not true in somecases; supporting time-changing preference is an interesting futurework• Grid resolution and zooming levels: currently, we manually selectthe grid resolution and the number of levels, but this may be adjusted
dynamically by analyzing the just-noticeable diﬀerence (JND)• Prior knowledge: when some prior data is available, we could build

a rough approximation of the goodness function and then use it as aprior of the Bayesian inference; this would make the optimizationeven more eﬃcient

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

Refer to the paperfor details

86Summary

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)87

Summary• Sequential Gallery is an interactive system for user-in-the-loop visualdesign optimization• Its eﬃciency is enabled by sequential plane search, which is a new variantof preferential Bayesian optimization and is able to find the solution with
only a minimal number of iterations
Sequential Gallery

Initial…

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)

Optimized…

88References• [Akiba+, KDD 2019] Takuya Akiba, Shotaro Sano, Toshihiko• [Koyama+, SIGGRAPH 2017] Yuki Koyama, Issei Sato,• [Brochu+, NIPS 2007] Eric Brochu, Nando de Freitas, and• [Koyama+, Computational Interaction (2018)] Yuki

Yanase, Takeru Ohta, and Masanori Koyama. 2019. Optuna:
A Next-generation Hyperparameter Optimization Framework.
In Proc. KDD ’19. 2623–2631. DOI:https://doi.org/
10.1145/3292500.3330701
Abhijeet Ghosh. 2007. Active Preference Learning with
Discrete Choice Data. In Proc. NIPS '07. 409–416. http://papers.nips.cc/paper/3219-active-preference-learning-withdiscrete-choice-data.pdf• [Brochu+, SCA 2010] Eric Brochu, Tyson Brochu, andNando de Freitas. 2010. A Bayesian Interactive Optimization
Approach to Procedural Animation Design. In Proc. SCA ’10.
103–112. https://doi.org/10.2312/SCA/SCA10/103-112• [Chiu+, SIGGRAPH 2020] Chia-Hsing Chiu, Yuki

Koyama, Yu-Chi Lai, Takeo Igarashi, and Yonghao
Yue. 2020. Human-in-the-Loop Diﬀerential Subspace Searchin High-Dimensional Latent Space. ACM Trans. Graph. 39, 4,
pp.85:1–85:15 (July 2020). DOI:https://doi.org/
10.1145/3386569.3392409

Daisuke Sakamoto, and Takeo Igarashi. 2017. Sequential line
search for eﬃcient visual design optimization by crowds.
ACM Trans. Graph. 36, 4, Article 48 (July 2017), 11 pages.
DOI:https://doi.org/10.1145/3072959.3073598
Koyama and Takeo Igarashi. 2018. Computational Design
with Crowds. In Computational Interaction (Eds. Antti
Oulasvirta, Per Ola Kristensson, Xiaojun Bi, and Andrew
Howes), Oxford University Press, pp.153—184. https://
arxiv.org/abs/2002.08657• [Shahriari+, Proc. IEEE (2016)] Bobak Shahriari, Kevin

Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas.
2016. Taking the Human Out of the Loop: A Review ofBayesian Optimization. Proc. IEEE 104,1(January 2016),148–
175. https://doi.org/10.1109/JPROC.2015.2494218• [Wang+, JAIR (2016)] Ziyu Wang, Frank Hutter, Masrour

Zoghi, David Matheson, and Nando de Feitas. 2016.
Bayesian Optimization in a Billion Dimensions via Random
Embeddings. J. Artif. Intell. Res. 55 (February 2016), 361–
387. https://doi.org/10.1613/jair.4806

Yuki Koyama, Issei Sato, and Masataka Goto. Sequential Gallery for Interactive Visual Design Optimization. ACM Trans. Graph. (SIGGRAPH 2020)89Sequential Gallery

Initial…

Optimized…

Sequential Gallery
for Interactive Visual Design Optimization
Yuki Koyama1, Issei Sato2, and Masataka Goto1
1. National Institute of Advanced Industrial Science and Technology (AIST)
2. The University of Tokyo