BO as Assistant: Using Bayesian OptimizationfiDe ned byhumanpreferenceHuman-in-the-loopoptimization🤔

solves optimization problems withperceptual objective functions(e.g., preference)🤖

✔︎"Which do you like?"

Designer

Evaluate

Ask

System

"I like this one."

Ask• Intelligent sampling strategy
Designerffifi

💪Evaluate🤔

✔︎"Which do you like?"

Bayesian optimization (BO)BO• "Exploration and exploitation"• Able to nd a good solution withina small number of iterations(sample e cient)"I like this one."• Less burden on the evaluatorPrevious Work: Human-in-the-Loop Bayesian Optimization

Photo Enhancement & 3D Computer Graphics
Koyama+Koyama+SIGGRAPH 2017
SIGGRAPH 2020

Melody GenerationZhou+
IUI 2020→ Successfully used in previous work to enable e

Photographic Lighting
Yamamoto+
UIST 2022
cient searchffi

8Problem & Motivation

t from BO’s
intelligence while providing thedesigner with freedom?

fi😢

I want to explore
freely!

🤖

✔︎Problem: No freedom for designer,
reducing agency and creativity"Which do you like?"

Designer

BayesianOptimization(BO)"I like this one."

Our Framework: "BO as Assistant"Learning the designer’s preference• There is no explicit loop
• BO requires no explicit inputs• BO just "assists" the designerTechnical Background:Preferential Bayesian Optimization (PBO)

Bayesian Optimization (BO)
• A global “black-box” optimization algorithm

Ask• Runs with relative preference data(rather than absolute scoring)• Better suited for human-in-the-loop🤔

Q: Can we extract relative preference
data implicitly from slider manipulation?

🤖

✔︎• A variant of BO [Brochu+, NIPS 2007]

DesignerEvaluatePBOOur Technique:
Extraction of Preferential Data for PBO

Design goal

A Closer Look: Slider Manipulation Session

0.0

1.024

A Closer Look: Slider Manipulation Session

The photo looks too dark.
Design goal

Let’s manipulate "brightness".

0.0

1.0

Brightness
25

A Closer Look: Slider Manipulation Session

Design goal

Let’s increase the value.

0.0

1.0
Press
26

A Closer Look: Slider Manipulation Session

Design goal

Still dark. Increase more.

0.0

1.0

Move27

A Closer Look: Slider Manipulation Session

Oh! Too much!!
Design goal

Let’s decrease the value.

0.0

1.0
Move28

A Closer Look: Slider Manipulation Session

Now the photo looks nice.
Design goal

Let’s stop.

0.0

1.0

Move & Release29

A Closer Look: Slider Manipulation Session

Observations:Estimated preference

Predicted standard deviation
Estimation uncertainty

Acquisition function
Priority in sampling→ Please refer to the paper for details

Predicted mean

Estimated preference

Predicted standard deviation
Estimation uncertainty

Acquisition function
Priority in sampling: Data points extracted from slider manipulation: Suggestions generated by BO→ Please refer to the paper for details32

Predicted mean

Estimated preference

Predicted standard deviation
Estimation uncertainty

Acquisition function
Priority in sampling: Data points extracted from slider manipulation: Suggestions generated by BO→ Please refer to the paper for details

33

Predicted mean

Estimated preference

Predicted standard deviation
Estimation uncertainty

Acquisition function
Priority in sampling: Data points extracted from slider manipulation: Suggestions generated by BO→ Please refer to the paper for details

34

Demonstrations

Photo Color Enhancement• #Parameters: 12• Brightness• Contrast• Saturation• Lift (RGB)• Gamma (RGB)• Gain (RGB)

x5

x5

Procedural Material Design• #Parameters: 8• Threshold• Noise Scale• Noise Detail• Noise Roughness• Noise Distortion• Ambient Occlusion• Bump Strength• Peel Boundary Strength• Design goal: Rusted metal with peeled painting