The Future of
Mixed Reality Interactions
Hrvoje Benko
IEEE AIVR 2019

Facebook F8 2017

• Sensory and social superpowers

A Vision ofAll-day MR• Communicate and collaborate at adistance• Next computing platform

Facebook F8 2017

• Sensory and social superpowers

A Vision ofAll-day MR• Communicate and collaborate at adistance• Next computing platform

Facebook F8 2017

Hollerer, T., Bell, B., Feiner, S., et al.
Mobile Augmented Reality System, ISAR 2001

Hollerer, T., Bell, B., Feiner, S., et al.
Mobile Augmented Reality System, ISAR 2001

What is taking so long?

Display

ComputeOptics

AudioBatteryTracking

Display
Audio

ComputeOptics
Interactions & Interfaces

BatteryTracking

Command LineInterfaces(keyboard)

Graphical User
Interfaces(mouse)

1960s

1980s

Natural User Interfaces(touch/gestures, tablets,smartphones)
2000s

New Computing Era =New Display Form Factor+ New Input Method+ New Interactions

Mixed Reality Interfaces

2020sNot convinced?

Command LineInterfaces(keyboard)

Graphical User
Interfaces(mouse)

1960s

1980s

Natural User Interfaces(touch/gestures, tablets,smartphones)
2000s

New Computing Era =New Display Form Factor+ New Input Method+ New Interactions

Mixed Reality Interfaces

2020sCommand LineInterfaces(keyboard)

Graphical UserInterfaces(mouse)

1960s

1980s

1968
Engelbart & English
MOAD

Natural User Interfaces(touch/gestures, tablets,smartphones)
2000s

1983
Microsoft Mouse / Apple Mouse

Mixed Reality Interfaces

2020sCommand LineInterfaces(keyboard)

Graphical User
Interfaces(mouse)

1960s

1980s

1985
MultiTouch Tablet

Natural User Interfaces(touch/gestures, tablets,smartphones)
2000s

Mixed Reality Interfaces

2020s

2007
Apple iPhone

S.K. Lee, W. Buxton, and K. C. Smith.
A multi-touch three dimensional touch-sensitive tablet.
In Proc. of the ACM CHI '85.

Command LineInterfaces(keyboard)

Graphical UserInterfaces(mouse)

1960s

1980s

Natural User Interfaces(touch/gestures, tablets,smartphones)
2000s

Mixed Reality Interfaces

2020s“The future is already here – it’s just not
evenly distributed yet.”
William Gibson

What makes for compelling
Mixed Reality interfaces?

What makes for compelling
Mixed Reality interfaces?

Compelling MR interfaces areadaptive,believable, andcomputational.

IllumiRoom: Peripheral Projected Illusions for Interactive Experiences.
Jones, B., Benko, H., Ofek, E., and Wilson, A. D. In Proc. of ACM CHI 2013.

IllumiRoomIllumiRoom

Jones, Benko, Ofek and Wilson, CES Las Vegas and ACM SIGCHI 2013Magic of MR interactions
happens when they are tightly
coupled to the user’senvironment.
context

Context
environment(space geometry, object semantics, people around,…)task(communication, navigation, calendar,…)user actions(gestures, body pose, bio-signals,…)user’s mental state(emotional, mental load, cognitive focus,…)Context not known at design
time.

Person

HCI

ComputerPersonComputer

HCIMR

RealitySensingWorldCompelling MR interfaces areadaptive,
believable, andcomputational.

Compelling MR interfaces areadaptive,believable, andcomputational.

In MR, we are obsessed withcreating a rich sense of reality!

For interactions,realistic is not always better

Faces or vase?

Straight or crooked?

Moving or static?

Think about MR interfaces as perceptual
illusions that give the user a believableexperience!

Believable ≠ Realistic

Example…

But, passive haptics don’t
scale!

Haptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality Experiences.
Azmandian, M., Hancock, M., Benko, H., Ofek, E., and Wilson, A. In Proc. of ACM CHI 2016.

Leverage the dominance ofvision to retarget people's
hand motions.

Body Warping

The Rendered BodyShifts to The Rightphysicalcube

virtualcube

World Warping physicalcube

30°
virtualcube

The World Also Rotates(At Different Rate)
90°

Putting it all together…

We can even estimate the mass effect!

Samad, M., Gatti, E., Hermes, A., Benko, H., and Parise, C. (2019). Pseudo-Haptic Weight: Changing thePerceived Weight of Virtual Objects By Manipulating Control-Display Ratio. In Proc. of ACM CHI 2019.

Focusing on “as real aspossible” designs can lead tosub-optimal MR experience.
Design for BELIEVABILITY, not REALISM.

Compelling MR interfaces areadaptive,believable, andcomputational.

Command LineInterfaces(keyboard)

Graphical UserInterfaces(mouse)

1960s

1980s

Location fixed
Precise and accurate inputs
Sensing poor
Explicit (command driven)

Natural User Interfaces(touch/gestures, tablets,smartphones)
2000s

Mixed Reality Interfaces

2020s

MobileImprecise and noisy inputs
Sensing rich
Implicit (context assisted)How to deal with imprecise,noisy, but sensing-richinputs?

Compelling MR interfaces areadaptive,believable, andcomputational.

Can you type on a phone keyboard?

Probabilistic Phone Touch Keyboard
Keyboard geometry model+ Touch precision model+ Dictionary model+ Language model+ N-best list UI for error
correction+ Gesture modelSmart virtual keyboard can be better than aphysical keyboard

MID AIRNo haptics upon contact

ON SURFACE
Passive haptics upon contact

Dudley, J., Benko, H., Wigdor, D., and Kristensson, P.O. (2019). Performance Envelopes of Virtual
Keyboard Text Input Strategies in Virtual Reality. In Proc. of IEEE ISMAR 2019.

Entry Rate Results
A. 2 fingers, mid-air
B. 2 fingers, on-surface
C. 10 fingers, mid-air
D. 10 fingers, on-surface

Plot shows participant q1, median and q3 (sorted by median) entry rates as well as
lumped condition q1, median and q3 entry rates. Only entries where error rate < 10%.

2 Finger VR Typing at >100 WPM

Stimulus: How are things with you?

The role of MR Interaction Designer
Iterative
DesignSensor
FusionMachineLearning

Antii Oulasvirta, Per Ola Kristenson,
Xiaojun Bi and Otmar HilligesMR Interaction Pipeline

UnderstandContext

Infer Goals

Adapt
Interaction

User Input

MR Interaction Pipeline

UnderstandContext

Infer Goals

Adapt
Interaction

User Input

MR Interaction Pipeline

UnderstandContext

Infer Goals

Adapt
Interaction

User Input

MR Interaction Pipeline

UnderstandContext

Infer Goals

Adapt
Interaction

User Input

MR Interaction Pipeline

UnderstandContext

Infer Goals

Adapt
Interaction

User Input

Learning MR UI Policies from Gaze Data
Trained RL agents to predict when an MR label is meaningful to the user.

Context: User’s gaze behavior +task + environment

Output: Inferring taskspecific goals + reduced
clutter of MR labels
https://ait.ethz.ch/label-agent/Gebhardt et al. “Learning Cooperative Personalized Policies from Gaze Data”
ACM UIST 2019

Key Challenge = Missing Data

Replica: 3D Spaces and Object Semantics

https://github.com/facebookresearch/Replica-Dataset
Straub et al. “Replica Dataset: A digital replica of indoor spaces” arXiv 2019

AI Habitat

https://www.aihabitat.org/

EPIC-Kitchens Dataset

Damen et al. Scaling Egocentric Vision: The EPIC-KITCHENS DATASET. ECCV2018
https://epic-kitchens.github.io/2019

Call to action!
Collect and release datasets of MR interactions withdifferent:• Environments• Objects (both real and virtual)• Tasks• People
Including eye-tracking, bio-signals, hand interactions, bodymovements, etc.

Summary

Command LineInterfaces(keyboard)

Graphical User
Interfaces(mouse)

1960s

1980s

Natural User Interfaces(touch/gestures, tablets,smartphones)
2000s

New Computing Era =New Display Form Factor+ New Input Method+ New Interactions

Mixed Reality Interfaces

2020sCompelling MR interactions areadaptive,believable, andcomputational.

Design interactions that adapt
to the user’s actions, the world
around them, and the context
of use.

Focus on interaction believability.
Reality is overrated!

Harness the computational
methods to overcome
uncertainty, scale, noise, andenable personalization.

MR Interaction Pipeline

UnderstandContext

Infer Goals

Adapt
Interaction

User Input

Thanks to all my collaborators!

Hrvoje Benko
benko@fb.com
Facebook Reality Labs
We are hiring!