The ABC of
Mixed Reality Interactions
Hrvoje Benko
Facebook Reality Labs

1

A new computing era!

2Command LineInterfaces(text based)

Graphical User
Interfaces(mouse + keyboard)

1960s

1980s

Natural User Interfaces(touch/gestures, tablets,smartphones)
2000s

Mixed Reality Interfaces

2020s

3

Command LineInterfaces(text based)

Graphical User
Interfaces(mouse + keyboard)

1960s

1980s

1968
Engelbart & English
MOAD

Natural User Interfaces(touch/gestures, tablets,smartphones)
2000s

Mixed Reality Interfaces

2020s

1983
Microsoft Mouse / Apple Mouse

4Command LineInterfaces(text based)

Graphical User
Interfaces(mouse + keyboard)

1960s

1980s

Natural User Interfaces(touch/gestures, tablets,smartphones)
2000s

1985
U of T - MultiTouch Tablet

Mixed Reality Interfaces

2020s

2007
Apple iPhone

SK Lee, W. Buxton, and K. C. Smith. A multi-touch
three dimensional touch-sensitive tablet. In Proc. ofthe CHI '85.
5Command LineInterfaces(text based)

Graphical UserInterfaces(mouse + keyboard)

1960s

1980s

Natural User Interfaces(touch/gestures, tablets,smartphones)
2000s

Mixed Reality Interfaces

2020s???

6

New York 2001

7

Bell B., Feiner, S., and Hollerer, T. Columbia
Touring Machine - ACM ISAR 2001 8

Bell B., Feiner, S., and Hollerer, T. Columbia
Touring Machine - ACM ISAR 2001 9

Bell B., Feiner, S., and Hollerer, T. Columbia Touring Machine - ACM ISAR 2001

10

Bell B., Feiner, S., and Hollerer, T. Columbia Touring Machine - ACM ISAR 2001

11

Bell B., Feiner, S., and Hollerer, T. Columbia Touring Machine - ACM ISAR 2001

12

Bell B., Feiner, S., and Hollerer, T. Columbia Touring Machine - ACM ISAR 2001

13

Command LineInterfaces(text based)

Graphical User
Interfaces(mouse + keyboard)

1960s

1980s

Natural User Interfaces(touch/gestures, tablets,smartphones)
2000s

Mixed Reality Interfaces

2020s

2001
MARS - Columbia Touring
Machine @ ISAR 2001

14

Michael Abrash“Imagine AR glasses that are sociallyacceptable and all-day wearable, that give
you useful virtual objects like your phone,
your TV and virtual work spaces, that give
you perceptual super powers, a contextaware personal assistant, and above all
the ability to connect, share andcollaborate with others anywhere, anytime. If those glasses existed today, we’d all
be wearing them right now.”15

Might be “obvious”…

16

… why not available today?

17

2001 vs. 2018 - Same Challenges
Display
OpticsBattery
Tracking/localization
Compute power
Spatial understanding
Spatial audio
Input/Interactions
18

2001 vs. 2018 - Same Challenges
Display
OpticsBattery
Tracking/localization
Compute power
Spatial understanding
Spatial audio
Input/Interactions
19

What kind of interactions willdefine the MR era?

20Compelling MR interactions will beadaptive,believable, andcomputational.
21

Compelling MR interactions will beadaptive,believable, andcomputational.
22

Magic of MR interactions happens
when they are tightly coupled to theuser’s environment.

23

Projection AR

IllumiRoom: Peripheral Projected Illusions for Interactive Experiences. Jones, B.,
Benko, H., Ofek, E., and Wilson, A. D. In Proc. of ACM CHI 2013.

31

32

Room2Room: Enabling Life-Size Telepresence in a Projected Augmented Reality Environment.
Tomislav Pejsa, Julian Kantor, Hrvoje Benko, Eyal Ofek, and Andy Wilson. In Proc. of ACM CSCW 2016.

33

Magic of MR interactions happens
when they are tightly coupled to thecontext
user’s environment.

34

Contextenvironment(e.g., space geometry, object semantics, people around)​user actions(e.g., gestures, body pose, bio-signals)
user’s mental state(e.g., emotional, mental load, cognitive, focus)​task(e.g., communication, navigation, calendar)​35

PersonHCIComputer

36How the computer sees us!

Tom Igoe and Dan O’Sullivan - Physical Computing. 2004.
37

How the phone sees us!

Modified Tom Igoe and Dan O’Sullivan - Physical Computing. 2004.
38

Person

HCI

ComputerMR
Reality

Perception

World
39

Example: Context-aware Tools
40

CLAW:
Multi-purpose controller that adapts tothe user’s context of use.

Choi, I., Ofek, E., Benko, H., Sinclair, M. and Holz, C. CLAW: A Multifunctional Handheld Haptic Controller for Grasping,41
Touching, and Triggering in Virtual Reality. In Proc. of ACM CHI '18.

CLAW

Choi, I., Ofek, E., Benko, H., Sinclair, M. and Holz, C. CLAW: A Multifunctional Handheld Haptic Controller for Grasping,Touching, and Triggering in Virtual Reality. In Proc. of ACM CHI '18.

Servo Motor

Force Sensor
Voice Coil

Example: Hand interactions
adapted to the environment
45

Shangchen Han, Beibei Liu, Robert Wang, Yuting Ye, Christopher D. Twigg, and Kenrick Kin. 2018. Online optical marker-based hand46
tracking with deep labels. ACM Trans. Graph. 37, 4, Article 166 (July 2018)

47

One small problem…

People don’t generally interact in mid air!
People interact with objects and on surfaces!

48

Hand interaction context = any available surface

49

MRTouch: Adding Touch to Head Mounted Mixed Reality.
Robert Xiao, Julia Schwarz, Nick Throm, Andrew D. Wilson, Hrvoje Benko. IEEE TVCG 2018 (Vol 24, No 4, April
50 2018)

51

FYI….
Hololens Research Mode API
Depth and reflectivity data now part of the public Research Mode API(developer mode only)• https://github.com/Microsoft/HoloLensForCV• Possible now to implement MRTouch using only public APIs

52

Compelling MR interactions will beadaptive,believable, andcomputational.
53

In MR, we are obsessed with creating arich sense of reality!

54

55

For interactions,realistic is not always better

56

Believable ≠ Realistic

57

Believable =consistent with user’s
expectations + non surprising58

In AR/VR we want to induce thesuspension of disbelief, whereby
users suspend their critical faculties(i.e., sacrifice logic and realism)
to believe the unbelievable.
59

60

61

Hand Movement

Wheel Movement62

Haptic Retargeting
Azmandian, M., Hancock, M., Benko, H., Ofek, E., and Wilson, A. Haptic
Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality
Experiences. In Proc. of ACM CHI 2016.

63

Passive Haptics Rocks!

64

65

66

Putting it all together…

67

Focusing on “as real as possible”
designs can lead to sub-optimal
MR experience.
Design for BELIEVABILITY, not REALISM.

68

Compelling MR interactions will beadaptive,believable, andcomputational.
69

Command LineInterfaces(text based)

Graphical UserInterfaces(mouse + keyboard)

1960s

1980s

Natural User Interfaces(touch/gestures, tablets,smartphones)
2000s

Mixed Reality Interfaces

2020sLocation fixed

MobileExplicit (command driven)

Implicit

Sensing poor

Sensing richSingle modality

Multimodal

Dedicated/cognitively capturing
Precise and accurate inputs

Glanceable/always-on
Imprecise and noisy inputs

70

AR Interactions
Iterative
DesignSensorFusionMachineLearning
71

Can you type on a phone touch keyboard?

72Modern Android touch keyboard (two thumbs)
No correction – 27.5 WPM, error rate 6.5%
With correction – 31 WPM, error rate 1.1%

Reyal, S., Zhai, S. and Kristensson, P.O. Performance and user experience oftouchscreen and gesture keyboards in a lab setting and in the wild. In Proc. ACM
CHI 2015. 679-688.

73

Probabilistic phone touch keyboard
Layers of probabilistic models:• Touch precision model• Dictionary model• Language model+ N-best list UI for error
correction+ Gesture model

74

Similar optimizations needed in MR• Text entry

75

Similar optimizations needed in MR• Text entry• Object selectionOlwal, A., Benko, H., Feiner, S. SenseShapes: Using StatisticalGeometry for Object Selection in a Multimodal Augmented Reality
System. In Proc. ISMAR '03. 300-301.

76

Similar optimizations needed in MR• Text entry• Object selection• (Multimodal) input fusion

77

Similar optimizations needed in MR• Text entry• Object selection• Multimodal input fusion• Output optimizations

Bell B., Feiner, S., and Hollerer, T. View Management for
Virtual and Augmented Reality. In Proc. ACM UIST 2001.

78

Similar optimizations needed in MR• Text entry• Probabilistic object selection• Multimodal input fusion• Output optimizations
Gal R., Shapira, L., Ofek, E., Kohli, P. FLARE: Fast
Layout for Augmented Reality Applications. In
Proc. IEEE ISMAR 2014.
79

formulate the UI challenges ascomputational problems(inferring, sensor fusing,predicting, tolerating noise)that adapt the interface
depending on the user and worldcontext
80

Compelling MR interactions will beadaptive,believable, andcomputational.
81

Maybe it should have been the CAB of MR Interactions?

Computational
Approaches

AdaptiveInterfaces

Believable
Experiences82

Design interactions that adapt to theuser’s actions, the world aroundthem, and the context of use.

83

Focus on believability. Reality isoverrated!

84

Harness the computational methods
to overcome uncertainty, scale,noise, and enable adaptivity.

85

Hrvoje Benko
benko@fb.com
Facebook Reality Labs

Thanks to all mycollaborators!
86Extra Slides

87

88

[Smith et al., Hand Eye Coordination Patterns inTarget Selection, ETRA 2000]

SMI eye-tracking module
250 hz eye-tracker
on Oculus DK2

We can predict with 97.5% accuracy
what is the user’s intended target
2 seconds before reaching the target!

92Body Warping physicalcube

virtualcube

The Rendered BodyShifts to The Right93

World Warping

physicalcube30°virtualcube

The World Also Rotates(At Different Rate)
90°

94

IllumiRoom
Peripheral Projected Illusions for Interactive Experiences.
Jones, B., Benko, H., Ofek, E., and Wilson, A. D. In Proc. of ACM CHI 2013.

95

What is “real” vs. what we“perceive as real” is notnecessarily the same!

99

Faces or vase?

100Straight or crooked?

101

Moving or static?

102

Insisting on “as real as possible” designs can leadto sub-optimal MR experience.
Think about MR interfaces as perceptual illusions
that give the user a believable experience!
103

Typical HCI model
usersystem

104

How the computer sees us!

Tom Igoe and Dan O’Sullivan - Physical Computing. 2004.
105

How the phone sees us!

Modified Tom Igoe and Dan O’Sullivan - Physical Computing. 2004.
106

MR interaction model
userMR system

world

107

MR interaction model
userMR systemuser/world model
input understanding

world

application logic
output generation

108