Augmenting Human
Capabilities for the MixedReality Future
Hrvoje BenkoDirector, Research Science @ Reality Labs Research@ MIT HCI Seminar
November 8, 2022

Facebook F8 2017

• Sensory and social superpowers​

A Vision ofAll-day MR• Communicate and collaborate at adistance• Next computing platform

Facebook F8 2017

Metaverse

Meta Connect 2021

Hollerer, T., Bell, B., Feiner, S., et al.
Mobile Augmented Reality System, ISAR 2001

Hollerer, T., Bell, B., Feiner, S., et al.
Mobile Augmented Reality System, ISAR 2001

Bell B., Feiner, S., and Hollerer, T. Columbia Touring Machine - ACM ISAR 2001

Bell B., Feiner, S., and Hollerer, T. Columbia Touring Machine - ACM ISAR 2001

Bell B., Feiner, S., and Hollerer, T. Columbia Touring Machine - ACM ISAR 2001

9

Bell B., Feiner, S., and Hollerer, T. Columbia Touring Machine - ACM ISAR 2001

10

What is taking so long?

Networking
DisplayComputeOptics

Audio
Battery

Tracking

Networking
Display

Audio

ComputeOpticsInteractions & Interfaces

Battery

Tracking

Command Line Interfaces Graphical User Interfaces(mainframes, keyboard) (personal computers,keyboard & mouse)
1960s

1980s

Natural User Interfaces(tablets, smartphones,touch/gestures)
2000s

New Computing Era =New Display Form Factor+ New Input Method+ New Interface

Mixed Reality Interfaces(MR glasses, wristbands,???)
2020s

Novel Displays

Novel Interfaces
Novel I/O Devices

https://research.fb.com/blog/2017/05/oculus-research-spotlight-meet-the-team-behind-focal-surface-displays/

https://research.fb.com/blog/2021/08/display-systems-research-reverse-passthrough-vr/

Novel Displays

Novel Interfaces
Novel I/O Devices

Novel XR Wristbands

Penfield & Rasmussen. 1950

Gray’s Anatomy Plates. 1918

Electromyography Wristbands

31

Fusion of Optical and Inertial Sensing1x IMU

8x optical proximity
sensors

4x vibrotactors

Parizi, F. S., Kienzle, W., Whitmire, E., Gupta, A., and Benko, H. (2021) RotoWrist: Continuous Infrared Wrist Angle Tracking using a Wristband. In Proceedings of ACM VRST '21.

Active Electrical Sensing of Touch andContact

Zhang, Y., Kienzle, W., Ma, Y., Ng, S. S., Harrison, C., and Benko, H. (2019). ActiTouch: Robust Touch Detection for On-Skin AR/VR Interfaces. In Proc. of ACM UIST 2019.

Active Electrical Sensing of Touch and Contact

Kienzle, W., Whitmire, E., Rittaler, C., and Benko, H. (2021) ElectroRing: Subtle Pinch and Touch Detection with aRing. In Proceedings of ACM CHI '21.

Acoustic Touch Sensing on Any SurfaceGong, J., Gupta, A. and Benko, H. (2020). Acustico: Surface Tap Detection and Localization
using Wrist-based Acoustic TDOA Sensing. In Proceedings of ACM UIST '20.

Wrist Haptics

Penzent, E., Israr, A., Samad, M., Robinson, S., Agrawal, P., Benko, H., and Colonnese, N. (2019). Tasbi: Multisensory Squeeze andVibrotactile Wrist Haptics for Augmented and Virtual Reality. In Proc. of World Haptics Conference (WHC 2019).

Haptic Gloves

Novel Displays

Novel Interfaces
Novel I/O Devices

Magic of MR interactions
happens when they are tightly
coupled to the user’senvironment.
context

Contextenvironment(space geometry, object semantics, people around,…)​task(communication, navigation, calendar,…)​user actions(gestures, body pose, bio-signals,…)user’s mental state(emotional, mental load, cognitive focus,…)​Context not known at design
time.

Person

HCI

ComputerPersonComputer

HCIMR

RealitySensingWorld

How to deal with imprecise,noisy, but sensing-richinputs?

Can you type on a phone keyboard?

Probabilistic Phone Touch Keyboard
Keyboard geometry model+ Touch precision model+ Dictionary model+ Language model+ N-best list UI for error
correction+ Gesture modelSmart virtual keyboard can be better than aphysical keyboard

MID AIRNo haptics upon contact

ON SURFACE
Passive haptics upon contact

Dudley, J., Benko, H., Wigdor, D., and Kristensson, P.O. (2019). Performance Envelopes of Virtual Keyboard Text Input Strategies in Virtual Reality. In Proc. of IEEE ISMAR 2019.

Entry Rate Results
A. 2 fingers, mid-air
B. 2 fingers, on-surface
C. 10 fingers, mid-air
D. 10 fingers, on-surface

Plot shows participant q1, median and q3 (sorted by median) entry rates as well as
lumped condition q1, median and q3 entry rates. Only entries where error rate < 10%.

2 Finger VR Typing at >100 WPM

Stimulus: How are things with you?

ComputationalApproaches
Needed• Text entry• Hand, body, touch input• Object selection• Multimodal fusion• Layout optimizations• Action recommendations• Error mitigations• Personalization

| Confidential58

MR Interaction Pipeline

UnderstandContext

Infer Goals

Adapt
Interaction

User Input

MR Interaction Pipeline

UnderstandContext

Infer Goals

Adapt
Interaction

User Input

MR Interaction Pipeline

UnderstandContext

Infer Goals

Adapt
Interaction

User Input

MR Interaction Pipeline

UnderstandContext

Infer Goals

Adapt
Interaction

User Input

ActionSensing

Prob. of Gesture

PinchMLUnderstand the Environment
Where am I? What is around me?
Project Aria - Research glasses device to help
build the 3D map of the world together with all
the objects, people and their relationships

https://about.facebook.com/realitylabs/projectaria/

Inferring user actions from sparse sensors
What am I doing?
Aria headset is doing SLAM + 2
wristbands with IMUs only areproviding full upper-body poseand helping with action
recognition

Intent to Interact using gaze dynamics
What am I trying to accomplish?
Predict user’s intent to interact with a virtual object using eye-tracking andpupillometry features alone. (AUC-ROC = 0.77, chance 0.5)
These features are consistent across individuals.

David-John, B., Peacock, C., Murdison, T. S., Benko, H., Jonker, T. Towards gaze-based prediction of the intent to interact in virtual reality. ETRA 2021

Predictive PointingHenrikson, R., Grossman, T., Trowbridge, S., Wigdor, D., and Benko, H. (2020). Head-Coupled Kinematic
Template Matching: A Prediction Model for Ray Pointing in VR. In Proceedings of ACM CHI '20.

Fusion Module

Model Intent-to-interact(Gaze + Hand)• Logistic Regression
• HMM
• CNN
• LSTM• Weighted Sum• Dempster Shafer Theory• Choquet's fuzzy integral

Prob. of Intention

Prob. ofSelection

Interface Adaptation to Minimize NoiseNo adaptation – Raw gaze highlighting

Adaptation based on I2I modelDavid-John, B., Peacock, C., Murdison, T. S., Benko, H., Jonker, T.
Towards gaze-based prediction of the intent to interact in virtual reality. ETRA 2021

Yu, D., Desai, R., Zhang, T.,
Benko, H., Jonker, T.R., andGupta, A. (2022). Optimizing theTiming of Intelligent
Suggestion in VirtualReality. In Proceedings of ACM
User Interface Systems andTechnology (ACM UIST '22).

Many other adaptations possible• Move content around• Filter information content• Correct user errors (auto-correct)• Make it easier to complete an action (auto-complete)• Suggest the next most likely action• Provide optimal guidance for a task
• Change the level of detail presented

The Intelligent ClickCommand LineInterfaces(keyboard)

Graphical User
Interfaces(mouse)

1960s

1980s

Natural User Interfaces(touch/gestures, tablets,smartphones)
2000s

Mixed Reality Interfaces

2020s

New Computing Era =Displays
NewNovel
DisplayForm Factor+ New Input Method+ New Interface

Novel Interfaces

Novel I/O Devices

Thanks to all my collaborators!
My teams are looking for interns for 2023!

Hrvoje BenkoDirector, Research Science
Reality Labs Researchbenko@meta.com

Summary

Command LineInterfaces(keyboard)

Graphical UserInterfaces(mouse)

1960s

1980s

Natural User Interfaces(touch/gestures, tablets,smartphones)
2000s

Mixed Reality Interfaces

2020s

New Computing Era =Displays
NewNovel
DisplayForm Factor+ New Input Method+ New Interface

Novel Interfaces

Novel I/O Devices

Compelling MR interactions
areadaptive and computational.

Wrist and hands are thekey to subtle XRinteractions

Design interactions that adapt
to the user’s actions, the world
around them, and the contextof use.

Harness the computational
methods to overcome
uncertainty, scale, noise, andenable personalization.

MR Interaction Pipeline

UnderstandContext

Infer Goals

Adapt
Interaction

User Input

Thanks to all my collaborators!
My teams are looking for interns for 2023!

Hrvoje Benko
Director, Research Science
Reality Labs Researchbenko@meta.com

Computational
Interactions for the XR
FutureResearch at Reality Labs

Computational
Interactions for the XR
FutureResearch at Reality Labs

Mission

Solve the interaction problem for
future all-day wearable virtual and augmentedreality

The Future of AR Interactions
Benko, ISMAR 2018

The Future of AR Interactions
Benko, ISMAR 2018

Mixed reality is a new era ofcomputing…

…contextually adaptive interfaces areessential

AgendaComputational Input @ RL Research

Computational Interfaces @ RL Research

Panel Discussion and Q&A

Computational Input

Computational Input

Wearable Sensing

AcusticoGong et al. UIST 2020

[RESEARCH PROTOTYPE]

96

97

Computational Input

Mid-air Typing

The Coactivation Problem

Understanding, Detecting and Mitigating the Effects ofCoactivations in Ten-Finger Mid-Air Typing in VirtualReality. Foy et al. CHI 2021

Computational Input

Mid-air Typing

Understanding, Detecting and Mitigating the Effects ofCoactivations in Ten-Finger Mid-Air Typing in VirtualReality. Foy et al. CHI 2021

Computational Input

Predictive Pointing

Head-Coupled Kinematic Template MatchingHenrikson et al. CHI 2020

Computational Interfaces

Computational Interfaces

Intent to interact using gaze dynamics
Input
EventInput
EventTowards gaze-based prediction of theintent to interact in virtual reality
David-John et al. ETRA 2021

Computational Interfaces

Predicting Focus of Visual Attention

Predicting visual attention using the hidden
structure in eye-gaze dynamics
Lengyal et al. CHI 2021 WorkshopComputational Interfaces

Investigating the Costs of Input Errors

False Positives vs. False Negatives: The Effects of Recovery Time and Cognitive Costs
on Input Error Preference
Lafreniere et al. UIST 2021

Computational Interfaces

Intelligent Error Mediation

Gaze as an Indicator of Input Recognition Errors
Peacock et al. ACM ETRA 2022

Computational Interfaces

Novel Approaches to Evaluating Error Acceptability

Investigating Cross-Modal Approaches for Evaluating
Error Acceptability of a Recognition-Based Input TechniqueHenderson et al. ACM IMWUT 2022

Intelligent Clickto Accept
JAZZMUSICLISTEN

JAZZ
MUSIC

ELECTRONIC
MUSIC

WATCHDRAMA
MOVIESROMANTIC
MOVIES

ENTERTAINMENTComputational Interfaces

Simulation-based Interface Adaptations

Computational Adaptation of Extended Reality Interfaces Through Interaction Simulation
Todi et al. CHI 2022 WorkshopComputational Input

Computational Interfaces

Mixed reality is a new era ofcomputing…

…contextually adaptive interfaces are
essential

Building the future
XR interface together
Full-time, Post-Doc, Interns
Research Scientists – HCI, Haptics, ML, AI
Research Engineers – SW, ML, HW
Design Technologists and Prototypers

www.meta.com/careers

0.88
0.84

0.84

0.81

Increased0.8 12%I2I+Inputdetection(PinchRecognizer)

0.84
0.82

Pinch Recognizer
DST
Weighted Sum

0.76
0.72

0.83

0.72

0.68

0.64
Precision(Recall𝑻𝑷)
𝑻𝑷 + 𝑭𝑷115

When and how to adapt theUI?
Lafreniere, B., Jonker, T. R., Santosa, S., Parent, M., Glueck, M., Grossman, T., Benko, H., Wigdor, D.
(2021) False Positives vs. False Negatives: The Effects of Recovery Time and Cognitive Costs on InputError Preference. In Proceedings of ACM UIST '21.

Project SuccessResearch Question: How can a system assess and adapt to the costs of errors?

False Negative

False PositiveProject SuccessProject Success
System Action
Mediation

Normalized Frequency

No Action

False Positives vs.
False Negatives
The Effects of Recovery Time and Cognitive Costs on Input Error Preference
Ben Lafreniere, Tanya R. Jonker, Stephanie Santosa, Mark Parent,
Michael Glueck, Tovi Grossman, Hrvoje Benko, and Daniel Wigdor

False Negative Errors

User intentionally

System fails to recognize the gesture;

performs a gesture

No action is performed

False Positive Errors

User is not intentionally

System recognizes a gesture anyway;

performing a gestureAn unwanted action is performed

Compose EmailAttach File
Delete Selected Items
Reply All
Send Message

124

Key Takeaways•••

Error-type preference can be driven by differences in the temporal costof FP and FN errors
Users exhibit a bias against FP errors, which can be equivalent to
1.5 seconds or more of added recovery time

FP errors impose greater attentional demands on users as compared toFN errors, which may partially explain this bias

125

Hand tracking

Hands in Oculus Quest 2Shangchen Han, Beibei Liu, Robert Wang, Yuting Ye, Christopher
D. Twigg, and Kenrick Kin. 2018. Online optical marker-based
hand tracking with deep labels. ACM Trans. Graph. 37, 4, Article
166 (July 2018)

Voice Assistant Interface

MR Interface• User initiates interaction• Limited contextual
understanding• Turn taking dialogue fordisambiguation• Proactive• Understands context anduser goals• Engages continuous
multimodal feedback toreduce errors and enabledisambiguation.

Learning MR UI Policies from Gaze Data
Trained RL agents to predict when an MR label is meaningful to the user.

Context: User’s gaze behavior +task + environment

Output: Inferring taskspecific goals + reduced
clutter of MR labels
https://ait.ethz.ch/label-agent/Gebhardt et al. “Learning Cooperative Personalized Policies from Gaze Data”
ACM UIST 2019