Creating Interactive Projected Augmented Reality Experiences

Hrvoje Benko
Microsoft ResearchOctober 2015Ivan Sutherland, 1968.

Columbia Touring Machine. ISAR 2001.

To create authentic augmented reality experiences that are situatedin the real world, don’t require additional gear to be worn, yet
enable a high degree of interactivity with computer-generatedcontent.

GPU-based computation

Underkoffler & Ishii, CHI ’98

Raskar et al., SIGGRAPH ‘98

Pinhanez, UBICOMP ‘01

555 Kubik - UrbanScreen

DisneyReal-time! We don’t assume static geometry.
Any surface is a display.
Procedural behaviors and rendering based on the current
conditions in the environment.
Our experiences scale from 1 to many nodes.
We want the experiences to be highly interactive, beyond thesimple controller input.

We want to enable “analog” interactions (i.e., mimic theexperience of the real world), while still offering the user
“supernatural” powers when interacting with computergenerated content.

MirageTable

Benko, Jota & Wilson, ACM CHI 2012

Benko, Jota & Wilson, ACM CHI 2012

Beamatron

Wilson, Benko, Izadi and Hilliges, ACM UIST 2012

Wilson, Benko, Izadi and Hilliges, ACM UIST 2012IllumiRoom

Jones, Benko, Ofek and Wilson, CES Las Vegas and ACM SIGCHI 2013RoomAlive

Jones, Sodhi, Murdock, Mehra, Benko, Wilson, Ofek, MacIntyre, Raghuvanshi, and Shapira. In ACM UIST 2014.

Kinect

KinectKinectKinectKinectKinectProjector

Projector

Projector

Projector

Projector

Projector

PC(UnityPCEngine)PC(UnityPCEngine)

PC(UnityPCEngine)PC(UnityPCEngine)PC(UnityPCEngine)PC(UnityPCEngine)…

1 living room (99/3319) 5x6 meters
6 projectors (6.3 megapixels at 60Hz)
6 Kinect cameras (3.7 megapixels of image data processed at 30 Hz)
6 computers

Distributed game engine (Unity)
2 head-tracked users
Triton 3D sound spatializationThe environment is unknown at design time.

Chairfloor

N. Silberman (NYU), L. Shapira, R. Gal & E. Ofek

The environment is unknown at design time.
Designer is not in complete control of the experience.
If storytelling, need to direct the person’s focus. More liketheater/theme park than a movie/game.
Magic happens when virtual stuff interacts with the real world.

Dyadic Spatial Augmented Reality

Benko, Wilson, and Zannier. In ACM UIST 2014.IllumiRoom, CHI 2013

LIghtSpace, UIST 2010

RoomAlive, UIST 2014

Edgar Mueller

Felice Varini

ProCam 1

User 1

ProCam 2

User 2

TargetUser’s POV

ProjectedProjected objects are perceived
as spatial even without stereo
rendering.

Users can understand theircollaborator’s spatial references.

FoveAR

Benko, Ofek, Zheng, and Wilson. To appear in ACM UIST 2015

Benko, Ofek, Zheng, and Wilson. To appear in ACM UIST 2015

Room2RoomPejsa, Kantor, Benko, Ofek, and Wilson. To appear in ACM CSCW 2016.

Pejsa, Kantor, Benko, Ofek, and Wilson. To appear in ACM CSCW 2016.

Open source multi-Kinect multi-projector calibration tool and viewdependent rendering samples.

Get the code: https://github.com/Kinect/RoomAliveToolkit/

Andy Wilson

Eyal Ofek

Bret Jones
Rajinder Sodhi
Michael Murdock
Tomislav Pejsa
Julian Kantor
Feng Zheng
Ravish Mehra
Yan Wang
Ricardo Costa Jota

Chris Harrison
Federico ZanierBlair MacIntyre
Shahram Izadi
Nikunj Raghuvanshi
Lior Shapira
Ran Galbenko@microsoft.com
http://research.microsoft.com/~benko