Ubiquitous Substitutional Reality: Re-Experiencing the Past in Immersion
Kevin Fan1, Yuta Sugiura1, Kouta Minamizawa1, Sohei Wakisaka2, Masahiko Inami1, and Naotaka Fujii2
1
Keio University, and 2RIKEN BSI

Figure 1: Left: Re-experiencing past reality. Center: Recording past reality. Right: Experiencing at different locations of household.

1. Introduction
We propose an immersive Substitutional Reality (SR) system thatenables users to experience alternate realities as they walk aroundin the live reality’s environment by substituting or blending inpre-recorded realities. SR is fundamentally a concept of presenting
realities that are of a different time than the live reality to the users
through seamless transitions so the users feel as one coherent
experience. Suzuki et al. construct the alternate realities frompre-recorded panoramic video images of the past and aim to study
humans’ brain behavior when they perceive the past reality as
happening before their eyes and interact in their SR system [1].
We install our system in the home environment as to provide an
immersive way for people to record and re-experience theirtreasured memories. To achieve seamless transition, we integratesensors in the furniture that can sense the users’ interactions so thetransition can be implicitly triggered as users subjectively interact
with the furniture. As the sensor interfaces are integrated invisiblywith the furniture and the experience occurs naturally wheninteracted, we consider this to be a kind of ubiquitous experience.

2. System Overview
The key to an immersive SR experience is the seamless transition
between the live reality and past reality. We suggest the method ofimplicit transition when users subjectively interact with thefurniture. Implicit transition improves immersion in that the users
do not have a preconceived idea that reality is about to change.
Also, the cross-modality of visual and audio sensation improvesrealism. We add the element of binaural audio to the originalvisual focused SR system so the audio environment is just as
realistic as listening from our own ears. The system has three corecomponents: recording, experiencing, and transition.
Recording The recording of the past reality is through apanoramic video camera with two mono microphones placed at adistance close to a human’s ear distance to record in binaural. The
recording of the visual and audio sensation has to be placed at thesame location as the interaction is in the live reality (Figure 1).
Experiencing The users experience through a video see-throughhead mount display that lets them see the live and past reality. A 9
DOF sensor is attached to the HMD so the users’ head orientation

Figure 2: Sensor network implicitly triggers the transition to beexperienced by the users when they interact.
is tracked and the correct view of the recorded panoramic video
can be presented to the users. Two microphones are fixed beside
the user’s ears to capture live binaural audio. The recorded
binaural audio will also change according to orientation. The users
are able to walk around in the live reality for SR experience at
different locations in the household (Figure 1).
Transition The implicit transition is subjectively initiated by theusers as they interact with the sensor-integrated furniture (Figure2). We implemented three examples: A infrared distance sensor
that senses how much a door opens, a tracking sensor such as
Kinect that tracks the user’s position at a certain location andtriggers a binaural audio to elicit the user’s head turning which inturn triggers transition, and a sensor-integrated cushion that senses
the sitting pressure to trigger the transition. Theoretically, we canexpand to any kind and number of sensors in a household.

3. User Experience
The users put on the video see-through HMD and binauralmicrophone and can observe their surroundings while interactingwith the people in the live reality. The users can explore theenvironment any way they prefer, and when they interact with thesensor furniture, the past reality will be implicitly transitioned.
They can turn to view different angles of the past reality incross-modality with binaural sound. The implicit seamless
transition and the ability to immerse in both realities will confuse
the users into doubting what they are experiencing is the actual
live reality or a recorded one.

4. Acknowledgement
This work is supported by Microsoft Research CORE9 Project.
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor commercial advantage and that copies bear this notice and the full citation on thefirst page. Copyrights for third-party components of this work must be honored. For allother uses, contact the Owner/Author.
SIGGRAPH 2014, August 10 – 14, 2014, Vancouver, British Columbia, Canada.
2014 Copyright held by the Owner/Author.
ACM 978-1-4503-2958-3/14/08

References
1. Suzuki, K., Wakisaka, S. and Fujii, N. 2012. Substitutional Reality
System: A Novel Experimental Platform for Experiencing AlternativeReality. In Sci. Rep. 2, 459.