Session: Haptics

CHI 2013, Changing Perspectives, Paris, France

Reality Jockey: Lifting the Barrier between Alternate
Realities through Audio and Haptic Feedback
Kevin Fan1, Hideyuki Izumi1, Yuta Sugiura1, Kouta Minamizawa1,
Sohei Wakisaka2, Masahiko Inami1, Naotaka Fujii2, Susumu Tachi1
12Graduate School of Media
Laboratory for Adaptive
Design, Keio University,
Intelligence, Brain Science Institute,4-1-1 Hiyoshi, Kohoku,
RIKEN, 2-1 Hirosawa, Wako,
Yokohama, 223-8526 Japan
Saitama, 351-0198 Japan{kevinfan | hid1979 | y-sugiura | kouta | inami | tachi}@kmd.keio.ac.jp, {wakisaka | na}@brain.riken.jp

ABSTRACT

We present Reality Jockey, a system that confuses theparticipant’s perception of the reality by mixing in arecorded past-reality. The participant will be immersed in a
spatialized 3D sound environment that is a mix of sounds
from the reality and from the past. The sound environment
from the past is augmented with haptic feedback in crossmodality. The haptic feedback is associated with certain
sounds such as the vibration in the table when stuff isplaced on the table to make the illusion of it happening inlive. The seamless transition between live and past creates
immersive experience of past events. The blending of live
and past allows interactivity. To validate our system, we
conducted user studies on 1) does blending live sensations
improve such experiences, and 2) how beneficial is it toprovide haptic feedbacks in recorded pasts. Potential
applications are suggested to illustrate the significance ofReality Jockey.
Author Keywords

Substitutional Reality; Haptic; Spatial Sound; CrossModality; Illusion	 Figure 1. The participant perceives the experimenter fromthe past is the actual reality as she talks with theexperimenter.

cherished moments, and re-experience them any time wewant as though they are happening right around us. Thosemoments are all realities that once happened, and yet they
do not exist right here and now, therefore, we call them the“alternate realities.”

ACM Classification Keywords

H.5.m. Information interfaces and presentation (e.g., HCI):
Miscellaneous.
General Terms

Design; Human Factors;
INTRODUCTION

Ever since the human civilization began, we humans havesought many ways to leave traces of our lives. We have thisambition because we would like to treasure our memories
that happened at a different time and place. For example,
every parent would surely like to go back to the momentthat the child first walked. Imagine if we can save all those

The ambition of interacting with or re-experiencing the past
may come from wanting to have a conversation withoneself from the past [11], or from wanting to see thehistorical view of renowned places [7]. These ambitions are
best achieved with an idea popular in literatures, the timemachine. With current technologies, it is not yet physicallypossible; however, we can create such kind of illusion thatenables the users to re-experience the past with immersion,as though the past events are happening again.
Our memories from the past play an important role in ourpresent lives and have impact on our decisions for the
future. Re-experiencing the pleasant past memories could
give us pleasure. As Csikszentmihalyi stated “…toremember the past is not only meaningful in order to create
and maintain your personal identity, but it can also in itself

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. To copy otherwise,or republish, to post on servers or to redistribute to lists, requires priorspecific permission and/or a fee.
CHI 2013, April 27–May 2, 2013, Paris, France.
Copyright © 2013 ACM 978-1-4503-1899-0/13/04...$15.00.

2557

Session: Haptics

CHI 2013, Changing Perspectives, Paris, France

be a very pleasurable activity.” and “It makes it possible tochoose and in our memory keep events that have been
particularly pleasant and meaningful and thereby ‘create’ apast that helps us to deal with the future [5].” Therefore, we
see our research goal to remove the barrier between
alternate realities, which may lead to an immersive reexperiencing of the past, has a high value in our lives.
Our goal, therefore, is to develop a system that erases thebarrier between “the past” and “the present”, so that theparticipants would not regard recorded past as being simply“the past”. The “past”, in this case, is a recorded subset of
everything that has happened up to now, and we will referthese “recorded past”, which we choose to record because
they are closely related to our personal lives, as “pastreality” through this paper. We want to achieve that when
we re-experience these recorded past reality in the future,
we would perceive them to be happening again at that timeand not feel awkward about the sudden emerging of the pastreality. To achieve that, the users have to be tricked into
losing their conviction about reality, that is, they have tolose their sense of time and space, and doubt what portionof the sensations they are experiencing is “real”, so thatthey will perceive the past as the actual reality (Figure 1).
As a result, the alternate realities could gradually fade into
the present seamlessly, and users can relive their memories
or other people’s experiences vividly as though they are
happening at the present moment. In achieving that, theillusion of the past reality happening again is presented tothe participants.
The idea of being able to manipulate one’s mind throughthe act of presenting alternate realities has already been
proven by Suzuki et al. and their Substitutional Reality (SR)
System, which is visual-sensation based [17]. We intend tobuild on their idea, and suggest that with the addition of
immersive 3D spatial sound environment and haptic
feedbacks, we could add great values to the whole SRexperience and the participants are more likely to believe
the recorded alternate realities to be happening live.
The system we have developed is called “Reality Jockey.”
It is a lightweight, mobile, immersive VR system thatblends or alternately presents the sounds from the past andthe sounds from the present to the participant. Sounds fromthe past scene are augmented with haptic feedbacks tofurther strengthen the sense of they happening in the reality
such as the vibration on the table associated with recorded
sounds of something being placed on the table. We mix thelive scene and the past scene so the whole experience canbe interactive. For example, during experiencing the past
the user might ask a question, then the user would get
appropriate response because live events are also presented.
Reality Jockey consists of a noise-cancelling headphone, amotion sensor, a USB audio interface, an equalizer, anamplifier, a tactile sound transducer, a binaural microphoneembedded dummy head, and a contact microphone. The
equalizer and amplifier are needed to power and play the2558

desired frequency range of conduction sound in the tactile
sound transducer that generates the haptic feedback.
Our system has intentionally left out the visual component
in the VR experience. The reason is that we humans rely on
our visual sensation the most, and yet we can form vivid
images of our surroundings in our mind with our eyes
closed when stimulated by other sensations such as hearing,
touch, taste, and smell. Researches have shown that audio
sensation alone can provide a self-motion illusion to theparticipant [9]. Riecke et al. further stated that audio alongwith haptic feedback in cross-modality can strengthen thesense of such illusions [15]. This proves that audio andtouching sensation are important factors in how humans
perceive the events that are happening around them andthey can fabricate images and experiences in their headeven when visual sensation is not available to them.
In essence, our system has to be an immersive virtual
reality system for the users to believe in their presence bothin the live and in the past. Tachi stated three fundamental
elements of virtual reality: 3D space, real-time interaction,and self projection [18]. These elements make a virtual
experience immersive. In our system, we address each ofthese elements accordingly. The user constructs the vivid
3D space from imagination; real-time interaction isachieved by blending the live feedbacks; self projection is
intuitive because the user thinks everything is happening inlive.
We conducted user studies to validate our system on 1)
does blending live sensations make participants to perceive
the past reality to be live more easily, and 2) how beneficial
is it to provide haptic feedbacks in the recorded alternate
realities.
Our work to develop a system that erases the barrier in timeand space pays tribute to the Turing Test [21]. We aim tocreate an immersive experience to the participants so theywould believe every interaction they make with the system
is indeed real, even though it may be computer-generated
past realities. Our research has the following significances.
• A system that lifts the barrier between alternate realities
has been developed. The approach is to blend sensations
from alternate realities and transition between themseamlessly.
• We suggest that having realistic audio and haptic
sensations in the alternate realities can make a better
illusion of them happening in live.
• Our user studies revealed how humans can believe
something that is not happening to be real.
• Potential applications that can be achieved with oursystem were conceptualized.
RELATED WORK

Our research builds on the following research areas. They
are 1) confusing the conviction about reality, 2)

Session: Haptics

CHI 2013, Changing Perspectives, Paris, France

substitutional reality, 3) HCI communication systems, and 4)
audio and haptic sensations in VR systems.
Conviction about Reality

There are many researches regarding how human perceives
the reality and how they act when they lose their conviction
about reality. The rubber hand illusion phenomenon
suggests that people can actually believe a rubber hand tobe their own hand when stimulated long enough with visualand haptic sensations [2]. Roediger et al. states that people
can remember things that never happened [16].
The participants have to be first confused in theirconviction about reality in order for the alternate reality, be
it from the past or from remote venues, to sink in and seemhappening right here and now. Reality Jockey implements asimilar method suggested by Suzuki et al. and their SR
System, which aims to aid the researches in neuroscienceby alternately presenting the past scene and the live scene tothe participants [17]. The SR system focuses on the visual
perception and uses panoramic videos recorded in the past.
Therefore, it has the pitfall of having to restrict the exact
place of where the system is used; otherwise the live scene
and recorded scene would look completely different. Also
the participants are prohibited in certain movements such as
looking at themselves because they would not see their own
body in the recorded scenes. Our system, on the other hand,
can be installed anywhere and permits full movement to theparticipants’ head with the help of motion sensing.
Substitutional Reality

Superimposing an alternate reality from the past on the live
reality has been a popular topic for performance arts. The
Mirage project superimposes images of the dancer from thepast in the real world where the live dancer dances [13].
Iwai designed a performance that captures live video
images of the attendees, and then manipulate them to createtime-lapse, slow motion effects [8]. Inspired by the above
work, Takeuchi et al. created PRIMA, a system thatprovides interaction between the participant in the present
time and the people in the past time in the same place [19].
The goal of our research is similar in concept. We aim tocreate the illusion that the past events are happening vividly
again through substitutional reality. Our work addresses anissue that the previous works lack by constructing an
immersive audio environment and providing haptic
feedback, a key component in realism.

Both ELIZA and PARRY aim to demonstrate that whenhumans interact with a computer, it is possible for them tobe immersed in an illusion that they are chatting with a real
person. Our system takes a similar concept of aiming tocreate an illusion during the communication and interactionof the participants and the system that everything is thereality. Whereas the previous works used text-based
communications, our focus is on verbal and tactile
communications.
Audio and Haptic Sensation

Up until now, VR systems have mainly focused on one outof the human’s five senses, the visual. It has been the
tradeoff between interactivity and realism. However,realism of a VR system can be greatly increased if we haveat least two sensation feedbacks in cross-modality [3]. Law
et al. have developed a multi-modal interactive floor that
includes visual, audio, and haptic sensations [10]. However,their focus is only on the interactive floor as all thesensations come from it. Our system addresses the wholeenvironment surrounding the participant.
3D sound has been proven to be an important factor in
virtual environments [1]. Naef et al. developed a spatialized
audio rendering system that utilizes many speakers [14].
Our system aims for compactness by using a headphone,
and we add haptic feedback.
It has been proven that realistic haptic feedbacks can begenerated from sound. Minamizawa et al. have developed
TECHTILE toolkit, an easy-to-use haptic device that cangenerate realistic haptic feedbacks through either playing
recorded sounds or streaming live sounds to the haptic
reactor [12]. We therefore use a similar concept ingenerating haptic feedbacks in real-time from recorded
sounds.
In Reality Jockey, we construct a spatialized soundenvironment and add haptic feedback in cross-modal effect.
For example, the participants would feel the realistic
vibrations on the table when they hear something is placedon the table.
THE REALITY JOCKEY SYSTEM

Reality Jockey is a VR system that erases the barrier
between the reality and the alternate reality that happened ata different time and space. We do so by immersing theparticipants in an environment with audio and haptic

HCI Communication Systems

The Turing Test suggests a method to measure if acomputer can imitate a real human being and thus
demonstrating machine intelligence [21]. Weizenbaum
developed ELIZA, a computer program that acts the role ofa psychotherapist and responds to the user’s typing toprovide the illusion that it is a human being [22]. Colby etal. also created a similar system, PARRY, that takes therole of a paranoid patient [4].

Figure 2. The Reality Jockey system.

2559

Session: Haptics

CHI 2013, Changing Perspectives, Paris, France

feedbacks. The audio feedback consists of a spatialized 3D
sound system that plays back a recording of the past sounds
such as conversations between people and environment
sounds. The 3D sound system mimics the source where thesounds would be actually coming from if the participant
were placed in the same environment as to provide an
immersive audio component in our VR system. The audio
component is augmented with haptic feedback associatedwith certain sounds. The key point in providing a sense of“being there” in the past is the seamless transition between
the live reality and the past reality. We continuously blend
or alternate between the realities to confuse the participantsin their conviction of true reality.
Overview

The Reality Jockey system (Figure 2) consists of a noisecancelling headphone (Bose QuietComfort 15), a motion
sensor (ATR-Promotions TSND121), a USB audio
interface (Cakewalk UA-101), an equalizer (DBX Graphics
Equalizer 131), an amplifier (Classic Pro DCP800), a tactile
sound transducer (Clark Synthesis TST239), a binauralmicrophone embedded dummy head (Kenwood DH-67),
and a contact microphone (Sun Mechatronics MW-22). The
USB audio interface expands the number of inputs andoutputs to the PC’s sound interface, and the microphones,
headphone, and transducer connect to it. The equalizer cuts
off the sound in the high frequency range and only sends
low frequency sounds (<400 Hz) that symbolize theconduction sounds of the vibration recorded with thecontact microphone. The amplifier powers and amplifies
the sounds sent from the equalizer and sends them to thetactile sound transducer, as it needs external power. The
tactile sound transducer is fixed on a wooden table, and it
generates haptic feedback to the participant’s hands. The
dummy head, with binaural microphone embedded, canrecord and also stream live spatialized sound; theparticipants then hear the 3D spatialized environment soundfrom the headphone. To compensate for the possible
movement of the participant during the usage of the system,
we use the motion sensor so that when, for example, theparticipants turn their heads the sound sources would stayin their original places as to provide a realistic real-worldsound environment.
DesignReality Jockey aims to lift the barrier in time and space.
Our method to do so is to present both the reality and thepast to the participants and continuously alternates between
the two. The switching is intended to be done seamlessly so
the participants would then begin to lose track of which is
the live reality, and therefore would believe that the past isactually happening right here and now.
Our approach is to only consider audio and haptic
sensations and investigate the cross-modal effect. Since ourgoal is to mix live and past realities, and gradually fade into
the past, we have to reduce the presence of the live reality,
and increase the presence of the past reality. The easiest and

2560

Figure 3. Recording is divided into two tracks.

most intuitive way of doing so is to simply close our eyes.
Without visual sensation, our minds are free to create
images of the surroundings that may not be the actual
reality. For example, people tend to close their eyes while
listening to fantasy stories to imagine actually being in
those virtual worlds. Therefore, we have excluded thevisual component.
The haptic sensations involved in our system are implicit.
We consider these implicit haptic sensations, such as thevibration in the table when stuff is put on it, because thesesensations are natural. They are regarded as part of oureveryday lives so having them greatly increases the realismof virtual experiences.
We will separate the design of the system into the recording
phase, where we discuss how we record the past scene
contents to be experienced by the system, and theexperiencing phase, where we discuss how we blend andsubstitute realities and present them to the participant.
Recording phase

Reality Jockey substitutes in an alternate reality from thepast. Therefore, it is crucial how we do the recording of thesounds in the past. Since we would like to provide audio
and haptic sensations in cross-modal effect, we have torecord the sounds in two different tracks: one for thegeneral environment sound that we humans realize ineveryday life, and one for the conduction sound associatedwith the vibration in materials (Figure 3).
We humans can perceive sounds coming from particularpoint source or direction thanks to how our heads and ears
are shaped. The acoustic signals are modified and received
at each ear differently according to the head-related transfer
function (HRTF). To simulate how a human head receives
sound signals and thus recording it, we use a dummy headthat has one microphone embedded in each of the ears. The
two sound channels recorded using such a method, whenplayed together, will produce realistic binaural sounds thatare the same as though we are hearing it with our ears inreal time, and thus we can identify where the sound iscoming from.

Session: Haptics

CHI 2013, Changing Perspectives, Paris, France

Figure 4. The participants experience the blending of realand past.

To record the vibrating conduction sounds that will later beplayed through the tactile sound transducer to mimic
vibrations in real time, we attach the contact microphone ona table made of materials that allow easy vibration such as
wood. The contact microphone will pick up conduction
audios caused by the vibrations as people interact with thetable.
Experiencing phase

To confuse the participants in their conviction about reality
so that they begin to doubt what portion of the sensations
they feel is the true reality, we blend and alternate the live
reality and past reality and present them to the participants
with audio and haptic feedbacks (Figure 4). The seamless
transition between live scene and recorded scene is the key
factor that immerses the participants in a reality that is boththe present and the past.
Our system plays the recorded sound to the participants
though the noise-cancelling headphone. To manipulate
where the recorded sound would be coming from in a
spatialized sound environment as to provide a realistic
feeling, we developed a program with FMOD Ex
Programmer’s API [6]. The simple user interface allows us
to do simple click-and-drag movements of the soundsources so they can move around the participant if therecorded audio is not binaural (Figure 5). The program also
takes as an input the motion sensor readings to determine if
the participants have moved their heads, and adjust thesound environment accordingly. The participant’s positionand head orientation is also indicated by the triangle.
The audio vibration to the feet is played in a separate track
to the tactile sound transducer, which is fixed on the same
type of table used in the recording phase. The recorded
conduction sound first goes through the equalizer to be
filtered so that only frequencies of 400 Hz and less aretransmitted to the tactile sound transducer. This step isnecessary because during the recording phase, the contact

2561

Figure 5. PC side user interface. Visualizes theparticipant’s position (triangle) and allows real-time
addition of virtual sounds sources if needed (square).

microphone does not only record the low frequency
conduction sounds, but will also record high frequency
environment sounds. Once the desired frequency range isfiltered, the sound signal is sent to the amplifier to be
amplified so that when the tactile transducer plays thesound, the vibration is a close approximation to what theparticipants would feel if a real person actually touches thetable or something is placed on the table. The audio
vibration is played in sync with the environment soundplayed in the headphone to make the transducer generate
vibrations at the exact timing to mimic as if the interaction
with the table happens in real time and not in the past.
Aside from playing sounds from the recorded scene, we
blend in the sounds from the live scene. The same dummy
head used in the recording phase is placed in close
proximity to the participant’s head and captures the live
sounds. The FMOD program corrects the offset in the initialdifference in distance between the participant’s head andthe dummy head. The blending of the past and live isimportant as it allows participants to interact with thesensations they are feeling through chatting and they would
get live responses based on their statements, which leads tothe perception that everything presented to them arehappening right “here” and “now”, including the past
events.
USER STUDY

We conducted user studies to validate our design andimplementation of our system. Our main goal is to study 1)
does blending live and past make participants to perceive
the past reality to be live more easily, and 2) how beneficial
is it to provide haptic feedbacks in the recorded alternate
realities. The hypothesis is that if we transition between live
and past seamlessly, and augment the past with haptic
feedbacks, the participants are more easily immersed in theSession: Haptics

CHI 2013, Changing Perspectives, Paris, France
experimenter, and they experienced our system with live
feedbacks from the live reality. The other ten participants
were presented only with the past and received no live
feedbacks from the experimenter, and they could notconfirm that the experimenter is actually in the room
because a different person led them into the room while
they were blindfolded. The key point is to compare whether
having a transition from live to past and live feedbacks,
such as the constant touch from the experimenter, cancreate the illusion in the participant’s head that everythingis happening in live, even including the past reality.
The scenario used in this test focuses on the participants’reactions to four different scenes that were recorded
beforehand.
• The experimenter asks the participant about the audio
level of the headphone.

Figure 6. User study.

• The experimenter asks to take a picture of the participant.
past realities. We observed how the participants reacted tolive and past events.

• The experimenter’s colleague joins, and the experimenterasks them to greet each other.

Participants• The experimenter gets a phone call and leaves the room,
then comes back and ends the experiment.

We recruited 20 participants; 10 were male and 10 werefemale. They are aged between 20 to 35 years old (average:
26.6, SD: 3.53). All participants had no previous knowledgeof our system. The 20 participants were divided into 2 equalgroups. Each group was tested in two experiments but thetwo groups were tested in different conditions.
Environmental Setup

The experiments were conducted in a small meeting room
as to provide a closed, quiet environment. The participants,
blindfolded, were seated across a table from theexperimenter and put their hands on the table (Figure 6).
The table has a tactile sound transducer fixed to it. A
dummy head is also placed in close distance from theparticipant’s head. Each experiment was video-recorded.
Experimental Design

We designed two distinctive experiments to validate thetwo different element of our system. The first experiment isfocused on the effectiveness of blending live and pastrealities. The second experiment is designed to test thehaptic component in alternate realities. In both experiments,
the participants are instructed to not react to anything if
they do not believe the sensations they feel are actually
happening in the live reality. We measure if they react toany past reality events.
Blending live and past

The goal of this user study is to examine the whole conceptof whether having a sensation of live feedback and live
interaction can make the participants believe that the pastreality is actually happening before them and how oursystem performs in providing such kind of a experience thatimmerses the participants in a environment that is both now
and the past. Ten participants saw the experimenter
beforehand personally and were led into the room by the2562

The four recorded-scenes were all presented to the 20
participants. The difference is that in the situation for thebatch of participants where they experienced the blendingof the live and the past, the experimenter encouraged theparticipants to chat and constantly touched the participants
before playing the recorded-scenes, whereas for the otherbatch, the experimenter refrained from interacting with theparticipants as much as possible.
Haptic in alternate realities

As mentioned earlier, haptic feedbacks can greatly increase
the feeling of immersion in VR. We designed this user
study to examine if haptic feedbacks from the past reality
can indeed help the participants to believe in the past reality
more easily. Ten participants were tested with haptic
feedbacks, whereas the other ten were tested without.
The idea is to present recorded past scenes to theparticipants that would normally result in haptic sensations
had them happened in live situations. The four recorded
past scenes presented to the participants are the following.
• The experimenter taps the table while making a
conversation.
• The experimenter taps the table to instruct the participants
to move their hands• The experimenter places a book on the table and slides itto the participant.
• The experimenter stands up but trips over towards theparticipant across the table.
For the ten participants that were tested with haptic
feedbacks, they would feel vibrations in the table with theirhands although the scenes were not really occurring. The

Session: Haptics

CHI 2013, Changing Perspectives, Paris, France

other ten participants were not presented with any haptic
feedbacks.
RESULTS
Blending Live and Past

We compared the results and reactions of the participants inthe two different test situations. The first situation is thatthe participants knew beforehand that the experimenter isactually in the room and interacted with the experimenter;we consider this situation to be “with blending.” Thesecond situation is that the participants have no priorconfirmation that the experimenter actually exists in theroom, and received as little interaction as possible; we
consider this situation to be “without blending.”
With blending

10 participants experienced this situation. We observed ifthey reacted to recorded past scenes. Except for the case of
1 participant, all 9 participants reacted to at least one scene.
Among them, 4 participants reacted to all 4 scenes. It wasobserved that the typical reaction include raising hand to
signal the audio level is perfect, sitting straight for the
picture, and turning and greeting the recorded colleague
walking in. The most interesting reaction is during the last
scenario where the recorded experimenter comes back into
the room and walks to the front of the participant to end theexperiment. Meanwhile, the live experimenter sneaks
behind the participant’s back. As a result, when theparticipants took off their blindfold, they were trying tosearch for the experimenter, who were nowhere in theirfield of sight. 5 participants were surprised when they were
pat on the back by the experimenter. During the interview
after the experiment, we learned that most participants
reacted because they heard the live experimenter talking
back to them so they felt everything has got to be real. For
the one participant who did not react to any scenes, it wasnoted that there was a small difference in sound quality
between live and past, and it did not escape the participant’s
ears, who was a musician in spare time.
Without blending

The other 10 participants experienced this situation. It wasobserved that half of the participants still reacted to thequestion about audio level because they thought it was aformal procedure to begin the experiment. Most participants
did not react to any of the scenes because they felt no realinteraction between the record experimenter’s voice andthem. 3 people also noted that they felt the experimenter
was not actually in the room because they did not see andtouch the experimenter beforehand.
Haptic in Alternate Realities

The participants were again divided into two groups. We
observed how they reacted to the recorded events thatwould normally generate haptic feedback in the table hadthem happened in live reality. One group experienced withcomputer-generated haptic, and the other had no haptic
feedback.

2563

Figure 7. Comparison of the number of people who reacted
to the past in the experiment “blending live and past.”

Figure 8. Comparison of the number of people who reacted
to the past in the experiment “haptic in alternate realities.”
With haptic

It was observed that more than half of the participants
reacted to each scenes except the last two scenarios. The
reason is that they felt the vibration in the table seemed too
weak to be real had the scenes really happened. Aninteresting phenomenon is that during the recorded scene
where the experimenter taps the table to instruct theparticipants to move their hands to the left, although thevibration should have came from one single point that is
between the participants’ two hands, all participants
claimed that they felt the vibration came from the left
because they also heard tapping sound from the left. Thisconfirms our original design in that when audio and haptic
feedback is in cross-modal effect, it creates more realistic
illusion.
Without haptic

The result of this experiment is close to our hypothesis. The
participants could easily tell that the scenes were recorded.
They told us when they felt no vibration in the table when
they heard the book being dropped and experimenter
tripping over, they instantly knew the scenes were not real.
When asked why some reacted to the first two scenarios,
they said because although they heard tapping sound on thetable, they were unsure that the gentle tapping would haveactually caused haptic feedback in real life.

Session: Haptics

CHI 2013, Changing Perspectives, Paris, France

PUBLIC DEMONSTRATION

We had the chance to do a public demo of Reality Jockey ata Japan domestic conference. During a total time span of 6hours in 2 days, a total of 42 participants experienced oursystem. For our demonstration, to better visualize the wholesystem and situation to the audiences observing theparticipant experiencing the system, we designed scenariosthat involve the experimenter walking around theparticipant and asking the participant to point in the
direction of the experimenter. This way, the audiences
could get a rough idea of Reality Jockey even though they
are not experiencing the blending of realties themselvesbecause they could see, for example, the participant
pointing at the recorded experimenter’s voice on the rightwhile in reality, the experimenter is actually on the left. The
switch from generating haptic from the table to generatingfrom the floor was easy. All we needed to do was toconstruct a wooden plank floor and record footstep
conduction sound on it. During the demonstration, theparticipants place their feet on the plank; the tactile
transducer is fixed beneath the plank.
The participants were not told the exact details of thedemonstration before experiencing the system besides
knowing that we are mixing realities through audio andhaptic feedback. The participants were asked to either wear
a blindfold or close their eyes. During the demonstration,
we interacted with the participants through chatting, asking
them to raise their hands or point, and lightly touching on
the arms. Example scenes of blending of the realities
include 1) the live experimenter stays stationary, but theparticipant hears and feels the past experimenter walking
around and talking, 2) the participant hears a recorded sceneof the experimenter getting a phone call during theirconversation, and 3) the past experimenter talks before theparticipant while the live experimenter sneaks behind theparticipant to give the element of surprise through a light
pat on the back.
Since the goal was to demonstrate our system, theparticipants were told the truth after every time they
responded to an event that happened in the past reality. Still,
during each demonstration that consisted of several
instances of blending, even though the participants had theirguard up after the first instance, they were still generally
tricked in the instances that followed. For example,
participants pointed or faced in the direction of an empty
space while talking, or trying to search for their cellphones
when they heard the ring, just like what people usually dowhen they hear a cellphone ring in a public place, and many
participants tried to search the place of the experimenter
before them after they took of their blindfold because thedemonstration ended with the recorded experimenterstanding right before them, only to be surprised by the
gentle pat on the back by the live experimenter whosneaked behind them. Through this demonstration, we
observed priceless expressions on the participants’ faces(Figure 9). These expressions come from the realization of
having just been tricked. It also confirmed that Reality
Jockey could be used and appreciated by the public as amean of experiencing past realities.
DISCUSSION
Seamless Transition

A problem of experiencing current virtual reality systems isthat there has always been a barrier between the real reality
and the virtual (alternate) realities. The alternate realities
may have truly realistic sensation feedbacks that provide
immersive feeling, but without a seamless transition
between the realities, the participants would know that
although what they are feeling is so realistic, it is not really
happening and is computer-generated.
In Reality Jockey, the whole experience first starts with live.
Then gradually the past (alternate) reality blends in, andbecause the participant can still interact with people in thelive reality, the participant feels that everything should still
be happening in live. Finally, the portion of the past reality
increases and the live reality decreases and the seamless
transition is complete. The effect of seamless transition is
confirmed in the user study that experimented on theblending of live and past. In the last part of the experiment,
we played only recorded past of the experimenter asking for
permission to answer a phone, walks out of the room, thencoming back to end the experiment. During that whole time,
the portion of live sensation is already less than the pastsensation, and yet half of the participants still reacted to thepast reality as though everything is live.
Live and Past Modality

For the transition between live reality and past reality toseem seamless, the live and past modalities in the sensationof audio and touch should feel closely the same. For
example, if the audio level and tone of the live
experimenter and the recorded experimenter is too different,
the participants could instantly detect a transition between
live and past. This happened in one of the case of theparticipants. That participant has a particular precise sensein hearing and could differentiate a single flick in the audio
quality when recorded audio is played. To solve this, during
the recording of past audios, we should conduct in a spacethat is absolute quiet. Then in the live experience, playing a

Figure 9. Participants enjoying Reality Jockey.

2564

Session: Haptics

CHI 2013, Changing Perspectives, Paris, France

small background static white noise may also help. For thehaptic sensation, we could achieve close to reality
sensations for small vibrations by frequency filtering andamplifying. However, the sensation of weight could not be
simply generated from the tactile sound transducer so someof the participants detected that the book on the table andthe experimenter tripping over is indeed not in the reality.

One suggested treatment to mental trauma patients is tohave them experience the situation again and to havepsychiatrists guide them during the re-experiencing. Our
system can aid those situations, being able to focus on thepatient’s mental imagination ability and create an
immersive experience of the past easily and safely.

Effect of Audio and Haptic Cross-Modality

Digital shrine

Our system excludes the visual sensation and focuses on the
cross-modal effect of audio and touch sensation. The idea isthat excluding the visual sensation is the easiest and mostintuitive way of decreasing the presence of the live reality
and increasing the presence of the alternate realities throughimagination and stimuli in other sensations.
The effectiveness of the cross-modality is proved in theuser study and the public demo. Although the haptic
sensation comes from one single point in our systembecause we only implemented one tactile sound transducer,
the participants claimed that they felt the origin of thevibration moving such as when the recorded experimenter
taps the desk in different locations in the user study, or
when the recorded experimenter walks around during the
public demo. The reason is that the spatial moving soundgave them the illusion that the vibration source was also
moving.
Importance of Blending LiveAs opposed to presenting everything that is recorded in an
immersive experience, we take the approach of blending
live sensations. This is important because just presenting
the past reality cannot provide a fully interactive experience.
The interactivity makes everything seem more real, eventhe past events. This was confirmed by the participants who
experimented with the blending of live and past who told us
although they kind of doubted some of the conversation
was not real, but because the live experimenter responded
to their statements during the conversations and also
touched them, they believed everything really did happen
before them.
Potential Applications

Since our system dwells around the concept of time, space,and reality, we have conceptualized several potential
application areas that are best used by using our system.
Reliving memories

Our system creates the illusion that the past is happening
right “here” and “now.” Therefore, it is the perfect solution
for using with keeping permanent records of the memories
that one would like to keep. Then, one could always relive
the experience as if went back in time, no matter how longit has passed. For example, parents could record the babies’
first words, first steps, and experience them many years
after. The difference between our system and thecommercial recording techniques now is that we provide an
immersive 3D spatial sound environment enhanced withhaptic feedbacks.

2565

Treating mental trauma

To talk with their passed-away loved ones have been one ofpeople’s most desired dream. Unlike the non-immersive
and non-interactive way of simply listening to voice records,
with Reality Jockey, if we have recordings of their voice
conversations, we can create an experience that is
immersive so one would feel like actually talking to thepassed-ones again in real time because the alternate reality
has blended into the live reality.
Immersive entertainment platform

Our system blends alternate realities through a realistic
vibration environment that generates haptic feedbacks insync with the 3D sounds that the participants hear. They
would experience the seamless transition from reality tovirtual reality, and before they know it, they could be
experiencing everything from the virtual world and yet feelno barriers during the transition. Therefore, it can be used
as an entertainment platform to bring the virtual worlds to
the reality.
Limitations and Future Work

Our system’s goal is to become the foundation of creating aillusion of the past so users can re-experience theirmemories. An illusion is fully immersive if we can provide
all of human’s five senses. As of now, we only have thehearing and touching sensation. We are looking to add in
the other three sensations once we are certain in their crossmodal value with hearing and touching. The first sensation
we are looking to add is visual because works like The
Rubber Hand Illusion has already suggested theeffectiveness of visual and touching cross-modality.
Another limitation is the lack of a more versatile haptic
feedback. We have yet to implement other kinds of haptic
sensation such as the explicit grabbing action from theparticipant. Adding them would allow more varied
scenarios capable to be experienced with the system.
The capability of providing a communication experience
makes our system interactive and immersive. However, thisinteractivity requires us to do versatile recordings
beforehand, and yet there is the possibility ofcommunication gap had the participant asks very specificquestions while trying to interact with the past. Also,
although our audio component is complete with headorientation tracking so the participants can move their headsaround, we have yet to implement body tracking, a featurethat can allow the participants to actually stand up and walkaround.

Session: Haptics

CHI 2013, Changing Perspectives, Paris, France

One interesting idea that can be derived from our system isthe usage in Out-of-Body Experience. Since our system canrecord the participant’s actions, such as talking andknocking the table, and play it back to the participant tocreate such illusions. The illusion can be even more
effective if we combine our system with means of affecting
how the user moves such as the PossessedHand [20].
CONCLUSION

Reality Jockey is a system that aims to lift the barriers when
one experiences alternate realities. To do that, the seamless
transition from the real reality to the alternate realities isimportant. We therefore blend the sensation from live andfrom past that the participants would feel. Our goal is tocreate a system that can provide such an immersive feelingof the past events that the illusion of them happening again
is achieved. As opposed to the conventional way ofproviding visual sensations in virtual reality systems, we
exclude visual and implements audio and haptic feedback incross-modality. The idea is that excluding visual is theintuitive way of decreasing the presence of the live reality
and increasing the presence of the alternate realities.
The participants can feel live sensation and interaction frompeople in live time. while feeling blended-in recorded audio
and haptic sensations from the past. The result is they are
immersed in an environment that is both the present and thepast. We conducted user studies to validate our concept on
blending the live and past, and on providing audio andhaptic feedback in cross-modality. The results showed thatthe participants felt more immersed and would react to therecorded events even though it was not actually happening
in real-time if both the blending and the haptic sensation ispresent in the system. We demonstrated Reality Jockey at aJapan domestic conference and received valuable feedbacks.
We also conceptualized potential applications of Reality
Jockey.
ACKNOWLEDGMENTS

This work was partly supported by JSPS KAKENHI Grant
Number 23240017.
REFERENCES

6. FMOD. http://www.fmod.org/
7. Fuji Television, Time Trip View.
http://blog.fujitv.co.jp/airmail/E20120627002.html
8. Iwai, T. Another Time, Another Space.
http://www.leonardo.info/gallery/gallery332/iwai.html
9. Larsson, P., Västfjäll, D., and Kleiner, M. Perception ofself-motion and presence in auditory virtualenvironments. In Proc. Seventh Annual International
Workshop Presence (2004), 252-258.
10. Law, A. W., Peck, B., Visell, Y., Kry, P., andCooperstock, J. 2008. A multi-modal ﬂoor-space forexperiencing material deformation underfoot in virtual
reality. In Proc. IEEE HAVE’08, 126-131.
11. MacDonald, J. A Conversation With My 12 Year Old
Self. http://youtu.be/XFGAQrEUaeU
12. Minamizawa, K., Kakehi, Y., Nakatani, M., Mihara, S.,
and Tachi, S. TECHTILE toolkit: a prototyping tool for
design and education of haptic media. In Proc. VRIC
'12, ACM (2012), 2 pages.
13. MIRAGE: Performance Art with Substitutional Reality
system. http://mirage.grinder-man.com/ (in Japanese)
14. Naef, M., Staadt, O., and Gross, M. Spatialized audio
rendering for immersive virtual environments. In Proc.
VRST '02, ACM, pp. 65-72
15. Riecke, B. E., Feuereissen, D., and Rieser, J. J. Auditory
self-motion illusions ("circular vection") can befacilitated by vibrations and the potential for actual
motion. In Proc. APGV '08.
16. Roediger, H. L., and McDermott, K. B. Creating False
Memories: Remembering Words Not Presented in Lists.
Journal of Experimental Psychology: Learning,
Memory, and Cognition, Vol 21(4), 1995, 803-814
17. Suzuki, K., Wakisaka, S. and Fujii, N. Substitutional
Reality System: A Novel Experimental Platform for
Experiencing Alternative Reality. Sci. Rep. 2, 459
18. Tachi, S. Telexistence. World Scientific, 2009.
19. Takeuchi, T., Nakashima, T., Nishimura, K., and Hirose,
M. PRIMA: Parallel Reality-based Interactive Motion
Area. In ACM SIGGRAPH 2011 Posters (SIGGRAPH'11). ACM (2011), 1 page.

1. Begault, D. 3-D Sound for Virtual Reality andMultimedia. AP Professional, 1994.
2. Botvinick, M., and Cohen, J. Rubber hands “feel” touchthat eyes see. Nature 391(6669):756.
3. Chalmers, A., Howard, D., and Moir, C. Real
Virtuality: A step change from Virtual Reality. In Proc.
SCCG '09, Stephen N. Spencer (Ed.). ACM (2009), 9-16
4. Colby, K. M., Hilf, F. D., Weber, S., and Kraemer, H.
Turing-like Indistinguishability Tests for the Validationof a Computer Simulation of Paranoid Processes.
Artificial Intelligence 3: 199–221.

20. Tamaki, E., Miyaki, T., and Rekimoto, J.
PossessedHand: a hand gesture manipulation systemusing electrical stimuli. In Proc. AH '10. ACM (2010),
Article 2 , 5 pages.
21. Turing, A. M. Computing Machinery and Intelligence.
Mind LIX (236): 433-460.
22. Weizenbaum, J. ELIZA — A Computer Program For
the Study of Natural Language Communication
Between Man And Machine. Communications of theACM 9 (1): 36–45

5. Csikszentmihalyi, M. Flow: the Psychology of Optimal
Experience. Harper and Row, 1990.

2566