Haptic Retargeting: Dynamic Repurposing ofPassive Haptics for Enhanced Virtual Reality Experiences
Mahdi Azmandian1,2, Mark Hancock1,3, Hrvoje Benko1, Eyal Ofek1, Andrew D. Wilson1
1
2
3
Microsoft Research
University of Southern California
University of Waterloo
Redmond, WA, USA
Los Angeles, CA, USA
Waterloo, ON, Canada{benko, eyalofek, awilson}
mazmandian@ict.usc.edu
mark.hancock@uwaterloo.ca@microsoft.com

Figure 1. User building a virtual castle by arranging and stacking multiple virtual cubes mapped to a single physical cube.
ABSTRACT

Author Keywords

Manipulating a virtual object with appropriate passive haptic
cues provides a satisfying sense of presence in virtual reality.
However, scaling such experiences to support multiple virtual objects is a challenge as each one needs to be accompanied with a precisely-located haptic proxy object. We propose a solution that overcomes this limitation by hacking human perception. We have created a framework for repurposing passive haptics, called haptic retargeting, that leverages
the dominance of vision when our senses conflict. With haptic retargeting, a single physical prop can provide passive
haptics for multiple virtual objects. We introduce three approaches for dynamically aligning physical and virtual objects: world manipulation, body manipulation and a hybrid
technique which combines both world and body manipulation. Our study results indicate that all our haptic retargeting
techniques improve the sense of presence when compared totypical wand-based 3D control of virtual objects. Furthermore, our hybrid haptic retargeting achieved the highest satisfaction and presence scores while limiting the visible sideeffects during interaction.

Virtual Reality; Haptics; Perception.

Permission to make digital or hard copies of all or part of this work forpersonal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on
servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from Permissions@acm.org.
CHI'16, May 07-12, 2016, San Jose, CA, USA© 2016 ACM. ISBN 978-1-4503-3362-7/16/05…$15.00
DOI: http://dx.doi.org/10.1145/2858036.2858226

ACM Classification Keywords

H.5.1 [Information Interfaces and Presentation]: Multimedia
Information Systems-Artificial, Augmented, and Virtual Realities; H.5.2 [User Interfaces]: Haptic I/O.
INTRODUCTION

Over the past few years, virtual reality (VR) has experienced
a resurgence. With the proliferation of consumer-level headmounted displays and motion tracking devices, an unprecedented quantity of immersive experiences have been created.
Though optics, rendering, and audio technologies have improved substantially, a component that has remained underdeveloped is haptics—the sense of touch expected whenreaching out and grabbing virtual objects.
A key objective in virtual reality is establishing a sense ofpresence. When shown an environment with photorealistic
rendering, people can be convinced by the illusion of reality,and describe the experience as immersive [37], but whenreaching out to touch a virtual object, the illusion can be shattered when one’s hand unexpectedly passes through the rendered visuals.
One method for adding a sense of touch to virtual objects isthrough passive haptics [17], where physical props areplaced around the environment to match their virtual counterparts. The result can be a compelling tactile sensation
when reaching and touching an object. However, this technique requires each virtual object to have a physical prop of
the same size and shape, in the correct position, to create thisillusion. Furthermore, each time the virtual scene is changed,
the infrastructure must be modified and possibly rebuilt. This
limits the ability to develop complex virtual environmentswith passive haptics.
In this paper, we explore a solution that overcomes these limitations by hacking human perception. We have created aframework for repurposing passive haptics, called haptic retargeting, that leverages the dominance of vision when oursenses conflict. In essence, our approach repurposes a single
physical prop to provide a passive haptic sensation for a variety of virtual world objects thus increasing the user’s senseof presence and improving the overall quality of the experience (Figure 1).
We do this by dynamically aligning physical and virtual objects as the person is interacting in the environment. Specifically, we investigate three approaches: a) world manipulation, where the virtual world surrounding the person is
moved to better align with a passive haptic prop, b) body manipulation, where the virtual representation of the person’s
body is warped to meet up with the passive haptic prop, andc) a hybrid technique which combines both world and bodymanipulation. This work improves upon redirected touching[25] by adding dynamic detection and remapping within thesame experience for the whole hand instead of one finger,
and also introducing a hybrid continuum, allowing retargeting in limited-movement situations. Our user study results
confirmed that our haptic retargeting techniques improved
the sense of presence when compared to typical wand-based
3D control of virtual objects.
Specific contributions of our work include:• The concept and detailed analysis of haptic retargeting asa method for repurposing passive haptics from the same
physical object over multiple virtual objects.
• The implementation and analysis of 3 specific haptic retargeting approaches: world, body and hybrid warping.
• The results of a user study evaluation of our three haptic
retargeting approaches in comparison with typical wandbased virtual manipulation.
RELATED WORK

There are several areas of related work that are relevant toour current design: haptic redirection, dominance of visual
perception, and repurposing physical props.
Haptic Redirection

Perhaps the most closely related work to our own is redirected touching [25], which addresses the inflexibility ofpassive haptic displays by warping the virtual space, introducing a discrepancy between a person’s real and virtual
hand motion. As a result, a physical display can be mapped
to a virtual display rotated about the vertical axis by up to 18
degrees, without being noticed. In contrast to our approach,this work explored world warping along a single dimension(rotation along a single vertical axis) and was limited to a
one-finger touch interaction without solving the problem ofrepresenting a full body under a variety of warped space conditions.
The inflexibility of passive haptics can be overcome with theaid of robotics using active haptics. In the case of Robotic
Shape Displays [28]: when a user reaches for a virtual object,a robotic arm places a real object correctly in front of theuser’s hand. One such robot has a Shape Approximation Device as its end-effector, which has several corners withcurved and flat edges to approximate different shapes [35].
While impressive, these haptic displays are expensive, require sophisticated control mechanisms, and miscalculations
and latency could be dangerous to users.
Studies have shown that using haptics can lead to significantly increased presence and spatial knowledge training
transfer [17]. Though haptics are traditionally implemented
using a one-to-one mapping between the real and virtualworld, in redirected touching [25] spatial warping is used togenerate mappings between the real and virtual space, suchthat a single real object can provide haptic feedback for virtual objects of various shapes. This results in discrepancies
between a person’s real and virtual hand motions so that theyreach the real and virtual objects simultaneously. Studies
have shown that people can adapt to spatial warping [24] andperform tasks as effectively under non-warped conditions[23].
In the context of walking, a single physical prop can be used
to provide haptics for multiple virtual objects [22]. This isachieved by using redirected walking [30], a technique that
injects additional translations and rotations to the user’s head
movements, causing users to walk on a physical path that isdifferent from the perceived virtual path. By having the user
walk back to the same physical object when moving fromone virtual target to the next, the object provides haptic feedback for multiple virtual targets. Though for an imperceptible execution of this method, large tracking spaces are required [34].
Dominance of Visual Perception

Some of these haptic redirection techniques are based on the
key idea that when senses conflict, vision often dominates. A
person moving their hand along a straight surface while
wearing distorting glasses feels the straight surface as curved[10]. In Rock and Victor’s study [31], participants holding
an object through a cloth while viewing the same object
through a distorting lens believed the object was most similar
to the distorted visual image, rather than the shape that theyfelt. Lécuyer et al. [26] had participants push with theirthumb on a piston mounted on a passive isometric input device. Simultaneously, participants were visually shown a virtual spring that compressed as force was applied to the piston. Even though the piston did not physically move, perception of spring stiffness was influenced by the virtual spring.
Vision is also found to dominate proprioception [5]. Thisfinding is exploited in a technique to allow visually correct
hand-object collisions in the virtual world when there is no

collision in the real world (since there is no real object).
Upon “collision”, the virtual and real hand positions can differ. Burns’ technique exploits visual dominance over proprioception to bring the two hands back to the same position
without the user noticing they were ever apart. Visual dominance is not always complete; during sensory conflict, sensory signals are weighted by their reliability [13]. Whenmixed-reality users are presented with real haptic and virtual
cube-shaped objects with discrepant edge curvatures, they
perceive the curvature to be intermediate [20]. Using a video
see-through system, users’ hand displacements were manipulated to perceive visual angles that were incongruent withphysical edges up to 30 degrees. This phenomenon has also
been leveraged to enable fine-controlled selection [16] andextend users’ reach and selection abilities [1,6].
Vision can also dominate vestibular cues [21], which allows
head rotations and translations to be scaled up or down inredirected walking [30]. This discrepancy causes a decoupling between the user’s path in the real and virtual world.
Spatial warping not only affects hand motions, but also how
the user’s body is represented (virtual body). Experimental
findings from body ownership illusions [33] show that under
specific multisensory conditions, we can experience artificial
body parts or fake bodies as our own body parts or body, respectively [19]. For instance, in one study, when the entire
scene including the body was rotated by 15 degrees upwards,
perceived ownership of the virtual body was only slightly diminished [4]. Scaling the body proportions has also been investigated by creating a long arm illusion, extending thelength of the user’s virtual hand by a factor of 3 while stillsustaining high ownership levels [18].
Repurposing Physical Props

While reusing physical props has seen little attention in virtual reality environments, other research has extensively explored the use of physical props to provide haptic feedback
in other domains [32]. Specifically, Hinckley et al. [15] introduced the notion of using physical models to control 3D
virtual models on a separate screen, which led to many others
extending this work to provide 3D interaction of objects in
virtual environments in general [5,8,6,29], though the problem of providing a one-to-one mapping of physical controls
in one’s immediate surroundings is not considered.
Corston et al. [7] extend this idea by providing the ability torecognize these physical props through recognition via a Kinect sensor, however they too do not superimpose this control in a virtual reality environment so that the physical controls and digital artifacts coincide. In an augmented reality
environment, Henderson and Feiner [14] provide opportunistic controls, which do connect the digital display environment to physical surfaces which are repurposed to provide a
haptic response to specific widget-like digital controls, suchas sliders and buttons. Our work builds upon these ideas byextending the notion of mapping digital objects in a virtual
reality scene to physical props in a person’s surroundings toprovide a haptic response, but we focus on providing a seamless transition between these spaces, so that a person wearing
the head-mounted display is largely unaware that these props
are being repurposed.
HAPTIC RETARGETING

Our vision of haptic retargeting is to create a design space
where we may reuse the physical objects in a person’s vicinity to provide a sense of touch when reaching, grabbing, lifting, or otherwise manipulating artifacts in a virtual reality
scene, but to not be limited by what specific objects arearound and whether they are located in the “right” place.
We define the term haptic retargeting to encompass the general notion of reusing the passive haptics of the same real
physical objects across multiple virtual objects. This can begeneralized to finding a mapping between the virtual and real
coordinate systems. Note that the mapping could be more
complex than a simple affine transformation.
In addition to the core mapping requirement, we stipulatethat good haptic retargeting solutions will also accomplish
the following two requirements:••

The mapping is dynamic; i.e., the solution should support interactive manipulations and not require carefulplacement of objects or pre-computed solutions;
The mapping is minimally noticeable (or unnoticeable)
and therefore minimally disturbing to the person.

Implementation Environment

To investigate the capabilities of haptic retargeting, we havecreated a virtual Minecraft world (Figure 1), where a person
seated at a desk, wearing a VR headset, constructs complex
virtual shapes out of virtual cubes by lifting and placing themon top of one another. We focus on a specific alignment
problem that involves manipulating cubes and constructing
models out of them, similar to the basic premise of the Minecraft game (http://www.minecraft.net).
We used the Oculus Rift DK2 headset, tracked by the Oculus
camera in combination with an overhead Microsoft Kinect
v2.0 camera. The Kinect camera is used to capture the geometry and the appearance of the person’s body, track the position of the person’s hand, as well as track the passive haptic
proxy object on the tabletop (Figure 2). Depth and color images from the Kinect are used to render realistic 3D views of
the user’s own body.
While there are many possible VR configurations, we had
the user seated at the desk for three reasons: a) this arrangement provides the person with an interactive surface (desk)
on top of which the manipulations can be easily performed,b) the person’s overall movements are restricted in space,thus making our hand/object tracking easier, and c) we expect many people using VR technology for work or play inFigure 2. Top-down view of our setup with Kinect (green),
Oculus HMD (blue) and Oculus DK2 Camera (red).

longer durations would prefer to be seated rather than walking around a room. In addition, many leading VR products(e.g., Oculus Rift, Sony Playstation VR) appear to be designed for the seated configuration.
Haptic Retargeting Mappings

We explore three mapping techniques to accomplish haptic
retargeting:•••

Body Warping – Manipulating the virtual representation of the person’s body (e.g., arms) such that, at the
point of contact with the real object, the body part also
appears to be at the virtual object;World Warping – Manipulating the virtual world’s coordinate system to align virtual and physical objects;
Hybrid Warping – A dynamic combination of Body
and World Warping.

We now detail each of these methods.
BODY WARPINGAs an illustrative example, consider a simple arrangementwith a real cube ! positioned on the desk in front of the user,
and the virtual cube !′ shifted slightly to the right. In thiscase, as the user reaches for the cube, to ensure that the virtual hand meets the real cube, a translation to the right must
be applied. A straightforward way of achieving this is to shift
the entire rendering of the body to the right, effectively translating the user's hands and arms to the right.
Incremental vs. Instantaneous Warp

Applying an overall shift, the user’s virtual representation of
their hands and arms may be noticeable and possibly disruptive. To avoid this, we apply the warp incrementally to respect the hands’ position at the beginning of the warp, yetstill align the cubes once the hand reaches the physical cube.
To define an incremental warp, we first measure the position#$ of the user’s hand (the hand used to grab the object) when
Body warping is activated and define it as the warping origin%& . The warping end %' is then set to hand position #( atthe time of Body Warping activation.
The warping ratio ) is then:) = min 0, max 1,%' − %& . #$ − %&%' − %& 5Figure 3. Body-friendly adjustment - Original approach (a)causes hand to drift away from expected body position, butwith a body-friendly warp (b), alignment can be achieved
without drastically separating hand from body.

The warping ratio quantifies the hand’s progression towards! and determines the amount of warp applied to the hand’s
position. By applying the warp incrementally, when theuser’s hand reaches the physical cube, the representation of
the hand will meet the virtual representation of the cube; andas the user’s hand is retracted, the warp is “undone”.
Body-Friendly Adjustment

Consider the case where the user reaches with their righthand and body warping effects a translation to the right. The
greater the translation, the more separated the arm appears
from the body. Similarly, a large translation to the left willcause the arm to appear very close to the body. Large mismatches between the visual and proprioceptive cues can be
disturbing, particularly when the arm appears to be obviously
detached from the user’s torso, or protrudes unnaturally fromthe torso. Such mismatches can be disruptive and distracting
to the user.
Our solution employs a combination of translation and rotation about the user’s body position (Figure 3). This method
does not guarantee that hand rotation matches that of the object (except in the case of a sphere). One way to resolve thisdiscrepancy would be to rotate the user’s rendering aroundthe wrist, to match the orientation of the object. Such nonaffine transforms were not investigated in this study. In an
informal proof-of-concept study, none of the 8 users that experienced the body-friendly-warp reported the discrepancy
in orientation, suggesting that in some cases convincing theuser that the physical and virtual cubes are aligned may notalways require perfect alignment.
Target Prediction

A successful remapping needs to be timely in addition to accurate. This requires determining the user’s intent early on toensure alignment when the hand reaches the virtual target.
Adopting a heuristic similar to drag-and-pop [3], we compare the hand’s velocity with the direct path to candidate targets and thus predict the next virtual target. A remapping isthen calculated and incremental warping is begun.

Figure 5. Arrangement of three cubes before the demonstration task begins (left) and with two cubes removed once the
HMD is put on (right). The user handles only a single physical cube.

Figure 4. 1 for 3 Illusion - User touches 3 different virtual
cubes while in reality the same cube is touched each time.
1 for 3 Illusion

We developed a demonstration task that involves reaching totouch one of three static virtual cubes aligned horizontally on
the desk before the user (Figure 4). At each phase, the user is
instructed to tap a button placed in front of them on the desk.
After the button is pressed, an instruction appears directing
them to reach for and touch one of the cubes. Before thedemo, they would see three cubes placed on the desk, leadingthem to believe that the three virtual cubes corresponded tothe same three physical cubes (Figure 5). Once the user putson the HMD, unbeknownst to them, the left and right cubes
were removed from the desk, leaving only the center cube on
the table. In each phase, pressing the button initiates a newtrial, mapping the single physical cube to one of the threevirtual cubes using Body Warping.
Users report that as they performed the task, they were uncertain what the purpose of the demonstration was and found
it repetitive; but once they were instructed to take off the
HMD, they realized what had happened. Observing the users’ behavior, we noticed that as they reached to grab a cube,
the virtual hand would start warping and they would—perhaps subconsciously—correct the hand trajectory, effectively countering the warp to meet the real cube.
Cube Stacking with Body Warping

In the current proposed implementation of Body Warping,
the warping factor is at its maximum when the user’s hand isin contact with the cube and no further warping is applied
while the user moves the cube. Though Body Warping was
originally intended to align the user’s virtual hand with thereal object, it can be extended to accomplish other illusions.
For example, we explored stacking virtual cubes on a table.
The task is to grab a virtual cube resting on the table, andplace it on top of another cube, also sitting on the table. Under normal circumstances, if the user were to perform thistask with the tracked cube, when the user brings the cube
above the bottom virtual cube, not only would there be no
haptic feedback indicating the cube’s placement on the lower
cube, the cube would fall to the table once released. To
properly simulate the haptics of stacking the cube, the idea isto add an upward translation to the body warp, so that when
the user attempts to place the upper cube, the user sees thevirtual cube resting on the lower cube. Meanwhile, haptic
feedback indicating that the upper cube is stacked on thelower cube is provided by the table’s surface. This creates
the illusion that the cubes are being stacked.
WORLD WARPING

World warping involves manipulating the virtual world toalign virtual objects with the real. In the cube alignment example, where physical cube ! is placed on the table, and thevirtual cube !′ is positioned to the right of !, alignment using
World Warping involves translating the entire virtual world
to the left, so that !′ meets !. Users would likely find thischange jarring if applied instantaneously. However, from theredirected walking literature we know that a user’s head motions can be scaled by certain factors without user noticing.
For instance, if the user performs a 90 degree rotation to theright, the virtual world can also rotate with/against the user’s
head rotation, resulting in a rightward/leftward 10 degree rotation about the user’s head position. From previous studies[34], when walking, translations can be scaled up by 26% orscaled down by 14% without being noticed. Similarly, rotations can be imperceptibly scaled up by 49% and down by
20% (note that the scaling factors are not symmetric and may
be different if seated).
Implementation

In a redirected walking implementation [30], translations androtations in the virtual world are injected at each simulation
frame. Our implementation follows a similar approach. We
calculate the user’s instantaneous change in position and orientation, and scale the desired translation and rotation proportionally. Such translations and rotations are applied continually, frame to frame, until alignment is achieved.
In a seated virtual reality experience, head rotations are morecommon than head translations. In our implementation ofworld warping, we exclusively employ rotations. This approach can be sufficient to accomplish large displacements
especially for farther targets.
Towards Ensuring Alignment

When a large amount of head rotation is expected, as when
the user faces away from the virtual target, we can assume aminimal amount of head rotation to face the target. Given theexisting rotation-scaling factors, the user’s head rotations
may not be sufficient to guarantee alignment by the time theFigure 7. Hybrid Warping - The real cube first aligned withthe green cube (left) moves towards the red cube with World
Warping while the user looks around (middle) and the alignment is completed with Body Warping as they reach for it(right).
Figure 6. Ring of Cubes Illusion - World warping progressively
maps a single physical cube to different virtual cubes arranged
in an arc around the button, and within the user’s reach.

user faces the target. To ensure alignment, we developed amechanism to calculate the required scale factor for alignment, which may result in applying a potentially higher scale
factor to ensure alignment. In this case, ensuring alignment
can come at the cost of being noticed by the user, and evenpossibly inducing additional simulator sickness. Furthermore, rotations were injected about the position of the button
as opposed to the exact position of the head to guarantee perfect alignment.
The Ring of Cubes Illusion

To demonstrate the functionality of world warping, we created a demonstration similar to the 1 for 3 Illusion. In thistask, the user is instructed to look to their right to view a billboard to trigger the next instruction (Figure 6). The user sees
an arrangement of cubes forming a quarter of a circle and ineach phase, one of the cubes is deactivated and the adjacentcube is activated. As before, the user is instructed to touch
the active cube in each trial.
The cubes are arranged so that mapping one cube to the othercan be accomplished by rotating the world around the button
placed in front of the user. Each time the user looks to theboard and looks back, the required scale factor is calculated
and applied. The rotations are applied about the position of
the button in the scene since the button is always the starting
location of the manipulating hand. Since the button itself wascylindrical, its own rotation during the warp was not easily
detectable.
This Ring of Cubes Illusion demonstrates how World Warping manipulations can be compounded and extended as longas there are sufficient head motions. This contrasts with
Body Warping, where manipulations are constrained byphysical limitations of the body.
HYBRID WARPING

Both Body Warping and World Warping can produce a
noticeable distortion of the user’s sense of space if themagnitude of the warp is large. To minimize artifacts of eachapproach and thereby improve the user’s overall sense ofpresence, we propose a Hybrid Warping technique which

distributes the alignment task between both Body Warping
and World Warping techniques simultaneously.
Implementation

In this hybrid remapping, the user’s head translation and rotation enables our World Warping technique to inject somesmall amounts of translations and rotations that assist in thealignment. The algorithm also simultaneously employs Body
Warping to aid in task completion: the required warp is calculated based on the current hand position and the difference
between positions of the target object and its virtual representation. Scaling factors are set so that users are typicallyunable to detect either of the warping.
In essence, Hybrid Warping works by World Warping improving the alignment as much as it can, while the remaining
warp is accommodated by Body Warping. In theory, if
enough head motions are observed, the alignment may becompleted entirely by World Warping, and no manipulation
will be applied by Body Warping.
The Hybrid Warping approach combines Body and World
Warping such that head motions activate World Warping,and hand motions activate Body Warping. In typical use, Hybrid Warping alternates regularly between World and Body
Warping as head and body motions occur. Note that in Figure
7, since head motions occur first as the user looks at the target, World Warping is applied before hand motions enable
Body Warping, but the algorithm does not require this order.
EVALUATION

We designed an experiment to compare Body Warping,
World Warping, and Hybrid Warping. We used a variant ofHybrid Warping which combines Body and World Warpingsuch that each is responsible for exactly half of the alignment
task. These three mechanisms were compared to a control
condition where the user interacts with a wand (no haptic
feedback). As target prediction was only possible in Body
and Hybrid Warping techniques, we removed this functionality in all conditions in order to create a comparable task andinvestigate the question of which technique most effectively
creates a haptic illusion.
Participants

Twenty participants (17 male, 3 female, aged 23-52, Mdn=
29) took part in our study and were compensated with $10
gift cards to a local restaurant. Participants were screened us-

ing the Titmus Fly Stereotest to ensure all users could identify disparities as low as 100 seconds of an arc. All participants self-reported as being right-handed.
Apparatus

We used a Kinect 2.0 mounted on the ceiling facing downward for rendering and tracking the participant’s body. Hand
tracking was assisted by requiring participants to wear a Velcro-fastened 1cm-wide strap with a retro-reflective marker
positioned on the back of the hand. A 5cm-sided cube wastracked by a strip of retro-reflective tape placed on the top,
providing three degrees of freedom (3DOF) of positional
tracking and 1DOF of rotation. The Kinect also tracked a“Virtual Wand” comprised of a wireless presenter with fourrods attached, each with a retro-reflective marker on one end.
This provided 3DOF positional tracking and 1DOF of rotation (similar to the cube) (Figure 8). A Griffin PowerMate
knob was also used as a virtual button. Participants wore an
Oculus Rift DK 2 head-mounted display (HMD) with tracking provided by the Oculus Camera (in addition to IMUs).
The experience was developed on the Unity3D game engine.
Procedure

After screening, participants were told about the series oftasks they were to perform and questionnaires to answer.
This was followed by the Miles Ocular Dominance test [29].
They were then asked to complete a questionnaire for demographic information (age, gender, handedness, ocular dominance, eyewear usage). Next was Witmer and Singer’s Immersive Tendencies Questionnaire [37]. Finally, they answered three Visualization of Rotations and three Visualization of Views questions from the Purdue Spatial Visualization
Test [12].
After gathering this preliminary information, participants
were asked to perform a task once for each of the four conditions. At the end of each condition, participants completedWitmer and Singer’s Presence Questionnaire [37] with two
added questions: “How stable did the world seem?” and“How easy was it to reach and grab objects?” Participants’hand positions were also logged for analysis.
Conditions

Participants completed the following four conditions:• World: Participants interacted with a tracked cube using
our World Warping technique.
• Hybrid: Participants interacted with a tracked cube using
our Hybrid Warping technique.
We used a within-participants design, and conditions werecounterbalanced using a Latin Square.
Task

The primary task performed in each of the conditions was a
block-stacking task, where participants were asked to placea cube (blue) in a target location (indicated in red). The basic
progression of the task was (a) tap a button to begin, (b) lookup and to the left to receive instructions, (c) look back at thetable to see the target cube (blue) and destination (red), (d)reach and grab the blue cube, (e) place it “inside” the destination marker, and (f) return to the button to end the task. In
total, each participant completed 9 repetitions, in the form of
three horizontally stacked rows, for each condition. They
used only their right (dominant) hand.
To begin each condition, the participant put on the HMD tosee a virtual world with a desk (aligned with a physical desk)with a button (aligned with the knob) near them on the desk,
as well as three virtual brick walls. The wall in front of theparticipant contained a billboard with instructions. A second
billboard was placed up and to the left (Figure 9), for step(b), and was intentionally placed out of the HMD’s forwardfacing view to be able to invoke world-centric and hybrid
haptic responses, and to maintain consistency in the othertwo conditions. Participants were asked to look at this board
with the instruction to inspect the next stacking operation,
which was described graphically in 2D.
Participants were given a chance to practice examining(moving and rotating) the blue cube with their hands at the
beginning of each condition to familiarize themselves withthe technique before each set of trials for as long as theyneeded. Participants practiced for one minute on average.
A trial was considered complete when the blue cube wasplaced such that it had 75% overlap with the destination (red)“ghost” cube. At this point the blue cube turned yellow. Once

• Wand: Participants used a virtual wand with grab and release functionality to interact with a virtual cube (no haptic feedback).
• Body: Participants interacted with a tracked cube using
our Body Warping technique.

Figure 9. The hand, cube, and virtual wand are tracked withretroreflective markers.

Figure 8. Participants first studied arrangement instructions(left) and then grabbed the blue cube and placed it in the target position indicated as a red cube (right).

the button had been pressed to end the trial, the yellow cube
would “solidify” and change texture, and the virtual blue
cube would disappear. The blue cube then would reappear instep (c) of the next trial and, despite having just been placed
elsewhere in a previous trial, the haptic remapping technique
was used to match it to the same physical cube. The respawning of the blue cube was done by taking the actual positionof the cube and rotating it by 16 degrees around the positionof the button, alternating between clockwise and counterclockwise. In Wand condition, this did not require haptic
feedback, but in the other three cases, the haptic remapping
was solved by the appropriate warping technique. Hybrid
condition split the warping evenly between Body Warping
and World Warping (8 degrees each). Note that with Body,
when the hand is retracted, the warp is undone, but in thecase of World, when a new remapping was required, the previous warp was first undone, requiring a total of 32 degreesof rotation (and 16 degrees in Hybrid). We required the participant to look at the upper left instruction board to ensurethat they performed some head rotations. The scaling factors
for World were dynamically adjusted to guarantee alignmentby the time the user looked back at the cube.
To provide the illusion of stacking, the table itself was used
to provide a haptic response when placing the cube, and so
the base of the target (i.e., the top of the previous layer) was
positioned to align with the physical table surface. The Body
Warping technique could handle this directly as the cube waslifted, however World Warping could not. To address for
this, the blue cube would appear in step (c) on a virtual pedestal at the same height as the destination. The Hybrid technique used Body Warping manipulation for 50% of heightadjustment, and so started on a pedestal half-way between
the table and the bottom layer. Wand did not require any adjustment, as no haptic response was provided. Only 1 of the
20 participants reported noticing that the table had been
“lowered.”
RESULTSThe following results are based on the data collected fromour user study via questionnaires and hand tracking data.
Presence Analysis

We measured the sense of presence and the perceived effectiveness of the haptic response using Witmer and Singer’s

2.0

wand

body

world

hybrid0.0
presence* realism* interface* haptic*
Figure 10. Results of the Presence Questionnaire. Error
bars represented standard error (SE).

Presence (overall):Realism
Control
Quality of Interface
Ability to Examine
Performance
Haptic

F3,27
3.1
2.9
1.9
3.2
1.2
1.3
13.9

P
.03*
.04*
.14
.03*
.32
.29
< .001**

Table 1. Results of RM-ANOVA for Presence andits six subscales. * = p < .05 or ** = p < .001.

21-question Presence Questionnaire [37], including questions relating to the realism, control, interface quality, ability
to examine, performance, and haptic subscales. Participant
responses to these questions were reliable (Kronbach’s α =.972). To analyze presence and these subscales, we used aone-way RM-ANOVA on the remapping technique factor (4levels), a common statistical method for robustly analyzing
Likert-scale responses [27]. Significant effects were found
for the overall presence scale, as well as the realism, qualityof interface, and haptic subscales (Table 1). Each question
was asked on a 7-point scale, which has been normalized
to -3 (lowest), 0 (neutral), 3 (highest).
Overall Presence

Post-hoc analysis revealed that participants rated Hybrid significantly higher than the Wand baseline (p = .01), and marginally higher than World (p = .06) on our measure of presence. All other pairwise differences were not significant (p >.05). Figure 10 shows that Hybrid received the highest average score for all 21 questions on a 7-point scale (M = 1.8, SE= 0.2).
Realism

Post-hoc analysis revealed that participants rated Hybrid significantly higher than Body (p = .02), and marginally higher
than World (p = .054) on the realism subscale. All other pairwise differences were not significant (p > .05). This subscale
included seven questions, such as “how natural was themechanism which controlled movement through the environment?” and “how compelling was your sense of objects moving through space?”
Quality of Interface

Post-hoc analysis revealed that participants rated Hybrid significantly higher than the Wand (p = .01) on the quality ofinterface subscale, and all other pairwise differences were
not significant (p > .05). This subscale included three questions: “how much delay did you experience between your actions and expected outcomes?”, “how much did the visualdisplay quality interfere or distract you from performing assigned tasks or required activities?”, and “how much did thecontrol devices interfere with the performance of assigned
tasks or with other activities?”
Perceived Effectiveness of Haptic Response

The Presence Questionnaire includes two optional questions
in a haptic (sense of touch) subscale: “how well could you
actively survey or search the virtual environment using

Figure 11. Horizontal paths (in meters) of our 20 participants when reaching for the target cube. Note: while thepaths differ, the target and user locations are similar acrossconditions.

touch?” and “how well could you move or manipulate objects in the virtual environment?” Post-hoc analysis revealed
that participants rated all warping techniques higher than
Wand (World: p = .001; Body: p = .03; Hybrid: p < .001).

Figure 12. Examples of paths shown that exhibit a last moment correction (a kink) in the movement (left) vs. no correction (right).

••

Path Analysis

To further understand the participant’s experience, we analyzed the paths of the participants’ hands when reaching forthe passive haptic target cube. We logged hand position fromthe moment the participant pressed on the button, until theyacquired the target cube. We excluded paths that had any
missing data due to noise in hand tracking (~10%) and wedropped the Wand condition from our analysis, since wewere primarily interested in the effect that our various warping conditions had on the paths.
We assume that if the user has a good agreement between
their visually-perceived virtual world and their physical reality, then their reaching paths should be direct, straight, andwithout any corrective movements.
When plotting all paths taken by 20 participants over 3 conditions (Figure 11), it is clear that the paths are substantiallydifferent across conditions. The paths are the straightest in
the World condition, which we expect, since there is a perfectmatch between the physical and the virtual world. The paths
are progressively more curved in Hybrid and Body conditions, likely indicating that the users adapted their movements more to match their expectations.
We also noticed that some of the paths had an explicit “kink”
near the end of the movement (Figure 12). These ‘last moment corrections’ indicate that the participant missed the target and adjusted their movement to find the cube. Two independent scorers manually inspected the paths to count these
correction events. We observed very high agreement between scores (Krippendorff’s α= .904).
Counts of corrections were statistically significantly different across conditions (Friedman Test χ2= 25.9, p < .001).
Post-hoc Wilcoxon Signed Ranks tests revealed Hybrid andWorld both did very well with only 7.5% and 8.75% of paths
exhibiting last moment correction (Z = 0.6, p = .53, n.s.).
Body had 31.25% of such corrections which was significantly different than both World (Z = 3.7, p < .001) and Hybrid (Z = 3.5, p = .001).
DISCUSSION

The main findings of our evaluation are as follows:••

Participants reported the highest sense of presence whenusing Hybrid, and this difference was significant compared to Wand and World techniques.
Hybrid also had advantages in terms of realism and quality of interface, and all three of the retargeting techniques provided a better sense of haptics than the Wandcondition.
Our path analysis revealed that World had the most direct paths to targets.
Body led to more corrections near the target object thanboth World and Hybrid, suggesting that combining techniques can help reduce the need for path correction.

Given these findings, we recommend designers use the Hybrid Warping approach when applying haptic retargeting, inorder to achieve an appropriate balance between providing agreater sense of presence and avoiding the need for path correction, thus enhancing the illusion by making these path corrections less noticeable. Participants generally reacted to theexperience of using haptic retargeting with enthusiasm andsurprise, making comments such as “knowing that the cube
was just placed there and it appeared somewhere else freakedme out”, “this is surreal; it’s definitely messing with me”,
and “my mind is blown”.
Based on these results, as well as our experience with designing and implementing these retargeting techniques, we canalso provide additional advice.
Reduce the Need for Virtual World Adjustment

Our path analysis confirms that when the participant’s actions match their expectations, they are able to perform direct
movements without errors. While the paths are slightly morecurved in the Hybrid condition, our analysis confirms thatparticipants did not perform significantly more corrections
than in the World condition. This is very encouraging, sinceWorld Warping requires a significant amount of manipulation of the virtual world around the user. This manipulationmust be masked during head movement or it will be noticed
by the user. Hybrid Warping is a good compromise: theamount of world movement can be kept to a minimum, while
few mistakes are made when reaching for the target. In addition, as our presence analysis shows, this combination of low
world and low body movement achieved the highest presence score of all conditions.

Make Use of Visual DominanceIt appears that users are proficient at accurately performing
the required set of muscular actions to initially reach for an
observed object, though they often rely on visual feedback tomake fine adjustments towards the end of the movement.
Therefore, it may seem natural to be slightly off when initially reaching for an object. The tendency to tolerate initialerrors in movement, corrected in the last phase by visualfeedback, argues for the use of Body Warping to retarget
haptics. But the dominance of visuals over proprioceptive
cues probably has a limit, and beyond a certain point, the user
will question the magnitude of their error.
Encourage Slow Movements When Possible

The more slowly the hand moves, the more likely it is for theuser to gradually correct the trajectory with visual feedback.
In the these cases, the user is less likely to notice the smaller
errors over the length of the reach. This suggests the possibility of extending the limits of Body Warping when users
are more likely move their hands more slowly. In our experience, the applied warp is more noticeable when people perform quick ballistic motions to reach the target, and as a result, they see their hand miss the target. We hypothesize thatthe visual angular difference between the physical and visualtargets is the key element in limiting Body Warping efficacy.
In the Body condition this difference was higher than the Hybrid condition, increasing the odds of the user noticing thewarp and decreasing performance.
Avoid Enforcing Head Motions by using Body or HybridWorld Warping exploits the user’s head translations and rotations, and therefore alignment cannot be guaranteed if headmotions are not required. Even if some type of head motions
are performed, they must be relevant to the necessary manipulation. For example, the motion of the user’s head left andright will not help alignment when a forward translation isrequired. Furthermore, the magnitude of these motions mustbe sufficient to achieve the desired alignment. For example,
if the alignment requires a 5cm translation and the maximum
permitted translation-scaling factor is 25%, then the user
must move at least 20cm forward. Magnifying the scale factors to guarantee alignment increases the likelihood of theuser noticing the injected motions. Hybrid Warping may beless noticeable than World Warping in part because it works
to minimize the scaling factor over time.
Limitations & Future Work

Tasks presented in our study have been deterministic in nature: in each case the virtual target is designated by the system rather than determined by the user. We previously described a method for path prediction that can be used in boththe Body and Hybrid Warping approaches. Path prediction ismore difficult with the World Warping technique, as it performed when the user is likely already looking at the object,
reducing the chances that there would be sufficient head
movements to effect alignment. Redirected walking [30] hassimilar a similar limitation. We thus removed path prediction
from our study. Even with this omission, the Hybrid Warping

technique was shown to provide a greater sense of presence,
and we thus recommend using it.
In our design, each warp has been followed by an unwarp.
This was enforced in the Body condition by retracting thehand to press a button, and in World by requiring the user tolook back at a sign. Our remapping mechanism cannot support releasing a virtual object and immediately grabbing another object using the same physical object, though by using
multiple physical cubes (e.g., two), this may be possible.
In our experiments, a single cube is mapped to multiple virtual cubes of identical dimensions. While using a physicalcube may not be effective for other virtual shapes, our focus
has been on the illusion of a prop’s location, rather than its
other material properties. Nonetheless, we can still effectively “fool” users into believing a real object is at the position of the virtual object, despite shape mismatch. For example, if a VR experience requires object movement or building, it may not be essential for material properties to match.
Rather, it may be more important to simulate the physicalmovement of a prop. Moreover, recent work [2] suggests thatsome geometric substitutions could further improve this experience. Exploration of what constitutes appropriate substitutions, and their effectiveness in whole-hand contexts, areareas of potentially fruitful future work.
In the current implementation, we have only considered simple translations and rotations of the body and world representations in the virtual environment. We would like to explore more complex, perhaps non-linear mappings and test
whether they provide a more compelling illusion. We would
also like to explore incorporating our technique into a more
completely developed environment, such as a video game,and evaluate the experience of using our technique in combination with other design constraints. Lastly, our techniquescurrently map virtual objects to similarly shaped physicalones, and we would like to explore more complex shapes andsizes and larger discrepancies between the two.
CONCLUSION

In this paper, we present a system for providing haptic feedback in a virtual reality environment by warping the virtual
space to match the location of a physical prop in a person’s
immediate surroundings. We provide three alternative methods for achieving this warping: Body Warping and World
Warping, which can be combined into a Hybrid Warping
technique that uses both. These techniques allow a person toexperience passive haptics in a virtual environment that differs from the physical one in potentially dramatic ways, allowing the designer to consider more complex virtual scenes
and experiences, while still providing a sense of touch. We
also present the results of a user study which shows that people experience a greater sense of presence when these techniques are combined, and we provide valuable insights into
how to design virtual worlds that make use of haptic retargeting. Through haptic retargeting, designers of virtualworlds can consider a wider range of scenarios where passive
haptics can be used.

REFERENCE

1.

2.

Merwan Achibet, Adrien Girard, Anthony Talvas,
Maud Marchal, and Anatole Lecuyer. 2015. ElasticArm: Human-scale passive haptic feedback foraugmenting interaction and perception in virtualenvironments. Virtual Reality (VR), 2015 IEEE, 63–68.
Yuki Ban, Takashi Kajinami, Takuji Narumi,
Tomohiro Tanikawa, and Michitaka Hirose. 2012.
Modifying an identified angle of edged shapes using
pseudo-haptic effects. Haptics: Perception, Devices,
Mobility, and Communication. Springer Berlin
Heidelberg, 2012. 25-36.. http://doi.org/10.1007/978-3642-31401-8_3

3.

Patrick Baudisch, Edward Cutrell, Dan Robbins, et al.
2003. Drag-and-pop and drag-and-pick: Techniques foraccessing remote screen content on touch-and penoperated systems. Proceedings of INTERACT, 57–64.

4.

Kristopher J. Blom, Jorge Arroyo-Palacios, and Mel
Slater. 2014. The effects of rotating the self out of the
body in the full virtual body ownership illusion.
Perception 43, 4: 275–294.
http://doi.org/10.1068/p7618

5.

6.

7.

8.

9.

Eric Burns, Sharif Razzaque, Mary C. Whitton, andFrederick P. Brooks. 2007. MACBETH: The avatar
which I see before me and its movement toward myhand. Proceedings - IEEE Virtual Reality: 295–296.
http://doi.org/10.1109/VR.2007.352509
François Conti and Oussama Khatib. 2005. Spanninglarge workspaces using small haptic devices.
Eurohaptics Conference, 2005 and Symposium on
Haptic Interfaces for Virtual Environment andTeleoperator Systems, 2005. World Haptics 2005. First
Joint: 183–188. http://doi.org/10.1109/WHC.2005.118
Christian Corsten, Ignacio Avellino, Max Möllers, andJan Borchers. 2013. Instant User Interfaces:
Repurposing Everyday Objects as Input Devices.
Proceedings of the 2013 ACM international conferenceon Interactive tabletops and surfaces - ITS ’13, 71–80.
http://doi.org/10.1145/2512349.2512799
George W. Fitzmaurice, Hiroshi Ishii, and William
A.S. Buxton. 1995. Bricks: laying the foundations forgraspable user interfaces. Proceedings of the SIGCHI
conference on Human factors in computing systems:
442–449. http://doi.org/10.1145/223904.223964
Bernd Frohlich and John Plate. 2000. The cubic mouse:a new device for three-dimensional input. Conferenceon Human Factors in Computing Systems: Proceedingsof the SIGCHI conference on Human factors incomputing systems 1, 06: 526–531.
http://doi.org/10.1145/332040.332491

10. James J. Gibson. 1933. Adaptation, after-effect andcontrast in the perception of curved lines. Journal of
Experimental Psychology 16, 1: 1–31.
http://doi.org/10.1037/h0074626

11. John C. Goble, Ken Hinckley, Randy Pausch, John W.
Snell, and Neal P. Kassell. 1995. Two-handed spatial
interface tools for neurosurgical planning. Computer
28, 7: 20–26. http://doi.org/10.1109/2.391037
12. Roland Guay. 1977. Purdue Spatial Visualization TestVisualization of Views. Purdue Research Foundation,
West Lafayette, IN.
13. Hannah B. Helbig and Marc O. Ernst. 2007. Optimal
integration of shape information from vision and touch.
Experimental Brain Research 179, 4: 595–606.
http://doi.org/10.1007/s00221-006-0814-y
14. Steven J Henderson and Steven Feiner. 2008.
Opportunistic Controls: Leveraging Natural
Affordances as Tangible User Interfaces forAugmented Reality. Proceedings of the 2008 ACM
symposium on Virtual reality software and technology VRST ’08, 211–218.
http://doi.org/10.1145/1450579.1450625
15. Ken Hinckley, Randy Pausch, John C Goble, and Neal
F Kassell. 1994. Passive Real-world Interface Props for
Neurosurgical Visualization. Proceedings of the
SIGCHI conference on Human factors in computing
systems celebrating interdependence - CHI ’94, 452–
458. http://doi.org/10.1145/191666.191821
16. Ralph L. Hollis, Septimiu E. Salcudean, and A. Peter
Allan. 1991. A six-degree-of-freedom magneticallylevitated variable compliance fine-motion wrist:
Design, modeling, and control. IEEE Transactions on
Robotics and Automation 7, 3: 320–332.
http://doi.org/10.1109/70.88141
17. Brent E. Insko. 2001. Passive haptics significantly
enhances virtual environments. Computer: 100.
18. Konstantina Kilteni, Jean Marie Normand, Maria V.
Sanchez-Vives, and Mel Slater. 2012. Extending bodyspace in immersive virtual reality: A very long arm
illusion. PLoS ONE 7, 7.
http://doi.org/10.1371/journal.pone.0040867
19. Konstantina Kilteni. 2015. Over my fake body: bodyownership illusions for studying the multisensory basisof own-body perception. Frontiers in Human
Neuroscience 9, March.
http://doi.org/10.3389/fnhum.2015.00141
20. Itaru Kitahara, Morio Nakahara, and Yuichi Ohta.
2010. Sensory Properties in Fusion of Visual / Haptic
Stimuli Using Mixed Reality. In Advances in Haptics.
InTech. 565–583.
21. Roberta L. Klatzky, Jack M. Loomis, Andrew C. Beall,
Sarah S. Chance, and Reginald G. Golledge. 1998.
Spatial Updating of Self-Position and Orientation
During Real, Imagined, and Virtual Locomotion.
Psychological Science 9, 4: 293–298.
http://doi.org/10.1111/1467-9280.00058

22. Luv Kohli, Eric Burns, Dorian Miller, and Henry
Fuchs. 2005. Combining passive haptics withredirected walking. Proceedings of the 2005
International Conference on Augmented Teleexistence
57, 4: 253–254.
http://doi.org/10.1145/1152399.1152451
23. Luv Kohli, Mary C. Whitton, and Frederick P. Brooks.
2012. Redirected touching: The effect of warping space
on task performance. IEEE Symposium on 3D User
Interfaces 2012, 3DUI 2012 - Proceedings: 105–112.
http://doi.org/10.1109/3DUI.2012.6184193
24. Luv Kohli, Mary C. Whitton, and Frederick P. Brooks.
2013. Redirected Touching: Training and adaptation in
warped virtual spaces. IEEE Symposium on 3D User
Interface 2013, 3DUI 2013 - Proceedings: 79–86.
http://doi.org/10.1109/3DUI.2013.6550201
25. Luv Kohli. 2010. Redirected touching: Warping space
to remap passive haptics. 3DUI 2010 - IEEE
Symposium on 3D User Interfaces 2010, Proceedings:
129–130. http://doi.org/10.1109/3DUI.2010.5444703
26. Anatole Lécuyer, Sabine Coquillart, Aberrahmane
Kheddar, Paul Richard, and Phillipe Coiffet. 2000.
Pseudo-haptic feedback: can isometric input devices
simulate forcefeedback? Proceedings IEEE Virtual
Reality 2000 (Cat. No.00CB37048).
http://doi.org/10.1109/VR.2000.840369
27. Rensis Likert. 1932. A technique for the measurementof attitudes. Archives of Psychology 22 140: 55.
http://doi.org/2731047
28. William A. McNeely. 1993. Robotic graphics: a new
approach to force feedback for virtual reality.
Proceedings of IEEE Virtual Reality Annual
International Symposium.
http://doi.org/10.1109/VRAIS.1993.380761
29. Walter R Miles. 1930. Ocular dominance in humanadults. Journal of General Psychology 3: 412–430.

30. Sharif Razzaque, Zachariah Kohn, and Mary C
Whitton. 2001. Redirected Walking. Proceedings of
EUROGRAPHICS: 289–294.
31. Irvin Rock and Jack Victor. 1964. Vision and Touch:
An Experimentally Created Confilict Between The
Two Senses. Science (New York, N.Y.) 143: 594–596.
http://doi.org/10.1126/science.143.3606.594
32. Adalberto L Simeone, Eduardo Velloso, and Hans
Gellersen. 2015. Substitutional Reality. Proceedings ofthe 33rd Annual ACM Conference on Human Factors
in Computing Systems - CHI ’15, 3307–3316.
http://doi.org/10.1145/2702123.2702389
33. Bernhard Spanlang, Jean-Marie Normand, David
Borland, et al. 2014. How to Build an Embodiment
Lab: Achieving Body Representation Illusions in
Virtual Reality. Frontiers in Robotics and AI 1: 1–22.
http://doi.org/10.3389/frobt.2014.00009
34. Frank Steinicke, Gerd Bruder, Jason Jerald, Harald
Frenz, and Markus Lappe. 2010. Estimation of
Detection Thresholds for Redirected Walking
Thechniques. IEEE Transactions on Visualization andComputer Graphics 16, 1: 17–27.
http://doi.org/10.1109/TVCG.2009.62
35. Susumu Tachi, Taro Maeda, Ryokichi Hirata, andHiroshi Hoshino. 1994. A Construction Method ofVirtual Haptic Space. Proceedings of the 4thInternational Conference on Artificial Reality andTele-Existence (ICAT’94).
36. Colin Ware and Jeff Rose. 1999. Rotating virtual
objects with real handles. ACM Transactions on
Computer-Human Interaction 6, 162–180.
http://doi.org/10.1145/319091.319102
37. Bob G. Witmer and Michael J. Singer. 1998.
Measuring Presence in Virtual Environments: A
Presence Questionnaire. Presence: Teleoperators andVirtual Environments 7, 3: 225–240.
http://doi.org/10.1162/105474698565686