Haptic Feedback

CHI 2017, May 6–11, 2017, Denver, CO, USA

Sparse Haptic Proxy: Touch Feedback in
Virtual Environments Using a General Passive Prop
Lung-Pan Cheng2,1, Eyal Ofek1, Christian Holz1, Hrvoje Benko1, Andrew D. Wilson1
1
2
Microsoft Research
Hasso Plattner InstituteRedmond, WA, USA
Potsdam, Germany{eyalofek, cholz, benko, awilson}@microsoft.com
lung-pan.cheng@hpi.deFigure 1. (a) Our hemispherical prop is an example of a Sparse Haptic Proxy. It simulates both, (b) a room and (c) a cockpit scene
to provide physical touch feedback during interaction. (White lines on the prop added for visibility on the black background).
ABSTRACT

over better-matching surfaces of our Sparse Haptic Proxy
when receiving haptic feedback for single-finger touch input.

We propose a class of passive haptics that we call Sparse
Haptic Proxy: a set of geometric primitives that simulate
touch feedback in elaborate virtual reality scenes. Unlike
previous passive haptics that replicate the virtual environment in physical space, a Sparse Haptic Proxy simulates ascene’s detailed geometry by redirecting the user’s hand to amatching primitive of the proxy. To bridge the divergence ofthe scene from the proxy, we augment an existing Haptic Retargeting technique with an on-the-fly target remapping: We
predict users’ intentions during interaction in the virtual
space by analyzing their gaze and hand motions, and consequently redirect their hand to a matching part of the proxy.

We demonstrate our system with two virtual scenes: a flight
cockpit and a room quest game. While their scene geometries
differ substantially, both use the same sparse haptic proxy toprovide haptic feedback to the user during task completion.
Author Keywords

Virtual reality; passive haptics; perception; retargeting;
ACM Classification Keywords

H.5.1 [Information Interfaces and Presentation]: Multimedia
Information Systems-Artificial, Augmented, and Virtual Realities; H.5.2 [User Interfaces]: Haptic I/O.

We conducted three user studies on our haptic retargeting
technique and implemented a system from the three main results: 1) The maximum angle participants found acceptable
for retargeting their hand is 40°, rated 4.6 out of 5 on average.
2) Tracking participants’ eye gaze reliably predicts their
touch intentions (97.5%), even while simultaneously manipulating the user’s hand-eye coordination for retargeting.
3) Participants preferred minimized retargeting distances

INTRODUCTION

Since the conception of the ultimate display in 1965 [17], researchers have sought to better convey the physicality of virtual worlds and reality (VR) to enhance immersion using active and passive haptics. Active haptics dynamically matchthe location of virtual shapes through active components,
such as robotic arms [23,35,36] or human actors that adjust
props on-the-fly [7]. Passive haptics are statically-positioned
physical replica that match the virtual environment [14].

Permission to make digital or hard copies of all or part of this work for personalor classroom use is granted without fee provided that copies are not made or
distributed for profit or commercial advantage and that copies bear this noticeand the full citation on the first page. Copyrights for components of this work
owned by others than the author(s) must be honored. Abstracting with credit ispermitted. To copy otherwise, or republish, to post on servers or to redistributeto lists, requires prior specific permission and/or a fee. Request permissions from
Permissions@acm.org.
CHI 2017, May 06 - 11, 2017, Denver, CO, USA
Copyright is held by the owner/author(s). Publication rights licensed to ACM.
ACM 978-1-4503-4655-9/17/05…$15.00
DOI: http://dx.doi.org/10.1145/3025453.3025753

While passive haptics are easier to set up, their need for static
positions limits their capability to represent virtual environments with complex geometries, large dimensions, or dynamic objects. Researchers have thus examined reversing
this approach and fitting the virtual scene to the geometry of
existing physical objects [13,30].

3718

Haptic Feedback

CHI 2017, May 6–11, 2017, Denver, CO, USA

In this paper, we present Sparse Haptic Proxy (SHP), a classof passive haptics [14] that simulates touch feedback for avariety of different virtual environments. Unlike traditionalpassive haptics, which replicate virtual scenes with high fidelity, SHP’s geometry comprises a sparse set of primitives
that simulates virtual environments with geometries that mayvary largely in sizes and shapes.

virtual scene in any order while their physical hand is continuously redirected to touch the physical prop. In our cockpit scene, the user operates the spaceship by pushing buttons
and turning a dial. Though the cockpit geometry does notmatch the proxy shape, reaching for a button allows the user
to feel the physical feedback of a solid surface on the proxy.
A single prop provides touch feedback to two scenes that are
different in geometries, size, and appearance. Whenever auser is moving the virtual hand towards a virtual target, thephysical hand is redirected to touch a plane of the proxy.

We bridge the difference between a Sparse Haptic Proxy andthe simulated virtual geometries with Haptic Retargeting [1].
That is, we manipulate the user’s hand-eye coordination to
redirect their hand to a physical proxy while they are approaching a target in the virtual environment.

RELATED WORKThis paper builds upon the research fields of simulating
touch feedback in virtual reality, including active haptics,
passive haptics and haptic retargeting.

To preserve unscripted and natural properties of passive
haptics while using Haptic Retargeting, we contribute thefollowing necessary enhancements:

Active Haptics

1) We evaluate haptic retargeting in a user study to informthe limits of retargeting that users still rate as tolerable.
The further the virtual geometry can stray from a given
SHP while still being well represented, the larger therange of scenes that proxy can simulate.

Simulating touch feedback using active machinery has been
researched for decades since Goertz and Thompson in 1954
proposed a robotic arm for tele-operation that transmitted theforce feedback on the end effector to users [11]. Project
GROPE [6] in 1967 then used the robotic arm to provide
force feedback when users were operating the arm to touchand grasp virtual objects in virtual reality.

2) We predict the user’s touch intention to make use of Haptic Retargeting in unscripted scenarios. We found that a
combination of the user’s gaze fixation and hand motion
is an accurate predictor for touch targets and we evaluate
the accuracy of our prediction algorithm in a user study.

Exoskeletons have been used to directly apply the force feedback to users’ body instead of through a controlling interface[2]. Robotic Graphics [36] used a robotic arm to move aboard to where users were about to touch, which physically“rendered” the entire geometry of the virtual world. Iwata et
al. used shape displays [16] to render geometries and provide
touch feedback.

3) We investigate and evaluate three mappings between thevirtual object and a target location on the SHP, ranging
from a technique that minimizes hand travel to a technique that better matches virtual and physical geometries.

While able to represent a large variety of scenes, these machineries have high cost, large size and safety concerns limits
most of them to stay in professional labs. Thus, TurkDeck[7] implements this using a series of human actors that moveand deform the geometry using a set of primitives, such as
wall parts to fit the on-line geometry in the virtual world.
Passive Haptics

Hinckley et al. [14] pioneered the concept of using passive
props to as haptic feedback when controlling 3D virtual models followed by [5,8,10]. Low et al. project augmented reality experiences onto Styrofoam walls [22], allowing users toexperience different virtual worlds in the same physicalroom. Similarly, Mixed Reality for military operations in urban terrain [15] uses passive haptics to create a haptic senseof objects, terrain, and walls in the virtual world.

Figure 2. Our Sparse Haptic Proxy provides touch feedback
for touch and drag interactions on these virtual objects.
Example Scenarios for Sparse Haptic Proxies

To demonstrate our system, we constructed a SHP made ofplanar plates of different orientations (Figure 1a) and created
two substantially different virtual scenes: a room scene(Figure 1b) and a flight cockpit scene (Figure 1c). The samesparse haptic proxy provides touch feedback for both worlds.

FlatWorld integrated large props into a physical world; between experiences these props can be rearranged to matchthe next virtual world [26]. Similarly, the Personal Interaction Panel used a pen and a pad to manipulate objects in seethrough augmented reality [34]. Sheng et al. used a spongelike prop to create a range of interaction techniques [29].
Corston et al. [9] provide the ability to recognize and trackthese physical props using Kinect. However, they used a

As shown in Figure 2, the user in our virtual room scene turns
on the light by touching the switch on the lamp, opens thedrawer, grabs the key, and reveals a secret message by touching the picture frame. The user finally opens the safe by turning its dial. Note that the user can touch all features of the3719

Haptic Feedback

CHI 2017, May 6–11, 2017, Denver, CO, USA

fixed one-to-one mapping of the physical controls to the virtual world. All these approaches require the physical objects
that match virtual geometries to align prior to interaction.

During interaction, users should be free to decide when andwhat they want to touch and be provided with touch feedback
that matches their expectation based on their visual impression. The retargeted virtual hand movement should movesmoothly and comfortably.

Opportunistic controls for Augmented Reality [12] connect
the displayed environment to physical surfaces that are repurposed to provide haptic responses to specific widget-like
digital controls, such as sliders and buttons. Annexing reality[13] analyzes the environment and opportunistically assigns
objects as passive proxies, but a given physical environment
may not support all the application needs, and the virtual geometry has to deform to fit existing geometries. Substitutional reality [30] investigated the limits of mismatches between physical and virtual objects.

To achieve touch feedback, the system needs to map a suitable primitive of the proxy that will provide adequate feedback. To achieve comfortable hand motion, retargetingshould be limited to a tolerable range, thus limiting the range
in which such a primitive can be found. Natural, unscriptedinteraction requires the ability to predict users’ touch intentions on-the-fly, so that retargeting can be applied.
We now investigate how to achieve and balance these needs
by implementing and evaluating a system through studies.

Haptic Retargeting

Redirected touching was suggested by [18] as a mean to use
manipulation of the virtual hand-eye coordination. This results in discrepancies between a person’s real and virtual
hand motions, so that they reach the real and virtual objects
simultaneously.

SYSTEM IMPLEMENTATION

To investigate the different aspects of realizing Sparse Haptic Proxy, we implemented our VR system with an Oculus
Rift DK2 headset that incorporated an SMI eye-tracker at
60 Hz. The headset is tracked by the Oculus camera in combination with an overhead OptiTrack V120:Trio tracking
system. Depending on the specific virtual reality application,
the user sits or stands in front of our sparse haptic proxy.

Haptic Retargeting [1] redirects the user’s hand when touching any one virtual cube on the table to touch a single physical cube prop. However, [1] assumed a-priori-known targets
and scripted interaction. Our work extends their technique tosupport natural unscripted interaction. The amount of possible retargeting effects the range of geometries that can be
represented by a single passive proxy. We explore the range
in which haptic retargeting can be applied under 1) changeof retargeting distance, 2) effect on hand eye coordination,and 3) change of surface normal. All of them are essential todesign a practical system.

For the support of single-finger interaction across the physical surface, we assembled the SHP with a set of 25 solid planar surfaces, at orientations ranging from -60° to +60° in horizontal and vertical directions as shown in Figure 3.

Studies have shown that people adapt to spatial warping [21]and perform tasks as effectively as they do under non-warped
conditions [20]. In the context of walking, a single physicalprop can be used to provide haptics for multiple virtual objects [19]. This is achieved by using redirected walking [28],a technique that injects additional translations and rotations
to the user’s head movements. This causes users to walk ona physical path that is different from the perceived virtual
path. By having the user walk back to the same physical object when moving from one virtual target to the next, the object provides haptic feedback for multiple virtual targets, but
imperceptible redirections require large tracking spaces [33].
Figure 3. Left: Our system set-up. Right: The glove & Oculus
DK2 HMD modified by SMI to include eye tracking.

Spillmann et al. [32] proposed adaptive space warping to
warp different organ geometries onto one physical mock-up.
We are inspired from their work and extend the idea to amore general scenario where one general passive object provides touch feedback for many other objects in virtual environments.

Our system tracks the user’s finger using a worn glove andthe OptiTrack system. A Kinect V2.0 depth camera is used
at a calibration stage to measure the physical proxy location.
The displayed hand in the virtual world is a generic hand.
The offset of the user’s fingertip to the tracked marker, isfound by a short calibration routine where each user touches
proxy plates. Our system and applications are implemented
using Unity game engine 5.2.3.

REALIZING SPARSE HAPTIC PROXY

Whenever the user touches a virtual geometry, their hand isredirected using Haptic Retargeting to one of the SHP primitives. A Sparse Haptic Proxy’s geometry should therefore
be abstract enough to support multiple virtual environments.

3720

Haptic Feedback

CHI 2017, May 6–11, 2017, Denver, CO, USA

MAKING HAPTIC RETARGETING WORK DYNAMICALLY=

Figure 4 illustrates how Haptic Retargeting redirects theuser’s hand to a physical location different than the virtual
target location [1]. The virtual hand (skin color) and thephysical hand (blue color) are initially at the same position.
As the users’ hand moves towards to the virtual target, thevirtual hand position is gradually deviating in a direction
away from the haptic proxy, causing the user to correct the
trajectory toward the virtual target. Thus, the physical handis gradually moved toward the haptic proxy.
virtual target

Pp

Where
=
−
, and
new touch target is acquired.

virtual handT==at the frame where aThis general retargeting should be reduced to zero whenever
the users retract their hand close to the body. This is done to
avoid visible position difference between the physical handand its rendering closer than 20 cm to the body as shown in
Figure 5. Virtual hand retargeting starts only when the user’sin the previous equation is now
hand crosses the line anddefined as the distance between the line and the physical
hand.

Hv

Pv

,

+ (1 − )Ds

H0physical handHpphysical proxyzero-warp
zone

Figure 4. The virtual hand (skin color) shifts from its true
physical location (semi-transparent blue color), inducing theuser to correct the hand movement to touch the physical prop.

Figure 5. A zero-warp zone to help users reset their virtual
hand position when retracting it.

To perform haptic retargeting in a gradual manner along thehand path, the system needs to be supplied with the hand thestarting point, the location of the virtual target that will betouched, and the location of the corresponding physical target. The retargeting is described as follows:

As the distance between the user’s physical hand and theplace it is rendered at grows, it can generate uneasiness due
to conflict of the user senses. The goal of the first study is to
map the limit of comfortable retargeting distance. [1] showeduseful redirection beyond noticeable difference JND foundby [24]. To explore the useful range of redirection we lookat the user’s tolerance which is a combination of touch feedback and hand redirection.

Study 1: Maximum Tolerable retargetingbe the physical position of the user hand, andbeLet
the position in which the virtual hand is rendered to the user.
Assume that a virtual scene point should be mapped to aphysical proxy point .
=Participants

We recruited 12 right-handed participants (4 female) from
our institution, ages 21 to 30 (M=27.0). 6 participants had
experience with head-mounted VR. Participants received asmall gratuity for their time.

−

This offset between the real and virtual hand position is gradually added to the position of the virtual hand as the user’s
toward :hand starts moving from initial position
=

,

=

Task

.

The task was a target-acquisition task. The participant saw avirtual bullseye target whose location was mapped to ourphysical proxy with a controlled offset. The participant thentouched the virtual target with their right hand while our system redirected their hand to touch the physical prop. Hitting
the virtual target completed a trial (the virtual finger collides
with the virtual target). The participant then rated the acceptance of the experience on a 5-point Likert scale. We assigned a tolerability of ‘5’ to ‘no redirection’, representing
the base condition (and demonstrated to participants prior to
the study). ‘1’ represents intolerable redirection, i.e., the participant struggled to reach the target. We chose ‘3’ as ourthreshold, balancing easy reach with distance of redirection.

Where is the shift ratio, ranging between 0 and 1 from the
beginning of the motion to a full offset T when the virtual,
hand touches , and the physical hand touches
=
−
and
=
− .
Extension 1: Changing Retargeting On-the-Fly

Azmandian et al. [1] looked at a motion of user’s hand froma fixed point, , (near the body) to the targets arranged in
front of the user. However, during natural interaction, a decision on a new touch target can be made while the user’s
hand has already been moving under some other retargeting
amount to a prior target. To maintain a smooth movement of
the virtual hand, we modify the original Haptic Retargeting
offset to interpolate between the current retargeting and the
updated retargeting to the new target:

Procedure

After a brief introduction to our system, participants sat
down one at a time and put on the HMD and the marker glove
and familiarized themselves with moving their hand in VR.

3721

Haptic Feedback

CHI 2017, May 6–11, 2017, Denver, CO, USA

To sample the proxy space, we used 9 different locations on
the proxy (Figure 6 left). For each proxy location, 33 virtual
target locations were generated with 0, 15, 30, 45, 60° offset
in 8 different directions as shown in Figure 6 right. We shuffled all possible conditions and used one condition for eachtrial. Whenever a virtual target fell outside the proxy (suchas in the case of a left-wise virtual target of the left most location on the proxy), it was removed from the experiment toprevent extreme rotations of the participants.

30°

45°

(SD=1.71), 2.07 seconds (SD=1.66) with 30° offset, 2.84
seconds (SD=2.13) with 45° offset, and 4.03 seconds(SD=3.35) with 60° offset. The time for reaching targets had
positive correlation (r= 0.91) with the distance of redirection.
Scores:22%

15° offset

45°

23%

20%

39%

35%

30° offset

45° offset

26%

34%

60° offset

Following this experiment, the following experiment and thefinal system were limited to retargeting distances of up to 40°.
Extension 2: Predicting the User’s Intention

In the original implementation of haptic retargeting, user wasalways directed towards a known target. In order to enable
the user to freely interact with any object in the scene, oursystem predicts the user intended target in the virtual space.
Following [31], we used both the user’s gaze fixation and theuser’s hand movement to predict touch targets.
We detect gaze fixation using Binsted et al.’s approach [4],using a moving window of 0.6 seconds and check whether
the standard deviation of the angle of gaze samples within
this temporal window falls beneath 1°.

Figure 7 shows the overall distribution of participants’ ratings and the time spent with all tested offsets and directions.
We used one-way ANOVA, but found no significant difference in the rating and time spent between different physicallocations, different directions within the same offset(F7,40=0.111, 0.298, 0.408, 0.421 for the rating, F7,40 =0.465,
2.014, 0.932, 3.446 for the time spend within 15°, 30°, 45°,60° respectively, all p>.05).

30°

no offset

17% 26%

Figure 8 shows the rating population for each offset. Retargeting of up to 15° generated similar rating to no retargeting.
Retargeting up to 45°, received lower but above natural ratings. Beyond 45° the average and the standard deviation oftime spent increased dramatically, perhaps because the retargeting is more prominent and less natural to the user.

Results

30°

69%

31%

2 1

Figure 8. The rating population for each offset.

Before the study, participants practiced 10 trials with increasing offset up to 60°. We did not include these practice
trials in our analysis. Throughout all trials, we logged participants’ gaze, hand, and head motion for offline analysis. After each task, participants filled out a questionnaire. Eachparticipant completed all 169 trials in about 30 minutes.

60°

72%

60°

Figure 6. The 9 physical locations on the physical proxy (left)
and all possible 33 offsets of a vertical target (right). The middle location is the physical location (zero retargeting).

45°

22%

5 4 3

We use the state machine shown in Figure 9 to predict theuser’s intention and to determine the touch target. The threestates represent the user’s current activity: (a) Seeking a newtarget – the user gaze scans the scene, (b) looking at a target
for more than a minimal time (fixation) and finally reachingfor it. We used a significant hand velocity Vhand to detect thatthe user started to move her hand toward the target, and report a possible touch event. The threshold of 3 cm/s wasfound empirically from our pilot study

60°

actionintentionuser!gazeFix
rating

0

5

time
(s) 0

SparseHaptics

7

Target[returntarget]Seek

Figure 7. The distribution of user tolerance and time spent
with 15°, 30°, 45°, 60° offset in 8 directions.

Look

The average time spent on each task without offset was 1.81
seconds (SD=1.85). With 15° offset it was 1.80 seconds

Figure 9. Our target prediction state machine.

3722

Vhand > 3cm/s

Haptic Feedback

CHI 2017, May 6–11, 2017, Denver, CO, USA

Study 2: Predicting User Touch Targets

sets 1) the prediction accuracy, 2) the time between the prediction and the touch event, and 3) the distance from hand
position to the touch point at the time of prediction. The
mean accuracy without retargeting was 97.6%, 97.8% with20° retargeting, and 97.1% with 40° retargeting.

Our system introduces haptic retargeting and manipulate thehand-eye coordination. The goal of this study was to evaluate
any effect of retargeting on the performance of the touch
events prediction. This study focused only on prediction accuracy and participants touched virtual targets with no haptic
feedback.
Participants

We recruited a fresh group of 12 right-handed participants (7females, ages 24-30, M=27.6). 8 participants had previous
experience with head-mounted VR. Participants received asmall gratuity for their time.

Figure 11. The confusion matrix of our target prediction. (‘a’= actual, ‘p’ = predicted)

From Figure 12, we used one-way ANOVA and found thatthe time from prediction to touch event and the distance fromprediction point to touch point were affected by the offset(F2,1004=12.424 and F2,1004=62.9807 respectively, all p<.005).
The mean last prediction time without retargeting was 2.88
seconds (SD=1.90), 2.11 seconds (SD=1.30) with 20° retargeting, and 1.88 seconds (SD=1.15) with 40° retargeting.
The mean hand-to-target distance without retargeting was
28.7 cm (SD=14.90), 19.5 cm (SD=12.54) with 20° retargeting, and 15.2 cm (SD=4.01) with 40° retargeting.

Figure 10. Left: The testing scene for target selection. The letters on the balls were shuffled for every trial. Numbers were
not displayed, and are used for correspondence to Figure 11
only. Right: Possible retargeting distances and direction used.
Task

We showed a 4x4 grid of virtual balls with letters to participants as shown in Figure 10, left. For each trial, our systemread a letter out loud. Participants then searched and touched
the corresponding target, completing the trial upon contact.
A new trial is started by pushing a ‘next’ button. Each trial
has a different arrangement of letters on the balls. Since thisstudy focused on our prediction, no physical prop was used.

Accuracy (%)

all

Procedure

no offset

20° offset

40° offset

98
97
96
95

We first calibrated the eye-tracking HMD for each participant. Participants then followed a similar procedure as in thefirst study. We simulated retargeting to distances of 0°, 20°,and 40° in 4 directions (up, down, left and right) from thetouch target as shown in Figure 10. Each participant completed 16 targets x 9 retargeting = 144 trials in total.
Each participant trained with 10 trials without any offset tofamiliarize themselves with the system before the trial. Overall, each participant took about 30 minutes to finish.

4
3
2
1
0

Distance (cm)Time (s)

5
30
20
10
0

Figure 12. The accuracy, last target time and last target distance with different warping.

Results

Figure 11 shows the confusion matrix for the results of ourpredictions. Each target was tested during 108 trials. Overall,our prediction was 97.5% accurate. On average, our system
predicted the correct target 2.04 (SD=1.28) seconds before
participants touched it with an average distance of 23.72 cm(SD=14.33) from touching the target.

Discussion

The overall prediction accuracy was high (above 97%) andwas not significantly affected by the introduction of retargeting. We do see some shortening of the time from prediction
to touch: 73% and 66% for retargeting of 20° and 40°. Thisis caused by extra user gaze wandering preventing the systemfrom making a prediction until later. This will increase therate of retargeting needed, but since the prediction is successful, it did not hamper the system performance.

To verify the hypothesis that retargeting affects hand-eye coordination and prediction accuracy, we compared for all off-3723

Haptic Feedback

CHI 2017, May 6–11, 2017, Denver, CO, USA

We did notice a lower accuracy (about 90°) at the lower sidetargets (12 and 15) due to the slightly lower reliability of the
OptiTrack hand tracking around the corner targets.

between them. To enable the user to slide on the proxy surface, and is seen sliding on the virtual surface, we used
Pseudo-haptic effect [2]. We project the virtual geometry
onto the proxy interaction area so that each point touched by
the virtual hand has a corresponding point on the physicalproxy as shown in Figure 14.

Extension 3: Mapping Virtual Targets to the Physical Proxy

Mapping from the virtual geometry to the SHP can be optimized to achieve different goals. One can map the nearest
physical primitive to minimize the retargeting distance. Another can fit the most similar primitive in order to enhance
the range of possible haptic interaction. We present threemapping strategies that maps the predicted touch point to asingle primitive on the Sparse Haptic Proxy.
One of the simplest interaction mode is a single finger touching the surface at a single point. Although simple, it enables
a very versatile interaction: pressing a button, selecting or
picking objects and more.

Figure 14. Remapping the virtual dial on the virtual safe to anarea on the physical prop.

We investigated and implemented three approaches to remap
the geometry touch point to a primitive of the Sparse Haptic
Proxy as shown in Figure 13.

Study 3: Comparison of MAPPING strategies

The goal of our final study was to determine the mapping
technique participants would prefer most.
Participants

We recruited 12 new right-handed participants (2 female),ages 26-47 (M=30.8). 5 participants had experience withhead-mounted VR. Participants received a meal card as a
gratuity for their time.

n

a

b

Task

c

The task in this study was similar to the task in Study 1. A
trial began by showing the participant a virtual bullseye target and finished with the participant hitting the virtual target
with their right hand.

Figure 13. Remapping the trigger element using (a) line ofsight, (b) closest point, (c) similar surface normal.

1) Line-of-Sight casts a ray to the virtual target and determine the intersection with the 3D model of the proxy as
the remapping point. This mapping, dependent on theuser’s position, minimizes the angular offset between thetouch point and the proxy, resulting in only back and forward redirection.

The mapping to the physical proxy was determined by thechosen techniques (line-of-sight, closest-point and similarnormal). The virtual bullseye target was selected from two
sets of nine orientations and locations as shown in Figure 15
to simulate both a simple virtual geometry and a complex
one. The participants were instructed to move a step to theright to vary their location and perspective in the virtual andphysical environment after they have done both sets of thetargets.

2) Closest-Point. The selected proxy point is the nearest
point to the touch. This mapping, independent of theuser’s location, will minimize the retargeting distance,
but can exhibit an angular offset between the proxy andthe virtual geometry point.
3) Similar-Normal remaps the virtual target to the most similar direction surface normal in the Sparse Haptic Proxy,
limited within a maximal angular deviation of 40° as
found in Study 1. This lets the user feel a surface of similar direction, but can require larger retargeting distances.

Figure 15. Virtual bullseye targets in Study 3 were rendered in
two set of different orientations and locations in the 3D scene.

Similar-Normal mapped primitive enables the user’s handslides along a larger area around the initial touch point, or
use multiple fingers as touching the surface. Such operations
can simulate dials, sliders, touch screen etc., such as the dialon the safe in our room scene in Figure 2.

Procedure

The procedure was similar to that in Study 2. Each participant completed 36 trials per technique (18 virtual targets x 2
points of view), resulting in 108 total trials. The order oftechniques was counterbalanced across participants.

Although we match an interaction surface to a similar primitive, there are still differences in surface orientation or shape3724

Haptic Feedback

CHI 2017, May 6–11, 2017, Denver, CO, USA

Between techniques, participants filled out a questionnaire.
We asked 1) how smooth your hand could move and 2) how
similar you feel the touch feedback as it should be in the virtual scene. Participants saw a text label for each presented
strategy, but they were never told how a strategy works orwhat the label means: T0 (line-of-sight), T1 (closest-point)
and T2 (similar-normal). Participants chose their preferred
one among those labels after they had tried all three strategies. Overall the experiment took 20 minutes per participant.

usage of a sparse set of physical primitives that can bemapped and reused for haptic simulation of mismatched virtual geometries (Figure 17).
Current commercial VR generates haptic feedback via hand
held controllers, or wearables. The proposed approach has
several major advantages, e.g., resistance felt by the user using a grounded sparse haptic proxy and hands-free interaction (no extra weight or limit to natural touch).

similarity
preference

4
2
0

41.7%
58.3%

line-of-sight

8
6
4
2
0

1

2

3

4

5

smoothness12closest-distance3

Figure 17. The mismatch between our proxy (blue overlay)
and our cockpit and room scene.

45
similar-normal

One of our limitation is that each interaction with the virtual
geometry is constrained to the span of the primitive. For example, when the user drags the finger over a planar primitive,
it will eventually reach the end of the primitive where themapping changes to another primitive, which can be at a different location. However, in many applications and games
the common interactions are quite localized: open a lock,pick an object, push a door and so on. Such interactions are
mapped well to a Sparse Haptic Proxy.

Figure 16. Participants’ preference (left). Participants’ rating
histograms (Likert scale, 1= totally not) for the similarity oftouch feedback (top right) and the smoothness of the hand
movement (bottom right).
Results

As shown in Figure 16, no participant preferred Similar-Normal, which participants mostly rated low in smoothness. One
user noted the difficulty to sense a surface normal from a single touch point. Another participant said the surface normal
felt more correct under T2 but still preferred less redirection.
One explanation could be that the haptic sensation for the
single point touch is too limited to be a reward for thatamount of redirection.

Direct motion between adjacent touch targets may not leave
enough distance for redirection. This is future work. In our
scenarios, the virtual objects are separated by at least 5 degrees, and we will look at ways to improve it in a future work.
Another limitation that raises from our use of planar primitives, is lack of representation of corner, edges and strong
curvatures. Although planar or near planar surfaces are common in many geometries, there are many objects, such as
handles, buttons, sliders and more, whose haptic representation will elevate the haptic experience. While even the simplest physical objects (e.g., boxes) can serve as Sparse Haptic
Proxies for some scenes, more complex environments are
better served by a Sparse Haptic Proxy that comprises a
range of primitives. We see this as a future work.

7 participants preferred the line-of-sight approach and 5 participants preferred the closest-point approach. We found no
significant difference between both ratings (p>.05). The concave shape of the proxy and the limited motion possible forthe users limited the differences between these mappings. It
may be that a convex proxy will generate stronger differences when users move around it.
Since most participants preferred Line-of-Sight mapping, weused it as the mapping technique in our two demo applications. Line-of-sight always creates a 0° angle offset, such that
retargeting only happens when a user is moving forward.
This can result in having to stretch one’s arm, similar to the
GO-GO technique [27]. While the stretch is fixed in distancewith GO-GO, it dynamically changes in our technique depending on the location of the target.

Another dimension of variety could be the materials of which
the primitives are made of. Some can represent soft materials
while other can be hard. Textures and temperature may also
be represented by different primitives.
The arrangement of the primitives should be derived fromthe range of applications to be supported. Our current arrangement of a hemisphere supports a symmetric set of primitives at arm reach. However, an analysis of VR applications
may lead to a different arrangement.

DISCUSSION AND FUTURE WORK

We have presented Sparse Haptic Proxy and its use to provide touch feedback to a variety of virtual applications of different geometry. The main idea of Sparse Haptic Proxy is the3725

Haptic Feedback

CHI 2017, May 6–11, 2017, Denver, CO, USA

The proxy we used for our experiments covers about 120° ofview direction in front of the user at about arm’s reach. Thereis no limit to build a proxy that will cover the complete directions around the user as long as the user can get in and outof the contraption. Another possibility is to actively rotate
the proxy to face the user when turning. The speed of therotation can be much slower than the rotation of the user
hand, as the proxy covers about third of the full rotation. This
allows for a slower and safer active operation.

3.

Bergamasco M., Allotta B., Bosio L., Ferretti L., Parrini G., Prisco G.M., Salsedo F., Sartini G.. 1994. An
arm exoskeleton system for teleoperation and virtualenvironments applications. In Proceedings of IEEE International Conference on Robotics and Automation. 2,
1449–1454.
http://doi.org/10.1109/ROBOT.1994.351286

4.

Binsted, G., Chua, R., Helsen, W., and Elliott, D. Eyehand coordination in goal-directed aiming. Human
Movement Science 20, 4–5 (2001), 563–585.

5.

Burns, E., Razzaque, S., Whitton, M.C., and Brooks,
F.P. 2007. MACBETH: The avatar which I see before
me and its movement toward my hand. In Proceedingsof IEEE Virtual Reality Conference, 295–296.

We explored the possibilities and limitations of providing
touch feedback for a variety of applications using a Sparse
Haptic Proxy—a set of haptic primitives that is dynamically
mapped and reused to fit virtual geometries.

6.

We conducted a set of experiments that tested the performance of haptic retargeting and built on the results of eachstudy to design the next one. Study 2 examined prediction
accuracies under the maximum redirection we found in
Study 1. Study 3 used the prediction system we evaluated inStudy 2 and the redirection limit we found in Study 1.

Frederick P. Brooks, Jr., Ming Ouh-Young, James J.
Batter, and P. Jerome Kilpatrick. 1990. Project
GROPEHaptic displays for scientific visualization.
SIGGRAPH Comput. Graph. 24, 4 (September 1990),
177-185. DOI=http://dx.doi.org/10.1145/97880.97899

7.

Lung-Pan Cheng, Thijs Roumen, Hannes Rantzsch,
Sven Köhler, Patrick Schmidt, Robert Kovacs, Johannes Jasper, Jonas Kemper, and Patrick Baudisch. 2015.
TurkDeck: Physical Virtual Reality Based on People.
In Proceedings of the 28th Annual ACM Symposium on
User Interface Software & Technology (UIST '15).
ACM, New York, NY, USA, 417-426. DOI:
http://dx.doi.org/10.1145/2807442.2807463

8.

Conti, F. and Khatib, O. 2005. Spanning large workspaces using small haptic devices. In Proceedings ofFirst Joint Eurohaptics Conference and Symposium on
Haptic Interfaces for Virtual Environment and Teleoperator Systems, 183–188.

9.

Christian Corsten, Ignacio Avellino, Max Möllers, andJan Borchers. 2013. Instant user interfaces: repurposing
everyday objects as input devices. In Proceedings ofthe 2013 ACM international conference on Interactive
tabletops and surfaces (ITS '13). ACM, New York,NY, USA, 71-80.
DOI=http://dx.doi.org/10.1145/2512349.2512799

Currently, we use a single proxy in front of which the user
stands or sits. One can imagine a system that uses redirectedwalking to bring a user to multiple or one such proxies tosimulate a larger environment.
CONCLUSION

We see Sparse Haptic Proxy as a “compressed” representation of the haptic feedback for a given scene. That is, Sparse
Haptic Proxy uses fewer physical “bits” to represent thesame amount of haptic information by removing repetitions.
This simplifies the proxy and the space for passive haptics.
On the flipside, our approach to passive haptics using aSparse Haptic Proxy is not a lossless compression. Proprioception and tactile sensation changes compared to the original representation. Nevertheless, Sparse Haptic Proxy enables users to perform a rich set of interactions that are common in different virtual reality experiences as we have shownin this paper, such as simulation, games and others.
ACKNOWLEDGMENTS

We thank our hardware lab for their insightful suggestions
and thank all participants of our studies for their time.

10. George W. Fitzmaurice, Hiroshi Ishii, and William A.
S. Buxton. 1995. Bricks: laying the foundations forgraspable user interfaces. In Proceedings of the
SIGCHI Conference on Human Factors in Computing
Systems (CHI '95), Irvin R. Katz, Robert Mack, Linn
Marks, Mary Beth Rosson, and Jakob Nielsen (Eds.).
ACM Press/Addison-Wesley Publishing Co., New
York, NY, USA, 442-449. DOI:
http://dx.doi.org/10.1145/223904.223964

REFERENCES

1.

2.

Azmandian, M., Hancock, M., Benko, H., Ofek, E.,
and Wilson, A. 2016. Haptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality Experiences. In Proceedings of the 2016 CHI
Conference on Human Factors in Computing Systems(CHI '16). ACM, New York, NY, USA, 1968-1979.
DOI: http://dx.doi.org/10.1145/2858036.2858226

11. Goertz, R. and Thompson, R. 1954. Electronically controlled manipulator. Nucleonics, 12, 11, 46-47.

Ban, Y., Kajinami, T., Narumi, T., Tanikawa, T., andHirose, M. (2012). Modifying an identified angle ofedged shapes using pseudo-haptic effects. In Proceedings of the International Conference on Haptics: Perception, Devices, Mobility, and Communication, 25 of

12. Steven J. Henderson and Steven Feiner. 2008. Opportunistic controls: leveraging natural affordances as tangible user interfaces for augmented reality. In Proceedings of the 2008 ACM symposium on Virtual reality3726

Haptic Feedback

CHI 2017, May 6–11, 2017, Denver, CO, USA

New York, NY, USA, 93-101.
DOI=http://dx.doi.org/10.1145/505008.505026

software and technology (VRST '08). ACM, New
York, NY, USA, 211-218.
DOI=http://dx.doi.org/10.1145/1450579.1450625

23. Massie, T.H. and Salisbury, J.K. 1994. The
PHANTOM Haptic Interface: A Device for Probing
Virtual Objects Three Enabling Observations Three
Necessary Criteria for an Effective Interface. In Proceedings HAPTICS ’94, 55, 1, 295–300.

13. Anuruddha Hettiarachchi and Daniel Wigdor. 2016.
Annexing Reality: Enabling Opportunistic Use of Everyday Objects as Tangible Proxies in Augmented Reality. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM,
New York, NY, USA, 1957-1967. DOI:
http://dx.doi.org/10.1145/2858036.2858134

24. Matsuoka, Y., Allin, S. J., & Klatzky, R. L. (2002).
The tolerance for visual feedback distortions in a virtual environment. Physiology & behavior, 77(4), 651655.

14. Ken Hinckley, Randy Pausch, John C. Goble, and Neal
F. Kassell. 1994. Passive real-world interface props forneurosurgical visualization. In Proceedings of the
SIGCHI Conference on Human Factors in Computing
Systems (CHI '94), Beth Adelson, Susan Dumais, andJudith Olson (Eds.). ACM, New York, NY, USA, 452458. DOI=http://dx.doi.org/10.1145/191666.191821

25. Patrick, N. 1990. Design, Construction, and Testing ofa Fingertip Tactile Display for Interaction with Virtualand Remote Environments, Masters Thesis, Department of Mechanical Engineering, MIT, Boston, MA.
26. Pair, J., Neumann, U., Piepol, D., and Swartout, W.R.
FlatWorld: Combining Hollywood Set-Design Techniques with VR. IEEE Computer Graphics and Applications 23, 1 (2003), 12–15.

15. Hughes, C.E., Stapleton, C.B., Hughes, D.E., Smith,
E.M. 2005. Mixed reality in education, entertainment,
and training. Computer Graphics and Applications, 25,
6, 24-30.

27. Ivan Poupyrev, Mark Billinghurst, Suzanne Weghorst,
and Tadao Ichikawa. 1996. The go-go interaction technique: non-linear mapping for direct manipulation in
VR. In Proceedings of the 9th annual ACM symposium
on User interface software and technology (UIST '96).
ACM, New York, NY, USA, 79-80.
DOI=http://dx.doi.org/10.1145/237091.237102

16. Hiroo Iwata, Hiroaki Yano, Fumitaka Nakaizumi, and
Ryo Kawamura. 2001. Project FEELEX: adding haptic
surface to graphics. In Proceedings of the 28th annual
conference on Computer graphics and interactive techniques (SIGGRAPH '01). ACM, New York, NY, USA,469-476.
DOI=http://dx.doi.org/10.1145/383259.383314

28. Razzaque, S., Kohn, Z., and Whitton, M.C. 2001. Redirected Walking. In Proceedings of EUROGRAPHICS,
289–294.

17. Ivan E. Sutherland. 1965. The ultimate display. In Proceedings of the Congress of the International Federation of Information Processing (IFIP), 506–508.
http://doi.org/10.1109/MC.2005.274

29. Jia Sheng, Ravin Balakrishnan, and Karan Singh. 2006.
An interface for virtual 3D sculpting via physicalproxy. In Proceedings of the 4th international conference on Computer graphics and interactive techniquesin Australasia and Southeast Asia (GRAPHITE '06).
ACM, New York, NY, USA, 213-220.
DOI=http://dx.doi.org/10.1145/1174429.1174467

18. Kohli, L. 2010. Redirected touching: Warping space toremap passive haptics. In Proceedings of IEEE Symposium on 3D User Interfaces (3DUI), 2010, 129–130.
http://doi.org/10.1109/3DUI.2010.5444703
19. Kohli, L., Burns, E., Miller, D., and Fuchs, H. 2005.
Combining Passive Haptics with Redirected Walking.
In Proceedings of the 2005 International Conference
on Augmented Tele-existence, 253–254.

30. Adalberto L. Simeone, Eduardo Velloso, and Hans
Gellersen. 2015. Substitutional Reality: Using thePhysical Environment to Design Virtual Reality Experiences. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI
'15). ACM, New York, NY, USA, 3307-3316. DOI:
http://dx.doi.org/10.1145/2702123.2702389

20. Kohli, L., Whitton, M.C., and Brooks, F.P. 2012. Redirected touching: The effect of warping space on task
performance. IEEE Symposium on 3D User Interfaces(3DUI), 105–112.

31. Barton A. Smith, Janet Ho, Wendy Ark, and Shumin
Zhai. 2000. Hand eye coordination patterns in target
selection. In Proceedings of the 2000 symposium on
Eye tracking research & applications (ETRA '00).
ACM, New York, NY, USA, 117-122.
DOI=http://dx.doi.org/10.1145/355017.355041

21. Kohli, L., Whitton, M.C., and Brooks, F.P. 2013. Redirected Touching: Training and adaptation in warped
virtual spaces. IEEE Symposium on 3D User Interfaces(3DUI), 79–86.
22. Kok-Lim Low, Greg Welch, Anselmo Lastra, andHenry Fuchs. 2001. Life-sized projector-based dioramas. In Proceedings of the ACM symposium on Virtual
reality software and technology (VRST '01). ACM,

32. Spillmann, J., Tuchschmid, S., and Harders, M. (2013).
Adaptive space warping to enhance passive haptics inan arthroscopy surgical simulator. IEEE Transactions

3727

Haptic Feedback

CHI 2017, May 6–11, 2017, Denver, CO, USA

35. Tachi, S., Maeda, T., Hirata, R., and Hoshino, H. A.
1994. Construction Method of Virtual Haptic Space. In
Proceedings of the 4th International Conference on Artificial Reality and Tele-Existence, 131–138.

on Visualization and Computer Graphics, 19(4):626–
633.
33. Steinicke, F., Bruder, G., Jerald, J., Frenz, H., andLappe, M. 2010. Estimation of Detection Thresholds
for Redirected Walking Techniques. IEEE Transactions on Visualization and Computer Graphics 16, 1,
17–27.

36. William A. McNeely. 1993. Robotic graphics: a new
approach to force feedback for virtual reality. In Proceedings of IEEE Virtual Reality Annual International
Symposium, 1993, 336–341.
http://doi.org/10.1109/VRAIS.1993.380761

34. Szalavári, Z. and Gervautz, M. 2008. The Personal Interaction Panel - a Two-Handed Interface for Augmented Reality. Computer Graphics Forum 16, 3, 335–
346.

3728