Touch&Fold: A Foldable Haptic Actuator for Rendering Touch in
Mixed Reality
Shan-Yuan Teng

University of Chicago, USA

Pengyu Li

University of Chicago, USA

Joshua Fonseca

Romain Nith

University of Chicago, USA

Pedro Lopes

University of Chicago, USA

University of Chicago, USA

Figure 1: We engineered a foldable haptic device worn on the user’s fngernail that renders touch in mixed reality (MR) without
preventing users from also touching real objects. Here, a user follows MR instructions to repair their bicycle. (a) When theuser touches the virtual tire, a cover slides down from our nail-device and pushes against their fngerpad to create the contact& pressure with this virtual object. To further increase realism, the cover also includes a linear resonant actuator that renders
virtual textures using vibrations. This allows, for instance, to feel the roughness of the virtual mountain tire. Then, (b) when
the user turns to interact with real objects, the cover folds back, leaving their fngerpads free to feel the texture of their realtire or operate physical tools.

ABSTRACT
We propose a nail-mounted foldable haptic device that provides
tactile feedback to mixed reality (MR) environments by pressing
against the user’s fngerpad when a user touches a virtual object.
What is novel in our device is that it quickly tucks away when
the user interacts with real-world objects. Its design allows it tofold back on top of the user’s nail when not in use, keeping theuser’s fngerpad free to, for instance, manipulate handheld tools andother objects while in MR. To achieve this, we engineered a wireless and self-contained haptic device, which measures 24×24×41
mm and weighs 9.5 g. Furthermore, our foldable end-efector also
features a linear resonant actuator, allowing it to render not only
touch contacts (i.e., pressure) but also textures (i.e., vibrations). We
demonstrate how our device renders contacts with MR surfaces,
buttons, low- and high-frequency textures. In our frst user study,
we found that participants perceived our device to be more realistic
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor proft or commercial advantage and that copies bear this notice and the full citationon the frst page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specifc permission
and/or a fee. Request permissions from permissions@acm.org.
CHI ’21, May 08–13, 2021, Yokohama, Japan© 2021 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-8096-6/21/05. . . $15.00
https://doi.org/10.1145/3411764.3445099

than a previous haptic device that also leaves the fngerpad free (i.e.,fngernail vibration). In our second user study, we investigated theparticipants’ experience while using our device in a real-world taskthat involved physical objects. We found that our device allowed
participants to use the same fnger to manipulate handheld tools,
small objects, and even feel textures and liquids, without much
hindrance to their dexterity, while feeling haptic feedback whentouching MR interfaces.

KEYWORDS
Haptics, Wearable, Mixed Reality
ACM Reference Format:
Shan-Yuan Teng, Pengyu Li, Romain Nith, Joshua Fonseca, and Pedro Lopes.
2021. Touch&Fold: A Foldable Haptic Actuator for Rendering Touch in
Mixed Reality. In CHI Conference on Human Factors in Computing Systems(CHI ’21), May 08–13, 2021, Yokohama, Japan. ACM, New York, NY, USA,
14 pages. https://doi.org/10.1145/3411764.3445099

1

INTRODUCTION

Mixed Reality (MR) allows overlaying digital content with ourreal-world surroundings, creating powerful new tools. Many argue
that the next challenge of mixed reality is the addition of haptics.
Over the last decades, an impressive number of haptic devices
have allowed users to feel the forces (e.g., exoskeleton gloves [7–9, 19, 52]) and contacts from interacting with virtual objects (e.g.,

CHI ’21, May 08–13, 2021, Yokohama, Japan

Shan-Yuan Teng et al.

Figure 2: (a) When not in use our device keeps the fngerpad free for feeling the haptics of real-world objects. When activated,
it unfolds (b) via a rack and pinion, and (c) a hinge redirects the force against the user’s fngerpad, which allows it to create
three types of haptic efects: contact (pressure of the cover), (d) high-frequency textures (using an LRA embedded in its cover),and (e) low-frequency textures (by rocking the cover back and forth).
vibration on the fngerpads [13, 22, 23, 50]). However, researchers
argue that haptics for MR are inherently diferent from haptics forvirtual reality (VR), as they must leave the user’s hands free so thatthe user can also interact with real objects [3, 6, 33, 49, 50].
This is imperative as many powerful uses of MR involve holding tools such as for repairs [17], construction [41], or tangibleinteractions [18]. Therefore, haptics for MR must not impair theuser’s ability to feel the world through their fngerpads. Recently,
researchers proposed unencumbered force-feedback in MR by using
electrical muscle stimulation instead of the traditional exoskeletons[33]. However, this only provides the sense of pushing against an
object and does not stimulate the sense of touching an object.
As of now, there is no device that provides users with a sense ofcontact at the fngerpad in MR without covering up their fngerpads.
Existing approaches involve applying actuators on the fngerpads,
such as thin electrodes [21, 49, 50] or soft actuators [15]. While someof these are also driven by the goal of minimal interference with theuser’s tactile sensation, adding these thin patches decreases one’s
ability to perform discriminate textured surfaces because thesepatches impair the tactile acuity of our fngerpads [34].
When it comes to rendering touch in MR without covering up thefngerpads, the most promising solution remains placing a vibration
motor on the user’s fngernail [2]. While this leaves the fngerpad
free, it has two main disadvantages: it does not create pressure, andits feedback is unrealistic, as it occurs on the fngernail rather thanon the fngerpad.
We tackle this challenge by engineering a foldable haptic device that provides virtual objects with haptic feedback by pressing
against the user’s fngerpad, yet, quickly tucking away when theuser grasps real objects. Our device, depicted in Figure 1, works
by unfolding a cover that wraps around and presses against theuser’s fngerpad. The key to its compact form factor is that theunfolding cover can be retracted and stored on top of the fngernail
via a motor-driven rail. Furthermore, besides rendering the senseof touch, it also renders textures by means of a linear resonant
actuator (LRA) embedded in its cover, as depicted in Figure 2
Besides being a one-of-a-kind device for MR haptics, it is also
completely untethered and self-contained, a feature not seen in any
existing haptic device of this kind. In its small footprint (24×24×41
mm and 9.5 g), it includes actuators, electronics, battery, and wireless communication through Bluetooth. We demonstrate how ourdevice renders haptic sensations such as taps, button presses, andlow- or high- frequency textures. In our frst user study, we foundthat participants felt that our device was more realistic than ourbaseline (attaching a vibration actuator to the fngernail). In our
second user study, we investigated the participants’ experience
while using our device in a real-world task that involved physicalobjects. We found that our device allowed participants to use thesame fnger to manipulate handheld tools, small objects, and evenfeel textures and liquids, without much hindrance to their dexterity,
while feeling haptic feedback when touching MR interfaces.

2

SYSTEM WALK-THROUGH

We demonstrate the key qualities of our device with the example
of a MR application for bike repairs. Here, the user wears our nailmounted device as well as a HoloLens 2 (Microsoft), which displays
the MR content and tracks their hands via built-in depth cameras.

2.1

Keeping the User’s Fingers Free To
Manipulate Tools

The user cycles between interacting with the MR repair guide andworking with physical tools (Figure 3a). The user taps the mid-air“next” button to reveal the instructions for removing the wheel
hub nuts (Figure 3b). Immediately, our device unfolds and pushes
its cover against the user’s fngerpad, rendering the pressure of
the contact with the MR button. Our device takes 92 ms to unfold.
Then, as soon as the user’s fnger leaves the MR interface, the cover
retracts backward, leaving the user’s fngerpad free to adjust thewrench’s jaw while still even wearing our device, as depicted in
Figure 3c. Note this is a dexterous task, and would not be possible
with existing haptic devices, as they would interfere with the user’s
tactile acuity and disrupt the task.
Now that the wheel has been removed, the MR guide displays
two diferent virtual tire profles (mountain tires vs. racing tires).
The user compares their tire with the virtual tires. Our device not
only keeps the user’s fngerpad free to feel their actual tire’s texture(Figure 4a) but also unfolds to creates the textures of diferent virtual
tires.
In fact, our device renders two diferent textures for two diferent
virtual tires. First, as depicted in Figure 4b, the user strokes theirfnger across the virtual mountain tire; our device unfolds its coverTouch&Fold: A Foldable Haptic Actuator for Rendering Touch in Mixed Reality

Figure 3: (a) The user holds a wrench while wearing our device. (b) When the user reaches out to touch the virtual button, our device unfolds and renders the contact with the MR
button; then (c) as soon as the user’s fnger leaves the MR interface, our device folds back, allowing the user to feel andmanipulate the wrench with their bare fngerpad.

CHI ’21, May 08–13, 2021, Yokohama, Japan

Figure 5: (a) User grips the bike handle while wearing ourdevice . (b) As they type the address, our device renders the
clicking of the keys. (c) When they drags the slider, our device renders haptic detents that depict the magnifcation levels.

delivering haptic sensations anywhere. In the case of MR, thiswearability requirement is paramount since most of AR devices
are untethered, such as HoloLens [54], MagicLeap [55], or evensmartphones [38].

3.2
Figure 4: (a) The user feels the texture of their real tire. (b)Our device moves its cover back and forth to render thebumps of a rough, virtual mountain tire. (c) Our device vibrates its LRA to render the fne texture of a virtual racing
tire.

around the fngerpad and quickly rocks the cover back and forth
to render the low-frequency texture of the tire bumps. Then, as
depicted in Figure 4c, the user feels the profle of the virtual racing
tire. Here, our device unfolds and pushes the cover against the user’s
fngerpad. Then, as the user strokes across the tire, the embedded
vibration actuator (LRA) renders the fne, high-frequency texture
typical of a racing tire.

2.2

Providing Haptics To General MR Widgets

After replacing the tires, the user bikes to their friend’s house using
the MR interface for driving directions. Figure 5a depicts the user
grasping the bike’s handle, while still wearing our device. When theuser types in the address on a virtual keyboard, our device creates
contact feedback for each keystroke by unfolding and pressing its
cover against the user’s fngerpad (Figure 5b). When the direction
is being shown in a virtual map, the user zooms using the map’sslider. Here, our device provides not only contact but also haptic
detents using vibration motor to depict the diferent magnifcationlevels (Figure 5c).

3

RELATED WORK

Our work builds on the feld of haptics; in particular, on wearable
tactile haptics for virtual and mixed reality.

3.1

VR and MR Demand Wearable Haptics

As real-walking VR increases in popularity (e.g., VIVE’s lighthouse [53]), most recent advances in haptic feedback have focusedon wearables. These are devices mounted onto the user’s body,

Force Feedback & Tactile Feedback in VR

Haptics devices can be categorized into two main streams: force
feedback and tactile feedback [28]. While force feedback devices render the forces that arise with contacting a virtual object (e.g., feeling the weight of a virtual object [20, 37]), tactile feedback devices render the contact with the object’s surface (e.g., texture [40, 47], temperature [56]). To maximize realism, researchers seek combinations of tactile and force feedback(e.g., [26, 45]).
3.2.1 Wearable Force Feedback. Wearable force feedback, especially in VR, is often provided through the means of motor-based
exoskeletons [52] or ferromagnetic fuids [48]. More recently, researchers have explored alternative avenues to miniaturize these
exoskeletons using smaller semi-passive actuators, such as brakemechanisms [8, 9, 14, 19], or pulley mechanisms that provide force
feedback to the arm [44] or fngertips [12, 46]. Pseudo-force feedback was proposed as an alternative [1]. On the other hand, electrical muscle stimulation (EMS), which bypasses motors entirely
and directly stimulates the user’s muscles [30], has been used toenable a smaller hardware footprint for force feedback in both VR[32] and AR [33].
3.2.2 Wearable Tactile Feedback. Wearable tactile feedback is typically achieved by attaching vibration motors to the user’s body,most commonly on the fngerpads [57]. Vibration was used to emulate textures [42], compliance [24], or even to create force illusions[36]. The advantage of vibrotactile devices is that the actuators are
small and thus wearable. The typical vibrotactile device takes theshape of a “haptic glove”, typically containing multiple vibration
motors underneath each fngerpad [57].
However, feeling a vibration is diferent from feeling pressure[27, 31]. Pressure triggers the Merkel cells, while vibrations mainly
activate Pacinian corpuscles [27]. As such, when prioritizing haptic
realism over form factor, researchers opt for devices that generate
the sense of pressure at the user’s fngerpad. Examples of thisinclude: Schorr et al.’s fnger-mounted and motor-based device,
which creates pressure in 3DOF [39]; HapCube, a fnger mounted
device that renders stifness, friction, and pressure by deforming

CHI ’21, May 08–13, 2021, Yokohama, Japanthe skin surface [23]; FinGAR, a fnger glove device that renders
pressure, low- and high- frequency vibration, and shear forcesusing mechanical actuation and electrical stimulation [50]; and,lastly, HapThimble, a fnger glove that emulates button presses byusing a servo motor to push a pad against the fnger in addition
to a vibration motor [22]. A more general and thorough review of
wearable tactile haptics can be found in [35].
Unfortunately, while these devices provide realistic haptic feedback (i.e., pressure and not just vibration), they are too cumbersome
to use in MR as they cover the user’s fngerpads, hindering the haptic sensations elicited by real-world objects. Users would have to
remove these devices before they can touch a real-world object.
Thus, existing tactile devices have found their use in VR but rarely
in MR.
3.2.3 VR Tactile Haptic Devices That Avoid Encumbering The Hand.
Because preserving user’s dexterity and tactile feedback is important, researchers started turning into mechanisms that deliver haptic feedback to the user’s hands without constantly covering it up.
PuPoP is an example of such a haptic device that mounts infatablepads into the user’s hand; these pads only infate on demand tocreate haptic feedback as the grabs virtual objects [43]. Similarly,
Haptic PIVOT is a VR haptic device that leverages a similar approach
but is implemented via motors; it mounts a motorized handle to
the wrist, which can pivot, on demand, to touch the user’s palmas they grab virtual objects [25]. However, none of these devices
can target a particular fngerpad; they simply push their actuatoragainst the user’s entire hand at once.
We take inspiration from these approaches that leave the user’s
hand free and take a step further into leaving the user’s fngerpads free. This requires engineering mechanisms much smaller andlighter than any of these hand-based feedback devices because we
intend to target the fngerpad itself and not the hand.

3.3

Tactile Haptics Specifcally Engineered for
Mixed Reality (MR)Researchers have been exploring ways to provide haptic feedback
without obstructing the fngerpads for MR. Arguably, the most MRready device is the fngernail vibration by Ando et al. [2]. In theirapproach, a vibration motor mounted on the fngernail augments
touches on virtual objects. While this device keeps the fngerpads
absolutely free, it has two main limitations: it does not generatepressure (only vibrations), and its tactile feedback happens at thefngernail instead of at the fngerpad, which can lead to unrealistic
sensations due to the spatial incongruence.
Another popular approach is to use thin actuators that, while still
covering the fngerpad, try to minimize this interference. Examples
include: Haptic ring, a device that squeezes the fngerpad by pullingwires wrapped around it [4]; Tacttoo, a device comprised of a thinelectrode sheet on the fngerpad and a stimulator at the wrist, whichelectrically stimulates the fngerpad [49]; or HydroRing, a fngerworn liquid chamber and a wearable pump, which creates pressure,vibration, and temperature on the fnger using hydraulics [15].
Unfortunately, while using the aforementioned devices, the user
still feels these thin actuators (e.g., wires, electrodes, etc.) every
time they touch a real object, impacting their sensation of textured
surfaces [34]. Furthermore, with thin actuators, the resulting haptic

Shan-Yuan Teng et al.

feedback is mostly limited to vibrations (e.g., [49]) or small contact
areas (e.g., [4]). Unlike these approaches, we propose the frst tactile
device capable of rendering both pressure and vibration, that canalso tuck away when the user is interacting with real objects.

4

KEY CONTRIBUTIONS AND LIMITATIONS

Our key contribution is engineering the frst foldable actuator thatcan provide the missing tactile haptics of mixed reality, while also
being able to tuck away to let the user still interact with real objects.
The key technical insight that enables this unencumbered feedback
is our compact folding mechanism.
Our approach has four benefts. (1) It provides on-demand haptic
feedback to the user’s fngerpad, i.e., the fngerpad remain unobstructed when the user is not interacting with virtual objects. (2) Its
tactile feedback is realistic as it combines pressure with vibrations,thus rendering a wide range of sensations such as contacts, mechanisms, and textures. (3) Our device is untethered and self-contained,
ftting entirely around the user’s fngernail. In contrast, most haptic
devices of this kind require cables running from the user’s hand toexternal power supplies, circuitry, pumps, etc. Lastly, (4) not onlyis the hardware footprint optimized for mixed reality by using only
surface mounted electronics, but its encasing was printed in clear
materials to minimize visual interference. Furthermore, as we willshow later, other actuators (e.g., such as a heating Peltier element)
can also be integrated into our device, allowing an even wider rangeof haptic sensations.
Our device is not without its limitations: (1) while we optimized
its footprint, it still covers up the user’s fngernail, and obstructs theside of the fngertip at certain angles; (2) while we fabricated it to ft
the average adult fnger [10], it might need adjusting for diferent
fnger sizes; lastly, (3) as with any mechanical haptic device, it has
its inherent latency (92ms), which we currently compensate byenlarging the collision detector volumes in our MR applications.

5

IMPLEMENTATION

To help readers replicate our design, we now provide the necessary technical details. Furthermore, to accelerate replication, we
provide all the source code, 3D fles, frmware, and schematics of
our implementation1 .
Figure 6 depicts our self-contained prototype, which was 3D
printed using a Form Labs 3 with clear resin to minimize visualinterference with the real world. Its complete footprint is 24×24×41
mm and weighs 9.5 g, including its battery. It attaches to the user’s
fngernail using double-sided tape.

5.1

Folding Mechanism

At the core of our contribution is our folding mechanism, depicted
in Figure 7. Its key design feature is a hemispheric rail from which
our “cover” unfolds. Once extended, the cover presses against theuser’s fngerpad.
The cover is comprised of two segments connected via a thinplastic sheet. To fold or unfold, we actuate the cover using a rackand pinion drive. Figure 7a depicts the initial confguration, in
which the cover’s front segment stays in the case. Figure 7b depicts
the cover’s front segment is pushed as it is driven by the pinion
1 http://lab.plopes.org/#touchfold

Touch&Fold: A Foldable Haptic Actuator for Rendering Touch in Mixed Reality

CHI ’21, May 08–13, 2021, Yokohama, Japan

Figure 6: (a) Our self-contained haptic device; (b) detail of its cover; (c) top view of our device with ruler unit in cm.

Figure 7: The key to our mechanism’s unfolding is that (a)its cover is comprised of two segments; (b) our hinge is basedon two detents that create a stopper and (c) force the cover
to pivot and be pushed by the wedge against the fngerpad.
to the point where the front segment is fully extended and hits ahinge stopper at the end of the rail. Figure 7c depicts the last stage
in which a wedge in the cover pushes and causes the front segment
to pivot around the hinge and land fush against the fngerpad. The
shape of our casing, cover and its hemispherical rail are all conicalin order to ergonomically follow the fnger’s shape, allowing thecover to fully contact with the fngerpad.

5.2

Actuation: Pressure And Vibration

Our device unfolds using a DC motor (26:1 Sub-Micro Planetary
Gearmotor 0.1 kg-cm, Pololu) mounted on our 3D printed rail drive(rack with 26 teeth and pinion with 12 teeth).
To increase the expressivity of our device, we embedded a linear resonant actuator (LRA C10-100, Precision Micro Drives) inthe cover that touches the user’s fnger. Our LRA renders highfrequency textures, allowing our device to render both contactpressure and a wider range of textures. We drive our LRA using a
MOSFET between 150-190 Hz; its resonant frequency is at 170Hz.
Lastly, a force sensor (FSR 400, Interlink Electronics) and a photointerrupter (SG-105F, Kodenshi) close the control loop by acting as
limit switches. The force sensor also doubles as a feedback signal for
fne-tuning the amount of pressure applied on the fngerpad. Thus,
our device not only creates pressure but also constantly measures
the applied force, which we demonstrate next in a brief technicalevaluation. Finally, we used the photo interrupter to sense whenever the cover is fully retracted, which serves as the signal to stop
actuating our DC motor.

5.2.1 Measuring Speed, Force & Noise. We further characterized
our device by measuring its speed, haptic force, and noise level.
Latency: The mechanism takes 92 ms to actuate, which is measured using slow motion camera (240 fps). This relatively fast speed
allows us to create low-frequency vibrations by driving our motor
back and forth when already in contact with the fngerpad. We
measured the Bluetooth communication latency using the same
method to be 40 ms.
Force: We measured force using a load cell with error of 9.8×10-5
N, with our device clamped on a 3D printed support. When unfolded,
it creates a sustained force of 0.34 N against the user’s fngerpad,
with a short overshoot peak of 0.41N before the force stabilizes.
Operational noise: We measured its operational noise using
a microphone. Our mechanics are quiet, producing around 22.25
RMS dB when unfolding. This measurement was recorded at arm’s
length from the device and in reference to a quiet background. As
reference, a clap is 65.01 RMS dB in this recording setup.
5.2.2 Measuring Contact Area. A relevant factor in haptic perception is the contact between the actuator and the user’s fngerpad. As
such, we set out to measure the contact area that our device makeswith a fnger. To measure this, we constructed a simple artifcial
fnger made from a rigid portion (using PLA, which emulates thenail) and a soft portion (using Ecofex 00-30), depicted in Figure 9a.
Then, we attached our device to the artifcial fnger and coated ourdevice’s cover (the part that contacts the fnger) with washable red
ink. This is a typical method used to determine contact between
fngerpads and haptic actuators employed, for instance, by Hauser
et al. [16]. Then, we actuated our device so that it contacts the artifcial fnger and leaves an ink imprint, which we depict in Figure
9b. Our result shows that, as expected, our device makes stablecontact with the fnger at its center, which is ideal as the vibration
motor is placed in that location. However, we also observed someunwanted side, yet much smaller, contact caused by the unfolding
mechanism’s rail.

5.3

Electronics: Printed Circuit Board And
Schematics

Figure 10 depicts the electronics schematic of our device. Our
16.8x10.3 mm PCB houses at its core a microcontroller with Bluetooth Low Energy (nRF52811, Nordic Semiconductor). To decrease

CHI ’21, May 08–13, 2021, Yokohama, Japan

Shan-Yuan Teng et al.

Figure 8: Our device produces a constant pushing force around 0.34N against the user’s fngerpad. Furthermore, it folds andunfolds very fast, taking only around 92 ms.

5.4

Rendering Four Haptic Sensations

To enable a range of haptic sensations, we control the actuation profles of both the rack & pinion (pressure) and of the linear resonant
actuator (texture) as follows:

Figure 9: (a) To measure the contact area of our folding mechanism with the user’s fnger pad we utilized an ink contact
test between our device and an artifcial (yet realistic) fnger.
(b) result from a contact test. (Arrows point to the fngertip).
its footprint, we used a ceramic chip antenna (W3008C, Pulse
Larsen), instead of the traditional zig-zag PCB antennas.
We power our device using a 40 mAh LiPo battery. We measured
a current of 200 mA when it unfolds, which takes 184 ms per interaction. As such, our device can be used for 12 min of continuous
tactile feedback. It is worth noting that in typical interactions with
MR interfaces one just expects to feel a few hundred milliseconds
of contact (e.g., tapping a button), thus our device’s battery tends
to last for many hours of on-demand use.

1. Simple surface contacts. To render touches with virtualsurfaces, we unfold the actuator until the force sensor reports contact with the fngerpad. Then, the mechanism keeps
pushing at a predetermined speed while the fnger remains
in contact with the virtual object.
2. Mechanisms (e.g., a button’s spring). To render mechanisms with variable force, we unfold the mechanism until
force sensor detects contact. As the user presses further downon the virtual button, the device pushes the unfolded covereven harder against the fngerpad to simulate the counterforce of the button’s spring.
3. Low-frequency textures (e.g., corrugated paper). To
render textures up to 25 Hz, we unfold the mechanism until
there is contact with the fngerpad. Then, we drive the coverup and down, against the fngerpad as the user runs acrosstheir fnger on the virtual textured surface.

Figure 10: Electronics schematic of our PCB.

Touch&Fold: A Foldable Haptic Actuator for Rendering Touch in Mixed Reality

CHI ’21, May 08–13, 2021, Yokohama, Japan

4. High-frequency textures (e.g., sand paper). To rendertextures from 150-190 Hz, we frst unfold the mechanism until there is contact with the fngerpad. We then drive the LRAat the desired frequency to render the actual texture as theuser runs across their fnger on the virtual textured surface.

5.5

Tracking and Display

To display the graphics to the user, we used a HoloLens 2. Built-in
depth cameras on the headset are used to track their hands. We
unfold our device to tap the user’s fngerpad only when the fnger
collides with a virtual object. Then, whenever the user’s fnger
leaves the virtual object’s collider (the "touch" area programmed in
our Unity3D demos), we immediately retract the cover back onto
the fngernail to leave the user’s fngerpad free. We implemented
this strategy as it is ideal for MR, since it prioritizes real-worldinteractions over virtual interactions. To trigger our device when
the user touches an MR object, we expand the fnger’s collisionbox in Unity3D to the radius of our device, which further compensates for its aforementioned latency. While we utilized the built-in
tracking from a HoloLens 2 headset, our system can also be pairedwith alternative tracking systems, such as the proximity sensors or
optical motion capture.

6

DEMO APPLICATIONS

To illustrate the key benefts that our device ofers to mixed reality,
we implemented three additional demos beyond the bike repair
guide, which was previously shown in Figures 3-5. Our demos werebuilt in Unity3D and displayed using two MR headsets (ProjectNorth Star & HoloLens 2), which were connected to our device via
Bluetooth.

6.1

Application#2: feeling haptic transitions
between physical and virtual

We depict how our foldable haptic actuator allows to render surface
contact and texture even as the user transitions between physicalvirtual objects, or vice versa. To demonstrate this, we implemented
a simple MR furniture editor inspired by [29, 33]. Figure 12 depicts
a user transitioning between feeling their real table to feeling the
texture and contact of the table’s virtual extension.

6.3

Figure 12: In our interior design application, (a) users canfeel both the real table but also, (b) the virtually-extended
table.

Application#1: Feeling On-Body Interfaces

This application depicts how our foldable haptic actuator does notinterfere with the haptic sensations elicited by touching an on-bodyinterface. Figure 11 shows an interface displaying physiological data
(steps, temperature, and heart rate from health trackers) projected
onto the user’s non-dominant arm; this was inspired by popular
MR interface designs, such as TapTap [5] or HoverUI [51]. Here,our foldable haptic actuator allows the user to feel their skin whentapping on the interface projected on their arm as well as feelinghaptic feedback when touching midair interfaces.

6.2

Figure 11: Our quantifed-self MR application uses a menu
anchored to the user’s arm. (a) As our device tucks awaywhen not in use, the user’s fngerpad can touch their own
arm. (b) When the user touches a mid-air slider, our device
then creates the feeling of contact and renders the slider’shaptic detents.

Application #3: Enabling Multi-Finger
Haptic Feedback That Does Not Occlude
Fingerpads

To render the feeling of contacting with more complex objects,
which typically requires multiple fngers, the user can wear two

Figure 13: A user wearing two of our foldable haptic actuators to interact with a MR kitchen-timer. (a) Users can grasp
the physical knob on the stove as our device leaves their fngerpads free, but also, (b) as they turn the virtual timer knob,
they feel the contact and detents (vibration) generated by thetwo devices they wear on their index and thumb).
(or more) of our devices. Figure 13a depicts how, while the user
is cooking, our two devices leave the fngerpads free, allowing
the user to rotate the physical knob on their stove to turn on thegas. Then, as depicted in Figure 13b, the user sets the MR kitchen
timer to fve minutes. When the user grasps the timer knob using
their index fnger and thumb, both of our devices unfold and press
the fngerpads to create the sensation of contact. Also, our device
creates vibrations to render the timer knob’s detents.

CHI ’21, May 08–13, 2021, Yokohama, Japan

Shan-Yuan Teng et al.

Figure 14: Adding contact haptic feedback to GUI widgets in existing Microsoft HoloLens Mixed Reality Toolkit (MRTK): (a)buttons (b) piano (c) sliders, (d) grabbing objects, a cofee mug in this case.

6.4

Generalizing by Adding Haptics to Existing
MR Toolkits

Finally, we generalize the usage of our haptic actuator to existing
MR toolkits. In this case, we add it to the HoloLens Mixed Reality
toolkit (MRTK). As depicted in Figure 14, we add the missing haptics
to MRTK demos (from the Hand Interaction package [58]). Here,contact haptic feedback is rendered when the user presses any
widget, such as buttons (Figure 14a) or even piano keys (Figure 14b).
Furthermore, by wearing two of our devices, we also render contact
haptic feedback for pinching (Figure 14c) or grabbing virtual objects(Figure 14d).

7

USER STUDY #1: HAPTIC FEEDBACK

The objective of our frst study was to validate our device’s ability to render touch in MR without encumbering the user’s fnger.
Therefore, we compared it to fngernail vibration (inspired by [2]),which is the most relevant haptic device specifcally designed to
not obstruct the fngerpad. Our hypothesis was that touching MR
objects with our device would feel more realistic than with thebaseline device, since our device provides not only vibration but
also pressure. Our study was approved by our ethics committee(IRB20-0276).

7.1

Apparatus

Participants wore a North Star headset [59]. We used the headset’s
depth camera for fnger tracking. Our Unity3D application provided
a simple MR environment that included one interactive object ateach time.

7.2

Conditions

Participants were asked to touch these objects in two conditions:(1) a USB version of our foldable actuator, which bypassed its
Bluetooth communication, and (2) fngernail vibration, by meansof an LRA at 170Hz frmly attached to the participants’ fngernails.
Condition order was counterbalanced across participants.

7.3

Tasks

Participants were asked to touch fve diferent mid-air virtual objects rendered in MR, which visually resembled a slab of concrete,
cloth, a button, corrugated paper, and sandpaper (Figure 15). For

Figure 15: The fve MR objects presented in our study: a hardsurface (slab of concrete), a soft surface (cloth), a button witha spring mechanism, a low-frequency texture (corrugatedpaper), and a high-frequency texture (sandpaper).
the slab and cloth, participants were instructed to touch on thesurface. For the button, participants were told to press it. For thecorrugated paper and sandpaper, participants were told to run theirfnger across the object’s surface.
After touching an object with either our device or the baseline,
participants were asked to rate the perceived realism of the haptic
feedback on a 7-point Likert scale, ranging from 1=“felt artifcial”
to 7=“felt real”.
Participants performed a total of 30 trials: 3 repetitions × 5 objects × 2 conditions. Trials were presented in randomized order.
At the end of the study, we asked participants which interface
condition they preferred.

7.4

Participants

We recruited 10 participants from our institution (M=25.6 years old,
SD=2.2; seven identifed as female, three as male). Four participants
had previously experienced MR, but none with haptics. Participants
were compensated with 10 USD for their time.

7.5

Results

Our main fndings are depicted in Figure 16. We analyzed our datausing two-way repeated measured ANOVA. We found signifcant
diference in average yield by both conditions (foldable actuatorand fngernail vibration, F(1)=164.51, p<.001) and virtual objects(F(4)=83.9, p<.001). We also found a signifcant diference in interaction of these terms (F(4)=69.1, p<.001). Thus, pair-wise Tukey
multiple comparisons were conducted.
We present our fndings regarding perceived realism while interacting with each object in both conditions:Touch&Fold: A Foldable Haptic Actuator for Rendering Touch in Mixed Reality

CHI ’21, May 08–13, 2021, Yokohama, Japan

Figure 16: Participants’ perceived realism in both conditions.

Figure 17: Participants’ preferred interface for each MR object.
1. Hard surface: We found a statistically signifcant diference
between conditions (p=0): foldable actuator was perceived
as more realistic (M=5.27, SE=0.20) than fngernail vibration (M=2.4, SE=0.21).
2. Soft surface: No signifcant diference was found in between conditions (p>.5), while comparing foldable actuator (M=3.03, SE=0.26) to fngernail vibration (M=2.63,
SE=0.21).
3. Button: We found a statistically signifcant diference between conditions (p <.001): foldable actuator was perceived as more realistic (M=5.23, SE=0.20) than fngernail
vibration (M=3.63, SE=0.33).
4. Low-frequency texture: We found a statistically signifcant diference between conditions (p=0): foldable actuator was perceived as more realistic (M=5.5, SE=0.19) than
fngernail vibration (M=2.57, SE=0.18).
5. High-frequency texture: We found a statistically signifcant diference between conditions (p<.05). Participants perceived our foldable actuator to be more realistic (M=4.57,
SE=0.21) than fngernail vibration (M=3.2, SE=0.22).
Figure 17 shows participants’ preferred interface for interacting with each of our MR objects: all 10 participants preferred
the foldable actuator for hard surfaces; preferences were split
regarding soft surfaces, with half of the participants preferring
the foldable actuator; eight participants (out of 10) preferred thefoldable actuator for buttons; and, lastly, all 10 participants preferred foldable actuator for both high- and low-frequency textures.

7.6

Qualitative Feedback

When asked about their experience, all participants mentioned thatusing the fngernail vibration felt unrealistic as sensations did notarise at the fngerpad. For example, P9 stated “very obvious it’s thenail” and P2 added “nothing is felt with my fngerpad”. Furthermore,
P10 and P7 commented on their perception: “vibration [alone] isweird for simulating touch” (P10) and “it feels real having something
covering my fngerpad” (P7).
Interacting with the soft surface using our foldable actuator
led to a haptic mismatch, as some participants perceived it as “toohard” or “too strong” (P1, P6, P8, P9), or unrealistic due to thefact that our “[pad] is solid” (P2). P4 added that “both [interfaces]
are not good [for feeling the cloth]”. P3 explained their preference for the fngernail vibration by stating that “[its] vibration islighter”.
When asked about the button, participants had varied expectations of the button’s feedback. Some consider vibration more
important than pressure: “I expect buttons to have vibrations like[a] rusty spring” (P3, and similarly P4), while others believed pressure added realism as it related to “pressing something with force”(P5). Lastly, P4 added, “they are both suitable for [simulating] abutton”.

CHI ’21, May 08–13, 2021, Yokohama, Japan

Shan-Yuan Teng et al.

Figure 18: (a) The participants tapped on MR interfaces, in which our foldable actuator provided touch feedback. The participants then followed the instructions shown in MR to: (b) fnd out which brake pad contains small scratches, (c) unscrew thebrake pad using the screwdriver and screw a new one, (d) screw a new brake pad using fngers, (e) feel which part of the v-brakeis oily, and fnally (f) apply more lubricant onto it. All shots were taken during the task in user study.
Many participants commented that the foldable actuator was
more realistic for textures (P7, P8, P1, P6). P1 explained that they“can feel the curves on the cardboard” and P6 stated that “frstfeeling contact with a surface and then feel the texture makes more
sense”.
When asked if they felt any movement from the deployment of
the foldable actuator, all participants reported to not have felt
any motion artifacts. P2 and P8 commented that this could havebeen caused by the fact that “[they] were so focused on the objects”.
P6 added that they might have felt some directional pressure, butthat “the feeling became unnoticeable the longer I touched”.

8

USER STUDY #2: INTERACTING WITH
REAL-WORLD OBJECTS AND VIRTUAL
INTERFACES

The objective of our second study was to understand the user experience of using our foldable actuator when engaged in a task thatinvolved both real-world objects and virtual interfaces. Specifcally,
we wanted to interview participants regarding how our device
preserves or encumbers the haptic feedback of the real world.
We asked participants to perform a physical repair task by following instructions depicted in Mixed Reality, which they browsed
by touching the MR interface while wearing our device. This study
was designed to test using our device in conjunction with manipulating virtual interfaces alongside handheld tools (e.g., screwdrivers)and even oily parts, all of which require signifcant dexterity andunimpaired tactile acuity.

8.1

Apparatus

Participants wore our foldable actuator on the index fnger of their
dominant hand and a HoloLens 2 MR headset. We used the headset’s depth camera for fnger tracking. Our Unity3D application
provided a simple MR interactive guide with repair instructions.
We also provided participants with real objects, including two pairsof bicycle V-brakes (detached from the bicycle), a screwdriver, anda bottle of lubricant.

8.2

Task Design

Participants were asked to "fx the brakes by following the MR
instructions". These instructions we comprised of a step-by-stepguide, depicted in Figure 18a. To navigate the next instruction, participants tapped on the mid-air graphics displayed by the HoloLens.
For instance, tapping “next” to proceed to the next step. Every single interaction with the MR interfaces was accompanied by haptic
feedback rendered by our device.
The experimental task involved fve steps: (1) fnd out whichbrake pad needs to be replaced by feeling for any scratches on its
surface; (2) unscrew the brake pad using the screwdriver; (3) screw
a new brake pad onto the V-brake by holding and turning the nut
using the fngers; (4) fnd the oily part on the bicycle brake, (5) put
more oil on the part using the plastic oil bottle. All together, these
sub-tasks account for three interaction types: (1) feeling textures(scratches on brake pad and oil); (2) using handheld tools (screwdriver and bottle); and (3) manipulating small objects (screwing thenut).

Touch&Fold: A Foldable Haptic Actuator for Rendering Touch in Mixed Reality

CHI ’21, May 08–13, 2021, Yokohama, Japan

Figure 19: (a) Participant visually examined the brake pad by holding it using the fnger that our device sits on. (b,c) Participants
used their index fnger to touch the surface of the brake pad. (d) Participant used multiple fngers, including the index fnger,
to feel the oily part. (e) Participant rubbed their index fnger and thumb to test if it felt oily.
Prior to starting the task, we encouraged participants to “thinkaloud”. We recorded participants by means of the HoloLens camera,
which overlays also the MR content and via an external camera.
After the task was completed, we conducted a semi-structured
interview with each participant.

8.3

Participants

We recruited seven participants from our institution (M=25.6 years
old, SD=3.5; four identifed as female, three as male). Four participants had previously experienced an MR headset but not in
conjunction with haptics. Participants were compensated with 20
USD for their time.

8.4

Qualitative Feedback

We present participants’ feedback organized by the three types ofinteraction in our task: (1) feeling textures (scratches on brake padand oil); (2) using handheld tools (screwdriver and bottle); and, (3)manipulating small objects (screw). Lastly, we also present overall
comments about the experience.
8.4.1 Feeling Textures (Brake Pads & Oily Part). First, we asked theparticipants how they distinguished between the two brake pads.
All but one participant (P3) mentioned that they could not visually
distinguish the brakes, so they explored them by touching with
their fnger, which was instrumented with our device. For instance,
P1 added "I looked at them [the brake pads]. They are both blackand it’s hard to distinguish. So I use my fngerpad to feel." While itis unsurprising that most participants used their index fnger (wepurposely added our device on their dominant index fnger to create
this situation), we observed that most participants did not perceive
any impediment caused by our device in feeling the brake pad’s
texture. For instance, P2, P6 and P4 added explicitly "[the device]
did not impede anything." (P2 and similarly P4). Only P5 and P7 felt
the device during this interaction. P7 still used the index fnger tocomplete the task but mentioned that "I worried [the device] would
fall". P5 was the only participant that did not use the index fnger
for this task, remarking "I avoid using index fnger because [of thedevice]".
Next, we asked participants about their experience while feelingwhich part was oily. All participants mentioned that they did not
feel any impediment from our device while feeling the oily part. The
majority (six out of seven) used the index fnger that was wearing
our device and stated, for example, “No, it did not afect me at all, I
could feel the oil easily" (P6, but also similarly P1, P2, P4, P5, P7).

All participants used multiple fngers to feel the oily part. Only P3
did not use the index fnger that wore our device for feeling the oil
but remarked that "I did not think of it, I fnd it more convenient
to feel that with [my] thumb". P4 and P6 even rubbed their index
fnger and thumb to feel the friction while determining if it was
oily (Figure 19e).
8.4.2 Using Handheld Tools (Screwdriver And Lubricant Botle).
First, we asked participants about their experience while manipulating the screwdriver, which requires dexterity from any fnger thatgrips. All participants turned the screwdriver while wearing ourdevice, most of them stating they felt no impediment. For instance,"The device did not afect me when turning the screwdriver." (P2)or "I didn’t even notice the device when I started turning the screwdriver." (P1). P3 mentioned that they felt that "the sides of the device
seem to touch the screwdriver" and we observed them adjusting
their index fnger angle on the grip. P5 added "It worked well! [I]grip it properly [and] it feels fne", adding later, "I can very easily
become used to it". P6 noticed that they raised the index fnger
unconsciously, but put the fnger back on the screwdriver handle
when they need to apply more force, as depicted in Figure 20c.
Next, we asked participants about their experience while manipulating the lubricant bottle, which requires a controlled force tosqueeze the right amount of liquid. Six participants out of seven
used the index fnger that wore our device to squeeze the bottle(usually alongside other fngers as depicted in Figure 20d,e), only
P3 added that they unconsciously did not use that fnger at all.
From the six that used the fnger wearing our device, fve reported
that the device did not interfere with manipulating the bottle inany way. For instance, “ [I] Grip with all fngers without problems(P6) or " it felt smooth when I did the task." (P4). P6 and P7 even
added that they felt that the device did not interfere with their force
control, adding "I slightly squeezed the bottle with my fnger." (P6)and "I squeezed with all my fngers slightly without impediment"(P7). Only P2 refected on a possible impediment, " I noticed that I
sometimes unconsciously raised my index fnger. I guess I was justnot used to it. It was a bit like bandage, so I intuitively didn’t want
to use that [fnger]".
8.4.3 Manipulating A Small Object (Manually Tightening The Nut).
We asked participants about their experiencing while tightening
the small nut by hand, which requires dexterity from any fngerthat grips it. All seven participants performed the task using thefnger that the device was attached to. Five out of seven reported
no difculties nor that the device got in the way. For instance, "TheCHI ’21, May 08–13, 2021, Yokohama, Japan

Shan-Yuan Teng et al.

Figure 20: (a) Participant held the screwdriver ready to turn it. (b) Participant exerted force on the screwdriver using index
fnger when turning it. (c) Participant unconsciously raised their index fnger while holding the screwdriver, they put down
after noticed it. (d) Participant held the oil bottle. (e) Participant slightly squeeze the bottle using all fngers.
device did not get in the way." (P1) or "Not interfering with mygrip." (P5). Only P2 and P4 noted some interference in this task. P2
noted "Because the place to screw is small, I was worried to hit thefngernail device when turning the nut.", and P4 added "Sometimes
the device touched the brake".
8.4.4 Haptics While Touching the MR Repair Guide. All participants
said they always felt the haptic feedback, which our device renderedusing its unfolding mechanism, when tapping any virtual interface,such as “next” or “back” buttons. Unprompted, three participants
added that it felt “pretty satisfying [to get tactile feedback in MR]”
(P6, similarly P1) and “felt natural” (also P6).
8.4.5 General Feedback. Lastly, we let participants add any openended feedback they felt was important. P4 added "Overall, it
doesn’t afect touching things. But the shell sometimes gets incontact with things." P5 added "With more task I might get used
to it". P3 added “I am not used to wearing anything when doing
precision tasks.". P3 also had a larger fngerpad than most users,
and we noted that our device was not custom ft for any user.

9 DISCUSSION
9.1 Limitations of Our Foldable Actuator
First, our device still covers up the user’s fngernail, and obstructs
the side of the fngertip at certain angles. However, in our second user study, we found little perceived impediment for users
manipulating and feeling physical objects.
Second, while our device provides pressure with a slightly tilted
angle due to the compact mechanism design, only one out of our
17 participants realized that. Moreover, as any foldable actuatormoves, it creates inertia, which could generate unwanted tactile
perception. However, our participants never mentioned perceiving
inertial forces; this was likely due to our lightweight cover (3g).
Also, as noted in Figure 8, we found that our mechanical design
resulted in an overshoot of <0.1 N when contacting the fngerpad.
One potential way to mitigate the overshoot would be to add a PID
controller.
Third, we did not fnd that our device was able to realistically simulate a soft surface, which was achieved by stopping the DC motor
as soon as the force sensor detected a contact with the fngerpad. A
more refned approach would include slowing the unfolding mechanism when the cover approaches the fngerpad, which requires aposition encoder with higher resolution.

Figure 21: Example of integrating a Peltier element onto ourfoldable actuator: (a) The Peltier element attached to thecover; and, (b) as the user touches the virtual cofee cup they
feel not only contact but also actual heat.

Finally, with any mechanical haptic device, it has its inherent
latency (92ms), which we currently compensate by enlarging thecollision detector volumes in our MR applications.

9.2

Integrating A Wide Range Of Haptic
Actuators

To expand the expressivity of our device, we can add more actuators
to it. For instance, in Figure 21, we repurpose the LRA driver todrive thermoelectric elements (e.g., Peltier). This allows our device
to render not only the contact with a hot cofee cup but also its
temperature. Optionally, adding a second h-bridge would enable
this Peltier element to be in either hot or cold state. Furthermore,
we believe that other haptic actuators, such as electrodes [49] canbe integrated into our device as well.

10

CONCLUSION & FUTURE WORK

We proposed the frst foldable, nail-mounted haptic device thatprovides tactile feedback to mixed reality (MR) experiences while
quickly tucking away when the user interacts with real-world objects. To achieve this unencumbered haptic feedback, we engineereda wireless haptic device, which measures 24×24×41mm and weighs
9.5g. Furthermore, our foldable end-efector also features a linear
resonant actuator allowing it to render not only touch contacts (i.e.,pressure) but also textures (i.e., vibrations).
We demonstrated how our device renders contact with MR surfaces, buttons, low- and high-frequency textures. In our frst user
study, participants felt that our device provided a more realistic
haptic experience than back-of-the-fnger vibration did when interacting with a variety of objects, such as surfaces, textures, andTouch&Fold: A Foldable Haptic Actuator for Rendering Touch in Mixed Reality

button mechanisms, but not soft objects. In our second user study,
we found that our device preserves the dexterity and haptic perception for manipulating and feeling physical objects, while providing
haptic feedback for virtual interfaces, in MR.
We believe that our device might inspire researchers to explore
new types of foldable haptic designs that optimize their form-factor
and are therefore more suited for MR. Future research might consider how small heating/cooling elements [15] or electrodes [49]
could be included in our foldable cover, and how sensing and actuation can be integrated in one single LRA [11]. Lastly, while ourdevice creates pressure against the fngerpad, it does not explore
skin-stretch [39] or force illusions (by using asymmetric vibrationof its LRA [36]).

ACKNOWLEDGMENTS
We would like to thank our colleague Daniel Steinberg for assisting
us in building a Project North Star headset. Secondly, we would
like to thank Leap Motion for open-sourcing the aforementioned
mixed reality headset and Microsoft for open-sourcing the Mixed
Reality Toolkit. Moreover, we would like to thank the University of
Chicago’s The Center for Data and Computing (CDAC) for theirsupport with the HoloLens headset. Lastly, we sincerely thank ourfunding sources for making this work possible. This work was partially funded by the University of Chicago’s Center for UnstoppableComputing (CERES) and the Sony Research Award program.

REFERENCES[1] Merwan Achibet, Benoît Le Gouis, Maud Marchal, Pierre-Alexandre Léziart,
Ferran Argelaguet, Adrien Girard, Anatole Lécuyer, and Hiroyuki Kajimoto.
2017. FlexiFingers: Multi-fnger interaction in VR combining passive haptics andpseudo-haptics. In 2017 IEEE Symposium on 3D User Interfaces (3DUI), 103–106.
https://doi.org/10.1109/3DUI.2017.7893325[2] Hideyuki Ando, Eisuke Kusachi, and Junji Watanabe. 2007. Nail-mounted Tactile
Display for Boundary/Texture Augmentation. In Proceedings of the International
Conference on Advances in Computer Entertainment Technology (ACE ’07), 292–293.
https://doi.org/10.1145/1255047.1255131[3] Hideyuki Ando, Takeshi Miki, Masahiko Inami, and Taro Maeda. 2002. The
Nail-mounted Tactile Display for the Behavior Modeling. In ACM SIGGRAPH
2002 Conference Abstracts and Applications (SIGGRAPH ’02), 264–264. https://doi.org/10.1145/1242073.1242277[4] Takafumi Aoki, Hironori Mitake, Shoichi Hasegawa, and Makoto Sato. 2009. Haptic ring: touching virtual creatures in mixed reality environments. In SIGGRAPH’09: Posters (SIGGRAPH ’09), 1. https://doi.org/10.1145/1599301.1599401[5] Takumi Azai, Syunsuke Ushiro, Junlin Li, Mai Otsuki, Fumihisa Shibata, andAsako Kimura. 2018. Tap-tap menu: body touching for virtual interactive menus.
In Proceedings of the 24th ACM Symposium on Virtual Reality Software and Technology (VRST ’18), 1–2. https://doi.org/10.1145/3281505.3281561[6] Olivier Bau and Ivan Poupyrev. 2012. REVEL: tactile feedback technology foraugmented reality. ACM Transactions on Graphics 31, 4: 89:1-89:11. https://doi.
org/10.1145/2185520.2185585[7] M. Bouzit, G. Burdea, G. Popescu, and R. Boian. 2002. The Rutgers Master IInew design force-feedback glove. IEEE/ASME Transactions on Mechatronics 7, 2:
256–263. https://doi.org/10.1109/TMECH.2002.1011262[8] Inrak Choi, Heather Culbertson, Mark R. Miller, Alex Olwal, and Sean Follmer.
2017. Grabity: A Wearable Haptic Interface for Simulating Weight and Graspingin Virtual Reality. In Proceedings of the 30th Annual ACM Symposium on User
Interface Software and Technology (UIST ’17), 119–130. https://doi.org/10.1145/
3126594.3126599[9] Inrak Choi and Sean Follmer. 2016. Wolverine: A Wearable Haptic Interface for
Grasping in VR. In Proceedings of the 29th Annual Symposium on User Interface
Software and Technology (UIST ’16 Adjunct), 117–119. https://doi.org/10.1145/
2984751.2985725[10] Kiran Dandekar, Balasundar I. Raju, and Mandayam A. Srinivasan. 2003. 3-D
Finite-Element Models of Human and Monkey Fingertips to Investigate the
Mechanics of Tactile Sense. Journal of Biomechanical Engineering 125, 5: 682–691.
https://doi.org/10.1115/1.1613673[11] Artem Dementyev, Alex Olwal, and Richard F. Lyon. 2020. Haptics with Input: Back-EMF in Linear Resonant Actuators to Enable Touch, Pressure and

CHI ’21, May 08–13, 2021, Yokohama, Japan

Environmental Awareness. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST ’20), 420–429. https://doi.org/10.1145/3379337.3415823[12] Cathy Fang, Yang Zhang, Matthew Dworman, and Chris Harrison. 2020. Wireality:
Enabling Complex Tangible Geometries in Virtual Reality with Worn Multi-String
Haptics. In Proceedings of the 2020 CHI Conference on Human Factors in Computing
Systems (CHI ’20), 1–10. https://doi.org/10.1145/3313831.3376470[13] Yuan-Ling Feng, Charith Lasantha Fernando, Jan Rod, and Kouta Minamizawa.
2017. Submerged haptics: a 3-DOF fngertip haptic display using miniature 3D
printed airbags. 1–2. https://doi.org/10.1145/3084822.3084835[14] Xiaochi Gu, Yifei Zhang, Weize Sun, Yuanzhe Bian, Dao Zhou, and Per Ola
Kristensson. 2016. Dexmo: An Inexpensive and Lightweight Mechanical Exoskeleton for Motion Capture and Force Feedback in VR. 1991–1995. https://doi.org/10.1145/2858036.2858487[15] Teng Han, Fraser Anderson, Pourang Irani, and Tovi Grossman. 2018. HydroRing:
Supporting Mixed Reality Haptics Using Liquid Flow. In Proceedings of the 31st
Annual ACM Symposium on User Interface Software and Technology (UIST ’18),
913–925. https://doi.org/10.1145/3242587.3242667[16] Steven C. Hauser and Gregory J. Gerling. 2016. Measuring tactile cues at thefngerpad for object compliances harder and softer than the skin. IEEE Haptics
Symposium: [proceedings]. IEEE Haptics Symposium 2016: 247–252. https://doi.
org/10.1109/HAPTICS.2016.7463185[17] Steven J. Henderson and Steven Feiner. 2009. Evaluating the benefts of augmented reality for task localization in maintenance of an armored personnel
carrier turret. In 2009 8th IEEE International Symposium on Mixed and Augmented
Reality, 135–144. https://doi.org/10.1109/ISMAR.2009.5336486[18] Anuruddha Hettiarachchi and Daniel Wigdor. 2016. Annexing Reality: Enabling
Opportunistic Use of Everyday Objects as Tangible Proxies in Augmented Reality.
1957–1967. https://doi.org/10.1145/2858036.2858134[19] Ronan Hinchet, Velko Vechev, Herbert Shea, and Otmar Hilliges. 2018. DextrES:
Wearable Haptic Feedback for Grasping in VR via a Thin Form-Factor Electrostatic
Brake. In Proceedings of the 31st Annual ACM Symposium on User Interface Software
and Technology (UIST ’18), 901–912. https://doi.org/10.1145/3242587.3242657[20] Seungwoo Je, Myung Jin Kim, Woojin Lee, Byungjoo Lee, Xing-Dong Yang, Pedro
Lopes, and Andrea Bianchi. 2019. Aero-plane: A Handheld Force-Feedback Device
that Renders Weight Motion Illusion on a Virtual 2D Plane. In Proceedings of the
32nd Annual ACM Symposium on User Interface Software and Technology (UIST’19), 763–775. https://doi.org/10.1145/3332165.3347926[21] Hiroyuki Kajimoto. 2012. Electrotactile Display with Real-Time Impedance Feedback Using Pulse Width Modulation. IEEE Transactions on Haptics 5, 2: 184–188.
https://doi.org/10.1109/TOH.2011.39[22] Hwan Kim, Minhwan Kim, and Woohun Lee. 2016. HapThimble: A Wearable
Haptic Device Towards Usable Virtual Touch Screen. In Proceedings of the 2016
CHI Conference on Human Factors in Computing Systems (CHI ’16), 3694–3705.
https://doi.org/10.1145/2858036.2858196[23] Hwan Kim, HyeonBeom Yi, Hyein Lee, and Woohun Lee. 2018. HapCube: A
Wearable Tactile Device to Provide Tangential and Normal Pseudo-Force Feedback on a Fingertip. In Proceedings of the 2018 CHI Conference on Human Factors
in Computing Systems (CHI ’18), 1–13. https://doi.org/10.1145/3173574.3174075[24] Sunjun Kim and Geehyuk Lee. 2013. Haptic feedback design for a virtual button along force-displacement curves. In Proceedings of the 26th annual ACM
symposium on User interface software and technology (UIST ’13), 91–96. https://doi.org/10.1145/2501988.2502041[25] Robert Kovacs, Eyal Ofek, Mar Gonzalez Franco, Alexa Fay Siu, Sebastian Marwecki, Christian Holz, and Mike Sinclair. 2020. Haptic PIVOT: On-Demand Handhelds in VR. In Proceedings of the 33rd Annual ACM Symposium on User Interface
Software and Technology (UIST ’20), 1046–1059. https://doi.org/10.1145/3379337.
3415854[26] James F. Kramer. 1993. Force feedback and textures simulating interface device.
Retrieved May 4, 2020 from https://patents.google.com/patent/US5184319A/en[27] Shinobu Kuroki, Hiroyuki Kajimoto, Hideaki Nii, Naoki Kawakami, and Susumu
Tachi. 2007. Proposal for tactile sense presentation that combines electrical andmechanical stimulus. In Second Joint EuroHaptics Conference and Symposium
on Haptic Interfaces for Virtual Environment and Teleoperator Systems (WHC’07),
121–126. https://doi.org/10.1109/WHC.2007.92[28] Susan J. Lederman and Lynette A. Jones. 2011. Tactile and Haptic Illusions. IEEE
Transactions on Haptics 4, 4: 273–294. https://doi.org/10.1109/TOH.2011.2[29] David Lindlbauer and Andy D. Wilson. 2018. Remixed Reality: Manipulating
Space and Time in Augmented Reality. In Proceedings of the 2018 CHI Conferenceon Human Factors in Computing Systems (CHI ’18), 129:1-129:13. https://doi.org/
10.1145/3173574.3173703[30] Pedro Lopes and Patrick Baudisch. 2013. Muscle-propelled Force Feedback:Bringing Force Feedback to Mobile Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13), 2577–2580. https://doi.org/10.1145/2470654.2481355[31] Pedro Lopes, Alexandra Ion, and Patrick Baudisch. 2015. Impacto: Simulating
Physical Impact by Combining Tactile Stimulation with Electrical Muscle Stimulation. In Proceedings of the 28th Annual ACM Symposium on User Interface

CHI ’21, May 08–13, 2021, Yokohama, Japan

Software & Technology (UIST ’15), 11–19. https://doi.org/10.1145/2807442.2807443[32] Pedro Lopes, Sijing You, Lung-Pan Cheng, Sebastian Marwecki, and Patrick
Baudisch. 2017. Providing Haptics to Walls & Heavy Objects in Virtual Reality by
Means of Electrical Muscle Stimulation. In Proceedings of the 2017 CHI Conferenceon Human Factors in Computing Systems (CHI ’17), 1471–1482. https://doi.org/10.
1145/3025453.3025600[33] Pedro Lopes, Sijing You, Alexandra Ion, and Patrick Baudisch. 2018. Adding ForceFeedback to Mixed Reality Experiences and Games Using Electrical Muscle Stimulation. In Proceedings of the 2018 CHI Conference on Human Factors in Computing
Systems (CHI ’18), 446:1-446:13. https://doi.org/10.1145/3173574.3174020[34] Aditya Shekhar Nittala, Klaus Kruttwig, Jaeyeon Lee, Roland Bennewitz, Eduard Arzt, and Jürgen Steimle. 2019. Like A Second Skin: Understanding How
Epidermal Devices Afect Human Tactile Perception. In Proceedings of the 2019
CHI Conference on Human Factors in Computing Systems (CHI ’19), 380:1-380:16.
https://doi.org/10.1145/3290605.3300610[35] Claudio Pacchierotti, Stephen Sinclair, Massimiliano Solazzi, Antonio Frisoli, Vincent Hayward, and Domenico Prattichizzo. 2017. Wearable Haptic Systems for the
Fingertip and the Hand: Taxonomy, Review, and Perspectives. IEEE Transactions
on Haptics 10, 4: 580–600. https://doi.org/10.1109/TOH.2017.2689006[36] Jun Rekimoto. 2013. Traxion: a tactile interaction device with virtual force sensation. 427–432. https://doi.org/10.1145/2501988.2502044[37] Stefano Scheggi, Gionata Salvietti, and Domenico Prattichizzo. 2010. Shape andWeight Rendering for Haptic Augmented Reality. 44–49. https://doi.org/10.1109/
ROMAN.2010.5598632[38] Dieter Schmalstieg and Daniel Wagner. 2007. Experiences with Handheld Augmented Reality. 3–18. https://doi.org/10.1109/ISMAR.2007.4538819[39] Samuel B. Schorr and Allison M. Okamura. 2017. Fingertip Tactile Devices for
Virtual Object Manipulation and Exploration. In Proceedings of the 2017 CHI
Conference on Human Factors in Computing Systems (CHI ’17), 3115–3119. https://doi.org/10.1145/3025453.3025744[40] Mike Sinclair, Michel Pahud, and Hrvoje Benko. 2014. TouchMover 2.0 - 3D touchscreen with force feedback and haptic texture. In 2014 IEEE Haptics Symposium(HAPTICS), 1–6. https://doi.org/10.1109/HAPTICS.2014.6775425[41] Anthony Webster Steven, Steven Feiner, Blair Macintyre, William Massie, andTheodore Krueger. 1996. Augmented Reality in Architectural Construction, Inspection, and Renovation.
[42] Paul Strohmeier and Kasper Hornb et al.. 2017. Generating Haptic Textures witha Vibrotactile Actuator. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI ’17), 4994–5005. https://doi.org/10.1145/3025453.
3025812[43] Shan-Yuan Teng, Tzu-Sheng Kuo, Chi Wang, Chi-huan Chiang, Da-Yuan Huang,
Liwei Chan, and Bing-Yu Chen. 2018. PuPoP: Pop-up Prop on Palm for Virtual Reality. In Proceedings of the 31st Annual ACM Symposium on User Interface Software
and Technology (UIST ’18), 5–17. https://doi.org/10.1145/3242587.3242628

Shan-Yuan Teng et al.

[44] Dzmitry Tsetserukou, Katsunari Sato, and Susumu Tachi. 2010. FlexTorque: Exoskeleton Interface for Haptic Interaction with the Digital World. In Proceedingsof the 2010 International Conference on Haptics - Generating and Perceiving Tangible Sensations: Part II (EuroHaptics’10), 166–171. Retrieved July 14, 2019 fromhttp://dl.acm.org/citation.cfm?id=1893760.1893786[45] Scott Varga. HaptX | Haptic gloves for VR training, simulation, and design. HaptX.
Retrieved May 4, 2020 from https://haptx.com/[46] Chiu-Hsuan Wang, Chen-Yuan Hsieh, Neng-Hao Yu, Andrea Bianchi, and Liwei
Chan. 2019. HapticSphere: Physical Support To Enable Precision Touch Interaction in Mobile Mixed-Reality. In 2019 IEEE Conference on Virtual Reality and 3D
User Interfaces (VR), 331–339. https://doi.org/10.1109/VR.2019.8798255[47] Graham Wilson, Tom Carter, Sriram Subramanian, and Stephen Brewster. 2014.
Perception of Ultrasonic Haptic Feedback on the Hand: Localisation and Apparent
Motion. https://doi.org/10.1145/2556288.2557033[48] S. H. Winter and M. Bouzit. 2007. Use of Magnetorheological Fluid in a Force Feedback Glove. IEEE Transactions on Neural Systems and Rehabilitation Engineering
15, 1: 2–8. https://doi.org/10.1109/TNSRE.2007.891401[49] Anusha Withana, Daniel Groeger, and Jürgen Steimle. 2018. Tacttoo: A Thin andFeel-Through Tattoo for On-Skin Tactile Output. In Proceedings of the 31st Annual
ACM Symposium on User Interface Software and Technology (UIST ’18), 365–378.
https://doi.org/10.1145/3242587.3242645[50] Vibol Yem, Ryuta Okazaki, and Hiroyuki Kajimoto. 2016. FinGAR: combinationof electrical and mechanical stimulation for high-fdelity tactile presentation.
In ACM SIGGRAPH 2016 Emerging Technologies (SIGGRAPH ’16), 1–2. https://doi.org/10.1145/2929464.2929474[51] 2020. aestheticinteractive/Hover-UI-Kit. Aesthetic Interactive. Retrieved May 4,2020 from https://github.com/aestheticinteractive/Hover-UI-Kit[52] CyberGrasp. CyberGlove Systems LLC. Retrieved March 8, 2020 from http://www.
cyberglovesystems.com/cybergrasp[53] VIVETM | Discover Virtual Reality Beyond Imagination. Retrieved May 4, 2020
from https://www.vive.com/us/[54] Microsoft HoloLens | Mixed Reality Technology for Business. Retrieved May 4,2020 from https://www.microsoft.com/en-us/hololens[55] Magic Leap 1. Retrieved May 4, 2020 from https://www.magicleap.com/magicleap-1[56] (PDF) Towards multimodal haptics for teleoperation: Design of a tactile thermal
display. ResearchGate. http://dx.doi.org/10.1109/AMC.2012.6197145[57] CyberTouch. CyberGlove Systems LLC. Retrieved March 8, 2020 from http://www.
cyberglovesystems.com/cybertouch[58] Hand interaction examples scene | Mixed Reality Toolkit Documentation. Retrieved September 16, 2020 from https://microsoft.github.io/MixedRealityToolkitUnity/Documentation/README_HandInteractionExamples.html[59] Project North Star. Leap Motion Developer. Retrieved March 30, 2020 from http://developer.leapmotion.com/northstar