Investigating Eyes-away Mid-air Typing in Virtual Reality using
Squeeze haptics-based Postural Reinforcement
Aakar Gupta

Naveen Sendhilnathan

Jess Hartcher-O’Brien

aakar.hci@gmail.com
Meta Inc., Reality Labs Research
Redmond, Washington, USA

naveensn@meta.com
Meta Inc., Reality Labs Research
Redmond, Washington, USAjesshartcher@meta.com
Meta Inc., Reality Labs Research
Redmond, Washington, USA

Evan Pezent

Hrvoje Benko

Tanya R. Jonker

epezent@meta.com
Meta Inc., Reality Labs Research
Redmond, Washington, USA

benko@meta.com
Meta Inc., Reality Labs Research
Redmond, Washington, USA

tanya.jonker@meta.com
Meta Inc., Reality Labs Research
Redmond, Washington, USA

ABSTRACTIn this paper, we investigate postural reinforcement haptics for
mid-air typing using squeeze actuation on the wrist. We propose
and validate eye-tracking based objective metrics that capture theimpact of haptics on the user’s experience, which traditional performance metrics like speed and accuracy are not able to capture.
To this end, we design four wrist-based haptic feedback conditions:no haptics, vibrations on keypress, squeeze+vibrations on keypress,
and squeeze posture reinforcement + vibrations on keypress. We
conduct a text input study with 48 participants to compare the four
conditions on typing and gaze metrics. Our results show that forexpert qwerty users, posture reinforcement haptics significantly
benefit typing by reducing the visual attention on the keyboard by
up to 44% relative to no haptics, thus enabling eyes-away behaviors.

CCS CONCEPTS• Human-centered computing → HCI theory, concepts andmodels; Mixed / augmented reality; Virtual reality.
ACM Reference Format:
Aakar Gupta, Naveen Sendhilnathan, Jess Hartcher-O’Brien, Evan Pezent,
Hrvoje Benko, and Tanya R. Jonker. 2023. Investigating Eyes-away Mid-air
Typing in Virtual Reality using Squeeze haptics-based Postural Reinforcement. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI ’23), April 23–28, 2023, Hamburg, Germany. ACM, New
York, NY, USA, 11 pages. https://doi.org/10.1145/3544548.3581467

1

INTRODUCTION

Freehand qwerty typing on a virtual keyboard floating in air isincreasingly being explored for AR/VR (augmented reality, virtual
reality) [10, 12, 19]. However, a virtual keyboard lacks the haptic
feedback that makes physical keyboard typing an efficient andpleasant task. The physical keyboard enables touch typing where an
expert user feels comfortable enough to avert their gaze away from
Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than theauthor(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or
republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.
CHI ’23, April 23–28, 2023, Hamburg, Germany© 2023 Copyright held by the owner/author(s). Publication rights licensed to ACM.
ACM ISBN 978-1-4503-9421-5/23/04. . . $15.00
https://doi.org/10.1145/3544548.3581467

the keyboard ("eyes-away") and towards where the words are being
typed. The physical keyboard’s haptic assistance can be broken
down into two components: 1) Kinesthetic: the physical surfacesof the keys and the desk provides the user kinesthetic awarenessof their hands relative to the keyboard and helps reinforce theappropriate hand posture. 2) Tactile: the actuation of keys provides
tactile feedback upon key press.
While the work in the space of providing mid-air haptic feedback for such 10-finger qwerty typing is sparse, the most relevant
recent work [19] investigated the tactile component and found thatvibrotactile feedback on the corresponding fingers for key presses
resulted in lower user effort and mental demand. However, thework found the typing performance with tactile feedback to becomparable to when tactile feedback was not provided. The paper explained this mismatch between subjective scores and typing
performance by theorizing that the speeds remained similar because users tended to compensate for the lack of tactile feedback
by increasing their cognitive and visual attention to the task in
the absence of tactile feedback. However, beyond the user quotes,
the paper did not present any objective evidence to support this
explanation.
In this paper, we advance upon the above work in two major
respects:• First, we investigate the addition of squeeze feedback as
a new haptic modality to convey kinesthetic feedback forpostural reinforcement, as well as for keypresses.
• Second, we propose and validate eye-tracking based objective metrics that capture the impact of haptics on the user’s
experience that traditional performance metrics like speed
and accuracy do not.
To this end, we designed three wrist-based haptic feedback conditions: vibrations on keypress, squeeze+vibrations on keypress, andsqueeze posture reinforcement + vibrations on keypress. We conducteda text input study with 48 participants to investigate the effects ofthese three haptics conditions against a baseline condition withno haptics on typing performance and gaze metrics. Our results
show that for expert qwerty users, posture reinforcement haptics
significantly benefited typing by reducing the visual attention on
the keyboard by 44% relative to no haptics, thus enabling moreeyes-away behaviors.

CHI ’23, April 23–28, 2023, Hamburg, Germany

2

Aakar Gupta, Naveen Sendhilnathan, Jess Hartcher-O’Brien, Evan Pezent, Hrvoje Benko, and Tanya R. Jonker

RELATED WORK

Other than the existing work on tactile feedback for mid-air typing described in the Introduction [19], there is sparse work in thespace of mid-air haptic feedback for qwerty-style text-input. Our
discussion of related work focuses on the related threads of text
entry in VR and on wrist haptics for VR.

2.1

Encumbered Text-entry in VR

Encumbered encompasses techniques where the user interacts withexternally grounded devices, such as a physical keyboard, as wellas techniques where the user’s hands are significantly encumberedby controllers or other devices. Current commercial VR devices
use controllers for text-entry where a ray cast from the controllers
is used to select keys on a qwerty keyboard [20, 46]. Prior work
has investigated dedicated handheld devices for VR typing such as
Twiddler [6], 9-key keypads [14], smartphones [23], and bimanual
touchpads with hover detection [43]. Speicher et al. [44] evaluatedmultiple techniques based on current commercial controllers including raycasted pointing, direct tapping, controller as gamepad,
and found raycasted pointing as the fastest with 15.4 words-perminute (WPM). Other studies have investigated non-qwerty layouts
with controllers including circular [14, 51] and cubic [49] layouts.
Multiple glove-based or optical tracking-based techniques havebeen proposed that map a keyboard layout on to the hand/fingers[14, 21, 33, 39], reporting speeds in the range of 5–10 WPM.
The work on physical keyboards in VR can be classified into one
of two categories: variations in visual feedback based on tracking
hand-finger motion in the real world [5, 16, 22, 28] and variationsin hand representations [15, 24, 32]. Most studies in this category
report typing speeds in the range of 25–45 WPM, which are muchhigher than other alternatives in VR. This lends credence to ourapproach of investigating 10-finger typing in VR and whether tactile feedback can help mitigate the problems arising from lack oftangibility in air. For a detailed review of physical keyboards in VR(and VR text-entry in general), we refer the reader to Dube et al.
[9].

2.2

Freehand qwerty Text-entry in VR

Freehand refers to unencumbered techniques that specifically focus
on qwerty keyboard typing using hand tracking. In 2003, ARKB[27] used vision-based tracking of fingertips for multi-finger typing
on a virtual qwerty keyboard. Markussen et al. [30] showed that asingle-finger mid-air vertical keyboard on a large display yields a
speed of 13.2 WPM in its final session, after ∼ 75 mins of practice.
VISAR [11] uses word-level decoding for a single-finger mid-air
vertical keyboard in VR yielding 17.8 WPM after ∼ 90 minutes ofpractice. ATK [50] uses Leap Motion to implement a 10-finger midair horizontal qwerty keyboard supported by a word-level decoder
and reports speeds of 29.2 WPM for a limited vocabulary phraseset after ∼ 1 hour of practice. None of these works examined theinfluence of haptic feedback on typing performance. Dudley et al.
[10] investigate the differences in on-surface vs. mid-air qwerty
keyboard typing in VR and conclude that when using a Wizard of
Oz decoder, the performance of 10-finger on-surface typing (51.6
WPM) is higher than mid-air (34.5 WPM). Wu et al. [47] propose
a glove that provides vibration feedback on the fingertips for a

mid-air qwerty keyboard. However, they do not investigate typing
performance.
Even though the performance reported in prior work is specificto their study design, physical keyboard style text-entry appears
to be one of the most promising techniques for performant typing
in VR. It is freehand, has potentially high speeds, and resembles
physical keyboard typing.

2.3

Wrist Haptics: Vibrations and Squeeze

While there have been multiple works on mid-air haptics pertaining
to gloves (see [34] for a review) and controllers (see [42] for areview), we focus our attention on wrist haptics. Several works have
explored placing multiple vibrotactors on the wrist [7, 8, 18, 31].
Matscheko et al. [31] concluded that placing tactors around thewrist circumference was best. Carcedo et al. [7] found that users
can reliably detect five vibration motors around the wrist.
Squeeze feedback on the wrist has recently received particularinterest in the literature. Pohl et al. [38] used pneumatic actuation
to create uniform compression around the wrist. Zhu et al. [52] built
a pneumatically actuated sleeve that enabled vibrations, squeeze,
and skin stretch. Gupta et al. [17] used shape-memory alloy springs
to overcome size constraints of pneumatic actuation. Pezent et al.
[37] built Tasbi that delivers squeeze using a mechanical string
retraction mechanism and also consists of six vibrotactors. In thispaper, we use the Tasbi device for our investigation.
A related area of work here is haptics guidance which has beenused for motor learning, teleoperation, rehabilitation use cases. Thisa huge area of work with applications in multiple engineering andmedical disciplines. Please refer to [2, 4, 26] for surveys of priorwork in the field.

3

HAPTIC DESIGN

Wrist-worn haptics enables the user to receive haptic feedback close
to the hand while leaving the hands free for interactions. Haptic
components on the fingers may also interfere with the hand tracking driven by headset-based cameras. Prior work on vibrotactile
feedback on fingers [19] used fiducial markers for hand tracking.
In this paper, we focus on wrist-worn haptic feedback and use thenative hand tracking in the Quest VR headset.
While vibrations are good at providing short-term feedback, thesensations can become annoying if played continuously for a longtime. Wrist squeeze feedback at the appropriate strength, on theother hand, can be applied continuously and enables sensations
that are useful for making multisensory experiences with virtual
objects in VR more believeable [37].
To design our haptic feedback conditions, we used the Tasbi
wristband device [37] which enables a wide range of squeeze force
and includes six vibrotactile motors around the band. We prototyped various haptic designs consisting of different combinationsof squeeze and vibrations towards invoking the kinesthetic and tactile components of a typing interaction. We finalized the following
three haptic condition designs, and a baseline condition with nohaptics, for our study:
3.0.1 No haptics (NH). This condition consisted of no haptic feedback. Participants received visual feedback upon key collision (key

Investigating Eyes-away Mid-air Typing in Virtual Reality using Squeeze haptics-based Postural Reinforcement

Figure 1: (left) Apparatus consisting of the Quest2 VR headset
with a Tobii eye tracker inside, and Tasbi squeeze+vibration
wristbands on both hands. (right) Virtual keyboard in VR
turns lighter) and keypress (key turns yellow). Note that the keypress is only registered when the key is pressed up to a certain
predefined depth (0.9cm). If the user collides with a key, but does
not press it all the way, then it is considered a key collision. Overall, the virtual keyboard design Fig 1 adhered close to the virtual
keyboard in prior work [19].
3.0.2 Spatial Wrist Vibration on Keypress (VK). This condition presented vibration feedback for key surface contact and key press
events. Each finger was mapped to an individual vibration actuator in the haptic bracelet (see Section 4.2). The vibration feedback
consisted of a 100 ms decaying sinusoidal signal starting with thecollision of the finger on the key surface until the keypress. Previous
research has successfully used decaying sinusoids to approximatecontact transients, e.g., a finger tapping a surface [25]. For surface
contact the frequency was set to 300 Hz, and for the keypress eventthe frequency was 150 Hz (in the optimum region for the vibration
actuators). If a user stops a fingertip at the location where the key
collision event occurs, the decaying sinusoidal still completes its
100ms feedback. Same applies if a user presses a key slowly. We
made this choice since based on our observation of mid-air typing
dynamics, such cases occurred very infrequently and we therefore
erred on the side of making the keypress feel good when it is performed as expected. The lower frequency content for the keypress
event produces the sensation of a less rigid surface interaction,
getting closer to the sensation of a displacing key.
3.0.3 Squeeze + Spatial Wrist Vibration on Keypress (SVK). Thiscondition extended the VK condition (ii) with squeeze feedback
that was proportional to key displacement. As users collided witha key and progressed towards a keypress, squeeze increased linearly through the key’s range of travel from 0 to MAXSQUEEZE(described later). Thus, squeeze served as a substitute for the springforces once would feel when pressing real keys and was similar to
the mid-air button concept presented in [37]. If multiple fingers
were in simultaneous contact, the total amount of wrist squeeze
was the sum across all contributing fingers (with an upper bound).
The spatial wrist vibration on contact was identical to the one used
in VK. Note that the squeeze feedback here is meant to provide thefeeling of a key in addition to the vibration feedback in VK.
3.0.4 Squeeze for Posture Reinforcement + Spatial Wrist Vibration
on Keypress (SPVK). This condition extended the VK condition(ii) with squeeze feedback that was inversely proportional to the

CHI ’23, April 23–28, 2023, Hamburg, Germany

distance between the user’s wrist location and a pre-determined
optimal wrist location. Squeeze was at its highest value when users
were at the optimal location, and decreased as their wrists driftedaway from this location. The optimal location was set based on
the wrist center point being in the position such that the index
fingers rested on f and j keys. The following equation determined
the squeeze strength S for a distance D (in meters) of the wrist fromthe optimal location:
𝑀𝐴𝑋 _𝑆𝑄𝑈 𝐸𝐸𝑍 𝐸
𝑆=(1)(1 + 𝐷) 300∗𝐷This was obtained based on iterative prototyping and ensured an
initially slow, but subsequently rapid reduction in squeeze strength
as distance increases. Thus, SPVK provided a sustained reinforcement stimuli when users were in the correct posture and tried to
invoke the feedback of resting one’s wrists on a physical surface asis typical for typing. Our hypothesis is that posture reinforcement
will enable the users to be more comfortable in averting their gazeaway from their keyboard since they can be more confident about
the position of their hands and fingers in space.

3.1

Posture Reinforcement vs. Posture
Correction

We also considered providing posture correction feedback where
the squeeze increases as the wrist’s distance from its optimal location increases. We conducted a pilot with four users to compare
reinforcement vs. correction and found that 3 out of 4 users preferred posture reinforcement since it was more analogous to thephysical keyboard.
Note that we don’t have a squeeze condition without vibrations. Since squeeze actuation has higher bulk and power needs,
we wanted to focus on the incremental significance of squeeze inaddition to vibration feedback on the wrist.

4

USER STUDY: EFFECT OF HAPTICFEEDBACK ON MID-AIR TYPINGPERFORMANCE AND GAZE BEHAVIOR
4.1 Participants
48 participants (22 female, 22 male, 2 non-binary and 2 preferred not
to say, age range: 22-65, mean: 39, 3 left handed and 1 ambidextrous)
did the study. Ten of them had a prior experience with VR, but noneof them were habitual users.

4.2

Apparatus

Participants did the study in virtual reality wearing the Quest 2 VR
headset. We fitted a Tobii eye tracker inside the headset to provide
eye tracking data. Quest 2’s native library was used for hand tracking. The Tasbi haptic bracelet [35] was used as the haptic apparatus
for this study. The bracelet was capable of rendering both vibrationand squeeze stimuli. Vibration stimuli were rendered via six linear
resonant actuators (LRA) (MPlus 1040W, 𝑓0 = 170 Hz) spaced evenly
around the circumference of the wrist. Vibration patterns wereauthored using the Syntacts framework [36]. Squeeze stimuli wererendered using a cord tensioning mechanism capable of delivering between 1 N and 12 N of uniform squeeze force. The amountof squeeze stimuli rendered was controlled for using incremental

CHI ’23, April 23–28, 2023, Hamburg, Germany

Aakar Gupta, Naveen Sendhilnathan, Jess Hartcher-O’Brien, Evan Pezent, Hrvoje Benko, and Tanya R. Jonker

encoder feedback and closed-loop position control on the cord tensioning mechanism. Two bracelets were worn by participants, one
on each wrist (see Fig. 1), and were tethered to the primary desktop
through Quanser Q8-USB and MOTU 24 AO hardware interfaces.
Prior to beginning the experiment, the Tasbis were calibrated toparticipants’ wrists so that the initial fit was comfortable and so thata consistent range of squeeze stimuli would be rendered across all
subjects. The calibration procedure has been previously describedin [37] and [35]. The magnitude of the vibrotactile stimuli were notcalibrated. Only the squeeze feedback was calibrated.

4.3

Study Design

Typing on the mid-air virtual keyboard is an unfamiliar experienceand the asymmetric learning effects can be too high for counterbalancing to overcome [19]. We thus adopted a between-subjects
design with 12 participants each in four conditions (NH, VK, SVK,and SPVK). All participants wore the wristband to keep the encumbrance constant. We recorded participants’ physical keyboard
typing speeds using an online test [45].
One of our hypotheses is that haptics will benefit expert qwerty
typists more than novices. This is because even on a physical keyboard, novice typists aren’t able to take their eyes away from the
keyboard as much as more expert users. Without haptics, novices
and experts will have to pay similar attention to the keyboard andtheir key presses. We therefore split the participant set based on
the physical typing median value, <47.5 WPM ("novices") and >47.5
WPM ("experts"). Each of the four conditions had 6 novices and 6
experts. The novice and expert nomenclature in this case, while notentirely accurate, allows us a suitable enough distinction to test out
the differentiating impact of haptics on novice and expert qwerty
users.
To capture the impact of haptics over time, each participant did
4 blocks of typing. Each block consisted of 5 phrases sourced fromthe standard MacKenzie phrase-set [29], for a total of 20 phrases
per participant. We believe that the benefit of haptics will increaseas users’ get more practice on the mid-air keyboard. Typing on the
mid-air virtual keyboard is an unfamiliar experience and involves
learning to deal with issues such as coactivation due to the lackof the physical resistance of the keys [10]. Thus, the user willneed some time to get familiar with mid-air typing with their full
attention before they can start using haptics to lower their attention.
However, building sufficient familiarity with the keyboard can takelonger than a single study session. We therefore enabled a speed
up the learning process by using the same 5 phrases for all 4 blocks.
This allowed the user to get quickly climb their learning curve forthose phrases without making the study too long.

4.4

Procedure

Participants sat on a chair and first wore the band and the headset.
We then calibrated the eye tracker. We did not collect data for participants whose eye tracking results were below the required eye
tracking accuracy threshold. After calibrating the eye tracker foreach user, the eye tracking error was determined based on the average gaze deviations of 5 targets. If the avg. deviation from the centreof the targets was higher than half the size of the target (which wassame as size of key), the participant was removed. We removed twoparticipants based on this criteria. The final participant set after
removing such participants is the balanced 12 participants ×4 conditions = 48 participant set reported above. They were then asked
to place their hands in a comfortable position as if to type on ahorizontal keyboard in air while keeping their shoulders completely
relaxed and their elbows on the armrest so as not to fatigue theirarms. The virtual keyboard was then placed under their fingers
and adjusted according to their preference. Participants were thenasked to complete two practice phrases on the keyboard without
any feedback to get familiar with the typing interface. Participants
were explicitly instructed on the keyboard design and that they
were free to use as many fingers as they want to use. In keepingwith prior unconstrained text-entry evaluations, participants were
instructed to type as quickly and accurately as possible, and thatthey could correct errors (using Backspace) they noticed immediately, but could also choose to ignore errors which they notice aftera few characters have been typed. Participants were then explained
the haptic feedback condition and were asked to try it out for arandom phrase. They then started the first block. Pressing Enter
took them to the next phrase. To prevent inadvertent Enter presses,
the press was only registered if the minimum string edit distance(MSD) of the transcribed phrase from the stimulus phrase was < 8.
Participants were given a 2 min break between blocks. Participants
did a post-study preference and NASA-TLX questionnaire. The participant responses were not under observation during this time tominimize response bias.

4.5

MeasuresWe measured Speed and Uncorrected Error Rate (UER) using standard metrics [3]. Speed is measured in words-per-minute: 𝑊 𝑃𝑀 =((|𝑇 | − 1) × 60)/(𝑆 × 5) where |T| is the transcribed phrase lengthand S is the time starting from the first key press until the last
key press before Enter including time spent in correcting errors.
UER represents the rate of errors that were not corrected by theuser (referred to as incorrect not fixed in [3]). 𝑈 𝐸𝑅 = 𝑀𝑆𝐷 (𝑃,𝑇 ) ×
100/𝑚𝑎𝑥 (|𝑃 |, |𝑇 |) where P is the stimulus phrase. We further analyzed other typing micro-metrics, subjective scores, and gaze behavioral metrics that we discuss in the relevant subsections.

5

RESULTSWe measured the effect of three different haptic conditions (VK, SVK
and SPVK) against the baseline condition (NH) using a betweensubjects study design (N = 48 participants).

5.1

Speed and accuracy did not differ among
haptic conditionsFirst, we investigated the effect of different haptic conditions on
the speed and the error rate of typing in VR. We calculated theaggregate VR typing speed for all four blocks for each of the three
haptic conditions (VK, SVK and SPVK) and the NH baseline condition. Consistent with prior reports [19], VR typing speed was
not affected by presence or absence of haptic feedback (one-way
ANOVA 𝐹 (3, 44) = 0.67; 𝑝 = 0.57, 𝜂 2 = 0.04, Fig 2A). However,there was a practice effect after consistent use of haptics over time.
Participants’ VR typing speed significantly improved for all conditions (including NH baseline condition) after four blocks (NH:Investigating Eyes-away Mid-air Typing in Virtual Reality using Squeeze haptics-based Postural Reinforcement

𝐹 = 5.71𝑝 < .01, rmANOVA for all blocks; 𝑝 < .01 pairwise
Tukey test between first and the last blocks; VK: 𝐹 = 2.97, 𝑝 < .05,
rmANOVA ; 𝑝 < .01 pairwise Tukey test; SVK and SPVK: 𝐹 = 11.29
and 𝐹 = 8.32, 𝑝 < .001, rmANOVA; 𝑝 < .001 pairwise Tukey test;
Fig 2B). Nevertheless, the VR typing speed in the last block forany of the haptic conditions was still not significantly different
from the NH baseline condition in the last block (one-way ANOVA
𝐹 (3, 44) = 1.52; 𝑝 = 0.22, 𝜂 2 = 0.09, Fig 2C).
Furthermore, the uncorrected error rate of typing in VR (UER)
for any of the haptics conditions also did not differ significantly
from the NH baseline for aggregate over all the blocks (one-way
ANOVA 𝐹 (3, 44) = 2.17; 𝑝 = 0.10, 𝜂 2 = 0.13, Fig 3A) or for the last
block alone (one-way ANOVA 𝐹 (3, 44) = 2.28; 𝑝 = 0.09, 𝜂 2 = 0.13,
Fig 3C). We did not observe a significant improvement in error ratewith practice Fig 3B, unlike improvement in speed.
As seen in Fig 2B, the mean speeds for all four haptic conditions
start out at very similar values, with the lines diverging over thecourse of the blocks. While they do not diverge enough to be significantly different as far as speeds are concerned, we observe thetrend in our subsequent metrics as we had earlier speculated in
section 3.1. Therefore, we will only discuss the analysis of the last
block henceforth.

5.2

Key Collisions reduced in SVK

Apart from speed and accuracy, we measured other typing micrometrics such as key collisions per character (a user may perform
multiple incorrect key collisions when trying to hit a key due tothe coactivation problem [10]), and time between consecutive key
presses to gain further insight on the effect of haptics on mid-air
VR typing.
Participants had significantly lower key collisions per key during
the SVK condition, relative to the baseline NH condition (𝑝 <.05 t-test, Fig 4A). This is interesting since this indicates that theadditional squeeze feedback on key press indeed helped the user
avoid unintended collisions.
The time between presses was not different for any of the haptic
conditions(one-way ANOVA 𝐹 (3, 44) = 1.02; 𝑝 = 0.39, 𝜂 2 = 0.06,
Fig 4B).

5.3

Effect of haptics on gaze behavior metrics

As prior work suggests [19], users might compensate for the lackof haptic feedback by increasing their visual attention. Further,
participants’ spatial and temporal gaze patterns along with gaze
metrics are strong correlates of cognitive and motor processes
such as visual attention [13], motor preparation and execution[41], decision making [40], and performance monitoring [1] andtherefore, gaze behavior is a valuable modality for investigating
implicit user behaviors. We thus come up with the following four
hypotheses on the effect of haptics, each pertaining to a different
gaze behavior:• H1: Users will spend less time looking at the keyboard when
there’s haptic feedback than without haptics. Haptics will
help users confirm their keypresses and determine the position of their hand with respect to the keyboard layout.
• H2: With haptic feedback, users’ gaze will spend less timealigned with their finger or with the next key they are about

CHI ’23, April 23–28, 2023, Hamburg, Germany

to press. Users will feel more comfortable not following theirfinger or eyeing the next key when there’s haptics present.
• H3: With haptic feedback, users will switch their gaze lessfrequently between the i/o prompt and the keyboard.
• H4: With haptic feedback, whenever users’ gaze falls overthe keyboard, it will be able to scan the keyboard faster.
5.3.1 Time spent looking at the keyboard. First, we assessed theamount of time the participants spent looking at the keyboard(Figure 1) while typing in VR (again, the analysis is for the last
block). Consistent with this hypothesis, the participants spent 14.5%
and 7.8% less time looking at the keyboard for the VK and SVK
conditions respectively, relative to the NH baseline condition. The
difference however is not statistically significant. The effect of
SPVK is statistically significant where participants spent 23% lesstime looking at the keyboard relative to the NH baseline condition(𝑝 < .05 t-test, Fig 5A).
Furthermore, we found that the amount of time participants
spent looking at the keyboard while typing in VR varied inverselywith their typing speed on a physical qwerty keyboard (expertisewith the qwerty layout). This was specifically true for the SPVK
condition (Fig 5B; Regression model: 𝑦 = −0.59𝑥 + 73.2, where 𝑦 isamount of time looking at the VR keyboard while typing and 𝑥 is thetyping speed in the physical keyboard. The overall regression was
statistically significant 𝑝 < .05; Pearson 𝑟 : −0.65). As previously
mentioned, if a participant were not familiar with the qwerty layout
and has limited expertise typing on a physical keyboard, their levelof expertise typing in VR even with haptics would still be expected
to be limited. Haptics should therefore benefit expert qwerty typists
more.
To validate our reasoning, we analyzed gaze behaviors for novices
and experts separately (See the Appendix for speed and accuracy
graphs for novices and experts). As mentioned earlier, the noviceexpert separation was based on users’ physical keyboard typing
speeds (Fig 6A). Novice typists’ looking behavior during the haptics
conditions was not very different from that of the NH baseline condition (Fig 6B). In fact, with VK and SVK conditions, they looked
13%, and 2% less at the keyboard, relative to the NH baseline. For
expert typists, with VK and SVK conditions, participants looked
15.33% and 13.02% less at the keyboard, relative to the NH baseline
condition. None of these were statistically significant, however.
The SPVK condition was statistically significantly different though,with a huge 44.1% significant reduction in the time spent looking at
the keyboard relative to the NH baseline condition (𝑝 < .01, t-test;Fig 6C).
It is also worth noting that the experts adapted to SPVK condition,over four blocks, much faster than the novice participants (𝐹 =3.25, 𝑝 < .05, rmANOVA and 𝑝 < .01 post-hoc Tukey test, Fig 7).
Together, these results suggest that with SPVK condition, theexpert participants spent significantly less time on the keyboard,
enabling more eyes-away typing behavior during mid-air VR typing,validating H1.
5.3.2 Angular distance between the gaze ray and the finger or next
key. To test H2, we calculated the extent of eye-hand disassociation
by measuring the angular distance between the gaze ray and thefinger / next key. We found that expert participants had 27% higher

All blocks

A 20

B 14

15

Speed (WPM)

speed (WPM)

Aakar Gupta, Naveen Sendhilnathan, Jess Hartcher-O’Brien, Evan Pezent, Hrvoje Benko, and Tanya R. Jonker

10
5
0

NH

SVK (***)SPVK (***)
NH (**)VK (**)

12
10
8
6
0

VK SVK SPVK

Last block

C 20
Speed (WPM)

CHI ’23, April 23–28, 2023, Hamburg, Germany

15
10
5

3
1
2
block number

0

VK SVK SPVKNH

Figure 2: A) Speed of typing across all blocks, all participants. B) Speed of typing by block across all participants. Pairwise
post-hoc tukey test results (after an rm-ANOVA) between the first and the last block for each conditions is shown at the right.
∗ ∗ ∗ means 𝑝 < .001, ∗∗ means 𝑝 < .01. C) Speed of typing for last block only, across all participants.

6
uer

uer

15
10

SPVK

4
NH
VK
SVK

2

5
0

C

B 8

0
NH

VK SVK SPVK0

Last block

20
15

uer

All blocks

A 20

10
5
0

3
1
2
block number

VK SVK SPVKNH

Figure 3: A) Uncorrected Error Rate (UER) across all blocks and all participants. B) UER for each block across all participants.
C) UER for last block only, across all participants.

4
2
0

NH

VK

SVK SPVK

0.8

100

0.6
0.4
0.2
0.0

B

NH

VK

SVK SPVK

Figure 4: A) Collisions per character B) Time between presses.
∗ means 𝑝 < .05.

eye-hand dissociation for SPVK condition relative to the baseline,
but the difference was not significant (𝑃 = .19, t-test, Fig 8A).
5.3.3 Gaze Location Switching. To test H3, we calculated the number of times the participants switched back and forth, normalized

80

*

60
40
20
0

NH

VK SVK SPVK

100
% time spent looking
at keyboard

*

A

1.0

% time spent looking
at keyboard

B

6

Time between presses

Collisions per character

A

80
60

NH (ns)40

SVK (ns)
VK (ns)20

SPVK (*)

0

20

40
60
80
Typing speed inphysical keyboard (WPM)

Figure 5: A) Proportion of time participants spent looking
at the keyboard during typing. B) Dot plot of the proportionof time each participant spent looking at the keyboard by
typing speed, along with regression lines. ∗ means 𝑝 < .05.
Error bars are 95% CI
by duration, between the i/o prompts and the keyboard while typing (Figure 1). We found that the expert participants made 52%Investigating Eyes-away Mid-air Typing in Virtual Reality using Squeeze haptics-based Postural Reinforcement

ExpertB% time spent looking
at keyboard

Frequency

0.03
0.02
0.01
0.00

0

50
100
Typing speed inphysical keyboard (WPM)

C

Novice

100
80
60
40
20
0

NH

Expert100% time spent looking
at keyboard

NoviceA

CHI ’23, April 23–28, 2023, Hamburg, Germany

**

80
60
40
20
0

VK SVK SPVKNH

VK SVK SPVK

Figure 6: A) Distribution of typing speed in physical keyboard for all participants. The participants were split into novice andexpert based on the median split (vertical red line) at 47.5 WPM. B) Proportion of time novice participants spent looking at the
keyboard during typing for each of the three haptic conditions and the NH baseline condition. C) same as B but for expert
participants. ∗∗ means 𝑝 < .01. Error bars are 95% CI.

B

Condition NH

100% Time spentlooking at keyboard

A

80

80
N
E

60
40
20
0

C

Condition VK

100

4
2
3
block number

100

80

60

N
E

40
20

1

D

Condition SVK

100

80

60

60

N
E

40

1

0

4
2
3
block number

N

40

20

0

Condition SPVK

E**

20
1

0

4
2
3
block number

1

4
2
3
block number

Figure 7: Proportion of time novice (broken line) and expert (solid line) participants spent looking at the keyboard during NH(A), VK (B), SVK (C) SPVK (D) conditions for each of the four blocks. ∗∗ means 𝑝 < .01.
ExpertNovice
30

30

20

20

10

10

0

NH

VK SVK SPVK0NH

VK SVK SPVK

B

Number of Switches per sec

Angular distance between
gaze and finger/key (deg)A

Novice

1.5
1.0

1.0

0.5

0.5

0.0

NH

VK SVK SPVK

Expert

1.5

0.0

NH

VK SVK SPVK

Figure 8: A) Angular distance between the gaze and the finger/ next key for novice (left) and expert (right) participants during
typing for each of the three haptic conditions and the NH baseline condition. B) Same as A but for number of gaze switches per
second.
fewer back and forth gaze switches between the i/o prompts andthe keyboard while typing with SPVK conditions (𝑝 = .14 t-test,
Fig 8B).

5.3.4 Gaze Velocity. Finally, we measured participants’ normalized
gaze velocity while they looked at the keyboard and typed. Higher
gaze velocity indicates that whenever the participants’ gaze falls
on the keyboard, they are able to scan the keyboard more quickly.

z-scored gaze velocity

CHI ’23, April 23–28, 2023, Hamburg, Germany

Novice

0.6

Aakar Gupta, Naveen Sendhilnathan, Jess Hartcher-O’Brien, Evan Pezent, Hrvoje Benko, and Tanya R. Jonker

0.4

0.4

0.2

0.2

0.0

NH

VK

SVK SPVK

Expert

0.6

0.0

*

NH

VK

SVK SPVK

UER. This could be because of the speed-accuracy trade-off [48].
That is, with more practice, haptics improved users’ typing speed,since it gets less attentionally and physically demanding, while
potentially compromising typing accuracy.
Finally, the benefit of haptics increased as users got more practiceon the mid-air keyboard (Fig 7). This confirms our assumption thatusers needed some time to get familiar with mid-air typing with
their full attention before they could start using haptics to lower
their attention. Although we see trends in speed, accuracy andgaze metrics with more practice, they did not stabilize during the 4
blocks we studied here. We expect the trends to stabilize over a few
additional blocks and thus a future study to evaluate longer timeeffects of haptics would be highly informative.

Figure 9: Z-scored gaze velocity while participants lookedat the keyboard during typing for novice (left) and expert(right) participants. ∗ means 𝑝 < .05. Error bars are 95% CI

6.1
Expert participants had significantly higher gaze velocity in the
SPVK condition compared to the NH baseline (𝑝 < .05 t-test, Fig 9),
validating H4.

5.4

Subjective scores

While subjective scores in between-participant studies are subject
to uncertainty due to the variation in how different participantsinterpret the scale, we report the scores here. Expert participants
reported significantly lower physical demand with haptics than thebaseline NH condition (𝑝 < .01) (Fig 10). This suggests that haptic
feedback has a meaningful impact on the experience of expert users.

6

Comparison of different haptics conditions

Here we studied the effects of three haptic conditions (VK, SVK and
SPVK) on user performance. Since any wrist device will probably
include vibrotactiles already, here, our purpose was to find theincremental value of squeeze in addition to vibrotactile feedback.
We suspect that incremental value of adding a feedback such as
squeeze for interaction would be one of the important questions
while developing a consumer device and we believe our findings
can inform such decisions.
Further, both VK and SVK have comparable speed and error rates
since the differences are non significant. Therefore, we can’t definitively say much about the speed accuracy comparisons between
them. SPVK in this respect provides a significant improvement.

DISCUSSION

Here we first summarize our key findings and then discuss theirimpact and future work.
First, SPVK benefited the participants the most especially interms of lowering the need for visually attention as demonstrated
by the 44% reduction in the time spent looking at the keyboard by
experts as well as in the angular distance, gaze switching, and gazevelocity metrics (Fig 5, 6, Fig 8A, Fig 8B, Fig 9). This shows that thekinesthetic posture reinforcement component of physical typing
can indeed be provided to a certain extent using squeeze in thismanner. P37: "I felt that the squeeze for reinforcement helped you
stay in the right position to press the keys most accurately."
Second, VK and SVK showed lower visual attention in the gaze
metrics, but the differences were not statistically significant. However, SVK resulted in fewer collisions while mid-air typing. This
indicates that the additional squeeze feedback on key press helped
the user avoid unintended collisions.
Third, haptics benefited expert qwerty typists more than novices(Fig 6, 10). Even on a physical keyboard, novice typists aren’t able
to take their eyes away from the keyboard as much as experts.
Without haptics, novices and experts paid similar attention to the
keyboard and their key presses. As the fidelity of haptic feedback
increases, users hewed closer to their physical behaviors, whichmeant that expert typists benefited from the haptic feedback morethan novices. Furthermore, although more practice with haptics(specifically, SPVK condition) promoted eyes-away behavior inexpert users, it also led to slight but non-significant increase in

6.2

Impact of Haptics: Participant Variations

The difference in novice and expert typists can be explained fromthe physical keyboard behaviors. If a novice user can’t take theireyes away from a physical keyboard, they won’t be able to do sowith mid-air haptics regardless of how good it may be.
We observe that there are multiple metrics in our results where
the means appear to be quite lower or higher, but do not end up
being significant. While this may partially be due to the number ofparticipants, it also indicates inter-participant variability even after
splitting novice and experts. While some participants found thefeedback to be useful, some did not like the vibration or squeeze
sensations and tried to ignore them. Thus there’s a component of
individual preference to the impact of haptics beyond just the task
(typing) expertise. The effect of SPVK is strong enough to overcome
this variability for certain metrics, but that is not the case for VK
and SVK. While there may be a multitude of reasons behind a user
preferring haptic feedback or not, one way we can improve thisin our system is to personalize the squeeze and vibration ranges
based on the user’s perceptual sensitivity and comfort. While wecalibrated the wristband for each participant’s wrist size, someparticipants did report a feeling of discomfort over time. Personalization for individual sensitivity can make the feedback more
pleasant and make the user more amenable to use the information
conveyed by it.

Investigating Eyes-away Mid-air Typing in Virtual Reality using Squeeze haptics-based Postural Reinforcement

CHI ’23, April 23–28, 2023, Hamburg, Germany

0 NH VK SVK SPVK

20 NH VK SVK SPVK*

42
0 NH VK SVK SPVK

64
20 NH VK SVK SPVK

6

**
* **

4
2
0 NH VK SVK SPVK

Mental Demand

2

4

6

Phhysical Demand

4

6

Confidence

6

ExpertMental Demand

Phhysical Demand

Confidence

Novice642
0 NH VK SVK SPVK

Figure 10: Participant subjective score box plots for confidence, physical demand and mental demand for novice (left) andexperts (right). ∗ means 𝑝 < .05 and ∗∗ means 𝑝 < .01.

6.3

Gaze Metrics as Objective Evidence for the
Impact of Haptics

This paper provides evidence of the utility of using gaze behavioral
metrics as objective measures of the impact of haptics on user interactions. This is especially true in scenarios where the impact of
haptics is not captured in the traditional task performance metrics,
but is subjectively felt when using it. Gaze metrics can thus help
product designers weigh pros and cons of including different haptic
capabilities in future devices more empirically. While some our metrics are specific to typing, others such as gaze velocity and angulardistance can potentially be adapted for use in other interactions
such as grabbing, pointing, manipulation etc.

6.4

Posture Reinforcement Haptics forEyes-away Interactions

Providing physical resistance haptics in mid-air akin to physicalobjects is a hard problem. A crucial outcome of physical resistanceis that the user’s hand is inherently holds its place relative to that
object. Thus posture reinforcement haptics even without physicalresistance, as we demonstrate, can be useful in providing this pieceof information to the user helping them maintain their hand posture.
It is crucial however for such feedback to be not annoying overlong durations. Future work in this space can explore other haptic
modalities such as pneumatic and glove approaches which may beable to provide even more nuanced forms of reinforcement.

7

CONCLUSION

Our work is the first investigation of the value of remote wrist
posture reinforcement haptic feedback for mid-air text input in VR. Our results suggest that while haptic feedback does not resultin significant improvements in user speed and accuracy, posturereinforcement haptics resulted in a significant difference in termsof their visual attention on the keyboard. One potential reason forthis trend is that in the absence of haptic feedback users use theirvisual and cognitive attention more, thus maintaining the same
performance but expending more effort. This shows that the valueof haptic feedback needs to be measured by going beyond traditional
performance metrics. Our work conclusively demonstrates the useof gaze metrics as objective evidence for the impact of haptics inthe task. We believe haptic feedback is a crucial component for
text-input in VR and hope that our work on posture reinforcement
feedback and on gaze metrics serves as a guide and impetus forresearchers and product designers.

ACKNOWLEDGMENTS
REFERENCES[1] [n.d.].
[2] Adilzhan Adilkhanov, Matteo Rubagotti, and Zhanat Kappassov. 2022. Haptic
Devices: Wearability-Based Taxonomy and Literature Review. IEEE Access (2022).
[3] A. S. Arif and W. Stuerzlinger. 2009. Analysis of text entry performance metrics.
In 2009 IEEE Toronto International Conference Science and Technology for Humanity(TIC-STH). 100–105. https://doi.org/10.1109/TIC-STH.2009.5444533[4] Carlos Bermejo and Pan Hui. 2017. A survey on haptic technologies for mobile
augmented reality. arXiv preprint arXiv:1709.00698 (2017).
[5] Sidney Bovet, Aidan Kehoe, Katie Crowley, Noirin Curran, Mario Gutierrez,
Mathieu Meisser, Damien O Sullivan, and Thomas Rouvinez. 2018. Using
Traditional Keyboards in VR: SteamVR Developer Kit and Pilot Game User
Study. In 2018 IEEE Games, Entertainment, Media Conference (GEM). IEEE, 1–
9. https://doi.org/10.1109/GEM.2018.8516449[6] Doug A. Bowman, Christopher J. Rhoton, and Marcio S. Pinho. 2002. Text Input
Techniques for Immersive Virtual Environments: An Empirical Comparison.
Proceedings of the Human Factors and Ergonomics Society Annual Meeting 46, 26(sep 2002), 2154–2158. https://doi.org/10.1177/154193120204602611[7] Marta G. Carcedo, Soon Hau Chua, Simon Perrault, Paweł Wozniak, Raj Joshi,
Mohammad Obaid, Morten Fjeld, and Shengdong Zhao. 2016. HaptiColor: Interpolating Color Information as Haptic Feedback to Assist the Colorblind. In
Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems(San Jose, California, USA) (CHI ’16). Association for Computing Machinery, New
York, NY, USA, 3572–3583. https://doi.org/10.1145/2858036.2858220[8] Hsiang-Yu Chen, Joseph Santos, Matthew Graves, Kwangtaek Kim, and Hong Z.
Tan. 2008. Tactor Localization at the Wrist. In Haptics: Perception, Devices andScenarios, Manuel Ferre (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg,
209–218.
[9] Tafadzwa Joseph Dube and Ahmed Sabbir Arif. 2019. Text Entry in Virtual
Reality: A Comprehensive Review of the Literature. Springer, Cham, 419–437.
https://doi.org/10.1007/978-3-030-22643-5_33[10] J. Dudley, H. Benko, D. Wigdor, and P. O. Kristensson. 2019. Performance
Envelopes of Virtual Keyboard Text Input Strategies in Virtual Reality. In 2019
IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 289–300.
https://doi.org/10.1109/ISMAR.2019.00027[11] John J. Dudley, Keith Vertanen, and Per Ola Kristensson. 2018. Fast and Precise
Touch-Based Text Entry for Head-Mounted Augmented Reality with VariableOcclusion. ACM Transactions on Computer-Human Interaction 25, 6 (dec 2018),
1–40. https://doi.org/10.1145/3232163[12] Conor R. Foy, John J. Dudley, Aakar Gupta, Hrvoje Benko, and Per Ola Kristensson.
2021. Understanding, Detecting and Mitigating the Effects of Coactivations in
Ten-Finger Mid-Air Typing in Virtual Reality. In Proceedings of the 2021 CHI
Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI ’21).
Association for Computing Machinery, New York, NY, USA, Article 287, 11 pages.
https://doi.org/10.1145/3411764.3445671

CHI ’23, April 23–28, 2023, Hamburg, Germany

Aakar Gupta, Naveen Sendhilnathan, Jess Hartcher-O’Brien, Evan Pezent, Hrvoje Benko, and Tanya R. Jonker[13] Alexandra Frischen, Andrew P Bayliss, and Steven P Tipper. 2007. Gaze cueing of attention: visual attention, social cognition, and individual differences.
Psychological bulletin 133, 4 (2007), 694.
[14] Gabriel González, José P. Molina, Arturo S. García, Diego Martínez, and Pascual
González. 2009. Evaluation of Text Input Techniques in Immersive Virtual Environments. In New Trends on Human-Computer Interaction. Springer London,
London, 1–10. https://doi.org/10.1007/978-1-84882-352-5_11[15] Jens Grubert, Lukas Witzani, Eyal Ofek, Michel Pahud, Matthias Kranz, and
Per Ola Kristensson. 2018. Effects of Hand Representations for Typing in Virtual
Reality. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).
IEEE, 151–158. https://doi.org/10.1109/VR.2018.8446250[16] Jens Grubert, Lukas Witzani, Eyal Ofek, Michel Pahud, Matthias Kranz, and
Per Ola Kristensson. 2018. Text Entry in Immersive Head-Mounted DisplayBased Virtual Reality Using Standard Keyboards. In 2018 IEEE Conference on
Virtual Reality and 3D User Interfaces (VR). IEEE, 159–166. https://doi.org/10.
1109/VR.2018.8446059[17] Aakar Gupta, Antony Albert Raj Irudayaraj, and Ravin Balakrishnan. 2017. HapticClench: Investigating Squeeze Sensations Using Memory Alloys. In Proceedingsof the 30th Annual ACM Symposium on User Interface Software and Technology(Québec City, QC, Canada) (UIST ’17). Association for Computing Machinery,
New York, NY, USA, 109–117. https://doi.org/10.1145/3126594.3126598[18] Aakar Gupta, Thomas Pietrzak, Nicolas Roussel, and Ravin Balakrishnan. 2016.
Direct Manipulation in Tactile Displays. In Proceedings of the 2016 CHI Conferenceon Human Factors in Computing Systems (San Jose, California, USA) (CHI ’16).
Association for Computing Machinery, New York, NY, USA, 3683–3693. https://doi.org/10.1145/2858036.2858161[19] Aakar Gupta, Majed Samad, Kenrick Kin, Per Ola Kristensson, and Hrvoje Benko.
2020. Investigating remote tactile feedback for mid-air text-entry in virtual reality.
In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR).
IEEE, 350–360.
[20] HTC. 2019. Vive. https://www.vive.com[21] Haiyan Jiang, Dongdong Weng, Zhenliang Zhang, Feng Chen, Haiyan Jiang,
Dongdong Weng, Zhenliang Zhang, and Feng Chen. 2019. HiFinger: One-Handed
Text Entry Technique for Virtual Environments Based on Touches between
Fingers. Sensors 19, 14 (jul 2019), 3063. https://doi.org/10.3390/s19143063[22] Sooyoung Kim and Gerard Jounghyun Kim. 2004. Using keyboards with head
mounted displays. In Proceedings of the 2004 ACM SIGGRAPH international conference on Virtual Reality continuum and its applications in industry - VRCAI ’04. ACM
Press, New York, New York, USA, 336. https://doi.org/10.1145/1044588.1044662[23] Youngwon R. Kim and Gerard J. Kim. 2016. HoVR-type: smartphone as a typing
interface in VR using hovering. In Proceedings of the 22nd ACM Conference on
Virtual Reality Software and Technology - VRST ’16. ACM Press, New York, New
York, USA, 333–334. https://doi.org/10.1145/2993369.2996330[24] Pascal Knierim, Valentin Schwind, Anna Maria Feit, Florian Nieuwenhuizen, andNiels Henze. 2018. Physical Keyboards in Virtual Reality. In Proceedings of the2018 CHI Conference on Human Factors in Computing Systems - CHI ’18. ACM
Press, New York, New York, USA, 1–9. https://doi.org/10.1145/3173574.3173919[25] Katherine Kuchenbecker, Jonathan Fiene, and Gunter Niemeyer. 2006. Improving contact realism through event-based haptic feedback. IEEE transactions on
visualization and computer graphics 12, 2 (2006), 219–230.
[26] Jaebong Lee and Seungmoon Choi. 2010. Effects of haptic guidance and disturbance on motor learning: Potential advantage of haptic disturbance. In 2010 IEEE
Haptics Symposium. IEEE, 335–342.
[27] Minkyung Lee and Woontack Woo. 2003. ARKB: 3D vision-based Augmented
Reality Keyboard.. In International Conferece on Artificial Reality and Telexisitence.
Tokyo, Japan, 54–57. http://www.vrsj.org/ic-at/ICAT2003/php/upload/153_
camera_38251238b626ed52ab0344b67e112d3d.pdf[28] Jia-Wei Lin, Ping-Hsuan Han, Jiun-Yu Lee, Yang-Sheng Chen, Ting-Wei Chang,Kuan-Wen Chen, and Yi-Ping Hung. 2017. Visualizing the keyboard in virtual
reality for enhancing immersive experience. In ACM SIGGRAPH 2017 Posters on SIGGRAPH ’17. ACM Press, New York, New York, USA, 1–2. https://doi.org/10.
1145/3102163.3102175[29] I. Scott MacKenzie and R. William Soukoreff. 2003. Phrase Sets for Evaluating Text
Entry Techniques. In CHI ’03 Extended Abstracts on Human Factors in Computing
Systems (Ft. Lauderdale, Florida, USA) (CHI EA ’03). ACM, New York, NY, USA,754–755. https://doi.org/10.1145/765891.765971[30] Anders Markussen, Mikkel R. Jakobsen, and Kasper Hornbæk. 2013. SelectionBased Mid-Air Text Entry on Large Displays. Springer, Berlin, Heidelberg, 401–
418. https://doi.org/10.1007/978-3-642-40483-2_28[31] Michael Matscheko, Alois Ferscha, Andreas Riener, and Manuel Lehner. 2010. Tactor placement in wrist worn wearables. In International Symposium on Wearable
Computers (ISWC) 2010. 1–8. https://doi.org/10.1109/ISWC.2010.5665867[32] Mark McGill, Daniel Boland, Roderick Murray-Smith, and Stephen Brewster.
2015. A Dose of Reality: Overcoming Usability Challenges in VR Head-Mounted
Displays. In Proceedings of the 33rd Annual ACM Conference on Human Factors in
Computing Systems - CHI ’15. ACM Press, New York, New York, USA, 2143–2152.
https://doi.org/10.1145/2702123.2702382[33] Taihei Ogitani, Yoshitaka Arahori, Yusuke Shinyama, and Katsuhiko Gondow.
2018. Space Saving Text Input Method for Head Mounted Display with Virtual 12key Keyboard. In 2018 IEEE 32nd International Conference on Advanced Information
Networking and Applications (AINA). IEEE, 342–349. https://doi.org/10.1109/
AINA.2018.00059[34] J. Perret and E. Vander Poorten. 2018. Touching Virtual Reality: A Review ofHaptic Gloves. In ACTUATOR 2018; 16th International Conference on New Actuators.
1–5.
[35] Evan Pezent, Priyanshu Agarwal, Jessica Hartcher-O’Brien, Nicholas Colonnese,and Marcia K. O’Malley. 2022. Design, Control, and Psychophysics of Tasbi:
A Force-Controlled Multimodal Haptic Bracelet. IEEE Transactions on Robotics(2022), 1–17. https://doi.org/10.1109/TRO.2022.3164840[36] Evan Pezent, Brandon Cambio, and Marcia K. O’Malley. 2021. Syntacts: OpenSource Software and Hardware for Audio-Controlled Haptics. IEEE Transactions
on Haptics 14, 1 (2021), 225–233. https://doi.org/10.1109/TOH.2020.3002696[37] E. Pezent, A. Israr, M. Samad, S. Robinson, P. Agarwal, H. Benko, and N. Colonnese.
2019. Tasbi: Multisensory Squeeze and Vibrotactile Wrist Haptics for Augmented
and Virtual Reality. In 2019 IEEE World Haptics Conference (WHC). 1–6. https://doi.org/10.1109/WHC.2019.8816098[38] Henning Pohl, Peter Brandes, Hung Ngo Quang, and Michael Rohs. 2017. Squeezeback: Pneumatic Compression for Notifications. In Proceedings of the 2017 CHI
Conference on Human Factors in Computing Systems (Denver, Colorado, USA)(CHI ’17). Association for Computing Machinery, New York, NY, USA, 5318–5330.
https://doi.org/10.1145/3025453.3025526[39] Manuel Prätorius, Dimitar Valkov, Ulrich Burgbacher, and Klaus Hinrichs. 2014.
DigiTap: an eyes-free VR/AR symbolic input device. In Proceedings of the 20th
ACM Symposium on Virtual Reality Software and Technology - VRST ’14. ACM
Press, New York, New York, USA, 9–18. https://doi.org/10.1145/2671015.2671029[40] Thomas R Reppert, Karolina M Lempert, Paul W Glimcher, and Reza Shadmehr.
2015. Modulation of saccade vigor during value-based decision making. Journalof Neuroscience 35, 46 (2015), 15369–15378.
[41] Naveen Sendhilnathan, Debaleena Basu, Michael E Goldberg, Jeffrey D Schall,and Aditya Murthy. 2021. Neural correlates of goal-directed and non–goaldirected movements. Proceedings of the National Academy of Sciences 118, 6(2021), e2006372118.
[42] Mike Sinclair, Eyal Ofek, Mar Gonzalez-Franco, and Christian Holz. 2019. CapstanCrunch: A Haptic VR Controller with User-Supplied Force Feedback. In
Proceedings of the 32nd Annual ACM Symposium on User Interface Software andTechnology (New Orleans, LA, USA) (UIST ’19). Association for Computing Machinery, New York, NY, USA, 815–829. https://doi.org/10.1145/3332165.3347891[43] Jeongmin Son, Sunggeun Ahn, Sunbum Kim, and Geehyuk Lee. 2019. Improving
Two-Thumb Touchpad Typing in Virtual Reality. In Extended Abstracts of the2019 CHI Conference on Human Factors in Computing Systems - CHI EA ’19. ACM
Press, New York, New York, USA, 1–6. https://doi.org/10.1145/3290607.3312926[44] Marco Speicher, Anna Maria Feit, Pascal Ziegler, and Antonio Krüger. 2018.
Selection-based Text Entry in Virtual Reality. In Proceedings of the 2018 CHI
Conference on Human Factors in Computing Systems - CHI ’18. ACM Press, New
York, New York, USA, 1–13. https://doi.org/10.1145/3173574.3174221[45] Typing.com. 2022. Typing Test. https://www.typing.com/student/typing-test/1minute[46] UploadVR. 2019. Oculus Claims Breakthrough in Hand-tracking Accuracy. https://www.roadtovr.com/oculus-claims-breakthrough-in-hand-tracking-accuracy/[47] Chien-Min Wu, Chih-Wen Hsu, Tzu-Kuei Lee, and Shana Smith. 2017. A virtual reality keyboard with realistic haptic feedback in a fully immersive virtual environment. Virtual Reality 21, 1 (01 Mar 2017), 19–29. https://doi.org/10.1007/s10055016-0296-6[48] Motonori Yamaguchi, Matthew JC Crump, and Gordon D Logan. 2013. Speed–
accuracy trade-off in skilled typewriting: Decomposing the contributions ofhierarchical control loops. Journal of Experimental Psychology: Human Perception
and Performance 39, 3 (2013), 678.
[49] Naoki Yanagihara and Buntarou Shizuki. 2018. Cubic Keyboard for Virtual Reality.
In Proceedings of the Symposium on Spatial User Interaction - SUI ’18. ACM Press,
New York, New York, USA, 170–170. https://doi.org/10.1145/3267782.3274687[50] Xin Yi, Chun Yu, Mingrui Zhang, Sida Gao, Ke Sun, and Yuanchun Shi. 2015. ATK:
Enabling Ten-Finger Freehand Typing in Air Based on 3D Hand Tracking Data.
In Proceedings of the 28th Annual ACM Symposium on User Interface Software &Technology - UIST ’15. ACM Press, New York, New York, USA, 539–548. https://doi.org/10.1145/2807442.2807504[51] Difeng Yu, Kaixuan Fan, Heng Zhang, Diego Monteiro, Wenge Xu, and HaiNing Liang. 2018. PizzaText: Text Entry for Virtual Reality Systems Using Dual
Thumbsticks. IEEE Transactions on Visualization and Computer Graphics 24, 11(nov 2018), 2927–2935. https://doi.org/10.1109/TVCG.2018.2868581[52] Mengjia Zhu, Amirhossein H. Memar, Aakar Gupta, Majed Samad, Priyanshu
Agarwal, Yon Visell, Sean J. Keller, and Nicholas Colonnese. 2020. PneuSleeve:
In-Fabric Multimodal Actuation and Sensing in a Soft, Compact, and Expressive
Haptic Sleeve. In Proceedings of the 2020 CHI Conference on Human Factors in
Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing
Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3313831.3376333

Investigating Eyes-away Mid-air Typing in Virtual Reality using Squeeze haptics-based Postural Reinforcement

10
5
0

NH

VK SVK SPVK

Novice

20
15

15

10

10

10

5
0

5
NH

VK SVK SPVK

Figure A1: Speed of typing (in WPM) for the last block for
novice and expert participants.

0

Expert

20

15

uer

Speed (WPM)

Speed (WPM)

15

Expert20

uer

Novice20

CHI ’23, April 23–28, 2023, Hamburg, Germany

5
NH

VK SVK SPVK0NH

VK SVK SPVK

Figure A2: UER for the last block for novice and expert participants.