Understanding, Detecting and Mitigating the Efects ofCoactivations in Ten-Finger Mid-Air Typing in Virtual Reality
Conor R. Foy, John J. Dudley

Aakar Gupta, Hrvoje Benko

Per Ola Kristensson

Department of Engineering
University of Cambridge
Cambridge, United Kingdom

Facebook Reality Labs
Seattle, Washington, USA

Department of Engineering
University of Cambridge
Cambridge, United Kingdom

ABSTRACT
Typing with ten fngers on a virtual keyboard in virtual or augmented reality exposes a challenging input interpretation problem.
There are many sources of noise in this interaction context andthese exacerbate the challenge of accurately translating human actions into text. A particularly challenging input noise source arises
from the physiology of the hand. Intentional fnger movements can
produce unintentional coactivations in other fngers. On a physical keyboard, the resistance of the keys alleviates this issue. On
a virtual keyboard, coactivations are likely to introduce spurious
input events under a naïve solution to input detection. In this paper
we examine the features that discriminate intentional activations
from coactivations. Based on this analysis, we demonstrate threealternative coactivation detection strategies with high discrimination power. Finally, we integrate coactivation detection into a
probabilistic decoder and demonstrate its ability to further reduce
uncorrected character error rates by approximately 10% relative
and 0.9% absolute.

CCS CONCEPTS• Human-centered computing → Virtual reality; Text input.

KEYWORDS
virtual reality, text entry
ACM Reference Format:
Conor R. Foy, John J. Dudley, Aakar Gupta, Hrvoje Benko, and Per Ola
Kristensson. 2021. Understanding, Detecting and Mitigating the Efectsof Coactivations in Ten-Finger Mid-Air Typing in Virtual Reality. In CHI
Conference on Human Factors in Computing Systems (CHI ’21), May 8–13,
2021, Yokohama, Japan. ACM, New York, NY, USA, 11 pages. https://doi.org/
10.1145/3411764.3445671

1

INTRODUCTION

The freedom aforded by modern virtual and augmented reality
headsets providing egocentric hand tracking encourages the exploration of efective bare-handed mid-air text entry methods. Ten
fnger typing is a familiar and efcient text entry strategy in conventional computing environments and with further hand tracking

This work is licensed under a Creative Commons Attribution International
4.0 License.
CHI ’21, May 8–13, 2021, Yokohama, Japan© 2021 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-8096-6/21/05.
https://doi.org/10.1145/3411764.3445671

improvements may become feasible on virtual and augmented reality headsets. Mid-air typing with ten fngers sufers, however, fromthe lack of a fxed physical reference frame and passive resistanceto key presses. This fundamental distinction from typing on a physical keyboard exposes the nontrivial challenge of discriminating
intended fnger movements from those produced by coupled motionof the hand and fngers. In this paper, we examine the features that
distinguish intentional fnger movements from unintentional fnger
movements in mid-air typing. For conciseness, we subsequently
refer to these unintentional fnger movements as coactivations.
A simple and highly extensible strategy for providing a virtual
keyboard is the virtualization of the input surface. Intersections
between the tracked fngertips and this virtual surface can thus betreated as input observations. Under this detection strategy, however, coactivations that intersect with the input surface introduce
spurious observations. Unintentional observations that insert corresponding characters into the input feld are frustrating to users
and likely to yield high error rates.
The input detection strategy based on intersections with a virtualsurface (described above) is simple and efective but sufers morefrom coactivations than other potential strategies. A deep neuralnetwork might, for example, be trained to decode tracked fnger
movements directly into characters or words and learn to ignore
fnger coactivations. This approach is, however, less extensible thanthe virtualized input surface and would necessitate large amounts
of user data which is currently difcult to collect and not widely
available. In this paper, we present the design of a novel softwaremodule capable of detecting coactivations based on interpretable
dynamic features of fnger motion while typing. This design strategy ensures that the proposed technique has utility beyond theconfnes of the specifc keyboard confguration examined. Further,
the module is readily integrated into a probabilistic decoding framework based on the virtual input surface paradigm to complement
deletion assessments for input observations in the decoder.
To better understand coactivations and evaluate detection strategies, we leverage a dataset previously collected by Dudley et al. [6]of 24 users typing on a mid-air virtual keyboard in virtual reality(VR) with high-precision fnger tracking. We mine this dataset todevelop an understanding of the causal factors and distributions
related to coactivations. This understanding supports the identifcation of the dominant features that distinguish coactivations from
intended fnger movements. The dataset is subsequently partitioned
to evaluate the performance of three alternative detection strategies
in terms of detection accuracy. Finally, to demonstrate the utility ofthe developed technique we integrate coactivation detection functionality into a probabilistic input decoding framework. The key
contributions of this paper are therefore:

CHI ’21, May 8–13, 2021, Yokohama, Japan(1) A characterization of coactivations as encountered in ten
fnger mid-air typing in virtual reality.
(2) An exposition of the features that distinguish coactivations
from intentional fnger movements in this setting.
(3) Three coactivation detection strategies based on the identifed features, evaluated on the partitioned dataset.
(4) An integration of coactivation detection into a statistical
decoder and a demonstration of the beneft coactivation
detection can provide in reducing character error rates.

2

RELATED WORKThis work is informed by prior research eforts exploring mid-air
text entry and other more general studies examining synergistic
movements of the hand. These two areas are briefy reviewed in
the following subsections.

2.1

Mid-Air Text Entry

There have been many mid-air text entry strategies proposed andevaluated but no dominant method has emerged [4, 7, 10, 12]. Bowman et al. [4] evaluated four diferent text input techniques for
immersive virtual environments: speech, glove, pen and chording keyboard approaches achieving 13 wpm1 , 6 wpm, 10 wpm and
4 wpm respectively. Bowman et al. [4]’s work serves to highlightthat diferent techniques ofer certain trade-ofs in terms of speed
and accuracy, and that the most appropriate choice may depend oncontext.
The virtualization of the keyboard opens up new opportunities
in terms of exploring exotic key layouts [22, 31] and interactions[3, 14, 19, 21, 27]. However, the Qwerty layout and touch interaction remain the de-facto standard in modern computing andit is highly desirable to leverage this familiarity where possible.
ATK [30] demonstrates the potential (29 wpm) of ten-fnger mid-air
typing with fngers tracked using a leap motion. Dudley et al. [6] investigated the performance limits of typing on a virtual keyboard in
VR. Users were observed to have signifcant difculty in efectively
utilizing all ten fngers in this setting [6]. Nevertheless, participants
achieved notional entry rates in the order of 30 to 35 wpm on a
mid-air virtual keyboard with ten fngers, albeit with relatively higherror rates [6]. These results suggest that traditional designs fail tofully support the user in ten fnger typing on a virtual keyboard.
Others have tackled related mid-air text entry scenarios, such assupporting input on large wall displays [18, 19, 25]. However, suchcontexts rarely demand prolonged or high rate text input.
The new capabilities aforded by virtual keyboards also expose
new challenges. For example, mid-air text entry can quickly become
uncomfortable due to the need to maintain the arm cantilevered inspace. Yanagihara et al. [29] describe a curved virtual keyboard forword-gesture text entry, in part designed to reduce fatigue. Anotherimportant consideration exposed in VR is how best to represent
hands and fngertips to support text entry [11, 17]. These studies
and the broader related work covered in this section highlight that
supporting efcient and enjoyable text entry in VR is nontrivial.
An efective solution, particularly when all ten fngers are engaged,
1 wpm = words per minute, with a word defned as fve consecutive characters including

space.

Foy et al.

hinges on being able to mitigate the aspects that make a virtual
keyboard distinct from a physical keyboard.

2.2

Unintended Finger Movements in Typing

Our hands are capable of an extremely wide range of both dynamic
and fne motion. This is achieved by a complicated structure consisting of several types of tissue including tendons, ligaments, andmuscles [16, 20, 28]. The variety of tissues connecting diferent
components within the hand also imposes certain constraints. For
example, the middle and ring digits share ligaments and other connective tissues which prevents one from moving through its fullrange of motion without movement of the other.
As well as being physically constrained, the movement of ourmuscles is also constrained by our neuromuscular system. These
constraints are used to reduce the number of degrees of freedomthat need to be independently controlled, making it easier to control
a group of muscles at once [24]. These physical and neuromuscular constraints combine to make it difcult to control one muscle
without unintentionally moving other ones. This coupled motionof the human digits is fundamental to the mechanisms that cause
coactivations. Understanding these mechanisms may provide useful insights into how best to detect them. Soechting and Flanders[26] investigated the 22 degrees of freedom of the hand and fngers and found hierarchical relationships between these degrees
of freedom producing consistent repeatability of given keystrokes.
This indicates that the nervous system imposes constraints to reduce the number of independent degrees of freedom of the hand.
Baker et al. [1, 2] explore the angles, velocities and accelerationsof the fnger joints during typing on standard and ergonomic keyboards. Baker et al. [2] found that user hand kinematics adapt to theenvironment imposed by the keyboard confguration. This work
provides a useful reference regarding the dynamic movements occurring during typing and suggests that users of a virtual keyboard
will also adjust their behavior based on the keyboard shape. Particularly informative to this work, Fish and Soechting [9] investigatedthe correlations between fnger lengths and between fnger orientations during typing tasks. Fish and Soechting [9] found thatin most cases there were signifcant correlations between digits
and that the highest was between neighboring digits. They also
found that while digits move in coordinated motions, once one
was needed to complete a task, it was decoupled from the groupand the correlation dropped substantially. This work was however
performed on a physical keyboard with passive resistance provided
by the keys. This difers from the setup used in this paper, but it isexpected that these observations will be relevant because up to themoment a digit contacts a key, its motion is unrestricted, which istrue for both setups.
Häger-Ross and Schieber [15] investigated the degree to whichhuman fngers can be moved independently. The objective of thisstudy was to determine whether humans move other digits whenattempting to move just one and, if so, quantify this movement.
Häger-Ross and Schieber [15] showed that there was signifcant
motion of the digits that were intended to be held stationary. These
results agree with the fndings of Fish and Soechting [9]. Individuation is used as a metric to indicate how much other digits movewhen the digit of focus moves while stationarity measures howUnderstanding, Detecting and Mitigating the Efects of Coactivations in Ten-Finger Mid-Air Typing in VR

little the digit of focus moves when each of the other digits move.
The thumb and index fnger were found to have the greatest individuation and stationarity from the other digits, followed by the
pinky fnger, with the middle and ring fngers having the least [9].
This suggest that the middle, ring and pinky digits are more likely
to cause coactivations. Feit et al. [8] provide a complementary perspective by identifying common typing behaviors as patterns froman interaction rather than kinematic point of view. Although pursuing a fundamentally diferent approach to text, Sridhar et al. [27]highlight the issue of coactivations in mid-air text entry. In summary, although fnger coactivations is a well recognized issue intext entry there has been limited specifc exploration of its characteristics in mid-air virtual typing. Further, there are no establishedmethods for detecting or mitigating spurious observations arisingfrom coactivations.

3

APPROACH

As briefy outlined in section 1, we frst seek to understand whenand how coactivations occur before attempting to detect and eliminate them. This approach sits in contrast to, for example, a deeplearning strategy that might circumvent the need for understanding the mechanisms that produce coactivations. The motivation
behind this strategy is: i) to provide a generalizable characterization of coactivations in mid-air ten fnger typing; ii) to avoid therequirement for very large amounts of training data; and iii) todeliver a detection strategy that can be readily adapted to alternative keyboard confgurations. This results in an understanding ofcoactivations that is relevant beyond the specifc confnes of thisstudy and an approach to detection that is easily realized by others.
This investigation therefore involves two distinct stages: Stage 1)
understanding when and how coactivations occur; and Stage 2)
implementing and evaluating strategies for coactivation detectionand elimination.
In Stage 1, we begin by studying the occurrence of coactivations
in a recorded dataset [6] of 24 users typing on a mid-air virtual
keyboard in VR. The dataset contains precision fnger tracking
data captured while users transcribe set phrases (170 phrases peruser). Users were not given the facility to correct errors when
encountered, however, an oracle strategy (detailed in [6]) was used
to simulate the efect of typing with high quality auto-corrections.
This confguration was used to capture representative data of how
users would type under ideal conditions. The dataset was manuallyreviewed to identify possible characteristics of coactivations. This
analysis is reported in section 4. The outcome of Stage 1 is an
improved understanding of the nature of coactivations encounteredin ten fnger mid-air typing.
In Stage 2, the discriminative power of each of the possible
characteristics identifed in Stage 1 is evaluated to determine theirpotential as features in a feature vector. The promising features
are then used to construct three alternative coactivation detection
techniques: naïve Bayes, support vector machine, and neural network. The careful approach taken in Stage 1 ensures that thesedetectors achieve high accuracy with minimal features. Finally, thebest performing model, the neural network, is incorporated into an
existing statistical decoder. This provides a demonstration of how
coactivation detection can provide reduced error rates in mid-air
ten fnger typing.

4

CHI ’21, May 8–13, 2021, Yokohama, JapanUNDERSTANDING COACTIVATIONS

The mid-air ten fnger typing dataset was manually reviewed to
identify the circumstances in which coactivations occur. We limited
this analysis to coactivations specifcally causing unintended input
observations. This review also provided an initial set of features
distinguishing a coactivation from an intended fnger movement.

4.1

When and Where do Coactivations Occur?

Coactivations can be divided into two types characterized by the order of the intended and unwanted key selection. If the intended key
is selected frst, the unwanted key is a post-intended-coactivationand if the unwanted key is selected frst is it a pre-intendedcoactivation. Post-intended-coactivations made up 3.4% of touch
events recorded in the dataset and the majority (92%) of all coactivations. Subsequent references to coactivations imply the combinedgroup of post- and pre-intended-coactivations. However, we limit
the scope of coactivation detection and mitigation to post-intendedcoactivations since: i) these are the overwhelming majority of coactivations; and ii) post-intended-coactivations can be dealt withimmediately in a relatively simple online detection implementation.
As outlined in section 2, coactivations are caused by the coupledmotion of a pair of digits. We hypothesized that the occurrence ofcoactivations would vary depending on which fnger is involved ina key press and, related to this, which key on the layout this press istargeting. The distribution of coactivation counts per digit is shownin Figure 1(a). In Figure 1(a) and throughout we use the following
initials for conciseness: (T)humb, (I)ndex, (M)iddle, (R)ing, (P)inky,right hand (RH), left hand (LH). This fgure shows that the majority
of coactivations observed were caused by the ring digit and not
the pinky fnger as expected. This fgure also shows that the righthand was responsible for two thirds of the coactivations which isconsistent with a right handed bias among participants. Figure 1(b)shows a similar hand split for intended counts, which was equatedto the underlying hand usage. Normalizing the coactivation countper fnger by hand usage yields an approximately equal distributionof coactivations across the fngers for each hand. This indicates thatneither hand is more prone to coactivations. The prior probabilitiesof each digit to cause a coactivation were also calculated and areshown in Figure 2. This fgure shows that the ring and pinky fngers
produce a much higher proportion of coactivations than the otherdigits.
The normalized proportion of ordered pairs of digits containing
a post-intended-coactivation is shown in Figure 3 along with theirstandard errors. There are 40 pairs in total, each digit followed byanother digit on the same hand. Coactivations, by defnition, cannot
be caused by the same digit although this case is included in thegraph for clarity of presentation. Figure 3 shows that the majority of
post-intended-coactivations are caused by a digit less dominant to
the intended one. For example, coactivations of the middle, ring andpinky occur after the intended use of the index digit, coactivationsof the ring and pinky occur after intended use of the middle digit,
and so on. It also shows that the intended use of the middle digit
followed by a post-intended-coactivation of the ring digit makes
up a signifcant proportion of the times that pair is used, consistent
with observations made by Häger-Ross and Schieber [15].

CHI ’21, May 8–13, 2021, Yokohama, Japan

1500
1500
1250
1250

LH: LH:
34%34%
RH: RH:
66%66%

30000
30000
25000
25000

Count
Count

750750

15000
15000

500500

10000
10000

250250

5000
5000

0

LH: LH:
42%42%
RH: RH:
58%58%

20000
20000

Count
Count

1000
1000

Foy et al.

0
T TI

0

0
T TI

IM MR RP P(a) Count of coactivations.

IM MR RP P(b) Count of intended touch events.

Figure 1: Distribution of intended and coactivated touch
events by each digit of each hand.

0.5 0.5

Proportion Of Total Count
Proportion Of Total Count

Proportion Of Total Count
Proportion Of Total Count0.5 0.5
0.4 0.4

0.4 0.4

0.3 0.3

0.3 0.3

0.2 0.2

0.2 0.2

0.1 0.10

0.1 0.1

00

T TI M
I M
R R
P P(a) Left hand.

4.2.3 Inter-Key Interval. Due to inherent limitations in motor skills
and the speed at which one can process information, there is anupper bound on realistic entry rates. Dhakal et al. [5] found theaverage fast typist had an interval between touch events of around
120 ms and a lower bound of around 60 ms. For a typical user, thissuggests that touch events within 120 ms are likely to be unintended
and ones within 60 ms are almost certainly unintended. The interval
between touch events is therefore a likely indicator of coactivations.
4.2.4 Averaged Digit Velocity Correlation. Coupled motion of digits
causing a coactivation was observed to exhibit similar motion paths
between digits. This observation is consistent with Fish and Soechting [9], who found that while typing there is a reasonable degree ofcorrelated digit movement when the digits were not engaged in a
keypress task. Once a digit was selected for a task, its correlation tothe other digits decreased substantially. It is reasonable to assume
that this decoupling may not always occur in the virtual keyboard
setting given the lack of physical resistance. This motivates consideration of correlation in the average digit velocity profle of two
digits as an indicator of coactivations.

0
T TI M
I M
R R
P P

5

(b) Right hand.

The four potential features identifed in section 4.2 are now analyzed by plotting histograms of the intended and coactivated touch
events. This distribution analysis was performed with half of thedataset (i.e. data from 12 of the 24 participants) to maintain the remainder as an unseen validation set. The intended and coactivateddistributions are compared to determine the efectiveness of eachas a discriminative feature. Note that in subsequent descriptions,the keyboard frame is defned as a fxed frame of reference where
the Z -axis is normal to the keyboard plane such that z is zero at
the keyboard surface and is increasingly positive at greater depth
into the keyboard (further from the user).

Figure 2: Proportion of all touch events for a given digit thatwere coactivations.

The distribution of coactivations across the layout was analyzed by plotting the raw coactivation counts and the proportion oftouches on a key that were coactivations as heat-maps, shown in
Figures 4(a) and 4(b), respectively. These fgures reveal that coactivations occur most frequently at the sides of the keyboard closest to
the ring and pinky digits. Also, there are higher counts on the rightside, consistent with the hand usage shown in Figure 1. Figure 4(b)shows the majority of the edge key events were coactivations. These
results ofer an insight into the prior probability for each key to becoactivated.

4.2

4.2.2 Digit Velocity. The intended activation of a digit produces amotion path with a given velocity. As the motion of coactivating
digits is not intended, we hypothesize that these digits move witha lower velocity than the intended digit, particularly at the point ofintersection with the virtual keyboard plane.

Features Indicative of a Coactivation

Four features hypothesized to be indicative of a coactivation wereidentifed through the dataset inspection exercise and with reference to the literature examining coupled fnger movement. These
are: i) digit depth of travel; ii) digit velocity; iii) inter-key interval;
and iv) averaged digit velocity correlation. Each of these features
are briefy described below.
4.2.1 Digit Depth. Coactivating digits were observed to typicallytrail behind the intended digit. There were many cases in the dataset
where, upon a touch event, the coactivating digit would reach ashallower depth before both digits were retracted.

5.1

FEATURE EVALUATION

Digit Depth

The depth of a touch event was computed based on the maximum
depth of travel immediately following entry within the virtual
keyboard plane. The depth of the touch event was taken as thedeepest point of a continuous downward movement from the detection plane. The resulting distributions of intended and coactivatedtouch event depths are shown in Figure 5(a). Figure 5(a) shows thatthe mean of each distribution occurs at a diferent depth. The mean
intended touch event depth and its standard deviation was 15 ± 7mm whereas the average coactivation depth was much shallower,
5 ± 3 mm. This is a promising result indicating that digit depth may
be an efective feature.
Due to the coupled motion of the digits, it is possible that deeper
intended touch events would be more likely to pull a neighboring
fnger down into the detecting plane, producing a coactivation on
the next touch event. To examine this, the depth of the previous digit
was plotted and classed according to the current touch event. This

Right Hand
Proportion Of Total Count

Left Hand
Proportion Of Total Count

Understanding, Detecting and Mitigating the Efects of Coactivations in Ten-Finger Mid-Air Typing in VR

T keypress then:

0.8

I keypress then:

0.8

CHI ’21, May 8–13, 2021, Yokohama, JapanM keypress then:

0.8

R keypress then:

0.8

0.6

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.2

0

T

I

M

R

0

P

T keypress then:

0.8

T

I

M

R

0

P

I keypress then:

0.8

T

I

M

R

0

P

M keypress then:

0.8

T

I

M

R

0

P

R keypress then:

0.8

0.6

0.6

0.6

0.6

0.4

0.4

0.4

0.4

0.4

0.2

0.2

0.2

0.2

0.2

T

I

M

R

0

P

T

I

M

R

0

P

T

I

M

R

0

P

T

I

M

R

T

0

P

I

M

R

P

P keypress then:

0.8

0.6

0

P keypress then:

0.8

T

I

M

R

P

Figure 3: Proportion of all touch events for a given pair of digits that were coactivations. Digit pairings with a total count ofless than 100 have been removed.

D

U

V
]

G
[

J

I
F


\

W

Y

X
M

K
E

L

Q

R

S

N

O

P



T






Z

H

D
"

U

V
]

  
&RXQW

J

I

[

\

W

G
F

Y

X

L
M

K
E

Q

R
N

P

(a) Total coactivation counts.

(b) Proportion of touches that were coactivations.

Figure 4: Coactivation count distribution over the letters of the keyboard.
0.2
0.08

Intended Touch Events
Coactivated Touch Events

Intended Touch Events
Coactivated Touch Events

0.07

0.15

0.1

0.05

0.06
0.05
0.04
0.03
0.02

0
0

10

20

30

Depth [mm](a) Current digit depth feature.

40

5
4
3
2
1

0.01
0

Intended Touch Events
Coactivated Touch Events

6

0
0

10

20

30

Depth [mm](b) Previous digit depth feature.

40

0

0.2

0.4

0.6

0.8

S
O




3URSRUWLRQRI&RDFWLYDWLRQV

Probability Density

"H

Probability Density

Z

Probability Density

T

1

Velocity [mm/ms](c) Digit velocity feature.

Figure 5: Depth and velocity feature results for the combined dataset of participant 1 to participant 12.

CHI ’21, May 8–13, 2021, Yokohama, Japan

Foy et al.

0.02

6

0.015

0.01

0.005

Intended Touch Events
Coactivated Touch Events

5

Probability Density

Probability Density

Intended Touch Events
Coactivated Touch Events

4
3
2
1

0

0
0

200

400

600

800

-1

Time [ms]

-0.5

0

0.5

1

Correlation(a) Inter-key interval feature.

(b) Digit correlation.

Figure 6: Inter-key and correlation feature results for the combined dataset of participant 1 to participant 12.

is shown in Figure 5(b). Figure 5(b) shows only a slight diference
between the distribution means but has the unique property of only
relying on data from the previous touch event.

5.2

Digit Velocity

The touch event velocity was calculated as the average velocity of
the activating digit from approximately 30 ms before to approximately 30 ms after the touch event. This range was chosen using
the following logic. Dhakal et al. [5] found the average interval
between keypresses was 239 ms. In this time, four movements needto take place. The activating digit must move from its rest heightto the detection plane, from there to its max depth, back to thedetection plane, and then back to the rest height. If each of theseactions take approximately the same amount of time, then the fulldownwards motion will occur from about 60 ms before a keypress
to about 60 ms after it. Taking the center half of this motion toremove the portions where the digit was accelerating and decelerating, results in a 60 ms window over which to average. As before,
this feature was calculated for half the dataset and plotted as histograms shown in Figure 5(c). This feature was evaluated in the
keyboard frame of reference. This fgure shows a clear separation
between both the peaks and means of the distributions, which isa promising result. The average intended and coactivated touch
event velocities were 0.31 ± 0.17 mm/ms and 0.12 ± 0.08 mm/ms
respectively.

5.3

Inter-Key Interval

The interval assigned to each coactivated touch event was the time
between that event and the coactivation’s corresponding intended
touch event. Intended touch events can cause multiple coactivations and so this interval may not necessarily be calculated between
consecutive events. The interval assigned to intended events was
the time between that event and the previous intended touch event.
The resulting distributions are shown in Figure 6(a). Figure 6(a)shows the greatest separation of the distributions among the features examined. However, it does not identify which of the twotouch events, used in its calculation, is the coactivation. This feature must therefore be combined with others to distinguish thecoactivation.

5.4

Average Digit Velocity Correlation

The average velocity correlation assigned to each touch event wascalculated between pairs selected in the same way as for the interkey interval feature. The correlation was calculated as the average
dot product of the digits’ velocity vectors, normalized by the magnitude of the vectors. The average was taken from 50 ms before
the frst touch event to 50 ms after the second touch event. The
times were chosen using a similar logic to the velocity feature. The
motion to select a key begins approximately 60 ms before the touch
event and ends approximately 60 ms after it. The frst and last 10
ms of the motion were removed to reduce the part of the motion
where there is little to no movement. Using a window that extends
from before the frst touch event to after the second ensured the
motion of the two events was captured. The equation used is givenjbelow, where XT is the three-dimensional position of digit j atitime Ti :VD 1 · VD 2
Corr =||VD 1 || ||VD 2 ||

j

where

VD j =j

XT − XT2

T2 − T1

1.

(1)

This feature was also evaluated in the keyboard frame. The
resulting distribution is shown in Figure 6(b). The mean and standard deviation of the intended and coactivated distributions were0.43 ± 0.29 and 0.84 ± 0.13 respectively. This shows that on averagethe velocity of a coactivating digit was highly correlated with its
corresponding intended digit. It also shows, however, that there isa non-insignifcant degree of correlation between all digits.

6

DETECTING COACTIVATIONSIn this section, we describe the design of a coactivation detection
technique implemented using three alternative models. We begin
with an overview of the chosen features before describing the model
selection and training procedure in section 6.2. Finally, we reporton the cross-validated accuracy evaluation of the developed models
in section 6.3.

6.1

Feature Selection

From among the features tested in section 5, the current and previous digit depth and velocity, inter-key interval, and averagecorrelation features were identifed as worthy elements of theFalse Positive Rate(a) Previous depth feature.

False Positive Rate(b) Interval feature.

False Positive Rate(c) Current velocity feature.

CHI ’21, May 8–13, 2021, Yokohama, Japan

1.0
0.8
0.6
0.4
0.2
Area=0.53
0.00.0 0.2 0.4 0.6 0.8 1.0

False Positive Rate(d) Prev. velocity feature.

True Positive Rate

1.0
0.8
0.6
0.4
0.2
Area=0.67
0.00.0 0.2 0.4 0.6 0.8 1.0

True Positive Rate

1.0
0.8
0.6
0.4
0.2
Area=0.97
0.00.0 0.2 0.4 0.6 0.8 1.0

True Positive Rate

1.0
0.8
0.6
0.4
0.2
Area=0.72
0.00.0 0.2 0.4 0.6 0.8 1.0

True Positive Rate

True Positive Rate

Understanding, Detecting and Mitigating the Efects of Coactivations in Ten-Finger Mid-Air Typing in VR

1.0
0.8
0.6
0.4
0.2
Area=0.90
0.00.0 0.2 0.4 0.6 0.8 1.0

False Positive Rate(e) Vel. correlation feature.

Figure 7: ROC curves for potential feature vector elements.
feature vector. These features combine data from multiple time
steps, extract useful information and reduce the size of the feature
vector.
One problem with some of these features are their dependence on
information occurring after the current touch point to be classifed.
Although using all the features listed above is likely to produce thebest performance, for a model to run easily online, it should run
in real time with no such dependencies. The depth of the current
touch event relied on the current digit’s z position several time
steps after the event had occurred and therefore it was removed.
The average correlation, shown in subsection 5.4, also relied ondata from time steps recorded after the touch event. The extra 50 msof data included after the second touch event was used to achieve amore accurate average. The correlation calculation was altered toremove these extra time steps. The extra time steps included before
the frst touch event were also removed to reduce the complexityof implementation. Removing the current depth feature and modifying the calculation of the correlation feature, leaves a total of fve
input features with no post-event information dependencies: thecurrent and previous digit velocity, the depth of the previous touch
event, the time interval, and the average velocity correlation. These
alterations limit the features, and therefore the feature vector, toonly data prior to the current touch event, allowing the model to be
more easily incorporated into an online system. Further evaluationof these fve features in subsection 6.2.1 reveals that the previous
digit velocity performs poorly and is subsequently removed fromthe fnal feature vector, leaving four features implemented into thefnal models.

6.2

Model Selection

Three alternative models were selected for evaluation. These were anaïve Bayes model, a support vector machine and a neural network.
Each of these models was trained and tested using cross-validation.
We partitioned the dataset into training and test sets for K-foldcross-validation. Efort was taken to assign participants into representative groupings. There was high variation between the users
in the dataset in terms of coactivation counts and typing speed.
Stratifed folds were therefore assembled based on two criteria.
First, participants were grouped to minimize the diference in totalcoactivation counts per group. An exhaustive search found several
hundred groupings where the maximum diference between totals
was one. Second, fnal group assignments were made based on theobjective of minimizing the diference in average words per minuteacross groups.

The design of each coactivation detection model is now summarized in the following subsections. Note that we describe the designof the neural network frst since we leverage its ability to isolate theinfuence of individual features to further refne the feature vector
used by the other models.
6.2.1 Neural Network. Although the task of detecting coactivations is inherently a time dependent problem, it was decided toaddress the problem with a time independent solution. This choice
was motivated by the fact that time dependent solutions are often
more complex and more difcult to train, and thus demand moredata, which is expensive and difcult to collect for this problem.
In preliminary results, the time independent features investigatedalso proved efective in identifying coactivations and therefore wepursued a time independent solution.
The architecture chosen for the neural network comprises thefollowing layers: 32 node dense with batch normalization, ReLU
activation and 20% dropout, followed by another 32 node densewith batch normalization, ReLU activation and 20% dropout, thena 16 node dense with batch normalization, ReLU activation and
20% dropout, and fnally a 1 node dense with a sigmoid activation
function. The network therefore has just less than 2,000 trainable
parameters. The model was trained on three of the groups, with thefourth used for testing and validation. To train, all coactivations
and an equal number of intended touch events were selected fromeach participant in the training group. This ensured the model wastrained on an equal number of positive and negative cases. As aresult, approximately 7,500 data points were used in the trainingof the model’s 2,000 parameters—3.75 data points per parameter.
We used 32 nodes per layer to maximize the complexity of therelationships that can be learned while maintaining a sufcientnumber of training points per parameter [23]. The specifc choiceof 32 nodes stems from the convention of assigning node counts at
powers of base two.
Calibration of the output is also an important consideration given
the objective of integrating the model with a statistical decoder. A
binary cross-entropy loss function was selected given its suitability to this type of problem. The cross-entropy loss function is a
probabilistic loss and is minimized when the model’s predictions
approximate the ground truth conditional distribution [13].
To test the efectiveness of each feature, the feature vector was
reduced to a single feature at a time for testing. The model wastrained and tested using cross-validation. A confusion matrix was
created for each fold of the validation and used to calculate themean and standard deviation of the accuracy, precision, recall andCHI ’21, May 8–13, 2021, Yokohama, Japan

Foy et al.

Table 1: Table of metrics showing the performance of various features (± 1 standard deviation).
FeatureAccuracy

Precision

RecallAUC

Previous Digit Depth
Current Digit Velocity
Previous Digit Velocity
Interval
Correlation

0.58 ± 0.07
0.61 ± 0.03
0.54 ± 0.16
0.96 ± 0.02
0.85 ± 0.02

0.20 ± 0.07
0.22 ± 0.03
0.14 ± 0.04
0.80 ± 0.11
0.48 ± 0.07

0.64 ± 0.11
0.72 ± 0.01
0.47 ± 0.21
0.96 ± 0.01
0.76 ± 0.05

0.64 ± 0.05
0.70 ± 0.02
0.50 ± 0.03
0.98 ± 0.01
0.88 ± 0.01

Table 2: Metrics showing the performance of the three models tested (± 1 standard deviation).
Model

Accuracy

Precision

RecallF1 Score

AUC

Neural Net
SVMNaïve Bayes

0.978 ± 0.011
0.973 ± 0.011
0.976 ± 0.005

0.882 ± 0.072
0.854 ± 0.065
0.922 ± 0.028

0.968 ± 0.009
0.971 ± 0.010
0.902 ± 0.021

0.921 ± 0.044
0.907 ± 0.042
0.912 ± 0.023

0.990 ± 0.006
0.985 ± 0.006
0.984 ± 0.008

F1 score. These evaluation metrics are given for each feature in
Table 1. The ROC curves for each feature from the frst fold of thecross-validation are shown in Figure 7.
An examination of the evaluation metrics presented in Table 1
and the ROC curves reveals that the interval feature achieves thebest performance while the previous key velocity feature performs
the worst. The remaining features demonstrate good performance.
As a result, it was decided to remove the previous key velocity
feature from the feature vector. This was done as its ROC curve
showed it contributed very little to the classifcation prediction
and reducing the feature vector would increase the data points perparameter, potentially improving the ft of the other features. The
fnal feature vector is therefore composed of the following features:previous depth, current velocity in the keyboard frame, interval,
and average velocity correlation in the keyboard frame.
6.2.2 Naïve Bayes. The naïve Bayes model was chosen as it is a
simple, commonly used classifer. The probability of a coactivation
given the values of the four selected features was found using
Bayes theorem in combination with the empirical distributions.
The conditional distributions of the features were found by frst
normalizing the histograms presented in section 5.
6.2.3 Support Vector Machine. The fnal model selected was a calibrated support vector machine (SVM). SVMs are well known andwell understood classifers, which serve as reliable baselines forevaluating classifcation performance. A linear kernel was chosen
for simplicity and the SVM was calibrated using Platt scaling.

6.3

Detection Accuracy

To evaluate coactivation detection accuracy, the three models weretrained on the chosen features and tested using the same crossvalidation splits to ensure consistency. This was done using crossvalidation in the same way the individual features were tested
using the neural network described above. A confusion matrix
was created for each fold of the validation and used to calculate
the mean and standard deviation of the accuracy, precision, recalland F1 score. This was repeated for each model and the resulting
evaluation metrics are given in Table 2.

All models show a high level of accuracy in detecting coactivations and none of the models have signifcantly higher accuracy
than any of the others. The naïve Bayes model’s precision is higher
than the SVM, but its recall is lower. This suggests that the SVM andnaïve Bayes trade of between false positives and false negatives.
The neural network’s precision and recall sits between these two.
The neural network achieved the highest F1 score and was therefore chosen as the model for subsequent integration and evaluation
with the decoder.

7

USING COACTIVATION DETECTION IN
STATISTICAL DECODING

To validate the utility of the developed coactivation detection functionality, we integrated it as a module within an existing probabilistic text input decoder. As an illustration of the behavior of themodule, independent of the decoder, Figure 8(a) demonstrates how
discrete input events are scored based on a continuous stream offnger tracking data. Figure 8(a) plots the z ofset of the right hand
digits only during the entry of “tomorrow with”. The vertical lines
indicate when a letter (shown at the top of the plot) was activated.
The markers indicate the ground truth of whether a touch event
was intended (green), or a coactivation (red). A vertical line with nocorresponding marker indicates the letter was activated by the left
hand. Beside each marker is the system’s coactivation prediction
for the touch event. Figure 8(a) shows the original text typed, “tomoorrokw wipth” and shows the system correctly identifying thecoactivations “o”, “k” and “p”. Removing these letters would returnthe correct phrase “tomorrow with”. This fgure also highlights thatthe system can correctly identify coactivations by diferent digits,
as the features are digit independent.
The promising behavior observed from the isolated detection
module and illustrated in Figure 8(a) motivated its complete integration with an existing statistical decoder. The decoder integration
operates as follows: at each touch event the probability that theevent is a coactivation is calculated. If the probability is greater thana certain threshold the character is removed from the input feld.
The system can also be confgured to remove this observation fromthe set passed to the decoder. If the character is not removed, theUnderstanding, Detecting and Mitigating the Efects of Coactivations in Ten-Finger Mid-Air Typing in VR

CHI ’21, May 8–13, 2021, Yokohama, Japan

Full Dataset

Coactivation Subset

CER without auto-correct
CER with auto-correct but without coactivation detection
CER with auto-correct and coactivation detection
Relaxed Error Rate

10.7 ± 0.5%
8.7 ± 0.5%
7.8 ± 0.7%
7.7 ± 0.2%

13.8 ± 1.0%
11.0 ± 1.1%
9.0 ± 1.1%
10.7 ± 0.5%

U

RN

Z 

Z LS

W

K














U

R PR R




W






Error Metric

=3RV
>PP@53

=3RV
>PP@55

=3RV
>PP@50

=3RV
>PP@
5,

=3RV
>PP@57

Table 3: Resulting error rates under diferent decoder confgurations (± 1 standard deviation).





 
7LPHPV





(a) The z (mm) ofset of the right hand digits plotted against time, showing model
coactivation predictions (0: intended, 1: coactivated) and ground truths (green: intended, red: coactivated).

(b) Illustrated operation of the decoder’s coactivation detection and auto-correction components. 1: Intended ‘n’. 2: Intended ‘o’. 3: Coactivated ‘p’. 4: ‘p’ removed by coactivation
module. 5: ‘tee’ entered by user. 6: Auto-correction of ‘notee’ to desired ‘notes’.

Figure 8: Sample evaluations of the decoder with the integrated coactivation detection.
weighted coactivation probability is combined with the likelihoodthat the character is an insertion error. This is done for each observation until a space or other word delimiter is activated to force a
decode. We are interested in the efect that coactivation detection
can have on reducing the character error rate (CER) when typing.
The CER is calculated based on how many insertion, deletion or
substitution corrections are needed to convert the fnal text into
the stimulus text the user was trying to replicate. It is normalized
by the length of the stimulus text. Using the same cross-validation
partitioning described in subsection 6.2, we determine appropriate hyperparameter values from the training sets and evaluate theresulting CER with the corresponding test sets. Approximate hyperparameters were frst found using Bayesian optimization, followed
by grid search to fne tune the parameters.

7.1

Eliminating Errors in Typing

The dataset upon which the models are trained and evaluated withcross-validation involved typing using a simulated decoder withno facility for error correction. Therefore subsequent error values
should be interpreted as uncorrected error rates. The mean CERand standard deviation of the fnal hyperparameter setting of thedecoder across the four folds of the cross-validation were calculatedwith and without the coactivation module enabled. These values
are summarized in Table 3. Also included in Table 3 is the CER
without auto-correct and the relaxed error rate, a metric introducedby Dudley et al. [6] based on a distance tolerance attached to eachtarget key that can only be calculated if the system has complete
knowledge of what the user is trying to type. The CER without
auto-correct refects direct interpretation of observations basedon the closest key to the touch point and therefore serves as a
pessimistic baseline. By contrast, the relaxed error rate refects theperformance under the oracle decoder, which allows for a highdegree of touch inaccuracy, and therefore serves as an optimistic
baseline. The Full Dataset column lists the results for all typed
phrases (i.e. 1,020 phrases per fold). The Coactivation Subset column
reports the subset of phrases where coactivations are present (n =[531, 547, 519, 659] across the four folds).
The results show that with the coactivation module enabled in
the statistical decoder a further reduction in character error rateby approximately 10% relative and 0.9% absolute can be achieved.
This improvement is more signifcant when the dataset is restricted
to just samples containing coactivations, achieving character error
rate reductions of approximately 18% relative and 2.0% absolute.
Figure 8(b) illustrates the keyboard behavior with the integrated
coactivation module. Figure 8(b) shows the keyboard state just as
the stream of characters, ‘nop’, have been entered using the rightindex (1.), middle (2.) and ring (3.) fngers. The user is trying totype ‘notes’ but has caused a coactivation with their ring fnger
and inserted the ‘p’ into the input feld. The coactivation module
subsequently identifes and removes the coactivated ‘p’ character(4.). The remaining characters, ‘tee’, are entered (5.), before theCHI ’21, May 8–13, 2021, Yokohama, Japan

literal input, ‘notee’, is corrected to the desired word, ‘notes’, by
the decoder (6.) upon typing the period.

8

DISCUSSION

This investigation has shown that identifcation and removal ofcoactivations from a text input stream can be completed with simple
lightweight features. Four features have been identifed that can aid
the detection of coactivations. Combined, these features can achievea detection accuracy exceeding 97%. The module integrated into
the input decoder also demonstrates how coactivation detection
can efectively reduce typing error rates. The simplest and mostefective feature is the interval between touch events. This featuresuggests that a naïve strategy for mitigating coactivations could
rely on simply enforcing a minimum time (50–100 ms) before thenext letter can be selected.
Despite the immediate demonstrated efcacy of the coactivation detection strategy presented, limitations and opportunities for
future work remain. First, the model is limited by its simplicity
and relatively small number of features. This was a design choicebased on the size of the dataset and the challenge in collecting alarge representative dataset for this application context. As a result,most of the raw digits’ motion is unavailable to the model or isonly available in a condensed form. While it would be theoreticallypossible to overcome the data sparseness issue with representativesynthetic coactivation data, this would necessitate a generativemodel capturing both physical and neuromuscular behaviors encountered while typing. This would, however, be a challenging
research project in its own right. Second, as described in section 4,
we exclude pre-intended-coactivations from consideration due tothe pursuit of streamlined online detection. Although pre-intendedcoactivations are a very small proportion of all coactivations, there
is potential value in robustly detecting all spurious input events.
Related to this point, the choice of features was also constrained tothose available up until the current time step (see section 6.1). Moreadvanced strategies that continually reassess past potential coactivations as new data streams in are feasible but would require more
complex accounting and event handling functionality. As a result,they would therefore likely demand a tighter coupling between thecoactivation detection module and the decoding framework. A specifc design objective in this work was to avoid a tight coupling ofcomponents, as this increases the complexity of the system design.
However, a more fexible strategy for handling coactivation detection that can propagate backwards to recent events is a promisingavenue for future work.
In general, it would likely be fruitful to extend the model toinclude prior probabilities of digits causing coactivations and characters becoming coactivated. The potential contribution of thisprior information is clearly illustrated in Figure 4 and could bereadily incorporated. Finally, the current pandemic has prevented
the execution of a user study to evaluate the potential beneft ofcoactivation detection and mitigation during ‘live’ text entry.

9

CONCLUSIONSIn this paper, we presented a detailed characterization of coactivations encountered in ten fnger mid-air typing in VR. The mostinformative features of coactivations were found to be: inter-key

Foy et al.

interval, key depth, key velocity, and digit velocity correlation. We
demonstrated that coactivations can indeed be efectively detected.
The fnal models successfully identifed 97 ± 1% of coactivations
while only incorrectly labeling 2 ± 1% of intended touch points ascoactivations. As 4% of characters typed were coactivations, thisequates to a 2 ± 1% reduction in errors. Finally, we demonstrated
how coactivation detection can be integrated into a statistical decoding framework to achieve a reduction in uncorrected CER of
approximately 10% relative and 0.9% absolute.

ACKNOWLEDGMENTS
This work was supported by Facebook Reality Labs. J. J. Dudleyand P. O. Kristensson were also supported in part by EPSRC (grants
EP/R004471/1 and EP/S027432/1). Supporting data for this publication is available at https://doi.org/10.17863/CAM.63030.

REFERENCES[1] Nancy A. Baker, Rakié Cham, Erin Hale Cidboy, James Cook, and Mark S. Redfern.
2007. Kinematics of the fngers and hands during computer keyboard use. Clinical
Biomechanics 22, 1 (2007), 34 – 43. https://doi.org/10.1016/j.clinbiomech.2006.08.
008[2] Nancy A. Baker, Rakié Cham, Erin Hale, James Cook, and Mark S. Redfern.
2007. Digit kinematics during typing with standard and ergonomic keyboard
confgurations. International Journal of Industrial Ergonomics 37, 4 (2007), 345 –
355. https://doi.org/10.1016/j.ergon.2006.12.004[3] Garrett Benoit, G. Michael Poor, and Alvin Jude. 2017. Bimanual Word Gesture Keyboards for Mid-air Gestures. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA ’17).
Association for Computing Machinery, Denver, Colorado, USA, 1500–1507.
https://doi.org/10.1145/3027063.3053137[4] Doug A. Bowman, Christopher J. Rhoton, and Marcio S. Pinho. 2002. Text Input
Techniques for Immersive Virtual Environments: An Empirical Comparison.
Proceedings of the Human Factors and Ergonomics Society Annual Meeting 46, 26(2002), 2154–2158. https://doi.org/10.1177/154193120204602611[5] Vivek Dhakal, Anna Maria Feit, Per Ola Kristensson, and Antti Oulasvirta. 2018.
Observations on Typing from 136 Million Keystrokes. In Proceedings of the 2018
CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada)(CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–12.
https://doi.org/10.1145/3173574.3174220[6] John J. Dudley, Hrvoje Benko, Daniel Wigdor, and Per Ola Kristensson. 2019.
Performance Envelopes of Virtual Keyboard Text Input Strategies in Virtual
Reality. In 2019 IEEE International Symposium on Mixed and Augmented Reality(ISMAR). 289–300. https://doi.org/10.1109/ISMAR.2019.00027[7] John J. Dudley, Keith Vertanen, and Per Ola Kristensson. 2018. Fast and Precise
Touch-Based Text Entry for Head-Mounted Augmented Reality with VariableOcclusion. ACM Trans. Comput.-Hum. Interact. 25, 6, Article 30 (Dec. 2018),
40 pages. https://doi.org/10.1145/3232163[8] Anna Maria Feit, Daryl Weir, and Antti Oulasvirta. 2016. How We Type: Movement Strategies and Performance in Everyday Typing. In Proceedings of the 2016
CHI Conference on Human Factors in Computing Systems (CHI ’16). ACM, New
York, NY, USA, 4262–4273. https://doi.org/10.1145/2858036.2858233[9] Jack T. Fish and John F. Soechting. 1992. Synergistic fnger movements in askilled motor task. Experimental Brain Research 91 (1992), 327–334.
[10] Jens Grubert, Lukas Witzani, Eyal Ofek, Michel Pahud, Matthias Kranz, and
Per Ola Kristensson. 2018. Efects of Hand Representations for Typing in Virtual
Reality. IEEE, 151–158.
[11] Jens Grubert, Lukas Witzani, Eyal Ofek, Michel Pahud, Matthias Kranz, and
Per Ola Kristensson. 2018. Efects of Hand Representations for Typing in Virtual
Reality. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR).
151–158. https://doi.org/10.1109/VR.2018.8446250[12] Jens Grubert, Lukas Witzani, Eyal Ofek, Michel Pahud, Matthias Kranz, and
Per Ola Kristensson. 2018. Text Entry in Immersive Head-Mounted DisplayBased Virtual Reality Using Standard Keyboards. IEEE, 159–166.
[13] Chuan Guo, Geof Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On Calibrationof Modern Neural Networks. arXiv:1706.04599 [cs.LG][14] Aakar Gupta, Cheng Ji, Hui-Shyong Yeo, Aaron Quigley, and Daniel Vogel. 2019.
RotoSwype: Word-Gesture Typing using a Ring. In Proceedings of the 2019 CHI
Conference on Human Factors in Computing Systems (CHI ’19). Association for
Computing Machinery, Glasgow, Scotland Uk, 1–12. https://doi.org/10.1145/
3290605.3300244

Understanding, Detecting and Mitigating the Efects of Coactivations in Ten-Finger Mid-Air Typing in VR[15] Charlotte Häger-Ross and Marc H. Schieber. 2000. Quantifying the Independenceof Human Finger Movements: Comparisons of Digits, Hands, and Movement
Frequencies. Journal of Neuroscience 20, 22 (2000), 8542–8550. https://doi.org/10.
1523/JNEUROSCI.20-22-08542.2000[16] InformedHealth.org [Internet]. Cologne, Germany: Institute for Quality andEfciency in Health Care (IQWiG). 2018. How do hands work? https://www.ncbi.
nlm.nih.gov/books/NBK279362/[17] Pascal Knierim, Valentin Schwind, Anna Maria Feit, Florian Nieuwenhuizen, andNiels Henze. 2018. Physical Keyboards in Virtual Reality: Analysis of Typing Performance and Efects of Avatar Hands. In Proceedings of the 2018 CHI Conferenceon Human Factors in Computing Systems (CHI ’18). ACM, New York, NY, USA,345:1–345:9. https://doi.org/10.1145/3173574.3173919[18] Anders Markussen, Mikkel R. Jakobsen, and Kasper Hornbaek. 2013. SelectionBased Mid-Air Text Entry on Large Displays. In Human-Computer Interaction– INTERACT 2013: 14th IFIP TC 13 International Conference, Cape Town, South
Africa, September 2-6, 2013, Proceedings, Part I, Paula Kotzé, Gary Marsden, Gitte
Lindgaard, Janet Wesson, and Marco Winckler (Eds.). Springer Berlin Heidelberg,
Berlin, Heidelberg, 401–418. https://doi.org/10.1007/978-3-642-40483-2_28[19] Anders Markussen, Mikkel Rønne Jakobsen, and Kasper Hornbaek. 2014. Vulture:
A Mid-air Word-gesture Keyboard. In Proceedings of the SIGCHI Conference on
Human Factors in Computing Systems (CHI ’14). ACM, New York, NY, USA, 1073–
1082. https://doi.org/10.1145/2556288.2556964[20] Keith L. Moore, Arthur F. Dalley, and A.M.R. Agur. 2013. Clinically Oriented
Anatomy. Wolters Kluwer Health/Lippincott Williams & Wilkins.
[21] Tao Ni, Doug Bowman, and Chris North. 2011. AirStroke: Bringing Unistroke
Text Entry to Freehand Gesture Interfaces. In Proceedings of the SIGCHI Conferenceon Human Factors in Computing Systems (CHI ’11). ACM, New York, NY, USA,2473–2476. https://doi.org/10.1145/1978942.1979303[22] Taihei Ogitani, Yoshitaka Arahori, Yusuke Shinyama, and Katsuhiko Gondow.
2018. Space Saving Text Input Method for Head Mounted Display with Virtual 12key Keyboard. In 2018 IEEE 32nd International Conference on Advanced Information
Networking and Applications (AINA). 342–349. https://doi.org/10.1109/AINA.
2018.00059

CHI ’21, May 8–13, 2021, Yokohama, Japan[23] Kevin L. Priddy and Paul E. Keller. 2005. Artifcial Neural Networks: An Introduction.
SPIE Press.
[24] Marc H. Schieber and Marco Santello. 2004. Hand function: peripheral and central
constraints on performance. Journal of Applied Physiology 96, 6 (2004), 2293–2300.
https://doi.org/10.1152/japplphysiol.01063.2003 PMID: 15133016.
[25] Garth Shoemaker, Leah Findlater, Jessica Q. Dawson, and Kellogg S. Booth. 2009.
Mid-air text input techniques for very large wall displays. In Proceedings of Graphics Interface 2009 (GI ’09). Canadian Information Processing Society, Kelowna,
British Columbia, Canada, 231–238.
[26] John F. Soechting and Martha Flanders. 1997. Flexibility and Repeatability ofFinger Movements During Typing: Analysis of Multiple Degrees of Freedom.
Journal of Computational Neuroscience 4, 1 (1997), 29–46.
[27] Srinath Sridhar, Anna Maria Feit, Christian Theobalt, and Antti Oulasvirta. 2015.
Investigating the Dexterity of Multi-Finger Input for Mid-Air Text Entry. In
Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing
Systems (CHI ’15). ACM, New York, NY, USA, 3643–3652. https://doi.org/10.
1145/2702123.2702136[28] Craig L. Taylor and Robert J. Schwarz. 1955. The anatomy and mechanics of thehuman hand. Artifcial limbs 2, 2 (1955), 22–35.
[29] Naoki Yanagihara, Buntarou Shizuki, and Shin Takahashi. 2019. Text Entry
Method for Immersive Virtual Environments Using Curved Keyboard. In 25th
ACM Symposium on Virtual Reality Software and Technology (VRST ’19). Association for Computing Machinery, Parramatta, NSW, Australia, 1–2. https://doi.org/10.1145/3359996.3365026[30] Xin Yi, Chun Yu, Mingrui Zhang, Sida Gao, Ke Sun, and Yuanchun Shi. 2015. ATK:
Enabling Ten-Finger Freehand Typing in Air Based on 3D Hand Tracking Data.
In Proceedings of the 28th Annual ACM Symposium on User Interface Software &Technology (UIST ’15). ACM, New York, NY, USA, 539–548. https://doi.org/10.
1145/2807442.2807504[31] Difeng Yu, Kaixuan Fan, Heng Zhang, Diego Monteiro, Wenge Xu, and HaiNing Liang. 2018. PizzaText: Text Entry for Virtual Reality Systems Using Dual
Thumbsticks. IEEE Transactions on Visualization and Computer Graphics 24, 11(Nov. 2018), 2927–2935. https://doi.org/10.1109/TVCG.2018.2868581