Head-Coupled Kinematic Template Matching:
A Prediction Model for Ray Pointing in VR

Rorik Henrikson*
Tovi Grossman†
Sean Trowbridge‡*†
Chatham Labs
University of Toronto
Toronto, ON
Toronto, ON{rorik | daniel}@chathamlabs.com
tovi@dgp.toronto.edu

Daniel Wigdor*
Hrvoje Benko‡‡
Facebook Reality Labs
Redmond, Washington{stro, benko}@fb.com

ABSTRACTThis paper presents a new technique to predict the ray pointerlanding position for selection movements in virtual reality(VR) environments. The technique adapts and extends aprior 2D kinematic template matching method to VR environments where ray pointers are used for selection. It builds
on the insight that the kinematics of a controller and HeadMounted Display (HMD) can be used to predict the ray’s final landing position and angle. An initial study provides evidence that the motion of the head is a key input channel for
improving prediction models. A second study validates thistechnique across a continuous range of distances, angles, andtarget sizes. On average, the technique’s predictions were
within 7.3° of the true landing position when 50% of the way
through the movement and within 3.4° when 90%. Furthermore, compared to a direct extension of Kinematic Template
Matching, which only uses controller movement, this headcoupled approach increases prediction accuracy by a factorof 1.8x when 40% of the way through the movement.
Author Keywords

Endpoint prediction; target prediction; virtual reality; VR;
kinematics; ray pointing; template matching
CSS Concepts• Human-centered computing~Human computerinteraction (HCI); Interactive paradigms; Virtual reality
INTRODUCTION

In the last several years, there has been a significant increasein the popularity of virtual reality (VR) technologies. Despite
decades of research in the HCI community, many interaction
challenges are still prevalent, and interface paradigms have
yet to converge. Specifically, target selection via pointing,one of the core tasks in VR systems [35], remains
problematic due to the spatial nature of VR environments.
One of the most common selection techniques to use in VR
is a ray pointer, which acts like a laser emanating from a 6DOF controller, and can be used to acquire distant targets.
Ray pointers, however, are error prone as when targets are
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are notmade or distributed for profit or commercial advantage and that copies bear
this notice and the full citation on the first page. Copyrights for componentsof this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on
servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from Permissions@acm.org.
CHI '20, April 25–30, 2020, Honolulu, HI, USA© 2020 Copyright is held by the owner/author(s). Publication rights licensed to ACM.
ACM 978-1-4503-6708-0/20/04…$15.00

Figure 1. During a ray pointer target acquisition movement, thevelocity of both the controller and the head mounted display
are tracked (a, b). These velocity profiles are matched to alibrary of templates to predict the final ray landing position (c).

far away, they require high angular precision for successfulacquisition. Numerous techniques have been developed tofacilitate ray pointing, but they often introduce additional
steps [22, 29] or input mechanisms [7]. To better leverage
pointing facilitation techniques, it would be advantageous ifa system could predict the target or region that a user intends
to point towards, while the movement is still in progress.
In 2D environments, many endpoint predictive models havebeen developed that could be used to facilitate pointing tasks[33, 45, 53, 63]. With such models, the cursor trajectory is
continuously analyzed as it moves towards an intended
target, and the model predicts where the final endpoint of the
trajectory will be. One recent, promising technique isKinematic Template Matching (KTM), which matches
cursor velocity profiles to a library of templates from known
movements to predict the trajectory’s endpoint [45]. Despite
the promise of such techniques, little work has applied
endpoint prediction to VR environments.
This paper describes a novel endpoint prediction model,
Head-Coupled Kinematic Template Matching (HC-KTM),
for ray pointing in VR (Figure 1). It builds upon 2D
Kinematic Template Matching, with important adaptations
and enhancements. Most notably, the prior model only
considers the cursor trajectory to build and match templategestures. Our key insight is to integrate head movement
trajectories into the templates – an information channel
inherently available from the Head-Mounted Displays(HMD) of VR platforms. This allows predictions to be basedon where users look, along with the pointer trajectory.
Two data collection experiments were performed to validate

the efficacy of this new model. The first experiment analyzed
acquisition behaviors to build and tune the model undercontrolled conditions. It revealed that during the first half ofa movement, head trajectory data is more indicative of the
final landing position than controller trajectory data. The
second experiment validates the HC-KTM model bycapturing target acquisition data across a continuous range ofangles and target sizes. The prediction model was thenapplied to the captured data, and its accuracy analyzed.
Results show that our new model’s predictions are within
7.3° of the true landing position, 50% of the way through the
movement, and within 3.4° at 90%. Furthermore, compared
to a direct extension of KTM, the head-coupled approachincreases prediction accuracy by a factor of 1.8x at 40% of
the way through the movement.

This can be done by highlighting the closest target [34],having the last intersected object persist until a new object isintersected [56], or by using more weighting schemes that
continuously update as the cursor moves [14, 22, 55].
Another method to disambiguate between targets is toprovide an explicit mechanism for users to indicate a target
of interest. Techniques such as using a secondary radial
menu [22, 48], supporting progressive refinement using aseries of quad-menus [30], or explicitly controlling a marker
along the depth of the ray [6, 22, 51], have all been proposed.

After describing the results of our studies, we discuss two
potential ways in which the predictive model could
potentially be utilized to improve VR user experiences. First,
pointing facilitation techniques could be enhanced by biasing
towards the model’s predicted region. Second, we discussbroader applications, beyond pointing facilitation, such as
haptic retargeting [12], foveated rendering [2], and latency
reduction techniques [61], which could all be improved if auser's intentions could be inferred prior to the associatedactions occurring.

Ray (or virtual laser) pointing is also a common selection
metaphor to use when interacting at a distance with large
displays [25, 28, 40]. To address the performance detriments
associated with the required angular accuracy, researchers
have explored techniques to increase precision, such as
adapting the CD ratio based on cursor velocity [28].

The key contributions of this work are (1) the adaptation of
KTM endpoint prediction to VR environments, (2) a headcoupled kinematic template matching (HC-KTM) algorithm
that integrates velocity data from both the controller and thehead, and (3) empirical data showing that this new technique
can predict ray pointer landing positions and outperform
more direct adaptations of prior approaches.
RELATED WORKThis work builds upon prior research in VR selection, distantpointing on large displays, cursor endpoint prediction, andgaze-based user interfaces.
VR Selection Techniques

Numerous techniques have been developed to facilitate
pointing in 3D environments (for a comprehensive survey,
see the work of Argelaguet & Andujar [3]). Early research
identified two main classes of selection techniques - raypointing or ray casting [17, 34, 47] and 3D cursors (i.e.,
virtual hand) [18, 24, 35, 48]. Facilitation techniques havebeen developed for 3D cursors [48, 57], but ray-based
approaches have proved to be more efficient [22].

Although these techniques improve the final selection
process, the literature is lacking in ways that can predict thelanding position of a ray pointer while moving.
Distant Pointing on Large Displays

Another approach leverages a technique where ray pointing
is first used for coarse positioning and then for precise
positioning [59]. Nancel et al. thoroughly explore the design
space of such “dual-precision” techniques [41, 42]. Most
related to our work, they propose a “dual-channel” technique
where head orientation provides coarse control of the cursor
and a handheld device handles precise positioning [42]. We
extend this approach by instead leveraging the headmovement to inform a predictive model of the ray pointerlanding position while the controller is still in motion.
Cursor Endpoint Prediction

In desktop configurations, significant efforts have been madeto develop endpoint prediction techniques while the mouseis still in motion. Three main approaches have been proposedin the literature: regression-based extrapolation, target
classification, and kinematic template matching.
With regression-based extrapolation, existing models of
cursor movement behaviors are used to predict the locationof a distant target based on partial movements [3, 25]. Most
successful is Lank et al.’s motion kinematics approach [33],which they subsequently improved to take into account thestability of the prediction [53].

With ray pointing, a cursor emanates like a laser pointer fromthe hand or a 6-DOF handheld controller. Despite its
prominence in current VR platforms, it is difficult to selectsmall and distant targets due to the angular accuracy needed.
One general solution is to use a conic area for the ray [18,35], however, this can require disambiguation techniques if
multiple targets are in the selection region.

An alternative approach is to use target classification, which
integrates knowledge about targets in the environment to
identify the most probable candidate target [39]. Recent
work has shown that Neural Networks and Kalman filters
can be used to predict user intent based on the kinematics ofthe cursor [5, 8]. Ziebart et al. assigned probabilities totargets, using inverse optimal control and Bayes’ rule [63].
While such techniques are promising, they are complex andrequire knowledge of the target locations.

The use of predictive heuristics can enable one toautomatically differentiate between multiple selected targets.

A final approach is Kinematic Template Matching [45]. With
this technique, the velocity profile of a partial pointing

movement is compared to a library of known “template”
movements to predict the final cursor location. Althoughuntested, the authors suggest that this prediction could thenbe combined with target selection techniques such as target
expansion [20] or gravity wells [9]. Template matchingoffers a number of advantages over the other techniques: it istarget-agnostic, user-adaptable, and easy to implement [45].
As such, this work builds upon this approach and will be
reviewed later in more detail.
Outside the domain of 2D cursors, target prediction for
touch-based interfaces has been explored, often to reduce
perceived latency [11, 43, 44]. For example, Xia et al.
leveraged hover information to predict the time and locationof a touch just before it occurred [61]. Ahmad et al. applied
similar predictive models for in-car, mid-air selection [1]. In
VR, LaViola explored filtering techniques for making
predictions to reduce the perceived latency in VR
environments [34]. These techniques have also been used to
predict saccade landing positions to reduce the latency of
foveated rendering in head-mounted displays [2].
Despite the promise of these techniques, we are unaware ofwork that applies prediction models within the realm of VR
ray pointing to predict selection intent.
Gaze and Head Input

Gaze is an established method to provide input to interactive
systems, in particular for contexts which require hands-free
operations [15]. Gaze can also be used as an implicit channel
to detect users’ intent [23]. For example, the MAGIC
technique leverages gaze information to warp the cursor to a
general area of interest [16, 62]. Similar efforts have also
been made in 3D environments, such as combining gaze withdwell or pinch [13, 46, 58]. However, gaze can suffer from
the Midas touch problem, and can be slower than a handcontrolled ray pointer [13].
Cassallas et al. demonstrated that by coupling head and hand
movement features, one can predict intended moving targets[10]. Kyto et al. showed that gaze and head pointing could
be combined with refinement to support the precise selectionof 2D targets in AR [31]. In automotive user interfaces,
Roider and Gross showed that gaze data could be used to
improve the accuracy of pointing gestures [52]. In VR,Cheng et al. presented a technique for predicting user
intention by analyzing gaze and hand movement, so that thehand can be redirected towards a physical proxy [12].
Such work demonstrates the benefit of coupling handcontrolled pointing with gaze or head movements. Our goal
is to build upon such work by using head movement data to
predict the landing position of a ray pointer movement.
REVIEW OF 2D KINEMATIC TEMPLATE MATCHING

The technique proposed in this paper is inspired by Pasqualand Wobbrock’s Kinematic Template Matching (KTM)
technique, one of the more accurate endpoint prediction
techniques in the literature [45]. It also has the practicaladvantages of being both target-agnostic and simple toimplement. The concept behind KTM is to utilize a cursor’s
velocity profile as a 2D stroke gesture, allowing it to berecognized using a template matching algorithm (Figure 2).
The approach uses a four-step process: building a template
library, preprocessing new candidate pointing movements,
matching the template, and estimating the cursor endpoint.
In the first step, a library of templates is generated using acollection of previous pointing movements. Each template
consists of (i) a cursor velocity curve as it progresses towards
the target and (ii) the total distance travelled. The velocity
profiles are truncated to remove overshoots and resampled to
20 Hz. It is important to note that their technique compares auser’s movement to the user’s own template library. This
allows results to be personalized to individual’s pointing
behaviors, at the cost of requiring individual data collection.
The next step occurs when a new pointing movement is being
made (i.e., the candidate movement). In real time, thevelocity profile is resampled to 20 Hz and smoothed using a
Gaussian filter. To prepare for template matching, eachtemplate in the library is truncated to match the duration of
the candidate movement, and then the same smoothing isthen applied. Note that the smoothing of the templates
happens after they are truncated, which was found to lead tobetter matched templates and higher accuracy.
Once preprocessing is completed, the candidate movement iscompared to each library template. This comparison occurs
when each new candidate movement point arrives. A
cumulative scoring function compares the candidate
movement to the template:𝑆𝑆(𝑇𝑇𝑖𝑖 ) = 𝑆𝑆(𝑇𝑇𝑖𝑖∗ ) + � 𝑛𝑛𝑡𝑡

𝑛𝑛

𝑐𝑐
∑𝑗𝑗=0
�𝐶𝐶𝑗𝑗 −𝑇𝑇𝑖𝑖𝑖𝑖 �

𝑛𝑛𝑐𝑐

,

𝑛𝑛𝑐𝑐
∑𝑗𝑗=0�𝐶𝐶𝑗𝑗 −𝑇𝑇𝑖𝑖𝑖𝑖 �+∑𝑗𝑗=𝑛𝑛
𝐶𝐶𝑡𝑡 +1 𝑗𝑗

𝑛𝑛𝑐𝑐,

𝑛𝑛𝑐𝑐 ≤ 𝑛𝑛𝑡𝑡

𝑛𝑛𝑐𝑐 > 𝑛𝑛𝑡𝑡

(1)

where Ti is the ith template in the library; S(Ti*) is the priorcalculated score; Cj and Tij are the jth velocity values from thecandidate and current template smoothed velocity profiles,
respectively; and nc and nt are the number of points in thecandidate and current template smoothed velocity profiles,
respectively. Once the candidate movement has been
compared to all templates, the template with the lowest scoreis selected as the best match.
Finally, to predict the candidate movement’s final endpoint,
the technique uses the travel distance associated with the best
matched template and applies that distance to the current
direction of the candidate’s movement from the original startpoint. As with other endpoint prediction techniques, theaccuracy of KTM improves as the candidate movement
progresses towards the target. It was found that on average,
KTM predicts the endpoint location within 83 pixels of thetrue endpoint when 50% of the movement has been
completed, 48 pixels at 75%, and 39 pixels at 90%. This
performance was better than prior kinematic endpoint
prediction methods [33].

Figure 2. The KTM approach considers the velocity profile ofa movement as a 2D gesture. A partial candidate movement iscompared to a known template, and the endpoint is inferred.
Figure taken from [45]HEAD-COUPLED KINEMATIC TEMPLATE MATCHING

To adapt the KTM approach for ray pointing in VR
environments, we consider a traditional ray pointer whichacts like a virtual laser pointer. The virtual ray has 5 degrees
of freedom: the user must specify the origin (X, Y, Z) anddirection (θh, θv) of the ray. The goal of our technique is topredict the final controller location and direction while thepointer movement is still in progress. We adapt and extend
the KTM technique in three important ways:
1) The KTM method was built for 2D cursor pointing,predicting the (X, Y) coordinates of the movement’s
endpoint. To adapt the technique for 3D ray pointing, weare not predicting an “end point” per se, rather the final
landing position of a ray. Thus, estimates of not only the
3D coordinates of the handheld controller, but also theangle at which the ray is being emitted, are needed.
2) The KTM method only considers the velocity profiles of
the cursor in the template matching procedure. Thus, acursor’s velocity profile across targets with different
distances may not be distinguishable in the first part ofits movement [19, 26]. We extend this method toconsider user head movement, hypothesizing that thisadditional channel may increase prediction accuracy.
We call this head-coupled variation HC-KTM.
3) The KTM method selects one, best matching template,to estimate the endpoint distance. We extend the method
to a top-n approach, where the weighted average ofmultiple matching templates may be used, allowing forthe compensation of a poorly matching individual
template. We call this top-n variation KTM-N.
Together, these 3 enhancements form the basis of our HeadCoupled Kinematic Template Matching (HC-KTM-N) technique, which maintains the desirable properties of KTM. The

Figure 4. Each template consists of four velocities: a)Controller positional velocity. b) Controller angular velocity.
c) Head positional velocity. d) Head angular velocity.

technique is target-agnostic, straight-forward to implement,
and can be personalized to individual users. The approach
follows the same 4 general steps of KTM, described below.
Step 1. Building the Template Library

The template library is built by capturing selection
movements for known targets, considering both the motionof the controller and the head during selection. Because theseare spatial input channels, we must consider both the locationand the angle of the controller and head (Figure 3). As such,
each template consists of four velocity profiles (Figure 4):• CP (mm/s): Controller positional velocity - the change inthe controller’s (X, Y, Z) origin coordinates over time.
• CA (deg/s): Controller angular velocity - the change inangle of the controller’s forward-facing vector over time.
• HP (mm/s): Head positional velocity - the change in the
HMD’s (X, Y, Z) origin coordinates over time.
• HA (deg/s): Head angular velocity - the change in angle ofthe HMD’s forward-facing vector over time.
In the previous movement angle distance KTM technique,the template library was modified to crop any backtracking
from a template; initial testing found that adequate results
were achieved without performing this step and it wasomitted. Unlike KTM, we found it necessary to do an initialsmoothing of the templates, given the noise introduced bymidair 6-DOF devices. A Gaussian smoothing operation was
performed on each of the velocities using a 5-point window.
The profiles were then resampled to 20 Hz in preparation forcomparison to subsequent candidate movements.
Step 2. Preprocessing Candidate Pointing Movements

Figure 3. Top view of a ray pointer acquisition movement.
Both the head and controller change in position and angle.

As a new candidate movement is captured, the position andangle values of the head and controller are collected. They
are used to create the four partial velocity profiles, which aresmoothed using a 5-point Gaussian window and resampled
to 20 Hz as each new point is collected. As with KTM, eachvelocity profile in the template library that is longer induration than the candidate movement is truncated to thesame length as the candidate movement.

Step 3. Matching Candidate Movements

The candidate movement, C, is then compared to eachtemplate, Ti, at the arrival of each new movement point using
the same cumulative controller scoring function presented in
Equation 1. However, in this case, the scoring calculation isrepeated four times, once for each of the four velocity
profiles, resulting in four scores (i.e., Scp, Sca, Shp, and Sha),that correspond to the velocity profiles CP, CA, HP, and HA.
The final cumulative scoring function, S(Ti), is defined as aweighted sum of the four individual scores:(2)

S(Ti) = aScp(Ti) + bSca(Ti) + cShp(Ti) + dSha (Ti)

Where, a, b, c, and d are tuning parameters. Note that by
setting a, c, and d to 0, the model reduces to the KTM modeland uses only the velocity profile of the controller angle.
Step 4. Calculating the Expected Landing Position

Unlike KTM, which considers only top matching templates,
the n-best template matches are ranked by the minimum values of S(Ti). To calculate the expected final movement angle
distance of the ray, a weighted average of the movement angle distances of the top-n templates is computed. The template’s weight, wi, is defined as the reciprocal of S(Ti), andthe template’s movement angle distance as di. Using thesevalues, the weighted average angular distance is calculated:∑𝑛𝑛 (𝑤𝑤𝑖𝑖 ∗𝑑𝑑𝑖𝑖 )μ = 𝑖𝑖=1𝑛𝑛∑𝑖𝑖=1 𝑤𝑤𝑖𝑖

(3)

Using this weighted average angular distance (µ), thecontroller’s initial angle is rotated by the magnitude of µ,along the current angle of motion. The same approach is used
to calculate the expected controller location. Using theweighted average of the top-n template’s controller distances, the magnitude of this average is added to the initial
controller’s position along the current direction of movement. By combining the expected angle and location, thefinal ray pointer landing position is calculated (Figure 5).
EXPERIMENT 1 – HUMAN POINTING BEHAVIOR

The first experiment gathered initial data to better understandhuman head and hand behaviors during ray pointer movements using controlled target distances and sizes. The

Figure 5. To predict the final landing position of theray, the prediction for the final angle and position of
the controller are combined.

Figure 6. First person view of the study environment.

experiment consisted of a pointing task in a VR environment,using a ray pointer, without prediction enabled.
Participants

Seventeen participants (11 female), with no major motor
impairments and normal or corrected-to-normal vision (onlycontact lenses were allowed) were recruited. They ranged inage from 18 to 26 (M=20.9, SD=2.0). Participants werecompensated $30 for their time. A Randot Stereo Optical
Test was administered prior to the experiment to ensure
adequate stereo vision. All participants were right-handed
and operated the controller with their right hand.
Apparatus

The experiment was conducted using an Oculus Rift CV1
head-mounted display, with a resolution of 2160×1200,
using a single Oculus Touch handheld controller for input.
The Index Trigger button was used for selection. The
position and angle of the HMD and controller was tracked
using Oculus Constellation Sensors. The system ran on a 3.7
GHz Intel Core i7-8700k desktop computer with an NVIDIA
GeForce RTX2080 graphics card and was developed inUnity3D. The HMD output updated at a frequency of 90 Hz,
and both the HMD and controller positions and angles were
updated at a rate of 90 Hz. The handheld controller
manipulated a ray pointer using an absolute mapping, withthe ray originating from the tip of the controller, aligned withthe z-axis of the local handheld controller coordinate system.
Procedure

The task was a reciprocal three-dimensional pointing task,
wherein participants pointed back and forth, in succession,
between a start and end target. No distractor targets were
included. The target to be selected was yellow, and the otherwas semi-transparent gray. The background of the scene wasa gray gradient, and in the virtual environment, subjects
stood on an elevated platform above an infinite grid groundplane. The two targets were rendered as spheres (Figure 6).
The goal target turned green when it was intersected by theray, to indicate the target could be selected. Upon successful
selection, the targets swapped colors. If the ray did notintersect the goal target when a button-click occurred, thetrial counted as an error, and the participant tried again until
successful. Subjects were asked to complete the task as
quickly as possible, without exceeding an error rate of 4%.
The error rate was displayed after each block of trials.

same Theta and W) were removed; which was 6% of the data.
All remaining trials were analyzed, however, trials where
errors occurred (1.7%) were only analyzed to the first click.
Movement Time

Figure 7. The target layout. a) Reciprocal targets were locatedon opposite sides of the z-axis at varying depths, with equal
angular widths. b) Targets appeared at one of 8 angles.

During the study, participants stood on a marked floor
position and were asked not to move their feet. The softwareand the experimenter ensured their feet were in the proper
position prior to each trial. To calibrate, prior to the study, thecoordinate system was reset with the participant on a marked
spot with the HMD in a resting state. The experimenter could
recalibrate at any time during the study, if needed. The point
between the eyes was the origin, with the positive axes being:left to right (X), bottom to top (Y), and back to front (Z).
Before each session, participants performed practice trials to
become familiar with the task – lasting about 2 minutes.
DesignA repeated-measures, within-participant design was used.
The position of the goal target varied based on three
independent variables – Depth (3m, 6m, 9m), Theta (25°,50°, 75°), and Position (0°-315° at 45° increments)(Figure 7). Depth manipulated the distance between thetarget center and the origin. Theta changed the magnitude of
the angle between the vectors generated by connecting eachtarget to the origin, with the center vector of these two
vectors laying along the Z-axis. For each combination of
Theta and Depth, there was a ring of possible target locations(i.e., Position), evenly distributed at 45° increments (Figure7b). In each reciprocal task, targets were placed in opposite
locations of the ring, but their depth values could vary(Figure 7a).
A target’s size was determined by its angular width, W (4.5°,9.0°), relative to the origin. With W fixed, the further thetargets the larger the radius; but the angle needed to place theray within its boundaries remained fixed. In each reciprocal
task, the angular width of both targets was equal (Figure 7a).

Movement time was defined as the duration between selecting the start target and the next subsequent selection, regardless of the success of the selection. A repeated measures
analysis of variance showed main effects of Width (F1, 16 =581.5, p < .0001) and Theta (F2, 32 = 693.4, p < .0001). Also,
significant was the interaction between Width and Theta (F2,32 = 7.7, p < .005). Target depth (Depth) did not have asignificant effect on movement time (F2, 32 = 2.8, ns).
Interestingly, across the 6 combinations of Theta and Width,
the performance trend followed an angular derivation of
Fitts’s Law [17, 27], with an extremely high fit (R2 = 0.993)(Figure 8):𝑇𝑇ℎ𝑒𝑒𝑒𝑒𝑒𝑒+ 1�
𝑊𝑊𝑊𝑊𝑊𝑊𝑊𝑊ℎ

𝑀𝑀𝑀𝑀 = 𝑎𝑎 + 𝑏𝑏 log 2 �

While such an angular derivation has been examinedpreviously for 2D large display interactions, our results
contrast prior work. In particular, Jota et al. achieved a fit ofonly 0.61 for horizontal pointing and 0.33 for verticalpointing on large displays [25]. Kopper et al. achieved ahigher fit of 0.96 [29], but used a more complex angularmodel (IDDP), which resulted in a fit of only 0.74 with ourdata. While our data provides an initial validation of Fitts’Law for ray pointing in VR environments, this contrast toprior work warrants future investigation.
Head and Controller Angular Movements

When measuring the cumulative angular distance that thecontroller and HMD travelled, both the HMD Angle (F2, 32 =272.4, p < .0001) and Controller Angle (F2, 32 = 116673, p <.0001) were significantly impacted by Theta. For HMD
angle, there was a significant effect of both Width (F1, 16 =79.1, p < .0001) and Position (F7, 112 = 110.1, p < .0001).
These effects were also observed for Width (F1, 16 = 18.2, p <.005) and Position (F7, 112 = 7.6, p < .0001) on the Controller
Angle. Notably, the HMD moved further for smaller targets– likely, to better see smaller targets. This shows that for a
predictive model, head movement may provide information
not apparent from just the controller movement. Depth did
not have a significant effect on HMD or Controller Angle.

The experiment was done in one, approximately 60 minute,session. The study had 54 blocks for each of the 54 possiblecombinations of Depth (start target), Depth (end target),
Theta, and W in random order. Participants could take breaks
between blocks to prevent fatigue. For each block, 4 sets ofreciprocal trials were performed for the eight Positions (i.e.,
4 pairs), and consisted of nine clicks (i.e., 8 reciprocal
selections between the two targets at opposite positions).
This resulted in 54 x 4 x 8 = 1728 trials per participant.
Results and Analysis

Prior to all analysis, outliers (trials where time was more than
two standard deviations in length compared to trials with the

Figure 8. The effect of width and angular distance on
movement time results in a high degree of fit with anangular derivation of Fitts’ Law.

Figure 9. The HMD and Controller angular movements, withrespect to the Angular Distance to the target. Error bars
represent standard error.

Comparing the magnitude of the angular movement between
the two devices, the HMD moved a fraction of the angle thatthe controller moved (F1, 16 = 422.0, p < .0001) (Figure 9).
This is intuitive – the controller angle is required to move
within the bounds of the goal target, while the HMD isrequired to move just enough so that the target is in the user’s
field of view. This is further illustrated in Figure 10, whichshows a scatterplot of the final focal point of the HMD andController when each target was acquired. It can be seen thatthe controller focus (blue) was within the bounds of thetarget, while the HMD only travelled a fraction of therequired distance, with observable variation between trials.
Head and Controller Velocity Profiles

In addition to looking at the angular movements of the HMD
and controller, it is also interesting to observed how velocity
profile templates differ across different movement angles. To
generate “representative” velocity profiles, the velocity
profiles were resampled to 20 Hz, and the average velocity
at each interval was computed (Figure 11). Consistent withprior findings [19, 26], the velocity profiles for the handheld
controller were not distinguishable in the initial stage ofmovement across the three movement angles. However, thevelocity profiles for the HMD diverge immediately. Thissuggests that incorporating head movements within apredictive model may allow predictions to be made earlier.
Model Parameters and Performance

To analyze the performance of our model and tune its parameters, intra-participant template libraries were created fromeach user’s trials. With outliers removed, each movement

Figure 10. A scatter plot of the final controller (blue) and HMD(orange) locations, for each position. Points are the intersection
with a plane centered at the target location.

was compared against the other templates in the user’s
template library. To calculate accuracy for individual predictions, the angular distance between the predicted ray and theray from the controller origin through the center of theintended target was computed. This angular accuracy
informed the precision of the model, similar to pixel distanceused in 2D models. Since the data collection experiment onlyused three discrete angular distances, it served as an initial
testbed of the model – if the model worked as expected,
candidate motions would match templates with the sameangle of movement.
The model had two main factors to manipulate: how many
templates to use in the top-n matching algorithm and the bestvalues to assign to the weights of the scoring algorithm forthe four velocity profiles (Eqn. 2). To determine n for the
top-n matching templates, four equally weighted components were used (i.e., a, b, c, d = 1), and the cumulative
accuracy of all trials, of all participants, for different valuesof n was calculated. Results improved gradually from n=1 to4, whereas n = 4 to 10 were similar, with n = 7 providing thebest result, which is the value used for our implementation.
To determine the best weighting values for the scoring function (a, b, c and d - Eqn. 2), various values were tried to optimize the model’s accuracy at 40% of the movement progress, prioritizing early prediction. The importance of eachindividual component is shown in Figure 12, showing the average angular accuracy when each component was used individually. The data suggests the HMD angle may provide a
better indicator for the first half of the movement, while the

Figure 11. Representative angular velocity profiles for the HMD and Controller by movement angle. The highlighted regions
illustrate that in the first 150 ms of movement, profiles only diverge for the HMD, not the controller.

The only controlled variable was Theta, which tested allangles from 15° to 85° at 1-degree intervals. All other task
variables were randomized. The Depth of both targets ranged
continuously from 3m to 9m, Position ranged from 0° to359°, and angular width (W) ranged from 4.5° to 9.0°.

Figure 12. The prediction accuracy derived from each input
channel at different stages of the movement.

controller angle may be a more accurate predictor by the end.
This may be due to the velocity profiles of the HMD being
more distinguishable in the first phase of movement (Figure11). This is an important result, as it demonstrates the potential benefits of incorporating data from head movements. By
considering the relative performance of each input channel,
setting the best at 1 and worst at 0.5, and interpolating theremain two values, we derived values of a = 0.95, b = 0.5, c= 0.86, and d = 1. We call this specific model HC-KTM-7,
for “Head-Coupled KTM” with n = 7.
For this controlled study, the calculated parameter values
resulted in average accuracies of 6.4°, 3.3°, and 2.3° at 50%,
70%, and 90% of the way through the task, respectively.
While promising, it is important to note that this should only
be considered initial validation, as the test set only used three
discrete angular distances. The next study validates the HCKTM-7 model with a continuous range of target conditions.
EXPERIMENT 2 – MODEL VALIDATION

A second experiment was conducted to further validate themodel and compare it to baseline approaches. Experiment 1
only examined three movement angles where this second experiment evaluates the robustness of the model against a continuous range of angles, depths, positions, and widths. Testing the model in a second experiment helps avoid the potential over-fitting of model parameters, which were set using
data gathered in Experiment 1. Again, this study collectedmovement behaviors without any prediction enabled.
Participants

A total of 12 participants (9 female) completed Experiment 2
and were recruited from the pool of 17 participants inExperiment 1. Participants were compensated $30.
Apparatus and Procedure

The same apparatus and procedure that was used inExperiment 1 was used in Experiment 2.
DesignA repeated measures within-participant design was used. As
with the first experiment, the task was a reciprocal threedimensional pointing task with no distractor targets.

The experiment was performed in one session lasting
approximately sixty minutes, with each angle of Theta
repeated five times in random order. To prevent fatigue andto allow for breaks, the experiment was divided into 50 equal
sections; each section presented 7 pairs of targets withrandomized Theta values, requiring 6 reciprocal selections.
This resulted in 50 x 7 x 6 = 2100 trials per participant.
Before each session, participants were given practice trials tofamiliarize themselves with the task, lasting about 2 minutes.
Results

All trials were analyzed, however, trials where errors
occurred (2.7%) were only analyzed up until the first click.
To validate our model (HC-KTM-7), it was compared to a
direct adaptation of the kinematic template matching (KTM)
model. Recall, our model varies from KTM in two ways: aweighted average of the top 7 matching templates is used,
and a head-coupled approach is utilized, using 4 velocity
profiles instead of one. As such, we provide a comparison to
KTM-7 (KTM, n=7) and HC-KTM-1 (KTM, with headcoupling), to understand the relative benefit of these twovariations. As in prior endpoint prediction work [63], theresults are also compared to a Baseline that utilizes thecurrent position of the ray without any prediction applied.
The results indicate that HC-KTM-7 performs better than the
KTM technique, and the other 2 intermediate variants (KTM7, HC-KTM-1; Figure 13). This is encouraging, as it shows
both of the proposed enhancements can increase the accuracyof predictions. Compared to the Baseline with no prediction,
HC-KTM-7 improved accuracy by a factor of 2.7x at 40%
progress. However, near the end of the movement, thebaseline seems to be most accurate. This is consistent withprior findings [63] and suggests that adaptive techniques thatcan detect when a movement is ending are promising.
It is interesting that HC-KTM-1 outperforms KTM-7,
indicating that the improvement of our technique primarily
stems from using a head-coupled approach. The differencesin accuracy are most pronounced at 40% progress, withaccuracies of 10.0° for HC-KTM-7, 11.6° for HC-KTM-1,
15.0° for KTM-7, 17.9° for KTM, and 26.4° for the Baseline.
Overall, our new model, HC-KTM-7 provided promising
results. Prediction accuracy was 7.3° at 50% of the way
through the task, 4.4° at 70%, and 3.4° at 90%. When looking
at the target hit rate, i.e., how often predictions fell within thebounds of the goal target, HC-KTM-7 outperforms the basic
KTM condition, achieving an average hit rate of 46.2%, 70%of the way through a movement, compared to 37.6% (Table1). While these predictions can still be improved, they are on
par with prior work which used a similar protocol [33, 53],
but are also dependent on the width of the goal targets.

Target Hit Rate
Distance
Travelled

KTM(n = 1)KTM-7(n = 7)

HC-KTM-1(n = 1)

HC-KTM-7(n = 7)

50%

15.1%

14.8%

12.2%

22.7%

60%

26.1%

28.0%

18.4%

34.8%

70%

37.6%

42.9%

23.8%

46.2%

80%

47.8%

56.2%

28.5%

55.6%

90%

55.2%

65.8%

31.8%

61.8%

Table 1. The percentage of predictcions where the predicted
ray hits the intended target; for 4 main conditions.
Figure 13. Accuracy curves for the Baseline and 4 variantsof the model, i.e., HC-KTM-7, HC-KTM-1, KTM-7, KTM.

Furthermore, to utilize the model, a target hit is only required
to predict the intended target if it is in a densely populated
area. Thus, there may be a novel opportunity to combine
selection refinement techniques [21, 30] with this predictivemodel in some instances. Alternatively, the model could also
be used to inform interface layouts, ensuring that adequate
spacing between targets in VR environments is provided.

model were only based on 2D angular data of the ray, thenthe predictions would be less accurate. Fortunately, mostvirtual ray pointing controllers provide full 6-DOF input.
Of course, HC-KTM-7 is only one possible predictive model.
Comparing our results to similar efforts such as regressionbased extrapolation and target classification could yield interesting results. Future models could include probability
distributions across possible targets to influence predictions.
Limitations

To examine the necessity of personalized template libraries,
accuracies were calculated for each participant when using
other user’s template libraries (Table 2). In all cases,
matching against the participant’s own template library,achieved the best accuracies. The average result whencomparing to other templates is still within 3° (9.96° vs.
12.94°). It is interesting to note that that personalized data
may be more important for some users (e.g. P3) likely
depending on the uniqueness of their pointing behaviors. For
example, the angular head movements of P3 were only
29.2% of the average across all other participants.

One aspect of 3D VR pointing that was not addressed, is thatmultiple targets can be located along the same projected pathat varying depths, requiring disambiguation. The HC-KTM
model only predicts the location of the ray, not the depth of
the target. While the study varied the target depth, the analysis did not show any effect from depth on pointing behaviors,
making this a challenging problem for prediction. Future
work should explore additional input channels, such as gaze[60], to predict object depth and extend the model into a truly
3D prediction. Selection refinement techniques [21, 30]
could also be used when multiple targets fall along the ray.

DISCUSSION AND FUTURE WORK

Our study also only examined movements across the center
point of the user’s field of view. Our initial implementation
indicates the captured templates are still applicable (just as
templates are applicable across direction of movement). Our
hope is that the top-n approach is useful in this context. Thiscertainly requires careful evaluation and validation in the future. The alternative, capturing movements for every possible target location, would probably not be feasible, due to theresulting size of the template library.

The presented model shows promise for VR ray-pointerpredictions. By modifying the KTM model and introducing
head-coupling and considering the top-n matches, the HCKTM-7 model’s predictions were 1.8x and 2.7x moreaccurate than KTM and the baseline respectively, 40% of theway through a user’s movements.
Although developed and evaluated within a 3D VR
environment, the model should generalize to 2D platforms.
Targets were shown in 3D space, but the task itself could be
decomposed into the 2D angular movements of the ray
pointer. Specifically, it would be interesting to use the modelfor distant pointing on large, high resolution displays, where
2D angular ray pointing is also used. Within this context, thepresent work can be seen as building on prior literature which
has coupled head and hand movements to divide large
display pointing into coarse and precise modes [42]. Notably,
while the model can be applied to 2D tasks, it still requires
and uses 3D input channels. In particular, the model must
estimate the final 3D position of the controller (Figure 5).
The 3D movements of the hand can be substantial – onaverage the hand moved 18.7cm, and as much as 45cm insome trials (average maximum across all participants). If theAnother limitation of the study was the lack of distractor targets, which were omitted to simplify the task environmentand capture raw target acquisition movements. The visualpresence of distractors could influence a user’s behavior,

Table 2. Accuracies (at 40%) for each participant (rows) whenusing another user’s template library (columns).

which could in turn interfere with the model. In particular, auser may duck, lean, or reach around distractors to get a goodview of their intended target. Incorporating, or filtering out,
such behaviors is an interesting topic for future studies.
Our study may also be limited in that the same participant
pool was used for both studies. It is possible that the parameter values which were tuned may have been biased towards
this participant pool, and validating across a larger spectrumof users would be worthwhile future work.
Last, our results show predictions may not be as accurate as
the actual ray position during the final stage of pointing. This
creates an interesting opportunity for a hybrid approach,using our model in the early ballistic phase of pointing, andconsiders the current ray position in the adjustment phase.
Personalization of Template Libraries

An advantage of the model personalization is that it can betuned to each user; a drawback is that training data is needed.
One solution is to start a new user with a generic template
library; slowly replacing that library with the user’s own data
as movements are collected. There may also be classes ofusers with similar behaviors, who could share predetermined
template libraries. For example, users could be classified
based on the extent with which they tend to move their head.

early stages of the movement. For a snapping technique, therequired prediction accuracy would depend on target layout.
In a dense environment, the prediction could be drawn to a
nearby, inaccurate target. Based on our study results, we
believe our model could work well for target snapping when
there is an average of approximately 7° between targets. This
would be large enough for larger UI buttons (e.g. the Oculus
Quest Home screen buttons, which range from around 10° 20°), but may be inadequate when selecting small and dense
scene content in a VR environment.
It is important to note that any facilitation technique is likely
to change the user’s behaviors due to the perception/action
loop in target acquisition [37]. This in turn may impact themodel’s performance. This work collected data when
prediction was inactive but future studies should collect data
and determine user behaviors when predictions are active.
Other Potential Applications

Complexity

Predictive modelling could improve other aspects of VR user
experiences beyond target selection. Recent work has shown
the potential of haptic retargeting in VR [6], where a user’s
movements are biased towards a physical proxy. The successof such techniques required the ability to predict what regiona user is moving towards [12]. Similarly, it has been recently
shown that foveated rendering [2], which provides a higher
resolution rendering at the user’s point of focus, can benefit
from predicting where a user is going to gaze next. Byleveraging our technique, such predictions could be madeearly enough to be unnoticeable by the user. Finally, by
predicting earlier what a user will do, associated processing
could begin proactively, reducing latency for associatedintended actions. This could be especially useful in VR
where the reduction of latency is particularly important.

Selection Facilitation Techniques

It is important to note that our technique is target agnostic,
and as such predicts regions and not targets. In some use
cases, (e.g. foveated rendering, CD gain adjustment), thetechnique could be useful on its own, without making a
specific target prediction. In other cases, the model could be
paired with facilitation techniques (e.g. target snapping) thatare target-aware. In such circumstances, the predicted
regions could be used to identify target likelihoods (e.g.
[63]). This would be a useful area of future research.

A related factor is the number of templates that are in an
individual’s template library. Our implementation usedapproximately 2000 templates for each user. The model wasable to run in real-time, with minimal optimization, with noperceivable impact on performance, at an input rate of 90 Hz.
With each incoming input event frame, a prediction occurs
in just under 11ms. However, any technique that requires
additional resources must be used with caution, given thenature of computation-heavy virtual environments. Limits
should be placed on the template library size, and thepredictions could be performed on a separate thread.
An exciting line of future research is developing new VR
pointing enabling techniques that use movement predictions.
Techniques that dynamically adapt the CD ratio [28] could
benefit particularly from early prediction. As the user
initially moves the cursor, it could accelerate towards thepredicted region, and decelerate when it arrives. As ourmodel performs best at the early stages of movement (e.g.
40% mark), this type of facilitation may be particularly
suitable for our model. In this case, the predicted landing
position would not need to be precisely located at theintended target, as the technique would benefit fromaccelerating towards the general target region.
Alternatively, target snapping (e.g. [21, 57]) could benefit
from early predictions – instead of just snapping to theclosest target to the cursor, the technique could snap to theclosest target in the predicted region. This could support
faster access to targets, when predictions are made in the

CONCLUSION

This work has demonstrated that by using an adapted
kinematic template matching technique (HC-KTM-7), raypointer landing positions can be predicted early in a user’s
movement. A key insight is that it is beneficial to integrate
the movements of the head into the model. The presented
approach increases landing position prediction accuracy by afactor of 1.8x at 40% of the way through a movement, whencompared to traditional KTM, and by a factor of 2.7x whencompared to a baseline using the current ray position. This
work has provided a first exploration into pointer prediction
in VR, opening the door for future enhancements to 3D user
experiences.

REFERENCES

1.

2.

Bashar I. Ahmad, Patrick M. Langdon, Simon J.
Godsill, Richard Donkor, Rebecca Wilde, and Lee
Skrypchuk. 2016. You Do Not Have to Touch to
Select: A Study on Predictive In-car Touchscreen with
Mid-air Selection. In Proceedings of the 8thInternational Conference on Automotive User
Interfaces and Interactive Vehicular Applications(Automotive'UI 16). ACM, New York, NY, USA, 113120. DOI: https://doi.org/10.1145/3003715.3005461
Elena Arabadzhiyska, Okan Tarhan Tursun, Karol
Myszkowski, Hans-Peter Seidel, and Piotr Didyk.
2017. Saccade landing position prediction for gazecontingent rendering. ACM Trans. Graph. 36, 4,
Article 50 (July 2017), 12 pages. DOI:
https://doi.org/10.1145/3072959.3073642

3.

Ferran Argelaguet, Carlos Andujar. 2013. A survey of
3D object selection techniques for virtualenvironments. Computers & Graphics, 37,(3), 121136. DOI: https://doi.org/10.1016/j.cag.2012.12.003

4.

Takeshi Asano, Ehud Sharlin, Yoshifumi Kitamura,
Kazuki Takashima, and Fumio Kishino. 2005.
Predictive interaction using the delphian desktop. In
Proceedings of the 18th annual ACM symposium on
User interface software and technology (UIST '05).
ACM, New York, NY, USA, 133-141. DOI:
http://dx.doi.org/10.1145/1095034.1095058
Gökçen Aslan Aydemir, Patrick M. Langdon,
Simon Godsil. 2013. User target intention recognition
from cursor position using kalman filter.
In International Conference on Universal Access in
Human-Computer Interaction. 419-426. Springer,
Berlin, Heidelberg. DOI: 10.1007/978-3-642-391880_45
Mahdi Azmandian, Mark Hancock, Hrvoje Benko,
Eyal Ofek, and Andrew D. Wilson. 2016. Haptic
Retargeting: Dynamic Repurposing of Passive Haptics
for Enhanced Virtual Reality Experiences.
In Proceedings of the 2016 CHI Conference on Human
Factors in Computing Systems (CHI '16). ACM, New
York, NY, USA, 1968-1979. DOI:
https://doi.org/10.1145/2858036.2858226
Marc Baloup, Thomas Pietrzak, and Géry Casiez.
2019. RayCursor: A 3D Pointing Facilitation
Technique based on Raycasting. In Proceedings of the2019 CHI Conference on Human Factors in
Computing Systems (CHI '19). ACM, New York, NY,
USA, Paper 101, 12 pages. DOI:
https://doi.org/10.1145/3290605.3300331
Pradipta Biswas, Gokcen Aslan Aydemir, Pat Langdon,
Simon Godsill. 2013. Intent recognition using neuralnetworks and Kalman filters. In International
Workshop on Human-Computer Interaction andKnowledge Discovery in Complex, Unstructured, Big
Data. 112-123. Springer, Berlin, Heidelberg. DOI:
10.1007/978-3-642-39146-0_11

5.

6.

7.

8.

9.

Renaud Blanch, Yves Guiard, and Michel BeaudouinLafon. 2004. Semantic pointing: improving target
acquisition with control-display ratio adaptation. In
Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (CHI '04). ACM, New
York, NY, USA, 519-526. DOI:
https://doi.org/10.1145/985692.985758
10. Juan Sebastian Casallas, James H. Oliver, Jonathan W.
Kelly, Frederic Merienne, Samir Garbaya. 2014. Using
relative head and hand-target features to predict
intention in 3D moving-target selection. In 2014 IEEE
Virtual Reality (VR). 51-56. IEEE. DOI:
10.1109/VR.2014.6802050
11. Elie Cattan, Amélie Rochet-Capellan, Pascal Perrier,and François Bérard. 2015. Reducing Latency with aContinuous Prediction: Effects on Users' Performancein Direct-Touch Target Acquisitions. In Proceedings ofthe 2015 International Conference on Interactive
Tabletops & Surfaces (ITS '15). ACM, New York, NY,
USA, 205-214. DOI:
https://doi.org/10.1145/2817721.2817736
12. Lung-Pan Cheng, Eyal Ofek, Christian Holz, Hrvoje
Benko, and Andrew D. Wilson. 2017. Sparse Haptic
Proxy: Touch Feedback in Virtual Environments Usinga General Passive Prop. In Proceedings of the 2017
CHI Conference on Human Factors in Computing
Systems (CHI '17). ACM, New York, NY, USA, 37183728. DOI: https://doi.org/10.1145/3025453.3025753
13. Nathan Cournia, John D. Smith, and Andrew T.
Duchowski. 2003. Gaze- vs. hand-based pointing invirtual environments. In CHI '03 Extended Abstracts
on Human Factors in Computing Systems (CHI EA'03). ACM, New York, NY, USA, 772-773. DOI:
http://dx.doi.org/10.1145/765891.765982
14. Haan, Gerwin de and Koutek, Michal and Post, Frits H.
2005. IntenSelect: Using Dynamic Object Rating for
Assisting 3D Object Selection. In IPT/EGVE 2005.
201-209. The Eurographics Association. DOI:
10.2312/EGVE/IPT_EGVE2005/201-209
15. Augusto Esteves, Eduardo Velloso, Andreas Bulling,and Hans Gellersen. 2015. Orbits: Gaze Interaction for
Smart Watches using Smooth Pursuit Eye Movements.
In Proceedings of the 28th Annual ACM Symposium on
User Interface Software & Technology (UIST '15).
ACM, New York, NY, USA, 457-466. DOI:
https://doi.org/10.1145/2807442.2807499
16. Ribel Fares, Shaomin Fang, and Oleg Komogortsev.
2013. Can we beat the mouse with MAGIC?. In
Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (CHI '13). ACM, New
York, NY, USA, 1387-1390. DOI:
https://doi.org/10.1145/2470654.2466183
17. Paul M. Fitts. 1954. The information capacity of thehuman motor system in controlling the amplitude of
movement. Journal of experimental psychology, 47(6).
381-391. DOI: http://dx.doi.org/10.1037/h0055392

18. Andrew Forsberg, Kenneth Herndon, and RobertZeleznik. 1996. Aperture based selection for immersive
virtual environments. In Proceedings of the 9th annual
ACM symposium on User interface software andtechnology (UIST '96). ACM, New York, NY, USA,95-96. DOI: http://dx.doi.org/10.1145/237091.237105
19. C. C. A. M. Gielen, K. van den Oosten, F. Pull ter
Gunne. 1985. Relation Between EMG Activation
Patterns and Kinematic Properties of Aimed Arm
Movements, Journal of Motor Behavior, 17(4). 421442. DOI: 10.1080/00222895.1985.10735359
20. Tovi Grossman and Ravin Balakrishnan. 2004.
Pointing at trivariate targets in 3D environments. In
Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (CHI '04). ACM, New
York, NY, USA, 447-454. DOI:
https://doi.org/10.1145/985692.985749
21. Tovi Grossman and Ravin Balakrishnan. 2005. The
bubble cursor: enhancing target acquisition by dynamic
resizing of the cursor's activation area. In Proceedingsof the SIGCHI Conference on Human Factors in
Computing Systems (CHI '05). ACM, New York, NY,
USA, 281-290. DOI:
https://doi.org/10.1145/1054972.1055012
22. Tovi Grossman and Ravin Balakrishnan. 2006. The
design and evaluation of selection techniques for 3Dvolumetric displays. In Proceedings of the 19th annual
ACM symposium on User interface software andtechnology (UIST '06). ACM, New York, NY, USA, 312. DOI: https://doi.org/10.1145/1166253.1166257
23. Qi Guo and Eugene Agichtein. 2010. Ready to buy orjust browsing?: detecting web searcher goals frominteraction data. In Proceedings of the 33rd
international ACM SIGIR conference on Research anddevelopment in information retrieval (SIGIR '10).
ACM, New York, NY, USA, 130-137. DOI:
https://doi.org/10.1145/1835449.1835473
24. Ken Hinckley, Randy Pausch, John C. Goble, and Neal
F. Kassell. 1994. A survey of design issues in spatial
input. In Proceedings of the 7th annual ACM
symposium on User interface software and technology(UIST '94). ACM, New York, NY, USA, 213-222.
DOI: http://dx.doi.org/10.1145/192426.192501
25. Ricardo Jota, Miguel A. Nacenta, Joaquim A. Jorge,
Sheelagh Carpendale, and Saul Greenberg. 2010. A
comparison of ray pointing techniques for very large
displays. In Proceedings of Graphics Interface 2010
(GI '10). Canadian Information Processing Society,
Toronto, Ont., Canada, Canada, 269-276.
26. Hilde Keuning-van Oirschot and Adrian J. M.
Houtsma. 2001. Cursor displacement and velocity
profiles for targets in various locations. In Proceedingsof Eurohaptics. 108-112.
27. George V. Kondrask. 1994. An Angular Motion Fitts'
Law for Human Performance Modeling and Prediction.
In Proceedings of Engineering in Medicine andBiology Society. 307.

28. Werner A. König, Jens Gerken, Stefan Dierdorf, andHarald Reiterer. 2009. Adaptive Pointing --- Design
and Evaluation of a Precision Enhancing Technique for
Absolute Pointing Devices. In Proceedings of the 12th
IFIP TC 13 International Conference on HumanComputer Interaction: Part I(INTERACT '09), Tom
Gross, Jan Gulliksen, Paula Kotzé, Lars Oestreicher,Philippe Palanque, Raquel Oliveira Prates, and Marco
Winckler (Eds.). Springer-Verlag, Berlin, Heidelberg,
658-671. DOI: https://doi.org/10.1007/978-3-64203655-2_73
29. Regis Kopper, Doug A.Bowman, Mara G.Silva, Ryan
P.McMahan. 2010. A human motor behavior model for
distal pointing tasks. International journal of humancomputer studies, 68(10). 603-615. Elsevier. DOI:
10.1016/j.ijhcs.2010.05.001
30. Regis Kopper, Felipe Bacim, Doug A. Bowman. 2011.
Rapid and accurate 3D selection by progressiverefinement. In 2011 IEEE Symposium on 3D User
Interfaces (3DUI). 67-74. IEEE. DOI:
10.1109/3DUI.2011.5759219
31. Mikko Kytö, Barrett Ens, Thammathip Piumsomboon,
Gun A. Lee, and Mark Billinghurst. 2018. Pinpointing:
Precise Head- and Eye-Based Target Selection forAugmented Reality. In Proceedings of the 2018 CHI
Conference on Human Factors in Computing Systems(CHI '18). ACM, New York, NY, USA, Paper 81, 14
pages. DOI: https://doi.org/10.1145/3173574.3173655
32. David M. Lane, S. Camille Peres, Aniko Sandor, H.
Albert Napier. 2005. A process for anticipating andexecuting icon selection in graphical userinterfaces. International Journal of Human-Computer
Interaction, 19(2). 241-252. DOI:
10.1207/s15327590ijhc1902_5
33. Edward Lank, Yi-Chun Nikko Cheng, and Jaime Ruiz.
2007. Endpoint prediction using motion kinematics. In
Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (CHI '07). ACM, New
York, NY, USA, 637-646. DOI:
https://doi.org/10.1145/1240624.1240724
34. Joseph J. LaViola. 2003. Double exponential
smoothing: an alternative to Kalman filter-based
predictive tracking. In Proceedings of the workshop on
Virtual environments 2003 (EGVE '03). ACM, New
York, NY, USA, 199-206. DOI:
http://dx.doi.org/10.1145/769953.769976
35. Jiandong Liang, Mark Green. 1994. JDCAD: A highlyinteractive 3D modeling system. Computers andGraphics. 18(4). 499-506. Elsevier. DOI:
10.1016/0097-8493(94)90062-0
36. Michael McGuffin and Ravin Balakrishnan. 2002.
Acquisition of expanding targets. In Proceedings of the
SIGCHI Conference on Human Factors in Computing
Systems (CHI '02). ACM, New York, NY, USA, 57-64.
DOI: https://doi.org/10.1145/503376.503388
37. David E. Meyer, Richard A. Abrams, Sylvan
Kornblum, Charles E. Wright, JE Keith Smith. 1988.

Optimality in human motor performance: ideal controlof rapid aimed movements. Psychological review,
95(3), 340.
38. Mark R. Mine. 1995. Virtual environment interaction
techniques. UNC Technical Report. TR95-018.
39. Atsuo Murata. 1998. Improvement of pointing time by
predicting targets in pointing with a PC mouse.
International Journal of Human-Computer Interaction,10(1). 23-32. DOI:
https://doi.org/10.1207/s15327590ijhc1001_2
40. Brad A. Myers, Rishi Bhatnagar, Jeffrey Nichols,
Choon Hong Peck, Dave Kong, Robert Miller, and A.
Chris Long. 2002. Interacting at a distance: measuring
the performance of laser pointers and other devices. In
Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (CHI '02). ACM, New
York, NY, USA, 33-40. DOI:
https://doi.org/10.1145/503376.503383
41. Mathieu Nancel, Olivier Chapuis, Emmanuel Pietriga,
Xing-Dong Yang, Pourang P. Irani, and Michel
Beaudouin-Lafon. 2013. High-precision pointing onlarge wall displays using small handheld devices. In
Proceedings of the SIGCHI Conference on Human
Factors in Computing Systems (CHI '13). ACM, New
York, NY, USA, 831-840. DOI:
https://doi.org/10.1145/2470654.2470773
42. Mathieu Nancel, Emmanuel Pietriga, Olivier Chapuis,
and Michel Beaudouin-Lafon. 2015. Mid-Air Pointing
on Ultra-Walls. ACM Trans. Comput.-Hum. Interact.
22, 5, Article 21 (August 2015), 62 pages. DOI:
https://doi.org/10.1145/2766448
43. Mathieu Nancel, Daniel Vogel, Bruno De Araujo,
Ricardo Jota, and Géry Casiez. 2016. Next-Point
Prediction Metrics for Perceived Spatial Errors. In
Proceedings of the 29th Annual Symposium on User
Interface Software and Technology (UIST '16). ACM,
New York, NY, USA, 271-285. DOI:
https://doi.org/10.1145/2984511.2984590
44. Mathieu Nancel, Stanislav Aranovskiy, Rosane
Ushirobira, Denis Efimov, Sebastien Poulmane,
Nicolas Roussel, and Géry Casiez. 2018. Next-Point
Prediction for Direct Touch Using Finite-Time
Derivative Estimation. In Proceedings of the 31st
Annual ACM Symposium on User Interface Software
and Technology (UIST '18). ACM, New York, NY,
USA, 793-807. DOI:
https://doi.org/10.1145/3242587.3242646
45. Phillip T. Pasqual and Jacob O. Wobbrock. 2014.
Mouse pointing endpoint prediction using kinematic
template matching. In Proceedings of the SIGCHI
Conference on Human Factors in Computing Systems(CHI '14). ACM, New York, NY, USA, 743-752. DOI:
https://doi.org/10.1145/2556288.2557406
46. Ken Pfeuffer, Benedikt Mayer, Diako Mardanbegi, andHans Gellersen. 2017. Gaze + pinch interaction in
virtual reality. In Proceedings of the 5th Symposium on

Spatial User Interaction (SUI '17). ACM, New York,NY, USA, 99-108. DOI:
https://doi.org/10.1145/3131277.3132180
47. Jeffrey S. Pierce, Andrew S. Forsberg, Matthew J.
Conway, Seung Hong, Robert C. Zeleznik, and MarkR. Mine. 1997. Image plane interaction techniques in
3D immersive environments. In Proceedings of the1997 symposium on Interactive 3D graphics (I3D '97).
ACM, New York, NY, USA, 39-ff.. DOI:
http://dx.doi.org/10.1145/253284.253303
48. Ivan Poupyrev, Mark Billinghurst, Suzanne Weghorst,
and Tadao Ichikawa. 1996. The go-go interaction
technique: non-linear mapping for direct manipulationin VR. In Proceedings of the 9th annual ACM
symposium on User interface software and technology(UIST '96). ACM, New York, NY, USA, 79-80. DOI:
http://dx.doi.org/10.1145/237091.237102
49. Kai Riege, Thorsten Holtkamper, Gerold Wesche, andBernd Frohlich. 2006. The Bent Pick Ray: An
Extended Pointing Technique for Multi-User
Interaction. In Proceedings of the 3D User Interfaces(3DUI '06). IEEE Computer Society, Washington, DC,
USA, 62-65. DOI:
https://doi.org/10.1109/VR.2006.127
50. Gang Ren, Eamonn O'Neill. 2013. 3D selection withfreehand gesture. Computers & Graphics, 37(3). 101–
120. DOI: https://doi.org/10.1016/j.cag.2012.12.006
51. Hyocheol Ro, Seungho Chae, Inhwan Kim, Junghyun
Byun, Yoonsik Yang, Yoonjung Park, Tackdon Han.
2017. A dynamic depth-variable ray-casting interface
for object manipulation in AR environments. In 2017
IEEE International Conference on Systems, Man, andCybernetics (SMC). 2873-2878. IEEE. DOI:
10.1109/SMC.2017.8123063
52. Florian Roider and Tom Gross. 2018. I See Your Point:
Integrating Gaze to Enhance Pointing Gesture
Accuracy While Driving. In Proceedings of the 10th
International Conference on Automotive User
Interfaces and Interactive Vehicular Applications(AutomotiveUI '18). ACM, New York, NY, USA, 351358. DOI: https://doi.org/10.1145/3239060.3239084
53. Jaime Ruiz, Edward Lank. 2009. Effects of target sizeand distance on kinematic endpoint prediction.
Technical Report CS-2009-25, University of Waterloo.
54. Jaime Ruiz and Edward Lank. 2010. Speeding pointing
in tiled widgets: understanding the effects of target
expansion and misprediction. In Proceedings of the
15th international conference on Intelligent userinterfaces (IUI '10). ACM, New York, NY, USA, 229238. DOI: https://doi.org/10.1145/1719970.1720002
55. G. Schmidt, Y. Baillot, D.G. Brown, E.B. Tomlin, J.E.
Swan. 2006. Toward disambiguating multiple
selections for frustum-based pointing. In 3D User
Interfaces (3DUI'06). 87-94. IEEE. DOI:
10.1109/VR.2006.133
56. Frank Steinicke, Timo Ropinski, Klaus Hinrichs. 2006.
Object selection in virtual environments using animproved virtual pointer metaphor. In Computer Vision
and Graphics. 320-326. Springer, Dordrecht. DOI:
10.1007/1-4020-4179-9_46
57. Lode Vanacken, Tovi Grossman, Karin Coninx. 2007.
March. Exploring the effects of environment density
and target visibility on object selection in 3D virtualenvironments. In IEEE 3D user interfaces. 117-124.
IEEE. DOI: 10.1109/3DUI.2007.340783
58. Eduardo Velloso, Jayson Turner, Jason Alexander,
Andreas Bulling, Hans Gellersen. 2015. An empiricalinvestigation of gaze selection in mid-air gestural 3D
manipulation. In Human-Computer Interaction. 315330. Springer. DOI: 10.1007/978-3-319-22668-2_25
59. Daniel Vogel and Ravin Balakrishnan. 2005. Distant
freehand pointing and clicking on very large, highresolution displays. In Proceedings of the 18th annual
ACM symposium on User interface software andtechnology (UIST '05). ACM, New York, NY, USA,33-42. DOI:
http://dx.doi.org/10.1145/1095034.1095041
60. Martin Weier, Thorsten Roth, André Hinkenjann, andPhilipp Slusallek. 2018. Predicting the gaze depth inhead-mounted displays using multiple feature
regression. In Proceedings of the 2018 ACM

Symposium on Eye Tracking Research & Applications(ETRA '18). ACM, New York, NY, USA, Article 19, 9
pages. DOI: https://doi.org/10.1145/3204493.3204547
61. Haijun Xia, Ricardo Jota, Benjamin McCanny, Zhe Yu,
Clifton Forlines, Karan Singh, and Daniel Wigdor.
2014. Zero-latency tapping: using hover information topredict touch locations and eliminate touchdown
latency. In Proceedings of the 27th annual ACM
symposium on User interface software and technology(UIST '14). ACM, New York, NY, USA, 205-214.
DOI: https://doi.org/10.1145/2642918.2647348
62. Shumin Zhai, Carlos Morimoto, and Steven Ihde. 1999.
Manual and gaze input cascaded (MAGIC) pointing. In
Proceedings of the SIGCHI conference on Human
Factors in Computing Systems (CHI '99). ACM, New
York, NY, USA, 246-253. DOI:
http://dx.doi.org/10.1145/302979.303053
63. Brian Ziebart, Anind Dey, and J. Andrew Bagnell.
2012. Probabilistic pointing target prediction via
inverse optimal control. In Proceedings of the 2012
ACM international conference on Intelligent User
Interfaces (IUI '12). ACM, New York, NY, USA, 1-10.
DOI: https://doi.org/10.1145/2166966.2166968