# Investigating Hand-Bound Pads for AR Input Using Hand-Tracking Only

Camille Dupré, Emmanuel Pietriga, Olivier Gladin, Stéphanie Rey, Houssem Saidi, Caroline Appert

► To cite this version:

Camille Dupré, Emmanuel Pietriga, Olivier Gladin, Stéphanie Rey, Houssem Saidi, et al. Investigating Hand-Bound Pads for AR Input Using Hand-Tracking Only. Proceedings of the ACM on Human-Computer Interaction, 2025, 9 (5), pp.MHCI018. 10.1145/3743707. hal-05140350

HAL Id: hal-05140350

https://inria.hal.science/hal-05140350v1

Submitted on 2 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Investigating Hand-Bound Pads for AR Input Using
Hand-Tracking Only

CAMILLE DUPRÉ, Université Paris-Saclay, CNRS, Inria, France and Carl Berger-Levrault, France

EMMANUEL PIETRIGA, Université Paris-Saclay, CNRS, Inria, France

OLIVIER GLADIN, Inria, France

STÉPHANIE REY, Berger-Levrault, France

HOUSSEM SAIDI, Carl Berger-Levrault, France

CAROLINE APPERT, Université Paris-Saclay, CNRS, Inria, France

<figure><img src="image_2.png" /><figcaption>Fig. 1. Using a hand-bound pad for AR input. Sample interactions: (a) discrete input to manage a virtual meeting; (b) 1D continuous input to scrub through a video; (c) 2D continuous input to control a cursor over a document.</figcaption></figure>

Interaction in Augmented Reality primarily relies on raycast pointing and mid-air touch. An alternative consists of using the non-dominant hand as a touch-sensitive surface, enabling more comfortable, less fatiguing input. AR UI design guidelines have so far discouraged this alternative because of poor hand tracking performance when the hands overlap, favoring touchpads *in the air* near the hand, rather than *on* the hand. But significant improvements to the hand tracking capabilities of recent commodity headsets suggest that on-hand pads may now be feasible. We develop an on-hand touchpad prototype and conduct two studies that involve both discrete input and continuous control tasks. The first study compares such *on-hand* pads to baseline *in-air* and *on-object* pads, showing comparable performance despite some limitations in tracking accuracy. The second study quantifies the advantage of on-hand and in-air pads over on-object pads during transitions between touchpad input and other physical hand activities.

CCS Concepts: • **Human-centered computing** → **Interaction techniques; Mixed / augmented reality; Gestural input.**

Additional Key Words and Phrases: augmented reality; touch input

Authors' Contact Information: Camille Dupré, LISN, Université Paris-Saclay, CNRS, Inria, Gif-sur-Yvette, France and Carl Berger-Levrault, Paris, France, camille.dupre@universite-paris-saclay.fr; Emmanuel Pietriga, LISN, Université Paris-Saclay, CNRS, Inria, Gif-sur-Yvette, France, emmanuel.pietriga@inria.fr; Olivier Gladin, SED Saclay, Inria, Palaiseau, France, olivier.gladin@inria.fr; Stéphanie Rey, Berger-Levrault, Toulouse, France, stephanie.rey@berger-levrault.com; Houssem Saidi, Carl Berger-Levrault, Paris, France, houssem.saidi@berger-levrault.com; Caroline Appert, LISN, Université Paris-Saclay, CNRS, Inria, Orsay, France, caroline.appert@universite-paris-saclay.fr.

<figure><img src="image_3.png" /></figure>

This work is licensed under a Creative Commons Attribution 4.0 International License.

© 2025 Copyright held by the owner/author(s).

ACM 2573-0142/2025/9-ARTpn8859

https://doi.org/10.1145/3743707
**ACM Reference Format:**

Camille Dupré, Emmanuel Pietriga, Olivier Gladin, Stéphanie Rey, Houssem Saidi, and Caroline Appert. 2025. Investigating Hand-Bound Pads for AR Input Using Hand-Tracking Only. *Proc. ACM Hum.-Comput. Interact.* 9, 5, Article pn8859 (September 2025), 25 pages. https://doi.org/10.1145/3743707

# 1 Introduction

Interaction in Augmented Reality involves not only immersive 3D scenes but a variety of 2D applications as well in spatial computing scenarios. AR headsets such as the Microsoft HoloLens and the Apple Vision Pro showcase many such scenarios where users watch videos, browse the Web, read documents, or work with productivity tools [1]. These interfaces typically appear as virtual windows, either anchored in the physical environment or in the user's field of view. Designing input techniques that let users interact effectively with this content remains challenging [29], however. Techniques such as mid-air gestures or raycast-based input often require users to keep their arms raised at gaze level, causing fatigue and raising concerns about social acceptance [24, 40, 44, 58]. Indirect interaction via touchpads offers an interesting alternative, allowing users to control a cursor displayed in AR by moving their finger on a pad located in an easy-to-reach zone. Not only is this approach more comfortable [4], but it also facilitates a seamless transition from familiar desktop experiences to spatial computing environments.

Such touchpads can be anchored in the physical environment, typically on flat surfaces such as walls or tables (e.g., [11, 56]). But in mobile situations, where physical surfaces are not always within reach, a touchpad can be attached to the non-dominant hand [6]. Such hand-bound touchpads can take different forms. In-Air pads are fully virtual, floating next to the hand [4]. On-Hand pads are directly placed on top of the hand [39]. On-Object pads are positioned on a tangible object held in the hand, such as a notebook or smartphone [6, 20]. Each pad type has its pros and cons. On-object pads provide haptic feedback when the finger comes into physical contact with the surface. On-object pads can thus improve precision, but they monopolize the non-dominant hand. In-air pads, on the contrary, leave the hands free for other tasks, but lack any haptic feedback. On-hand pads offer a compromise between those two, preserving haptic feedback while keeping the hands unencumbered, with proprioception facilitating finger positioning on the pad [16, 17, 37].

Prior prototype implementations of on-hand pads have relied on additional sensors and advanced computer vision pipelines [7, 28, 39, 47, 57–59] that make them impractical for real-world deployment, and are often not robust enough to effectively support more elaborate interactions than simple button activations [20]. One key challenge has lied in the limitations of early hand-tracking technology, which struggled with occlusion caused by overlapping hands [18, 19]. As a result, industry guidelines have historically discouraged hand-hand interactions [35]. However, recent advances in hand-tracking accuracy [45] suggest that an on-hand touchpad using only the built-in capabilities of commercial AR headsets may have become viable. This hand-tracking-only approach not only simplifies implementation and deployment, but it also enables the hand itself to serve as an opportunistic input surface [11, 23].

This article explores the feasibility of on-hand pads using the built-in hand-tracking capabilities of commodity AR headsets only. Our prototype runs on a Meta Quest 3, aligning a virtual panel with the non-dominant hand and detecting input through hand-tracking-based movement and collision detection. We assess the performance of such pads with two studies. The first study evaluates input performance by comparing On-Hand, In-Air, and On-Object pads across the three fundamental interactions illustrated in Figure 1: discrete command invocation (e.g., button); 1D continuous control (e.g., slider); 2D pointing (e.g., cursor control). It also measures the cost of relying solely on hand tracking, comparing On-Object with a fourth pad, On-Phone, which uses the phone's capacitive screen for finger contact and movement tracking. This latter condition serves
as a reference for an ideal, high-fidelity tracking method. Results show that while on-hand pads perform comparably to in-air and on-object pads, the high-fidelity on-phone pad outperforms all pads that use hand tracking only by 15% (discrete input) to 55% (2D continuous input). The second study measures the time required to reach the pad when interleaving pad-based input with other tasks that require the hands to be free for other purposes. While in-air and on-hand pads feature a consistent 1-second access time, on-phone pads exhibit a significant overhead, ranging from .75 to 2s when interactions are chained or interleaved with other actions. This cost had been suggested in prior research [2, 15, 16] but had never been empirically quantified.

Overall, these findings demonstrate that on-hand touchpads can be implemented using the hand-tracking capabilities of off-the-shelf AR hardware only. The hand itself becomes an opportunistic touch surface, that performs comparably to mid-air pads in terms of input performance but provides haptic feedback as well as proprioception cues [16, 17, 37]. Higher-fidelity tracking can yield better input performance, but in practice requires holding a device in the non-dominant hand. On the contrary, on-hand pads allow users to keep both of their hands free, which has significant practical value especially in contexts where users have to frequently alternate between different uses of their hands.

## 2 Related Work

The literature on pad-based interaction is extensive, spanning various types of pads, including pen-operated pads (e.g., [13, 36]), motion-controlled pads (e.g., [27, 38]), or keyboard pads designed for text entry (e.g., [14, 32, 34, 49]). In this section, we focus specifically on *touchpads*, which are two-dimensional surfaces that sense finger movement and position. We review prior work that investigates how such touchpads are implemented and evaluated, focusing on their ability to support fundamental input actions such as *invoking discrete commands* (e.g., button presses or gesture triggers) and *controlling continuous parameters* (e.g., 1D sliders or 2D cursor control).

### 2.1 Touchpad Realization

The idea of using a pad held in the non-dominant hand to interact with AR in a mobile context dates back to pioneering AR systems in the 1990's [13, 30, 37]. Since then, pads have been used in multiple AR projects, either as a surface to interact directly with menus and widgets displayed on the pad itself [13], or as pointing devices to control a cursor displayed in the AR view [4, 6].

**Physical pads.** In many projects, the pad is a handheld device such as a smartphone (see, e.g., [5, 27, 61]). While such devices provide high input fidelity thanks to their touchscreen [15, 60], they also have drawbacks: they monopolize the non-dominant hand, which is no longer free to perform tasks in the physical world; and users must carry them, typically in their pocket or on the hip, which entails a relatively high access cost each time they want to interact [2, 15, 16]. While physical pads often use handheld devices with capacitive screens for reliable contact detection (e.g., [6, 9, 43]), some approaches enhance passive surfaces with touch functionality using external motion-capture systems to track both pad and finger positions in 3D (e.g., [16, 27, 55]). A few pads eliminate the need for external trackers, using cameras embedded in headsets instead. For instance, MRTouch [56] relies on a 3D scan of the environment to detect static surfaces and transform them into touch-sensitive interfaces. Similarly, TriPad [11] uses hand tracking to achieve touch functionality without additional hardware than the headset itself. These latter solutions are limited to stationary surfaces and do not support handheld pads.

**Mid-air pads.** Mid-air touchpads offer an alternative to physical surfaces. They do not require holding any device. Compared to physical pads, they leave the hands free and are always available, featuring minimal access cost. But users may have trouble estimating depth precisely because of
their intangibility [8]. A common UI design pattern in AR is to display widgets on a mid-air pad that floats near the user's non-dominant hand (e.g., the hand menu in Microsoft's MRTK). As they are intangible, other placement options can be considered. For example, Gunslinger [31] positions a virtual pad near the user's thigh to support gestures in a relaxed, arms-down posture. The ARPads design space [4] further investigates alternative placements to optimize comfort and accessibility, such as near the waist or above the wrist. However, positions that fall outside the headset's field of view typically require additional body-mounted cameras for hand tracking.

*Skin-Based pads.* The user's own body can also serve as an interaction surface. One approach is to overlay the skin with stretchable materials embedded with capacitive or proximity sensors (e.g.,[52, 54]). These research prototypes typically support only contact detection and do not robustly track finger position or movement. Other techniques do not modify the skin directly but rather rely on wearables such as smartwatches (e.g.,[28, 57, 59]) or electromagnetic sensors embedded in rings or wristbands (e.g.,[7, 47, 58]). While innovative, these systems remain laboratory prototypes with limited robustness, and often support discrete input only. An exception is ActiTouch [58], which combines a wrist-mounted sensor to enable contact detection with a headset-mounted Leap Motion device for finger motion tracking. A more recent project, EgoTouch [39], uses computer vision and machine learning to detect finger contact on the user's skin using cameras embedded in the headset, suggesting that instrumentation-free skin pads may become feasible. However, EgoTouch currently supports only contact detection and remains computationally demanding.

*Summary.* While prior work has explored a wide range of touchpad implementations, from high-fidelity physical devices to flexible mid-air and skin-based alternatives, most approaches rely on external instrumentation or additional wearables. Mid-air touchpads are a notable exception that can be implemented without any additional hardware. However, despite significant advances in hand tracking, the feasibility of using the hand itself as a touchpad implemented exclusively with the headset's built-in sensing capabilities has not been investigated.

## 2.2 Touchpad Evaluation

Our goal is to provide empirical findings about the performance of on-hand touchpads implemented with AR headset built-in sensing. Our survey of the literature in the previous section indicates that there is no prior work about such pads. But the HCI literature reports on empirical work evaluating touchpad interaction relying on other implementations. Table 1 gives a structured overview of this literature. Many evaluations are *internal*, focusing on a single surface type to, e.g., assess the robustness of a specific prototype, or compare design variations on the same surface type. Since our primary interest is on comparing the viability of on-hand touchpads to other surface types, we concentrate here on comparative studies across different surfaces. Our focus being specifically on on-hand touchpads, we are mostly interested in skin as a touch sensitive surface. However, as skin can also be considered as a type of physical surface, findings about physical surfaces can also provide valuable information about the potential of on-hand touchpads.

We organize our analysis of the literature in two broad input categories: *command invocation*, which covers location-based touch actions and gesture-triggered commands (e.g., tap, swipe, or shape gestures), and *parameter control*, which requires continuous tracking of finger position for continuous 1D or 2D input, such as sliders or cursor movements.

### 2.2.1 Command Invocation.
Three studies directly compared skin-based and mid-air input. The first two studies – the second reported in Mine et al.'s seminal paper [37] and the study investigating the concept of an Imaginary Phone [16] – report consistent observations. Both studies observe that participants were able to position or reposition their index finger with greater precision on their
<table><thead><tr><th rowspan="2">YEAR</th><th rowspan="2">PAPER ID</th><th colspan="6">SURFACE TYPE</th><th colspan="2">INTERACTION TYPE</th><th rowspan="2">STUDY TYPE</th></tr><tr><th colspan="2">physical</th><th colspan="2">skin</th><th colspan="2">mid-air</th><th rowspan="2">command invocation</th><th rowspan="2">parameter control</th></tr><tr><th></th><th></th><th>headset cameras only</th><th>additional instrumentation</th><th>headset cameras only</th><th>additional instrumentation</th><th>headset cameras only</th><th>additional instrumentation</th><th></th><th></th><th></th></tr></thead><tbody><tr><td>1997</td><td>Moving Objects In Space [37]</td><td>-</td><td>-</td><td>-</td><td>✓*</td><td>-</td><td>✓*</td><td>-</td><td>✓</td><td>C</td></tr><tr><td>1999</td><td>HARP [30]</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>✓</td><td>✓</td><td>C</td></tr><tr><td>2010</td><td>Touching the Void [8]</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>✓</td><td>-</td><td>I</td></tr><tr><td>2010</td><td>Skinput [21]</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>I</td></tr><tr><td>2011</td><td>Imaginary Phone [16]</td><td>-</td><td>-</td><td>✓NE</td><td>✓</td><td>-</td><td>✓</td><td>✓</td><td>-</td><td>C</td></tr><tr><td>2011</td><td>OmniTouch [20]</td><td>-</td><td>✓</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>✓</td><td>✓phys (1)</td><td>I, C</td></tr><tr><td>2012</td><td>PalmRC [10]</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>I</td></tr><tr><td>2013</td><td>Gustafson et al. [17]</td><td>-</td><td>✓</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>✓</td><td>✓</td><td>C</td></tr><tr><td>2013</td><td>FingerPad [7]</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>✓</td><td>✓</td><td>I</td></tr><tr><td>2013</td><td>HHDwithHMD [6]</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>I</td></tr><tr><td>2014</td><td>Tablet- &amp; HMD-based HVE [51]</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>✓</td><td>S</td></tr><tr><td>2014</td><td>SkinButton [28]</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>I</td></tr><tr><td>2014</td><td>SleeD [46]</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>✓</td><td>S</td></tr><tr><td>2014</td><td>Smarties [9]</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>✓</td><td>S</td></tr><tr><td>2015</td><td>Gunslinger [31]</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>✓</td><td>✓</td><td>I, C<sub>±</sub></td></tr><tr><td>2015</td><td>PalmGesture [50]</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>I</td></tr><tr><td>2016</td><td>SkinTrack [59]</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>✓</td><td>✓NE</td><td>I</td></tr><tr><td>2016</td><td>TapSkin [57]</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>I</td></tr><tr><td>2017</td><td>WatchSense [42]</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>✓</td><td>-</td><td>✓NE</td><td>I</td></tr><tr><td>2018</td><td>David Meets Goliath [25]</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>S</td></tr><tr><td>2018</td><td>MRTouch [56]</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>✓</td><td>I</td></tr><tr><td>2019</td><td>ActiTouch [58]</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>✓</td><td>✓</td><td>I</td></tr><tr><td>2019</td><td>TabletInVR [43]</td><td>-</td><td>✓</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>✓</td><td>✓</td><td>S</td></tr><tr><td>2020</td><td>ARPads [4]</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>✓</td><td>✓</td><td>C</td></tr><tr><td>2021</td><td>Brasier et al. [5]</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>✓</td><td>✓</td><td>C, C<sub>±</sub></td></tr><tr><td>2021</td><td>HPUI [12] (2)</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>✓</td><td>✓</td><td>-</td><td>C</td></tr><tr><td>2022</td><td>CleAR Sight [27]</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>✓</td><td>S</td></tr><tr><td>2023</td><td>Medeiros et al. [33]</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>✓</td><td>✓</td><td>C</td></tr><tr><td>2023</td><td>Z-Ring [47]</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>I</td></tr><tr><td>2024</td><td>TriPad [11]</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>✓</td><td>✓</td><td>I, C<sub>±</sub></td></tr><tr><td>2024</td><td>PhoneInVR [61]</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>✓</td><td>✓</td><td>C</td></tr><tr><td>2024</td><td>MouseRing [41]</td><td>-</td><td>✓</td><td>-</td><td>-</td><td>-</td><td>✓</td><td>-</td><td>✓</td><td>I, C<sub>±</sub></td></tr></tbody><tfoot><tr><td colspan="2">Hand-Bound Pads</td><td>✓</td><td>✓</td><td>✓</td><td>-</td><td>✓</td><td>-</td><td>✓</td><td>✓</td><td>C</td></tr></tfoot></table>

I Internal evaluation where only one Surface Type is considered

C Comparative evaluation between Surface Type conditions

C<sub>±</sub> Comparative evaluation between conditions differing by Surface Type but also other interaction design aspects

S System evaluation (user feedback on systems with a pad, without quantitative pad input performance measurements)

<caption>Table 1. Comparative table organizing papers with at least one experiment that involves touchpad interaction for either *command invocation* or *parameter control*. The symbol ✓NE means that this specific condition is not tested in the experiment reported in the paper; and an asterisk * signifies that it is our best guess based on information available in the paper.</caption>

<footnote>(1) Only the physical surface is evaluated for parameter control.</footnote>

<footnote>(2) The pad spans both the fingers and the air around them. We consider only the study that reports on the comparative performance between targets on skin and in the air.</footnote>
palm compared to a virtual floating touchpad. The third study evaluates the concept of HPUI [12], which consists of placing buttons on and between the fingers of the non-dominant hand, allowing the thumb of the same hand to tap these buttons. The study found that participants were faster at acquiring targets located on the fingers compared to those positioned mid-air between the fingers. However, it is important to note that HPUIs are touchpads specifically designed for the specific case of *one-handed* interaction.

A couple of studies also suggest an advantage of physical surfaces over mid-air for command invocation. Brasier et al. [5] report significant performance drops when buttons are displayed in mid-air around a phone, as opposed to being displayed on the phone's tactile screen. PhoneInVR [61] is a system designed to enable users to interact with their smartphones while immersed in Virtual Reality. In the reported study, participants selected commands most effectively when interacting with a virtual phone anchored to the real, physical phone, rather than when the virtual phone was placed in mid-air. However, while surface type may play a role in both of these studies, contact detection on the physical phone relies on capacitive touch sensing, which introduces a bias when assessing the effect of surface type alone.

Few studies compare skin-based and physical surfaces. OmniTouch [20] is implemented with a wearable device that combines both a projector and a depth-sensing camera. The study compares interactions on the palm and a passive physical surface (a notepad held in the hand). In the reported study, participants achieved similar levels of precision when acquiring discrete targets on both surface types.

### 2.2.2 *Parameter Control*
Very few studies have considered skin surfaces for continuous input. One specific exception is Gustafson et al. [17], which investigated “touch-and-explore” interactions, where users continuously slide their index finger across a surface to locate and identify targets, with target names voiced when the finger enters them. The study compared the palm of the hand to a phone-sized physical surface held by the hand. Participants were more efficient with the palm, as proprioception aided navigation and memorization of targets laid on the skin. OmniTouch [20] also supports input on skin and physical surfaces and considered continuous input. However, their study evaluated continuous input only on a passive physical surface (notepad). No equivalent task was conducted on skin, likely due to robustness limitations of the prototype.

Several studies have compared mid-air and physical surfaces for parameter control. Gunslinger [31] compares indirect mid-air interaction (virtual touchpad near the thigh) with direct touch on a vertical display, observing better performance with direct touch. Similarly, TriPad [11] observed improved pointing time with a passive physical surface than with mid-air touch. However, both studies compare conditions that differ in both surface type and control type: Gunslinger compared indirect mid-air control to direct touch on screen, and TriPad compared indirect cursor control on a physical surface to direct touch in mid-air. Brasier et al. [5] also studied parameter control (slider and slate widgets), but their mid-air interactions relied on raycasting, whereas widgets on the smartphone's screen used direct touch. Together, these studies suggest potential performance benefits of physical surfaces, but they involve other factors that make it difficult to isolate the specific effect of surface type.

Two VR studies [30, 33] investigated dragging tasks and found that participants performed better on physical surfaces than in mid-air. But these tasks involved direct touch on physical surfaces without the need for indirect cursor movements or clutching actions. Finally, MouseRing [41] compared indirect input using three physical surface conditions with a mid-air condition that relied on active input with a gyroscopic mouse. The mid-air condition performed slightly worse, but it relied on a gyroscopic mouse, unlike the physical surface conditions that relied on touch input.
*Summary.* Prior work suggests that both physical and skin-based surfaces may offer interaction advantages over mid-air input, particularly for discrete command invocation tasks. The skin, in particular, seems to enable more precise finger placement, benefiting from proprioceptive cues. For continuous parameter control, physical surfaces generally appear to outperform mid-air alternatives, but studies often entangle surface type with other variables such as contact detection sensing, or interaction design aspects. This makes it difficult to isolate the effect of surface type. Empirical comparisons involving skin surfaces for continuous control are almost inexistent, as laboratory prototypes are usually not robust-enough to support finger sliding movements over the skin.

As shown in Table 1, our contribution addresses these gaps by evaluating touchpad interaction using solely the hand tracking capabilities built in recent AR headsets. Our goal is to assess practical implementations that involve hand tracking only and are thus easy to deploy in real-world settings. We test three surface types (On-Hand, In-Air, On-Object) and two task types: command invocation and parameter control. We keep all other interaction design aspects constant to isolate the impact of surface type, and we measure the performance trade-offs of relying on hand tracking compared to an ideal implementation based on sensors that provide high-fidelity tracking.

# 3 Experiment 1: Comparative Evaluation of Pad Type Input Performance

Our first experiment had two main goals: (1) compare task performance across touchpads with different surface types using only the hand tracking capabilities of AR headsets, and (2) estimate the performance drop when using hand tracking only instead of high-fidelity tracking.

To assess the effect of surface type, we tested three touchpad configurations: a virtual touchpad floating in mid-air near the non-dominant hand (In-Air), the non-dominant hand itself (On-Hand), and a physical surface held in the non-dominant hand (On-Object). Hand tracking is expected to perform best in the mid-air condition (In-Air), where the fingers are never occluded. The physical surface (On-Object) serves as an intermediate condition, where the fingers get in contact with a flat surface visually distinct from the dominant hand. It helps isolate tracking challenges specific to skin-based interactions.

Quantifying the cost of relying on hand tracking only is more complex. Ideally, we would compare a high-fidelity touchpad implementation for each surface type to its hand-tracking counterpart. However, as noted earlier, current technologies do not support robust sensing of finger-pad contact and movement (discrete taps and sliding gestures) without heavy instrumentation – neither on skin nor in the air. We thus focused on the comparison of the physical surface condition, specifically comparing On-Object (hand-tracking-based implementation) with On-Phone, a touchpad in the form of a smartphone with high-fidelity capacitive sensing. The two share the same form factor and interaction design, differing only in the sensing technology. Although this comparison does not isolate the cost of hand tracking for each surface type, it estimates the impact of using hand tracking versus an ideal, very-high-fidelity setup.

## 3.1 Touchpad Designs

Figure 2 illustrates the four touchpad conditions evaluated in this experiment.

3.1.1 *In-air Pad.* As illustrated in Figure 2-a, this pad lies next to the non-dominant hand as if it were floating in mid air. It follows the hand's movements, and is oriented so as to always face the user. This pad is fully virtual. It will automatically appear every time users turn their non-dominant hand palm up in the hand tracking field, and is thus easy to carry and access – actually users do not have to carry anything. The implementation relies solely on the hand tracking API, which tracks finger position and triggers finger-pad contact events when the dominant hand's index
<figure><img src="image_4.png" /><figcaption>Fig. 2. The four types of touchpads tested in Experiment 1: (a) pad floating in mid-air next to the non-dominant hand; (b) articulated pad that fits in the non-dominant hand; (c) pad on a flat physical surface held in the non-dominant hand; (d) pad on the same flat physical surface, but with high-fidelity tracking.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>Fig. 3. On-Hand Pads are articulated: one part is aligned with the palm, and the other with the fingers: (left) the hand is nearly straight, and the pad is almost flat; (right) the fingers form an angle with the palm, and the pad adapts accordingly.</figcaption></figure>

finger enters or exits the pad's collider (see details later in this section). This pad does not provide
any haptic feedback.

3.1.2 On-Hand Pad. The hand, and the palm in particular [44], is a promising support surface for touchpads [17, 53]. As illustrated in Figure 2-b, this pad lies on, and is aligned with, the non-dominant hand. It is articulated, consisting of two parts (Figure 3): one is aligned with the palm, and the other with the fingers to better match the human hand’s anatomy. As in the case of the mid-air pad, users do not have to carry any object and the pad is readily accessible simply by turning the non-dominant hand palm up. Finger-pad contact detection and movement tracking are the same as described above, using the hand tracking API. This time there are two colliders, aligned with the two parts of the articulated pad. This pad provides both haptic feedback and proprioceptive cues.

3.1.3 On-Object and On-Phone Pads. The last two pads are aligned (both position and orientation) with a flat physical surface held in the hand. Like the On-Hand pad, these pads provide haptic feedback, but on a smooth surface made from a material distinct from the interacting fingers. To isolate and quantify the effect of relying solely on hand tracking, we compare two physical pads with the same smartphone form factor, differing only in their implementation:

* On-Phone Pad: uses the smartphone's capacitive screen for high-fidelity finger-pad contact detection and movement tracking (Figure 2-d), as seen in [6, 43].

* On-Object Pad: The smartphone is *turned off*. Finger-pad contact and movements are detected via the headset's hand-tracking API, similar to the In-Air and On-Hand pads described above.
To align the pad with the phone's position, a VR controller is attached to the phone (Figure 2-c), similar to Bai et al.'s setup [3].

While tracking the phone's position is unnecessary for the On-Phone pad, the VR controller is attached in both conditions to ensure that the two conditions only differ in the tracker implementation (hand tracking only vs. high-fidelity sensors). This prevents confounding factors such as variations in physical surface properties such as friction and weight.

The only minor difference between the On-Phone pad and the three other pads using hand tracking only is how visual output is rendered: the widgets participants interact with in this condition are rendered by the smartphone and passed through by the Quest 3 cameras, whereas in other conditions those widgets are rendered directly in the headset. The video passthrough rendering makes those widgets slightly blurry in the On-Phone condition, but UI elements and font size have been set so as to minimize this difference.

All three pads that rely on hand tracking only use colliders with the following characteristics: rectangular, 5cm thick, positioned so as to extend 1cm only above the surface. Participants go through a quick calibration phase for the On-Hand Pad and the On-Object Pad aiming to provide the best fit, taking into account variability in hand morphology and tracking discrepancies. The calibration involves participants pointing at three reference points on the pad using the index finger of their dominant hand. Pad position and orientation is adjusted based on the distance between the index finger tip position as reported by the hand tracking API and the corresponding reference point.

In all conditions, the system provides audio feedback when performing interactions. Such feedback is key to user experience in Augmented Reality. For instance, Chan et al. [8] found that providing any form of feedback (visual or audio) significantly improved target acquisition time compared to no feedback at all. Audio feedback is implemented as follows: a short sound played once for discrete interactions; a sound played in loop for continuous interactions.

## 3.2 Hypotheses

As discussed in section 2, comparisons between touchpads often involve other factors than the surface type alone, and rarely consider both discrete and continuous tasks. Drawing specific hypotheses regarding comparative performance between surfaces is thus difficult. Nevertheless, the literature generally suggests that touch interaction on a physical surface, thanks to haptic feedback, tends to be more precise than interaction in mid-air [10, 36, 60]. Also, while hand tracking has seen substantial improvements, occasional tracking inconsistencies may still occur. We thus formulate the following two hypotheses:

H1: *Among hand-tracking-only pads, those featuring haptic feedback (On-Object and On-Hand) will yield better precision than the In-Air pad.*

H2: *The pad featuring high-fidelity tracking (On-Phone) yields better overall performance than its hand-tracking-only counterpart (On-Object), because of its higher precision.*

## 3.3 Participants

24 participants volunteered for the experiment (14 male, 9 female, 1 preferred not to disclose). 22 of them were right-handed, 2 were ambidextrous and used their right hand as their dominant hand. The experiment started with a question about their prior experience with Mixed Reality (MR) headsets. We purposefully recruited 12 *expert* and 12 *novice* AR users. An *expert* is a user who self-reports as being very familiar with AR technologies, while a *novice* self-reports as having very little familiarity or no familiarity at all with the technology.
<figure><img src="image_6.png" /><figcaption>Fig. 4. Experiment 1 setup illustrated with the Continuous 2D task and the On-Hand Pad.</figcaption></figure>

## 3.4 Apparatus

The experiment was developed using Unity v2021.3.29f1 and MRTK v2.8.3 and ran on a Meta Quest 3 headset with video passthrough activated. We used a smartphone Samsung Galaxy Note 10 for the On-Object and On-Phone pads. Communication between the smartphone and the AR headset (required in the On-Phone condition only) was implemented using Photon Pun Version 4.1.8.4. UI widgets running on the phone were also developed in Unity.

As illustrated in Figure 4, participants were seated at a desk to minimize the risks of potential cybersickness sensations. The task panel showing instructions and providing feedback about the task was positioned 2m in front of the participant, and had a width of 1m.

## 3.5 Task

Tasks involve command invocation and parameter control in both one dimension (1D) and two dimensions (2D), operationalizing interactions with common user interfaces in AR, which are illustrated in Figure 1. For each task, participants interact with one of the four pads to control an interface displayed on the virtual canvas depicted in Figure 4. Participants are always instructed to be as fast and as accurate as possible.

*Discrete* (command invocation). Participants have to click on a sequence of 8 targets located in different positions. Only one target is visible at a time. At the beginning of each task, participants are instructed to wait (min 2s, max 4s) until a red balloon appears (Figure 5). They then have to pop the balloon by pointing at the target displayed on their pad. This wait time, displayed in the task panel, forces participants to focus on the task panel in front of them and is aimed at making sure that all trials start in the same conditions. The target location on the pad is visible right from the start of the trial. Once the balloon appears, participants hit the target on the pad. Targets are 2cm in diameter. An animation shows the balloon popping, with audio feedback.

*1D_Continuous* (parameter control). Participants have to adjust the value of four sliders, in sequence. Sliders are located in one of two pad areas (lower and upper half of the pad), and can be oriented vertically or horizontally. All sliders are 5cm on the pad. The corresponding visual representation on the task panel (Figure 6) is 25cm in width. There is a 1cm movement tolerance, which visually maps to a 5cm white rectangle in the slider's gutter on the task panel. The target
<figure><img src="image_7.png" /><figcaption>Fig. 5. Experiment 1 Task - Discrete Command Invocation.</figcaption></figure>

<figure><img src="image_8.png" /><figcaption>Fig. 6. Experiment 1 Task - Continuous 1D Parameter Control.</figcaption></figure>

value, visually represented by a thick black line in the middle of the tolerance rectangle, turns green when the cursor falls within the tolerance, this green color getting increasingly bright and saturated as the cursor gets nearer to the ideal position. The task ends as soon as participants release control, provided they do so inside the tolerance area.

2D_Continuous (parameter control). Each trial consists of a series of 8 target acquisitions laid out radially (Figure 7, ISO 9241-9 standard [26]). The next target to acquire is colored white with a black cross painted inside it. That cross turns green as soon as the cursor enters the tolerance area. Similar to 1D parameter control, the cross gets increasingly bright and saturated as the cursor gets closer to the target’s center. All other targets are colored black. The cursor itself is represented as a gray cross. Targets are 10cm in diameter on the task panel and are laid out on a circle 60cm in diameter. The control display gain value is set to 5. Participants typically have to clutch to reach the next target. To validate each target acquisition, participants have to place the cursor inside the current target and then click to validate. Once a target has been validated, it turns black, and the next target turns white. We ignore the first pointing task in each series since the cursor is located in the center of the canvas when a trial starts.

## 3.6 Design and Procedure

The experimental procedure lasted 45 minutes per participant on average. Participants first had to read and sign a consent form.

Participants used four different types of pad (In-Air, On-Hand, On-Object, On-Phone) to perform three types of task (*Discrete*, *1D_Continuous*, and *2D_Continuous*). Trials were blocked by PADTYPE,
<figure><img src="image_9.png" /><figcaption>Fig. 7. Experiment 1 Task - Continuous 2D Parameter Control.</figcaption></figure>

and then per task type. Presentation order of PADTYPE blocks and task sub-blocks was counterbalanced across participants using a Latin Square. Each task sub-block consisted of a training trial (repeatable at will), followed by 2 measure trials. Widget placement (upper or lower half of the pad) was presented in a random order.

We thus collected:

* 24 participants × 4 PADTYPE × 2 repetitions × 8 positions = 1536 events for the *Discrete* task;

* 24 participants × 4 PADTYPE × 2 repetitions × 4 positions = 768 events for the *1D_Continuous* task;

* 24 participants × 4 PADTYPE × 2 repetitions × 7 positions = 1344 events for the *2D_Continuous* task.

At the end of each PADTYPE block, participants filled a questionnaire where they had to rate the pad for the three tasks in terms of comfort, ease of use, efficiency and tiredness using 5-point Likert scales. They also had to rank the different pads for the three tasks at the end of the experiment.

## 3.7 Results

We systematically analyze the effect of Tracking Method (hand-tracking-only vs. high-fidelity) and of Surface Type (hand, in air, on object) on the following two performance measures:

(1) *TaskDuration*: the time (in ms) to complete the task.

* For *Discrete* tasks, the timer starts when the balloon appears, and stops when participants lift their index finger off the pad.

* For *1D_Continuous* tasks, the timer starts when the slider appears, and stops when participants lift their index finger off the pad.

* For *2D_Continuous* tasks, the timer starts when the previous target is hit, and stops when participants click the correct target.

(2) *Precision*: the distance (in mm) in motor space to the target's location, at task completion time.

* For *Discrete* tasks, this is the distance between the index position on the pad and the center of the target.

* For *1D_Continuous* and *2D_Continuous* tasks, this is the cursor-target distance on the visual representation, divided by 5 to account for the control-display ratio and derive motor precision.

As we recruited two equally-sized groups of participants (12 novice and 12 expert users), we can also analyze the effect of the Expertise factor according to a between-subject design on *Precision* and *TaskDuration* across all three tasks. We anticipated an Expertise effect on performance in
<figure><img src="image_10.png" /><figcaption>Fig. 8. Speed and Precision performance for pads using hand tracking only – Charts report means of the differences between all pairs of PADTYPE conditions, with error bars representing 95% confidence intervals. Differences are computed as left condition minus right condition, and are reported for both the full dataset (black) and the dataset after outlier removal (gray).</figcaption></figure>

general due to the familiarity these users have with AR headsets. However, there was no significant effect of Expertise on either measure for 1D_Continuous and 2D_Continuous tasks. There was an effect only for Discrete tasks, where experts were significantly faster than novices (unpaired t-test: $t(22) = 3.12, p < 0.01$).

The rest of this section examines the effects of Tracking Method and Surface Type by task type using ANOVA and post hoc pairwise tests. First, we analyze the effect of Tracking Method by specifically comparing the On-Object and On-Phone conditions. Next, we evaluate the effect of Surface Type by comparing the On-Object, On-Hand, and In-Air conditions. All significant differences identified in post hoc pairwise tests are reported with Holm-corrected *p* values. As the surface in the On-Hand condition is not uniform because of the anatomical difference between fingers and palm – which could have an impact on performance – we systematically look at the effect of interacting in the lower (AREA=Bottom) or upper (AREA=Top) part of the pad in our analyses. We also systematically ran the analyses on the Novice subset and on the Expert subset. Unless stated otherwise, the effects and differences we observed were the same for the two groups.

Although accuracy in hand tracking has significantly improved in the last few years, we observed occasional tracking issues during the experiment. To account for these, we conducted analyses on two datasets: (i) a full dataset including all data points, and (ii) a filtered dataset excluding outliers, which should be more representative of perfect tracking conditions. Outliers were identified using the *TaskDuration* measure and defined as data points falling outside the range [*Q1* − 1.5×*IQR*, *Q3* + 1.5 × *IQR*]. Analyses in the following sections consider the full dataset. We then explicitly get back to the specific cases where outlier removal affects the significance of effects or differences. Figure 8 illustrates the analyses for both datasets.
<figure><img src="image_11.png" /><figcaption>Fig. 9. Speed and Precision cost of hand-tracking only pads – Charts report the average task duration (top) and precision (bottom) for the two pads that have the exact same form factor but differ in their tracking method: On-Object and On-Phone pads. Error bars represent 95% confidence intervals. All data points are included.</figcaption></figure>

### 3.7.1 Effects of Tracking Method and Surface Type.
The table below reports the results of anova tests, revealing a significant effect of PADTYPE on both *TaskDuration* and *Precision* measures for each of the three task types participants did during the experiment.

<table><thead><tr><th></th><th>Discrete</th><th>1D_Continuous</th><th>2D_Continuous</th></tr></thead><tbody><tr><th>PADTYPE effect on TaskDuration</th><td>F<sub>3,69</sub> = 10, p &lt; 0.001, η<sup>2</sup><sub>G</sub> = 0.14</td><td>F<sub>3,69</sub> = 19, p &lt; 0.001, η<sup>2</sup><sub>G</sub> = 0.23</td><td>F<sub>3,69</sub> = 36, p &lt; 0.001, η<sup>2</sup><sub>G</sub> = 0.38</td></tr><tr><th>PADTYPE effect on Precision</th><td>F<sub>3,69</sub> = 77, p &lt; 0.001, η<sup>2</sup><sub>G</sub> = 0.49</td><td>F<sub>3,69</sub> = 42, p &lt; 0.001, η<sup>2</sup><sub>G</sub> = 0.37</td><td>F<sub>3,69</sub> = 34, p &lt; 0.001, η<sup>2</sup><sub>G</sub> = 0.36</td></tr></tbody></table>

*Tracking Method.* To examine the effect of Tracking Method, we relied on post hoc pairwise tests to contrast the On-Phone and On-Object conditions across all task types. As illustrated in Figure 9, for *Discrete* tasks, On-Phone significantly outperformed On-Object, being both faster (*p* < 0.01) and more precise (*p* < 0.001). A similar pattern was observed in *1D_Continuous* tasks, where On-Phone again was significantly faster and more precise (all *p*'s < 0.001). For *2D_Continuous* tasks, tracking method also significantly affected both *TaskDuration* and *Precision* in the same way (all *p*'s < 0.001). For this latter task, looking at positions used by the participants across the different PADTYPE conditions provides insights about how accurate tracking influences finger movements. As illustrated in Figure 10, the most heavily used part is the pad's center in the four conditions, but the comparison between On-Phone and the other pads reveals that the area used is more condensed for On-Phone than for the other three hand tracking only pads. The high fidelity of the capacitive screen enabled users to perform finger lifts of very small amplitude when clutching, leading to
<figure><img src="image_12.png" /><figcaption>Fig. 10. HeatMap of Participants' positions inside the different PADTYPE during the Continuous 2D task of Experiment 1.</figcaption></figure>

smaller and quicker movements. In the other three conditions, participants had to lift their hand higher to clutch, which likely led them to prefer wide movements to limit the number of clutch actions.

**Surface Type.** To test the effect of Surface Type, we relied on post hoc pairwise tests to contrast the three hand tracking only pads. For *Discrete* tasks, there is no significant difference in *TaskDuration* across the three hand tracking only pads. However, *PADTYPE* influences Precision ($F_{3,69} = 77, p < 0.001$, $\eta_G^2 = 0.49$, reported in table above), along with AREA ($F_{1,23} = 17, p < 0.001$, $\eta_G^2 = 0.07$). Additionally, the interaction between PADTYPE and AREA is significant ($F_{3,69} = 4, p < 0.001$, $\eta_G^2 = 0.03$). Post hoc tests showed that On-Object yielded significantly higher precision than both On-Hand and In-Air ($p < 0.05$), and that On-Hand was more precise than In-Air ($p < 0.01$). Precision was also influenced by the interaction area, with movements in AREA = Bottom (the palm, for On-Hand) being more precise than in AREA = Top (the fingers). The significant PADTYPE × AREA interaction effect ($F_{3,69} = 3, p < 0.001$, $\eta_G^2 = 0.03$) suggests that surface type influences precision differently depending on where the interaction occurs. Specifically, On-Hand was more precise than In-Air ($p < 0.01$) when interacting on the palm, but less precise than On-Object ($p < 0.001$) when interacting on the fingers. This pattern likely reflects anatomical differences, with the flatter palm enabling more stable interactions than the more irregular surface over fingers.

For *1D_Continuous* tasks, unlike *Discrete* tasks, surface type did affect both speed and precision: On-Object was significantly faster and more precise than On-Hand ($p < 0.01$). For *2D_Continuous* continuous tasks, there was no significant difference in task duration between pads, but On-Object remained significantly more precise than both On-Hand ($p < 0.01$) and In-Air ($p < 0.05$).

**Outliers.** Looking at the number of outliers and their distribution across the different conditions gives some indication about how much hand tracking may have negatively impact the performance that we report. For *Discrete* tasks, we detected a total of 120 outliers (7.8%), with 119 values being too high and 1 too low. As expected, the On-Phone condition had fewer outliers, though there were a few (6). Interestingly, while we might have anticipated more tracking issues when the hand was over a surface or over the other hand, outliers were fairly evenly distributed across the three hand tracking only pads: 27 for On-Object, 45 for On-Hand, and 41 for In-Air. For *1D_Continuous* tasks, there were 52 outliers (6.7%) (52 too-high values) with the following distribution: 0 for On-Phone, 13 for On-Object, 23 for On-Hand and 16 for In-Air. For *2D_Continuous* tasks, there were 81 Outliers
<figure><img src="image_13.png" /><figcaption>Fig. 11. Distribution of participants' 5-pt ratings per Task and PADTYPE for statements "The technique was (comfortable, easy to use, efficient, tiring)". Asterisks indicate statistical significance (***: *p* < 0.001, **: *p* < 0.01, *: *p* < 0.05).</figcaption></figure>

<figure><img src="image_14.png" /><figcaption>Fig. 12. Distribution of participants' preferences per Task and PADTYPE (1 is the most preferred, 4 is the least preferred).</figcaption></figure>

(6%) in the 2D_Continuous tasks (81 too-high values) with the following distribution: 0 for On-Phone, 24 for On-Object, 33 for On-Hand and 24 for In-Air.

As illustrated in Figure 8, removing outliers has little overall impact on observations. There are only two notable differences. First, for 1D_Continuous tasks, removing outliers reveals a small but significant effect of AREA for TaskDuration ($F_{1,23} = 5, p < 0.001, \eta_G^2 = 0.008$), with participants being faster in AREA = Bottom (palm) than in AREA = Top (fingers), likely due to slider manipulation being harder across multiple fingers. This is in line with the significant PADTYPE × AREA PADTYPE × AREAinteraction effect on Precision ($F_{3,69} = 4, p < 0.001, \eta_G^2 = 0.06$): On-Object is more precise than On-Hand in the Bottom area, while On-Hand is significantly less precise than all other pads in the Top area. Second, for 2D_Continuous tasks, outlier removal reduces variability, making the difference between On-Object and On-Hand significant ($p < 0.01$).

### 3.7.2 Participants' Feedback.
Figures 11 and 12 report participants' answers to the questionnaire, and how they ranked the four pad conditions across the three tasks.

**Questionnaire.** We ran Friedman tests to analyze the effect of PADTYPE on participants' ratings about the following criteria: comfort, ease of use, perceived efficiency, and fatigue.

* Tracking Method. For 1D_Continuous and 2D_Continuous tasks, tracking method had a clear effect, with the On-Phone pad receiving significantly higher ratings than all three hand
tracking only pads across all criteria. The only exception was the On-Hand pad, which was not rated as significantly more tiring than the On-Phone pad for *1D_Continuous* tasks. However, differences for *Discrete* tasks were more subtle. While the active pad generally received slightly-higher ratings, significant differences (with the In-Air pad for comfort and ease of use, and with the On-Hand pad for ease of use) were marginal.

* Surface Type. Participants' ratings were consistent with the negligible effect of surface type observed in the quantitative performance metrics discussed above, as the three hand-tracking-only pads did not differ significantly from one another.

*Participants' preferences.* The On-Phone pad was the top choice for *Discrete* tasks for 16 participants, while 23 participants ranked it first for *1D_Continuous* tasks and 22 for *2D_Continuous* tasks. The On-Hand pad was ranked first 6 times for *Discrete* tasks, once for *1D_Continuous* tasks, and once for *2D_Continuous* tasks. The In-Air pad was ranked first twice for *Discrete* tasks and once for *2D_Continuous* tasks, and was frequently the least preferred—half the time for *Discrete* and *1D_Continuous* tasks and four times for *2D_Continuous* tasks. The On-Object pad was never ranked first.

Although the performance metrics did not reveal strong differences between the three surface types, these preferences suggest that the type of surface does influence participants' subjective experience. Among the hand tracking only pads, the On-Hand pad was generally favored, although participants expressed more reservations for *2D_Continuous* tasks. This may be due to the larger-amplitude movements required when clutching, because of the imperfect planarity of the hand's surface. Participants' preferences were rarely in favor of the In-Air pad. Throughout the experiment, many participants expressed concerns about interacting with the In-Air pad, with half of them (12 participants) specifically commenting on the absence of haptic feedback and the difficulties they encountered while interacting.

## 3.8 Summary of Findings

The key findings of this first experiment are:

* Tracking Method: across all tasks, On-Phone was significantly faster and more precise than On-Object. **H₂ is supported.** Interestingly, the number of outliers was moderate (~7%) and they were evenly distributed across the three hand-tracking-only pads (On-Object, On-Hand, In-Air). This suggests that tracking robustness remained equally good in all these conditions, even when challenged by overlapping hands (On-Hand) or when fingers are in contact with a physical object (On-Object).

* Surface Type: there were no significant differences between hand-tracking-only pads in terms of task completion time, except for *1D_Continuous* tasks that were performed faster on the On-Object surface than on the hand (On-Hand). There were more differences in terms of precision. The On-Hand surface was systematically less precise than the On-Object surface. The In-Air surface was systematically less precise than the On-Object surface, though the difference was not significant for *1D_Continuous* tasks. Finally, the On-Hand surface was more precise than the In-Air surface, but for *Discrete* tasks only. **H₁ is supported.** Interestingly, participants' preferences were not particularly in favor of the On-Object pad. They actually expressed a slight preference for the On-Hand pad when performing *Discrete* and *1D_Continuous* tasks.

* Expertise: participants' familiarity with AR headsets did not have an effect on the comparative performance between pad types.
# 4 Experiment 2: Quantifying the Overhead of Hand-bound Pad Access Time

As mentioned earlier, some types of pads leave users' hands free [10, 44], while others require holding a physical object in the non-dominant hand (On-Object, On-Phone). Monopolizing the non-dominant hand with a physical object can be very impractical – in mobile situations in particular – as users will often use both of their hands for other tasks in their physical environment. Physical pads will thus need to be put aside, for instance on a nearby surface, in one's pocket, or attached to a belt near the hip [2], introducing a non-negligible access cost when users want to interact with them. On the opposite, In-Air and On-Hand pads leave the user's hands free and are always ready-to-use, requiring only a rotation of the non-dominant hand to make it face upward.

Experiment 1 evaluated pad performance assuming each pad was readily available and bound to the non-dominant hand. To get a complete picture of pad performance, Experiment 2 focuses on pad access cost, i.e., the time needed to begin interacting with a pad when it is not bound to the hand yet.

We consider the three pad types from Experiment 1: In-Air, On-Hand, and On-Phone. Since Experiment 2 focuses on the access cost rather than tracking fidelity, we retained only one of the two physical pads, On-Phone. In Experiment 1, a VR controller was attached to the On-Phone to match the form factor of On-Object and isolate the effect of the tracking method. Here, there is no such need and we remove the VR controller, evaluating the physical pad in its usual configuration. We also consider a best-case storage scenario: the physical pad lies on a table, in front of participants (Figure 13) – as opposed to worst case scenarios such as having to retrieve the pad from a pocket that is zippered-up.

This access cost, or *homing time*, is the time it takes to make the pad visible and accessible to the dominant-hand. It applies only once, at the beginning of a sequence of contiguous interactions. The experiment measures homing time in two conditions:

* *ToBeBound*: when the pad first has to be bound to the non-dominant hand;

* *AlreadyBound*: when the pad is already bound to the non-dominant hand.

## 4.1 Hypotheses

Two time intervals need to be considered: *setup time* corresponds to the time it takes to bind the pad to the non-dominant hand, i.e., to make the pad visible and ready to receive input; and *reach time* corresponds to the time it takes the dominant hand to reach the pad and effectively initiate an interaction. When the pad first had to be bound to the non-dominant hand, homing time will consist of a *setup time* and a *reach time*. The two operations are not necessarily sequential, and the two time intervals can partially overlap. When the pad is already bound to the non-dominant hand, homing time will consist of *reach time* only.

Neither the mid-air pad nor the hand pad involve fetching physical objects to be held in the non-dominant hand. They only require users to rotate their hand. They should thus have a shorter *setup time* than the physical pad. We hypothesize that:

* *H3*: when the pad is *ToBeBound*, *homing time* will be longer with the On-Phone pad than with the other two types of pads;

* *H4*: when the pad is *AlreadyBound*, *homing time* will be shorter with the On-Phone pad, as the homing interaction in this condition is fairly similar to the *Discrete* task from Experiment 1.

## 4.2 Participants & Apparatus

All participants performed both Experiment 1 then Experiment 2 in a row, with a break between them. The apparatus remained the same, but, as explained above, the phone did not have a VR controller attached to it in Experiment 2.
<figure><img src="image_15.png" /><figcaption>Fig. 13. Experiment 2 - a trial illustrating the task performed on a On-Phone pad *ToBeBound*: (a) resizing the virtual 3D cube using both hands; (b) grabbing the phone and holding it in the non-dominant hand; (c) performing the horizontal slide gesture on the pad.</figcaption></figure>

## 4.3 Task

The task consists of performing a coarse horizontal slide gesture with an amplitude of at least 3cm on the pad, in an area of 5cm × 5cm positioned at the center of the pad. Participants perform half of the trials with the pad already bound to the non-dominant hand (*AlreadyBound*), and the other half with both hands free, in which case they first have to bind the pad to the non-dominant hand before interacting with it (*ToBeBound*).

We operationalize these two conditions as follows:

* *ToBeBound*: each trial starts with participants first having to resize a virtual 3D cube using both of their hands (Figure 13-a). This forces them to have their hands free, in a configuration that does not allow holding any pad, neither physical nor virtual. Once the cube has been resized correctly, it disappears, and participants can then bind the pad to their non-dominant hand (Figure 13-b), reach it with their dominant hand, and perform the slide gesture (Figure 13-c).

* *AlreadyBound*: participants start each trial with the pad already bound to their non-dominant hand. They only have to reach it with their dominant hand and perform the slide gesture.

All trials start with participants pressing a Start button. In order to properly isolate *homing time* in both conditions, the timer starts:

* immediately in *AlreadyBound* trials;

* as soon as the cube disappears (after resizing) in *ToBeBound* trials.

The timer stops when participants effectively *start* interacting with the pad, i.e., we ignore the time spent performing the gesture, which by definition comes after *homing time*.

## 4.4 Design and Procedure

The experimental procedure lasted 15 minutes per participant on average. Participants experienced 3 PADTYPE {In-Air, On-Hand, On-Phone} × 2 PADSTATE {AlreadyBound, ToBeBound} conditions. Trials were blocked by PADTYPE, with sub-blocks per PADSTATE. Presentation order of both blocks and sub-blocks was counterbalanced across participants with a Latin Square. Each sub-block consisted of 10 repetitions (2 training trials, 8 measures). We thus collected 24 participants × 3 PADTYPE × 2 PADSTATE × 8 repetitions = 1152 measures.

## 4.5 Results

We analyze the effect of PADTYPE and PADSTATE on *Homing Time*. We find a significant effect of PADTYPE ($F_{2,22} = 12, p = 0.001, \eta_G^2 = 0.22$), PADSTATE ($F_{1,11} = 98, p = 0.001, \eta_G^2 = 0.56$), as well as an interaction effect between those two factors ($F_{2,22} = 50, p = 0.001, \eta_G^2 = 0.50$).

Pads *ToBeBound*: as illustrated in Figure 14, PADTYPE has a significant effect on *Homing Time*. Engaging with a On-Phone pad takes significantly longer than with the other two pad types. $H_3$
<figure><img src="image_16.png" /><figcaption>Fig. 14. Experiment 2 - Homing time (in ms) per PADSTATE (*AlreadyBound*, *ToBeBound*) and per PADTYPE. Asterisks indicate statistical significance (***: *p* < 0.001, **: *p* < 0.01, *: *p* < 0.05).</figcaption></figure>

**is supported:** homing time is 73% longer with the On-Phone pad than with the On-Hand pad, and 63% longer than with the In-Air pad.

**Pads AlreadyBound:** more surprisingly, we observe a significant effect of PADTYPE on *Homing Time* as well. Although the difference is smaller, the On-Phone pad performs better than the other two: homing time is 37% shorter than with the On-Hand pad and 33% shorter than with the In-Air pad. H₄ **is supported**. They are in strong accordance with those observed for the *Discrete* task in Experiment 1, and we tentatively attribute them to the high-fidelity tracking of the On-Phone pad.

In summary, engaging with an On-Phone pad takes significantly longer when that pad has to be grabbed – even in the best case scenario considered here – compared to simply rotating the hand to make an On-Hand pad or an In-Air pad appear. But it takes less time to engage with an On-Phone pad than with the other two pads when the pad is already bound to the non-dominant hand.

# 5 Summary of Findings

On-hand pads are a promising input method for AR. They allow users to interact in a comfortable posture, combining the haptic benefits of a physical surface with proprioceptive advantages, while also allowing the user to seamlessly integrate touchpad interactions with other virtual and physical object manipulations. However, implementing such pads remains technically challenging, and state-of-the-art laboratory prototypes involve much custom instrumentation and lack robustness. This article presents the first empirical investigation of an on-hand pad implemented using only the built-in hand tracking capabilities of a commercially available AR headset (Meta Quest 3), and supporting all types of interactions with 2D GUIs, from button presses to point-and-click cursor manipulations. Our results show that on-hand pads perform on par with mid-air pads, which are already widely adopted in AR UI design. This suggests that on-hand pads can be a viable input method even with today's hand tracking capabilities.

We also quantify the trade-off of relying solely on hand tracking in comparison with high-fidelity tracking. We show that the potential performance gap is of about 55% for 2D continuous input, but is of only 15% for discrete input. This relatively small gap indicates that it is possible to use hand tracking-only to implement on-hand pads today, particularly for discrete tasks such as button activations. As for now, achieving high-fidelity input still requires relying on a physical pad in the form of a handheld device with a capacitive screen. However, our second experiment shows that, when such a device is not already held in the hand, even simple interactions like button presses can take twice as long to initiate. This emphasizes the practical value of on-hand pads, especially in mobile or dynamic AR settings where users may frequently alternate between different uses of their hands.
Finally, the presence of some outliers in Experiment 1 shows that there are still some issues with current hand tracking APIs. However, the number of outliers was relatively low (~7%), balanced between conditions, and statistical analyses confirmed that the overall comparative performance of the pad remained consistent with the other conditions despite the presence of outliers.

# 6 Limitations and Future Work

As with any laboratory experiment, our study has limitations regarding the external validity of the collected observations. First, these observations are based on the capabilities of current commodity hand tracking technology. They provide a baseline that may still be improved upon, and that future studies should compare to the upper limits of human capabilities regardless of the tracking technology considered. Second, the touchpads we evaluated represent only a subset of all possible designs. Variations in factors such as shape and size could influence performance and should be explored in future studies.

To isolate the effect of tracking method, we ensured the On-Object and On-Phone pads had the exact same form factor. This imposed certain constraints on both pads. First, we had to attach a VR controller to the phone in order to track it accurately. This may have slightly degraded performance compared to a regular physical pad without additional hardware. Second, using the same smartphone as the passive physical surface meant that finger movements of the dominant hand occurred in front of a screen which, even if it was turned off in this condition, remained reflective and could have impacted hand tracking robustness.

These considerations also influenced the design of the other two hands-free pads (In-Air and On-Hand), as we maintained a consistent pad size and shape across all pad conditions to properly isolate the effect of surface type. This may have led to suboptimal hand-pad sizes for some participants. Larger hands could extend beyond the pad, while the pad could extend beyond smaller hands. The mid-air pad had the same size as the other pads. While its lack of haptic support is a drawback, its fully virtual nature means that in practice it can be resized to create a larger and potentially more comfortable interaction area. Future designs could explore adaptive or variable-size pads to better accommodate the user's hand anatomy and match task-specific needs.

Very recent work [22, 39] has aimed to improve index-to-palm contact detection using a single RGB camera, leveraging computer vision and machine learning. These projects could enable more robust, instrumentation-free On-Hand pads. However, He et al. [22], who explicitly compare their RGB-based method to a hand-tracking-only baseline, report only marginal performance gains. With AR headsets like the Meta Quest 3 now granting developers access to RGB camera feeds, there is a good potential to integrate such vision-based techniques to develop On-Hand pads that rely solely on built-in sensing. But these techniques remain computationally demanding, and future work should aim to balance performance improvements with computational cost.

Interaction with an On-Hand pad involves not only detecting contact but also tracking spatial finger position and supporting continuous sliding, going beyond what is typically addressed in prior work that has largely focused on discrete (on/off) contact detection on skin. Very recent projects [22, 39] go toward a more accurate contact detection using RGB video alone. However, the improvement over hand tracking only implementation remains marginal [22]. As AR headset manufacturers have recently started granting developers access to the RGB camera feed on devices such as the Meta Quest 3, future work combining computer vision and machine learning may further enhance instrumentation-free On-Hand pads.

Finally, to ensure participant comfort and minimize the risk of cybersickness, our experiments were conducted with seated participants. Future studies should explore a wider range of contexts for the sake of generalizability, such as outdoor environments with variable conditions, and users standing or even walking, as in, e.g., [39, 48].
Acknowledgments

We would like to thank Damien Verani for his contribution to the development of early hand-
bound pad prototypes. We would also like to thank all participants to our studies. This project has
been partially supported by the following grants: ANRT (CIFRE 2022/0586) and ANR Continuum
(ANR-21-ESRE-0030).

References

[1] Apple. 2025. Design for Spatial Input. https://developer.apple.com/videos/play/wwdc2023/10073 - last accessed 2025-07-01.

[2] Daniel L. Ashbrook, James R. Clawson, Kent Lyons, Thad E. Starner, and Nirmal Patel. 2008. Quickdraw: the impact of mobility and on-body placement on device access time. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (Florence, Italy) (CHI '08). Association for Computing Machinery, New York, NY, USA, 219–222. https://doi.org/10.1145/1357054.1357092

[3] Huidong Bai, Li Zhang, Jing Yang, and Mark Billinghurst. 2021. Bringing full-featured mobile phone interaction into virtual reality. *Comput. Graph.* 97, C (June 2021), 42–53. https://doi.org/10.1016/j.cag.2021.04.004

[4] Eugenie Brasier, Olivier Chapuis, Nicolas Ferey, Jeanne Vezien, and Caroline Appert. 2020. ARPads: Mid-air Indirect Input for Augmented Reality. In *2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)*. IEEE, 332–343. https://doi.org/10.1109/ISMAR50242.2020.00060

[5] Eugenie Brasier, Emmanuel Pietriga, and Caroline Appert. 2021. AR-enhancedWidgets for Smartphone-centric Interaction. In *Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction* (Toulouse & Virtual, France) (MobileHCI ’21). Association for Computing Machinery, New York, NY, USA, Article 32, 12 pages. https://doi.org/10.1145/3447526.3472019

[6] Rahul Budhiraja, Gun A. Lee, and Mark Billinghurst. 2013. Using a HHD with a HMD for mobile AR interaction. In *2013 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)*. 1–6. https://doi.org/10.1109/ISMAR.2013.6671837

[7] Liwei Chan, Rong-Hao Liang, Ming-Chang Tsai, Kai-Yin Cheng, Chao-Huai Su, Mike Y. Chen, Wen-Huang Cheng, and Bing-Yu Chen. 2013. FingerPad: private and subtle interaction using fingertips. In *Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology* (St. Andrews, Scotland, United Kingdom) (UIST ’13). Association for Computing Machinery, New York, NY, USA, 255–260. https://doi.org/10.1145/2501988.2502016

[8] Li-Wei Chan, Hui-Shan Kao, Mike Y. Chen, Ming-Sui Lee, Jane Hsu, and Yi-Ping Hung. 2010. Touching the Void: Direct-Touch Interaction for Intangible Displays. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (Atlanta, Georgia, USA) (CHI ’10). Association for Computing Machinery, New York, NY, USA, 2625–2634. https://doi.org/10.1145/1753326.1753725

[9] Olivier Chapuis, Anastasia Bezerianos, and Stelios Frantzeskakis. 2014. Smarties: An Input System for Wall Display Development. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 2763–2772. https://doi.org/10.1145/2556288.2556956

[10] Niloofar Dezfuli, Mohammadreza Khalilbeigi, Jochen Huber, Florian Müller, and Max Mühlhäuser. 2012. PalmRC: imaginary palm-based remote control for eyes-free television interaction. In *Proceedings of the 10th European Conference on Interactive TV and Video* (Berlin, Germany) (EurolTV ’12). Association for Computing Machinery, New York, NY, USA, 27–34. https://doi.org/10.1145/2325616.2325623

[11] Camille Dupré, Caroline Appert, Stéphanie Rey, Houssem Saidi, and Emmanuel Pietriga. 2024. TriPad: Touch Input in AR on Ordinary Surfaces with Hand Tracking Only. In *Proceedings of the CHI Conference on Human Factors in Computing Systems* (Honolulu, HI, USA) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 754, 18 pages. https://doi.org/10.1145/3613904.3642323

[12] Shariff AM Faleel, Michael Gammon, Kevin Fan, Da-Yuan Huang, Wei Li, and Pourang Irani. 2021. HPUI: Hand Proximate User Interfaces for One-Handed Interactions on Head Mounted Displays. *IEEE Transactions on Visualization and Computer Graphics* 27, 11 (2021), 4215–4225. https://doi.org/10.1109/TVCG.2021.3106493

[13] Steven Feiner, Blair MacIntyre, Tobias Höllerer, and Anthony Webster. 1997. A touring machine: Prototyping 3D mobile augmented reality systems for exploring the urban environment. *Personal Technologies* 1 (1997), 208–217.

[14] Patrick Grady, Jeremy A. Collins, Chengcheng Tang, Christopher D. Twigg, Kunal Aneja, James Hays, and Charles C. Kemp. 2024. PressureVision++: Estimating Fingertip Pressure from Diverse RGB Images. In *2024 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)*. 8683–8693. https://doi.org/10.1109/WACV57701.2024.00850

[15] Jens Grubert, Matthias Heinisch, Aaron Quigley, and Dieter Schmalstieg. 2015. MultiFi: Multi Fidelity Interaction with Displays On and Around the Body. In *Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems*
*Systems* (Seoul, Republic of Korea) (CHI '15). Association for Computing Machinery, New York, NY, USA, 3933–3942.
https://doi.org/10.1145/2702123.2702331

[16] Sean Gustafson, Christian Holz, and Patrick Baudisch. 2011. Imaginary phone: learning imaginary interfaces by transferring spatial memory from a familiar device. In *Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology* (Santa Barbara, California, USA) (UIST '11). Association for Computing Machinery, New York, NY, USA, 283–292. https://doi.org/10.1145/2047196.2047233

[17] Sean G. Gustafson, Bernhard Rabe, and Patrick M. Baudisch. 2013. Understanding palm-based imaginary interfaces: the role of visual and tactile cues when browsing. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (Paris, France) (CHI '13). Association for Computing Machinery, New York, NY, USA, 889–898.
https://doi.org/10.1145/2470654.2466114

[18] Shangchen Han, Beibei Liu, Randi Cabezas, Christopher D. Twigg, Peizhao Zhang, Jeff Petkau, Tsz-Ho Yu, Chun-Jung Tai, Muzaffer Akbay, Zheng Wang, Asaf Nitzan, Gang Dong, Yuting Ye, Lingling Tao, Chengde Wan, and Robert Wang. 2020. MEgATrack: Monochrome Egocentric Articulated Hand-Tracking for Virtual Reality. *ACM Trans. Graph*. 39, 4, Article 87 (aug 2020), 13 pages. https://doi.org/10.1145/3386569.3392452

[19] Shangchen Han, Po-Chen Wu, Yubo Zhang, Beibei Liu, Linguang Zhang, Zheng Wang, Weiguang Si, Peizhao Zhang, Yujun Cai, Tomas Hodan, Randi Cabezas, Luan Tran, Muzaffer Akbay, Tsz-Ho Yu, Cem Keskin, and Robert Wang. 2022. UmeTrack: Unified Multi-View End-to-End Hand Tracking for VR. In *SIGGRAPH Asia 2022 Conference Papers* (Daegu, Republic of Korea) (SA '22). Association for Computing Machinery, New York, NY, USA, Article 50, 9 pages.
https://doi.org/10.1145/3550469.3555378

[20] Chris Harrison, Hrvoje Benko, and Andrew D. Wilson. 2011. OmniTouch: wearable multitouch interaction everywhere. In *Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology* (Santa Barbara, California, USA) (UIST '11). Association for Computing Machinery, New York, NY, USA, 441–450. https://doi.org/10.1145/2047196.2047255

[21] Chris Harrison, Desney Tan, and Dan Morris. 2010. Skininput: Appropriating the Body as an Input Surface. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (Atlanta, Georgia, USA) (CHI '10). Association for Computing Machinery, New York, NY, USA, 453–462. https://doi.org/10.1145/1753326.1753394

[22] Zhe He, Xiangyang Wang, Yuanchun Shi, Chi Hsia, Chen Liang, and Chun Yu. 2025. Palmpad: Enabling Real-Time Index-to-Palm Touch Interaction with a Single RGB Camera. In *Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems* (CHI '25). Association for Computing Machinery, New York, NY, USA, Article 551, 16 pages. https://doi.org/10.1145/3706598.3714130

[23] Steven J. Henderson and Steven Feiner. 2008. Opportunistic controls: leveraging natural affordances as tangible user interfaces for augmented reality. In *Proceedings of the 2008 ACM Symposium on Virtual Reality Software and Technology* (Bordeaux, France) (VRST '08). Association for Computing Machinery, New York, NY, USA, 211–218.
https://doi.org/10.1145/1450579.1450625

[24] Juan David Hincapié-Ramos, Xiang Guo, Paymahn Moghadasian, and Pourang Irani. 2014. Consumed endurance: a metric to quantify arm fatigue of mid-air interactions. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (Toronto, Ontario, Canada) (CHI '14). Association for Computing Machinery, New York, NY, USA, 1063–1072. https://doi.org/10.1145/2556288.2557130

[25] Tom Horak, Sriram Karthik Badam, Niklas Elmqvist, and Raimund Dachselt. 2018. When David Meets Goliath: Combining Smartwatches with a Large Vertical Display for Visual Data Exploration. In *Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems* (Montreal QC, Canada) (CHI '18). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3173574.3173593

[26] ISO. 2000. 9241-9 Ergonomic requirements for office work with visual display terminals (VDTs)-Part 9: Requirements for non-keyboard input devices. *International Organization for Standardization* (2000), 47 pages.

[27] Katja Krug, Wolfgang Büschel, Konstantin Klamka, and Raimund Dachselt. 2022. CleAR Sight: Exploring the Potential of Interacting with Transparent Tablets in Augmented Reality. In *IEEE International Symposium on Mixed and Augmented Reality (ISMAR)*. 196–205. https://doi.org/10.1109/ISMAR55827.2022.00034

[28] Gierad Laput, Robert Xiao, Xiang 'Anthony' Chen, Scott E. Hudson, and Chris Harrison. 2014. Skin buttons: cheap, small, low-powered and clickable fixed icon laser projectors. In *Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology* (Honolulu, Hawaii, USA) (UIST '14). Association for Computing Machinery, New York, NY, USA, 389–394. https://doi.org/10.1145/2642918.2647356

[29] Lik-Hang Lee and Pan Hui. 2018. Interaction Methods for Smart Glasses: A Survey. *IEEE Access* 6 (2018), 28712–28732.
https://doi.org/10.1109/ACCESS.2018.2831081

[30] Robert W. Lindeman, John L. Sibert, and James K. Hahn. 1999. Towards Usable VR: An Empirical Study of User Interfaces for Immersive Virtual Environments. In *Proceedings of the SIGCHI Conference on Human Factors in Computing Systems* (Pittsburgh, Pennsylvania, USA) (CHI '99). Association for Computing Machinery, New York, NY, USA, 64–71.
https://doi.org/10.1145/302979.302995
[31] Mingyu Liu, Mathieu Nancel, and Daniel Vogel. 2015. Gunslinger: Subtle Arms-down Mid-air Interaction. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology (Charlotte, NC, USA) (UIST '15). Association for Computing Machinery, New York, NY, USA, 63–71. https://doi.org/10.1145/2807442.2807489

[32] Anders Markussen, Mikkel Rønne Jakobsen, and Kasper Hornbæk. 2014. Vulture: a mid-air word-gesture keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI '14). Association for Computing Machinery, New York, NY, USA, 1073–1082. https://doi.org/10.1145/2556288.2556964

[33] Daniel Medeiros, Graham Wilson, Mark Mcgill, and Stephen Anthony Brewster. 2023. The Benefits of Passive Haptics and Perceptual Manipulation for Extended Reality Interactions in Constrained Passenger Spaces. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 232, 19 pages. https://doi.org/10.1145/3544548.3581079

[34] Manuel Meier, Paul Streli, Andreas Fender, and Christian Holz. 2021. TapID: Rapid Touch Interaction in Virtual Reality using Wearable Sensing. In 2021 IEEE Virtual Reality and 3D User Interfaces (VR). 519–528. https://doi.org/10.1109/VR50410.2021.00076

[35] Microsoft. 2021. Hand menu. https://learn.microsoft.com/en-us/windows/mixed-reality/design/hand-menu - last accessed 2024-12-17.

[36] Alexandre Millette and Michael J. McGuffin. 2016. DualCAD: Integrating Augmented Reality with a Desktop GUI and Smartphone Interaction. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). 21–26. https://doi.org/10.1109/ISMAR-Adjunct.2016.0030

[37] Mark R. Mine, Frederick P. Brooks, and Carlo H. Sequin. 1997. Moving objects in space: exploiting proprioception in virtual-environment interaction. In Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '97). ACM Press/Addison-Wesley Publishing Co., USA, 19–26. https://doi.org/10.1145/258734.258747

[38] Peter Mohr, Markus Tatzgern, Tobias Langlotz, Andreas Lang, Dieter Schmalstieg, and Denis Kalkofen. 2019. TrackCap: Enabling Smartphones for 3D Interaction on Mobile Head-Mounted Displays. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland UK) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3290605.3300815

[39] Vimal Mollyn and Chris Harrison. 2024. EgoTouch: On-Body Touch Input Using AR/VR Headset Cameras. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology (Pittsburgh, PA, USA) (UIST '24). Association for Computing Machinery, New York, NY, USA, Article 69, 11 pages. https://doi.org/10.1145/3654777.3676455

[40] Jeffrey S. Pierce, Andrew S. Forsberg, Matthew J. Conway, Seung Hong, Robert C. Zeleznik, and Mark R. Mine. 1997. Image plane interaction techniques in 3D immersive environments. In Proceedings of the 1997 Symposium on Interactive 3D Graphics (Providence, Rhode Island, USA) (I3D '97). Association for Computing Machinery, New York, NY, USA, 39–ff. https://doi.org/10.1145/253284.253303

[41] Xiyuan Shen, Chun Yu, Xutong Wang, Chen Liang, Haozhan Chen, and Yuanchun Shi. 2024. MouseRing: Always-available Touchpad Interaction with IMU Rings. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '24). Association for Computing Machinery, New York, NY, USA, Article 412, 19 pages. https://doi.org/10.1145/3613904.3642225

[42] Srinath Sridhar, Anders Markussen, Antti Oulasvirta, Christian Theobalt, and Sebastian Boring. 2017. WatchSense: On-and Above-Skin Input Sensing through a Wearable Depth Sensor. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI '17). Association for Computing Machinery, New York, NY, USA, 3891–3902. https://doi.org/10.1145/3025453.3026005

[43] Hemant Bhaskar Surale, Aakar Gupta, Mark Hancock, and Daniel Vogel. 2019. TabletInVR: Exploring the Design Space for Using a Multi-Touch Tablet in Virtual Reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland UK) (CHI '19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300243

[44] Ying-Chao Tung, Chun-Yen Hsu, Han-Yu Wang, Silvia Chyou, Jhe-Wei Lin, Pei-Jung Wu, Andries Valstar, and Mike Y. Chen. 2015. User-Defined Game Input for Smart Glasses in Public Space. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI '15). Association for Computing Machinery, New York, NY, USA, 3327–3336. https://doi.org/10.1145/2702123.2702214

[45] David Heaney UPLOAD. 2024. Quest v72 Brings Hand Tracking 2.3, Passthrough Keyboard Cutout & More. https://www uploadvr.com/quest-v72-hand-tracking-2-3/ - last accessed 2024-12-30.

[46] Ulrich von Zadow, Wolfgang Büschel, Ricardo Langner, and Raimund Dachselt. 2014. SleeD: Using a Sleeve Display to Interact with Touch-Sensitive Display Walls. In Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces (Dresden, Germany) (ITS '14). Association for Computing Machinery, New York, NY, USA, 129–138. https://doi.org/10.1145/2669485.2669507
[47] Anandghan Waghmare, Youssef Ben Taleb, Ishan Chatterjee, Arjun Narendra, and Shwetak Patel. 2023. Z-Ring: Single-Point Bio-Impedance Sensing for Gesture, Touch, Object and User Recognition. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI '23). Association for Computing Machinery, New York, NY, USA, Article 150, 18 pages. https://doi.org/10.1145/3544548.3581422

[48] Chiu-Hsuan Wang, Chen-Yuan Hsieh, Neng-Hao Yu, Andrea Bianchi, and Liwei Chan. 2019. HapticSphere: Physical Support To Enable Precision Touch Interaction in Mobile Mixed-Reality. In *2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR)*. 331–339. https://doi.org/10.1109/VR.2019.8798255

[49] Cheng-Yao Wang, Wei-Chen Chu, Po-Tsung Chiu, Min-Chieh Hsiu, Yih-Harn Chiang, and Mike Y Chen. 2015. PalmType: Using palms as keyboards for smart glasses. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services. 153–160.

[50] Cheng-Yao Wang, Min-Chieh Hsiu, Po-Tsung Chiu, Chiao-Hui Chang, Liwei Chan, Bing-Yu Chen, and Mike Y. Chen. 2015. PalmGesture: Using Palms as Gesture Interfaces for Eyes-free Input. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services (Copenhagen, Denmark) (MobileHCI '15). Association for Computing Machinery, New York, NY, USA, 217–226. https://doi.org/10.1145/2785830.2785885

[51] Jia Wang and Robert Lindeman. 2014. Coordinated 3D interaction in tablet- and HMD-based hybrid virtual environments. In Proceedings of the 2nd ACM Symposium on Spatial User Interaction (Honolulu, Hawaii, USA) (SUI '14). Association for Computing Machinery, New York, NY, USA, 70–79. https://doi.org/10.1145/2659766.2659777

[52] Martin Weigel, Tong Lu, Gilles Bailly, Antti Oulasvirta, Carmel Majidi, and Jürgen Steimle. 2015. iSkin: Flexible, Stretchable and Visually Customizable On-Skin Touch Sensors for Mobile Computing. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI '15). Association for Computing Machinery, New York, NY, USA, 2991–3000. https://doi.org/10.1145/2702123.2702391

[53] Martin Weigel, Vikram Mehta, and Jürgen Steimle. 2014. More than touch: understanding how people use skin as an input surface for mobile computing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI '14). Association for Computing Machinery, New York, NY, USA, 179–188. https://doi.org/10.1145/2556288.2557239

[54] Michael Wessely, Theophanis Tsandilas, and Wendy E. Mackay. 2016. Stretchis: Fabricating Highly Stretchable User Interfaces. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (Tokyo, Japan) (UIST '16). Association for Computing Machinery, New York, NY, USA, 697–704. https://doi.org/10.1145/2984511.2984521

[55] Ziyi Xia, Xincheng Huang, Sidney S Fels, and Robert Xiao. 2025. HaloTouch: Using IR Multi-Path Interference to Support Touch Interactions with General Surfaces. In Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems (CHI '25). Association for Computing Machinery, New York, NY, USA, Article 548, 17 pages. https://doi.org/10.1145/3706598.3714179

[56] Robert Xiao, Julia Schwarz, Nick Throm, Andrew D. Wilson, and Hrvoje Benko. 2018. MRTouch: Adding Touch Input to Head-Mounted Mixed Reality. IEEE Transactions on Visualization and Computer Graphics 24, 4 (apr 2018), 1653–1660. https://doi.org/10.1109/TVCG.2018.2794222

[57] Cheng Zhang, AbdelKareem Bedri, Gabriel Reyes, Bailey Bercik, Omer T. Inan, Thad E. Starner, and Gregory D. Abowd. 2016. TapSkin: Recognizing On-Skin Input for Smartwatches. In Proceedings of the 2016 ACM International Conference on Interactive Surfaces and Spaces (Niagara Falls, Ontario, Canada) (ISS '16). Association for Computing Machinery, New York, NY, USA, 13–22. https://doi.org/10.1145/2992154.2992187

[58] Yang Zhang, Wolf Kienzle, Yanjun Ma, Shiu S. Ng, Hrvoje Benko, and Chris Harrison. 2019. ActiTouch: Robust Touch Detection for On-Skin AR/VR Interfaces. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology (New Orleans, LA, USA) (UIST '19). Association for Computing Machinery, New York, NY, USA, 1151–1159. https://doi.org/10.1145/3332165.3347869

[59] Yang Zhang, Junhan Zhou, Gierad Laput, and Chris Harrison. 2016. SkinTrack: Using the Body as an Electrical Waveguide for Continuous Finger Tracking on the Skin. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI '16). Association for Computing Machinery, New York, NY, USA, 1491–1503. https://doi.org/10.1145/2858036.2858082

[60] Fengyuan Zhu and Tovi Grossman. 2020. BISHARE: Exploring Bidirectional Interactions Between Smartphones and Head-Mounted Augmented Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI '20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376233

[61] Fengyuan Zhu, Mauricio Sousa, Ludwig Sidenmark, and Tovi Grossman. 2024. PhoneInVR: An Evaluation of Spatial Anchoring and Interaction Techniques for Smartphone Usage in Virtual Reality. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI '24). Association for Computing Machinery, New York, NY, USA, Article 639, 16 pages. https://doi.org/10.1145/3613904.3642582