title: Advancing human-computer interaction: acoustic-based intelligent hand tracking and recognition for the future
Creators: name: Han, Wei
Publisher: name: The University of Edinburgh
Year: 2025
Subjects: subject: Human-Computer Interaction
subject: HCI mechanisms
subject: tracking algorithms
subject: air-writing
subject: dual-user system
subject: 3D hand tracking
subject: acoustic-based hand sensing technology
subject: Virtual Reality

description: Acoustic-based hand tracking technology is heralding a new era in Human-Computer Interaction (HCI) by utilising the speakers and microphones already embedded in many commercial devices. This approach leverages acoustic signals to detect and process echoes, thereby determining the position of a user’s hand in space. This thesis commences with the construction of a circuit prototype designed to refine algorithm development and reduce the constraints associated with direct testing on commercial devices. The prototype features adjustable placements for speakers and microphones, allowing for flexibility in their positioning. Initial evaluations on this prototype demonstrated a 1D hand tracking accuracy of 3.7 mm, also confirming the system’s capability to accurately track 2D trajectories and visually recognise them. A key application of acoustic-based hand tracking explored in this thesis is airwriting, where the user can write characters in mid-air, and the system records and recognises the resulting trajectories. Existing tracking approaches do not support multi-stroke input, as a result, the trajectory is incapable of character recognition with models trained by open source hand written characters images. As a result, a virtual writing system is proposed to estimate the status of the hand with the energy information acquired from the echo. Subsequently, Zadoff-Chu (ZC) sequences and sinusoidal waves are used to obtain the initial position of the hand, and track the hand continuously with the change of phase for a smooth trajectory. While inputting the characters, the virtual writing system allows the user to input multiple strokes to get rid of redundant trajectory which affects the recognition. The results show that the virtual writing system achieves an average error of 4.3 mm for 2D tracking and 94.8% recognition accuracy for 52 English letters, 10 numbers, and 20 Chinese characters. Another critical challenge with existing approaches is the limited accuracy in accommodating multiple users, which restricts their utility in scenarios such as multiplayer gaming. A dual-user interaction system is then proposed to enable precise hand tracking for cursor control. Notably, it supports two users to operate independently. For the first user, a dedicated set of one speaker and two microphones are employed. To enable the participation of a second user, an additional speaker and two supplementary microphones were positioned in the opposite direction. To minimise the interference between users, distinct frequencies of sinusoidal waves and ZC signals generated by the co-prime root indexes were allocated for each user. The proposed method demonstrates remarkable performance, achieving an average hand tracking error of 5.7 mm, and maintains consistent accuracy in dual-user scenarios. Finally, the thesis explores the expansion from 2D to 3D hand tracking, a necessary evolution to capture more complex hand gestures and movements. Implementing a 3D tracking system requires at least three, and ideally four microphones to accurately compute hand coordinates in 3D space. Modifications to the original prototype allow for the integration of additional microphones, and a robust mathematical framework is developed to translate multiple distance measurements into precise 3D coordinates, achieving an average 3D hand tracking error of 1.9 cm with four microphones. In summary, this thesis advances the field of HCI by developing robust and innovative solutions for acoustic-based hand sensing, specifically designed for close-range interactions within 5 – 30 cm. It lays the groundwork for more natural and intuitive user interactions, providing a strong foundation for future research aimed at refining these systems and integrating them into everyday technological applications.