Interactive Gesture Elicitation through Immersive Computing and Semantic Modelling
Creators: name: Perera, Madhawa
Publisher: name: The Australian National University
Year: 2023

description: Technological advances constantly enable new methods for empirical studies. In this context, immersive computing shows potential for future empirical research by making knowledge discoveries possible, even in resource-constrained environments. Yet only recently has immersive computing increased its reach to consumers and became available for behaviour research. Thus, to date, it is unclear whether and how immersive computing can be utilised, and what its effects on different empirical studies in human-computer interaction (HCI) may be. This thesis investigates the use of immersive environments (IEs) as a technology to enhance end-user elicitation studies. It focuses on hand gesture elicitation and empirical study design. End-user gesture elicitation studies (GESs) are the cornerstone of gesture design research. They are used extensively when designing gesture-controlled interfaces. Despite the perceived effectiveness of such studies, researchers have identified notable limitations in their still predominantly lab-based research designs, and in how the technology at a given time can hamper creative solutions to the problems that behavioural research faces. For example, when investigating users' gesture interactions, the environment setup is crucial for uncovering nuanced interaction information. Researchers aspire to collect a diverse set of participant data from different demographics, yet end up collecting homogeneous data samples in time-consuming, chronologically conducted lab-based GESs. Researchers continue to face the hurdle of removing the gulf between execution and eliciting natural user interactions for emerging futuristic systems without having sophisticated instruments and lab setups. The use of emerging technologies, such as Augmented Reality (AR) and Virtual Reality (VR), as a medium to facilitate future empirical research such as GESs remains under-explored. With the increasing accessibility of consumer-grade immersive devices (sometimes referred to as Metaverse devices), this research investigates the use of IEs to facilitate future end-user GES research. In this research, a novel empirical research design is proposed which is capable of remote, encapsulated, and self-guided GESs. The study then goes on to evaluate the feasibility of overcoming major criticisms of GESs, i.e., legacy bias and lack of diverse samples. Further, this research investigates how users interact in IEs compared to similar real-world setups, and postulates that IEs are useful in this kind of empirical research. This novel design enables observation and recording of qualitative and quantitative data such as three-dimensional kinematic data of human hands to understand the effects on interaction choices/decisions with no or minimum involvement of the investigator. Finally, to address the challenge of elicited data representation, this research proposes a technique to model ubiquitous hand data semantically, with observed relationships, using an ontological approach to give the data human- and computer-understandable and usable representations. The work demonstrates how ontological data representation can increase the usability of GES outcomes and help gestural interface designers and developers to extract meaningful information for their application development, without having to conduct redundant GESs. The findings of this research set examples for intuitive, remote and privacy-preserving ways of conducting self-guided GESs using VR, which can also overcome challenges imposed by unforeseen circumstances such as a pandemic. The findings of this research have value not only for the HCI research community that uses the end-user gesture elicitation but also for design practitioners to critically re-evaluate conventional approaches to doing research and enter a futuristic domain where research is not limited by the constraints of the physical world.