Gestural Interaction in Virtual Environments: User Studies and Applications

With the current available technology there has been an increased interest in the development of virtual reality (VR) applications, some of those already becoming commercial products. This fact also rose many issues related to their usability. One of the main challenges is the design of interfaces to interact with these virtual environments (VE), in particular for those setups relying on hand tracking as a mean of input. This research project aims to tackle some of the critical aspects of the interaction based on hand tracking in VE by proposing novel interaction technique for object manipulation and gesture based interfaces with intent of addressing relevant issues affecting usability of this kind of VR applications. 1 Background of the research project Virtual Reality (VR), is a term used for those applications that can simulate physical presence in places of the real world or imagined worlds. These VR applications recreate sensory experiences, which could in theory also include virtual taste, sight, smell, sound, touch, etc. Most current VR environments are mainly composed of visual experiences, through the use of a regular computer screen or with special stereoscopic displays, and some additional sensory information, like sound through headphones or speakers targeted towards users. The availability of low-cost devices for visualization and interaction gestures are reviving the interest in VR and the technology matured enough to offer many applications of potential interest, not only in games and entertainment, but also for other specific uses such as scientific and medical visualization, augmented reality, etc. What limits the use of immersive 3D environments in these areas, are often the difficulties of interaction that, despite the increasingly sophisticated and relatively cheap tracking devices, make unusable applications that could have significant utility and a large number of users. Therefore it becomes crucial to study and test the different tasks to evaluate which interaction paradigm is the best, before proposing the applications themselves to potential users. Even though the ability to interact with the virtual environment is not required in order to speak of VR, the most interesting applications of this technology come along with the faculty of performing one or more tasks based on the application purpose. These tasks involve an interaction between both the user and the virtual objects present in the environment (i.e. grabbing a virtual object). Such feature is anything but trivial; depending on the complexity of the task, a large number of problems may arise. One of the main goal of research in this field is to achieve naturalness of interaction with virtual environments. [4] In order to achieve such goal, research work is currently focusing on two main aspects. The first is the understanding of peoples mental models of interaction with these virtual environments. In fact, due to the interface/devices layer of interaction, users may develop different interaction models from those used in a real physical environment regardless of the kind of task they wish to perform. Understanding this is a key factor in the development of the mid-air interaction interface for a different number of tasks (e.g. virtual assembly, shape modeling, etc.) in order to design an interface perceived as natural by most people. The second one is about the development of actual interaction interfaces possibly based on guidelines derived from the knowledge acquired from the kind of research described above. [8] The aim of my PhD activity is to investigate on open issues related to object manipulation and gesture-based interfaces in immersive virtual environments both analyzing user preferences and interaction metaphors that implementing and testing practical solutions with low cost hardware. 2 Open issues and research aims Since the early days of virtual environments, interaction with virtual objects in the scene has been one of the main object of studies. Considering threedimensional virtual environments, interaction isn’t trivial, mainly due to the required mapping between traditional input devices (2D) and the virtual environment (3D). Most common solutions resort to techniques that somehow relate the actions performed in the two-dimensional space of the input device (e.g. mouse cursor or touch) to three-dimensional transformations. Since it is usual for people to interact with these kind of environments with traditional displays, 3D content is displayed in a 2D rendered image, which hinders content’s perception. To overcome both the limitations of the input and the output devices, mainstream solutions for creating and editing 3D virtual content, namely computer-aided design (CAD) tools, resort to different orthogonal views of the environment. This allows a more direct two-dimensional interaction with limited degrees of freedom. Solutions that offer a single perspective view usually either apply the transformation in a plane parallel to the view plane, or resort to widgets that constraint interactions and ease the 2D-3D mapping. [9] Research has shown that the first approach can sometimes result in unexpected transformations when users are allowed to freely navigate through the virtual environment, and that constrained interactions allow for more accurate manipulations. [10] Recent technological advances lead to an increased interest in immersive virtual reality settings. Affordable hardware for immersive visualization of virtual environments, such as the Oculus Rift head-mounted display (HMD), ease the perception of three-dimensional content. Moreover, advances in user tracking solutions make possible to know where users’ head, limbs and hands are in space. This allows for more direct interactions, mimicking the ones with physical objects. First results of our work showed that mid-air direct interactions with 3D virtual content can reduce tasks’ duration and are appealing to users. Although mid-air interactions show promising results, the accuracy of human spatial interactions is limited. Moreover, the limited dexterity of mid-air hand gestures, aggravated by lack of precision from tracking systems and lowdefinition of current HMDs, constrain precise manipulations. This precision is of extreme importance when creating or assembling engineering models or architectural mock-ups, for instance. Even if a large number of methods have been proposed in the literature, most demo systems do not provide easy to use interfaces and the possibility of reaching a high level of precision. This is also due to more intrinsic issues of the tracking task such as the inaccuracy in tracking, occlusions, difficulty in segmentation of different gestural primitives, and other non-trivial problems. There is surely a lot of room for improvements, not necessarily related to a better tracking of body landmark but also to a smart global processing of 3D keypoint trajectories. Several methods have been recently presented for characterizing salient points and for global shape description, with invariance properties and robustness against various kinds of perturbation. Methods with similar characteristics but adapted to the different kind of shape data provided by tracking devices could in principle be applied for the solution of open issues in gestural interaction. An example of smart application of simple geometric processing to realize effective interaction comes from 2D touchscreen interaction, where many gesture recognition applications are not based on complex time series analysis, but on the reduction of the problem to a simple template matching of 2D shapes. The popular 1-dollar recognizer [17] and similar derived methods (also proposed for 3D interaction, e.g. [11] are a clear demonstration of the usefulness of this simplification. This may seem to indicate that in this case the dynamic information may be neglected without losing the meaningful part of the signal. In the following two of the main aspects covered by the thesis are presented along with respective issues that are relevant topic of interest for the HCI community and still need to be further investigated. Object manipulation in VR environments There are different ways to interact with virtual object present in a VR scene. The type of interaction is determined by the kind of task the user is allowed to perform and ultimately by the purpose of the application featuring such interactions. One specific kind of interaction is the object manipulation. In this case the user is allowed more or less directly depending on the metaphor proposed by the implemented technique to select an object in the scene e perform one or more different transformations (i.e. translation rather than rotations or scaling operations). This kind of interaction has many limitations, however. First, user’s arm has a limited range of interaction due to real world constraints (e.g. it’s safe to assume arms are always anchored to user’s bodies) and sensors detection volume in case of deviceless setups. This limit can be worked around with the use of a navigation technique. Such technique is required as it also allows different visual perspectives, but it should not be required depending on the application. Manipulation of large objects is also an issue as they tend to obscure the users view during positioning tasks unless additional workarounds are implemented to give the user a way to change the point of a view for a more convenient perspective. [3] Gestural interaction for VR interfaces Gestural interaction is particular successful on 2D touchscreens where it is easy to determine the beginning and the end of the gesture by using the contact between fingers and display surface. In 3D deviceless interaction, the beginning of a gesture must be automatically detected from the gesture itself. The use of pattern recognition tools may allow a robust recognition of a gesture learned after the gesture realization, that may