Reviewerly/exHarmony huggingface dataset:
Extended Abstract: Interaction-Triggered Estimation of AR Object Placement on Indeterminate Meshes

Current Augmented Reality devices rely heavily on live environment mapping to provide convincing world-relative experiences through user interaction with the real world. This mapping is obtained and updated through many different algorithms but often contains holes and other mesh artifacts when generated in less ideal scenarios, like outdoors and with fast movement. In this paper, we present the Interaction-Triggered Estimation of AR Object Placement on Indeterminate Meshes, a work-in-progress application providing a quick, interaction-triggered method to estimate the normal and position of missing mesh in real-time with low computational overhead. We achieve this by extending the user's hand using a group of additional raycast sample points, aggregating results according to different algorithms, and then using the resulting values to place an object.
Cybersickness assessment framework(CSAF): An Open Source Repository for Standardized Cybersickness Experiments

Studies of cybersickness (CS) often require a significant investment in creating the primary VR environment and other experiment-related features. In addition, minor differences in VR content in independent studies may lead to opposite results. This motivates us to introduce a standardized CS assessment framework to facilitate the development of CS-related experiments. This paper describes the concept of the framework, the current progress of the technical implementation, and future goals.
Privacy Threats of Behaviour Identity Detection in VR

Modern VR technology generates large volumes of data by attaching various sensors around the user. Over time, researchers have used these data to create unique behavioural keys to provide authentication to immersive applications. However, these approaches come with various behavioural privacy risks. Hence, through this work, we investigate the privacy risks associated with behavioural identity detection in VR and how users' physical appearance affects this behavioural detection. We found that users could be identified across various sessions with 80% accuracy once tracked, and that physical appearance did not impact the behaviour detection.
Real-Time Recognition of In-Place Body Actions and Head Gestures using Only a Head-Mounted Display

Body actions and head gestures are natural interfaces for interaction in virtual environments. Existing methods for in-place body action recognition often require hardware more than a head-mounted display (HMD), making body action interfaces difficult to be introduced to ordinary virtual reality (VR) users as they usually only possess an HMD. In addition, there lacks a unified solution to recognize in-place body actions and head gestures. This potentially hinders the exploration of the use of in-place body actions and head gestures for novel interaction experiences in virtual environments. We present a unified two-stream 1-D convolutional neural network (CNN) for recognition of body actions when a user performs walking-in-place (WIP) and for recognition of head gestures when a user stands still wearing only an HMD. Compared to previous approaches, our method does not require specialized hardware and/or additional tracking devices other than an HMD and can recognize a significantly larger number of body actions and head gestures than other existing methods. In total, ten in-place body actions and eight head gestures can be recognized with the proposed method, which makes this method a readily available body action interface (head gestures included) for interaction with virtual environments. We demonstrate one utility of the interface through a virtual locomotion task. Results show that the present body action interface is reliable in detecting body actions for the VR locomotion task but is physically demanding compared to a touch controller interface. The present body action interface is promising for new VR experiences and applications, especially for VR fitness applications where workouts are intended.
Empirically Evaluating the Effects of Eye Height and Self-Avatars on Dynamic Passability Affordances in Virtual Reality

Over the past two decades self-avatars have been shown to affect the perception of both oneself and of environmental properties including the sizes and distances of elements in immersive virtual environments. However, virtual avatars that accurately match the body proportions of their users remain inaccessible to the general public. As such, most virtual experiences that represent the user have a generic avatar that does not fit the proportions of the users' body. This can negatively affect judgments involving affordances, such as passability and maneuverability, which pertain to the relationship between the properties of environmental elements relative to the properties of the user providing information about actions that can be enacted. This is especially true when the task requires the user to maneuver around moving objects like in games. Therefore, it is necessary to understand how different sized self-avatars affect the perception of affordances in dynamic virtual environments. To better understand this, we conducted an experiment investigating how a self-avatar that is either the same size, 20% shorter, or 20% taller, than the user's own body affects passability judgments in a dynamic virtual environment. Our results suggest that the presence of self-avatars results in better regulatory and safer road crossing behavior, and helps participants synchronize self-motion to external stimuli quicker than in the absence of self-avatars.
Enhancing the Reading Experience on AR HMDs by Using Smartphones as Assistive Displays

The reading experience on current augmented reality (AR) head mounted displays (HMDs) is often impeded by the devices' low perceived resolution, translucency, and small field of view, especially in situations involving lengthy text. Although many researchers have proposed methods to resolve this issue, the inherent characteristics prevent these displays from delivering a readability on par with that of more traditional displays. As a solution, we explore the use of smartphones as assistive displays to AR HMDs. To validate the feasibility of our approach, we conducted a user study in which we compared a smartphone-assisted hybrid interface against using the HMD only for two different text lengths. The results demonstrate that the hybrid interface yields a lower task load regardless of the text length, although it does not improve task performance. Furthermore, the hybrid interface provides a better experience regarding user comfort, visual fatigue, and perceived readability. Based on these results, we claim that joining the spatial output capabilities of the HMD with the high-resolution display of the smartphone is a viable solution for improving the reading experience in AR.
Heat Metaphor for Attention Estimation for Educational VR

We prototype a technique, for educational VR applications, to estimate each student's level of attention in real time. Our system attaches scores to both students and objects, which change in response to eye-tracked gaze intersections. Compared to a simple angle-based approach, our system provides a dynamic and granular representation of object importance and frees the lesson designer from having to fully define objects of interest and timings. Our system takes into account simultaneous behaviors of multiple students and filters out brief behavioral deviations of attentive students. The results may help a teacher or a virtual agent better guide students.
[DC] Supporting Embodied Sensemaking in Immersive Environment

Traditional desktop-based visualisations have their limitations in supporting and improving the sensemaking of data that is inherent 3D, such as spatio-temporal data. Immersive analytics and embodiment provide new opportunities to understand this kind of data in a more engaging way. However, little research explored how to support the whole sensemaking process in immersive environment, from low level of data understanding and exploration to higher level of provenance. My doctoral research focuses on supporting spatio-temporal data sensemaking using embodiment in immersive environment. We hope to provide guidance for developing embodied tools that aim to accomplish various levels of sensemaking tasks in immersive spaces.
An Augmented Reality Application and User Study for Understanding and Learning Architectural Representations

Advances in computational technology provide opportunities to explore new methods to improve spatial abilities and the understanding of buildings in architecture education. The research employed BIMxAR, a Building Information Modeling-enabled AR educational tool with novel visualization features to support learning and understanding construction systems, materials configuration, and 3D section views of complex building structures. We validated the research through a test case based on a quasi-experimental research design, in which BIMxAR was used as an intervention. Two study groups were employed - non-AR and AR. The learning gain differences within and between the groups were not statistically significant, however, the AR group perceived significantly less workload and higher performance compared to the non-AR group. These findings suggest that the AR version is an easy, useful, and convenient learning tool.
Towards Trustworthy Augmented Reality in The Metaverse Era: Probing Manipulative Designs in Virtual-Physical Commercial Platforms

E-commerce has become an important activity where new advances in technology shape the shopper experience. At the same time, the metaverse is seen as the next milestone to revolutionize the e-commerce experience, where immersion, realism, and ubiquity are its main features. However, under such circumstances, manipulative designs to ‘trick’ users toward intended choices or outcomes can become more effective. This paper sheds light on the design space of manipulative techniques in e-commerce applications for the meta-verse, reinforcing our understanding of interface design guidelines and counteracting malicious practices.
Effect of Look-Alike Avatars on Students' Perceptions of Teaching Effectiveness

This paper presents a study investigating the influence of look-alike avatars on students' perceptions of teaching effectiveness in three-dimensional virtual and augmented reality environments. This study investigated three avatar representations: i) look-alike avatar of the instructor; ii) stick avatar; and iii) video recording of the instructor. Eighteen participants were asked to rank the three avatar representations, as well as the immersion experience of the teaching simulation on the virtual and augmented reality displays. The result of this study suggests that look-alike avatars can be used to represent an instructor in virtual environments.
Virtual Optical Bench: Teaching Spherical Lens Layout in VR with Real-Time Ray Tracing

Teaching in optical systems design is usually performed on an optical bench. While experimentation plays an important role in education, experiments involving expensive or dangerous components are usually limited to short, heavily supervised sessions. Computer simulations, on the other hand, offer high accessibility, but suffer from reduced realism and tangibility when presented on a 2D screen. For this reason, we present the virtual optical bench, an application that lets users explore spherical lens layouts in virtual reality (VR). We implemented a numerically accurate simulation of optical systems using Nvidia OptiX, as well as a prototypical VR application, which we then evaluated in an expert review with 6 optics experts. Based on their feedback, we re-implemented our VR application in Unreal Engine 4. The re-implementation has since been actively used for teaching optical layouts, where we performed a qualitative evaluation with 18 students. We show that our virtual optical bench achieves good usability and is perceived to enhance the understanding of course contents.
Generating Co-Speech Gestures for Virtual Agents from Multimodal Information Based on Transformer

To generate co-speech gestures for virtual agents and enhance the correlation between gestures and input modalities, we propose a Transformer-based model, which encodes four-modal-like information (Audio Waveform, Mel-Spectrogram, Text, and SpeakerIDs). For the Mel-Spectrogram modal, we design a Mel-Spectrogram encoder based on the Swin Transformer pre-trained model to extract the audio spectrum features hierarchically. For the Text modal, we use the Transformer encoder to extract text features aligned with the audio. We evaluate on the TED-Gesture dataset. Compared with the state-of-art methods, we improve the mean absolute joint error by 2.33%, the mean acceleration difference by 15.01%, and the Fréchet gesture distance by 59.32%.
Exploring Influences of Design and Environmental Factors on Recognizing Teacher's Facial Expressions in Educational VR

Teachers' facial expressions significantly influence students' willingness to learn. However, insufficient resolution of current consumer head-mounted displays leads to difficulties in capturing nuances of virtual teachers in VR, which may be solved by scaling up the heads of virtual teachers. In this paper, we explore the effects of design and environmental factors on the ideal head scales for recognizing the virtual teacher's facial expressions. Our results show that the facial visualization style and the distance from the user to the virtual teacher are important factors affecting facial expression recognition, which could contribute to the design and optimization of educational VR.
Immersive and Interactive 3D Visualization of Large-Scale Geo-Scientific Data

In this paper, we present a software architecture and framework developed over the past decade to enable scalable and highly interactive visualizations for large datasets and display sizes. The framework integrates distributed data processing, data streaming, and dynamic scheduling to allow for view-dependent feature extraction and progressive data streaming. Additionally, the framework has been designed to support visualizations from local desktop workstations to large multi-display virtual environments.
HandAttNet: Attention 3D Hand Mesh Estimation Network

Hand pose estimation and reconstruction become increasingly compelling in the metaverse era. But in reality hands are often heavily occluded, which makes the estimation of occluded 3D hand meshes challenging.Previous work tends to ignore the information of the occluded regions, we believe that the occluded regions hand information can be highly utilized, Therefore, in this study, we propose hand mesh estimation network, HandAttNet.We design the cross-attention mechanism module and the DUO-FIT module to inject hand information into the occluded region.Finally, we use the self-attention regression module for 3D hand mesh estimation.Our HandAttNet achieves SOTA performance.
Replicability and Transparency for the Creation of Public Human User Video Game Datasets

Replicability is absent in games research; a lack of transparency in protocol detail hinders scientific consensus and willingness to publish public datasets, impacting the application of these techniques in video games research. To combat this, we propose and give an example of the use of a set of experimental considerations, such as games and materials choice. This work promotes the communication of research protocols when publishing datasets, benefiting researchers when designing experiments.
Descriptive Linguistic Patterns of Group Conversations in VR

Although talking is one of the most common activities in social VR, there is little empirical work identifying what people say and how they communicate in immersive, virtual settings. The current paper addresses this opportunity by performing automated text analysis on over 4,800 minutes of in-Vr, small group conversations. These conversations took place over the span of two months during a university course where 171 students attended discussion sections via head-mounted displays. We provide a methodology for analyzing verbal communication along two dimensions: content and structure. We implement methods to describe linguistic patterns from the class and introduce a preliminary VR Dictionary.
Magic, Superpowers, or Empowerment? A Conceptual Framework for Magic Interaction Techniques

This poster presents an approach to systematically distinguish between interaction techniques (ITs) in the context of magic ITs in immersive virtual environments. Currently, heterogeneous terms are used in research to describe the concept of enhancing the abilities of users beyond the limits of the real world, such as magic, super-natural, hyper-natural, superhuman abilities, superpowers, augmentation or empowerment. As a first step towards clarifying and systematically defining the terminology, we propose using the orthogonal concepts of interalizability, congruence, and enhancement (or ICE-cube) as a simple yet expressive conceptual framework.
Style-aware Augmented Virtuality Embeddings (SAVE)

We present an augmented virtuality (AV) pipeline that enables the user to interact with real-world objects through stylised representations which match the VR scene and thereby preserve immersion. It consists of three stages: First, the object of interest is reconstructed from images and corresponding camera poses recorded with the VR headset, or alternatively a retrieval model finds a fitting mesh from the ShapeNet dataset. Second, a style transfer technique adapts the mesh to the VR game scene in order to preserve consistent immersion. Third, the stylised mesh is superimposed on the real object in real time to ensure interactivity even if the real object is moved. Our pipeline serves as proof of concept for style-aware AV embeddings.
High Levels of Visibility of Virtual Agents Increase the Social Presence of Users

Virtual humanoid agents can perform a variety of tasks, and their representation has an influence on several psychological aspects. We compared the effects of agent visibility during an anagram solving task in virtual reality on social presence and cognitive task performance. We increased the visibility from (i) voice-only, (ii) mouth-only, (iii) full head, (iv) upper body, to (v) full body representations of the same virtual agent in a within-subject design study. The re-sults revealed significant differences in the perceived social presence, especially for the two least visible representations, i.e., voice- and mount -only, but no significant effects on task performance could be found.
Enhancing Culinary Training with Spatial Augmented Reality: A User Study Comparing sAR Kitchen and Video Tutorials

This paper presents sAR Kitchen, a cooking assistant that aims to enhance culinary training through the use of spatial augmented reality (sAR). We conducted a user study with twenty-two participants to investigate the effects of instructions given by our proposed sAR system and a monitor display with video tutorials in a playdough-making task representing the cooking task. Our study measured participants' perceived workload, usability, and performance regarding task completion time, product quality, and cooking station messiness. The results showed that sAR significantly decreased the perceived workload and increased the system's usability compared to traditional video-based tutorials. Additionally, our open-ended questions revealed that participants found the sAR instructions more engaging and interactive. Overall, this study demonstrates the potential of sAR to provide more user-friendly learning methods in culinary training.
Children's evaluation of a virtual reality boating safety education resource

Research on designing and using immersive virtual reality with children for educative purposes is an emerging field. It is important to deploy participatory design techniques to highlight the perspectives of children so that the design of content, characters, interactive elements and narrative resonate with young learners. This paper reports on an evaluation of a proof-of-concept 360-degree learning resource designed to educate children about boating safety. The resource was developed as part of a government service safety initiative to be integrated into a short in-school boating safety session facilitated by maritime safety officers. A mixed methodology was used during a focus group with primary (elementary) school aged children. Findings were that children wanted to experience the learning resource via a virtual reality headset with their Systems Usability Scale results being positive. However, most had difficulty adjusting the headset for visual clarity and a minority managed to experience all 360-degree scenes and their embedded content. Children enjoyed the interactive nature of the resource, felt transported to the boating location and liked human and animated characters in the story. The evaluation was useful in clarifying aspects of the learning resource that were valued by children and it raised questions about whether the resource might best be used during the short in-school boating safety sessions in computer or mobile device screen mode rather than via a virtual reality headset.
Co-building Viewer's Representation: Animated Documentary Extended by Virtual Reality from the Perspective of Anthropology

Since the crisis of representation in the 1980s, the anthropological community has adhered to the interdisciplinary experimental spirit and sought innovative models of expression for contemporary ethnography. This paper regards animated documentary as an important force in participating in the contemporary anthropological representation experiment, and it places a strong emphasis on how animated documentary is extended by virtual reality; furthermore, it analyzes the development from the creator's representation to the viewer's representation. The representation of viewers based on their experiences also reshapes the creator's representation and expands the dialogue relationship between multiple agents.
AR Interfaces for Disocclusion—A Comparative Study

An important application of augmented reality (AR) is the design of interfaces that reveal parts of the real world to which the user does not have line of sight. The design space for such interfaces is vast, with many options for integrating the visualization of the occluded parts of the scene into the user's main view. This paper compares four AR interfaces for disocclusion: X-ray, Cutaway, Picture-in-picture, and Multiperspective. The interfaces are compared in a within-subjects study (N = 33) over four tasks: counting dynamic spheres, pointing to the direction of an occluded person, finding the closest object to a given object, and finding pairs of matching numbers. The results show that Cutaway leads to poor performance in tasks where the user needs to see both the occluder and the occludee; that Picture-in-picture and Multiperspective have a visualization comprehensiveness advantage over Cutaway and X-ray, but a disadvantage in terms of directional guidance; that X-ray has a task completion time disadvantage due to the visualization complexity; and that participants gave Cutaway and Picture-in-picture high, and Multiperspective and X-ray low usability scores.