Reviewerly/exHarmony huggingface dataset:
Enhancing Auditory Immersion in Interactive Virtual Reality Environments

The realm of sound remains relatively underexplored in comparison to its visual counterpart within the context of interactive virtual reality (IVR). It is a well-known practice to create a digital representation of the user’s human body in the virtual space, thereby enhancing human-computer interaction and facilitating navigation amid fellow users and objects within these environments, commonly referred to as avatars. Human sensory feedback is predominantly oriented towards visual stimuli. The sonic design, presence and interactions of avatars are often neglected in research. However, a more accurate representation of the virtual body, both visual and sonic, can increase immersion and the overall effectiveness of the user experience.By studying how users react to their own sonic environments and to the sounds created by themselves/their avatars, developers could create more personalized scenarios for IVR applications. To the best of my knowledge, there is still a lot to explore in this field.The goal of this research is to investigate the effects of sonic interactions between humans and computers in VR, where the users have six degrees of freedom to interact with their environment. This doctoral research aims to find the answers to the following questions: How can auditory immersion be enhanced in IVR? How do interactive sounds affect body ownership in a virtual environment? How do procedurally generated sounds affect the agency of the user in IVR?The focus of this study will be on finding ways to enhance the plausibility illusion experienced by the users, which allows them to believe that the given scenario is truly happening, even though they know it is not real. Both sample-based, and procedural audio productions will be explored to create a more effective user experience. Advancements in this field of study could be applied for both entertainment, serious games, health, and other IVR applications, to increase their desired impact.
OCTOPUS: Open-vocabulary Content Tracking and Object Placement Using Semantic Understanding in Mixed Reality

One key challenge in augmented reality is the placement of virtual content in natural locations. Existing automated techniques are only able to work with a closed-vocabulary, fixed set of objects. In this paper, we introduce a new open-vocabulary method for object placement. Our eight-stage pipeline leverages recent advances in segmentation models, vision-language models, and LLMs to place any virtual object in any AR camera frame or scene. In a preliminary user study, we show that our method performs at least as well as human experts 57% of the time. <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup>
Study of Natural Language Dialog System for Avatar Communication in Metaverse

Communication between humans and avatars will increase in the Metaverse, and the quality of that conversation will be important. Natural language communication of man-machine interface is a key technology. In addition, the Metaverse also requires automatic avatar responses that consider local content. In this research, we are developing a system that implements natural language processing using sentence transformers to achieve accurate local communication between humans and avatars in the Metaverse.
Exploring the Impact of User and System Factors on Human-AI Interactions in Head-Worn Displays

Empowered by the rich sensory capabilities and the advancements in artificial intelligence (AI), head-worn displays (HWD) could understand the user’s contexts and provide just-in-time assistance to users’ tasks to augment their everyday lives. However, there has been limited understanding of how users perceive interacting with AI services, and how different factors impact the user experience in HWD applications. In this research, we investigated broadly what user and system factors play important roles in human-AI experiences during an AI-assisted spatial task. We conducted a user study to simulate an everyday scenario where augmented reality (AR) glasses could provide suggestions/assistance. We researched three AI system factors (performance, initiation, transparency) with multiple user factors (personality traits, trust propensity, and prior trust with AI). We not only identified the impact of user traits such as the levels of conscientiousness and prior trust with the AI, but also found interesting interactions between them and system factors such as AI’s performance and initiation strategy. Based on the findings, we suggest that future AI assistance on HWD needs to take users’ individual characteristics into account and customize the system design accordingly.
Implementing First-Person Shooter Game AI in WILD-SCAV with Rule-Enhanced Deep Reinforcement Learning

Deep Reinforcement Learning (DRL) algorithms have achieved remarkable performance in various types of games. However, in First-Person Shooter (FPS) games the progress is relatively limited due to the partial observability, sparsity of reward function, complexity in state and action spaces brought by a 3D open environment, and the lack of satisfactory training and testing benchmarks. To this end, Wilderness Scavenger (WILD-SCAV), a novel FPS game environment was developed and used for an online competition held by Inspire-AI and IEEE Conference on Games <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> , in which we introduced a rule-based agent and wined all three tracks as navigation, supply gathering, and supply battle. In this work, we propose a rule-enhanced deep reinforcement learning algorithm, adopting localization and reconstruction techniques to create 3D occupancy grid maps based on visual observations, and uses them together with heat map and other game variables as the input of a reinforcement learning neural network to conduct basic navigation tasks. Further enhanced and expanded by rules, this navigation agent can handle even more complicated supply gathering and battle tasks. Pre-trained with the previously proposed winning rules in the competition through imitation learning, then trained with distributed PPO algorithm, our agent has better performance over multiple rule-based and RL-based agents, including the agent using winning policy.
Predictive Models and Monte Carlo Tree Search: A Pipeline for Believable Agents

Developing and assessing believable agents remains a sought out challenge. Recently, research has approached this problem by treating and assessing believability as a time-continuous phenomenon, learning from collected data to predict believability of games and game states. Our study will build on this work: by integrating this believability model with a game agent to affect its behaviour. In this short paper, we first describe our methodology and then the results obtained from our user study, which suggests that this methodology can help creating more believable agents, opening the possibility of integrating this type of models into game development. We also discuss the limitations of this approach, possible variants to tackle these, and ideas for future work to extend this preliminary work.
Comparative Evaluation of Virtual Reality and In-Person Onboarding for Assembly Trainings in Manufacturing

In recent years, employee turnover has increased to the point where the manufacturing workforce is constantly changing. The onboarding and training processes are time-consuming and costly. Virtual reality (VR) allows skills to be trained before working on the actual production line and learning in a safe environment. This approach promises cost and productivity benefits for companies and a personalized learning experience for users. In this paper, we present the results of a user evaluation of an industrial VR use case for training the assembly of a compressor. The goal of the evaluation is to compare conventional onboarding training in-person with VR training in terms of learning success, workload, net promotion score, and qualitative feedback. Additionally, user acceptance and usability of the VR training are questioned. The results show that the users still prefer conventional training, as the net promoter score is higher for the in-person training. Also, the results of the learning success show better results for in-person training than the VR training.
SwiftDepth: An Efficient Hybrid CNN-Transformer Model for Self-Supervised Monocular Depth Estimation on Mobile Devices

Self-supervised Monocular Depth Estimation (MDE) models trained solely on single-camera video have gained significant popularity. Recent studies have shown that Vision Transformers (ViT) can improve depth estimation quality, despite their high computational demands. In Extended Reality (XR) context, lightweight and fast models are crucial for seamless operation on mobile devices. This paper proposes SwiftDepth, a hybrid MDE framework that fulfils these requirements. The model combines the benefits of Convolutional Neural Network (CNN), which provides speed and shift invariance, and ViT, which offers a global receptive field. We utilize SwiftFormer, a low-latency feature extraction network with efficient additive attention. Also, we introduce a novel two-level decoder to enhance depth estimation quality without an increase in the number of parameters. Our model achieves comparable results to the state-of-the-art lightweight Lite-Mono on the KITTI outdoor dataset while demonstrating better generalization capability on the NYUv2 indoor dataset.
Cross-Device Augmented Reality Systems for Fire and Rescue based on Thermal Imaging and Live Tracking

Augmented Reality (AR) may enhance safety and operational efficiency during firefighting operations. By leveraging modern head-worn AR equipped with embedded sensors and cross-device collaboration, firefighters gain situational awareness. Cross-reality interaction challenges still exist, in low-visibility, high-temperature, life-threatening conditions. This paper introduces two systems targeting distinct firefighting scenarios employing hands-free gaze-based interaction. The first system named Thermal Imaging System addresses fire and rescue operations when thick smoke is a challenge. By integrating a heat-wavelength infrared camera on the Microsoft HoloLens 2, thermal data are streamed directly to the operator’s field of view, enhancing their visibility. Unlike conventional handheld thermal cameras, this approach allows hands-free operation. The second system named Live Tracking System focuses on wildfire, enabling live tracking of each firefighting team member and displaying their locations on a 3D map, accessible remotely on a computer. An AR headset provides the team captain on the field with real-time visual signals signifying team members’ location on the field and in headquarters. A thorough proof-of-concept evaluation with firefighters indicated that the system enabled cross-device collaboration during fire and rescue operations.
Comparison of Virtual Reality Teleportation Targeting Method Performance depending on the Teleport Distance

Teleportation is a prominent locomotion technique in virtual reality that allows users to move across a wider area without physical constraints. While various teleportation methods have been proposed in previous studies, there is a lack of comparing their performance and determining which method suits specific environmental situations. This study measures and compares the efficiency and accuracy of three targeting methods (hand controller, head-tracking, and eye-tracking) during teleportation. A behavioral experiment involving 16 participants was conducted to collect data on elapsed time and targeting offset. Furthermore, the study analyzes the tendencies of the three targeting methods based on the three distance levels (short, medium, long) to the locomotion target. The experimental results indicate the significant effects of the targeting methods and interval distances, and we also provide teleportation design guidelines based on these findings. In future studies, we will enhance our findings by investigating locomotion usability and efficiency in dynamic environments.
Don’t Pull the Balrog — Lessons Learned from Designing Wizualization: a Magic-inspired Data Analytics System in XR

This paper presents lessons learned in the design and development of Wizualization, a ubiquitous analytics system for authoring visualizations in WebXR using a magic metaphor. The system is based on a fundamentally hybrid and multimodal approach utilizing AR/XR, gestures, sound, and speech to support the mobile setting. Our lessons include how to overcome mostly technical challenges, such as view management and combining multiple sessions in the same analytical 3D space, but also user-based, design-oriented, and even social ones. Our intention in sharing these teachings is to help fellow travellers navigate the same troubled waters we have traversed.
PSA: A Cross-Platform Framework for Situated Analytics in MR and VR

Using immersive environments for training, education, and recreation is becoming increasingly popular among individuals. In recent years these environments have also been used to facilitate data analytics processes involving interaction and task completion. In this work, we discuss Personal Situated Analytics (PSA) a cross-platform framework to embed users into recorded conversations with support for multiple degrees of immersion in the Reality-Virtuality spectrum. Our development pipeline consisted of several stages: tracking, data capturing, data cleaning, data synchronization, prototype building, and deploying the final product to end-user hardware. We evaluate PSA by analyzing data from a recorded meeting involving human participants and a visual conversational AI agent. Our pilot study (n = 12) using this framework compares user experiences for MR and VR environments.
Expanding Targets in Virtual Reality Environments: A Fitts’ Law Study

Target pointing selection is a fundamental task. According to Fitts’ law, users need more time to select targets with smaller sizes. Expanding the target to a larger size is a practical approach that can facilitate pointing selection. It has been well-examined and -deployed in 2D user interfaces. However, limited research has investigated target expansion methods using an immersive virtual reality (VR) head-mounted display (HMD). In this work, we aimed to fill this gap by conducting a user study using ISO 9241-411 multi-directional pointing task to examine the effect of target expansion on target selection performance in VR HMD. Based on our results, we found that compared to not expanding the target, expanding the target width by 1.5 and 2.5 times during the movement can significantly reduce the selection time. We hope that the design and results derived from the study can help frame future work.
Model-aware 3D Eye Gaze from Weak and Few-shot Supervisions

The task of predicting 3D eye gaze from eye images can be performed either by (a) end-to-end learning for image-to-gaze mapping or by (b) fitting a 3D eye model onto images. The former case requires 3D gaze labels, while the latter requires eye semantics or landmarks to facilitate the model fitting. Although obtaining eye semantics and landmarks is relatively easy, fitting an accurate 3D eye model on them remains to be very challenging due to its ill-posed nature in general. On the other hand, obtaining large-scale 3D gaze data is cumbersome due to the required hardware setups and computational demands.In this work, we propose to predict 3D eye gaze from weak supervision of eye semantic segmentation masks and direct supervision of a few 3D gaze vectors. The proposed method combines the best of both worlds by leveraging large amounts of weak annotations–which are easy to obtain, and only a few 3D gaze vectors–which alleviate the difficulty of fitting 3D eye models on the semantic segmentation of eye images. Thus, the eye gaze vectors, used in the model fitting, are directly supervised using the few-shot gaze labels. Additionally, we propose a transformer-based network architecture, that serves as a solid baseline for our improvements. Our experiments in diverse settings illustrate the significant benefits of the proposed method, achieving about 5 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">◦</sup> lower angular gaze error over the baseline, when only 0.05% 3D annotations of the training images are used.
The Impact of Different Virtual Work Environments on Flow, Performance, User Emotions, and Preferences

This research explores how different virtual work environments, differing in the type and amount of elements they include, impact users’ flow, performance, emotional state, and preferences. Pre-study interviews were conducted to inform the design of three VR work environments: the Dark Room, the Empty Room, and the Furnished Room. Fifteen participants took part in a user study where they engaged in a logic-based task simulating deep work while experiencing each environment. The findings suggest that while objective performance measures did not differ significantly, subjective experiences and perceptions varied across the environments. Participants reported feeling less distracted and more focused in the Dark Room and the Empty Room compared to the Furnished Room. The Empty Room was associated with the highest levels of relaxation and calmness, while the Furnished Room was perceived as visually appealing yet more distracting. These findings highlight the variability of user preferences and emphasise the importance of considering user comfort and well-being in the design of virtual work environments. The study contributes to the better understanding of virtual workspaces and provides insights for designing environments that promote flow, productivity, and user well-being.
Augmented Reality for Quality Inspection: A User-Centred Systematic Review of Use Cases, Trends and Technology

Augmented reality (AR)-based quality inspection (QI) has the potential to aid operators in industry by providing additional information and real-time feedback on their actions. To realise the full potential of AR, users must be considered in the design of applications because their perception and behaviour influence the effectiveness of an AR-based quality inspection system. However, how the design of AR experiences affects the performance of QI remains under-explored. To address this research gap, we conducted a systematic literature review, considered 27 use cases, and investigated how QI-related challenges have been addressed from the user’s perspective. We identified six tasks that tend to benefit from AR and mapped common user interface (UI) elements and interaction modalities. Furthermore, we observed concepts that address design challenges, such as visibility and obstructions of physical and digital objects. However, we also found that current systems lack a focus on ergonomics and safety requirements.
The Benefits of Utilizing Augmented Reality as a Tool for Assessments

In means of improving the accessibility of performing home assessments, we have developed a novel set of augmented reality (AR) tools. To test the impact of these tools, a user study was run that compares the developed tools against traditional techniques. The findings show that the AR assessment tools generally provide results that are both more accurate and quicker to obtain compared to traditional methods. In addition, participants rated the AR assessment tools as easier to use and were more confident in their assessments compared to when using traditional measures. These results demonstrate that current generation phone-based AR technologies can be utilized as a powerful tool in the assessment process.
Content Extraction Model of Bilingual Loose-leaf Textbooks for Vocational Minority Students Based on Neural Networks

The semantic feature vectors of teaching texts were determined for information extraction using the characteristics of bilingual loose-leaf textbooks for ethnic minorities in vocational colleges. The corpus of bilingual loose-leaf teaching was trained in neural networks to extract template semantic feature vectors independently. Then, the required data was extracted in task modules. In this method, lexical analysis was not necessary owing to its relatively fine information processing and granularity. The algorithm proposed in this study showed an improved accuracy upto 30.5% compared to those of traditional algorithms. It also had the adaptability to standardize language semantic elements and structures.
Rapid Prototyping of Distributed Musical Things using Web Technologies

This short paper reports on recent advances in the development of an open-source prototyping platform for the creation of distributed and embedded musical systems based on Web technologies. The proposed architecture is based on user-grade hardware and open-source software and aims at fostering rapid-prototyping and experimentation in the context of colocated distributed systems for musical research, creation and performance. After a short review of related works, the paper describes the general design and the different building blocks of the system. Then, it exposes a first characterization of the proposed design, and concludes with the description of a prototype that highlights different features of the platform.
Research on Zhuang's Speech-to-Text and Translation Cloud Platform

The Zhuang people distribute in Guangxi province maintaining their language without characters. To convert the speech of the Zhuang into Chinese characters and translate it into other languages, we built a database to construct an end-to-end automatic speech recognition using a neural network SaformerNet. The network used a transformer-based deep learning model for the translation. An edge computing-based private cloud system using FPGA acceleration was proposed for the rapid response of the network. The results demonstrated that the proposed solution was suitable for small-scale conversion of natural languages in a real-time application.
Unique Feature Extraction and Consistency Network for Skeleton Body Keypoints Configuration and Enhancement

The machine learning algorithm recognizes skeleton poses in many ways but previous methods suffered from higher computation and low latency. To increase solitary person posture accurateness, we developed a novel strategy for fitness-related appliances. The developed design included a basic network that offered a starting point for eminent modification using a Unique Features Extraction Network (UFE Net) for long-term consistency. The UFE Net imposed innovative constraints through local element changes and enhancement of global body properties. The recommended module decreased the effect of varying body key points and inconsistent postures. Using a standard dataset, we showed how effective the suggested approach was. The proposed framework operated rapidly and increased precision on the CPU environment platform. In comparison to previous results, the developed design enhanced accuracy by 98.5% at real-time proceeding rates of 32 FPS.
Designing Situated Dashboards: Challenges and Opportunities

Situated Visualization is an emerging field that unites several areas - visualization, augmented reality, human-computer interaction, and internet-of-things, to support human data activities within the ubiquitous world. Likewise, dashboards are broadly used to simplify complex data through multiple views. However, dashboards are only adapted for desktop settings, and requires visual strategies to support situatedness. We propose the concept of AR-based situated dashboards and present design considerations and challenges developed over interviews with experts. These challenges aim to propose directions and opportunities for facilitating the effective designing and authoring of situated dashboards.