Reviewerly/exHarmony huggingface dataset:
How Visualising Emotions Affects Interpersonal Trust and Task Collaboration in a Shared Virtual Space

Emotion is dynamic. Changes in emotion can be hard to process during face-to-face interaction, yet transferring them into a shared virtual space becomes more challenging. This research first explores nine visual representations to amplify emotions in a virtual space, leading to a bi-directional emotion-sharing system (FeelMoji i/o). The second study investigates the effect of explicit emotion-sharing in interpersonal trust and task collaboration through three conditions - verbal only, verbal+positive visual, and verbal+honest visual using FeelMoji through the proposal of a framework of four factors (usability, integrity, behaviour, and collaboration). The results indicate that FeelMoji yields frequent emotion consensus as task milestones and positive interdependent behaviours between collaborators, which help develop conversations, affirm decision-making, and build familiarity and trust between strangers. Moreover, we discuss how our study can inspire future investigation in human-AI agent behaviours and large-scale multi-user virtual environments.
Perception and Proxemics with Virtual Humans on Transparent Display Installations in Augmented Reality

It is not uncommon for science fiction movies to portray futuristic user interfaces that can only be realized decades later with state-of-the-art technology. In this work, we present a prototypical augmented reality (AR) installation that was inspired by the movie The Time Machine (2002). It consists of a transparent screen that acts as a window through which users can see the stereoscopic projection of a three-dimensional virtual human (VH). However, there are some key differences between the vision of this technology and the way VHs on these displays are actually perceived. In particular, the additive light model of these displays causes darker VHs to appear more transparent, while light in the physical environment further increases transparency, which may affect the way VHs are perceived, to what degree they are trusted, and the distances one maintains from them in a spatial setting. In this paper, we present a user study in which we investigate how transparency in the scope of transparent AR screens affects the perception of a VH’s appearance, social presence with the VH, and the social space around users as defined by proxemics theory. Our results indicate that appearances are comparatively robust to transparency, while social presence improves in darker physical environments, and proxemic distances to the VH largely depend on one’s distance from the screen but are not noticeably affected by transparency. Overall, our results suggest that such transparent AR screens can be an effective technology for facilitating social interactions between users and VHs in a shared physical space.
Into the Portraits: Face Swapping System for Art Gallery Visitors

To provide a unique and personalized experience for visitors at the art gallery, we have developed a face swapping system that seamlessly replaces the original face in a painting with the face of visitor. This system aims to create a sense of immersion, making the visitor feel as if the artist had originally depicted their portrait and they had become part of the artwork itself. In order to achieve this, we constructed a dataset of portrait paintings, including manual annotation of facial landmarks for a selected subset, which serves as the target images for our system. We developed a photo- to-paint face swapping method by building upon the FaceShifter and fine-tuning it using the constructed dataset. To ensure the high-quality results, we have incorporated several post-processing steps into the pipeline. To make this system accessible to gallery visitors, we implemented a photo kiosk based on the developed method. This kiosk has been successfully installed at Théâtre des Lumières, a digital art center located in Seoul, South Korea. Since its installation, the kiosk has gained significant popularity, with around 20,000 visitors utilizing it and generating a substantial number of 31,820 images. This high level of visitor engagement demonstrates the widespread appeal and popularity of the system.
Realistic Fusion of Virtual Materials and Hands for AR through Intrinsic Decomposition

Intrinsic decomposition is an inherent issue in computer vision and graphics, which usually decomposes the image into reflectance and shading. Reflectance and shading can be applied to achieve virtual and real information fusion with illumination consistency in augmented reality, but this demands higher performance and speed of the intrinsic decomposition method. In this study, we propose a novel neural network architecture for intrinsic decomposition that enables real-time generation of high-quality reflectance and shading. Furthermore, we extend our framework by integrating the hands segmentation network, resulting in an augmented reality framework capable of seamlessly fusing virtual materials and hands. Through comprehensive experiments, we demonstrate the effectiveness of our framework in achieving realistic fusion of multiple virtual materials and hands, even in the presence of dynamic gestures and diverse illumination conditions. We believe that our study will serve as an inspiration for the development of more creative applications in fields such as art, film, and advertising.
Training for Open-Ended Drilling through a Virtual Reality Simulation

Virtual Reality (VR) can support effective and scalable training of psychomotor skills in manufacturing. However, many industry training modules offer experiences that are close-ended and do not allow for human error. We aim to address this gap in VR training tools for psychomotor skills training by exploring an open-ended approach to the system design. We designed a VR training simulation prototype to perform open-ended practice of drilling using a 3-axis milling machine. The simulation employs near “endto-end” instruction through a safety module, a setup and drilling tutorial, open-ended practice complete with warnings of mistakes and failures, and a function to assess the geometries and locations of drilled holes against an engineering drawing. We developed and conducted a user study within an undergraduate-level introductory fabrication course to investigate the impact of open-ended VR practice on learning outcomes. Study results reveal positive trends, with the VR group successfully completing the machining task of drilling at a higher rate (75% vs 64%), with fewer mistakes (1.75 vs 2.14 score), and in less time (17.67 mins vs 21.57 mins) compared to the control group. We discuss our findings and limitations and implications for the design of open-ended VR training systems for learning psychomotor skills.
What And How Together: A Taxonomy On 30 Years Of Collaborative Human-Centered XR Tasks

We present a taxonomy of human-centered collaborative XR tasks. XR technologies have extended into the realm of collaboration, improving the quality and accessibility of teamwork. However, after a comprehensive assessment of the literature on the interaction between XR technologies and collaboration, no comprehensive method that emphasizes task actions and properties exists to classify collaborative tasks. Thus, our suggested taxonomy represents a classification system for collaborative tasks. After conducting a thorough literature review across different research venues, we conducted several exhaustive classification and review cycles for over 800 papers collected, which resulted in 148 papers retained to create the taxonomy. We dissected the actions and properties that the collaborative endeavors and tasks of these papers encompass as well as the types of categorizations and relations these papers illustrate. We expand on the design choices and usage of our taxonomy, followed by its limitations and future work. We built this taxonomy in order to reduce ambiguities and confusion regarding the design and comprehension of human-based collaborative tasks that use XR technology, which could prove useful in aiding the development and understanding of these tasks. Our taxonomy reveals a framework for understanding how collaborative tasks are designed and a systematic way of classifying different methods by which people can collaborate and interact in environments that involve XR, while still promoting efficient communication, teamwork, goal achievement and productivity.
Manifest: Public Speaking Training Using Virtual Reality

Public speaking is pivotal in personal and professional realms, evaluating communication skills. Employing VR training offers realistic practice, identifying weaknesses, improving speaking, and fostering confidence to overcome the fear of public speaking. This paper introduces a novel VR-based public speaking training system that comprehensively analyzes the speaker’s speech, body language, and nervousness levels. The system requires the use of a VR headset that immerses the user in a virtual classroom setting where they deliver speeches, while simultaneously recording the speech, capturing video through an external camera, and monitoring the user’s heart rate using a dedicated sensor. To evaluate the speech quality, the recorded speech is processed to calculate the scores of speaking rate, clarity, listenability, pronunciation, and number of pauses during the speech. The recorded video is further utilized to analyze body language, employing a pose estimation framework-YOLOv7. Furthermore, heart sensor data is employed to calculate the user’s level of nervousness. The system provides detailed analysis for each training session, providing users with performance feedback that can be accessed through the website. By integrating immersive VR technology, speech analysis, body language assessment, and heart rate monitoring, this novel public speaking training application provides a comprehensive way to evaluate public speaking skills and receive feedback and thus is a viable solution for coaching in the field of public speaking.
Exploring Trajectory Data in Augmented Reality: A Comparative Study of Interaction Modalities

The visual exploration of trajectory data is crucial in domains such as animal behavior, molecular dynamics, and transportation. With the emergence of immersive technology, trajectory data, which is often inherently three-dimensional, can be analyzed in stereoscopic 3D, providing new opportunities for perception, engagement, and understanding. However, the interaction with the presented data remains a key challenge. While most applications depend on hand tracking, we see eye tracking as a promising yet under-explored interaction modality, while challenges such as imprecision or inadvertently triggered actions need to be addressed. In this work, we explore the potential of eye gaze interaction for the visual exploration of trajectory data within an AR environment. We integrate hand- and eye-based interaction techniques specifically designed for three common use cases and address known eye tracking challenges. We refine our techniques and setup based on a pilot user study (n=6) and find in a follow-up study (n=20) that gaze interaction can compete with hand-tracked interaction regarding effectiveness, efficiency, and task load for selection and cluster exploration tasks. However, time step analysis comes with higher answer times and task load. In general, we find the results and preferences to be user-dependent. Our work contributes to the field of immersive data exploration, underscoring the need for continued research on eye tracking interaction.
Chenrezig Sadhana VR: A Novel Arousal-Based Virtual Reality Neurofeedback Meditation System

This paper introduces a novel arousal-based virtual reality (VR) neurofeedback meditation system based on Buddhist visualization techniques, which generates meditation imagery based on the users state of physiological arousal. Therapists have increasingly incorporated mindfulness-based VR meditation systems into their interventions to treat a wide range of psychological conditions, however the effectiveness of mindfulness based VR meditation systems is still unproven. We propose that visualization based meditation presented in VR will have greater efficacy for the improvement of psychological well-being than mindfulness based VR meditation. We base this on the hypothesis that the visual affordances of VR and the arousal of the autonomic nervous system it produces through the phenomenon of virtual presence, links the technology directly with visualization based meditation techniques found in Buddhism which also induce arousal in the body. We have designed a pilot study where participants practice a Buddhist visualization meditation technique using our arousal-based VR neurofeedback system, while a control group will experience a VR mindfulness system. We hypothesize that the visualization based VR experience will have greater efficacy for improving psychological well-being than the mindfulness based systems.
Cloud-Based Face Recognition for Augmented Reality Glasses

Traditional face recognition systems are built to be deployed within a geographically constrained environment using a combination of cameras, processing units, and display monitors. However, this can limit the mobility and operational area of these systems. A wireless system with wearable display offers a potential solution to this problem. To address this issue, we propose a cloud-based face recognition system with augmented reality (AR) glasses. The system consists of three main components: AR glasses to capture images of a scene and display recognition results, mobile phone to operate the AR glasses and communicate with the server via Internet connection, and server to run a deep learning model to perform face recognition on images received. Our system allows users to connect to our server through the Internet, such as cellular networks, as well as public and local WiFi networks. Additionally, experimental results indicate that our system can simultaneously recognize up to two individuals while maintaining an average frame rate of approximately 14 frames per second, sufficient for enabling users to move around and constantly identify individuals in a scene.
DM-VTON: Distilled Mobile Real-time Virtual Try-On

The fashion e-commerce industry has witnessed significant growth in recent years, prompting exploring image-based virtual try-on techniques to incorporate Augmented Reality (AR) experiences into online shopping platforms. However, existing research has primarily overlooked a crucial aspect - the runtime of the underlying machine-learning model. While existing methods prioritize enhancing output quality, they often disregard the execution time, which restricts their applications on a limited range of devices. To address this gap, we propose Distilled Mobile Real-time Virtual Try-On (DM-VTON), a novel virtual try-on framework designed to achieve simplicity and efficiency. Our approach is based on a knowledge distillation scheme that leverages a strong Teacher network as supervision to guide a Student network without relying on human parsing. Notably, we introduce an efficient Mobile Generative Module within the Student network, significantly reducing the runtime while ensuring high-quality output. Additionally, we propose Virtual Try-on-guided Pose for Data Synthesis to address the limited pose variation observed in training images. Experimental results show that the proposed method can achieve 40 frames per second on a single Nvidia Tesla T4 GPU and only take up 37 MB of memory while producing almost the same output quality as other state-of-the-art methods. DM-VTON stands poised to facilitate the advancement of real-time AR applications, in addition to the generation of lifelike attired human figures tailored for diverse specialized training tasks.
Reality Distortion Room: A Study of User Locomotion Responses to Spatial Augmented Reality Effects

Reality Distortion Room (RDR) is a proof-of-concept augmented reality system using projection mapping and unencumbered interaction with the Microsoft RoomAlive system to study a user’s locomotive response to visual effects that seemingly transform the physical room the user is in. This study presents five effects that augment the appearance of a physical room to subtly encourage user motion. Our experiment demonstrates users’ reactions to the different distortion and augmentation effects in a standard living room, with the distortion effects projected as wall grids, furniture holograms, and small particles in the air. The augmented living room can give the impression of becoming elongated, wrapped, shifted, elevated, and enlarged. The study results support the implementation of AR experiences in limited physical spaces by providing an initial understanding of how users can be subtly encouraged to move throughout a room.
Evaluating 3D User Interaction Techniques on Spatial Working Memory for 3D Scatter Plot Exploration in Immersive Analytics

This work evaluates three 3D user interaction techniques to investigate their visuo-spatial working memory support for users’ data exploration in immersive analytics. Two techniques are the common VR locomotion technique, Walking and Teleportation, while the other one is Grab, an object manipulation technique. We present two formal user studies in VR and AR. Our study is designed based on the Corsi block-tapping Task, a psychological test for assessing visuo-spatial working memory. Our study results show that Walking supports spatial memory best, and Grab follows. Though Teleportation is found to support it the least, participants rated Teleportation as the easiest way to move in the VR study. We also compare the Walking and Grab results in the VR and AR studies and discuss differences. At last, we discuss our limitations and future work.
Towards Eyeglasses Refraction in Appearance-based Gaze Estimation

For myopia and hyperopia subjects, eyeglasses would change the position of objects in their views, leading to different eyeball rotations for the same gaze target (Fig. 1). Existing appearance-based gaze estimation methods ignore this effect, while this paper investigates it and proposes an effective method to consider it in gaze estimation, achieving noticeable improvements. Specifically, we discover that the appearance-gaze mapping differs for spectacled and unspectacled conditions, and the deviations are nearly consistent with the physical laws of the ideal lens. Based on this discovery, we propose a novel multi-task training strategy that encourages networks to regress gaze and classify the wearing conditions simultaneously. We apply the proposed strategy to some popular methods, including supervised and unsupervised ones, and evaluate them on different datasets with various backbones. The results show that the multi-task training strategy could be used on the existing methods to improve the performance of gaze estimation. To the best of our knowledge, we are the first to clearly reveal and explicitly consider eyeglasses refraction in appearance-based gaze estimation. Data and code are available at https://github.com/StoryMY/RefractionGaze.
HOCTOPUS: An Open-Source Cross-Reality tool to Augment Live-Streaming Remote Classes

Extended Reality (XR) has gained significant attention, offering novel pathways for learning based on the sharing of 3D content. Nevertheless, additional efforts should be spent to understand the advantages set forth by Cross-Reality (CR) scenarios, for example including Augmented Reality (AR) and Mixed Reality (MR) experiences in the context of remote live-streaming classes. In this process, a necessary step entails designing, implementing, and making educational tools available in an open-source format. To this aim, we present HOCTOPUS, a CR system designed to support live remote teaching. HOCTOPUS provides a cross-reality educational experience: an MR application for teachers and an AR mobile app for students. HOCTOPUS lets teachers visualize, manipulate, and share 3D objects in a live-streaming fashion, supporting bidirectional communication and interactions with all students. The MR component is developed for the Hololens 2, offering flexible manipulation capabilities, while the AR component runs on Android mobile devices, providing affordable interactive visualization and manipulation. The proposed tool may be used as a building block to enhance the learning process and is made available to the educational community for its use, assessment, and extension.
Feel the Breeze: Promoting Relaxation in Virtual Reality using Mid-Air Haptics

Mid-air haptic interfaces employ focused ultrasound waves to generate touchless haptic sensations on the skin. Prior studies have demonstrated the potential positive impact of mid-air haptic feedback on virtual experiences, enhancing aspects such as enjoyment, immersion, and sense of agency. As a highly immersive environment, Virtual Reality (VR) is being explored as a tool for stress management and relaxation in current research. However, the impact of incorporating mid-air haptic stimuli into relaxing experiences in VR has not been studied thus far. In this paper, for the first time, we design a mid-air haptic stimulation that is congruent with a relaxing scene in VR, and conduct a user study investigating the effectiveness of this experience. Our user study encompasses three different conditions: a control group with no relaxation intervention, a VR-only relaxation experience, and a VR+Haptics relaxation experience that includes the mid-air haptic feedback. While we did not find any significant differences between the conditions, a trend suggesting that the VR+Haptics condition might be associated with greater pleasure emerged, requiring further validation with a larger sample size. These initial findings set the foundation for future investigations into leveraging multimodal interventions in VR, utilising mid-air haptics to potentially enhance relaxation experiences.
ARPuzzle: Evaluating the Effectiveness of Collaborative Augmented Reality

Collaborative Augmented Reality (CAR) offers disruptive ways for people to collaborate. However, this emerging technology must improve its acceptance, efficiency, and usability to scale up and, for example, support augmented operations executed by technicians. This paper presents our CAR system and its experimentation during a cooperative puzzle-solving task. Our system provides collaborators with a shared virtual space allowing verbal and non-verbal interpersonal communications, and intuitive interactions with shared virtual replicas of real objects. Our system also integrates avatars embodied by remote users. We conducted a dual-user study comparing colocated and remote solving of a puzzle virtual replica with its real solving. We evaluated task performance, collaboration, mutual awareness, spatial presence, and copresence, usability, and preference. We found that, if real is preferred and more efficient than our CAR system, CAR is reaching favorable usability levels. We also found that remote augmented reality including full-body avatars offers similar results to colocated augmented reality. This preliminary work paves the way for future research aiming to support and enhance the design and making of Collaborative Augmented Reality systems dedicated to augmented operations.