Reviewerly/exHarmony huggingface dataset:
Dynascape : Immersive Authoring of Real-World Dynamic Scenes with Spatially Tracked RGB-D Videos

In this paper, we present Dynascape, an immersive approach to the composition and playback of dynamic real-world scenes in mixed and virtual reality. We use spatially tracked RGB-D cameras to capture point cloud representations of arbitrary dynamic real-world scenes. Dynascape provides a suite of tools for spatial and temporal editing and composition of such scenes, as well as fine control over their visual appearance. We also explore strategies for spatiotemporal navigation and different tools for the in situ authoring and viewing of mixed and virtual reality scenes. Dynascape is intended as a research platform for exploring the creative potential of dynamic point clouds captured with mobile, tracked RGB-D cameras. We believe our work represents a first attempt to author and playback spatially tracked RGB-D video in an immersive environment, and opens up new possibilities for involving dynamic 3D scenes in virtual space.
Ready Worker One? High-Res VR for the Home Office

Many employees prefer to work from home, yet struggle to squeeze their office into an already fully-utilized space. Virtual Reality (VR) seemingly offered a solution with its ability to transform even modest physical spaces into spacious, productive virtual offices, but hardware challenges—such as low resolution—have prevented this from becoming a reality. Now that hardware issues are being overcome, we are able to investigate the suitability of VR for daily work. To do so, we (1) studied the physical space that users typically dedicate to home offices and (2) conducted an exploratory study of users working in VR for one week. For (1) we used digital ethnography to study 430 self-published images of software developer workstations in the home, confirming that developers faced myriad space challenges. We used speculative design to re-envision these as VR workstations, eliminating many challenges. For (2) we asked 10 developers to work in their own home using VR for about two hours each day for four workdays, and then interviewed them. We found that working in VR improved focus and made mundane tasks more enjoyable. While some subjects reported issues—annoyances with the fit, weight, and umbilical cord of the headset—the vast majority of these issues seem to be addressable. Together, these studies show VR technology has the potential to address many key problems with home workstations, and, with continued improvements, may become an integral part of creating an effective workstation in the home.
Exploring User Engagement in Immersive Virtual Reality Games through Multimodal Body Movements

User engagement in Virtual Reality (VR) games is crucial for creating immersive and captivating gaming experiences that meet the expectations of players. However, understanding and measuring these levels in VR games presents a challenge for game designers, as current methods, such as self-reports, may be limited in capturing the full extent of user engagement. Additionally, approaches based on biological signals to measure engagement in VR games present complications and challenges, including signal complexity, interpretation difficulties, and ethical concerns. This study explores body movements, as a novel approach to measure user engagement in VR gaming. We employ E4, emteqPRO, and off-the-shelf IMUs to measure the body movements from diverse participants engaged in multiple VR games. Further, we examine the simultaneous occurrence of player motivation and physiological responses to explore potential associations with body movements. Our findings suggest that body movements hold promise as a reliable and objective indicator of user engagement, offering game designers valuable insights on generating more engaging and immersive experiences.
Discrete Diffusion for Co-Speech Gesture Synthesis

In this paper, we describe the gesture synthesis system we developed for our entry to the GENEA Challenge 2023. One challenge in learning the co-speech gesture model is that there may be multiple viable gesture motions for the same speech utterance. Therefore compared to a deterministic regression model, a probabilistic model will be preferred to handle the one-to-many mapping problem. Our system utilizes the vector-quantized variational autoencoder (VQ-VAE) and discrete diffusion as the framework for predicting co-speech gestures. Since the gesture motions are produced via sampling the discrete gesture tokens using the discrete diffusion process, the method is able to produce diverse gestures given the same speech input. Based on the user evaluation results, we further discuss about the strength and limitations of our system, and provide the lessons learned when developing and tuning the system. The subjective evaluation results show that our method ranks in the middle for human-likeness among all submitted entries. In the the speech appropriateness evaluations, our method has preferences of 55.4% for matched agent gesture and 51.1% for matched interlocutor gestures. Overall, we demonstrated the potential of discrete diffusion models in gesture generation.
Redirected Placement: Evaluating the Redirection of Passive Props during Reach-to-Place in Virtual Reality

Hand redirection is an effective technique that can provide users with haptic feedback in virtual reality (VR) when a disparity exists between virtual objects and their physical counterparts. Psychophysiological research has revealed the distinct motion profiles of different kinematic phases when people operate hand-object interaction. In this paper, we proposed the Redirected Placement (RP), which determines the new placement of a physical prop using a constrained optimization problem. The visual illusion is used during the "reach-to-place" kinematic phase in the proposed RP method rather than the "reach-to-grasp" phase in the typical Redirected Reach (RR) method. We conducted two experiments based on the proposed RP method. Our first experiment showed that detection thresholds are generally higher with the proposed method compared to the RR method. The second experiment evaluated the embodiment experience with hand redirection using RR-only, RP-only, and RR&RP methods. The results report an enhanced sense of embodiment with the combined use of both RR and RP techniques. Our study further indicates that a 1:1 combination ratio of RR&RP resulted in the closest subjective experience to the baseline.
The Detectability of Saccadic Hand Offset in Virtual Reality

On the way towards novel hand redirection (HR) techniques that make use of change blindness, the next step is to take advantage of saccades for hiding body warping. A prerequisite for saccadic HR algorithms, however, is to know how much the user’s virtual hand can unnoticeably be offset during saccadic suppression. We contribute this knowledge by conducting a psychophysical experiment, which lays the ground for upcoming HR techniques by exploring the perceptual detection thresholds (DTs) of hand offset injected during saccades. Our findings highlight the pivotal role of saccade direction for unnoticeable hand jumps, and reveal that most offset goes unnoticed when the saccade and hand move in opposite directions. Based on the gathered perceptual data, we derived a model that considers the angle between saccade and hand offset direction to predict the DTs of saccadic hand jumps.
Leveraging gaze for potential error prediction in AI-support systems: An exploratory analysis of interaction with a simulated robot

With the increasing prevalence of AI–driven systems that assist humans in daily tasks, ensuring safety and fostering trust is pivotal. Moreover, addressing system errors is important, as conventional methods may be inaccessible and time consuming for people with motor impairments. This study explores implicit communication using eye-tracking features. We conducted a Virtual Reality (VR) experiment where participants used a robotic arm, encountering errors to be prevented through explicit communication. Findings reveal significant differences in eye-tracking features based on error recognition and demonstrate automated error detection feasibility. Using machine learning, we were able to classify when a user intends to stop the system before a potential collision using the eye features (86 – 90% accuracy). This research contributes to understanding eye movement behavior for error detection and offers insights for implementing implicit communication in AI –- driven assistive systems.
Pigments of Imagination: An Interactive Virtual Reality Composition

Pigments of Imagination is an artistic interactive virtual reality experience based on an original fixed media composition. It is designed to reimagine the popular music video in a virtual space as a dynamic, emotionally engaging experience through exploration of novel approaches toward audiovisual reactivity and interactivity. In this piece the user can interact, directly affect, and build upon prior user interpolations of the environment’s sonic and visual qualities, allowing a narrative immersion that maintains a structured arc and conclusion but unique experience with each use.
Enforcing Information System Security

Every year brings numerous security breaches that lead to highly destructive ransomware attacks, data leaks, and reputational damage to governments, companies, and other organizations around the world. As a result, there is a growing need to ensure that workers comply with critical policies put in place to avoid such incidents. This study investigated how factors from social bond theory and involvement theory affected compliance with information security policies and procedures. All of the factors examined were found to have a significant influence on attitudes about compliance, and attitude had a significant impact on intention to comply. The findings of this study revealed that it is vital to raise employees' awareness about compliance with security policies by improving their information security behavior. Moreover, all the factors were found to have a significant influence on the attitude of employees towards compliance with their organizational information security policies and procedures.
Revisiting Consumed Endurance: A NICE Way to Quantify Shoulder Fatigue in Virtual Reality

Virtual Reality (VR) is increasingly being adopted in fitness, gaming, and workplace productivity applications for its natural interaction with body movement. A widely accepted method for quantifying the physical fatigue caused by VR interactions is through metrics such as Consumed Endurance (CE). Proposed in 2014, CE calculates the shoulder torque to infer endurance time (ET)—i.e. the maximum amount of time a pose can be maintained—during mid-air interactions. This model remains widely cited but has not been closely examined beyond its initial evaluation, leaving untested assumptions about exertion from low-intensity interactions and its basis on torque. In this paper, we present two VR studies where we (1) collect a baseline dataset that replicates the foundation of CE and (2) extend the initial evaluation in a pointing task from a two-dimensional (2D) screen to a three-dimensional (3D) immersive environment. Our baseline dataset collected from a high-precision tracking system found that the CE model overestimates ET for low-exertion interactions. Further, our studies reveal that a biomechanical model based on only torque cannot account for additional exertion measured when the shoulder angle exceeds 90° elevation. Based on these findings, we propose a revised formulation of CE to highlight the need for a hybrid approach in future fatigue modelling.
eXtended Reality of socio-motor interactions: Current Trends and Ethical Considerations for Mixed Reality Environments Design

Social interactions are multi-modal, and their translation into virtuous and smooth interactions in digital, hybrid reality spaces constitute a technological challenge with profound but often dismissed ethical considerations (such as harmful consequences). In particular, the increasing reliance on algorithms (i.e., artificial intelligence) to emulate “seamless” communication patterns between human and artificial agents calls for caution in designing such systems. This short paper reviews current trends in rendering hyper-realistic human bodies and motor behaviors, focusing on the ethical issues of manipulating kinematics in human-to-human and human-to-artificial agent interactions.
Towards Effective Automatic Evaluation of Generated Reflections for Motivational Interviewing

Reflection is an essential counselling skill where the therapist communicates their understanding of the client’s words to the client. Recent studies have explored language-model-based reflection generation, but automatic quality evaluation of generated reflections remains under-explored. In this work, we investigate automatic evaluation on one fundamental quality aspect: coherence and context-consistency. We test a range of automatic evaluators/metrics and examine their correlations with expert judgement. We find that large language models (LLMs) as zero-shot evaluators achieve the best performance, while other metrics correlate poorly with expert judgement. We also demonstrate that diverse LLM-as-evaluator configurations need to be explored to find the best setup.
Gaze Assistance for Older Adults during Throwing in Virtual Reality and its Effects on Performance and Motivation

Initial motivation when starting exergaming is a key factor towards enabling long-term engagement and adherence, especially among older adults. To increase, in particular, the initial motivation of older adults, we introduce the concept of diminishing gaze assistance (GA), assess its feasibility for virtual reality (VR) exergames, and investigate the effects on motor learning, performance, and motivation in older adult users. First, we conducted a focus group followed by a pre-study on the development of VR exergames for older adults and VR gaze assistance. The results informed the design and implementation of our gaze-assisted throwing exergame, which was then evaluated in a follow-up main study. Participants of the main study were randomly assigned to the GA and Motor (control) group, and had to complete a VR throwing task, in which participants had to aim and throw at three targets at varying angles. The GA group received declining gaze assistance, in which the ball trajectory was initially guided by their gaze (rather than their physical (motor) throwing) before guidance was gradually reduced until their physical (motor) throwing ability was solely responsible for hitting the target. Motivation and user experience were assessed using the Questionnaire on Current Motivation before and during, and the short scale of intrinsic motivation questionnaire after the task. The results show that the GA was generally perceived positively. In particular, the initial confidence of the GA group was rated higher, and we observed evidence suggesting increased confidence throughout the trial.
ActioNet: A Lightweight Architecture for Efficient Action Recognition

We present ActioNet, a groundbreaking lightweight neural network architecture optimized for action recognition tasks, particularly in resource-constrained environments such as drones and edge devices. Utilizing a strategically modified 3D U-Net encoder followed by fully connected layers for fine-grained classification, ActioNet manages to achieve a promising validation accuracy of 72%. This is accomplished with a notably compact model size of just 46MB, making it uniquely suitable for devices with limited computational capabilities. Although ActioNet may not surpass state-of-the-art models in terms of sheer accuracy, it distinguishes itself through its fast inference times and small footprint. These attributes make real-time action recognition not only feasible but also efficient in constrained operational settings. We argue that ActioNet serves as a meaningful contribution to the emerging field of efficient deep learning and provides a solid foundation for future advancements in lightweight action recognition models.
Insights Into the Importance of Linguistic Textual Features on the Persuasiveness of Public Speaking

There is a growing need for public speaking skills in both professional and private life. With this background, our research project’s long-term aims are to develop tools that can analyse public speeches and provide helpful feedback. The impact of audio and visual characteristics on the automatic analysis of speech quality has been widely explored in the existing literature. However, only a few studies have focused on textual features. In response to this shortcoming, this paper investigates the importance of textual content for the automatic analysis of public speaking. We created an open-source Python library of textual features and integrated them as inputs of simple machine-learning models for automatic public-speaking analysis, and persuasiveness prediction, in particular. The best result (accuracy of ) is obtained using a logistic regression. We then evaluated the impact of these features on persuasiveness prediction using both correlation analysis and Explainable AI methods. This evaluation was conducted on the French data set 3MT_French, including student performances in the "Ma Thèse en 180 Secondes" competition.
UniteXR: Joint Exploration of a Real-World Museum and its Digital Twin

The combination of smartphone Augmented Reality (AR) and Virtual Reality (VR) makes it possible for on-site and remote users to simultaneously explore a physical space and its digital twin through an asymmetric Collaborative Virtual Environment (CVE). In this paper, we investigate two spatial awareness visualizations to enable joint exploration of a space for dyads consisting of a smartphone AR user and a head-mounted display VR user. Our study revealed that both, a mini-map-based method and an egocentric compass method with a path visualization, enabled the on-site visitors to locate and follow a virtual companion reliably and quickly. Furthermore, the embodiment of the AR user by an inverse kinematics avatar allowed the use of natural gestures such as pointing and waving which was preferred over text messages by the participants of our study. In an expert review in a museum and its digital twin we observed an overall high social presence for on-site AR and remote VR visitors and found that the visualizations and the avatar embodiment successfully facilitated their communication and collaboration.
Do Body Expressions Leave Good Impressions? - Predicting Investment Decisions based on Pitcher's Body Expressions

This work aims to investigate whether displayed body expressions during venture pitches hold predictive value over the success of gathering funding. To this end the performance of several traditional regression models trained on aggregated body expression features is compared to that of deep recurrent models. Moreover, the impact of features and the effect of oversampling to counteract data imbalance are investigated. The results showed that temporal modeling of body expressions with GRU yielded the best performance (average MAE = 16.9). Oversampling was found to not improve models’ performance. The analysis of the impact of features suggest that higher levels of expressed body movements negatively affect investment scores whereas a more open body posture can have a positive effect. These findings show that body expressions can be predictive of investment decisions. Future work will be directed towards integrating these cues into a multimodal model to aid entrepreneurs in improving their pitching skills and further the understanding of the investment decision-making process.
Hands-on DNA: Exploring the Impact of Virtual Reality on Teaching DNA Structure and Function

Molecular biology is a demanding subject, requiring students to master abstract, three-dimensional (3D) concepts across a range of spatial scales. Virtual reality (VR) is a medium that excels at portraying scale and 3D concepts, and allows people to have tangible experiences of otherwise intangible subjects. This paper describes Hands-on DNA, a virtual reality learning experience for teaching undergraduate university students about the scale and structure of deoxyribose nucleic acid (DNA), a central molecule in molecular biology. The intention of Hands-on DNA is to leverage the advantages of virtual reality against specific challenges faced in teaching molecular biology. We derive design requirements motivated by pedagogy, provide guidelines, and discuss lessons learned during development. Our user study shows that students perceive Hands-on DNA as a fun, engaging, effective learning tool, and that it addresses some of the weaknesses in molecular biology education. Our results also suggest that new interaction techniques to support learning in VR need to be developed (e.g., for note taking) and that the increasing penetration of recreational VR increases students’ expectations and hence the risk of students being disappointed of VR learning tools.
Effect of voice imitation using voice conversion by avatar on customer service in Virtual Environments

We investigate the impact of voice imitation on rapport building in customer service in a virtual environment(VE). We simulated a VE store and conducted a within-subject experiment in three customer service scenarios with 16 participants. The imitation condition used a voice generated by a machine learning model to reproduce the participant’s linguistic content, with the voice identity set midway between the participant and the salesperson. In a group of men, voice imitation significantly improved their impression of salespeople. The findings of this study can be used to design interpersonal services in VE with the help of salesperson avatar voice design.
Exploring Real-time Precision Feedback for AR-assisted Manual Adjustment in Mechanical Assembly

Augmented Reality (AR) based manual assembly nowadays enables to guide the process of physical tasks, providing intuitive instructions and detailed information in real-time. However, very limited studies have explored AR manual adjustment tasks with precision requirements. In this paper, we develop an AR-assisted guidance system for manual adjustments with relatively high-precision requirements. We first assessed the accuracy of the special-set OptiTrack system to determine the threshold of precision requirements for our user study. We further evaluated the performance of Number-based and Bar-based precision feedback by comparing orienting assembly errors and task completion time, as well as the usability in the user study. We found that the assembly errors of orientation in the Number-based and Bar-based interfaces were significantly lower than the baseline condition, while there was no significant difference between the Number-based and Bar-based interfaces. Furthermore, the Number-based showed faster task completion time, lower workload, and higher usability than the Bar-based condition.
Waddle: using virtual penguin embodiment as a vehicle for empathy and informal learning

This paper presents, Waddle, a virtual experience to promote informal learning by embodying the user as an Adélie Penguin to partake in a narrative-based virtual reality application that shares the story of the lives of these unique animals. We test the effects of this experience on informal learning and empathy, an important component for fostering social engagement with ecology. The research demonstrates that the developed experience is able to support informal learning, virtual embodiment, and is able to create a positive change in empathy.