Reviewerly/exHarmony huggingface dataset:
Comparing Visual Attention with Leading and Following Virtual Agents in a Collaborative Perception-Action Task in VR

This paper presents a within-subject study to investigate the effects of leading and following behaviors on user visual attention behaviors when collaborating with a virtual agent (VA) during performing transportation tasks in immersive virtual environments. The task was to carry a target object from a location to a predefined location. There were two conditions, namely leader VA (LVA) and follower VA (FVA). The leader gave instructions to the follower to perform actions. In the FVA condition, users played the leader role, while they played the follower role in the LVA condition. The users and the VA communicated via spoken language. During the experiment, participants wore a head-mounted display and performed real walking in a room. In each condition, each participant performed 20 trials of object transportation for different types of objects. Our preliminary results revealed significant differences in the user visual attention behaviors between the follower and leader VA conditions during the transportation tasks.
Offloading Visual-Inertial Odometry for Low Power Extended Reality

Power consumption is a constraint in designing all-day wear extended reality (XR) devices. To reduce power consumption, we present a novel end-to-end XR system with visual-inertial odometry (VIO) offloaded to a remote server. Even with long network latency, the system produces an end-to-end XR experience comparable to a system without offloading while providing significant power savings. This latency and bandwidth-resilient nature of head tracking is a new and surprising result.
A Case Study on Providing Immersive Visualization for Neuronal Network Data Using COTS Soft- and Hardware

COTS VR hardware and modern game engines create the impression that bringing even complex data into VR has become easy. In this work, we investigate to what extent game engines can support the development of immersive visualization software with a case study. We discuss how the engine can support the development and where it falls short, e.g., failing to provide acceptable rendering performance for medium and large-sized data sets without using more sophisticated features.
Shrink or grow the kids? Scale cognition in an immersive virtual environment for K-12 summer camp

Virtual reality (VR) has been widely used for education and affords embodied learning experiences. Here we describe: Scale Worlds (SW), an immersive virtual environment to allow users to shrink or grow by powers of ten (10 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">X</sup> ) and experience entities from molecular to astronomical levels; and students' impressions and outcomes from experiencing SW in a CAVE (Figure 1) during experiential summer outreach sessions. Data collected from post-visit surveys of 69 students, and field observations, revealed that VR technologies: enabled interactive learning experiences; encouraged active engagement and discussions among participating students; enhanced the understanding of size and scale; and increased interest in STEM careers.
Adaptive Immersive VR Training Based on Performance and Self-Efficacy

Effective training requires that the training experience fits the ability and confidence (i.e., self-efficacy) of the trainee. Specifically, the individual's self-efficacy should ideally be slightly higher than the difficulty of a given task. A significant benefit of immersive virtual reality (IVR) is the potential to utilize measures of trainee behavior to continuously adapt the training content to the individual. However, a major challenge involves the identification of relevant measures that can be used to adapt training content in a way that increases training output. The current paper aims to inspire further research on adaptive IVR training by describing the design and development of a study on adaptive IVR training where the use of self-efficacy measures for adaptation marks a point of departure from prior literature. The design of the proposed study is informed by analyses of results from previous IVR studies.
Like a Rolling Stone: Effects of Space Deformation During Linear Acceleration on Slope Perception and Cybersickness

The decoupled relationship between the optical and inertial information in virtual reality is commonly acknowledged as a major factor contributing to cybersickness. Based on laws of physics, we noticed that a slope naturally affords acceleration, and the gravito-inertial force we experience when we are accelerating freely on a slope has the same relative direction and approximately the same magnitude as the gravity we experience when standing on the ground. This provides the opportunity to simulate a slope by manipulating the orientation of virtual objects accordingly with the accelerating optical flow. In this paper, we present a novel space deformation technique that deforms the virtual environment to replicate the structure of a slope when the user accelerates virtually. As a result, we can restore the physical relationship between the optical and inertial information available to the user. However, the changes to the geometry of the virtual environment during space deformation remain perceptible to users. Consequently, we created two different transition effects, pinch and tilt, which provide different visual experiences of ground bending. A human subject study (N=87) was conducted to evaluate the effects of space deformation on both slope perception and cyber-sickness. The results confirmed that the proposed technique created a strong feeling of traveling on a slope, but no significant differences were found on measures of discomfort and cybersickness.
A Continuous Authentication Technique for XR Utilizing Time-Based One Time Passwords, Haptics, and Kinetic Activity

Authentication in Extended Reality (XR) applications typically re-quires the user to enter a pattern or traditional password into an adapted two-dimensional UI or to enter information from outside the XR environment such as a pairing code on a mobile device. The existing solutions are far from ideal due to the inconvenience of repeatedly exiting and entering the XR environment to transfer codes, the risk associated with relying on static passwords, and the vulnerability caused by only authenticating at the start of the session. We present an authentication method developed for XR that offers robust security and an uninterrupted user experience. Our method uses a web-connected device able to generate time-based one-time passwords (TOTP) via haptics and maintain continuous authentication by tracking the user's kinetic activity. We refer to this theoretical device as the authentication device and emulate it for this paper using either an XR tracker or a networked microcontroller with an attached IMU.
DC-“Born Again”: En/De-Roling, Character Identification, &amp; VR Effects

With the rise of the metaverse, humans are increasingly spending time in mediated spaces, emphasizing the importance to understand how immersive experiences shape individual and collective behavior in various contexts (e.g., enterprise, entertainment). Two particularly important factors emerging in metaverse technologies relate to how users of virtual reality (VR) adapt and relinquish their character roles (en/de-roling), and the extent to which they identify with said roles.. Despite the prominent reliance of character roles in most VR experiences, and its noted influence on aspects of user experience (UX), little is understood about (a) how users adapt and relinquish their roles, and (b) how this process may shape affective and cognitive responses through altered identification. To address this gap, the current work employs a mixed-methods approach to understand how VR effects are influenced by these two factors.
Towards a Mixed Reality Agent to Support Multi-Modal Interactive Mini-Lessons That Help Users Learn Educational Concepts in Context

The discipline of education and training is in a state of transformation currently and is shifting toward more online learning environments as such, researchers are beginning to explore and apply new emerging technologies for online education. However, the current online learning platforms face new challenges in presenting learning content. There is a need for ways to support self-directed learning using new technological resources to create personally meaningful curiosity-driven learning experiences. In this work, an architectural framework for a mixed reality learning system is designed and presented to address this need. This is presented as an approach toward building tools for future educators to enable them to create multi-modal interactive mini-lessons in mixed reality that students can experience when they are within the appropriate learning contexts. Two proof-of-concept use case scenarios are presented using this framework. Together, this provides a step toward future multi-modal interactive educational agents that help users learn educational concepts in mixed reality.
An Immersive Geometry Environment for Mathematics Education: Taxonomy and Preliminary Evaluation

Virtual Reality (VR) offers a promising new approach to teaching fundamental aspects of mathematics, such as geometry. In this paper, we present a taxonomy of learning environments in geometry education and a study, involving expert users, evaluating VR-based geometry instruction. Based on the findings of the study, we present a series of design considerations and suggest directions for future work.
AR-MoCap: Using Augmented Reality to Support Motion Capture Acting

Technology is disrupting the way films involving visual effects are produced. Chroma-key, LED walls, motion capture (mocap), 3D visual storyboards, and simulcams are only a few examples of the many changes introduced in the cinema industry over the last years. Although these technologies are getting commonplace, they are presenting new, unexplored challenges to the actors. In particular, when mocap is used to record the actors' movements with the aim of animating digital character models, an increase in the workload can be easily expected for people on stage. In fact, actors have to largely rely on their imagination to understand what the digitally created characters will be actually seeing and feeling. This paper focuses on this specific domain, and aims to demonstrate how Augmented Reality (AR) can be helpful for actors when shooting mocap scenes. To this purpose, we devised a system named AR-MoCap that can be used by actors for rehearsing the scene in AR on the real set before actually shooting it. Through an Optical See-Through Head-Mounted Display (OST-HMD), an actor can see, e.g., the digital characters of other actors wearing mocap suits overlapped in real-time to their bodies. Experimental results showed that, compared to the traditional approach based on physical props and other cues, the devised system can help the actors to position themselves and direct their gaze while shooting the scene, while also improving spatial and social presence, as well as perceived effectiveness.
Designing Prototype XRI Workspaces with Mixed Reality and IoT Devices for Immersive Adaptive Environments

Mixed Reality (XR) and Internet-of-Things (IoT) are converging paradigms which are evolving into the metaverse. The XRI research domain addresses these hybrid IoT and XR ecosystems with a focus on embodied environmental objects and avatars. This work contributes the results of an XRI design rapid prototype process. Specifically, it presents i) an XRI system architecture, ii) a context-driven workspace prototype, and iii) an early design-science evaluation, toward new immersive adaptive spaces.
Scaling VR Video Conferencing

Virtual Reality (VR) telepresence platforms are being challenged to support live performances, sporting events, and conferences with thousands of users across seamless virtual worlds. Current systems have struggled to meet these demands which has led to high-profile performance events with groups of users isolated in parallel sessions. The core difference in scaling VR environments compared to classic 2D video content delivery comes from the dynamic peer-to-peer spatial dependence on communication. Users have many pair-wise interactions that grow and shrink as they explore spaces. In this paper, we discuss the challenges of VR scaling and present an architecture that supports hundreds of users with spatial audio and video in a single virtual environment. We leverage the property of spatial locality with two key optimizations: (1) a Quality of Service (QoS) scheme to prioritize audio and video traffic based on users' locality, and (2) a resource manager that allocates client connections across multiple servers based on user proximity within the virtual world. Through real-world deployments and extensive evaluations under real and simulated environments, we demonstrate the scalability of our platform while showing improved QoS compared with existing approaches.
Is Immersive Virtual Reality in K-12 Education Ready for Primetime? Challenges, Possibilities, and Considerations

As education enters a digitalized era, educators are embracing new tools such as immersive virtual reality (IVR) for K-12 teaching. Here, we examine the benefits of using IVR in K-12 education and highlight interesting existing use cases. Then, we acknowledge the lack of a framework for IVR use with children and propose key components that should be incorporated into such a framework. Additionally, we outline pressing topics and research questions that are essential to advance the field. We conclude that, although IVR holds great promises for shaping the digital future of education, there are still several challenges to consider.
Table2Table: Merging “Similar” Workspaces and Supporting Adaptive Telepresence Demonstration Guidance

In a typical tele-presence system, the remote participant is “trans-ported” into the local space to be shared. However, such a set-up limits the remote participant in terms of physical interaction and demonstration with the objects in the local space. One alternative is to merge the two spaces, such that the student in one space can closely watch and mimic the actions of the instructor in its space - as often occurs in science laboratories or cooking classes. As the local set-ups in the two spaces may not be exactly the same, the student has to “adapt” to the situation and identify the right objects in mimicking the instructor. We propose to semi-automatically iden-tify such object mappings, and based on the mapping, recognize the instructor's action, and replicate it in the student's space. Our preliminary user testing showed the possibility of the framework in improving the communication and process of learning.
A Comparison Study on Stress Relief in VR

In modern life, people encounter various stressors, which may cause negative mental and bodily reactions to make them feel frustrated, angry, or irritated. Earlier studies have revealed that Virtual Reality (VR) experiences are effective as traditional methods in reality in stress management and reduction. However, it is still unclear how different activities in VR and their combinations could moderate the effects on stress reduction. In this work, we conduct a comparative study to investigate the effects of two virtual activities: 1) meditation in VR, 2) a smash room experience in VR, and 3) a traditional meditation method of Sitting in Silence in the real world. The results show that VR meditation has the highest short-term effects on temporal stress, followed by VR smash room and Sitting in Silence.
Comparing Scatterplot Variants for Temporal Trends Visualization in Immersive Virtual Environments

Trends are changes in variables or attributes over time, often represented by line plots or scatterplot variants, with time being one of the axes. Interpreting tendencies and estimating trends require observing the lines or points behavior regarding increments, decrements, or both (reversals) in the value of the observed variable. Previous work assessed variants of scatterplots like Animation, Small Multiples, and Overlaid Trails for comparing the effectiveness of trends representation using large and small displays and found differences between them. In this work, we study how best to enable the analyst to explore and perform temporal trend tasks with these same techniques in immersive virtual environments. We designed and conducted a user study based on the approaches followed by previous works regarding visualization and interaction techniques, as well as tasks for comparisons in three-dimensional settings. Results show that Overlaid Trails are the fastest overall, followed by Animation and Small Multiples, while accuracy is task-dependent. We also report results from interaction measures and questionnaires.
ARPad: Compound Spatial-Semantic Gesture Interaction in Augmented Reality

In this paper, we propose the concept of compound spatial-semantic gesture interaction, in which spatial semantics is defined as the combination of world-level spatial semantics and object-level spatial semantics. The semantics of compound spatial-semantic gesture is determined by both the gesture and the spatial location of the gesture, which can increase the interaction bandwidth of gestures and improve the user experience. We present three typical compound spatial-semantic gesture application scenarios. Finally, compound spatial-semantic concepts can be applied to other interaction channels, which provides a feasible and general implementation method for expanding the interaction in Augmented Reality (AR) scenarios.
Center for Cyber-Physical Systems: Immersive Visualization and Simulation Environment

Immersive display systems in the form of head-mounted displays or full-size, walkable display systems can provide a very intuitive environment for a multitude of applications. Such applications include exploration, simulation for training, experimental studies to learn about people's behavior, and many more. Similarly, large-scale high-resolution display systems can also be very effective in data exploration and visualization. These systems can also be fully immersive. This paper describes the infrastructure available at Wright State University with its advantages and disadvantages and discusses some of its use cases as well as its setup and administration.
Behaviourally-based Synthesis of Scene-aware Footstep Sound

This paper proposes a behaviourally-based scene-aware (BBSA) footstep sound synthesis method for congruent interaction in virtual environments. Most existing methods focus on how to simulate timbre of footstep sound appropriately, but ignore the impact of user behaviour and virtual scenes on footstep sound, which reduces the sense of immersion and presence. To tackle this issue, we classify common user behaviours and design a set of mapping functions between different behaviours and footsteps. To further enhance user immersion, we propose a ray-casting based scene aware sound synthesis method, which allows to synthesize corresponding footstep sounds in real time for indoor and outdoor scenes with different material surfaces. User studies demonstrate that our proposed BBSA method achieves a higher level of immersion and presence. The complete project code is available: https://github.com/OlyMarco/Behaviourally-based-Synthesis-of-Scene-aware-Footstep-Sound-Demo-.
You Make Me Sick! The Effect of Stairs on Presence, Cybersickness, and Perception of Embodied Conversational Agents

Virtual reality (VR) technologies are used in a diverse range of applications. Many of these involve an embodied conversational agent (ECA), a virtual human who exchanges information with the user. Unfortunately, VR technologies remain inaccessible to many users due to the phenomenon of cybersickness: a collection of negative symptoms such as nausea and headache that can appear when immersed in a simulation. Many factors are believed to affect a user's level of cybersickness, but little is known regarding how these factors may influence a user's opinion of an ECA. In this study, we examined the effects of virtual stairs, a factor associated with increased levels of cybersickness. We recruited 39 participants to complete a simulated airport experience. This involved a simple navigation task followed by a brief conversation with a virtual airport customs agent in Spanish. Participants completed the experience twice, once walking across flat hallways, and once traversing a series of staircases. We collected self-reported ratings of cybersickness, presence, and perception of the ECA. We additionally collected physiological data on heart rate and galvanic skin response. Results indicate that the virtual staircases increased user level's of cybersickness and reduced their perceived realism of the ECA, but increased levels of presence.
Beyond Action Recognition: Extracting Meaningful Information from Procedure Recordings

Understanding procedural actions is important, as it can be used to automatically analyze the execution of a procedure and provide assistance to users by warning for potential mistakes or forgotten steps. However, current approaches require a rigid, step-by-step execution order, laborious and impractical datasets. Furthermore, they are unreliable to variations in viewpoint, or measure the performance of actions rather than the actual completion of actions. To address these limitations and stimulate research in this field, this work proposes the novel task of procedure state recognition (PSR) together with a set of evaluation metrics.
Design and evaluation of CNN hardware accelerator designs for in-seat passenger anomaly detection

Passenger distress detection and its notification to the connected ecosystem is of high priority to offer timely help and rescue their lives. Today's Artificial Intelligent (AI) models are capable to extract temporal and spectral features needed to autonomously recognize the passenger state using sensory signals. The work proposes convolutional neural network (CNN) based hardware accelerator design to detect passenger anomaly state, based on the digitized signals acquired from 15 spatially distributed cushion embedded pressure-sensitive-conductive (PSC) sensors. The design was aimed to work successfully on the seated passenger inside the vehicle for three different vehicle conditions including stationary vehicle, vehicle moving on well maintained plain road, and vehicle running on a sub-standard road. The hardware CNN was designed and evaluated for five different configurations by incorporating the reused processing elements that are programmed with the available resources of the targeted FPGA. All the hardware designs were evaluated with respect to power, throughput, resources utilized, and inference delay. The hardware accelerator designed FPGA system exhibited 64.70X, and 22.57X faster inference when compared with other embedded computing implementations including Raspberry Pi, and single core processor in the ZYNQ board.