Reviewerly/exHarmony huggingface dataset:
Towards Using Generative AI for Facilitating Image Creation in Spatial Augmented Reality

The intersection of Generative Artificial Intelligence (GenAI) and Spatial Augmented Reality (SAR) constitutes a relatively unexplored field. The effectiveness of general-purpose generative AI models in numerous distinct scenarios and tasks remains wanting. Addressing this predicament, our paper concentrates on image generation in SAR contexts. We present the Contrastive Learning Integrated with Depth Perceptual Fusion (CLIPF) method, a refined text-plus-image-to-image generative technique, specifically devised to overcome projection-specific hurdles typically inherent in standard diffusion models. Additionally, to substantiate the efficacy of our approach, we construct a system through a cooperative amalgamation of software components, such as ChatGPT, and hardware devices, including the Intel RealSense D435 Depth Camera, thereby fostering an innovative interaction for projection content creation based on natural language. Comprehensive evaluation results validate our approach’s capacity to markedly enhance projection image quality. In summation, we illuminate potential utility avenues for our proposed approach, emphasizing its diverse prospective contributions towards enriching future SAR experiences.
Supporting Co-Presence in Populated Virtual Environments by Actor Takeover of Animated Characters

Online social virtual worlds are now becoming widely available on consumer devices including virtual reality headsets. One goal of a virtual world could be to give a user an experience of a crowded environment with many virtual humans. However, gathering enough personnel to control the necessary number of avatars for creating a realistic scene is usually difficult. Additionally, current technology is not capable of fully simulating avatars with behaviours, especially when interaction with users is required. In this paper, we develop a system that enables an actor to take over control of one of a set of avatars. We built an immersive interface that allows an actor to select an avatar to take over and then segue into the currently playing animation. By allowing one person to take control of multiple avatars, we can enhance the plausibility of environments inhabited by simulated characters. In an experiment, we show that in a cafe scenario, one actor can take over the roles of a barista and two customers. Experiment participants reported experiencing the scene as if it were populated by more than one actor. This system and experiment demonstrate the feasibility of one actor controlling multiple avatars sequentially, thus enhancing users’ feelings of being in a populated environment.
Is this the vReal Life? Manipulating Visual Fidelity of Immersive Environments for Medical Task Simulation

Recent developments and research advances contribute to an ever-increasing trend towards quality levels close to what we experience in reality. In this work, we investigate how different degrees of these quality characteristics affect user performance, qualia of user experience (UX), and sense of presence in an example medical task. To this end, a two-way within-subjects design user study was conducted, in which three different levels of visual fidelity were compared. In addition, two different interaction modalities were considered: (1) the use of conventional VR controllers and (2) natural hand interaction using 3D-printed, spatially-registered replicas of medical devices, to interact with their virtual representations. Consistent results indicate that higher degrees of visual fidelity evoke a higher sense of presence and UX. However, user performance was less affected. Moreover, no differences were detected between both interaction modalities for the examined task. Future work should investigate the discovered interaction effects between quality levels and interaction modalities in more detail and examine whether these results can be reproduced in tasks that require more precision. This work provides insights into the implications to consider when studying interactions in VR and paves the way for investigations into early phases of medical product development and workflow analysis.
Shopping in between Realities-Using an Augmented Virtuality Smartphone in a Virtual Supermarket

In this project, the full spectrum provided by Milgram’s RealityVirtuality Continuum is utilized to enhance presence, usability, and interactions in an authentic Virtual Reality (VR) supermarket simulation used as a standardized evaluation platform for mHealth apps. We introduce a unique Unity-based modeling platform for supermarket environments and the option to design high-quality customized products. To achieve that, solutions are demonstrated by focusing on a recognizable replica of a discounter and utilizing Augmented Virtuality (AV) to include a physical smartphone in the virtual simulation. The user is able to manipulate the simulation from within the smartphone app through this versatile, highly usability-centered controller. To achieve reliable tracking of the smartphone screen, we propose a hybrid approach, which combines fiducial marker tracking with data acquired through a WiFi connection between the VR system and the smartphone. Furthermore, the AV concept is utilized to build scenarios for Mixed Reality (MR) use cases such as simulated AR to navigate to a chosen product in the market. After an initial pre-study with important insights to strengthen the platform, a broad user study involving the physical smartphone with a simulated AR scenario has been conducted. The goals with 30 participants were to evaluate spatial presence, involvement, experienced realism (IPQ), and usability of the system (SUS). Results showed ”Good” (SUS) usability and recorded data as well as the participants’ feedback brought important insights. We plan to release this unique VR supermarket platform to contribute to the science community and mHealth industry.
Cueing Sequential 6DoF Rigid-Body Transformations in Augmented Reality

Augmented reality (AR) has been used to guide users in multi-step tasks, providing information about the current step (cueing) or future steps (precueing). However, existing work exploring cueing and precueing a series of rigid-body transformations requiring rotation has only examined one-degree-of-freedom (DoF) rotations alone or in conjunction with 3DoF translations. In contrast, we address sequential tasks involving 3DoF rotations and 3DoF translations. We built a testbed to compare two types of visualizations for cueing and precueing steps. In each step, a user picks up an object, rotates it in 3D while translating it in 3D, and deposits it in a target 6DoF pose. Action-based visualizations show the actions needed to carry out a step and goal-based visualizations show the desired end state of a step. We conducted a user study to evaluate these visualizations and the efficacy of precueing. Participants performed better with goal-based visualizations than with action-based visualizations, and most effectively with goal-based visualizations aligned with the Euler axis. However, only a few of our participants benefited from precues, most likely because of the cognitive load of 3D rotations.
“Can You Move It?”: The Design and Evaluation of Moving VR Shots in Sport Broadcast

Virtual Reality (VR) broadcasting has seen widespread adoption in major sports events, attributed to its ability to generate a sense of presence, curiosity, and excitement among viewers. However, we have noticed that still shots reveal a limitation in the movement of VR cameras and hinder the VR viewing experience in current VR sports broadcasts. This paper aims to bridge this gap by engaging in a quantitative user analysis to explore the design and impact of dynamic VR shots on viewing experiences. We conducted two user studies in a digital hockey game twin environment and asked participants to evaluate their viewing experience through two questionnaires. Our findings suggested that the viewing experiences demonstrated no notable disparity between still and moving shots for single clips. However, when considering entire events, moving shots improved the viewer’s immersive experience, with no notable increase in sickness compared to still shots. We further discuss the benefits of integrating moving shots into VR sports broadcasts and present a set of design considerations and potential improvements for future VR sports broadcasting.
Augmented Reality Annotations in a Collaborative Time-Critical Task

This research explores the use of Augmented Reality (AR) annotations for assisting with performance, decision-making, and collaboration in a time-critical task. A user study was conducted to compare the use of AR with and without annotations, as well as the use of an attention-guiding technique. The results revealed that participants felt more confident in making timely and informed decisions when having access to annotations. Performance metrics indicated that the consumption of AR annotations adds cognitive overhead, however, they are not detrimental to achieving task goals. These findings pave the way for further development and investigation of AR systems for assisting with performance, decision-making, and collaboration in time-critical tasks.
LeanOn: Simulating Balance Vehicle Locomotion in Virtual Reality

Locomotion plays a critical role in user experience in Virtual Reality (VR). This work presents a novel locomotion device, LeanOn, which aims to enhance immersion and feedback experience in VR. Inspired by balance vehicles, LeanOn is a leaning-based locomotion device that allows users to control their location by tilting a board on two balance wheels, with rotation enabled by two buttons near users’ feet. To create a more realistic riding experience, LeanOn is equipped with a terrain vibration system that generates varying levels of vibration based on the roughness of the terrain. We conducted a within-subjects experiment $(\mathrm{N}=24)$ and compared the use of LeanOn and joystick steering in four aspects: cybersickness, spatial presence, feedback experience, and task performance. Participants used LeanOn with and without the vibration system to investigate the necessity of tactile feedback. The results showed that LeanOn significantly improved users’ feedback experience, including autotelic, expressivity, harmony, and immersion, and maintained similar levels of cybersickness and spatial presence, compared to joystick steering. Our work contributes to the field of VR locomotion by validating a leaning-based steering prototype and showing its positive effect on improving users’ feedback experience in VR. We also showed that tactile feedback in locomotion is necessary to further enhance immersion in VR.
MRMAC: Mixed Reality Multi-user Asymmetric Collaboration

We present MRMAC, a Mixed Reality Multi-user Asymmetric Collaboration system that allows remote users to teleport virtually into a real-world collaboration space to communicate and collaborate with local users. Our system enables telepresence for remote users by live-streaming the physical environment of local users using a 360° camera while blending 3D virtual assets into the mixed-reality collaboration space. Our novel client-server architecture enables asymmetric collaboration for multiple AR and VR users and incorporates avatars, view controls, as well as synchronized low-latency audio, video, and asset streaming. We evaluated our implementation with two baseline conditions: conventional 2D and standard 360° videoconferencing. Results show that MRMAC outperformed both baselines in inducing a sense of presence, improving task performance, usability, and overall user preference, demonstrating its potential for immersive multi-user telecollaboration.
A Comparative Evaluation of AR Embodiments vs. Videos and Figures for Learning Bead Weaving

The most common learning materials for handcraft today are videos and figures, which are limited in their ability to express embodied knowledge as an in-person tutor could. We developed WeavAR, an application for headworn augmented reality (AR) displays designed to teach basic bead weaving patterns. WeavAR combines virtual 3D hands showing weaving sequences recorded from an experienced bead weaver and a dynamic 3D bead model showing how the work progresses. Using a mixed within/between-subjects user study (n=30), we compared learning materials (AR to videos and figures) and learning material placement (in the area of work or to the side). Results show that the AR learning materials had comparable effectiveness to video and figures. Hand visualizations were found to lack crucial context, however, making them less useful than the 3D bead model. Extra measures to prevent obstruction are required when placing learning materials at the area of work.
Edge-Centric Space Rescaling with Redirected Walking for Dissimilar Physical-Virtual Space Registration

We propose a novel space-rescaling technique for registering dissimilar physical-virtual spaces by utilizing the effects of adjusting physical space with redirected walking. Achieving a seamless and immersive Virtual Reality (VR) experience requires overcoming the spatial heterogeneities between the physical and virtual spaces and accurately aligning the VR environment with the user’s tracked physical space. However, existing space-matching algorithms that rely on one-to-one scale mapping are inadequate when dealing with highly dissimilar physical and virtual spaces, and redirected walking controllers could not utilize basic geometric information from physical space in the virtual space due to coordinate distortion. To address these issues, we apply relative translation gains to partitioned space grids based on the main interactable object’s edge, which enables space-adaptive modification effects of physical space without coordinate distortion. Our evaluation results demonstrate the effectiveness of our algorithm in aligning the main object’s edge, surface, and wall, as well as securing the largest registered area compared to alternative methods under all conditions. These findings can be used to create an immersive play area for VR content where users can receive passive feedback from the plane and edge in their physical environment.
Approximated Match Swiping: Exploring more Ergonomic Gaze-based Text Input for XR

In augmented and virtual reality, hands-free text input is still a significant challenge. Though some virtual keyboard systems exist, they often require the use of controllers, suffer from the Midas touch problem, or require significant activation times to allow for the selection of keys. To help address this problem, we present a system called Approximated Match Swiping (AMSwipe for short), a pure gaze-based text input system that allows for efficient typing in virtual environments. This system builds on prior eye-based path/swipe systems for word recognition and selection in which users look at the keys on a virtual keyboard in succession to trace out a word path. Unlike other systems such as reverse-crossing, our method makes use of the spacebar to start or stop input, and the user can chain words together with additional brief glances at the spacebar, allowing for faster transitions between words. This system works for virtual and augmented reality as well as desktop and mobile systems. In an experiment testing our design alongside other well-known input systems, participants achieved an average raw typing speed of 7.81 words per minute (WPM) with under five minutes of practice, with an expert user reaching an average of 23 WPM. We compared our technique with two others, including Dwell and reverse-crossing, which had similar novice typing speeds of 8.19 WPM and 6.40 WPM, respectively.
Reality Stack I/O: A Versatile and Modular Framework for Simplifying and Unifying XR Applications and Research

This paper introduces Reality Stack I/O (RSIO), a versatile and modular framework designed to facilitate the development of extended reality (XR) applications. Researchers and developers often spend a significant amount of time enabling cross-device and cross-platform compatibility, leading to delays and increased complexity. RSIO provides the essential features to simplify and unify the development of XR applications. It enhances cross-device and cross-platform compatibility, expedites integration, and allows developers to focus more on building XR experiences rather than device integration. We offer a public Unity reference implementation with examples.
Comparative Analysis of Change Blindness in Virtual Reality and Augmented Reality Environments

Change blindness is a phenomenon where an individual fails to notice alterations in a visual scene when a change occurs during a brief interruption or distraction. Understanding this phenomenon is specifically important for the technique that uses a visual stimulus, such as Virtual Reality (VR) or Augmented Reality (AR). Previous research had primarily focused on 2D environments or conducted limited controlled experiments in 3D immersive environments. In this paper, we design and conduct two formal user experiments to investigate the effects of different visual attention-disrupting conditions (Flickering and Head-Turning) and object alternative conditions (Removal, Color Alteration, and Size Alteration) on change blindness detection in VR and AR environments. Our results reveal that participants detected changes more quickly and had a higher detection rate with Flickering compared to Head-Turning. Furthermore, they spent less time detecting changes when an object disappeared compared to changes in color or size. Additionally, we provide a comparison of the results between VR and AR environments.
Theoretical Aspects of the Semantics of Generated Music

In the course of creating and analyzing music content, artificial intelligence systems are widely used, and according to the forecasts of analysts, their use in the field of music will grow steadily. Modern artificial intelligence systems successfully solve problems that cause difficulties for humans. However, these systems lack the ability to create and evaluate the semantics of musical works the same way as humans do. The problem of creating musical artificial intelligence with the creative abilities of the human mind is one of the most topical and complex problems in the music industry. The solution to this problem in the first place lies in studying and understanding the essence of the term “Musical Sense”. The purpose of the study is to develop a concept for the probabilistic assessment of the musical texts' semantics based on the concepts of information theory and information technologies. To achieve the goal, a convergent approach to the study of the problem of musical meaning, proposed by Professor A. B. Kayak, was used. As a result of the completed study, the main conceptual provisions of the theory of probabilistic evaluation of musical senses have been developed. The article is of a theoretical nature. However, the materials of the article may be of practical interest for researchers of the problem of sense and developers of artificial intelligence in the music industry.
Large Language Model Locally Fine-tuning (LLMLF) on Chinese Medical Imaging Reports

The emergence of large language models exerts significant impact in the field of natural language processing. These models, which are based on attention networks, have shown remarkable capabilities in understanding and generating human conversation context, surpassing most of the state-of-the-art models in Natural Language Processing (NLP). Though Generative Pre-trained GPT-series open their public Application Programming Interfaces (APIs) for direct inference, such services require users to upload (relinquish) their data to servers, thus not suitable for domains operating sensitive data, such as the medical field. Due to comparable smaller model size, existing local pre-trained model deployments are usually not that helpful on domain-specific data without further fine-tuning. In this paper, we leverage Microsoft's recent open source DeepSpeed-Chat platform and one of our selected pre-trained models, to locally conduct fine-tuning on our medical imaging reports (in Chinese), to infer diagnosis recommendations based on the imaging description. Our output models show very promising results according to both NLP metrics and experts' evaluation criteria. This paper provides an initial report on our latest progress on locally fine-tuning large language model for medical data.
Value-added evaluation study based on regression analysis

Student assessment is an important part of educational assessment, and value-added assessment is an assessment system that has been gradually developed in recent years, but student value-added is influenced by factors such as basic achievement. Theoretically, student value-added tends to regress to the center, and the empirical analysis also found a significant linear relationship between student value-added and basic grades. This paper first uses regression analysis to determine the quantitative relationship between student value-added and basic rank, and then uses the "residual", also known as the difference between the original value-added and the regression value, to propose a value-added evaluation model for teaching classes based on regression analysis to achieve value-added evaluation at the student and teacher levels, which complements the evaluation system.
Semi-supervised One-pass Learning under Distribution Shift

One-pass learning is one of the most popular and widely used online learning method for high-speed and large-scale streaming data. However, large-scale data usually has the problem of label scarcity, and the insufficient labeled data has a great negative impact on the performance of one-pass learning. The existing research on semi-supervised online learning has difficulty extending to one-pass learning due to the iterative nature of semi-supervised methods. In addition, the distribution of complex streaming data may also change over time. Just directly ignoring the change in data distribution will seriously affect the value and significance of learning. Therefore, one-pass learning under distribution shift when labels are scarce is a challenging task. In this paper, we propose a semi-supervised one-pass learning method with forgetting factor to handle the change of data distribution efficiently. In this method, we use an improved mini-batch label propagation algorithm to obtain pseudo-labels for the incoming unlabeled data. The forgetting factor is introduced to calculate the memory matrix for the labeled data and the pseudo-labeled data to deal with the data distribution change. We also introduce a semi-supervised confidence factor to balance the weight of labeled and pseudo-labeled data. Extensive experiments on streaming datasets demonstrate the effectiveness of our approach. Especially under the one-pass constraint, it has achieved far better performance than the existing semi-supervised online learning.
A Data Collection Method based on Topic Weight Hypergraph

With the rapid development of technologies such as big data and artificial intelligence, the global internet data production speed is increasing. The traditional manual keyword-based retrieval approach is unable to meet the needs of information retrieval. Therefore, it is essential to efficiently and quickly obtain high-value data from massive internet data. In this paper, we propose an automatic data collection method based on the weighted hypergraph of topic words, initiating the retrieval process from a seed keyword search. By iteratively exploring new topic words around the seed keyword, we achieve the collection of topic data.