Reviewerly/exHarmony huggingface dataset:
Object Detection and Classification Based on their 3D Models

This paper proposes a new approach to detect and classify terrestrial and aquatic objects based on their 3D models. The advantage of this approach is its independence from the number of output classes, which allows to dynamically add or remove them, as well as no need for a training sample with images of real objects. We propose two-stage image processing: 1) preliminary object detection and segmentation using YOLOv8, 2) transformation of the segmented object into an embedding vector for matching with vectors in the available database based on 3D models. The second step allows to match the nearest 3D object angle to the detected object and unambiguously classify it. The conversion of the segmented object into an embedding vector is performed using a neural network trained based on triplet loss. The approach was tested on a set of 3D ship models and their real videos captured from drones and other locations. The results showed an accuracy of 95% on the test sample when classifying a given set of objects.
A Comparative Evaluation of Tabs and Linked Panels for Program Understanding in Augmented Reality

Integrated development environments (IDEs) commonly employ a tab-based interface for displaying source code, which often poses challenges in efficient code navigation and retrieval. Previous research has proposed several novel approaches that have in common that they place code fragments on a 2D canvas and draw visual connections between them. In this paper, we investigate the extension of such interfaces to augmented reality (AR) environments. As AR allows to display information in three dimensions, the restriction to a 2D canvas for placing code fragments is not justified, and we implement it by allowing users to place code panels freely in 3D space. We call the resulting interface linked panels. We present the results of a quantitative user study conducted with 24 participants, aiming to explore whether the benefits observed for the canvas-based approach in traditional 2D screen environments can be replicated with linked panels in augmented reality. The participants were given tasks to identify and resolve two bugs in two different software projects using the traditional tab-based and the panel-based approaches in AR. To find possible explanations of our quantitative results we also conducted a qualitative analysis evaluating participants’ comments and different placement strategies of panels in the panel-based approach. Our results indicate that participants found more bugs with the tabs-version, but were equally fast with both tools. We also found that less skilled participants were faster with the tabs, while more skilled ones were faster with the panels. Although, participants experienced problems with the cluttered spatial arrangement of the panels, they preferred the panels version over the tabs version as it made better use of AR.
A Real Time Retrieval Method of Massive Arrears User Profile Data Based on HBase

Due to the large amount, fast speed and incompatibility of the massive arrears users' file data, it has brought some impact on their retrieval work. In order to improve the accuracy and efficiency of massive arrears user archive data retrieval, a real-time retrieval method for massive arrears user archive data based on HBase is proposed. Based on the Hadoop architecture, the massive arrears user profile data is collected, the massive arrears user profile data is linearly converted, the dimensions are reduced, and the massive arrears user profile data is preprocessed. On this basis, HBase storage model and HBase two-level fast index are established to realize real-time retrieval of massive arrears user file data based on HBase. The experimental results show that the retrieval accuracy of the proposed method is 97.6%, and the retrieval time is only 6.8 ms, which can effectively improve the retrieval accuracy and efficiency of the massive arrears user file data.
Beyond the Wizard of Oz: Using Imperfect Machine Learning to Examine the Impact of Reliability of Augmented Reality Cues on Visual Search Performance

Searching for an object in their visual field, even when the person knows exactly what the object looks like, is a complex task. This is time-consuming and can be error-prone. We present findings of automation bias when imperfect cues are presented using real-time machine learning, presented in the Augmented Reality head-mounted display using the YOLOv5 machine learning model. We validated that Wizard-of-Oz studies produce results that are equivalent to using a real-time machine-learning system for visual search aided by cues.
A Framework for Automatic Generation of Augmented Reality Objects

Currently, periodic inspection for maintenance of social facilities such as domestic bridges and highways are being carried out. Recently, methods using AR glasses and mobile devices have been attempted to improve such an existing document-oriented inspection method. However, many steps and transformations are required to create 3D objects or augmented reality objects used in these devices. That is, it is necessary to use various software to extract and convert information about objects in each step, and there are many difficulties in processing and optimizing the created objects. To solve these problems, this study aims to develop a framework that can automatically create augmented reality objects from the basic data of facilities by reducing the complex creation process of 3D objects. The characteristics of this research system are, first, it can automatically create 3D objects by reducing the complex steps for object conversion, Second, it made it possible to process a partial area of the facility, Third, it can be converted into various 3D objects, Fourth, optimization work was performed to reduce the size of the created object. Currently, a pilot system was built, and the data optimization experiments were performed, and various creations of 3D objects can be automatically generated.
Transitional Blended Realities: Linking between Video and Mixed Reality for Distributed Team Collaboration

This paper envisions Transitional Blended Realities (TBR)—systems that integrate interfaces for Video Conferencing and Collaborative Mixed Reality. By utilizing a visual overlap in the reference space between remote physical spaces, TBR supports fluid transitions between video conferencing and mixed reality collaboration. This approach has the potential to enhance both collaboration flexibility and accessibility, fostering more dynamic distributed team meetings.
Effect of Degrees of Freedom on the Quality of Use of Virtual Theatre Experiences

Immersive technologies create new possibilities for media consumption. Especially in the case of cultural events such as theatre performances, new forms of participation are emerging. However, due to technical barriers, such offerings are still limited to 360° video content. In a study with 26 experienced VR users, we analyzed the influence of the degrees of freedom (DoF) of movement in seated applications on the perception of presence, motion sickness, quality of experience (QoE) and on user acceptance. The results show that the perception of presence and QoE were significantly higher with six degrees of freedom (6DoF) than with three degrees of freedom (3DoF). In a direct comparison, all participants preferred the 6DoF variant. The majority rejected the 3DoF variant, mainly due to physical discomfort. The findings indicate that solutions for 6DoF-enabled content should also be developed and provided for immersive content consumed in a seated position, such as a theater performance.
Towards Universal Interaction for Extended Reality

Extended Reality (XR) is a rapidly growing field offering unique immersive experiences, social networking, learning, and collaboration opportunities. The continuous advancements in XR technology and industry efforts are gradually moving this technology toward end consumers. However, a universal one-size-fits-all solution for seamless XR interaction still needs to be discovered. Currently, we face a diverse landscape of interaction modalities that depend on the environment, user preferences, task, and device capabilities. Commercially available input methods like handheld controllers, hand gestures, voice commands, and combinations of those need universal flexibility and expressiveness. Additionally, hybrid user interfaces, such as smartwatches and smartphones as ubiquitous input and output devices, expand this interaction design space. In this position paper, we discuss the idea of a universal interaction concept for XR. We present challenges and opportunities for implementing hybrid user interfaces, emphasizing Environment, Task, and User. We explore the potential to enhance user experiences, interaction capabilities, and the development of seamless and efficient XR interaction methods. We examine challenges and aim to stimulate a discussion on the design of generic, universal interfaces for XR.
Stop Bad Real-time Feedback!: Estimation of the Timing of Feedback that Negatively Impacts Presenters for Presentation Training in Virtual Reality

Training in virtual reality (VR) using a head-mounted display (HMD) has been applied to various fields, and recent attention has been focused on real-time feedback to the trainee. However, trainees’ acceptance of the presented feedback is dependent on the state of the ongoing task as well as their internal state. If the trainees are not ready to accept the feedback from the system, it may potentially disturb the ongoing training. This study is premised on it being desirable for a VR training system to recognize the trainee’s state and present feedback at an appropriate time. As a first step, this study focused on using a case scenario about public speaking training to confirm the appropriate timing of feedback in terms of trainees’ subjective impressions. Specifically, a presentation training system that uses VR-HMD to collect observable multimodal information (including body posture and movement, gaze, heartbeat, perspiration, voice, and slide operation) was developed, which provided real-time feedback using icons. We collected the trainee’s subjective scores of disturbance for each feedback in various situations through a Wizard of Oz experiment. Data analysis revealed two things; (1) Real-time feedback negatively affects the trainee’s presentation in as many as 30 % of cases, (2) More than half of such negating-impact feedback can be estimated from observable information. This finding is expected to contribute to the development of future training systems that provide real-time feedback avoiding inappropriate timing for the trainee.
Where We Stand and Where to Go: Building Bridges Between Real and Virtual Worlds for Collaboration

Cross reality has redefined how individuals interact in the virtual and physical realms, offering a comprehensive paradigm for communication and collaboration. A rigorous examination of the literature from ACM, IEEE, SCOPUS, and ScienceDirect databases was conducted using a focused search string, resulting in the review of 42 relevant papers as of July 10th, 2023. Our analysis provides an understanding of the current state of cross reality collaboration across various fields, including hardware and software technology, applications, and hybrid user interfaces. Future research directions such as addressing non-adopters, implementing ‘Privacy by Design’ principles, and enhancing latency reduction are highlighted. We hope that our paper will serve as a stepping stone for researchers and practitioners aiming to develop novel cross reality collaboration experiences.
SAR Image Registration Based on Feature Points Using Improved CSP-DenseNet

Synthetic aperture radar (SAR) image registration is widely used in integrated navigation system with development of guidance system in sensors and other applications. However, there exists some difficulty because of the quality and imaginary principle of SAR images. In this article, a brand-new method based on feature points using improved CSP-DenseNet is proposed to solve the problems of SAR image registration with weak and noisy texture. Deep features of interest points in SAR images are extracted using the proposed network from the crops of search image and template image respectively. The method has an advantage of preserving abundant feature information in SAR images under the influence of insufficient image information and noise of SAR images. The CSP-DenseNet architecture has a cross-stage construction optimizing fused by CSP-Net and DenseNet to extract matching features. The network is partially connected in specific convolutional layers for feature reusing and computation resources saving. Then, Brute Force Matcher and RANSAC voting are successively applied for a larger number of matching pairs and high-precise matching vertexes. Experimental results on various SAR image matching methods show that the proposed method provides better performance than other approaches compared.
Optimization of SVM Classification Accuracy with Bayesian Optimization Utilizing Data Augmentation

Optimizing the accuracy of SVM classification is one of the most important things in the world of machine learning. This study will use the SVM (Support Vector Machine) algorithm in classifying. This study aims to improve the classification accuracy of the Support Vector Machine (SVM) model by utilizing Bayesian Optimization techniques and data augmentation. The Bayesian Optimization technique will be used to find the most optimal values for the kernel and C parameters in the SVM model. While data augmentation is done to improve the quality of the training data by adding variations and data features. This research uses a synonym replacement technique in data augmentation. The data used was obtained from social media Twitter. The results of this study are in the form of a model with a combination of Bayesian Optimization and augmentation. The accuracy obtained with the SVM model using Bayesian Optimization and data augmentation is 89%. However, the conventional SVM model without Bayesian Optimization only produces 82.19% accuracy so the SVM model with Bayesian Optimization increases the accuracy by 7%.
Discussing Facets of Hybrid User Interfaces for the Medical Domain

Hybrid user interfaces are commonly discussed in the context of integrating conventional displays into Mixed Reality environments, as well as multi-modality and cross-device interaction. We believe that the consideration of these overarching categories does not suffice to inform the development of these interfaces in very distinct application domains, such as medicine. Instead, we argue that the medical domain comprises of various further aspects that influence the design of hybrid user interfaces. With this work, we want to discuss additional facets that emerge from the specific constraints and challenges of this particular field. To do this, we first introduce three selected research projects, in which we have been involved, showcasing the range of possible tasks and application areas in the context of medicine. As a result, we identify and discuss the following facets: Physical Location, Visualization Placement, Display Capabilities, Interaction Style, Enhancement Target, and Task Concurrency.
AiRcupid: Supporting Cucumber Picking for Intellectual Disabilities using Artificial Intelligence and Augmented Reality

As one of the sustainable development goals (SDGs) of the United Nation (UN), building an inclusive society is essential to fostering peace and harmony. Through social participation, persons with disabilities in particular can gain confidence and a sense of fulfilment. However, people with intellectual disabilities may have difficulty making decisions of their own initiative and may face various difficulties in employment. This paper proposes a new technology to support cucumber harvesting for individuals with intellectual disabilities by combining deep neural network (DNN)based harvest timing determination and augmented reality (AR)based harvest instruction presentation. Through evaluation experiments targeting four participants with different levels of intellectual disabilities in an actual cucumber field, the effectiveness of the proposed system is verified, demonstrating that the proposed technology can significantly improve the work performance of individuals with intellectual disabilities.
EyeTTS: Evaluating and Calibrating Eye Tracking for Mixed-Reality Locomotion

Eye tracking is an increasingly popular method for interacting with AR and measuring user attention. However, implementing and evaluating eye tracking across multiple platforms and use cases can be challenging due to the lack of standardized metrics and measurements. Additionally, existing calibration methods and accuracy measurements often do not account for the common scenarios of walking and scanning in mobile AR settings. To test and compare different eye tracking devices on various AR tasks and metrics, we developed EyeTTS, an eye tracking test suite specifically designed for scenarios involving head movement and locomotion in AR. We conducted user studies on the Magic Leap (n=36, 1 trial per task) and HoloLens 2 (n=54, 2 trials per task) devices to collect data and compare the precision and accuracy of each headset under different movement and reference frame conditions.
BerryScope: AR Strawberry Picking Aid

Strawberry picking farms have the problem that visitors do not choose fully ripe straw berries. This is because fully ripe strawberries can easily get damaged during transportation, meaning customers usually see strawberries that are not fully ripe in stores. Therefore, they are likely to pick the strawberries similar to what they see in stores. As a result, visitors may miss the opportunity to taste the most delicious strawberries, and for the farmers, it leads to waste of the ripe strawberries. To solve the problem, this study proposes BerryScope, an augmented reality (AR) system for supporting strawberry picking. Visitors wear an optical see-through head-mounted display (OSTHMD) to pick strawberries. Based on the images captured by the OSTHMD's camera, the system uses deep neural network (DNN) technology to analyze the ripeness of the strawberries and overlays harvest instructions on fully ripe strawberries. We conducted experiments at an actual strawberry farm involving 15 people and confirmed that using the proposed system enables users to choose more ripe strawberries.
Testbed for Intuitive Magnification in Augmented Reality

Humans strive to magnify portions of our visually perceived surroundings for various reasons, e.g., because they are too far away or too small to see. Different technologies have been introduced for magnification, from monoculars to binoculars, and telescopes to microscopes. A promising technology are modern high-resolution digital cameras, which are capable of optical or digital zoom and very flexible as their imagery can be presented to users in real-time with mobile or head-mounted displays and intuitive 3D user interfaces allowing control over the magnification. In this demo, we present a novel design space and testbed for intuitive augmented reality (AR) magnifications, where an AR optical see-through head-mounted display is used for the presentation of real-time magnified camera imagery. The testbed includes different unimanual and bi-manual AR interaction techniques for defining the scale factor and portion of the user’s visual field that should be magnified.
Exploring Hybrid User Interfaces for Surgery Planning

Hybrid user interfaces are a great opportunity to combine complementary interfaces to make use of the best interface for specific steps in a workflow. This position paper outlines one diverse application field: surgery planning. Planning a surgery is a complex task as the surgical team has to get an overview and understanding of a patient’s medical history and the internal anatomical structures of the organ or region of interest. In this position paper, we outline how different hardware (e.g., mixed reality head-worn devices and physical objects) and interaction concepts (e.g., gesture-based interaction or keyboard and mouse) can create an optimal workflow for surgery planning.
Reviving the Euston Arch: A Mixed Reality Approach to Cultural Heritage Tours

Augmented Reality (AR) and Virtual Reality (VR) users have distinct capabilities and experiences during Extended Reality (XR) collaborations: while AR users benefit from real-time contextual information due to physical presence, VR users enjoy the flexibility to transition between locations rapidly, unconstrained by physical space.Our research aims to utilize these spatial differences to facilitate engaging, shared XR experiences. Using Google Geospatial Creator, we enable large-scale outdoor authoring and precise localization to create a unified environment. We integrated Ubiq to allow simultaneous voice communication, avatar-based interaction and shared object manipulation across platforms.We apply AR and VR technologies in cultural heritage exploration. We selected the Euston Arch as our case study due to its dramatic architectural transformations over time. We enriched the co-exploration experience by integrating historical photos, a 3D model of the Euston Arch, and immersive audio narratives into the shared AR/VR environment.
Widget Design for Precise and Intuitive Guidance in Surgical Spatial Environments

Precision tasks require spatial 6DOF manual dexterity and minimal human error may cause irreversible damage or unwanted outcomes. Augmented reality technology assists surgeons in tool navigation tasks by conveying precision information with specific assets and widgets in literature. However, the current approaches are often quasi-static, the errors are still present, and no specific guidelines are established. While tracking technology is constantly evolving the knowledge of User interfaces is not ready Moreover, the user interface design is often underestimated in its importance and poor designs cause a high task load on the surgeon cognitively or physically. This Ph.D. thesis aims to design and evaluate novel methods for precision guidance tools for improving performance and user experience. As a first task, I made literature reviews followed by user studies with the implementation of intuitive, dynamic widgets that allow better precision outcomes and user preference. I believe that this Ismar 23 Doctoral Consortium can help me to gain opinions on followed methodology and discuss further steps of my doctoral thesis.
Folding Rays: a Bimanual Occluded Target Interaction Technique

As Virtual Reality becomes commonplace in the world, it is important for developers to focus on user interaction with the virtual world. Currently, there are limitations to some selection and navigation techniques that have not yet been completely overcome. Focusing specifically on enhancing ray-casting, we present the advanced technique of folding rays which allows for the selection of occluded targets without any unnecessary physical navigation around a virtual environment. By improving upon current approaches, our technique allows for the selection of these targets without any manipulation of the virtual environment itself using rays that can bend at user-determined points. With their potential to be used in conjunction with teleportation as a virtual navigation technique, folding rays can be used in a variety of scenarios to enhance a user’s interactive experience in virtual environments.