Reviewerly/exHarmony huggingface dataset:
Conjunctive use of mineralogy and elemental composition for empirical forensic provenancing of topsoil from Canberra, Australia

The capability to spatially triage geographical areas as low and high interest has the potential to provide valuable information as forensic intelligence to law enforcement operations, and related provenancing applications. Among others, our previously published work has largely been based on the elemental composition of topsoil samples, omitting other potentially useful compositional characteristics, such as mineralogy, that have proven valuable in forensic casework discriminations. In this contribution, a total of 334 topsoil (0–5 cm sampling depth; 0–75 µm fraction) samples collected from the Canberra region in Australia, were selected from a larger collection (n = 685) and their bulk mineralogy determined using X-ray powder diffraction (XRPD). Utilising an existing casework technique for discriminating soils by mineralogy, a total of twelve diagnostic peaks were selected representing commonly occurring minerals. Peak intensities were normalised relative to the sum of their intensities and used to create an indicative mineralogy dataset for the study region. Based on an existing algorithm for assigning investigative analytical similarities from overlapping areas between two Cauchy distributions, the provenance was estimated for thirteen blind topsoil samples. Provenance maps based on the mineralogy were subsequently combined with earlier elemental-based predictions, incorporating contrasting discriminatory capabilities from both techniques. Results indicate the mineralogical component of topsoils can provide accurate provenance predictions, and when combined with those based on the elemental composition, can further delineate areas as low interest that otherwise would not necessarily be differentiated from one technique alone.
DeAnomalyzer: Improving Determinism and Consistency in Anomaly Detection Implementations

Anomaly Detection (AD) is a popular unsupervised learning technique, but AD implementations are difficult to test, understand, and ultimately improve. Contributing factors for these difficulties include the lack of a specification to test against, output differences (on the same input) between toolkits that supposedly implement the same AD algorithm, and no linkage between learning parameters and undesirable outcomes. We have implemented DeAnomalyzer, a black-box tool that improves AD reliability by addressing two issues: nondeterminism (wide output variations across repeated runs of the same implementation on the same dataset) and inconsistency (wide output variations between toolkits on the same dataset). Specifically, DeAnomalyzer uses a feedback-directed, gradient descent-like approach to search for toolkit parameter settings that maximize determinism and consistency. DeAnomalyzer can operate in two modes: univariate, without ground truth, targeted to general users, and bivariate, with ground truth, targeted to algorithm designers and developers. We evaluated DeAnomalyzer on 54 AD datasets and the implementations of four AD algorithms in three popular ML toolkits: MATLAB, R, and Scikit-learn. The evaluation has revealed that DeAnomalyzer is effective at increasing determinism and consistency without sacrificing performance, and can even improve performance.
Validating Objective Evaluation Metric: Is Fréchet Motion Distance able to Capture Foot Skating Artifacts ?

Automatically generating character motion is one of the technologies required for virtual reality, graphics, and robotics. Motion synthesis with deep learning is an emerging research topic. A key component of the development of such an algorithm involves the design of a proper objective metric to evaluate the quality and diversity of the synthesized motion dataset, two key factors of the performance of generative models. The Fréchet distance is nowadays a common method to assess this performance. In the motion generation field, the validation of such evaluation methods relies on the computation of the Fréchet distance between embeddings of the ground truth dataset and motion samples polluted by synthetic noise to mimic the artifacts produced by generative algorithms. However, the synthetic noise degradation does not fully represent motion perturbations that are commonly perceived. One of these artifacts is foot skating: the unnatural foot slides on the ground during locomotion. In this work-in-progress paper, we tested how well the Fréchet Motion Distance (FMD), which was proposed in previous works, is able to measure foot skating artifacts, and we found that FMD is not able to measure efficiently the intensity of the skating degradation.
A Social Awareness Interface for Helping Immigrants Maintain Connections to Their Families and Cultural Roots: The Case of Venezuelan Immigrants

International migration forces people into an unfamiliar reality in which their customs and values lose relevance. Moreover, former relationships are left behind, which makes immigrants more likely to experience loneliness. This study focuses particularly on Venezuelan immigrants by incorporating cultural aspects into a solution aimed at reducing loneliness and increasing social connectedness. Among Venezuelans, coffee is a staple of their daily routine and their favorite social beverage. We propose KEPEIN, a coffee maker-shaped interface to transfer a sense of presence and share coffee over distance. Through an experimental study, we evaluated the user’s perception and reaction when communicating through the interface. The results show potential added value to communication by including KEPEIN in a traditional remote interaction scenario. We discuss the benefits and limitations of this type of tangible communication interface and the importance of incorporating culture into the design of solutions for immigrants.
Detecting Human Attitudes through Interactions with Responsive Environments

The paper is based on the developments of the research project “Paradigms of Ubiquitous Computing”, funded by the Swiss National Science Foundation (SNSF, 100016_185436 / 1), 2019-23. It investigates the impact of environmentally embedded sensor-actuator systems on humans. With a critical stance, we examine human-machine interfaces to make quantitative statements about behavior patterns and attitudes, only based on their physical interactions with a responsive environment. By staging different paradigms of Ubiquitous Computing in an experimental setup and evaluating them with test persons, we aim to gain insights into the human experience and appropriation of immersive and sometimes challenging situations. The artistic approach is based on strategies of New Media Art and Speculative Design and is not aligned with processes commonly used in applied research and development. The evaluation design is based on mixed methods with a strong emphasis on semantic differentials to quantify user interactions with electronically enhanced devices and furnishings. The focus is on interaction design strategies and evaluation design methods.
The Green Notebook - A Co-Creativity Partner for Facilitating Sustainability Reflection

AI is becoming increasingly popular in artistic work. Yet tools for calculating environmental impact of AI are more adapted for other contexts than creative practices, making them sometimes hard to comprehend for the non-expert. In this study, based on interviews with AI artists, a design artifact called The Green Notebook was developed: a physical notebook where the AI artist could discuss ideas and receive feedback of their expected environmental impact. The conversational experience between the artist and the interface was informed by online content analysis of artistic work processes. The Notebook was explored and assessed with five artists in Wizard-of-Oz and focus group studies. Generally, the participants found a co-creation process with the enhanced ability to reflect on sustainability an accessible way to engage with sustainability considerations of their AI artistic practices. We provide insights of the Notebook’s perceived role and the conversational strategies used by the artists. Furthermore, we discuss trade-offs between politeness vs. efficiency and focus vs. integration to inform future research.
Virtual Rehearsal Suite: An Environment and Framework for Virtual Performance Practice

Contemporary performance artists use Virtual Reality (VR) tools to create immersive narratives and extend the boundaries of traditional performance mediums. As the medium evolves, performance practice is changing with it. Our work explores ways to leverage VR to support the creative process by introducing the Virtual Rehearsal Suite (VRS) that provides users with the experience of a large-scale rehearsal or performance environment while occupying limited physical space and minimal real-world obstructions. In this paper, we discuss findings from scene study experiments conducted within the VRS. In addition, we contribute our thresholding protocols a framework designed to support user transitions into and out of VR experiences. Our integrated approach to digital performance practice and creative collaboration combines traditional and contemporary acting techniques with HCI research to harness the innovative capabilities of virtual reality technologies creating accessible, immersive experiences for actors while facilitating user presence through state change protocols.
Enhancing Arabic Content Generation with Prompt Augmentation Using Integrated GPT and Text-to-Image Models

With the current and continuous advancements in the field of text-to-image modeling, it has become critical to design prompts that make the best of these model capabilities and guides them to generate the most desirable images, and thus the field of prompt engineering has emerged. Here, we study a method to use prompt engineering to enhance text-to-image model representation of the Arabic culture. This work proposes a simple, novel approach for prompt engineering that uses the domain knowledge of a state-of-the-art language model, GPT, to perform the task of prompt augmentation, where a simple, initial prompt is used to generate multiple, more detailed prompts related to the Arabic culture from multiple categories through a GPT model through a process known as in-context learning. The augmented prompts are then used to generate images enhanced for the Arabic culture. We perform multiple experiments with a number of participants to evaluate the performance of the proposed method, which shows promising results, specially for generating prompts that are more inclusive of the different Arabic countries and with a wider variety in terms of image subjects, where we find that our proposed method generates image with more variety 85 % of the time and are more inclusive of the Arabic countries more than 72.66 % of the time, compared to the direct approach.
Human-Centered and AI-driven Generation of 6-DoF Extended Reality

In order to unlock the full potential of Extended Reality (XR) and its application to societal sectors such as health (e.g., training) or Industry 5.0 (e.g., remote control of infrastructure) there is a need for very realistic environments to enhance the presence of the user. However, current photo-realistic content generation methods (such as Light Fields) require a massive amount of data transmission (i.e., ultra-high bandwidths) and extreme computational power for displaying. Thus, they are not suited for interactive immersive and realistic applications. In this research, we hypothesize that is possible to generate realistic dynamic 3D environments by means of Deep Generative Networks. The work will consist of two parts: (1) a computer vision system that generates the 3D environment based on 2D images, and (2) a Human-Computer Interaction system (HCI) that predicts Region of Interest (RoI) for efficient 3D rendering, subjective and objective assessment of user perception (by means of presence) to enhance the 3D scene quality. This work aims to gain insights into how well deep generative methods can create realistic and immersive environments. This will significantly help future developments in realistic and immersive XR content creation.
LIFT - A System to Create Mixed 360° Video and 3D Content for Live Immersive Virtual Field Trip

Our paper presents LIFT, a system that enables educators to create immersive virtual field trip experiences for their students. LIFT overcomes the challenges of enabling non-technical educators to create their own content and allows educators to act as guides during the immersive experience. The system combines live-streamed 360° video, 3D models, and live instruction to create collaborative virtual field trips. To evaluate LIFT, we developed a field trip with biology educators from the University of Central Florida(UCF) and showcased it at a science festival. Our results suggest that LIFT can help educators create immersive educational content while out in the field. However, our pilot observational study at the museum highlighted the need for further research to explore the instructional design of mixed immersive content created with LIFT. Overall, our work provides an application development framework for educators to create immersive, hands-on field trip experiences.
Enhancing VR Gaming Experience using Computational Attention Models and Eye-Tracking

This study explores the potential of enhancing interaction experiences, such as virtual reality (VR) games, through the use of computational attention models. Our proposed approach utilizes a saliency map generated by attention models to dynamically adjust game difficulty levels and to help in the game level design, resulting in a more immersive and engaging experience for users. To inform the development of this approach, we present an experimental setup that is able tp collect data in a VR environment and intends to be able to validate the adaptation of attention models to this domain. Through this work, we aim to create a framework for VR game design that leverages attention models to offer a new level of immersion and engagement for users. We believe our contributions have significant potential to enhance VR experiences and advance the field of game design.
Analysing and Understanding Embodied Interactions in Virtual Reality Systems

Virtual reality (VR) offers opportunities in human-computer interaction research, to embody users in immersive environments and observe how they interact with 3D scenarios under well-controlled environments. VR content has stronger influences on users physical and emotional states as compared to traditional 2D media, however, a fuller understanding of this kind of embodied interaction is currently limited by the extent to which attention and behaviour can be observed in a VR environment, and the accuracy at which these observations can be interpreted as, and mapped to, real-world interactions and intentions. This thesis aims at the creation of a system to help designers in the understanding of the embodied user experience in VR environment: how they feel, what is their intentions when interacting with a certain object, provide them guidance based on their needs and attention. Controlled guided scenarios will help to reduce the gap of perception between the designer building an experience, and the user living it, leading to more efficient behaviour analysis in VR systems.
LatentKeypointGAN: Controlling Images via Latent Keypoints

Generative adversarial networks (GANs) can now generate photorealistic images. However, how to best control the image content remains an open challenge. We introduce LatentKeypointGAN, a two-stage GAN internally conditioned on a set of keypoints and associated appearance embeddings providing control of the position and style of the generated objects and their respective parts. A major difficulty that we address is disentangling the image into spatial and appearance factors with little domain knowledge and supervision signals. We demonstrate in a user study and quantitative experiments that LatentKeypointGAN provides an interpretable latent space that can be used to re-arrange the generated images by re-positioning, adding, removing, and exchanging keypoint embeddings, such as generating portraits by combining the eyes, and mouth from different images. Notably, our method does not require labels as it is self-supervised and thereby applies to diverse application domains, such as editing portraits, indoor rooms, and full-body human poses. In addition, the explicit generation of keypoints and matching images enables a new, GAN-based method for unsupervised keypoint detection.
Identifying the Developmental Challenges of Creating Virtual Reality Theatre

Virtual Reality Theatre continues to grow as a form of digital creative output for theatre practitioners. However, understanding the common challenges faced during the development of productions by practitioners and the barriers to entry produced by the complexity of platforms is under-researched. This paper provides an in-depth analysis of several challenges identified through semi-structured interviews of practitioners and a thematic review.
Generating Utterances for Companion Robots using Television Program Subtitles

This study presents a method for generating utterances for companion robots that watch TV with people, using TV program subtitles. To enable the robot to automatically generate relevant utterances while watching TV, we created a dataset of approximately 12,000 utterances that were manually added to the collected TV subtitles. Using this dataset, we fine-tuned a large-scale language model to construct an utterance generation model. The proposed model generates utterances based on multiple keywords extracted from the subtitles as topics, while also taking into account the context of the subtitles by inputting them. The evaluation of the generated utterances revealed that approximately 88% of the sentences were natural Japanese, and approximately 75% were relevant and natural in the context of the TV program. Moreover, approximately 99% of the sentences contained the extracted keywords, indicating that our proposed method can generate diverse and contextually appropriate utterances containing the targeted topics. These findings provide evidence of the effectiveness of our approach in generating natural utterances for companion robots that watch TV with people.
Kaskade-CNN and NB-KDE-CNN: CNN Architectures with KDE and Probabilistic Reasoning Layers

In recent years, several differentiable histogram layers have been proposed which are suitable for end-to-end training in a convolutional neural network (CNN) model. In this paper we examine the use of a closely related kernel density estimation (KDE) layer to represent a probability distribution in combination with two novel layers for principled probabilistic reasoning. One of these probabilistic reasoning layers is based on naive Bayes, and the other is based on the nonparametric Kolmogorov-Smirnov (KS) hypothesis test. We show excellent texture classification results using compact “NB-KDE-CNN” and “Kaskade-CNN” networks which combine KDE and probabilistic reasoning layers.
A Dataset of Gaze and Mouse Patterns in the Context of Facial Expression Recognition

Facial expression recognition is an important and challenging task for both the computer vision and affective computing communities, and even more specifically in the context of multimedia applications, where audience understanding is of particular interest. Recent data-oriented approaches have created the need for large-scale annotated datasets. However, most existing datasets present some weaknesses, because of the collecting methods used. In order to further highlight these issues, we investigate in this work how human visual attention is deployed when performing a facial expression recognition task. To do so, we carried out several complementary experiments, using the eye-tracking technology, as well as the BubbleView metaphor, both under laboratory and crowdsourcing settings. We show significant variations in gaze patterns depending on the emotion represented, but also on the difficulty of the task, i.e., whether the emotion is correctly recognised or not. Moreover, we use these results to propose recommendations on the ways to collect label data for facial expression recognition datasets.
Generalized Kronecker-based Adapters for Parameter-efficient Fine-tuning of Vision Transformers

While large transformer-based vision models have achieved remarkable performance on a variety of Computer Vision (CV) applications, they are cumbersome to fine-tune for target tasks. However, such models, pre-trained on large-scale datasets, are suspected to exhibit a low intrinsic dimension during fine-tuning. This suggests fewer parameters can be optimized during the fine-tuning stage to achieve a similar level of performance. Parameter-efficient Fine-tuning (PEFT) methods have been introduced to leverage this property by limiting the optimization to a subspace of the trainable parameters during fine-tuning. Utilizing low-rank projection matrices as adapter modules has been shown to be a good approach. However, it has been predominantly explored for transformer-based language models and their linear layers, while there is an increasing interest to use convolutional layers jointly with linear layers in transformer-based vision models. In this work, we introduce adapters to bypass gradient updating of the pre-trained transformer blocks of vision models during fine-tuning. Our adapters are designed using sequences of low-rank Kronecker products that provide a factorized representation of large tensors, resulting in an efficient fine-tuning parameter space for fine-tuning. Since our adapter weights are merged with the original pre-trained model weights, the proposed PEFT method does not add extra computations or memory footprints in the inference stage. Experimental results for image classification using ViT i.e., the original transformer-based vision model, and the convolution-enhanced transformer models; CeiT, and CvT substantiate the effectiveness of our proposed approach. Our proposed method outperforms state-of-the-art PEFT methods.
Transformer-Based Human Action Recognition with Dynamic Feature Selection

Human action recognition in videos is an important task of computer vision that aims to automatically recognize and classify human actions in video sequences. However, accurately recognizing human actions can be challenging due to the complexity and variability of human motion and appearance. In this paper, we propose ActiViT, a novel approach for human action recognition in videos based on a Transformer architecture. Unlike existing methods that rely on convolutional or recurrent layers, our model is entirely based on the Transformer encoder, enabling us to leverage valuable information in action image patches features. We demonstrate that by dynamically selecting key patches guided by specific human poses, our model learns informative features useful for distinguishing between different actions. Our experimental results on real-world datasets convincingly demonstrate the effectiveness of our model and the importance of selecting discriminative key poses for action recognition.