OpenRaiser/Intern-Atlas huggingface dataset:
Mixed Reality Heritage Performance As a Decolonising Tool for Heritage Sites
Venue: arXiv cs.HC 2024
Year: 2024

In this paper we introduce two world-first Mixed Reality (MR) experiences that fuse smart AR glasses and live theatre and take place in a heritage site with the purpose to reveal the site's hidden and difficult histories about slavery. We term these unique general audience experiences Mixed Reality Heritage Performances (MRHP). Along with the development of our initial two performances we designed and developed a tool and guidelines that can help heritage organisations with their decolonising process by critically engaging the public with under-represented voices and viewpoints of troubled European and colonial narratives. The evaluations showed the embodied and affective potential of MRHP to attract and educate heritage audiences visitors. Insights of the design process are being formulated into an extensive design toolkit that aims to support experience design, theatre and heritage professionals to collaboratively carry out similar projects.
SealMates: Supporting Communication in Video Conferencing using a Collective Behavior-Driven Avatar
Venue: arXiv cs.HC 2024
Year: 2024

The limited nonverbal cues and spatially distributed nature of remote communication make it challenging for unacquainted members to be expressive during social interactions over video conferencing. Though it enables seeing others' facial expressions, the visual feedback can instead lead to unexpected self-focus, resulting in users missing cues for others to engage in the conversation equally. To support expressive communication and equal participation among unacquainted counterparts, we propose SealMates, a behavior-driven avatar in which the avatar infers the engagement level of the group based on collective gaze and speech patterns and then moves across interlocutors' windows in the video conferencing. By conducting a controlled experiment with 15 groups of triads, we found the avatar's movement encouraged people to experience more self-disclosure and made them perceive everyone was equally engaged in the conversation than when there was no behavior-driven avatar. We discuss how a behavior-driven avatar influences distributed members' perceptions and the implications of avatar-mediated communication for future platforms.
Multimodal Emotion Recognition by Fusing Video Semantic in MOOC Learning Scenarios
Venue: arXiv cs.MM 2024
Year: 2024

In the Massive Open Online Courses (MOOC) learning scenario, the semantic information of instructional videos has a crucial impact on learners' emotional state. Learners mainly acquire knowledge by watching instructional videos, and the semantic information in the videos directly affects learners' emotional states. However, few studies have paid attention to the potential influence of the semantic information of instructional videos on learners' emotional states. To deeply explore the impact of video semantic information on learners' emotions, this paper innovatively proposes a multimodal emotion recognition method by fusing video semantic information and physiological signals. We generate video descriptions through a pre-trained large language model (LLM) to obtain high-level semantic information about instructional videos. Using the cross-attention mechanism for modal interaction, the semantic information is fused with the eye movement and PhotoPlethysmoGraphy (PPG) signals to obtain the features containing the critical information of the three modes. The accurate recognition of learners' emotional states is realized through the emotion classifier. The experimental results show that our method has significantly improved emotion recognition performance, providing a new perspective and efficient method for emotion recognition research in MOOC learning scenarios. The method proposed in this paper not only contributes to a deeper understanding of the impact of instructional videos on learners' emotional states but also provides a beneficial reference for future research on emotion recognition in MOOC learning scenarios.
Fine-Grained Side Information Guided Dual-Prompts for Zero-Shot Skeleton Action Recognition
Venue: arXiv cs.CV 2024
Year: 2024

Skeleton-based zero-shot action recognition aims to recognize unknown human actions based on the learned priors of the known skeleton-based actions and a semantic descriptor space shared by both known and unknown categories. However, previous works focus on establishing the bridges between the known skeleton representation space and semantic descriptions space at the coarse-grained level for recognizing unknown action categories, ignoring the fine-grained alignment of these two spaces, resulting in suboptimal performance in distinguishing high-similarity action categories. To address these challenges, we propose a novel method via Side information and dual-prompts learning for skeleton-based zero-shot action recognition (STAR) at the fine-grained level. Specifically, 1) we decompose the skeleton into several parts based on its topology structure and introduce the side information concerning multi-part descriptions of human body movements for alignment between the skeleton and the semantic space at the fine-grained level; 2) we design the visual-attribute and semantic-part prompts to improve the intra-class compactness within the skeleton space and inter-class separability within the semantic space, respectively, to distinguish the high-similarity actions. Extensive experiments show that our method achieves state-of-the-art performance in ZSL and GZSL settings on NTU RGB+D, NTU RGB+D 120, and PKU-MMD datasets.
Neural Fault Injection: Generating Software Faults from Natural Language
Venue: arXiv cs.SE 2024
Year: 2024

Traditional software fault injection methods, while foundational, face limitations in adequately representing real-world faults, offering customization, and requiring significant manual effort and expertise. This paper introduces a novel methodology that harnesses the capabilities of Large Language Models (LLMs) augmented with Reinforcement Learning from Human Feedback (RLHF) to overcome these challenges. The usage of RLHF emphasizes an iterative refinement process, allowing testers to provide feedback on generated faults, which is then used to enhance the LLM's fault generation capabilities, ensuring the generation of fault scenarios that closely mirror actual operational risks. This innovative methodology aims to significantly reduce the manual effort involved in crafting fault scenarios as it allows testers to focus on higher-level testing strategies, hence paving the way to new possibilities for enhancing the dependability of software systems.
Leveraging Data Augmentation for Process Information Extraction
Venue: arXiv cs.CL 2024
Year: 2024

Business Process Modeling projects often require formal process models as a central component. High costs associated with the creation of such formal process models motivated many different fields of research aimed at automated generation of process models from readily available data. These include process mining on event logs, and generating business process models from natural language texts. Research in the latter field is regularly faced with the problem of limited data availability, hindering both evaluation and development of new techniques, especially learning-based ones. To overcome this data scarcity issue, in this paper we investigate the application of data augmentation for natural language text data. Data augmentation methods are well established in machine learning for creating new, synthetic data without human assistance. We find that many of these methods are applicable to the task of business process information extraction, improving the accuracy of extraction. Our study shows, that data augmentation is an important component in enabling machine learning methods for the task of business process model generation from natural language text, where currently mostly rule-based systems are still state of the art. Simple data augmentation techniques improved the $F_1$ score of mention extraction by 2.9 percentage points, and the $F_1$ of relation extraction by $4.5$. To better understand how data augmentation alters human annotated texts, we analyze the resulting text, visualizing and discussing the properties of augmented textual data. We make all code and experiments results publicly available.
SFSORT: Scene Features-based Simple Online Real-Time Tracker
Venue: arXiv cs.CV 2024
Year: 2024

This paper introduces SFSORT, the world's fastest multi-object tracking system based on experiments conducted on MOT Challenge datasets. To achieve an accurate and computationally efficient tracker, this paper employs a tracking-by-detection method, following the online real-time tracking approach established in prior literature. By introducing a novel cost function called the Bounding Box Similarity Index, this work eliminates the Kalman Filter, leading to reduced computational requirements. Additionally, this paper demonstrates the impact of scene features on enhancing object-track association and improving track post-processing. Using a 2.2 GHz Intel Xeon CPU, the proposed method achieves an HOTA of 61.7\% with a processing speed of 2242 Hz on the MOT17 dataset and an HOTA of 60.9\% with a processing speed of 304 Hz on the MOT20 dataset. The tracker's source code, fine-tuned object detection model, and tutorials are available at \url{https://github.com/gitmehrdad/SFSORT}.
Lifting with Inner Functions of Polynomial Discrepancy
Venue: arXiv cs.CC 2024
Year: 2024

Lifting theorems are theorems that bound the communication complexity of a composed function $f\circ g^{n}$ in terms of the query complexity of $f$ and the communication complexity of $g$. Such theorems constitute a powerful generalization of direct-sum theorems for $g$, and have seen numerous applications in recent years. We prove a new lifting theorem that works for every two functions $f,g$ such that the discrepancy of $g$ is at most inverse polynomial in the input length of $f$. Our result is a significant generalization of the known direct-sum theorem for discrepancy, and extends the range of inner functions $g$ for which lifting theorems hold.
Separated Attention: An Improved Cycle GAN Based Under Water Image Enhancement Method
Venue: arXiv cs.CV 2024
Year: 2024

In this paper we have present an improved Cycle GAN based model for under water image enhancement. We have utilized the cycle consistent learning technique of the state-of-the-art Cycle GAN model with modification in the loss function in terms of depth-oriented attention which enhance the contrast of the overall image, keeping global content, color, local texture, and style information intact. We trained the Cycle GAN model with the modified loss functions on the benchmarked Enhancing Underwater Visual Perception (EUPV) dataset a large dataset including paired and unpaired sets of underwater images (poor and good quality) taken with seven distinct cameras in a range of visibility situation during research on ocean exploration and human-robot cooperation. In addition, we perform qualitative and quantitative evaluation which supports the given technique applied and provided a better contrast enhancement model of underwater imagery. More significantly, the upgraded images provide better results from conventional models and further for under water navigation, pose estimation, saliency prediction, object detection and tracking. The results validate the appropriateness of the model for autonomous underwater vehicles (AUV) in visual navigation.
rollama: An R package for using generative large language models through Ollama
Venue: arXiv cs.CL 2024
Year: 2024

rollama is an R package that wraps the Ollama API, which allows you to run different Generative Large Language Models (GLLM) locally. The package and learning material focus on making it easy to use Ollama for annotating textual or imagine data with open-source models as well as use these models for document embedding. But users can use or extend rollama to do essentially anything else that is possible through OpenAI's API, yet more private, reproducible and for free.
On Naisargik Images of Varshamov-Tenengolts and Helberg Codes
Venue: arXiv cs.IT 2024
Year: 2024

The VT and Helberg codes, both in binary and non-binary forms, stand as elegant solutions for rectifying insertion and deletion errors. In this paper we consider the quaternary versions of these codes. It is well known that many optimal binary non-linear codes like Kerdock and Prepreta can be depicted as Gray images (isometry) of codes defined over $\mathbb{Z}_4$. Thus a natural question arises: Can we find similar maps between quaternary and binary spaces which gives interesting properties when applied to the VT and Helberg codes. We found several such maps called Naisargik (natural) maps and we study the images of quaternary VT and Helberg codes under these maps. Naisargik and inverse Naisargik images gives interesting error-correcting properties for VT and Helberg codes. If two Naisargik images of VT code generates an intersecting one deletion sphere, then the images holds the same weights. A quaternary Helberg code designed to correct $s$ deletions can effectively rectify $s+1$ deletion errors when considering its Naisargik image, and $s$-deletion correcting binary Helberg code can corrects $\lfloor\frac{s}{2}\rfloor$ errors with inverse Naisargik image.
Integrating On-demand Ride-sharing with Mass Transit at-Scale
Venue: arXiv cs.DM 2024
Year: 2024

We are in the midst of a technology-driven transformation of the urban mobility landscape. However, unfortunately these new innovations are still dominated by car-centric personal mobility, which leads to concerns such as environmental sustainability, congestion, and equity. On the other hand, mass transit provides a means to move large amounts of travelers very efficiently, but is not very versatile and depends on an adequate concentration of demand. In this context, our overarching goal is to explore opportunities for new technologies such as ride-sharing to integrate with mass transit and provide a better service. More specifically, we envision a hybrid system that uses on-demand shuttles in conjunction with mass transit to move passengers efficiently, and provide an algorithmic framework for operational optimization. Our approach extends a state-of-the-art trip-vehicle assignment model to the multi-modal setting, where we develop a new integer-linear programming formulation to solve the problem efficiently. A comprehensive study covering five major cities in the United States based on real-world data is carried out to verify the advantages of such a system and the effectiveness of our algorithms. We show that our hybrid system provides significant improvements in comparison to a purely on-demand model by exploiting the efficiencies of the mass transit system.
Reflectance Estimation for Proximity Sensing by Vision-Language Models: Utilizing Distributional Semantics for Low-Level Cognition in Robotics
Venue: arXiv cs.RO 2024
Year: 2024

Large language models (LLMs) and vision-language models (VLMs) have been increasingly used in robotics for high-level cognition, but their use for low-level cognition, such as interpreting sensor information, remains underexplored. In robotic grasping, estimating the reflectance of objects is crucial for successful grasping, as it significantly impacts the distance measured by proximity sensors. We investigate whether LLMs can estimate reflectance from object names alone, leveraging the embedded human knowledge in distributional semantics, and if the latent structure of language in VLMs positively affects image-based reflectance estimation. In this paper, we verify that 1) LLMs such as GPT-3.5 and GPT-4 can estimate an object's reflectance using only text as input; and 2) VLMs such as CLIP can increase their generalization capabilities in reflectance estimation from images. Our experiments show that GPT-4 can estimate an object's reflectance using only text input with a mean error of 14.7%, lower than the image-only ResNet. Moreover, CLIP achieved the lowest mean error of 11.8%, while GPT-3.5 obtained a competitive 19.9% compared to ResNet's 17.8%. These results suggest that the distributional semantics in LLMs and VLMs increases their generalization capabilities, and the knowledge acquired by VLMs benefits from the latent structure of language.
Reframing the Mind-Body Picture: Applying Formal Systems to the Relationship of Mind and Matter
Venue: arXiv cs.AI 2024
Year: 2024

This paper aims to show that a simple framework, utilizing basic formalisms from set theory and category theory, can clarify and inform our theories of the relation between mind and matter.
Generating consistent PDDL domains with Large Language Models
Venue: arXiv cs.RO 2024
Year: 2024

Large Language Models (LLMs) are capable of transforming natural language domain descriptions into plausibly looking PDDL markup. However, ensuring that actions are consistent within domains still remains a challenging task. In this paper we present a novel concept to significantly improve the quality of LLM-generated PDDL models by performing automated consistency checking during the generation process. Although the proposed consistency checking strategies still can't guarantee absolute correctness of generated models, they can serve as valuable source of feedback reducing the amount of correction efforts expected from a human in the loop. We demonstrate the capabilities of our error detection approach on a number of classical and custom planning domains (logistics, gripper, tyreworld, household, pizza).
An Application Layer Multi-Hop Collective Perception Service for Vehicular Adhoc Networks
Venue: arXiv cs.NI 2024
Year: 2024

Collective Perception will play a crucial role for ensuring vehicular safety in the near future, enabling the sharing of local perceived objects with other Intelligent Transport System Stations (ITS-Ss). However, at the beginning of the roll-out, low market penetration rates are expected. This paper proposes and evaluates an application layer multi-hop Collective Perception Service (CPS) for vehicular ad-hoc networks. The goal is to improve the environmental awareness ratio in scenarios with low CPS market penetration. In such scenarios, the CPS service without forwarding enabled struggles to achieve complete awareness. A decentralized application layer forwarding algorithm is presented that shares perceived object information across multiple hops while maintaining a low age of information. The proposed approach is compared against standard CPS with no forwarding and CPS with geographically-scoped (GBC) multi-hop forwarding. Simulations according to standards of the European Telecommunications Standards Institute (ETSI) demonstrate that the application layer forwarding achieves near 100% awareness at 10% penetration rate versus 92% for standard CPS. The awareness improvement comes with moderate channel load, unlike GBC forwarding which quickly saturates the channel. The median age of information remains below 80 ms for the proposed scheme, enabling real-time CPS operation. Our application layer multi-hop approach effectively improves environmental awareness during initial CPS deployment while aligning with latency and channel load requirements.
Voice-Assisted Real-Time Traffic Sign Recognition System Using Convolutional Neural Network
Venue: arXiv cs.CV 2024
Year: 2024

Traffic signs are important in communicating information to drivers. Thus, comprehension of traffic signs is essential for road safety and ignorance may result in road accidents. Traffic sign detection has been a research spotlight over the past few decades. Real-time and accurate detections are the preliminaries of robust traffic sign detection system which is yet to be achieved. This study presents a voice-assisted real-time traffic sign recognition system which is capable of assisting drivers. This system functions under two subsystems. Initially, the detection and recognition of the traffic signs are carried out using a trained Convolutional Neural Network (CNN). After recognizing the specific traffic sign, it is narrated to the driver as a voice message using a text-to-speech engine. An efficient CNN model for a benchmark dataset is developed for real-time detection and recognition using Deep Learning techniques. The advantage of this system is that even if the driver misses a traffic sign, or does not look at the traffic sign, or is unable to comprehend the sign, the system detects it and narrates it to the driver. A system of this type is also important in the development of autonomous vehicles.
Question Generation in Knowledge-Driven Dialog: Explainability and Evaluation
Venue: arXiv cs.CL 2024
Year: 2024

We explore question generation in the context of knowledge-grounded dialogs focusing on explainability and evaluation. Inspired by previous work on planning-based summarisation, we present a model which instead of directly generating a question, sequentially predicts first a fact then a question. We evaluate our approach on 37k test dialogs adapted from the KGConv dataset and we show that, although more demanding in terms of inference, our approach performs on par with a standard model which solely generates a question while allowing for a detailed referenceless evaluation of the model behaviour in terms of relevance, factuality and pronominalisation.
The Power of Properties: Uncovering the Influential Factors in Emotion Classification
Venue: arXiv cs.CV 2024
Year: 2024

Facial expression-based human emotion recognition is a critical research area in psychology and medicine. State-of-the-art classification performance is only reached by end-to-end trained neural networks. Nevertheless, such black-box models lack transparency in their decision-making processes, prompting efforts to ascertain the rules that underlie classifiers' decisions. Analyzing single inputs alone fails to expose systematic learned biases. These biases can be characterized as facial properties summarizing abstract information like age or medical conditions. Therefore, understanding a model's prediction behavior requires an analysis rooted in causality along such selected properties. We demonstrate that up to 91.25% of classifier output behavior changes are statistically significant concerning basic properties. Among those are age, gender, and facial symmetry. Furthermore, the medical usage of surface electromyography significantly influences emotion prediction. We introduce a workflow to evaluate explicit properties and their impact. These insights might help medical professionals select and apply classifiers regarding their specialized data and properties.