OpenRaiser/Intern-Atlas huggingface dataset:
Back to Supervision: Boosting Word Boundary Detection through Frame Classification
Venue: arXiv cs.LG 2024
Year: 2024

Speech segmentation at both word and phoneme levels is crucial for various speech processing tasks. It significantly aids in extracting meaningful units from an utterance, thus enabling the generation of discrete elements. In this work we propose a model-agnostic framework to perform word boundary detection in a supervised manner also employing a labels augmentation technique and an output-frame selection strategy. We trained and tested on the Buckeye dataset and only tested on TIMIT one, using state-of-the-art encoder models, including pre-trained solutions (Wav2Vec 2.0 and HuBERT), as well as convolutional and convolutional recurrent networks. Our method, with the HuBERT encoder, surpasses the performance of other state-of-the-art architectures, whether trained in supervised or self-supervised settings on the same datasets. Specifically, we achieved F-values of 0.8427 on the Buckeye dataset and 0.7436 on the TIMIT dataset, along with R-values of 0.8489 and 0.7807, respectively. These results establish a new state-of-the-art for both datasets. Beyond the immediate task, our approach offers a robust and efficient preprocessing method for future research in audio tokenization.
VeriGraph: Scene Graphs for Execution Verifiable Robot Planning
Venue: arXiv cs.RO 2024
Year: 2024

Recent advancements in vision-language models (VLMs) offer potential for robot task planning, but challenges remain due to VLMs' tendency to generate incorrect action sequences. To address these limitations, we propose VeriGraph, a novel framework that integrates VLMs for robotic planning while verifying action feasibility. VeriGraph employs scene graphs as an intermediate representation, capturing key objects and spatial relationships to improve plan verification and refinement. The system generates a scene graph from input images and uses it to iteratively check and correct action sequences generated by an LLM-based task planner, ensuring constraints are respected and actions are executable. Our approach significantly enhances task completion rates across diverse manipulation scenarios, outperforming baseline methods by 58% for language-based tasks and 30% for image-based tasks.
Goetterfunke: Creativity in Machinae Sapiens. About the Qualitative Shift in Generative AI with a Focus on Text-To-Image
Venue: arXiv cs.HC 2024
Year: 2024

The year 2022 marks a watershed in technology, and arguably in human history, with the release of powerful generative AIs capable of convincingly performing creative tasks. With the help of these systems, anyone can create something that would previously have been considered a remarkable work of art. In human-AI collaboration, the computer seems to have become more than a tool. Many who have made their first contact with current generative AIs see them as "creativity machines" while for others the term "machine creativity" remains an oxymoron. This article is about (the possibility of) creativity in computers within the current Machine Learning paradigm. It outlines some of the key concepts behind the technologies and the innovations that have contributed to this qualitative shift, with a focus on text-to-image systems. The nature of Artificial Creativity as such is discussed, as well as what this might mean for art. AI may become a responsible collaborator with elements of independent machine authorship in the artistic process.
Love in Action: Gamifying Public Video Cameras for Fostering Social Relationships in Real World
Venue: arXiv cs.HC 2024
Year: 2024

In this paper, we create "Love in Action" (LIA), a body language-based social game utilizing video cameras installed in public spaces to enhance social relationships in real-world. In the game, participants assume dual roles, i.e., requesters, who issue social requests, and performers, who respond social requests through performing specified body languages. To mediate the communication between participants, we build an AI-enhanced video analysis system incorporating multiple visual analysis modules like person detection, attribute recognition, and action recognition, to assess the performer's body language quality. A two-week field study involving 27 participants shows significant improvements in their social friendships, as indicated by self-reported questionnaires. Moreover, user experiences are investigated to highlight the potential of public video cameras as a novel communication medium for socializing in public spaces.
Directing Avatars in Live Performances -- an Autonomy Simulacrum of Virtual Entities
Venue: arXiv cs.HC 2024
Year: 2024

We propose to review the main stages in the computer history of virtual actors, with a view to the exploration of virtual reality and discussion on different approaches to human simulation. The notion of autonomy emerges as a key issue for the virtual entities. We then explore one way of building elements of autonomy and conclude with an example of avatar stage direction leading to a simulacrum of autonomy in a live performance.
Evolutionary Multi-agent Reinforcement Learning in Group Social Dilemmas
Venue: arXiv cs.MA 2024
Year: 2024

Reinforcement learning (RL) is a powerful machine learning technique that has been successfully applied to a wide variety of problems. However, it can be unpredictable and produce suboptimal results in complicated learning environments. This is especially true when multiple agents learn simultaneously, which creates a complex system that is often analytically intractable. Our work considers the fundamental framework of Q-learning in Public Goods Games, where RL individuals must work together to achieve a common goal. This setting allows us to study the tragedy of the commons and free rider effects in AI cooperation, an emerging field with potential to resolve challenging obstacles to the wider application of artificial intelligence. While this social dilemma has been mainly investigated through traditional and evolutionary game theory, our approach bridges the gap between these two by studying agents with an intermediate level of intelligence. Specifically, we consider the influence of learning parameters on cooperation levels in simulations and a limiting system of differential equations, as well as the effect of evolutionary pressures on exploration rate in both of these models. We find selection for higher and lower levels of exploration, as well as attracting values, and a condition that separates these in a restricted class of games. Our work enhances the theoretical understanding of evolutionary Q-learning, and extends our knowledge of the evolution of machine behavior in social dilemmas.
Gamification and AI: Enhancing User Engagement through Intelligent Systems
Venue: arXiv cs.HC 2024
Year: 2024

Gamification applies game mechanics to non-game environments to motivate and engage users. Artificial Intelligence (AI) offers powerful tools for personalizing and optimizing gamification, adapting to users' needs, preferences, and performance levels. By integrating AI with gamification, systems can dynamically adjust game mechanics, deliver personalized feedback, and predict user behavior, significantly enhancing the effectiveness of gamification efforts. This paper examines the intersection of gamification and AI, exploring AI's methods to optimize gamified experiences and proposing mathematical models for adaptive and predictive gamification.
Prolegomena to a Post-Aesthetics of Artificial Imaginations
Venue: arXiv cs.HC 2024
Year: 2024

The acceleration of the use of generative artificial intelligences (AI), since 2015 and the turning point operated by Deepdream, tends to obscure a real analysis of what could be defined as artificial imagination. AIs are either reduced to simple instruments or thought of according to a form of techno-theologism. Our research tends to suspend any form of judgment in order to phenomenally grasp the emergence of these AIs. By taking up the question of Hegel's aesthetics and of art as the free production of the mind, but by moving it towards the question of generative AIs and therefore of a post-aesthetics, this article will show the phenomenal specificity of images generatedby AI.
A Survey on Importance of Homophones Spelling Correction Model for Khmer Authors
Venue: arXiv cs.CL 2024
Year: 2024

Homophones present a significant challenge to authors in any languages due to their similarities of pronunciations but different meanings and spellings. This issue is particularly pronounced in the Khmer language, rich in homophones due to its complex structure and extensive character set. This research aims to address the difficulties faced by Khmer authors when using homophones in their writing and proposes potential solutions based on an extensive literature review and survey analysis. A survey of 108 Khmer native speakers, including students, employees, and professionals, revealed that many frequently encounter challenges with homophones in their writing, often struggling to choose the correct word based on context. The survey also highlighted the absence of effective tools to address homophone errors in Khmer, which complicates the writing process. Additionally, a review of existing studies on spelling correction in other languages, such as English, Azerbaijani, and Bangla, identified a lack of research focused specifically on homophones, particularly in the Khmer language. In summary, this research highlights the necessity for a specialized tool to address Khmer homophone errors. By bridging current gaps in research and available resources, such a tool would enhance the confidence and accuracy of Khmer authors in their writing, thereby contributing to the enrichment and preservation of the language. Continued efforts in this domain are essential for ensuring that Khmer can leverage advancements in technology and linguistics effectively.
Boolean-aware Boolean Circuit Classification: A Comprehensive Study on Graph Neural Network
Venue: arXiv cs.LG 2024
Year: 2024

Boolean circuit is a computational graph that consists of the dynamic directed graph structure and static functionality. The commonly used logic optimization and Boolean matching-based transformation can change the behavior of the Boolean circuit for its graph structure and functionality in logic synthesis. The graph structure-based Boolean circuit classification can be grouped into the graph classification task, however, the functionality-based Boolean circuit classification remains an open problem for further research. In this paper, we first define the proposed matching-equivalent class based on its ``Boolean-aware'' property. The Boolean circuits in the proposed class can be transformed into each other. Then, we present a commonly study framework based on graph neural network~(GNN) to analyze the key factors that can affect the Boolean-aware Boolean circuit classification. The empirical experiment results verify the proposed analysis, and it also shows the direction and opportunity to improve the proposed problem. The code and dataset will be released after acceptance.
Gait Kinematics in Healthy Participants: A Motion Capture Dataset Under Weight Load and Knee Brace Conditions
Venue: arXiv cs.HC 2024
Year: 2024

The objective assessment of gait kinematics is crucial in evaluating human movement, informing clinical decisions, and advancing rehabilitation and assistive technologies. Assessing gait symmetry, in particular, holds significant importance in clinical rehabilitation, as it reflects the intricate coordination between nerves and muscles during human walking. In this research, a dataset has been compiled to improve the understanding of gait kinematics. The dataset encompasses motion capture data of the walking patterns of eleven healthy participants who were tasked with completing various activities on a circular path. These activities included normal walking, walking with a weighted dominant hand, walking with a braced dominant leg, and walking with both weight and brace. The walking tasks involving weight and brace were designed to emulate the asymmetry associated with common health conditions, shedding light on irregularities in individuals' walking patterns and reflecting the coordination between nerves and muscles. All tasks were performed at regular and fast speeds, offering valuable insights into upper and lower body kinematics. The dataset comprises raw sensor data, providing information on joint dynamics, angular velocities, and orientation changes during walking, as well as analyzed data, including processed data, Euler angles, and joint kinematics spanning various body segments. This dataset will serve as a valuable resource for researchers, clinicians, and engineers, facilitating the analysis of gait patterns and extracting relevant indices on mobility and balance.
The Future of Skill: What Is It to Be Skilled at Work?
Venue: arXiv cs.HC 2024
Year: 2024

In this short paper, we introduce work that is aiming to purposefully venture into this mesh of questions from a different starting point. Interjecting into the conversation, we want to ask: 'What is it to be skilled at work?' Building on work from scholars like Tim Ingold, and strands of longstanding research in workplace studies and CSCW, our interest is in turning the attention to the active work of 'being good', or 'being skilled', at what we as workers do. As we see it, skill provides a counterpoint to the version of intelligence that appears to be easily blackboxed in systems like Slack, and that ultimately reduces much of what people do to work well together. To put it slightly differently, skill - as we will argue below - gives us a way into thinking about work as a much more entangled endeavour, unfolding through multiple and interweaving sets of practices, places, tools and collaborations. In this vein, designing for the future of work seems to be about much more than where work is done or how we might bolt on discrete containers of intelligence. More fruitful would be attending to how we succeed in threading so many entities together to do our jobs well - in 'coming to be skilled'.
Structure Tensor Representation for Robust Oriented Object Detection
Venue: arXiv cs.CV 2024
Year: 2024

Oriented object detection predicts orientation in addition to object location and bounding box. Precisely predicting orientation remains challenging due to angular periodicity, which introduces boundary discontinuity issues and symmetry ambiguities. Inspired by classical works on edge and corner detection, this paper proposes to represent orientation in oriented bounding boxes as a structure tensor. This representation combines the strengths of Gaussian-based methods and angle-coder solutions, providing a simple yet efficient approach that is robust to angular periodicity issues without additional hyperparameters. Extensive evaluations across five datasets demonstrate that the proposed structure tensor representation outperforms previous methods in both fully-supervised and weakly supervised tasks, achieving high precision in angular prediction with minimal computational overhead. Thus, this work establishes structure tensors as a robust and modular alternative for encoding orientation in oriented object detection. We make our code publicly available, allowing for seamless integration into existing object detectors.
On the Privacy Risk of In-context Learning
Venue: arXiv cs.LG 2024
Year: 2024

Large language models (LLMs) are excellent few-shot learners. They can perform a wide variety of tasks purely based on natural language prompts provided to them. These prompts contain data of a specific downstream task -- often the private dataset of a party, e.g., a company that wants to leverage the LLM for their purposes. We show that deploying prompted models presents a significant privacy risk for the data used within the prompt by instantiating a highly effective membership inference attack. We also observe that the privacy risk of prompted models exceeds fine-tuned models at the same utility levels. After identifying the model's sensitivity to their prompts -- in the form of a significantly higher prediction confidence on the prompted data -- as a cause for the increased risk, we propose ensembling as a mitigation strategy. By aggregating over multiple different versions of a prompted model, membership inference risk can be decreased.
Breaking the mold: overcoming the time constraints of molecular dynamics on general-purpose hardware
Venue: arXiv cs.DC 2024
Year: 2024

The evolution of molecular dynamics (MD) simulations has been intimately linked to that of computing hardware. For decades following the creation of MD, simulations have improved with computing power along the three principal dimensions of accuracy, atom count (spatial scale), and duration (temporal scale). Since the mid-2000s, computer platforms have however failed to provide strong scaling for MD as scale-out CPU and GPU platforms that provide substantial increases to spatial scale do not lead to proportional increases in temporal scale. Important scientific problems therefore remained inaccessible to direct simulation, prompting the development of increasingly sophisticated algorithms that present significant complexity, accuracy, and efficiency challenges. While bespoke MD-only hardware solutions have provided a path to longer timescales for specific physical systems, their impact on the broader community has been mitigated by their limited adaptability to new methods and potentials. In this work, we show that a novel computing architecture, the Cerebras Wafer Scale Engine, completely alters the scaling path by delivering unprecedentedly high simulation rates up to 1.144M steps/second for 200,000 atoms whose interactions are described by an Embedded Atom Method potential. This enables direct simulations of the evolution of materials using general-purpose programmable hardware over millisecond timescales, dramatically increasing the space of direct MD simulations that can be carried out.
Advancing Autonomous Driving Perception: Analysis of Sensor Fusion and Computer Vision Techniques
Venue: arXiv cs.RO 2024
Year: 2024

In autonomous driving, perception systems are piv otal as they interpret sensory data to understand the envi ronment, which is essential for decision-making and planning. Ensuring the safety of these perception systems is fundamental for achieving high-level autonomy, allowing us to confidently delegate driving and monitoring tasks to machines. This re port aims to enhance the safety of perception systems by examining and summarizing the latest advancements in vision based systems, and metrics for perception tasks in autonomous driving. The report also underscores significant achievements and recognized challenges faced by current research in this field. This project focuses on enhancing the understanding and navigation capabilities of self-driving robots through depth based perception and computer vision techniques. Specifically, it explores how we can perform better navigation into unknown map 2D map with existing detection and tracking algorithms and on top of that how depth based perception can enhance the navigation capabilities of the wheel based bots to improve autonomous driving perception.
SoftLMs: Efficient Adaptive Low-Rank Approximation of Language Models using Soft-Thresholding Mechanism
Venue: arXiv cs.LG 2024
Year: 2024

Extensive efforts have been made to boost the performance in the domain of language models by introducing various attention-based transformers. However, the inclusion of linear layers with large dimensions contributes to significant computational and memory overheads. The escalating computational demands of these models necessitate the development of various compression techniques to ensure their deployment on devices, particularly in resource-constrained environments. In this paper, we propose a novel compression methodology that dynamically determines the rank of each layer using a soft thresholding mechanism, which clips the singular values with a small magnitude in a differentiable form. This approach automates the decision-making process to identify the optimal degree of compression for each layer. We have successfully applied the proposed technique to attention-based architectures, including BERT for discriminative tasks and GPT2 and TinyLlama for generative tasks. Additionally, we have validated our method on Mamba, a recently proposed state-space model. Our experiments demonstrate that the proposed technique achieves a speed-up of 1.33X to 1.72X in the encoder/ decoder with a 50% reduction in total parameters.
Attraction-Repulsion Swarming: A Generalized Framework of t-SNE via Force Normalization and Tunable Interactions
Venue: arXiv cs.LG 2024
Year: 2024

We propose a new method for data visualization based on attraction-repulsion swarming (ARS) dynamics, which we call ARS visualization. ARS is a generalized framework that is based on viewing the t-distributed stochastic neighbor embedding (t-SNE) visualization technique as a swarm of interacting agents driven by attraction and repulsion. Motivated by recent developments in swarming, we modify the t-SNE dynamics to include a normalization by the \emph{total influence}, which results in better posed dynamics in which we can use a data size independent time step (of $h=1$) and a simple iteration, without the need for the array of optimization tricks employed in t-SNE. ARS also includes the ability to separately tune the attraction and repulsion kernels, which gives the user control over the tightness within clusters and the spacing between them in the visualization. In contrast with t-SNE, our proposed ARS data visualization method is not gradient descent on the Kullback-Leibler divergence, and can be viewed solely as an interacting particle system driven by attraction and repulsion forces. We provide theoretical results illustrating how the choice of interaction kernel affects the dynamics, and experimental results to validate our method and compare to t-SNE on the MNIST and Cifar-10 data sets.
Electrical Load Forecasting in Smart Grid: A Personalized Federated Learning Approach
Venue: arXiv cs.LG 2024
Year: 2024

Electric load forecasting is essential for power management and stability in smart grids. This is mainly achieved via advanced metering infrastructure, where smart meters (SMs) are used to record household energy consumption. Traditional machine learning (ML) methods are often employed for load forecasting but require data sharing which raises data privacy concerns. Federated learning (FL) can address this issue by running distributed ML models at local SMs without data exchange. However, current FL-based approaches struggle to achieve efficient load forecasting due to imbalanced data distribution across heterogeneous SMs. This paper presents a novel personalized federated learning (PFL) method to load prediction under non-independent and identically distributed (non-IID) metering data settings. Specifically, we introduce meta-learning, where the learning rates are manipulated using the meta-learning idea to maximize the gradient for each client in each global round. Clients with varying processing capacities, data sizes, and batch sizes can participate in global model aggregation and improve their local load forecasting via personalized learning. Simulation results show that our approach outperforms state-of-the-art ML and FL methods in terms of better load forecasting accuracy.