GenAI4ELab/papercli-papers huggingface dataset:
CVAT-BWV: A Web-Based Video Annotation Platform for Police Body-Worn Video
Authors: Parsa Hejabi; Akshay Kiran Padte; Preni Golazizian; Rajat Hebbar; Jackson Trager; Georgios Chochlakis; Aditya Kommineni; Ellie Graeden; Shrikanth Narayanan; Benjamin A.T. Graham; Morteza Dehghani
Venue: IJCAI 2024 (main)

Abstract: We introduce an open-source platform for annotating body-worn video (BWV) footage aimed at enhancing transparency and accountability in policing. Despite the widespread adoption of BWVs in police departments, analyzing the vast amount of footage generated has presented significant challenges. This is primarily due to resource constraints, the sensitive nature of the data, which limits widespread access, and consequently, lack of annotations for training machine learning models. Our platform, called CVAT-BWV, offers a secure, locally hosted annotation environment that integrates several AI tools to assist in annotating multimodal data. With features such as automatic speech recognition, speaker diarization, object detection, and face recognition, CVAT-BWV aims to reduce the manual annotation workload, improve annotation quality, and allow for capturing perspectives from a diverse population of annotators. This tool aims to streamline the collection of annotations and the building of models, enhancing the use of BWV data for oversight and learning purposes to uncover insights into police-civilian interactions.
An LLM-enhanced Agent-based Simulation Tool for Information Propagation
Authors: Yuxuan Hu; Gemju Sherpa; Lan Zhang; Weihua Li; Quan Bai; Yijun Wang; Xiaodan Wang
Venue: IJCAI 2024 (main)

Abstract: Influence diffusion models are used for simulating information propagation in social networks. While most existing influence diffusion models are probabilistic, the emergence of Large Language Model (LLM) sheds light on the language-level inferences and interactions of user agents. This paper presents an LLM-enhanced Agent-based Influence Diffusion model (LAID), and a web-based visualization tool, LAIDSim, for simulating the information propagation in social networks.
InViTe: Individual Virtual Transfer for Personalized 3D Face Generation System
Authors: Mingyu Jang; Kyungjune Lee; Seongmin Lee; Hoseok Tong; Juwan Chung; Yusung Ro; Sanghoon Lee
Venue: IJCAI 2024 (main)

Abstract: With the expansion of the virtual communication industry using VR/AR, it has attracted increasing attention to enable users to represent their personalities in a 3D avatar. As the face of 3D avatars plays a crucial role in conveying human personality, a system that generates and manipulates 3D faces is desired. However, establishing the system is challenging due to the need for human effort and specialized knowledge. To fill this void, we present the Individual Virtual Transfer (InViTe), which enables the creation and customization of a 3D face according to the user's preference. Our proposed system is featured for 1) 3D face reconstruction with high fidelity texture map, 2) 3D face personalization, 3) realistic rendering results, and 4) real-time mobile virtual applications. We conduct an experiment to demonstrate that the proposed system can achieve sufficient individual personalization of 3D faces. Furthermore, we evaluate the system's data transmission protocol and demonstrate its efficiency. The demonstration video is available at https://www.youtube.com/watch?v=D 4pXZvGUWU.
Artificial Intelligence-Driven Video Indexing for Rapid Surveillance Footage Summarization and Review
Authors: Jaemin Jung; Soonyong Park; Harim Kim; Changha Lee; Charmgil Hong
Venue: IJCAI 2024 (main)

Abstract: This paper introduces VIDEX, an advanced tool designed to streamline the analysis of surveillance video through a user-friendly interface. VIDEX achieves high development efficiency and maintainability utilizing the Model-View-ViewModel (MVVM) design pattern. The core feature of VIDEX is a footage summary using object detection and anomaly detection. Its architecture ensures efficient data management by organizing detected objects and anomalies within an indexed database, thus facilitating a more rapid review process. Additionally, multi-threading was used to shorten the processing time. VIDEX provides a video summarization that can be used primarily in the criminal investigation stage using the information stored in a database. Discover more about VIDEX and access its resources at https://github.com/nth221/videx.
Demo: Enhancing Wildlife Acoustic Data Annotation Efficiency through Transfer and Active Learning
Authors: Hannes Kath; Patricia P. Serafini; Ivan B. Campos; Thiago S. Gouvêa; Daniel Sonntag
Venue: IJCAI 2024 (main)

Abstract: Passive Acoustic Monitoring (PAM) has become a key technology in wildlife monitoring, generating large amounts of acoustic data. However, the effective application of machine learning methods for sound event detection in PAM datasets is highly dependent on the accessibility of annotated data, a process that can be labour intensive. As a team of domain experts and machine learning researchers, in this paper we present a no-code annotation tool designed for PAM datasets that incorporates transfer learning and active learning strategies to address the data annotation challenge inherent in PAM. Transfer learning is applied to use pre-trained models to compute meaningful embeddings from the PAM audio files. Active learning iteratively identifies the most informative samples and then presents them to the user for annotation. This iterative approach improves the performance of the model compared to random sample selection. In a preliminary evaluation of the tool, a domain expert annotated part of a real PAM data set. Compared to conventional tools, the workflow of the proposed tool showed a speed improvement of 2-4 times. Further enhancements, such as the incorporation of sound examples, have the potential to further improve efficiency.
Using Large Language Models and Recruiter Expertise for Optimized Multilingual Job Offer – Applicant CV Matching
Authors: Hamit Kavas; Marc Serra-Vidal; Leo Wanner
Venue: IJCAI 2024 (main)

Abstract: In the context of the increasingly globalised economy and labour market, recruitment agencies face the challenge to deal with a magnitude of job offers and job applications written in a variety of languages, formats, and styles. Quite often, this leads to a suboptimal evaluation of the CVs of job seekers with respect to their relevance to a job offer. To address this challenge, we propose an interactive system that follows the ``human-in-the-loop'' approach, actively involving recruiters in the job offer -- applicant CV matching. The system uses a fine-tuned state-of-the-art classification model that aligns job seeker CVs with labels of the {\it European Skills, Competences, Qualifications and Occupations} taxonomy to propose an initial match between job offers with the CVs of job candidates. This match is refined in sequential LLM driven-interaction with the recruiter, which culminates in CV relevance scores and reports that justify them.
NEGOTIATOR: A Comprehensive Framework for Human-Agent Negotiation Integrating Preferences, Interaction, and Emotion
Authors: Mehmet Onur Keskin; Berk Buzcu; Berkecan Koçyiğit; Umut Çakan; Anıl Doğru; Reyhan Aydoğan
Venue: IJCAI 2024 (main)

Abstract: The paper introduces a comprehensive human-agent negotiation framework designed to facilitate the development and evaluation of research studies on human-agent negotiation without building each component from scratch. Leveraging the interoperability and reusability of its components, this framework offers various functionalities, including speech-to-text conversion, emotion recognition, a repository of negotiation strategies, and an interaction manager capable of managing gestures designed for Nao, Pepper, and QT, and coordinating message exchanges in a turn-taking fashion. This framework aims to lower the entry barrier for researchers in human-agent negotiation by providing a versatile platform that supports a wide range of research directions, including affective computing, natural language processing, decision-making, and non-verbal communication.
AUTODRAITEC: An Infrastructure-Based AUTOnomous DRiving System Using Artificial Intelligence and TEleCommunication Technologies
Authors: Zine el abidine Kherroubi; Fouzi Boukhalfa; Thierry Lestable; Carlos-Faouzi Bader
Venue: IJCAI 2024 (main)

Abstract: This paper introduces AUTODRAITEC, a novel AI-based system that is deployed on the road infrastructure to control the driving of Connected and Autonomous Vehicles (CAVs). For this purpose, we present a convincing proof of concept that demonstrates the effectiveness of our solution. The system deploys a hybrid machine learning approach comprised of a supervised learning classifier to characterize the behaviors of human drivers, with a deep reinforcement learning policy to provide speed recommendations for CAVs. This system is implemented using perception sensors and an industrial computer (IPC), which are intended to be deployed on the road infrastructure. Using a 1:18 scale testbed that faithfully replicates real-world driving scenarios, we demonstrate that AUTODRAITEC improves driving safety and efficiency while preserving the traffic flow rate.
ProMoAI: Process Modeling with Generative AI
Authors: Humam Kourani; Alessandro Berti; Daniel Schuster; Wil M.P. van der Aalst
Venue: IJCAI 2024 (main)

Abstract: ProMoAI is a novel tool that leverages Large Language Models (LLMs) to automatically generate process models from textual descriptions, incorporating advanced prompt engineering, error handling, and code generation techniques. Beyond automating the generation of complex process models, ProMoAI also supports process model optimization. Users can interact with the tool by providing feedback on the generated model, which is then used for refining the process model. ProMoAI utilizes the capabilities LLMs to offer a novel, AI-driven approach to process modeling, significantly reducing the barrier to entry for users without deep technical knowledge in process modeling.
SPARK: Harnessing Human-Centered Workflows with Biomedical Foundation Models for Drug Discovery
Authors: Bum Chul Kwon; Simona Rabinovici-Cohen; Beldine Moturi; Ruth Mwaura; Kezia Wahome; Oliver Njeru; Miguel Shinyenyi; Catherine Wanjiru; Sekou Remy; William Ogallo; Itai Guez; Partha Suryanarayanan; Joseph Morrone; Shreyans Sethi; Seung-Gu Kang; Tien Huynh; Kenney Ng; Diwakar Mahajan; Hongyang Li; Matan Ninio; Shervin Ayati; Efrat Hexter; Wendy Cornell
Venue: IJCAI 2024 (main)

Abstract: Biomedical foundation models, trained on diverse sources of small molecule data, hold great potential for accelerating drug discovery. However, their complex nature often presents a barrier for researchers seeking scientific insights and drug candidate generation. SPARK addresses this challenge by providing a user-friendly, web-based interface that empowers researchers to leverage these powerful models in their scientific workflows. Through SPARK, users can specify target proteins and desired molecule properties, adjust pre-trained models for tailored inferences, generate lists of potential drug candidates, analyze and compare molecules through interactive visualizations, and filter candidates based on key metrics (e.g., toxicity). By seamlessly integrating human knowledge and biomedical AI models' capabilities through an interactive web-based system, SPARK can improve the efficiency of collaboration between human experts and AI, thereby accelerating drug candidate discovery and ultimately leading to breakthroughs in finding cures for various diseases.
iFakeDetector: Real Time Integrated Web-based Deepfake Detection System
Authors: Kangjun Lee; Inho Jung; Simon S. Woo
Venue: IJCAI 2024 (main)

Abstract: Recently, deepfake detection research has been actively conducted. While many deepfake detectors have been proposed, validating the practicality of such systems against real world settings has not been explored much. Indeed, there are some gaps and disparities when they are applied in the real world. In this work, we developed a real time integrated web-based deepfake detection system, iFakeDetector, which incorporates the recent high performing deepfake detectors, and enables easy access for non-expert users to evaluate deepfake videos. Our system takes a deepfake video as input, allowing users to upload videos and select different detectors, and provides detection results on whether the uploaded video is a deepfake or not. Also, we provide an analysis tool that enables the video to be analyzed on a frame-by-frame basis with the probability of each frame being manipulated. Finally, we tested and deployed iFakeDetector in a real world scenario to verify its practicality and feasibility.
Interactive Visual Learning for Stable Diffusion
Authors: Seongmin Lee; Benjamin Hoover; Hendrik Strobelt; Zijie J. Wang; ShengYun Peng; Austin Wright; Kevin Li; Haekyu Park; Haoyang Yang; Duen Horng Chau
Venue: IJCAI 2024 (main)

Abstract: Diffusion-based generative models’ impressive ability to create convincing images has garnered global attention. However, their complex internal structures and operations often pose challenges for non-experts to grasp. We introduce Diffusion Explainer, the first interactive visualization tool designed to elucidate how Stable Diffusion transforms text prompts into images. It tightly integrates a visual overview of Stable Diffusion’s complex components with detailed explanations of their underlying operations. This integration enables users to fluidly transition between multiple levels of abstraction through animations and interactive elements. Offering real-time hands-on experience, Diffusion Explainer allows users to adjust Stable Diffusion’s hyperparameters and prompts without the need for installation or specialized hardware. Accessible via users’ web browsers, Diffusion Explainer is making significant strides in democratizing AI education, fostering broader public access. More than 7,200 users spanning 113 countries have used our open-sourced tool at https://poloclub.github.io/diffusion-explainer/. A video demo is available at https://youtu.be/MbkIADZjPnA.
Real-time Multi-modal Object Detection and Tracking on Edge for Regulatory Compliance Monitoring
Authors: Jia Syuen Lim; Ziwei Wang; Jiajun Liu; Abdelwahed Khamis; Reza Arablouei; Robert Barlow; Ryan McAllister
Venue: IJCAI 2024 (main)

Abstract: Regulatory compliance auditing in agrifood processing facilities is crucial for upholding the highest standards of quality assurance and traceability. However, the current manual and intermittent approaches to auditing present significant challenges and risks, potentially leading to gaps or loopholes in the system. To address these shortcomings, we introduce a real-time, multi-modal sensing system that utilizes 3D time-of-flight and RGB cameras and leverages unsupervised learning techniques on edge AI devices. The proposed system enables continuous object tracking, leading to improved efficiency in record-keeping and reduced manual labor. We demonstrate the effectiveness of the system in a knife sanitization monitoring scenario, showcasing its capability to overcome occlusion and low-light performance limitations commonly encountered with conventional RGB cameras.
Place Anything into Any Video
Authors: Ziling Liu; Jinyu Yang; Mingqi Gao; Feng Zheng
Venue: IJCAI 2024 (main)

Abstract: Controllable video editing has demonstrated remarkable potential across diverse applications, particularly in scenarios where capturing or re-capturing real-world videos is either impractical or costly. This paper introduces a novel and efficient system named Place-Anything, which facilitates the insertion of any object into any video solely based on a picture or text description of the target object or element. The system comprises three modules: 3D generation, video reconstruction, and 3D target insertion. This integrated approach offers an efficient and effective solution for producing and editing high-quality videos by naturally inserting realistic objects. Through experiment, we demonstrate that our system can effortlessly place any object into any video using just a photograph of the object. Our demo video can be found at https://youtu.be/afXqgLLRnTE. Please also visit our project page https://place-anything.github.io to get more information.
Mahjong AI Competition: Exploring AI Application in Complex Real-World Games
Authors: Yunlong Lu; Wenxin Li
Venue: IJCAI 2024 (main)

Abstract: This paper presents three Mahjong AI competitions we held at IJCAI. We briefly introduce the rule of Mahjong and its challenges to AI algorithms. By showing the results and the application of various algorithms in the competitions, we claim that existing algorithms show promising results in Mahjong, while open problems remain and more efforts are needed towards solving this complex game.
AESim: A Data-Driven Aircraft Engine Simulator
Authors: Abdellah Madane; Florent Forest; Hanane Azzag; Mustapha Lebbah; Jérôme Lacaille
Venue: IJCAI 2024 (main)

Abstract: We present AESim, a data-driven Aircraft Engine Simulator developed using transformer-based conditional generative adversarial networks. AESim generates samples of aircraft engine sensor measurements over full flights, conditioned on a given flight mission profile representing the flight conditions. It constitutes an essential tool in aircraft engine digital twins, capable of simulating their performance for different flight missions. It allows for comparison of the behavior of different engines under the same operational conditions, simulation of various scenarios for a given engine, facilitating applications like engine behavior analysis, performance limit identification, and optimization of maintenance schedules within a global Prognostics and Health Management (PHM) strategy. It also allows the imputation of missing flight data and addresses confidentiality concerns by generating synthetic flight datasets that can be shared for public research purposes or data challenges.
DFRP: A Dual-Track Feedback Recommendation System for Educational Resources
Authors: ChaoJun Meng; Changfan Pan; Zilong Li; Cong Zhou; Xinran Cao; Jia Zhu
Venue: IJCAI 2024 (main)

Abstract: The educational disparities among different regions are remarkably significant. The educational resource platform can effectively bridge the educational capability gap between regions. Most of the existing recommendation algorithms only consider interaction history, while we argue that the dependencies between knowledge points and education-related features are crucial for education resource recommendations. To address this, we propose DFRP, an educational resource recommendation platform based on knowledge graphs(KGs) and educational scale feedback. DFRP employs a recommendation algorithm based on teaching pathways and educational dimensions to achieve accurate recommendations and active feedback on educational resources. We also provide a detailed description of the system framework and present a demonstration scenario that uses educational scales for active feedback and KGs to show knowledge point dependencies.
SaGol: Using MiniGPT-4 to Generate Alt Text for Improving Image Accessibility
Authors: Yunseo Moon; Hyunmin Lee; SeungYoung Oh; Hyunggu Jung
Venue: IJCAI 2024 (main)

Abstract: SaGol is an AI-powered application to improve image accessibility for people with visual impairments (PVI) users. Alternative (alt) text, a general method of web accessibility for PVI users, is text or phrases that describe images on a website in an understandable way. SaGol generates alt text with the images on the user's smartphone using a vision large language model called MiniGPT-4. SaGol searches for similar images based on the generated alt text. We evaluated the length of alt text and the search accuracy. This paper shows a potential opportunity to improve image accessibility for PVI users.
ReportParse: A Unified NLP Tool for Extracting Document Structure and Semantics of Corporate Sustainability Reporting
Authors: Gaku Morio; Soh Young In; Jungah Yoon; Harri Rowlands; Christopher Manning
Venue: IJCAI 2024 (main)

Abstract: We introduce ReportParse, a Python-based tool designed to parse corporate sustainability reports. It combines document structure analysis with natural language processing (NLP) models to extract sustainability-related information from the reports. We also provide easy-to-use web and command interfaces. The tool is expected to aid researchers and analysts in evaluating corporate commitment to and management of sustainability efforts.
LangXAI: Integrating Large Vision Models for Generating Textual Explanations to Enhance Explainability in Visual Perception Tasks
Authors: Hung Nguyen; Tobias Clement; Loc Nguyen; Nils Kemmerzell; Binh Truong; Khang Nguyen; Mohamed Abdelaal; Hung Cao
Venue: IJCAI 2024 (main)

Abstract: LangXAI is a framework that integrates Explainable Artificial Intelligence (XAI) with advanced vision models to generate textual explanations for visual recognition tasks. Despite XAI advancements, an understanding gap persists for end-users with limited domain knowledge in artificial intelligence and computer vision. LangXAI addresses this by furnishing text-based explanations for classification, object detection, and semantic segmentation model outputs to end-users. Preliminary results demonstrate LangXAI's enhanced plausibility, with high BERTScore across tasks, fostering a more transparent and reliable AI framework on vision tasks for end-users. The code and demo of this work can be found at https://analytics-everywhere-lab.github.io/langxai.io/.
ComVas: Contextual Moral Values Alignment System
Authors: Inkit Padhi; Pierre Dognin; Jesus Rios; Ronny Luss; Swapnaja Achintalwar; Matthew Riemer; Miao Liu; Prasanna Sattigeri; Manish Nagireddy; Kush R. Varshney; Djallel Bouneffouf
Venue: IJCAI 2024 (main)

Abstract: In contemporary society, the integration of artificial intelligence (AI) systems into various aspects of daily life raises significant ethical concerns. One critical aspect is to ensure that AI systems align with the moral values of the endusers. To that end, we introduce the Contextual Moral Value Alignment System, ComVas. Unlike traditional AI systems which have moral values predefined, ComVas empowers users to dynamically select and customize the desired moral values thereby guiding the system’s decision-making process. Through a user-friendly interface, individuals can specify their preferred morals, allowing the system to steer the model’s responses and actions accordingly. ComVas utilizes advanced natural language processing techniques to engage with the users in a meaningful dialogue, understanding their preferences, and reasoning about moral dilemmas in diverse contexts. This demo article showcases the functionality of ComVas, illustrating its potential to foster ethical decision-making in AI systems while respecting individual autonomy and promoting user-centric design principles.
AVIN-Chat: An Audio-Visual Interactive Chatbot System with Emotional State Tuning
Authors: Chanhyuk Park; Jungbin Cho; Junwan Kim; Seongmin Lee; Jungsu Kim; Sanghoon Lee
Venue: IJCAI 2024 (main)

Abstract: This work presents an audio-visual interactive chatbot (AVIN-Chat) system that allows users to have face-to-face conversations with 3D avatars in real-time. Compared to the previous chatbot services, which provide text-only or speech-only communications, the proposed AVIN-Chat can offer audio-visual communications providing users with a superior experience quality. In addition, the proposed AVIN-Chat emotionally speaks and expresses according to the user's emotional state. Thus, it enables users to establish a strong bond with the chatbot system, increasing the user's immersion. Through user subjective tests, it is demonstrated that the proposed system provides users with a higher sense of immersion than previous chatbot systems. The demonstration video is available at https://www.youtube.com/watch?v=Z74uIV9k7_k.