OpenRaiser/Intern-Atlas huggingface dataset:
Multi-Agent Deep Reinforcement Learning for Cooperative and Competitive Autonomous Vehicles using AutoDRIVE Ecosystem
Venue: arXiv cs.RO 2023
Year: 2023

This work presents a modular and parallelizable multi-agent deep reinforcement learning framework for imbibing cooperative as well as competitive behaviors within autonomous vehicles. We introduce AutoDRIVE Ecosystem as an enabler to develop physically accurate and graphically realistic digital twins of Nigel and F1TENTH, two scaled autonomous vehicle platforms with unique qualities and capabilities, and leverage this ecosystem to train and deploy multi-agent reinforcement learning policies. We first investigate an intersection traversal problem using a set of cooperative vehicles (Nigel) that share limited state information with each other in single as well as multi-agent learning settings using a common policy approach. We then investigate an adversarial head-to-head autonomous racing problem using a different set of vehicles (F1TENTH) in a multi-agent learning setting using an individual policy approach. In either set of experiments, a decentralized learning architecture was adopted, which allowed robust training and testing of the approaches in stochastic environments, since the agents were mutually independent and exhibited asynchronous motion behavior. The problems were further aggravated by providing the agents with sparse observation spaces and requiring them to sample control commands that implicitly satisfied the imposed kinodynamic as well as safety constraints. The experimental results for both problem statements are reported in terms of quantitative metrics and qualitative remarks for training as well as deployment phases.
What does ChatGPT know about natural science and engineering?
Venue: arXiv cs.HC 2023
Year: 2023

ChatGPT is a powerful language model from OpenAI that is arguably able to comprehend and generate text. ChatGPT is expected to have a large impact on society, research, and education. An essential step to understand ChatGPT's expected impact is to study its domain-specific answering capabilities. Here, we perform a systematic empirical assessment of its abilities to answer questions across the natural science and engineering domains. We collected 594 questions from 198 faculty members across 5 faculties at Delft University of Technology. After collecting the answers from ChatGPT, the participants assessed the quality of the answers using a systematic scheme. Our results show that the answers from ChatGPT are on average perceived as ``mostly correct''. Two major trends are that the rating of the ChatGPT answers significantly decreases (i) as the complexity level of the question increases and (ii) as we evaluate skills beyond scientific knowledge, e.g., critical attitude.
Hierarchy Builder: Organizing Textual Spans into a Hierarchy to Facilitate Navigation
Venue: arXiv cs.CL 2023
Year: 2023

Information extraction systems often produce hundreds to thousands of strings on a specific topic. We present a method that facilitates better consumption of these strings, in an exploratory setting in which a user wants to both get a broad overview of what's available, and a chance to dive deeper on some aspects. The system works by grouping similar items together and arranging the remaining items into a hierarchical navigable DAG structure. We apply the method to medical information extraction.
GAME: Generalized deep learning model towards multimodal data integration for early screening of adolescent mental disorders
Venue: arXiv cs.LG 2023
Year: 2023

The timely identification of mental disorders in adolescents is a global public health challenge.Single factor is difficult to detect the abnormality due to its complex and subtle nature. Additionally, the generalized multimodal Computer-Aided Screening (CAS) systems with interactive robots for adolescent mental disorders are not available. Here, we design an android application with mini-games and chat recording deployed in a portable robot to screen 3,783 middle school students and construct the multimodal screening dataset, including facial images, physiological signs, voice recordings, and textual transcripts.We develop a model called GAME (Generalized Model with Attention and Multimodal EmbraceNet) with novel attention mechanism that integrates cross-modal features into the model. GAME evaluates adolescent mental conditions with high accuracy (73.34%-92.77%) and F1-Score (71.32%-91.06%).We find each modality contributes dynamically to the mental disorders screening and comorbidities among various mental disorders, indicating the feasibility of explainable model. This study provides a system capable of acquiring multimodal information and constructs a generalized multimodal integration algorithm with novel attention mechanisms for the early screening of adolescent mental disorders.
Ontology-Based Feedback to Improve Runtime Control for Multi-Agent Manufacturing Systems
Venue: arXiv cs.MA 2023
Year: 2023

Improving the overall equipment effectiveness (OEE) of machines on the shop floor is crucial to ensure the productivity and efficiency of manufacturing systems. To achieve the goal of increased OEE, there is a need to develop flexible runtime control strategies for the system. Decentralized strategies, such as multi-agent systems, have proven effective in improving system flexibility. However, runtime multi-agent control of complex manufacturing systems can be challenging as the agents require extensive communication and computational efforts to coordinate agent activities. One way to improve communication speed and cooperation capabilities between system agents is by providing a common language between these agents to represent knowledge about system behavior. The integration of ontology into multi-agent systems in manufacturing provides agents with the capability to continuously update and refine their knowledge in a global context. This paper contributes to the design of an ontology for multi-agent systems in manufacturing, introducing an extendable knowledge base and a methodology for continuously updating the production data by agents during runtime. To demonstrate the effectiveness of the proposed framework, a case study is conducted in a simulated environment, which shows improvements in OEE during runtime.
Continuous Integration and Software Quality: A Causal Explanatory Study
Venue: arXiv cs.SE 2023
Year: 2023

Continuous Integration (CI) is a software engineering practice that aims to reduce the cost and risk of code integration among teams. Recent empirical studies have confirmed associations between CI and the software quality (SQ). However, no existing study investigates causal relationships between CI and SQ. This paper investigates it by applying the causal Direct Acyclic Graphs (DAGs) technique. We combine two other strategies to support this technique: a literature review and a Mining Software Repository (MSR) study. In the first stage, we review the literature to discover existing associations between CI and SQ, which help us create a "literature-based causal DAG" in the second stage. This DAG encapsulates the literature assumptions regarding CI and its influence on SQ. In the third stage, we analyze 12 activity months for 70 opensource projects by mining software repositories -- 35 CI and 35 no-CI projects. This MSR study is not a typical "correlation is not causation" study because it is used to verify the relationships uncovered in the causal DAG produced in the first stages. The fourth stage consists of testing the statistical implications from the "literature-based causal DAG" on our dataset. Finally, in the fifth stage, we build a DAG with observations from the literature and the dataset, the "literature-data DAG". In addition to the direct causal effect of CI on SQ, we find evidence of indirect effects of CI. For example, CI affects teams' communication, which positively impacts SQ. We also highlight the confounding effect of project age.
360$^\circ$ Reconstruction From a Single Image Using Space Carved Outpainting
Venue: arXiv cs.CV 2023
Year: 2023

We introduce POP3D, a novel framework that creates a full $360^\circ$-view 3D model from a single image. POP3D resolves two prominent issues that limit the single-view reconstruction. Firstly, POP3D offers substantial generalizability to arbitrary categories, a trait that previous methods struggle to achieve. Secondly, POP3D further improves reconstruction fidelity and naturalness, a crucial aspect that concurrent works fall short of. Our approach marries the strengths of four primary components: (1) a monocular depth and normal predictor that serves to predict crucial geometric cues, (2) a space carving method capable of demarcating the potentially unseen portions of the target object, (3) a generative model pre-trained on a large-scale image dataset that can complete unseen regions of the target, and (4) a neural implicit surface reconstruction method tailored in reconstructing objects using RGB images along with monocular geometric cues. The combination of these components enables POP3D to readily generalize across various in-the-wild images and generate state-of-the-art reconstructions, outperforming similar works by a significant margin. Project page: \url{http://cg.postech.ac.kr/research/POP3D}
Multi-dimension Queried and Interacting Network for Stereo Image Deraining
Venue: arXiv cs.CV 2023
Year: 2023

Eliminating the rain degradation in stereo images poses a formidable challenge, which necessitates the efficient exploitation of mutual information present between the dual views. To this end, we devise MQINet, which employs multi-dimension queries and interactions for stereo image deraining. More specifically, our approach incorporates a context-aware dimension-wise queried block (CDQB). This module leverages dimension-wise queries that are independent of the input features and employs global context-aware attention (GCA) to capture essential features while avoiding the entanglement of redundant or irrelevant information. Meanwhile, we introduce an intra-view physics-aware attention (IPA) based on the inverse physical model of rainy images. IPA extracts shallow features that are sensitive to the physics of rain degradation, facilitating the reduction of rain-related artifacts during the early learning period. Furthermore, we integrate a cross-view multi-dimension interacting attention mechanism (CMIA) to foster comprehensive feature interaction between the two views across multiple dimensions. Extensive experimental evaluations demonstrate the superiority of our model over EPRRNet and StereoIRR, achieving respective improvements of 4.18 dB and 0.45 dB in PSNR. Code and models are available at \url{https://github.com/chdwyb/MQINet}.
Computational Approaches for App-to-App Retrieval and Design Consistency Check
Venue: arXiv cs.HC 2023
Year: 2023

Extracting semantic representations from mobile user interfaces (UI) and using the representations for designers' decision-making processes have shown the potential to be effective computational design support tools. Current approaches rely on machine learning models trained on small-sized mobile UI datasets to extract semantic vectors and use screenshot-to-screenshot comparison to retrieve similar-looking UIs given query screenshots. However, the usability of these methods is limited because they are often not open-sourced and have complex training pipelines for practitioners to follow, and are unable to perform screenshot set-to-set (i.e., app-to-app) retrieval. To this end, we (1) employ visual models trained with large web-scale images and test whether they could extract a UI representation in a zero-shot way and outperform existing specialized models, and (2) use mathematically founded methods to enable app-to-app retrieval and design consistency analysis. Our experiments show that our methods not only improve upon previous retrieval models but also enable multiple new applications.
Anti-Aliased Neural Implicit Surfaces with Encoding Level of Detail
Venue: arXiv cs.CV 2023
Year: 2023

We present LoD-NeuS, an efficient neural representation for high-frequency geometry detail recovery and anti-aliased novel view rendering. Drawing inspiration from voxel-based representations with the level of detail (LoD), we introduce a multi-scale tri-plane-based scene representation that is capable of capturing the LoD of the signed distance function (SDF) and the space radiance. Our representation aggregates space features from a multi-convolved featurization within a conical frustum along a ray and optimizes the LoD feature volume through differentiable rendering. Additionally, we propose an error-guided sampling strategy to guide the growth of the SDF during the optimization. Both qualitative and quantitative evaluations demonstrate that our method achieves superior surface reconstruction and photorealistic view synthesis compared to state-of-the-art approaches.
Prompt, Condition, and Generate: Classification of Unsupported Claims with In-Context Learning
Venue: arXiv cs.CL 2023
Year: 2023

Unsupported and unfalsifiable claims we encounter in our daily lives can influence our view of the world. Characterizing, summarizing, and -- more generally -- making sense of such claims, however, can be challenging. In this work, we focus on fine-grained debate topics and formulate a new task of distilling, from such claims, a countable set of narratives. We present a crowdsourced dataset of 12 controversial topics, comprising more than 120k arguments, claims, and comments from heterogeneous sources, each annotated with a narrative label. We further investigate how large language models (LLMs) can be used to synthesise claims using In-Context Learning. We find that generated claims with supported evidence can be used to improve the performance of narrative classification models and, additionally, that the same model can infer the stance and aspect using a few training examples. Such a model can be useful in applications which rely on narratives , e.g. fact-checking.
Improving CLIP Robustness with Knowledge Distillation and Self-Training
Venue: arXiv cs.CV 2023
Year: 2023

This paper examines the robustness of a multi-modal computer vision model, CLIP (Contrastive Language-Image Pretraining), in the context of unsupervised learning. The main objective is twofold: first, to evaluate the robustness of CLIP, and second, to explore strategies for augmenting its robustness. To achieve this, we introduce a novel approach named LP-CLIP. This technique involves the distillation of CLIP features through the incorporation of a linear probing layer positioned atop its encoding structure. This newly added layer is trained utilizing pseudo-labels produced by CLIP, coupled with a self-training strategy. The LP-CLIP technique offers a promising approach to enhance the robustness of CLIP without the need for annotations. By leveraging a simple linear probing layer, we aim to improve the model's ability to withstand various uncertainties and challenges commonly encountered in real-world scenarios. Importantly, our approach does not rely on annotated data, which makes it particularly valuable in situations where labeled data might be scarce or costly to obtain. Our proposed approach increases the robustness of CLIP with SOTA results compared to supervised technique on various datasets.
Automating Bird Diverter Installation through Multi-Aerial Robots and Signal Temporal Logic Specifications
Venue: arXiv cs.RO 2023
Year: 2023

This paper tackles the task assignment and trajectory generation problem for bird diverter installation using a fleet of multi-rotors. The proposed motion planner considers payload capacity, recharging constraints, and utilizes Signal Temporal Logic (STL) specifications for encoding mission objectives and temporal requirements. An event-based replanning strategy is introduced to handle unexpected failures and ensure operational continuity. An energy minimization term is also employed to implicitly save multi-rotor flight time during installation. Simulations in MATLAB and Gazebo, as well as field experiments, demonstrate the effectiveness and validity of the approach in a mock-up scenario.
Exploring Different Levels of Supervision for Detecting and Localizing Solar Panels on Remote Sensing Imagery
Venue: arXiv cs.CV 2023
Year: 2023

This study investigates object presence detection and localization in remote sensing imagery, focusing on solar panel recognition. We explore different levels of supervision, evaluating three models: a fully supervised object detector, a weakly supervised image classifier with CAM-based localization, and a minimally supervised anomaly detector. The classifier excels in binary presence detection (0.79 F1-score), while the object detector (0.72) offers precise localization. The anomaly detector requires more data for viable performance. Fusion of model results shows potential accuracy gains. CAM impacts localization modestly, with GradCAM, GradCAM++, and HiResCAM yielding superior results. Notably, the classifier remains robust with less data, in contrast to the object detector.
Gaining a better understanding of online polarization by approaching it as a dynamic process
Venue: arXiv cs.SI 2023
Year: 2023

Polarization is often a clich{é}, its conceptualization remains approximate and no consensus has been reached so far. Often simply seen as an inevitable result of the use of social networks, polarization nevertheless remains a complex social phenomenon that must be placed in a wider context. To contribute to a better understanding of polarization, we approach it as an evolving process, drawing on a dual expertise in political and data sciences. We compare the polarization process between one mature debate (COVID-19 vaccine) and one emerging debate (Ukraine conflict) at the time of data collection. Both debates are studied on Twitter users, a highly politicized population, and on the French population to provide key elements beyond the traditional US context. This unprecedented analysis confirms that polarization varies over time, through a succession of specific periods, whose existence and duration depend on the maturity of the debate. Importantly, we highlight that polarization is paced by context-related events. Bearing this in mind, we pave the way for a new generation of personalized depolarization strategies, adapted to the context and maturity of debates.
Functional requirements to mitigate the Risk of Harm to Patients from Artificial Intelligence in Healthcare
Venue: arXiv cs.AI 2023
Year: 2023

The Directorate General for Parliamentary Research Services of the European Parliament has prepared a report to the Members of the European Parliament where they enumerate seven main risks of Artificial Intelligence (AI) in medicine and healthcare: patient harm due to AI errors, misuse of medical AI tools, bias in AI and the perpetuation of existing inequities, lack of transparency, privacy and security issues, gaps in accountability, and obstacles in implementation. In this study, we propose fourteen functional requirements that AI systems may implement to reduce the risks associated with their medical purpose: AI passport, User management, Regulation check, Academic use only disclaimer, data quality assessment, Clinicians double check, Continuous performance evaluation, Audit trail, Continuous usability test, Review of retrospective/simulated cases, Bias check, eXplainable AI, Encryption and use of field-tested libraries, and Semantic interoperability. Our intention here is to provide specific high-level specifications of technical solutions to ensure continuous good performance and use of AI systems to benefit patients in compliance with the future EU regulatory framework.
Posterior sampling algorithms for unsupervised speech enhancement with recurrent variational autoencoder
Venue: arXiv cs.CV 2023
Year: 2023

In this paper, we address the unsupervised speech enhancement problem based on recurrent variational autoencoder (RVAE). This approach offers promising generalization performance over the supervised counterpart. Nevertheless, the involved iterative variational expectation-maximization (VEM) process at test time, which relies on a variational inference method, results in high computational complexity. To tackle this issue, we present efficient sampling techniques based on Langevin dynamics and Metropolis-Hasting algorithms, adapted to the EM-based speech enhancement with RVAE. By directly sampling from the intractable posterior distribution within the EM process, we circumvent the intricacies of variational inference. We conduct a series of experiments, comparing the proposed methods with VEM and a state-of-the-art supervised speech enhancement approach based on diffusion models. The results reveal that our sampling-based algorithms significantly outperform VEM, not only in terms of computational efficiency but also in overall performance. Furthermore, when compared to the supervised baseline, our methods showcase robust generalization performance in mismatched test conditions.
Decoupling the Curve Modeling and Pavement Regression for Lane Detection
Venue: arXiv cs.CV 2023
Year: 2023

The curve-based lane representation is a popular approach in many lane detection methods, as it allows for the representation of lanes as a whole object and maximizes the use of holistic information about the lanes. However, the curves produced by these methods may not fit well with irregular lines, which can lead to gaps in performance compared to indirect representations such as segmentation-based or point-based methods. We have observed that these lanes are not intended to be irregular, but they appear zigzagged in the perspective view due to being drawn on uneven pavement. In this paper, we propose a new approach to the lane detection task by decomposing it into two parts: curve modeling and ground height regression. Specifically, we use a parameterized curve to represent lanes in the BEV space to reflect the original distribution of lanes. For the second part, since ground heights are determined by natural factors such as road conditions and are less holistic, we regress the ground heights of key points separately from the curve modeling. Additionally, we have unified the 2D and 3D lane detection tasks by designing a new framework and a series of losses to guide the optimization of models with or without 3D lane labels. Our experiments on 2D lane detection benchmarks (TuSimple and CULane), as well as the recently proposed 3D lane detection datasets (ONCE-3Dlane and OpenLane), have shown significant improvements. We will make our well-documented source code publicly available.
Model Leeching: An Extraction Attack Targeting LLMs
Venue: arXiv cs.LG 2023
Year: 2023

Model Leeching is a novel extraction attack targeting Large Language Models (LLMs), capable of distilling task-specific knowledge from a target LLM into a reduced parameter model. We demonstrate the effectiveness of our attack by extracting task capability from ChatGPT-3.5-Turbo, achieving 73% Exact Match (EM) similarity, and SQuAD EM and F1 accuracy scores of 75% and 87%, respectively for only $50 in API cost. We further demonstrate the feasibility of adversarial attack transferability from an extracted model extracted via Model Leeching to perform ML attack staging against a target LLM, resulting in an 11% increase to attack success rate when applied to ChatGPT-3.5-Turbo.
Addressing the Scalability Bottleneck of Semantic Technologies at Bosch
Venue: arXiv cs.DC 2023
Year: 2023

At the heart of smart manufacturing is real-time semi-automatic decision-making. Such decisions are vital for optimizing production lines, e.g., reducing resource consumption, improving the quality of discrete manufacturing operations, and optimizing the actual products, e.g., optimizing the sampling rate for measuring product dimensions during production. Such decision-making relies on massive industrial data thus posing a real-time processing bottleneck.