OpenRaiser/Intern-Atlas huggingface dataset:
SelfPiCo: Self-Guided Partial Code Execution with LLMs
Venue: arXiv cs.SE 2024
Year: 2024

Code executability plays a vital role in software debugging and testing (e.g., detecting runtime exceptions or assertion violations). However, code execution, especially partial or arbitrary code execution, is a non-trivial task due to missing definitions and complex third-party dependencies. To make partial code (such as code snippets posted on the web or code fragments deep inside complex software projects) executable, the existing study has proposed a machine learning model to predict the undefined element types and inject the pre-defined dummy values into execution. However, the performance of their tool is limited due to its simply designed dummy values and the inability to continue learning. In this paper, we design and implement a novel framework, named SelfPiCo (Self Guided Partial Code Executor), to dynamically guide partial code execution by incorporating the open-source LLM (i.e., Code Llama) within an interactive loop. Particularly, SelfPiCo leverages few-shot in-context learning and chain-of-thought reasoning to elicit human knowledge and logical reasoning based on fine-tuning the Code Llama model. SelfPiCo continuously learns from code execution results and refines its predictions step after step. Our evaluations demonstrate that SelfPiCo can execute 72.7% and 83.3% of all lines in the open-source code and Stack Overflow snippets, outperforming the most recent state-of-the-art Lexecutor by 37.9% and 33.5%, respectively. Moreover, SelfPiCo successfully detected 18 and 33 runtime type error issues by executing the partial code from eight GitHub software projects and 43 Stack Overflow posts, demonstrating the practical usage and potential application of our framework in practice.
Simultaneous Trajectory Optimization and Contact Selection for Contact-rich Manipulation with High-Fidelity Geometry
Venue: arXiv cs.RO 2024
Year: 2024

Contact-implicit trajectory optimization (CITO) is an effective method to plan complex trajectories for various contact-rich systems including manipulation and locomotion. CITO formulates a mathematical program with complementarity constraints (MPCC) that enforces that contact forces must be zero when points are not in contact. However, MPCC solve times increase steeply with the number of allowable points of contact, which limits CITO's applicability to problems in which only a few, simple geometries are allowed to make contact. This paper introduces simultaneous trajectory optimization and contact selection (STOCS), as an extension of CITO that overcomes this limitation. The innovation of STOCS is to identify salient contact points and times inside the iterative trajectory optimization process. This effectively reduces the number of variables and constraints in each MPCC invocation. The STOCS framework, instantiated with key contact identification subroutines, renders the optimization of manipulation trajectories computationally tractable even for high-fidelity geometries consisting of tens of thousands of vertices.
Quotient complex (QC)-based machine learning for 2D perovskite design
Venue: arXiv cs.CE 2024
Year: 2024

With remarkable stability and exceptional optoelectronic properties, two-dimensional (2D) halide layered perovskites hold immense promise for revolutionizing photovoltaic technology. Presently, inadequate representations have substantially impeded the design and discovery of 2D perovskites. In this context, we introduce a novel computational topology framework termed the quotient complex (QC), which serves as the foundation for the material representation. Our QC-based features are seamlessly integrated with learning models for the advancement of 2D perovskite design. At the heart of this framework lies the quotient complex descriptors (QCDs), representing a quotient variation of simplicial complexes derived from materials unit cell and periodic boundary conditions. Differing from prior material representations, this approach encodes higher-order interactions and periodicity information simultaneously. Based on the well-established New Materials for Solar Energetics (NMSE) databank, our QC-based machine learning models exhibit superior performance against all existing counterparts. This underscores the paramount role of periodicity information in predicting material functionality, while also showcasing the remarkable efficiency of the QC-based model in characterizing materials structural attributes.
Q-Ground: Image Quality Grounding with Large Multi-modality Models
Venue: arXiv cs.CV 2024
Year: 2024

Recent advances of large multi-modality models (LMM) have greatly improved the ability of image quality assessment (IQA) method to evaluate and explain the quality of visual content. However, these advancements are mostly focused on overall quality assessment, and the detailed examination of local quality, which is crucial for comprehensive visual understanding, is still largely unexplored. In this work, we introduce Q-Ground, the first framework aimed at tackling fine-scale visual quality grounding by combining large multi-modality models with detailed visual quality analysis. Central to our contribution is the introduction of the QGround-100K dataset, a novel resource containing 100k triplets of (image, quality text, distortion segmentation) to facilitate deep investigations into visual quality. The dataset comprises two parts: one with human-labeled annotations for accurate quality assessment, and another labeled automatically by LMMs such as GPT4V, which helps improve the robustness of model training while also reducing the costs of data collection. With the QGround-100K dataset, we propose a LMM-based method equipped with multi-scale feature learning to learn models capable of performing both image quality answering and distortion segmentation based on text prompts. This dual-capability approach not only refines the model's understanding of region-aware image quality but also enables it to interactively respond to complex, text-based queries about image quality and specific distortions. Q-Ground takes a step towards sophisticated visual quality analysis in a finer scale, establishing a new benchmark for future research in the area. Codes and dataset are available at https://github.com/Q-Future/Q-Ground.
High Efficiency Image Compression for Large Visual-Language Models
Venue: arXiv cs.CV 2024
Year: 2024

In recent years, large visual language models (LVLMs) have shown impressive performance and promising generalization capability in multi-modal tasks, thus replacing humans as receivers of visual information in various application scenarios. In this paper, we pioneer to propose a variable bitrate image compression framework consisting of a pre-editing module and an end-to-end codec to achieve promising rate-accuracy performance for different LVLMs. In particular, instead of optimizing an adaptive pre-editing network towards a particular task or several representative tasks, we propose a new optimization strategy tailored for LVLMs, which is designed based on the representation and discrimination capability with token-level distortion and rank. The pre-editing module and the variable bitrate end-to-end image codec are jointly trained by the losses based on semantic tokens of the large model, which introduce enhanced generalization capability for various data and tasks. {Experimental results demonstrate that the proposed framework could efficiently achieve much better rate-accuracy performance compared to the state-of-the-art coding standard, Versatile Video Coding.} Meanwhile, experiments with multi-modal tasks have revealed the robustness and generalization capability of the proposed framework.
AI-based Density Recognition
Venue: arXiv cs.CV 2024
Year: 2024

Learning-based analysis of images is commonly used in the fields of mobility and robotics for safe environmental motion and interaction. This requires not only object recognition but also the assignment of certain properties to them. With the help of this information, causally related actions can be adapted to different circumstances. Such logical interactions can be optimized by recognizing object-assigned properties. Density as a physical property offers the possibility to recognize how heavy an object is, which material it is made of, which forces are at work, and consequently which influence it has on its environment. Our approach introduces an AI-based concept for assigning physical properties to objects through the use of associated images. Based on synthesized data, we derive specific patterns from 2D images using a neural network to extract further information such as volume, material, or density. Accordingly, we discuss the possibilities of property-based feature extraction to improve causally related logics.
News Ninja: Gamified Annotation of Linguistic Bias in Online News
Venue: arXiv cs.HC 2024
Year: 2024

Recent research shows that visualizing linguistic bias mitigates its negative effects. However, reliable automatic detection methods to generate such visualizations require costly, knowledge-intensive training data. To facilitate data collection for media bias datasets, we present News Ninja, a game employing data-collecting game mechanics to generate a crowdsourced dataset. Before annotating sentences, players are educated on media bias via a tutorial. Our findings show that datasets gathered with crowdsourced workers trained on News Ninja can reach significantly higher inter-annotator agreements than expert and crowdsourced datasets with similar data quality. As News Ninja encourages continuous play, it allows datasets to adapt to the reception and contextualization of news over time, presenting a promising strategy to reduce data collection expenses, educate players, and promote long-term bias mitigation.
SDoH-GPT: Using Large Language Models to Extract Social Determinants of Health (SDoH)
Venue: arXiv cs.CL 2024
Year: 2024

Extracting social determinants of health (SDoH) from unstructured medical notes depends heavily on labor-intensive annotations, which are typically task-specific, hampering reusability and limiting sharing. In this study we introduced SDoH-GPT, a simple and effective few-shot Large Language Model (LLM) method leveraging contrastive examples and concise instructions to extract SDoH without relying on extensive medical annotations or costly human intervention. It achieved tenfold and twentyfold reductions in time and cost respectively, and superior consistency with human annotators measured by Cohen's kappa of up to 0.92. The innovative combination of SDoH-GPT and XGBoost leverages the strengths of both, ensuring high accuracy and computational efficiency while consistently maintaining 0.90+ AUROC scores. Testing across three distinct datasets has confirmed its robustness and accuracy. This study highlights the potential of leveraging LLMs to revolutionize medical note classification, demonstrating their capability to achieve highly accurate classifications with significantly reduced time and cost.
Mapping the individual, social, and biospheric impacts of Foundation Models
Venue: arXiv cs.CY 2024
Year: 2024

Responding to the rapid roll-out and large-scale commercialization of foundation models, large language models, and generative AI, an emerging body of work is shedding light on the myriad impacts these technologies are having across society. Such research is expansive, ranging from the production of discriminatory, fake and toxic outputs, and privacy and copyright violations, to the unjust extraction of labor and natural resources. The same has not been the case in some of the most prominent AI governance initiatives in the global north like the UK's AI Safety Summit and the G7's Hiroshima process, which have influenced much of the international dialogue around AI governance. Despite the wealth of cautionary tales and evidence of algorithmic harm, there has been an ongoing over-emphasis within the AI governance discourse on technical matters of safety and global catastrophic or existential risks. This narrowed focus has tended to draw attention away from very pressing social and ethical challenges posed by the current brute-force industrialization of AI applications. To address such a visibility gap between real-world consequences and speculative risks, this paper offers a critical framework to account for the social, political, and environmental dimensions of foundation models and generative AI. We identify 14 categories of risks and harms and map them according to their individual, social, and biospheric impacts. We argue that this novel typology offers an integrative perspective to address the most urgent negative impacts of foundation models and their downstream applications. We conclude with recommendations on how this typology could be used to inform technical and normative interventions to advance responsible AI.
FIIH: Fully Invertible Image Hiding for Secure and Robust
Venue: arXiv cs.CV 2024
Year: 2024

Image hiding is the study of techniques for covert storage and transmission, which embeds a secret image into a container image and generates stego image to make it similar in appearance to a normal image. However, existing image hiding methods have a serious problem that the hiding and revealing process cannot be fully invertible, which results in the revealing network not being able to recover the secret image losslessly, which makes it impossible to simultaneously achieve high fidelity and secure transmission of the secret image in an insecure network environment. To solve this problem,this paper proposes a fully invertible image hiding architecture based on invertible neural network,aiming to realize invertible hiding of secret images,which is invertible on both data and network. Based on this ingenious architecture, the method can withstand deep learning based image steganalysis. In addition, we propose a new method for enhancing the robustness of stego images after interference during transmission. Experiments demonstrate that the FIIH proposed in this paper significantly outperforms other state-of-the-art image hiding methods in hiding a single image, and also significantly outperforms other state-of-the-art methods in robustness and security.
KnowTD-An Actionable Knowledge Representation System for Thermodynamics
Venue: arXiv cs.CE 2024
Year: 2024

We demonstrate that thermodynamic knowledge acquired by humans can be transferred to computers so that the machine can use it to solve thermodynamic problems and produce explainable solutions with a guarantee of correctness. The actionable knowledge representation system that we have created for this purpose is called KnowTD. It is based on an ontology of thermodynamics that represents knowledge of thermodynamic theory, material properties, and thermodynamic problems. The ontology is coupled with a reasoner that sets up the problem to be solved based on user input, extracts the correct, pertinent equations from the ontology, solves the resulting mathematical problem, and returns the solution to the user, together with an explanation of how it was obtained. KnowTD is presently limited to simple thermodynamic problems, similar to those discussed in an introductory course in Engineering Thermodynamics. This covers the basic theory and working principles of thermodynamics. KnowTD is designed in a modular way and is easily extendable.
Domain Generalized Recaptured Screen Image Identification Using SWIN Transformer
Venue: arXiv cs.CV 2024
Year: 2024

An increasing number of classification approaches have been developed to address the issue of image rebroadcast and recapturing, a standard attack strategy in insurance frauds, face spoofing, and video piracy. However, most of them neglected scale variations and domain generalization scenarios, performing poorly in instances involving domain shifts, typically made worse by inter-domain and cross-domain scale variances. To overcome these issues, we propose a cascaded data augmentation and SWIN transformer domain generalization framework (DAST-DG) in the current research work Initially, we examine the disparity in dataset representation. A feature generator is trained to make authentic images from various domains indistinguishable. This process is then applied to recaptured images, creating a dual adversarial learning setup. Extensive experiments demonstrate that our approach is practical and surpasses state-of-the-art methods across different databases. Our model achieves an accuracy of approximately 82\% with a precision of 95\% on high-variance datasets.
Robust Point Cloud Registration in Robotic Inspection with Locally Consistent Gaussian Mixture Model
Venue: arXiv cs.RO 2024
Year: 2024

In robotic inspection of aviation parts, achieving accurate pairwise point cloud registration between scanned and model data is essential. However, noise and outliers generated in robotic scanned data can compromise registration accuracy. To mitigate this challenge, this article proposes a probability-based registration method utilizing Gaussian Mixture Model (GMM) with local consistency constraint. This method converts the registration problem into a model fitting one, constraining the similarity of posterior distributions between neighboring points to enhance correspondence robustness. We employ the Expectation Maximization algorithm iteratively to find optimal rotation matrix and translation vector while obtaining GMM parameters. Both E-step and M-step have closed-form solutions. Simulation and actual experiments confirm the method's effectiveness, reducing root mean square error by 20% despite the presence of noise and outliers. The proposed method excels in robustness and accuracy compared to existing methods.
The Sketchfab 3D Creative Commons Collection (S3D3C)
Venue: arXiv cs.MM 2024
Year: 2024

The technology to capture, create, and use three-dimensional (3D) models has become increasingly accessible in recent years. With increasing numbers of use cases for 3D models and collections of rapidly increasing size, better methods to analyze the content of 3D models are required. While previously proposed 3D model collections for research purposes exist, these often contain only untextured geometry and are typically designed for a specific application, which limits their use in quantitative evaluations of modern 3D model analysis methods. In this paper, we introduce the Sketchfab 3D Creative Commons Collection (S3D3C), a new 3D model research collection consisting of 40,802 creative commons licensed models downloaded from the 3D model platform Sketchfab. By including popular freely available models with a wide variety of technical properties, such as textures, materials, and animations, we enable its use in the evaluation of state-of-the-art geometry-based and view-based 3D model analysis and retrieval techniques.
SCIsegV2: A Universal Tool for Segmentation of Intramedullary Lesions in Spinal Cord Injury
Venue: arXiv cs.CV 2024
Year: 2024

Spinal cord injury (SCI) is a devastating incidence leading to permanent paralysis and loss of sensory-motor functions potentially resulting in the formation of lesions within the spinal cord. Imaging biomarkers obtained from magnetic resonance imaging (MRI) scans can predict the functional recovery of individuals with SCI and help choose the optimal treatment strategy. Currently, most studies employ manual quantification of these MRI-derived biomarkers, which is a subjective and tedious task. In this work, we propose (i) a universal tool for the automatic segmentation of intramedullary SCI lesions, dubbed \texttt{SCIsegV2}, and (ii) a method to automatically compute the width of the tissue bridges from the segmented lesion. Tissue bridges represent the spared spinal tissue adjacent to the lesion, which is associated with functional recovery in SCI patients. The tool was trained and validated on a heterogeneous dataset from 7 sites comprising patients from different SCI phases (acute, sub-acute, and chronic) and etiologies (traumatic SCI, ischemic SCI, and degenerative cervical myelopathy). Tissue bridges quantified automatically did not significantly differ from those computed manually, suggesting that the proposed automatic tool can be used to derive relevant MRI biomarkers. \texttt{SCIsegV2} and the automatic tissue bridges computation are open-source and available in Spinal Cord Toolbox (v6.4 and above) via the \texttt{sct\_deepseg -task seg\_sc\_lesion\_t2w\_sci} and \texttt{sct\_analyze\_lesion} functions, respectively.
Component Matching as a Graph Matching Problem
Venue: arXiv cs.SE 2024
Year: 2024

The development of an IT strategy and ensuring that it is the best possible one for business is a key problem many organizations face. This problem is that of linking business architecture to IT architecture in general and application architecture specifically. In our earlier work we proposed Category theory as the formal language to unify the business and IT worlds with the ability to represent the concepts and relations between the two in a unified way. We used rCOS as the underlying model for the specification of interfaces, contracts, and components. The concept of pseudo-category was then utilized to represent the business and application architecture specifications and the relationships contained within. Contracts are used for the specification of both IT and Business architecture components. The linkages between them is now established using the matching of the business component contracts with the application component contracts. Typically, the matching was based on manual process, in this paper we extend the work by considering automated component matching process. In this paper we provide implementation of the matching process using graph matching.
Reacting on human stubbornness in human-machine trajectory planning
Venue: arXiv cs.RO 2024
Year: 2024

In this paper, a method for a cooperative trajectory planning between a human and an automation is extended by a behavioral model of the human. This model can characterize the stubbornness of the human, which measures how strong the human adheres to his preferred trajectory. Accordingly, a static model is introduced indicating a link between the force in haptically coupled human-robot interactions and humans's stubbornness. The introduced stubbornness parameter enables an application-independent reaction of the automation for the cooperative trajectory planning. Simulation results in the context of human-machine cooperation in a care application show that the proposed behavioral model can quantitatively estimate the stubbornness of the interacting human, enabling a more targeted adaptation of the automation to the human behavior.