OpenRaiser/Intern-Atlas huggingface dataset:
RAG-Enhanced Commit Message Generation
Venue: arXiv cs.SE 2024
Year: 2024

Commit message is one of the most important textual information in software development and maintenance. However, it is time-consuming to write commit messages manually. Commit Message Generation (CMG) has become a research hotspot. Recently, several pre-trained language models (PLMs) and large language models (LLMs) with code capabilities have been introduced, demonstrating impressive performance on code-related tasks. Meanwhile, prior studies have explored the utilization of retrieval techniques for CMG, but it is still unclear what effects would emerge from combining advanced retrieval techniques with various generation models. This paper proposed REACT, a REtrieval-Augmented framework for CommiT message generation. It integrates advanced retrieval techniques with different PLMs and LLMs, to enhance the performance of these models on the CMG task. Specifically, a hybrid retriever is designed and used to retrieve the most relevant code diff and commit message pair as an exemplar. Then, the retrieved pair is utilized to guide and enhance the CMG task by PLMs and LLMs through fine-tuning and in-context learning. The experimental results show that REACT significantly enhances these models' performance on the CMG task, improving the BLEU score of CodeT5 by up to 55%, boosting Llama 3's BLEU score by 102%, and substantially surpassing all baselines.
"Violation of my body:" Perceptions of AI-generated non-consensual (intimate) imagery
Venue: arXiv cs.CY 2024
Year: 2024

AI technology has enabled the creation of deepfakes: hyper-realistic synthetic media. We surveyed 315 individuals in the U.S. on their views regarding the hypothetical non-consensual creation of deepfakes depicting them, including deepfakes portraying sexual acts. Respondents indicated strong opposition to creating and, even more so, sharing non-consensually created synthetic content, especially if that content depicts a sexual act. However, seeking out such content appeared more acceptable to some respondents. Attitudes around acceptability varied further based on the hypothetical creator's relationship to the participant, the respondent's gender and their attitudes towards sexual consent. This study provides initial insight into public perspectives of a growing threat and highlights the need for further research to inform social norms as well as ongoing policy conversations and technical developments in generative AI.
A Shape Change Enhancing Hierarchical Layout for the Pairwise Comparison of Directed Acyclic Graphs
Venue: arXiv cs.HC 2024
Year: 2024

Comparing directed acyclic graphs is essential in various fields such as healthcare, social media, finance, biology, and marketing. DAGs often result from contagion processes over networks, including information spreading, retweet activity, disease transmission, financial crisis propagation, malware spread, and gene mutations. For instance, in disease spreading, an infected patient can transmit the disease to contacts, making it crucial to analyze and predict scenarios. Similarly, in finance, understanding the effects of saving or not saving specific banks during a crisis is vital. Experts often need to identify small differences between DAGs, such as changes in a few nodes or edges. Even the presence or absence of a single edge can be significant. Visualization plays a crucial role in facilitating these comparisons. However, standard hierarchical layout algorithms struggle to visualize subtle changes effectively. The typical hierarchical layout, with the root on top, is preferred due to its performance in comparison to other layouts. Nevertheless, these standard algorithms prioritize single-graph aesthetics over comparison suitability, making it challenging for users to spot changes. To address this issue, we propose a layout that enhances shape changes in DAGs while minimizing the impact on aesthetics. Our approach involves outwardly swapping changes, altering the DAG's shape. We introduce new drawing criteria. Our layout builds upon a Sugiyama-like hierarchical layout and implements these criteria through two extensions. We designed it this way to maintain interchangeability and accommodate future optimizations, such as pseudo-nodes for edge crossing minimization. In our evaluations, our layout achieves excellent results, with edge crossing aesthetics averaging around 0.8 (on a scale of 0 to 1). Additionally, our layout outperforms the base implementation by an average of 60-75\%.
Learning Human Detected Differences in Directed Acyclic Graphs
Venue: arXiv cs.HC 2024
Year: 2024

Prior research has shown that human perception of similarity differs from mathematical measures in visual comparison tasks, including those involving directed acyclic graphs. This divergence can lead to missed differences and skepticism about algorithmic results. To address this, we aim to learn the structural differences humans detect in graphs visually. We want to visualize these human-detected differences alongside actual changes, enhancing credibility and aiding users in spotting overlooked differences. Our approach aligns with recent research in machine learning capturing human behavior. We provide a data augmentation algorithm, a dataset, and a machine learning model to support this task. This work fills a gap in learning differences in directed acyclic graphs and contributes to better comparative visualizations.
Prioritizing Potential Wetland Areas via Region-to-Region Knowledge Transfer and Adaptive Propagation
Venue: arXiv cs.IR 2024
Year: 2024

Wetlands are important to communities, offering benefits ranging from water purification, and flood protection to recreation and tourism. Therefore, identifying and prioritizing potential wetland areas is a critical decision problem. While data-driven solutions are feasible, this is complicated by significant data sparsity due to the low proportion of wetlands (3-6\%) in many areas of interest in the southwestern US. This makes it hard to develop data-driven models that can help guide the identification of additional wetland areas. To solve this limitation, we propose two strategies: (1) The first of these is knowledge transfer from regions with rich wetlands (such as the Eastern US) to sparser regions (such as the Southwestern area with few wetlands). Recognizing that these regions are likely to be very different from each other in terms of soil characteristics, population distribution, and land use, we propose a domain disentanglement strategy that identifies and transfers only the applicable aspects of the learned model. (2) We complement this with a spatial data enrichment strategy that relies on an adaptive propagation mechanism. This mechanism differentiates between node pairs that have positive and negative impacts on each other for Graph Neural Networks (GNNs). To summarize, given two spatial cells belonging to different regions, we identify domain-specific and domain-shareable features, and, for each region, we rely on adaptive propagation to enrich features with the features of surrounding cells. We conduct rigorous experiments to substantiate our proposed method's effectiveness, robustness, and scalability compared to state-of-the-art baselines. Additionally, an ablation study demonstrates that each module is essential in prioritizing potential wetlands, which justifies our assumption.
Observation Denoising in CYRUS Soccer Simulation 2D Team For RoboCup 2024
Venue: arXiv cs.RO 2024
Year: 2024

In the Soccer Simulation 2D environment, accurate observation is crucial for effective decision making. However, challenges such as partial observation and noisy data can hinder performance. To address these issues, we propose a denoising algorithm that leverages predictive modeling and intersection analysis to enhance the accuracy of observations. Our approach aims to mitigate the impact of noise and partial data, leading to improved gameplay performance. This paper presents the framework, implementation, and preliminary results of our algorithm, demonstrating its potential in refining observations in Soccer Simulation 2D. Cyrus 2D Team is using a combination of Helios, Gliders, and Cyrus base codes.
How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden States
Venue: arXiv cs.CL 2024
Year: 2024

Large language models (LLMs) rely on safety alignment to avoid responding to malicious user inputs. Unfortunately, jailbreak can circumvent safety guardrails, resulting in LLMs generating harmful content and raising concerns about LLM safety. Due to language models with intensive parameters often regarded as black boxes, the mechanisms of alignment and jailbreak are challenging to elucidate. In this paper, we employ weak classifiers to explain LLM safety through the intermediate hidden states. We first confirm that LLMs learn ethical concepts during pre-training rather than alignment and can identify malicious and normal inputs in the early layers. Alignment actually associates the early concepts with emotion guesses in the middle layers and then refines them to the specific reject tokens for safe generations. Jailbreak disturbs the transformation of early unethical classification into negative emotions. We conduct experiments on models from 7B to 70B across various model families to prove our conclusion. Overall, our paper indicates the intrinsical mechanism of LLM safety and how jailbreaks circumvent safety guardrails, offering a new perspective on LLM safety and reducing concerns. Our code is available at https://github.com/ydyjya/LLM-IHS-Explanation.
A Superalignment Framework in Autonomous Driving with Large Language Models
Venue: arXiv cs.RO 2024
Year: 2024

Over the last year, significant advancements have been made in the realms of large language models (LLMs) and multi-modal large language models (MLLMs), particularly in their application to autonomous driving. These models have showcased remarkable abilities in processing and interacting with complex information. In autonomous driving, LLMs and MLLMs are extensively used, requiring access to sensitive vehicle data such as precise locations, images, and road conditions. These data are transmitted to an LLM-based inference cloud for advanced analysis. However, concerns arise regarding data security, as the protection against data and privacy breaches primarily depends on the LLM's inherent security measures, without additional scrutiny or evaluation of the LLM's inference outputs. Despite its importance, the security aspect of LLMs in autonomous driving remains underexplored. Addressing this gap, our research introduces a novel security framework for autonomous vehicles, utilizing a multi-agent LLM approach. This framework is designed to safeguard sensitive information associated with autonomous vehicles from potential leaks, while also ensuring that LLM outputs adhere to driving regulations and align with human values. It includes mechanisms to filter out irrelevant queries and verify the safety and reliability of LLM outputs. Utilizing this framework, we evaluated the security, privacy, and cost aspects of eleven large language model-driven autonomous driving cues. Additionally, we performed QA tests on these driving prompts, which successfully demonstrated the framework's efficacy.
Evolution-aware VAriance (EVA) Coreset Selection for Medical Image Classification
Venue: arXiv cs.CV 2024
Year: 2024

In the medical field, managing high-dimensional massive medical imaging data and performing reliable medical analysis from it is a critical challenge, especially in resource-limited environments such as remote medical facilities and mobile devices. This necessitates effective dataset compression techniques to reduce storage, transmission, and computational cost. However, existing coreset selection methods are primarily designed for natural image datasets, and exhibit doubtful effectiveness when applied to medical image datasets due to challenges such as intra-class variation and inter-class similarity. In this paper, we propose a novel coreset selection strategy termed as Evolution-aware VAriance (EVA), which captures the evolutionary process of model training through a dual-window approach and reflects the fluctuation of sample importance more precisely through variance measurement. Extensive experiments on medical image datasets demonstrate the effectiveness of our strategy over previous SOTA methods, especially at high compression rates. EVA achieves 98.27% accuracy with only 10% training data, compared to 97.20% for the full training set. None of the compared baseline methods can exceed Random at 5% selection rate, while EVA outperforms Random by 5.61%, showcasing its potential for efficient medical image analysis.
Diverse 3D Human Pose Generation in Scenes based on Decoupled Structure
Venue: arXiv cs.CV 2024
Year: 2024

This paper presents a novel method for generating diverse 3D human poses in scenes with semantic control. Existing methods heavily rely on the human-scene interaction dataset, resulting in a limited diversity of the generated human poses. To overcome this challenge, we propose to decouple the pose and interaction generation process. Our approach consists of three stages: pose generation, contact generation, and putting human into the scene. We train a pose generator on the human dataset to learn rich pose prior, and a contact generator on the human-scene interaction dataset to learn human-scene contact prior. Finally, the placing module puts the human body into the scene in a suitable and natural manner. The experimental results on the PROX dataset demonstrate that our method produces more physically plausible interactions and exhibits more diverse human poses. Furthermore, experiments on the MP3D-R dataset further validates the generalization ability of our method.
Demystifying the Characteristics for Smart Contract Upgrades
Venue: arXiv cs.SE 2024
Year: 2024

Upgradable smart contracts play an important role in the decentralized application ecosystem, to support routine maintenance, security patching, and feature additions. In this paper, we conduct an empirical study on proxy-based upgradable smart contracts to understand the characteristics of contract upgrading. Through our study on 57,118 open source proxy contracts, we found that 583 contracts have ever been upgraded on Ethereum, involving 973 unique implementation contract versions. The results show that developers often intend to improve usability of contracts if upgrading, where functionality addition and update are the most frequent upgrade intentions. We investigated the practical impacts of contract upgrades, e.g., breaking changes causing compatibility issues, storage collisions and initialization risks leading to security vulnerabilities. The results demonstrate that there are 4,334 ABI breaking changes due to the upgrades of 276 proxies, causing real-world broken usages within 584 transactions witnessed by the blockchain; 36 contract upgrades had storage collisions and five proxies with 59 implementation contracts are vulnerable to initialization attacks.
Region of Interest Loss for Anonymizing Learned Image Compression
Venue: arXiv cs.CV 2024
Year: 2024

The use of AI in public spaces continually raises concerns about privacy and the protection of sensitive data. An example is the deployment of detection and recognition methods on humans, where images are provided by surveillance cameras. This results in the acquisition of great amounts of sensitive data, since the capture and transmission of images taken by such cameras happens unaltered, for them to be received by a server on the network. However, many applications do not explicitly require the identity of a given person in a scene; An anonymized representation containing information of the person's position while preserving the context of them in the scene suffices. We show how using a customized loss function on region of interests (ROI) can achieve sufficient anonymization such that human faces become unrecognizable while persons are kept detectable, by training an end-to-end optimized autoencoder for learned image compression that utilizes the flexibility of the learned analysis and reconstruction transforms for the task of mutating parts of the compression result. This approach enables compression and anonymization in one step on the capture device, instead of transmitting sensitive, nonanonymized data over the network. Additionally, we evaluate how this anonymization impacts the average precision of pre-trained foundation models on detecting faces (MTCNN) and humans (YOLOv8) in comparison to non-ANN based methods, while considering compression rate and latency.
CorrMAE: Pre-training Correspondence Transformers with Masked Autoencoder
Venue: arXiv cs.CV 2024
Year: 2024

Pre-training has emerged as a simple yet powerful methodology for representation learning across various domains. However, due to the expensive training cost and limited data, pre-training has not yet been extensively studied in correspondence pruning. To tackle these challenges, we propose a pre-training method to acquire a generic inliers-consistent representation by reconstructing masked correspondences, providing a strong initial representation for downstream tasks. Toward this objective, a modicum of true correspondences naturally serve as input, thus significantly reducing pre-training overhead. In practice, we introduce CorrMAE, an extension of the mask autoencoder framework tailored for the pre-training of correspondence pruning. CorrMAE involves two main phases, \ie correspondence learning and matching point reconstruction, guiding the reconstruction of masked correspondences through learning visible correspondence consistency. Herein, we employ a dual-branch structure with an ingenious positional encoding to reconstruct unordered and irregular correspondences. Also, a bi-level designed encoder is proposed for correspondence learning, which offers enhanced consistency learning capability and transferability. Extensive experiments have shown that the model pre-trained with our CorrMAE outperforms prior work on multiple challenging benchmarks. Meanwhile, our CorrMAE is primarily a task-driven pre-training method, and can achieve notable improvements for downstream tasks by pre-training on the targeted dataset. We hope this work can provide a starting point for correspondence pruning pre-training.
A Survey on Text-guided 3D Visual Grounding: Elements, Recent Advances, and Future Directions
Venue: arXiv cs.CV 2024
Year: 2024

Text-guided 3D visual grounding (T-3DVG), which aims to locate a specific object that semantically corresponds to a language query from a complicated 3D scene, has drawn increasing attention in the 3D research community over the past few years. Compared to 2D visual grounding, this task presents great potential and challenges due to its closer proximity to the real world and the complexity of data collection and 3D point cloud source processing. In this survey, we attempt to provide a comprehensive overview of the T-3DVG progress, including its fundamental elements, recent research advances, and future research directions. To the best of our knowledge, this is the first systematic survey on the T-3DVG task. Specifically, we first provide a general structure of the T-3DVG pipeline with detailed components in a tutorial style, presenting a complete background overview. Then, we summarize the existing T-3DVG approaches into different categories and analyze their strengths and weaknesses. We also present the benchmark datasets and evaluation metrics to assess their performances. Finally, we discuss the potential limitations of existing T-3DVG and share some insights on several promising research directions. The latest papers are continually collected at https://github.com/liudaizong/Awesome-3D-Visual-Grounding.
Multi-Stain Multi-Level Convolutional Network for Multi-Tissue Breast Cancer Image Segmentation
Venue: arXiv cs.CV 2024
Year: 2024

Digital pathology and microscopy image analysis are widely employed in the segmentation of digitally scanned IHC slides, primarily to identify cancer and pinpoint regions of interest (ROI) indicative of tumor presence. However, current ROI segmentation models are either stain-specific or suffer from the issues of stain and scanner variance due to different staining protocols or modalities across multiple labs. Also, tissues like Ductal Carcinoma in Situ (DCIS), acini, etc. are often classified as Tumors due to their structural similarities and color compositions. In this paper, we proposed a novel convolutional neural network (CNN) based Multi-class Tissue Segmentation model for histopathology whole-slide Breast slides which classify tumors and segments other tissue regions such as Ducts, acini, DCIS, Squamous epithelium, Blood Vessels, Necrosis, etc. as a separate class. Our unique pixel-aligned non-linear merge across spatial resolutions empowers models with both local and global fields of view for accurate detection of various classes. Our proposed model is also able to separate bad regions such as folds, artifacts, blurry regions, bubbles, etc. from tissue regions using multi-level context from different resolutions of WSI. Multi-phase iterative training with context-aware augmentation and increasing noise was used to efficiently train a multi-stain generic model with partial and noisy annotations from 513 slides. Our training pipeline used 12 million patches generated using context-aware augmentations which made our model stain and scanner invariant across data sources. To extrapolate stain and scanner invariance, our model was evaluated on 23000 patches which were for a completely new stain (Hematoxylin and Eosin) from a completely new scanner (Motic) from a different lab. The mean IOU was 0.72 which is on par with model performance on other data sources and scanners.
Dispersive Vertex Guarding for Simple and Non-Simple Polygons
Venue: arXiv cs.CG 2024
Year: 2024

We study the Dispersive Art Gallery Problem with vertex guards: Given a polygon $\mathcal{P}$, with pairwise geodesic Euclidean vertex distance of at least $1$, and a rational number $\ell$; decide whether there is a set of vertex guards such that $\mathcal{P}$ is guarded, and the minimum geodesic Euclidean distance between any two guards (the so-called dispersion distance) is at least $\ell$. We show that it is NP-complete to decide whether a polygon with holes has a set of vertex guards with dispersion distance $2$. On the other hand, we provide an algorithm that places vertex guards in simple polygons at dispersion distance at least $2$. This result is tight, as there are simple polygons in which any vertex guard set has a dispersion distance of at most $2$.
STARLING: Self-supervised Training of Text-based Reinforcement Learning Agent with Large Language Models
Venue: arXiv cs.LG 2024
Year: 2024

Interactive fiction games have emerged as an important application to improve the generalization capabilities of language-based reinforcement learning (RL) agents. Existing environments for interactive fiction games are domain-specific or time-consuming to generate and do not train the RL agents to master a specific set of skills. In this work, we introduce an interactive environment for self-supervised RL, STARLING, for text-based games that bootstraps the text-based RL agents with automatically generated games (based on the seed set of game ideas) to boost the performance and generalization capabilities to reach a goal of the target environment. These games let the agent hone their skills on a predefined set of tasks. We create and test an environment with 100 games, generated using this automated framework that uses large language models (GPT-3) and an interactive fiction game engine (based on Inform7) to provide the user with the ability to generate more games under minimal human supervision. Experimental results based on both the human participants and baseline text-based RL agents reveal that current state-of-the-art text-based RL agents cannot use previously learned skills in new situations at the level humans can. These results enforce STARLING's potential to serve as a sandbox environment for further research in self-supervised text-based RL.