OpenRaiser/Intern-Atlas huggingface dataset:
HieraFashDiff: Hierarchical Fashion Design with Multi-stage Diffusion Models
Venue: arXiv cs.CV 2024
Year: 2024

Fashion design is a challenging and complex process.Recent works on fashion generation and editing are all agnostic of the actual fashion design process, which limits their usage in practice.In this paper, we propose a novel hierarchical diffusion-based framework tailored for fashion design, coined as HieraFashDiff. Our model is designed to mimic the practical fashion design workflow, by unraveling the denosing process into two successive stages: 1) an ideation stage that generates design proposals given high-level concepts and 2) an iteration stage that continuously refines the proposals using low-level attributes. Our model supports fashion design generation and fine-grained local editing in a single framework. To train our model, we contribute a new dataset of full-body fashion images annotated with hierarchical text descriptions. Extensive evaluations show that, as compared to prior approaches, our method can generate fashion designs and edited results with higher fidelity and better prompt adherence, showing its promising potential to augment the practical fashion design workflow. Code and Dataset are available at https://github.com/haoli-zbdbc/hierafashdiff.
Model Editing at Scale leads to Gradual and Catastrophic Forgetting
Venue: arXiv cs.CL 2024
Year: 2024

Editing knowledge in large language models is an attractive capability to have which allows us to correct incorrectly learnt facts during pre-training, as well as update the model with an ever-growing list of new facts. While existing model editing techniques have shown promise, they are usually evaluated using metrics for reliability, specificity and generalization over one or few edits. We argue that for model editing to have practical utility, we must be able to make multiple edits to the same model. With this in mind, we evaluate the current model editing methods at scale, focusing on two state of the art methods: ROME and MEMIT. We find that as the model is edited sequentially with multiple facts, it continually forgets previously edited facts and the ability to perform downstream tasks. This forgetting happens in two phases -- an initial gradual but progressive forgetting phase followed by abrupt or catastrophic forgetting phase. Both gradual and catastrophic forgetting limit the usefulness of model editing methods at scale -- the former making model editing less effective as multiple edits are made to the model while the latter caps the scalability of such model editing methods. Our analysis also highlights other key limitations of ROME and MEMIT at scale. With our work, we push for the development and evaluation of model editing methods keeping scalability in mind.
Your Instructions Are Not Always Helpful: Assessing the Efficacy of Instruction Fine-tuning for Software Vulnerability Detection
Venue: arXiv cs.SE 2024
Year: 2024

Software, while beneficial, poses potential cybersecurity risks due to inherent vulnerabilities. Detecting these vulnerabilities is crucial, and deep learning has shown promise as an effective tool for this task due to its ability to perform well without extensive feature engineering. However, a challenge in deploying deep learning for vulnerability detection is the limited availability of training data. Recent research highlights the deep learning efficacy in diverse tasks. This success is attributed to instruction fine-tuning, a technique that remains under-explored in the context of vulnerability detection. This paper investigates the capability of models, specifically a recent language model, to generalize beyond the programming languages used in their training data. It also examines the role of natural language instructions in enhancing this generalization. Our study evaluates the model performance on a real-world dataset to predict vulnerable code. We present key insights and lessons learned, contributing to understanding the deep learning application in software vulnerability detection.
GWPT: A Green Word-Embedding-based POS Tagger
Venue: arXiv cs.CL 2024
Year: 2024

As a fundamental tool for natural language processing (NLP), the part-of-speech (POS) tagger assigns the POS label to each word in a sentence. A novel lightweight POS tagger based on word embeddings is proposed and named GWPT (green word-embedding-based POS tagger) in this work. Following the green learning (GL) methodology, GWPT contains three modules in cascade: 1) representation learning, 2) feature learning, and 3) decision learning modules. The main novelty of GWPT lies in representation learning. It uses non-contextual or contextual word embeddings, partitions embedding dimension indices into low-, medium-, and high-frequency sets, and represents them with different N-grams. It is shown by experimental results that GWPT offers state-of-the-art accuracies with fewer model parameters and significantly lower computational complexity in both training and inference as compared with deep-learning-based methods.
Feature Selection via Maximizing Distances between Class Conditional Distributions
Venue: arXiv cs.LG 2024
Year: 2024

For many data-intensive tasks, feature selection is an important preprocessing step. However, most existing methods do not directly and intuitively explore the intrinsic discriminative information of features. We propose a novel feature selection framework based on the distance between class conditional distributions, measured by integral probability metrics (IPMs). Our framework directly explores the discriminative information of features in the sense of distributions for supervised classification. We analyze the theoretical and practical aspects of IPMs for feature selection, construct criteria based on IPMs. We propose several variant feature selection methods of our framework based on the 1-Wasserstein distance and implement them on real datasets from different domains. Experimental results show that our framework can outperform state-of-the-art methods in terms of classification accuracy and robustness to perturbations.
Study Features via Exploring Distribution Structure
Venue: arXiv cs.LG 2024
Year: 2024

In this paper, we present a novel framework for data redundancy measurement based on probabilistic modeling of datasets, and a new criterion for redundancy detection that is resilient to noise. We also develop new methods for data redundancy reduction using both deterministic and stochastic optimization techniques. Our framework is flexible and can handle different types of features, and our experiments on benchmark datasets demonstrate the effectiveness of our methods. We provide a new perspective on feature selection, and propose effective and robust approaches for both supervised and unsupervised learning problems.
Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applications
Venue: arXiv cs.CR 2024
Year: 2024

The critical challenge of prompt injection attacks in Large Language Models (LLMs) integrated applications, a growing concern in the Artificial Intelligence (AI) field. Such attacks, which manipulate LLMs through natural language inputs, pose a significant threat to the security of these applications. Traditional defense strategies, including output and input filtering, as well as delimiter use, have proven inadequate. This paper introduces the 'Signed-Prompt' method as a novel solution. The study involves signing sensitive instructions within command segments by authorized users, enabling the LLM to discern trusted instruction sources. The paper presents a comprehensive analysis of prompt injection attack patterns, followed by a detailed explanation of the Signed-Prompt concept, including its basic architecture and implementation through both prompt engineering and fine-tuning of LLMs. Experiments demonstrate the effectiveness of the Signed-Prompt method, showing substantial resistance to various types of prompt injection attacks, thus validating its potential as a robust defense strategy in AI security.
A Human-Powered Public Display that Nudges Social Biking via Motion Gesturing
Venue: arXiv cs.HC 2024
Year: 2024

The WeWatt bike serves as an energy station that enables passers-by to charge their mobile devices through physical activity. However, despite multiple people using it simultaneously, the bike is typically used individually. To address this limitation, we developed the WeWattTree, an installation utilising human-powered energy to filter environmental air. Through the orchestration of subtle motion gestures, our goal is to entice passers-by to participate and encourage them to socially interact, synchronising their pace. In this work-in-progress, we provide insights into the prototyping process, combining physical experimentation and computational simulation, and delve into the underlying concepts of our grammar of motion gestures. We highlight how a single design effectively merged multiple functionalities, how the role of material characteristics shaped the interaction design, and discuss the potential for social performances as captivating public displays.
Go-Explore for Residential Energy Management
Venue: arXiv cs.AI 2024
Year: 2024

Reinforcement learning is commonly applied in residential energy management, particularly for optimizing energy costs. However, RL agents often face challenges when dealing with deceptive and sparse rewards in the energy control domain, especially with stochastic rewards. In such situations, thorough exploration becomes crucial for learning an optimal policy. Unfortunately, the exploration mechanism can be misled by deceptive reward signals, making thorough exploration difficult. Go-Explore is a family of algorithms which combines planning methods and reinforcement learning methods to achieve efficient exploration. We use the Go-Explore algorithm to solve the cost-saving task in residential energy management problems and achieve an improvement of up to 19.84\% compared to the well-known reinforcement learning algorithms.
Sparsity-based background removal for STORM super-resolution images
Venue: arXiv cs.CV 2024
Year: 2024

Single-molecule localization microscopy techniques, like stochastic optical reconstruction microscopy (STORM), visualize biological specimens by stochastically exciting sparse blinking emitters. The raw images suffer from unwanted background fluorescence, which must be removed to achieve super-resolution. We introduce a sparsity-based background removal method by adapting a neural network (SLNet) from a different microscopy domain. The SLNet computes a low-rank representation of the images, and then, by subtracting it from the raw images, the sparse component is computed, representing the frames without the background. We compared our approach with widely used background removal methods, such as the median background removal or the rolling ball algorithm, on two commonly used STORM datasets, one glial cell, and one microtubule dataset. The SLNet delivers STORM frames with less background, leading to higher emitters' localization precision and higher-resolution reconstructed images than commonly used methods. Notably, the SLNet is lightweight and easily trainable (<5 min). Since it is trained in an unsupervised manner, no prior information is required and can be applied to any STORM dataset. We uploaded a pre-trained SLNet to the Bioimage model zoo, easily accessible through ImageJ. Our results show that our sparse decomposition method could be an essential and efficient STORM pre-processing tool.
Metalevel transformation of strategies
Venue: arXiv cs.LO 2024
Year: 2024

In the reflective Maude specification language, based on rewriting logic, a strategy language has been introduced to control rule rewriting while avoiding complex and verbose metalevel programs. However, just as multiple levels of reflection are required for some metaprogramming tasks, reflective manipulation and generation of strategies are convenient in multiple situations. Some examples of reflective strategy transformations are presented, which implement special forms of evaluation or extend the strategy language while preserving its advantages.
Activations and Gradients Compression for Model-Parallel Training
Venue: arXiv cs.LG 2024
Year: 2024

Large neural networks require enormous computational clusters of machines. Model-parallel training, when the model architecture is partitioned sequentially between workers, is a popular approach for training modern models. Information compression can be applied to decrease workers communication time, as it is often a bottleneck in such systems. This work explores how simultaneous compression of activations and gradients in model-parallel distributed training setup affects convergence. We analyze compression methods such as quantization and TopK compression, and also experiment with error compensation techniques. Moreover, we employ TopK with AQ-SGD per-batch error feedback approach. We conduct experiments on image classification and language model fine-tuning tasks. Our findings demonstrate that gradients require milder compression rates than activations. We observe that $K=10\%$ is the lowest TopK compression level, which does not harm model convergence severely. Experiments also show that models trained with TopK perform well only when compression is also applied during inference. We find that error feedback techniques do not improve model-parallel training compared to plain compression, but allow model inference without compression with almost no quality drop. Finally, when applied with the AQ-SGD approach, TopK stronger than with $ K=30\%$ worsens model performance significantly.
Phenotyping calcification in vascular tissues using artificial intelligence
Venue: arXiv cs.CV 2024
Year: 2024

Vascular calcification is implicated as an important factor in major adverse cardiovascular events (MACE), including heart attack and stroke. A controversy remains over how to integrate the diverse forms of vascular calcification into clinical risk assessment tools. Even the commonly used calcium score for coronary arteries, which assumes risk scales positively with total calcification, has important inconsistencies. Fundamental studies are needed to determine how risk is influenced by the diverse calcification phenotypes. However, studies of these kinds are hindered by the lack of high-throughput, objective, and non-destructive tools for classifying calcification in imaging data sets. Here, we introduce a new classification system for phenotyping calcification along with a semi-automated, non-destructive pipeline that can distinguish these phenotypes in even atherosclerotic tissues. The pipeline includes a deep-learning-based framework for segmenting lipid pools in noisy micro-CT images and an unsupervised clustering framework for categorizing calcification based on size, clustering, and topology. This approach is illustrated for five vascular specimens, providing phenotyping for thousands of calcification particles across as many as 3200 images in less than seven hours. Average Dice Similarity Coefficients of 0.96 and 0.87 could be achieved for tissue and lipid pool, respectively, with training and validation needed on only 13 images despite the high heterogeneity in these tissues. By introducing an efficient and comprehensive approach to phenotyping calcification, this work enables large-scale studies to identify a more reliable indicator of the risk of cardiovascular events, a leading cause of global mortality and morbidity.
VeCAF: Vision-language Collaborative Active Finetuning with Training Objective Awareness
Venue: arXiv cs.CV 2024
Year: 2024

Finetuning a pretrained vision model (PVM) is a common technique for learning downstream vision tasks. However, the conventional finetuning process with randomly sampled data points results in diminished training efficiency. To address this drawback, we propose a novel approach, Vision-language Collaborative Active Finetuning (VeCAF). With the emerging availability of labels and natural language annotations of images through web-scale crawling or controlled generation, VeCAF makes use of these information to perform parametric data selection for PVM finetuning. VeCAF incorporates the finetuning objective to select significant data points that effectively guide the PVM towards faster convergence to meet the performance goal. This process is assisted by the inherent semantic richness of the text embedding space which we use to augment image features. Furthermore, the flexibility of text-domain augmentation allows VeCAF to handle out-of-distribution scenarios without external data. Extensive experiments show the leading performance and high computational efficiency of VeCAF that is superior to baselines in both in-distribution and out-of-distribution image classification tasks. On ImageNet, VeCAF uses up to 3.3x less training batches to reach the target performance compared to full finetuning, and achieves an accuracy improvement of 2.7% over the state-of-the-art active finetuning method with the same number of batches.
The What, Why, and How of Context Length Extension Techniques in Large Language Models -- A Detailed Survey
Venue: arXiv cs.CL 2024
Year: 2024

The advent of Large Language Models (LLMs) represents a notable breakthrough in Natural Language Processing (NLP), contributing to substantial progress in both text comprehension and generation. However, amidst these advancements, it is noteworthy that LLMs often face a limitation in terms of context length extrapolation. Understanding and extending the context length for LLMs is crucial in enhancing their performance across various NLP applications. In this survey paper, we delve into the multifaceted aspects of exploring why it is essential, and the potential transformations that superior techniques could bring to NLP applications. We study the inherent challenges associated with extending context length and present an organized overview of the existing strategies employed by researchers. Additionally, we discuss the intricacies of evaluating context extension techniques and highlight the open challenges that researchers face in this domain. Furthermore, we explore whether there is a consensus within the research community regarding evaluation standards and identify areas where further agreement is needed. This comprehensive survey aims to serve as a valuable resource for researchers, guiding them through the nuances of context length extension techniques and fostering discussions on future advancements in this evolving field.
Survey of Learning-based Approaches for Robotic In-Hand Manipulation
Venue: arXiv cs.RO 2024
Year: 2024

Human dexterity is an invaluable capability for precise manipulation of objects in complex tasks. The capability of robots to similarly grasp and perform in-hand manipulation of objects is critical for their use in the ever changing human environment, and for their ability to replace manpower. In recent decades, significant effort has been put in order to enable in-hand manipulation capabilities to robotic systems. Initial robotic manipulators followed carefully programmed paths, while later attempts provided a solution based on analytical modeling of motion and contact. However, these have failed to provide practical solutions due to inability to cope with complex environments and uncertainties. Therefore, the effort has shifted to learning-based approaches where data is collected from the real world or through a simulation, during repeated attempts to complete various tasks. The vast majority of learning approaches focused on learning data-based models that describe the system to some extent or Reinforcement Learning (RL). RL, in particular, has seen growing interest due to the remarkable ability to generate solutions to problems with minimal human guidance. In this survey paper, we track the developments of learning approaches for in-hand manipulations and, explore the challenges and opportunities. This survey is designed both as an introduction for novices in the field with a glossary of terms as well as a guide of novel advances for advanced practitioners.
A Lexicon for Studying Radicalization in Incel Communities
Venue: arXiv cs.CL 2024
Year: 2024

Incels are an extremist online community of men who believe in an ideology rooted in misogyny, racism, the glorification of violence, and dehumanization. In their online forums, they use an extensive, evolving cryptolect - a set of ingroup terms that have meaning within the group, reflect the ideology, demonstrate membership in the community, and are difficult for outsiders to understand. This paper presents a lexicon with terms and definitions for common incel root words, prefixes, and affixes. The lexicon is text-based for use in automated analysis and is derived via a Qualitative Content Analysis of the most frequent incel words, their structure, and their meaning on five of the most active incel communities from 2016 to 2023. This lexicon will support future work examining radicalization and deradicalization/disengagement within the community.
ADMIn: Attacks on Dataset, Model and Input. A Threat Model for AI Based Software
Venue: arXiv cs.CR 2024
Year: 2024

Machine learning (ML) and artificial intelligence (AI) techniques have now become commonplace in software products and services. When threat modelling a system, it is therefore important that we consider threats unique to ML and AI techniques, in addition to threats to our software. In this paper, we present a threat model that can be used to systematically uncover threats to AI based software. The threat model consists of two main parts, a model of the software development process for AI based software and an attack taxonomy that has been developed using attacks found in adversarial AI research. We apply the threat model to two real life AI based software and discuss the process and the threats found.
Convolutional Neural Network Compression via Dynamic Parameter Rank Pruning
Venue: arXiv cs.CV 2024
Year: 2024

While Convolutional Neural Networks (CNNs) excel at learning complex latent-space representations, their over-parameterization can lead to overfitting and reduced performance, particularly with limited data. This, alongside their high computational and memory demands, limits the applicability of CNNs for edge deployment. Low-rank matrix approximation has emerged as a promising approach to reduce CNN parameters, but its application presents challenges including rank selection and performance loss. To address these issues, we propose an efficient training method for CNN compression via dynamic parameter rank pruning. Our approach integrates efficient matrix factorization and novel regularization techniques, forming a robust framework for dynamic rank reduction and model compression. We use Singular Value Decomposition (SVD) to model low-rank convolutional filters and dense weight matrices and we achieve model compression by training the SVD factors with back-propagation in an end-to-end way. We evaluate our method on an array of modern CNNs, including ResNet-18, ResNet-20, and ResNet-32, and datasets like CIFAR-10, CIFAR-100, and ImageNet (2012), showcasing its applicability in computer vision. Our experiments show that the proposed method can yield substantial storage savings while maintaining or even enhancing classification performance.