OpenRaiser/Intern-Atlas huggingface dataset:
A Combined Feature Embedding Tools for Multi-Class Software Defect and Identification
Venue: arXiv cs.SE 2024
Year: 2024

In software, a vulnerability is a defect in a program that attackers might utilize to acquire unauthorized access, alter system functions, and acquire information. These vulnerabilities arise from programming faults, design flaws, incorrect setups, and a lack of security protective measures. To mitigate these vulnerabilities, regular software upgrades, code reviews, safe development techniques, and the use of security tools to find and fix problems have been important. Several ways have been delivered in recent studies to address difficulties related to software vulnerabilities. However, previous approaches have significant limitations, notably in feature embedding and precisely recognizing specific vulnerabilities. To overcome these drawbacks, we present CodeGraphNet, an experimental method that combines GraphCodeBERT and Graph Convolutional Network (GCN) approaches, where, CodeGraphNet reveals data in a high-dimensional vector space, with comparable or related properties grouped closer together. This method captures intricate relationships between features, providing for more exact identification and separation of vulnerabilities. Using this feature embedding approach, we employed four machine learning models, applying both independent testing and 10-fold cross-validation. The DeepTree model, which is a hybrid of a Decision Tree and a Neural Network, outperforms state-of-the-art approaches. In additional validation, we evaluated our model using feature embeddings from LSA, GloVe, FastText, CodeBERT and GraphCodeBERT, and found that the CodeGraphNet method presented improved vulnerability identification with 98% of accuracy. Our model was tested on a real-time dataset to determine its capacity to handle real-world data and to focus on defect localization, which might influence future studies.
Semi-analytical model for the calculation of solar radiation pressure and its effects on a LEO satellite with predicting the change in position vectors using machine learning techniques
Venue: arXiv cs.CE 2024
Year: 2024

The rapid increase in the deployment of Low Earth Orbit (LEO) satellites, catering to diverse applications such as communication, Earth observation, environmental monitoring, and scientific research, has significantly amplified the complexity of trajectory management. The current work focuses on calculating and analyzing perturbation effects on a satellite's anticipated trajectory in LEO, considering Solar Radiation Pressure (SRP) as the main perturbing force. The acceleration due to SRP and it's effects on the satellite was calculated using a custom-built Python module mainly based on the hypothesis of the cannonball model. The study demonstrates the effectiveness of the proposed model through comprehensive simulations and comparisons with existing analytical and numerical methods. Here, the primary Keplerian orbital characteristics were employed to analyze a simulated low-earth orbit LEO satellite, initially visualizing the satellite's trajectory and ground tracks at a designated altitude. The study also focuses on a comparative analysis of ground stations, primarily considering the main regions of the subcontinent, with revisit time as the key parameter for comparison. In the end, we combine analytical techniques with Machine Learning (ML) algorithms to predict changes in the position vectors of the satellite. Using ML techniques, the model can adaptively learn and refine predictions based on historical data and real-time input, thus improving accuracy over time. In addition, the incorporation of analytical methods allows for a deeper understanding of the underlying physics governing satellite motion, enabling more precise adjustments and corrections.
Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokens
Venue: arXiv cs.LG 2024
Year: 2024

We reveal that low-bit quantization favors undertrained large language models (LLMs) by observing that models with larger sizes or fewer training tokens experience less quantization-induced degradation (QiD) when applying low-bit quantization, whereas smaller models with extensive training tokens suffer significant QiD. To gain deeper insights into this trend, we study over 1500 quantized LLM checkpoints of various sizes and at different training levels (undertrained or fully trained) in a controlled setting, deriving scaling laws for understanding the relationship between QiD and factors such as the number of training tokens, model size and bit width. With the derived scaling laws, we propose a novel perspective that we can use QiD to measure an LLM's training levels and determine the number of training tokens required for fully training LLMs of various sizes. Moreover, we use the scaling laws to predict the quantization performance of different-sized LLMs trained with 100 trillion tokens. Our projection shows that the low-bit quantization performance of future models, which are expected to be trained with over 100 trillion tokens, may NOT be desirable. This poses a potential challenge for low-bit quantization in the future and highlights the need for awareness of a model's training level when evaluating low-bit quantization research. To facilitate future research on this problem, we release all the 1500+ quantized checkpoints used in this work at https://huggingface.co/Xu-Ouyang.
The Partially Observable Off-Switch Game
Venue: arXiv cs.GT 2024
Year: 2024

A wide variety of goals could cause an AI to disable its off switch because "you can't fetch the coffee if you're dead" (Russell 2019). Prior theoretical work on this shutdown problem assumes that humans know everything that AIs do. In practice, however, humans have only limited information. Moreover, in many of the settings where the shutdown problem is most concerning, AIs might have vast amounts of private information. To capture these differences in knowledge, we introduce the Partially Observable Off-Switch Game (PO-OSG), a game-theoretic model of the shutdown problem with asymmetric information. Unlike when the human has full observability, we find that in optimal play, even AI agents assisting perfectly rational humans sometimes avoid shutdown. As expected, increasing the amount of communication or information available always increases (or leaves unchanged) the agents' expected common payoff. But counterintuitively, introducing bounded communication can make the AI defer to the human less in optimal play even though communication mitigates information asymmetry. In particular, communication sometimes enables new optimal behavior requiring strategic AI deference to achieve outcomes that were previously inaccessible. Thus, designing safe artificial agents in the presence of asymmetric information requires careful consideration of the tradeoffs between maximizing payoffs (potentially myopically) and maintaining AIs' incentives to defer to humans.
Efficient Multi-modal Large Language Models via Visual Token Grouping
Venue: arXiv cs.CV 2024
Year: 2024

The development of Multi-modal Large Language Models (MLLMs) enhances Large Language Models (LLMs) with the ability to perceive data formats beyond text, significantly advancing a range of downstream applications, such as visual question answering and image captioning. However, the substantial computational costs associated with processing high-resolution images and videos pose a barrier to their broader adoption. To address this challenge, compressing vision tokens in MLLMs has emerged as a promising approach to reduce inference costs. While existing methods conduct token reduction in the feature alignment phase. In this paper, we introduce VisToG, a novel grouping mechanism that leverages the capabilities of pre-trained vision encoders to group similar image segments without the need for segmentation masks. Specifically, we concatenate semantic tokens to represent image semantic segments after the linear projection layer before feeding into the vision encoder. Besides, with the isolated attention we adopt, VisToG can identify and eliminate redundant visual tokens utilizing the prior knowledge in the pre-trained vision encoder, which effectively reduces computational demands. Extensive experiments demonstrate the effectiveness of VisToG, maintaining 98.1% of the original performance while achieving a reduction of over 27\% inference time.
DreamCache: Finetuning-Free Lightweight Personalized Image Generation via Feature Caching
Venue: arXiv cs.CV 2024
Year: 2024

Personalized image generation requires text-to-image generative models that capture the core features of a reference subject to allow for controlled generation across different contexts. Existing methods face challenges due to complex training requirements, high inference costs, limited flexibility, or a combination of these issues. In this paper, we introduce DreamCache, a scalable approach for efficient and high-quality personalized image generation. By caching a small number of reference image features from a subset of layers and a single timestep of the pretrained diffusion denoiser, DreamCache enables dynamic modulation of the generated image features through lightweight, trained conditioning adapters. DreamCache achieves state-of-the-art image and text alignment, utilizing an order of magnitude fewer extra parameters, and is both more computationally effective and versatile than existing models.
Leveraging Large Language Models and Topic Modeling for Toxicity Classification
Venue: arXiv cs.CL 2024
Year: 2024

Content moderation and toxicity classification represent critical tasks with significant social implications. However, studies have shown that major classification models exhibit tendencies to magnify or reduce biases and potentially overlook or disadvantage certain marginalized groups within their classification processes. Researchers suggest that the positionality of annotators influences the gold standard labels in which the models learned from propagate annotators' bias. To further investigate the impact of annotator positionality, we delve into fine-tuning BERTweet and HateBERT on the dataset while using topic-modeling strategies for content moderation. The results indicate that fine-tuning the models on specific topics results in a notable improvement in the F1 score of the models when compared to the predictions generated by other prominent classification models such as GPT-4, PerspectiveAPI, and RewireAPI. These findings further reveal that the state-of-the-art large language models exhibit significant limitations in accurately detecting and interpreting text toxicity contrasted with earlier methodologies. Code is available at https://github.com/aheldis/Toxicity-Classification.git.
A Parallel Scan Algorithm in the Tensor Core Unit Model
Venue: arXiv cs.DC 2024
Year: 2024

We present a parallel scan (prefix sum) algorithm in the Tensor Core Unit (TCU) model of computation. The TCU model assumes that multiplication between two square matrices of constant size $s$ is a basic operation. In the $(s^2, \ell)$-TCU model, we show that for inputs of size $n$, the algorithm has depth at most $2\lfloor \log_s (n)\rfloor$ and runs in $O(n(1 + \ell /s^2)/p + (s^2 + \ell) \log_s (n))$ time assuming $p$ tensor core units. Equivalently, the algorithm performs $O(n/s^2)$ multiplications of square matrices of size s.
Automating grapevine LAI features estimation with UAV imagery and machine learning
Venue: arXiv cs.CV 2024
Year: 2024

The leaf area index determines crop health and growth. Traditional methods for calculating it are time-consuming, destructive, costly, and limited to a scale. In this study, we automate the index estimation method using drone image data of grapevine plants and a machine learning model. Traditional feature extraction and deep learning methods are used to obtain helpful information from the data and enhance the performance of the different machine learning models employed for the leaf area index prediction. The results showed that deep learning based feature extraction is more effective than traditional methods. The new approach is a significant improvement over old methods, offering a faster, non-destructive, and cost-effective leaf area index calculation, which enhances precision agriculture practices.
Enhancing Project Performance Forecasting using Machine Learning Techniques
Venue: arXiv cs.LG 2024
Year: 2024

Accurate forecasting of project performance metrics is crucial for successfully managing and delivering urban road reconstruction projects. Traditional methods often rely on static baseline plans and fail to consider the dynamic nature of project progress and external factors. This research proposes a machine learning-based approach to forecast project performance metrics, such as cost variance and earned value, for each Work Breakdown Structure (WBS) category in an urban road reconstruction project. The proposed model utilizes time series forecasting techniques, including Autoregressive Integrated Moving Average (ARIMA) and Long Short-Term Memory (LSTM) networks, to predict future performance based on historical data and project progress. The model also incorporates external factors, such as weather patterns and resource availability, as features to enhance the accuracy of forecasts. By applying the predictive power of machine learning, the performance forecasting model enables proactive identification of potential deviations from the baseline plan, which allows project managers to take timely corrective actions. The research aims to validate the effectiveness of the proposed approach using a case study of an urban road reconstruction project, comparing the model's forecasts with actual project performance data. The findings of this research contribute to the advancement of project management practices in the construction industry, offering a data-driven solution for improving project performance monitoring and control.
AI2T: Building Trustable AI Tutors by Interactively Teaching a Self-Aware Learning Agent
Venue: arXiv cs.HC 2024
Year: 2024

AI2T is an interactively teachable AI for authoring intelligent tutoring systems (ITSs). Authors tutor AI2T by providing a few step-by-step solutions and then grading AI2T's own problem-solving attempts. From just 20-30 minutes of interactive training, AI2T can induce robust rules for step-by-step solution tracking (i.e., model-tracing). As AI2T learns it can accurately estimate its certainty of performing correctly on unseen problem steps using STAND: a self-aware precondition learning algorithm that outperforms state-of-the-art methods like XGBoost. Our user study shows that authors can use STAND's certainty heuristic to estimate when AI2T has been trained on enough diverse problems to induce correct and complete model-tracing programs. AI2T-induced programs are more reliable than hallucination-prone LLMs and prior authoring-by-tutoring approaches. With its self-aware induction of hierarchical rules, AI2T offers a path toward trustable data-efficient authoring-by-tutoring for complex ITSs that normally require as many as 200-300 hours of programming per hour of instruction.
Structural Parameterization of Locating-Dominating Set and Test Cover
Venue: arXiv cs.DS 2024
Year: 2024

We investigate structural parameterizations of two identification problems: LOCATING-DOMINATING SET and TEST COVER. In the first problem, an input is a graph $G$ on $n$ vertices and an integer $k$, and one asks if there is a subset $S$ of $k$ vertices such that any two distinct vertices not in $S$ are dominated by distinct subsets of $S$. In the second problem, an input is a set of items $U$, a set of subsets $\mathcal{F}$ of $U$ called $tests$ and an integer $k$, and one asks if there is a set $S$ of at most $k$ tests such that any two items belong to distinct subsets of tests of $S$. These two problems are "identification" analogues of DOMINATING SET and SET COVER, respectively. Chakraborty et al. [ISAAC 2024] proved that both the problems admit conditional double-exponential lower bounds and matching algorithms when parameterized by treewidth of the input graph. We continue this line of investigation and consider parameters larger than treewidth, like vertex cover number and feedback edge set number. We design a nontrivial dynamic programming scheme to solve TEST COVER in "slightly super-exponential" time $2^{O(|U|\log |U|)}(|U|+|\mathcal{F}|)^{O(1)}$ in the number $|U|$ of items and LOCATING-DOMINATING SET in time $2^{O(\textsf{vc} \log \textsf{vc})} \cdot n^{O(1)}$, where $\textsf{vc}$ is the vertex cover number and $n$ is the order of the graph. This shows that the lower bound results with respect to treewidth from Chakraborty et al. [ISAAC 2024] cannot be extended to vertex cover number. We also show that, parameterized by feedback edge set number, LOCATING-DOMINATING SET admits a linear kernel thereby answering an open question in [Cappelle et al., LAGOS 2021]. Finally, we show that neither LOCATING-DOMINATING SET nor TEST COVER is likely to admit a compression algorithm returning an input with a subquadratic number of bits, unless $\textsf{NP} \subseteq \textsf{coNP}/poly$.
Revisiting Misalignment in Multispectral Pedestrian Detection: A Language-Driven Approach for Cross-modal Alignment Fusion
Venue: arXiv cs.CV 2024
Year: 2024

Multispectral pedestrian detection is a crucial component in various critical applications. However, a significant challenge arises due to the misalignment between these modalities, particularly under real-world conditions where data often appear heavily misaligned. Conventional methods developed on well-aligned or minimally misaligned datasets fail to address these discrepancies adequately. This paper introduces a new framework for multispectral pedestrian detection designed specifically to handle heavily misaligned datasets without the need for costly and complex traditional pre-processing calibration. By leveraging Large-scale Vision-Language Models (LVLM) for cross-modal semantic alignment, our approach seeks to enhance detection accuracy by aligning semantic information across the RGB and thermal domains. This method not only simplifies the operational requirements but also extends the practical usability of multispectral detection technologies in practical applications.
Exploring Visual Vulnerabilities via Multi-Loss Adversarial Search for Jailbreaking Vision-Language Models
Venue: arXiv cs.CV 2024
Year: 2024

Despite inheriting security measures from underlying language models, Vision-Language Models (VLMs) may still be vulnerable to safety alignment issues. Through empirical analysis, we uncover two critical findings: scenario-matched images can significantly amplify harmful outputs, and contrary to common assumptions in gradient-based attacks, minimal loss values do not guarantee optimal attack effectiveness. Building on these insights, we introduce MLAI (Multi-Loss Adversarial Images), a novel jailbreak framework that leverages scenario-aware image generation for semantic alignment, exploits flat minima theory for robust adversarial image selection, and employs multi-image collaborative attacks for enhanced effectiveness. Extensive experiments demonstrate MLAI's significant impact, achieving attack success rates of 77.75% on MiniGPT-4 and 82.80% on LLaVA-2, substantially outperforming existing methods by margins of 34.37% and 12.77% respectively. Furthermore, MLAI shows considerable transferability to commercial black-box VLMs, achieving up to 60.11% success rate. Our work reveals fundamental visual vulnerabilities in current VLMs safety mechanisms and underscores the need for stronger defenses. Warning: This paper contains potentially harmful example text.
Pixel-aligned RGB-NIR Stereo Imaging and Dataset for Robot Vision
Venue: arXiv cs.CV 2024
Year: 2024

Integrating RGB and NIR stereo imaging provides complementary spectral information, potentially enhancing robotic 3D vision in challenging lighting conditions. However, existing datasets and imaging systems lack pixel-level alignment between RGB and NIR images, posing challenges for downstream vision tasks. In this paper, we introduce a robotic vision system equipped with pixel-aligned RGB-NIR stereo cameras and a LiDAR sensor mounted on a mobile robot. The system simultaneously captures pixel-aligned pairs of RGB stereo images, NIR stereo images, and temporally synchronized LiDAR points. Utilizing the mobility of the robot, we present a dataset containing continuous video frames under diverse lighting conditions. We then introduce two methods that utilize the pixel-aligned RGB-NIR images: an RGB-NIR image fusion method and a feature fusion method. The first approach enables existing RGB-pretrained vision models to directly utilize RGB-NIR information without fine-tuning. The second approach fine-tunes existing vision models to more effectively utilize RGB-NIR information. Experimental results demonstrate the effectiveness of using pixel-aligned RGB-NIR images across diverse lighting conditions.
Lightweight Gaze Estimation Model Via Fusion Global Information
Venue: arXiv cs.CV 2024
Year: 2024

Deep learning-based appearance gaze estimation methods are gaining popularity due to their high accuracy and fewer constraints from the environment. However, existing high-precision models often rely on deeper networks, leading to problems such as large parameters, long training time, and slow convergence. In terms of this issue, this paper proposes a novel lightweight gaze estimation model FGI-Net(Fusion Global Information). The model fuses global information into the CNN, effectively compensating for the need of multi-layer convolution and pooling to indirectly capture global information, while reducing the complexity of the model, improving the model accuracy and convergence speed. To validate the performance of the model, a large number of experiments are conducted, comparing accuracy with existing classical models and lightweight models, comparing convergence speed with models of different architectures, and conducting ablation experiments. Experimental results show that compared with GazeCaps, the latest gaze estimation model, FGI-Net achieves a smaller angle error with 87.1% and 79.1% reduction in parameters and FLOPs, respectively (MPIIFaceGaze is 3.74°, EyeDiap is 5.15°, Gaze360 is 10.50° and RT-Gene is 6.02°). Moreover, compared with different architectural models such as CNN and Transformer, FGI-Net is able to quickly converge to a higher accuracy range with fewer iterations of training, when achieving optimal accuracy on the Gaze360 and EyeDiap datasets, the FGI-Net model has 25% and 37.5% fewer iterations of training compared to GazeTR, respectively.
Derivation of Closed Form of Expected Improvement for Gaussian Process Trained on Log-Transformed Objective
Venue: arXiv cs.LG 2024
Year: 2024

Expected Improvement (EI) is arguably the most widely used acquisition function in Bayesian optimization. However, it is often challenging to enhance the performance with EI due to its sensitivity to numerical precision. Previously, Hutter et al. (2009) tackled this problem by using Gaussian process trained on the log-transformed objective function and it was reported that this trick improves the predictive accuracy of GP, leading to substantially better performance. Although Hutter et al. (2009) offered the closed form of their EI, its intermediate derivation has not been provided so far. In this paper, we give a friendly derivation of their proposition.