OpenRaiser/Intern-Atlas huggingface dataset:
Sharper Guarantees for Learning Neural Network Classifiers with Gradient Methods
Venue: arXiv cs.LG 2024
Year: 2024

In this paper, we study the data-dependent convergence and generalization behavior of gradient methods for neural networks with smooth activation. Our first result is a novel bound on the excess risk of deep networks trained by the logistic loss, via an alogirthmic stability analysis. Compared to previous works, our results improve upon the shortcomings of the well-established Rademacher complexity-based bounds. Importantly, the bounds we derive in this paper are tighter, hold even for neural networks of small width, do not scale unfavorably with width, are algorithm-dependent, and consequently capture the role of initialization on the sample complexity of gradient descent for deep nets. Specialized to noiseless data separable with margin $γ$ by neural tangent kernel (NTK) features of a network of width $Ω(\text{poly}(\log(n)))$, we show the test-error rate to be $e^{O(L)}/{γ^2 n}$, where $n$ is the training set size and $L$ denotes the number of hidden layers. This is an improvement in the test loss bound compared to previous works while maintaining the poly-logarithmic width conditions. We further investigate excess risk bounds for deep nets trained with noisy data, establishing that under a polynomial condition on the network width, gradient descent can achieve the optimal excess risk. Finally, we show that a large step-size significantly improves upon the NTK regime's results in classifying the XOR distribution. In particular, we show for a one-hidden-layer neural network of constant width $m$ with quadratic activation and standard Gaussian initialization that mini-batch SGD with linear sample complexity and with a large step-size $η=m$ reaches the perfect test accuracy after only $\ceil{\log(d)}$ iterations, where $d$ is the data dimension.
A Hybrid Sampling and Multi-Objective Optimization Approach for Enhanced Software Defect Prediction
Venue: arXiv cs.SE 2024
Year: 2024

Accurate early prediction of software defects is essential to maintain software quality and reduce maintenance costs. However, the field of software defect prediction (SDP) faces challenges such as class imbalances, high-dimensional feature spaces, and suboptimal prediction accuracy. To mitigate these challenges, this paper introduces a novel SDP framework that integrates hybrid sampling techniques, specifically Borderline SMOTE and Tomek Links, with a suite of multi-objective optimization algorithms, including NSGA-II, MOPSO, and MODE. The proposed model applies feature fusion through multi-objective optimization, enhancing both the generalization capability and stability of the predictions. Furthermore, the integration of parallel processing for these optimization algorithms significantly boosts the computational efficiency of the model. Comprehensive experiments conducted on datasets from NASA and PROMISE repositories demonstrate that the proposed hybrid sampling and multi-objective optimization approach improves data balance, eliminates redundant features, and enhances prediction accuracy. The experimental results also highlight the robustness of the feature fusion approach, confirming its superiority over existing state-of-the-art techniques in terms of predictive performance and applicability across diverse datasets.
Towards Bridging Generalization and Expressivity of Graph Neural Networks
Venue: arXiv cs.LG 2024
Year: 2024

Expressivity and generalization are two critical aspects of graph neural networks (GNNs). While significant progress has been made in studying the expressivity of GNNs, much less is known about their generalization capabilities, particularly when dealing with the inherent complexity of graph-structured data. In this work, we address the intricate relationship between expressivity and generalization in GNNs. Theoretical studies conjecture a trade-off between the two: highly expressive models risk overfitting, while those focused on generalization may sacrifice expressivity. However, empirical evidence often contradicts this assumption, with expressive GNNs frequently demonstrating strong generalization. We explore this contradiction by introducing a novel framework that connects GNN generalization to the variance in graph structures they can capture. This leads us to propose a $k$-variance margin-based generalization bound that characterizes the structural properties of graph embeddings in terms of their upper-bounded expressive power. Our analysis does not rely on specific GNN architectures, making it broadly applicable across GNN models. We further uncover a trade-off between intra-class concentration and inter-class separation, both of which are crucial for effective generalization. Through case studies and experiments on real-world datasets, we demonstrate that our theoretical findings align with empirical results, offering a deeper understanding of how expressivity can enhance GNN generalization.
Innovative Deep Learning Techniques for Obstacle Recognition: A Comparative Study of Modern Detection Algorithms
Venue: arXiv cs.CV 2024
Year: 2024

This study explores a comprehensive approach to obstacle detection using advanced YOLO models, specifically YOLOv8, YOLOv7, YOLOv6, and YOLOv5. Leveraging deep learning techniques, the research focuses on the performance comparison of these models in real-time detection scenarios. The findings demonstrate that YOLOv8 achieves the highest accuracy with improved precision-recall metrics. Detailed training processes, algorithmic principles, and a range of experimental results are presented to validate the model's effectiveness.
Interaction-Guided Two-Branch Image Dehazing Network
Venue: arXiv cs.CV 2024
Year: 2024

Image dehazing aims to restore clean images from hazy ones. Convolutional Neural Networks (CNNs) and Transformers have demonstrated exceptional performance in local and global feature extraction, respectively, and currently represent the two mainstream frameworks in image dehazing. In this paper, we propose a novel dual-branch image dehazing framework that guides CNN and Transformer components interactively. We reconsider the complementary characteristics of CNNs and Transformers by leveraging the differential relationships between global and local features for interactive guidance. This approach enables the capture of local feature positions through global attention maps, allowing the CNN to focus solely on feature information at effective positions. The single-branch Transformer design ensures the network's global information recovery capability. Extensive experiments demonstrate that our proposed method yields competitive qualitative and quantitative evaluation performance on both synthetic and real public datasets. Codes are available at https://github.com/Feecuin/Two-Branch-Dehazing
DecKG: Decentralized Collaborative Learning with Knowledge Graph Enhancement for POI Recommendation
Venue: arXiv cs.IR 2024
Year: 2024

Decentralized collaborative learning for Point-of-Interest (POI) recommendation has gained research interest due to its advantages in privacy preservation and efficiency, as it keeps data locally and leverages collaborative learning among clients to train models in a decentralized manner. However, since local data is often limited and insufficient for training accurate models, a common solution is integrating external knowledge as auxiliary information to enhance model performance. Nevertheless, this solution poses challenges for decentralized collaborative learning. Due to private nature of local data, identifying relevant auxiliary information specific to each user is non-trivial. Furthermore, resource-constrained local devices struggle to accommodate all auxiliary information, which places heavy burden on local storage. To fill the gap, we propose a novel decentralized collaborative learning with knowledge graph enhancement framework for POI recommendation (DecKG). Instead of directly uploading interacted items, users generate desensitized check-in data by uploading general categories of interacted items and sampling similar items from same category. The server then pretrains KG without sensitive user-item interactions and deploys relevant partitioned sub-KGs to individual users. Entities are further refined on the device, allowing client to client communication to exchange knowledge learned from local data and sub-KGs. Evaluations across two real-world datasets demonstrate DecKG's effectiveness recommendation performance.
A First Look at Package-to-Group Mechanism: An Empirical Study of the Linux Distributions
Venue: arXiv cs.SE 2024
Year: 2024

Reusing third-party software packages is a common practice in software development. As the scale and complexity of open-source software (OSS) projects continue to grow (e.g., Linux distributions), the number of reused third-party packages has significantly increased. Therefore, maintaining effective package management is critical for developing and evolving OSS projects. To achieve this, a package-to-group mechanism (P2G) is employed to enable unified installation, uninstallation, and updates of multiple packages at once. To better understand this mechanism, this paper takes Linux distributions as a case study and presents an empirical study focusing on its application trends, evolutionary patterns, group quality, and developer tendencies. By analyzing 11,746 groups and 193,548 packages from 89 versions of 5 popular Linux distributions and conducting questionnaire surveys with Linux practitioners and researchers, we derive several key insights. Our findings show that P2G is increasingly being adopted, particularly in popular Linux distributions. P2G follows six evolutionary patterns (\eg splitting and merging groups). Interestingly, packages no longer managed through P2G are more likely to remain in Linux distributions rather than being directly removed. To assess the effectiveness of P2G, we propose a metric called {\sc GValue} to evaluate the quality of groups and identify issues such as inadequate group descriptions and insufficient group sizes. We also summarize five types of packages that tend to adopt P2G, including graphical desktops, networks, etc. To the best of our knowledge, this is the first study focusing on the P2G mechanisms. We expect our study can assist in the efficient management of packages and reduce the burden on practitioners in rapidly growing Linux distributions and other open-source software projects.
First Creating Backgrounds Then Rendering Texts: A New Paradigm for Visual Text Blending
Venue: arXiv cs.CV 2024
Year: 2024

Diffusion models, known for their impressive image generation abilities, have played a pivotal role in the rise of visual text generation. Nevertheless, existing visual text generation methods often focus on generating entire images with text prompts, leading to imprecise control and limited practicality. A more promising direction is visual text blending, which focuses on seamlessly merging texts onto text-free backgrounds. However, existing visual text blending methods often struggle to generate high-fidelity and diverse images due to a shortage of backgrounds for synthesis and limited generalization capabilities. To overcome these challenges, we propose a new visual text blending paradigm including both creating backgrounds and rendering texts. Specifically, a background generator is developed to produce high-fidelity and text-free natural images. Moreover, a text renderer named GlyphOnly is designed for achieving visually plausible text-background integration. GlyphOnly, built on a Stable Diffusion framework, utilizes glyphs and backgrounds as conditions for accurate rendering and consistency control, as well as equipped with an adaptive text block exploration strategy for small-scale text rendering. We also explore several downstream applications based on our method, including scene text dataset synthesis for boosting scene text detectors, as well as text image customization and editing. Code and model will be available at \url{https://github.com/Zhenhang-Li/GlyphOnly}.
Identity-Focused Inference and Extraction Attacks on Diffusion Models
Venue: arXiv cs.CV 2024
Year: 2024

The increasing reliance on diffusion models for generating synthetic images has amplified concerns about the unauthorized use of personal data, particularly facial images, in model training. In this paper, we introduce a novel identity inference framework to hold model owners accountable for including individuals' identities in their training data. Our approach moves beyond traditional membership inference attacks by focusing on identity-level inference, providing a new perspective on data privacy violations. Through comprehensive evaluations on two facial image datasets, Labeled Faces in the Wild (LFW) and CelebA, our experiments demonstrate that the proposed membership inference attack surpasses baseline methods, achieving an attack success rate of up to 89% and an AUC-ROC of 0.91, while the identity inference attack attains 92% on LDM models trained on LFW, and the data extraction attack achieves 91.6% accuracy on DDPMs, validating the effectiveness of our approach across diffusion models.
SkillAggregation: Reference-free LLM-Dependent Aggregation
Venue: arXiv cs.CL 2024
Year: 2024

Large Language Models (LLMs) are increasingly used to assess NLP tasks due to their ability to generate human-like judgments. Single LLMs were used initially, however, recent work suggests using multiple LLMs as judges yields improved performance. An important step in exploiting multiple judgements is the combination stage, aggregation. Existing methods in NLP either assign equal weight to all LLM judgments or are designed for specific tasks such as hallucination detection. This work focuses on aggregating predictions from multiple systems where no reference labels are available. A new method called SkillAggregation is proposed, which learns to combine estimates from LLM judges without needing additional data or ground truth. It extends the Crowdlayer aggregation method, developed for image classification, to exploit the judge estimates during inference. The approach is compared to a range of standard aggregation methods on HaluEval-Dialogue, TruthfulQA and Chatbot Arena tasks. SkillAggregation outperforms Crowdlayer on all tasks, and yields the best performance over all approaches on the majority of tasks.
The Importance of Justified Patient Trust in unlocking AI's potential in mental healthcare
Venue: arXiv cs.CY 2024
Year: 2024

Without trust, patients may hesitate to engage with AI systems, significantly limiting the technology's potential in mental healthcare. This paper focuses specifically on the trust that mental health patients, as direct users, must have in AI systems, highlighting the most sensitive and direct relationship between AI systems and those whose mental healthcare is impacted by them. We explore the concept of justified trust, why it is important for patient positive care outcomes, and the strategies needed to foster and maintain this trust. By examining these aspects, we highlight how cultivating justified trust is key to unlocking AI's potential impact in mental healthcare.
Quasilinear-time eccentricities computation, and more, on median graphs
Venue: arXiv cs.DS 2024
Year: 2024

Computing the diameter, and more generally, all eccentricities of an undirected graph is an important problem in algorithmic graph theory and the challenge is to identify graph classes for which their computation can be achieved in subquadratic time. Using a new recursive scheme based on the structural properties of median graphs, we provide a quasilinear-time algorithm to determine all eccentricities for this well-known family of graphs. Our recursive technique manages specifically balanced and unbalanced parts of the $Θ$-class decomposition of median graphs. The exact running time of our algorithm is O(n log^4 n). This outcome not only answers a question asked by B{é}n{é}teau et al. (2020) but also greatly improves a recent result which presents a combinatorial algorithm running in time O(n^1.6408 log^{O(1)} n) for the same problem.Furthermore we also propose a distance oracle for median graphs with both polylogarithmic size and query time. Speaking formally, we provide a combinatorial algorithm which computes for any median graph G, in quasilinear time O(n log^4 n), vertex-labels of size O(log^3 n) such that any distance of G can be retrieved in time O(log^4 n) thanks to these labels.
LOBG:Less Overfitting for Better Generalization in Vision-Language Model
Venue: arXiv cs.CV 2024
Year: 2024

Existing prompt learning methods in Vision-Language Models (VLM) have effectively enhanced the transfer capability of VLM to downstream tasks, but they suffer from a significant decline in generalization due to severe overfitting. To address this issue, we propose a framework named LOBG for vision-language models. Specifically, we use CLIP to filter out fine-grained foreground information that might cause overfitting, thereby guiding prompts with basic visual concepts. To further mitigate overfitting, we devel oped a structural topology preservation (STP) loss at the feature level, which endows the feature space with overall plasticity, allowing effective reshaping of the feature space during optimization. Additionally, we employed hierarchical logit distilation (HLD) at the output level to constrain outputs, complementing STP at the output end. Extensive experimental results demonstrate that our method significantly improves generalization capability and alleviates overfitting compared to state-of-the-art approaches.
big.LITTLE Vision Transformer for Efficient Visual Recognition
Venue: arXiv cs.CV 2024
Year: 2024

In this paper, we introduce the big.LITTLE Vision Transformer, an innovative architecture aimed at achieving efficient visual recognition. This dual-transformer system is composed of two distinct blocks: the big performance block, characterized by its high capacity and substantial computational demands, and the LITTLE efficiency block, designed for speed with lower capacity. The key innovation of our approach lies in its dynamic inference mechanism. When processing an image, our system determines the importance of each token and allocates them accordingly: essential tokens are processed by the high-performance big model, while less critical tokens are handled by the more efficient little model. This selective processing significantly reduces computational load without sacrificing the overall performance of the model, as it ensures that detailed analysis is reserved for the most important information. To validate the effectiveness of our big.LITTLE Vision Transformer, we conducted comprehensive experiments on image classification and segment anything task. Our results demonstrate that the big.LITTLE architecture not only maintains high accuracy but also achieves substantial computational savings. Specifically, our approach enables the efficient handling of large-scale visual recognition tasks by dynamically balancing the trade-offs between performance and efficiency. The success of our method underscores the potential of hybrid models in optimizing both computation and performance in visual recognition tasks, paving the way for more practical and scalable deployment of advanced neural networks in real-world applications.
Intelligent Reflecting Surface-Assisted Symbiotic Radio Systems: A Double-Reflection Covert Communication Design
Venue: arXiv cs.IT 2024
Year: 2024

We investigate covert communication in an intelligent reflecting surface (IRS)-assisted symbiotic radio (SR) system under the parasitic SR (PSR) and the commensal SR (CSR) cases, where an IRS is exploited to create a double reflection link for legitimate users and degrade the detection performance of the warden (W). Specifically, we derive an analytical expression for the average detection error probability of W and design an optimal strategy to determine the transmit power and backscatter reflection coefficient. To further enhance the covert performance, the joint optimization of the source transmit power, backscatter device (BD) reflection coefficient, and IRS phase-shifter is formulated as an expectation-based quadratic-fractional (EQF) problem. By reformulating the original problem into a fraction-eliminated backscatter power leakage minimization problem, we further develop the phase alignment pursuit and the power leakage minimization algorithms for the PSR and the CSR cases, respectively. Numerical results confirm the accuracy of the derived results and the superiority of our proposed strategy in terms of covertness.
On Representation of 3D Rotation in the Context of Deep Learning
Venue: arXiv cs.CV 2024
Year: 2024

This paper investigates various methods of representing 3D rotations and their impact on the learning process of deep neural networks. We evaluated the performance of ResNet18 networks for 3D rotation estimation using several rotation representations and loss functions on both synthetic and real data. The real datasets contained 3D scans of industrial bins, while the synthetic datasets included views of a simple asymmetric object rendered under different rotations. On synthetic data, we also assessed the effects of different rotation distributions within the training and test sets, as well as the impact of the object's texture. In line with previous research, we found that networks using the continuous 5D and 6D representations performed better than the discontinuous ones.
Optimizing Instruction Synthesis: Effective Exploration of Evolutionary Space with Tree Search
Venue: arXiv cs.AI 2024
Year: 2024

Instruction tuning is a crucial technique for aligning language models with humans' actual goals in the real world. Extensive research has highlighted the quality of instruction data is essential for the success of this alignment. However, creating high-quality data manually is labor-intensive and time-consuming, which leads researchers to explore using LLMs to synthesize data. Recent studies have focused on using a stronger LLM to iteratively enhance existing instruction data, showing promising results. Nevertheless, previous work often lacks control over the evolution direction, resulting in high uncertainty in the data synthesis process and low-quality instructions. In this paper, we introduce a general and scalable framework, IDEA-MCTS (Instruction Data Enhancement using Monte Carlo Tree Search), a scalable framework for efficiently synthesizing instructions. With tree search and evaluation models, it can efficiently guide each instruction to evolve into a high-quality form, aiding in instruction fine-tuning. Experimental results show that IDEA-MCTS significantly enhances the seed instruction data, raising the average evaluation scores of quality, diversity, and complexity from 2.19 to 3.81. Furthermore, in open-domain benchmarks, experimental results show that IDEA-MCTS improves the accuracy of real-world instruction-following skills in LLMs by an average of 5\% in low-resource settings.