OpenRaiser/Intern-Atlas huggingface dataset:
StegaVision: Enhancing Steganography with Attention Mechanism
Venue: arXiv cs.CV 2024
Year: 2024

Image steganography is the technique of embedding secret information within images. The development of deep learning has led to significant advances in this field. However, existing methods often struggle to balance image quality, embedding capacity, and security. This paper proposes a novel approach to image steganography by enhancing an encoder-decoder architecture with attention mechanisms, specifically focusing on channel and spatial attention modules. We systematically investigate five configurations: (1) channel attention, (2) spatial attention, (3) sequential channel followed by spatial attention, (4) spatial attention followed by channel attention and (5) parallel channel and spatial attention. Our experiments show that adding attention mechanisms improves the ability to embed hidden information while maintaining the visual quality of the images. The increase in the PSNR and SSIM scores shows that using a parallel combination of channel and spatial attention improves image quality and hiding capacity simultaneously. This is in contrast to previous works where there is a tradeoff between them. This study shows that attention mechanisms in image steganography lead to better hiding of secret information. Our code is available at https://github.com/vlgiitr/StegaVision.
Neural Precision Polarization: Simplifying Neural Network Inference with Dual-Level Precision
Venue: arXiv cs.LG 2024
Year: 2024

We introduce a precision polarization scheme for DNN inference that utilizes only very low and very high precision levels, assigning low precision to the majority of network weights and activations while reserving high precision paths for targeted error compensation. This separation allows for distinct optimization of each precision level, thereby reducing memory and computation demands without compromising model accuracy. In the discussed approach, a floating-point model can be trained in the cloud and then downloaded to an edge device, where network weights and activations are directly quantized to meet the edge devices' desired level, such as NF4 or INT8. To address accuracy loss from quantization, surrogate paths are introduced, leveraging low-rank approximations on a layer-by-layer basis. These paths are trained with a sensitivity-based metric on minimal training data to recover accuracy loss under quantization as well as due to process variability, such as when the main prediction path is implemented using analog acceleration. Our simulation results show that neural precision polarization enables approximately 464 TOPS per Watt MAC efficiency and reliability by integrating rank-8 error recovery paths with highly efficient, though potentially unreliable, bit plane-wise compute-in-memory processing.
Reducing catastrophic forgetting of incremental learning in the absence of rehearsal memory with task-specific token
Venue: arXiv cs.LG 2024
Year: 2024

Deep learning models generally display catastrophic forgetting when learning new data continuously. Many incremental learning approaches address this problem by reusing data from previous tasks while learning new tasks. However, the direct access to past data generates privacy and security concerns. To address these issues, we present a novel method that preserves previous knowledge without storing previous data. This method is inspired by the architecture of a vision transformer and employs a unique token capable of encapsulating the compressed knowledge of each task. This approach generates task-specific embeddings by directing attention differently based on the task associated with the data, thereby effectively mimicking the impact of having multiple models through tokens. Our method incorporates a distillation process that ensures efficient interactions even after multiple additional learning steps, thereby optimizing the model against forgetting. We measured the performance of our model in terms of accuracy and backward transfer using a benchmark dataset for different task-incremental learning scenarios. Our results demonstrate the superiority of our approach, which achieved the highest accuracy and lowest backward transfer among the compared methods. In addition to presenting a new model, our approach lays the foundation for various extensions within the spectrum of vision-transformer architectures.
Saliency Assisted Quantization for Neural Networks
Venue: arXiv cs.CV 2024
Year: 2024

Deep learning methods have established a significant place in image classification. While prior research has focused on enhancing final outcomes, the opaque nature of the decision-making process in these models remains a concern for experts. Additionally, the deployment of these methods can be problematic in resource-limited environments. This paper tackles the inherent black-box nature of these models by providing real-time explanations during the training phase, compelling the model to concentrate on the most distinctive and crucial aspects of the input. Furthermore, we employ established quantization techniques to address resource constraints. To assess the effectiveness of our approach, we explore how quantization influences the interpretability and accuracy of Convolutional Neural Networks through a comparative analysis of saliency maps from standard and quantized models. Quantization is implemented during the training phase using the Parameterized Clipping Activation method, with a focus on the MNIST and FashionMNIST benchmark datasets. We evaluated three bit-width configurations (2-bit, 4-bit, and mixed 4/2-bit) to explore the trade-off between efficiency and interpretability, with each configuration designed to highlight varying impacts on saliency map clarity and model accuracy. The results indicate that while quantization is crucial for implementing models on resource-limited devices, it necessitates a trade-off between accuracy and interpretability. Lower bit-widths result in more pronounced reductions in both metrics, highlighting the necessity of meticulous quantization parameter selection in applications where model transparency is paramount. The study underscores the importance of achieving a balance between efficiency and interpretability in the deployment of neural networks.
Rethinking Deep Learning: Non-backpropagation and Non-optimization Machine Learning Approach Using Hebbian Neural Networks
Venue: arXiv cs.NE 2024
Year: 2024

Developing strong AI could provide a powerful tool for addressing social and scientific challenges. Neural networks (NNs), inspired by biological systems, have the potential to achieve this. However, weight optimization techniques using error backpropagation are not observed in biological systems, raising doubts about current NNs approaches. In this context, Itoh (2024) solved the MNIST classification problem without using objective functions or backpropagation. However, weight updates were not used, so it does not qualify as machine learning AI. In this study, I develop a machine learning method that mimics biological neural systems by implementing Hebbian learning in NNs without backpropagation and optimization method to solve the MNIST classification problem and analyze its output. Development proceeded in three stages. In the first stage, I applied the Hebbian learning rule to the MNIST character recognition algorithm by Itoh (2024), resulting in lower accuracy than non-Hebbian NNs, highlighting the limitations of conventional training procedures for Hebbian learning. In the second stage, I examined the properties of individually trained NNs using norm-based cognition, showing that NNs trained on a specific label respond powerfully to that label. In the third stage, I created an MNIST character recognition program using vector norm magnitude as the criterion, achieving an accuracy of approximately 75%. This demonstrates that the Hebbian learning NNs can recognize handwritten characters without objective functions, backpropagation, optimization processes, and large data set. Based on these results, developing a mechanism based on norm-based cognition as a fundamental unit and then increasing complexity to achieve indirect similarity cognition should help mimic biological neural systems and contribute to realizing strong AI.
Boosting the Efficiency of Metaheuristics Through Opposition-Based Learning in Optimum Locating of Control Systems in Tall Buildings
Venue: arXiv cs.NE 2024
Year: 2024

Opposition-based learning (OBL) is an effective approach to improve the performance of metaheuristic optimization algorithms, which are commonly used for solving complex engineering problems. This chapter provides a comprehensive review of the literature on the use of opposition strategies in metaheuristic optimization algorithms, discussing the benefits and limitations of this approach. An overview of the opposition strategy concept, its various implementations, and its impact on the performance of metaheuristic algorithms are presented. Furthermore, case studies on the application of opposition strategies in engineering problems are provided, including the optimum locating of control systems in tall building. A shear frame with Magnetorheological (MR) fluid damper is considered as a case study. The results demonstrate that the incorporation of opposition strategies in metaheuristic algorithms significantly enhances the quality and speed of the optimization process. This chapter aims to provide a clear understanding of the opposition strategy in metaheuristic optimization algorithms and its engineering applications, with the ultimate goal of facilitating its adoption in real-world engineering problems.
Bilinear Fuzzy Genetic Algorithm and Its Application on the Optimum Design of Steel Structures with Semi-rigid Connections
Venue: arXiv cs.NE 2024
Year: 2024

An improved bilinear fuzzy genetic algorithm (BFGA) is introduced in this chapter for the design optimization of steel structures with semi-rigid connections. Semi-rigid connections provide a compromise between the stiffness of fully rigid connections and the flexibility of fully pinned connections. However, designing such structures is challenging due to the nonlinear behavior of semi-rigid connections. The BFGA is a robust optimization method that combines the strengths of fuzzy logic and genetic algorithm to handle the complexity and uncertainties of structural design problems. The BFGA, compared to standard GA, demonstrated to generate high-quality solutions in a reasonable time. The application of the BFGA is demonstrated through the optimization of steel structures with semirigid connections, considering the weight and performance criteria. The results show that the proposed BFGA is capable of finding optimal designs that satisfy all the design requirements and constraints. The proposed approach provides a promising solution for the optimization of complex structures with nonlinear behavior.
Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization
Venue: arXiv cs.LG 2024
Year: 2024

Direct Preference Optimization (DPO) and its variants have become the de facto standards for aligning large language models (LLMs) with human preferences or specific goals. However, DPO requires high-quality preference data and suffers from unstable preference optimization. In this work, we aim to improve the preference optimization pipeline by taking a closer look at preference data generation and training regularization techniques. For preference data generation, we demonstrate that existing scoring-based reward models produce unsatisfactory preference data and perform poorly on out-of-distribution tasks. This significantly impacts the LLM alignment performance when using these data for preference tuning. To ensure high-quality preference data generation, we propose an iterative pairwise ranking mechanism that derives preference ranking of completions using pairwise comparison signals. For training regularization, we observe that preference optimization tends to achieve better convergence when the LLM predicted likelihood of preferred samples gets slightly reduced. However, the widely used supervised next-word prediction regularization strictly prevents any likelihood reduction of preferred samples. This observation motivates our design of a budget-controlled regularization formulation. Empirically we show that combining the two designs leads to aligned models that surpass existing SOTA across two popular benchmarks.
A Comparative Analysis of Machine Learning Models for DDoS Detection in IoT Networks
Venue: arXiv cs.CR 2024
Year: 2024

This paper presents the detection of DDoS attacks in IoT networks using machine learning models. Their rapid growth has made them highly susceptible to various forms of cyberattacks, many of whose security procedures are implemented in an irregular manner. It evaluates the efficacy of different machine learning models, such as XGBoost, K-Nearest Neighbours, Stochastic Gradient Descent, and Naïve Bayes, in detecting DDoS attacks from normal network traffic. Each model has been explained on several performance metrics, such as accuracy, precision, recall, and F1-score to understand the suitability of each model in real-time detection and response against DDoS threats. This comparative analysis will, therefore, enumerate the unique strengths and weaknesses of each model with respect to the IoT environments that are dynamic and hence moving in nature. The effectiveness of these models is analyzed, showing how machine learning can greatly enhance IoT security frameworks, offering adaptive, efficient, and reliable DDoS detection capabilities. These findings have shown the potential of machine learning in addressing the pressing need for robust IoT security solutions that can mitigate modern cyber threats and assure network integrity.
Streaming Bayes GFlowNets
Venue: arXiv cs.LG 2024
Year: 2024

Bayes' rule naturally allows for inference refinement in a streaming fashion, without the need to recompute posteriors from scratch whenever new data arrives. In principle, Bayesian streaming is straightforward: we update our prior with the available data and use the resulting posterior as a prior when processing the next data chunk. In practice, however, this recipe entails i) approximating an intractable posterior at each time step; and ii) encapsulating results appropriately to allow for posterior propagation. For continuous state spaces, variational inference (VI) is particularly convenient due to its scalability and the tractability of variational posteriors. For discrete state spaces, however, state-of-the-art VI results in analytically intractable approximations that are ill-suited for streaming settings. To enable streaming Bayesian inference over discrete parameter spaces, we propose streaming Bayes GFlowNets (abbreviated as SB-GFlowNets) by leveraging the recently proposed GFlowNets -- a powerful class of amortized samplers for discrete compositional objects. Notably, SB-GFlowNet approximates the initial posterior using a standard GFlowNet and subsequently updates it using a tailored procedure that requires only the newly observed data. Our case studies in linear preference learning and phylogenetic inference showcase the effectiveness of SB-GFlowNets in sampling from an unnormalized posterior in a streaming setting. As expected, we also observe that SB-GFlowNets is significantly faster than repeatedly training a GFlowNet from scratch to sample from the full posterior.
Towards Multi-Modal Mastery: A 4.5B Parameter Truly Multi-Modal Small Language Model
Venue: arXiv cs.LG 2024
Year: 2024

We present a novel 4.5B parameter small language model that can handle multiple input and output modalities, including text, images, videos, and audio. Despite its small size, the model achieves near state-of-the-art performance on a variety of tasks, demonstrating the potential of multi-modal models to tackle complex real-world problems. Our approach leverages recent advancements in language modeling and multi-task learning to create a versatile and high-performing model that can even be deployed for edge inference. Experimental results show the model's strong performance across multiple benchmarks, paving the way for further progress in multi-modal artificial intelligence.
Bridging Nodes and Narrative Flows: Identifying Intervention Targets for Disinformation on Telegram
Venue: arXiv cs.CY 2024
Year: 2024

In recent years, mass-broadcast messaging platforms like Telegram have gained prominence for both, serving as a harbor for private communication and enabling large-scale disinformation campaigns. The encrypted and networked nature of these platforms makes it challenging to identify intervention targets since most channels that promote misleading information are not originators of the message. In this work, we examine the structural mechanisms that facilitate the propagation of debunked misinformation on Telegram, focusing on the role of cross-community hubs-nodes that bridge otherwise isolated groups in amplifying misinformation. We introduce a multi-dimensional 'bridging' metric to quantify the influence of nodal Telegram channels, exploring their role in reshaping network topology during key geopolitical events. By analyzing over 1740 Telegram channels and applying network analysis we uncover the small subset of nodes, and identify patterns that are emblematic of information 'flows' on this platform. Our findings provide insights into the structural vulnerabilities of distributed platforms, offering practical suggestions for interventions to mitigate networked disinformation flows.
DNAMite: Interpretable Calibrated Survival Analysis with Discretized Additive Models
Venue: arXiv cs.LG 2024
Year: 2024

Survival analysis is a classic problem in statistics with important applications in healthcare. Most machine learning models for survival analysis are black-box models, limiting their use in healthcare settings where interpretability is paramount. More recently, glass-box machine learning models have been introduced for survival analysis, with both strong predictive performance and interpretability. Still, several gaps remain, as no prior glass-box survival model can produce calibrated shape functions with enough flexibility to capture the complex patterns often found in real data. To fill this gap, we introduce a new glass-box machine learning model for survival analysis called DNAMite. DNAMite uses feature discretization and kernel smoothing in its embedding module, making it possible to learn shape functions with a flexible balance of smoothness and jaggedness. Further, DNAMite produces calibrated shape functions that can be directly interpreted as contributions to the cumulative incidence function. Our experiments show that DNAMite generates shape functions closer to true shape functions on synthetic data, while making predictions with comparable predictive performance and better calibration than previous glass-box and black-box models.
Aligned Vector Quantization for Edge-Cloud Collabrative Vision-Language Models
Venue: arXiv cs.CV 2024
Year: 2024

Vision Language Models (VLMs) are central to Visual Question Answering (VQA) systems and are typically deployed in the cloud due to their high computational demands. However, this cloud-only approach underutilizes edge computational resources and requires significant bandwidth for transmitting raw images. In this paper, we introduce an edge-cloud collaborative VQA system, called LLaVA-AlignedVQ, which features a novel Aligned Vector Quantization algorithm (AlignedVQ) that efficiently compress intermediate features without compromising accuracy to support partitioned execution. Our experiments demonstrate that LLaVA-AlignedVQ achieves approximately 1365x compression rate of intermediate features, reducing data transmission overhead by 96.8% compared to transmitting JPEG90-compressed images to the cloud. LLaVA-AlignedVQ achieves an inference speedup of 2-15x while maintaining high accuracy, remaining within -2.23% to +1.6% of the original model's accuracy performance across eight VQA datasets, compared to the cloud-only solution.
Game-theoretic LLM: Agent Workflow for Negotiation Games
Venue: arXiv cs.AI 2024
Year: 2024

This paper investigates the rationality of large language models (LLMs) in strategic decision-making contexts, specifically within the framework of game theory. We evaluate several state-of-the-art LLMs across a spectrum of complete-information and incomplete-information games. Our findings reveal that LLMs frequently deviate from rational strategies, particularly as the complexity of the game increases with larger payoff matrices or deeper sequential trees. To address these limitations, we design multiple game-theoretic workflows that guide the reasoning and decision-making processes of LLMs. These workflows aim to enhance the models' ability to compute Nash Equilibria and make rational choices, even under conditions of uncertainty and incomplete information. Experimental results demonstrate that the adoption of these workflows significantly improves the rationality and robustness of LLMs in game-theoretic tasks. Specifically, with the workflow, LLMs exhibit marked improvements in identifying optimal strategies, achieving near-optimal allocations in negotiation scenarios, and reducing susceptibility to exploitation during negotiations. Furthermore, we explore the meta-strategic considerations of whether it is rational for agents to adopt such workflows, recognizing that the decision to use or forgo the workflow constitutes a game-theoretic issue in itself. Our research contributes to a deeper understanding of LLMs' decision-making capabilities in strategic contexts and provides insights into enhancing their rationality through structured workflows. The findings have implications for the development of more robust and strategically sound AI agents capable of navigating complex interactive environments. Code and data supporting this study are available at \url{https://github.com/Wenyueh/game_theory}.
A Toolkit for Measuring the Impacts of Public Funding on Open Source Software Development
Venue: arXiv cs.CY 2024
Year: 2024

Governments are increasingly employing funding for open source software (OSS) development as a policy lever to support the security of software supply chains, digital sovereignty, economic growth, and national competitiveness in science and innovation, among others. However, the impacts of public funding on OSS development remain poorly understood, with a lack of consensus on how to meaningfully measure them. This gap hampers assessments of the return on public investment and impedes the optimisation of public-interest funding strategies. We address this gap with a toolkit of methodological considerations that may inform such measurements, drawing on prior work on OSS valuations and community health metrics by the Community Health Analytics Open Source Software (CHAOSS) project as well as our first-hand learnings as practitioners tasked with evaluating funding programmes by the Next Generation Internet initiative and the Sovereign Tech Agency. We discuss salient considerations, including the importance of accounting for funding objectives, project life stage and social structure, and regional and organisational cost factors. Next, we present a taxonomy of potential social, economic, and technological impacts that can be both positive and negative, direct and indirect, internal (i.e. within a project) and external (i.e. among a project's ecosystem of dependents and users), and manifest over various time horizons. Furthermore, we discuss the merits and limitations of qualitative, quantitative, and mixed-methods approaches, as well as options for and hazards of estimating multiplier effects. With this toolkit, we contribute to the multi-stakeholder conversation about the value and impacts of funding on OSS developers and society at large.