OpenRaiser/Intern-Atlas huggingface dataset:
Research on the Application of Spark Streaming Real-Time Data Analysis System and large language model Intelligent Agents
Venue: arXiv cs.DC 2024
Year: 2024

This study explores the integration of Agent AI with LangGraph to enhance real-time data analysis systems in big data environments. The proposed framework overcomes limitations of static workflows, inefficient stateful computations, and lack of human intervention by leveraging LangGraph's graph-based workflow construction and dynamic decision-making capabilities. LangGraph allows large language models (LLMs) to dynamically determine control flows, invoke tools, and assess the necessity of further actions, improving flexibility and efficiency. The system architecture incorporates Apache Spark Streaming, Kafka, and LangGraph to create a high-performance sentiment analysis system. LangGraph's capabilities include precise state management, dynamic workflow construction, and robust memory checkpointing, enabling seamless multi-turn interactions and context retention. Human-in-the-loop mechanisms are integrated to refine sentiment analysis, particularly in ambiguous or high-stakes scenarios, ensuring greater reliability and contextual relevance. Key features such as real-time state streaming, debugging via LangGraph Studio, and efficient handling of large-scale data streams make this framework ideal for adaptive decision-making. Experimental results confirm the system's ability to classify inquiries, detect sentiment trends, and escalate complex issues for manual review, demonstrating a synergistic blend of LLM capabilities and human oversight. This work presents a scalable, adaptable, and reliable solution for real-time sentiment analysis and decision-making, advancing the use of Agent AI and LangGraph in big data applications.
On strict ranking by pairwise comparisons
Venue: arXiv cs.IT 2024
Year: 2024

We attack the problem of getting a strict ranking (i.e. a ranking without equally ranked items) of $n$ items from a pairwise comparisons matrix. Basic structures are described, a first heuristical approach based on a condition, the $\mathcal{R}-$condition, is proposed. Analyzing the limits of this ranking procedure, we finish with a minimization problem which can be applied to a wider class of pairwise comparisons matrices. If solved, it produces consistent pairwise comparisons that produce a strict ranking.
AI-Driven Health Monitoring of Distributed Computing Architecture: Insights from XGBoost and SHAP
Venue: arXiv cs.DC 2024
Year: 2024

With the rapid development of artificial intelligence technology, its application in the optimization of complex computer systems is becoming more and more extensive. Edge computing is an efficient distributed computing architecture, and the health status of its nodes directly affects the performance and reliability of the entire system. In view of the lack of accuracy and interpretability of traditional methods in node health status judgment, this paper proposes a health status judgment method based on XGBoost and combines the SHAP method to analyze the interpretability of the model. Through experiments, it is verified that XGBoost has superior performance in processing complex features and nonlinear data of edge computing nodes, especially in capturing the impact of key features (such as response time and power consumption) on node status. SHAP value analysis further reveals the global and local importance of features, so that the model not only has high precision discrimination ability but also can provide intuitive explanations, providing data support for system optimization. Research shows that the combination of AI technology and computer system optimization can not only realize the intelligent monitoring of the health status of edge computing nodes but also provide a scientific basis for dynamic optimization scheduling, resource management and anomaly detection. In the future, with the in-depth development of AI technology, model dynamics, cross-node collaborative optimization and multimodal data fusion will become the focus of research, providing important support for the intelligent evolution of edge computing systems.
ABACUS: A FinOps Service for Cloud Cost Optimization
Venue: arXiv cs.DC 2024
Year: 2024

In recent years, as more enterprises have moved their infrastructure to the cloud, significant challenges have emerged in achieving holistic cloud spend visibility and cost optimization. FinOps practices provide a way for enterprises to achieve these business goals by optimizing cloud costs and bringing accountability to cloud spend. This paper presents ABACUS - Automated Budget Analysis and Cloud Usage Surveillance, a FinOps solution for optimizing cloud costs by setting budgets, enforcing those budgets through blocking new deployments, and alerting appropriate teams if spending breaches a budget threshold. ABACUS also leverages best practices like Infrastructure-as-Code to alert engineering teams of the expected cost of deployment before resources are deployed in the cloud. Finally, future research directions are proposed to advance the state of the art in this important field.
Hybrid Firefly-Genetic Algorithm for Single and Multi-dimensional 0-1 Knapsack Problems
Venue: arXiv cs.NE 2024
Year: 2024

This paper addresses the challenges faced by algorithms, such as the Firefly Algorithm (FA) and the Genetic Algorithm (GA), in constrained optimization problems. While both algorithms perform well for unconstrained problems, their effectiveness diminishes when constraints are introduced due to limitations in exploration, exploitation, and constraint handling. To overcome these challenges, a hybrid FAGA algorithm is proposed, combining the strengths of both algorithms. The hybrid algorithm is validated by solving unconstrained benchmark functions and constrained optimization problems, including design engineering problems and combinatorial problems such as the 0-1 Knapsack Problem. The proposed algorithm delivers improved solution accuracy and computational efficiency compared to conventional optimization algorithm. This paper outlines the development and structure of the hybrid algorithm and demonstrates its effectiveness in handling complex optimization problems.
Green AI: Which Programming Language Consumes the Most?
Venue: arXiv cs.CY 2024
Year: 2024

AI is demanding an evergrowing portion of environmental resources. Despite their potential impact on AI environmental sustainability, the role that programming languages play in AI (in)efficiency is to date still unknown. With this study, we aim to understand the impact that programming languages can have on AI environmental sustainability. To achieve our goal, we conduct a controlled empirical experiment by considering five programming languages (C++, Java, Python, MATLAB, and R), seven AI algorithms (KNN, SVC, AdaBoost, decision tree, logistic regression, naive bayses, and random forest), three popular datasets, and the training and inference phases. The collected results show that programming languages have a considerable impact on AI environmental sustainability. Compiled and semi-compiled languages (C++, Java) consistently consume less than interpreted languages (Python, MATLAB, R), which require up to 54x more energy. Some languages are cumulatively more efficient in training, while others in inference. Which programming language consumes the most highly depends on the algorithm considered. Ultimately, algorithm implementation might be the most determining factor in Green AI, regardless of the language used. As conclusion, while making AI more environmentally sustainable is paramount, a trade-off between energy efficiency and implementation ease should always be considered. Green AI can be achieved without the need of completely disrupting the development practices and technologies currently in place.
Efficient Distributed Training through Gradient Compression with Sparsification and Quantization Techniques
Venue: arXiv cs.LG 2024
Year: 2024

This study investigates the impact of gradient compression on distributed training performance, focusing on sparsification and quantization techniques, including top-k, DGC, and QSGD. In baseline experiments, random-k compression results in severe performance degradation, highlighting its inefficacy. In contrast, using top-k and DGC at 50 times compression yields performance improvements, reducing perplexity by up to 0.06 compared to baseline. Experiments across 1, 2, and 4 workers demonstrate that conservative sparsification can have a regularizing effect, especially for smaller models, while compression ratios above 5000 times impair performance, particularly for DGC. Communication times are reduced across all compression methods, with top-k and DGC decreasing communication to negligible levels at high compression ratios. However, increased computation times offset this efficiency for top-k due to sorting demands, making it less scalable than DGC or QSGD. In convergence tests, sparsification techniques show accelerated convergence, requiring fewer epochs than the baseline, which has implications for computational savings. Although precision trade-offs emerge, floating point errors are mitigated by compression. This study's findings underscore the need to tune hyperparameters specifically for each compression technique to achieve optimal model performance, especially in distributed training systems.
Counterexample Guided Program Repair Using Zero-Shot Learning and MaxSAT-based Fault Localization
Venue: arXiv cs.SE 2024
Year: 2024

Automated Program Repair (APR) for introductory programming assignments (IPAs) is motivated by the large number of student enrollments in programming courses each year. Since providing feedback on IPAs requires substantial time and effort from faculty, personalized feedback often involves suggesting fixes to students' programs. Formal Methods (FM)-based semantic repair approaches, check a program's execution against a test suite or reference solution, are effective but limited. These tools excel at identifying buggy parts but can only fix programs if the correct implementation and the faulty one share the same control flow graph. Conversely, Large Language Models (LLMs) are used for APR but often make extensive instead of minimal rewrites. This leads to more invasive fixes, making it harder for students to learn from their mistakes. In summary, LLMs excel at completing strings, while FM-based fault localization excel at identifying buggy parts of a program. In this paper, we propose a novel approach that combines the strengths of both FM-based fault localization and LLMs, via zero-shot learning, to enhance APR for IPAs. Our method uses MaxSAT-based fault localization to identify buggy parts of a program, then presents the LLM with a program sketch devoid of these buggy statements. This hybrid approach follows a CEGIS loop to iteratively refine the program. We ask the LLM to synthesize the missing parts, which are then checked against a test suite. If the suggested program is incorrect, a counterexample from the test suite is fed back to the LLM. Our experiments show that our counterexample guided approach, using MaxSAT-based bug-free program sketches, significantly improves the repair capabilities of all six evaluated LLMs. This method allows LLMs to repair more programs with smaller fixes, outperforming other configurations and state-of-the-art symbolic program repair tools.
Momentum Posterior Regularization for Multi-hop Dense Retrieval
Venue: arXiv cs.CL 2024
Year: 2024

Multi-hop question answering (QA) often requires sequential retrieval (multi-hop retrieval), where each hop retrieves missing knowledge based on information from previous hops. To facilitate more effective retrieval, we aim to distill knowledge from a posterior retrieval, which has access to posterior information like an answer, into a prior retrieval used during inference when such information is unavailable. Unfortunately, current methods for knowledge distillation in one-time retrieval are ineffective for multi-hop QA due to two issues: 1) Posterior information is often defined as the response (i.e. the answer), which may not clearly connect to the query without intermediate retrieval; and 2) The large knowledge gap between prior and posterior retrievals makes existing distillation methods unstable, even resulting in performance loss. As such, we propose MoPo (Momentum Posterior Regularization) with two key innovations: 1) Posterior information of one hop is defined as a query-focus summary from the golden knowledge of the previous and current hops; 2) We develop an effective training strategy where the posterior retrieval is updated along with the prior retrieval via momentum moving average method, allowing smoother and effective distillation. Experiments on HotpotQA and StrategyQA demonstrate that MoPo outperforms existing baselines in both retrieval and downstream QA tasks.
Network Analysis, Plot Theory: Revisiting French Literature through Character Networks
Venue: arXiv cs.DL 2024
Year: 2024

Character recognition is a technique that enables the automated extraction of characters from texts, while coreference resolution establishes connections between various mentions of the same character, collectively facilitating the creation of expansive character networks (Moretti, 2011). Together, these technologies make it possible to navigate and analyze large literary corpora, opening new avenues for in-depth exploration and understanding of literature. We have created a system specifically for the French language, based on BookNLP-fr (the French counterpart of BookNLP) and NetworkX (a Python package for the manipulation and visualization of complex networks). This allows us to establish connections between series of literary works based on structural features (such as typical relationships between characters) or specific subgenres (for instance, adventure novels featuring a group of young heroes). In this paper, as an illustration, we show the networks obtained at different stages of the short novel Boule de Suif from Maupassant (a French 19th century novelist). These figures effectively illustrate how the relationships between the characters develop over the course of the story.
Blockchain-Driven Solutions for Carbon Credit Trading: A Decentralized Platform for SMEs
Venue: arXiv cs.CR 2024
Year: 2024

The increasing demand for sustainability and compliance with global carbon regulations has posed significant challenges for small and medium-sized enterprises (SMEs). This paper proposes a blockchain-based decentralized carbon credit trading platform tailored for SMEs in Taiwan, aiming to simplify the complex carbon trading process and lower market entry barriers. Drawing upon the Diffusion of Innovations theory and transaction cost economics, we illustrate how blockchain technology can reduce informational asymmetry and intermediary costs in carbon markets. By integrating Ethereum-based smart contracts, the platform automates transactions, enhances transparency, and reduces administrative burdens - addressing key obstacles such as technical complexity and market risks. A controlled experimental design was conducted to compare the proposed system with a conventional centralized carbon trading platform. Statistical analysis confirms its effectiveness in minimizing time and expenses while ensuring compliance with the Carbon Border Adjustment Mechanism (CBAM) and the Clean Competition Act (CCA). User satisfaction was measured using the Kano model, with the results identifying essential features and prioritizing future enhancements. This study contributes a more comprehensive solution for SMEs seeking to achieve carbon neutrality, underscoring the transformative potential of blockchain technology in global carbon markets.
Probability Bracket Notation, Markov Chains, Stochastic Processes, and Microscopic Probabilistic Processes
Venue: arXiv cs.OH 2024
Year: 2024

Inspired by the Dirac vector probability notation (VPN), we propose the Probability Bracket Notation (PBN), a new set of symbols defined similarly (but not identically) as in the VPN. Applying the PBN to fundamental definitions and theorems for discrete and continuous random variables, we show that the PBN could play a similar role in the probability space as the VBN in the Hilbert vector. Our system P-kets are identified with the probability vectors in Markov chains (MC). The master equation of homogeneous MC in the Schrodinger pictures can be basis-independent. Our system P-bra is linked to the Doi state function and the Peliti standard bra. Transformed from the Schrodinger picture to the Heisenberg picture, the time dependence of the system P-ket of a homogeneous MC (HMC) is shifted to the observable as a stochastic process. Using the correlations established by the special Wick rotation (SWR), the microscopic probabilistic processes (MPPs) are investigated for single and many-particle systems. The expected occupation number of particles in quantum statistics is reproduced by associating time with temperature (the Wick-Matsubara relation).
Measuring Scientific Group Performance: Integrating h-Group and Homogeneity into the $α$-Index
Venue: arXiv cs.IT 2025
Year: 2025

Ranking groups of researchers is important in several contexts and can serve many purposes such as the fair distribution of grants based on the scientist's publication output, concession of research projects, classification of journal editorial boards and many other applications in a social context. In this paper, we propose a method for measuring the performance of groups of researchers. The proposed method is called alpha-index and it is based on two parameters: (i) the homogeneity of the h-indexes of the researchers in the group; and (ii) the h-group, which is an extension of the h-index for groups. Our method integrates the concepts of homogeneity and absolute value of the h-index into a single measure which is appropriate for the evaluation of groups. We report on experiments that assess computer science conferences based on the h-indexes of their program committee members. Our results are similar to a manual classification scheme adopted by a research agency.
Considerations for Cloud Security Operations
Venue: arXiv cs.CR 2025
Year: 2025

Information Security in Cloud Computing environments is explored. Cloud Computing is presented, security needs are discussed, and mitigation approaches are listed. Topics covered include Information Security, Cloud Computing, Private Cloud, Public Cloud, SaaS, PaaS, IaaS, ISO 27001, OWASP, Secure SDLC.
Optimal quantization for a probability measure on a nonuniform stretched Sierpiński triangle
Venue: arXiv cs.IT 2025
Year: 2025

Quantization for a Borel probability measure refers to the idea of estimating a given probability by a discrete probability with support containing a finite number of elements. In this paper, we have considered a Borel probability measure $P$ on $\mathbb R^2$, which has support a nonuniform stretched Sierpiński triangle generated by a set of three contractive similarity mappings on $\mathbb R^2$. For this probability measure, we investigate the optimal sets of $n$-means and the $n$th quantization errors for all positive integers $n$.
A Framework for Algorithm Stability
Venue: arXiv cs.DS 2025
Year: 2025

We say that an algorithm is stable if small changes in the input result in small changes in the output. This kind of algorithm stability is particularly relevant when analyzing and visualizing time-varying data. Stability in general plays an important role in a wide variety of areas, such as numerical analysis, machine learning, and topology, but is poorly understood in the context of (combinatorial) algorithms. In this paper we present a framework for analyzing the stability of algorithms. We focus in particular on the trade-off between the stability of an algorithm and the quality of the solution it computes. Our framework allows for three types of stability analysis with increasing degrees of complexity: event stability, topological stability, and Lipschitz stability. In addition, we need to refine the model of an algorithm based on how it interacts with the time-varying data, for which we consider several options. We demonstrate the use of our stability framework by applying it to kinetic Euclidean minimum spanning trees.
Innovative Software Development and Project Management Framework for Technology Startups
Venue: arXiv cs.SE 2025
Year: 2025

This paper introduces a new process that integrates inventive problem-solving methods into modern software development. The central research question addresses how tech startups can enhance their software development processes with minimal project management expertise. The Systematic Innovation Mounted Software Development Process, which blends Agile and Systematic Innovation, offers an alternative framework to facilitate idea generation in software products. This intuitive project management framework empowers technology-driven companies to manage their projects more effectively. The study aims to provide guidelines for entrepreneurs to manage projects successfully. By collaborating with the existing Agile model, the Systematic Innovation model fosters creativity and innovative problem-solving. Ultimately, this new software development process and its techniques have the potential to significantly impact the software industry, particularly for startups, as they alleviate managerial burdens and allow companies to focus on their core technologies.
Paradigm Completion for Derivational Morphology
Venue: arXiv cs.CL 2025
Year: 2025

The generation of complex derived word forms has been an overlooked problem in NLP; we fill this gap by applying neural sequence-to-sequence models to the task. We overview the theoretical motivation for a paradigmatic treatment of derivational morphology, and introduce the task of derivational paradigm completion as a parallel to inflectional paradigm completion. State-of-the-art neural models, adapted from the inflection task, are able to learn a range of derivation patterns, and outperform a non-neural baseline by 16.4%. However, due to semantic, historical, and lexical considerations involved in derivational morphology, future work will be needed to achieve performance parity with inflection-generating systems.
Cross-lingual, Character-Level Neural Morphological Tagging
Venue: arXiv cs.CL 2025
Year: 2025

Even for common NLP tasks, sufficient supervision is not available in many languages -- morphological tagging is no exception. In the work presented here, we explore a transfer learning scheme, whereby we train character-level recurrent neural taggers to predict morphological taggings for high-resource languages and low-resource languages together. Learning joint character representations among multiple related languages successfully enables knowledge transfer from the high-resource languages to the low-resource ones, improving accuracy by up to 30% over a monolingual model.