OpenRaiser/Intern-Atlas huggingface dataset:
Securing Proof of Stake Blockchains: Leveraging Multi-Agent Reinforcement Learning for Detecting and Mitigating Malicious Nodes
Venue: arXiv cs.DC 2024
Year: 2024

Proof of Stake (PoS) blockchains offer promising alternatives to traditional Proof of Work (PoW) systems, providing scalability and energy efficiency. However, blockchains operate in a decentralized manner and the network is composed of diverse users. This openness creates the potential for malicious nodes to disrupt the network in various ways. Therefore, it is crucial to embed a mechanism within the blockchain network to constantly monitor, identify, and eliminate these malicious nodes without involving any central authority. In this paper, we propose MRL-PoS+, a novel consensus algorithm to enhance the security of PoS blockchains by leveraging Multi-agent Reinforcement Learning (MRL) techniques. Our proposed consensus algorithm introduces a penalty-reward scheme for detecting and eliminating malicious nodes. This approach involves the detection of behaviors that can lead to potential attacks in a blockchain network and hence penalizes the malicious nodes, restricting them from performing certain actions. Our developed Proof of Concept demonstrates effectiveness in eliminating malicious nodes for six types of major attacks. Experimental results demonstrate that MRL-PoS+ significantly improves the attack resilience of PoS blockchains compared to the traditional schemes without incurring additional computation overhead.
PIXELMOD: Improving Soft Moderation of Visual Misleading Information on Twitter
Venue: arXiv cs.CV 2024
Year: 2024

Images are a powerful and immediate vehicle to carry misleading or outright false messages, yet identifying image-based misinformation at scale poses unique challenges. In this paper, we present PIXELMOD, a system that leverages perceptual hashes, vector databases, and optical character recognition (OCR) to efficiently identify images that are candidates to receive soft moderation labels on Twitter. We show that PIXELMOD outperforms existing image similarity approaches when applied to soft moderation, with negligible performance overhead. We then test PIXELMOD on a dataset of tweets surrounding the 2020 US Presidential Election, and find that it is able to identify visually misleading images that are candidates for soft moderation with 0.99% false detection and 2.06% false negatives.
The Dual-Edged Sword of Technical Debt: Benefits and Issues Analyzed Through Developer Discussions
Venue: arXiv cs.SE 2024
Year: 2024

Background. Technical debt (TD) has long been one of the key factors influencing the maintainability of software products. It represents technical compromises that sacrifice long-term software quality for potential short-term benefits. Objective. This work is to collectively investigate the practitioners' opinions on the various perspectives of TD from a large collection of articles. We find the topics and latent details of each, where the sentiments of the detected opinions are also considered. Method. For such a purpose, we conducted a grey literature review on the articles systematically collected from three mainstream technology forums. Furthermore, we adopted natural language processing techniques like topic modeling and sentiment analysis to achieve a systematic and comprehensive understanding. However, we adopted ChatGPT to support the topic interpretation. Results. In this study, 2,213 forum posts and articles were collected, with eight main topics and 43 sub-topics identified. For each topic, we obtained the practitioners' collective positive and negative opinions. Conclusion. We identified 8 major topics in TD related to software development. Identified challenges by practitioners include unclear roles and a lack of engagement. On the other hand, active management supports collaboration and mitigates the impact of TD on the source code.
Data-Driven Abstractions via Binary-Tree Gaussian Processes for Formal Verification
Venue: arXiv cs.LO 2024
Year: 2024

To advance formal verification of stochastic systems against temporal logic requirements for handling unknown dynamics, researchers have been designing data-driven approaches inspired by breakthroughs in the underlying machine learning techniques. As one promising research direction, abstraction-based solutions based on Gaussian process (GP) regression have become popular for their ability to learn a representation of the latent system from data with a quantified error. Results obtained based on this model are then translated to the true system via various methods. In a recent publication, GPs using a so-called binary-tree kernel have demonstrated a polynomial speedup w.r.t. the size of the data compared to their vanilla version, outcompeting all existing sparse GP approximations. Incidentally, the resulting binary-tree Gaussian process (BTGP) is characteristic for its piecewise-constant posterior mean and covariance functions, naturally abstracting the input space into discrete partitions. In this paper, we leverage this natural abstraction of the BTGP for formal verification, eliminating the need for cumbersome abstraction and error quantification procedures. We show that the BTGP allows us to construct an interval Markov chain model of the unknown system with a speedup that is polynomial w.r.t. the size of the abstraction compared to alternative approaches. We provide a delocalized error quantification via a unified formula even when the true dynamics do not live in the function space of the BTGP. This allows us to compute upper and lower bounds on the probability of satisfying reachability specifications that are robust to both aleatoric and epistemic uncertainties.
Mapping Patient Trajectories: Understanding and Visualizing Sepsis Prognostic Pathways from Patients Clinical Narratives
Venue: arXiv cs.CL 2024
Year: 2024

In recent years, healthcare professionals are increasingly emphasizing on personalized and evidence-based patient care through the exploration of prognostic pathways. To study this, structured clinical variables from Electronic Health Records (EHRs) data have traditionally been employed by many researchers. Presently, Natural Language Processing models have received great attention in clinical research which expanded the possibilities of using clinical narratives. In this paper, we propose a systematic methodology for developing sepsis prognostic pathways derived from clinical notes, focusing on diverse patient subgroups identified by exploring comorbidities associated with sepsis and generating explanations of these subgroups using SHAP. The extracted prognostic pathways of these subgroups provide valuable insights into the dynamic trajectories of sepsis severity over time. Visualizing these pathways sheds light on the likelihood and direction of disease progression across various contexts and reveals patterns and pivotal factors or biomarkers influencing the transition between sepsis stages, whether toward deterioration or improvement. This empowers healthcare providers to implement more personalized and effective healthcare strategies for individual patients.
Modular RAG: Transforming RAG Systems into LEGO-like Reconfigurable Frameworks
Venue: arXiv cs.CL 2024
Year: 2024

Retrieval-augmented Generation (RAG) has markedly enhanced the capabilities of Large Language Models (LLMs) in tackling knowledge-intensive tasks. The increasing demands of application scenarios have driven the evolution of RAG, leading to the integration of advanced retrievers, LLMs and other complementary technologies, which in turn has amplified the intricacy of RAG systems. However, the rapid advancements are outpacing the foundational RAG paradigm, with many methods struggling to be unified under the process of "retrieve-then-generate". In this context, this paper examines the limitations of the existing RAG paradigm and introduces the modular RAG framework. By decomposing complex RAG systems into independent modules and specialized operators, it facilitates a highly reconfigurable framework. Modular RAG transcends the traditional linear architecture, embracing a more advanced design that integrates routing, scheduling, and fusion mechanisms. Drawing on extensive research, this paper further identifies prevalent RAG patterns-linear, conditional, branching, and looping-and offers a comprehensive analysis of their respective implementation nuances. Modular RAG presents innovative opportunities for the conceptualization and deployment of RAG systems. Finally, the paper explores the potential emergence of new operators and paradigms, establishing a solid theoretical foundation and a practical roadmap for the continued evolution and practical deployment of RAG technologies.
Learning Optimal Signal Temporal Logic Decision Trees for Classification: A Max-Flow MILP Formulation
Venue: arXiv cs.LG 2024
Year: 2024

This paper presents a novel framework for inferring timed temporal logic properties from data. The dataset comprises pairs of finite-time system traces and corresponding labels, denoting whether the traces demonstrate specific desired behaviors, e.g. whether the ship follows a safe route or not. Our proposed approach leverages decision-tree-based methods to infer Signal Temporal Logic classifiers using primitive formulae. We formulate the inference process as a mixed integer linear programming optimization problem, recursively generating constraints to determine both data classification and tree structure. Applying a max-flow algorithm on the resultant tree transforms the problem into a global optimization challenge, leading to improved classification rates compared to prior methodologies. Moreover, we introduce a technique to reduce the number of constraints by exploiting the symmetry inherent in STL primitives, which enhances the algorithm's time performance and interpretability. To assess our algorithm's effectiveness and classification performance, we conduct three case studies involving two-class, multi-class, and complex formula classification scenarios.
Entropy, Thermodynamics and the Geometrization of the Language Model
Venue: arXiv cs.CL 2024
Year: 2024

In this paper, we discuss how pure mathematics and theoretical physics can be applied to the study of language models. Using set theory and analysis, we formulate mathematically rigorous definitions of language models, and introduce the concept of the moduli space of distributions for a language model. We formulate a generalized distributional hypothesis using functional analysis and topology. We define the entropy function associated with a language model and show how it allows us to understand many interesting phenomena in languages. We argue that the zero points of the entropy function and the points where the entropy is close to 0 are the key obstacles for an LLM to approximate an intelligent language model, which explains why good LLMs need billions of parameters. Using the entropy function, we formulate a conjecture about AGI. Then, we show how thermodynamics gives us an immediate interpretation to language models. In particular we will define the concepts of partition function, internal energy and free energy for a language model, which offer insights into how language models work. Based on these results, we introduce a general concept of the geometrization of language models and define what is called the Boltzmann manifold. While the current LLMs are the special cases of the Boltzmann manifold.
Teaching Survey Research in Software Engineering
Venue: arXiv cs.SE 2024
Year: 2024

In this chapter, we provide advice on how to effectively teach survey research based on lessons learned from several international teaching experiences on the topic and from conducting large-scale surveys published at various scientific conferences and journals. First, we provide teachers with a potential syllabus for teaching survey research, including learning objectives, lectures, and examples of practical assignments. Thereafter, we provide actionable advice on how to teach the topics related to each learning objective, including survey design, sampling, data collection, statistical and qualitative analysis, threats to validity and reliability, and ethical considerations. The chapter is complemented by online teaching resources, including slides covering an entire course.
Deduction Game Framework and Information Set Entropy Search
Venue: arXiv cs.AI 2024
Year: 2024

We present a game framework tailored for deduction games, enabling structured analysis from the perspective of Shannon entropy variations. Additionally, we introduce a new forward search algorithm, Information Set Entropy Search (ISES), which effectively solves many single-player deduction games. The ISES algorithm, augmented with sampling techniques, allows agents to make decisions within controlled computational resources and time constraints. Experimental results on eight games within our framework demonstrate the significant superiority of our method over the Single Observer Information Set Monte Carlo Tree Search(SO-ISMCTS) algorithm under limited decision time constraints. The entropy variation of game states in our framework enables explainable decision-making, which can also be used to analyze the appeal of deduction games and provide insights for game designers.
LFFR: Logistic Function For (multi-output) Regression
Venue: arXiv cs.CR 2024
Year: 2024

In this manuscript, we extend our previous work on privacy-preserving regression to address multi-output regression problems using data encrypted under a fully homomorphic encryption scheme. We build upon the simplified fixed Hessian approach for linear and ridge regression and adapt our novel LFFR algorithm, initially designed for single-output logistic regression, to handle multiple outputs. We further refine the constant simplified Hessian method for the multi-output context, ensuring computational efficiency and robustness. Evaluations on multiple real-world datasets demonstrate the effectiveness of our multi-output LFFR algorithm, highlighting its capability to maintain privacy while achieving high predictive accuracy. Normalizing both data and target predictions remains essential for optimizing homomorphic encryption parameters, confirming the practicality of our approach for secure and efficient multi-output regression tasks.
Diffusion-Based Generation of Neural Activity from Disentangled Latent Codes
Venue: arXiv cs.LG 2024
Year: 2024

Recent advances in recording technology have allowed neuroscientists to monitor activity from thousands of neurons simultaneously. Latent variable models are increasingly valuable for distilling these recordings into compact and interpretable representations. Here we propose a new approach to neural data analysis that leverages advances in conditional generative modeling to enable the unsupervised inference of disentangled behavioral variables from recorded neural activity. Our approach builds on InfoDiffusion, which augments diffusion models with a set of latent variables that capture important factors of variation in the data. We apply our model, called Generating Neural Observations Conditioned on Codes with High Information (GNOCCHI), to time series neural data and test its application to synthetic and biological recordings of neural activity during reaching. In comparison to a VAE-based sequential autoencoder, GNOCCHI learns higher-quality latent spaces that are more clearly structured and more disentangled with respect to key behavioral variables. These properties enable accurate generation of novel samples (unseen behavioral conditions) through simple linear traversal of the latent spaces produced by GNOCCHI. Our work demonstrates the potential of unsupervised, information-based models for the discovery of interpretable latent spaces from neural data, enabling researchers to generate high-quality samples from unseen conditions.
Automated Quantification of Hyperreflective Foci in SD-OCT With Diabetic Retinopathy
Venue: arXiv cs.AI 2024
Year: 2024

The presence of hyperreflective foci (HFs) is related to retinal disease progression, and the quantity has proven to be a prognostic factor of visual and anatomical outcome in various retinal diseases. However, lack of efficient quantitative tools for evaluating the HFs has deprived ophthalmologist of assessing the volume of HFs. For this reason, we propose an automated quantification algorithm to segment and quantify HFs in spectral domain optical coherence tomography (SD-OCT). The proposed algorithm consists of two parallel processes namely: region of interest (ROI) generation and HFs estimation. To generate the ROI, we use morphological reconstruction to obtain the reconstructed image and histogram constructed for data distributions and clustering. In parallel, we estimate the HFs by extracting the extremal regions from the connected regions obtained from a component tree. Finally, both the ROI and the HFs estimation process are merged to obtain the segmented HFs. The proposed algorithm was tested on 40 3D SD-OCT volumes from 40 patients diagnosed with non-proliferative diabetic retinopathy (NPDR), proliferative diabetic retinopathy (PDR), and diabetic macular edema (DME). The average dice similarity coefficient (DSC) and correlation coefficient (r) are 69.70%, 0.99 for NPDR, 70.31%, 0.99 for PDR, and 71.30%, 0.99 for DME, respectively. The proposed algorithm can provide ophthalmologist with good HFs quantitative information, such as volume, size, and location of the HFs.
Fine-grained Metrics for Point Cloud Semantic Segmentation
Venue: arXiv cs.CV 2024
Year: 2024

Two forms of imbalances are commonly observed in point cloud semantic segmentation datasets: (1) category imbalances, where certain objects are more prevalent than others; and (2) size imbalances, where certain objects occupy more points than others. Because of this, the majority of categories and large objects are favored in the existing evaluation metrics. This paper suggests fine-grained mIoU and mAcc for a more thorough assessment of point cloud segmentation algorithms in order to address these issues. Richer statistical information is provided for models and datasets by these fine-grained metrics, which also lessen the bias of current semantic segmentation metrics towards large objects. The proposed metrics are used to train and assess various semantic segmentation algorithms on three distinct indoor and outdoor semantic segmentation datasets.
SimpleLLM4AD: An End-to-End Vision-Language Model with Graph Visual Question Answering for Autonomous Driving
Venue: arXiv cs.CV 2024
Year: 2024

Many fields could benefit from the rapid development of the large language models (LLMs). The end-to-end autonomous driving (e2eAD) is one of the typically fields facing new opportunities as the LLMs have supported more and more modalities. Here, by utilizing vision-language model (VLM), we proposed an e2eAD method called SimpleLLM4AD. In our method, the e2eAD task are divided into four stages, which are perception, prediction, planning, and behavior. Each stage consists of several visual question answering (VQA) pairs and VQA pairs interconnect with each other constructing a graph called Graph VQA (GVQA). By reasoning each VQA pair in the GVQA through VLM stage by stage, our method could achieve e2e driving with language. In our method, vision transformers (ViT) models are employed to process nuScenes visual data, while VLM are utilized to interpret and reason about the information extracted from the visual inputs. In the perception stage, the system identifies and classifies objects from the driving environment. The prediction stage involves forecasting the potential movements of these objects. The planning stage utilizes the gathered information to develop a driving strategy, ensuring the safety and efficiency of the autonomous vehicle. Finally, the behavior stage translates the planned actions into executable commands for the vehicle. Our experiments demonstrate that SimpleLLM4AD achieves competitive performance in complex driving scenarios.
Pathology Foundation Models
Venue: arXiv cs.CV 2024
Year: 2024

Pathology has played a crucial role in the diagnosis and evaluation of patient tissue samples obtained from surgeries and biopsies for many years. The advent of Whole Slide Scanners and the development of deep learning technologies have significantly advanced the field, leading to extensive research and development in pathology AI (Artificial Intelligence). These advancements have contributed to reducing the workload of pathologists and supporting decision-making in treatment plans. Recently, large-scale AI models known as Foundation Models (FMs), which are more accurate and applicable to a wide range of tasks compared to traditional AI, have emerged, and expanded their application scope in the healthcare field. Numerous FMs have been developed in pathology, and there are reported cases of their application in various tasks, such as disease diagnosis, rare cancer diagnosis, patient survival prognosis prediction, biomarker expression prediction, and the scoring of immunohistochemical expression intensity. However, several challenges remain for the clinical application of FMs, which healthcare professionals, as users, must be aware of. Research is ongoing to address these challenges. In the future, it is expected that the development of Generalist Medical AI, which integrates pathology FMs with FMs from other medical domains, will progress, leading to the effective utilization of AI in real clinical settings to promote precision and personalized medicine.
Big Cooperative Learning
Venue: arXiv cs.LG 2024
Year: 2024

Cooperation plays a pivotal role in the evolution of human intelligence; moreover, it also underlies the recent revolutionary advancement of artificial intelligence (AI) that is driven by foundation models. Specifically, we reveal that the training of foundation models can be interpreted as a form of big cooperative learning (\textit{abbr.} big learning), where massive learning individuals/tasks \emph{cooperate} to approach the unique essence of data from diverse perspectives of data prediction, leveraging a universal model. The presented big learning therefore unifies most training objectives of foundation models within a consistent framework, where their underlying assumptions are exposed simultaneously. We design tailored simulations to demonstrate the principle of big learning, based on which we provide learning-perspective justifications for the successes of foundation models, with interesting side-products. Furthermore, we reveal that big learning is a new dimension for upgrading conventional machine learning paradigms, valuable for endowing reinvigorations to associated applications; as an illustrative example, we propose the BigLearn-GAN, which is a novel adversarially-trained foundation model with versatile data sampling capabilities. Code is available at \texttt{https://github.com/YulaiCong/BigCooperativeLearning}.
CAMAv2: A Vision-Centric Approach for Static Map Element Annotation
Venue: arXiv cs.CV 2024
Year: 2024

The recent development of online static map element (a.k.a. HD map) construction algorithms has raised a vast demand for data with ground truth annotations. However, available public datasets currently cannot provide high-quality training data regarding consistency and accuracy. For instance, the manual labelled (low efficiency) nuScenes still contains misalignment and inconsistency between the HD maps and images (e.g., around 8.03 pixels reprojection error on average). To this end, we present CAMAv2: a vision-centric approach for Consistent and Accurate Map Annotation. Without LiDAR inputs, our proposed framework can still generate high-quality 3D annotations of static map elements. Specifically, the annotation can achieve high reprojection accuracy across all surrounding cameras and is spatial-temporal consistent across the whole sequence. We apply our proposed framework to the popular nuScenes dataset to provide efficient and highly accurate annotations. Compared with the original nuScenes static map element, our CAMAv2 annotations achieve lower reprojection errors (e.g., 4.96 vs. 8.03 pixels). Models trained with annotations from CAMAv2 also achieve lower reprojection errors (e.g., 5.62 vs. 8.43 pixels).