OpenRaiser/Intern-Atlas huggingface dataset:
LLM-based Continuous Intrusion Detection Framework for Next-Gen Networks
Venue: arXiv cs.CR 2024
Year: 2024

In this paper, we present an adaptive framework designed for the continuous detection, identification and classification of emerging attacks in network traffic. The framework employs a transformer encoder architecture, which captures hidden patterns in a bidirectional manner to differentiate between malicious and legitimate traffic. Initially, the framework focuses on the accurate detection of malicious activities, achieving a perfect recall of 100\% in distinguishing between attack and benign traffic. Subsequently, the system incrementally identifies unknown attack types by leveraging a Gaussian Mixture Model (GMM) to cluster features derived from high-dimensional BERT embeddings. This approach allows the framework to dynamically adjust its identification capabilities as new attack clusters are discovered, maintaining high detection accuracy. Even after integrating additional unknown attack clusters, the framework continues to perform at a high level, achieving 95.6\% in both classification accuracy and recall.The results demonstrate the effectiveness of the proposed framework in adapting to evolving threats while maintaining high accuracy in both detection and identification tasks. Our ultimate goal is to develop a scalable, real-time intrusion detection system that can continuously evolve with the ever-changing network threat landscape.
Balancing Profit and Traveller Acceptance in Ride-Pooling Personalised Fares
Venue: arXiv cs.GT 2024
Year: 2024

In a ride-pooling system, travellers experience discomfort associated with a detour and a longer travel time, which is compensated with a sharing discount. Most studies assume travellers receive either a flat discount or, in rare cases, a proportional to the inconvenience. We show the system benefits from individually tailored fares. We argue that fares that optimise an expected profit of an operator also improve system-wide performance if they include travellers' acceptance. Our pricing method is set in a heterogeneous population, where travellers have varying levels of value-of-time and willingness-to-share, unknown to the operator. A high fare discourages clients from the service, while a low fare reduces the profit margin. Notably, a shared ride is only realised if accepted by all co-travellers (decision is driven by the latent behavioural factors). Our method reveals intriguing properties of the shareability topology. Not only identifies rides efficient for the system and supports them with reduced fares (to increase their realisation probability), but also identifies travellers unattractive for the system (e.g. due to incompatibility with other travellers) and effectively shifts them to private rides via high fares. Unlike in previous methods, such approach naturally balances the travellers satisfaction and the profit maximisation. With an experiment set in NYC, we show that this leads to significant improvements over the flat discount baseline: the mileage (proxy for environmental externalities) is reduced by 4.5% and the operator generates more profit per mile (over 20% improvement). We argue that ride pooling systems with fares that maximise profitability are more sustainable and efficient if they include travellers' satisfaction. Keywords: ride-pooling, personalised pricing, individual discounts
An Open API Architecture to Discover the Trustworthy Explanation of Cloud AI Services
Venue: arXiv cs.DC 2024
Year: 2024

This article presents the design of an open-API-based explainable AI (XAI) service to provide feature contribution explanations for cloud AI services. Cloud AI services are widely used to develop domain-specific applications with precise learning metrics. However, the underlying cloud AI services remain opaque on how the model produces the prediction. We argue that XAI operations are accessible as open APIs to enable the consolidation of the XAI operations into the cloud AI services assessment. We propose a design using a microservice architecture that offers feature contribution explanations for cloud AI services without unfolding the network structure of the cloud models. We can also utilize this architecture to evaluate the model performance and XAI consistency metrics showing cloud AI services trustworthiness. We collect provenance data from operational pipelines to enable reproducibility within the XAI service. Furthermore, we present the discovery scenarios for the experimental tests regarding model performance and XAI consistency metrics for the leading cloud vision AI services. The results confirm that the architecture, based on open APIs, is cloud-agnostic. Additionally, data augmentations result in measurable improvements in XAI consistency metrics for cloud AI services.
GRATEV2.0: Computational Tools for Real-time Analysis of High-throughput High-resolution TEM (HRTEM) Images of Conjugated Polymers
Venue: arXiv cs.CE 2024
Year: 2024

Automated analysis of high-resolution transmission electron microscopy (HRTEM) images is increasingly essential for advancing research in organic electronics, where precise characterization of nanoscale crystal structures is crucial for optimizing material properties. This paper introduces an open-source computational framework called GRATEV2.0 (GRaph-based Analysis of TEM), designed for real-time analysis of HRTEM data, with a focus on characterizing complex microstructures in conjugated polymers, illustrated using Poly[N-9'-heptadecanyl-2,7-carbazole-alt-5,5-(4',7'-di-2-thienyl-2',1',3'-benzothiadiazole)] (PCDTBT), a key material in organic photovoltaics. GRATEV2.0 employs fast, automated image processing algorithms, enabling rapid extraction of structural features like d-spacing, orientation, and crystal shape metrics. Gaussian process optimization rapidly identifies the user-defined parameters in the approach, reducing the need for manual parameter tuning and thus enhancing reproducibility and usability. Additionally, GRATEV2.0 is compatible with high-performance computing (HPC) environments, allowing for efficient, large-scale data processing at near real-time speeds. A unique feature of GRATEV2.0 is a Wasserstein distance-based stopping criterion, which optimizes data collection by determining when further sampling no longer adds statistically significant information. This capability optimizes the amount of time the TEM facility is used while ensuring data adequacy for in-depth analysis. Open-source and tested on a substantial PCDTBT dataset, this tool offers a powerful, robust, and accessible solution for high-throughput material characterization in organic electronics.
Automatic Generation of Question Hints for Mathematics Problems using Large Language Models in Educational Technology
Venue: arXiv cs.CL 2024
Year: 2024

The automatic generation of hints by Large Language Models (LLMs) within Intelligent Tutoring Systems (ITSs) has shown potential to enhance student learning. However, generating pedagogically sound hints that address student misconceptions and adhere to specific educational objectives remains challenging. This work explores using LLMs (GPT-4o and Llama-3-8B-instruct) as teachers to generate effective hints for students simulated through LLMs (GPT-3.5-turbo, Llama-3-8B-Instruct, or Mistral-7B-instruct-v0.3) tackling math exercises designed for human high-school students, and designed using cognitive science principles. We present here the study of several dimensions: 1) identifying error patterns made by simulated students on secondary-level math exercises; 2) developing various prompts for GPT-4o as a teacher and evaluating their effectiveness in generating hints that enable simulated students to self-correct; and 3) testing the best-performing prompts, based on their ability to produce relevant hints and facilitate error correction, with Llama-3-8B-Instruct as the teacher, allowing for a performance comparison with GPT-4o. The results show that model errors increase with higher temperature settings. Notably, when hints are generated by GPT-4o, the most effective prompts include prompts tailored to specific errors as well as prompts providing general hints based on common mathematical errors. Interestingly, Llama-3-8B-Instruct as a teacher showed better overall performance than GPT-4o. Also the problem-solving and response revision capabilities of the LLMs as students, particularly GPT-3.5-turbo, improved significantly after receiving hints, especially at lower temperature settings. However, models like Mistral-7B-Instruct demonstrated a decline in performance as the temperature increased.
SynthSet: Generative Diffusion Model for Semantic Segmentation in Precision Agriculture
Venue: arXiv cs.CV 2024
Year: 2024

This paper introduces a methodology for generating synthetic annotated data to address data scarcity in semantic segmentation tasks within the precision agriculture domain. Utilizing Denoising Diffusion Probabilistic Models (DDPMs) and Generative Adversarial Networks (GANs), we propose a dual diffusion model architecture for synthesizing realistic annotated agricultural data, without any human intervention. We employ super-resolution to enhance the phenotypic characteristics of the synthesized images and their coherence with the corresponding generated masks. We showcase the utility of the proposed method for wheat head segmentation. The high quality of synthesized data underscores the effectiveness of the proposed methodology in generating image-mask pairs. Furthermore, models trained on our generated data exhibit promising performance when tested on an external, diverse dataset of real wheat fields. The results show the efficacy of the proposed methodology for addressing data scarcity for semantic segmentation tasks. Moreover, the proposed approach can be readily adapted for various segmentation tasks in precision agriculture and beyond.
Model-based Deep Learning for QoS-Aware Rate-Splitting Multiple Access Wireless Systems
Venue: arXiv cs.NI 2024
Year: 2024

Next generation communications demand for better spectrum management, lower latency, and guaranteed quality-of-service (QoS). Recently, Artificial intelligence (AI) has been widely introduced to advance these aspects in next generation wireless systems. However, such AI applications suffer from limited training data, low robustness, and poor generalization capabilities. To address these issues, a model-driven deep unfolding (DU) algorithm is introduced in this paper to bridge the gap between traditional model-driven communication algorithms and data-driven deep learning. Focusing on the QoS-aware rate-splitting multiple access (RSMA) resource allocation problem in multi-user communications, a conventional fractional programming (FP) algorithm is first applied as a benchmark. The solution is then refined by the application of projection gradient descent (PGD). DU is employed to further speed up convergence procedure, hence improving the efficiency of PGD. Moreover, the feasibility of results is guaranteed by designing a low-complexity projection based on scale factors, plus adding violation control mechanisms into the loss function that minimizes error rates. Finally, we provide a detailed analysis of the computational complexity and analysis design of the proposed DU algorithm. Extensive simulations are conducted and the results demonstrate that the proposed DU algorithm can reach the optimal communication efficiency with a mere $0.024\%$ violation rate for 4 layers DU. The DU algorithm also exhibits robustness in out-of-distribution tests and can be effectively trained with as few as 50 samples.
Change Is the Only Constant: Dynamic LLM Slicing based on Layer Redundancy
Venue: arXiv cs.CL 2024
Year: 2024

This paper introduces a novel model compression approach through dynamic layer-specific pruning in Large Language Models (LLMs), enhancing the traditional methodology established by SliceGPT. By transitioning from constant to dynamic slicing, our method leverages the newly proposed Layer Redundancy (LR) score, which assesses how much change each layer changes its input by measuring the cosine similarity of the input to the output of the layer. We use this score to prune parts of individual layers based on redundancy in such a way that the average pruned percentage for all layers is a fixed value. We conducted extensive experiments using models like Llama3-8B and Mistral-7B on multiple datasets, evaluating different slicing bases and percentages to determine optimal configurations that balance efficiency and performance. Our findings show that our dynamic slicing approach not only maintains but, in many cases, enhances model performance compared to the baseline established by constant slicing methods. For instance, in several settings, we see performance improvements of up to 5% over the SliceGPT baseline. Additionally, a perplexity decrease by as much as 7% was observed across multiple benchmarks, validating the effectiveness of our method. The code, model weights, and datasets are open-sourced at https://github.com/RazvanDu/DynamicSlicing.
Towards Personalized Federated Learning via Comprehensive Knowledge Distillation
Venue: arXiv cs.LG 2024
Year: 2024

Federated learning is a distributed machine learning paradigm designed to protect data privacy. However, data heterogeneity across various clients results in catastrophic forgetting, where the model rapidly forgets previous knowledge while acquiring new knowledge. To address this challenge, personalized federated learning has emerged to customize a personalized model for each client. However, the inherent limitation of this mechanism is its excessive focus on personalization, potentially hindering the generalization of those models. In this paper, we present a novel personalized federated learning method that uses global and historical models as teachers and the local model as the student to facilitate comprehensive knowledge distillation. The historical model represents the local model from the last round of client training, containing historical personalized knowledge, while the global model represents the aggregated model from the last round of server aggregation, containing global generalized knowledge. By applying knowledge distillation, we effectively transfer global generalized knowledge and historical personalized knowledge to the local model, thus mitigating catastrophic forgetting and enhancing the general performance of personalized models. Extensive experimental results demonstrate the significant advantages of our method.
Beyond Regularity: Simple versus Optimal Mechanisms, Revisited
Venue: arXiv cs.GT 2024
Year: 2024

A large proportion of the Bayesian mechanism design literature is restricted to the family of regular distributions $\mathbb{F}_{\tt reg}$ [Mye81] or the family of monotone hazard rate (MHR) distributions $\mathbb{F}_{\tt MHR}$ [BMP63], which overshadows this beautiful and well-developed theory. We (re-)introduce two generalizations, the family of quasi-regular distributions $\mathbb{F}_{\tt Q-reg}$ and the family of quasi-MHR distributions $\mathbb{F}_{\tt Q-MHR}$. All four families together form the following hierarchy: $\mathbb{F}_{\tt MHR} \subsetneq (\mathbb{F}_{\tt reg} \cap \mathbb{F}_{\tt Q-MHR}) \subsetneq \mathbb{F}_{\tt Q-reg}$ and $\mathbb{F}_{\tt Q-MHR} \subsetneq (\mathbb{F}_{\tt reg} \cup \mathbb{F}_{\tt Q-MHR}) \subsetneq \mathbb{F}_{\tt Q-reg}$. The significance of our new families is manifold. First, their defining conditions are immediate relaxations of the regularity/MHR conditions (i.e., monotonicity of the virtual value functions and/or the hazard rate functions), which reflect economic intuition. Second, they satisfy natural mathematical properties (about order statistics) that are violated by both original families $\mathbb{F}_{\tt reg}$ and $\mathbb{F}_{\tt MHR}$. Third but foremost, numerous results [BK96, HR09a, CD15, DRY15, HR14, AHN+19, JLTX20, JLQ+19b, FLR19, GHZ19b, JLX23, LM24] established before for regular/MHR distributions now can be generalized, with or even without quantitative losses.
From Medprompt to o1: Exploration of Run-Time Strategies for Medical Challenge Problems and Beyond
Venue: arXiv cs.CL 2024
Year: 2024

Run-time steering strategies like Medprompt are valuable for guiding large language models (LLMs) to top performance on challenging tasks. Medprompt demonstrates that a general LLM can be focused to deliver state-of-the-art performance on specialized domains like medicine by using a prompt to elicit a run-time strategy involving chain of thought reasoning and ensembling. OpenAI's o1-preview model represents a new paradigm, where a model is designed to do run-time reasoning before generating final responses. We seek to understand the behavior of o1-preview on a diverse set of medical challenge problem benchmarks. Following on the Medprompt study with GPT-4, we systematically evaluate the o1-preview model across various medical benchmarks. Notably, even without prompting techniques, o1-preview largely outperforms the GPT-4 series with Medprompt. We further systematically study the efficacy of classic prompt engineering strategies, as represented by Medprompt, within the new paradigm of reasoning models. We found that few-shot prompting hinders o1's performance, suggesting that in-context learning may no longer be an effective steering approach for reasoning-native models. While ensembling remains viable, it is resource-intensive and requires careful cost-performance optimization. Our cost and accuracy analysis across run-time strategies reveals a Pareto frontier, with GPT-4o representing a more affordable option and o1-preview achieving state-of-the-art performance at higher cost. Although o1-preview offers top performance, GPT-4o with steering strategies like Medprompt retains value in specific contexts. Moreover, we note that the o1-preview model has reached near-saturation on many existing medical benchmarks, underscoring the need for new, challenging benchmarks. We close with reflections on general directions for inference-time computation with LLMs.
SEGMN: A Structure-Enhanced Graph Matching Network for Graph Similarity Learning
Venue: arXiv cs.LG 2024
Year: 2024

Graph similarity computation (GSC) aims to quantify the similarity score between two graphs. Although recent GSC methods based on graph neural networks (GNNs) take advantage of intra-graph structures in message passing, few of them fully utilize the structures presented by edges to boost the representation of their connected nodes. Moreover, previous cross-graph node embedding matching lacks the perception of the overall structure of the graph pair, due to the fact that the node representations from GNNs are confined to the intra-graph structure, causing the unreasonable similarity score. Intuitively, the cross-graph structure represented in the assignment graph is helpful to rectify the inappropriate matching. Therefore, we propose a structure-enhanced graph matching network (SEGMN). Equipped with a dual embedding learning module and a structure perception matching module, SEGMN achieves structure enhancement in both embedding learning and cross-graph matching. The dual embedding learning module incorporates adjacent edge representation into each node to achieve a structure-enhanced representation. The structure perception matching module achieves cross-graph structure enhancement through assignment graph convolution. The similarity score of each cross-graph node pair can be rectified by aggregating messages from structurally relevant node pairs. Experimental results on benchmark datasets demonstrate that SEGMN outperforms the state-of-the-art GSC methods in the GED regression task, and the structure perception matching module is plug-and-play, which can further improve the performance of the baselines by up to 25%.
StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding
Venue: arXiv cs.CV 2024
Year: 2024

The rapid development of Multimodal Large Language Models (MLLMs) has expanded their capabilities from image comprehension to video understanding. However, most of these MLLMs focus primarily on offline video comprehension, necessitating extensive processing of all video frames before any queries can be made. This presents a significant gap compared to the human ability to watch, listen, think, and respond to streaming inputs in real time, highlighting the limitations of current MLLMs. In this paper, we introduce StreamingBench, the first comprehensive benchmark designed to evaluate the streaming video understanding capabilities of MLLMs. StreamingBench assesses three core aspects of streaming video understanding: (1) real-time visual understanding, (2) omni-source understanding, and (3) contextual understanding. The benchmark consists of 18 tasks, featuring 900 videos and 4,500 human-curated QA pairs. Each video features five questions presented at different time points to simulate a continuous streaming scenario. We conduct experiments on StreamingBench with 13 open-source and proprietary MLLMs and find that even the most advanced proprietary MLLMs like Gemini 1.5 Pro and GPT-4o perform significantly below human-level streaming video understanding capabilities. We hope our work can facilitate further advancements for MLLMs, empowering them to approach human-level video comprehension and interaction in more realistic scenarios.
Digital Twin-Assisted Robust and Adaptive Resource Slicing in LEO Satellite Networks
Venue: arXiv cs.NI 2024
Year: 2024

Resource slicing in low Earth orbit satellite networks (LSN) is essential to support diversified services. In this paper, we investigate a resource slicing problem in LSN to reserve resources in satellites to achieve efficient resource provisioning. To address the challenges of non-stationary service demands, inaccurate prediction, and satellite mobility, we propose an adaptive digital twin (DT)-assisted resource slicing scheme for robust and adaptive resource management in LSN. Specifically, a slice DT, being able to capture the service demand prediction uncertainty through collected service demand data, is constructed to enhance the robustness of resource slicing decisions for dynamic service demands. In addition, the constructed DT can emulate resource slicing decisions for evaluating their performance, enabling adaptive slicing decision updates to efficiently reserve resources in LSN. Simulation results demonstrate that the proposed scheme outperforms benchmark methods, achieving low service demand violations with efficient resource consumption.
Evaluating Moral Beliefs across LLMs through a Pluralistic Framework
Venue: arXiv cs.CL 2024
Year: 2024

Proper moral beliefs are fundamental for language models, yet assessing these beliefs poses a significant challenge. This study introduces a novel three-module framework to evaluate the moral beliefs of four prominent large language models. Initially, we constructed a dataset containing 472 moral choice scenarios in Chinese, derived from moral words. The decision-making process of the models in these scenarios reveals their moral principle preferences. By ranking these moral choices, we discern the varying moral beliefs held by different language models. Additionally, through moral debates, we investigate the firmness of these models to their moral choices. Our findings indicate that English language models, namely ChatGPT and Gemini, closely mirror moral decisions of the sample of Chinese university students, demonstrating strong adherence to their choices and a preference for individualistic moral beliefs. In contrast, Chinese models such as Ernie and ChatGLM lean towards collectivist moral beliefs, exhibiting ambiguity in their moral choices and debates. This study also uncovers gender bias embedded within the moral beliefs of all examined language models. Our methodology offers an innovative means to assess moral beliefs in both artificial and human intelligence, facilitating a comparison of moral values across different cultures.