OpenRaiser/Intern-Atlas huggingface dataset:
Software analytics for software engineering: A tertiary review
Venue: arXiv cs.SE 2024
Year: 2024

Software analytics (SA) is frequently proposed as a tool to support practitioners in software engineering (SE) tasks. We have observed that several secondary studies on SA have been published. Some of these studies have overlapping aims and some have even been published in the same calendar year. This presents an opportunity to analyze the congruence or divergence of the conclusions in these studies. Such an analysis can help identify broader generalizations beyond any of the individual secondary studies. We identified five secondary studies on the use of SA for SE. These secondary studies cover primary research from 2000 to 2021. Despite the overlapping objectives and search time frames of these secondary studies, there is negligible overlap of primary studies between these secondary studies. Thus, each of them provides an isolated view, and together, they provide a fragmented view, i.e., there is no ``common picture'' of the area. Thus, we conclude that an overview of the literature identified by these secondary studies would be useful in providing a more comprehensive overview of the topic.
Retrieving, Rethinking and Revising: The Chain-of-Verification Can Improve Retrieval Augmented Generation
Venue: arXiv cs.CL 2024
Year: 2024

Recent Retrieval Augmented Generation (RAG) aims to enhance Large Language Models (LLMs) by incorporating extensive knowledge retrieved from external sources. However, such approach encounters some challenges: Firstly, the original queries may not be suitable for precise retrieval, resulting in erroneous contextual knowledge; Secondly, the language model can easily generate inconsistent answer with external references due to their knowledge boundary limitation. To address these issues, we propose the chain-of-verification (CoV-RAG) to enhance the external retrieval correctness and internal generation consistency. Specifically, we integrate the verification module into the RAG, engaging in scoring, judgment, and rewriting. To correct external retrieval errors, CoV-RAG retrieves new knowledge using a revised query. To correct internal generation errors, we unify QA and verification tasks with a Chain-of-Thought (CoT) reasoning during training. Our comprehensive experiments across various LLMs demonstrate the effectiveness and adaptability compared with other strong baselines. Especially, our CoV-RAG can significantly surpass the state-of-the-art baselines using different LLM backbones.
Gradual Learning: Optimizing Fine-Tuning with Partially Mastered Knowledge in Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

During the pretraining phase, large language models (LLMs) acquire vast amounts of knowledge from extensive text corpora. Nevertheless, in later stages such as fine-tuning and inference, the model may encounter knowledge not covered in the initial training, which can lead to hallucinations and degraded performance. This issue has a profound impact on the model's capabilities, as it will inevitably face out-of-scope knowledge after pretraining. Furthermore, fine-tuning is often required to adapt LLMs to domain-specific tasks. However, this phenomenon limits the model's ability to learn and integrate new information during fine-tuning. The effectiveness of fine-tuning largely depends on the type of knowledge involved. Existing research suggests that fine-tuning the model on partially mastered knowledge-for instance, question-answer pairs where the model has a chance of providing correct responses under non-greedy decoding-can enable the model to acquire new knowledge while mitigating hallucination. Notably, this approach can still lead to the forgetting of fully mastered knowledge, constraining the fine-tuning dataset to a narrower range and limiting the model's overall potential for improvement. Given the model's intrinsic reasoning abilities and the interconnectedness of different knowledge areas, it is likely that as the model's capacity to utilize existing knowledge improves during fine-tuning, previously unmastered knowledge may become more understandable. To explore this hypothesis, we conducted experiments and, based on the results, proposed a two-stage fine-tuning strategy. This approach not only improves the model's overall test accuracy and knowledge retention but also preserves its accuracy on previously mastered content. When fine-tuning on the WikiQA dataset, our method increases the amount of knowledge acquired by the model in this stage by 24%.
Probing Language Models on Their Knowledge Source
Venue: arXiv cs.CL 2024
Year: 2024

Large Language Models (LLMs) often encounter conflicts between their learned, internal (parametric knowledge, PK) and external knowledge provided during inference (contextual knowledge, CK). Understanding how LLMs models prioritize one knowledge source over the other remains a challenge. In this paper, we propose a novel probing framework to explore the mechanisms governing the selection between PK and CK in LLMs. Using controlled prompts designed to contradict the model's PK, we demonstrate that specific model activations are indicative of the knowledge source employed. We evaluate this framework on various LLMs of different sizes and demonstrate that mid-layer activations, particularly those related to relations in the input, are crucial in predicting knowledge source selection, paving the way for more reliable models capable of handling knowledge conflicts effectively.
Bottom-up Anytime Discovery of Generalised Multimodal Graph Patterns for Knowledge Graphs
Venue: arXiv cs.AI 2024
Year: 2024

Vast amounts of heterogeneous knowledge are becoming publicly available in the form of knowledge graphs, often linking multiple sources of data that have never been together before, and thereby enabling scholars to answer many new research questions. It is often not known beforehand, however, which questions the data might have the answers to, potentially leaving many interesting and novel insights to remain undiscovered. To support scholars during this scientific workflow, we introduce an anytime algorithm for the bottom-up discovery of generalized multimodal graph patterns in knowledge graphs. Each pattern is a conjunction of binary statements with (data-) type variables, constants, and/or value patterns. Upon discovery, the patterns are converted to SPARQL queries and presented in an interactive facet browser together with metadata and provenance information, enabling scholars to explore, analyse, and share queries. We evaluate our method from a user perspective, with the help of domain experts in the humanities.
MDAP: A Multi-view Disentangled and Adaptive Preference Learning Framework for Cross-Domain Recommendation
Venue: arXiv cs.IR 2024
Year: 2024

Cross-domain Recommendation systems leverage multi-domain user interactions to improve performance, especially in sparse data or new user scenarios. However, CDR faces challenges such as effectively capturing user preferences and avoiding negative transfer. To address these issues, we propose the Multi-view Disentangled and Adaptive Preference Learning (MDAP) framework. Our MDAP framework uses a multiview encoder to capture diverse user preferences. The framework includes a gated decoder that adaptively combines embeddings from different views to generate a comprehensive user representation. By disentangling representations and allowing adaptive feature selection, our model enhances adaptability and effectiveness. Extensive experiments on benchmark datasets demonstrate that our method significantly outperforms state-of-the-art CDR and single-domain models, providing more accurate recommendations and deeper insights into user behavior across different domains.
Deep learning-based fault identification in condition monitoring
Venue: arXiv cs.LG 2024
Year: 2024

Vibration-based condition monitoring techniques are commonly used to identify faults in rolling element bearings. Accuracy and speed of fault detection procedures are critical performance measures in condition monitoring. Delay is especially important in remote condition monitoring and time-sensitive industrial applications. While most existing methods focus on accuracy, little attention has been given to the inference time in the fault identification process. In this paper, we address this gap by presenting a Convolutional Neural Network (CNN) based approach for real-time fault identification in rolling element bearings. We encode raw vibration signals into two-dimensional images using various encoding methods and use these with a CNN to classify several categories of bearing fault types and sizes. We analyse the interplay between fault identification accuracy and processing time. For training and evaluation we use a bearing failure CWRU dataset.
Towards an Autonomous Surface Vehicle Prototype for Artificial Intelligence Applications of Water Quality Monitoring
Venue: arXiv cs.RO 2024
Year: 2024

The use of Autonomous Surface Vehicles, equipped with water quality sensors and artificial vision systems, allows for a smart and adaptive deployment in water resources environmental monitoring. This paper presents a real implementation of a vehicle prototype that to address the use of Artificial Intelligence algorithms and enhanced sensing techniques for water quality monitoring. The vehicle is fully equipped with high-quality sensors to measure water quality parameters and water depth. Furthermore, by means of a stereo-camera, it also can detect and locate macro-plastics in real environments by means of deep visual models, such as YOLOv5. In this paper, experimental results, carried out in Lago Mayor (Sevilla), has been presented as proof of the capabilities of the proposed architecture. The overall system, and the early results obtained, are expected to provide a solid example of a real platform useful for the water resource monitoring task, and to serve as a real case scenario for deploying Artificial Intelligence algorithms, such as path planning, artificial vision, etc.
Bayesian model of individual learning to control a motor imagery BCI
Venue: arXiv cs.HC 2024
Year: 2024

The cognitive mechanisms underlying subjects' self-regulation in Brain-Computer Interface (BCI) and neurofeedback (NF) training remain poorly understood. Yet, a mechanistic computational model of each individual learning trajectory is required to improve the reliability of BCI applications. The few existing attempts mostly rely on model-free (reinforcement learning) approaches. Hence, they cannot capture the strategy developed by each subject and neither finely predict their learning curve. In this study, we propose an alternative, model-based approach rooted in cognitive skill learning within the Active Inference framework. We show how BCI training may be framed as an inference problem under high uncertainties. We illustrate the proposed approach on a previously published synthetic Motor Imagery ERD laterality training. We show how simple changes in model parameters allow us to qualitatively match experimental results and account for various subject. In the near future, this approach may provide a powerful computational to model individual skill learning and thus optimize and finely characterize BCI training.
EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment
Venue: arXiv cs.CV 2024
Year: 2024

Mamba-based architectures have shown to be a promising new direction for deep learning models owing to their competitive performance and sub-quadratic deployment speed. However, current Mamba multi-modal large language models (MLLM) are insufficient in extracting visual features, leading to imbalanced cross-modal alignment between visual and textural latents, negatively impacting performance on multi-modal tasks. In this work, we propose Empowering Multi-modal Mamba with Structural and Hierarchical Alignment (EMMA), which enables the MLLM to extract fine-grained visual information. Specifically, we propose a pixel-wise alignment module to autoregressively optimize the learning and processing of spatial image-level features along with textual tokens, enabling structural alignment at the image level. In addition, to prevent the degradation of visual information during the cross-model alignment process, we propose a multi-scale feature fusion (MFF) module to combine multi-scale visual features from intermediate layers, enabling hierarchical alignment at the feature level. Extensive experiments are conducted across a variety of multi-modal benchmarks. Our model shows lower latency than other Mamba-based MLLMs and is nearly four times faster than transformer-based MLLMs of similar scale during inference. Due to better cross-modal alignment, our model exhibits lower degrees of hallucination and enhanced sensitivity to visual details, which manifests in superior performance across diverse multi-modal benchmarks. Code will be provided.
The Cyber Alliance Game: How Alliances Influence Cyber-Warfare
Venue: arXiv cs.GT 2024
Year: 2024

Cyber-warfare has become the norm in current ongoing military conflicts. Over the past decade, numerous examples have shown the extent to which nation-states become vulnerable if they do not focus on building their cyber capacities. Adding to the inherent complexity of cyberwar scenarios, a state is usually a member of one or more alliances. Alliance policies and internal struggles could shape the individual actions of member states; intuitively, this also holds for the cyber domain. In this paper, we define and study a simple Cyber Alliance Game with the objective of understanding the fundamental influence of alliances on cyber conflicts between nation-states. Specifically, we focus on the decision of whether to exploit a newly found vulnerability individually or share it with the alliance. First, we characterize the impact of vulnerability-sharing rewards on the resulting equilibrium. Second, we study the implications of the internal power structure of alliances on cyberwar outcomes and infer the expected behavior of Dictator, Veto, and Dummy players. Finally, we investigate how alliances can nudge their members via rewards and punishments to adhere to their defensive or offensive cyber policy. We believe that our results contribute to the fundamental understanding of real-world cyber-conflicts by characterizing the impact of alliances.
Generalizing to any diverse distribution: uniformity, gentle finetuning and rebalancing
Venue: arXiv cs.LG 2024
Year: 2024

As training datasets grow larger, we aspire to develop models that generalize well to any diverse test distribution, even if the latter deviates significantly from the training data. Various approaches like domain adaptation, domain generalization, and robust optimization attempt to address the out-of-distribution challenge by posing assumptions about the relation between training and test distribution. Differently, we adopt a more conservative perspective by accounting for the worst-case error across all sufficiently diverse test distributions within a known domain. Our first finding is that training on a uniform distribution over this domain is optimal. We also interrogate practical remedies when uniform samples are unavailable by considering methods for mitigating non-uniformity through finetuning and rebalancing. Our theory provides a mathematical grounding for previous observations on the role of entropy and rebalancing for o.o.d. generalization and foundation model training. We also provide new empirical evidence across tasks involving o.o.d. shifts which illustrate the broad applicability of our perspective.
Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG
Venue: arXiv cs.CL 2024
Year: 2024

Retrieval-augmented generation (RAG) empowers large language models (LLMs) to utilize external knowledge sources. The increasing capacity of LLMs to process longer input sequences opens up avenues for providing more retrieved information, to potentially enhance the quality of generated outputs. It is plausible to assume that a larger retrieval set would contain more relevant information (higher recall), that might result in improved performance. However, our empirical findings demonstrate that for many long-context LLMs, the quality of generated output initially improves first, but then subsequently declines as the number of retrieved passages increases. This paper investigates this phenomenon, identifying the detrimental impact of retrieved "hard negatives" as a key contributor. To mitigate this and enhance the robustness of long-context LLM-based RAG, we propose both training-free and training-based approaches. We first showcase the effectiveness of retrieval reordering as a simple yet powerful training-free optimization. Furthermore, we explore training-based methods, specifically RAG-specific implicit LLM fine-tuning and RAG-oriented fine-tuning with intermediate reasoning, demonstrating their capacity for substantial performance gains. Finally, we conduct a systematic analysis of design choices for these training-based methods, including data distribution, retriever selection, and training context length.
Utilizing Lyapunov Exponents in designing deep neural networks
Venue: arXiv cs.LG 2024
Year: 2024

Training large deep neural networks is resource intensive. This study investigates whether Lyapunov exponents can accelerate this process by aiding in the selection of hyperparameters. To study this I formulate an optimization problem using neural networks with different activation functions in the hidden layers. By initializing model weights with different random seeds, I calculate the Lyapunov exponent while performing traditional gradient descent on these model weights. The findings demonstrate that variations in the learning rate can induce chaotic changes in model weights. I also show that activation functions with more negative Lyapunov exponents exhibit better convergence properties. Additionally, the study also demonstrates that Lyapunov exponents can be utilized to select effective initial model weights for deep neural networks, potentially enhancing the optimization process.
AIVIO: Closed-loop, Object-relative Navigation of UAVs with AI-aided Visual Inertial Odometry
Venue: arXiv cs.RO 2024
Year: 2024

Object-relative mobile robot navigation is essential for a variety of tasks, e.g. autonomous critical infrastructure inspection, but requires the capability to extract semantic information about the objects of interest from raw sensory data. While deep learning-based (DL) methods excel at inferring semantic object information from images, such as class and relative 6 degree of freedom (6-DoF) pose, they are computationally demanding and thus often not suitable for payload constrained mobile robots. In this letter we present a real-time capable unmanned aerial vehicle (UAV) system for object-relative, closed-loop navigation with a minimal sensor configuration consisting of an inertial measurement unit (IMU) and RGB camera. Utilizing a DL-based object pose estimator, solely trained on synthetic data and optimized for companion board deployment, the object-relative pose measurements are fused with the IMU data to perform object-relative localization. We conduct multiple real-world experiments to validate the performance of our system for the challenging use case of power pole inspection. An example closed-loop flight is presented in the supplementary video.
SpecTrack: Learned Multi-Rotation Tracking via Speckle Imaging
Venue: arXiv cs.ET 2024
Year: 2024

Precision pose detection is increasingly demanded in fields such as personal fabrication, Virtual Reality (VR), and robotics due to its critical role in ensuring accurate positioning information. However, conventional vision-based systems used in these systems often struggle with achieving high precision and accuracy, particularly when dealing with complex environments or fast-moving objects. To address these limitations, we investigate Laser Speckle Imaging (LSI), an emerging optical tracking method that offers promising potential for improving pose estimation accuracy. Specifically, our proposed LSI-Based Tracking (SpecTrack) leverages the captures from a lensless camera and a retro-reflector marker with a coded aperture to achieve multi-axis rotational pose estimation with high precision. Our extensive trials using our in-house built testbed have shown that SpecTrack achieves an accuracy of 0.31° (std=0.43°), significantly outperforming state-of-the-art approaches and improving accuracy up to 200%.
KwicKwocKwac, a tool for rapidly generating concordances and marking up a literary text
Venue: arXiv cs.DL 2024
Year: 2024

This paper introduces KwicKwocKwac 1.0 (KwicKK), a web application designed to enhance the annotation and enrichment of digital texts in the humanities. KwicKK provides a user-friendly interface that enables scholars and researchers to perform semi-automatic markup of textual documents, facilitating the identification of relevant entities such as people, organizations, and locations. Key functionalities include the visualization of annotated texts using KeyWord in Context (KWIC), KeyWord Out Of Context (KWOC), and KeyWord After Context (KWAC) methodologies, alongside automatic disambiguation of generic references and integration with Wikidata for Linked Open Data connections. The application supports metadata input and offers multiple download formats, promoting accessibility and ease of use. Developed primarily for the National Edition of Aldo Moro's works, KwicKK aims to lower the technical barriers for users while fostering deeper engagement with digital scholarly resources. The architecture leverages contemporary web technologies, ensuring scalability and reliability. Future developments will explore user experience enhancements, collaborative features, and integration of additional data sources.
"Diversity is Having the Diversity": Unpacking and Designing for Diversity in Applicant Selection
Venue: arXiv cs.HC 2024
Year: 2024

When selecting applicants for scholarships, universities, or jobs, practitioners often aim for a diverse cohort of qualified recipients. However, differing articulations, constructs, and notions of diversity prevents decision-makers from operationalising and progressing towards the diversity they all agree is needed. To understand this challenge of translation from values, to requirements, to decision support tools (DSTs), we conducted participatory design studies exploring professionals' varied perceptions of diversity and how to build for them. Our results suggest three definitions of diversity: bringing together different perspectives; ensuring representativeness of a base population; and contextualising applications, which we use to create the Diversity Triangle. We experience-prototyped DSTs reflecting each angle of the Diversity Triangle to enhance decision-making around diversity. We find that notions of diversity are highly diverse; efforts to design DSTs for diversity should start by working with organisations to distil 'diversity' into definitions and design requirements.
Gaussian-Based and Outside-the-Box Runtime Monitoring Join Forces
Venue: arXiv cs.LG 2024
Year: 2024

Since neural networks can make wrong predictions even with high confidence, monitoring their behavior at runtime is important, especially in safety-critical domains like autonomous driving. In this paper, we combine ideas from previous monitoring approaches based on observing the activation values of hidden neurons. In particular, we combine the Gaussian-based approach, which observes whether the current value of each monitored neuron is similar to typical values observed during training, and the Outside-the-Box monitor, which creates clusters of the acceptable activation values, and, thus, considers the correlations of the neurons' values. Our experiments evaluate the achieved improvement.