OpenRaiser/Intern-Atlas huggingface dataset:
Let your LLM generate a few tokens and you will reduce the need for retrieval
Venue: arXiv cs.CL 2024
Year: 2024

In this paper, we investigate how efficiently large language models (LLM) can be trained to check whether an answer is already stored in their parametric memory. We distill an LLM-as-a-judge to compute the IK (I Know) score. We found that this method is particularly beneficial in the context of retrieval-assisted augmented generation (RAG), with a respectable accuracy of 80%. It enables a significant reduction (more than 50%) in the number of search and reranking steps required for certain data sets. We have also introduced the IK score, which serves as a useful tool for characterising datasets by facilitating the classification task. Interestingly, through the inclusion of response tokens as input, our results suggest that only about 20,000 training samples are required to achieve good performance. The central element of this work is the use of a teacher model - the LLM as a judge - to generate training data. We also assess the robustness of the IK classifier by evaluating it with various types of teachers, including both string-based methods and LLMs, with the latter providing better results.
Efficient Avoidance of Ellipsoidal Obstacles with Model Predictive Control for Mobile Robots and Vehicles
Venue: arXiv cs.RO 2024
Year: 2024

In real-world applications of mobile robots, collision avoidance is of critical importance. Typically, global motion planning in constrained environments is addressed through high-level control schemes. However, additionally integrating local collision avoidance into robot motion control offers significant advantages. For instance, it reduces the reliance on heuristics and conservatism that can arise from a two-stage approach separating local collision avoidance and control. Moreover, using model predictive control (MPC), a robot's full potential can be harnessed by considering jointly local collision avoidance, the robot's dynamics, and actuation constraints. In this context, the present paper focuses on obstacle avoidance for wheeled mobile robots, where both the robot's and obstacles' occupied volumes are modeled as ellipsoids. To this end, a computationally efficient overlap test, that works for arbitrary ellipsoids, is conducted and novelly integrated into the MPC framework. We propose a particularly efficient implementation tailored to robots moving in the plane. The functionality of the proposed obstacle-avoiding MPC is demonstrated for two exemplary types of kinematics by means of simulations. A hardware experiment using a real-world wheeled mobile robot shows transferability to reality and real-time applicability. The general computational approach to ellipsoidal obstacle avoidance can also be applied to other robotic systems and vehicles as well as three-dimensional scenarios.
Enhancing Healthcare Recommendation Systems with a Multimodal LLMs-based MOE Architecture
Venue: arXiv cs.IR 2024
Year: 2024

With the increasing availability of multimodal data, many fields urgently require advanced architectures capable of effectively integrating these diverse data sources to address specific problems. This study proposes a hybrid recommendation model that combines the Mixture of Experts (MOE) framework with large language models to enhance the performance of recommendation systems in the healthcare domain. We built a small dataset for recommending healthy food based on patient descriptions and evaluated the model's performance on several key metrics, including Precision, Recall, NDCG, and MAP@5. The experimental results show that the hybrid model outperforms the baseline models, which use MOE or large language models individually, in terms of both accuracy and personalized recommendation effectiveness. The paper finds image data provided relatively limited improvement in the performance of the personalized recommendation system, particularly in addressing the cold start problem. Then, the issue of reclassification of images also affected the recommendation results, especially when dealing with low-quality images or changes in the appearance of items, leading to suboptimal performance. The findings provide valuable insights into the development of powerful, scalable, and high-performance recommendation systems, advancing the application of personalized recommendation technologies in real-world domains such as healthcare.
The Role of Natural Language Processing Tasks in Automatic Literary Character Network Construction
Venue: arXiv cs.CL 2024
Year: 2024

The automatic extraction of character networks from literary texts is generally carried out using natural language processing (NLP) cascading pipelines. While this approach is widespread, no study exists on the impact of low-level NLP tasks on their performance. In this article, we conduct such a study on a literary dataset, focusing on the role of named entity recognition (NER) and coreference resolution when extracting co-occurrence networks. To highlight the impact of these tasks' performance, we start with gold-standard annotations, progressively add uniformly distributed errors, and observe their impact in terms of character network quality. We demonstrate that NER performance depends on the tested novel and strongly affects character detection. We also show that NER-detected mentions alone miss a lot of character co-occurrences, and that coreference resolution is needed to prevent this. Finally, we present comparison points with 2 methods based on large language models (LLMs), including a fully end-to-end one, and show that these models are outperformed by traditional NLP pipelines in terms of recall.
Self-Adaptive Paraphrasing and Preference Learning for Improved Claim Verifiability
Venue: arXiv cs.CL 2024
Year: 2024

In fact-checking, structure and phrasing of claims critically influence a model's ability to predict verdicts accurately. Social media content in particular rarely serves as optimal input for verification systems, which necessitates pre-processing to extract the claim from noisy context before fact checking. Prior work suggests extracting a claim representation that humans find to be checkworthy and verifiable. This has two limitations: (1) the format may not be optimal for a fact-checking model, and (2), it requires annotated data to learn the extraction task from. We address both issues and propose a method to extract claims that is not reliant on labeled training data. Instead, our self-adaptive approach only requires a black-box fact checking model and a generative language model (LM). Given a tweet, we iteratively optimize the LM to generate a claim paraphrase that increases the performance of a fact checking model. By learning from preference pairs, we align the LM to the fact checker using direct preference optimization. We show that this novel setup extracts a claim paraphrase that is more verifiable than their original social media formulations, and is on par with competitive baselines. For refuted claims, our method consistently outperforms all baselines.
C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness
Venue: arXiv cs.CL 2024
Year: 2024

Generating Chain-of-Thought (CoT) before deriving the answer can effectively improve the reasoning capabilities of large language models (LLMs) and significantly improve the accuracy of the generated answer. However, in most cases, the length of the generated CoT is much longer than the desired final answer, which results in additional decoding costs. Furthermore, existing research has discovered that shortening the reasoning steps in CoT, even while preserving the key information, diminishes LLMs' abilities. These phenomena make it difficult to use LLMs and CoT in many real-world applications that only require the final answer and are sensitive to latency, such as search and recommendation. To reduce the costs of model decoding and shorten the length of the generated CoT, this paper presents $\textbf{C}$onditioned $\textbf{C}$ompressed $\textbf{C}$hain-of-$\textbf{T}$hought (C3oT), a CoT compression framework that involves a compressor to compress an original longer CoT into a shorter CoT while maintaining key information and interpretability, a conditioned training method to train LLMs with both longer CoT and shorter CoT simultaneously to learn the corresponding relationships between them, and a conditioned inference method to gain the reasoning ability learned from longer CoT by generating shorter CoT. We conduct experiments over four datasets from arithmetic and commonsense scenarios, showing that the proposed method is capable of compressing the length of generated CoT by up to more than 50% without compromising its effectiveness.
LLM-DaaS: LLM-driven Drone-as-a-Service Operations from Text User Requests
Venue: arXiv cs.AI 2024
Year: 2024

We propose LLM-DaaS, a novel Drone-as-a-Service (DaaS) framework that leverages Large Language Models (LLMs) to transform free-text user requests into structured, actionable DaaS operation tasks. Our approach addresses the key challenge of interpreting and structuring natural language input to automate drone service operations under uncertain conditions. The system is composed of three main components: free-text request processing, structured request generation, and dynamic DaaS selection and composition. First, we fine-tune different LLM models such as Phi-3.5, LLaMA-3.2 7b and Gemma 2b on a dataset of text user requests mapped to structured DaaS requests. Users interact with our model in a free conversational style, discussing package delivery requests, while the fine-tuned LLM extracts DaaS metadata such as delivery time, source and destination locations, and package weight. The DaaS service selection model is designed to select the best available drone capable of delivering the requested package from the delivery point to the nearest optimal destination. Additionally, the DaaS composition model composes a service from a set of the best available drones to deliver the package from the source to the final destination. Second, the system integrates real-time weather data to optimize drone route planning and scheduling, ensuring safe and efficient operations. Simulations demonstrate the system's ability to significantly improve task accuracy, operational efficiency, and establish LLM-DaaS as a robust solution for DaaS operations in uncertain environments.
NEST: A Neuromodulated Small-world Hypergraph Trajectory Prediction Model for Autonomous Driving
Venue: arXiv cs.RO 2024
Year: 2024

Accurate trajectory prediction is essential for the safety and efficiency of autonomous driving. Traditional models often struggle with real-time processing, capturing non-linearity and uncertainty in traffic environments, efficiency in dense traffic, and modeling temporal dynamics of interactions. We introduce NEST (Neuromodulated Small-world Hypergraph Trajectory Prediction), a novel framework that integrates Small-world Networks and hypergraphs for superior interaction modeling and prediction accuracy. This integration enables the capture of both local and extended vehicle interactions, while the Neuromodulator component adapts dynamically to changing traffic conditions. We validate the NEST model on several real-world datasets, including nuScenes, MoCAD, and HighD. The results consistently demonstrate that NEST outperforms existing methods in various traffic scenarios, showcasing its exceptional generalization capability, efficiency, and temporal foresight. Our comprehensive evaluation illustrates that NEST significantly improves the reliability and operational efficiency of autonomous driving systems, making it a robust solution for trajectory prediction in complex traffic environments.
Multimodal LLM for Intelligent Transportation Systems
Venue: arXiv cs.LG 2024
Year: 2024

In the evolving landscape of transportation systems, integrating Large Language Models (LLMs) offers a promising frontier for advancing intelligent decision-making across various applications. This paper introduces a novel 3-dimensional framework that encapsulates the intersection of applications, machine learning methodologies, and hardware devices, particularly emphasizing the role of LLMs. Instead of using multiple machine learning algorithms, our framework uses a single, data-centric LLM architecture that can analyze time series, images, and videos. We explore how LLMs can enhance data interpretation and decision-making in transportation. We apply this LLM framework to different sensor datasets, including time-series data and visual data from sources like Oxford Radar RobotCar, D-Behavior (D-Set), nuScenes by Motional, and Comma2k19. The goal is to streamline data processing workflows, reduce the complexity of deploying multiple models, and make intelligent transportation systems more efficient and accurate. The study was conducted using state-of-the-art hardware, leveraging the computational power of AMD RTX 3060 GPUs and Intel i9-12900 processors. The experimental results demonstrate that our framework achieves an average accuracy of 91.33\% across these datasets, with the highest accuracy observed in time-series data (92.7\%), showcasing the model's proficiency in handling sequential information essential for tasks such as motion planning and predictive maintenance. Through our exploration, we demonstrate the versatility and efficacy of LLMs in handling multimodal data within the transportation sector, ultimately providing insights into their application in real-world scenarios. Our findings align with the broader conference themes, highlighting the transformative potential of LLMs in advancing transportation technologies.
Testing Medical Rules Web Services in Practice
Venue: arXiv cs.SE 2024
Year: 2024

The Cancer Registry of Norway (CRN) collects and processes cancer-related data for patients in Norway. For this, it employs a sociotechnical software system that evolves with changing requirements and medical standards. The current practice is to manually test CRN's system to prevent faults and ensure its dependability. This paper focuses on automatically testing GURI, the CRN's medical rule engine, using a system-level testing tool, EvoMaster, in both its black-box and white-box modes, and a novel CRN-specific EvoMaster-based tool, EvoGURI. We empirically evaluate the tools' effectiveness regarding code coverage, errors found, domain-specific rule coverage, and ability to identify artificial faults ten versions of GURI. Our results show that all the tools achieve similar code coverage and identified a similar number of errors. For rule coverage, EvoGURI and EvoMaster's black-box mode produce test suites that cover the highest number of rules with Pass, Fail, and Warning results. The test suites of EvoGURI and two EvoMaster white-box tools identify the most faults in a mutation testing experiment. Based on our findings, we recommend using EvoGURI in CRN's current practice. Finally, we present key takeaways and outline open research questions for the research community.
Beyond Graph Convolution: Multimodal Recommendation with Topology-aware MLPs
Venue: arXiv cs.IR 2024
Year: 2024

Given the large volume of side information from different modalities, multimodal recommender systems have become increasingly vital, as they exploit richer semantic information beyond user-item interactions. Recent works highlight that leveraging Graph Convolutional Networks (GCNs) to explicitly model multimodal item-item relations can significantly enhance recommendation performance. However, due to the inherent over-smoothing issue of GCNs, existing models benefit only from shallow GCNs with limited representation power. This drawback is especially pronounced when facing complex and high-dimensional patterns such as multimodal data, as it requires large-capacity models to accommodate complicated correlations. To this end, in this paper, we investigate bypassing GCNs when modeling multimodal item-item relationship. More specifically, we propose a Topology-aware Multi-Layer Perceptron (TMLP), which uses MLPs instead of GCNs to model the relationships between items. TMLP enhances MLPs with topological pruning to denoise item-item relations and intra (inter)-modality learning to integrate higher-order modality correlations. Extensive experiments on three real-world datasets verify TMLP's superiority over nine baselines. We also find that by discarding the internal message passing in GCNs, which is sensitive to node connections, TMLP achieves significant improvements in both training efficiency and robustness against existing models.
The Black Ninjas and the Sniper: On Robustness of Population Protocols
Venue: arXiv cs.FL 2024
Year: 2024

Population protocols are a model of distributed computation in which an arbitrary number of indistinguishable finite-state agents interact in pairs to decide some property of their initial configuration. We investigate the behaviour of population protocols under adversarial faults that cause agents to silently crash and no longer interact with other agents. As a starting point, we consider the property ``the number of agents exceeds a given threshold $t$'', represented by the predicate $x \geq t$, and show that the standard protocol for $x \geq t$ is very fragile: one single crash in a computation with $x:=2t-1$ agents can already cause the protocol to answer incorrectly that $x \geq t$ does not hold. However, a slightly less known protocol is robust: for any number $t' \geq t$ of agents, at least $t' - t+1$ crashes must occur for the protocol to answer that the property does not hold. We formally define robustness for arbitrary population protocols, and investigate the question whether every predicate computable by population protocols has a robust protocol. Angluin et al. proved in 2007 that population protocols decide exactly the Presburger predicates, which can be represented as Boolean combinations of threshold predicates of the form $\sum_{i=1}^n a_i \cdot x_i \geq t$ for $a_1,...,a_n, t \in \mathbb{Z}$ and modulo prdicates of the form $\sum_{i=1}^n a_i \cdot x_i \bmod m \geq t $ for $a_1, \ldots, a_n, m, t \in \mathbb{N}$. We design robust protocols for all threshold and modulo predicates. We also show that, unfortunately, the techniques in the literature that construct a protocol for a Boolean combination of predicates given protocols for the conjuncts do not preserve robustness. So the question remains open.
Uncovering Student Engagement Patterns in Moodle with Interpretable Machine Learning
Venue: arXiv cs.CY 2024
Year: 2024

Understanding and enhancing student engagement through digital platforms is critical in higher education. This study introduces a methodology for quantifying engagement across an entire module using virtual learning environment (VLE) activity log data. Using study session frequency, immediacy, and diversity, we create a cumulative engagement metric and model it against weekly VLE interactions with resources to identify critical periods and resources predictive of student engagement. In a case study of a computing module at University College London's Department of Statistical Science, we further examine how delivery methods (online, hybrid, in-person) impact student behaviour. Across nine regression models, we validate the consistency of the random forest model and highlight the interpretive strengths of generalised additive models for analysing engagement patterns. Results show weekly VLE clicks as reliable engagement predictors, with early weeks and the first assessment period being key. However, the impact of delivery methods on engagement is inconclusive due to inconsistencies across models. These findings support early intervention strategies to assist students at risk of disengagement. This work contributes to learning analytics research by proposing a refined VLE-based engagement metric and advancing data-driven teaching strategies in higher education.
Event-based Motion Deblurring via Multi-Temporal Granularity Fusion
Venue: arXiv cs.CV 2024
Year: 2024

Conventional frame-based cameras inevitably produce blurry effects due to motion occurring during the exposure time. Event camera, a bio-inspired sensor offering continuous visual information could enhance the deblurring performance. Effectively utilizing the high-temporal-resolution event data is crucial for extracting precise motion information and enhancing deblurring performance. However, existing event-based image deblurring methods usually utilize voxel-based event representations, losing the fine-grained temporal details that are mathematically essential for fast motion deblurring. In this paper, we first introduce point cloud-based event representation into the image deblurring task and propose a Multi-Temporal Granularity Network (MTGNet). It combines the spatially dense but temporally coarse-grained voxel-based event representation and the temporally fine-grained but spatially sparse point cloud-based event. To seamlessly integrate such complementary representations, we design a Fine-grained Point Branch. An Aggregation and Mapping Module (AMM) is proposed to align the low-level point-based features with frame-based features and an Adaptive Feature Diffusion Module (AFDM) is designed to manage the resolution discrepancies between event data and image data by enriching the sparse point feature. Extensive subjective and objective evaluations demonstrate that our method outperforms current state-of-the-art approaches on both synthetic and real-world datasets.
Using Instruction-Tuned Large Language Models to Identify Indicators of Vulnerability in Police Incident Narratives
Venue: arXiv cs.CL 2024
Year: 2024

Objectives: Compare qualitative coding of instruction tuned large language models (IT-LLMs) against human coders in classifying the presence or absence of vulnerability in routinely collected unstructured text that describes police-public interactions. Evaluate potential bias in IT-LLM codings. Methods: Analyzing publicly available text narratives of police-public interactions recorded by Boston Police Department, we provide humans and IT-LLMs with qualitative labelling codebooks and compare labels generated by both, seeking to identify situations associated with (i) mental ill health; (ii) substance misuse; (iii) alcohol dependence; and (iv) homelessness. We explore multiple prompting strategies and model sizes, and the variability of labels generated by repeated prompts. Additionally, to explore model bias, we utilize counterfactual methods to assess the impact of two protected characteristics - race and gender - on IT-LLM classification. Results: Results demonstrate that IT-LLMs can effectively support human qualitative coding of police incident narratives. While there is some disagreement between LLM and human generated labels, IT-LLMs are highly effective at screening narratives where no vulnerabilities are present, potentially vastly reducing the requirement for human coding. Counterfactual analyses demonstrate that manipulations to both gender and race of individuals described in narratives have very limited effects on IT-LLM classifications beyond those expected by chance. Conclusions: IT-LLMs offer effective means to augment human qualitative coding in a way that requires much lower levels of resource to analyze large unstructured datasets. Moreover, they encourage specificity in qualitative coding, promote transparency, and provide the opportunity for more standardized, replicable approaches to analyzing large free-text police data sources.