OpenRaiser/Intern-Atlas huggingface dataset:
Efficient algorithms for computing bisimulations for nondeterministic fuzzy transition systems
Venue: arXiv cs.DS 2024
Year: 2024

Fuzzy transition systems offer a robust framework for modeling and analyzing systems with inherent uncertainties and imprecision, which are prevalent in real-world scenarios. As their extension, nondeterministic fuzzy transition systems (NFTSs) have been studied in a considerable number of works. Wu et al. (2018) provided an algorithm for computing the greatest crisp bisimulation of a finite NFTS $\mathcal{S} = \langle S, A, δ\rangle$, with a time complexity of order $O(|S|^4 \cdot |δ|^2)$ under the assumption that $|δ| \geq |S|$. Qiao {\em et al.} (2023) provided an algorithm for computing the greatest fuzzy bisimulation of a finite NFTS $\mathcal{S}$ under the Gödel semantics, with a time complexity of order $O(|S|^4 \cdot |δ|^2 \cdot l)$ under the assumption that $|δ| \geq |S|$, where $l$ is the number of fuzzy values used in $\mathcal{S}$ plus 1. In this work, we provide efficient algorithms for computing the partition corresponding to the greatest crisp bisimulation of a finite NFTS $\mathcal{S}$, as well as the compact fuzzy partition corresponding to the greatest fuzzy bisimulation of $\mathcal{S}$ under the Gödel semantics. Their time complexities are of the order $O((size(δ) \log{l} + |S|) \log{(|S| + |δ|)})$, where $l$ is the number of fuzzy values used in $\mathcal{S}$ plus 2. When $|δ| \geq |S|$, this order is within $O(|S| \cdot |δ| \cdot \log^2{|δ|})$. The reduction of time complexity from $O(|S|^4 \cdot |δ|^2)$ and $O(|S|^4 \cdot |δ|^2 \cdot l)$ to $O(|S| \cdot |δ| \cdot \log^2{|δ|})$ is a significant contribution of this work. In addition, we introduce nondeterministic fuzzy labeled transition systems, which extend NFTSs with fuzzy state labels, and we define and provide results on simulations and bisimulations between them.
Deep Domain Specialisation for single-model multi-domain learning to rank
Venue: arXiv cs.IR 2024
Year: 2024

Information Retrieval (IR) practitioners often train separate ranking models for different domains (geographic regions, languages, stores, websites,...) as it is believed that exclusively training on in-domain data yields the best performance when sufficient data is available. Despite their performance gains, training multiple models comes at a higher cost to train, maintain and update compared to having only a single model responsible for all domains. Our work explores consolidated ranking models that serve multiple domains. Specifically, we propose a novel architecture of Deep Domain Specialisation (DDS) to consolidate multiple domains into a single model. We compare our proposal against Deep Domain Adaptation (DDA) and a set of baseline for multi-domain models. In our experiments, DDS performed the best overall while requiring fewer parameters per domain as other baselines. We show the efficacy of our method both with offline experimentation and on a large-scale online experiment on Amazon customer traffic.
Physics-Informed Holomorphic Neural Networks (PIHNNs): Solving Linear Elasticity Problems
Venue: arXiv cs.CE 2024
Year: 2024

We propose physics-informed holomorphic neural networks (PIHNNs) as a method to solve boundary value problems where the solution can be represented via holomorphic functions. Specifically, we consider the case of plane linear elasticity and, by leveraging the Kolosov-Muskhelishvili representation of the solution in terms of holomorphic potentials, we train a complex-valued neural network to fulfill stress and displacement boundary conditions while automatically satisfying the governing equations. This is achieved by designing the network to return only approximations that inherently satisfy the Cauchy-Riemann conditions through specific choices of layers and activation functions. To ensure generality, we provide a universal approximation theorem guaranteeing that, under basic assumptions, the proposed holomorphic neural networks can approximate any holomorphic function. Furthermore, we suggest a new tailored weight initialization technique to mitigate the issue of vanishing/exploding gradients. Compared to the standard PINN approach, noteworthy benefits of the proposed method for the linear elasticity problem include a more efficient training, as evaluations are needed solely on the boundary of the domain, lower memory requirements, due to the reduced number of training points, and $C^\infty$ regularity of the learned solution. Several benchmark examples are used to verify the correctness of the obtained PIHNN approximations, the substantial benefits over traditional PINNs, and the possibility to deal with non-trivial, multiply-connected geometries via a domain-decomposition strategy.
$\text{Memory}^3$: Language Modeling with Explicit Memory
Venue: arXiv cs.CL 2024
Year: 2024

The training and inference of large language models (LLMs) are together a costly process that transports knowledge from raw data to meaningful computation. Inspired by the memory hierarchy of the human brain, we reduce this cost by equipping LLMs with explicit memory, a memory format cheaper than model parameters and text retrieval-augmented generation (RAG). Conceptually, with most of its knowledge externalized to explicit memories, the LLM can enjoy a smaller parameter size, training cost, and inference cost, all proportional to the amount of remaining "abstract knowledge". As a preliminary proof of concept, we train from scratch a 2.4B LLM, which achieves better performance than much larger LLMs as well as RAG models, and maintains higher decoding speed than RAG. The model is named $\text{Memory}^3$, since explicit memory is the third form of memory in LLMs after implicit memory (model parameters) and working memory (context key-values). We introduce a memory circuitry theory to support the externalization of knowledge, and present novel techniques including a memory sparsification mechanism that makes storage tractable and a two-stage pretraining scheme that facilitates memory formation.
Deep Learning Approach for Enhanced Transferability and Learning Capacity in Tool Wear Estimation
Venue: arXiv cs.LG 2024
Year: 2024

As an integral part of contemporary manufacturing, monitoring systems obtain valuable information during machining to oversee the condition of both the process and the machine. Recently, diverse algorithms have been employed to detect tool wear using single or multiple sources of measurements. In this study, a deep learning approach is proposed for estimating tool wear, considering cutting parameters. The model's accuracy and transferability in tool wear estimation were assessed with milling experiments conducted under varying cutting parameters. The results indicate that the proposed method outperforms conventional methods in terms of both transferability and rapid learning capabilities.
The African Woman is Rhythmic and Soulful: An Investigation of Implicit Biases in LLM Open-ended Text Generation
Venue: arXiv cs.CL 2024
Year: 2024

This paper investigates the subtle and often concealed biases present in Large Language Models (LLMs), focusing on implicit biases that may remain despite passing explicit bias tests. Implicit biases are significant because they influence the decisions made by these systems, potentially perpetuating stereotypes and discrimination, even when LLMs appear to function fairly. Traditionally, explicit bias tests or embedding-based methods are employed to detect bias, but these approaches can overlook more nuanced, implicit forms of bias. To address this, we introduce two novel psychological-inspired methodologies: the LLM Implicit Association Test (IAT) Bias and the LLM Decision Bias, designed to reveal and measure implicit biases through prompt-based and decision-making tasks. Additionally, open-ended generation tasks with thematic analysis of word generations and storytelling provide qualitative insights into the model's behavior. Our findings demonstrate that the LLM IAT Bias correlates with traditional methods and more effectively predicts downstream behaviors, as measured by the LLM Decision Bias, offering a more comprehensive framework for detecting subtle biases in AI systems. This research advances the field of AI ethics by proposing new methods to continually assess and mitigate biases in LLMs, highlighting the importance of qualitative and decision-focused evaluations to address challenges that previous approaches have not fully captured.
First Place Solution of 2023 Global Artificial Intelligence Technology Innovation Competition Track 1
Venue: arXiv cs.CL 2024
Year: 2024

In this paper, we present our champion solution to the Global Artificial Intelligence Technology Innovation Competition Track 1: Medical Imaging Diagnosis Report Generation. We select CPT-BASE as our base model for the text generation task. During the pre-training stage, we delete the mask language modeling task of CPT-BASE and instead reconstruct the vocabulary, adopting a span mask strategy and gradually increasing the number of masking ratios to perform the denoising auto-encoder pre-training task. In the fine-tuning stage, we design iterative retrieval augmentation and noise-aware similarity bucket prompt strategies. The retrieval augmentation constructs a mini-knowledge base, enriching the input information of the model, while the similarity bucket further perceives the noise information within the mini-knowledge base, guiding the model to generate higher-quality diagnostic reports based on the similarity prompts. Surprisingly, our single model has achieved a score of 2.321 on leaderboard A, and the multiple model fusion scores are 2.362 and 2.320 on the A and B leaderboards respectively, securing first place in the rankings.
Quaternion-based Adaptive Backstepping Fast Terminal Sliding Mode Control for Quadrotor UAVs with Finite Time Convergence
Venue: arXiv cs.RO 2024
Year: 2024

This paper proposes a novel quaternion-based approach for tracking the translation (position and linear velocity) and rotation (attitude and angular velocity) trajectories of underactuated Unmanned Aerial Vehicles (UAVs). Quadrotor UAVs are challenging regarding accuracy, singularity, and uncertainties issues. Controllers designed based on unit-quaternion are singularity-free for attitude representation compared to other methods (e.g., Euler angles), which fail to represent the vehicle's attitude at multiple orientations. Quaternion-based Adaptive Backstepping Control (ABC) and Adaptive Fast Terminal Sliding Mode Control (AFTSMC) are proposed to address a set of challenging problems. A quaternion-based ABC, a superior recursive approach, is proposed to generate the necessary thrust handling unknown uncertainties and UAV translation trajectory tracking. Next, a quaternion-based AFTSMC is developed to overcome parametric uncertainties, avoid singularity, and ensure fast convergence in a finite time. Moreover, the proposed AFTSMC is able to significantly minimize control signal chattering, which is the main reason for actuator failure and provide smooth and accurate rotational control input. To ensure the robustness of the proposed approach, the designed control algorithms have been validated considering unknown time-variant parametric uncertainties and significant initialization errors. The proposed techniques has been compared to state-of-the-art control technique. Keywords: Adaptive Backstepping Control (ABC), Adaptive Fast Terminal Sliding Mode Control (AFTSMC), Unit-quaternion, Unmanned Aerial Vehicles, Singularity Free, Pose Control
Small Aerial Target Detection for Airborne Infrared Detection Systems using LightGBM and Trajectory Constraints
Venue: arXiv cs.CV 2024
Year: 2024

Factors, such as rapid relative motion, clutter background, etc., make robust small aerial target detection for airborne infrared detection systems a challenge. Existing methods are facing difficulties when dealing with such cases. We consider that a continuous and smooth trajectory is critical in boosting small infrared aerial target detection performance. A simple and effective small aerial target detection method for airborne infrared detection system using light gradient boosting model (LightGBM) and trajectory constraints is proposed in this article. First, we simply formulate target candidate detection as a binary classification problem. Target candidates in every individual frame are detected via interesting pixel detection and a trained LightGBM model. Then, the local smoothness and global continuous characteristic of the target trajectory are modeled as short-strict and long-loose constraints. The trajectory constraints are used efficiently for detecting the true small infrared aerial targets from numerous target candidates. Experiments on public datasets demonstrate that the proposed method performs better than other existing methods. Furthermore, a public dataset for small aerial target detection in airborne infrared detection systems is constructed. To the best of our knowledge, this dataset has the largest data scale and richest scene types within this field.
Human-Robot Mutual Learning through Affective-Linguistic Interaction and Differential Outcomes Training [Pre-Print]
Venue: arXiv cs.RO 2024
Year: 2024

Owing to the recent success of Large Language Models, Modern A.I has been much focused on linguistic interactions with humans but less focused on non-linguistic forms of communication between man and machine. In the present paper, we test how affective-linguistic communication, in combination with differential outcomes training, affects mutual learning in a human-robot context. Taking inspiration from child-caregiver dynamics, our human-robot interaction setup consists of a (simulated) robot attempting to learn how best to communicate internal, homeostatically-controlled needs; while a human "caregiver" attempts to learn the correct object to satisfy the robot's present communicated need. We studied the effects of i) human training type, and ii) robot reinforcement learning type, to assess mutual learning terminal accuracy and rate of learning (as measured by the average reward achieved by the robot). Our results find mutual learning between a human and a robot is significantly improved with Differential Outcomes Training (DOT) compared to Non-DOT (control) conditions. We find further improvements when the robot uses an exploration-exploitation policy selection, compared to purely exploitation policy selection. These findings have implications for utilizing socially assistive robots (SAR) in therapeutic contexts, e.g. for cognitive interventions, and educational applications.
Bridging Smoothness and Approximation: Theoretical Insights into Over-Smoothing in Graph Neural Networks
Venue: arXiv cs.LG 2024
Year: 2024

In this paper, we explore the approximation theory of functions defined on graphs. Our study builds upon the approximation results derived from the $K$-functional. We establish a theoretical framework to assess the lower bounds of approximation for target functions using Graph Convolutional Networks (GCNs) and examine the over-smoothing phenomenon commonly observed in these networks. Initially, we introduce the concept of a $K$-functional on graphs, establishing its equivalence to the modulus of smoothness. We then analyze a typical type of GCN to demonstrate how the high-frequency energy of the output decays, an indicator of over-smoothing. This analysis provides theoretical insights into the nature of over-smoothing within GCNs. Furthermore, we establish a lower bound for the approximation of target functions by GCNs, which is governed by the modulus of smoothness of these functions. This finding offers a new perspective on the approximation capabilities of GCNs. In our numerical experiments, we analyze several widely applied GCNs and observe the phenomenon of energy decay. These observations corroborate our theoretical results on exponential decay order.
Data After Death: Australian User Preferences and Future Solutions to Protect Posthumous User Data
Venue: arXiv cs.CY 2024
Year: 2024

The digital footprints of today's internet-active individuals are a testament to their lives, and have the potential to become digital legacies once they pass on. Future descendants of those alive today will greatly appreciate the unprecedented insight into the lives of their long since deceased ancestors, but this can only occur if today we have a process for data preservation and handover after death. Many prominent online platforms offer nebulous or altogether absent policies regarding posthumous data handling, and despite recent advances it is currently unclear who the average Australian would like their data to be managed after their death (i.e., social media platforms, a trusted individual, or another digital executor). While at present the management of deceased accounts is largely performed by the platform (e.g., Facebook), it is conceivable that many Australians may not trust such platforms to do so with integrity. This study aims to further the academic conversation around posthumous data by delving deeper into the preferences of the Australian Public regarding the management of their data after death, ultimately to inform future development of research programs and industry solutions. A survey of 1020 Australians revealed that most desired a level of control over how their data is managed after death. Australians currently prefer to entrust the management of their data to a trusted close individual or third party software that they can administrate themselves. As expected, social media companies ranked low regarding both trust and convenience to manage data after death. Future research focus should be to conceptualise and develop a third-party solution that enables these preferences to be realised. Such a solution could interface with the major online vendors (social media, cloud hosting etc.) to action the deceased's will.
Formal Verification of Deep Neural Networks for Object Detection
Venue: arXiv cs.CV 2024
Year: 2024

Deep neural networks (DNNs) are widely used in real-world applications, yet they remain vulnerable to errors and adversarial attacks. Formal verification offers a systematic approach to identify and mitigate these vulnerabilities, enhancing model robustness and reliability. While most existing verification methods focus on image classification models, this work extends formal verification to the more complex domain of emph{object detection} models. We propose a formulation for verifying the robustness of such models and demonstrate how state-of-the-art verification tools, originally developed for classification, can be adapted for this purpose. Our experiments, conducted on various datasets and networks, highlight the ability of formal verification to uncover vulnerabilities in object detection models, underscoring the need to extend verification efforts to this domain. This work lays the foundation for further research into formal verification across a broader range of computer vision applications.
ToCoAD: Two-Stage Contrastive Learning for Industrial Anomaly Detection
Venue: arXiv cs.CV 2024
Year: 2024

Current unsupervised anomaly detection approaches perform well on public datasets but struggle with specific anomaly types due to the domain gap between pre-trained feature extractors and target-specific domains. To tackle this issue, this paper presents a two-stage training strategy, called \textbf{ToCoAD}. In the first stage, a discriminative network is trained by using synthetic anomalies in a self-supervised learning manner. This network is then utilized in the second stage to provide a negative feature guide, aiding in the training of the feature extractor through bootstrap contrastive learning. This approach enables the model to progressively learn the distribution of anomalies specific to industrial datasets, effectively enhancing its generalizability to various types of anomalies. Extensive experiments are conducted to demonstrate the effectiveness of our proposed two-stage training strategy, and our model produces competitive performance, achieving pixel-level AUROC scores of 98.21\%, 98.43\% and 97.70\% on MVTec AD, VisA and BTAD respectively.
Evaluating Knowledge-based Cross-lingual Inconsistency in Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

This paper investigates the cross-lingual inconsistencies observed in Large Language Models (LLMs), such as ChatGPT, Llama, and Baichuan, which have shown exceptional performance in various Natural Language Processing (NLP) tasks. Despite their successes, these models often exhibit significant inconsistencies when processing the same concepts across different languages. This study focuses on three primary questions: the existence of cross-lingual inconsistencies in LLMs, the specific aspects in which these inconsistencies manifest, and the correlation between cross-lingual consistency and multilingual capabilities of LLMs.To address these questions, we propose an innovative evaluation method for Cross-lingual Semantic Consistency (xSC) using the LaBSE model. We further introduce metrics for Cross-lingual Accuracy Consistency (xAC) and Cross-lingual Timeliness Consistency (xTC) to comprehensively assess the models' performance regarding semantic, accuracy, and timeliness inconsistencies. By harmonizing these metrics, we provide a holistic measurement of LLMs' cross-lingual consistency. Our findings aim to enhance the understanding and improvement of multilingual capabilities and interpretability in LLMs, contributing to the development of more robust and reliable multilingual language models.
UAV Trajectory Planning with Path Processing
Venue: arXiv cs.RO 2024
Year: 2024

This paper examines the influence of initial guesses on trajectory planning for Unmanned Aerial Vehicles (UAVs) formulated in terms of Optimal Control Problem (OCP). The OCP is solved numerically using the Pseudospectral collocation method. Our approach leverages a path identified through Lazy Theta* and incorporates known constraints and a model of the UAV's behavior for the initial guess. Our findings indicate that a suitable initial guess has a beneficial influence on the planned trajectory. They also suggest promising directions for future research.
Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapters
Venue: arXiv cs.CL 2024
Year: 2024

This paper explores the integration of graph knowledge from linguistic ontologies into multilingual Large Language Models (LLMs) using adapters to improve performance for low-resource languages (LRLs) in sentiment analysis (SA) and named entity recognition (NER). Building upon successful parameter-efficient fine-tuning techniques, such as K-ADAPTER and MAD-X, we propose a similar approach for incorporating knowledge from multilingual graphs, connecting concepts in various languages with each other through linguistic relationships, into multilingual LLMs for LRLs. Specifically, we focus on eight LRLs -- Maltese, Bulgarian, Indonesian, Nepali, Javanese, Uyghur, Tibetan, and Sinhala -- and employ language-specific adapters fine-tuned on data extracted from the language-specific section of ConceptNet, aiming to enable knowledge transfer across the languages covered by the knowledge graph. We compare various fine-tuning objectives, including standard Masked Language Modeling (MLM), MLM with full-word masking, and MLM with targeted masking, to analyse their effectiveness in learning and integrating the extracted graph data. Through empirical evaluation on language-specific tasks, we assess how structured graph knowledge affects the performance of multilingual LLMs for LRLs in SA and NER, providing insights into the potential benefits of adapting language models for low-resource scenarios.