OpenRaiser/Intern-Atlas huggingface dataset:
Will LLMs Replace the Encoder-Only Models in Temporal Relation Classification?
Venue: arXiv cs.CL 2024
Year: 2024

The automatic detection of temporal relations among events has been mainly investigated with encoder-only models such as RoBERTa. Large Language Models (LLM) have recently shown promising performance in temporal reasoning tasks such as temporal question answering. Nevertheless, recent studies have tested the LLMs' performance in detecting temporal relations of closed-source models only, limiting the interpretability of those results. In this work, we investigate LLMs' performance and decision process in the Temporal Relation Classification task. First, we assess the performance of seven open and closed-sourced LLMs experimenting with in-context learning and lightweight fine-tuning approaches. Results show that LLMs with in-context learning significantly underperform smaller encoder-only models based on RoBERTa. Then, we delve into the possible reasons for this gap by applying explainable methods. The outcome suggests a limitation of LLMs in this task due to their autoregressive nature, which causes them to focus only on the last part of the sequence. Additionally, we evaluate the word embeddings of these two models to better understand their pre-training differences. The code and the fine-tuned models can be found respectively on GitHub.
Structuring data analysis projects in the Open Science era with Kerblam!
Venue: arXiv cs.HC 2024
Year: 2024

Structuring data analysis projects, that is, defining the layout of files and folders needed to analyze data using existing tools and novel code, largely follows personal preferences. In this work, we look at the structure of several data analysis project templates and find little structural overlap. We highlight the parts that are similar between them, and propose guiding principles to keep in mind when one wishes to create a new data analysis project. Finally, we present Kerblam!, a project management tool that can expedite project data management, execution of workflow managers, and sharing of the resulting workflow and analysis outputs. We hope that, by following these principles and using Kerblam!, the landscape of data analysis projects can become more transparent, understandable, and ultimately useful to the wider community.
A Method to Generate Multi-interval Pairwise Compatibility Graphs
Venue: arXiv cs.DM 2024
Year: 2024

Reconstruction of evolutionary relationships between species is an important topic in the field of computational biology. Pairwise compatibility graphs (PCGs) are used to model such relationships. A graph is a PCG if its edges can be represented by the distance between the leaves of an edge-weighted tree within a fixed interval. If the number of intervals is more than one, then the graph with such a tree representation is called a multi-interval PCG. The aim of this paper is to generate all multi-interval PCGs with a given number of vertices. For this purpose, we propose a method to generate almost all multi-interval PCGs corresponding to a given tree by randomly assigning edge weights and selecting typical intervals. To reduce the exponential tree search space, we theoretically prove that for each multi-interval PCG there exists a tree whose internal vertices have degree exactly three, and developed an algorithm to enumerate such trees. The proposed method is applied to enumerate all two-interval PCGs with up to ten vertices. Our computational results establish that all graphs with up to ten vertices are 2-IPCGs, making significant progress towards the open problem of determining whether a non-2-IPCG exists with fewer than 135 vertices.
Motion-guided small MAV detection in complex and non-planar scenes
Venue: arXiv cs.CV 2024
Year: 2024

In recent years, there has been a growing interest in the visual detection of micro aerial vehicles (MAVs) due to its importance in numerous applications. However, the existing methods based on either appearance or motion features encounter difficulties when the background is complex or the MAV is too small. In this paper, we propose a novel motion-guided MAV detector that can accurately identify small MAVs in complex and non-planar scenes. This detector first exploits a motion feature enhancement module to capture the motion features of small MAVs. Then it uses multi-object tracking and trajectory filtering to eliminate false positives caused by motion parallax. Finally, an appearance-based classifier and an appearance-based detector that operates on the cropped regions are used to achieve precise detection results. Our proposed method can effectively and efficiently detect extremely small MAVs from dynamic and complex backgrounds because it aggregates pixel-level motion features and eliminates false positives based on the motion and appearance features of MAVs. Experiments on the ARD-MAV dataset demonstrate that the proposed method could achieve high performance in small MAV detection under challenging conditions and outperform other state-of-the-art methods across various metrics
ZONIA: a Zero-Trust Oracle System for Blockchain IoT Applications
Venue: arXiv cs.ET 2024
Year: 2024

The rapid expansion of the Internet of Things (IoT) has led to significant data reliability and system transparency challenges, aggravated by the centralized nature of existing IoT architectures. This centralization often results in siloed data ecosystems, where interoperability issues and opaque data handling practices compromise both the utility and trustworthiness of IoT applications. To address these issues, we introduce ZONIA (Zero-trust Oracle Network for IoT Applications), a novel blockchain oracle system designed to enhance data integrity and decentralization in IoT environments. Unlike traditional approaches that rely on Trusted Execution Environments and centralized data sources, ZONIA utilizes a decentralized, zero-trust model that allows for anonymous participation and integrates multiple data sources to ensure fairness and reliability. This paper outlines ZONIA's architecture, which supports semantic and geospatial queries, details its data reliability mechanisms, and presents a comprehensive evaluation demonstrating its scalability and resilience against data falsification and collusion attacks. Both analytical and experimental results demonstrate ZONIA's scalability, showcasing its feasibility to handle an increasing number of nodes in the system under different system conditions and workloads. Furthermore, the implemented reputation mechanism significantly enhances data accuracy, maintaining high reliability even when 40\% of nodes exhibit malicious behavior.
RICASSO: Reinforced Imbalance Learning with Class-Aware Self-Supervised Outliers Exposure
Venue: arXiv cs.CV 2024
Year: 2024

In real-world scenarios, deep learning models often face challenges from both imbalanced (long-tailed) and out-of-distribution (OOD) data. However, existing joint methods rely on real OOD data, which leads to unnecessary trade-offs. In contrast, our research shows that data mixing, a potent augmentation technique for long-tailed recognition, can generate pseudo-OOD data that exhibit the features of both in-distribution (ID) data and OOD data. Therefore, by using mixed data instead of real OOD data, we can address long-tailed recognition and OOD detection holistically. We propose a unified framework called Reinforced Imbalance Learning with Class-Aware Self-Supervised Outliers Exposure (RICASSO), where "self-supervised" denotes that we only use ID data for outlier exposure. RICASSO includes three main strategies: Norm-Odd-Duality-Based Outlier Exposure: Uses mixed data as pseudo-OOD data, enabling simultaneous ID data rebalancing and outlier exposure through a single loss function. Ambiguity-Aware Logits Adjustment: Utilizes the ambiguity of ID data to adaptively recalibrate logits. Contrastive Boundary-Center Learning: Combines Virtual Boundary Learning and Dual-Entropy Center Learning to use mixed data for better feature separation and clustering, with Representation Consistency Learning for robustness. Extensive experiments demonstrate that RICASSO achieves state-of-the-art performance in long-tailed recognition and significantly improves OOD detection compared to our baseline (27% improvement in AUROC and 61% reduction in FPR on the iNaturalist2018 dataset). On iNaturalist2018, we even outperforms methods using real OOD data. The code will be made public soon.
Reflexive Input-Output Causality Mechanisms
Venue: arXiv cs.RO 2024
Year: 2024

This paper explores the concept of reflexive actuation, examining how robots may leverage both internal and external stimuli to trigger changes in the motion, performance, or physical characteristics of the robot, such as its size, shape, or configuration, and so on. These changes themselves may in turn be sequentially re-used as input to drive further adaptations. Drawing inspiration from biological systems, where reflexes are an essential component of the response to environmental changes, reflexive actuation is critical to enable robots to adapt to diverse situations and perform complex tasks. The underlying principles of reflexive actuation are analyzed, with examples provided from existing implementations such as contact-sensitive reflexive arms, physical counters, and their applications. The paper also outlines future directions and challenges for advancing this research area, emphasizing its significance in the development of adaptive, responsive robotic systems.
Development of a 3D virtual world tool for sustainable energy education
Venue: arXiv cs.HC 2024
Year: 2024

The UNESCO (2022) points out that the gap between the existing awareness of a person or a community, and the actual habits of everyday life, is attributed to: low levels of understanding the environmental issues at stake; low levels of knowledge regarding energy and climate issues; and lack of attention to social, emotional or behavioral learning. In this context, as environmental and energetic concerns grow rapidly in our daily lives, RAISE - Raising environmental knowledge & awareness, an ERAMUS+ project with partners from three countries (Greece, Portugal, and Italy), was born. The project's first and main objective is to increase the environmental knowledge and awareness (EK&A) of schoolchildren. To achieve this goal, a survey was first carried out to ascertain the students' knowledge and, based on these results, scenarios and educational material were drawn up to build a pedagogical tool - a 3D Virtual World Learning Environment (3D VWLE). This paper presents the desktop research results and the main features of the game that are intended to match the identified educational needs. The 3D VWLE puts students in situations where they can acquire transferable skills. The project demonstrated that the simulation effect of 3D VWLE on skills provides an excellent online learning environment for students to work on and improve their abilities. In other words, virtual worlds open the door to a new way of learning and teaching.
Intelligent prospector v2.0: exploration drill planning under epistemic model uncertainty
Venue: arXiv cs.AI 2024
Year: 2024

Optimal Bayesian decision making on what geoscientific data to acquire requires stating a prior model of uncertainty. Data acquisition is then optimized by reducing uncertainty on some property of interest maximally, and on average. In the context of exploration, very few, sometimes no data at all, is available prior to data acquisition planning. The prior model therefore needs to include human interpretations on the nature of spatial variability, or on analogue data deemed relevant for the area being explored. In mineral exploration, for example, humans may rely on conceptual models on the genesis of the mineralization to define multiple hypotheses, each representing a specific spatial variability of mineralization. More often than not, after the data is acquired, all of the stated hypotheses may be proven incorrect, i.e. falsified, hence prior hypotheses need to be revised, or additional hypotheses generated. Planning data acquisition under wrong geological priors is likely to be inefficient since the estimated uncertainty on the target property is incorrect, hence uncertainty may not be reduced at all. In this paper, we develop an intelligent agent based on partially observable Markov decision processes that plans optimally in the case of multiple geological or geoscientific hypotheses on the nature of spatial variability. Additionally, the artificial intelligence is equipped with a method that allows detecting, early on, whether the human stated hypotheses are incorrect, thereby saving considerable expense in data acquisition. Our approach is tested on a sediment-hosted copper deposit, and the algorithm presented has aided in the characterization of an ultra high-grade deposit in Zambia in 2023.
On the sparsity of binary numbers
Venue: arXiv cs.DM 2024
Year: 2024

We introduce the concept of negative coefficients in various number-based systems, with a focus on decimal and binary systems. We demonstrate that every binary number can be transformed into a sparse form, significantly enhancing computational speed by converting binary numbers into this form.
Thinking LLMs: General Instruction Following with Thought Generation
Venue: arXiv cs.CL 2024
Year: 2024

LLMs are typically trained to answer user questions or follow instructions similarly to how human experts respond. However, in the standard alignment framework they lack the basic ability of explicit thinking before answering. Thinking is important for complex questions that require reasoning and planning -- but can be applied to any task. We propose a training method for equipping existing LLMs with such thinking abilities for general instruction following without use of additional human data. We achieve this by an iterative search and optimization procedure that explores the space of possible thought generations, allowing the model to learn how to think without direct supervision. For each instruction, the thought candidates are scored using a judge model to evaluate their responses only, and then optimized via preference optimization. We show that this procedure leads to superior performance on AlpacaEval and Arena-Hard, and shows gains from thinking on non-reasoning categories such as marketing, health and general knowledge, in addition to more traditional reasoning & problem-solving tasks.
Kub: Enabling Elastic HPC Workloads on Containerized Environments
Venue: arXiv cs.DC 2024
Year: 2024

The conventional model of resource allocation in HPC systems is static. Thus, a job cannot leverage newly available resources in the system or release underutilized resources during the execution. In this paper, we present Kub, a methodology that enables elastic execution of HPC workloads on Kubernetes so that the resources allocated to a job can be dynamically scaled during the execution. One main optimization of our method is to maximize the reuse of the originally allocated resources so that the disruption to the running job can be minimized. The scaling procedure is coordinated among nodes through remote procedure calls on Kubernetes for deploying workloads in the cloud. We evaluate our approach using one synthetic benchmark and two production-level MPI-based HPC applications -- GROMACS and CM1. Our results demonstrate that the benefits of adapting the allocated resources depend on the workload characteristics. In the tested cases, a properly chosen scaling point for increasing resources during execution achieved up to 2x speedup. Also, the overhead of checkpointing and data reshuffling significantly influences the selection of optimal scaling points and requires application-specific knowledge.
Combinatorial Multi-armed Bandits: Arm Selection via Group Testing
Venue: arXiv cs.LG 2024
Year: 2024

This paper considers the problem of combinatorial multi-armed bandits with semi-bandit feedback and a cardinality constraint on the super-arm size. Existing algorithms for solving this problem typically involve two key sub-routines: (1) a parameter estimation routine that sequentially estimates a set of base-arm parameters, and (2) a super-arm selection policy for selecting a subset of base arms deemed optimal based on these parameters. State-of-the-art algorithms assume access to an exact oracle for super-arm selection with unbounded computational power. At each instance, this oracle evaluates a list of score functions, the number of which grows as low as linearly and as high as exponentially with the number of arms. This can be prohibitive in the regime of a large number of arms. This paper introduces a novel realistic alternative to the perfect oracle. This algorithm uses a combination of group-testing for selecting the super arms and quantized Thompson sampling for parameter estimation. Under a general separability assumption on the reward function, the proposed algorithm reduces the complexity of the super-arm-selection oracle to be logarithmic in the number of base arms while achieving the same regret order as the state-of-the-art algorithms that use exact oracles. This translates to at least an exponential reduction in complexity compared to the oracle-based approaches.
SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators
Venue: arXiv cs.LG 2024
Year: 2024

Large Language Models (LLMs) have transformed natural language processing, but face significant challenges in widespread deployment due to their high runtime cost. In this paper, we introduce SeedLM, a novel post-training compression method that uses seeds of pseudo-random generators to encode and compress model weights. Specifically, for each block of weights, we find a seed that is fed into a Linear Feedback Shift Register (LFSR) during inference to efficiently generate a random matrix. This matrix is then linearly combined with compressed coefficients to reconstruct the weight block. SeedLM reduces memory access and leverages idle compute cycles during inference, effectively speeding up memory-bound tasks by trading compute for fewer memory accesses. Unlike state-of-the-art compression methods that rely on calibration data, our approach is data-free and generalizes well across diverse tasks. Our experiments with Llama 3 70B, which is particularly challenging to compress, show that SeedLM achieves significantly better zero-shot accuracy retention at 4- and 3-bit than state-of-the-art techniques, while maintaining performance comparable to FP16 baselines. Additionally, FPGA-based tests demonstrate that 4-bit SeedLM, as model size increases to 70B, approaches a 4x speed-up over an FP16 Llama 2/3 baseline.
Towards LLM-guided Efficient and Interpretable Multi-linear Tensor Network Rank Selection
Venue: arXiv cs.LG 2024
Year: 2024

We propose a novel framework that leverages large language models (LLMs) to guide the rank selection in tensor network models for higher-order data analysis. By utilising the intrinsic reasoning capabilities and domain knowledge of LLMs, our approach offers enhanced interpretability of the rank choices and can effectively optimise the objective function. This framework enables users without specialised domain expertise to utilise tensor network decompositions and understand the underlying rationale within the rank selection process. Experimental results validate our method on financial higher-order datasets, demonstrating interpretable reasoning, strong generalisation to unseen test data, and its potential for self-enhancement over successive iterations. This work is placed at the intersection of large language models and higher-order data analysis.
DragEntity: Trajectory Guided Video Generation using Entity and Positional Relationships
Venue: arXiv cs.CV 2024
Year: 2024

In recent years, diffusion models have achieved tremendous success in the field of video generation, with controllable video generation receiving significant attention. However, existing control methods still face two limitations: Firstly, control conditions (such as depth maps, 3D Mesh) are difficult for ordinary users to obtain directly. Secondly, it's challenging to drive multiple objects through complex motions with multiple trajectories simultaneously. In this paper, we introduce DragEntity, a video generation model that utilizes entity representation for controlling the motion of multiple objects. Compared to previous methods, DragEntity offers two main advantages: 1) Our method is more user-friendly for interaction because it allows users to drag entities within the image rather than individual pixels. 2) We use entity representation to represent any object in the image, and multiple objects can maintain relative spatial relationships. Therefore, we allow multiple trajectories to control multiple objects in the image with different levels of complexity simultaneously. Our experiments validate the effectiveness of DragEntity, demonstrating its excellent performance in fine-grained control in video generation.
Denial-of-Service Poisoning Attacks against Large Language Models
Venue: arXiv cs.CR 2024
Year: 2024

Recent studies have shown that LLMs are vulnerable to denial-of-service (DoS) attacks, where adversarial inputs like spelling errors or non-semantic prompts trigger endless outputs without generating an [EOS] token. These attacks can potentially cause high latency and make LLM services inaccessible to other users or tasks. However, when there are speech-to-text interfaces (e.g., voice commands to a robot), executing such DoS attacks becomes challenging, as it is difficult to introduce spelling errors or non-semantic prompts through speech. A simple DoS attack in these scenarios would be to instruct the model to "Keep repeating Hello", but we observe that relying solely on natural instructions limits output length, which is bounded by the maximum length of the LLM's supervised finetuning (SFT) data. To overcome this limitation, we propose poisoning-based DoS (P-DoS) attacks for LLMs, demonstrating that injecting a single poisoned sample designed for DoS purposes can break the output length limit. For example, a poisoned sample can successfully attack GPT-4o and GPT-4o mini (via OpenAI's finetuning API) using less than $1, causing repeated outputs up to the maximum inference length (16K tokens, compared to 0.5K before poisoning). Additionally, we perform comprehensive ablation studies on open-source LLMs and extend our method to LLM agents, where attackers can control both the finetuning dataset and algorithm. Our findings underscore the urgent need for defenses against P-DoS attacks to secure LLMs. Our code is available at https://github.com/sail-sg/P-DoS.
Boosting Camera Motion Control for Video Diffusion Transformers
Venue: arXiv cs.CV 2024
Year: 2024

Recent advancements in diffusion models have significantly enhanced the quality of video generation. However, fine-grained control over camera pose remains a challenge. While U-Net-based models have shown promising results for camera control, transformer-based diffusion models (DiT)-the preferred architecture for large-scale video generation - suffer from severe degradation in camera motion accuracy. In this paper, we investigate the underlying causes of this issue and propose solutions tailored to DiT architectures. Our study reveals that camera control performance depends heavily on the choice of conditioning methods rather than camera pose representations that is commonly believed. To address the persistent motion degradation in DiT, we introduce Camera Motion Guidance (CMG), based on classifier-free guidance, which boosts camera control by over 400%. Additionally, we present a sparse camera control pipeline, significantly simplifying the process of specifying camera poses for long videos. Our method universally applies to both U-Net and DiT models, offering improved camera control for video generation tasks.