OpenRaiser/Intern-Atlas huggingface dataset:
No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantization
Venue: arXiv cs.LG 2024
Year: 2024

Key-Value (KV) Caching has become an essential technique for accelerating the inference speed and throughput of generative Large Language Models~(LLMs). However, the memory footprint of the KV cache poses a critical bottleneck in LLM deployment as the cache size grows with batch size and sequence length, often surpassing even the size of the model itself. Although recent methods were proposed to select and evict unimportant KV pairs from the cache to reduce memory consumption, the potential ramifications of eviction on the generative process are yet to be thoroughly examined. In this paper, we examine the detrimental impact of cache eviction and observe that unforeseen risks arise as the information contained in the KV pairs is exhaustively discarded, resulting in safety breaches, hallucinations, and context loss. Surprisingly, we find that preserving even a small amount of information contained in the evicted KV pairs via reduced precision quantization substantially recovers the incurred degradation. On the other hand, we observe that the important KV pairs must be kept at a relatively higher precision to safeguard the generation quality. Motivated by these observations, we propose \textit{Mixed-precision KV cache}~(MiKV), a reliable cache compression method that simultaneously preserves the context details by retaining the evicted KV pairs in low-precision and ensure generation quality by keeping the important KV pairs in high-precision. Experiments on diverse benchmarks and LLM backbones show that our proposed method offers a state-of-the-art trade-off between compression ratio and performance, compared to other baselines.
Hierarchical Multi-Relational Graph Representation Learning for Large-Scale Prediction of Drug-Drug Interactions
Venue: arXiv cs.LG 2024
Year: 2024

Most existing methods for predicting drug-drug interactions (DDI) predominantly concentrate on capturing the explicit relationships among drugs, overlooking the valuable implicit correlations present between drug pairs (DPs), which leads to weak predictions. To address this issue, this paper introduces a hierarchical multi-relational graph representation learning (HMGRL) approach. Within the framework of HMGRL, we leverage a wealth of drug-related heterogeneous data sources to construct heterogeneous graphs, where nodes represent drugs and edges denote clear and various associations. The relational graph convolutional network (RGCN) is employed to capture diverse explicit relationships between drugs from these heterogeneous graphs. Additionally, a multi-view differentiable spectral clustering (MVDSC) module is developed to capture multiple valuable implicit correlations between DPs. Within the MVDSC, we utilize multiple DP features to construct graphs, where nodes represent DPs and edges denote different implicit correlations. Subsequently, multiple DP representations are generated through graph cutting, each emphasizing distinct implicit correlations. The graph-cutting strategy enables our HMGRL to identify strongly connected communities of graphs, thereby reducing the fusion of irrelevant features. By combining every representation view of a DP, we create high-level DP representations for predicting DDIs. Two genuine datasets spanning three distinct tasks are adopted to gauge the efficacy of our HMGRL. Experimental outcomes unequivocally indicate that HMGRL surpasses several leading-edge methods in performance.
OccTransformer: Improving BEVFormer for 3D camera-only occupancy prediction
Venue: arXiv cs.CV 2024
Year: 2024

This technical report presents our solution, "occTransformer" for the 3D occupancy prediction track in the autonomous driving challenge at CVPR 2023. Our method builds upon the strong baseline BEVFormer and improves its performance through several simple yet effective techniques. Firstly, we employed data augmentation to increase the diversity of the training data and improve the model's generalization ability. Secondly, we used a strong image backbone to extract more informative features from the input data. Thirdly, we incorporated a 3D unet head to better capture the spatial information of the scene. Fourthly, we added more loss functions to better optimize the model. Additionally, we used an ensemble approach with the occ model BevDet and SurroundOcc to further improve the performance. Most importantly, we integrated 3D detection model StreamPETR to enhance the model's ability to detect objects in the scene. Using these methods, our solution achieved 49.23 miou on the 3D occupancy prediction track in the autonomous driving challenge.
Generation of skill-specific maps from graph world models for robotic systems
Venue: arXiv cs.RO 2024
Year: 2024

With the increase in the availability of Building Information Models (BIM) and (semi-) automatic tools to generate BIM from point clouds, we propose a world model architecture and algorithms to allow the use of the semantic and geometric knowledge encoded within these models to generate maps for robot localization and navigation. When heterogeneous robots are deployed within an environment, maps obtained from classical SLAM approaches might not be shared between all agents within a team of robots, e.g. due to a mismatch in sensor type, or a difference in physical robot dimensions. Our approach extracts the 3D geometry and semantic description of building elements (e.g. material, element type, color) from BIM, and represents this knowledge in a graph. Based on queries on the graph and knowledge of the skills of the robot, we can generate skill-specific maps that can be used during the execution of localization or navigation tasks. The approach is validated with data from complex build environments and integrated into existing navigation frameworks.
HearHere: Mitigating Echo Chambers in News Consumption through an AI-based Web System
Venue: arXiv cs.HC 2024
Year: 2024

Considerable efforts are currently underway to mitigate the negative impacts of echo chambers, such as increased susceptibility to fake news and resistance towards accepting scientific evidence. Prior research has presented the development of computer systems that support the consumption of news information from diverse political perspectives to mitigate the echo chamber effect. However, existing studies still lack the ability to effectively support the key processes of news information consumption and quantitatively identify a political stance towards the information. In this paper, we present HearHere, an AI-based web system designed to help users accommodate information and opinions from diverse perspectives. HearHere facilitates the key processes of news information consumption through two visualizations. Visualization 1 provides political news with quantitative political stance information, derived from our graph-based political classification model, and users can experience diverse perspectives (Hear). Visualization 2 allows users to express their opinions on specific political issues in a comment form and observe the position of their own opinions relative to pro-liberal and pro-conservative comments presented on a map interface (Here). Through a user study with 94 participants, we demonstrate the feasibility of HearHere in supporting the consumption of information from various perspectives. Our findings highlight the importance of providing political stance information and quantifying users' political status as a means to mitigate political polarization. In addition, we propose design implications for system development, including the consideration of demographics such as political interest and providing users with initiatives.
Affective State Detection using fNIRs and Machine Learning
Venue: arXiv cs.HC 2024
Year: 2024

Affective states regulate our day to day to function and has a tremendous effect on mental and physical health. Detection of affective states is of utmost importance for mental health monitoring, smart entertainment selection and dynamic workload management. In this paper, we discussed relevant literature on affective state detection using physiology data, the benefits and limitations of different sensors and methods used for collecting physiology data, and our rationale for selecting functional near-infrared spectroscopy. We present the design of an experiment involving nine subjects to evoke the affective states of meditation, amusement and cognitive load and the results of the attempt to classify using machine learning. A mean accuracy of 83.04% was achieved in three class classification with an individual model; 84.39% accuracy was achieved for a group model and 60.57% accuracy was achieved for subject independent model using leave one out cross validation. It was found that prediction accuracy for cognitive load was higher (evoked using a pen and paper task) than the other two classes (evoked using computer bases tasks). To verify that this discrepancy was not due to motor skills involved in the pen and paper task, a second experiment was conducted using four participants and the results of that experiment has also been presented in the paper.
The Cost of Permissionless Liquidity Provision in Automated Market Makers
Venue: arXiv cs.GT 2024
Year: 2024

Automated market makers (AMMs) allocate fee revenue \textit{proportional} to the amount of liquidity investors deposit. In this paper, we study the economic consequences of the competition between passive liquidity providers (LPs) caused by this allocation rule. We employ a game-theoretic model in which $N$ strategic agents optimally provide liquidity and two types of liquidity traders trade. In this setting, we find that competition drives LPs to provide excess liquidity. Excess liquidity is costly as more capital is exposed to adverse selection costs. One of our main results is that the price of anarchy, defined over the liquidity provider performance, is $O(N)$, implying that the welfare loss scales linearly with the number of liquidity providers. This inefficient capital allocation is masked when considering the welfare of elastic liquidity traders as the total price of anarchy is $O(1)$. Since this result is driven by elastic liquidity traders benefiting from the liquidity provided because of inelastic liquidity traders, we show that different types of liquidity traders complement each other. Finally, we show that AMM designs that reduce the arbitrage intensity per unit of liquidity do increase utility for liquidity traders but importantly not for LPs nor do they necessarily decrease total arbitrage volume.
Hierarchical Multimodal Pre-training for Visually Rich Webpage Understanding
Venue: arXiv cs.CL 2024
Year: 2024

The growing prevalence of visually rich documents, such as webpages and scanned/digital-born documents (images, PDFs, etc.), has led to increased interest in automatic document understanding and information extraction across academia and industry. Although various document modalities, including image, text, layout, and structure, facilitate human information retrieval, the interconnected nature of these modalities presents challenges for neural networks. In this paper, we introduce WebLM, a multimodal pre-training network designed to address the limitations of solely modeling text and structure modalities of HTML in webpages. Instead of processing document images as unified natural images, WebLM integrates the hierarchical structure of document images to enhance the understanding of markup-language-based documents. Additionally, we propose several pre-training tasks to model the interaction among text, structure, and image modalities effectively. Empirical results demonstrate that the pre-trained WebLM significantly surpasses previous state-of-the-art pre-trained models across several webpage understanding tasks. The pre-trained models and code are available at https://github.com/X-LANCE/weblm.
Low-Modeling of Software Systems
Venue: arXiv cs.SE 2024
Year: 2024

There is a growing need for better development methods and tools to keep up with the increasing complexity of new software systems. New types of user interfaces, the need for intelligent components, sustainability concerns, ... bring new challenges that we need to handle. In the last years, model-driven engineering has been key to improving the quality and productivity of software development, but models themselves are becoming increasingly complex to specify and manage. In this paper, we present the concept of low-modeling as a solution to enhance current model-driven engineering techniques and get them ready for this new generation of software systems.
Neuromorphic Event-Driven Semantic Communication in Microgrids
Venue: arXiv cs.ET 2024
Year: 2024

Synergies between advanced communications, computing and artificial intelligence are unraveling new directions of coordinated operation and resiliency in microgrids. On one hand, coordination among sources is facilitated by distributed, privacy-minded processing at multiple locations, whereas on the other hand, it also creates exogenous data arrival paths for adversaries that can lead to cyber-physical attacks amongst other reliability issues in the communication layer. This long-standing problem necessitates new intrinsic ways of exchanging information between converters through power lines to optimize the system's control performance. Going beyond the existing power and data co-transfer technologies that are limited by efficiency and scalability concerns, this paper proposes neuromorphic learning to implant communicative features using spiking neural networks (SNNs) at each node, which is trained collaboratively in an online manner simply using the power exchanges between the nodes. As opposed to the conventional neuromorphic sensors that operate with spiking signals, we employ an event-driven selective process to collect sparse data for training of SNNs. Finally, its multi-fold effectiveness and reliable performance is validated under simulation conditions with different microgrid topologies and components to establish a new direction in the sense-actuate-compute cycle for power electronic dominated grids and microgrids.
DevPhish: Exploring Social Engineering in Software Supply Chain Attacks on Developers
Venue: arXiv cs.SE 2024
Year: 2024

The Software Supply Chain (SSC) has captured considerable attention from attackers seeking to infiltrate systems and undermine organizations. There is evidence indicating that adversaries utilize Social Engineering (SocE) techniques specifically aimed at software developers. That is, they interact with developers at critical steps in the Software Development Life Cycle (SDLC), such as accessing Github repositories, incorporating code dependencies, and obtaining approval for Pull Requests (PR) to introduce malicious code. This paper aims to comprehensively explore the existing and emerging SocE tactics employed by adversaries to trick Software Engineers (SWEs) into delivering malicious software. By analyzing a diverse range of resources, which encompass established academic literature and real-world incidents, the paper systematically presents an overview of these manipulative strategies within the realm of the SSC. Such insights prove highly beneficial for threat modeling and security gap analysis.
CafkNet: GNN-Empowered Forward Kinematic Modeling for Cable-Driven Parallel Robots
Venue: arXiv cs.RO 2024
Year: 2024

Cable-driven parallel robots (CDPRs) have gained significant attention due to their promising advantages. When deploying CDPRs in practice, the kinematic modeling is a key question. Unlike serial robots, CDPRs have a simple inverse kinematics problem but a complex forward kinematics (FK) issue. So, the development of accurate and efficient FK solvers has been a prominent research focus in CDPR applications. By observing the topology within CDPRs, in this paper, we propose a graph-based representation to model CDPRs and introduce CafkNet, a fast and general FK solving method, leveraging Graph Neural Network (GNN) to learn the topological structure and yield the real FK solutions with superior generality, high accuracy, and low time cost. CafkNet is extensively tested on 3D and 2D CDPRs in different configurations, both in simulators and real scenarios. The results demonstrate its ability to learn CDPRs' internal topology and accurately solve the FK problem. Then, the zero-shot generalization from one configuration to another is validated. Also, the sim2real gap can be bridged by CafkNet using both simulation and real-world data. To the best of our knowledge, it is the first study that employs the GNN to solve the FK problem for CDPRs.
Emotion Classification in Low and Moderate Resource Languages
Venue: arXiv cs.CL 2024
Year: 2024

It is important to be able to analyze the emotional state of people around the globe. There are 7100+ active languages spoken around the world and building emotion classification for each language is labor intensive. Particularly for low-resource and endangered languages, building emotion classification can be quite challenging. We present a cross-lingual emotion classifier, where we train an emotion classifier with resource-rich languages (i.e. \textit{English} in our work) and transfer the learning to low and moderate resource languages. We compare and contrast two approaches of transfer learning from a high-resource language to a low or moderate-resource language. One approach projects the annotation from a high-resource language to low and moderate-resource language in parallel corpora and the other one uses direct transfer from high-resource language to the other languages. We show the efficacy of our approaches on 6 languages: Farsi, Arabic, Spanish, Ilocano, Odia, and Azerbaijani. Our results indicate that our approaches outperform random baselines and transfer emotions across languages successfully. For all languages, the direct cross-lingual transfer of emotion yields better results. We also create annotated emotion-labeled resources for four languages: Farsi, Azerbaijani, Ilocano and Odia.
A Relational Inductive Bias for Dimensional Abstraction in Neural Networks
Venue: arXiv cs.AI 2024
Year: 2024

The human cognitive system exhibits remarkable flexibility and generalization capabilities, partly due to its ability to form low-dimensional, compositional representations of the environment. In contrast, standard neural network architectures often struggle with abstract reasoning tasks, overfitting, and requiring extensive data for training. This paper investigates the impact of the relational bottleneck -- a mechanism that focuses processing on relations among inputs -- on the learning of factorized representations conducive to compositional coding and the attendant flexibility of processing. We demonstrate that such a bottleneck not only improves generalization and learning efficiency, but also aligns network performance with human-like behavioral biases. Networks trained with the relational bottleneck developed orthogonal representations of feature dimensions latent in the dataset, reflecting the factorized structure thought to underlie human cognitive flexibility. Moreover, the relational network mimics human biases towards regularity without pre-specified symbolic primitives, suggesting that the bottleneck fosters the emergence of abstract representations that confer flexibility akin to symbols.
Semantic Information in MC: Chemotaxis Beyond Shannon
Venue: arXiv cs.IT 2024
Year: 2024

The recently emerged molecular communication (MC) paradigm intends to leverage communication engineering tools for the design of synthetic chemical communication systems. These systems are envisioned to operate at nanoscale and in biological environments, such as the human body, and catalyze the emergence of revolutionary applications in the context of early disease monitoring and drug targeting. Despite the abundance of theoretical (and recently also experimental) MC system designs proposed over the past years, some fundamental questions remain unresolved, hindering the breakthrough of MC in real-world applications. One of these questions is: What can be a useful measure of information in the context of MC applications? While most existing works on MC build upon the concept of syntactic information as introduced by Shannon, in this paper, we explore the framework of semantic information as introduced by Kolchinsky and Wolpert for the information-theoretic analysis of a natural MC system, namely bacterial chemotaxis. Exploiting computational agent-based modeling (ABM), we are able to quantify, for the first time, the amount of information that the considered chemotactic bacterium (CB) utilizes to adapt to and survive in a dynamic environment. In other words, we show how the flow of information between the environment and the CB is related to the effectiveness of communication. Effectiveness here refers to the adaptation of the CB to the dynamic environment in order to ensure survival. Our analysis reveals that it highly depends on the environmental conditions how much information the CB can effectively utilize for improving their survival chances. Encouraged by our results, we envision that the proposed semantic information framework can open new avenues for the development of theoretical and experimental MC system designs for future nanoscale applications.
Implementing Online Reinforcement Learning with Clustering Neural Networks
Venue: arXiv cs.NE 2024
Year: 2024

An agent employing reinforcement learning takes inputs (state variables) from an environment and performs actions that affect the environment in order to achieve some objective. Rewards (positive or negative) guide the agent toward improved future actions. This paper builds on prior clustering neural network research by constructing an agent with biologically plausible neo-Hebbian three-factor synaptic learning rules, with a reward signal as the third factor (in addition to pre- and post-synaptic spikes). The classic cart-pole problem (balancing an inverted pendulum) is used as a running example throughout the exposition. Simulation results demonstrate the efficacy of the approach, and the proposed method may eventually serve as a low-level component of a more general method.
NewsQs: Multi-Source Question Generation for the Inquiring Mind
Venue: arXiv cs.CL 2024
Year: 2024

We present NewsQs (news-cues), a dataset that provides question-answer pairs for multiple news documents. To create NewsQs, we augment a traditional multi-document summarization dataset with questions automatically generated by a T5-Large model fine-tuned on FAQ-style news articles from the News On the Web corpus. We show that fine-tuning a model with control codes produces questions that are judged acceptable more often than the same model without them as measured through human evaluation. We use a QNLI model with high correlation with human annotations to filter our data. We release our final dataset of high-quality questions, answers, and document clusters as a resource for future work in query-based multi-document summarization.