OpenRaiser/Intern-Atlas huggingface dataset:
InfuserKI: Enhancing Large Language Models with Knowledge Graphs via Infuser-Guided Knowledge Integration
Venue: arXiv cs.CL 2024
Year: 2024

Large Language Models (LLMs) have achieved exceptional capabilities in open generation across various domains, yet they encounter difficulties with tasks that require intensive knowledge. To address these challenges, methods for integrating knowledge have been developed, which augment LLMs with domain-specific knowledge graphs through external modules. These approaches, however, face data inefficiency issues as they necessitate the processing of both known and unknown knowledge for fine-tuning. Thus, our research focuses on a novel problem: efficiently integrating unknown knowledge into LLMs without unnecessary overlap of known knowledge. A risk of introducing new knowledge is the potential forgetting of existing knowledge. To mitigate this risk, we propose the innovative {\method} framework. This framework employs transformer internal states to determine when to enrich LLM outputs with additional information, effectively preventing knowledge forgetting. Performance evaluations using the UMLS-2.5k and MetaQA domain knowledge graphs reveal that {\method} not only successfully integrates new knowledge but also outperforms state-of-the-art baselines, reducing knowledge forgetting by 9\% and 6\%, respectively.
Age of $k$-out-of-$n$ Systems on a Gossip Network
Venue: arXiv cs.IT 2024
Year: 2024

We consider information update systems on a gossip network, which consists of a single source and $n$ receiver nodes. The source encrypts the information into $n$ distinct keys with version stamps, sending a unique key to each node. For decoding the information in a $k$-out-of-$n$ system, each receiver node requires at least $k+1$ different keys with the same version, shared over peer-to-peer connections. Each node determines $k$ based on a given function, ensuring that as $k$ increases, the precision of the decoded information also increases. We consider two different schemes: a memory scheme (in which the nodes keep the source's current and previous encrypted messages) and a memoryless scheme (in which the nodes are allowed to only keep the source's current message). We measure the ''timeliness'' of information updates by using the $k$-keys version age of information. Our work focuses on determining closed-form expressions for the time average age of information in a heterogeneous random graph under both with memory and memoryless schemes.
A Three-Party Repeated Coalition Formation Game for PLS in Wireless Communications with IRSs
Venue: arXiv cs.GT 2024
Year: 2024

In this paper, a repeated coalition formation game (RCFG) with dynamic decision-making for physical layer security (PLS) in wireless communications with intelligent reflecting surfaces (IRSs) has been investigated. In the considered system, one central legitimate transmitter (LT) aims to transmit secret signals to a group of legitimate receivers (LRs) under the threat of a proactive eavesdropper (EV), while there exist a number of third-party IRSs (TIRSs) which can choose to form a coalition with either legitimate pairs (LPs) or the EV to improve their respective performances in exchange for potential benefits (e.g., payments). Unlike existing works that commonly restricted to friendly IRSs or malicious IRSs only, we study the complicated dynamic ally-adversary relationships among LPs, EV and TIRSs, under unpredictable wireless channel conditions, and introduce a RCFG to model their long-term strategic interactions. Particularly, we first analyze the existence of Nash equilibrium (NE) in the formulated RCFG, and then propose a switch operations-based coalition selection along with a deep reinforcement learning (DRL)-based algorithm for obtaining such equilibrium. Simulations examine the feasibility of the proposed algorithm and show its superiority over counterparts.
GenAD: Generative End-to-End Autonomous Driving
Venue: arXiv cs.CV 2024
Year: 2024

Directly producing planning results from raw sensors has been a long-desired solution for autonomous driving and has attracted increasing attention recently. Most existing end-to-end autonomous driving methods factorize this problem into perception, motion prediction, and planning. However, we argue that the conventional progressive pipeline still cannot comprehensively model the entire traffic evolution process, e.g., the future interaction between the ego car and other traffic participants and the structural trajectory prior. In this paper, we explore a new paradigm for end-to-end autonomous driving, where the key is to predict how the ego car and the surroundings evolve given past scenes. We propose GenAD, a generative framework that casts autonomous driving into a generative modeling problem. We propose an instance-centric scene tokenizer that first transforms the surrounding scenes into map-aware instance tokens. We then employ a variational autoencoder to learn the future trajectory distribution in a structural latent space for trajectory prior modeling. We further adopt a temporal model to capture the agent and ego movements in the latent space to generate more effective future trajectories. GenAD finally simultaneously performs motion prediction and planning by sampling distributions in the learned structural latent space conditioned on the instance tokens and using the learned temporal model to generate futures. Extensive experiments on the widely used nuScenes benchmark show that the proposed GenAD achieves state-of-the-art performance on vision-centric end-to-end autonomous driving with high efficiency. Code: https://github.com/wzzheng/GenAD.
PreAct: Prediction Enhances Agent's Planning Ability
Venue: arXiv cs.CL 2024
Year: 2024

Addressing the disparity between forecasts and actual results can enable individuals to expand their thought processes and stimulate self-reflection, thus promoting accurate planning. In this research, we present **PreAct**, an agent framework that integrates **pre**diction, **rea**soning, and **act**ion. By utilizing the information derived from predictions, the large language model (LLM) agent can provide a wider range and more strategically focused reasoning. This leads to more efficient actions that aid the agent in accomplishing intricate tasks. Our experimental results show that PreAct surpasses the ReAct method in completing complex tasks and that PreAct's performance can be further improved when paired with other memory or selection strategy techniques. We presented the model with varying quantities of historical predictions and discovered that these predictions consistently enhance LLM planning.The variances in single-step reasoning between PreAct and ReAct indicate that PreAct indeed has benefits in terms of diversity and strategic orientation over ReAct.
A Temporally Disentangled Contrastive Diffusion Model for Spatiotemporal Imputation
Venue: arXiv cs.LG 2024
Year: 2024

Spatiotemporal data analysis is pivotal across various domains, such as transportation, meteorology, and healthcare. The data collected in real-world scenarios are often incomplete due to device malfunctions and network errors. Spatiotemporal imputation aims to predict missing values by exploiting the spatial and temporal dependencies in the observed data. Traditional imputation approaches based on statistical and machine learning techniques require the data to conform to their distributional assumptions, while graph and recurrent neural networks are prone to error accumulation problems due to their recurrent structures. Generative models, especially diffusion models, can potentially circumvent the reliance on inaccurate, previously imputed values for future predictions; However, diffusion models still face challenges in generating stable results. We propose to address these challenges by designing conditional information to guide the generative process and expedite the training process. We introduce a conditional diffusion framework called C$^2$TSD, which incorporates disentangled temporal (trend and seasonality) representations as conditional information and employs contrastive learning to improve generalizability. Our extensive experiments on three real-world datasets demonstrate the superior performance of our approach compared to a number of state-of-the-art baselines.
A novel Fourier neural operator framework for classification of multi-sized images: Application to three dimensional digital porous media
Venue: arXiv cs.CV 2024
Year: 2024

Fourier neural operators (FNOs) are invariant with respect to the size of input images, and thus images with any size can be fed into FNO-based frameworks without any modification of network architectures, in contrast to traditional convolutional neural networks (CNNs). Leveraging the advantage of FNOs, we propose a novel deep-learning framework for classifying images with varying sizes. Particularly, we simultaneously train the proposed network on multi-sized images. As a practical application, we consider the problem of predicting the label (e.g., permeability) of three-dimensional digital porous media. To construct the framework, an intuitive approach is to connect FNO layers to a classifier using adaptive max pooling. First, we show that this approach is only effective for porous media with fixed sizes, whereas it fails for porous media of varying sizes. To overcome this limitation, we introduce our approach: instead of using adaptive max pooling, we use static max pooling with the size of channel width of FNO layers. Since the channel width of the FNO layers is independent of input image size, the introduced framework can handle multi-sized images during training. We show the effectiveness of the introduced framework and compare its performance with the intuitive approach through the example of the classification of three-dimensional digital porous media of varying sizes.
Visual In-Context Learning for Large Vision-Language Models
Venue: arXiv cs.CV 2024
Year: 2024

In Large Visual Language Models (LVLMs), the efficacy of In-Context Learning (ICL) remains limited by challenges in cross-modal interactions and representation disparities. To overcome these challenges, we introduce a novel Visual In-Context Learning (VICL) method comprising Visual Demonstration Retrieval, Intent-Oriented Image Summarization, and Intent-Oriented Demonstration Composition. Our approach retrieves images via ''Retrieval & Rerank'' paradigm, summarises images with task intent and task-specific visual parsing, and composes language-based demonstrations that reduce token count and alleviate cross-modal interaction problem. Experimental evaluations on five visual reasoning datasets demonstrate the effectiveness of our method. Moreover, our extensive experiments leverage information flow analysis to elucidate the effectiveness of our method, and investigate the impact of length and position of demonstrations for LVLM. The use of in-context unlearning further shows promise in resetting specific model knowledge without retraining.
Toward Humanity-Centered Design without Hubris
Venue: arXiv cs.HC 2024
Year: 2024

Humanity-centered design is a concept of emerging interest in HCI, one motivated by the limitations of human-centered design. As discussed to date, humanity-centered design is compatible with but goes beyond human-centered design in that it considers entire ecosystems and populations over the long term and centers participatory design. Though the intentions of humanity-centered design are laudable, current articulations of humanity-centered design are incoherent in a number of ways, leading to questions of how exactly it can or should be implemented. In this article, I delineate four ways in which humanity-centered design is incoherent, which can be boiled down to a tendency toward hubris, and propose a more fruitful way forward, a humble approach to humanity-centered design. Rather than a contradiction in terms, "humility" here refers to an organic, piecemeal, patterns-based approach to design that will be good for our being on this earth.
Using rule engine in self-healing systems and MAPE model
Venue: arXiv cs.SE 2024
Year: 2024

Software malfunction presents a significant hurdle within the computing domain, carrying substantial risks for systems, enterprises, and users universally. To produce software with high reliability and quality, effective debugging is essential. Program debugging is an activity to reduce software maintenance costs. In this study, a failure repair method that uses a rule engine is presented. The simulation on mRUBIS showed that the proposed method could be efficient in the operational environment. Through a thorough grasp of software failure and the adoption of efficient mitigation strategies, stakeholders can bolster the dependability, security, and adaptability of software systems. This, in turn, reduces the repercussions of failures and cultivates increased confidence in digital technologies.
PolypNextLSTM: A lightweight and fast polyp video segmentation network using ConvNext and ConvLSTM
Venue: arXiv cs.CV 2024
Year: 2024

Commonly employed in polyp segmentation, single image UNet architectures lack the temporal insight clinicians gain from video data in diagnosing polyps. To mirror clinical practices more faithfully, our proposed solution, PolypNextLSTM, leverages video-based deep learning, harnessing temporal information for superior segmentation performance with the least parameter overhead, making it possibly suitable for edge devices. PolypNextLSTM employs a UNet-like structure with ConvNext-Tiny as its backbone, strategically omitting the last two layers to reduce parameter overhead. Our temporal fusion module, a Convolutional Long Short Term Memory (ConvLSTM), effectively exploits temporal features. Our primary novelty lies in PolypNextLSTM, which stands out as the leanest in parameters and the fastest model, surpassing the performance of five state-of-the-art image and video-based deep learning models. The evaluation of the SUN-SEG dataset spans easy-to-detect and hard-to-detect polyp scenarios, along with videos containing challenging artefacts like fast motion and occlusion. Comparison against 5 image-based and 5 video-based models demonstrates PolypNextLSTM's superiority, achieving a Dice score of 0.7898 on the hard-to-detect polyp test set, surpassing image-based PraNet (0.7519) and video-based PNSPlusNet (0.7486). Notably, our model excels in videos featuring complex artefacts such as ghosting and occlusion. PolypNextLSTM, integrating pruned ConvNext-Tiny with ConvLSTM for temporal fusion, not only exhibits superior segmentation performance but also maintains the highest frames per speed among evaluated models. Access code here https://github.com/mtec-tuhh/PolypNextLSTM
Dynamic planning in hierarchical active inference
Venue: arXiv cs.AI 2024
Year: 2024

By dynamic planning, we refer to the ability of the human brain to infer and impose motor trajectories related to cognitive decisions. A recent paradigm, active inference, brings fundamental insights into the adaptation of biological organisms, constantly striving to minimize prediction errors to restrict themselves to life-compatible states. Over the past years, many studies have shown how human and animal behaviors could be explained in terms of active inference - either as discrete decision-making or continuous motor control - inspiring innovative solutions in robotics and artificial intelligence. Still, the literature lacks a comprehensive outlook on effectively planning realistic actions in changing environments. Setting ourselves the goal of modeling complex tasks such as tool use, we delve into the topic of dynamic planning in active inference, keeping in mind two crucial aspects of biological behavior: the capacity to understand and exploit affordances for object manipulation, and to learn the hierarchical interactions between the self and the environment, including other agents. We start from a simple unit and gradually describe more advanced structures, comparing recently proposed design choices and providing basic examples. This study distances itself from traditional views centered on neural networks and reinforcement learning, and points toward a yet unexplored direction in active inference: hybrid representations in hierarchical models.
Opening the black box of language acquisition
Venue: arXiv cs.CL 2024
Year: 2024

Recent advances in large language models using deep learning techniques have renewed interest on how languages can be learned from data. However, it is unclear whether or how these models represent grammatical information from the learned languages. In addition, the models must be pre-trained on large corpora before they can be used. In this work, we propose an alternative, more transparent and cognitively plausible architecture for learning language. Instead of using deep learning, our approach uses a minimal cognitive architecture based on sequence memory and chunking. The learning mechanism is based on the principles of reinforcement learning. We test our architecture on a number of natural-like toy languages. Results show that the model can learn these artificial languages from scratch and extract grammatical information that supports learning. Our study demonstrates the power of this simple architecture and stresses the importance of sequence memory as a key component of the language learning process. Since other animals do not seem to have a faithful sequence memory, this may explain why only humans have developed complex languages.
Variability-Aware Noise-Induced Dynamic Instability of Ultra-Low-Voltage SRAM Bitcells
Venue: arXiv cs.AR 2024
Year: 2024

Stability of ultra-low-voltage SRAM bitcells in retention mode is threatened by two types of uncertainty: process variability and intrinsic noise. While variability dominates the failure probability, noise-induced bit flips in weakened bitcells lead to dynamic instability. We study both effects jointly in a unified SPICE simulation framework. Starting from a synthetic representation of process variations introduced in a previous work, we identify the cases of poor noise immunity that require thorough noise analyses. Relying on a rigorous and systematic methodology, we simulate them in the time domain so as to emulate a true data retention operation. Short times to failure, unacceptable for a practical ultra-low-power memory system application, are recorded. The transient bit-flip mechanism is analysed and a dynamic failure criterion involving the unstable point is established. We conclude that, beyond static variability, the dynamic noise inflates defectiveness among SRAM bitcells. We also discuss the limits of existing analytical formulas from the literature, which rely on a linear near-equilibrium approximation of the SRAM dynamics to, inaccurately, predict the mean time to failure.
Enabling Software Defined Optical Networks
Venue: arXiv cs.NI 2024
Year: 2024

This paper gives an overview of Software Defined Optical Networks or SDONs and how they can be implemented. It traces the evolution of Optical networks upto GMPLS and traces the idea of SDN and builds upto OpenFlow. The paper explores the need for SDONs and explains what a SDON solution could look like, including the hardware. It also seeks to explain how OpenFlow could be used as a part of this solution to overcome the limitations of GMPLS.
GNNavi: Navigating the Information Flow in Large Language Models by Graph Neural Network
Venue: arXiv cs.CL 2024
Year: 2024

Large Language Models (LLMs) exhibit strong In-Context Learning (ICL) capabilities when prompts with demonstrations are used. However, fine-tuning still remains crucial to further enhance their adaptability. Prompt-based fine-tuning proves to be an effective fine-tuning method in low-data scenarios, but high demands on computing resources limit its practicality. We address this issue by introducing a prompt-based parameter-efficient fine-tuning (PEFT) approach. GNNavi leverages insights into ICL's information flow dynamics, which indicates that label words act in prompts as anchors for information propagation. GNNavi employs a Graph Neural Network (GNN) layer to precisely guide the aggregation and distribution of information flow during the processing of prompts by hardwiring the desired information flow into the GNN. Our experiments on text classification tasks with GPT-2 and Llama2 show GNNavi surpasses standard prompt-based fine-tuning methods in few-shot settings by updating just 0.2% to 0.5% of parameters. We compare GNNavi with prevalent PEFT approaches, such as prefix tuning, LoRA and Adapter in terms of performance and efficiency. Our analysis reveals that GNNavi enhances information flow and ensures a clear aggregation process.
Deployment of Microcells by integrating LTE-U with LTE
Venue: arXiv cs.NI 2024
Year: 2024

With increasing data requirements of users, cellular operators are finding new ways to fulfil these requirements. These attempts involve the practice of deploying Wi-Fi access points nearer to the user and backhauling it to the nearest eNB (in case of LTE and LTE-A). The paper studies LTE-U, an extension of LTE which works in the unlicensed spectrum, as a potential solution to this problem. It is based on the idea of densification. Network deployments incorporating LTE-U will be able to better cater to the growing data rate demand of voice and video, thus reducing the load on eNB. Further we explore the possibility of LTE-U as an alternative to Wi-Fi or co-existing with Wi-Fi deployments and issues revolving around this idea. We show that LTE-U deployment solves the problem of capacity in both cases.
Shaping Human-AI Collaboration: Varied Scaffolding Levels in Co-writing with Language Models
Venue: arXiv cs.HC 2024
Year: 2024

Advances in language modeling have paved the way for novel human-AI co-writing experiences. This paper explores how varying levels of scaffolding from large language models (LLMs) shape the co-writing process. Employing a within-subjects field experiment with a Latin square design, we asked participants (N=131) to respond to argumentative writing prompts under three randomly sequenced conditions: no AI assistance (control), next-sentence suggestions (low scaffolding), and next-paragraph suggestions (high scaffolding). Our findings reveal a U-shaped impact of scaffolding on writing quality and productivity (words/time). While low scaffolding did not significantly improve writing quality or productivity, high scaffolding led to significant improvements, especially benefiting non-regular writers and less tech-savvy users. No significant cognitive burden was observed while using the scaffolded writing tools, but a moderate decrease in text ownership and satisfaction was noted. Our results have broad implications for the design of AI-powered writing tools, including the need for personalized scaffolding mechanisms.
Compression Repair for Feedforward Neural Networks Based on Model Equivalence Evaluation
Venue: arXiv cs.LG 2024
Year: 2024

In this paper, we propose a method of repairing compressed Feedforward Neural Networks (FNNs) based on equivalence evaluation of two neural networks. In the repairing framework, a novel neural network equivalence evaluation method is developed to compute the output discrepancy between two neural networks. The output discrepancy can quantitatively characterize the output difference produced by compression procedures. Based on the computed output discrepancy, the repairing method first initializes a new training set for the compressed networks to narrow down the discrepancy between the two neural networks and improve the performance of the compressed network. Then, we repair the compressed FNN by re-training based on the training set. We apply our developed method to the MNIST dataset to demonstrate the effectiveness and advantages of our proposed repair method.