OpenRaiser/Intern-Atlas huggingface dataset:
Formulation Comparison for Timeline Construction using LLMs
Venue: arXiv cs.CL 2024
Year: 2024

Constructing a timeline requires identifying the chronological order of events in an article. In prior timeline construction datasets, temporal orders are typically annotated by either event-to-time anchoring or event-to-event pairwise ordering, both of which suffer from missing temporal information. To mitigate the issue, we develop a new evaluation dataset, TimeSET, consisting of single-document timelines with document-level order annotation. TimeSET features saliency-based event selection and partial ordering, which enable a practical annotation workload. Aiming to build better automatic timeline construction systems, we propose a novel evaluation framework to compare multiple task formulations with TimeSET by prompting open LLMs, i.e., Llama 2 and Flan-T5. Considering that identifying temporal orders of events is a core subtask in timeline construction, we further benchmark open LLMs on existing event temporal ordering datasets to gain a robust understanding of their capabilities. Our experiments show that (1) NLI formulation with Flan-T5 demonstrates a strong performance among others, while (2) timeline construction and event temporal ordering are still challenging tasks for few-shot LLMs. Our code and data are available at https://github.com/kimihiroh/timeset.
Advancing parabolic operators in thermodynamic MHD models II: Evaluating a Practical Time Step Limit for Unconditionally Stable Methods
Venue: arXiv cs.CE 2024
Year: 2024

Unconditionally stable time stepping schemes are useful and often practically necessary for advancing parabolic operators in multi-scale systems. However, serious accuracy problems may emerge when taking time steps that far exceed the explicit stability limits. In our previous work, we compared the accuracy and performance of advancing parabolic operators in a thermodynamic MHD model using an implicit method and an explicit super time-stepping (STS) method. We found that while the STS method outperformed the implicit one with overall good results, it was not able to damp oscillatory behavior in the solution efficiently, hindering its practical use. In this follow-up work, we evaluate an easy-to-implement method for selecting a practical time step limit (PTL) for unconditionally stable schemes. This time step is used to `cycle' the operator-split thermal conduction and viscosity parabolic operators. We test the new time step with both an implicit and STS scheme for accuracy, performance, and scaling. We find that, for our test cases here, the PTL dramatically improves the STS solution, matching or improving the solution of the original implicit scheme, while retaining most of its performance and scaling advantages. The PTL shows promise to allow more accurate use of unconditionally stable schemes for parabolic operators and reliable use of STS methods.
BasedAI: A decentralized P2P network for Zero Knowledge Large Language Models (ZK-LLMs)
Venue: arXiv cs.CR 2024
Year: 2024

BasedAI is a distributed network of machines which introduces decentralized infrastructure capable of integrating Fully Homomorphic Encryption (FHE) with any large language model (LLM) connected to its network. The proposed framework embeds a default mechanism, called "Cerberus Squeezing", into the mining process which enables the transformation of a standard LLMs into encrypted zero-knowledge LLMs, or "ZK-LLMs", leveraging insights from generative adversarial networks for data privacy. This novel quantization mechanism empowers BasedAI miners to process and respond to prompts derived from User interaction with LLMs without the need for decrypting either the queries or their corresponding responses. The introduction of Cerberus Squeezing significantly improves performance degradation caused by quantized functions in current FHE-compliant computing environments by proactively optimizing calls between users, miners, and validators.
Nonlinear dynamics and stability analysis of locally-active Mott memristors using a physics-based compact model
Venue: arXiv cs.ET 2024
Year: 2024

Locally-active memristors are a class of emerging nonlinear dynamic circuit elements that hold promise for scalable yet biomimetic neuromorphic circuits. Starting from a physics-based compact model, we performed small-signal linearization analyses and applied Chua's local activity theory to a one-dimensional locally-active vanadium dioxide Mott memristor based on an insulator-to-metal phase transition. This approach allows a connection between the dynamical behaviors of a Mott memristor and its physical device parameters as well as a complete mapping of the locally passive and edge of chaos domains in the frequency and current operating parameter space, which could guide materials and device development for neuromorphic circuit applications. We also examined the applicability of local analyses on a second-order relaxation oscillator circuit that consists of a voltage-biased vanadium dioxide memristor coupled to a parallel reactive capacitor element and a series resistor. We show that global nonlinear techniques, including nullclines and phase portraits, provide insights on instabilities and persistent oscillations near non-hyperbolic fixed points, such as a supercritical Hopf-like bifurcation from an unstable spiral to a stable limit cycle, with each of the three circuit parameters acting as a bifurcation parameter. The abruptive growth in the limit cycle resembles the Canard explosion phenomenon in systems exhibiting relaxation oscillations. Finally, we show that experimental limit cycle oscillations in a vanadium dioxide nano-device relaxation oscillator match well with SPICE simulations built upon the compact model.
A Library of Mirrors: Deep Neural Nets in Low Dimensions are Convex Lasso Models with Reflection Features
Venue: arXiv cs.LG 2024
Year: 2024

We prove that training neural networks on 1-D data is equivalent to solving convex Lasso problems with discrete, explicitly defined dictionary matrices. We consider neural networks with piecewise linear activations and depths ranging from 2 to an arbitrary but finite number of layers. We first show that two-layer networks with piecewise linear activations are equivalent to Lasso models using a discrete dictionary of ramp functions, with breakpoints corresponding to the training data points. In certain general architectures with absolute value or ReLU activations, a third layer surprisingly creates features that reflect the training data about themselves. Additional layers progressively generate reflections of these reflections. The Lasso representation provides valuable insights into the analysis of globally optimal networks, elucidating their solution landscapes and enabling closed-form solutions in certain special cases. Numerical results show that reflections also occur when optimizing standard deep networks using standard non-convex optimizers. Additionally, we demonstrate our theory with autoregressive time series models.
GraphRCG: Self-Conditioned Graph Generation
Venue: arXiv cs.LG 2024
Year: 2024

Graph generation generally aims to create new graphs that closely align with a specific graph distribution. Existing works often implicitly capture this distribution through the optimization of generators, potentially overlooking the intricacies of the distribution itself. Furthermore, these approaches generally neglect the insights offered by the learned distribution for graph generation. In contrast, in this work, we propose a novel self-conditioned graph generation framework designed to explicitly model graph distributions and employ these distributions to guide the generation process. We first perform self-conditioned modeling to capture the graph distributions by transforming each graph sample into a low-dimensional representation and optimizing a representation generator to create new representations reflective of the learned distribution. Subsequently, we leverage these bootstrapped representations as self-conditioned guidance for the generation process, thereby facilitating the generation of graphs that more accurately reflect the learned distributions. We conduct extensive experiments on generic and molecular graph datasets across various fields. Our framework demonstrates superior performance over existing state-of-the-art graph generation methods in terms of graph quality and fidelity to training data.
Teaching MLP More Graph Information: A Three-stage Multitask Knowledge Distillation Framework
Venue: arXiv cs.LG 2024
Year: 2024

We study the challenging problem for inference tasks on large-scale graph datasets of Graph Neural Networks: huge time and memory consumption, and try to overcome it by reducing reliance on graph structure. Even though distilling graph knowledge to student MLP is an excellent idea, it faces two major problems of positional information loss and low generalization. To solve the problems, we propose a new three-stage multitask distillation framework. In detail, we use Positional Encoding to capture positional information. Also, we introduce Neural Heat Kernels responsible for graph data processing in GNN and utilize hidden layer outputs matching for better performance of student MLP's hidden layers. To the best of our knowledge, it is the first work to include hidden layer distillation for student MLP on graphs and to combine graph Positional Encoding with MLP. We test its performance and robustness with several settings and draw the conclusion that our work can outperform well with good stability.
Towards Accurate Lip-to-Speech Synthesis in-the-Wild
Venue: arXiv cs.MM 2024
Year: 2024

In this paper, we introduce a novel approach to address the task of synthesizing speech from silent videos of any in-the-wild speaker solely based on lip movements. The traditional approach of directly generating speech from lip videos faces the challenge of not being able to learn a robust language model from speech alone, resulting in unsatisfactory outcomes. To overcome this issue, we propose incorporating noisy text supervision using a state-of-the-art lip-to-text network that instills language information into our model. The noisy text is generated using a pre-trained lip-to-text model, enabling our approach to work without text annotations during inference. We design a visual text-to-speech network that utilizes the visual stream to generate accurate speech, which is in-sync with the silent input video. We perform extensive experiments and ablation studies, demonstrating our approach's superiority over the current state-of-the-art methods on various benchmark datasets. Further, we demonstrate an essential practical application of our method in assistive technology by generating speech for an ALS patient who has lost the voice but can make mouth movements. Our demo video, code, and additional details can be found at \url{http://cvit.iiit.ac.in/research/projects/cvit-projects/ms-l2s-itw}.
MulCogBench: A Multi-modal Cognitive Benchmark Dataset for Evaluating Chinese and English Computational Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Pre-trained computational language models have recently made remarkable progress in harnessing the language abilities which were considered unique to humans. Their success has raised interest in whether these models represent and process language like humans. To answer this question, this paper proposes MulCogBench, a multi-modal cognitive benchmark dataset collected from native Chinese and English participants. It encompasses a variety of cognitive data, including subjective semantic ratings, eye-tracking, functional magnetic resonance imaging (fMRI), and magnetoencephalography (MEG). To assess the relationship between language models and cognitive data, we conducted a similarity-encoding analysis which decodes cognitive data based on its pattern similarity with textual embeddings. Results show that language models share significant similarities with human cognitive data and the similarity patterns are modulated by the data modality and stimuli complexity. Specifically, context-aware models outperform context-independent models as language stimulus complexity increases. The shallow layers of context-aware models are better aligned with the high-temporal-resolution MEG signals whereas the deeper layers show more similarity with the high-spatial-resolution fMRI. These results indicate that language models have a delicate relationship with brain language representations. Moreover, the results between Chinese and English are highly consistent, suggesting the generalizability of these findings across languages.
LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization
Venue: arXiv cs.LG 2024
Year: 2024

Recent breakthroughs in Large-scale language models (LLMs) have demonstrated impressive performance on various tasks. The immense sizes of LLMs have led to very high resource demand and cost for running the models. Though the models are largely served using uniform high-caliber GPUs nowadays, utilizing a heterogeneous cluster with a mix of available high- and low-capacity GPUs can potentially substantially reduce the serving cost. There is a lack of designs to support efficient LLM serving using a heterogeneous cluster, while the current solutions focus on model partition and uniform compression among homogeneous devices. This paper proposes LLM-PQ, a system that advocates adaptive model quantization and phase-aware partition to improve LLM serving efficiency on heterogeneous GPU clusters. We carefully decide on mixed-precision model quantization together with phase-aware model partition and micro-batch sizing in distributed LLM serving with an efficient algorithm, to greatly enhance inference throughput while fulfilling user-specified model quality targets. Extensive experiments on production inference workloads in 11 different clusters demonstrate that LLM-PQ achieves up to 2.88x (2.26x on average) throughput improvement in inference, showing great advantages over state-of-the-art works.
A Hybrid Model for Traffic Incident Detection based on Generative Adversarial Networks and Transformer Model
Venue: arXiv cs.LG 2024
Year: 2024

In addition to enhancing traffic safety and facilitating prompt emergency response, traffic incident detection plays an indispensable role in intelligent transportation systems by providing real-time traffic status information. This enables the realization of intelligent traffic control and management. Previous research has identified that apart from employing advanced algorithmic models, the effectiveness of detection is also significantly influenced by challenges related to acquiring large datasets and addressing dataset imbalances. A hybrid model combining transformer and generative adversarial networks (GANs) is proposed to address these challenges. Experiments are conducted on four real datasets to validate the superiority of the transformer in traffic incident detection. Additionally, GANs are utilized to expand the dataset and achieve a balanced ratio of 1:4, 2:3, and 1:1. The proposed model is evaluated against the baseline model. The results demonstrate that the proposed model enhances the dataset size, balances the dataset, and improves the performance of traffic incident detection in various aspects.
Misconfiguration in O-RAN: Analysis of the impact of AI/ML
Venue: arXiv cs.NI 2024
Year: 2024

User demand on network communication infrastructure has never been greater with applications such as extended reality, holographic telepresence, and wireless brain-computer interfaces challenging current networking capabilities. Open RAN (O-RAN) is critical to supporting new and anticipated uses of 6G and beyond. It promotes openness and standardisation, increased flexibility through the disaggregation of Radio Access Network (RAN) components, supports programmability, flexibility, and scalability with technologies such as Software-Defined Networking (SDN), Network Function Virtualization (NFV), and cloud, and brings automation through the RAN Intelligent Controller (RIC). Furthermore, the use of xApps, rApps, and Artificial Intelligence/Machine Learning (AI/ML) within the RIC enables efficient management of complex RAN operations. However, due to the open nature of O-RAN and its support for heterogeneous systems, the possibility of misconfiguration problems becomes critical. In this paper, we present a thorough analysis of the potential misconfiguration issues in O-RAN with respect to integration and operation, the use of SDN and NFV, and, specifically, the use of AI/ML. The opportunity for AI/ML to be used to identify these misconfigurations is investigated. A case study is presented to illustrate the direct impact on the end user of conflicting policies amongst xApps along with a potential AI/ML-based solution to this problem. This research presents a first analysis of the impact of AI/ML on misconfiguration challenges in O-RAN.
Shaping Multi-Robot Patrol Performance with Heterogeneity in Individual Learning Behavior
Venue: arXiv cs.RO 2024
Year: 2024

Individual differences in learning behavior within social groups, whether in humans, other animals, or among robots, can have significant effects on collective task performance. This is because it can affect individuals' response to the environment and their interactions with each other. In recent years there has been rising interest in the question of how individual differences, whether in learning or other traits, affect collective outcomes: studied, for example, in social insect foraging behavior. Multi-robot, 'swarm' systems have a heritage of bioinspiration from such examples, and here we consider whether heterogeneity in a learning behavior called latent inhibition (LI) may be useful for a team of patrolling robots tasked with environmental monitoring and anomaly detection. Individuals with high LI can be seen as better at learning to be inattentive to irrelevant or unrewarding stimuli, while low LI individuals might be seen as 'distractible' and yet, more positively, more exploratory. We introduce a simple model of the effects of LI as the probability of re-searching a location for a reward (anomalous reading) where it has previously been found to be unrewarding (irrelevant). In simulated patrols, we find that a negatively skewed distribution of mostly high LI robots, and just a single low LI robot, is collectively most effective at monitoring dynamic environments. These results are an example of 'functional heterogeneity' in 'swarm engineering' and could inform predictions for ecological distributions of learning traits within social groups.
d-DSE: Distinct Dynamic Searchable Encryption Resisting Volume Leakage in Encrypted Databases
Venue: arXiv cs.CR 2024
Year: 2024

Dynamic Searchable Encryption (DSE) has emerged as a solution to efficiently handle and protect large-scale data storage in encrypted databases (EDBs). Volume leakage poses a significant threat, as it enables adversaries to reconstruct search queries and potentially compromise the security and privacy of data. Padding strategies are common countermeasures for the leakage, but they significantly increase storage and communication costs. In this work, we develop a new perspective to handle volume leakage. We start with distinct search and further explore a new concept called \textit{distinct} DSE (\textit{d}-DSE). We also define new security notions, in particular Distinct with Volume-Hiding security, as well as forward and backward privacy, for the new concept. Based on \textit{d}-DSE, we construct the \textit{d}-DSE designed EDB with related constructions for distinct keyword (d-KW-\textit{d}DSE), keyword (KW-\textit{d}DSE), and join queries (JOIN-\textit{d}DSE) and update queries in encrypted databases. We instantiate a concrete scheme \textsf{BF-SRE}, employing Symmetric Revocable Encryption. We conduct extensive experiments on real-world datasets, such as Crime, Wikipedia, and Enron, for performance evaluation. The results demonstrate that our scheme is practical in data search and with comparable computational performance to the SOTA DSE scheme (\textsf{MITRA}*, \textsf{AURA}) and padding strategies (\textsf{SEAL}, \textsf{ShieldDB}). Furthermore, our proposal sharply reduces the communication cost as compared to padding strategies, with roughly 6.36 to 53.14x advantage for search queries.
Machine Translation in the Covid domain: an English-Irish case study for LoResMT 2021
Venue: arXiv cs.CL 2024
Year: 2024

Translation models for the specific domain of translating Covid data from English to Irish were developed for the LoResMT 2021 shared task. Domain adaptation techniques, using a Covid-adapted generic 55k corpus from the Directorate General of Translation, were applied. Fine-tuning, mixed fine-tuning and combined dataset approaches were compared with models trained on an extended in-domain dataset. As part of this study, an English-Irish dataset of Covid related data, from the Health and Education domains, was developed. The highest-performing model used a Transformer architecture trained with an extended in-domain Covid dataset. In the context of this study, we have demonstrated that extending an 8k in-domain baseline dataset by just 5k lines improved the BLEU score by 27 points.
Boosting Box-supervised Instance Segmentation with Pseudo Depth
Venue: arXiv cs.CV 2024
Year: 2024

The realm of Weakly Supervised Instance Segmentation (WSIS) under box supervision has garnered substantial attention, showcasing remarkable advancements in recent years. However, the limitations of box supervision become apparent in its inability to furnish effective information for distinguishing foreground from background within the specified target box. This research addresses this challenge by introducing pseudo-depth maps into the training process of the instance segmentation network, thereby boosting its performance by capturing depth differences between instances. These pseudo-depth maps are generated using a readily available depth predictor and are not necessary during the inference stage. To enable the network to discern depth features when predicting masks, we integrate a depth prediction layer into the mask prediction head. This innovative approach empowers the network to simultaneously predict masks and depth, enhancing its ability to capture nuanced depth-related information during the instance segmentation process. We further utilize the mask generated in the training process as supervision to distinguish the foreground from the background. When selecting the best mask for each box through the Hungarian algorithm, we use depth consistency as one calculation cost item. The proposed method achieves significant improvements on Cityscapes and COCO dataset.
Results and Lessons Learned from Autonomous Driving Transportation Services in Airfield, Crowded Indoor, and Urban Environments
Venue: arXiv cs.RO 2024
Year: 2024

Autonomous vehicles have been actively investigated over the past few decades. Several recent works show the potential of autonomous vehicles in urban environments with impressive experimental results. However, these works note that autonomous vehicles are still occasionally inferior to expert drivers in complex scenarios. Furthermore, they do not focus on the possibilities of autonomous driving transportation services in other areas beyond urban environments. This paper presents the research results and lessons learned from autonomous driving transportation services in airfield, crowded indoor, and urban environments. We discuss how we address several unique challenges in these diverse environments. We also offer an overview of remaining challenges that have not received much attention but must be addressed. This paper aims to share our unique experience to support researchers who are interested in exploring autonomous driving transportation services in various real-world environments.