OpenRaiser/Intern-Atlas huggingface dataset:
LongRAG: Enhancing Retrieval-Augmented Generation with Long-context LLMs
Venue: arXiv cs.CL 2024
Year: 2024

In traditional RAG framework, the basic retrieval units are normally short. The common retrievers like DPR normally work with 100-word Wikipedia paragraphs. Such a design forces the retriever to search over a large corpus to find the `needle' unit. In contrast, the readers only need to generate answers from the short retrieved units. The imbalanced `heavy' retriever and `light' reader design can lead to sub-optimal performance. The loss of contextual information in the short, chunked units may increase the likelihood of introducing hard negatives during the retrieval stage. Additionally, the reader might not fully leverage the capabilities of recent advancements in LLMs. In order to alleviate the imbalance, we propose a new framework LongRAG, consisting of a `long retriever' and a `long reader'. In the two Wikipedia-based datasets, NQ and HotpotQA, LongRAG processes the entire Wikipedia corpus into 4K-token units by grouping related documents. By increasing the unit size, we significantly reduce the total number of units. This greatly reduces the burden on the retriever, resulting in strong retrieval performance with only a few (less than 8) top units. Without requiring any training, LongRAG achieves an EM of 62.7% on NQ and 64.3% on HotpotQA, which are on par with the (fully-trained) SoTA model. Furthermore, we test on two non-Wikipedia-based datasets, Qasper and MultiFieldQA-en. LongRAG processes each individual document as a single (long) unit rather than chunking them into smaller units. By doing so, we achieve an F1 score of 25.9% on Qasper and 57.5% on MultiFieldQA-en. Our study offers insights into the future roadmap for combining RAG with long-context LLMs.
Rethinking Remote Sensing Change Detection With A Mask View
Venue: arXiv cs.CV 2024
Year: 2024

Remote sensing change detection aims to compare two or more images recorded for the same area but taken at different time stamps to quantitatively and qualitatively assess changes in geographical entities and environmental factors. Mainstream models usually built on pixel-by-pixel change detection paradigms, which cannot tolerate the diversity of changes due to complex scenes and variation in imaging conditions. To address this shortcoming, this paper rethinks the change detection with the mask view, and further proposes the corresponding: 1) meta-architecture CDMask and 2) instance network CDMaskFormer. Components of CDMask include Siamese backbone, change extractor, pixel decoder, transformer decoder and normalized detector, which ensures the proper functioning of the mask detection paradigm. Since the change query can be adaptively updated based on the bi-temporal feature content, the proposed CDMask can adapt to different latent data distributions, thus accurately identifying regions of interest changes in complex scenarios. Consequently, we further propose the instance network CDMaskFormer customized for the change detection task, which includes: (i) a Spatial-temporal convolutional attention-based instantiated change extractor to capture spatio-temporal context simultaneously with lightweight operations; and (ii) a scene-guided axial attention-instantiated transformer decoder to extract more spatial details. State-of-the-art performance of CDMaskFormer is achieved on five benchmark datasets with a satisfactory efficiency-accuracy trade-off. Code is available at https://github.com/xwmaxwma/rschange.
Fine-grained Attention in Hierarchical Transformers for Tabular Time-series
Venue: arXiv cs.LG 2024
Year: 2024

Tabular data is ubiquitous in many real-life systems. In particular, time-dependent tabular data, where rows are chronologically related, is typically used for recording historical events, e.g., financial transactions, healthcare records, or stock history. Recently, hierarchical variants of the attention mechanism of transformer architectures have been used to model tabular time-series data. At first, rows (or columns) are encoded separately by computing attention between their fields. Subsequently, encoded rows (or columns) are attended to one another to model the entire tabular time-series. While efficient, this approach constrains the attention granularity and limits its ability to learn patterns at the field-level across separate rows, or columns. We take a first step to address this gap by proposing Fieldy, a fine-grained hierarchical model that contextualizes fields at both the row and column levels. We compare our proposal against state of the art models on regression and classification tasks using public tabular time-series datasets. Our results show that combining row-wise and column-wise attention improves performance without increasing model size. Code and data are available at https://github.com/raphaaal/fieldy.
Masked Extended Attention for Zero-Shot Virtual Try-On In The Wild
Venue: arXiv cs.CV 2024
Year: 2024

Virtual Try-On (VTON) is a highly active line of research, with increasing demand. It aims to replace a piece of garment in an image with one from another, while preserving person and garment characteristics as well as image fidelity. Current literature takes a supervised approach for the task, impairing generalization and imposing heavy computation. In this paper, we present a novel zero-shot training-free method for inpainting a clothing garment by reference. Our approach employs the prior of a diffusion model with no additional training, fully leveraging its native generalization capabilities. The method employs extended attention to transfer image information from reference to target images, overcoming two significant challenges. We first initially warp the reference garment over the target human using deep features, alleviating "texture sticking". We then leverage the extended attention mechanism with careful masking, eliminating leakage of reference background and unwanted influence. Through a user study, qualitative, and quantitative comparison to state-of-the-art approaches, we demonstrate superior image quality and garment preservation compared unseen clothing pieces or human figures.
Introducing Syllable Tokenization for Low-resource Languages: A Case Study with Swahili
Venue: arXiv cs.CL 2024
Year: 2024

Many attempts have been made in multilingual NLP to ensure that pre-trained language models, such as mBERT or GPT2 get better and become applicable to low-resource languages. To achieve multilingualism for pre-trained language models (PLMs), we need techniques to create word embeddings that capture the linguistic characteristics of any language. Tokenization is one such technique because it allows for the words to be split based on characters or subwords, creating word embeddings that best represent the structure of the language. Creating such word embeddings is essential to applying PLMs to other languages where the model was not trained, enabling multilingual NLP. However, most PLMs use generic tokenization methods like BPE, wordpiece, or unigram which may not suit specific languages. We hypothesize that tokenization based on syllables within the input text, which we call syllable tokenization, should facilitate the development of syllable-aware language models. The syllable-aware language models make it possible to apply PLMs to languages that are rich in syllables, for instance, Swahili. Previous works introduced subword tokenization. Our work extends such efforts. Notably, we propose a syllable tokenizer and adopt an experiment-centric approach to validate the proposed tokenizer based on the Swahili language. We conducted text-generation experiments with GPT2 to evaluate the effectiveness of the syllable tokenizer. Our results show that the proposed syllable tokenizer generates syllable embeddings that effectively represent the Swahili language.
The TikToking troll and weaponization of conscience: A systems perspective case study
Venue: arXiv cs.CY 2024
Year: 2024

Cybercrime is a pervasive threat that impacts every facet of society. Its reach transcends geographic borders and extends far beyond the digital realm, often serving as the catalyst for offline crimes. As modern conflicts become increasingly intertwined with cyber warfare, the need for interdisciplinary cooperation to grasp and combat this escalating threat is paramount. This case study centers around a controversial TikToker, highlighting how the weaponization of conscience can be leveraged to manipulate multiple actors within a propagandist's target population. Weaponization of conscience is a tactic used by fraudsters to camouflage their activity, deceive their victims, and extend the effectiveness of their modi operandi. Research shows that 95 percent of cybersecurity incidents are the result of human error and 90 percent begin with a phishing attempt. Honing the capacity to identify and dissect strategies employed by fraudsters along with how individual reactions unfold in the larger system is an essential skill for organizations and individuals to safeguard themselves. Understanding cybercrime and its many interconnected systems requires examination through the lens of complexity science.
Barriers facing e-service adopting and implementation at local environment level in Nigeria
Venue: arXiv cs.CY 2024
Year: 2024

E-Government services offer a great deal of potential to improve government activities and citizen support. However, there is a lack of research covering E-Government services at the local government level, particularly in developing countries. However, implementing successful E-Service technology in this part of the world will not come without its barriers considering the unstable and fragile economies in most developing countries. The research aim is to identify the barriers facing E-Service adoption and implementation at a local environment level, using Nigeria as a case example. This thesis adopts an interpretive paradigm and uses action research. It consists of a large field study in Nigeria (interviews), an online survey of government officials, online focus groups, and analyses government documents and E-Service initiatives. A structured literature review method consisted of sifting through 3,245 papers. The main theoretical tools used in this thesis are the diffusion of innovation (DOI) theory and the theory of change.
RadEx: A Framework for Structured Information Extraction from Radiology Reports based on Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Annually and globally, over three billion radiography examinations and computer tomography scans result in mostly unstructured radiology reports containing free text. Despite the potential benefits of structured reporting, its adoption is limited by factors such as established processes, resource constraints and potential loss of information. However, structured information would be necessary for various use cases, including automatic analysis, clinical trial matching, and prediction of health outcomes. This study introduces RadEx, an end-to-end framework comprising 15 software components and ten artifacts to develop systems that perform automated information extraction from radiology reports. It covers the complete process from annotating training data to extracting information by offering a consistent generic information model and setting boundaries for model development. Specifically, RadEx allows clinicians to define relevant information for clinical domains (e.g., mammography) and to create report templates. The framework supports both generative and encoder-only models and the decoupling of information extraction from template filling enables independent model improvements. Developing information extraction systems according to the RadEx framework facilitates implementation and maintenance as components are easily exchangeable, while standardized artifacts ensure interoperability between components.
On the Principles behind Opinion Dynamics in Multi-Agent Systems of Large Language Models
Venue: arXiv cs.MA 2024
Year: 2024

We study the evolution of opinions inside a population of interacting large language models (LLMs). Every LLM needs to decide how much funding to allocate to an item with three initial possibilities: full, partial, or no funding. We identify biases that drive the exchange of opinions based on the LLM's tendency to find consensus with the other LLM's opinion, display caution when specifying funding, and consider ethical concerns in its opinion. We find these biases are affected by the perceived absence of compelling reasons for opinion change, the perceived willingness to engage in discussion, and the distribution of allocation values. Moreover, tensions among biases can lead to the survival of funding for items with negative connotations. We also find that the final distribution of full, partial, and no funding opinions is more diverse when an LLM freely forms its opinion after an interaction than when its opinion is a multiple-choice selection among the three allocation options. In the latter case, consensus is mostly attained. When agents are aware of past opinions, they seek to maintain consistency with them, changing the opinion dynamics. Our study is performed using Llama 3 and Mistral LLMs.
System Description for the Displace Speaker Diarization Challenge 2023
Venue: arXiv cs.CL 2024
Year: 2024

This paper describes our solution for the Diarization of Speaker and Language in Conversational Environments Challenge (Displace 2023). We used a combination of VAD for finding segfments with speech, Resnet architecture based CNN for feature extraction from these segments, and spectral clustering for features clustering. Even though it was not trained with using Hindi, the described algorithm achieves the following metrics: DER 27. 1% and DER 27. 4%, on the development and phase-1 evaluation parts of the dataset, respectively.
Cyclic Scheduler Design for Minimizing Age of Information in Massive Scale Networks Susceptible to Packet Errors
Venue: arXiv cs.IT 2024
Year: 2024

In multi-source status update systems, sources need to be scheduled appropriately to maintain timely communication between each of the sources and the monitor. A cyclic schedule is an age-agnostic schedule in which the sources are served according to a fixed finite transmission pattern, which upon completion, repeats itself. Such a scheme has a low $O(1)$ runtime complexity, which is desirable in large networks. This paper's focus is on designing transmission patterns so as to be used in massive scale networking scenarios involving a very large number of sources, e.g., up to thousands of IoT sources, with service time requirements and weights being heterogeneous in nature. The goal is to minimize the weighted sum age of information (AoI), called weighted AoI, when transmitting users' packets over a channel susceptible to heterogeneous packet errors. The main tool we use is a stochastic modeling framework using either Markov chains (MC) or moment generating functions (MGF), by which we obtain the weighted AoI for a given transmission pattern, which is not straightforward in the presence of packet drops. Using this framework, we provide a lower bound on the weighted AoI for the particular case of two sources, and also an algorithm to attain this lower bound. Then, by using the same framework, we design a cyclic scheduler for general number of sources with reasonable complexity using convex optimization and well-established packet spreading algorithms, and comparatively evaluate the proposed algorithm and existing age-agnostic scheduling schemes for general number of sources (resp.~two sources) when the lower bound is not available (resp.~when it is available). We present extensive numerical results to validate the effectiveness of the proposed approach.
How to Rent GPUs on a Budget
Venue: arXiv cs.DC 2024
Year: 2024

The explosion in Machine Learning (ML) over the past ten years has led to a dramatic increase in demand for GPUs to train ML models. Because it is prohibitively expensive for most users to build and maintain a large GPU cluster, large cloud providers (Microsoft Azure, Amazon AWS, Google Cloud) have seen explosive growth in demand for renting cloud-based GPUs. In this cloud-computing paradigm, a user must specify their demand for GPUs at every moment in time, and will pay for every GPU-hour they use. ML training jobs are known to be parallelizable to different degrees. Given a stream of ML training jobs, a user typically wants to minimize the mean response time across all jobs. Here, the response time of a job denotes the time from when a job arrives until it is complete. Additionally, the user is constrained by some operating budget. Specifically, in this paper the user is constrained to use no more than $b$ GPUs per hour, over a long-run time average. The question is how to minimize mean response time while meeting the budget constraint. Because training jobs receive a diminishing marginal benefit from running on additional GPUs, allocating too many GPUs to a single training job can dramatically increase the overall cost paid by the user. Hence, an optimal rental policy must balance a tradeoff between training cost and mean response time. This paper derives the optimal rental policy for a stream of training jobs where the jobs have different levels of parallelizability (specified by a speedup function) and different job sizes (amounts of inherent work). We make almost no assumptions about the arrival process and about the job size distribution. Our optimal policy specifies how many GPUs to rent at every moment in time and how to allocate these GPUs.
Unseen Object Reasoning with Shared Appearance Cues
Venue: arXiv cs.CV 2024
Year: 2024

This paper introduces an innovative approach to open world recognition (OWR), where we leverage knowledge acquired from known objects to address the recognition of previously unseen objects. The traditional method of object modeling relies on supervised learning with strict closed-set assumptions, presupposing that objects encountered during inference are already known at the training phase. However, this assumption proves inadequate for real-world scenarios due to the impracticality of accounting for the immense diversity of objects. Our hypothesis posits that object appearances can be represented as collections of "shareable" mid-level features, arranged in constellations to form object instances. By adopting this framework, we can efficiently dissect and represent both known and unknown objects in terms of their appearance cues. Our paper introduces a straightforward yet elegant method for modeling novel or unseen objects, utilizing established appearance cues and accounting for inherent uncertainties. This representation not only enables the detection of out-of-distribution objects or novel categories among unseen objects but also facilitates a deeper level of reasoning, empowering the identification of the superclass to which an unknown instance belongs. This novel approach holds promise for advancing open world recognition in diverse applications.
Physics Informed Machine Learning (PIML) methods for estimating the remaining useful lifetime (RUL) of aircraft engines
Venue: arXiv cs.LG 2024
Year: 2024

This paper is aimed at using the newly developing field of physics informed machine learning (PIML) to develop models for predicting the remaining useful lifetime (RUL) aircraft engines. We consider the well-known benchmark NASA Commercial Modular Aero-Propulsion System Simulation (C-MAPSS) data as the main data for this paper, which consists of sensor outputs in a variety of different operating modes. C-MAPSS is a well-studied dataset with much existing work in the literature that address RUL prediction with classical and deep learning methods. In the absence of published empirical physical laws governing the C-MAPSS data, our approach first uses stochastic methods to estimate the governing physics models from the noisy time series data. In our approach, we model the various sensor readings as being governed by stochastic differential equations, and we estimate the corresponding transition density mean and variance functions of the underlying processes. We then augment LSTM (long-short term memory) models with the learned mean and variance functions during training and inferencing. Our PIML based approach is different from previous methods, and we use the data to first learn the physics. Our results indicate that PIML discovery and solutions methods are well suited for this problem and outperform previous data-only deep learning methods for this data set and task. Moreover, the framework developed herein is flexible, and can be adapted to other situations (other sensor modalities or combined multi-physics environments), including cases where the underlying physics is only partially observed or known.
Good things come in three: Generating SO Post Titles with Pre-Trained Models, Self Improvement and Post Ranking
Venue: arXiv cs.SE 2024
Year: 2024

Stack Overflow is a prominent Q and A forum, supporting developers in seeking suitable resources on programming-related matters. Having high-quality question titles is an effective means to attract developers' attention. Unfortunately, this is often underestimated, leaving room for improvement. Research has been conducted, predominantly leveraging pre-trained models to generate titles from code snippets and problem descriptions. Yet, getting high-quality titles is still a challenging task, attributed to both the quality of the input data (e.g., containing noise and ambiguity) and inherent constraints in sequence generation models. In this paper, we present FILLER as a solution to generating Stack Overflow post titles using a fine-tuned language model with self-improvement and post ranking. Our study focuses on enhancing pre-trained language models for generating titles for Stack Overflow posts, employing a training and subsequent fine-tuning paradigm for these models. To this end, we integrate the model's predictions into the training process, enabling it to learn from its errors, thereby lessening the effects of exposure bias. Moreover, we apply a post-ranking method to produce a variety of sample candidates, subsequently selecting the most suitable one. To evaluate FILLER, we perform experiments using benchmark datasets, and the empirical findings indicate that our model provides high-quality recommendations. Moreover, it significantly outperforms all the baselines, including Code2Que, SOTitle, CCBERT, M3NSCT5, and GPT3.5-turbo. A user study also shows that FILLER provides more relevant titles, with respect to SOTitle and GPT3.5-turbo.
Simple Games on Complex Networks
Venue: arXiv cs.SI 2024
Year: 2024

The relationship between topology and dynamics of complex systems has motivated continuing interest from the scientific community. In the present work, we address this interesting topic from the perspective of simple games, involving two teams playing according to a small set of simple rules, taking place on four types of complex networks. Starting from a minimalist game, characterized by full symmetry always leading to ties, four other games are described in progressive order of complexity, taking into account the presence of neighbors as well as strategies. Each of these five games, as well as their specific changes when implemented in four types of networks, are studied in terms of statistics of the total duration of the game as well as the number of victories and ties, with several interesting results that substantiate, in some cases, the importance of the network topology on the respective dynamics. As a subsidiary result, the visualization of relationships between the data elements in terms of coincidence similarity networks allowed a more complete and direct interpretation of the obtained results.