OpenRaiser/Intern-Atlas huggingface dataset:
LiVOS: Light Video Object Segmentation with Gated Linear Matching
Venue: arXiv cs.CV 2024
Year: 2024

Semi-supervised video object segmentation (VOS) has been largely driven by space-time memory (STM) networks, which store past frame features in a spatiotemporal memory to segment the current frame via softmax attention. However, STM networks face memory limitations due to the quadratic complexity of softmax matching, restricting their applicability as video length and resolution increase. To address this, we propose LiVOS, a lightweight memory network that employs linear matching via linear attention, reformulating memory matching into a recurrent process that reduces the quadratic attention matrix to a constant-size, spatiotemporal-agnostic 2D state. To enhance selectivity, we introduce gated linear matching, where a data-dependent gate matrix is multiplied with the state matrix to control what information to retain or discard. Experiments on diverse benchmarks demonstrated the effectiveness of our method. It achieved 64.8 J&F on MOSE and 85.1 J&F on DAVIS, surpassing all non-STM methods and narrowing the gap with STM-based approaches. For longer and higher-resolution videos, it matched STM-based methods with 53% less GPU memory and supports 4096p inference on a 32G consumer-grade GPU--a previously cost-prohibitive capability--opening the door for long and high-resolution video foundation models.
Enhancing EmoBot: An In-Depth Analysis of User Satisfaction and Faults in an Emotion-Aware Chatbot
Venue: arXiv cs.HC 2024
Year: 2024

The research community has traditionally shown a keen interest in emotion modeling, with a notable emphasis on the detection aspect. In contrast, the exploration of emotion generation has received less attention.This study delves into an existing state-of-the-art emotional chatbot, EmoBot, designed for generating emotions in general-purpose conversations. This research involves a comprehensive examination, including a survey to evaluate EmoBot's proficiency in key dimensions like usability, accuracy, and overall user satisfaction, with a specific focus on fault tolerance. By closely examining the chatbot's operations, we identified some noteworthy shortcomings in the existing model. We propose some solutions designed to address and overcome the identified issues.
PersianRAG: A Retrieval-Augmented Generation System for Persian Language
Venue: arXiv cs.CL 2024
Year: 2024

Retrieval augmented generation (RAG) models, which integrate large-scale pre-trained generative models with external retrieval mechanisms, have shown significant success in various natural language processing (NLP) tasks. However, applying RAG models in Persian language as a low-resource language, poses distinct challenges. These challenges primarily involve the preprocessing, embedding, retrieval, prompt construction, language modeling, and response evaluation of the system. In this paper, we address the challenges towards implementing a real-world RAG system for Persian language called PersianRAG. We propose novel solutions to overcome these obstacles and evaluate our approach using several Persian benchmark datasets. Our experimental results demonstrate the capability of the PersianRAG framework to enhance question answering task in Persian.
Metaheuristics for the Template Design Problem: Encoding, Symmetry and Hybridisation
Venue: arXiv cs.NE 2024
Year: 2024

The template design problem (TDP) is a hard combinatorial problem with a high number of symmetries which makes solving it more complicated. A number of techniques have been proposed in the literature to optimise its resolution, ranging from complete methods to stochastic ones. However, although metaheuristics are considered efficient methods that can find enough-quality solutions at a reasonable computational cost, these techniques have not proven to be truly efficient enough to deal with this problem. This paper explores and analyses a wide range of metaheuristics to tackle the problem with the aim of assessing their suitability for finding template designs. We tackle the problem using a wide set of metaheuristics whose implementation is guided by a number of issues such as problem formulation, solution encoding, the symmetrical nature of the problem, and distinct forms of hybridisation. For the TDP, we also propose a slot-based alternative problem formulation (distinct to other slot-based proposals), which represents another option other than the classical variation-based formulation of the problem. An empirical analysis, assessing the performance of all the metaheuristics (i.e., basic, integrative and collaborative algorithms working on different search spaces and with/without symmetry breaking) shows that some of our proposals can be considered the state-of-the-art when they are applied to specific problem instances.
Enhancing Adversarial Robustness via Uncertainty-Aware Distributional Adversarial Training
Venue: arXiv cs.LG 2024
Year: 2024

Despite remarkable achievements in deep learning across various domains, its inherent vulnerability to adversarial examples still remains a critical concern for practical deployment. Adversarial training has emerged as one of the most effective defensive techniques for improving model robustness against such malicious inputs. However, existing adversarial training schemes often lead to limited generalization ability against underlying adversaries with diversity due to their overreliance on a point-by-point augmentation strategy by mapping each clean example to its adversarial counterpart during training. In addition, adversarial examples can induce significant disruptions in the statistical information w.r.t. the target model, thereby introducing substantial uncertainty and challenges to modeling the distribution of adversarial examples. To circumvent these issues, in this paper, we propose a novel uncertainty-aware distributional adversarial training method, which enforces adversary modeling by leveraging both the statistical information of adversarial examples and its corresponding uncertainty estimation, with the goal of augmenting the diversity of adversaries. Considering the potentially negative impact induced by aligning adversaries to misclassified clean examples, we also refine the alignment reference based on the statistical proximity to clean examples during adversarial training, thereby reframing adversarial training within a distribution-to-distribution matching framework interacted between the clean and adversarial domains. Furthermore, we design an introspective gradient alignment approach via matching input gradients between these domains without introducing external models. Extensive experiments across four benchmark datasets and various network architectures demonstrate that our approach achieves state-of-the-art adversarial robustness and maintains natural performance.
Approximation Algorithms for the Freeze Tag Problem inside Polygons
Venue: arXiv cs.CG 2024
Year: 2024

The freeze tag problem (FTP) aims to awaken a swarm of robots with one or more initial awake robots as soon as possible. Each awake robot must touch a sleeping robot to wake it up. Once a robot is awakened, it can assist in awakening other sleeping robots. We study this problem inside a polygonal domain and present approximation algorithms for it.
Codes with restricted overlaps: expandability, constructions, and bounds
Venue: arXiv cs.IT 2024
Year: 2024

Consider a $q$-ary block code satisfying the property that no $l$-letters long codeword's prefix occurs as a suffix of any codeword for $l$ inside some interval. We determine a general upper bound on the maximum size of these codes and a tighter bound for codes where overlaps with lengths not exceeding $k$ are prohibited. We then provide constructions for codes with various restrictions on overlap lengths and use them to determine lower bounds on the maximum sizes. In particular, we construct $(1,k)$-overlap-free codes where $k \geq n/2$ and $n$ denotes the block size, expand a known construction of $(k,n-1)$-overlap-free codes, and combine the ideas behind both constructions to obtain $(t_1,t_2)$-overlap-free codes and codes that are simultaneously $(1,k)$- and $(n-k,n-1)$-overlap-free for some $k < n/2$. In the case when overlaps of lengths between 1 and $k$ are prohibited, we complete the characterisation of non-expandable codes started by Cai, Wang, and Feng (2023).
Photon: Federated LLM Pre-Training
Venue: arXiv cs.LG 2024
Year: 2024

Scaling large language models (LLMs) demands extensive data and computing resources, which are traditionally constrained to data centers by the high-bandwidth requirements of distributed training. Low-bandwidth methods like federated learning (FL) could enable collaborative training of larger models across weakly-connected GPUs if they can effectively be used for pre-training. To achieve this, we introduce Photon, the first complete system for federated end-to-end LLM training, leveraging cross-silo FL for global-scale training with minimal communication overheads. Using Photon, we train the first federated family of decoder-only LLMs from scratch. We show that: (1) Photon can train model sizes up to 7B in a federated fashion while reaching an even better perplexity than centralized pre-training; (2) Photon model training time decreases with available compute, achieving a similar compute-time trade-off to centralized; and (3) Photon outperforms the wall-time of baseline distributed training methods by 35% via communicating 64x-512xless. Our proposal is robust to data heterogeneity and converges twice as fast as previous methods like DiLoCo. This surprising data efficiency stems from a unique approach combining small client batch sizes with extremely high learning rates, enabled by federated averaging's robustness to hyperparameters. Photon thus represents the first economical system for global internet-wide LLM pre-training.
Textual Aesthetics in Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Image aesthetics is a crucial metric in the field of image generation. However, textual aesthetics has not been sufficiently explored. With the widespread application of large language models (LLMs), previous work has primarily focused on the correctness of content and the helpfulness of responses. Nonetheless, providing responses with textual aesthetics is also an important factor for LLMs, which can offer a cleaner layout and ensure greater consistency and coherence in content. In this work, we introduce a pipeline for aesthetics polishing and help construct a textual aesthetics dataset named TexAes. We propose a textual aesthetics-powered fine-tuning method based on direct preference optimization, termed TAPO, which leverages textual aesthetics without compromising content correctness. Additionally, we develop two evaluation methods for textual aesthetics based on text and image analysis, respectively. Our experiments demonstrate that using textual aesthetics data and employing the TAPO fine-tuning method not only improves aesthetic scores but also enhances performance on general evaluation datasets such as AlpacalEval and Anera-hard.
Counting random $k$-SAT near the satisfiability threshold
Venue: arXiv cs.DS 2024
Year: 2024

We present efficient counting and sampling algorithms for random $k$-SAT when the clause density satisfies $α\le \frac{2^k}{\mathrm{poly}(k)}.$ In particular, the exponential term $2^k$ matches the satisfiability threshold $Θ(2^k)$ for the existence of a solution and the (conjectured) algorithmic threshold $2^k (\ln k) / k$ for efficiently finding a solution. Previously, the best-known counting and sampling algorithms required far more restricted densities $α\lesssim 2^{k/3}$ [He, Wu, Yang, SODA '23]. Notably, our result goes beyond the lower bound $d\gtrsim 2^{k/2}$ for worst-case $k$-SAT with bounded-degree $d$ [Bezáková et al, SICOMP '19], showing that for counting and sampling, the average-case random $k$-SAT model is computationally much easier than the worst-case model. At the heart of our approach is a new refined analysis of the recent novel coupling procedure by [Wang, Yin, FOCS '24], utilizing the structural properties of random constraint satisfaction problems (CSPs). Crucially, our analysis avoids reliance on the $2$-tree structure used in prior works, which cannot extend beyond the worst-case threshold $2^{k/2}$. Instead, we employ a witness tree similar to that used in the analysis of the Moser-Tardos algorithm [Moser, Tardos, JACM '10] for the Lovász Local lemma, which may be of independent interest. Our new analysis provides a universal framework for efficient counting and sampling for random atomic CSPs, including, for example, random hypergraph colorings. At the same time, it immediately implies as corollaries several structural and probabilistic properties of random CSPs that have been widely studied but rarely justified, including replica symmetry and non-reconstruction.
Graph Agnostic Causal Bayesian Optimisation
Venue: arXiv cs.LG 2024
Year: 2024

We study the problem of globally optimising a target variable of an unknown causal graph on which a sequence of soft or hard interventions can be performed. The problem of optimising the target variable associated with a causal graph is formalised as Causal Bayesian Optimisation (CBO). We study the CBO problem under the cumulative regret objective with unknown causal graphs for two settings, namely structural causal models with hard interventions and function networks with soft interventions. We propose Graph Agnostic Causal Bayesian Optimisation (GACBO), an algorithm that actively discovers the causal structure that contributes to achieving optimal rewards. GACBO seeks to balance exploiting the actions that give the best rewards against exploring the causal structures and functions. To the best of our knowledge, our work is the first to study causal Bayesian optimization with cumulative regret objectives in scenarios where the graph is unknown or partially known. We show our proposed algorithm outperforms baselines in simulated experiments and real-world applications.
Alternate Learning and Compression Approaching R(D)
Venue: arXiv cs.IT 2024
Year: 2024

The inherent trade-off in on-line learning is between exploration and exploitation. A good balance between these two (conflicting) goals can achieve a better long-term performance. Can we define an optimal balance? We propose to study this question through a backward-adaptive lossy compression system, which exhibits a "natural" trade-off between exploration and exploitation.
Equivalence of Deterministic Weighted Real-time One-Counter Automata
Venue: arXiv cs.FL 2024
Year: 2024

This paper introduces deterministic weighted real-time one-counter automaton (DWROCA). A DWROCA is a deterministic real-time one-counter automaton whose transitions are assigned a weight from a field. Two DWROCAs are equivalent if every word accepted by one is accepted by the other with the same weight. DWROCA is a sub-class of weighted one-counter automata with counter-determinacy. It is known that the equivalence problem for this model is in P. This paper gives a simpler proof and a better polynomial-time algorithm for checking the equivalence of two DWROCAs.
Multi-dimensional Approximate Counting
Venue: arXiv cs.DS 2024
Year: 2024

The celebrated Morris counter uses $\log_2\log_2 n + O(\log_2 σ^{-1})$ bits to count up to $n$ with a relative error $σ$, where if $\hatλ$ is the estimate of the current count $λ$, then $\mathbb{E}|\hatλ-λ|^2 <σ^2λ^2$. A natural generalization is \emph{multi-dimensional} approximate counting. Let $d\geq 1$ be the dimension. The count vector $x\in \mathbb{N}^d$ is incremented entry-wisely over a stream of coordinates $(w_1,\ldots,w_n)\in [d]^n$, where upon receiving $w_k\in[d]$, $x_{w_k}\gets x_{w_k}+1$. A \emph{$d$-dimensional approximate counter} is required to count $d$ coordinates simultaneously and return an estimate $\hat{x}$ of the count vector $x$. Aden-Ali, Han, Nelson, and Yu \cite{aden2022amortized} showed that the trivial solution of using $d$ Morris counters that track $d$ coordinates separately is already optimal in space, \emph{if each entry only allows error relative to itself}, i.e., $\mathbb{E}|\hat{x}_j-x_j|^2<σ^2|x_j|^2$ for each $j\in [d]$. However, for another natural error metric -- the \emph{Euclidean mean squared error} $\mathbb{E} |\hat{x}-x|^2$ -- we show that using $d$ separate Morris counters is sub-optimal. In this work, we present a simple and optimal $d$-dimensional counter with Euclidean relative error $σ$, i.e., $\mathbb{E} |\hat{x}-x|^2 <σ^2|x|^2$ where $|x|=\sqrt{\sum_{j=1}^d x_j^2}$, with a matching lower bound. The upper and lower bounds are proved with ideas that are strikingly simple. The upper bound is constructed with a certain variable-length integer encoding and the lower bound is derived from a straightforward volumetric estimation of sphere covering.
Receiver-Centric Generative Semantic Communications
Venue: arXiv cs.IT 2024
Year: 2024

This paper investigates semantic communications between a transmitter and a receiver, where original data, such as videos of interest to the receiver, is stored at the transmitter. Although significant process has been made in semantic communications, a fundamental design problem is that the semantic information is extracted based on certain criteria at the transmitter alone, without considering the receiver's specific information needs. As a result, critical information of primary concern to the receiver may be lost. In such cases, the semantic transmission becomes meaningless to the receiver, as all received information is irrelevant to its interests. To solve this problem, this paper presents a receiver-centric generative semantic communication system, where each transmission is initialized by the receiver. Specifically, the receiver first sends its request for the desired semantic information to the transmitter at the start of each transmission. Then, the transmitter extracts the required semantic information accordingly. A key challenge is how the transmitter understands the receiver's requests for semantic information and extracts the required semantic information in a reasonable and robust manner. We address this challenge by designing a well-structured framework and leveraging off-the-shelf generative AI products, such as GPT-4, along with several specialized tools for detection and estimation. Evaluation results demonstrate the feasibility and effectiveness of the proposed new semantic communication system.
Machine Learning Innovations in CPR: A Comprehensive Survey on Enhanced Resuscitation Techniques
Venue: arXiv cs.LG 2024
Year: 2024

This survey paper explores the transformative role of Machine Learning (ML) and Artificial Intelligence (AI) in Cardiopulmonary Resuscitation (CPR). It examines the evolution from traditional CPR methods to innovative ML-driven approaches, highlighting the impact of predictive modeling, AI-enhanced devices, and real-time data analysis in improving resuscitation outcomes. The paper provides a comprehensive overview, classification, and critical analysis of current applications, challenges, and future directions in this emerging field.
Hardware for converting floating-point to the microscaling (MX) format
Venue: arXiv cs.AR 2024
Year: 2024

This paper proposes hardware converters for the microscaling format (MX-format), a reduced representation of floating-point numbers. We present an algorithm and a memory-free hardware model for converting 32 single-precision floating-point numbers to MX-format. The proposed model supports six different types of MX-format: E5M2, E4M3, E3M2, E2M3, E2M1, and INT8. The conversion process consists of three steps: calculating the maximum absolute value among 32 inputs, generating a shared scale, and producing 32 outputs in the selected MX-format type. The hardware converters were implemented in FPGA, and experimental results demonstrate.
ZipCache: A DRAM/SSD Cache with Built-in Transparent Compression
Venue: arXiv cs.DB 2024
Year: 2024

As a core component in modern data centers, key-value cache provides high-throughput and low-latency services for high-speed data processing. The effectiveness of a key-value cache relies on its ability of accommodating the needed data. However, expanding the cache capacity is often more difficult than commonly expected because of many practical constraints, such as server costs, cooling issues, rack space, and even human resource expenses. A potential solution is compression, which virtually extends the cache capacity by condensing data in cache. In practice, this seemingly simple idea has not gained much traction in key-value cache system design, due to several critical issues: the compression-unfriendly index structure, severe read/write amplification, wasteful decompression operations, and heavy computing cost. This paper presents a hybrid DRAM-SSD cache design to realize a systematic integration of data compression in key-value cache. By treating compression as an essential component, we have redesigned the indexing structure, data management, and leveraged the emerging computational SSD hardware for collaborative optimizations. We have developed a prototype, called ZipCache. Our experimental results show that ZipCache can achieve up to 72.4% higher throughput and 42.4% lower latency, while reducing the write amplification by up to 26.2 times.
On the Role of Constraints in the Complexity of Min-Max Optimization
Venue: arXiv cs.GT 2024
Year: 2024

We investigate the role of constraints in the computational complexity of min-max optimization. The work of Daskalakis, Skoulakis, and Zampetakis [2021] was the first to study min-max optimization through the lens of computational complexity, showing that min-max problems with nonconvex-nonconcave objectives are PPAD-hard. However, their proof hinges on the presence of joint constraints between the maximizing and minimizing players. The main goal of this paper is to understand the role of these constraints in min-max optimization. The first contribution of this paper is a fundamentally new proof of their main result, which improves it in multiple directions: it holds for degree 2 polynomials, it is essentially tight in the parameters, and it is much simpler than previous approaches, clearly highlighting the role of constraints in the hardness of the problem. Second, we show that with general constraints (i.e., the min player and max player have different constraints), even convex-concave min-max optimization becomes PPAD-hard. Along the way, we also provide PPAD-membership of a general problem related to quasi-variational inequalities, which has applications beyond our problem.