OpenRaiser/Intern-Atlas huggingface dataset:
DFIMat: Decoupled Flexible Interactive Matting in Multi-Person Scenarios
Venue: arXiv cs.CV 2024
Year: 2024

Interactive portrait matting refers to extracting the soft portrait from a given image that best meets the user's intent through their inputs. Existing methods often underperform in complex scenarios, mainly due to three factors. (1) Most works apply a tightly coupled network that directly predicts matting results, lacking interpretability and resulting in inadequate modeling. (2) Existing works are limited to a single type of user input, which is ineffective for intention understanding and also inefficient for user operation. (3) The multi-round characteristics have been under-explored, which is crucial for user interaction. To alleviate these limitations, we propose DFIMat, a decoupled framework that enables flexible interactive matting. Specifically, we first decouple the task into 2 sub-ones: localizing target instances by understanding scene semantics and the flexible user inputs, and conducting refinement for instance-level matting. We observe a clear performance gain from decoupling, as it makes sub-tasks easier to learn, and the flexible multi-type input further enhances both effectiveness and efficiency. DFIMat also considers the multi-round interaction property, where a contrastive reasoning module is designed to enhance cross-round refinement. Another limitation for multi-person matting task is the lack of training data. We address this by introducing a new synthetic data generation pipeline that can generate much more realistic samples than previous arts. A new large-scale dataset SMPMat is subsequently established. Experiments verify the significant superiority of DFIMat. With it, we also investigate the roles of different input types, providing valuable principles for users. Our code and dataset can be found at https://github.com/JiaoSiyi/DFIMat.
Model Predictive Control for Optimal Motion Planning of Unmanned Aerial Vehicles
Venue: arXiv cs.RO 2024
Year: 2024

Motion planning is an essential process for the navigation of unmanned aerial vehicles (UAVs) where they need to adapt to obstacles and different structures of their operating environment to reach the goal. This paper presents an optimal motion planner for UAVs operating in unknown complex environments. The motion planner receives point cloud data from a local range sensor and then converts it into a voxel grid representing the surrounding environment. A local trajectory guiding the UAV to the goal is then generated based on the voxel grid. This trajectory is further optimized using model predictive control (MPC) to enhance the safety, speed, and smoothness of UAV operation. The optimization is carried out via the definition of several cost functions and constraints, taking into account the UAV's dynamics and requirements. A number of simulations and comparisons with a state-of-the-art method have been conducted in a complex environment with many obstacles to evaluate the performance of our method. The results show that our method provides not only shorter and smoother trajectories but also faster and more stable speed profiles. It is also energy efficient making it suitable for various UAV applications.
Unraveling the Potential of Large Language Models in Code Translation: How Far Are We?
Venue: arXiv cs.SE 2024
Year: 2024

While large language models (LLMs) exhibit state-of-the-art performance in various tasks, recent studies have revealed their struggle for code translation. This is because they haven't been extensively pre-trained with parallel multilingual code, which code translation heavily depends on. Moreover, existing benchmarks only cover a limited subset of common programming languages, and thus cannot reflect the full potential of LLMs in code translation. In this paper, we conduct a large-scale empirical study to exploit the capabilities and incapabilities of LLMs in code translation tasks. We first craft a novel benchmark called PolyHumanEval by extending HumanEval to a multilingual benchmark of 14 languages. With PolyHumanEval, we then perform over 110,000 translations with bleeding-edge code LLMs. The result shows LLMs' suboptimal performance on Python to other languages and the negligible impact of widely adopted LLM optimization techniques such as conventional pre-training and instruction tuning on code translation. To further uncover the potential of LLMs in code translation, we propose two methods: (1) intermediary translation which selects an intermediary language between the source and target ones; and (2) self-training which fine-tunes LLMs on self-generated parallel data. Evaluated with CodeLlama-13B, our approach yields an average improvement of 11.7% computation accuracy on Python-to-other translations. Notably, we interestingly find that Go can serve as a lingua franca for translating between any two studied languages.
TopOC: Topological Deep Learning for Ovarian and Breast Cancer Diagnosis
Venue: arXiv cs.CV 2024
Year: 2024

Microscopic examination of slides prepared from tissue samples is the primary tool for detecting and classifying cancerous lesions, a process that is time-consuming and requires the expertise of experienced pathologists. Recent advances in deep learning methods hold significant potential to enhance medical diagnostics and treatment planning by improving accuracy, reproducibility, and speed, thereby reducing clinicians' workloads and turnaround times. However, the necessity for vast amounts of labeled data to train these models remains a major obstacle to the development of effective clinical decision support systems. In this paper, we propose the integration of topological deep learning methods to enhance the accuracy and robustness of existing histopathological image analysis models. Topological data analysis (TDA) offers a unique approach by extracting essential information through the evaluation of topological patterns across different color channels. While deep learning methods capture local information from images, TDA features provide complementary global features. Our experiments on publicly available histopathological datasets demonstrate that the inclusion of topological features significantly improves the differentiation of tumor types in ovarian and breast cancers.
RISC-V Needs Secure 'Wheels': the MCU Initiator-Side Perspective
Venue: arXiv cs.CR 2024
Year: 2024

The automotive industry is experiencing a massive paradigm shift. Cars are becoming increasingly autonomous, connected, and computerized. Modern electrical/electronic (E/E) architectures are pushing for an unforeseen functionality integration density, resulting in physically separate Electronic Control Units (ECUs) becoming virtualized and mapped to logical partitions within a single physical microcontroller (MCU). While functional safety (FuSa) has been pivotal for vehicle certification for decades, the increasing connectivity and advances have opened the door for a number of car hacks and attacks. This development drives (cyber-)security requirements in cars, and has paved the way for the release of the new security certification standard ISO21434. RISC-V has great potential to transform automotive computing systems, but we argue that current ISA/extensions are not ready yet. This paper provides our critical perspective on the existing RISC-V limitations, particularly on the upcoming WorldGuard technology, to address virtualized MCU requirements in line with foreseen automotive applications and ISO21434 directives. We then present our proposal for the required ISA extensions to address such limitations, mainly targeting initiator-side protection. Finally, we explain our roadmap towards a full open-source proof-of-concept (PoC), which includes extending QEMU, an open-source RISC-V core, and building a complete software stack.
Understanding Robustness of Parameter-Efficient Tuning for Image Classification
Venue: arXiv cs.CV 2024
Year: 2024

Parameter-efficient tuning (PET) techniques calibrate the model's predictions on downstream tasks by freezing the pre-trained models and introducing a small number of learnable parameters. However, despite the numerous PET methods proposed, their robustness has not been thoroughly investigated. In this paper, we systematically explore the robustness of four classical PET techniques (e.g., VPT, Adapter, AdaptFormer, and LoRA) under both white-box attacks and information perturbations. For white-box attack scenarios, we first analyze the performance of PET techniques using FGSM and PGD attacks. Subsequently, we further explore the transferability of adversarial samples and the impact of learnable parameter quantities on the robustness of PET methods. Under information perturbation attacks, we introduce four distinct perturbation strategies, including Patch-wise Drop, Pixel-wise Drop, Patch Shuffle, and Gaussian Noise, to comprehensively assess the robustness of these PET techniques in the presence of information loss. Via these extensive studies, we enhance the understanding of the robustness of PET methods, providing valuable insights for improving their performance in computer vision applications. The code is available at https://github.com/JCruan519/PETRobustness.
ImagineNav: Prompting Vision-Language Models as Embodied Navigator through Scene Imagination
Venue: arXiv cs.RO 2024
Year: 2024

Visual navigation is an essential skill for home-assistance robots, providing the object-searching ability to accomplish long-horizon daily tasks. Many recent approaches use Large Language Models (LLMs) for commonsense inference to improve exploration efficiency. However, the planning process of LLMs is limited within texts and it is difficult to represent the spatial occupancy and geometry layout only by texts. Both are important for making rational navigation decisions. In this work, we seek to unleash the spatial perception and planning ability of Vision-Language Models (VLMs), and explore whether the VLM, with only on-board camera captured RGB/RGB-D stream inputs, can efficiently finish the visual navigation tasks in a mapless manner. We achieve this by developing the imagination-powered navigation framework ImagineNav, which imagines the future observation images at valuable robot views and translates the complex navigation planning process into a rather simple best-view image selection problem for VLM. To generate appropriate candidate robot views for imagination, we introduce the Where2Imagine module, which is distilled to align with human navigation habits. Finally, to reach the VLM preferred views, an off-the-shelf point-goal navigation policy is utilized. Empirical experiments on the challenging open-vocabulary object navigation benchmarks demonstrates the superiority of our proposed system.
The Roles of Contextual Semantic Relevance Metrics in Human Visual Processing
Venue: arXiv cs.CV 2024
Year: 2024

Semantic relevance metrics can capture both the inherent semantics of individual objects and their relationships to other elements within a visual scene. Numerous previous research has demonstrated that these metrics can influence human visual processing. However, these studies often did not fully account for contextual information or employ the recent deep learning models for more accurate computation. This study investigates human visual perception and processing by introducing the metrics of contextual semantic relevance. We evaluate semantic relationships between target objects and their surroundings from both vision-based and language-based perspectives. Testing a large eye-movement dataset from visual comprehension, we employ state-of-the-art deep learning techniques to compute these metrics and analyze their impacts on fixation measures on human visual processing through advanced statistical models. These metrics could also simulate top-down and bottom-up processing in visual perception. This study further integrates vision-based and language-based metrics into a novel combined metric, addressing a critical gap in previous research that often treated visual and semantic similarities separately. Results indicate that all metrics could precisely predict fixation measures in visual perception and processing, but with distinct roles in prediction. The combined metric outperforms other metrics, supporting theories that emphasize the interaction between semantic and visual information in shaping visual perception/processing. This finding aligns with growing recognition of the importance of multi-modal information processing in human cognition. These insights enhance our understanding of cognitive mechanisms underlying visual processing and have implications for developing more accurate computational models in fields such as cognitive science and human-computer interaction.
Separable Drawings: Extendability and Crossing-Free Hamiltonian Cycles
Venue: arXiv cs.CG 2024
Year: 2024

Generalizing pseudospherical drawings, we introduce a new class of simple drawings, which we call separable drawings. In a separable drawing, every edge can be closed to a simple curve that intersects each other edge at most once. Curves of different edges might interact arbitrarily. Most notably, we show that (1) every separable drawing of any graph on $n$ vertices in the plane can be extended to a simple drawing of the complete graph $K_{n}$, (2) every separable drawing of $K_{n}$ contains a crossing-free Hamiltonian cycle and is plane Hamiltonian connected, and (3) every generalized convex drawing and every 2-page book drawing is separable. Further, the class of separable drawings is a proper superclass of the union of generalized convex and 2-page book drawings. Hence, our results on plane Hamiltonicity extend recent work on generalized convex drawings by Bergold et al. (SoCG 2024).
Analysis and Design of a Personalized Recommendation System Based on a Dynamic User Interest Model
Venue: arXiv cs.IR 2024
Year: 2024

With the rapid development of the internet and the explosion of information, providing users with accurate personalized recommendations has become an important research topic. This paper designs and analyzes a personalized recommendation system based on a dynamic user interest model. The system captures user behavior data, constructs a dynamic user interest model, and combines multiple recommendation algorithms to provide personalized content to users. The research results show that this system significantly improves recommendation accuracy and user satisfaction. This paper discusses the system's architecture design, algorithm implementation, and experimental results in detail and explores future research directions.
The Case for DBMS Live Patching [Extended Version]
Venue: arXiv cs.DB 2024
Year: 2024

Traditionally, when the code of a database management system (DBMS) needs to be updated, the system is restarted and database clients suffer downtime, or the provider instantiates hot-standby instances and rolls over the workload. We investigate a third option, live patching of the DBMS binary. For certain code changes, live patching allows to modify the application code in memory, without restart. The memory state and all client connections can be maintained. Although live patching has been explored in the operating systems research community, it remains a blind spot in DBMS research. In this Experiment, Analysis & Benchmark article, we systematically explore this field from the DBMS perspective. We discuss what distinguishes database management systems from generic multi-threaded applications when it comes to live patching. We then propose domain-specific strategies for injecting quiescence points into the DBMS source code, so that threads can safely migrate to the patched process version. We experimentally investigate the interplay between the query workload and different quiescence methods, monitoring both transaction throughput and tail latencies. We show that live patching can be a viable option for updating database management systems, since database providers can make informed decisions w.r.t. the latency overhead on the client side.
A resource-efficient model for deep kernel learning
Venue: arXiv cs.LG 2024
Year: 2024

According to the Hughes phenomenon, the major challenges encountered in computations with learning models comes from the scale of complexity, e.g. the so-called curse of dimensionality. There are various approaches for accelerate learning computations with minimal loss of accuracy. These approaches range from model-level to implementation-level approaches. To the best of our knowledge, the first one is rarely used in its basic form. Perhaps, this is due to theoretical understanding of mathematical insights of model decomposition approaches, and thus the ability of developing mathematical improvements has lagged behind. We describe a model-level decomposition approach that combines both the decomposition of the operators and the decomposition of the network. We perform a feasibility analysis on the resulting algorithm, both in terms of its accuracy and scalability.
Towards a Parameterized Approximation Dichotomy of MinCSP for Linear Equations over Finite Commutative Rings
Venue: arXiv cs.DS 2024
Year: 2024

We consider the MIN-r-LIN(R) problem: given a system S of length-r linear equations over a ring R, find a subset of equations Z of minimum cardinality such that S-Z is satisfiable. The problem is NP-hard and UGC-hard to approximate within any constant even when r=|R|=2, so we focus on parameterized approximability with solution size as the parameter. For a large class of infinite rings R called Euclidean domains, Dabrowski et al. [SODA-2023] obtained an FPT-algorithm for MIN-2-LIN(R) using an LP-based approach based on work by Wahlström [SODA-2017]. Here, we consider MIN-r-LIN(R) for finite commutative rings R, initiating a line of research with the ultimate goal of proving dichotomy theorems that separate problems that are FPT-approximable within a constant from those that are not. A major motivation is that our project is a promising step for more ambitious classification projects concerning finite-domain MinCSP and VCSP. Dabrowski et al.'s algorithm is limited to rings without zero divisors, which are only fields among finite commutative rings. Handling zero divisors seems to be an insurmountable obstacle for the LP-based approach. In response, we develop a constant-factor FPT-approximation algorithm for a large class of finite commutative rings, called Bergen rings, and thus prove approximability for chain rings, principal ideal rings, and Z_m for all m>1. We complement the algorithmic result with powerful lower bounds. For r>2, we show that the problem is not FPT-approximable within any constant (unless FPT=W[1]). We identify the class of non-Helly rings for which MIN-2-LIN(R) is not FPT-approximable. Under ETH, we also rule out (2-e)-approximation for every e>0 for non-lineal rings, which includes e.g. rings Z_{pq} where p and q are distinct primes. Towards closing the gaps between upper and lower bounds, we lay the foundation of a geometric approach for analysing rings.
Dynamic Estimation of Learning Rates Using a Non-Linear Autoregressive Model
Venue: arXiv cs.LG 2024
Year: 2024

We introduce a new class of adaptive non-linear autoregressive (Nlar) models incorporating the concept of momentum, which dynamically estimate both the learning rates and momentum as the number of iterations increases. In our method, the growth of the gradients is controlled using a scaling (clipping) function, leading to stable convergence. Within this framework, we propose three distinct estimators for learning rates and provide theoretical proof of their convergence. We further demonstrate how these estimators underpin the development of effective Nlar optimizers. The performance of the proposed estimators and optimizers is rigorously evaluated through extensive experiments across several datasets and a reinforcement learning environment. The results highlight two key features of the Nlar optimizers: robust convergence despite variations in underlying parameters, including large initial learning rates, and strong adaptability with rapid convergence during the initial epochs.
MisinfoEval: Generative AI in the Era of "Alternative Facts"
Venue: arXiv cs.CL 2024
Year: 2024

The spread of misinformation on social media platforms threatens democratic processes, contributes to massive economic losses, and endangers public health. Many efforts to address misinformation focus on a knowledge deficit model and propose interventions for improving users' critical thinking through access to facts. Such efforts are often hampered by challenges with scalability, and by platform users' personal biases. The emergence of generative AI presents promising opportunities for countering misinformation at scale across ideological barriers. In this paper, we introduce a framework (MisinfoEval) for generating and comprehensively evaluating large language model (LLM) based misinformation interventions. We present (1) an experiment with a simulated social media environment to measure effectiveness of misinformation interventions, and (2) a second experiment with personalized explanations tailored to the demographics and beliefs of users with the goal of countering misinformation by appealing to their pre-existing values. Our findings confirm that LLM-based interventions are highly effective at correcting user behavior (improving overall user accuracy at reliability labeling by up to 41.72%). Furthermore, we find that users favor more personalized interventions when making decisions about news reliability and users shown personalized interventions have significantly higher accuracy at identifying misinformation.
Tight Bounds and Phase Transitions for Incremental and Dynamic Retrieval
Venue: arXiv cs.DS 2024
Year: 2024

Retrieval data structures are data structures that answer key-value queries without paying the space overhead of explicitly storing keys. The problem can be formulated in four settings (static, value-dynamic, incremental, or dynamic), each of which offers different levels of dynamism to the user. In this paper, we establish optimal bounds for the final two settings (incremental and dynamic) in the case of a polynomial universe. Our results complete a line of work that has spanned more than two decades, and also come with a surprise: the incremental setting, which has long been viewed as essentially equivalent to the dynamic one, actually has a phase transition, in which, as the value size $v$ approaches $\log n$, the optimal space redundancy actually begins to shrink, going from roughly $n \log \log n$ (which has long been thought to be optimal) all the way down to $Θ(n)$ (which is the optimal bound even for the seemingly much-easier value-dynamic setting).
NARAIM: Native Aspect Ratio Autoregressive Image Models
Venue: arXiv cs.CV 2024
Year: 2024

While vision transformers are able to solve a wide variety of computer vision tasks, no pre-training method has yet demonstrated the same scaling laws as observed in language models. Autoregressive models show promising results, but are commonly trained on images that are cropped or transformed into square images, which distorts or destroys information present in the input. To overcome this limitation, we propose NARAIM, a vision model pre-trained with an autoregressive objective that uses images in their native aspect ratio. By maintaining the native aspect ratio, we preserve the original spatial context, thereby enhancing the model's ability to interpret visual information. In our experiments, we show that maintaining the aspect ratio improves performance on a downstream classification task.