OpenRaiser/Intern-Atlas huggingface dataset:
XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference
Venue: arXiv cs.LG 2024
Year: 2024

Recently the generative Large Language Model (LLM) has achieved remarkable success in numerous applications. Notably its inference generates output tokens one-by-one, leading to many redundant computations. The widely-used KV-Cache framework makes a compromise between time and space complexities. However, caching data generates the increasingly growing memory demand, that can quickly exhaust the limited memory capacity of the modern accelerator like GPUs, particularly in long-context inference tasks. Existing studies reduce memory consumption by evicting some of cached data that have less important impact on inference accuracy. But the benefit in practice is far from ideal due to the static cache allocation across different LLM network layers. This paper observes that the layer-specific cached data have very different impacts on accuracy. We quantify this difference, and give experimental and theoretical validation. We accordingly make a formal analysis and shows that customizing the cache size for each layer in a personalized manner can yield a significant memory reduction, while still providing comparable accuracy. We simulate the cache allocation as a combinatorial optimization problem and give a global optimal solution. In particular, we devise a mini- and sampling-based inference over a lightweight variant of the LLM model, so as to quickly capture the difference and then feed it into the personalized algorithms. Extensive experiments on real-world datasets demonstrate that our proposals can reduce KV cache memory consumption by 61.6% on average, improve computational efficiency by 2.1x and then increase the throughput by up to 5.5x.
Exploring Multi-Grained Concept Annotations for Multimodal Large Language Models
Venue: arXiv cs.CV 2024
Year: 2024

Multimodal Large Language Models (MLLMs) excel in vision--language tasks by pre-training solely on coarse-grained concept annotations (e.g., image captions). We hypothesize that integrating fine-grained concept annotations (e.g., object labels and object regions) will further improve performance, as both data granularities complement each other in terms of breadth and depth in concept representation. We introduce a new dataset featuring Multimodal Multi-Grained Concept annotations (MMGiC) for MLLMs. In constructing MMGiC, we explore the impact of different data recipes on multimodal comprehension and generation. Our analyses reveal that multi-grained concept annotations integrate and complement each other, under our structured template and a general MLLM framework. We clearly explore and demonstrate the potential of MMGiC to help MLLMs better locate and learn concepts, aligning vision and language at multiple granularities. We further validate our hypothesis by investigating the fair comparison and effective collaboration between MMGiC and image--caption data on 12 multimodal comprehension and generation benchmarks, e.g., their appropriate combination achieve 3.95% and 2.34% absolute improvements over image--caption data alone on POPE and SEED-Bench. Code, data and models will be available at https://github.com/LooperXX/MMGiC.
When Vision Models Meet Parameter Efficient Look-Aside Adapters Without Large-Scale Audio Pretraining
Venue: arXiv cs.SD 2024
Year: 2024

Recent studies show that pretrained vision models can boost performance in audio downstream tasks. To enhance the performance further, an additional pretraining stage with large scale audio data is typically required to infuse audio specific knowledge into the vision model. However, such approaches require extensive audio data and a carefully designed objective function. In this work, we propose bypassing the pretraining stage by directly fine-tuning the vision model with our Look Aside Adapter (LoAA) designed for efficient audio understanding. Audio spectrum data is represented across two heterogeneous dimensions time and frequency and we refine adapters to facilitate interactions between tokens across these dimensions. Our experiments demonstrate that our adapters allow vision models to reach or surpass the performance of pretrained audio models in various audio and speech tasks, offering a resource efficient and effective solution for leveraging vision models in audio applications.
Lightweight Spatial Embedding for Vision-based 3D Occupancy Prediction
Venue: arXiv cs.CV 2024
Year: 2024

Occupancy prediction has garnered increasing attention in recent years for its comprehensive fine-grained environmental representation and strong generalization to open-set objects. However, cumbersome voxel features and 3D convolution operations inevitably introduce large overheads in both memory and computation, obstructing the deployment of occupancy prediction approaches in real-time autonomous driving systems. Although some methods attempt to efficiently predict 3D occupancy from 2D Bird's-Eye-View (BEV) features through the Channel-to-Height mechanism, BEV features are insufficient to store all the height information of the scene, which limits performance. This paper proposes LightOcc, an innovative 3D occupancy prediction framework that leverages Lightweight Spatial Embedding to effectively supplement the height clues for the BEV-based representation while maintaining its deployability. Firstly, Global Spatial Sampling is used to obtain the Single-Channel Occupancy from multi-view depth distribution. Spatial-to-Channel mechanism then takes the arbitrary spatial dimension of Single-Channel Occupancy as the feature dimension and extracts Tri-Perspective Views (TPV) Embeddings by 2D convolution. Finally, TPV Embeddings will interact with each other by Lightweight TPV Interaction module to obtain the Spatial Embedding that is optimal supplementary to BEV features. Sufficient experimental results show that LightOcc significantly increases the prediction accuracy of the baseline and achieves state-of-the-art performance on the Occ3D-nuScenes benchmark.
Latent-Reframe: Enabling Camera Control for Video Diffusion Model without Training
Venue: arXiv cs.CV 2024
Year: 2024

Precise camera pose control is crucial for video generation with diffusion models. Existing methods require fine-tuning with additional datasets containing paired videos and camera pose annotations, which are both data-intensive and computationally costly, and can disrupt the pre-trained model distribution. We introduce Latent-Reframe, which enables camera control in a pre-trained video diffusion model without fine-tuning. Unlike existing methods, Latent-Reframe operates during the sampling stage, maintaining efficiency while preserving the original model distribution. Our approach reframes the latent code of video frames to align with the input camera trajectory through time-aware point clouds. Latent code inpainting and harmonization then refine the model latent space, ensuring high-quality video generation. Experimental results demonstrate that Latent-Reframe achieves comparable or superior camera control precision and video quality to training-based methods, without the need for fine-tuning on additional datasets.
Optimizing Location Allocation in Urban Management: A Brief Review
Venue: arXiv cs.CY 2024
Year: 2024

Regarding the concepts of urban management, digital transformation, and smart cities, various issues are presented. Currently, we like to attend to location allocation problems that can be a new part of digital transformation in urban management (such as locating and placing facilities, locating and arranging centers such as aid and rescue centers, or even postal hubs, telecommunications, electronic equipment, and data centers, and routing in transportation optimization). These issues, which are seemingly simple but in practice complex, are important in urban environments, and the issue of accurate location allocation based on existing criteria directly impacts cost management, profit, efficiency, and citizen satisfaction. In recent years, researchers have used or presented various models and methods for location allocation problems, some of which will be mentioned in this article. Given the nature of these problems, which are optimization problems, this article will also examine existing research from an optimization perspective in summary. Finally, a brief conclusion will be made of the existing methods and their weaknesses, and suggestions will be made for continuing the path and improving scientific and practical research in this field.
The AI Double Standard: Humans Judge All AIs for the Actions of One
Venue: arXiv cs.AI 2024
Year: 2024

Robots and other artificial intelligence (AI) systems are widely perceived as moral agents responsible for their actions. As AI proliferates, these perceptions may become entangled via the moral spillover of attitudes towards one AI to attitudes towards other AIs. We tested how the seemingly harmful and immoral actions of an AI or human agent spill over to attitudes towards other AIs or humans in two preregistered experiments. In Study 1 (N = 720), we established the moral spillover effect in human-AI interaction by showing that immoral actions increased attributions of negative moral agency (i.e., acting immorally) and decreased attributions of positive moral agency (i.e., acting morally) and moral patiency (i.e., deserving moral concern) to both the agent (a chatbot or human assistant) and the group to which they belong (all chatbot or human assistants). There was no significant difference in the spillover effects between the AI and human contexts. In Study 2 (N = 684), we tested whether spillover persisted when the agent was individuated with a name and described as an AI or human, rather than specifically as a chatbot or personal assistant. We found that spillover persisted in the AI context but not in the human context, possibly because AIs were perceived as more homogeneous due to their outgroup status relative to humans. This asymmetry suggests a double standard whereby AIs are judged more harshly than humans when one agent morally transgresses. With the proliferation of diverse, autonomous AI systems, HCI research and design should account for the fact that experiences with one AI could easily generalize to perceptions of all AIs and negative HCI outcomes, such as reduced trust.
PyPOD-GP: Using PyTorch for Accelerated Chip-Level Thermal Simulation of the GPU
Venue: arXiv cs.CE 2024
Year: 2024

The rising demand for high-performance computing (HPC) has made full-chip dynamic thermal simulation in many-core GPUs critical for optimizing performance and extending device lifespans. Proper orthogonal decomposition (POD) with Galerkin projection (GP) has shown to offer high accuracy and massive runtime improvements over direct numerical simulation (DNS). However, previous implementations of POD-GP use MPI-based libraries like PETSc and FEniCS and face significant runtime bottlenecks. We propose a $\textbf{Py}$Torch-based $\textbf{POD-GP}$ library (PyPOD-GP), a GPU-optimized library for chip-level thermal simulation. PyPOD-GP achieves over $23.4\times$ speedup in training and over $10\times$ speedup in inference on a GPU with over 13,000 cores, with just $1.2\%$ error over the device layer.
Efficient Probabilistic Workflow Scheduling for IaaS Clouds
Venue: arXiv cs.DC 2024
Year: 2024

The flexibility and the variety of computing resources offered by the cloud make it particularly attractive for executing user workloads. However, IaaS cloud environments pose non-trivial challenges in the case of workflow scheduling under deadlines and monetary cost constraints. Indeed, given the typical uncertain performance behavior of cloud resources, scheduling algorithms that assume deterministic execution times may fail, thus requiring probabilistic approaches. However, existing probabilistic algorithms are computationally expensive, mainly due to the greater complexity of the workflow scheduling problem in its probabilistic form, and they hardily scale with the size of the problem instance. In this article, we propose EPOSS, a novel workflow scheduling algorithm for IaaS cloud environments based on a probabilistic formulation. Our solution blends together the low execution latency of state-of-the-art scheduling algorithms designed for the case of deterministic execution times and the capability to enforce probabilistic constraints.Designed with computational efficiency in mind, EPOSS achieves one to two orders lower execution times in comparison with existing probabilistic schedulers. Furthermore, it ensures good scaling with respect to workflow size and number of heterogeneous virtual machine types offered by the IaaS cloud environment. We evaluated the benefits of our algorithm via an experimental comparison over a variety of workloads and characteristics of IaaS cloud environments.
Hyperspectral Image Spectral-Spatial Feature Extraction via Tensor Principal Component Analysis
Venue: arXiv cs.CV 2024
Year: 2024

This paper addresses the challenge of spectral-spatial feature extraction for hyperspectral image classification by introducing a novel tensor-based framework. The proposed approach incorporates circular convolution into a tensor structure to effectively capture and integrate both spectral and spatial information. Building upon this framework, the traditional Principal Component Analysis (PCA) technique is extended to its tensor-based counterpart, referred to as Tensor Principal Component Analysis (TPCA). The proposed TPCA method leverages the inherent multi-dimensional structure of hyperspectral data, thereby enabling more effective feature representation. Experimental results on benchmark hyperspectral datasets demonstrate that classification models using TPCA features consistently outperform those using traditional PCA and other state-of-the-art techniques. These findings highlight the potential of the tensor-based framework in advancing hyperspectral image analysis.
Verifying Shortest Paths in Linear Time
Venue: arXiv cs.DS 2024
Year: 2024

In this paper we propose a linear-time certifying algorithm for the single-source shortest-path problem capable of verifying graphs with positive, negative, and zero arc weights. Previously proposed linear-time approaches only work for graphs with positive arc weights.
SGIA: Enhancing Fine-Grained Visual Classification with Sequence Generative Image Augmentation
Venue: arXiv cs.CV 2024
Year: 2024

In Fine-Grained Visual Classification (FGVC), distinguishing highly similar subcategories remains a formidable challenge, often necessitating datasets with extensive variability. The acquisition and annotation of such FGVC datasets are notably difficult and costly, demanding specialized knowledge to identify subtle distinctions among closely related categories. Our study introduces a novel approach employing the Sequence Latent Diffusion Model (SLDM) for augmenting FGVC datasets, called Sequence Generative Image Augmentation (SGIA). Our method features a unique Bridging Transfer Learning (BTL) process, designed to minimize the domain gap between real and synthetically augmented data. This approach notably surpasses existing methods in generating more realistic image samples, providing a diverse range of pose transformations that extend beyond the traditional rigid transformations and style changes in generative augmentation. We demonstrate the effectiveness of our augmented dataset with substantial improvements in FGVC tasks on various datasets, models, and training strategies, especially in few-shot learning scenarios. Our method outperforms conventional image augmentation techniques in benchmark tests on three FGVC datasets, showcasing superior realism, variability, and representational quality. Our work sets a new benchmark and outperforms the previous state-of-the-art models in classification accuracy by 0.5% for the CUB-200-2011 dataset and advances the application of generative models in FGVC data augmentation.
Towards Long Video Understanding via Fine-detailed Video Story Generation
Venue: arXiv cs.CV 2024
Year: 2024

Long video understanding has become a critical task in computer vision, driving advancements across numerous applications from surveillance to content retrieval. Existing video understanding methods suffer from two challenges when dealing with long video understanding: intricate long-context relationship modeling and interference from redundancy. To tackle these challenges, we introduce Fine-Detailed Video Story generation (FDVS), which interprets long videos into detailed textual representations. Specifically, to achieve fine-grained modeling of long-temporal content, we propose a Bottom-up Video Interpretation Mechanism that progressively interprets video content from clips to video. To avoid interference from redundant information in videos, we introduce a Semantic Redundancy Reduction mechanism that removes redundancy at both the visual and textual levels. Our method transforms long videos into hierarchical textual representations that contain multi-granularity information of the video. With these representations, FDVS is applicable to various tasks without any fine-tuning. We evaluate the proposed method across eight datasets spanning three tasks. The performance demonstrates the effectiveness and versatility of our method.
Modeling, Planning, and Control for Hybrid UAV Transition Maneuvers
Venue: arXiv cs.RO 2024
Year: 2024

Small unmanned aerial vehicles (UAVs) have become standard tools in reconnaissance and surveying for both civilian and defense applications. In the future, UAVs will likely play a pivotal role in autonomous package delivery, but current multi-rotor candidates suffer from poor energy efficiency leading to insufficient endurance and range. In order to reduce the power demands of package delivery UAVs while still maintaining necessary hovering capabilities, companies like Amazon are experimenting with hybrid Vertical Take-Off and Landing (VTOL) platforms. Tailsitter VTOLs offer a mechanically simple and cost-effective solution compared to other hybrid VTOL configurations, and while advances in hardware and microelectronics have optimized the tailsitter for package delivery, the software behind its operation has largely remained a critical barrier to industry adoption. Tailsitters currently lack a generic, computationally efficient method of control that can provide strong safety and robustness guarantees over the entire flight domain. Further, tailsitters lack a closed-form method of designing dynamically feasible transition maneuvers between hover and cruise. In this paper, we survey the modeling and control methods currently implemented on small-scale tailsitter UAVs, and attempt to leverage a nonlinear dynamic model to design physically realizable, continuous-pitch transition maneuvers at constant altitude. Primary results from this paper isolate potential barriers to constant-altitude transition, and a novel approach to bypassing these barriers is proposed. While initial results are unsuccessful at providing feasible transition, this work acts as a stepping stone for future efforts to design new transition maneuvers that are safe, robust, and computationally efficient.
Applying Machine Learning Tools for Urban Resilience Against Floods
Venue: arXiv cs.LG 2024
Year: 2024

Floods are among the most prevalent and destructive natural disasters, often leading to severe social and economic impacts in urban areas due to the high concentration of assets and population density. In Iran, particularly in Tehran, recurring flood events underscore the urgent need for robust urban resilience strategies. This paper explores flood resilience models to identify the most effective approach for District 6 in Tehran. Through an extensive literature review, various resilience models were analyzed, with the Climate Disaster Resilience Index (CDRI) emerging as the most suitable model for this district due to its comprehensive resilience dimensions: Physical, Social, Economic, Organizational, and Natural Health resilience. Although the CDRI model provides a structured approach to resilience measurement, it remains a static model focused on spatial characteristics and lacks temporal adaptability. An extensive literature review enhances the CDRI model by integrating data from 2013 to 2022 in three-year intervals and applying machine learning techniques to predict resilience dimensions for 2025. This integration enables a dynamic resilience model that can accommodate temporal changes, providing a more adaptable and data driven foundation for urban flood resilience planning. By employing artificial intelligence to reflect evolving urban conditions, this model offers valuable insights for policymakers and urban planners to enhance flood resilience in Tehrans critical District 6.
MSCrackMamba: Leveraging Vision Mamba for Crack Detection in Fused Multispectral Imagery
Venue: arXiv cs.CV 2024
Year: 2024

Crack detection is a critical task in structural health monitoring, aimed at assessing the structural integrity of bridges, buildings, and roads to prevent potential failures. Vision-based crack detection has become the mainstream approach due to its ease of implementation and effectiveness. Fusing infrared (IR) channels with red, green and blue (RGB) channels can enhance feature representation and thus improve crack detection. However, IR and RGB channels often differ in resolution. To align them, higher-resolution RGB images typically need to be downsampled to match the IR image resolution, which leads to the loss of fine details. Moreover, crack detection performance is restricted by the limited receptive fields and high computational complexity of traditional image segmentation networks. Inspired by the recently proposed Mamba neural architecture, this study introduces a two-stage paradigm called MSCrackMamba, which leverages Vision Mamba along with a super-resolution network to address these challenges. Specifically, to align IR and RGB channels, we first apply super-resolution to IR channels to match the resolution of RGB channels for data fusion. Vision Mamba is then adopted as the backbone network, while UperNet is employed as the decoder for crack detection. Our approach is validated on the large-scale Crack Detection dataset Crack900, demonstrating an improvement of 3.55% in mIoU compared to the best-performing baseline methods.
A Scalable Decentralized Reinforcement Learning Framework for UAV Target Localization Using Recurrent PPO
Venue: arXiv cs.RO 2024
Year: 2024

The rapid advancements in unmanned aerial vehicles (UAVs) have unlocked numerous applications, including environmental monitoring, disaster response, and agricultural surveying. Enhancing the collective behavior of multiple decentralized UAVs can significantly improve these applications through more efficient and coordinated operations. In this study, we explore a Recurrent PPO model for target localization in perceptually degraded environments like places without GNSS/GPS signals. We first developed a single-drone approach for target identification, followed by a decentralized two-drone model. Our approach can utilize two types of sensors on the UAVs, a detection sensor and a target signal sensor. The single-drone model achieved an accuracy of 93%, while the two-drone model achieved an accuracy of 86%, with the latter requiring fewer average steps to locate the target. This demonstrates the potential of our method in UAV swarms, offering efficient and effective localization of radiant targets in complex environmental conditions.