OpenRaiser/Intern-Atlas huggingface dataset:
SkelMamba: A State Space Model for Efficient Skeleton Action Recognition of Neurological Disorders
Venue: arXiv cs.CV 2024
Year: 2024

We introduce a novel state-space model (SSM)-based framework for skeleton-based human action recognition, with an anatomically-guided architecture that improves state-of-the-art performance in both clinical diagnostics and general action recognition tasks. Our approach decomposes skeletal motion analysis into spatial, temporal, and spatio-temporal streams, using channel partitioning to capture distinct movement characteristics efficiently. By implementing a structured, multi-directional scanning strategy within SSMs, our model captures local joint interactions and global motion patterns across multiple anatomical body parts. This anatomically-aware decomposition enhances the ability to identify subtle motion patterns critical in medical diagnosis, such as gait anomalies associated with neurological conditions. On public action recognition benchmarks, i.e., NTU RGB+D, NTU RGB+D 120, and NW-UCLA, our model outperforms current state-of-the-art methods, achieving accuracy improvements up to $3.2\%$ with lower computational complexity than previous leading transformer-based models. We also introduce a novel medical dataset for motion-based patient neurological disorder analysis to validate our method's potential in automated disease diagnosis.
In-Vehicle Edge System for Real-Time Dashcam Video Analysis
Venue: arXiv cs.DC 2024
Year: 2024

Modern vehicles equip dashcams that primarily collect visual evidence for traffic accidents. However, most of the video data collected by dashcams that is not related to traffic accidents is discarded without any use. In this paper, we present a use case for dashcam videos that aims to improve driving safety. By analyzing the real-time videos captured by dashcams, we can detect driving hazards and driver distractedness to alert the driver immediately. To that end, we design and implement a Distributed Edge-based dashcam Video Analytics system (DEVA), that analyzes dashcam videos using personal edge (mobile) devices in a vehicle. DEVA consolidates available in-vehicle edge devices to maintain the resource pool, distributes video frames for analysis to devices considering resource availability in each device, and dynamically adjusts frame rates of dashcams to control the overall workloads. The entire video analytics task is divided into multiple independent phases and executed in a pipelined manner to improve the overall frame processing throughput. We implement DEVA in an Android app and also develop a dashcam emulation app to be used in vehicles that are not equipped with dashcams. Experimental results using the apps and commercial smartphones show that DEVA can process real-time videos from two dashcams with frame rates of around 22~30 FPS per camera within 200 ms of latency, using three high-end devices.
LDA-AQU: Adaptive Query-guided Upsampling via Local Deformable Attention
Venue: arXiv cs.CV 2024
Year: 2024

Feature upsampling is an essential operation in constructing deep convolutional neural networks. However, existing upsamplers either lack specific feature guidance or necessitate the utilization of high-resolution feature maps, resulting in a loss of performance and flexibility. In this paper, we find that the local self-attention naturally has the feature guidance capability, and its computational paradigm aligns closely with the essence of feature upsampling (\ie feature reassembly of neighboring points). Therefore, we introduce local self-attention into the upsampling task and demonstrate that the majority of existing upsamplers can be regarded as special cases of upsamplers based on local self-attention. Considering the potential semantic gap between upsampled points and their neighboring points, we further introduce the deformation mechanism into the upsampler based on local self-attention, thereby proposing LDA-AQU. As a novel dynamic kernel-based upsampler, LDA-AQU utilizes the feature of queries to guide the model in adaptively adjusting the position and aggregation weight of neighboring points, thereby meeting the upsampling requirements across various complex scenarios. In addition, LDA-AQU is lightweight and can be easily integrated into various model architectures. We evaluate the effectiveness of LDA-AQU across four dense prediction tasks: object detection, instance segmentation, panoptic segmentation, and semantic segmentation. LDA-AQU consistently outperforms previous state-of-the-art upsamplers, achieving performance enhancements of 1.7 AP, 1.5 AP, 2.0 PQ, and 2.5 mIoU compared to the baseline models in the aforementioned four tasks, respectively. Code is available at \url{https://github.com/duzw9311/LDA-AQU}.
Tortho-Gaussian: Splatting True Digital Orthophoto Maps
Venue: arXiv cs.CV 2024
Year: 2024

True Digital Orthophoto Maps (TDOMs) are essential products for digital twins and Geographic Information Systems (GIS). Traditionally, TDOM generation involves a complex set of traditional photogrammetric process, which may deteriorate due to various challenges, including inaccurate Digital Surface Model (DSM), degenerated occlusion detections, and visual artifacts in weak texture regions and reflective surfaces, etc. To address these challenges, we introduce TOrtho-Gaussian, a novel method inspired by 3D Gaussian Splatting (3DGS) that generates TDOMs through orthogonal splatting of optimized anisotropic Gaussian kernel. More specifically, we first simplify the orthophoto generation by orthographically splatting the Gaussian kernels onto 2D image planes, formulating a geometrically elegant solution that avoids the need for explicit DSM and occlusion detection. Second, to produce TDOM of large-scale area, a divide-and-conquer strategy is adopted to optimize memory usage and time efficiency of training and rendering for 3DGS. Lastly, we design a fully anisotropic Gaussian kernel that adapts to the varying characteristics of different regions, particularly improving the rendering quality of reflective surfaces and slender structures. Extensive experimental evaluations demonstrate that our method outperforms existing commercial software in several aspects, including the accuracy of building boundaries, the visual quality of low-texture regions and building facades. These results underscore the potential of our approach for large-scale urban scene reconstruction, offering a robust alternative for enhancing TDOM quality and scalability.
Enhancing Security in Third-Party Library Reuse -- Comprehensive Detection of 1-day Vulnerability through Code Patch Analysis
Venue: arXiv cs.SE 2024
Year: 2024

Nowadays, software development progresses rapidly to incorporate new features. To facilitate such growth and provide convenience for developers when creating and updating software, reusing open-source software (i.e., thirdparty library reuses) has become one of the most effective and efficient methods. Unfortunately, the practice of reusing third-party libraries (TPLs) can also introduce vulnerabilities (known as 1-day vulnerabilities) because of the low maintenance of TPLs, resulting in many vulnerable versions remaining in use. If the software incorporating these TPLs fails to detect the introduced vulnerabilities and leads to delayed updates, it will exacerbate the security risks. However, the complicated code dependencies and flexibility of TPL reuses make the detection of 1-day vulnerability a challenging task. To support developers in securely reusing TPLs during software development, we design and implement VULTURE, an effective and efficient detection tool, aiming at identifying 1-day vulnerabilities that arise from the reuse of vulnerable TPLs. It first executes a database creation method, TPLFILTER, which leverages the Large Language Model (LLM) to automatically build a unique database for the targeted platform. Instead of relying on code-level similarity comparison, VULTURE employs hashing-based comparison to explore the dependencies among the collected TPLs and identify the similarities between the TPLs and the target projects. Recognizing that developers have the flexibility to reuse TPLs exactly or in a custom manner, VULTURE separately conducts version-based comparison and chunk-based analysis to capture fine-grained semantic features at the function levels. We applied VULTURE to 10 real-world projects to assess its effectiveness and efficiency in detecting 1-day vulnerabilities. VULTURE successfully identified 175 vulnerabilities from 178 reused TPLs.
Fast Mutual Information Computation for Large Binary Datasets
Venue: arXiv cs.LG 2024
Year: 2024

Mutual Information (MI) is a powerful statistical measure that quantifies shared information between random variables, particularly valuable in high-dimensional data analysis across fields like genomics, natural language processing, and network science. However, computing MI becomes computationally prohibitive for large datasets where it is typically required a pairwise computational approach where each column is compared to others. This work introduces a matrix-based algorithm that accelerates MI computation by leveraging vectorized operations and optimized matrix calculations. By transforming traditional pairwise computational approaches into bulk matrix operations, the proposed method enables efficient MI calculation across all variable pairs. Experimental results demonstrate significant performance improvements, with computation times reduced up to 50,000 times in the largest dataset using optimized implementations, particularly when utilizing hardware optimized frameworks. The approach promises to expand MI's applicability in data-driven research by overcoming previous computational limitations.
Know Your RAG: Dataset Taxonomy and Generation Strategies for Evaluating RAG Systems
Venue: arXiv cs.IR 2024
Year: 2024

Retrieval Augmented Generation (RAG) systems are a widespread application of Large Language Models (LLMs) in the industry. While many tools exist empowering developers to build their own systems, measuring their performance locally, with datasets reflective of the system's use cases, is a technological challenge. Solutions to this problem range from non-specific and cheap (most public datasets) to specific and costly (generating data from local documents). In this paper, we show that using public question and answer (Q&A) datasets to assess retrieval performance can lead to non-optimal systems design, and that common tools for RAG dataset generation can lead to unbalanced data. We propose solutions to these issues based on the characterization of RAG datasets through labels and through label-targeted data generation. Finally, we show that fine-tuned small LLMs can efficiently generate Q&A datasets. We believe that these observations are invaluable to the know-your-data step of RAG systems development.
SoK: Detection and Repair of Accessibility Issues
Venue: arXiv cs.SE 2024
Year: 2024

There is an increasing global emphasis on information accessibility, with numerous researchers actively developing automated tools to detect and repair accessibility issues, thereby ensuring that individuals with diverse abilities can independently access software products and services. However, current research still encounters significant challenges in two key areas: the absence of a comprehensive taxonomy of accessibility issue types, and the lack of comprehensive analysis of the capabilities of detection and repair tools, as well as the status of corresponding datasets. To address these challenges, this paper introduces the Accessibility Issue Analysis (AIA) framework. Utilizing this framework, we develop a comprehensive taxonomy that categorizes 55 types of accessibility issues across four pivotal dimensions: Perceivability, Operability, Understandability, and Robustness. This taxonomy has been rigorously recognized through a questionnaire survey (n=130). Building on this taxonomy, we conduct an in-depth analysis of existing detection and repair tools, as well as the status of corresponding datasets. In terms of tools, our findings indicate that 14 detection tools can identify 31 issue types, achieving a 56.3% rate (31/55). Meanwhile, 9 repair tools address just 13 issue types, with a 23.6% rate. In terms of datasets, those for detection tools cover 21 issue types, at a 38.1% coverage rate, whereas those for repair tools cover only 7 types, at a 12.7% coverage rate.
Real-Time Anomaly Detection in Video Streams
Venue: arXiv cs.CV 2024
Year: 2024

This thesis is part of a CIFRE agreement between the company Othello and the LIASD laboratory. The objective is to develop an artificial intelligence system that can detect real-time dangers in a video stream. To achieve this, a novel approach combining temporal and spatial analysis has been proposed. Several avenues have been explored to improve anomaly detection by integrating object detection, human pose detection, and motion analysis. For result interpretability, techniques commonly used for image analysis, such as activation and saliency maps, have been extended to videos, and an original method has been proposed. The proposed architecture performs binary or multiclass classification depending on whether an alert or the cause needs to be identified. Numerous neural networkmodels have been tested, and three of them have been selected. You Only Looks Once (YOLO) has been used for spatial analysis, a Convolutional Recurrent Neuronal Network (CRNN) composed of VGG19 and a Gated Recurrent Unit (GRU) for temporal analysis, and a multi-layer perceptron for classification. These models handle different types of data and can be combined in parallel or in series. Although the parallel mode is faster, the serial mode is generally more reliable. For training these models, supervised learning was chosen, and two proprietary datasets were created. The first dataset focuses on objects that may play a potential role in anomalies, while the second consists of videos containing anomalies or non-anomalies. This approach allows for the processing of both continuous video streams and finite videos, providing greater flexibility in detection.
Improving generalization of robot locomotion policies via Sharpness-Aware Reinforcement Learning
Venue: arXiv cs.RO 2024
Year: 2024

Reinforcement learning often requires extensive training data. Simulation-to-real transfer offers a promising approach to address this challenge in robotics. While differentiable simulators offer improved sample efficiency through exact gradients, they can be unstable in contact-rich environments and may lead to poor generalization. This paper introduces a novel approach integrating sharpness-aware optimization into gradient-based reinforcement learning algorithms. Our simulation results demonstrate that our method, tested on contact-rich environments, significantly enhances policy robustness to environmental variations and action perturbations while maintaining the sample efficiency of first-order methods. Specifically, our approach improves action noise tolerance compared to standard first-order methods and achieves generalization comparable to zeroth-order methods. This improvement stems from finding flatter minima in the loss landscape, associated with better generalization. Our work offers a promising solution to balance efficient learning and robust sim-to-real transfer in robotics, potentially bridging the gap between simulation and real-world performance.
A Comprehensive Content Verification System for ensuring Digital Integrity in the Age of Deep Fakes
Venue: arXiv cs.CR 2024
Year: 2024

In an era marked by the widespread sharing of digital content, the need for a robust content-integrity verification goes beyond the confines of individual social media platforms. While verified profiles (such as blue ticks on platforms like Instagram and X) have become synonymous with credibility, the content they share often traverses a complex network of interconnected platforms, by means of re-sharing, re-posting, etc., leaving a void in the authentication process of the content itself. With the advent of easily accessible AI tools (like DALL-E, Sora, and the tools that are explicitly built for generating deepfakes & face swaps), the risk of misinformation through social media platforms is growing exponentially. This paper discusses a solution, a Content Verification System, designed to authenticate images and videos shared as posts or stories across the digital landscape. Going beyond the limitations of blue ticks, this system empowers individuals and influencers to validate the authenticity of their digital footprint, safeguarding their reputation in an interconnected world.
Noncommutative Model Selection and the Data-Driven Estimation of Real Cohomology Groups
Venue: arXiv cs.CG 2024
Year: 2024

We propose three completely data-driven methods for estimating the real cohomology groups $H^k (X ; \mathbb{R})$ of a compact metric-measure space $(X, d_X, μ_X)$ embedded in a metric-measure space $(Y,d_Y,μ_Y)$, given a finite set of points $S$ sampled from a uniform distrbution $μ_X$ on $X$, possibly corrupted with noise from $Y$. We present the results of several computational experiments in the case that $X$ is embedded in $\mathbb{R}^n$, where two of the three algorithms performed well.
Handling irresolvable conflicts in the Semantic Web: an RDF-based conflict-tolerant version of the Deontic Traditional Scheme
Venue: arXiv cs.AI 2024
Year: 2024

This paper presents a new ontology that implements the well-known Deontic Traditional Scheme in RDFs and SPARQL, fit to handle irresolvable conflicts, i.e., situations in which two or more statements prescribe conflicting obligations, prohibitions, or permissions, with none of them being "stronger" than the other one(s). In our view, this paper marks a significant advancement in standard theoretical research in formal Deontic Logic. Most contemporary approaches in this field are confined to the propositional level, mainly focus on the notion of obligation, and lack implementations. The proposed framework is encoded in RDF, which is not only a first-order language but also the most widely used knowledge representation language, as it forms the foundation of the Semantic Web. Moreover, the proposed computational ontology formalizes all deontic modalities defined in the Deontic Traditional Scheme, without specifically focusing on obligations, and offers constructs to model and reason with various types of irresolvable conflicts, violations, and the interaction between deontic modalities and contextual constraints in a given state of affairs. To the best of our knowledge, no existing approach in the literature addresses all these aspects within a unified integrated framework. All examples presented and discussed in this paper, together with Java code and clear instructions to re-execute them locally, are available at https://github.com/liviorobaldo/conflict-tolerantDeonticTraditionalScheme
A Supercomputing Based Distributed Cloud Marketplace
Venue: arXiv cs.CR 2024
Year: 2024

The once mythological 51% attack has moved beyond the hypothetical and now poses a legitimate widespread threat to blockchain technology. Current blockchains provide inferior throughput capacity when compared to that of centralized systems, creating an obvious vulnerability which allows the 51% attack to occur within decentralized systems. Despite recent advancements in blockchain which introduce interesting models that achieve high throughputs with enhanced security and privacy, no current networks have evolved to deploy the optimal solution of combining scalability, security, and distributed systems to create a legitimate supercomputing enterprise-grade developer sandbox. In this paper, we introduce an infinitely scalable, secure, and high throughput blockchain capable of amassing supercomputer speeds with off-the-shelf hardware, LuluChain. LuluChain simplifies the blockchain model to obtain greater functionality, speed, scalability, privacy, and flexibility, that works to combat the inflated pricing models set by the oligopolistic cloud computing market as it requires minimal computational work. By eliminating the need for timestamp synchronization and majority agreement among all participants, LuluChain opens the door to reliable trust, low-cost instant transactions, and flexible instant smart contracts. The supercomputing, high throughput distributed system is the ideal foundation for an essential distributed cloud marketplace.
Deep Learning-Based Electricity Price Forecast for Virtual Bidding in Wholesale Electricity Market
Venue: arXiv cs.LG 2024
Year: 2024

Virtual bidding plays an important role in two-settlement electric power markets, as it can reduce discrepancies between day-ahead and real-time markets. Renewable energy penetration increases volatility in electricity prices, making accurate forecasting critical for virtual bidders, reducing uncertainty and maximizing profits. This study presents a Transformer-based deep learning model to forecast the price spread between real-time and day-ahead electricity prices in the ERCOT (Electric Reliability Council of Texas) market. The proposed model leverages various time-series features, including load forecasts, solar and wind generation forecasts, and temporal attributes. The model is trained under realistic constraints and validated using a walk-forward approach by updating the model every week. Based on the price spread prediction results, several trading strategies are proposed and the most effective strategy for maximizing cumulative profit under realistic market conditions is identified through backtesting. The results show that the strategy of trading only at the peak hour with a precision score of over 50% produces nearly consistent profit over the test period. The proposed method underscores the importance of an accurate electricity price forecasting model and introduces a new method of evaluating the price forecast model from a virtual bidder's perspective, providing valuable insights for future research.
Targeted Therapy in Data Removal: Object Unlearning Based on Scene Graphs
Venue: arXiv cs.CV 2024
Year: 2024

Users may inadvertently upload personally identifiable information (PII) to Machine Learning as a Service (MLaaS) providers. When users no longer want their PII on these services, regulations like GDPR and COPPA mandate a right to forget for these users. As such, these services seek efficient methods to remove the influence of specific data points. Thus the introduction of machine unlearning. Traditionally, unlearning is performed with the removal of entire data samples (sample unlearning) or whole features across the dataset (feature unlearning). However, these approaches are not equipped to handle the more granular and challenging task of unlearning specific objects within a sample. To address this gap, we propose a scene graph-based object unlearning framework. This framework utilizes scene graphs, rich in semantic representation, transparently translate unlearning requests into actionable steps. The result, is the preservation of the overall semantic integrity of the generated image, bar the unlearned object. Further, we manage high computational overheads with influence functions to approximate the unlearning process. For validation, we evaluate the unlearned object's fidelity in outputs under the tasks of image reconstruction and image synthesis. Our proposed framework demonstrates improved object unlearning outcomes, with the preservation of unrequested samples in contrast to sample and feature learning methods. This work addresses critical privacy issues by increasing the granularity of targeted machine unlearning through forgetting specific object-level details without sacrificing the utility of the whole data sample or dataset feature.