OpenRaiser/Intern-Atlas huggingface dataset:
AR-Facilitated Safety Inspection and Fall Hazard Detection on Construction Sites
Venue: arXiv cs.HC 2024
Year: 2024

Together with industry experts, we are exploring the potential of head-mounted augmented reality to facilitate safety inspections on high-rise construction sites. A particular concern in the industry is inspecting perimeter safety screens on higher levels of construction sites, intended to prevent falls of people and objects. We aim to support workers performing this inspection task by tracking which parts of the safety screens have been inspected. We use machine learning to automatically detect gaps in the perimeter screens that require closer inspection and remediation and to automate reporting. This work-in-progress paper describes the problem, our early progress, concerns around worker privacy, and the possibilities to mitigate these.
FedPAW: Federated Learning with Personalized Aggregation Weights for Urban Vehicle Speed Prediction
Venue: arXiv cs.AI 2024
Year: 2024

Vehicle speed prediction is crucial for intelligent transportation systems, promoting more reliable autonomous driving by accurately predicting future vehicle conditions. Due to variations in drivers' driving styles and vehicle types, speed predictions for different target vehicles may significantly differ. Existing methods may not realize personalized vehicle speed prediction while protecting drivers' data privacy. We propose a Federated learning framework with Personalized Aggregation Weights (FedPAW) to overcome these challenges. This method captures client-specific information by measuring the weighted mean squared error between the parameters of local models and global models. The server sends tailored aggregated models to clients instead of a single global model, without incurring additional computational and communication overhead for clients. To evaluate the effectiveness of FedPAW, we collected driving data in urban scenarios using the autonomous driving simulator CARLA, employing an LSTM-based Seq2Seq model with a multi-head attention mechanism to predict the future speed of target vehicles. The results demonstrate that our proposed FedPAW ranks lowest in prediction error within the time horizon of 10 seconds, with a 0.8% reduction in test MAE, compared to eleven representative benchmark baselines. The source code of FedPAW and dataset CarlaVSP are open-accessed at: https://github.com/heyuepeng/PFLlibVSP and https://pan.baidu.com/s/1qs8fxUvSPERV3C9i6pfUIw?pwd=tl3e.
The Seeds of the FUTURE Sprout from History: Fuzzing for Unveiling Vulnerabilities in Prospective Deep-Learning Libraries
Venue: arXiv cs.SE 2024
Year: 2024

The widespread application of large language models (LLMs) underscores the importance of deep learning (DL) technologies that rely on foundational DL libraries such as PyTorch and TensorFlow. Despite their robust features, these libraries face challenges with scalability and adaptation to rapid advancements in the LLM community. In response, tech giants like Apple and Huawei are developing their own DL libraries to enhance performance, increase scalability, and safeguard intellectual property. Ensuring the security of these libraries is crucial, with fuzzing being a vital solution. However, existing fuzzing frameworks struggle with target flexibility, effectively testing bug-prone API sequences, and leveraging the limited available information in new libraries. To address these limitations, we propose FUTURE, the first universal fuzzing framework tailored for newly introduced and prospective DL libraries. FUTURE leverages historical bug information from existing libraries and fine-tunes LLMs for specialized code generation. This strategy helps identify bugs in new libraries and uses insights from these libraries to enhance security in existing ones, creating a cycle from history to future and back. To evaluate FUTURE's effectiveness, we conduct comprehensive evaluations on three newly introduced DL libraries. Evaluation results demonstrate that FUTURE significantly outperforms existing fuzzers in bug detection, success rate of bug reproduction, validity rate of code generation, and API coverage. Notably, FUTURE has detected 148 bugs across 452 targeted APIs, including 142 previously unknown bugs. Among these, 10 have been assigned CVE IDs. Additionally, FUTURE detects 7 bugs in PyTorch, demonstrating its ability to enhance security in existing libraries in reverse.
The "LLM World of Words" English free association norms generated by large language models
Venue: arXiv cs.CL 2024
Year: 2024

Free associations have been extensively used in cognitive psychology and linguistics for studying how conceptual knowledge is organized. Recently, the potential of applying a similar approach for investigating the knowledge encoded in LLMs has emerged, specifically as a method for investigating LLM biases. However, the absence of large-scale LLM-generated free association norms that are comparable with human-generated norms is an obstacle to this new research direction. To address this limitation, we create a new dataset of LLM-generated free association norms modeled after the "Small World of Words" (SWOW) human-generated norms consisting of approximately 12,000 cue words. We prompt three LLMs, namely Mistral, Llama3, and Haiku, with the same cues as those in the SWOW norms to generate three novel comparable datasets, the "LLM World of Words" (LWOW). Using both SWOW and LWOW norms, we construct cognitive network models of semantic memory that represent the conceptual knowledge possessed by humans and LLMs. We demonstrate how these datasets can be used for investigating implicit biases in humans and LLMs, such as the harmful gender stereotypes that are prevalent both in society and LLM outputs.
A Versatile Influence Function for Data Attribution with Non-Decomposable Loss
Venue: arXiv cs.LG 2024
Year: 2024

Influence function, a technique rooted in robust statistics, has been adapted in modern machine learning for a novel application: data attribution -- quantifying how individual training data points affect a model's predictions. However, the common derivation of influence functions in the data attribution literature is limited to loss functions that can be decomposed into a sum of individual data point losses, with the most prominent examples known as M-estimators. This restricts the application of influence functions to more complex learning objectives, which we refer to as non-decomposable losses, such as contrastive or ranking losses, where a unit loss term depends on multiple data points and cannot be decomposed further. In this work, we bridge this gap by revisiting the general formulation of influence function from robust statistics, which extends beyond M-estimators. Based on this formulation, we propose a novel method, the Versatile Influence Function (VIF), that can be straightforwardly applied to machine learning models trained with any non-decomposable loss. In comparison to the classical approach in statistics, the proposed VIF is designed to fully leverage the power of auto-differentiation, hereby eliminating the need for case-specific derivations of each loss function. We demonstrate the effectiveness of VIF across three examples: Cox regression for survival analysis, node embedding for network analysis, and listwise learning-to-rank for information retrieval. In all cases, the influence estimated by VIF closely resembles the results obtained by brute-force leave-one-out retraining, while being up to $10^3$ times faster to compute. We believe VIF represents a significant advancement in data attribution, enabling efficient influence-function-based attribution across a wide range of machine learning paradigms, with broad potential for practical use cases.
MoTrans: Customized Motion Transfer with Text-driven Video Diffusion Models
Venue: arXiv cs.CV 2024
Year: 2024

Existing pretrained text-to-video (T2V) models have demonstrated impressive abilities in generating realistic videos with basic motion or camera movement. However, these models exhibit significant limitations when generating intricate, human-centric motions. Current efforts primarily focus on fine-tuning models on a small set of videos containing a specific motion. They often fail to effectively decouple motion and the appearance in the limited reference videos, thereby weakening the modeling capability of motion patterns. To this end, we propose MoTrans, a customized motion transfer method enabling video generation of similar motion in new context. Specifically, we introduce a multimodal large language model (MLLM)-based recaptioner to expand the initial prompt to focus more on appearance and an appearance injection module to adapt appearance prior from video frames to the motion modeling process. These complementary multimodal representations from recaptioned prompt and video frames promote the modeling of appearance and facilitate the decoupling of appearance and motion. In addition, we devise a motion-specific embedding for further enhancing the modeling of the specific motion. Experimental results demonstrate that our method effectively learns specific motion pattern from singular or multiple reference videos, performing favorably against existing methods in customized video generation.
DeFi: Concepts and Ecosystem
Venue: arXiv cs.CE 2024
Year: 2024

This paper investigates the evolving landscape of decentralized finance (DeFi) by examining its foundational concepts, research trends, and ecosystem. A bibliometric analysis was conducted to identify thematic clusters and track the evolution of DeFi research. Additionally, a thematic review was performed to analyze the roles and interactions of key participants within the DeFi ecosystem, focusing on its opportunities and inherent risks. The bibliometric analysis identified a progression in research priorities, transitioning from an initial focus on technological innovation to addressing sustainability, environmental impacts, and regulatory challenges. Key thematic clusters include decentralization, smart contracts, tokenization, and sustainability concerns. The analysis of participants highlighted the roles of developers, liquidity providers, auditors, and regulators while identifying critical risks such as smart contract vulnerabilities, liquidity constraints, and regulatory uncertainties. The study underlines the transformative potential of DeFi to enhance financial inclusion and transparency while emphasizing the need for robust security frameworks and regulatory oversight to ensure long-term stability. This paper comprehensively explains the DeFi ecosystem by integrating bibliometric and thematic analyses. It offers valuable insights for researchers, practitioners, and policymakers, contributing to the ongoing discourse on the sustainable development and integration of DeFi into the global financial system.
HoloDrive: Holistic 2D-3D Multi-Modal Street Scene Generation for Autonomous Driving
Venue: arXiv cs.CV 2024
Year: 2024

Generative models have significantly improved the generation and prediction quality on either camera images or LiDAR point clouds for autonomous driving. However, a real-world autonomous driving system uses multiple kinds of input modality, usually cameras and LiDARs, where they contain complementary information for generation, while existing generation methods ignore this crucial feature, resulting in the generated results only covering separate 2D or 3D information. In order to fill the gap in 2D-3D multi-modal joint generation for autonomous driving, in this paper, we propose our framework, \emph{HoloDrive}, to jointly generate the camera images and LiDAR point clouds. We employ BEV-to-Camera and Camera-to-BEV transform modules between heterogeneous generative models, and introduce a depth prediction branch in the 2D generative model to disambiguate the un-projecting from image space to BEV space, then extend the method to predict the future by adding temporal structure and carefully designed progressive training. Further, we conduct experiments on single frame generation and world model benchmarks, and demonstrate our method leads to significant performance gains over SOTA methods in terms of generation metrics.
CellSeg1: Robust Cell Segmentation with One Training Image
Venue: arXiv cs.CV 2024
Year: 2024

Recent trends in cell segmentation have shifted towards universal models to handle diverse cell morphologies and imaging modalities. However, for continuously emerging cell types and imaging techniques, these models still require hundreds or thousands of annotated cells for fine-tuning. We introduce CellSeg1, a practical solution for segmenting cells of arbitrary morphology and modality with a few dozen cell annotations in 1 image. By adopting Low-Rank Adaptation of the Segment Anything Model (SAM), we achieve robust cell segmentation. Tested on 19 diverse cell datasets, CellSeg1 trained on 1 image achieved 0.81 average mAP at 0.5 IoU, performing comparably to existing models trained on over 500 images. It also demonstrated superior generalization in cross-dataset tests on TissueNet. We found that high-quality annotation of a few dozen densely packed cells of varied sizes is key to effective segmentation. CellSeg1 provides an efficient solution for cell segmentation with minimal annotation effort.
Network Simulation with Complex Cyber-attack Scenarios
Venue: arXiv cs.CR 2024
Year: 2024

Network Intrusion Detection (NID) systems can benefit from Machine Learning (ML) models to detect complex cyber-attacks. However, to train them with a great amount of high-quality data, it is necessary to perform reliable simulations of multiple interacting machines. This paper presents a network simulation solution for the creation of NID datasets with complex attack scenarios. This solution was integrated in the Airbus CyberRange platform to benefit from its simulation capabilities of generating benign and malicious traffic patterns that represent realistic cyber-attacks targeting a computer network. A realistic vulnerable network topology was configured in the CyberRange and three different attack scenarios were implemented: Man-in-the-Middle (MitM), Denial-of-Service (DoS), and Brute-Force (BF).
MamKPD: A Simple Mamba Baseline for Real-Time 2D Keypoint Detection
Venue: arXiv cs.CV 2024
Year: 2024

Real-time 2D keypoint detection plays an essential role in computer vision. Although CNN-based and Transformer-based methods have achieved breakthrough progress, they often fail to deliver superior performance and real-time speed. This paper introduces MamKPD, the first efficient yet effective mamba-based pose estimation framework for 2D keypoint detection. The conventional Mamba module exhibits limited information interaction between patches. To address this, we propose a lightweight contextual modeling module (CMM) that uses depth-wise convolutions to model inter-patch dependencies and linear layers to distill the pose cues within each patch. Subsequently, by combining Mamba for global modeling across all patches, MamKPD effectively extracts instances' pose information. We conduct extensive experiments on human and animal pose estimation datasets to validate the effectiveness of MamKPD. Our MamKPD-L achieves 77.3% AP on the COCO dataset with 1492 FPS on an NVIDIA GTX 4090 GPU. Moreover, MamKPD achieves state-of-the-art results on the MPII dataset and competitive results on the AP-10K dataset while saving 85% of the parameters compared to ViTPose. Our project page is available at https://mamkpd.github.io/.
Definite Descriptions and Hybrid Tense Logic
Venue: arXiv cs.LO 2024
Year: 2024

We provide a version of first-order hybrid tense logic with predicate abstracts and definite descriptions as the only non-rigid terms. It is formalised by means of a tableau calculus working on sat-formulas. A particular theory of DD exploited here is essentially based on the approach of Russell, but with descriptions treated as genuine terms. However, the reductionist aspect of the Russellian approach is retained in several ways. Moreover, a special form of tense definite descriptions is formally developed. A constructive proof of the interpolation theorem for this calculus is given, which is an extension of the result provided by Blackburn and Marx.
Scaling Law for Language Models Training Considering Batch Size
Venue: arXiv cs.CL 2024
Year: 2024

Large language models (LLMs) have made remarkable advances in recent years, with scaling laws playing a critical role in this rapid progress. In this paper, we empirically investigate how a critical hyper-parameter, i.e., the global batch size, influences the LLM training prdocess. We begin by training language models ranging from 125 million to 2.6 billion parameters, using up to 300 billion high-quality tokens. Through these experiments, we establish a basic scaling law on model size and training data amount. We then examine how varying batch sizes and learning rates affect the convergence and generalization of these models. Our analysis yields batch size scaling laws under two different cases: with a fixed compute budget, and with a fixed amount of training data. Extrapolation experiments on models of increasing sizes validate our predicted laws, which provides guidance for optimizing LLM training strategies under specific resource constraints.
Traversing the Subspace of Adversarial Patches
Venue: arXiv cs.CV 2024
Year: 2024

Despite ongoing research on the topic of adversarial examples in deep learning for computer vision, some fundamentals of the nature of these attacks remain unclear. As the manifold hypothesis posits, high-dimensional data tends to be part of a low-dimensional manifold. To verify the thesis with adversarial patches, this paper provides an analysis of a set of adversarial patches and investigates the reconstruction abilities of three different dimensionality reduction methods. Quantitatively, the performance of reconstructed patches in an attack setting is measured and the impact of sampled patches from the latent space during adversarial training is investigated. The evaluation is performed on two publicly available datasets for person detection. The results indicate that more sophisticated dimensionality reduction methods offer no advantages over a simple principal component analysis.
Beyond SICP -- Design and Implementation of a Notional Machine for Scheme
Venue: arXiv cs.PL 2024
Year: 2024

Computer science education has been at the heart of Scheme from the beginning. The language was designed in the 1970s concurrently with the MIT course 6.001 and the textbook "Structure and Interpretation of Computer Programs" (SICP). To explain the scope of variables at run time in the presence of higher-order procedures, SICP introduces a mental model called the environment model, along with a pictorial representation of environments and data structures. Recently, the concept of notional machines has emerged in computer science education: a predictive set of abstractions that define the structure and behavior of a computational device. Proponents of notional machines argue that learners benefit when complex dynamic concepts such as the computational structure of Scheme are accompanied with concise notional machines. In this paper, we start with a sublanguage of Scheme sufficient for all programs in SICP that we call SICP Scheme. We extend the environment model to a full notional machine for SICP Scheme that is simple enough to serve as the central mental model in a CS1 course and demonstrate the machine with computer-generated visualizations. Moving beyond SICP Scheme, we show how the notional machine can be further extended to explain Scheme's call/cc and thus make this powerful concept accessible to beginners through a coherent mental model. The presented notional machine serves as the core of a web-based implementation of Scheme that is under development at our university.
SfM-Free 3D Gaussian Splatting via Hierarchical Training
Venue: arXiv cs.CV 2024
Year: 2024

Standard 3D Gaussian Splatting (3DGS) relies on known or pre-computed camera poses and a sparse point cloud, obtained from structure-from-motion (SfM) preprocessing, to initialize and grow 3D Gaussians. We propose a novel SfM-Free 3DGS (SFGS) method for video input, eliminating the need for known camera poses and SfM preprocessing. Our approach introduces a hierarchical training strategy that trains and merges multiple 3D Gaussian representations -- each optimized for specific scene regions -- into a single, unified 3DGS model representing the entire scene. To compensate for large camera motions, we leverage video frame interpolation models. Additionally, we incorporate multi-source supervision to reduce overfitting and enhance representation. Experimental results reveal that our approach significantly surpasses state-of-the-art SfM-free novel view synthesis methods. On the Tanks and Temples dataset, we improve PSNR by an average of 2.25dB, with a maximum gain of 3.72dB in the best scene. On the CO3D-V2 dataset, we achieve an average PSNR boost of 1.74dB, with a top gain of 3.90dB. The code is available at https://github.com/jibo27/3DGS_Hierarchical_Training.
3DSceneEditor: Controllable 3D Scene Editing with Gaussian Splatting
Venue: arXiv cs.CV 2024
Year: 2024

The creation of 3D scenes has traditionally been both labor-intensive and costly, requiring designers to meticulously configure 3D assets and environments. Recent advancements in generative AI, including text-to-3D and image-to-3D methods, have dramatically reduced the complexity and cost of this process. However, current techniques for editing complex 3D scenes continue to rely on generally interactive multi-step, 2D-to-3D projection methods and diffusion-based techniques, which often lack precision in control and hamper real-time performance. In this work, we propose 3DSceneEditor, a fully 3D-based paradigm for real-time, precise editing of intricate 3D scenes using Gaussian Splatting. Unlike conventional methods, 3DSceneEditor operates through a streamlined 3D pipeline, enabling direct manipulation of Gaussians for efficient, high-quality edits based on input prompts.The proposed framework (i) integrates a pre-trained instance segmentation model for semantic labeling; (ii) employs a zero-shot grounding approach with CLIP to align target objects with user prompts; and (iii) applies scene modifications, such as object addition, repositioning, recoloring, replacing, and deletion directly on Gaussians. Extensive experimental results show that 3DSceneEditor achieves superior editing precision and speed with respect to current SOTA 3D scene editing approaches, establishing a new benchmark for efficient and interactive 3D scene customization.