OpenRaiser/Intern-Atlas huggingface dataset:
SafeSynthDP: Leveraging Large Language Models for Privacy-Preserving Synthetic Data Generation Using Differential Privacy
Venue: arXiv cs.LG 2024
Year: 2024

Machine learning (ML) models frequently rely on training data that may include sensitive or personal information, raising substantial privacy concerns. Legislative frameworks such as the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act (CCPA) have necessitated the development of strategies that preserve privacy while maintaining the utility of data. In this paper, we investigate the capability of Large Language Models (LLMs) to generate synthetic datasets integrated with Differential Privacy (DP) mechanisms, thereby enabling data-driven research and model training without direct exposure of sensitive information. Our approach incorporates DP-based noise injection methods, including Laplace and Gaussian distributions, into the data generation process. We then evaluate the utility of these DP-enhanced synthetic datasets by comparing the performance of ML models trained on them against models trained on the original data. To substantiate privacy guarantees, we assess the resilience of the generated synthetic data to membership inference attacks and related threats. The experimental results demonstrate that integrating DP within LLM-driven synthetic data generation offers a viable balance between privacy protection and data utility. This study provides a foundational methodology and insight into the privacy-preserving capabilities of LLMs, paving the way for compliant and effective ML research and applications.
Enhancing Visual Representation for Text-based Person Searching
Venue: arXiv cs.CV 2024
Year: 2024

Text-based person search aims to retrieve the matched pedestrians from a large-scale image database according to the text description. The core difficulty of this task is how to extract effective details from pedestrian images and texts, and achieve cross-modal alignment in a common latent space. Prior works adopt image and text encoders pre-trained on unimodal data to extract global and local features from image and text respectively, and then global-local alignment is achieved explicitly. However, these approaches still lack the ability of understanding visual details, and the retrieval accuracy is still limited by identity confusion. In order to alleviate the above problems, we rethink the importance of visual features for text-based person search, and propose VFE-TPS, a Visual Feature Enhanced Text-based Person Search model. It introduces a pre-trained multimodal backbone CLIP to learn basic multimodal features and constructs Text Guided Masked Image Modeling task to enhance the model's ability of learning local visual details without explicit annotation. In addition, we design Identity Supervised Global Visual Feature Calibration task to guide the model learn identity-aware global visual features. The key finding of our study is that, with the help of our proposed auxiliary tasks, the knowledge embedded in the pre-trained CLIP model can be successfully adapted to text-based person search task, and the model's visual understanding ability is significantly enhanced. Experimental results on three benchmarks demonstrate that our proposed model exceeds the existing approaches, and the Rank-1 accuracy is significantly improved with a notable margin of about $1\%\sim9\%$. Our code can be found at https://github.com/zhangweifeng1218/VFE_TPS.
Environmental and Economic Impact of I/O Device Obsolescence
Venue: arXiv cs.CY 2024
Year: 2024

This paper analyzes the proportion of Input/output devices made obsolete by changes in technology generations. This obsolescence may be by new software/hardware generations rendering otherwise functional devices unusable. Concluding with brief analysis on the economic and environmental impacts of the e-waste produced.
Blockchain-Empowered Cyber-Secure Federated Learning for Trustworthy Edge Computing
Venue: arXiv cs.DC 2024
Year: 2024

Federated Learning (FL) is a privacy-preserving distributed machine learning scheme, where each participant data remains on the participating devices and only the local model generated utilizing the local computational power is transmitted throughout the database. However, the distributed computational nature of FL creates the necessity to develop a mechanism that can remotely trigger any network agents, track their activities, and prevent threats to the overall process posed by malicious participants. Particularly, the FL paradigm may become vulnerable due to an active attack from the network participants, called a poisonous attack. In such an attack, the malicious participant acts as a benign agent capable of affecting the global model quality by uploading an obfuscated poisoned local model update to the server. This paper presents a cross-device FL model that ensures trustworthiness, fairness, and authenticity in the underlying FL training process. We leverage trustworthiness by constructing a reputation-based trust model based on contributions of agents toward model convergence. We ensure fairness by identifying and removing malicious agents from the training process through an outlier detection technique. Further, we establish authenticity by generating a token for each participating device through a distributed sensing mechanism and storing that unique token in a blockchain smart contract. Further, we insert the trust scores of all agents into a blockchain and validate their reputations using various consensus mechanisms that consider the computational task.
Online Adaptive Platoon Control for Connected and Automated Vehicles via Physics Enhanced Residual Learning
Venue: arXiv cs.RO 2024
Year: 2024

This paper introduces a physics enhanced residual learning (PERL) framework for connected and automated vehicle (CAV) platoon control, addressing the dynamics and unpredictability inherent to platoon systems. The framework first develops a physics-based controller to model vehicle dynamics, using driving speed as input to optimize safety and efficiency. Then the residual controller, based on neural network (NN) learning, enriches the prior knowledge of the physical model and corrects residuals caused by vehicle dynamics. By integrating the physical model with data-driven online learning, the PERL framework retains the interpretability and transparency of physics-based models and enhances the adaptability and precision of data-driven learning, achieving significant improvements in computational efficiency and control accuracy in dynamic scenarios. Simulation and robot car platform tests demonstrate that PERL significantly outperforms pure physical and learning models, reducing average cumulative absolute position and speed errors by up to 58.5% and 40.1% (physical model) and 58.4% and 47.7% (NN model). The reduced-scale robot car platform tests further validate the adaptive PERL framework's superior accuracy and rapid convergence under dynamic disturbances, reducing position and speed cumulative errors by 72.73% and 99.05% (physical model) and 64.71% and 72.58% (NN model). PERL enhances platoon control performance through online parameter updates when external disturbances are detected. Results demonstrate the advanced framework's exceptional accuracy and rapid convergence capabilities, proving its effectiveness in maintaining platoon stability under diverse conditions.
Learning to Rank Pre-trained Vision-Language Models for Downstream Tasks
Venue: arXiv cs.CV 2024
Year: 2024

Vision language models (VLMs) like CLIP show stellar zero-shot capability on classification benchmarks. However, selecting the VLM with the highest performance on the unlabeled downstream task is non-trivial. Existing VLM selection methods focus on the class-name-only setting, relying on a supervised large-scale dataset and large language models, which may not be accessible or feasible during deployment. This paper introduces the problem of \textbf{unsupervised vision-language model selection}, where only unsupervised downstream datasets are available, with no additional information provided. To solve this problem, we propose a method termed Visual-tExtual Graph Alignment (VEGA), to select VLMs without any annotations by measuring the alignment of the VLM between the two modalities on the downstream task. VEGA is motivated by the pretraining paradigm of VLMs, which aligns features with the same semantics from the visual and textual modalities, thereby mapping both modalities into a shared representation space. Specifically, we first construct two graphs on the vision and textual features, respectively. VEGA is then defined as the overall similarity between the visual and textual graphs at both node and edge levels. Extensive experiments across three different benchmarks, covering a variety of application scenarios and downstream datasets, demonstrate that VEGA consistently provides reliable and accurate estimates of VLMs' performance on unlabeled downstream tasks.
Powering the Future: Innovations in Electric Vehicle Battery Recycling
Venue: arXiv cs.CE 2024
Year: 2024

The global shift towards electric vehicles (EVs) as a sustainable alternative to traditional gasoline-powered cars has triggered a significant rise in the demand for lithium-ion batteries. However, as the adoption of EVs grows, the issue of battery disposal and recycling has emerged as a critical challenge. The recycling of EV batteries is essential not only for reducing the environmental impact of battery waste but also for ensuring the sustainable supply of critical raw materials such as lithium, cobalt, and nickel. This paper explores recent innovations in the field of electric vehicle battery recycling, examining advanced techniques such as direct recycling, hydrometallurgical processes, and sustainable battery design. It also highlights the role of policy and industry collaboration in improving recycling infrastructure and addressing the economic and environmental challenges associated with battery waste. By focusing on both the technical and regulatory aspects of EV battery recycling, this paper aims to provide a comprehensive overview of the state of the industry and the future outlook for recycling technologies, ultimately paving the way for a cleaner, more sustainable future in transportation.
Modeling and Simulating Agent-Based City Migration Using Conway's Game of Life
Venue: arXiv cs.CY 2024
Year: 2024

Agent-based modeling (ABM) has become a cornerstone of complexity science, enabling the study of heterogeneous agents interacting within dynamic environments. Among ABM frameworks, John Conway's Game of Life (GoL) stands out for its simplicity and ability to generate emergent macroscopic patterns from basic microscopic rules. In this paper, we propose and implement a novel GoL-based framework to simulate urban migration dynamics. Using a grid-within-a-grid approach, our approach encodes probabilistic tendencies for out-migration due to densification and sparsification, simulating the evolution of population centers. By initializing GoL grids with different distributions and parameterizing migration preferences, we explore how urban structures emerge and stabilize over time. Through a series of experiments, we demonstrate that even with simple rules, this framework shows promise for understanding emergent urban phenomena, providing insights into city growth and structure. Methodologically, our framework offers a versatile and computationally efficient tool for studying urban migration patterns, contributing to the broader application of ABMs in computational urban social science.
HFI: A unified framework for training-free detection and implicit watermarking of latent diffusion model generated images
Venue: arXiv cs.CV 2024
Year: 2024

Dramatic advances in the quality of the latent diffusion models (LDMs) also led to the malicious use of AI-generated images. While current AI-generated image detection methods assume the availability of real/AI-generated images for training, this is practically limited given the vast expressibility of LDMs. This motivates the training-free detection setup where no related data are available in advance. The existing LDM-generated image detection method assumes that images generated by LDM are easier to reconstruct using an autoencoder than real images. However, we observe that this reconstruction distance is overfitted to background information, leading the current method to underperform in detecting images with simple backgrounds. To address this, we propose a novel method called HFI. Specifically, by viewing the autoencoder of LDM as a downsampling-upsampling kernel, HFI measures the extent of aliasing, a distortion of high-frequency information that appears in the reconstructed image. HFI is training-free, efficient, and consistently outperforms other training-free methods in detecting challenging images generated by various generative models. We also show that HFI can successfully detect the images generated from the specified LDM as a means of implicit watermarking. HFI outperforms the best baseline method while achieving magnitudes of
M$^3$oralBench: A MultiModal Moral Benchmark for LVLMs
Venue: arXiv cs.CV 2024
Year: 2024

Recently, large foundation models, including large language models (LLMs) and large vision-language models (LVLMs), have become essential tools in critical fields such as law, finance, and healthcare. As these models increasingly integrate into our daily life, it is necessary to conduct moral evaluation to ensure that their outputs align with human values and remain within moral boundaries. Previous works primarily focus on LLMs, proposing moral datasets and benchmarks limited to text modality. However, given the rapid development of LVLMs, there is still a lack of multimodal moral evaluation methods. To bridge this gap, we introduce M$^3$oralBench, the first MultiModal Moral Benchmark for LVLMs. M$^3$oralBench expands the everyday moral scenarios in Moral Foundations Vignettes (MFVs) and employs the text-to-image diffusion model, SD3.0, to create corresponding scenario images. It conducts moral evaluation across six moral foundations of Moral Foundations Theory (MFT) and encompasses tasks in moral judgement, moral classification, and moral response, providing a comprehensive assessment of model performance in multimodal moral understanding and reasoning. Extensive experiments on 10 popular open-source and closed-source LVLMs demonstrate that M$^3$oralBench is a challenging benchmark, exposing notable moral limitations in current models. Our benchmark is publicly available.
Beam Codebook Refinement for mmWave Devices with Random Orientations: Concept and Experimental Validation
Venue: arXiv cs.IT 2024
Year: 2024

There is a growing interest in codebook-based beam-steering for millimeter-wave (mmWave) systems due to its potential for low complexity and rapid beam search. A key focus of recent research has been the design of codebooks that strike a trade-off between achievable gain and codebook size, which directly impacts beam search time. Statistical approaches have shown promise by leveraging the likelihood that certain beam directions (equivalently, sets of phase-shifter configurations) are more probable than others. Such approaches are shown to be valid for static, non-rotating transmission stations such as base stations. However, for the case of user terminals that are constantly changing orientation, the possible phase-shifter configurations become equally probable, rendering statistical methods less relevant. On the other hand, user terminals come with a large number of possible steering vector configurations, which can span up to six orders of magnitude. Therefore, efficient solutions to reduce the codebook size (set of possible steering vectors) without compromising array gain are needed. We address this challenge by proposing a novel and practical codebook refinement technique, aiming to reduce the codebook size while maintaining array gain within $γ$ dB of the maximum achievable gain at any random orientation of the user terminal. We project that a steering vector at a given angle could effectively cover adjacent angles with a small gain loss compared to the maximum achievable gain. We demonstrate experimentally that it is possible to reduce the codebook size from $1024^{16}$ to just a few configurations (e.g., less than ten), covering all angles while maintaining the gain within $γ=3$ dB of the maximum achievable gain.
Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering
Venue: arXiv cs.CV 2024
Year: 2024

Multimodal large language models (MLLMs), such as GPT-4o, Gemini, LLaVA, and Flamingo, have made significant progress in integrating visual and textual modalities, excelling in tasks like visual question answering (VQA), image captioning, and content retrieval. They can generate coherent and contextually relevant descriptions of images. However, they still face challenges in accurately identifying and counting objects and determining their spatial locations, particularly in complex scenes with overlapping or small objects. To address these limitations, we propose a novel framework based on multimodal retrieval-augmented generation (RAG), which introduces structured scene graphs to enhance object recognition, relationship identification, and spatial understanding within images. Our framework improves the MLLM's capacity to handle tasks requiring precise visual descriptions, especially in scenarios with challenging perspectives, such as aerial views or scenes with dense object arrangements. Finally, we conduct extensive experiments on the VG-150 dataset that focuses on first-person visual understanding and the AUG dataset that involves aerial imagery. The results show that our approach consistently outperforms existing MLLMs in VQA tasks, which stands out in recognizing, localizing, and quantifying objects in different spatial contexts and provides more accurate visual descriptions.
Efficient Parallel Genetic Algorithm for Perturbed Substructure Optimization in Complex Network
Venue: arXiv cs.NE 2024
Year: 2024

Evolutionary computing, particularly genetic algorithm (GA), is a combinatorial optimization method inspired by natural selection and the transmission of genetic information, which is widely used to identify optimal solutions to complex problems through simulated programming and iteration. Due to its strong adaptability, flexibility, and robustness, GA has shown significant performance and potentiality on perturbed substructure optimization (PSSO), an important graph mining problem that achieves its goals by modifying network structures. However, the efficiency and practicality of GA-based PSSO face enormous challenges due to the complexity and diversity of application scenarios. While some research has explored acceleration frameworks in evolutionary computing, their performance on PSSO remains limited due to a lack of scenario generalizability. Based on these, this paper is the first to present the GA-based PSSO Acceleration framework (GAPA), which simplifies the GA development process and supports distributed acceleration. Specifically, it reconstructs the genetic operation and designs a development framework for efficient parallel acceleration. Meanwhile, GAPA includes an extensible library that optimizes and accelerates 10 PSSO algorithms, covering 4 crucial tasks for graph mining. Comprehensive experiments on 18 datasets across 4 tasks and 10 algorithms effectively demonstrate the superiority of GAPA, achieving an average of 4x the acceleration of Evox. The repository is in https://github.com/NetAlsGroup/GAPA.
Reconfiguration of unit squares and disks: PSPACE-hardness in simple settings
Venue: arXiv cs.CG 2024
Year: 2024

We study two well-known reconfiguration problems. Given a start and a target configuration of geometric objects in a polygon, we wonder whether we can move the objects from the start configuration to the target configuration while avoiding collisions between the objects and staying within the polygon. Problems of this type have been considered since the early 80s by roboticists and computational geometers. In this paper, we study some of the simplest possible variants where the objects are unlabeled unit squares or unit disks. In unlabeled reconfiguration, the objects are identical, so that any object is allowed to end at any of the targets positions. We show that it is PSPACE-hard to decide whether there exists a reconfiguration of unit squares even in a simple polygon. Previously, it was only known to be PSPACE-hard in a polygon with holes [Solovey and Halperin, Int. J. Robotics Res. 2016]. Our proof is based on a result of independent interest, namely that reconfiguration between two satisfying assignments of a formula of Monotone-Planar-3SAT is also PSPACE-complete. The reduction from reconfiguration of Monotone-Planar-3SAT to reconfiguration of unit squares extends techniques recently developed to show NP-hardness of packing unit squares in a simple polygon [Abrahamsen and Stade, FOCS 2024]. We also show PSPACE-hardness of reconfiguration of unit disks in a polygon with holes. Previously, it was only known that reconfiguration of disks of two different sizes was PSPACE-hard [Brocken, van der Heijden, Kostitsyna, Lo-Wong and Surtel, FUN 2021].
EdgeRAG: Online-Indexed RAG for Edge Devices
Venue: arXiv cs.LG 2024
Year: 2024

Deploying Retrieval Augmented Generation (RAG) on resource-constrained edge devices is challenging due to limited memory and processing power. In this work, we propose EdgeRAG which addresses the memory constraint by pruning embeddings within clusters and generating embeddings on-demand during retrieval. To avoid the latency of generating embeddings for large tail clusters, EdgeRAG pre-computes and stores embeddings for these clusters, while adaptively caching remaining embeddings to minimize redundant computations and further optimize latency. The result from BEIR suite shows that EdgeRAG offers significant latency reduction over the baseline IVF index, but with similar generation quality while allowing all of our evaluated datasets to fit into the memory.
Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Model
Venue: arXiv cs.CV 2024
Year: 2024

We introduce Vinci, a real-time embodied smart assistant built upon an egocentric vision-language model. Designed for deployment on portable devices such as smartphones and wearable cameras, Vinci operates in an "always on" mode, continuously observing the environment to deliver seamless interaction and assistance. Users can wake up the system and engage in natural conversations to ask questions or seek assistance, with responses delivered through audio for hands-free convenience. With its ability to process long video streams in real-time, Vinci can answer user queries about current observations and historical context while also providing task planning based on past interactions. To further enhance usability, Vinci integrates a video generation module that creates step-by-step visual demonstrations for tasks that require detailed guidance. We hope that Vinci can establish a robust framework for portable, real-time egocentric AI systems, empowering users with contextual and actionable insights. We release the complete implementation for the development of the device in conjunction with a demo web platform to test uploaded videos at https://github.com/OpenGVLab/vinci.