OpenRaiser/Intern-Atlas huggingface dataset:
Fair Diagnosis: Leveraging Causal Modeling to Mitigate Medical Bias
Venue: arXiv cs.CV 2024
Year: 2024

In medical image analysis, model predictions can be affected by sensitive attributes, such as race and gender, leading to fairness concerns and potential biases in diagnostic outcomes. To mitigate this, we present a causal modeling framework, which aims to reduce the impact of sensitive attributes on diagnostic predictions. Our approach introduces a novel fairness criterion, \textbf{Diagnosis Fairness}, and a unique fairness metric, leveraging path-specific fairness to control the influence of demographic attributes, ensuring that predictions are primarily informed by clinically relevant features rather than sensitive attributes. By incorporating adversarial perturbation masks, our framework directs the model to focus on critical image regions, suppressing bias-inducing information. Experimental results across multiple datasets demonstrate that our framework effectively reduces bias directly associated with sensitive attributes while preserving diagnostic accuracy. Our findings suggest that causal modeling can enhance both fairness and interpretability in AI-powered clinical decision support systems.
DRDST: Low-latency DAG Consensus through Robust Dynamic Sharding and Tree-broadcasting for IoV
Venue: arXiv cs.NI 2024
Year: 2024

The Internet of Vehicles (IoV) is emerging as a pivotal technology for enhancing traffic management and safety. Its rapid development demands solutions for enhanced communication efficiency and reduced latency. However, traditional centralized networks struggle to meet these demands, prompting the exploration of decentralized solutions such as blockchain. Addressing blockchain's scalability challenges posed by the growing number of nodes and transactions calls for innovative solutions, among which sharding stands out as a pivotal approach to significantly enhance blockchain throughput. However, existing schemes still face challenges related to a) the impact of vehicle mobility on blockchain consensus, especially for cross-shard transaction; and b) the strict requirements of low latency consensus in a highly dynamic network. In this paper, we propose a DAG (Directed Acyclic Graph) consensus leveraging Robust Dynamic Sharding and Tree-broadcasting (DRDST) to address these challenges. Specifically, we first develop a standard for evaluating the network stability of nodes, combined with the nodes' trust values, to propose a novel robust sharding model that is solved through the design of the Genetic Sharding Algorithm (GSA). Then, we optimize the broadcast latency of the whole sharded network by improving the tree-broadcasting to minimize the maximum broadcast latency within each shard. On this basis, we also design a DAG consensus scheme based on an improved hashgraph protocol, which can efficiently handle cross-shard transactions. Finally, the simulation proves the proposed scheme is superior to the comparison schemes in latency, throughput, consensus success rate, and node traffic load.
Diff4Steer: Steerable Diffusion Prior for Generative Music Retrieval with Semantic Guidance
Venue: arXiv cs.SD 2024
Year: 2024

Modern music retrieval systems often rely on fixed representations of user preferences, limiting their ability to capture users' diverse and uncertain retrieval needs. To address this limitation, we introduce Diff4Steer, a novel generative retrieval framework that employs lightweight diffusion models to synthesize diverse seed embeddings from user queries that represent potential directions for music exploration. Unlike deterministic methods that map user query to a single point in embedding space, Diff4Steer provides a statistical prior on the target modality (audio) for retrieval, effectively capturing the uncertainty and multi-faceted nature of user preferences. Furthermore, Diff4Steer can be steered by image or text inputs, enabling more flexible and controllable music discovery combined with nearest neighbor search. Our framework outperforms deterministic regression methods and LLM-based generative retrieval baseline in terms of retrieval and ranking metrics, demonstrating its effectiveness in capturing user preferences, leading to more diverse and relevant recommendations. Listening examples are available at tinyurl.com/diff4steer.
Code generation and runtime techniques for enabling data-efficient deep learning training on GPUs
Venue: arXiv cs.DC 2024
Year: 2024

As deep learning models scale, their training cost has surged significantly. Due to both hardware advancements and limitations in current software stacks, the need for data efficiency has risen. Data efficiency refers to the effective hiding of data access latency and the avoidance of unnecessary data movements. Major challenges arise from the growing disparity between GPU memory bandwidth and computational throughput, imminent GPU memory capacity limitations, and inefficiencies in the PyTorch software stack, including a lack of device-specific PCIe transfer optimizations and high-level domain-specific abstractions. To effectively mitigate these data inefficiencies for deep learning training, this dissertation analyzes data inefficiency in representative deep training tasks, specifically in graph neural networks (GNNs) and large language models (LLMs). It then proposes novel runtime and code generation techniques to mitigate these challenges and implements these optimizations seamlessly within the PyTorch stack while maintaining strong programmability and interoperability. First, PyTorch-Direct is devised to incorporate the GPU-centric PCIe data transfer paradigm in PyTorch for GNN training. Next, Hector intermediate representation (IR) and its code generator are proposed to introduce domain-specific high-level abstraction and systematically address memory-intensive performance challenges for relational GNNs. Finally, in LLM training, the throughput has been increasingly constrained by GPU memory capacity. To mitigate this, the SSDTrain offloading framework is designed and implemented. Together, these contributions show that code generation and runtime techniques can systematically mitigate the data management bottlenecks in deep learning training, which stem from the data-intensive nature of workloads and the oversimplification inherent in the deep learning training software stack.
Distributed Massive MIMO-Aided Task Offloading in Satellite-Terrestrial Integrated Multi-Tier VEC Networks
Venue: arXiv cs.DC 2024
Year: 2024

This paper proposes a distributed massive multiple input multiple-output (DM-MIMO) aided multi-tier vehicular edge computing (VEC) system. In particular, each vehicle terminal (VT) offloads its computational task to the roadside unit (RSU) by orthogonal frequency division multiple access (OFDMA), which can be computed locally at the RSU and offloaded to the central processing unit (CPU) via massive satellite access points (SAPs) for remote computation. By considering the partial task offloading model, we consider the joint optimization of the task offloading, subchannel allocation and precoding optimization to minimize the total cost in terms of total delay and energy consumption. To solve this non-convex problem, we transform the original problem into three sub-problems and use the alternate optimization algorithm to solve it. First, we transform the subcarrier allocation problem of discrete variables into convex optimization problem of continuous variables. Then, we use multiple quadratic transformations and Lagrange multiplier method to transform the non-convex subproblem of optimizing precoding vectors into a convex problem, while the task offloading subproblem is a convex problem. Given the subcarrier and the task allocation and precoding result, we finally find the joint optimized results by iterative optimization algorithm. Simulation results show that our proposed algorithm is superior to other benchmarks.
Employee Well-being in the Age of AI: Perceptions, Concerns, Behaviors, and Outcomes
Venue: arXiv cs.CY 2024
Year: 2024

The growing integration of Artificial Intelligence (AI) into Human Resources (HR) processes has transformed the way organizations manage recruitment, performance evaluation, and employee engagement. While AI offers numerous advantages, such as improved efficiency, reduced bias, and hyper-personalization, it raises significant concerns about employee well-being, job security, fairness, and transparency. This study examines how AI shapes employee perceptions, job satisfaction, mental health, and retention. Key findings reveal that while AI can enhance efficiency and reduce bias, it also raises concerns about job security, fairness, and privacy. Transparency in AI systems emerges as a critical factor in fostering trust and positive employee attitudes. AI systems can both support and undermine employee well-being, depending on how they are implemented and perceived. The research introduces an AI-employee well-being Interaction Framework, illustrating how AI influences employee perceptions, behaviors, and outcomes. Organizational strategies, such as clear communication, upskilling programs, and employee involvement in AI implementation, are identified as crucial for mitigating negative impacts and enhancing positive outcomes. The study concludes that the successful integration of AI in HR requires a balanced approach that prioritizes employee well-being, facilitates human-AI collaboration, and ensures ethical and transparent AI practices alongside technological advancement.
CCS: Continuous Learning for Customized Incremental Wireless Sensing Services
Venue: arXiv cs.LG 2024
Year: 2024

Wireless sensing has made significant progress in tasks ranging from action recognition, vital sign estimation, pose estimation, etc. After over a decade of work, wireless sensing currently stands at the tipping point transitioning from proof-of-concept systems to the large-scale deployment. We envision a future service scenario where wireless sensing service providers distribute sensing models to users. During usage, users might request new sensing capabilities. For example, if someone is away from home on a business trip or vacation for an extended period, they may want a new sensing capability that can detect falls in elderly parents or grandparents and promptly alert them. In this paper, we propose CCS (continuous customized service), enabling model updates on users' local computing resources without data transmission to the service providers. To address the issue of catastrophic forgetting in model updates where updating model parameters to implement new capabilities leads to the loss of existing capabilities we design knowledge distillation and weight alignment modules. These modules enable the sensing model to acquire new capabilities while retaining the existing ones. We conducted extensive experiments on the large-scale XRF55 dataset across Wi-Fi, millimeter-wave radar, and RFID modalities to simulate scenarios where four users sequentially introduced new customized demands. The results affirm that CCS excels in continuous model services across all the above wireless modalities, significantly outperforming existing approaches like OneFi.
MTSpark: Enabling Multi-Task Learning with Spiking Neural Networks for Generalist Agents
Venue: arXiv cs.NE 2024
Year: 2024

Currently, state-of-the-art RL methods excel in single-task settings, but they still struggle to generalize across multiple tasks due to catastrophic forgetting challenges, where previously learned tasks are forgotten as new tasks are introduced. This multi-task learning capability is significantly important for generalist agents, where adaptation features are highly required (e.g., autonomous robots). On the other hand, Spiking Neural Networks (SNNs) have emerged as alternative energy-efficient neural network algorithms due to their sparse spike-based operations. Toward this, we propose MTSpark, a novel methodology to enable multi-task RL using spiking networks. Specifically, MTSpark develops a Deep Spiking Q-Network (DSQN) with active dendrites and dueling structure by leveraging task-specific context signals. Specifically, each neuron computes task-dependent activations that dynamically modulate inputs, forming specialized sub-networks for each task. Moreover, this bioplausible network model also benefits from SNNs, enhancing energy efficiency and making the model suitable for hardware implementation. Experimental results show that, our MTSpark effectively learns multiple tasks with higher performance compared to the state-of-the-art. Specifically, MTSpark successfully achieves high score in three Atari games (i.e., Pong: -5.4, Breakout: 0.6, and Enduro: 371.2), reaching human-level performance (i.e., Pong: -3, Breakout: 31, and Enduro: 368), where state-of-the-art struggle to achieve. In addition, our MTSpark also shows better accuracy in image classification tasks than the state-of-the-art. These results highlight the potential of our MTSpark methodology to develop generalist agents that can learn multiple tasks by leveraging both RL and SNN concepts.
Building a Family of Data Augmentation Models for Low-cost LLM Fine-tuning on the Cloud
Venue: arXiv cs.CL 2024
Year: 2024

Specializing LLMs in various domain-specific tasks has emerged as a critical step towards achieving high performance. However, the construction and annotation of datasets in specific domains are always very costly. Apart from using superior and expensive closed-source LLM APIs to construct datasets, some open-source models have become strong enough to handle dataset construction in many scenarios. Thus, we present a family of data augmentation models designed to significantly improve the efficiency for model fine-tuning. These models, trained based on sufficiently small LLMs, support key functionalities with low inference costs: instruction expansion, instruction refinement, and instruction-response pair expansion. To fulfill this goal, we first construct an automatic data collection system with seed datasets generated from both public repositories and our in-house datasets. This system leverages powerful LLMs to expand, refine and re-write the instructions and responses, incorporating quality assessment techniques. Following this, we introduce the training process of our models, which effectively distills task-solving and text synthesis abilities from teacher LLMs. Finally, we demonstrate how we integrate these functionalities into a machine learning platform to support low-cost LLM fine-tuning from both dataset preparation and training perspectives for users. Experiments and an application study prove the effectiveness of our approach.
A cyber-physical digital twin approach to replicating realistic multi-stage cyberattacks on smart grids
Venue: arXiv cs.CR 2024
Year: 2024

The integration of information and communication technology in distribution grids presents opportunities for active grid operation management, but also increases the need for security against power outages and cyberattacks. This paper examines the impact of cyberattacks on smart grids by replicating the power grid in a secure laboratory environment as a cyber-physical digital twin. A simulation is used to study communication infrastructures for secure operation of smart grids. The cyber-physical digital twin approach combines communication network emulation and power grid simulation in a common modular environment, and is demonstrated through laboratory tests and attack replications.
Large Language Models for Ingredient Substitution in Food Recipes using Supervised Fine-tuning and Direct Preference Optimization
Venue: arXiv cs.CL 2024
Year: 2024

In this paper, we address the challenge of recipe personalization through ingredient substitution. We make use of Large Language Models (LLMs) to build an ingredient substitution system designed to predict plausible substitute ingredients within a given recipe context. Given that the use of LLMs for this task has been barely done, we carry out an extensive set of experiments to determine the best LLM, prompt, and the fine-tuning setups. We further experiment with methods such as multi-task learning, two-stage fine-tuning, and Direct Preference Optimization (DPO). The experiments are conducted using the publicly available Recipe1MSub corpus. The best results are produced by the Mistral7-Base LLM after fine-tuning and DPO. This result outperforms the strong baseline available for the same corpus with a Hit@1 score of 22.04. Thus we believe that this research represents a significant step towards enabling personalized and creative culinary experiences by utilizing LLM-based ingredient substitution.
$S^3$: Synonymous Semantic Space for Improving Zero-Shot Generalization of Vision-Language Models
Venue: arXiv cs.CV 2024
Year: 2024

Recently, many studies have been conducted to enhance the zero-shot generalization ability of vision-language models (e.g., CLIP) by addressing the semantic misalignment between image and text embeddings in downstream tasks. Although many efforts have been made, existing methods barely consider the fact that a class of images can be described by notably different textual concepts due to well-known lexical variation in natural language processing, which heavily affects the zero-shot generalization of CLIP. Therefore, this paper proposes a \textbf{S}ynonymous \textbf{S}emantic \textbf{S}pace ($S^3$) for each image class, rather than relying on a single textual concept, achieving more stable semantic alignment and improving the zero-shot generalization of CLIP. Specifically, our $S^3$ method first generates several synonymous concepts based on the label of each class by using large language models, and constructs a continuous yet compact synonymous semantic space based on the Vietoris-Rips complex of the generated synonymous concepts. Furthermore, we explore the effect of several point-to-space metrics on our $S^3$, while presenting a point-to-local-center metric to compute similarity between image embeddings and the synonymous semantic space of each class, accomplishing effective zero-shot predictions. Extensive experiments are conducted across 17 benchmarks, including fine-grained zero-shot classification, natural distribution zero-shot classification, and open-vocabulary segmentation, and the results show that our $S^3$ outperforms state-of-the-art methods.
Continuous Video Process: Modeling Videos as Continuous Multi-Dimensional Processes for Video Prediction
Venue: arXiv cs.CV 2024
Year: 2024

Diffusion models have made significant strides in image generation, mastering tasks such as unconditional image synthesis, text-image translation, and image-to-image conversions. However, their capability falls short in the realm of video prediction, mainly because they treat videos as a collection of independent images, relying on external constraints such as temporal attention mechanisms to enforce temporal coherence. In our paper, we introduce a novel model class, that treats video as a continuous multi-dimensional process rather than a series of discrete frames. We also report a reduction of 75\% sampling steps required to sample a new frame thus making our framework more efficient during the inference time. Through extensive experimentation, we establish state-of-the-art performance in video prediction, validated on benchmark datasets including KTH, BAIR, Human3.6M, and UCF101. Navigate to the project page https://www.cs.umd.edu/~gauravsh/cvp/supp/website.html for video results.
DEYOLO: Dual-Feature-Enhancement YOLO for Cross-Modality Object Detection
Venue: arXiv cs.CV 2024
Year: 2024

Object detection in poor-illumination environments is a challenging task as objects are usually not clearly visible in RGB images. As infrared images provide additional clear edge information that complements RGB images, fusing RGB and infrared images has potential to enhance the detection ability in poor-illumination environments. However, existing works involving both visible and infrared images only focus on image fusion, instead of object detection. Moreover, they directly fuse the two kinds of image modalities, which ignores the mutual interference between them. To fuse the two modalities to maximize the advantages of cross-modality, we design a dual-enhancement-based cross-modality object detection network DEYOLO, in which semantic-spatial cross modality and novel bi-directional decoupled focus modules are designed to achieve the detection-centered mutual enhancement of RGB-infrared (RGB-IR). Specifically, a dual semantic enhancing channel weight assignment module (DECA) and a dual spatial enhancing pixel weight assignment module (DEPA) are firstly proposed to aggregate cross-modality information in the feature space to improve the feature representation ability, such that feature fusion can aim at the object detection task. Meanwhile, a dual-enhancement mechanism, including enhancements for two-modality fusion and single modality, is designed in both DECAand DEPAto reduce interference between the two kinds of image modalities. Then, a novel bi-directional decoupled focus is developed to enlarge the receptive field of the backbone network in different directions, which improves the representation quality of DEYOLO. Extensive experiments on M3FD and LLVIP show that our approach outperforms SOTA object detection algorithms by a clear margin. Our code is available at https://github.com/chips96/DEYOLO.
Solving 1D Poisson problem with a Variational Quantum Linear Solver
Venue: arXiv cs.CE 2024
Year: 2024

Different hybrid quantum-classical algorithms have recently been developed as a near-term way to solve linear systems of equations on quantum devices. However, the focus has so far been mostly on the methods, rather than the problems that they need to tackle. In fact, these algorithms have been run on real hardware only for problems in quantum physics, such as Hamiltonians of a few qubits systems. These problems are particularly favorable for quantum hardware, since their matrices are the sum of just a few unitary terms and since only shallow quantum circuits are required to estimate the cost function. However, for many interesting problems in linear algebra, it appears far less trivial to find an efficient decomposition and to trade it off with the depth of the cost quantum circuits. A first simple yet interesting instance to consider are tridiagonal systems of equations. These arise, for instance, in the discretization of one-dimensional finite element analyses. This work presents a method to solve a class of tridiagonal systems of equations with the variational quantum linear solver (VQLS), a recently proposed variational hybrid algorithm for solving linear systems. In particular, we present a new decomposition for this class of matrices based on both Pauli strings and multi--qubit gates, resulting in less terms than those obtained by just using Pauli gates. Based on this decomposition, we discuss the tradeoff between the number of terms and the near-term implementability of the quantum circuits. Furthermore, we present the first simulated and real-hardware results obtained by solving tridiagonal linear systems with VQLS, using the decomposition proposed.