OpenRaiser/Intern-Atlas huggingface dataset:
SARA: Controllable Makeup Transfer with Spatial Alignment and Region-Adaptive Normalization
Venue: arXiv cs.CV 2024
Year: 2024

Makeup transfer is a process of transferring the makeup style from a reference image to the source images, while preserving the source images' identities. This technique is highly desirable and finds many applications. However, existing methods lack fine-level control of the makeup style, making it challenging to achieve high-quality results when dealing with large spatial misalignments. To address this problem, we propose a novel Spatial Alignment and Region-Adaptive normalization method (SARA) in this paper. Our method generates detailed makeup transfer results that can handle large spatial misalignments and achieve part-specific and shade-controllable makeup transfer. Specifically, SARA comprises three modules: Firstly, a spatial alignment module that preserves the spatial context of makeup and provides a target semantic map for guiding the shape-independent style codes. Secondly, a region-adaptive normalization module that decouples shape and makeup style using per-region encoding and normalization, which facilitates the elimination of spatial misalignments. Lastly, a makeup fusion module blends identity features and makeup style by injecting learned scale and bias parameters. Experimental results show that our SARA method outperforms existing methods and achieves state-of-the-art performance on two public datasets.
Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization
Venue: arXiv cs.CV 2024
Year: 2024

Multimodal large language models have made significant advancements in recent years, yet they still suffer from a common issue known as the "hallucination problem", in which the models generate textual descriptions that inaccurately depict or entirely fabricate content from associated images. This paper introduces a novel solution, Hallucination-Aware Direct Preference Optimization (HA-DPO), which reframes the hallucination problem as a preference selection task. The model is trained to favor the non-hallucinating response when presented with two responses of the same image (one accurate and one hallucinatory). Furthermore, this paper proposes an efficient pipeline for constructing positive~(non-hallucinatory) and negative~(hallucinatory) sample pairs, ensuring a high-quality, style-consistent dataset for robust preference learning. When applied to three mainstream multimodal models, HA-DPO significantly reduced hallucination issues and amplified the models' generalization capabilities. Notably, the MiniGPT-4 model, when enhanced with HA-DPO, demonstrated a substantial improvement: POPE accuracy rose from 51.13% to 86.13% (an absolute improvement of 35%), and the MME score surged from 932.00 to 1326.46 (a relative improvement of 42.32%). The codes, models, and datasets are made accessible at https://opendatalab.github.io/HA-DPO.
Two-step dynamic obstacle avoidance
Venue: arXiv cs.RO 2024
Year: 2024

Dynamic obstacle avoidance (DOA) is a fundamental challenge for any autonomous vehicle, independent of whether it operates in sea, air, or land. This paper proposes a two-step architecture for handling DOA tasks by combining supervised and reinforcement learning (RL). In the first step, we introduce a data-driven approach to estimate the collision risk (CR) of an obstacle using a recurrent neural network, which is trained in a supervised fashion and offers robustness to non-linear obstacle movements. In the second step, we include these CR estimates into the observation space of an RL agent to increase its situational awareness. We illustrate the power of our two-step approach by training different RL agents in a challenging environment that requires to navigate amid multiple obstacles. The non-linear movements of obstacles are exemplarily modeled based on stochastic processes and periodic patterns, although our architecture is suitable for any obstacle dynamics. The experiments reveal that integrating our CR metrics into the observation space doubles the performance in terms of reward, which is equivalent to halving the number of collisions in the considered environment. We also perform a generalization experiment to validate the proposal in an RL environment based on maritime traffic and real-world vessel trajectory data. Furthermore, we show that the architecture's performance improvement is independent of the applied RL algorithm.
Power Hungry Processing: Watts Driving the Cost of AI Deployment?
Venue: arXiv cs.LG 2024
Year: 2024

Recent years have seen a surge in the popularity of commercial AI products based on generative, multi-purpose AI systems promising a unified approach to building machine learning (ML) models into technology. However, this ambition of ``generality'' comes at a steep cost to the environment, given the amount of energy these systems require and the amount of carbon that they emit. In this work, we propose the first systematic comparison of the ongoing inference cost of various categories of ML systems, covering both task-specific (i.e. finetuned models that carry out a single task) and `general-purpose' models, (i.e. those trained for multiple tasks). We measure deployment cost as the amount of energy and carbon required to perform 1,000 inferences on representative benchmark dataset using these models. We find that multi-purpose, generative architectures are orders of magnitude more expensive than task-specific systems for a variety of tasks, even when controlling for the number of model parameters. We conclude with a discussion around the current trend of deploying multi-purpose generative ML systems, and caution that their utility should be more intentionally weighed against increased costs in terms of energy and emissions. All the data from our study can be accessed via an interactive demo to carry out further exploration and analysis.
The Sky's the Limit: Re-lightable Outdoor Scenes via a Sky-pixel Constrained Illumination Prior and Outside-In Visibility
Venue: arXiv cs.CV 2024
Year: 2024

Inverse rendering of outdoor scenes from unconstrained image collections is a challenging task, particularly illumination/albedo ambiguities and occlusion of the illumination environment (shadowing) caused by geometry. However, there are many cues in an image that can aid in the disentanglement of geometry, albedo and shadows. Whilst sky is frequently masked out in state-of-the-art methods, we exploit the fact that any sky pixel provides a direct observation of distant lighting in the corresponding direction and, via a neural illumination prior, a statistical cue to derive the remaining illumination environment. The incorporation of our illumination prior is enabled by a novel `outside-in' method for computing differentiable sky visibility based on a neural directional distance function. This is highly efficient and can be trained in parallel with the neural scene representation, allowing gradients from appearance loss to flow from shadows to influence the estimation of illumination and geometry. Our method estimates high-quality albedo, geometry, illumination and sky visibility, achieving state-of-the-art results on the NeRF-OSR relighting benchmark. Our code and models can be found at https://github.com/JADGardner/neusky
DyRA: Portable Dynamic Resolution Adjustment Network for Existing Detectors
Venue: arXiv cs.CV 2024
Year: 2024

Achieving constant accuracy in object detection is challenging due to the inherent variability of object sizes. One effective approach to this problem involves optimizing input resolution, referred to as a multi-resolution strategy. Previous approaches to resolution optimization have often been based on pre-defined resolutions with manual selection. However, there is a lack of study on run-time resolution optimization for existing architectures. This paper introduces DyRA, a dynamic resolution adjustment network providing an image-specific scale factor for existing detectors. This network is co-trained with detectors utilizing specially designed loss functions, namely ParetoScaleLoss and BalanceLoss. ParetoScaleLoss determines an adaptive scale factor for robustness, while BalanceLoss optimizes overall scale factors according to the localization performance of the detector. The loss function is devised to minimize the accuracy drop across contrasting objectives of different-sized objects for scaling. Our proposed network can improve accuracy across various models, including RetinaNet, Faster-RCNN, FCOS, DINO, and H-Deformable-DETR. The code is available at https://github.com/DaEunFullGrace/DyRA.git.
Classification, Challenges, and Automated Approaches to Handle Non-Functional Requirements in ML-Enabled Systems: A Systematic Literature Review
Venue: arXiv cs.SE 2024
Year: 2024

Context: Machine learning (ML) is nowadays so pervasive and diffused that virtually no application can avoid its use. Nonetheless, its enormous potential is often tempered by the need to manage non-functional requirements and navigate pressing, contrasting trade-offs. Objective: In this respect, we notice the lack of a comprehensive synthesis of the non-functional requirements affecting ML-enabled systems, other than the major challenges faced to deal with them. Such a synthesis may not only provide a comprehensive summary of the state of the art, but also drive further research on the analysis, management, and optimization of non-functional requirements of ML-intensive systems. Method: In this paper, we propose a systematic literature review targeting two key aspects such as (1) the classification of the non-functional requirements investigated so far, and (2) the challenges to be faced when developing models in ML-enabled systems. Through the combination of well-established guidelines for conducting systematic literature reviews and additional search criteria, we survey a total amount of 69 research articles. Results: Our findings report that current research identified 30 different non-functional requirements, which can be grouped into six main classes. We also compiled a catalog of more than 23 software engineering challenges, based on which further research should consider the nonfunctional requirements of machine learning-enabled systems. Conclusion: We conclude our work by distilling implications and a future outlook on the topic.
An Efficient Illumination Invariant Tiger Detection Framework for Wildlife Surveillance
Venue: arXiv cs.CV 2024
Year: 2024

Tiger conservation necessitates the strategic deployment of multifaceted initiatives encompassing the preservation of ecological habitats, anti-poaching measures, and community involvement for sustainable growth in the tiger population. With the advent of artificial intelligence, tiger surveillance can be automated using object detection. In this paper, an accurate illumination invariant framework is proposed based on EnlightenGAN and YOLOv8 for tiger detection. The fine-tuned YOLOv8 model achieves a mAP score of 61% without illumination enhancement. The illumination enhancement improves the mAP by 0.7%. The approaches elevate the state-of-the-art performance on the ATRW dataset by approximately 6% to 7%.
A low-cost Framework for Decentralized Autonomous Intersection Management
Venue: arXiv cs.RO 2024
Year: 2024

This paper addresses the traffic management problem for autonomous vehicles at intersections without traffic signals. In the current system, a road junction has no traffic signals when the traffic volume is low to medium. Installing infrastructure at each unsignalled crossing to coordinate autonomous cars can be formidable. We propose a novel low-cost solution strategy where the vehicles use a harmony matrix to find the best possible combination of the cars to cross the intersection without any crashes. The harmony matrix defines the connection between different vehicle maneuvers and is queried online for intersection management. We maximize the throughput of the intersection by solving a maximal clique problem formulated based on the vehicles present at the intersection. The proposed algorithm relies on the intent perceived by the autonomous vehicles. We compare our work with a communication-based strategy that uses V2I communication protocols, and through extensive simulation, we showed that our algorithm is comparable when the traffic volume is less than 500 PCUs/hr/lane.
New Online Communities: Graph Deep Learning on Anonymous Voting Networks to Identify Sybils in Polycentric Governance
Venue: arXiv cs.LG 2024
Year: 2024

This research examines the polycentric governance of digital assets in blockchain-based Decentralized Autonomous Organizations (DAOs). It offers a theoretical framework and addresses a critical challenge facing decentralized governance by developing a method to identify Sybils, or spurious identities. Sybils pose significant organizational sustainability threats to DAOs and other, commons-based online communities, and threat models are identified. The experimental method uses an autoencoder architecture and graph deep learning techniques to identify Sybil activity in a DAO governance dataset (snapshot.org). Specifically, a Graph Convolutional Neural Network (GCNN) learned voting behaviours and a fast vector clustering algorithm used high-dimensional embeddings to identify similar nodes in a graph. The results reveal that deep learning can effectively identify Sybils, reducing the voting graph by 2-5%. This research underscores the importance of Sybil resistance in DAOs, identifies challenges and opportunities for forensics and analysis of anonymous networks, and offers a novel perspective on decentralized governance, informing future policy, regulation, and governance practices.
A Stochastic-Geometrical Framework for Object Pose Estimation based on Mixture Models Avoiding the Correspondence Problem
Venue: arXiv cs.CV 2024
Year: 2024

Background: Pose estimation of rigid objects is a practical challenge in optical metrology and computer vision. This paper presents a novel stochastic-geometrical modeling framework for object pose estimation based on observing multiple feature points. Methods: This framework utilizes mixture models for feature point densities in object space and for interpreting real measurements. Advantages are the avoidance to resolve individual feature correspondences and to incorporate correct stochastic dependencies in multi-view applications. First, the general modeling framework is presented, second, a general algorithm for pose estimation is derived, and third, two example models (camera and lateration setup) are presented. Results: Numerical experiments show the effectiveness of this modeling and general algorithm by presenting four simulation scenarios for three observation systems, including the dependence on measurement resolution, object deformations and measurement noise. Probabilistic modeling utilizing mixture models shows the potential for accurate and robust pose estimations while avoiding the correspondence problem.
DisMech: A Discrete Differential Geometry-based Physical Simulator for Soft Robots and Structures
Venue: arXiv cs.RO 2024
Year: 2024

Fast, accurate, and generalizable simulations are a key enabler of modern advances in robot design and control. However, existing simulation frameworks in robotics either model rigid environments and mechanisms only, or if they include flexible or soft structures, suffer significantly in one or more of these performance areas. To close this "sim2real" gap, we introduce DisMech, a simulation environment that models highly dynamic motions of rod-like soft continuum robots and structures, quickly and accurately, with arbitrary connections between them. Our methodology combines a fully implicit discrete differential geometry-based physics solver with fast and accurate contact handling, all in an intuitive software interface. Crucially, we propose a gradient descent approach to easily map the motions of hardware robot prototypes to control inputs in DisMech. We validate DisMech through several highly-nuanced soft robot simulations while demonstrating an order of magnitude speed increase over previous state of the art. Our real2sim validation shows high physical accuracy versus hardware, even with complicated soft actuation mechanisms such as shape memory alloy wires. With its low computational cost, physical accuracy, and ease of use, DisMech can accelerate translation of sim-based control for both soft robotics and deformable object manipulation.
RDMA-Based Algorithms for Sparse Matrix Multiplication on GPUs
Venue: arXiv cs.DC 2024
Year: 2024

Sparse matrix multiplication is an important kernel for large-scale graph processing and other data-intensive applications. In this paper, we implement various asynchronous, RDMA-based sparse times dense (SpMM) and sparse times sparse (SpGEMM) algorithms, evaluating their performance running in a distributed memory setting on GPUs. Our RDMA-based implementations use the NVSHMEM communication library for direct, asynchronous one-sided communication between GPUs. We compare our asynchronous implementations to state-of-the-art bulk synchronous GPU libraries as well as a CUDA-aware MPI implementation of the SUMMA algorithm. We find that asynchronous RDMA-based implementations are able to offer favorable performance compared to bulk synchronous implementations, while also allowing for the straightforward implementation of novel work stealing algorithms.
Privacy and Copyright Protection in Generative AI: A Lifecycle Perspective
Venue: arXiv cs.SE 2024
Year: 2024

The advent of Generative AI has marked a significant milestone in artificial intelligence, demonstrating remarkable capabilities in generating realistic images, texts, and data patterns. However, these advancements come with heightened concerns over data privacy and copyright infringement, primarily due to the reliance on vast datasets for model training. Traditional approaches like differential privacy, machine unlearning, and data poisoning only offer fragmented solutions to these complex issues. Our paper delves into the multifaceted challenges of privacy and copyright protection within the data lifecycle. We advocate for integrated approaches that combines technical innovation with ethical foresight, holistically addressing these concerns by investigating and devising solutions that are informed by the lifecycle perspective. This work aims to catalyze a broader discussion and inspire concerted efforts towards data privacy and copyright integrity in Generative AI.
Unclonable Cryptography with Unbounded Collusions and Impossibility of Hyperefficient Shadow Tomography
Venue: arXiv cs.CR 2024
Year: 2024

Quantum no-cloning theorem gives rise to the intriguing possibility of quantum copy protection where we encode a program or functionality in a quantum state such that a user in possession of k copies cannot create k+1 copies, for any k. Introduced by Aaronson (CCC'09) over a decade ago, copy protection has proven to be notoriously hard to achieve. Previous work has been able to achieve copy-protection for various functionalities only in restricted models: (i) in the bounded collusion setting where k -> k+1 security is achieved for a-priori fixed collusion bound k (in the plain model with the same computational assumptions as ours, by Liu, Liu, Qian, Zhandry [TCC'22]), or, (ii) only k -> 2k security is achieved (relative to a structured quantum oracle, by Aaronson [CCC'09]). In this work, we give the first unbounded collusion-resistant (i.e. multiple-copy secure) copy-protection schemes, answering the long-standing open question of constructing such schemes, raised by multiple previous works starting with Aaronson (CCC'09). More specifically, we obtain the following results. - We construct (i) public-key encryption, (ii) public-key functional encryption, (iii) signature and (iv) pseudorandom function schemes whose keys are copy-protected against unbounded collusions in the plain model (i.e. without any idealized oracles), assuming (post-quantum) subexponentially secure iO and LWE. - We show that any unlearnable functionality can be copy-protected against unbounded collusions, relative to a classical oracle. - As a corollary of our results, we rule out the existence of hyperefficient quantum shadow tomography, * even given non-black-box access to the measurements, assuming subexponentially secure iO and LWE, or, * unconditionally relative to a quantumly accessible classical oracle, and hence answer an open question by Aaronson (STOC'18).
Tight Bounds for The Price of Fairness
Venue: arXiv cs.GT 2024
Year: 2024

A central decision maker (CDM), who seeks an efficient allocation of scarce resources among a finite number of players, often has to incorporate fairness criteria to avoid unfair outcomes. Indeed, the Price of Fairness (POF), a term coined in the seminal work by Bertsimas et al. (2011), refers to the efficiency loss due to the incorporation of fairness criteria into the allocation method. Quantifying the POF would help the CDM strike an appropriate balance between efficiency and fairness. In this paper we improve upon existing results in the literature, by providing tight bounds for the POF for the proportional fairness criterion for any $n$, when the maximum achievable utilities of the players are equal or are not equal. Further, while Bertsimas et al. (2011) have already derived a tight bound for the max-min fairness criterion for the case that all players have equal maximum achievable utilities, we also provide a tight bound in scenarios where these utilities are not equal. For both criteria, we characterize the conditions where the POF reaches its peak and provide the supremum bounds of our bounds over all maximum achievable utility vectors, which are shown to be asymptotically strictly smaller than the supremum of the Bertsimas et al. (2011) bounds. Finally, we investigate the sensitivity of our bounds and the bounds in Bertsimas et al. (2011) for the POF to the variability of the maximum achievable utilities.
Investigating Collaborative Data Practices: a Case Study on Artificial Intelligence for Healthcare Research
Venue: arXiv cs.HC 2024
Year: 2024

Developing artificial intelligence (AI) tools for healthcare is a collaborative effort, bringing data scientists, clinicians, patients and other disciplines together. In this paper, we explore the collaborative data practices of research consortia tasked with applying AI tools to understand and manage multiple long-term conditions in the UK. Through an inductive thematic analysis of 13 semi-structured interviews with participants of these consortia, we aimed to understand how collaboration happens based on the tools used, communication processes and settings, as well as the conditions and obstacles for collaborative work. Our findings reveal the adaptation of tools that are used for sharing knowledge and the tailoring of information based on the audience, particularly those from a clinical or patient perspective. Limitations on the ability to do this were also found to be imposed by the use of electronic healthcare records and access to datasets. We identified meetings as the key setting for facilitating exchanges between disciplines and allowing for the blending and creation of knowledge. Finally, we bring to light the conditions needed to facilitate collaboration and discuss how some of the challenges may be navigated in future work.