OpenRaiser/Intern-Atlas huggingface dataset:
Follow Anything: Open-set detection, tracking, and following in real-time
Venue: arXiv cs.RO 2024
Year: 2024

Tracking and following objects of interest is critical to several robotics use cases, ranging from industrial automation to logistics and warehousing, to healthcare and security. In this paper, we present a robotic system to detect, track, and follow any object in real-time. Our approach, dubbed ``follow anything'' (FAn), is an open-vocabulary and multimodal model -- it is not restricted to concepts seen at training time and can be applied to novel classes at inference time using text, images, or click queries. Leveraging rich visual descriptors from large-scale pre-trained models (foundation models), FAn can detect and segment objects by matching multimodal queries (text, images, clicks) against an input image sequence. These detected and segmented objects are tracked across image frames, all while accounting for occlusion and object re-emergence. We demonstrate FAn on a real-world robotic system (a micro aerial vehicle) and report its ability to seamlessly follow the objects of interest in a real-time control loop. FAn can be deployed on a laptop with a lightweight (6-8 GB) graphics card, achieving a throughput of 6-20 frames per second. To enable rapid adoption, deployment, and extensibility, we open-source all our code on our project webpage at https://github.com/alaamaalouf/FollowAnything . We also encourage the reader to watch our 5-minutes explainer video in this https://www.youtube.com/watch?v=6Mgt3EPytrw .
Zero Grads: Learning Local Surrogate Losses for Non-Differentiable Graphics
Venue: arXiv cs.CV 2024
Year: 2024

Gradient-based optimization is now ubiquitous across graphics, but unfortunately can not be applied to problems with undefined or zero gradients. To circumvent this issue, the loss function can be manually replaced by a ``surrogate'' that has similar minima but is differentiable. Our proposed framework, ZeroGrads, automates this process by learning a neural approximation of the objective function, which in turn can be used to differentiate through arbitrary black-box graphics pipelines. We train the surrogate on an actively smoothed version of the objective and encourage locality, focusing the surrogate's capacity on what matters at the current training episode. The fitting is performed online, alongside the parameter optimization, and self-supervised, without pre-computed data or pre-trained models. As sampling the objective is expensive (it requires a full rendering or simulator run), we devise an efficient sampling scheme that allows for tractable run-times and competitive performance at little overhead. We demonstrate optimizing diverse non-convex, non-differentiable black-box problems in graphics, such as visibility in rendering, discrete parameter spaces in procedural modelling or optimal control in physics-driven animation. In contrast to other derivative-free algorithms, our approach scales well to higher dimensions, which we demonstrate on problems with up to 35k interlinked variables.
GPLaSDI: Gaussian Process-based Interpretable Latent Space Dynamics Identification through Deep Autoencoder
Venue: arXiv cs.CE 2024
Year: 2024

Numerically solving partial differential equations (PDEs) can be challenging and computationally expensive. This has led to the development of reduced-order models (ROMs) that are accurate but faster than full order models (FOMs). Recently, machine learning advances have enabled the creation of non-linear projection methods, such as Latent Space Dynamics Identification (LaSDI). LaSDI maps full-order PDE solutions to a latent space using autoencoders and learns the system of ODEs governing the latent space dynamics. By interpolating and solving the ODE system in the reduced latent space, fast and accurate ROM predictions can be made by feeding the predicted latent space dynamics into the decoder. In this paper, we introduce GPLaSDI, a novel LaSDI-based framework that relies on Gaussian process (GP) for latent space ODE interpolations. Using GPs offers two significant advantages. First, it enables the quantification of uncertainty over the ROM predictions. Second, leveraging this prediction uncertainty allows for efficient adaptive training through a greedy selection of additional training data points. This approach does not require prior knowledge of the underlying PDEs. Consequently, GPLaSDI is inherently non-intrusive and can be applied to problems without a known PDE or its residual. We demonstrate the effectiveness of our approach on the Burgers equation, Vlasov equation for plasma physics, and a rising thermal bubble problem. Our proposed method achieves between 200 and 100,000 times speed-up, with up to 7% relative error.
Learning Team-Based Navigation: A Review of Deep Reinforcement Learning Techniques for Multi-Agent Pathfinding
Venue: arXiv cs.AI 2024
Year: 2024

Multi-agent pathfinding (MAPF) is a critical field in many large-scale robotic applications, often being the fundamental step in multi-agent systems. The increasing complexity of MAPF in complex and crowded environments, however, critically diminishes the effectiveness of existing solutions. In contrast to other studies that have either presented a general overview of the recent advancements in MAPF or extensively reviewed Deep Reinforcement Learning (DRL) within multi-agent system settings independently, our work presented in this review paper focuses on highlighting the integration of DRL-based approaches in MAPF. Moreover, we aim to bridge the current gap in evaluating MAPF solutions by addressing the lack of unified evaluation metrics and providing comprehensive clarification on these metrics. Finally, our paper discusses the potential of model-based DRL as a promising future direction and provides its required foundational understanding to address current challenges in MAPF. Our objective is to assist readers in gaining insight into the current research direction, providing unified metrics for comparing different MAPF algorithms and expanding their knowledge of model-based DRL to address the existing challenges in MAPF.
Semantic-embedded Similarity Prototype for Scene Recognition
Venue: arXiv cs.CV 2024
Year: 2024

Due to the high inter-class similarity caused by the complex composition and the co-existing objects across scenes, numerous studies have explored object semantic knowledge within scenes to improve scene recognition. However, a resulting challenge emerges as object information extraction techniques require heavy computational costs, thereby burdening the network considerably. This limitation often renders object-assisted approaches incompatible with edge devices in practical deployment. In contrast, this paper proposes a semantic knowledge-based similarity prototype, which can help the scene recognition network achieve superior accuracy without increasing the computational cost in practice. It is simple and can be plug-and-played into existing pipelines. More specifically, a statistical strategy is introduced to depict semantic knowledge in scenes as class-level semantic representations. These representations are used to explore correlations between scene classes, ultimately constructing a similarity prototype. Furthermore, we propose to leverage the similarity prototype to support network training from the perspective of Gradient Label Softening and Batch-level Contrastive Loss, respectively. Comprehensive evaluations on multiple benchmarks show that our similarity prototype enhances the performance of existing networks, all while avoiding any additional computational burden in practical deployments. Code and the statistical similarity prototype will be available at https://github.com/ChuanxinSong/SimilarityPrototype
Unveiling the Tricks: Automated Detection of Dark Patterns in Mobile Applications
Venue: arXiv cs.HC 2024
Year: 2024

Mobile apps bring us many conveniences, such as online shopping and communication, but some use malicious designs called dark patterns to trick users into doing things that are not in their best interest. Many works have been done to summarize the taxonomy of these patterns and some have tried to mitigate the problems through various techniques. However, these techniques are either time-consuming, not generalisable or limited to specific patterns. To address these issues, we propose UIGuard, a knowledge-driven system that utilizes computer vision and natural language pattern matching to automatically detect a wide range of dark patterns in mobile UIs. Our system relieves the need for manually creating rules for each new UI/app and covers more types with superior performance. In detail, we integrated existing taxonomies into a consistent one, conducted a characteristic analysis and distilled knowledge from real-world examples and the taxonomy. Our UIGuard consists of two components, Property Extraction and Knowledge-Driven Dark Pattern Checker. We collected the first dark pattern dataset, which contains 4,999 benign UIs and 1,353 malicious UIs of 1,660 instances spanning 1,023 mobile apps. Our system achieves a superior performance in detecting dark patterns (micro averages: 0.82 in precision, 0.77 in recall, 0.79 in F1 score). A user study involving 58 participants further shows that \tool{} significantly increases users' knowledge of dark patterns.
Simple Analysis of Priority Sampling
Venue: arXiv cs.DS 2024
Year: 2024

We prove a tight upper bound on the variance of the priority sampling method (aka sequential Poisson sampling). Our proof is significantly shorter and simpler than the original proof given by Mario Szegedy at STOC 2006, which resolved a conjecture by Duffield, Lund, and Thorup.
Embracing Errors Is More Efficient Than Avoiding Them Through Constrained Coding for DNA Data Storage
Venue: arXiv cs.IT 2024
Year: 2024

DNA is an attractive medium for digital data storage. When data is stored on DNA, errors occur, which makes error-correcting coding techniques critical for reliable DNA data storage. To reduce the errors, a common technique is to include constraints that avoid homopolymers (consecutive repeated nucleotides) and balance the GC content, as sequences with homopolymers and unbalanced GC content are often associated with higher error rates. However, constrained coding comes at the cost of an increase in redundancy. An alternative is to control errors by randomizing the sequences, embracing errors, and paying for them with additional coding redundancy. In this paper, we determine the error regimes in which embracing substitutions is more efficient than constrained coding for DNA data storage. Our results suggest that constrained coding for substitution errors is inefficient for existing DNA data storage systems. Theoretical analysis indicates that for constrained coding to be efficient, the increase in substitution errors for nucleotides in homopolymers and sequences with unbalanced GC content must be very large. Additionally, empirical results show that the increase in substitution, deletion, and insertion rates for these nucleotides is minimal in existing DNA storage systems.
Semihierarchical Reconstruction and Weak-area Revisiting for Robotic Visual Seafloor Mapping
Venue: arXiv cs.CV 2024
Year: 2024

Despite impressive results achieved by many on-land visual mapping algorithms in the recent decades, transferring these methods from land to the deep sea remains a challenge due to harsh environmental conditions. Images captured by autonomous underwater vehicles (AUVs), equipped with high-resolution cameras and artificial illumination systems, often suffer from heterogeneous illumination and quality degradation caused by attenuation and scattering, on top of refraction of light rays. These challenges often result in the failure of on-land SLAM approaches when applied underwater or cause SfM approaches to exhibit drifting or omit challenging images. Consequently, this leads to gaps, jumps, or weakly reconstructed areas. In this work, we present a navigation-aided hierarchical reconstruction approach to facilitate the automated robotic 3D reconstruction of hectares of seafloor. Our hierarchical approach combines the advantages of SLAM and global SfM that is much more efficient than incremental SfM, while ensuring the completeness and consistency of the global map. This is achieved through identifying and revisiting problematic or weakly reconstructed areas, avoiding to omit images and making better use of limited dive time. The proposed system has been extensively tested and evaluated during several research cruises, demonstrating its robustness and practicality in real-world conditions.
DIG In: Evaluating Disparities in Image Generations with Indicators for Geographic Diversity
Venue: arXiv cs.CV 2024
Year: 2024

The unprecedented photorealistic results achieved by recent text-to-image generative systems and their increasing use as plug-and-play content creation solutions make it crucial to understand their potential biases. In this work, we introduce three indicators to evaluate the realism, diversity and prompt-generation consistency of text-to-image generative systems when prompted to generate objects from across the world. Our indicators complement qualitative analysis of the broader impact of such systems by enabling automatic and efficient benchmarking of geographic disparities, an important step towards building responsible visual content creation systems. We use our proposed indicators to analyze potential geographic biases in state-of-the-art visual content creation systems and find that: (1) models have less realism and diversity of generations when prompting for Africa and West Asia than Europe, (2) prompting with geographic information comes at a cost to prompt-consistency and diversity of generated images, and (3) models exhibit more region-level disparities for some objects than others. Perhaps most interestingly, our indicators suggest that progress in image generation quality has come at the cost of real-world geographic representation. Our comprehensive evaluation constitutes a crucial step towards ensuring a positive experience of visual content creation for everyone.
Split-State Non-Malleable Codes and Secret Sharing Schemes for Quantum Messages
Venue: arXiv cs.CR 2024
Year: 2024

Non-malleable codes are fundamental objects at the intersection of cryptography and coding theory. These codes provide security guarantees even in settings where error correction and detection are impossible, and have found applications to several other cryptographic tasks. One of the strongest and most well-studied adversarial tampering models is $2$-split-state tampering. Here, a codeword is split into two parts and the adversary can then independently tamper with each part using arbitrary functions. This model can be naturally extended to the secret sharing setting with several parties by having the adversary independently tamper with each share. Previous works on non-malleable coding and secret sharing in the split-state tampering model only considered the encoding of \emph{classical} messages. Furthermore, until recent work by Aggarwal, Boddu, and Jain (IEEE Trans.\ Inf.\ Theory 2024), adversaries with quantum capabilities and \emph{shared entanglement} had not been considered, and it is a priori not clear whether previous schemes remain secure in this model. In this work, we introduce the notions of split-state non-malleable codes and secret sharing schemes for quantum messages secure against quantum adversaries with shared entanglement. Then, we present explicit constructions of such schemes that achieve low-error non-malleability. More precisely, we construct efficiently encodable and decodable split-state non-malleable codes and secret sharing schemes for quantum messages preserving entanglement with external systems and achieving security against quantum adversaries having shared entanglement with codeword length $n$, any message length at most $n^{Ω(1)}$, and error $ε=2^{-{n^{Ω(1)}}}$. In the easier setting of \emph{average-case} non-malleability, we achieve efficient non-malleable coding with rate close to $1/11$.
On Cooperative Coevolution and Global Crossover
Venue: arXiv cs.NE 2024
Year: 2024

Cooperative coevolutionary algorithms (CCEAs) divide a given problem in to a number of subproblems and use an evolutionary algorithm to solve each subproblem. This short paper is concerned with the scenario under which only a single, global fitness measure exists. By removing the typically used subproblem partnering mechanism, it is suggested that such CCEAs can be viewed as making use of a generalised version of the global crossover operator introduced in early Evolution Strategies. Using the well-known NK model of fitness landscapes, the effects of varying aspects of global crossover with respect to the ruggedness of the underlying fitness landscape are explored. Results suggest improvements over the most widely used form of CCEAs, something further demonstrated using other well-known test functions.
Optimal FIFO grouping in public transit networks
Venue: arXiv cs.DS 2024
Year: 2024

This technical report is about grouping vehicles in public transport into routes so that two vehicles of a route do not overtake each other. We say that such a set of routes satisfies the FIFO property. A natural question is: Given a set of trips, find a minimal FIFO grouping into routes. This question is especially interesting for route planning algorithms since a better route grouping leads to a better runtime.
Untangling Carbon-free Energy Attribution and Carbon Intensity Estimation for Carbon-aware Computing
Venue: arXiv cs.DC 2024
Year: 2024

Many organizations, including governments, utilities, and businesses, have set ambitious targets to reduce carbon emissions for their Environmental, Social, and Governance (ESG) goals. To achieve these targets, these organizations increasingly use power purchase agreements (PPAs) to obtain renewable energy credits, which they use to compensate for the ``brown'' energy consumed from the grid. However, the details of these PPAs are often private and not shared with important stakeholders, such as grid operators and carbon information services, who monitor and report the grid's carbon emissions. This often results in incorrect carbon accounting, where the same renewable energy production could be factored into grid carbon emission reports and separately claimed by organizations that own PPAs. Such ``double counting'' of renewable energy production could lead organizations with PPAs to understate their carbon emissions and overstate their progress toward sustainability goals, and also provide significant challenges to consumers using common carbon reduction measures to decrease their carbon footprint. Unfortunately, there is no consensus on accurately computing the grid's carbon intensity by properly accounting for PPAs. The goal of our work is to shed quantitative and qualitative light on the renewable energy attribution and the incorrect carbon intensity estimation problems.
Machine Unlearning: Solutions and Challenges
Venue: arXiv cs.LG 2024
Year: 2024

Machine learning models may inadvertently memorize sensitive, unauthorized, or malicious data, posing risks of privacy breaches, security vulnerabilities, and performance degradation. To address these issues, machine unlearning has emerged as a critical technique to selectively remove specific training data points' influence on trained models. This paper provides a comprehensive taxonomy and analysis of the solutions in machine unlearning. We categorize existing solutions into exact unlearning approaches that remove data influence thoroughly and approximate unlearning approaches that efficiently minimize data influence. By comprehensively reviewing solutions, we identify and discuss their strengths and limitations. Furthermore, we propose future directions to advance machine unlearning and establish it as an essential capability for trustworthy and adaptive machine learning models. This paper provides researchers with a roadmap of open problems, encouraging impactful contributions to address real-world needs for selective data removal.
Enhancing State Estimator for Autonomous Racing : Leveraging Multi-modal System and Managing Computing Resources
Venue: arXiv cs.RO 2024
Year: 2024

This paper introduces an approach that enhances the state estimator for high-speed autonomous race cars, addressing challenges from unreliable measurements, localization failures, and computing resource management. The proposed robust localization system utilizes a Bayesian-based probabilistic approach to evaluate multimodal measurements, ensuring the use of credible data for accurate and reliable localization, even in harsh racing conditions. To tackle potential localization failures, we present a resilient navigation system which enables the race car to continue track-following by leveraging direct perception information in planning and execution, ensuring continuous performance despite localization disruptions. In addition, efficient computing is critical to avoid overload and system failure. Hence, we optimize computing resources using an efficient LiDAR-based state estimation method. Leveraging CUDA programming and GPU acceleration, we perform nearest points search and covariance computation efficiently, overcoming CPU bottlenecks. Simulation and real-world tests validate the system's performance and resilience. The proposed approach successfully recovers from failures, effectively preventing accidents and ensuring safety of the car.
Successive Refinement of Shannon Cipher System Under Maximal Leakage
Venue: arXiv cs.IT 2024
Year: 2024

We study the successive refinement setting of Shannon cipher system (SCS) under the maximal leakage secrecy metric for discrete memoryless sources under bounded distortion measures. Specifically, we generalize the threat model for the point-to-point rate-distortion setting of Issa, Wagner and Kamath (T-IT 2020) to the multiterminal successive refinement setting. Under mild conditions that correspond to partial secrecy, we characterize the asymptotically optimal normalized maximal leakage region for both the joint excess-distortion probability (JEP) and the expected distortion reliability constraints. Under JEP, in the achievability part, we propose a type-based coding scheme, analyze the reliability guarantee for JEP and bound the leakage of the information source through compressed messages. In the converse part, by analyzing a guessing scheme of the eavesdropper, we prove the optimality of our achievability result. Under expected distortion, the achievability part is established similarly to the JEP counterpart. The converse proof proceeds by generalizing the corresponding results for the rate-distortion setting of SCS by Schieler and Cuff (T-IT 2014) to the successive refinement setting. Somewhat surprisingly, the normalized maximal leakage regions under both JEP and expected distortion constraints are identical under certain conditions, although JEP appears to be a stronger reliability constraint.
Human-centered NLP Fact-checking: Co-Designing with Fact-checkers using Matchmaking for AI
Venue: arXiv cs.HC 2024
Year: 2024

While many Natural Language Processing (NLP) techniques have been proposed for fact-checking, both academic research and fact-checking organizations report limited adoption of such NLP work due to poor alignment with fact-checker practices, values, and needs. To address this, we investigate a co-design method, Matchmaking for AI, to enable fact-checkers, designers, and NLP researchers to collaboratively identify what fact-checker needs should be addressed by technology, and to brainstorm ideas for potential solutions. Co-design sessions we conducted with 22 professional fact-checkers yielded a set of 11 design ideas that offer a "north star", integrating fact-checker criteria into novel NLP design concepts. These concepts range from pre-bunking misinformation, efficient and personalized monitoring misinformation, proactively reducing fact-checker potential biases, and collaborative writing fact-check reports. Our work provides new insights into both human-centered fact-checking research and practice and AI co-design research.