OpenRaiser/Intern-Atlas huggingface dataset:
Robust Policy Learning for Multi-UAV Collision Avoidance with Causal Feature Selection
Venue: arXiv cs.RO 2024
Year: 2024

In unseen and complex outdoor environments, collision avoidance navigation for unmanned aerial vehicle (UAV) swarms presents a challenging problem. It requires UAVs to navigate through various obstacles and complex backgrounds. Existing collision avoidance navigation methods based on deep reinforcement learning show promising performance but suffer from poor generalization abilities, resulting in performance degradation in unseen environments. To address this issue, we investigate the cause of weak generalization ability in DRL and propose a novel causal feature selection module. This module can be integrated into the policy network and effectively filters out non-causal factors in representations, thereby reducing the influence of spurious correlations between non-causal factors and action predictions. Experimental results demonstrate that our proposed method can achieve robust navigation performance and effective collision avoidance especially in scenarios with unseen backgrounds and obstacles, which significantly outperforms existing state-of-the-art algorithms.
Enhancing Physical Layer Security in LEO Satellite-Enabled IoT Network Communications
Venue: arXiv cs.NI 2024
Year: 2024

The extensive deployment of Low Earth Orbit (LEO) satellites introduces significant security challenges for communication security issues in Internet of Things (IoT) networks. With the rising number of satellites potentially acting as eavesdroppers, integrating Physical Layer Security (PLS) into satellite communications has become increasingly critical. However, these studies are facing challenges such as dealing with dynamic topology difficulties, limitations in interference analysis, and the high complexity of performance evaluation. To address these challenges, for the first time, we investigate PLS strategies in satellite communications using the Stochastic Geometry (SG) analytical framework. We consider the uplink communication scenario in an LEO-enabled IoT network, where multi-tier satellites from different operators respectively serve as legitimate receivers and eavesdroppers. In this scenario, we derive low-complexity analytical expressions for the security performance metrics, namely availability probability, successful communication probability, and secure communication probability. By introducing the power allocation parameters, we incorporate the Artificial Noise (AN) technique, which is an important PLS strategy, into this analytical framework and evaluate the gains it brings to secure transmission. In addition to the AN technique, we also analyze the impact of constellation configuration, physical layer parameters, and network layer parameters on the aforementioned metrics.
MAPO: Boosting Large Language Model Performance with Model-Adaptive Prompt Optimization
Venue: arXiv cs.CL 2024
Year: 2024

Prompt engineering, as an efficient and effective way to leverage Large Language Models (LLM), has drawn a lot of attention from the research community. The existing research primarily emphasizes the importance of adapting prompts to specific tasks, rather than specific LLMs. However, a good prompt is not solely defined by its wording, but also binds to the nature of the LLM in question. In this work, we first quantitatively demonstrate that different prompts should be adapted to different LLMs to enhance their capabilities across various downstream tasks in NLP. Then we novelly propose a model-adaptive prompt optimizer (MAPO) method that optimizes the original prompts for each specific LLM in downstream tasks. Extensive experiments indicate that the proposed method can effectively refine prompts for an LLM, leading to significant improvements over various downstream tasks.
Biometric Authentication Based on Enhanced Remote Photoplethysmography Signal Morphology
Venue: arXiv cs.CV 2024
Year: 2024

Remote photoplethysmography (rPPG) is a non-contact method for measuring cardiac signals from facial videos, offering a convenient alternative to contact photoplethysmography (cPPG) obtained from contact sensors. Recent studies have shown that each individual possesses a unique cPPG signal morphology that can be utilized as a biometric identifier, which has inspired us to utilize the morphology of rPPG signals extracted from facial videos for person authentication. Since the facial appearance and rPPG are mixed in the facial videos, we first de-identify facial videos to remove facial appearance while preserving the rPPG information, which protects facial privacy and guarantees that only rPPG is used for authentication. The de-identified videos are fed into an rPPG model to get the rPPG signal morphology for authentication. In the first training stage, unsupervised rPPG training is performed to get coarse rPPG signals. In the second training stage, an rPPG-cPPG hybrid training is performed by incorporating external cPPG datasets to achieve rPPG biometric authentication and enhance rPPG signal morphology. Our approach needs only de-identified facial videos with subject IDs to train rPPG authentication models. The experimental results demonstrate that rPPG signal morphology hidden in facial videos can be used for biometric authentication. The code is available at https://github.com/zhaodongsun/rppg_biometrics.
Annotating Control-Flow Graphs for Formalized Test Coverage Criteria
Venue: arXiv cs.SE 2024
Year: 2024

Control flow coverage criteria are an important part of the process of qualifying embedded software for safety-critical systems. Criteria such as modified condition/decision coverage (MC/DC) as defined by DO-178B are used by regulators to judge the adequacy of testing and by QA engineers to design tests when full path coverage is impossible. Despite their importance, these coverage criteria are often misunderstood. One problem is that their definitions are typically written in natural language specification documents, making them imprecise. Other works have proposed formal definitions using binary predicate logic, but these definitions are difficult to apply to the analysis of real programs. Control-Flow Graphs (CFGs) are the most common model for analyzing program logic in compilers, and seem to be a good fit for defining and analyzing coverage criteria. However, CFGs discard the explicit concept of a decision, making their use for this task seem impossible. In this paper, we show how to annotate a CFG with decision information inferred from the graph itself. We call this annotated model a Control-Flow Decision Graph (CFDG) and we use it to formally define several common coverage criteria. We have implemented our algorithms in a tool which we show can be applied to automatically annotate CFGs output from popular compilers.
Securing Multi-turn Conversational Language Models From Distributed Backdoor Triggers
Venue: arXiv cs.CL 2024
Year: 2024

Large language models (LLMs) have acquired the ability to handle longer context lengths and understand nuances in text, expanding their dialogue capabilities beyond a single utterance. A popular user-facing application of LLMs is the multi-turn chat setting. Though longer chat memory and better understanding may seemingly benefit users, our paper exposes a vulnerability that leverages the multi-turn feature and strong learning ability of LLMs to harm the end-user: the backdoor. We demonstrate that LLMs can capture the combinational backdoor representation. Only upon presentation of triggers together does the backdoor activate. We also verify empirically that this representation is invariant to the position of the trigger utterance. Subsequently, inserting a single extra token into two utterances of 5%of the data can cause over 99% Attack Success Rate (ASR). Our results with 3 triggers demonstrate that this framework is generalizable, compatible with any trigger in an adversary's toolbox in a plug-and-play manner. Defending the backdoor can be challenging in the chat setting because of the large input and output space. Our analysis indicates that the distributed backdoor exacerbates the current challenges by polynomially increasing the dimension of the attacked input space. Canonical textual defenses like ONION and BKI leverage auxiliary model forward passes over individual tokens, scaling exponentially with the input sequence length and struggling to maintain computational feasibility. To this end, we propose a decoding time defense - decayed contrastive decoding - that scales linearly with assistant response sequence length and reduces the backdoor to as low as 0.35%.
Learning Interpretable Differentiable Logic Networks
Venue: arXiv cs.LG 2024
Year: 2024

The ubiquity of neural networks (NNs) in real-world applications, from healthcare to natural language processing, underscores their immense utility in capturing complex relationships within high-dimensional data. However, NNs come with notable disadvantages, such as their "black-box" nature, which hampers interpretability, as well as their tendency to overfit the training data. We introduce a novel method for learning interpretable differentiable logic networks (DLNs) that are architectures that employ multiple layers of binary logic operators. We train these networks by softening and differentiating their discrete components, e.g., through binarization of inputs, binary logic operations, and connections between neurons. This approach enables the use of gradient-based learning methods. Experimental results on twenty classification tasks indicate that differentiable logic networks can achieve accuracies comparable to or exceeding that of traditional NNs. Equally importantly, these networks offer the advantage of interpretability. Moreover, their relatively simple structure results in the number of logic gate-level operations during inference being up to a thousand times smaller than NNs, making them suitable for deployment on edge devices.
Attention Normalization Impacts Cardinality Generalization in Slot Attention
Venue: arXiv cs.CV 2024
Year: 2024

Object-centric scene decompositions are important representations for downstream tasks in fields such as computer vision and robotics. The recently proposed Slot Attention module, already leveraged by several derivative works for image segmentation and object tracking in videos, is a deep learning component which performs unsupervised object-centric scene decomposition on input images. It is based on an attention architecture, in which latent slot vectors, which hold compressed information on objects, attend to localized perceptual features from the input image. In this paper, we demonstrate that design decisions on normalizing the aggregated values in the attention architecture have considerable impact on the capabilities of Slot Attention to generalize to a higher number of slots and objects as seen during training. We propose and investigate alternatives to the original normalization scheme which increase the generalization capabilities of Slot Attention to varying slot and object counts, resulting in performance gains on the task of unsupervised image segmentation. The newly proposed normalizations represent minimal and easy to implement modifications of the usual Slot Attention module, changing the value aggregation mechanism from a weighted mean operation to a scaled weighted sum operation.
ChartGemma: Visual Instruction-tuning for Chart Reasoning in the Wild
Venue: arXiv cs.AI 2024
Year: 2024

Given the ubiquity of charts as a data analysis, visualization, and decision-making tool across industries and sciences, there has been a growing interest in developing pre-trained foundation models as well as general purpose instruction-tuned models for chart understanding and reasoning. However, existing methods suffer crucial drawbacks across two critical axes affecting the performance of chart representation models: they are trained on data generated from underlying data tables of the charts, ignoring the visual trends and patterns in chart images, and use weakly aligned vision-language backbone models for domain-specific training, limiting their generalizability when encountering charts in the wild. We address these important drawbacks and introduce ChartGemma, a novel chart understanding and reasoning model developed over PaliGemma. Rather than relying on underlying data tables, ChartGemma is trained on instruction-tuning data generated directly from chart images, thus capturing both high-level trends and low-level visual information from a diverse set of charts. Our simple approach achieves state-of-the-art results across $5$ benchmarks spanning chart summarization, question answering, and fact-checking, and our elaborate qualitative studies on real-world charts show that ChartGemma generates more realistic and factually correct summaries compared to its contemporaries. We release the code, model checkpoints, dataset, and demos at https://github.com/vis-nlp/ChartGemma.
Pathfinder: Exploring Path Diversity for Assessing Internet Censorship Inconsistency
Venue: arXiv cs.CR 2024
Year: 2024

Internet censorship is typically enforced by authorities to achieve information control for a certain group of Internet users. So far existing censorship studies have primarily focused on country-level characterization because (1) in many cases, censorship is enabled by governments with nationwide policies and (2) it is usually hard to control how the probing packets are routed to trigger censorship in different networks inside a country. However, the deployment and implementation of censorship could be highly diverse at the ISP level. In this paper, we investigate Internet censorship from a different perspective by scrutinizing the diverse censorship deployment inside a country. Specifically, by leveraging an end-to-end measurement framework, we deploy multiple geo-distributed back-end control servers to explore various paths from one single vantage point. The generated traffic with the same domain but different control servers' IPs could be forced to traverse different transit networks, thereby being examined by different censorship devices if present. Through our large-scale experiments and in-depth investigation, we reveal that the diversity of Internet censorship caused by different routing paths inside a country is prevalent, implying that (1) the implementations of centralized censorship are commonly incomplete or flawed and (2) decentralized censorship is also common. Moreover, we identify that different hosting platforms also result in inconsistent censorship activities due to different peering relationships with the ISPs in a country. Finally, we present extensive case studies in detail to illustrate the configurations that lead to censorship inconsistency and explore the causes.
Drop it All or Pick it Up? How Developers Responded to the Log4JShell Vulnerability
Venue: arXiv cs.SE 2024
Year: 2024

Although using third-party libraries has become prevalent in contemporary software development, developers often struggle to update their dependencies. Prior works acknowledge that due to the migration effort, priority and other issues cause lags in the migration process. The common assumption is that developers should drop all other activities and prioritize fixing the vulnerability. Our objective is to understand developer behavior when facing high-risk vulnerabilities in their code. We explore the prolific, and possibly one of the cases of the Log4JShell, a vulnerability that has the highest severity rating ever, which received widespread media attention. Using a mixed-method approach, we analyze 219 GitHub Pull Requests (PR) and 354 issues belonging to 53 Maven projects affected by the Log4JShell vulnerability. Our study confirms that developers show a quick response taking from 5 to 6 days. However, instead of dropping everything, surprisingly developer activities tend to increase for all pending issues and PRs. Developer discussions involved either giving information (29.3\%) and seeking information (20.6\%), which is missing in existing support tools. Leveraging this possibly-one of a kind event, insights opens up a new line of research, causing us to rethink best practices and what developers need in order to efficiently fix vulnerabilities.
A High-Quality Workflow for Multi-Resolution Scientific Data Reduction and Visualization
Venue: arXiv cs.DC 2024
Year: 2024

Multi-resolution methods such as Adaptive Mesh Refinement (AMR) can enhance storage efficiency for HPC applications generating vast volumes of data. However, their applicability is limited and cannot be universally deployed across all applications. Furthermore, integrating lossy compression with multi-resolution techniques to further boost storage efficiency encounters significant barriers. To this end, we introduce an innovative workflow that facilitates high-quality multi-resolution data compression for both uniform and AMR simulations. Initially, to extend the usability of multi-resolution techniques, our workflow employs a compression-oriented Region of Interest (ROI) extraction method, transforming uniform data into a multi-resolution format. Subsequently, to bridge the gap between multi-resolution techniques and lossy compressors, we optimize three distinct compressors, ensuring their optimal performance on multi-resolution data. Lastly, we incorporate an advanced uncertainty visualization method into our workflow to understand the potential impacts of lossy compression. Experimental evaluation demonstrates that our workflow achieves significant compression quality improvements.
Systematic Evaluation of Online Speaker Diarization Systems Regarding their Latency
Venue: arXiv cs.CL 2024
Year: 2024

In this paper, different online speaker diarization systems are evaluated on the same hardware with the same test data with regard to their latency. The latency is the time span from audio input to the output of the corresponding speaker label. As part of the evaluation, various model combinations within the DIART framework, a diarization system based on the online clustering algorithm UIS-RNN-SML, and the end-to-end online diarization system FS-EEND are compared. The lowest latency is achieved for the DIART-pipeline with the embedding model pyannote/embedding and the segmentation model pyannote/segmentation. The FS-EEND system shows a similarly good latency. In general there is currently no published research that compares several online diarization systems in terms of their latency. This makes this work even more relevant.
Knowledge-based Drug Samples' Comparison
Venue: arXiv cs.AI 2024
Year: 2024

Drug sample comparison is a process used by the French National police to identify drug distribution networks. The current approach is based on manual comparison done by forensic experts. In this article, we present our approach to acquire, formalise, and specify expert knowledge to improve the current process. For modelling the underlying knowledge we use an ontology coupled with logical rules. The different steps of our approach are designed to be reused in other application domains. The results obtained are explainable making them usable by experts in different fields.
Temporal fingerprints: Identity matching across fully encrypted domain
Venue: arXiv cs.CR 2024
Year: 2024

Technological advancements have significantly transformed communication patterns, introducing a diverse array of online platforms, thereby prompting individuals to use multiple profiles for different domains and objectives. Enhancing the understanding of cross domain identity matching capabilities is essential, not only for practical applications such as commercial strategies and cybersecurity measures, but also for theoretical insights into the privacy implications of data disclosure. In this study, we demonstrate that individual temporal data, in the form of inter-event times distribution, constitutes an individual temporal fingerprint, allowing for matching profiles across different domains back to their associated real-world entity. We evaluate our methodology on encrypted digital trading platforms within the Ethereum Blockchain and present impressing results in matching identities across these privacy-preserving domains, while outperforming previously suggested models. Our findings indicate that simply knowing when an individual is active, even if information about who they talk to and what they discuss is lacking, poses risks to users' privacy, highlighting the inherent challenges in preserving privacy in today's digital landscape.
Data-Driven Tissue- and Subject-Specific Elastic Regularization for Medical Image Registration
Venue: arXiv cs.CV 2024
Year: 2024

Physics-inspired regularization is desired for intra-patient image registration since it can effectively capture the biomechanical characteristics of anatomical structures. However, a major challenge lies in the reliance on physical parameters: Parameter estimations vary widely across the literature, and the physical properties themselves are inherently subject-specific. In this work, we introduce a novel data-driven method that leverages hypernetworks to learn the tissue-dependent elasticity parameters of an elastic regularizer. Notably, our approach facilitates the estimation of patient-specific parameters without the need to retrain the network. We evaluate our method on three publicly available 2D and 3D lung CT and cardiac MR datasets. We find that with our proposed subject-specific tissue-dependent regularization, a higher registration quality is achieved across all datasets compared to using a global regularizer. The code is available at https://github.com/compai-lab/2024-miccai-reithmeir.
Romanization Encoding For Multilingual ASR
Venue: arXiv cs.CL 2024
Year: 2024

We introduce romanization encoding for script-heavy languages to optimize multilingual and code-switching Automatic Speech Recognition (ASR) systems. By adopting romanization encoding alongside a balanced concatenated tokenizer within a FastConformer-RNNT framework equipped with a Roman2Char module, we significantly reduce vocabulary and output dimensions, enabling larger training batches and reduced memory consumption. Our method decouples acoustic modeling and language modeling, enhancing the flexibility and adaptability of the system. In our study, applying this method to Mandarin-English ASR resulted in a remarkable 63.51% vocabulary reduction and notable performance gains of 13.72% and 15.03% on SEAME code-switching benchmarks. Ablation studies on Mandarin-Korean and Mandarin-Japanese highlight our method's strong capability to address the complexities of other script-heavy languages, paving the way for more versatile and effective multilingual ASR systems.
A Mapping Strategy for Interacting with Latent Audio Synthesis Using Artistic Materials
Venue: arXiv cs.SD 2024
Year: 2024

This paper presents a mapping strategy for interacting with the latent spaces of generative AI models. Our approach involves using unsupervised feature learning to encode a human control space and mapping it to an audio synthesis model's latent space. To demonstrate how this mapping strategy can turn high-dimensional sensor data into control mechanisms of a deep generative model, we present a proof-of-concept system that uses visual sketches to control an audio synthesis model. We draw on emerging discourses in XAIxArts to discuss how this approach can contribute to XAI in artistic and creative contexts, we also discuss its current limitations and propose future research directions.