OpenRaiser/Intern-Atlas huggingface dataset:
Multimodal Collaboration Networks for Geospatial Vehicle Detection in Dense, Occluded, and Large-Scale Events
Venue: arXiv cs.CV 2024
Year: 2024

In large-scale disaster events, the planning of optimal rescue routes depends on the object detection ability at the disaster scene, with one of the main challenges being the presence of dense and occluded objects. Existing methods, which are typically based on the RGB modality, struggle to distinguish targets with similar colors and textures in crowded environments and are unable to identify obscured objects. To this end, we first construct two multimodal dense and occlusion vehicle detection datasets for large-scale events, utilizing RGB and height map modalities. Based on these datasets, we propose a multimodal collaboration network for dense and occluded vehicle detection, MuDet for short. MuDet hierarchically enhances the completeness of discriminable information within and across modalities and differentiates between simple and complex samples. MuDet includes three main modules: Unimodal Feature Hierarchical Enhancement (Uni-Enh), Multimodal Cross Learning (Mul-Lea), and Hard-easy Discriminative (He-Dis) Pattern. Uni-Enh and Mul-Lea enhance the features within each modality and facilitate the cross-integration of features from two heterogeneous modalities. He-Dis effectively separates densely occluded vehicle targets with significant intra-class differences and minimal inter-class differences by defining and thresholding confidence values, thereby suppressing the complex background. Experimental results on two re-labeled multimodal benchmark datasets, the 4K-SAI-LCS dataset, and the ISPRS Potsdam dataset, demonstrate the robustness and generalization of the MuDet. The codes of this work are available openly at \url{https://github.com/Shank2358/MuDet}.
Total Variation Distance for Product Distributions is $\#\mathsf{P}$-Complete
Venue: arXiv cs.CC 2024
Year: 2024

We show that computing the total variation distance between two product distributions is $\#\mathsf{P}$-complete. This is in stark contrast with other distance measures such as Kullback-Leibler, Chi-square, and Hellinger, which tensorize over the marginals leading to efficient algorithms.
T-Watch: Towards Timed Execution of Private Transaction in Blockchains
Venue: arXiv cs.CR 2024
Year: 2024

In blockchains such as Bitcoin and Ethereum, transactions represent the primary mechanism that the external world can use to trigger a change of blockchain state. Transactions serve as key sources of evidence and play a vital role in forensic analysis. Timed transaction refers to a specific class of service that enables a user to schedule a transaction to change the blockchain state during a chosen future time-frame. This paper proposes T-Watch, a decentralized and cost-efficient approach for users to schedule timed execution of any type of transaction in Ethereum with privacy guarantees. T-Watch employs a novel combination of threshold secret sharing and decentralized smart contracts. To protect the private elements of a scheduled transaction from getting disclosed before the future time-frame, T-Watch maintains shares of the decryption key of the scheduled transaction using a group of executors recruited in a blockchain network before the specified future time-frame and restores the scheduled transaction at a proxy smart contract to trigger the change of blockchain state at the required time-frame. To reduce the cost of smart contract execution in T-Watch, we carefully design the proposed protocol to run in an optimistic mode by default and then switch to a pessimistic mode once misbehaviors occur. Furthermore, the protocol supports users to form service request pooling to further reduce the gas cost. We rigorously analyze the security of T-Watch and implement the protocol over the Ethereum official test network. The results demonstrate that T-Watch is more scalable compared to the state of the art and could reduce the cost by over 90% through pooling.
Exploring Equilibrium Strategies in Network Games with Generative AI
Venue: arXiv cs.GT 2024
Year: 2024

Game theory offers a powerful framework for analyzing strategic interactions among decision-makers, providing tools to model, analyze, and predict their behavior. However, implementing game theory can be challenging due to difficulties in deriving solutions, understanding interactions, and ensuring optimal performance. Traditional non-AI and discriminative AI approaches have made valuable contributions but struggle with limitations in handling large-scale games and dynamic scenarios. In this context, generative AI emerges as a promising solution because of its superior data analysis and generation capabilities. This paper comprehensively summarizes the challenges, solutions, and outlooks of combining generative AI with game theory. We start with reviewing the limitations of traditional non-AI and discriminative AI approaches in employing game theory, and then highlight the necessity and advantages of integrating generative AI. Next, we explore the applications of generative AI in various stages of the game theory lifecycle, including model formulation, solution derivation, and strategy improvement. Additionally, from game theory viewpoint, we propose a generative AI-enabled framework for optimizing machine learning model performance against false data injection attacks, supported by a case study to demonstrate its effectiveness. Finally, we outline future research directions for generative AI-enabled game theory, paving the way for its further advancements and development.
Collateral Portfolio Optimization in Crypto-Backed Stablecoins
Venue: arXiv cs.CR 2024
Year: 2024

Stablecoins - crypto tokens whose value is pegged to a real-world asset such as the US Dollar - are an important component of the DeFi ecosystem as they mitigate the impact of token price volatility. In crypto-backed stablecoins, the peg is founded on the guarantee that in case of system shutdown, each stablecoin can be exchanged for a basket of other crypto tokens worth approximately its nominal value. However, price fluctuations that affect the collateral tokens may cause this guarantee to be invalidated. In this work, we investigate the impact of the collateral portfolio's composition on the resilience to this type of catastrophic event. For stablecoins whose developers maintain a significant portion of the collateral (e.g., MakerDAO's Dai), we propose two portfolio optimization methods, based on convex optimization and (semi)variance minimization, that account for the correlation between the various token prices. We compare the optimal portfolios to the historical evolution of Dai's collateral portfolio, and to aid reproducibility, we have made our data and code publicly available.
SpeechGuard: Exploring the Adversarial Robustness of Multimodal Large Language Models
Venue: arXiv cs.CL 2024
Year: 2024

Integrated Speech and Large Language Models (SLMs) that can follow speech instructions and generate relevant text responses have gained popularity lately. However, the safety and robustness of these models remains largely unclear. In this work, we investigate the potential vulnerabilities of such instruction-following speech-language models to adversarial attacks and jailbreaking. Specifically, we design algorithms that can generate adversarial examples to jailbreak SLMs in both white-box and black-box attack settings without human involvement. Additionally, we propose countermeasures to thwart such jailbreaking attacks. Our models, trained on dialog data with speech instructions, achieve state-of-the-art performance on spoken question-answering task, scoring over 80% on both safety and helpfulness metrics. Despite safety guardrails, experiments on jailbreaking demonstrate the vulnerability of SLMs to adversarial perturbations and transfer attacks, with average attack success rates of 90% and 10% respectively when evaluated on a dataset of carefully designed harmful questions spanning 12 different toxic categories. However, we demonstrate that our proposed countermeasures reduce the attack success significantly.
Cross-Dataset Generalization For Retinal Lesions Segmentation
Venue: arXiv cs.CV 2024
Year: 2024

Identifying lesions in fundus images is an important milestone toward an automated and interpretable diagnosis of retinal diseases. To support research in this direction, multiple datasets have been released, proposing groundtruth maps for different lesions. However, important discrepancies exist between the annotations and raise the question of generalization across datasets. This study characterizes several known datasets and compares different techniques that have been proposed to enhance the generalisation performance of a model, such as stochastic weight averaging, model soups and ensembles. Our results provide insights into how to combine coarsely labelled data with a finely-grained dataset in order to improve the lesions segmentation.
Perivascular space Identification Nnunet for Generalised Usage (PINGU)
Venue: arXiv cs.CV 2024
Year: 2024

Perivascular spaces(PVSs) form a central component of the brainś waste clearance system, the glymphatic system. These structures are visible on MRI images, and their morphology is associated with aging and neurological disease. Manual quantification of PVS is time consuming and subjective. Numerous deep learning methods for PVS segmentation have been developed, however the majority have been developed and evaluated on homogenous datasets and high resolution scans, perhaps limiting their applicability for the wide range of image qualities acquired in clinic and research. In this work we train a nnUNet, a top-performing biomedical image segmentation algorithm, on a heterogenous training sample of manually segmented MRI images of a range of different qualities and resolutions from 6 different datasets. These are compared to publicly available deep learning methods for 3D segmentation of PVS. The resulting model, PINGU (Perivascular space Identification Nnunet for Generalised Usage), achieved voxel and cluster level dice scores of 0.50(SD=0.15), 0.63(0.17) in the white matter(WM), and 0.54(0.11), 0.66(0.17) in the basal ganglia(BG). Performance on data from unseen sites was substantially lower for both PINGU(0.20-0.38(WM, voxel), 0.29-0.58(WM, cluster), 0.22-0.36(BG, voxel), 0.46-0.60(BG, cluster)) and the publicly available algorithms(0.18-0.30(WM, voxel), 0.29-0.38(WM cluster), 0.10-0.20(BG, voxel), 0.15-0.37(BG, cluster)), but PINGU strongly outperformed the publicly available algorithms, particularly in the BG. Finally, training PINGU on manual segmentations from a single site with homogenous scan properties gave marginally lower performances on internal cross-validation, but in some cases gave higher performance on external validation. PINGU stands out as broad-use PVS segmentation tool, with particular strength in the BG, an area of PVS related to vascular disease and pathology.
UnMarker: A Universal Attack on Defensive Image Watermarking
Venue: arXiv cs.CR 2024
Year: 2024

Reports regarding the misuse of Generative AI (GenAI) to create deepfakes are frequent. Defensive watermarking enables GenAI providers to hide fingerprints in their images and use them later for deepfake detection. Yet, its potential has not been fully explored. We present UnMarker -- the first practical universal attack on defensive watermarking. Unlike existing attacks, UnMarker requires no detector feedback, no unrealistic knowledge of the watermarking scheme or similar models, and no advanced denoising pipelines that may not be available. Instead, being the product of an in-depth analysis of the watermarking paradigm revealing that robust schemes must construct their watermarks in the spectral amplitudes, UnMarker employs two novel adversarial optimizations to disrupt the spectra of watermarked images, erasing the watermarks. Evaluations against SOTA schemes prove UnMarker's effectiveness. It not only defeats traditional schemes while retaining superior quality compared to existing attacks but also breaks semantic watermarks that alter an image's structure, reducing the best detection rate to $43\%$ and rendering them useless. To our knowledge, UnMarker is the first practical attack on semantic watermarks, which have been deemed the future of defensive watermarking. Our findings show that defensive watermarking is not a viable defense against deepfakes, and we urge the community to explore alternatives.
Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control
Venue: arXiv cs.LG 2024
Year: 2024

Disentangling model activations into meaningful features is a central problem in interpretability. However, the absence of ground-truth for these features in realistic scenarios makes validating recent approaches, such as sparse dictionary learning, elusive. To address this challenge, we propose a framework for evaluating feature dictionaries in the context of specific tasks, by comparing them against \emph{supervised} feature dictionaries. First, we demonstrate that supervised dictionaries achieve excellent approximation, control, and interpretability of model computations on the task. Second, we use the supervised dictionaries to develop and contextualize evaluations of unsupervised dictionaries along the same three axes. We apply this framework to the indirect object identification (IOI) task using GPT-2 Small, with sparse autoencoders (SAEs) trained on either the IOI or OpenWebText datasets. We find that these SAEs capture interpretable features for the IOI task, but they are less successful than supervised features in controlling the model. Finally, we observe two qualitative phenomena in SAE training: feature occlusion (where a causally relevant concept is robustly overshadowed by even slightly higher-magnitude ones in the learned features), and feature over-splitting (where binary features split into many smaller, less interpretable features). We hope that our framework will provide a useful step towards more objective and grounded evaluations of sparse dictionary learning methods.
Large-Scale Metric Computation in Online Controlled Experiment Platform
Venue: arXiv cs.DC 2024
Year: 2024

Online controlled experiment (also called A/B test or experiment) is the most important tool for decision-making at a wide range of data-driven companies like Microsoft, Google, Meta, etc. Metric computation is the core procedure for reaching a conclusion during an experiment. With the growth of experiments and metrics in an experiment platform, computing metrics efficiently at scale becomes a non-trivial challenge. This work shows how metric computation in WeChat experiment platform can be done efficiently using bit-sliced index (BSI) arithmetic. This approach has been implemented in a real world system and the performance results are presented, showing that the BSI arithmetic approach is very suitable for large-scale metric computation scenarios.
WaterMamba: Visual State Space Model for Underwater Image Enhancement
Venue: arXiv cs.CV 2024
Year: 2024

Underwater imaging often suffers from low quality due to factors affecting light propagation and absorption in water. To improve image quality, some underwater image enhancement (UIE) methods based on convolutional neural networks (CNN) and Transformer have been proposed. However, CNN-based UIE methods are limited in modeling long-range dependencies, and Transformer-based methods involve a large number of parameters and complex self-attention mechanisms, posing efficiency challenges. Considering computational complexity and severe underwater image degradation, a state space model (SSM) with linear computational complexity for UIE, named WaterMamba, is proposed. We propose spatial-channel omnidirectional selective scan (SCOSS) blocks comprising spatial-channel coordinate omnidirectional selective scan (SCCOSS) modules and a multi-scale feedforward network (MSFFN). The SCOSS block models pixel and channel information flow, addressing dependencies. The MSFFN facilitates information flow adjustment and promotes synchronized operations within SCCOSS modules. Extensive experiments showcase WaterMamba's cutting-edge performance with reduced parameters and computational resources, outperforming state-of-the-art methods on various datasets, validating its effectiveness and generalizability. The code will be released on GitHub after acceptance.
Understanding the performance gap between online and offline alignment algorithms
Venue: arXiv cs.LG 2024
Year: 2024

Reinforcement learning from human feedback (RLHF) is the canonical framework for large language model alignment. However, rising popularity in offline alignment algorithms challenge the need for on-policy sampling in RLHF. Within the context of reward over-optimization, we start with an opening set of experiments that demonstrate the clear advantage of online methods over offline methods. This prompts us to investigate the causes to the performance discrepancy through a series of carefully designed experimental ablations. We show empirically that hypotheses such as offline data coverage and data quality by itself cannot convincingly explain the performance difference. We also find that while offline algorithms train policy to become good at pairwise classification, it is worse at generations; in the meantime the policies trained by online algorithms are good at generations while worse at pairwise classification. This hints at a unique interplay between discriminative and generative capabilities, which is greatly impacted by the sampling process. Lastly, we observe that the performance discrepancy persists for both contrastive and non-contrastive loss functions, and appears not to be addressed by simply scaling up policy networks. Taken together, our study sheds light on the pivotal role of on-policy sampling in AI alignment, and hints at certain fundamental challenges of offline alignment algorithms.
Challenges and Opportunities in Text Generation Explainability
Venue: arXiv cs.CL 2024
Year: 2024

The necessity for interpretability in natural language processing (NLP) has risen alongside the growing prominence of large language models. Among the myriad tasks within NLP, text generation stands out as a primary objective of autoregressive models. The NLP community has begun to take a keen interest in gaining a deeper understanding of text generation, leading to the development of model-agnostic explainable artificial intelligence (xAI) methods tailored to this task. The design and evaluation of explainability methods are non-trivial since they depend on many factors involved in the text generation process, e.g., the autoregressive model and its stochastic nature. This paper outlines 17 challenges categorized into three groups that arise during the development and assessment of attribution-based explainability methods. These challenges encompass issues concerning tokenization, defining explanation similarity, determining token importance and prediction change metrics, the level of human intervention required, and the creation of suitable test datasets. The paper illustrates how these challenges can be intertwined, showcasing new opportunities for the community. These include developing probabilistic word-level explainability methods and engaging humans in the explainability pipeline, from the data design to the final evaluation, to draw robust conclusions on xAI methods.
Empowering Programmable Wireless Environments with Optical Anchor-based Positioning
Venue: arXiv cs.IT 2024
Year: 2024

The evolution toward sixth-generation (6G) wireless networks has introduced programmable wireless environments (PWEs) and reconfigurable intelligent surfaces (RISs) as transformative elements for achieving near-deterministic wireless communications. However, the enhanced capabilities of RISs within PWEs, especially as we move toward more complex electromagnetic functions by increasing the number of reflecting elements, underscore the need for high-precision user localization, since inaccurate localization could lead to erroneous configuration of RISs, which would then compromise the effectiveness of PWEs. In this direction, this paper investigates the integration of RISs and optical anchors within PWEs, emphasizing the crucial role of ultra-precise localization in unlocking advanced electromagnetic functionalities. Specifically, we present an in-depth analysis of various localization techniques, both RISbased and RIS-independent, while introducing the concept of empowering PWEs with optical anchors for enhanced localization precision. Our findings highlight that accurate localization is essential to fully exploit the capabilities of RISs, paving the way for future applications. Through this exploration, we contribute to the advancement of PWEs in line with the ambitious goals of the 6G standards and improve the quality of service in next generation wireless networks.
Self-Distillation Improves DNA Sequence Inference
Venue: arXiv cs.LG 2024
Year: 2024

Self-supervised pretraining (SSP) has been recognized as a method to enhance prediction accuracy in various downstream tasks. However, its efficacy for DNA sequences remains somewhat constrained. This limitation stems primarily from the fact that most existing SSP approaches in genomics focus on masked language modeling of individual sequences, neglecting the crucial aspect of encoding statistics across multiple sequences. To overcome this challenge, we introduce an innovative deep neural network model, which incorporates collaborative learning between a `student' and a `teacher' subnetwork. In this model, the student subnetwork employs masked learning on nucleotides and progressively adapts its parameters to the teacher subnetwork through an exponential moving average approach. Concurrently, both subnetworks engage in contrastive learning, deriving insights from two augmented representations of the input sequences. This self-distillation process enables our model to effectively assimilate both contextual information from individual sequences and distributional data across the sequence population. We validated our approach with preliminary pretraining using the human reference genome, followed by applying it to 20 downstream inference tasks. The empirical results from these experiments demonstrate that our novel method significantly boosts inference performance across the majority of these tasks. Our code is available at https://github.com/wiedersehne/FinDNA.
SecScore: Enhancing the CVSS Threat Metric Group with Empirical Evidences
Venue: arXiv cs.CR 2024
Year: 2024

Background: Timely prioritising and remediating vulnerabilities are paramount in the dynamic cybersecurity field, and one of the most widely used vulnerability scoring systems (CVSS) does not address the increasing likelihood of emerging an exploit code. Aims: We present SecScore, an innovative vulnerability severity score that enhances CVSS Threat metric group with statistical models from empirical evidences of real-world exploit codes. Method: SecScore adjusts the traditional CVSS score using an explainable and empirical method that more accurately and promptly captures the dynamics of exploit code development. Results: Our approach can integrate seamlessly into the assessment/prioritisation stage of several vulnerability management processes, improving the effectiveness of prioritisation and ensuring timely remediation. We provide real-world statistical analysis and models for a wide range of vulnerability types and platforms, demonstrating that SecScore is flexible according to the vulnerability's profile. Comprehensive experiments validate the value and timeliness of SecScore in vulnerability prioritisation. Conclusions: SecScore advances the vulnerability metrics theory and enhances organisational cybersecurity with practical insights.