OpenRaiser/Intern-Atlas huggingface dataset:
Silico-centric Theory of Mind
Venue: arXiv cs.AI 2024
Year: 2024

Theory of Mind (ToM) refers to the ability to attribute mental states, such as beliefs, desires, intentions, and knowledge, to oneself and others, and to understand that these mental states can differ from one's own and from reality. We investigate ToM in environments with multiple, distinct, independent AI agents, each possessing unique internal states, information, and objectives. Inspired by human false-belief experiments, we present an AI ('focal AI') with a scenario where its clone undergoes a human-centric ToM assessment. We prompt the focal AI to assess whether its clone would benefit from additional instructions. Concurrently, we give its clones the ToM assessment, both with and without the instructions, thereby engaging the focal AI in higher-order counterfactual reasoning akin to human mentalizing--with respect to humans in one test and to other AI in another. We uncover a discrepancy: Contemporary AI demonstrates near-perfect accuracy on human-centric ToM assessments. Since information embedded in one AI is identically embedded in its clone, additional instructions are redundant. Yet, we observe AI crafting elaborate instructions for their clones, erroneously anticipating a need for assistance. An independent referee AI agrees with these unsupported expectations. Neither the focal AI nor the referee demonstrates ToM in our 'silico-centric' test.
Anatomical Structure-Guided Medical Vision-Language Pre-training
Venue: arXiv cs.CV 2024
Year: 2024

Learning medical visual representations through vision-language pre-training has reached remarkable progress. Despite the promising performance, it still faces challenges, i.e., local alignment lacks interpretability and clinical relevance, and the insufficient internal and external representation learning of image-report pairs. To address these issues, we propose an Anatomical Structure-Guided (ASG) framework. Specifically, we parse raw reports into triplets <anatomical region, finding, existence>, and fully utilize each element as supervision to enhance representation learning. For anatomical region, we design an automatic anatomical region-sentence alignment paradigm in collaboration with radiologists, considering them as the minimum semantic units to explore fine-grained local alignment. For finding and existence, we regard them as image tags, applying an image-tag recognition decoder to associate image features with their respective tags within each sample and constructing soft labels for contrastive learning to improve the semantic association of different image-report pairs. We evaluate the proposed ASG framework on two downstream tasks, including five public benchmarks. Experimental results demonstrate that our method outperforms the state-of-the-art methods.
A complete logic for causal consistency
Venue: arXiv cs.LO 2024
Year: 2024

The $\mathrm{Caus}[-]$ construction takes a base category of ``raw materials'' and builds a category of higher order causal processes, that is a category whose types encode causal (a.k.a. signalling) constraints between collections of systems. Notable examples are categories of higher-order stochastic maps and higher-order quantum channels. Well-typedness in $\mathrm{Caus}[-]$ corresponds to a composition of processes being causally consistent, in the sense that any choice of local processes of the prescribed types yields an overall process respecting causality constraints. It follows that closed processes always occur with probability 1, ruling out e.g. causal paradoxes arising from time loops. It has previously been shown that $\mathrm{Caus}[\mathcal{C}]$ gives a model of MLL+MIX and BV logic, hence these logics give sufficient conditions for causal consistency, but they fail to provide a complete characterisation. In this follow-on work, we introduce graph types as a tool to examine causal structures over graphs in this model. We explore their properties, standard forms, and equivalent definitions; in particular, a process obeys all signalling constraints of the graph iff it is expressible as an affine combination of factorisations into local causal processes connected according to the edges of the graph. The properties of graph types are then used to prove completeness for causal consistency of a new causal logic that conservatively extends pomset logic. The crucial extra ingredient is a notion of distinguished atoms that correspond to first-order states, which only admit a flow of information in one direction. Using the fact that causal logic conservatively extends pomset logic, we finish by giving a physically-meaningful interpretation to a separating statement between pomset and BV.
Semi- and Weakly-Supervised Learning for Mammogram Mass Segmentation with Limited Annotations
Venue: arXiv cs.CV 2024
Year: 2024

Accurate identification of breast masses is crucial in diagnosing breast cancer; however, it can be challenging due to their small size and being camouflaged in surrounding normal glands. Worse still, it is also expensive in clinical practice to obtain adequate pixel-wise annotations for training deep neural networks. To overcome these two difficulties with one stone, we propose a semi- and weakly-supervised learning framework for mass segmentation that utilizes limited strongly-labeled samples and sufficient weakly-labeled samples to achieve satisfactory performance. The framework consists of an auxiliary branch to exclude lesion-irrelevant background areas, a segmentation branch for final prediction, and a spatial prompting module to integrate the complementary information of the two branches. We further disentangle encoded obscure features into lesion-related and others to boost performance. Experiments on CBIS-DDSM and INbreast datasets demonstrate the effectiveness of our method.
B-AVIBench: Towards Evaluating the Robustness of Large Vision-Language Model on Black-box Adversarial Visual-Instructions
Venue: arXiv cs.CV 2024
Year: 2024

Large Vision-Language Models (LVLMs) have shown significant progress in responding well to visual-instructions from users. However, these instructions, encompassing images and text, are susceptible to both intentional and inadvertent attacks. Despite the critical importance of LVLMs' robustness against such threats, current research in this area remains limited. To bridge this gap, we introduce B-AVIBench, a framework designed to analyze the robustness of LVLMs when facing various Black-box Adversarial Visual-Instructions (B-AVIs), including four types of image-based B-AVIs, ten types of text-based B-AVIs, and nine types of content bias B-AVIs (such as gender, violence, cultural, and racial biases, among others). We generate 316K B-AVIs encompassing five categories of multimodal capabilities (ten tasks) and content bias. We then conduct a comprehensive evaluation involving 14 open-source LVLMs to assess their performance. B-AVIBench also serves as a convenient tool for practitioners to evaluate the robustness of LVLMs against B-AVIs. Our findings and extensive experimental results shed light on the vulnerabilities of LVLMs, and highlight that inherent biases exist even in advanced closed-source LVLMs like GeminiProVision and GPT-4V. This underscores the importance of enhancing the robustness, security, and fairness of LVLMs. The source code and benchmark are available at https://github.com/zhanghao5201/B-AVIBench.
From Pro, Anti to Informative and Hesitant: An Infoveillance study of COVID-19 vaccines and vaccination discourse on Twitter
Venue: arXiv cs.SI 2024
Year: 2024

COVID-19 pandemic has brought unprecedented challenges to the world, and vaccination has been a key strategy to combat the disease. Since Twitter is one of the most widely used public microblogging platforms, researchers have analysed COVID-19 vaccines and vaccination Twitter discourse to explore the conversational dynamics around the topic. While contributing to the crisis informatics literature, we curate a large-scale geotagged Twitter dataset, GeoCovaxTweets Extended, and explore the discourse through multiple spatiotemporal analyses. This dataset covers a longer time span of 38 months, from the announcement of the first vaccine to the availability of booster doses. Results show that 43.4% of the collected tweets, although containing phrases and keywords related to vaccines and vaccinations, were unrelated to the COVID-19 context. In total, 23.1% of the discussions on vaccines and vaccinations were classified as Pro, 16% as Hesitant, 11.4% as Anti, and 6.1% as Informative. The trend shifted towards Pro and Informative tweets globally as vaccination programs progressed, indicating a change in the public's perception of COVID-19 vaccines and vaccination. Furthermore, we explored the discourse based on account attributes, i.e., followers counts and tweet counts. Results show a significant pattern of discourse differences. Our findings highlight the potential of harnessing a large-scale geotagged Twitter dataset to understand global public health communication and to inform targeted interventions aimed at addressing vaccine hesitancy.
REPQC: Reverse Engineering and Backdooring Hardware Accelerators for Post-quantum Cryptography
Venue: arXiv cs.CR 2024
Year: 2024

Significant research efforts have been dedicated to designing cryptographic algorithms that are quantum-resistant. The motivation is clear: robust quantum computers, once available, will render current cryptographic standards vulnerable. Thus, we need new Post-Quantum Cryptography (PQC) algorithms, and, due to the inherent complexity of such algorithms, there is also a demand to accelerate them in hardware. In this paper, we show that PQC hardware accelerators can be backdoored by two different adversaries located in the chip supply chain. We propose REPQC, a sophisticated reverse engineering algorithm that can be employed to confidently identify hashing operations (i.e., Keccak) within the PQC accelerator - the location of which serves as an anchor for finding secret information to be leaked. Armed with REPQC, an adversary proceeds to insert malicious logic in the form of a stealthy Hardware Trojan Horse (HTH). Using Dilithium as a study case, our results demonstrate that HTHs that increase the accelerator's layout density by as little as 0.1\% can be inserted without any impact on the performance of the circuit and with a marginal increase in power consumption. An essential aspect is that the entire reverse engineering in REPQC is automated, and so is the HTH insertion that follows it, empowering adversaries to explore multiple HTH designs and identify the most suitable one.
Joint Port Selection and Beamforming Design for Fluid Antenna Assisted Integrated Data and Energy Transfer
Venue: arXiv cs.IT 2024
Year: 2024

Integrated data and energy transfer (IDET) has been of fundamental importance for providing both wireless data transfer (WDT) and wireless energy transfer (WET) services towards low-power devices. Fluid antenna (FA) is capable of exploiting the huge spatial diversity of the wireless channel to enhance the receive signal strength, which is more suitable for the tiny-size low-power devices having the IDET requirements. In this letter, a multiuser FA assisted IDET system is studied and the weighted energy harvesting power at energy receivers (ERs) is maximized by jointly optimizing the port selection and transmit beamforming design under imperfect channel state information (CSI), while the signal-to-interference-plus-noise ratio (SINR) constraint for each data receiver (DR) is satisfied. An efficient algorithm is proposed to obtain the suboptimal solutions for the non-convex problem. Simulation results evaluate the performance of the FA-IDET system, while also demonstrate that FA outperforms the multi-input-multi-output (MIMO) counterpart in terms of the IDET performance, as long as the port number is large enough.
An Extensible Framework for Architecture-Based Data Flow Analysis for Information Security
Venue: arXiv cs.SE 2024
Year: 2024

The growing interconnection between software systems increases the need for security already at design time. Security-related properties like confidentiality are often analyzed based on data flow diagrams (DFDs). However, manually analyzing DFDs of large software systems is bothersome and error-prone, and adjusting an already deployed software is costly. Additionally, closed analysis ecosystems limit the reuse of modeled information and impede comprehensive statements about a system's security. In this paper, we present an open and extensible framework for data flow analysis. The central element of our framework is our new implementation of a well-validated data-flow-based analysis approach. The framework is compatible with DFDs and can also extract data flows from the Palladio architectural description language. We showcase the extensibility with multiple model and analysis extensions. Our evaluation indicates that we can analyze similar scenarios while achieving higher scalability compared to previous implementations.
Borrowing Treasures from Neighbors: In-Context Learning for Multimodal Learning with Missing Modalities and Data Scarcity
Venue: arXiv cs.LG 2024
Year: 2024

Multimodal machine learning with missing modalities is an increasingly relevant challenge arising in various applications such as healthcare. This paper extends the current research into missing modalities to the low-data regime, i.e., a downstream task has both missing modalities and limited sample size issues. This problem setting is particularly challenging and also practical as it is often expensive to get full-modality data and sufficient annotated training samples. We propose to use retrieval-augmented in-context learning to address these two crucial issues by unleashing the potential of a transformer's in-context learning ability. Diverging from existing methods, which primarily belong to the parametric paradigm and often require sufficient training samples, our work exploits the value of the available full-modality data, offering a novel perspective on resolving the challenge. The proposed data-dependent framework exhibits a higher degree of sample efficiency and is empirically demonstrated to enhance the classification model's performance on both full- and missing-modality data in the low-data regime across various multimodal learning tasks. When only 1% of the training data are available, our proposed method demonstrates an average improvement of 6.1% over a recent strong baseline across various datasets and missing states. Notably, our method also reduces the performance gap between full-modality and missing-modality data compared with the baseline.
Efficient Transferability Assessment for Selection of Pre-trained Detectors
Venue: arXiv cs.CV 2024
Year: 2024

Large-scale pre-training followed by downstream fine-tuning is an effective solution for transferring deep-learning-based models. Since finetuning all possible pre-trained models is computational costly, we aim to predict the transferability performance of these pre-trained models in a computational efficient manner. Different from previous work that seek out suitable models for downstream classification and segmentation tasks, this paper studies the efficient transferability assessment of pre-trained object detectors. To this end, we build up a detector transferability benchmark which contains a large and diverse zoo of pre-trained detectors with various architectures, source datasets and training schemes. Given this zoo, we adopt 7 target datasets from 5 diverse domains as the downstream target tasks for evaluation. Further, we propose to assess classification and regression sub-tasks simultaneously in a unified framework. Additionally, we design a complementary metric for evaluating tasks with varying objects. Experimental results demonstrate that our method outperforms other state-of-the-art approaches in assessing transferability under different target domains while efficiently reducing wall-clock time 32$\times$ and requires a mere 5.2\% memory footprint compared to brute-force fine-tuning of all pre-trained detectors.
Anomaly Detection by Adapting a pre-trained Vision Language Model
Venue: arXiv cs.CV 2024
Year: 2024

Recently, large vision and language models have shown their success when adapting them to many downstream tasks. In this paper, we present a unified framework named CLIP-ADA for Anomaly Detection by Adapting a pre-trained CLIP model. To this end, we make two important improvements: 1) To acquire unified anomaly detection across industrial images of multiple categories, we introduce the learnable prompt and propose to associate it with abnormal patterns through self-supervised learning. 2) To fully exploit the representation power of CLIP, we introduce an anomaly region refinement strategy to refine the localization quality. During testing, the anomalies are localized by directly calculating the similarity between the representation of the learnable prompt and the image. Comprehensive experiments demonstrate the superiority of our framework, e.g., we achieve the state-of-the-art 97.5/55.6 and 89.3/33.1 on MVTec-AD and VisA for anomaly detection and localization. In addition, the proposed method also achieves encouraging performance with marginal training data, which is more challenging.
A Reinforcement Learning Approach to Dairy Farm Battery Management using Q Learning
Venue: arXiv cs.LG 2024
Year: 2024

Dairy farming consumes a significant amount of energy, making it an energy-intensive sector within agriculture. Integrating renewable energy generation into dairy farming could help address this challenge. Effective battery management is important for integrating renewable energy generation. Managing battery charging and discharging poses significant challenges because of fluctuations in electrical consumption, the intermittent nature of renewable energy generation, and fluctuations in energy prices. Artificial Intelligence (AI) has the potential to significantly improve the use of renewable energy in dairy farming, however, there is limited research conducted in this particular domain. This research considers Ireland as a case study as it works towards attaining its 2030 energy strategy centered on the utilization of renewable sources. This study proposes a Q-learning-based algorithm for scheduling battery charging and discharging in a dairy farm setting. This research also explores the effect of the proposed algorithm by adding wind generation data and considering additional case studies. The proposed algorithm reduces the cost of imported electricity from the grid by 13.41%, peak demand by 2%, and 24.49% when utilizing wind generation. These results underline how reinforcement learning is highly effective in managing batteries in the dairy farming sector.
RANDAO-based RNG: Last Revealer Attacks in Ethereum 2.0 Randomness and a Potential Solution
Venue: arXiv cs.CR 2024
Year: 2024

Ethereum 2.0 is a major upgrade to improve its scalability, throughput, and security. In this version, RANDAO is the scheme to randomly select the users who propose, confirm blocks, and get rewards. However, a vulnerability, referred to as the `Last Revealer Attack' (LRA), compromises the randomness of this scheme by introducing bias to the Random Number Generator (RNG) process. This vulnerability is first clarified again in this study. After that, we propose a Shamir's Secret Sharing (SSS)-based RANDAO scheme to mitigate the LRA. Through our analysis, the proposed method can prevent the LRA under favorable network conditions.
GenAD: Generalized Predictive Model for Autonomous Driving
Venue: arXiv cs.CV 2024
Year: 2024

In this paper, we introduce the first large-scale video prediction model in the autonomous driving discipline. To eliminate the restriction of high-cost data collection and empower the generalization ability of our model, we acquire massive data from the web and pair it with diverse and high-quality text descriptions. The resultant dataset accumulates over 2000 hours of driving videos, spanning areas all over the world with diverse weather conditions and traffic scenarios. Inheriting the merits from recent latent diffusion models, our model, dubbed GenAD, handles the challenging dynamics in driving scenes with novel temporal reasoning blocks. We showcase that it can generalize to various unseen driving datasets in a zero-shot manner, surpassing general or driving-specific video prediction counterparts. Furthermore, GenAD can be adapted into an action-conditioned prediction model or a motion planner, holding great potential for real-world driving applications.
A Hybrid Intelligence Method for Argument Mining
Venue: arXiv cs.AI 2024
Year: 2024

Large-scale survey tools enable the collection of citizen feedback in opinion corpora. Extracting the key arguments from a large and noisy set of opinions helps in understanding the opinions quickly and accurately. Fully automated methods can extract arguments but (1) require large labeled datasets that induce large annotation costs and (2) work well for known viewpoints, but not for novel points of view. We propose HyEnA, a hybrid (human + AI) method for extracting arguments from opinionated texts, combining the speed of automated processing with the understanding and reasoning capabilities of humans. We evaluate HyEnA on three citizen feedback corpora. We find that, on the one hand, HyEnA achieves higher coverage and precision than a state-of-the-art automated method when compared to a common set of diverse opinions, justifying the need for human insight. On the other hand, HyEnA requires less human effort and does not compromise quality compared to (fully manual) expert analysis, demonstrating the benefit of combining human and artificial intelligence.
Fine-tuning vs Prompting, Can Language Models Understand Human Values?
Venue: arXiv cs.CL 2024
Year: 2024

Accurately handling the underlying support values in sentences is crucial for understanding the speaker's tendencies, yet it poses a challenging task in natural language understanding (NLU). In this article, we explore the potential of fine-tuning and prompt tuning in this downstream task, using the Human Value Detection 2023. Additionally, we attempt to validate whether models can effectively solve the problem based on the knowledge acquired during the pre-training stage. Simultaneously, our interest lies in the capabilities of large language models (LLMs) aligned with RLHF in this task, and some preliminary attempts are presented.
ClaimVer: Explainable Claim-Level Verification and Evidence Attribution of Text Through Knowledge Graphs
Venue: arXiv cs.CL 2024
Year: 2024

In the midst of widespread misinformation and disinformation through social media and the proliferation of AI-generated texts, it has become increasingly difficult for people to validate and trust information they encounter. Many fact-checking approaches and tools have been developed, but they often lack appropriate explainability or granularity to be useful in various contexts. A text validation method that is easy to use, accessible, and can perform fine-grained evidence attribution has become crucial. More importantly, building user trust in such a method requires presenting the rationale behind each prediction, as research shows this significantly influences people's belief in automated systems. Localizing and bringing users' attention to the specific problematic content is also paramount, instead of providing simple blanket labels. In this paper, we present ClaimVer, a human-centric framework tailored to meet users' informational and verification needs by generating rich annotations and thereby reducing cognitive load. Designed to deliver comprehensive evaluations of texts, it highlights each claim, verifies it against a trusted knowledge graph (KG), presents the evidence, and provides succinct, clear explanations for each claim prediction. Finally, our framework introduces an attribution score, enhancing applicability across a wide range of downstream tasks.