OpenRaiser/Intern-Atlas huggingface dataset:
COMMA: A Communicative Multimodal Multi-Agent Benchmark
Venue: arXiv cs.AI 2025
Year: 2025

The rapid advances of multimodal agents built on large foundation models have largely overlooked their potential for language-based communication between agents in collaborative tasks. This oversight presents a critical gap in understanding their effectiveness in real-world deployments, particularly when communicating with humans. Existing agentic benchmarks fail to address key aspects of inter-agent communication and collaboration, particularly in scenarios where agents have unequal access to information and must work together to achieve tasks beyond the scope of individual capabilities. To fill this gap, we introduce COMMA: a novel puzzle benchmark designed to evaluate the collaborative performance of multimodal multi-agent systems through language communication. Our benchmark features a variety of multimodal puzzles, providing a comprehensive evaluation across four key categories of agentic capability in a communicative collaboration setting. Our findings reveal surprising weaknesses in state-of-the-art models, including strong proprietary models like GPT-4o and reasoning models like o4-mini. Many chain of thought reasoning models such as R1-Onevision and LLaVA-CoT struggle to outperform even a random baseline in agent-agent collaboration, indicating a potential growth area in their communication abilities.
Ormer: A Manipulation-resistant and Gas-efficient Blockchain Pricing Oracle for DeFi
Venue: arXiv cs.CR 2025
Year: 2025

Price feeds of cryptocurrencies are essential for Decentralized Finance (DeFi) applications to realize fundamental trading and exchanging functionalities, which are retrieved from external price data sources such as exchanges and input to on-chain smart contracts in real-time. Currently, arithmetic mean based time-weighted average price (TWAP) oracles are widely used to process price feeds by averaging asset price with short time frame to achieve reliable and gas-efficient pricing. However, recent research indicates that TWAP is vulnerable to price manipulation attacks, resulting in abnormal price fluctuations and severe financial loss. Even worse, TWAP oracles usually set a relatively long time frame setting to prevent such attack. However, it would further introduce long delays and high price deviation errors from the market asset price. To address this issue, we propose a novel on-chain gas-efficient pricing algorithm (Ormer) that heuristically estimates the median of asset price within an observation window based on a piecewise-parabolic formula, while the time delay is suppressed by fusing estimations with different window sizes. Our evaluation based on multiple pairs of token swapping price feed across different chains show that Ormer reduces the mean absolute price error by 15.3% and the time delay by 49.3% compared to TWAP. For gas efficiency, regardless of the number of price observations, an encoding mechanism with constant storage requirement is employed without saving all the historical data for median estimation. Surprisingly, the lowest gas consumption of Ormer is even 15.2% less than TWAP, and the oracle querying fee would be saved up to ~20K USD per day for DeFi participants.
COMPL-AI Framework: A Technical Interpretation and LLM Benchmarking Suite for the EU Artificial Intelligence Act
Venue: arXiv cs.CL 2025
Year: 2025

The EU's Artificial Intelligence Act (AI Act) is a significant step towards responsible AI development, but lacks clear technical interpretation, making it difficult to assess models' compliance. This work presents COMPL-AI, a comprehensive framework consisting of (i) the first technical interpretation of the EU AI Act, translating its broad regulatory requirements into measurable technical requirements, with the focus on large language models (LLMs), and (ii) an open-source Act-centered benchmarking suite, based on thorough surveying and implementation of state-of-the-art LLM benchmarks. By evaluating 12 prominent LLMs in the context of COMPL-AI, we reveal shortcomings in existing models and benchmarks, particularly in areas like robustness, safety, diversity, and fairness. This work highlights the need for a shift in focus towards these aspects, encouraging balanced development of LLMs and more comprehensive regulation-aligned benchmarks. Simultaneously, COMPL-AI for the first time demonstrates the possibilities and difficulties of bringing the Act's obligations to a more concrete, technical level. As such, our work can serve as a useful first step towards having actionable recommendations for model providers, and contributes to ongoing efforts of the EU to enable application of the Act, such as the drafting of the GPAI Code of Practice.
Octopus Inspired Optimization (OIO): A Hierarchical Framework for Navigating Protein Fitness Landscapes
Venue: arXiv cs.NE 2025
Year: 2025

Navigating vast, rugged biological fitness landscapes to discover high-value functional patterns-such as optimal protein sequences-is a central challenge in health informatics. However, conventional algorithms often struggle with the exploration-exploitation dilemma, failing to synergize global search with deep local refinement, which leads to entrapment in suboptimal solutions. To overcome this barrier, we introduce Octopus Inspired Optimization (OIO), a novel hierarchical metaheuristic that mimics the octopus's unique neural architecture to intrinsically unify centralized global exploration and parallelized local exploitation. We validated OIO on a real-world protein engineering benchmark, where it surpassed 15 competing metaheuristics. This success is underpinned by OIO's architectural suitability for protein-like landscapes, confirmed by its top ranking on the NK-Landscape benchmark, and its powerful optimization engine, demonstrated by its first-place performance on the gold-standard CEC2022 benchmark. OIO thus provides a robust, nature-inspired computational tool for complex optimization problems in drug discovery and personalized medicine.
Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targets
Venue: arXiv cs.CL 2025
Year: 2025

The rise of online platforms exacerbated the spread of hate speech, demanding scalable and effective detection. However, the accuracy of hate speech detection systems heavily relies on human-labeled data, which is inherently susceptible to biases. While previous work has examined the issue, the interplay between the characteristics of the annotator and those of the target of the hate are still unexplored. We fill this gap by leveraging an extensive dataset with rich socio-demographic information of both annotators and targets, uncovering how human biases manifest in relation to the target's attributes. Our analysis surfaces the presence of widespread biases, which we quantitatively describe and characterize based on their intensity and prevalence, revealing marked differences. Furthermore, we compare human biases with those exhibited by persona-based LLMs. Our findings indicate that while persona-based LLMs do exhibit biases, these differ significantly from those of human annotators. Overall, our work offers new and nuanced results on human biases in hate speech annotations, as well as fresh insights into the design of AI-driven hate speech detection systems.
Insight Over Sight: Exploring the Vision-Knowledge Conflicts in Multimodal LLMs
Venue: arXiv cs.CL 2025
Year: 2025

This paper explores the problem of commonsense level vision-knowledge conflict in Multimodal Large Language Models (MLLMs), where visual information contradicts model's internal commonsense knowledge. To study this issue, we introduce an automated framework, augmented with human-in-the-loop quality control, to generate inputs designed to simulate and evaluate these conflicts in MLLMs. Using this framework, we have crafted a diagnostic benchmark consisting of 374 original images and 1,122 high-quality question-answer (QA) pairs. The benchmark covers two aspects of conflict and three question types, providing a thorough assessment tool. We apply this benchmark to assess the conflict-resolution capabilities of nine representative MLLMs from various model families. Our results indicate an evident over-reliance on parametric knowledge for approximately 20% of all queries, especially among Yes-No and action-related problems. Based on these findings, we evaluate the effectiveness of existing approaches to mitigating the conflicts and compare them to our "Focus-on-Vision" prompting strategy. Despite some improvement, the vision-knowledge conflict remains unresolved and can be further scaled through our data construction framework. Our proposed framework, benchmark, and analysis contribute to the understanding and mitigation of vision-knowledge conflicts in MLLMs.
Complete and bi-continuous invariant of protein backbones under rigid motion
Venue: arXiv cs.CG 2025
Year: 2025

Proteins are large biomolecules that regulate all living organisms and consist of one or several chains. The primary structure of a protein chain is a sequence of amino acid residues whose three main atoms (alpha-carbon, nitrogen, and carbonyl carbon) form a protein backbone. The tertiary structure is the rigid shape of a protein chain represented by atomic positions in 3-dimensional space. Because different geometric structures often have distinct functional properties, it is important to continuously quantify differences in rigid shapes of protein backbones. Unfortunately, many widely used similarities of proteins fail axioms of a distance metric and discontinuously change under tiny perturbations of atoms. This paper develops a complete invariant that identifies any protein backbone in 3-dimensional space, uniquely under rigid motion. This invariant is Lipschitz bi-continuous in the sense that it changes up to a constant multiple of a maximum perturbation of atoms, and vice versa. The new invariant has been used to detect thousands of (near-)duplicates in the Protein Data Bank, whose presence inevitably skews machine learning predictions. The resulting invariant space allows low-dimensional maps with analytically defined coordinates that reveal substantial variability in the protein universe.
Kernel Banzhaf: A Fast and Robust Estimator for Banzhaf Values
Venue: arXiv cs.LG 2025
Year: 2025

Banzhaf values provide a popular, interpretable alternative to the widely-used Shapley values for quantifying the importance of features in machine learning models. Like Shapley values, computing Banzhaf values exactly requires time exponential in the number of features, necessitating the use of efficient estimators. Existing estimators, however, are limited to Monte Carlo sampling methods. In this work, we introduce Kernel Banzhaf, the first regression-based estimator for Banzhaf values. Our approach leverages a novel regression formulation, whose exact solution corresponds to the exact Banzhaf values. Inspired by the success of Kernel SHAP for Shapley values, Kernel Banzhaf efficiently solves a sampled instance of this regression problem. Through empirical evaluations across eight datasets, we find that Kernel Banzhaf significantly outperforms existing Monte Carlo methods in terms of accuracy, sample efficiency, robustness to noise, and feature ranking recovery. Finally, we complement our experimental evaluation with strong theoretical guarantees on Kernel Banzhaf's performance.
Language model developers should report train-test overlap
Venue: arXiv cs.LG 2025
Year: 2025

Language models are extensively evaluated, but correctly interpreting evaluation results requires knowledge of train-test overlap which refers to the extent to which the language model is trained on the very data it is being tested on. The public currently lacks adequate information about train-test overlap: most models have no public train-test overlap statistics, and third parties cannot directly measure train-test overlap since they do not have access to the training data. To make this clear, we document the practices of 30 model developers, finding that just 9 developers report train-test overlap: 4 developers release training data under open-source licenses, enabling the community to directly measure train-test overlap, and 5 developers publish their train-test overlap methodology and statistics. By engaging with language model developers, we provide novel information about train-test overlap for three additional developers. Overall, we take the position that language model developers should publish train-test overlap statistics and/or training data whenever they report evaluation results on public test sets. We hope our work increases transparency into train-test overlap to increase the community-wide trust in model evaluations.
When does additional information lead to longer travel time in multi-origin-destination networks?
Venue: arXiv cs.GT 2025
Year: 2025

The Informational Braess' Paradox (IBP) illustrates a counterintuitive scenario where revelation of additional roadway segments to some self-interested travelers leads to increased travel times for these individuals. IBP extends the original Braess' paradox by relaxing the assumption that all travelers have identical and complete information about the network. In this paper, we study the conditions under which IBP does not occur in networks with non-atomic selfish travelers and multiple origin-destination pairs. Our results completely characterize the network topologies immune to IBP, thus resolving an open question proposed by Acemoglu et al.
Deeper Insights into Deep Graph Convolutional Networks: Stability and Generalization
Venue: arXiv cs.LG 2025
Year: 2025

Graph convolutional networks (GCNs) have emerged as powerful models for graph learning tasks, exhibiting promising performance in various domains. While their empirical success is evident, there is a growing need to understand their essential ability from a theoretical perspective. Existing theoretical research has primarily focused on the analysis of single-layer GCNs, while a comprehensive theoretical exploration of the stability and generalization of deep GCNs remains limited. In this paper, we bridge this gap by delving into the stability and generalization properties of deep GCNs, aiming to provide valuable insights by characterizing rigorously the associated upper bounds. Our theoretical results reveal that the stability and generalization of deep GCNs are influenced by certain key factors, such as the maximum absolute eigenvalue of the graph filter operators and the depth of the network. Our theoretical studies contribute to a deeper understanding of the stability and generalization properties of deep GCNs, potentially paving the way for developing more reliable and well-performing models.
JingZhao: A Framework for Rapid NIC Prototyping in the Domain-Specific-Network Era
Venue: arXiv cs.NI 2025
Year: 2025

The network is becoming domain-specific, which requires on-demand design of the network protocols, as well as the microarchitecture of the NIC. However, to develop such a NIC is not that easy. Since the scissor gap between network speed and the growth of CPU frequency is expanding, most of the protocols need to be offloaded to hardware. The process of designing, verifying and optimizing a domain-specific NIC usually takes great effort, which hinders the rapid iteration of new protocols and algorithms. In this paper, we propose JingZhao, an open-sourced framework for NIC prototyping, which could be leveraged to rapidly implement and verify a domain-specific NIC. Using this framework, we implement a fully-functional RDMA NIC (RNIC). To the best of our knowledge, this represents the first open-source RDMA NIC solution with complete compatibility to the standard OFED communication library. The RNIC was also taped out using TSMC's 28nm process to validate our design. Our evaluation results show that new network functions can be easily integrated into the framework, and achieve nearly line-rate packet processing.
Evaluating the effects of Data Sparsity on the Link-level Bicycling Volume Estimation: A Graph Convolutional Neural Network Approach
Venue: arXiv cs.LG 2025
Year: 2025

Accurate bicycling volume estimation is crucial for making informed decisions and planning about future investments in bicycling infrastructure. However, traditional link-level volume estimation models are effective for motorized traffic but face significant challenges when applied to the bicycling context because of sparse data and the intricate nature of bicycling mobility patterns. To the best of our knowledge, we present the first study to utilize a Graph Convolutional Network (GCN) architecture to model link-level bicycling volumes and systematically investigate the impact of varying levels of data sparsity (0%--99%) on model performance, simulating real-world scenarios. We have leveraged Strava Metro data as the primary source of bicycling counts across 15,933 road segments/links in the City of Melbourne, Australia. To evaluate the effectiveness of the GCN model, we benchmark it against traditional machine learning models, such as linear regression, support vector machines, and random forest. Our results show that the GCN model outperforms these traditional models in predicting Annual Average Daily Bicycle (AADB) counts, demonstrating its ability to capture the spatial dependencies inherent in bicycle traffic networks. While GCN remains robust up to 80% sparsity, its performance declines sharply beyond this threshold, highlighting the challenges of extreme data sparsity. These findings underscore the potential of GCNs in enhancing bicycling volume estimation, while also emphasizing the need for further research on methods to improve model resilience under high-sparsity conditions. Our findings offer valuable insights for city planners aiming to improve bicycling infrastructure and promote sustainable transportation.
Emergent social conventions and collective bias in LLM populations
Venue: arXiv cs.MA 2025
Year: 2025

Social conventions are the backbone of social coordination, shaping how individuals form a group. As growing populations of artificial intelligence (AI) agents communicate through natural language, a fundamental question is whether they can bootstrap the foundations of a society. Here, we present experimental results that demonstrate the spontaneous emergence of universally adopted social conventions in decentralized populations of large language model (LLM) agents. We then show how strong collective biases can emerge during this process, even when agents exhibit no bias individually. Last, we examine how committed minority groups of adversarial LLM agents can drive social change by imposing alternative social conventions on the larger population. Our results show that AI systems can autonomously develop social conventions without explicit programming and have implications for designing AI systems that align, and remain aligned, with human values and societal goals.
From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impacts
Venue: arXiv cs.HC 2025
Year: 2025

With advances in generative AI, there is increasing work towards creating autonomous agents that can manage daily tasks by operating user interfaces (UIs). While prior research has studied the mechanics of how AI agents might navigate UIs and understand UI structure, the effects of agents and their autonomous actions-particularly those that may be risky or irreversible-remain under-explored. In this work, we investigate the real-world impacts and consequences of mobile UI actions taken by AI agents. We began by developing a taxonomy of the impacts of mobile UI actions through a series of workshops with domain experts. Following this, we conducted a data synthesis study to gather realistic mobile UI screen traces and action data that users perceive as impactful. We then used our impact categories to annotate our collected data and data repurposed from existing mobile UI navigation datasets. Our quantitative evaluations of different large language models (LLMs) and variants demonstrate how well different LLMs can understand the impacts of mobile UI actions that might be taken by an agent. We show that our taxonomy enhances the reasoning capabilities of these LLMs for understanding the impacts of mobile UI actions, but our findings also reveal significant gaps in their ability to reliably classify more nuanced or complex categories of impact.
MedMobile: A mobile-sized language model with clinical capabilities
Venue: arXiv cs.CL 2025
Year: 2025

Language models (LMs) have demonstrated expert-level reasoning and recall abilities in medicine. However, computational costs and privacy concerns are mounting barriers to wide-scale implementation. To address these significant limitations, we introduce a parsimonious adaptation of phi-3-mini, MedMobile, a 3.8 billion parameter LM capable of running on a mobile device, for medical applications. We perform a careful set of pipeline additions and demonstrate that chain of thought, ensembling, and fine-tuning lead to the greatest performance gains, while unexpectedly retrieval augmented generation fails to demonstrate significant improvements. We evaluate the efficiency of our pipeline on the MultiMedQA and MedBullets. We demonstrate that MedMobile scores 75.7% on the MedQA (USMLE), surpassing the passing mark for licensed physicians (~60%) and rivaling scores of models 100 times its size. Across the entirety of the MultiMedQA, MedMobile achieves SOTA performance for models with less than 5B parameters and represents the smallest model to pass the MedQA (USMLE). MedMobile holds promise to democratize access to language models in medicine, bolstering lower compute needs and fast inference speeds. With the ability to combat the biggest barriers to entry for language models in medicine, we hope that MedMobile is a critical step forward in developing clinically relevant language models.
MFIT: Multi-Fidelity Thermal Modeling for 2.5D and 3D Multi-Chiplet Architectures
Venue: arXiv cs.AR 2025
Year: 2025

Rapidly evolving artificial intelligence and machine learning applications require ever-increasing computational capabilities, while monolithic 2D design technologies approach their limits. Heterogeneous integration of smaller chiplets using a 2.5D silicon interposer and 3D packaging has emerged as a promising paradigm to address this limit and meet performance demands. These approaches offer a significant cost reduction and higher manufacturing yield than monolithic 2D integrated circuits. However, the compact arrangement and high compute density exacerbate the thermal management challenges, potentially compromising performance. Addressing these thermal modeling challenges is critical, especially as system sizes grow and different design stages require varying levels of accuracy and speed. Since no single thermal modeling technique meets all these needs, this paper introduces MFIT, a range of multi-fidelity thermal models that effectively balance accuracy and speed. These multi-fidelity models can enable efficient design space exploration and runtime thermal management. Our extensive testing on systems with 16, 36, and 64 2.5D integrated chiplets and 16x3 3D integrated chiplets demonstrates that these models can reduce execution times from days to mere seconds and milliseconds with negligible loss in accuracy.
The Condorcet Dimension of Metric Spaces
Venue: arXiv cs.GT 2025
Year: 2025

A Condorcet winning set is a set of candidates such that no other candidate is preferred by at least half the voters over all members of the set. The Condorcet dimension, which is the minimum cardinality of a Condorcet winning set, is known to be at most logarithmic in the number of candidates. We study the case of elections where voters and candidates are located in a $2$-dimensional space with preferences based upon proximity voting. Our main result is that the Condorcet dimension is at most $3$, under both the Manhattan norm and the infinity norm, natural measures in electoral systems.