OpenRaiser/Intern-Atlas huggingface dataset:
The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understanding
Venue: arXiv cs.CL 2025
Year: 2025

In a systematic way, we investigate a widely asked question: Do LLMs really understand what they say?, which relates to the more familiar term Stochastic Parrot. To this end, we propose a summative assessment over a carefully designed physical concept understanding task, PhysiCo. Our task alleviates the memorization issue via the usage of grid-format inputs that abstractly describe physical phenomena. The grids represents varying levels of understanding, from the core phenomenon, application examples to analogies to other abstract patterns in the grid world. A comprehensive study on our task demonstrates: (1) state-of-the-art LLMs, including GPT-4o, o1 and Gemini 2.0 flash thinking, lag behind humans by ~40%; (2) the stochastic parrot phenomenon is present in LLMs, as they fail on our grid task but can describe and recognize the same concepts well in natural language; (3) our task challenges the LLMs due to intrinsic difficulties rather than the unfamiliar grid format, as in-context learning and fine-tuning on same formatted data added little to their performance.
DICE: Device-level Integrated Circuits Encoder with Graph Contrastive Pretraining
Venue: arXiv cs.LG 2025
Year: 2025

Pretraining models with unsupervised graph representation learning has led to significant advancements in domains such as social network analysis, molecular design, and electronic design automation (EDA). However, prior work in EDA has mainly focused on pretraining models for digital circuits, overlooking analog and mixed-signal circuits. To bridge this gap, we introduce DICE, a Device-level Integrated Circuits Encoder, which is the first graph neural network (GNN) pretrained via self-supervised learning specifically tailored for graph-level prediction tasks in both analog and digital circuits. DICE adopts a simulation-free pretraining approach based on graph contrastive learning, leveraging two novel graph augmentation techniques. Experimental results demonstrate substantial performance improvements across three downstream tasks, highlighting the effectiveness of DICE for both analog and digital circuits. The code is available at github.com/brianlsy98/DICE.
Modeling Time-evolving Causality over Data Streams
Venue: arXiv cs.LG 2025
Year: 2025

Given an extensive, semi-infinite collection of multivariate coevolving data sequences (e.g., sensor/web activity streams) whose observations influence each other, how can we discover the time-changing cause-and-effect relationships in co-evolving data streams? How efficiently can we reveal dynamical patterns that allow us to forecast future values? In this paper, we present a novel streaming method, ModePlait, which is designed for modeling such causal relationships (i.e., time-evolving causality) in multivariate co-evolving data streams and forecasting their future values. The solution relies on characteristics of the causal relationships that evolve over time in accordance with the dynamic changes of exogenous variables. ModePlait has the following properties: (a) Effective: it discovers the time-evolving causality in multivariate co-evolving data streams by detecting the transitions of distinct dynamical patterns adaptively. (b) Accurate: it enables both the discovery of time-evolving causality and the forecasting of future values in a streaming fashion. (c) Scalable: our algorithm does not depend on data stream length and thus is applicable to very large sequences. Extensive experiments on both synthetic and real-world datasets demonstrate that our proposed model outperforms state-of-the-art methods in terms of discovering the time-evolving causality as well as forecasting.
A Decade of Metric Differential Privacy: Advancements and Applications
Venue: arXiv cs.CR 2025
Year: 2025

Metric Differential Privacy (mDP) builds upon the core principles of Differential Privacy (DP) by incorporating various distance metrics, which offer adaptable and context-sensitive privacy guarantees for a wide range of applications, such as location-based services, text analysis, and image processing. Since its inception in 2013, mDP has garnered substantial research attention, advancing theoretical foundations, algorithm design, and practical implementations. Despite this progress, existing surveys mainly focus on traditional DP and local DP, and they provide limited coverage of mDP. This paper provides a comprehensive survey of mDP research from 2013 to 2024, tracing its development from the foundations of DP. We categorize essential mechanisms, including Laplace, Exponential, and optimization-based approaches, and assess their strengths, limitations, and application domains. Additionally, we highlight key challenges and outline future research directions to encourage innovation and real-world adoption of mDP. This survey is designed to be a valuable resource for researchers and practitioners aiming to deepen their understanding and drive progress in mDP within the broader privacy ecosystem.
Text-driven 3D Human Generation via Contrastive Preference Optimization
Venue: arXiv cs.CV 2025
Year: 2025

Recent advances in Score Distillation Sampling (SDS) have improved 3D human generation from textual descriptions. However, existing methods still face challenges in accurately aligning 3D models with long and complex textual inputs. To address this challenge, we propose a novel framework that introduces contrastive preferences, where human-level preference models, guided by both positive and negative prompts, assist SDS for improved alignment. Specifically, we design a preference optimization module that integrates multiple models to comprehensively capture the full range of textual features. Furthermore, we introduce a negation preference module to mitigate over-optimization of irrelevant details by leveraging static-dynamic negation prompts, effectively preventing ``reward hacking". Extensive experiments demonstrate that our method achieves state-of-the-art results, significantly enhancing texture realism and visual alignment with textual descriptions, particularly for long and complex inputs.
Beyond Shallow Behavior: Task-Efficient Value-Based Multi-Task Offline MARL via Skill Discovery
Venue: arXiv cs.LG 2025
Year: 2025

As a data-driven approach, offline MARL learns superior policies solely from offline datasets, ideal for domains rich in historical data but with high interaction costs and risks. However, most existing methods are task-specific, requiring retraining for new tasks, leading to redundancy and inefficiency. To address this issue, we propose a task-efficient value-based multi-task offline MARL algorithm, Skill-Discovery Conservative Q-Learning (SD-CQL). Unlike existing methods decoding actions from skills via behavior cloning, SD-CQL discovers skills in a latent space by reconstructing the next observation, evaluates fixed and variable actions separately, and uses conservative Q-learning with local value calibration to select the optimal action for each skill. It eliminates the need for local-global alignment and enables strong multi-task generalization from limited, small-scale source tasks. Substantial experiments on StarCraft II demonstrate the superior generalization performance and task-efficiency of SD-CQL. It achieves the best performance on $\textbf{13}$ out of $14$ task sets, with up to $\textbf{68.9%}$ improvement on individual task sets.
Task Generalization With AutoRegressive Compositional Structure: Can Learning From $D$ Tasks Generalize to $D^{T}$ Tasks?
Venue: arXiv cs.LG 2025
Year: 2025

Large language models (LLMs) exhibit remarkable task generalization, solving tasks they were never explicitly trained on with only a few demonstrations. This raises a fundamental question: When can learning from a small set of tasks generalize to a large task family? In this paper, we investigate task generalization through the lens of autoregressive compositional structure, where each task is a composition of $T$ operations, and each operation is among a finite family of $D$ subtasks. This yields a total class of size $D^T$. We first show that generalization to all $D^T$ tasks is theoretically achievable by training on only $\widetilde{O}(D)$ tasks. Empirically, we demonstrate that Transformers achieve such exponential task generalization on sparse parity functions via In-context Learning (ICL) and chain-of-thought (CoT) reasoning. We further show generalization in arithmetic and translation, beyond parity functions.
RED: Energy Optimization Framework for eDRAM-based PIM with Reconfigurable Voltage Swing and Retention-aware Scheduling
Venue: arXiv cs.AR 2025
Year: 2025

In the era of artificial intelligence (AI), Transformer demonstrates its performance across various applications. The excessive amount of parameters incurs high latency and energy overhead when processed in the von Neumann architecture. Processing-in-memory (PIM) has shown the potential in accelerating data-intensive applications by reducing data movement. While previous works mainly optimize the computational part of PIM to enhance energy efficiency, the importance of memory design, which consumes the most power in PIM, has been rather neglected. In this work, we present RED, an energy optimization framework for eDRAM-based PIM. We first analyze the PIM operations in eDRAM, obtaining two key observations: 1) memory access energy consumption is predominant in PIM, and 2) read bitline (RBL) voltage swing, sense amplifier power, and retention time are in trade-off relations. Leveraging them, we propose a novel reconfigurable eDRAM and retention-aware scheduling that minimizes the runtime energy consumption of the eDRAM macro. The framework pinpoints the optimal operating point by pre-estimating energy consumption across all possible tiling schemes and memory operations. Then, the reconfigurable eDRAM controls the RBL voltage swing at runtime according to the scheduling, optimizing the memory access power. Moreover, RED employs refresh skipping and sense amplifier power gating to mitigate the energy consumption overhead coming from the trade-off relation. Finally, the RED framework achieves up to 3.05x higher energy efficiency than the prior SRAM-based PIM, reducing the energy consumption of eDRAM macro up to 74.88% with reconfigurable eDRAM and optimization schemes, requiring only 3.5% area and 0.77% energy overhead for scheduling.
Diversity Enhances an LLM's Performance in RAG and Long-context Task
Venue: arXiv cs.CL 2025
Year: 2025

The rapid advancements in large language models (LLMs) have highlighted the challenge of context window limitations, primarily due to the quadratic time complexity of the self-attention mechanism (\(O(N^2)\), where \(N\) denotes the context window length). This constraint impacts tasks such as retrieval-augmented generation (RAG) in question answering (Q\&A) and long context summarization. A common approach involves selecting content with the highest similarity to the query; however, this often leads to redundancy and the exclusion of diverse yet relevant information. Building on principles from Maximal Marginal Relevance (MMR) and Farthest Point Sampling (FPS), we integrate diversity into the content selection process. Our findings reveal that incorporating diversity substantially increases the recall of selecting relevant sentences or chunks before LLM-based Q\&A and summarization. These results highlight the importance of maintaining diversity in future LLM applications to further improve summarization and Q\&A outcomes.
Zero-shot Concept Bottleneck Models
Venue: arXiv cs.LG 2025
Year: 2025

Concept bottleneck models (CBMs) are inherently interpretable and intervenable neural network models, which explain their final label prediction by the intermediate prediction of high-level semantic concepts. However, they require target task training to learn input-to-concept and concept-to-label mappings, incurring target dataset collections and training resources. In this paper, we present \textit{zero-shot concept bottleneck models} (Z-CBMs), which predict concepts and labels in a fully zero-shot manner without training neural networks. Z-CBMs utilize a large-scale concept bank, which is composed of millions of vocabulary extracted from the web, to describe arbitrary input in various domains. For the input-to-concept mapping, we introduce concept retrieval, which dynamically finds input-related concepts by the cross-modal search on the concept bank. In the concept-to-label inference, we apply concept regression to select essential concepts from the retrieved concepts by sparse linear regression. Through extensive experiments, we confirm that our Z-CBMs provide interpretable and intervenable concepts without any additional training. Code will be available at https://github.com/yshinya6/zcbm.
MTDP: A Modulated Transformer based Diffusion Policy Model
Venue: arXiv cs.RO 2025
Year: 2025

Recent research on robot manipulation based on Behavior Cloning (BC) has made significant progress. By combining diffusion models with BC, diffusion policiy has been proposed, enabling robots to quickly learn manipulation tasks with high success rates. However, integrating diffusion policy with high-capacity Transformer presents challenges, traditional Transformer architectures struggle to effectively integrate guiding conditions, resulting in poor performance in manipulation tasks when using Transformer-based models. In this paper, we investigate key architectural designs of Transformers and improve the traditional Transformer architecture by proposing the Modulated Transformer Diffusion Policy (MTDP) model for diffusion policy. The core of this model is the Modulated Attention module we proposed, which more effectively integrates the guiding conditions with the main input, improving the generative model's output quality and, consequently, increasing the robot's task success rate. In six experimental tasks, MTDP outperformed existing Transformer model architectures, particularly in the Toolhang experiment, where the success rate increased by 12\%. To verify the generality of Modulated Attention, we applied it to the UNet architecture to construct Modulated UNet Diffusion Policy model (MUDP), which also achieved higher success rates than existing UNet architectures across all six experiments. The Diffusion Policy uses Denoising Diffusion Probabilistic Models (DDPM) as the diffusion model. Building on this, we also explored Denoising Diffusion Implicit Models (DDIM) as the diffusion model, constructing the MTDP-I and MUDP-I model, which nearly doubled the generation speed while maintaining performance.
WhatsApp as an improvisation of health information systems in Southern African public hospitals: A socio-technical perspective
Venue: arXiv cs.CY 2025
Year: 2025

Digital health interventions, particularly electronic referrals (e-referrals) and health information systems, have revolutionised clinical workflows in public hospitals by automating processes. However, the utilization of e-referrals has yielded mixed outcomes, with varying levels of success in organisational processes.This paper explores improvisation of health information systems in Southern African public hospitals from a socio-technical perspective. In particular the paper explains the design-reality gaps giving rise to improvisations of mandated health information systems in order to understand their occurrence and impact on referral outcomes. We employed the design-reality framework and the Process framework for Healthcare Information System Workarounds and Impacts to explain the socio-technical issues related to the phenomenon of interest.We conducted semi-interviews with 31 respondents from health organisations as case studies.Respondents from two public hospitals in South Africa and two in Namibia were interviewed to examine how they devised improvisations to various health information systems in each setting.The findings showed that using WhatsApp or improvising existing health information systems (HIS) improved efficiency and productivity of healthcare practitioners (HCPs) referral activities. Additionally, HCPs reported positive outcomes related to continual professional development in the given settings.The findings further show a relationship between design-reality gaps and improvisations enacted by HCPs.The observed gaps are related to poor management systems and structures lack of HCPs' involvement in the roll-out of HIS and inadequacies of existing HIS to support referral tasks.These study findings can be insightful and useful to system developers and other stakeholders for devising measures to address the gaps.
Enhancing RAG with Active Learning on Conversation Records: Reject Incapables and Answer Capables
Venue: arXiv cs.CL 2025
Year: 2025

Retrieval-augmented generation (RAG) is a key technique for leveraging external knowledge and reducing hallucinations in large language models (LLMs). However, RAG still struggles to fully prevent hallucinated responses. To address this, it is essential to identify samples prone to hallucination or guide LLMs toward correct responses, which experts then annotate to develop high-quality datasets for refining LLMs. However, the growing scarcity of such datasets makes their creation challenging. This paper proposes using the vast amount of conversations from widespread LLM usage to build these datasets, training LLMs to avoid hallucination-prone questions while accurately responding to manageable ones. Given the impracticality of expert-annotating all conversation records, the paper introduces AL4RAG, which uses active learning to select the most suitable conversation samples for annotation, optimizing performance within an annotation budget. Additionally, recognizing that traditional active learning methods are not fully compatible with RAG due to unsuitable distance metrics, we develop a novel sample distance measurement for RAG active learning. Extensive experiments show that our method consistently outperforms baselines across multiple metrics.
Application of Tabular Transformer Architectures for Operating System Fingerprinting
Venue: arXiv cs.CR 2025
Year: 2025

Operating System (OS) fingerprinting is essential for network management and cybersecurity, enabling accurate device identification based on network traffic analysis. Traditional rule-based tools such as Nmap and p0f face challenges in dynamic environments due to frequent OS updates and obfuscation techniques. While Machine Learning (ML) approaches have been explored, Deep Learning (DL) models, particularly Transformer architectures, remain unexploited in this domain. This study investigates the application of Tabular Transformer architectures-specifically TabTransformer and FT-Transformer-for OS fingerprinting, leveraging structured network data from three publicly available datasets. Our experiments demonstrate that FT-Transformer generally outperforms traditional ML models, previous approaches and TabTransformer across multiple classification levels (OS family, major, and minor versions). The results establish a strong foundation for DL-based OS fingerprinting, improving accuracy and adaptability in complex network environments. Furthermore, we ensure the reproducibility of our research by providing an open-source implementation.
Unsupervised Anomaly Detection on Implicit Shape representations for Sarcopenia Detection
Venue: arXiv cs.CV 2025
Year: 2025

Sarcopenia is an age-related progressive loss of muscle mass and strength that significantly impacts daily life. A commonly studied criterion for characterizing the muscle mass has been the combination of 3D imaging and manual segmentations. In this paper, we instead study the muscles' shape. We rely on an implicit neural representation (INR) to model normal muscle shapes. We then introduce an unsupervised anomaly detection method to identify sarcopenic muscles based on the reconstruction error of the implicit model. Relying on a conditional INR with an auto-decoding strategy, we also learn a latent representation of the muscles that clearly separates normal from abnormal muscles in an unsupervised fashion. Experimental results on a dataset of 103 segmented volumes indicate that our double anomaly detection strategy effectively discriminates sarcopenic and non-sarcopenic muscles.
Incremental Approximate Maximum Flow via Residual Graph Sparsification
Venue: arXiv cs.DS 2025
Year: 2025

We give an algorithm that, with high probability, maintains a $(1-ε)$-approximate $s$-$t$ maximum flow in undirected, uncapacitated $n$-vertex graphs undergoing $m$ edge insertions in $\tilde{O}(m+ n F^*/ε)$ total update time, where $F^{*}$ is the maximum flow on the final graph. This is the first algorithm to achieve polylogarithmic amortized update time for dense graphs ($m = Ω(n^2)$), and more generally, for graphs where $F^*= \tilde{O}(m/n)$. At the heart of our incremental algorithm is the residual graph sparsification technique of Karger and Levine [SICOMP '15], originally designed for computing exact maximum flows in the static setting. Our main contributions are (i) showing how to maintain such sparsifiers for approximate maximum flows in the incremental setting and (ii) generalizing the cut sparsification framework of Fung et al. [SICOMP '19] from undirected graphs to balanced directed graphs.
A Novel Dialect-Aware Framework for the Classification of Arabic Dialects and Emotions
Venue: arXiv cs.CL 2025
Year: 2025

Arabic is one of the oldest languages still in use today. As a result, several Arabic-speaking regions have developed dialects that are unique to them. Dialect and emotion recognition have various uses in Arabic text analysis, such as determining an online customer's origin based on their comments. Furthermore, intelligent chatbots that are aware of a user's emotions can respond appropriately to the user. Current research in emotion detection in the Arabic language lacks awareness of how emotions are exhibited in different dialects, which motivates the work found in this study. This research addresses the problems of dialect and emotion classification in Arabic. Specifically, this is achieved by building a novel framework that can identify and predict Arabic dialects and emotions from a given text. The framework consists of three modules: A text-preprocessing module, a classification module, and a clustering module with the novel capability of building new dialect-aware emotion lexicons. The proposed framework generated a new emotional lexicon for different dialects. It achieved an accuracy of 88.9% in classifying Arabic dialects, which outperforms the state-of-the-art results by 6.45 percentage points. Furthermore, the framework achieved 89.1-79% accuracy in detecting emotions in the Egyptian and Gulf dialects, respectively.